@thinkingai/ae-cli 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +229 -0
- package/bin/ae-cli.js +2 -0
- package/dist/auth-JXELJNDS.js +19 -0
- package/dist/auth-L3G3A74I.js +67 -0
- package/dist/chunk-3GM5LJZR.js +145 -0
- package/dist/chunk-JD5DXMKL.js +172 -0
- package/dist/chunk-KM57HI5B.js +107 -0
- package/dist/chunk-OVQOK35G.js +108 -0
- package/dist/chunk-RJNLN3KQ.js +217 -0
- package/dist/client-5QSQBBDB.js +16 -0
- package/dist/community-GMALVSRF.js +684 -0
- package/dist/config-Q7J3Z7Z2.js +242 -0
- package/dist/index.js +277 -0
- package/dist/raw-ZNVURYMR.js +59 -0
- package/dist/te-analysis-7NN3SY3I.js +871 -0
- package/dist/te-audience-P43BWMHM.js +350 -0
- package/dist/te-common-H3IPTI2I.js +144 -0
- package/dist/te-dataops-WCGDBJLF.js +1853 -0
- package/dist/te-engage-UTRVNTGU.js +1470 -0
- package/dist/te-meta-HGYNXVGF.js +479 -0
- package/package.json +53 -0
- package/skills/te-analysis/SKILL.md +107 -0
- package/skills/te-analysis/references/build-entity-details-sql.md +58 -0
- package/skills/te-analysis/references/build-event-details-sql.md +61 -0
- package/skills/te-analysis/references/create-alert.md +47 -0
- package/skills/te-analysis/references/create-dashboard.md +49 -0
- package/skills/te-analysis/references/create-or-update-dashboard-note.md +39 -0
- package/skills/te-analysis/references/create-public-access-link.md +43 -0
- package/skills/te-analysis/references/create-report.md +60 -0
- package/skills/te-analysis/references/create-result-cluster.md +73 -0
- package/skills/te-analysis/references/drilldown-user-events.md +72 -0
- package/skills/te-analysis/references/drilldown-users.md +75 -0
- package/skills/te-analysis/references/get-alert-definition-schema.md +32 -0
- package/skills/te-analysis/references/get-alert.md +31 -0
- package/skills/te-analysis/references/get-analysis-query-schema.md +36 -0
- package/skills/te-analysis/references/get-filter-schema.md +33 -0
- package/skills/te-analysis/references/get-groupby-schema.md +33 -0
- package/skills/te-analysis/references/get-report-definition.md +33 -0
- package/skills/te-analysis/references/get-table-columns.md +38 -0
- package/skills/te-analysis/references/list-alerts.md +33 -0
- package/skills/te-analysis/references/list-dashboards.md +33 -0
- package/skills/te-analysis/references/list-public-access-links.md +31 -0
- package/skills/te-analysis/references/list-reports.md +33 -0
- package/skills/te-analysis/references/load-filters.md +44 -0
- package/skills/te-analysis/references/query-adhoc.md +71 -0
- package/skills/te-analysis/references/query-dashboard-detail.md +35 -0
- package/skills/te-analysis/references/query-dashboard-report-data.md +60 -0
- package/skills/te-analysis/references/query-entity-details.md +60 -0
- package/skills/te-analysis/references/query-event-details.md +63 -0
- package/skills/te-analysis/references/query-report-data.md +65 -0
- package/skills/te-analysis/references/update-alert.md +49 -0
- package/skills/te-analysis/references/update-dashboard.md +45 -0
- package/skills/te-analysis/references/update-public-access-link.md +41 -0
- package/skills/te-audience/SKILL.md +63 -0
- package/skills/te-audience/references/create-cluster.md +65 -0
- package/skills/te-audience/references/create-tag.md +60 -0
- package/skills/te-audience/references/get-cluster-definition-schema.md +37 -0
- package/skills/te-audience/references/get-clusters-by-name.md +32 -0
- package/skills/te-audience/references/get-tag-definition-schema.md +32 -0
- package/skills/te-audience/references/get-tags-by-name.md +32 -0
- package/skills/te-audience/references/list-cluster-members.md +35 -0
- package/skills/te-audience/references/list-clusters.md +32 -0
- package/skills/te-audience/references/list-tag-members.md +36 -0
- package/skills/te-audience/references/list-tags.md +32 -0
- package/skills/te-audience/references/refresh-cluster.md +31 -0
- package/skills/te-audience/references/refresh-tag.md +31 -0
- package/skills/te-audience/references/update-cluster.md +59 -0
- package/skills/te-audience/references/update-tag.md +60 -0
- package/skills/te-common/SKILL.md +135 -0
- package/skills/te-common/references/get-resource-url.md +32 -0
- package/skills/te-common/references/list-projects.md +37 -0
- package/skills/te-community/SKILL.md +133 -0
- package/skills/te-community/references/community-activity-analysis.md +68 -0
- package/skills/te-community/references/community-analyzing-official-content.md +72 -0
- package/skills/te-community/references/community-analyzing-theme-comment.md +124 -0
- package/skills/te-community/references/community-character-analysis.md +71 -0
- package/skills/te-community/references/community-daily-report.md +82 -0
- package/skills/te-community/references/community-hottopic-insight.md +120 -0
- package/skills/te-community/references/community-release-analysis.md +79 -0
- package/skills/te-community/references/community-weekly-report.md +84 -0
- package/skills/te-community/references/get_channel_info.md +21 -0
- package/skills/te-community/references/get_comment_tag_analysis.md +30 -0
- package/skills/te-community/references/get_comments_summary.md +33 -0
- package/skills/te-community/references/get_corpus_tags.md +21 -0
- package/skills/te-community/references/get_daily_summary.md +23 -0
- package/skills/te-community/references/get_hot_topics.md +26 -0
- package/skills/te-community/references/get_livestream_analysis.md +23 -0
- package/skills/te-community/references/get_livestream_detail.md +34 -0
- package/skills/te-community/references/get_livestream_list.md +26 -0
- package/skills/te-community/references/get_livestream_overview.md +25 -0
- package/skills/te-community/references/get_livestream_room_metrics.md +26 -0
- package/skills/te-community/references/get_livestream_rooms.md +26 -0
- package/skills/te-community/references/get_overview_metrics.md +26 -0
- package/skills/te-community/references/get_post_detail.md +38 -0
- package/skills/te-community/references/get_risk_content.md +43 -0
- package/skills/te-community/references/get_sentiment_overview.md +34 -0
- package/skills/te-community/references/get_tag_trends.md +28 -0
- package/skills/te-community/references/get_topic_detail.md +27 -0
- package/skills/te-community/references/search_posts.md +48 -0
- package/skills/te-dataops/SKILL.md +645 -0
- package/skills/te-dataops/references/dataops-flow-create.md +163 -0
- package/skills/te-dataops/references/dataops-flow-monitor.md +118 -0
- package/skills/te-dataops/references/dataops-integration.md +246 -0
- package/skills/te-dataops/references/dataops-operations.md +107 -0
- package/skills/te-dataops/references/dataops-query.md +111 -0
- package/skills/te-dataops/references/dataops-table.md +122 -0
- package/skills/te-engage/SKILL.md +211 -0
- package/skills/te-engage/references/add-approver.md +24 -0
- package/skills/te-engage/references/add-channel.md +74 -0
- package/skills/te-engage/references/approver-list.md +19 -0
- package/skills/te-engage/references/cancel-query-by-request-id.md +23 -0
- package/skills/te-engage/references/channel-detail.md +52 -0
- package/skills/te-engage/references/channel-list.md +41 -0
- package/skills/te-engage/references/config-channel-detail.md +32 -0
- package/skills/te-engage/references/config-channel-list.md +27 -0
- package/skills/te-engage/references/config-item-analysis-report.md +34 -0
- package/skills/te-engage/references/config-item-detail.md +30 -0
- package/skills/te-engage/references/config-item-list.md +19 -0
- package/skills/te-engage/references/config-item-strategy-comparison.md +25 -0
- package/skills/te-engage/references/config-item-trigger-report.md +42 -0
- package/skills/te-engage/references/copy-config-template.md +29 -0
- package/skills/te-engage/references/delete-channel.md +24 -0
- package/skills/te-engage/references/delete-config-channel.md +24 -0
- package/skills/te-engage/references/delete-config-item.md +25 -0
- package/skills/te-engage/references/delete-flow.md +24 -0
- package/skills/te-engage/references/flow-ab-split-node-report.md +54 -0
- package/skills/te-engage/references/flow-detail.md +79 -0
- package/skills/te-engage/references/flow-list.md +19 -0
- package/skills/te-engage/references/flow-node-config-schema.md +44 -0
- package/skills/te-engage/references/flow-node-detail-report.md +55 -0
- package/skills/te-engage/references/flow-node-overview-report.md +52 -0
- package/skills/te-engage/references/flow-process-report.md +59 -0
- package/skills/te-engage/references/manage-flow.md +94 -0
- package/skills/te-engage/references/manage-strategy.md +74 -0
- package/skills/te-engage/references/manage-task.md +37 -0
- package/skills/te-engage/references/modify-flow-base-info.md +27 -0
- package/skills/te-engage/references/save-flow.md +870 -0
- package/skills/te-engage/references/strategy-detail.md +73 -0
- package/skills/te-engage/references/strategy-list.md +21 -0
- package/skills/te-engage/references/task-data-detail.md +81 -0
- package/skills/te-engage/references/task-data-overview.md +48 -0
- package/skills/te-engage/references/task-detail.md +61 -0
- package/skills/te-engage/references/task-experiment-report.md +51 -0
- package/skills/te-engage/references/task-list.md +91 -0
- package/skills/te-engage/references/task-metric-detail.md +56 -0
- package/skills/te-engage/references/task-stats.md +89 -0
- package/skills/te-engage/references/update-channel-status.md +32 -0
- package/skills/te-engage/references/update-config-channel-status.md +32 -0
- package/skills/te-engage/references/validate-flow-node-config.md +52 -0
- package/skills/te-engage/references/whitelist-list.md +19 -0
- package/skills/te-meta/SKILL.md +71 -0
- package/skills/te-meta/references/batch-create-metadata.md +37 -0
- package/skills/te-meta/references/batch-edit-metadata.md +36 -0
- package/skills/te-meta/references/create-metric.md +62 -0
- package/skills/te-meta/references/create-project-mark-time.md +38 -0
- package/skills/te-meta/references/create-virtual-event.md +64 -0
- package/skills/te-meta/references/create-virtual-property.md +69 -0
- package/skills/te-meta/references/delete-project-mark-times.md +34 -0
- package/skills/te-meta/references/delete-track-items.md +35 -0
- package/skills/te-meta/references/get-metric.md +32 -0
- package/skills/te-meta/references/get-project-config.md +32 -0
- package/skills/te-meta/references/get-track-program.md +31 -0
- package/skills/te-meta/references/list-entities.md +34 -0
- package/skills/te-meta/references/list-events.md +33 -0
- package/skills/te-meta/references/list-metrics.md +33 -0
- package/skills/te-meta/references/list-project-mark-times.md +33 -0
- package/skills/te-meta/references/list-project-users.md +32 -0
- package/skills/te-meta/references/list-properties.md +35 -0
- package/skills/te-meta/references/save-track-items.md +34 -0
- package/skills/te-meta/references/update-metric.md +61 -0
- package/skills/te-meta/references/update-project-mark-time.md +39 -0
- package/skills/te-shared/SKILL.md +115 -0
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: dataops-flow-create
|
|
3
|
+
version: 1.0.0
|
|
4
|
+
description: "任务流创建与配置:创建任务流、添加任务节点、配置依赖关系、保存任务定义、配置调度、测试、发布到生产。触发关键词:创建任务流、新建工作流、配置调度、添加任务节点、发布、cron、定时执行。"
|
|
5
|
+
metadata:
|
|
6
|
+
requires:
|
|
7
|
+
bins: ["te-cli"]
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# DataOps 任务流创建与配置
|
|
11
|
+
|
|
12
|
+
> **前置条件:** 阅读 [`te-dataops/SKILL.md`](../SKILL.md) 了解通用规则。
|
|
13
|
+
|
|
14
|
+
使用 `dataops_flow` 子命令管理任务流生命周期。
|
|
15
|
+
|
|
16
|
+
**任务流生命周期:创建 → 配置节点 → 配置调度 → DEV 环境测试 → 发布到 PROD → 上线调度**
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## 完整任务流创建流程
|
|
21
|
+
|
|
22
|
+
按顺序执行以下步骤,从零创建一个可上线的任务流。
|
|
23
|
+
|
|
24
|
+
### Step 1: 创建任务流
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
te-cli dataops_flow +create_flow --spaceCode "${spaceCode}" \
|
|
28
|
+
--flowName "每日ETL流程" --remark "处理用户数据" --confirmed true
|
|
29
|
+
# 返回 flowCode,后续步骤都需要
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
### Step 2: 创建任务节点(可多次调用)
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
# 创建 SQL 任务
|
|
36
|
+
te-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
|
|
37
|
+
--flowCode "${flowCode}" --taskName "处理用户数据" --taskType "TRINO_SQL" --confirmed true
|
|
38
|
+
# 返回 taskCode
|
|
39
|
+
|
|
40
|
+
# 创建带前置依赖的任务
|
|
41
|
+
te-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
|
|
42
|
+
--flowCode "${flowCode}" --taskName "导出结果" --taskType "TRINO_SQL" \
|
|
43
|
+
--preTaskCode "${upstreamTaskCode}" --confirmed true
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
**任务节点类型:**
|
|
47
|
+
|
|
48
|
+
| 类型 | 说明 | 典型用途 |
|
|
49
|
+
|------|------|---------|
|
|
50
|
+
| TRINO_SQL | Trino SQL 查询 | 数据加工、ETL |
|
|
51
|
+
| SHELL | Shell 脚本 | 系统命令、文件操作 |
|
|
52
|
+
| FLOW_CHECK | 流程检查 | 检查上游任务流是否完成 |
|
|
53
|
+
| TASK_CHECK | 任务检查 | 检查指定任务是否完成 |
|
|
54
|
+
| OFFLINE_SYNC | 离线同步 | 集成方案数据同步(需 syncId) |
|
|
55
|
+
| APP_SYNC | 应用同步 | 应用表数据同步(需 syncId) |
|
|
56
|
+
| PLACE_HOLDER | 占位符 | 流程控制、逻辑分组 |
|
|
57
|
+
|
|
58
|
+
### Step 3: 保存任务定义(配置 SQL/Shell 内容)
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
# SQL 任务
|
|
62
|
+
te-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
|
|
63
|
+
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
64
|
+
--taskContentJson '{"sql":"SELECT * FROM dwd_user","repoCode":"te_etl","catalog":"hive","schema":"ws_default_dev"}' \
|
|
65
|
+
--failRetryTimes 3 --failRetryInterval 30 --timeout 60 --confirmed true
|
|
66
|
+
|
|
67
|
+
# Shell 任务
|
|
68
|
+
te-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
|
|
69
|
+
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
70
|
+
--taskContentJson '{"script":"#!/bin/bash\necho hello"}' \
|
|
71
|
+
--confirmed true
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
### Step 4: 校验 SQL(推荐,仅 TRINO_SQL 类型)
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
te-cli dataops_flow +validate_task_sql --spaceCode "${spaceCode}" \
|
|
78
|
+
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
79
|
+
--sql "SELECT * FROM dwd_user" --repoCode "te_etl" \
|
|
80
|
+
--catalog "hive" --schema "ws_default_dev"
|
|
81
|
+
# 检查 valid=true,否则查看 message 和 errorPosition
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### Step 5: 添加任务依赖关系(DAG 连线)
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
te-cli dataops_flow +add_task_relation --spaceCode "${spaceCode}" \
|
|
88
|
+
--flowCode "${flowCode}" --preTaskCode "${upstreamTaskCode}" --taskCode "${downstreamTaskCode}" \
|
|
89
|
+
--confirmed true
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
### Step 6: 配置调度
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
# CRON 表达式定时执行
|
|
96
|
+
te-cli dataops_flow +save_schedule_config --spaceCode "${spaceCode}" \
|
|
97
|
+
--flowCode "${flowCode}" --scheduled 1 --scheduleType "CRON" \
|
|
98
|
+
--cron "0 0 2 * * ?" --confirmed true
|
|
99
|
+
# 示例:0 0 2 * * ? = 每天凌晨2点
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
**CRON 格式**(6 字段):`秒 分 时 日 月 周`
|
|
103
|
+
- `0 0 */4 * * ?` — 每 4 小时
|
|
104
|
+
- `0 30 8 * * 1-5` — 工作日 8:30
|
|
105
|
+
|
|
106
|
+
### Step 7: DEV 环境测试
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
# 手动触发执行
|
|
110
|
+
te-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
111
|
+
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
112
|
+
|
|
113
|
+
# 查看执行状态
|
|
114
|
+
te-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
115
|
+
|
|
116
|
+
# 查看任务日志(从 DAG 获取 bsTaskInstanceId 和 taskCode)
|
|
117
|
+
te-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${instanceId}" --taskCode "${taskCode}"
|
|
118
|
+
|
|
119
|
+
# 根据日志调整配置后,重复测试
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
### Step 8: 发布到生产
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
te-cli dataops_flow +release_flow --spaceCode "${spaceCode}" \
|
|
126
|
+
--flowCode "${flowCode}" --confirmed true
|
|
127
|
+
# 发布后 PROD 调度立即生效,不会中断运行中的实例
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## 命令速查
|
|
133
|
+
|
|
134
|
+
| 命令 | 用途 | 关键 Flags |
|
|
135
|
+
|------|------|-----------|
|
|
136
|
+
| `+create_flow` | 创建任务流 | `--spaceCode` `--flowName` `--remark` `--confirmed` |
|
|
137
|
+
| `+create_task` | 创建任务节点 | `--spaceCode` `--flowCode` `--taskName` `--taskType` `--syncId` `--preTaskCode` `--confirmed` |
|
|
138
|
+
| `+save_task_definition` | 保存任务定义 | `--spaceCode` `--flowCode` `--taskCode` `--taskContentJson` `--failRetryTimes` `--failRetryInterval` `--timeout` `--confirmed` |
|
|
139
|
+
| `+validate_task_sql` | 校验 SQL | `--spaceCode` `--flowCode` `--taskCode` `--sql` `--repoCode` `--catalog` `--schema` |
|
|
140
|
+
| `+add_task_relation` | 添加依赖 | `--spaceCode` `--flowCode` `--preTaskCode` `--taskCode` `--confirmed` |
|
|
141
|
+
| `+save_schedule_config` | 配置调度 | `--spaceCode` `--flowCode` `--scheduled` `--scheduleType` `--cron` `--confirmed` |
|
|
142
|
+
| `+get_flow_dag` | 查看 DAG 结构 | `--spaceCode` `--flowCode` `--env` |
|
|
143
|
+
| `+get_task_params` | 查看任务参数 | `--spaceCode` `--flowCode` `--taskCode` `--env` |
|
|
144
|
+
| `+execute_flow` | 手动执行 | `--spaceCode` `--flowCode` `--env` `--confirmed` |
|
|
145
|
+
| `+get_execute_dag` | 查看运行态 DAG | `--spaceCode` `--executeId` |
|
|
146
|
+
| `+get_task_instance_log` | 查看任务日志 | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
|
|
147
|
+
| `+release_flow` | 发布到生产 | `--spaceCode` `--flowCode` `--confirmed` |
|
|
148
|
+
|
|
149
|
+
## 参数说明
|
|
150
|
+
|
|
151
|
+
- **参数引用**: 任务中通过 `${paramKey}` 引用空间参数(如 `${ws_run_date}`)
|
|
152
|
+
- **环境**: `DEV`(默认)| `PROD`
|
|
153
|
+
- **taskContentJson**: JSON 字符串,格式因 taskType 而异:
|
|
154
|
+
- TRINO_SQL: `{"sql":"SELECT ...","repoCode":"xxx","catalog":"hive","schema":"db"}`
|
|
155
|
+
- SHELL: `{"script":"#!/bin/bash\n..."}`
|
|
156
|
+
|
|
157
|
+
## 注意事项
|
|
158
|
+
|
|
159
|
+
1. **flowCode 获取**: 通过 `+list_flows` 获取(见 `dataops-flow-monitor` Skill)
|
|
160
|
+
2. **taskCode 获取**: 通过 `+create_task` 返回或 `+get_flow_dag` 查询
|
|
161
|
+
3. **发布影响**: 将 DEV 所有变更同步到 PROD,PROD 调度立即生效
|
|
162
|
+
4. **SQL 校验**: 保存 TRINO_SQL 任务前建议先校验
|
|
163
|
+
5. **不能创建循环依赖**
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: dataops-flow-monitor
|
|
3
|
+
version: 1.0.0
|
|
4
|
+
description: "任务流执行与监控:查看执行实例、运行态 DAG、任务日志、手动执行/停止任务流。触发关键词:执行任务流、运行实例、监控、日志、停止、DAG、排错、实例、flow instance。"
|
|
5
|
+
metadata:
|
|
6
|
+
requires:
|
|
7
|
+
bins: ["te-cli"]
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# DataOps 任务流执行与监控
|
|
11
|
+
|
|
12
|
+
> **前置条件:** 阅读 [`te-dataops/SKILL.md`](../SKILL.md) 了解通用规则。
|
|
13
|
+
|
|
14
|
+
使用 `dataops_flow` 子命令执行和监控任务流。
|
|
15
|
+
|
|
16
|
+
**核心概念:**
|
|
17
|
+
- **executeId** — flow_* 工具使用的执行记录 ID(通过 `+list_flow_instances` 获取)
|
|
18
|
+
- **flowInstanceId** — operations_* 工具使用的运维实例 ID(两者不互通,不可混用)
|
|
19
|
+
- **实例列表默认查询 PROD 环境**(与其他工具默认 DEV 不同)
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## 工作流 A:查找并查看任务流执行状态
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
# Step 1: 搜索任务流(获取 flowCode)
|
|
27
|
+
te-cli dataops_flow +list_flows --spaceCode "${spaceCode}" --keyword "etl"
|
|
28
|
+
|
|
29
|
+
# Step 2: 查看执行实例列表(默认 PROD,按时间倒序)
|
|
30
|
+
te-cli dataops_flow +list_flow_instances --spaceCode "${spaceCode}" \
|
|
31
|
+
--flowCode "${flowCode}" --env "PROD" --size 30
|
|
32
|
+
|
|
33
|
+
# Step 3: 查看运行态 DAG(含各节点执行状态)
|
|
34
|
+
te-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
35
|
+
|
|
36
|
+
# Step 4: 查看执行记录详情
|
|
37
|
+
te-cli dataops_flow +get_execute_record --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## 工作流 B:排查任务失败
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
# Step 1: 获取运行态 DAG,定位失败节点
|
|
46
|
+
te-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
47
|
+
# 从返回中找到 status=FAILED 的节点,获取其 bsTaskInstanceId 和 taskCode
|
|
48
|
+
|
|
49
|
+
# Step 2: 查看失败任务的日志
|
|
50
|
+
te-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${bsTaskInstanceId}" \
|
|
51
|
+
--taskCode "${taskCode}" --limit 100
|
|
52
|
+
|
|
53
|
+
# Step 3: 查看任务定义(确认 SQL/配置)
|
|
54
|
+
te-cli dataops_flow +get_task_params --spaceCode "${spaceCode}" \
|
|
55
|
+
--flowCode "${flowCode}" --taskCode "${taskCode}"
|
|
56
|
+
|
|
57
|
+
# Step 4: 修复后重新执行
|
|
58
|
+
te-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
59
|
+
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## 工作流 C:手动执行与停止
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
# 手动触发执行(默认 DEV,可指定 PROD)
|
|
68
|
+
te-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
69
|
+
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
70
|
+
# 返回 executeId
|
|
71
|
+
|
|
72
|
+
# 监控执行过程
|
|
73
|
+
te-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
74
|
+
|
|
75
|
+
# 必要时停止执行(不可撤销)
|
|
76
|
+
te-cli dataops_flow +stop_flow --spaceCode "${spaceCode}" \
|
|
77
|
+
--flowCode "${flowCode}" --executeId "${executeId}" --confirmed true
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## 工作流 D:查看任务流结构
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
# 查看任务流定义态 DAG(静态结构、依赖关系)
|
|
86
|
+
te-cli dataops_flow +get_flow_dag --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
87
|
+
|
|
88
|
+
# 查看任务流基本信息
|
|
89
|
+
te-cli dataops_flow +get_flow_detail --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
90
|
+
|
|
91
|
+
# 查看调度配置
|
|
92
|
+
te-cli dataops_flow +get_schedule_config --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
97
|
+
## 命令速查
|
|
98
|
+
|
|
99
|
+
| 命令 | 用途 | 关键 Flags |
|
|
100
|
+
|------|------|-----------|
|
|
101
|
+
| `+list_flows` | 搜索任务流 | `--spaceCode` `--keyword` |
|
|
102
|
+
| `+get_flow_detail` | 任务流信息 | `--spaceCode` `--flowCode` `--env` |
|
|
103
|
+
| `+get_flow_dag` | 定义态 DAG | `--spaceCode` `--flowCode` `--env` |
|
|
104
|
+
| `+get_schedule_config` | 调度配置 | `--spaceCode` `--flowCode` `--env` |
|
|
105
|
+
| `+execute_flow` | 手动执行 | `--spaceCode` `--flowCode` `--env` `--confirmed` |
|
|
106
|
+
| `+stop_flow` | 停止执行 | `--spaceCode` `--flowCode` `--executeId` `--confirmed` |
|
|
107
|
+
| `+list_flow_instances` | 实例列表 | `--spaceCode` `--flowCode` `--env` `--size` |
|
|
108
|
+
| `+get_execute_record` | 执行记录 | `--spaceCode` `--executeId` |
|
|
109
|
+
| `+get_execute_dag` | 运行态 DAG | `--spaceCode` `--executeId` |
|
|
110
|
+
| `+get_task_instance_log` | 任务日志 | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
|
|
111
|
+
| `+get_task_params` | 任务参数 | `--spaceCode` `--flowCode` `--taskCode` `--env` |
|
|
112
|
+
| `+update_flow` | 更新任务流 | `--spaceCode` `--flowCode` `--flowName` `--remark` `--confirmed` |
|
|
113
|
+
|
|
114
|
+
## 参数说明
|
|
115
|
+
|
|
116
|
+
- **env**: `DEV`(开发)| `PROD`(生产,实例列表默认 PROD)
|
|
117
|
+
- **任务状态**: `success` / `failure` / `running` / `waiting`
|
|
118
|
+
- **DAG 区别**: `+get_flow_dag` 返回静态定义结构,`+get_execute_dag` 返回含执行状态的运行态结构
|
|
@@ -0,0 +1,246 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: dataops-integration
|
|
3
|
+
version: 1.0.0
|
|
4
|
+
description: "数据源与数据集成:创建数据源、创建同步方案、配置字段映射、执行同步、监控同步。触发关键词:数据源、同步、integration、sync、字段映射、数据接入、datasource、MySQL、ClickHouse。"
|
|
5
|
+
metadata:
|
|
6
|
+
requires:
|
|
7
|
+
bins: ["te-cli"]
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# DataOps 数据源与数据集成
|
|
11
|
+
|
|
12
|
+
> **前置条件:** 阅读 [`te-dataops/SKILL.md`](../SKILL.md) 了解通用规则。
|
|
13
|
+
|
|
14
|
+
使用 `dataops_integration` 子命令管理数据源和同步方案。
|
|
15
|
+
|
|
16
|
+
**核心规则:**
|
|
17
|
+
- 预置仓(te_etl)与非预置仓的配置参数差异很大,必须严格按照模板生成
|
|
18
|
+
- sourceConfig/sinkConfig/channelConfig/fieldsMapping 都是 JSON 字符串
|
|
19
|
+
- 创建同步方案前先测试数据源连接
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## 工作流 A:创建数据源
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
# Step 1: 查看支持的数据源组件类型
|
|
27
|
+
te-cli dataops_integration +list_datasource_components
|
|
28
|
+
|
|
29
|
+
# Step 2: 测试连接(临时配置,不保存)
|
|
30
|
+
te-cli dataops_integration +test_datasource_connect --spaceCode "${spaceCode}" \
|
|
31
|
+
--componentName "MySQL" \
|
|
32
|
+
--configValue '{"host":"localhost","port":3306,"username":"root","password":"xxx","database":"test"}'
|
|
33
|
+
|
|
34
|
+
# Step 3: 创建数据源
|
|
35
|
+
te-cli dataops_integration +add_datasource --spaceCode "${spaceCode}" \
|
|
36
|
+
--componentName "MySQL" --dataSourceName "生产MySQL" \
|
|
37
|
+
--sharedConfig true \
|
|
38
|
+
--envJsonList '[{"host":"localhost","port":3306,"username":"root","password":"xxx"}]' \
|
|
39
|
+
--confirmed true
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
---
|
|
43
|
+
|
|
44
|
+
## 工作流 B:创建同步方案(完整流程)
|
|
45
|
+
|
|
46
|
+
### Step 1: 查看可用数据源
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
# 查看空间所有数据源
|
|
50
|
+
te-cli dataops_integration +list_space_datasources --spaceCode "${spaceCode}"
|
|
51
|
+
|
|
52
|
+
# 查看可用于同步的数据源(按 source/sink 分类)
|
|
53
|
+
te-cli dataops_integration +list_sync_datasources --spaceCode "${spaceCode}"
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
### Step 2: 浏览源表结构
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
# 列出数据源下的数据库
|
|
60
|
+
te-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}" \
|
|
61
|
+
--datasourceId "${datasourceId}"
|
|
62
|
+
|
|
63
|
+
# 列出数据库下的表
|
|
64
|
+
te-cli dataops_integration +list_datasource_tables --spaceCode "${spaceCode}" \
|
|
65
|
+
--datasourceId "${datasourceId}" --database "test"
|
|
66
|
+
|
|
67
|
+
# 获取表结构(列定义、分区)
|
|
68
|
+
te-cli dataops_integration +get_table_structure --spaceCode "${spaceCode}" \
|
|
69
|
+
--datasourceId "${datasourceId}" --database "test" --tablePath "users"
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### Step 3: 创建同步方案
|
|
73
|
+
|
|
74
|
+
**关键:必须严格按照下方 JSON 模板生成参数**
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
te-cli dataops_integration +add_sync_solution --spaceCode "${spaceCode}" \
|
|
78
|
+
--syncName "MySQL到预置仓同步" \
|
|
79
|
+
--srcComponent "MySQL" --srcDatasourceId "${mysqlDatasourceId}" \
|
|
80
|
+
--sinkComponent "hive" --sinkDatasourceId "te_etl@TASK_ENGINE_TRINO" \
|
|
81
|
+
--sourceConfig '{"component":"MySQL","datasourceId":"xxx","database":"test","tablePath":"users"}' \
|
|
82
|
+
--sinkConfig '{"component":"hive","datasourceId":"te_etl@TASK_ENGINE_TRINO","database":"","tablePath":"ods_users_mysql","tableType":"PHYSICAL_TABLE","bizClassify":"CURRENT","dbBizType":"TASK_ENV_DB","authedSpace":"","partitionKeys":[],"dataSaveMode":1,"batchSize":20000}' \
|
|
83
|
+
--channelConfig '{"limitType":"0","gatewayConfig":{"engineFlag":"TASK_ENGINE_TRINO","companyId":1,"appDefinition":"APP_GAIA","bizFlag":"BIZ_GAIA_TASK_RELEASE","repoCode":"te_etl","spaceCode":"default"}}' \
|
|
84
|
+
--fieldsMapping '{"mapping":[{"source":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false},"target":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false}}]}' \
|
|
85
|
+
--confirmed true
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
### Step 4: 执行同步
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
# 手动执行同步方案
|
|
92
|
+
te-cli dataops_integration +exec_sync_solution --spaceCode "${spaceCode}" \
|
|
93
|
+
--syncId "${syncId}" --confirmed true
|
|
94
|
+
|
|
95
|
+
# 查看执行历史
|
|
96
|
+
te-cli dataops_integration +list_sync_exec_histories --spaceCode "${spaceCode}" \
|
|
97
|
+
--syncId "${syncId}" --limit 20
|
|
98
|
+
|
|
99
|
+
# 查看执行详情
|
|
100
|
+
te-cli dataops_integration +get_sync_exec_info --spaceCode "${spaceCode}" \
|
|
101
|
+
--syncId "${syncId}" --taskId "${taskId}"
|
|
102
|
+
|
|
103
|
+
# 必要时停止执行
|
|
104
|
+
te-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
|
|
105
|
+
--syncId "${syncId}" --taskId "${taskId}" --confirmed true
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## JSON 配置模板
|
|
111
|
+
|
|
112
|
+
### 预置仓作为目标端(Sink)
|
|
113
|
+
|
|
114
|
+
```json
|
|
115
|
+
{
|
|
116
|
+
"component": "hive",
|
|
117
|
+
"datasourceId": "te_etl@TASK_ENGINE_TRINO",
|
|
118
|
+
"database": "",
|
|
119
|
+
"tablePath": "ods_users_mysql",
|
|
120
|
+
"tableType": "PHYSICAL_TABLE",
|
|
121
|
+
"bizClassify": "CURRENT",
|
|
122
|
+
"dbBizType": "TASK_ENV_DB",
|
|
123
|
+
"authedSpace": "",
|
|
124
|
+
"partitionKeys": [],
|
|
125
|
+
"dataSaveMode": 1,
|
|
126
|
+
"batchSize": 20000
|
|
127
|
+
}
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
### 非预置仓(如 MySQL)
|
|
131
|
+
|
|
132
|
+
```json
|
|
133
|
+
{
|
|
134
|
+
"component": "MySQL",
|
|
135
|
+
"datasourceId": "ds-uuid-xxx",
|
|
136
|
+
"database": "test",
|
|
137
|
+
"tablePath": "users"
|
|
138
|
+
}
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
### channelConfig(源端或目标端涉及预置仓时必须包含 gatewayConfig)
|
|
142
|
+
|
|
143
|
+
```json
|
|
144
|
+
{
|
|
145
|
+
"limitType": "0",
|
|
146
|
+
"gatewayConfig": {
|
|
147
|
+
"engineFlag": "TASK_ENGINE_TRINO",
|
|
148
|
+
"companyId": 1,
|
|
149
|
+
"appDefinition": "APP_GAIA",
|
|
150
|
+
"bizFlag": "BIZ_GAIA_TASK_RELEASE",
|
|
151
|
+
"repoCode": "te_etl",
|
|
152
|
+
"spaceCode": "default"
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
### fieldsMapping(双列映射)
|
|
158
|
+
|
|
159
|
+
```json
|
|
160
|
+
{
|
|
161
|
+
"mapping": [
|
|
162
|
+
{
|
|
163
|
+
"source": {
|
|
164
|
+
"name": "id",
|
|
165
|
+
"type": "int",
|
|
166
|
+
"manual": false,
|
|
167
|
+
"partitionKey": false,
|
|
168
|
+
"primaryKey": false,
|
|
169
|
+
"shardingKey": false,
|
|
170
|
+
"sortingKey": false,
|
|
171
|
+
"upsertKey": false
|
|
172
|
+
},
|
|
173
|
+
"target": {
|
|
174
|
+
"name": "id",
|
|
175
|
+
"type": "int",
|
|
176
|
+
"manual": false,
|
|
177
|
+
"partitionKey": false,
|
|
178
|
+
"primaryKey": false,
|
|
179
|
+
"shardingKey": false,
|
|
180
|
+
"sortingKey": false,
|
|
181
|
+
"upsertKey": false
|
|
182
|
+
}
|
|
183
|
+
}
|
|
184
|
+
]
|
|
185
|
+
}
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
---
|
|
189
|
+
|
|
190
|
+
## 命令速查
|
|
191
|
+
|
|
192
|
+
| 命令 | 用途 | 关键 Flags |
|
|
193
|
+
|------|------|-----------|
|
|
194
|
+
| `+list_datasource_components` | 支持的组件 | 无 |
|
|
195
|
+
| `+list_space_datasources` | 空间数据源 | `--spaceCode` `--componentName` `--dataSourceName` |
|
|
196
|
+
| `+add_datasource` | 创建数据源 | `--spaceCode` `--componentName` `--dataSourceName` `--sharedConfig` `--envJsonList` `--confirmed` |
|
|
197
|
+
| `+test_datasource_connect` | 测试连接 | `--spaceCode` `--componentName` `--configValue` |
|
|
198
|
+
| `+list_datasource_databases` | 列出数据库 | `--spaceCode` `--datasourceId` |
|
|
199
|
+
| `+list_datasource_tables` | 列出表 | `--spaceCode` `--datasourceId` `--database` |
|
|
200
|
+
| `+get_table_structure` | 表结构 | `--spaceCode` `--datasourceId` `--database` `--tablePath` |
|
|
201
|
+
| `+list_sync_solutions` | 同步方案列表 | `--spaceCode` |
|
|
202
|
+
| `+add_sync_solution` | 创建同步方案 | `--spaceCode` `--syncName` `--srcComponent` `--srcDatasourceId` `--sinkComponent` `--sinkDatasourceId` `--sourceConfig` `--sinkConfig` `--channelConfig` `--fieldsMapping` `--confirmed` |
|
|
203
|
+
| `+exec_sync_solution` | 执行同步 | `--spaceCode` `--syncId` `--confirmed` |
|
|
204
|
+
| `+list_sync_exec_histories` | 执行历史 | `--spaceCode` `--syncId` `--execType` `--limit` |
|
|
205
|
+
| `+get_sync_exec_info` | 执行详情 | `--spaceCode` `--syncId` `--taskId` |
|
|
206
|
+
| `+stop_sync_solution` | 停止同步 | `--spaceCode` `--syncId` `--taskId` `--confirmed` |
|
|
207
|
+
|
|
208
|
+
## 组件条件必填参数
|
|
209
|
+
|
|
210
|
+
部分组件存在根据部署模式不同而变化的条件必填参数,`+list_datasource_components` 的 requiredFields 可能不包含这些参数(它们在 optionalFields 或 importantNotes 中说明)。创建数据源时务必根据用户选择的 mode 补齐对应参数。
|
|
211
|
+
|
|
212
|
+
### MongoDB
|
|
213
|
+
|
|
214
|
+
| mode 值 | 额外必填参数 | 说明 |
|
|
215
|
+
|----------|-------------|------|
|
|
216
|
+
| `single` | 无 | 单节点模式 |
|
|
217
|
+
| `replicaSet` | `replicaSet` | 副本集名称(如 `rs0`),字段名为 `replicaSet` 而非 `replicaSetName` |
|
|
218
|
+
| `sharded` | 无 | 分片集群模式 |
|
|
219
|
+
|
|
220
|
+
**MongoDB 各模式 envJsonList 示例:**
|
|
221
|
+
|
|
222
|
+
单节点:
|
|
223
|
+
```json
|
|
224
|
+
[{"mode":"single","nodes":[{"host":"10.0.0.1","port":"27017"}],"database":"mydb","username":"admin","password":"xxx"}]
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
副本集群(注意 `replicaSet` 必填):
|
|
228
|
+
```json
|
|
229
|
+
[{"mode":"replicaSet","nodes":[{"host":"10.0.0.1","port":"27017"},{"host":"10.0.0.2","port":"27017"}],"database":"mydb","username":"admin","password":"xxx","replicaSet":"rs0"}]
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
分片集群:
|
|
233
|
+
```json
|
|
234
|
+
[{"mode":"sharded","nodes":[{"host":"10.0.0.1","port":"27017"},{"host":"10.0.0.2","port":"27017"}],"database":"mydb","username":"admin","password":"xxx"}]
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
---
|
|
238
|
+
|
|
239
|
+
## 关键规则
|
|
240
|
+
|
|
241
|
+
1. **表名规则**: 写入预置仓时,未指定表名则用 `ods_${来源表名}_${组件名小写}`
|
|
242
|
+
2. **tablePath**: PostgreSQL 用 `模式.表名`,其他组件直接用表名
|
|
243
|
+
3. **预置仓 database 为空**,非预置仓 database 必填
|
|
244
|
+
4. **channelConfig**: 涉及预置仓时必须包含 gatewayConfig
|
|
245
|
+
5. **字段映射**: 每个字段对象必须包含 manual/partitionKey/primaryKey/shardingKey/sortingKey/upsertKey 属性
|
|
246
|
+
6. **条件必填参数**: 部分组件(如 MongoDB)根据 mode 不同有额外必填字段,详见上方「组件条件必填参数」
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: dataops-operations
|
|
3
|
+
version: 1.0.0
|
|
4
|
+
description: "运维操作与补数管理:搜索运维实例、重跑失败任务、停止实例、补数任务管理。触发关键词:运维、监控、补数、重跑、告警、operations、实例、backfill。"
|
|
5
|
+
metadata:
|
|
6
|
+
requires:
|
|
7
|
+
bins: ["te-cli"]
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# DataOps 运维操作与补数管理
|
|
11
|
+
|
|
12
|
+
> **前置条件:** 阅读 [`te-dataops/SKILL.md`](../SKILL.md) 了解通用规则。
|
|
13
|
+
|
|
14
|
+
使用 `dataops_operations` 子命令进行运维操作。
|
|
15
|
+
|
|
16
|
+
**核心概念:**
|
|
17
|
+
- **flowInstanceId** — operations_* 工具使用的运维实例 ID
|
|
18
|
+
- **executeId** — flow_* 工具使用的执行记录 ID(两者不互通,不可混用)
|
|
19
|
+
- 运维视角关注生产环境的实例管理和补数操作
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## 工作流 A:运维实例查询与操作
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
# Step 1: 搜索任务流实例(支持按状态/负责人/时间筛选)
|
|
27
|
+
te-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
|
|
28
|
+
--pageNum 1 --pageSize 20
|
|
29
|
+
|
|
30
|
+
# Step 2: 查看实例下的所有任务节点及执行状态
|
|
31
|
+
te-cli dataops_operations +list_flow_task_instances --spaceCode "${spaceCode}" \
|
|
32
|
+
--flowInstanceId "${flowInstanceId}"
|
|
33
|
+
|
|
34
|
+
# Step 3: 获取任务节点定义详情
|
|
35
|
+
te-cli dataops_operations +get_task_definition --spaceCode "${spaceCode}" \
|
|
36
|
+
--flowInstanceId "${flowInstanceId}" --taskCode "${taskCode}"
|
|
37
|
+
|
|
38
|
+
# Step 4: 重跑失败的任务流实例(从失败节点继续)
|
|
39
|
+
te-cli dataops_operations +rerun_flow_instance --spaceCode "${spaceCode}" \
|
|
40
|
+
--flowInstanceId "${flowInstanceId}" --confirmed true
|
|
41
|
+
|
|
42
|
+
# Step 5: 停止正在运行的实例
|
|
43
|
+
te-cli dataops_operations +stop_flow_instance --spaceCode "${spaceCode}" \
|
|
44
|
+
--flowInstanceId "${flowInstanceId}" --confirmed true
|
|
45
|
+
|
|
46
|
+
# Step 6: 控制实例运行状态(暂停/强制暂停/恢复)
|
|
47
|
+
te-cli dataops_operations +control_flow_instance --spaceCode "${spaceCode}" \
|
|
48
|
+
--flowInstanceId "${flowInstanceId}" --action "PAUSE" --confirmed true
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## 工作流 B:补数任务管理
|
|
54
|
+
|
|
55
|
+
补数任务用于批量重跑历史数据。
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
# Step 1: 查询补数任务列表
|
|
59
|
+
te-cli dataops_operations +list_flow_jobs --spaceCode "${spaceCode}" \
|
|
60
|
+
--pageNum 1 --pageSize 20
|
|
61
|
+
|
|
62
|
+
# Step 2: 获取补数任务详情(配置、日期范围、并发设置)
|
|
63
|
+
te-cli dataops_operations +get_flow_job_detail --spaceCode "${spaceCode}" \
|
|
64
|
+
--jobId "${jobId}"
|
|
65
|
+
|
|
66
|
+
# Step 3: 获取补数任务的执行计划列表(每个日期的状态)
|
|
67
|
+
te-cli dataops_operations +list_flow_job_plans --spaceCode "${spaceCode}" \
|
|
68
|
+
--jobId "${jobId}"
|
|
69
|
+
|
|
70
|
+
# Step 4: 停止正在执行的补数任务
|
|
71
|
+
te-cli dataops_operations +stop_flow_job --spaceCode "${spaceCode}" \
|
|
72
|
+
--jobId "${jobId}" --confirmed true
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## 工作流 C:任务节点实例查询
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
# 搜索任务节点实例(支持按状态/类型/时间筛选)
|
|
81
|
+
te-cli dataops_operations +search_task_instances --spaceCode "${spaceCode}" \
|
|
82
|
+
--pageNum 1 --pageSize 20
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
---
|
|
86
|
+
|
|
87
|
+
## 命令速查
|
|
88
|
+
|
|
89
|
+
| 命令 | 用途 | 关键 Flags |
|
|
90
|
+
|------|------|-----------|
|
|
91
|
+
| `+search_flow_instances` | 搜索任务流实例 | `--spaceCode` `--pageNum` `--pageSize` |
|
|
92
|
+
| `+list_flow_task_instances` | 实例任务节点 | `--spaceCode` `--flowInstanceId` |
|
|
93
|
+
| `+get_task_definition` | 任务节点定义 | `--spaceCode` `--flowInstanceId` `--taskCode` |
|
|
94
|
+
| `+rerun_flow_instance` | 重跑实例 | `--spaceCode` `--flowInstanceId` `--confirmed` |
|
|
95
|
+
| `+stop_flow_instance` | 停止实例 | `--spaceCode` `--flowInstanceId` `--confirmed` |
|
|
96
|
+
| `+control_flow_instance` | 控制实例状态 | `--spaceCode` `--flowInstanceId` `--action` `--confirmed` |
|
|
97
|
+
| `+search_task_instances` | 搜索任务实例 | `--spaceCode` `--pageNum` `--pageSize` |
|
|
98
|
+
| `+list_flow_jobs` | 补数任务列表 | `--spaceCode` `--pageNum` `--pageSize` |
|
|
99
|
+
| `+get_flow_job_detail` | 补数任务详情 | `--spaceCode` `--jobId` |
|
|
100
|
+
| `+list_flow_job_plans` | 补数执行计划 | `--spaceCode` `--jobId` |
|
|
101
|
+
| `+stop_flow_job` | 停止补数任务 | `--spaceCode` `--jobId` `--confirmed` |
|
|
102
|
+
|
|
103
|
+
## 参数说明
|
|
104
|
+
|
|
105
|
+
- **action**: `PAUSE`(暂停)| `FORCE_PAUSE`(强制暂停)| `RESUME`(恢复)
|
|
106
|
+
- **实例状态**: `RUNNING` / `SUCCESS` / `FAILURE` / `WAITING` / `STOPPED`
|
|
107
|
+
- **flowInstanceId vs executeId**: 运维工具用 flowInstanceId,flow 工具用 executeId,不可混用
|