@thinkingai/ae-cli 1.0.1 → 1.0.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +38 -64
- package/README.zh.md +203 -0
- package/dist/{auth-JXELJNDS.js → auth-4GAHTBVH.js} +2 -2
- package/dist/{auth-L3G3A74I.js → auth-U2GHFMFY.js} +20 -6
- package/dist/{chunk-OVQOK35G.js → chunk-EBFUVWIZ.js} +25 -6
- package/dist/{chunk-3GM5LJZR.js → chunk-OQ4XEG77.js} +7 -4
- package/dist/{chunk-JD5DXMKL.js → chunk-TJIMDNAU.js} +10 -8
- package/dist/{chunk-RJNLN3KQ.js → chunk-UK6HO5O5.js} +7 -6
- package/dist/{client-5QSQBBDB.js → client-HU63573O.js} +3 -3
- package/dist/{config-Q7J3Z7Z2.js → config-OLF7RBPM.js} +5 -5
- package/dist/index.js +18 -17
- package/dist/{raw-ZNVURYMR.js → raw-GDO4IWNW.js} +8 -6
- package/dist/{te-analysis-7NN3SY3I.js → te-analysis-ZDJXMWZY.js} +64 -26
- package/dist/{te-audience-P43BWMHM.js → te-audience-T5B52EEB.js} +65 -25
- package/dist/{te-common-H3IPTI2I.js → te-common-BPTXXTHQ.js} +11 -38
- package/dist/{community-GMALVSRF.js → te-community-FPB5LH56.js} +59 -59
- package/dist/{te-dataops-WCGDBJLF.js → te-dataops-2NUJVG47.js} +3 -28
- package/dist/{te-engage-UTRVNTGU.js → te-engage-K3EANZ5P.js} +502 -520
- package/dist/{te-meta-HGYNXVGF.js → te-meta-JJMV75KB.js} +88 -20
- package/package.json +9 -7
- package/skills/ae-analysis/SKILL.md +229 -0
- package/skills/{te-meta/references/batch-create-metadata.md → ae-analysis/references/batch_create_metadata.md} +5 -5
- package/skills/{te-meta/references/batch-edit-metadata.md → ae-analysis/references/batch_edit_metadata.md} +4 -4
- package/skills/{te-analysis/references/build-entity-details-sql.md → ae-analysis/references/build_entity_details_sql.md} +14 -14
- package/skills/{te-analysis/references/build-event-details-sql.md → ae-analysis/references/build_event_details_sql.md} +14 -14
- package/skills/{te-analysis/references/create-alert.md → ae-analysis/references/create_alert.md} +7 -7
- package/skills/{te-audience/references/create-cluster.md → ae-analysis/references/create_cluster.md} +26 -22
- package/skills/{te-analysis/references/create-dashboard.md → ae-analysis/references/create_dashboard.md} +5 -5
- package/skills/{te-meta/references/create-metric.md → ae-analysis/references/create_metric.md} +16 -16
- package/skills/{te-analysis/references/create-or-update-dashboard-note.md → ae-analysis/references/create_or_update_dashboard_note.md} +5 -5
- package/skills/{te-meta/references/create-project-mark-time.md → ae-analysis/references/create_project_mark_time.md} +5 -5
- package/skills/{te-analysis/references/create-public-access-link.md → ae-analysis/references/create_public_access_link.md} +5 -5
- package/skills/{te-analysis/references/create-report.md → ae-analysis/references/create_report.md} +14 -14
- package/skills/{te-analysis/references/create-result-cluster.md → ae-analysis/references/create_result_cluster.md} +8 -8
- package/skills/{te-audience/references/create-tag.md → ae-analysis/references/create_tag.md} +18 -17
- package/skills/{te-meta/references/create-virtual-event.md → ae-analysis/references/create_virtual_event.md} +14 -14
- package/skills/{te-meta/references/create-virtual-property.md → ae-analysis/references/create_virtual_property.md} +11 -11
- package/skills/{te-meta/references/delete-project-mark-times.md → ae-analysis/references/delete_project_mark_times.md} +4 -4
- package/skills/{te-meta/references/delete-track-items.md → ae-analysis/references/delete_track_items.md} +4 -4
- package/skills/{te-analysis/references/drilldown-user-events.md → ae-analysis/references/drilldown_user_events.md} +11 -11
- package/skills/{te-analysis/references/drilldown-users.md → ae-analysis/references/drilldown_users.md} +12 -9
- package/skills/ae-analysis/references/generate_track_program.md +51 -0
- package/skills/ae-analysis/references/generate_track_sdk_sample.md +43 -0
- package/skills/{te-analysis/references/get-alert.md → ae-analysis/references/get_alert.md} +4 -4
- package/skills/{te-analysis/references/get-alert-definition-schema.md → ae-analysis/references/get_alert_definition_schema.md} +4 -4
- package/skills/ae-analysis/references/get_analysis_query_schema.md +61 -0
- package/skills/ae-analysis/references/get_cluster_definition_schema.md +53 -0
- package/skills/ae-analysis/references/get_clusters_by_name.md +69 -0
- package/skills/{te-analysis/references/get-filter-schema.md → ae-analysis/references/get_filter_schema.md} +4 -4
- package/skills/{te-analysis/references/get-groupby-schema.md → ae-analysis/references/get_groupby_schema.md} +4 -4
- package/skills/{te-meta/references/get-metric.md → ae-analysis/references/get_metric.md} +4 -4
- package/skills/{te-meta/references/get-project-config.md → ae-analysis/references/get_project_config.md} +4 -4
- package/skills/{te-analysis/references/get-report-definition.md → ae-analysis/references/get_report_definition.md} +4 -4
- package/skills/{te-common/references/get-resource-url.md → ae-analysis/references/get_resource_url.md} +7 -7
- package/skills/{te-analysis/references/get-table-columns.md → ae-analysis/references/get_table_columns.md} +4 -4
- package/skills/ae-analysis/references/get_tag_definition_schema.md +49 -0
- package/skills/ae-analysis/references/get_tags_by_name.md +69 -0
- package/skills/{te-meta/references/get-track-program.md → ae-analysis/references/get_track_program.md} +4 -4
- package/skills/{te-analysis/references/list-alerts.md → ae-analysis/references/list_alerts.md} +8 -5
- package/skills/{te-audience/references/list-cluster-members.md → ae-analysis/references/list_cluster_members.md} +13 -6
- package/skills/{te-audience/references/list-clusters.md → ae-analysis/references/list_clusters.md} +12 -6
- package/skills/{te-analysis/references/list-dashboards.md → ae-analysis/references/list_dashboards.md} +10 -6
- package/skills/{te-meta/references/list-entities.md → ae-analysis/references/list_entities.md} +12 -5
- package/skills/{te-meta/references/list-events.md → ae-analysis/references/list_events.md} +10 -7
- package/skills/ae-analysis/references/list_metrics.md +41 -0
- package/skills/ae-analysis/references/list_project_mark_times.md +42 -0
- package/skills/{te-meta/references/list-project-users.md → ae-analysis/references/list_project_users.md} +4 -4
- package/skills/{te-common/references/list-projects.md → ae-analysis/references/list_projects.md} +9 -7
- package/skills/{te-meta/references/list-properties.md → ae-analysis/references/list_properties.md} +11 -6
- package/skills/ae-analysis/references/list_public_access_links.md +41 -0
- package/skills/{te-analysis/references/list-reports.md → ae-analysis/references/list_reports.md} +10 -6
- package/skills/{te-audience/references/list-tag-members.md → ae-analysis/references/list_tag_members.md} +13 -6
- package/skills/{te-audience/references/list-tags.md → ae-analysis/references/list_tags.md} +12 -6
- package/skills/{te-analysis/references/load-filters.md → ae-analysis/references/load_filters.md} +5 -5
- package/skills/{te-analysis/references/query-adhoc.md → ae-analysis/references/query_adhoc.md} +19 -14
- package/skills/{te-analysis/references/query-dashboard-detail.md → ae-analysis/references/query_dashboard_detail.md} +4 -4
- package/skills/{te-analysis/references/query-dashboard-report-data.md → ae-analysis/references/query_dashboard_report_data.md} +13 -13
- package/skills/{te-analysis/references/query-entity-details.md → ae-analysis/references/query_entity_details.md} +15 -15
- package/skills/{te-analysis/references/query-event-details.md → ae-analysis/references/query_event_details.md} +17 -17
- package/skills/{te-analysis/references/query-report-data.md → ae-analysis/references/query_report_data.md} +16 -16
- package/skills/{te-audience/references/refresh-cluster.md → ae-analysis/references/refresh_cluster.md} +4 -4
- package/skills/{te-audience/references/refresh-tag.md → ae-analysis/references/refresh_tag.md} +4 -4
- package/skills/{te-meta/references/save-track-items.md → ae-analysis/references/save_track_items.md} +4 -4
- package/skills/{te-analysis/references/update-alert.md → ae-analysis/references/update_alert.md} +7 -7
- package/skills/{te-audience/references/update-cluster.md → ae-analysis/references/update_cluster.md} +17 -15
- package/skills/{te-analysis/references/update-dashboard.md → ae-analysis/references/update_dashboard.md} +5 -5
- package/skills/{te-meta/references/update-metric.md → ae-analysis/references/update_metric.md} +14 -14
- package/skills/{te-meta/references/update-project-mark-time.md → ae-analysis/references/update_project_mark_time.md} +5 -5
- package/skills/{te-analysis/references/update-public-access-link.md → ae-analysis/references/update_public_access_link.md} +5 -5
- package/skills/{te-audience/references/update-tag.md → ae-analysis/references/update_tag.md} +17 -16
- package/skills/ae-community/SKILL.md +305 -0
- package/skills/ae-community/references/community-activity-analysis.md +67 -0
- package/skills/ae-community/references/community-analyzing-official-content.md +71 -0
- package/skills/ae-community/references/community-analyzing-theme-comment.md +123 -0
- package/skills/ae-community/references/community-character-analysis.md +70 -0
- package/skills/ae-community/references/community-daily-report.md +84 -0
- package/skills/ae-community/references/community-hottopic-insight.md +113 -0
- package/skills/ae-community/references/community-weekly-report.md +84 -0
- package/skills/ae-community/references/get_channel_info.md +20 -0
- package/skills/ae-community/references/get_comment_tag_analysis.md +29 -0
- package/skills/ae-community/references/get_comments_summary.md +32 -0
- package/skills/ae-community/references/get_corpus_tags.md +20 -0
- package/skills/ae-community/references/get_daily_summary.md +22 -0
- package/skills/ae-community/references/get_hot_topics.md +25 -0
- package/skills/ae-community/references/get_livestream_analysis.md +22 -0
- package/skills/ae-community/references/get_livestream_detail.md +33 -0
- package/skills/ae-community/references/get_livestream_list.md +25 -0
- package/skills/ae-community/references/get_livestream_overview.md +24 -0
- package/skills/ae-community/references/get_livestream_room_metrics.md +25 -0
- package/skills/ae-community/references/get_livestream_rooms.md +25 -0
- package/skills/ae-community/references/get_overview_metrics.md +25 -0
- package/skills/ae-community/references/get_post_detail.md +37 -0
- package/skills/ae-community/references/get_risk_content.md +42 -0
- package/skills/ae-community/references/get_sentiment_overview.md +33 -0
- package/skills/ae-community/references/get_tag_trends.md +27 -0
- package/skills/ae-community/references/get_topic_detail.md +26 -0
- package/skills/ae-community/references/search_posts.md +47 -0
- package/skills/ae-dataops/SKILL.md +706 -0
- package/skills/ae-dataops/references/dataops-flow-create.md +163 -0
- package/skills/ae-dataops/references/dataops-flow-monitor.md +118 -0
- package/skills/ae-dataops/references/dataops-integration.md +251 -0
- package/skills/ae-dataops/references/dataops-query.md +111 -0
- package/skills/ae-dataops/references/dataops-table.md +122 -0
- package/skills/ae-engage/SKILL.md +311 -0
- package/skills/ae-engage/references/add-approver.md +24 -0
- package/skills/ae-engage/references/add-channel.md +74 -0
- package/skills/ae-engage/references/approver-list.md +19 -0
- package/skills/ae-engage/references/cancel-query-by-request-id.md +23 -0
- package/skills/ae-engage/references/channel-detail.md +52 -0
- package/skills/ae-engage/references/channel-list.md +41 -0
- package/skills/ae-engage/references/config-channel-detail.md +32 -0
- package/skills/ae-engage/references/config-channel-list.md +27 -0
- package/skills/ae-engage/references/config-item-analysis-report.md +34 -0
- package/skills/ae-engage/references/config-item-detail.md +30 -0
- package/skills/ae-engage/references/config-item-list.md +19 -0
- package/skills/ae-engage/references/config-item-strategy-comparison.md +25 -0
- package/skills/ae-engage/references/config-item-trigger-report.md +42 -0
- package/skills/ae-engage/references/copy-config-template.md +29 -0
- package/skills/ae-engage/references/delete-channel.md +24 -0
- package/skills/ae-engage/references/delete-config-channel.md +24 -0
- package/skills/ae-engage/references/delete-config-item.md +25 -0
- package/skills/ae-engage/references/delete-flow.md +24 -0
- package/skills/ae-engage/references/flow-ab-split-node-report.md +54 -0
- package/skills/ae-engage/references/flow-detail.md +79 -0
- package/skills/ae-engage/references/flow-list.md +19 -0
- package/skills/ae-engage/references/flow-node-config-schema.md +44 -0
- package/skills/ae-engage/references/flow-node-detail-report.md +55 -0
- package/skills/ae-engage/references/flow-node-overview-report.md +52 -0
- package/skills/ae-engage/references/flow-process-report.md +59 -0
- package/skills/ae-engage/references/manage-flow.md +94 -0
- package/skills/ae-engage/references/manage-strategy.md +74 -0
- package/skills/ae-engage/references/manage-task.md +37 -0
- package/skills/ae-engage/references/modify-flow-base-info.md +27 -0
- package/skills/ae-engage/references/save-flow.md +871 -0
- package/skills/ae-engage/references/strategy-detail.md +73 -0
- package/skills/ae-engage/references/strategy-list.md +21 -0
- package/skills/ae-engage/references/task-data-detail.md +74 -0
- package/skills/ae-engage/references/task-data-overview.md +42 -0
- package/skills/ae-engage/references/task-detail.md +61 -0
- package/skills/ae-engage/references/task-experiment-report.md +45 -0
- package/skills/ae-engage/references/task-list.md +91 -0
- package/skills/ae-engage/references/task-metric-detail.md +50 -0
- package/skills/ae-engage/references/task-stats.md +89 -0
- package/skills/ae-engage/references/update-channel-status.md +32 -0
- package/skills/ae-engage/references/update-config-channel-status.md +32 -0
- package/skills/ae-engage/references/validate-flow-node-config.md +52 -0
- package/skills/ae-engage/references/whitelist-list.md +19 -0
- package/skills/te-analysis/SKILL.md +0 -107
- package/skills/te-analysis/references/get-analysis-query-schema.md +0 -36
- package/skills/te-analysis/references/list-public-access-links.md +0 -31
- package/skills/te-audience/SKILL.md +0 -63
- package/skills/te-audience/references/get-cluster-definition-schema.md +0 -37
- package/skills/te-audience/references/get-clusters-by-name.md +0 -32
- package/skills/te-audience/references/get-tag-definition-schema.md +0 -32
- package/skills/te-audience/references/get-tags-by-name.md +0 -32
- package/skills/te-common/SKILL.md +0 -135
- package/skills/te-community/SKILL.md +0 -133
- package/skills/te-community/references/community-activity-analysis.md +0 -68
- package/skills/te-community/references/community-analyzing-official-content.md +0 -72
- package/skills/te-community/references/community-analyzing-theme-comment.md +0 -124
- package/skills/te-community/references/community-character-analysis.md +0 -71
- package/skills/te-community/references/community-daily-report.md +0 -82
- package/skills/te-community/references/community-hottopic-insight.md +0 -120
- package/skills/te-community/references/community-release-analysis.md +0 -79
- package/skills/te-community/references/community-weekly-report.md +0 -84
- package/skills/te-community/references/get_channel_info.md +0 -21
- package/skills/te-community/references/get_comment_tag_analysis.md +0 -30
- package/skills/te-community/references/get_comments_summary.md +0 -33
- package/skills/te-community/references/get_corpus_tags.md +0 -21
- package/skills/te-community/references/get_daily_summary.md +0 -23
- package/skills/te-community/references/get_hot_topics.md +0 -26
- package/skills/te-community/references/get_livestream_analysis.md +0 -23
- package/skills/te-community/references/get_livestream_detail.md +0 -34
- package/skills/te-community/references/get_livestream_list.md +0 -26
- package/skills/te-community/references/get_livestream_overview.md +0 -25
- package/skills/te-community/references/get_livestream_room_metrics.md +0 -26
- package/skills/te-community/references/get_livestream_rooms.md +0 -26
- package/skills/te-community/references/get_overview_metrics.md +0 -26
- package/skills/te-community/references/get_post_detail.md +0 -38
- package/skills/te-community/references/get_risk_content.md +0 -43
- package/skills/te-community/references/get_sentiment_overview.md +0 -34
- package/skills/te-community/references/get_tag_trends.md +0 -28
- package/skills/te-community/references/get_topic_detail.md +0 -27
- package/skills/te-community/references/search_posts.md +0 -48
- package/skills/te-dataops/SKILL.md +0 -645
- package/skills/te-dataops/references/dataops-flow-create.md +0 -163
- package/skills/te-dataops/references/dataops-flow-monitor.md +0 -118
- package/skills/te-dataops/references/dataops-integration.md +0 -246
- package/skills/te-dataops/references/dataops-operations.md +0 -107
- package/skills/te-dataops/references/dataops-query.md +0 -111
- package/skills/te-dataops/references/dataops-table.md +0 -122
- package/skills/te-engage/SKILL.md +0 -211
- package/skills/te-engage/references/add-approver.md +0 -24
- package/skills/te-engage/references/add-channel.md +0 -74
- package/skills/te-engage/references/approver-list.md +0 -19
- package/skills/te-engage/references/cancel-query-by-request-id.md +0 -23
- package/skills/te-engage/references/channel-detail.md +0 -52
- package/skills/te-engage/references/channel-list.md +0 -41
- package/skills/te-engage/references/config-channel-detail.md +0 -32
- package/skills/te-engage/references/config-channel-list.md +0 -27
- package/skills/te-engage/references/config-item-analysis-report.md +0 -34
- package/skills/te-engage/references/config-item-detail.md +0 -30
- package/skills/te-engage/references/config-item-list.md +0 -19
- package/skills/te-engage/references/config-item-strategy-comparison.md +0 -25
- package/skills/te-engage/references/config-item-trigger-report.md +0 -42
- package/skills/te-engage/references/copy-config-template.md +0 -29
- package/skills/te-engage/references/delete-channel.md +0 -24
- package/skills/te-engage/references/delete-config-channel.md +0 -24
- package/skills/te-engage/references/delete-config-item.md +0 -25
- package/skills/te-engage/references/delete-flow.md +0 -24
- package/skills/te-engage/references/flow-ab-split-node-report.md +0 -54
- package/skills/te-engage/references/flow-detail.md +0 -79
- package/skills/te-engage/references/flow-list.md +0 -19
- package/skills/te-engage/references/flow-node-config-schema.md +0 -44
- package/skills/te-engage/references/flow-node-detail-report.md +0 -55
- package/skills/te-engage/references/flow-node-overview-report.md +0 -52
- package/skills/te-engage/references/flow-process-report.md +0 -59
- package/skills/te-engage/references/manage-flow.md +0 -94
- package/skills/te-engage/references/manage-strategy.md +0 -74
- package/skills/te-engage/references/manage-task.md +0 -37
- package/skills/te-engage/references/modify-flow-base-info.md +0 -27
- package/skills/te-engage/references/save-flow.md +0 -870
- package/skills/te-engage/references/strategy-detail.md +0 -73
- package/skills/te-engage/references/strategy-list.md +0 -21
- package/skills/te-engage/references/task-data-detail.md +0 -81
- package/skills/te-engage/references/task-data-overview.md +0 -48
- package/skills/te-engage/references/task-detail.md +0 -61
- package/skills/te-engage/references/task-experiment-report.md +0 -51
- package/skills/te-engage/references/task-list.md +0 -91
- package/skills/te-engage/references/task-metric-detail.md +0 -56
- package/skills/te-engage/references/task-stats.md +0 -89
- package/skills/te-engage/references/update-channel-status.md +0 -32
- package/skills/te-engage/references/update-config-channel-status.md +0 -32
- package/skills/te-engage/references/validate-flow-node-config.md +0 -52
- package/skills/te-engage/references/whitelist-list.md +0 -19
- package/skills/te-meta/SKILL.md +0 -71
- package/skills/te-meta/references/list-metrics.md +0 -33
- package/skills/te-meta/references/list-project-mark-times.md +0 -33
- package/skills/te-shared/SKILL.md +0 -115
|
@@ -1,163 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: dataops-flow-create
|
|
3
|
-
version: 1.0.0
|
|
4
|
-
description: "任务流创建与配置:创建任务流、添加任务节点、配置依赖关系、保存任务定义、配置调度、测试、发布到生产。触发关键词:创建任务流、新建工作流、配置调度、添加任务节点、发布、cron、定时执行。"
|
|
5
|
-
metadata:
|
|
6
|
-
requires:
|
|
7
|
-
bins: ["te-cli"]
|
|
8
|
-
---
|
|
9
|
-
|
|
10
|
-
# DataOps 任务流创建与配置
|
|
11
|
-
|
|
12
|
-
> **前置条件:** 阅读 [`te-dataops/SKILL.md`](../SKILL.md) 了解通用规则。
|
|
13
|
-
|
|
14
|
-
使用 `dataops_flow` 子命令管理任务流生命周期。
|
|
15
|
-
|
|
16
|
-
**任务流生命周期:创建 → 配置节点 → 配置调度 → DEV 环境测试 → 发布到 PROD → 上线调度**
|
|
17
|
-
|
|
18
|
-
---
|
|
19
|
-
|
|
20
|
-
## 完整任务流创建流程
|
|
21
|
-
|
|
22
|
-
按顺序执行以下步骤,从零创建一个可上线的任务流。
|
|
23
|
-
|
|
24
|
-
### Step 1: 创建任务流
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
te-cli dataops_flow +create_flow --spaceCode "${spaceCode}" \
|
|
28
|
-
--flowName "每日ETL流程" --remark "处理用户数据" --confirmed true
|
|
29
|
-
# 返回 flowCode,后续步骤都需要
|
|
30
|
-
```
|
|
31
|
-
|
|
32
|
-
### Step 2: 创建任务节点(可多次调用)
|
|
33
|
-
|
|
34
|
-
```bash
|
|
35
|
-
# 创建 SQL 任务
|
|
36
|
-
te-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
|
|
37
|
-
--flowCode "${flowCode}" --taskName "处理用户数据" --taskType "TRINO_SQL" --confirmed true
|
|
38
|
-
# 返回 taskCode
|
|
39
|
-
|
|
40
|
-
# 创建带前置依赖的任务
|
|
41
|
-
te-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
|
|
42
|
-
--flowCode "${flowCode}" --taskName "导出结果" --taskType "TRINO_SQL" \
|
|
43
|
-
--preTaskCode "${upstreamTaskCode}" --confirmed true
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
**任务节点类型:**
|
|
47
|
-
|
|
48
|
-
| 类型 | 说明 | 典型用途 |
|
|
49
|
-
|------|------|---------|
|
|
50
|
-
| TRINO_SQL | Trino SQL 查询 | 数据加工、ETL |
|
|
51
|
-
| SHELL | Shell 脚本 | 系统命令、文件操作 |
|
|
52
|
-
| FLOW_CHECK | 流程检查 | 检查上游任务流是否完成 |
|
|
53
|
-
| TASK_CHECK | 任务检查 | 检查指定任务是否完成 |
|
|
54
|
-
| OFFLINE_SYNC | 离线同步 | 集成方案数据同步(需 syncId) |
|
|
55
|
-
| APP_SYNC | 应用同步 | 应用表数据同步(需 syncId) |
|
|
56
|
-
| PLACE_HOLDER | 占位符 | 流程控制、逻辑分组 |
|
|
57
|
-
|
|
58
|
-
### Step 3: 保存任务定义(配置 SQL/Shell 内容)
|
|
59
|
-
|
|
60
|
-
```bash
|
|
61
|
-
# SQL 任务
|
|
62
|
-
te-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
|
|
63
|
-
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
64
|
-
--taskContentJson '{"sql":"SELECT * FROM dwd_user","repoCode":"te_etl","catalog":"hive","schema":"ws_default_dev"}' \
|
|
65
|
-
--failRetryTimes 3 --failRetryInterval 30 --timeout 60 --confirmed true
|
|
66
|
-
|
|
67
|
-
# Shell 任务
|
|
68
|
-
te-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
|
|
69
|
-
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
70
|
-
--taskContentJson '{"script":"#!/bin/bash\necho hello"}' \
|
|
71
|
-
--confirmed true
|
|
72
|
-
```
|
|
73
|
-
|
|
74
|
-
### Step 4: 校验 SQL(推荐,仅 TRINO_SQL 类型)
|
|
75
|
-
|
|
76
|
-
```bash
|
|
77
|
-
te-cli dataops_flow +validate_task_sql --spaceCode "${spaceCode}" \
|
|
78
|
-
--flowCode "${flowCode}" --taskCode "${taskCode}" \
|
|
79
|
-
--sql "SELECT * FROM dwd_user" --repoCode "te_etl" \
|
|
80
|
-
--catalog "hive" --schema "ws_default_dev"
|
|
81
|
-
# 检查 valid=true,否则查看 message 和 errorPosition
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
### Step 5: 添加任务依赖关系(DAG 连线)
|
|
85
|
-
|
|
86
|
-
```bash
|
|
87
|
-
te-cli dataops_flow +add_task_relation --spaceCode "${spaceCode}" \
|
|
88
|
-
--flowCode "${flowCode}" --preTaskCode "${upstreamTaskCode}" --taskCode "${downstreamTaskCode}" \
|
|
89
|
-
--confirmed true
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
### Step 6: 配置调度
|
|
93
|
-
|
|
94
|
-
```bash
|
|
95
|
-
# CRON 表达式定时执行
|
|
96
|
-
te-cli dataops_flow +save_schedule_config --spaceCode "${spaceCode}" \
|
|
97
|
-
--flowCode "${flowCode}" --scheduled 1 --scheduleType "CRON" \
|
|
98
|
-
--cron "0 0 2 * * ?" --confirmed true
|
|
99
|
-
# 示例:0 0 2 * * ? = 每天凌晨2点
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
**CRON 格式**(6 字段):`秒 分 时 日 月 周`
|
|
103
|
-
- `0 0 */4 * * ?` — 每 4 小时
|
|
104
|
-
- `0 30 8 * * 1-5` — 工作日 8:30
|
|
105
|
-
|
|
106
|
-
### Step 7: DEV 环境测试
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
# 手动触发执行
|
|
110
|
-
te-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
111
|
-
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
112
|
-
|
|
113
|
-
# 查看执行状态
|
|
114
|
-
te-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
115
|
-
|
|
116
|
-
# 查看任务日志(从 DAG 获取 bsTaskInstanceId 和 taskCode)
|
|
117
|
-
te-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${instanceId}" --taskCode "${taskCode}"
|
|
118
|
-
|
|
119
|
-
# 根据日志调整配置后,重复测试
|
|
120
|
-
```
|
|
121
|
-
|
|
122
|
-
### Step 8: 发布到生产
|
|
123
|
-
|
|
124
|
-
```bash
|
|
125
|
-
te-cli dataops_flow +release_flow --spaceCode "${spaceCode}" \
|
|
126
|
-
--flowCode "${flowCode}" --confirmed true
|
|
127
|
-
# 发布后 PROD 调度立即生效,不会中断运行中的实例
|
|
128
|
-
```
|
|
129
|
-
|
|
130
|
-
---
|
|
131
|
-
|
|
132
|
-
## 命令速查
|
|
133
|
-
|
|
134
|
-
| 命令 | 用途 | 关键 Flags |
|
|
135
|
-
|------|------|-----------|
|
|
136
|
-
| `+create_flow` | 创建任务流 | `--spaceCode` `--flowName` `--remark` `--confirmed` |
|
|
137
|
-
| `+create_task` | 创建任务节点 | `--spaceCode` `--flowCode` `--taskName` `--taskType` `--syncId` `--preTaskCode` `--confirmed` |
|
|
138
|
-
| `+save_task_definition` | 保存任务定义 | `--spaceCode` `--flowCode` `--taskCode` `--taskContentJson` `--failRetryTimes` `--failRetryInterval` `--timeout` `--confirmed` |
|
|
139
|
-
| `+validate_task_sql` | 校验 SQL | `--spaceCode` `--flowCode` `--taskCode` `--sql` `--repoCode` `--catalog` `--schema` |
|
|
140
|
-
| `+add_task_relation` | 添加依赖 | `--spaceCode` `--flowCode` `--preTaskCode` `--taskCode` `--confirmed` |
|
|
141
|
-
| `+save_schedule_config` | 配置调度 | `--spaceCode` `--flowCode` `--scheduled` `--scheduleType` `--cron` `--confirmed` |
|
|
142
|
-
| `+get_flow_dag` | 查看 DAG 结构 | `--spaceCode` `--flowCode` `--env` |
|
|
143
|
-
| `+get_task_params` | 查看任务参数 | `--spaceCode` `--flowCode` `--taskCode` `--env` |
|
|
144
|
-
| `+execute_flow` | 手动执行 | `--spaceCode` `--flowCode` `--env` `--confirmed` |
|
|
145
|
-
| `+get_execute_dag` | 查看运行态 DAG | `--spaceCode` `--executeId` |
|
|
146
|
-
| `+get_task_instance_log` | 查看任务日志 | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
|
|
147
|
-
| `+release_flow` | 发布到生产 | `--spaceCode` `--flowCode` `--confirmed` |
|
|
148
|
-
|
|
149
|
-
## 参数说明
|
|
150
|
-
|
|
151
|
-
- **参数引用**: 任务中通过 `${paramKey}` 引用空间参数(如 `${ws_run_date}`)
|
|
152
|
-
- **环境**: `DEV`(默认)| `PROD`
|
|
153
|
-
- **taskContentJson**: JSON 字符串,格式因 taskType 而异:
|
|
154
|
-
- TRINO_SQL: `{"sql":"SELECT ...","repoCode":"xxx","catalog":"hive","schema":"db"}`
|
|
155
|
-
- SHELL: `{"script":"#!/bin/bash\n..."}`
|
|
156
|
-
|
|
157
|
-
## 注意事项
|
|
158
|
-
|
|
159
|
-
1. **flowCode 获取**: 通过 `+list_flows` 获取(见 `dataops-flow-monitor` Skill)
|
|
160
|
-
2. **taskCode 获取**: 通过 `+create_task` 返回或 `+get_flow_dag` 查询
|
|
161
|
-
3. **发布影响**: 将 DEV 所有变更同步到 PROD,PROD 调度立即生效
|
|
162
|
-
4. **SQL 校验**: 保存 TRINO_SQL 任务前建议先校验
|
|
163
|
-
5. **不能创建循环依赖**
|
|
@@ -1,118 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: dataops-flow-monitor
|
|
3
|
-
version: 1.0.0
|
|
4
|
-
description: "任务流执行与监控:查看执行实例、运行态 DAG、任务日志、手动执行/停止任务流。触发关键词:执行任务流、运行实例、监控、日志、停止、DAG、排错、实例、flow instance。"
|
|
5
|
-
metadata:
|
|
6
|
-
requires:
|
|
7
|
-
bins: ["te-cli"]
|
|
8
|
-
---
|
|
9
|
-
|
|
10
|
-
# DataOps 任务流执行与监控
|
|
11
|
-
|
|
12
|
-
> **前置条件:** 阅读 [`te-dataops/SKILL.md`](../SKILL.md) 了解通用规则。
|
|
13
|
-
|
|
14
|
-
使用 `dataops_flow` 子命令执行和监控任务流。
|
|
15
|
-
|
|
16
|
-
**核心概念:**
|
|
17
|
-
- **executeId** — flow_* 工具使用的执行记录 ID(通过 `+list_flow_instances` 获取)
|
|
18
|
-
- **flowInstanceId** — operations_* 工具使用的运维实例 ID(两者不互通,不可混用)
|
|
19
|
-
- **实例列表默认查询 PROD 环境**(与其他工具默认 DEV 不同)
|
|
20
|
-
|
|
21
|
-
---
|
|
22
|
-
|
|
23
|
-
## 工作流 A:查找并查看任务流执行状态
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
# Step 1: 搜索任务流(获取 flowCode)
|
|
27
|
-
te-cli dataops_flow +list_flows --spaceCode "${spaceCode}" --keyword "etl"
|
|
28
|
-
|
|
29
|
-
# Step 2: 查看执行实例列表(默认 PROD,按时间倒序)
|
|
30
|
-
te-cli dataops_flow +list_flow_instances --spaceCode "${spaceCode}" \
|
|
31
|
-
--flowCode "${flowCode}" --env "PROD" --size 30
|
|
32
|
-
|
|
33
|
-
# Step 3: 查看运行态 DAG(含各节点执行状态)
|
|
34
|
-
te-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
35
|
-
|
|
36
|
-
# Step 4: 查看执行记录详情
|
|
37
|
-
te-cli dataops_flow +get_execute_record --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
---
|
|
41
|
-
|
|
42
|
-
## 工作流 B:排查任务失败
|
|
43
|
-
|
|
44
|
-
```bash
|
|
45
|
-
# Step 1: 获取运行态 DAG,定位失败节点
|
|
46
|
-
te-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
47
|
-
# 从返回中找到 status=FAILED 的节点,获取其 bsTaskInstanceId 和 taskCode
|
|
48
|
-
|
|
49
|
-
# Step 2: 查看失败任务的日志
|
|
50
|
-
te-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${bsTaskInstanceId}" \
|
|
51
|
-
--taskCode "${taskCode}" --limit 100
|
|
52
|
-
|
|
53
|
-
# Step 3: 查看任务定义(确认 SQL/配置)
|
|
54
|
-
te-cli dataops_flow +get_task_params --spaceCode "${spaceCode}" \
|
|
55
|
-
--flowCode "${flowCode}" --taskCode "${taskCode}"
|
|
56
|
-
|
|
57
|
-
# Step 4: 修复后重新执行
|
|
58
|
-
te-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
59
|
-
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
---
|
|
63
|
-
|
|
64
|
-
## 工作流 C:手动执行与停止
|
|
65
|
-
|
|
66
|
-
```bash
|
|
67
|
-
# 手动触发执行(默认 DEV,可指定 PROD)
|
|
68
|
-
te-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
|
|
69
|
-
--flowCode "${flowCode}" --env "DEV" --confirmed true
|
|
70
|
-
# 返回 executeId
|
|
71
|
-
|
|
72
|
-
# 监控执行过程
|
|
73
|
-
te-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
|
|
74
|
-
|
|
75
|
-
# 必要时停止执行(不可撤销)
|
|
76
|
-
te-cli dataops_flow +stop_flow --spaceCode "${spaceCode}" \
|
|
77
|
-
--flowCode "${flowCode}" --executeId "${executeId}" --confirmed true
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
---
|
|
81
|
-
|
|
82
|
-
## 工作流 D:查看任务流结构
|
|
83
|
-
|
|
84
|
-
```bash
|
|
85
|
-
# 查看任务流定义态 DAG(静态结构、依赖关系)
|
|
86
|
-
te-cli dataops_flow +get_flow_dag --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
87
|
-
|
|
88
|
-
# 查看任务流基本信息
|
|
89
|
-
te-cli dataops_flow +get_flow_detail --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
90
|
-
|
|
91
|
-
# 查看调度配置
|
|
92
|
-
te-cli dataops_flow +get_schedule_config --spaceCode "${spaceCode}" --flowCode "${flowCode}"
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
---
|
|
96
|
-
|
|
97
|
-
## 命令速查
|
|
98
|
-
|
|
99
|
-
| 命令 | 用途 | 关键 Flags |
|
|
100
|
-
|------|------|-----------|
|
|
101
|
-
| `+list_flows` | 搜索任务流 | `--spaceCode` `--keyword` |
|
|
102
|
-
| `+get_flow_detail` | 任务流信息 | `--spaceCode` `--flowCode` `--env` |
|
|
103
|
-
| `+get_flow_dag` | 定义态 DAG | `--spaceCode` `--flowCode` `--env` |
|
|
104
|
-
| `+get_schedule_config` | 调度配置 | `--spaceCode` `--flowCode` `--env` |
|
|
105
|
-
| `+execute_flow` | 手动执行 | `--spaceCode` `--flowCode` `--env` `--confirmed` |
|
|
106
|
-
| `+stop_flow` | 停止执行 | `--spaceCode` `--flowCode` `--executeId` `--confirmed` |
|
|
107
|
-
| `+list_flow_instances` | 实例列表 | `--spaceCode` `--flowCode` `--env` `--size` |
|
|
108
|
-
| `+get_execute_record` | 执行记录 | `--spaceCode` `--executeId` |
|
|
109
|
-
| `+get_execute_dag` | 运行态 DAG | `--spaceCode` `--executeId` |
|
|
110
|
-
| `+get_task_instance_log` | 任务日志 | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
|
|
111
|
-
| `+get_task_params` | 任务参数 | `--spaceCode` `--flowCode` `--taskCode` `--env` |
|
|
112
|
-
| `+update_flow` | 更新任务流 | `--spaceCode` `--flowCode` `--flowName` `--remark` `--confirmed` |
|
|
113
|
-
|
|
114
|
-
## 参数说明
|
|
115
|
-
|
|
116
|
-
- **env**: `DEV`(开发)| `PROD`(生产,实例列表默认 PROD)
|
|
117
|
-
- **任务状态**: `success` / `failure` / `running` / `waiting`
|
|
118
|
-
- **DAG 区别**: `+get_flow_dag` 返回静态定义结构,`+get_execute_dag` 返回含执行状态的运行态结构
|
|
@@ -1,246 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: dataops-integration
|
|
3
|
-
version: 1.0.0
|
|
4
|
-
description: "数据源与数据集成:创建数据源、创建同步方案、配置字段映射、执行同步、监控同步。触发关键词:数据源、同步、integration、sync、字段映射、数据接入、datasource、MySQL、ClickHouse。"
|
|
5
|
-
metadata:
|
|
6
|
-
requires:
|
|
7
|
-
bins: ["te-cli"]
|
|
8
|
-
---
|
|
9
|
-
|
|
10
|
-
# DataOps 数据源与数据集成
|
|
11
|
-
|
|
12
|
-
> **前置条件:** 阅读 [`te-dataops/SKILL.md`](../SKILL.md) 了解通用规则。
|
|
13
|
-
|
|
14
|
-
使用 `dataops_integration` 子命令管理数据源和同步方案。
|
|
15
|
-
|
|
16
|
-
**核心规则:**
|
|
17
|
-
- 预置仓(te_etl)与非预置仓的配置参数差异很大,必须严格按照模板生成
|
|
18
|
-
- sourceConfig/sinkConfig/channelConfig/fieldsMapping 都是 JSON 字符串
|
|
19
|
-
- 创建同步方案前先测试数据源连接
|
|
20
|
-
|
|
21
|
-
---
|
|
22
|
-
|
|
23
|
-
## 工作流 A:创建数据源
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
# Step 1: 查看支持的数据源组件类型
|
|
27
|
-
te-cli dataops_integration +list_datasource_components
|
|
28
|
-
|
|
29
|
-
# Step 2: 测试连接(临时配置,不保存)
|
|
30
|
-
te-cli dataops_integration +test_datasource_connect --spaceCode "${spaceCode}" \
|
|
31
|
-
--componentName "MySQL" \
|
|
32
|
-
--configValue '{"host":"localhost","port":3306,"username":"root","password":"xxx","database":"test"}'
|
|
33
|
-
|
|
34
|
-
# Step 3: 创建数据源
|
|
35
|
-
te-cli dataops_integration +add_datasource --spaceCode "${spaceCode}" \
|
|
36
|
-
--componentName "MySQL" --dataSourceName "生产MySQL" \
|
|
37
|
-
--sharedConfig true \
|
|
38
|
-
--envJsonList '[{"host":"localhost","port":3306,"username":"root","password":"xxx"}]' \
|
|
39
|
-
--confirmed true
|
|
40
|
-
```
|
|
41
|
-
|
|
42
|
-
---
|
|
43
|
-
|
|
44
|
-
## 工作流 B:创建同步方案(完整流程)
|
|
45
|
-
|
|
46
|
-
### Step 1: 查看可用数据源
|
|
47
|
-
|
|
48
|
-
```bash
|
|
49
|
-
# 查看空间所有数据源
|
|
50
|
-
te-cli dataops_integration +list_space_datasources --spaceCode "${spaceCode}"
|
|
51
|
-
|
|
52
|
-
# 查看可用于同步的数据源(按 source/sink 分类)
|
|
53
|
-
te-cli dataops_integration +list_sync_datasources --spaceCode "${spaceCode}"
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
### Step 2: 浏览源表结构
|
|
57
|
-
|
|
58
|
-
```bash
|
|
59
|
-
# 列出数据源下的数据库
|
|
60
|
-
te-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}" \
|
|
61
|
-
--datasourceId "${datasourceId}"
|
|
62
|
-
|
|
63
|
-
# 列出数据库下的表
|
|
64
|
-
te-cli dataops_integration +list_datasource_tables --spaceCode "${spaceCode}" \
|
|
65
|
-
--datasourceId "${datasourceId}" --database "test"
|
|
66
|
-
|
|
67
|
-
# 获取表结构(列定义、分区)
|
|
68
|
-
te-cli dataops_integration +get_table_structure --spaceCode "${spaceCode}" \
|
|
69
|
-
--datasourceId "${datasourceId}" --database "test" --tablePath "users"
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
### Step 3: 创建同步方案
|
|
73
|
-
|
|
74
|
-
**关键:必须严格按照下方 JSON 模板生成参数**
|
|
75
|
-
|
|
76
|
-
```bash
|
|
77
|
-
te-cli dataops_integration +add_sync_solution --spaceCode "${spaceCode}" \
|
|
78
|
-
--syncName "MySQL到预置仓同步" \
|
|
79
|
-
--srcComponent "MySQL" --srcDatasourceId "${mysqlDatasourceId}" \
|
|
80
|
-
--sinkComponent "hive" --sinkDatasourceId "te_etl@TASK_ENGINE_TRINO" \
|
|
81
|
-
--sourceConfig '{"component":"MySQL","datasourceId":"xxx","database":"test","tablePath":"users"}' \
|
|
82
|
-
--sinkConfig '{"component":"hive","datasourceId":"te_etl@TASK_ENGINE_TRINO","database":"","tablePath":"ods_users_mysql","tableType":"PHYSICAL_TABLE","bizClassify":"CURRENT","dbBizType":"TASK_ENV_DB","authedSpace":"","partitionKeys":[],"dataSaveMode":1,"batchSize":20000}' \
|
|
83
|
-
--channelConfig '{"limitType":"0","gatewayConfig":{"engineFlag":"TASK_ENGINE_TRINO","companyId":1,"appDefinition":"APP_GAIA","bizFlag":"BIZ_GAIA_TASK_RELEASE","repoCode":"te_etl","spaceCode":"default"}}' \
|
|
84
|
-
--fieldsMapping '{"mapping":[{"source":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false},"target":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false}}]}' \
|
|
85
|
-
--confirmed true
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
### Step 4: 执行同步
|
|
89
|
-
|
|
90
|
-
```bash
|
|
91
|
-
# 手动执行同步方案
|
|
92
|
-
te-cli dataops_integration +exec_sync_solution --spaceCode "${spaceCode}" \
|
|
93
|
-
--syncId "${syncId}" --confirmed true
|
|
94
|
-
|
|
95
|
-
# 查看执行历史
|
|
96
|
-
te-cli dataops_integration +list_sync_exec_histories --spaceCode "${spaceCode}" \
|
|
97
|
-
--syncId "${syncId}" --limit 20
|
|
98
|
-
|
|
99
|
-
# 查看执行详情
|
|
100
|
-
te-cli dataops_integration +get_sync_exec_info --spaceCode "${spaceCode}" \
|
|
101
|
-
--syncId "${syncId}" --taskId "${taskId}"
|
|
102
|
-
|
|
103
|
-
# 必要时停止执行
|
|
104
|
-
te-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
|
|
105
|
-
--syncId "${syncId}" --taskId "${taskId}" --confirmed true
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
---
|
|
109
|
-
|
|
110
|
-
## JSON 配置模板
|
|
111
|
-
|
|
112
|
-
### 预置仓作为目标端(Sink)
|
|
113
|
-
|
|
114
|
-
```json
|
|
115
|
-
{
|
|
116
|
-
"component": "hive",
|
|
117
|
-
"datasourceId": "te_etl@TASK_ENGINE_TRINO",
|
|
118
|
-
"database": "",
|
|
119
|
-
"tablePath": "ods_users_mysql",
|
|
120
|
-
"tableType": "PHYSICAL_TABLE",
|
|
121
|
-
"bizClassify": "CURRENT",
|
|
122
|
-
"dbBizType": "TASK_ENV_DB",
|
|
123
|
-
"authedSpace": "",
|
|
124
|
-
"partitionKeys": [],
|
|
125
|
-
"dataSaveMode": 1,
|
|
126
|
-
"batchSize": 20000
|
|
127
|
-
}
|
|
128
|
-
```
|
|
129
|
-
|
|
130
|
-
### 非预置仓(如 MySQL)
|
|
131
|
-
|
|
132
|
-
```json
|
|
133
|
-
{
|
|
134
|
-
"component": "MySQL",
|
|
135
|
-
"datasourceId": "ds-uuid-xxx",
|
|
136
|
-
"database": "test",
|
|
137
|
-
"tablePath": "users"
|
|
138
|
-
}
|
|
139
|
-
```
|
|
140
|
-
|
|
141
|
-
### channelConfig(源端或目标端涉及预置仓时必须包含 gatewayConfig)
|
|
142
|
-
|
|
143
|
-
```json
|
|
144
|
-
{
|
|
145
|
-
"limitType": "0",
|
|
146
|
-
"gatewayConfig": {
|
|
147
|
-
"engineFlag": "TASK_ENGINE_TRINO",
|
|
148
|
-
"companyId": 1,
|
|
149
|
-
"appDefinition": "APP_GAIA",
|
|
150
|
-
"bizFlag": "BIZ_GAIA_TASK_RELEASE",
|
|
151
|
-
"repoCode": "te_etl",
|
|
152
|
-
"spaceCode": "default"
|
|
153
|
-
}
|
|
154
|
-
}
|
|
155
|
-
```
|
|
156
|
-
|
|
157
|
-
### fieldsMapping(双列映射)
|
|
158
|
-
|
|
159
|
-
```json
|
|
160
|
-
{
|
|
161
|
-
"mapping": [
|
|
162
|
-
{
|
|
163
|
-
"source": {
|
|
164
|
-
"name": "id",
|
|
165
|
-
"type": "int",
|
|
166
|
-
"manual": false,
|
|
167
|
-
"partitionKey": false,
|
|
168
|
-
"primaryKey": false,
|
|
169
|
-
"shardingKey": false,
|
|
170
|
-
"sortingKey": false,
|
|
171
|
-
"upsertKey": false
|
|
172
|
-
},
|
|
173
|
-
"target": {
|
|
174
|
-
"name": "id",
|
|
175
|
-
"type": "int",
|
|
176
|
-
"manual": false,
|
|
177
|
-
"partitionKey": false,
|
|
178
|
-
"primaryKey": false,
|
|
179
|
-
"shardingKey": false,
|
|
180
|
-
"sortingKey": false,
|
|
181
|
-
"upsertKey": false
|
|
182
|
-
}
|
|
183
|
-
}
|
|
184
|
-
]
|
|
185
|
-
}
|
|
186
|
-
```
|
|
187
|
-
|
|
188
|
-
---
|
|
189
|
-
|
|
190
|
-
## 命令速查
|
|
191
|
-
|
|
192
|
-
| 命令 | 用途 | 关键 Flags |
|
|
193
|
-
|------|------|-----------|
|
|
194
|
-
| `+list_datasource_components` | 支持的组件 | 无 |
|
|
195
|
-
| `+list_space_datasources` | 空间数据源 | `--spaceCode` `--componentName` `--dataSourceName` |
|
|
196
|
-
| `+add_datasource` | 创建数据源 | `--spaceCode` `--componentName` `--dataSourceName` `--sharedConfig` `--envJsonList` `--confirmed` |
|
|
197
|
-
| `+test_datasource_connect` | 测试连接 | `--spaceCode` `--componentName` `--configValue` |
|
|
198
|
-
| `+list_datasource_databases` | 列出数据库 | `--spaceCode` `--datasourceId` |
|
|
199
|
-
| `+list_datasource_tables` | 列出表 | `--spaceCode` `--datasourceId` `--database` |
|
|
200
|
-
| `+get_table_structure` | 表结构 | `--spaceCode` `--datasourceId` `--database` `--tablePath` |
|
|
201
|
-
| `+list_sync_solutions` | 同步方案列表 | `--spaceCode` |
|
|
202
|
-
| `+add_sync_solution` | 创建同步方案 | `--spaceCode` `--syncName` `--srcComponent` `--srcDatasourceId` `--sinkComponent` `--sinkDatasourceId` `--sourceConfig` `--sinkConfig` `--channelConfig` `--fieldsMapping` `--confirmed` |
|
|
203
|
-
| `+exec_sync_solution` | 执行同步 | `--spaceCode` `--syncId` `--confirmed` |
|
|
204
|
-
| `+list_sync_exec_histories` | 执行历史 | `--spaceCode` `--syncId` `--execType` `--limit` |
|
|
205
|
-
| `+get_sync_exec_info` | 执行详情 | `--spaceCode` `--syncId` `--taskId` |
|
|
206
|
-
| `+stop_sync_solution` | 停止同步 | `--spaceCode` `--syncId` `--taskId` `--confirmed` |
|
|
207
|
-
|
|
208
|
-
## 组件条件必填参数
|
|
209
|
-
|
|
210
|
-
部分组件存在根据部署模式不同而变化的条件必填参数,`+list_datasource_components` 的 requiredFields 可能不包含这些参数(它们在 optionalFields 或 importantNotes 中说明)。创建数据源时务必根据用户选择的 mode 补齐对应参数。
|
|
211
|
-
|
|
212
|
-
### MongoDB
|
|
213
|
-
|
|
214
|
-
| mode 值 | 额外必填参数 | 说明 |
|
|
215
|
-
|----------|-------------|------|
|
|
216
|
-
| `single` | 无 | 单节点模式 |
|
|
217
|
-
| `replicaSet` | `replicaSet` | 副本集名称(如 `rs0`),字段名为 `replicaSet` 而非 `replicaSetName` |
|
|
218
|
-
| `sharded` | 无 | 分片集群模式 |
|
|
219
|
-
|
|
220
|
-
**MongoDB 各模式 envJsonList 示例:**
|
|
221
|
-
|
|
222
|
-
单节点:
|
|
223
|
-
```json
|
|
224
|
-
[{"mode":"single","nodes":[{"host":"10.0.0.1","port":"27017"}],"database":"mydb","username":"admin","password":"xxx"}]
|
|
225
|
-
```
|
|
226
|
-
|
|
227
|
-
副本集群(注意 `replicaSet` 必填):
|
|
228
|
-
```json
|
|
229
|
-
[{"mode":"replicaSet","nodes":[{"host":"10.0.0.1","port":"27017"},{"host":"10.0.0.2","port":"27017"}],"database":"mydb","username":"admin","password":"xxx","replicaSet":"rs0"}]
|
|
230
|
-
```
|
|
231
|
-
|
|
232
|
-
分片集群:
|
|
233
|
-
```json
|
|
234
|
-
[{"mode":"sharded","nodes":[{"host":"10.0.0.1","port":"27017"},{"host":"10.0.0.2","port":"27017"}],"database":"mydb","username":"admin","password":"xxx"}]
|
|
235
|
-
```
|
|
236
|
-
|
|
237
|
-
---
|
|
238
|
-
|
|
239
|
-
## 关键规则
|
|
240
|
-
|
|
241
|
-
1. **表名规则**: 写入预置仓时,未指定表名则用 `ods_${来源表名}_${组件名小写}`
|
|
242
|
-
2. **tablePath**: PostgreSQL 用 `模式.表名`,其他组件直接用表名
|
|
243
|
-
3. **预置仓 database 为空**,非预置仓 database 必填
|
|
244
|
-
4. **channelConfig**: 涉及预置仓时必须包含 gatewayConfig
|
|
245
|
-
5. **字段映射**: 每个字段对象必须包含 manual/partitionKey/primaryKey/shardingKey/sortingKey/upsertKey 属性
|
|
246
|
-
6. **条件必填参数**: 部分组件(如 MongoDB)根据 mode 不同有额外必填字段,详见上方「组件条件必填参数」
|
|
@@ -1,107 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: dataops-operations
|
|
3
|
-
version: 1.0.0
|
|
4
|
-
description: "运维操作与补数管理:搜索运维实例、重跑失败任务、停止实例、补数任务管理。触发关键词:运维、监控、补数、重跑、告警、operations、实例、backfill。"
|
|
5
|
-
metadata:
|
|
6
|
-
requires:
|
|
7
|
-
bins: ["te-cli"]
|
|
8
|
-
---
|
|
9
|
-
|
|
10
|
-
# DataOps 运维操作与补数管理
|
|
11
|
-
|
|
12
|
-
> **前置条件:** 阅读 [`te-dataops/SKILL.md`](../SKILL.md) 了解通用规则。
|
|
13
|
-
|
|
14
|
-
使用 `dataops_operations` 子命令进行运维操作。
|
|
15
|
-
|
|
16
|
-
**核心概念:**
|
|
17
|
-
- **flowInstanceId** — operations_* 工具使用的运维实例 ID
|
|
18
|
-
- **executeId** — flow_* 工具使用的执行记录 ID(两者不互通,不可混用)
|
|
19
|
-
- 运维视角关注生产环境的实例管理和补数操作
|
|
20
|
-
|
|
21
|
-
---
|
|
22
|
-
|
|
23
|
-
## 工作流 A:运维实例查询与操作
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
# Step 1: 搜索任务流实例(支持按状态/负责人/时间筛选)
|
|
27
|
-
te-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
|
|
28
|
-
--pageNum 1 --pageSize 20
|
|
29
|
-
|
|
30
|
-
# Step 2: 查看实例下的所有任务节点及执行状态
|
|
31
|
-
te-cli dataops_operations +list_flow_task_instances --spaceCode "${spaceCode}" \
|
|
32
|
-
--flowInstanceId "${flowInstanceId}"
|
|
33
|
-
|
|
34
|
-
# Step 3: 获取任务节点定义详情
|
|
35
|
-
te-cli dataops_operations +get_task_definition --spaceCode "${spaceCode}" \
|
|
36
|
-
--flowInstanceId "${flowInstanceId}" --taskCode "${taskCode}"
|
|
37
|
-
|
|
38
|
-
# Step 4: 重跑失败的任务流实例(从失败节点继续)
|
|
39
|
-
te-cli dataops_operations +rerun_flow_instance --spaceCode "${spaceCode}" \
|
|
40
|
-
--flowInstanceId "${flowInstanceId}" --confirmed true
|
|
41
|
-
|
|
42
|
-
# Step 5: 停止正在运行的实例
|
|
43
|
-
te-cli dataops_operations +stop_flow_instance --spaceCode "${spaceCode}" \
|
|
44
|
-
--flowInstanceId "${flowInstanceId}" --confirmed true
|
|
45
|
-
|
|
46
|
-
# Step 6: 控制实例运行状态(暂停/强制暂停/恢复)
|
|
47
|
-
te-cli dataops_operations +control_flow_instance --spaceCode "${spaceCode}" \
|
|
48
|
-
--flowInstanceId "${flowInstanceId}" --action "PAUSE" --confirmed true
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
---
|
|
52
|
-
|
|
53
|
-
## 工作流 B:补数任务管理
|
|
54
|
-
|
|
55
|
-
补数任务用于批量重跑历史数据。
|
|
56
|
-
|
|
57
|
-
```bash
|
|
58
|
-
# Step 1: 查询补数任务列表
|
|
59
|
-
te-cli dataops_operations +list_flow_jobs --spaceCode "${spaceCode}" \
|
|
60
|
-
--pageNum 1 --pageSize 20
|
|
61
|
-
|
|
62
|
-
# Step 2: 获取补数任务详情(配置、日期范围、并发设置)
|
|
63
|
-
te-cli dataops_operations +get_flow_job_detail --spaceCode "${spaceCode}" \
|
|
64
|
-
--jobId "${jobId}"
|
|
65
|
-
|
|
66
|
-
# Step 3: 获取补数任务的执行计划列表(每个日期的状态)
|
|
67
|
-
te-cli dataops_operations +list_flow_job_plans --spaceCode "${spaceCode}" \
|
|
68
|
-
--jobId "${jobId}"
|
|
69
|
-
|
|
70
|
-
# Step 4: 停止正在执行的补数任务
|
|
71
|
-
te-cli dataops_operations +stop_flow_job --spaceCode "${spaceCode}" \
|
|
72
|
-
--jobId "${jobId}" --confirmed true
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
---
|
|
76
|
-
|
|
77
|
-
## 工作流 C:任务节点实例查询
|
|
78
|
-
|
|
79
|
-
```bash
|
|
80
|
-
# 搜索任务节点实例(支持按状态/类型/时间筛选)
|
|
81
|
-
te-cli dataops_operations +search_task_instances --spaceCode "${spaceCode}" \
|
|
82
|
-
--pageNum 1 --pageSize 20
|
|
83
|
-
```
|
|
84
|
-
|
|
85
|
-
---
|
|
86
|
-
|
|
87
|
-
## 命令速查
|
|
88
|
-
|
|
89
|
-
| 命令 | 用途 | 关键 Flags |
|
|
90
|
-
|------|------|-----------|
|
|
91
|
-
| `+search_flow_instances` | 搜索任务流实例 | `--spaceCode` `--pageNum` `--pageSize` |
|
|
92
|
-
| `+list_flow_task_instances` | 实例任务节点 | `--spaceCode` `--flowInstanceId` |
|
|
93
|
-
| `+get_task_definition` | 任务节点定义 | `--spaceCode` `--flowInstanceId` `--taskCode` |
|
|
94
|
-
| `+rerun_flow_instance` | 重跑实例 | `--spaceCode` `--flowInstanceId` `--confirmed` |
|
|
95
|
-
| `+stop_flow_instance` | 停止实例 | `--spaceCode` `--flowInstanceId` `--confirmed` |
|
|
96
|
-
| `+control_flow_instance` | 控制实例状态 | `--spaceCode` `--flowInstanceId` `--action` `--confirmed` |
|
|
97
|
-
| `+search_task_instances` | 搜索任务实例 | `--spaceCode` `--pageNum` `--pageSize` |
|
|
98
|
-
| `+list_flow_jobs` | 补数任务列表 | `--spaceCode` `--pageNum` `--pageSize` |
|
|
99
|
-
| `+get_flow_job_detail` | 补数任务详情 | `--spaceCode` `--jobId` |
|
|
100
|
-
| `+list_flow_job_plans` | 补数执行计划 | `--spaceCode` `--jobId` |
|
|
101
|
-
| `+stop_flow_job` | 停止补数任务 | `--spaceCode` `--jobId` `--confirmed` |
|
|
102
|
-
|
|
103
|
-
## 参数说明
|
|
104
|
-
|
|
105
|
-
- **action**: `PAUSE`(暂停)| `FORCE_PAUSE`(强制暂停)| `RESUME`(恢复)
|
|
106
|
-
- **实例状态**: `RUNNING` / `SUCCESS` / `FAILURE` / `WAITING` / `STOPPED`
|
|
107
|
-
- **flowInstanceId vs executeId**: 运维工具用 flowInstanceId,flow 工具用 executeId,不可混用
|