@thinkingai/ae-cli 1.0.1 → 1.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -13
- package/dist/auth-BQC5DBE7.js +19 -0
- package/dist/auth-DXKYLMO7.js +81 -0
- package/dist/chunk-3CDLNXTD.js +218 -0
- package/dist/chunk-53FNNCZW.js +174 -0
- package/dist/chunk-EBFUVWIZ.js +127 -0
- package/dist/chunk-ZYXIGPG7.js +148 -0
- package/dist/client-T4VODLA6.js +16 -0
- package/dist/community-FVDAWGS4.js +684 -0
- package/dist/config-V7AXKDMB.js +242 -0
- package/dist/index.js +16 -15
- package/dist/raw-G3GKHAPZ.js +61 -0
- package/dist/te-analysis-CHFTDN55.js +871 -0
- package/dist/te-audience-M6LQY6M7.js +350 -0
- package/dist/te-common-BLQ7DVGC.js +144 -0
- package/dist/te-dataops-V4OTKY35.js +1853 -0
- package/dist/te-engage-DXE4R5EW.js +1470 -0
- package/dist/te-meta-FMM7KCVI.js +479 -0
- package/package.json +7 -5
- package/skills/te-analysis/SKILL.md +10 -10
- package/skills/te-analysis/references/build-entity-details-sql.md +10 -10
- package/skills/te-analysis/references/build-event-details-sql.md +10 -10
- package/skills/te-analysis/references/create-alert.md +4 -4
- package/skills/te-analysis/references/create-dashboard.md +4 -4
- package/skills/te-analysis/references/create-or-update-dashboard-note.md +4 -4
- package/skills/te-analysis/references/create-public-access-link.md +4 -4
- package/skills/te-analysis/references/create-report.md +10 -10
- package/skills/te-analysis/references/create-result-cluster.md +4 -4
- package/skills/te-analysis/references/drilldown-user-events.md +6 -6
- package/skills/te-analysis/references/drilldown-users.md +4 -4
- package/skills/te-analysis/references/get-alert-definition-schema.md +3 -3
- package/skills/te-analysis/references/get-alert.md +3 -3
- package/skills/te-analysis/references/get-analysis-query-schema.md +4 -4
- package/skills/te-analysis/references/get-filter-schema.md +3 -3
- package/skills/te-analysis/references/get-groupby-schema.md +3 -3
- package/skills/te-analysis/references/get-report-definition.md +3 -3
- package/skills/te-analysis/references/get-table-columns.md +3 -3
- package/skills/te-analysis/references/list-alerts.md +4 -4
- package/skills/te-analysis/references/list-dashboards.md +4 -4
- package/skills/te-analysis/references/list-public-access-links.md +3 -3
- package/skills/te-analysis/references/list-reports.md +4 -4
- package/skills/te-analysis/references/load-filters.md +4 -4
- package/skills/te-analysis/references/query-adhoc.md +10 -10
- package/skills/te-analysis/references/query-dashboard-detail.md +3 -3
- package/skills/te-analysis/references/query-dashboard-report-data.md +9 -9
- package/skills/te-analysis/references/query-entity-details.md +10 -10
- package/skills/te-analysis/references/query-event-details.md +10 -10
- package/skills/te-analysis/references/query-report-data.md +11 -11
- package/skills/te-analysis/references/update-alert.md +4 -4
- package/skills/te-analysis/references/update-dashboard.md +4 -4
- package/skills/te-analysis/references/update-public-access-link.md +4 -4
- package/skills/te-audience/SKILL.md +5 -5
- package/skills/te-audience/references/create-cluster.md +10 -10
- package/skills/te-audience/references/create-tag.md +10 -10
- package/skills/te-audience/references/get-cluster-definition-schema.md +9 -9
- package/skills/te-audience/references/get-clusters-by-name.md +3 -3
- package/skills/te-audience/references/get-tag-definition-schema.md +3 -3
- package/skills/te-audience/references/get-tags-by-name.md +3 -3
- package/skills/te-audience/references/list-cluster-members.md +4 -4
- package/skills/te-audience/references/list-clusters.md +4 -4
- package/skills/te-audience/references/list-tag-members.md +4 -4
- package/skills/te-audience/references/list-tags.md +4 -4
- package/skills/te-audience/references/refresh-cluster.md +3 -3
- package/skills/te-audience/references/refresh-tag.md +3 -3
- package/skills/te-audience/references/update-cluster.md +9 -9
- package/skills/te-audience/references/update-tag.md +10 -10
- package/skills/te-common/SKILL.md +2 -2
- package/skills/te-common/references/get-resource-url.md +5 -5
- package/skills/te-common/references/list-projects.md +3 -3
- package/skills/te-community/SKILL.md +61 -69
- package/skills/te-community/references/community-activity-analysis.md +43 -43
- package/skills/te-community/references/community-analyzing-official-content.md +47 -47
- package/skills/te-community/references/community-analyzing-theme-comment.md +87 -87
- package/skills/te-community/references/community-character-analysis.md +44 -44
- package/skills/te-community/references/community-daily-report.md +56 -53
- package/skills/te-community/references/community-hottopic-insight.md +81 -87
- package/skills/te-community/references/community-weekly-report.md +54 -53
- package/skills/te-community/references/get_channel_info.md +9 -9
- package/skills/te-community/references/get_comment_tag_analysis.md +15 -15
- package/skills/te-community/references/get_comments_summary.md +15 -15
- package/skills/te-community/references/get_corpus_tags.md +9 -9
- package/skills/te-community/references/get_daily_summary.md +10 -10
- package/skills/te-community/references/get_hot_topics.md +12 -12
- package/skills/te-community/references/get_livestream_analysis.md +9 -9
- package/skills/te-community/references/get_livestream_detail.md +19 -19
- package/skills/te-community/references/get_livestream_list.md +13 -13
- package/skills/te-community/references/get_livestream_overview.md +11 -11
- package/skills/te-community/references/get_livestream_room_metrics.md +12 -12
- package/skills/te-community/references/get_livestream_rooms.md +12 -12
- package/skills/te-community/references/get_overview_metrics.md +12 -12
- package/skills/te-community/references/get_post_detail.md +21 -21
- package/skills/te-community/references/get_risk_content.md +23 -23
- package/skills/te-community/references/get_sentiment_overview.md +16 -16
- package/skills/te-community/references/get_tag_trends.md +13 -13
- package/skills/te-community/references/get_topic_detail.md +12 -12
- package/skills/te-community/references/search_posts.md +26 -26
- package/skills/te-dataops/SKILL.md +390 -392
- package/skills/te-dataops/references/dataops-flow-create.md +87 -87
- package/skills/te-dataops/references/dataops-flow-monitor.md +65 -65
- package/skills/te-dataops/references/dataops-integration.md +90 -85
- package/skills/te-dataops/references/dataops-query.md +58 -58
- package/skills/te-dataops/references/dataops-table.md +63 -63
- package/skills/te-engage/SKILL.md +103 -103
- package/skills/te-engage/references/add-approver.md +11 -11
- package/skills/te-engage/references/add-channel.md +34 -34
- package/skills/te-engage/references/approver-list.md +8 -8
- package/skills/te-engage/references/cancel-query-by-request-id.md +10 -10
- package/skills/te-engage/references/channel-detail.md +13 -13
- package/skills/te-engage/references/channel-list.md +15 -15
- package/skills/te-engage/references/config-channel-detail.md +10 -10
- package/skills/te-engage/references/config-channel-list.md +11 -11
- package/skills/te-engage/references/config-item-analysis-report.md +21 -21
- package/skills/te-engage/references/config-item-detail.md +10 -10
- package/skills/te-engage/references/config-item-list.md +8 -8
- package/skills/te-engage/references/config-item-strategy-comparison.md +14 -14
- package/skills/te-engage/references/config-item-trigger-report.md +26 -26
- package/skills/te-engage/references/copy-config-template.md +16 -16
- package/skills/te-engage/references/delete-channel.md +11 -11
- package/skills/te-engage/references/delete-config-channel.md +11 -11
- package/skills/te-engage/references/delete-config-item.md +12 -12
- package/skills/te-engage/references/delete-flow.md +11 -11
- package/skills/te-engage/references/flow-ab-split-node-report.md +29 -29
- package/skills/te-engage/references/flow-detail.md +19 -19
- package/skills/te-engage/references/flow-list.md +8 -8
- package/skills/te-engage/references/flow-node-config-schema.md +11 -11
- package/skills/te-engage/references/flow-node-detail-report.md +30 -30
- package/skills/te-engage/references/flow-node-overview-report.md +27 -27
- package/skills/te-engage/references/flow-process-report.md +32 -32
- package/skills/te-engage/references/manage-flow.md +44 -44
- package/skills/te-engage/references/manage-strategy.md +37 -37
- package/skills/te-engage/references/manage-task.md +20 -20
- package/skills/te-engage/references/modify-flow-base-info.md +14 -14
- package/skills/te-engage/references/save-flow.md +264 -263
- package/skills/te-engage/references/strategy-detail.md +16 -16
- package/skills/te-engage/references/strategy-list.md +10 -10
- package/skills/te-engage/references/task-data-detail.md +47 -47
- package/skills/te-engage/references/task-data-overview.md +23 -23
- package/skills/te-engage/references/task-detail.md +18 -18
- package/skills/te-engage/references/task-experiment-report.md +26 -26
- package/skills/te-engage/references/task-list.md +35 -35
- package/skills/te-engage/references/task-metric-detail.md +31 -31
- package/skills/te-engage/references/task-stats.md +31 -31
- package/skills/te-engage/references/update-channel-status.md +13 -13
- package/skills/te-engage/references/update-config-channel-status.md +14 -14
- package/skills/te-engage/references/validate-flow-node-config.md +15 -15
- package/skills/te-engage/references/whitelist-list.md +8 -8
- package/skills/te-meta/SKILL.md +5 -5
- package/skills/te-meta/references/batch-create-metadata.md +4 -4
- package/skills/te-meta/references/batch-edit-metadata.md +3 -3
- package/skills/te-meta/references/create-metric.md +12 -12
- package/skills/te-meta/references/create-project-mark-time.md +4 -4
- package/skills/te-meta/references/create-virtual-event.md +10 -10
- package/skills/te-meta/references/create-virtual-property.md +8 -8
- package/skills/te-meta/references/delete-project-mark-times.md +3 -3
- package/skills/te-meta/references/delete-track-items.md +3 -3
- package/skills/te-meta/references/get-metric.md +3 -3
- package/skills/te-meta/references/get-project-config.md +3 -3
- package/skills/te-meta/references/get-track-program.md +3 -3
- package/skills/te-meta/references/list-entities.md +4 -4
- package/skills/te-meta/references/list-events.md +4 -4
- package/skills/te-meta/references/list-metrics.md +4 -4
- package/skills/te-meta/references/list-project-mark-times.md +4 -4
- package/skills/te-meta/references/list-project-users.md +3 -3
- package/skills/te-meta/references/list-properties.md +4 -4
- package/skills/te-meta/references/save-track-items.md +3 -3
- package/skills/te-meta/references/update-metric.md +10 -10
- package/skills/te-meta/references/update-project-mark-time.md +4 -4
- package/skills/te-shared/SKILL.md +8 -8
- package/skills/te-community/references/community-release-analysis.md +0 -79
- package/skills/te-dataops/references/dataops-operations.md +0 -107
|
@@ -1,39 +1,39 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dataops-integration
|
|
3
3
|
version: 1.0.0
|
|
4
|
-
description: "
|
|
4
|
+
description: "Datasource and data integration: create datasources, create sync solutions, configure field mapping, execute sync, monitor sync. Trigger keywords: datasource, sync, integration, sync, field mapping, data ingestion, datasource, MySQL, ClickHouse."
|
|
5
5
|
metadata:
|
|
6
6
|
requires:
|
|
7
|
-
bins: ["
|
|
7
|
+
bins: ["ae-cli"]
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
-
# DataOps
|
|
10
|
+
# DataOps Datasource and Data Integration
|
|
11
11
|
|
|
12
|
-
>
|
|
12
|
+
> **Prerequisites:** Read [`te-dataops/SKILL.md`](../SKILL.md) for general rules.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
Use the `dataops_integration` subcommand to manage datasources and sync solutions.
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
-
|
|
18
|
-
- sourceConfig/sinkConfig/channelConfig/fieldsMapping
|
|
19
|
-
-
|
|
16
|
+
**Core Rules:**
|
|
17
|
+
- Configuration parameters differ significantly between preset repositories (te_etl) and non-preset repositories, must strictly follow templates
|
|
18
|
+
- sourceConfig/sinkConfig/channelConfig/fieldsMapping are all JSON strings
|
|
19
|
+
- Test datasource connection before creating sync solution
|
|
20
20
|
|
|
21
21
|
---
|
|
22
22
|
|
|
23
|
-
##
|
|
23
|
+
## Workflow A: Create Datasource
|
|
24
24
|
|
|
25
25
|
```bash
|
|
26
|
-
# Step 1:
|
|
27
|
-
|
|
26
|
+
# Step 1: View supported datasource component types
|
|
27
|
+
ae-cli dataops_integration +list_datasource_components
|
|
28
28
|
|
|
29
|
-
# Step 2:
|
|
30
|
-
|
|
29
|
+
# Step 2: Test connection (temporary configuration, not saved)
|
|
30
|
+
ae-cli dataops_integration +test_datasource_connect --spaceCode "${spaceCode}" \
|
|
31
31
|
--componentName "MySQL" \
|
|
32
32
|
--configValue '{"host":"localhost","port":3306,"username":"root","password":"xxx","database":"test"}'
|
|
33
33
|
|
|
34
|
-
# Step 3:
|
|
35
|
-
|
|
36
|
-
--componentName "MySQL" --dataSourceName "
|
|
34
|
+
# Step 3: Create datasource
|
|
35
|
+
ae-cli dataops_integration +add_datasource --spaceCode "${spaceCode}" \
|
|
36
|
+
--componentName "MySQL" --dataSourceName "Production MySQL" \
|
|
37
37
|
--sharedConfig true \
|
|
38
38
|
--envJsonList '[{"host":"localhost","port":3306,"username":"root","password":"xxx"}]' \
|
|
39
39
|
--confirmed true
|
|
@@ -41,41 +41,41 @@ te-cli dataops_integration +add_datasource --spaceCode "${spaceCode}" \
|
|
|
41
41
|
|
|
42
42
|
---
|
|
43
43
|
|
|
44
|
-
##
|
|
44
|
+
## Workflow B: Create Sync Solution (Complete Process)
|
|
45
45
|
|
|
46
|
-
### Step 1:
|
|
46
|
+
### Step 1: View Available Datasources
|
|
47
47
|
|
|
48
48
|
```bash
|
|
49
|
-
#
|
|
50
|
-
|
|
49
|
+
# View all datasources in workspace
|
|
50
|
+
ae-cli dataops_integration +list_space_datasources --spaceCode "${spaceCode}"
|
|
51
51
|
|
|
52
|
-
#
|
|
53
|
-
|
|
52
|
+
# View datasources available for sync (categorized by source/sink)
|
|
53
|
+
ae-cli dataops_integration +list_sync_datasources --spaceCode "${spaceCode}"
|
|
54
54
|
```
|
|
55
55
|
|
|
56
|
-
### Step 2:
|
|
56
|
+
### Step 2: Browse Source Table Structure
|
|
57
57
|
|
|
58
58
|
```bash
|
|
59
|
-
#
|
|
60
|
-
|
|
59
|
+
# List databases under datasource
|
|
60
|
+
ae-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}" \
|
|
61
61
|
--datasourceId "${datasourceId}"
|
|
62
62
|
|
|
63
|
-
#
|
|
64
|
-
|
|
63
|
+
# List tables under database
|
|
64
|
+
ae-cli dataops_integration +list_datasource_tables --spaceCode "${spaceCode}" \
|
|
65
65
|
--datasourceId "${datasourceId}" --database "test"
|
|
66
66
|
|
|
67
|
-
#
|
|
68
|
-
|
|
67
|
+
# Get table structure (column definitions, partitions)
|
|
68
|
+
ae-cli dataops_integration +get_table_structure --spaceCode "${spaceCode}" \
|
|
69
69
|
--datasourceId "${datasourceId}" --database "test" --tablePath "users"
|
|
70
70
|
```
|
|
71
71
|
|
|
72
|
-
### Step 3:
|
|
72
|
+
### Step 3: Create Sync Solution
|
|
73
73
|
|
|
74
|
-
|
|
74
|
+
**Key: Must strictly follow JSON templates below to generate parameters**
|
|
75
75
|
|
|
76
76
|
```bash
|
|
77
|
-
|
|
78
|
-
--syncName "MySQL
|
|
77
|
+
ae-cli dataops_integration +add_sync_solution --spaceCode "${spaceCode}" \
|
|
78
|
+
--syncName "MySQL to Preset Repository Sync" \
|
|
79
79
|
--srcComponent "MySQL" --srcDatasourceId "${mysqlDatasourceId}" \
|
|
80
80
|
--sinkComponent "hive" --sinkDatasourceId "te_etl@TASK_ENGINE_TRINO" \
|
|
81
81
|
--sourceConfig '{"component":"MySQL","datasourceId":"xxx","database":"test","tablePath":"users"}' \
|
|
@@ -85,31 +85,31 @@ te-cli dataops_integration +add_sync_solution --spaceCode "${spaceCode}" \
|
|
|
85
85
|
--confirmed true
|
|
86
86
|
```
|
|
87
87
|
|
|
88
|
-
### Step 4:
|
|
88
|
+
### Step 4: Execute Sync
|
|
89
89
|
|
|
90
90
|
```bash
|
|
91
|
-
#
|
|
92
|
-
|
|
91
|
+
# Manually execute sync solution
|
|
92
|
+
ae-cli dataops_integration +exec_sync_solution --spaceCode "${spaceCode}" \
|
|
93
93
|
--syncId "${syncId}" --confirmed true
|
|
94
94
|
|
|
95
|
-
#
|
|
96
|
-
|
|
95
|
+
# View execution history
|
|
96
|
+
ae-cli dataops_integration +list_sync_exec_histories --spaceCode "${spaceCode}" \
|
|
97
97
|
--syncId "${syncId}" --limit 20
|
|
98
98
|
|
|
99
|
-
#
|
|
100
|
-
|
|
99
|
+
# View execution details
|
|
100
|
+
ae-cli dataops_integration +get_sync_exec_info --spaceCode "${spaceCode}" \
|
|
101
101
|
--syncId "${syncId}" --taskId "${taskId}"
|
|
102
102
|
|
|
103
|
-
#
|
|
104
|
-
|
|
103
|
+
# Stop execution if necessary
|
|
104
|
+
ae-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
|
|
105
105
|
--syncId "${syncId}" --taskId "${taskId}" --confirmed true
|
|
106
106
|
```
|
|
107
107
|
|
|
108
108
|
---
|
|
109
109
|
|
|
110
|
-
## JSON
|
|
110
|
+
## JSON Configuration Templates
|
|
111
111
|
|
|
112
|
-
###
|
|
112
|
+
### Preset Repository as Target (Sink)
|
|
113
113
|
|
|
114
114
|
```json
|
|
115
115
|
{
|
|
@@ -127,7 +127,7 @@ te-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
|
|
|
127
127
|
}
|
|
128
128
|
```
|
|
129
129
|
|
|
130
|
-
###
|
|
130
|
+
### Non-Preset Repository (e.g., MySQL)
|
|
131
131
|
|
|
132
132
|
```json
|
|
133
133
|
{
|
|
@@ -138,7 +138,7 @@ te-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
|
|
|
138
138
|
}
|
|
139
139
|
```
|
|
140
140
|
|
|
141
|
-
### channelConfig
|
|
141
|
+
### channelConfig (must include gatewayConfig when source or target involves preset repository)
|
|
142
142
|
|
|
143
143
|
```json
|
|
144
144
|
{
|
|
@@ -154,7 +154,7 @@ te-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
|
|
|
154
154
|
}
|
|
155
155
|
```
|
|
156
156
|
|
|
157
|
-
### fieldsMapping
|
|
157
|
+
### fieldsMapping (bidirectional column mapping)
|
|
158
158
|
|
|
159
159
|
```json
|
|
160
160
|
{
|
|
@@ -187,60 +187,65 @@ te-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
|
|
|
187
187
|
|
|
188
188
|
---
|
|
189
189
|
|
|
190
|
-
##
|
|
191
|
-
|
|
192
|
-
|
|
|
193
|
-
|
|
194
|
-
| `+list_datasource_components` |
|
|
195
|
-
| `+
|
|
196
|
-
| `+add_datasource` |
|
|
197
|
-
| `+
|
|
198
|
-
| `+
|
|
199
|
-
| `+
|
|
200
|
-
| `+
|
|
201
|
-
| `+
|
|
202
|
-
| `+
|
|
203
|
-
| `+
|
|
204
|
-
| `+
|
|
205
|
-
| `+
|
|
206
|
-
| `+
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
190
|
+
## Command Quick Reference
|
|
191
|
+
|
|
192
|
+
| Command | Purpose | Key Flags |
|
|
193
|
+
|---------|---------|-----------|
|
|
194
|
+
| `+list_datasource_components` | List component types | None |
|
|
195
|
+
| `+test_datasource_connect` | Test connection | `--spaceCode` `--componentName` `--configValue` |
|
|
196
|
+
| `+add_datasource` | Create datasource | `--spaceCode` `--componentName` `--dataSourceName` `--sharedConfig` `--envJsonList` `--confirmed` |
|
|
197
|
+
| `+modify_datasource` | Modify datasource | `--spaceCode` `--dataSourceName` `--dataSourceRemark` `--sharedConfig` `--envJsonList` `--confirmed` |
|
|
198
|
+
| `+online_datasource` | Online datasource | `--spaceCode` `--dataSourceNames` `--confirmed` |
|
|
199
|
+
| `+list_space_datasources` | List datasources | `--spaceCode` `--componentName` `--dataSourceName` |
|
|
200
|
+
| `+list_sync_datasources` | Sync datasources | `--spaceCode` `--env` |
|
|
201
|
+
| `+list_datasource_databases` | List databases | `--spaceCode` `--datasourceId` `--env` |
|
|
202
|
+
| `+list_datasource_tables` | List tables | `--spaceCode` `--datasourceId` `--database` `--env` |
|
|
203
|
+
| `+get_table_structure` | Table structure | `--spaceCode` `--datasourceId` `--database` `--tablePath` `--env` |
|
|
204
|
+
| `+add_sync_solution` | Create sync solution | `--spaceCode` `--syncName` `--srcComponent` `--srcDatasourceId` `--sinkComponent` `--sinkDatasourceId` `--sourceConfig` `--sinkConfig` `--channelConfig` `--fieldsMapping` `--confirmed` |
|
|
205
|
+
| `+save_sync_solution` | Update sync solution | `--spaceCode` `--syncId` `--syncName` `--sourceConfig` `--sinkConfig` `--channelConfig` `--fieldsMapping` `--confirmed` |
|
|
206
|
+
| `+list_sync_solutions` | List sync solutions | `--spaceCode` |
|
|
207
|
+
| `+get_sync_detail` | Sync solution details | `--spaceCode` `--syncId` `--withParams` |
|
|
208
|
+
| `+exec_sync_solution` | Execute sync | `--spaceCode` `--syncId` `--execParams` `--confirmed` |
|
|
209
|
+
| `+list_sync_exec_histories` | Execution history | `--spaceCode` `--syncId` `--execType` `--limit` |
|
|
210
|
+
| `+get_sync_exec_info` | Execution details | `--spaceCode` `--syncId` `--taskId` |
|
|
211
|
+
| `+stop_sync_solution` | Stop sync | `--spaceCode` `--syncId` `--taskId` `--confirmed` |
|
|
212
|
+
|
|
213
|
+
## Component Conditional Required Parameters
|
|
214
|
+
|
|
215
|
+
Some components have conditional required parameters that vary based on deployment mode. The requiredFields from `+list_datasource_components` may not include these parameters (they are explained in optionalFields or importantNotes). When creating datasources, be sure to supplement corresponding parameters based on the user's selected mode.
|
|
211
216
|
|
|
212
217
|
### MongoDB
|
|
213
218
|
|
|
214
|
-
| mode
|
|
215
|
-
|
|
216
|
-
| `single` |
|
|
217
|
-
| `replicaSet` | `replicaSet` |
|
|
218
|
-
| `sharded` |
|
|
219
|
+
| mode value | Additional required parameters | Description |
|
|
220
|
+
|------------|-------------------------------|-------------|
|
|
221
|
+
| `single` | None | Single node mode |
|
|
222
|
+
| `replicaSet` | `replicaSet` | Replica set name (e.g., `rs0`), field name is `replicaSet` not `replicaSetName` |
|
|
223
|
+
| `sharded` | None | Sharded cluster mode |
|
|
219
224
|
|
|
220
|
-
**MongoDB
|
|
225
|
+
**MongoDB envJsonList examples for each mode:**
|
|
221
226
|
|
|
222
|
-
|
|
227
|
+
Single node:
|
|
223
228
|
```json
|
|
224
229
|
[{"mode":"single","nodes":[{"host":"10.0.0.1","port":"27017"}],"database":"mydb","username":"admin","password":"xxx"}]
|
|
225
230
|
```
|
|
226
231
|
|
|
227
|
-
|
|
232
|
+
Replica set (note `replicaSet` is required):
|
|
228
233
|
```json
|
|
229
234
|
[{"mode":"replicaSet","nodes":[{"host":"10.0.0.1","port":"27017"},{"host":"10.0.0.2","port":"27017"}],"database":"mydb","username":"admin","password":"xxx","replicaSet":"rs0"}]
|
|
230
235
|
```
|
|
231
236
|
|
|
232
|
-
|
|
237
|
+
Sharded cluster:
|
|
233
238
|
```json
|
|
234
239
|
[{"mode":"sharded","nodes":[{"host":"10.0.0.1","port":"27017"},{"host":"10.0.0.2","port":"27017"}],"database":"mydb","username":"admin","password":"xxx"}]
|
|
235
240
|
```
|
|
236
241
|
|
|
237
242
|
---
|
|
238
243
|
|
|
239
|
-
##
|
|
244
|
+
## Key Rules
|
|
240
245
|
|
|
241
|
-
1.
|
|
242
|
-
2. **tablePath**: PostgreSQL
|
|
243
|
-
3.
|
|
244
|
-
4. **channelConfig**:
|
|
245
|
-
5.
|
|
246
|
-
6.
|
|
246
|
+
1. **Table name rule**: When writing to preset repository, if table name not specified, use `ods_${source_table_name}_${component_name_lowercase}`
|
|
247
|
+
2. **tablePath**: PostgreSQL uses `schema.table_name`, other components use table name directly
|
|
248
|
+
3. **Preset repository database is empty**, non-preset repository database is required
|
|
249
|
+
4. **channelConfig**: Must include gatewayConfig when involving preset repository
|
|
250
|
+
5. **Field mapping**: Each field object must include manual/partitionKey/primaryKey/shardingKey/sortingKey/upsertKey properties
|
|
251
|
+
6. **Conditional required parameters**: Some components (e.g., MongoDB) have additional required fields based on mode, see "Component Conditional Required Parameters" above
|
|
@@ -1,111 +1,111 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dataops-query
|
|
3
3
|
version: 1.0.0
|
|
4
|
-
description: "
|
|
4
|
+
description: "Data exploration and SQL query: browse data warehouse metadata, search tables, execute SQL queries, get results. Trigger keywords: query, SQL, data exploration, search tables, view table structure, browse data, IDE, catalog, select."
|
|
5
5
|
metadata:
|
|
6
6
|
requires:
|
|
7
|
-
bins: ["
|
|
7
|
+
bins: ["ae-cli"]
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
-
# DataOps
|
|
10
|
+
# DataOps Data Exploration and SQL Query
|
|
11
11
|
|
|
12
|
-
>
|
|
12
|
+
> **Prerequisites:** Read [`te-dataops/SKILL.md`](../SKILL.md) for general rules.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
Use the `dataops_ide` subcommand for data exploration and SQL queries.
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
**Core Rule: IDE only allows query operations, prohibits creating/modifying/deleting data tables (use dataops_datatable for task table DDL).**
|
|
17
17
|
|
|
18
18
|
---
|
|
19
19
|
|
|
20
|
-
##
|
|
20
|
+
## Workflow A: Browse Data Warehouse Metadata
|
|
21
21
|
|
|
22
|
-
|
|
22
|
+
Explore data warehouse structure step by step: repository → catalog → schema → table → column.
|
|
23
23
|
|
|
24
24
|
```bash
|
|
25
|
-
# Step 1:
|
|
26
|
-
|
|
25
|
+
# Step 1: List available repositories
|
|
26
|
+
ae-cli dataops_ide +list_repos --spaceCode "${spaceCode}"
|
|
27
27
|
|
|
28
|
-
# Step 2:
|
|
29
|
-
|
|
28
|
+
# Step 2: List all catalogs and their schemas in repository
|
|
29
|
+
ae-cli dataops_ide +list_catalogs --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}"
|
|
30
30
|
|
|
31
|
-
# Step 3:
|
|
32
|
-
|
|
31
|
+
# Step 3: List tables/views under schema
|
|
32
|
+
ae-cli dataops_ide +list_tables --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
|
|
33
33
|
--catalog "${catalog}" --schema "${schema}" --isView false --pageNum 1 --pageSize 100
|
|
34
34
|
|
|
35
|
-
# Step 4:
|
|
36
|
-
|
|
35
|
+
# Step 4: Get detailed table metadata (column definitions, DDL, partitions, row count)
|
|
36
|
+
ae-cli dataops_ide +get_table_detail --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
|
|
37
37
|
--catalog "${catalog}" --schema "${schema}" --tableName "${tableName}" \
|
|
38
38
|
--engineType "TASK_ENGINE_TRINO" --isView false
|
|
39
39
|
```
|
|
40
40
|
|
|
41
41
|
---
|
|
42
42
|
|
|
43
|
-
##
|
|
43
|
+
## Workflow B: Search and View Tables
|
|
44
44
|
|
|
45
|
-
|
|
45
|
+
When you don't know the exact table location, search by keyword.
|
|
46
46
|
|
|
47
47
|
```bash
|
|
48
|
-
#
|
|
49
|
-
|
|
48
|
+
# Fuzzy search tables across catalog/schema
|
|
49
|
+
ae-cli dataops_ide +search_tables --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
|
|
50
50
|
--searchKey "user" --size 20
|
|
51
51
|
|
|
52
|
-
#
|
|
53
|
-
|
|
52
|
+
# Fuzzy search column names across tables
|
|
53
|
+
ae-cli dataops_ide +search_columns --spaceCode "${spaceCode}" --repoCode "${repoCode}" \
|
|
54
54
|
--searchKey "user_id" --tables '[{"catalog":"hive","isView":false,"schema":"ws_default_dev","tableName":"dwd_user"}]' \
|
|
55
55
|
--engineType "TASK_ENGINE_TRINO" --pageNum 1 --pageSize 100
|
|
56
56
|
|
|
57
|
-
#
|
|
58
|
-
|
|
57
|
+
# Generate SELECT SQL based on table and column names
|
|
58
|
+
ae-cli dataops_ide +generate_sql --spaceCode "${spaceCode}" --repoCode "${repoCode}" \
|
|
59
59
|
--catalog "hive" --schema "ws_default_dev" --tableName "dwd_user" \
|
|
60
60
|
--engineType "TASK_ENGINE_TRINO" --selectColumns '["user_id","user_name","age"]'
|
|
61
61
|
```
|
|
62
62
|
|
|
63
63
|
---
|
|
64
64
|
|
|
65
|
-
##
|
|
65
|
+
## Workflow C: Execute SQL Query (Complete Async Flow)
|
|
66
66
|
|
|
67
|
-
|
|
67
|
+
This is the most important workflow, containing 3 steps: execute → poll progress → get results.
|
|
68
68
|
|
|
69
69
|
```bash
|
|
70
|
-
# Step 1:
|
|
71
|
-
|
|
70
|
+
# Step 1: Execute SQL asynchronously (returns requestId)
|
|
71
|
+
ae-cli dataops_ide +execute_sql --spaceCode "${spaceCode}" --repoCode "te_etl" \
|
|
72
72
|
--sql "SELECT * FROM hive.ws_default_dev.dwd_user LIMIT 10" \
|
|
73
73
|
--engineType "TASK_ENGINE_TRINO" --confirmed true
|
|
74
74
|
|
|
75
|
-
# Step 2:
|
|
76
|
-
# state
|
|
77
|
-
|
|
75
|
+
# Step 2: Query execution progress (use requestId from Step 1, recommend 2-5 second polling)
|
|
76
|
+
# state values: QUEUED / RUNNING / FINISHED / FAILED / CANCELLED
|
|
77
|
+
ae-cli dataops_ide +get_query_progress --requestId "${requestId}"
|
|
78
78
|
|
|
79
|
-
# Step 3: state=FINISHED
|
|
80
|
-
# recordId
|
|
81
|
-
|
|
79
|
+
# Step 3: After state=FINISHED, get query results
|
|
80
|
+
# recordId obtained from query history or progress info
|
|
81
|
+
ae-cli dataops_ide +get_query_result --spaceCode "${spaceCode}" --connType "SPACE" \
|
|
82
82
|
--repoCode "te_etl" --recordId "${recordId}"
|
|
83
83
|
|
|
84
|
-
#
|
|
85
|
-
|
|
84
|
+
# (Optional) Cancel running query
|
|
85
|
+
ae-cli dataops_ide +cancel_query --requestId "${requestId}" --confirmed true
|
|
86
86
|
```
|
|
87
87
|
|
|
88
88
|
---
|
|
89
89
|
|
|
90
|
-
##
|
|
91
|
-
|
|
92
|
-
|
|
|
93
|
-
|
|
94
|
-
| `+list_repos` |
|
|
95
|
-
| `+list_catalogs` |
|
|
96
|
-
| `+list_tables` |
|
|
97
|
-
| `+get_table_detail` |
|
|
98
|
-
| `+get_schema_info` |
|
|
99
|
-
| `+search_tables` |
|
|
100
|
-
| `+search_columns` |
|
|
101
|
-
| `+generate_sql` |
|
|
102
|
-
| `+execute_sql` |
|
|
103
|
-
| `+get_query_progress` |
|
|
104
|
-
| `+get_query_result` |
|
|
105
|
-
| `+cancel_query` |
|
|
106
|
-
|
|
107
|
-
##
|
|
108
|
-
|
|
109
|
-
- **connType**: `SPACE
|
|
110
|
-
- **engineType**: `TASK_ENGINE_TRINO
|
|
111
|
-
- **isView**: `false`
|
|
90
|
+
## Command Quick Reference
|
|
91
|
+
|
|
92
|
+
| Command | Purpose | Key Flags |
|
|
93
|
+
|---------|---------|-----------|
|
|
94
|
+
| `+list_repos` | List repositories | `--spaceCode` |
|
|
95
|
+
| `+list_catalogs` | List catalog/schema | `--spaceCode` `--connType` `--repoCode` |
|
|
96
|
+
| `+list_tables` | List tables | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` `--isView` `--pageNum` `--pageSize` |
|
|
97
|
+
| `+get_table_detail` | Table metadata | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` `--tableName` `--engineType` `--isView` |
|
|
98
|
+
| `+get_schema_info` | Schema statistics | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` |
|
|
99
|
+
| `+search_tables` | Fuzzy search tables | `--spaceCode` `--connType` `--repoCode` `--searchKey` `--size` |
|
|
100
|
+
| `+search_columns` | Fuzzy search columns | `--spaceCode` `--repoCode` `--searchKey` `--tables` (JSON) `--engineType` `--pageNum` `--pageSize` |
|
|
101
|
+
| `+generate_sql` | Generate SELECT | `--spaceCode` `--repoCode` `--catalog` `--schema` `--tableName` `--engineType` `--selectColumns` (JSON) |
|
|
102
|
+
| `+execute_sql` | Execute SQL | `--spaceCode` `--repoCode` `--sql` `--engineType` `--confirmed` |
|
|
103
|
+
| `+get_query_progress` | Query progress | `--requestId` |
|
|
104
|
+
| `+get_query_result` | Get results | `--spaceCode` `--connType` `--repoCode` `--recordId` |
|
|
105
|
+
| `+cancel_query` | Cancel query | `--requestId` `--confirmed` |
|
|
106
|
+
|
|
107
|
+
## Parameter Notes
|
|
108
|
+
|
|
109
|
+
- **connType**: `SPACE` (data warehouse for daily queries, default) | `ETL` (ETL engine) | `APP` (app warehouse for external services)
|
|
110
|
+
- **engineType**: `TASK_ENGINE_TRINO` (default, interactive queries) | `TASK_ENGINE_STARROCKS` (real-time analytics, high concurrency)
|
|
111
|
+
- **isView**: `false` for physical tables | `true` for views
|