@thinkingai/ae-cli 1.0.1 → 1.0.15

package/skills/ae-dataops/references/dataops-flow-create.md

@@ -0,0 +1,163 @@
+---
+name: dataops-flow-create
+version: 1.0.0
+description: "Flow creation and configuration: create flows, add task nodes, configure dependencies, save task definitions, configure scheduling, test, and release to production. Trigger keywords: create flow, new workflow, configure schedule, add task node, release, cron, scheduled execution."
+metadata:
+  requires:
+    bins: ["ae-cli"]
+---
+
+# DataOps Flow Creation and Configuration
+
+> **Prerequisites:** Read [`ae-dataops/SKILL.md`](../SKILL.md) for general rules.
+
+Use the `dataops_flow` subcommand to manage flow lifecycle.
+
+**Flow Lifecycle: Create → Configure Nodes → Configure Schedule → Test in DEV → Release to PROD → Online Schedule**
+
+---
+
+## Complete Flow Creation Process
+
+Follow these steps in order to create a production-ready flow from scratch.
+
+### Step 1: Create Flow
+
+```bash
+ae-cli dataops_flow +create_flow --spaceCode "${spaceCode}" \
+  --flowName "Daily ETL Process" --remark "Process user data" --confirmed true
+# Returns flowCode, required for subsequent steps
+```
+
+### Step 2: Create Task Nodes (can be called multiple times)
+
+```bash
+# Create SQL task
+ae-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --taskName "Process User Data" --taskType "TRINO_SQL" --confirmed true
+# Returns taskCode
+
+# Create task with upstream dependency
+ae-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --taskName "Export Results" --taskType "TRINO_SQL" \
+  --preTaskCode "${upstreamTaskCode}" --confirmed true
+```
+
+**Task Node Types:**
+
+| Type | Description | Typical Use |
+|------|-------------|-------------|
+| TRINO_SQL | Trino SQL query | Data processing, ETL |
+| SHELL | Shell script | System commands, file operations |
+| FLOW_CHECK | Flow check | Check if upstream flow is complete |
+| TASK_CHECK | Task check | Check if specified task is complete |
+| OFFLINE_SYNC | Offline sync | Integration solution data sync (requires syncId) |
+| APP_SYNC | App sync | App table data sync (requires syncId) |
+| PLACE_HOLDER | Placeholder | Flow control, logical grouping |
+
+### Step 3: Save Task Definition (configure SQL/Shell content)
+
+```bash
+# SQL task
+ae-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --taskCode "${taskCode}" \
+  --taskContentJson '{"sql":"SELECT * FROM dwd_user","repoCode":"te_etl","catalog":"hive","schema":"ws_default_dev"}' \
+  --failRetryTimes 3 --failRetryInterval 30 --timeout 60 --confirmed true
+
+# Shell task
+ae-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --taskCode "${taskCode}" \
+  --taskContentJson '{"script":"#!/bin/bash\necho hello"}' \
+  --confirmed true
+```
+
+### Step 4: Validate SQL (recommended, TRINO_SQL type only)
+
+```bash
+ae-cli dataops_flow +validate_task_sql --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --taskCode "${taskCode}" \
+  --sql "SELECT * FROM dwd_user" --repoCode "te_etl" \
+  --catalog "hive" --schema "ws_default_dev"
+# Check valid=true, otherwise review message and errorPosition
+```
+
+### Step 5: Add Task Dependencies (DAG connections)
+
+```bash
+ae-cli dataops_flow +add_task_relation --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --preTaskCode "${upstreamTaskCode}" --taskCode "${downstreamTaskCode}" \
+  --confirmed true
+```
+
+### Step 6: Configure Schedule
+
+```bash
+# CRON expression for scheduled execution
+ae-cli dataops_flow +save_schedule_config --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --scheduled 1 --scheduleType "CRON" \
+  --cron "0 0 2 * * ?" --confirmed true
+# Example: 0 0 2 * * ? = Daily at 2 AM
+```
+
+**CRON Format** (6 fields): `second minute hour day month weekday`
+- `0 0 */4 * * ?` — Every 4 hours
+- `0 30 8 * * 1-5` — Weekdays at 8:30
+
+### Step 7: Test in DEV Environment
+
+```bash
+# Manually trigger execution
+ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --env "DEV" --confirmed true
+
+# Check execution status
+ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
+
+# View task logs (get bsTaskInstanceId and taskCode from DAG)
+ae-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${instanceId}" --taskCode "${taskCode}"
+
+# Adjust configuration based on logs, then repeat testing
+```
+
+### Step 8: Release to Production
+
+```bash
+ae-cli dataops_flow +release_flow --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --confirmed true
+# After release, PROD schedule takes effect immediately without interrupting running instances
+```
+
+---
+
+## Command Quick Reference
+
+| Command | Purpose | Key Flags |
+|---------|---------|-----------|
+| `+create_flow` | Create flow | `--spaceCode` `--flowName` `--remark` `--confirmed` |
+| `+create_task` | Create task node | `--spaceCode` `--flowCode` `--taskName` `--taskType` `--syncId` `--preTaskCode` `--confirmed` |
+| `+save_task_definition` | Save task definition | `--spaceCode` `--flowCode` `--taskCode` `--taskContentJson` `--failRetryTimes` `--failRetryInterval` `--timeout` `--confirmed` |
+| `+validate_task_sql` | Validate SQL | `--spaceCode` `--flowCode` `--taskCode` `--sql` `--repoCode` `--catalog` `--schema` |
+| `+add_task_relation` | Add dependency | `--spaceCode` `--flowCode` `--preTaskCode` `--taskCode` `--confirmed` |
+| `+save_schedule_config` | Configure schedule | `--spaceCode` `--flowCode` `--scheduled` `--scheduleType` `--cron` `--confirmed` |
+| `+get_flow_dag` | View DAG structure | `--spaceCode` `--flowCode` `--env` |
+| `+get_task_params` | View task parameters | `--spaceCode` `--flowCode` `--taskCode` `--env` |
+| `+execute_flow` | Manual execution | `--spaceCode` `--flowCode` `--env` `--confirmed` |
+| `+get_execute_dag` | View runtime DAG | `--spaceCode` `--executeId` |
+| `+get_task_instance_log` | View task logs | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
+| `+release_flow` | Release to production | `--spaceCode` `--flowCode` `--confirmed` |
+
+## Parameter Notes
+
+- **Parameter Reference**: Reference workspace parameters in tasks using `${paramKey}` (e.g., `${ws_run_date}`)
+- **Environment**: `DEV` (default) | `PROD`
+- **taskContentJson**: JSON string, format varies by taskType:
+  - TRINO_SQL: `{"sql":"SELECT ...","repoCode":"xxx","catalog":"hive","schema":"db"}`
+  - SHELL: `{"script":"#!/bin/bash\n..."}`
+
+## Important Notes
+
+1. **Getting flowCode**: Use `+list_flows` (see `dataops-flow-monitor` Skill)
+2. **Getting taskCode**: Returned by `+create_task` or query via `+get_flow_dag`
+3. **Release Impact**: Syncs all DEV changes to PROD, PROD schedule takes effect immediately
+4. **SQL Validation**: Recommended to validate before saving TRINO_SQL tasks
+5. **Cannot create circular dependencies**

package/skills/ae-dataops/references/dataops-flow-monitor.md

@@ -0,0 +1,118 @@
+---
+name: dataops-flow-monitor
+version: 1.0.0
+description: "Flow execution and monitoring: view execution instances, runtime DAG, task logs, manually execute/stop flows. Trigger keywords: execute flow, running instance, monitor, logs, stop, DAG, troubleshoot, instance, flow instance."
+metadata:
+  requires:
+    bins: ["ae-cli"]
+---
+
+# DataOps Flow Execution and Monitoring
+
+> **Prerequisites:** Read [`ae-dataops/SKILL.md`](../SKILL.md) for general rules.
+
+Use the `dataops_flow` subcommand to execute and monitor flows.
+
+**Core Concepts:**
+- **executeId** — Execution record ID used by flow_* tools (obtained via `+list_flow_instances`)
+- **flowInstanceId** — Operations instance ID used by operations_* tools (these two are not interchangeable)
+- **Instance list queries PROD environment by default** (unlike other tools which default to DEV)
+
+---
+
+## Workflow A: Find and View Flow Execution Status
+
+```bash
+# Step 1: Search flows (get flowCode)
+ae-cli dataops_flow +list_flows --spaceCode "${spaceCode}" --keyword "etl"
+
+# Step 2: View execution instance list (default PROD, sorted by time descending)
+ae-cli dataops_flow +list_flow_instances --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --env "PROD" --size 30
+
+# Step 3: View runtime DAG (with execution status for each node)
+ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
+
+# Step 4: View execution record details
+ae-cli dataops_flow +get_execute_record --spaceCode "${spaceCode}" --executeId "${executeId}"
+```
+
+---
+
+## Workflow B: Troubleshoot Task Failures
+
+```bash
+# Step 1: Get runtime DAG, locate failed node
+ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
+# Find node with status=FAILED, get its bsTaskInstanceId and taskCode
+
+# Step 2: View failed task logs
+ae-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${bsTaskInstanceId}" \
+  --taskCode "${taskCode}" --limit 100
+
+# Step 3: View task definition (confirm SQL/configuration)
+ae-cli dataops_flow +get_task_params --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --taskCode "${taskCode}"
+
+# Step 4: Re-execute after fixing
+ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --env "DEV" --confirmed true
+```
+
+---
+
+## Workflow C: Manual Execution and Stop
+
+```bash
+# Manually trigger execution (default DEV, can specify PROD)
+ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --env "DEV" --confirmed true
+# Returns executeId
+
+# Monitor execution progress
+ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
+
+# Stop execution if necessary (irreversible)
+ae-cli dataops_flow +stop_flow --spaceCode "${spaceCode}" \
+  --flowCode "${flowCode}" --executeId "${executeId}" --confirmed true
+```
+
+---
+
+## Workflow D: View Flow Structure
+
+```bash
+# View flow definition DAG (static structure, dependencies)
+ae-cli dataops_flow +get_flow_dag --spaceCode "${spaceCode}" --flowCode "${flowCode}"
+
+# View flow basic information
+ae-cli dataops_flow +get_flow_detail --spaceCode "${spaceCode}" --flowCode "${flowCode}"
+
+# View schedule configuration
+ae-cli dataops_flow +get_schedule_config --spaceCode "${spaceCode}" --flowCode "${flowCode}"
+```
+
+---
+
+## Command Quick Reference
+
+| Command | Purpose | Key Flags |
+|---------|---------|-----------|
+| `+list_flows` | Search flows | `--spaceCode` `--keyword` |
+| `+get_flow_detail` | Flow information | `--spaceCode` `--flowCode` `--env` |
+| `+get_flow_dag` | Definition DAG | `--spaceCode` `--flowCode` `--env` |
+| `+get_schedule_config` | Schedule configuration | `--spaceCode` `--flowCode` `--env` |
+| `+execute_flow` | Manual execution | `--spaceCode` `--flowCode` `--env` `--confirmed` |
+| `+stop_flow` | Stop execution | `--spaceCode` `--flowCode` `--executeId` `--confirmed` |
+| `+list_flow_instances` | Instance list | `--spaceCode` `--flowCode` `--env` `--size` |
+| `+get_execute_record` | Execution record | `--spaceCode` `--executeId` |
+| `+get_execute_dag` | Runtime DAG | `--spaceCode` `--executeId` |
+| `+get_task_instance_log` | Task logs | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
+| `+get_task_params` | Task parameters | `--spaceCode` `--flowCode` `--taskCode` `--env` |
+| `+update_flow` | Update flow | `--spaceCode` `--flowCode` `--flowName` `--remark` `--confirmed` |
+
+## Parameter Notes
+
+- **env**: `DEV` (development) | `PROD` (production, instance list defaults to PROD)
+- **Task Status**: `success` / `failure` / `running` / `waiting`
+- **DAG Difference**: `+get_flow_dag` returns static definition structure, `+get_execute_dag` returns runtime structure with execution status

package/skills/ae-dataops/references/dataops-integration.md

@@ -0,0 +1,251 @@
+---
+name: dataops-integration
+version: 1.0.0
+description: "Datasource and data integration: create datasources, create sync solutions, configure field mapping, execute sync, monitor sync. Trigger keywords: datasource, sync, integration, sync, field mapping, data ingestion, datasource, MySQL, ClickHouse."
+metadata:
+  requires:
+    bins: ["ae-cli"]
+---
+
+# DataOps Datasource and Data Integration
+
+> **Prerequisites:** Read [`ae-dataops/SKILL.md`](../SKILL.md) for general rules.
+
+Use the `dataops_integration` subcommand to manage datasources and sync solutions.
+
+**Core Rules:**
+- Configuration parameters differ significantly between preset repositories (te_etl) and non-preset repositories, must strictly follow templates
+- sourceConfig/sinkConfig/channelConfig/fieldsMapping are all JSON strings
+- Test datasource connection before creating sync solution
+
+---
+
+## Workflow A: Create Datasource
+
+```bash
+# Step 1: View supported datasource component types
+ae-cli dataops_integration +list_datasource_components
+
+# Step 2: Test connection (temporary configuration, not saved)
+ae-cli dataops_integration +test_datasource_connect --spaceCode "${spaceCode}" \
+  --componentName "MySQL" \
+  --configValue '{"host":"localhost","port":3306,"username":"root","password":"xxx","database":"test"}'
+
+# Step 3: Create datasource
+ae-cli dataops_integration +add_datasource --spaceCode "${spaceCode}" \
+  --componentName "MySQL" --dataSourceName "Production MySQL" \
+  --sharedConfig true \
+  --envJsonList '[{"host":"localhost","port":3306,"username":"root","password":"xxx"}]' \
+  --confirmed true
+```
+
+---
+
+## Workflow B: Create Sync Solution (Complete Process)
+
+### Step 1: View Available Datasources
+
+```bash
+# View all datasources in workspace
+ae-cli dataops_integration +list_space_datasources --spaceCode "${spaceCode}"
+
+# View datasources available for sync (categorized by source/sink)
+ae-cli dataops_integration +list_sync_datasources --spaceCode "${spaceCode}"
+```
+
+### Step 2: Browse Source Table Structure
+
+```bash
+# List databases under datasource
+ae-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}" \
+  --datasourceId "${datasourceId}"
+
+# List tables under database
+ae-cli dataops_integration +list_datasource_tables --spaceCode "${spaceCode}" \
+  --datasourceId "${datasourceId}" --database "test"
+
+# Get table structure (column definitions, partitions)
+ae-cli dataops_integration +get_table_structure --spaceCode "${spaceCode}" \
+  --datasourceId "${datasourceId}" --database "test" --tablePath "users"
+```
+
+### Step 3: Create Sync Solution
+
+**Key: Must strictly follow JSON templates below to generate parameters**
+
+```bash
+ae-cli dataops_integration +add_sync_solution --spaceCode "${spaceCode}" \
+  --syncName "MySQL to Preset Repository Sync" \
+  --srcComponent "MySQL" --srcDatasourceId "${mysqlDatasourceId}" \
+  --sinkComponent "hive" --sinkDatasourceId "te_etl@TASK_ENGINE_TRINO" \
+  --sourceConfig '{"component":"MySQL","datasourceId":"xxx","database":"test","tablePath":"users"}' \
+  --sinkConfig '{"component":"hive","datasourceId":"te_etl@TASK_ENGINE_TRINO","database":"","tablePath":"ods_users_mysql","tableType":"PHYSICAL_TABLE","bizClassify":"CURRENT","dbBizType":"TASK_ENV_DB","authedSpace":"","partitionKeys":[],"dataSaveMode":1,"batchSize":20000}' \
+  --channelConfig '{"limitType":"0","gatewayConfig":{"engineFlag":"TASK_ENGINE_TRINO","companyId":1,"appDefinition":"APP_GAIA","bizFlag":"BIZ_GAIA_TASK_RELEASE","repoCode":"te_etl","spaceCode":"default"}}' \
+  --fieldsMapping '{"mapping":[{"source":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false},"target":{"name":"id","type":"int","manual":false,"partitionKey":false,"primaryKey":false,"shardingKey":false,"sortingKey":false,"upsertKey":false}}]}' \
+  --confirmed true
+```
+
+### Step 4: Execute Sync
+
+```bash
+# Manually execute sync solution
+ae-cli dataops_integration +exec_sync_solution --spaceCode "${spaceCode}" \
+  --syncId "${syncId}" --confirmed true
+
+# View execution history
+ae-cli dataops_integration +list_sync_exec_histories --spaceCode "${spaceCode}" \
+  --syncId "${syncId}" --limit 20
+
+# View execution details
+ae-cli dataops_integration +get_sync_exec_info --spaceCode "${spaceCode}" \
+  --syncId "${syncId}" --taskId "${taskId}"
+
+# Stop execution if necessary
+ae-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
+  --syncId "${syncId}" --taskId "${taskId}" --confirmed true
+```
+
+---
+
+## JSON Configuration Templates
+
+### Preset Repository as Target (Sink)
+
+```json
+{
+  "component": "hive",
+  "datasourceId": "te_etl@TASK_ENGINE_TRINO",
+  "database": "",
+  "tablePath": "ods_users_mysql",
+  "tableType": "PHYSICAL_TABLE",
+  "bizClassify": "CURRENT",
+  "dbBizType": "TASK_ENV_DB",
+  "authedSpace": "",
+  "partitionKeys": [],
+  "dataSaveMode": 1,
+  "batchSize": 20000
+}
+```
+
+### Non-Preset Repository (e.g., MySQL)
+
+```json
+{
+  "component": "MySQL",
+  "datasourceId": "ds-uuid-xxx",
+  "database": "test",
+  "tablePath": "users"
+}
+```
+
+### channelConfig (must include gatewayConfig when source or target involves preset repository)
+
+```json
+{
+  "limitType": "0",
+  "gatewayConfig": {
+    "engineFlag": "TASK_ENGINE_TRINO",
+    "companyId": 1,
+    "appDefinition": "APP_GAIA",
+    "bizFlag": "BIZ_GAIA_TASK_RELEASE",
+    "repoCode": "te_etl",
+    "spaceCode": "default"
+  }
+}
+```
+
+### fieldsMapping (bidirectional column mapping)
+
+```json
+{
+  "mapping": [
+    {
+      "source": {
+        "name": "id",
+        "type": "int",
+        "manual": false,
+        "partitionKey": false,
+        "primaryKey": false,
+        "shardingKey": false,
+        "sortingKey": false,
+        "upsertKey": false
+      },
+      "target": {
+        "name": "id",
+        "type": "int",
+        "manual": false,
+        "partitionKey": false,
+        "primaryKey": false,
+        "shardingKey": false,
+        "sortingKey": false,
+        "upsertKey": false
+      }
+    }
+  ]
+}
+```
+
+---
+
+## Command Quick Reference
+
+| Command | Purpose | Key Flags |
+|---------|---------|-----------|
+| `+list_datasource_components` | List component types | None |
+| `+test_datasource_connect` | Test connection | `--spaceCode` `--componentName` `--configValue` |
+| `+add_datasource` | Create datasource | `--spaceCode` `--componentName` `--dataSourceName` `--sharedConfig` `--envJsonList` `--confirmed` |
+| `+modify_datasource` | Modify datasource | `--spaceCode` `--dataSourceName` `--dataSourceRemark` `--sharedConfig` `--envJsonList` `--confirmed` |
+| `+online_datasource` | Online datasource | `--spaceCode` `--dataSourceNames` `--confirmed` |
+| `+list_space_datasources` | List datasources | `--spaceCode` `--componentName` `--dataSourceName` |
+| `+list_sync_datasources` | Sync datasources | `--spaceCode` `--env` |
+| `+list_datasource_databases` | List databases | `--spaceCode` `--datasourceId` `--env` |
+| `+list_datasource_tables` | List tables | `--spaceCode` `--datasourceId` `--database` `--env` |
+| `+get_table_structure` | Table structure | `--spaceCode` `--datasourceId` `--database` `--tablePath` `--env` |
+| `+add_sync_solution` | Create sync solution | `--spaceCode` `--syncName` `--srcComponent` `--srcDatasourceId` `--sinkComponent` `--sinkDatasourceId` `--sourceConfig` `--sinkConfig` `--channelConfig` `--fieldsMapping` `--confirmed` |
+| `+save_sync_solution` | Update sync solution | `--spaceCode` `--syncId` `--syncName` `--sourceConfig` `--sinkConfig` `--channelConfig` `--fieldsMapping` `--confirmed` |
+| `+list_sync_solutions` | List sync solutions | `--spaceCode` |
+| `+get_sync_detail` | Sync solution details | `--spaceCode` `--syncId` `--withParams` |
+| `+exec_sync_solution` | Execute sync | `--spaceCode` `--syncId` `--execParams` `--confirmed` |
+| `+list_sync_exec_histories` | Execution history | `--spaceCode` `--syncId` `--execType` `--limit` |
+| `+get_sync_exec_info` | Execution details | `--spaceCode` `--syncId` `--taskId` |
+| `+stop_sync_solution` | Stop sync | `--spaceCode` `--syncId` `--taskId` `--confirmed` |
+
+## Component Conditional Required Parameters
+
+Some components have conditional required parameters that vary based on deployment mode. The requiredFields from `+list_datasource_components` may not include these parameters (they are explained in optionalFields or importantNotes). When creating datasources, be sure to supplement corresponding parameters based on the user's selected mode.
+
+### MongoDB
+
+| mode value | Additional required parameters | Description |
+|------------|-------------------------------|-------------|
+| `single` | None | Single node mode |
+| `replicaSet` | `replicaSet` | Replica set name (e.g., `rs0`), field name is `replicaSet` not `replicaSetName` |
+| `sharded` | None | Sharded cluster mode |
+
+**MongoDB envJsonList examples for each mode:**
+
+Single node:
+```json
+[{"mode":"single","nodes":[{"host":"10.0.0.1","port":"27017"}],"database":"mydb","username":"admin","password":"xxx"}]
+```
+
+Replica set (note `replicaSet` is required):
+```json
+[{"mode":"replicaSet","nodes":[{"host":"10.0.0.1","port":"27017"},{"host":"10.0.0.2","port":"27017"}],"database":"mydb","username":"admin","password":"xxx","replicaSet":"rs0"}]
+```
+
+Sharded cluster:
+```json
+[{"mode":"sharded","nodes":[{"host":"10.0.0.1","port":"27017"},{"host":"10.0.0.2","port":"27017"}],"database":"mydb","username":"admin","password":"xxx"}]
+```
+
+---
+
+## Key Rules
+
+1. **Table name rule**: When writing to preset repository, if table name not specified, use `ods_${source_table_name}_${component_name_lowercase}`
+2. **tablePath**: PostgreSQL uses `schema.table_name`, other components use table name directly
+3. **Preset repository database is empty**, non-preset repository database is required
+4. **channelConfig**: Must include gatewayConfig when involving preset repository
+5. **Field mapping**: Each field object must include manual/partitionKey/primaryKey/shardingKey/sortingKey/upsertKey properties
+6. **Conditional required parameters**: Some components (e.g., MongoDB) have additional required fields based on mode, see "Component Conditional Required Parameters" above

package/skills/ae-dataops/references/dataops-query.md

@@ -0,0 +1,111 @@
+---
+name: dataops-query
+version: 1.0.0
+description: "Data exploration and SQL query: browse data warehouse metadata, search tables, execute SQL queries, get results. Trigger keywords: query, SQL, data exploration, search tables, view table structure, browse data, IDE, catalog, select."
+metadata:
+  requires:
+    bins: ["ae-cli"]
+---
+
+# DataOps Data Exploration and SQL Query
+
+> **Prerequisites:** Read [`ae-dataops/SKILL.md`](../SKILL.md) for general rules.
+
+Use the `dataops_ide` subcommand for data exploration and SQL queries.
+
+**Core Rule: IDE only allows query operations, prohibits creating/modifying/deleting data tables (use dataops_datatable for task table DDL).**
+
+---
+
+## Workflow A: Browse Data Warehouse Metadata
+
+Explore data warehouse structure step by step: repository → catalog → schema → table → column.
+
+```bash
+# Step 1: List available repositories
+ae-cli dataops_ide +list_repos --spaceCode "${spaceCode}"
+
+# Step 2: List all catalogs and their schemas in repository
+ae-cli dataops_ide +list_catalogs --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}"
+
+# Step 3: List tables/views under schema
+ae-cli dataops_ide +list_tables --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
+  --catalog "${catalog}" --schema "${schema}" --isView false --pageNum 1 --pageSize 100
+
+# Step 4: Get detailed table metadata (column definitions, DDL, partitions, row count)
+ae-cli dataops_ide +get_table_detail --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
+  --catalog "${catalog}" --schema "${schema}" --tableName "${tableName}" \
+  --engineType "TASK_ENGINE_TRINO" --isView false
+```
+
+---
+
+## Workflow B: Search and View Tables
+
+When you don't know the exact table location, search by keyword.
+
+```bash
+# Fuzzy search tables across catalog/schema
+ae-cli dataops_ide +search_tables --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
+  --searchKey "user" --size 20
+
+# Fuzzy search column names across tables
+ae-cli dataops_ide +search_columns --spaceCode "${spaceCode}" --repoCode "${repoCode}" \
+  --searchKey "user_id" --tables '[{"catalog":"hive","isView":false,"schema":"ws_default_dev","tableName":"dwd_user"}]' \
+  --engineType "TASK_ENGINE_TRINO" --pageNum 1 --pageSize 100
+
+# Generate SELECT SQL based on table and column names
+ae-cli dataops_ide +generate_sql --spaceCode "${spaceCode}" --repoCode "${repoCode}" \
+  --catalog "hive" --schema "ws_default_dev" --tableName "dwd_user" \
+  --engineType "TASK_ENGINE_TRINO" --selectColumns '["user_id","user_name","age"]'
+```
+
+---
+
+## Workflow C: Execute SQL Query (Complete Async Flow)
+
+This is the most important workflow, containing 3 steps: execute → poll progress → get results.
+
+```bash
+# Step 1: Execute SQL asynchronously (returns requestId)
+ae-cli dataops_ide +execute_sql --spaceCode "${spaceCode}" --repoCode "te_etl" \
+  --sql "SELECT * FROM hive.ws_default_dev.dwd_user LIMIT 10" \
+  --engineType "TASK_ENGINE_TRINO" --confirmed true
+
+# Step 2: Query execution progress (use requestId from Step 1, recommend 2-5 second polling)
+# state values: QUEUED / RUNNING / FINISHED / FAILED / CANCELLED
+ae-cli dataops_ide +get_query_progress --requestId "${requestId}"
+
+# Step 3: After state=FINISHED, get query results
+# recordId obtained from query history or progress info
+ae-cli dataops_ide +get_query_result --spaceCode "${spaceCode}" --connType "SPACE" \
+  --repoCode "te_etl" --recordId "${recordId}"
+
+# (Optional) Cancel running query
+ae-cli dataops_ide +cancel_query --requestId "${requestId}" --confirmed true
+```
+
+---
+
+## Command Quick Reference
+
+| Command | Purpose | Key Flags |
+|---------|---------|-----------|
+| `+list_repos` | List repositories | `--spaceCode` |
+| `+list_catalogs` | List catalog/schema | `--spaceCode` `--connType` `--repoCode` |
+| `+list_tables` | List tables | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` `--isView` `--pageNum` `--pageSize` |
+| `+get_table_detail` | Table metadata | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` `--tableName` `--engineType` `--isView` |
+| `+get_schema_info` | Schema statistics | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` |
+| `+search_tables` | Fuzzy search tables | `--spaceCode` `--connType` `--repoCode` `--searchKey` `--size` |
+| `+search_columns` | Fuzzy search columns | `--spaceCode` `--repoCode` `--searchKey` `--tables` (JSON) `--engineType` `--pageNum` `--pageSize` |
+| `+generate_sql` | Generate SELECT | `--spaceCode` `--repoCode` `--catalog` `--schema` `--tableName` `--engineType` `--selectColumns` (JSON) |
+| `+execute_sql` | Execute SQL | `--spaceCode` `--repoCode` `--sql` `--engineType` `--confirmed` |
+| `+get_query_progress` | Query progress | `--requestId` |
+| `+get_query_result` | Get results | `--spaceCode` `--connType` `--repoCode` `--recordId` |
+| `+cancel_query` | Cancel query | `--requestId` `--confirmed` |
+
+## Parameter Notes
+
+- **connType**: `SPACE` (data warehouse for daily queries, default) | `ETL` (ETL engine) | `APP` (app warehouse for external services)
+- **engineType**: `TASK_ENGINE_TRINO` (default, interactive queries) | `TASK_ENGINE_STARROCKS` (real-time analytics, high concurrency)
+- **isView**: `false` for physical tables | `true` for views