@thinkingai/ae-cli 6.0.18 → 6.0.20

package/skills/ae-dataops/SKILL.md

@@ -1,22 +1,27 @@
 ---
 name: ae-dataops
 version: 2.0.0
-description: "AE Data Development and Operations: Data warehouse management, flow orchestration, IDE queries, data integration, operations and backfill management"
+description: "AE Data Development and Operations: Data warehouse management, flow orchestration, IDE queries, and data integration"
+metadata:
+  requires:
+    bins: ["ae-cli"]
 ---
 
 # ae-dataops
 
+> **CRITICAL - This skill is self-contained.** Use the Global AE CLI Rules below; do not require a separate shared skill for DataOps-side tasks.
+
 The AE Data Development and Operations domain provides capabilities for data warehouse management, flow orchestration, IDE SQL queries, data integration, operations and backfill management, including the following subcommands:
 
 | Subcommand | Responsibility | Corresponding Scenario Skill |
 |------------|----------------|------------------------------|
-| `dataops_repo` | Space and resource management | — |
+| `dataops_repo` | Space discovery | — |
 | `dataops_datatable` | Data table and view management | `dataops-table` |
 | `dataops_flow` | Flow creation and orchestration | `dataops-flow-create` |
 | `dataops_flow` | Flow execution and monitoring | `dataops-flow-monitor` |
+| `dataops_operations` | Operations instance search, details, and task logs | `dataops-flow-monitor` |
 | `dataops_ide` | Data exploration and SQL queries | `dataops-query` |
 | `dataops_integration` | Datasource and data integration | `dataops-integration` |
-| `dataops_operations` | Operations and backfill management | `dataops-operations` |
 
 ---
 
@@ -39,11 +44,11 @@ Output and errors:
 
 Safety constraints:
 - Read commands can execute directly after required IDs and references are verified.
-- Write commands require explicit user intent and normally keep the confirmation prompt.
+- Write-command examples omit `--yes`; add `--yes` only when the user explicitly wants to skip the confirmation prompt.
 - Never invent command names, flags, JSON payloads, `project_id`, resource IDs, field names, event names, property names, metric definitions, or date formats. Read the matching command reference and discover real project metadata first.
 - **NEVER fabricate or guess resource names** (reports, dashboards, events, properties, metrics, clusters, tags, alerts). Always use list commands to discover real resources first. If a resource is not found after fuzzy search and full list fallback, explicitly tell the user "resource not found" and stop - do not proceed with fabricated names.
 
-Domains for DataOps: `dataops_repo`, `dataops_datatable`, `dataops_flow`, `dataops_ide`, `dataops_integration`, `dataops_operations`
+Domains for DataOps: `dataops_repo`, `dataops_datatable`, `dataops_flow`, `dataops_operations`, `dataops_ide`, `dataops_integration`
 
 ---
 
@@ -51,22 +56,22 @@ Domains for DataOps: `dataops_repo`, `dataops_datatable`, `dataops_flow`, `datao
 
 You must understand the following key concepts before use, otherwise errors are highly likely.
 
-### ID System (Not Interchangeable)
+### ID System
 
 | ID | Source | Usage Scope |
 |----|--------|-------------|
-| **executeId** | Returned by `dataops_flow +execute_flow` / obtained via `+list_flow_instances` | All execution-related commands in `dataops_flow` |
-| **flowInstanceId** | Obtained via `dataops_operations +search_flow_instances` | All operations commands in `dataops_operations` |
-
-> **executeId and flowInstanceId are not interchangeable.** `dataops_flow` tools use executeId, `dataops_operations` tools use flowInstanceId.
+| **executeId** | Returned by `dataops_flow +execute_flow` | Early stop handle before the scheduler `flowInstanceId` is available |
+| **flowInstanceId** | Returned by `dataops_operations +search_flow_instances` | Operations perspective instance inspection and stop |
 
 ### Environment and Defaults
 
 | Scenario | Default Environment | Description |
 |----------|---------------------|-------------|
 | Most flow/ide/datatable commands | `DEV` | Development environment |
-| `+list_flow_instances` instance list | **PROD** | Operations scenarios default to production |
-| All `dataops_operations` commands | **PROD** | Operations perspective focuses on production |
+| `dataops_operations +search_flow_instances` | Operations instance search | Filter by keyword, execution date, status, and paging |
+| `dataops_operations +get_flow_instance_detail` | Instance detail | Inspect one instance DAG and task statuses |
+| `dataops_operations +get_task_instance_detail` | Task detail/logs | Inspect one task and include logs only when needed |
+| `dataops_operations +stop_flow_instance` | Instance stop | Stop by exactly one of `executeId` or `flowInstanceId` |
 
 ### Schema Naming Rules
 
@@ -79,12 +84,11 @@ You must understand the following key concepts before use, otherwise errors are
 |-----------|--------------|------------|
 | Execute SELECT queries | `dataops_ide` | — |
 | Create/modify/delete data tables (DDL) | `dataops_datatable` | `dataops_ide` |
-| Execute DDL on external datasources | `dataops_integration +execute_sql` | — |
 
 ### Flow Lifecycle
 
 ```
-Create Flow → Create Task Nodes → Save Task Definitions → Configure Dependencies → Configure Schedule → DEV Testing → Release to PROD
+Create DEV Flow → Create/Update DEV SQL Tasks → Configure Dependencies/Schedule → Preview Release → Release to PROD → PROD Manual Execution / Operations Troubleshooting
 ```
 
 ### CRON Format (6 fields)
@@ -109,119 +113,38 @@ Choose the appropriate scenario skill based on user intent to get complete step-
 |-------------|---------------|----------|
 | Create flow, add nodes, configure schedule, release | `dataops-flow-create` | create flow, new workflow, configure schedule, add task node, release, cron, scheduled execution |
 | View execution status, troubleshoot failures, view logs | `dataops-flow-monitor` | execute flow, running instance, monitor, logs, stop, DAG, troubleshoot |
-| Create datasource, configure sync solution, execute sync | `dataops-integration` | datasource, sync, integration, sync, field mapping, data ingestion, MySQL, ClickHouse |
-| Operations instance management, rerun failures, backfill operations | `dataops-operations` | operations, backfill, rerun, alert, operations, backfill |
+| Search operation instances across a space | `dataops-flow-monitor` | operations instance, flow instance search, status statistics, owner statistics |
+| Create datasource, configure sync solution, execute sync | `dataops-integration` | datasource, sync, integration, field mapping, data ingestion, MySQL, ClickHouse, DatabricksJdbc |
 | Browse metadata, search tables, execute SQL queries | `dataops-query` | query, SQL, data exploration, search tables, view table structure, IDE, catalog, select |
-| Create tables, create views, batch create linked views | `dataops-table` | create table, table creation, view, data dictionary, linked view, linkview, DDL |
+| Create tables and views | `dataops-table` | create table, table creation, view, data dictionary, table details, DDL |
 
 ---
 
-## 1. Space and Resource Management
-
-```bash
-# Get space list (entry command, get spaceCode)
-ae-cli dataops_repo +list_spaces
-
-# Get repository list
-ae-cli dataops_repo +list_repos --spaceCode "${spaceCode}"
-
-# Browse repository structure
-ae-cli dataops_repo +list_catalogs --spaceCode "${spaceCode}" --repoCode "te_etl"
-ae-cli dataops_repo +list_schemas --spaceCode "${spaceCode}" --repoCode "te_etl" --catalog "hive"
+## 1. Space Discovery
 
-# View space members
-ae-cli dataops_repo +list_space_members --spaceCode "${spaceCode}"
+`dataops_repo` exposes only one read command. Use it to discover a valid `spaceCode` before calling DataOps commands that require one. It returns `createTime`, `spaceCode`, and `spaceDisplayName`.
 
-# View space parameters
-ae-cli dataops_repo +list_support_params --spaceCode "${spaceCode}"
+- If the user already provided a trusted `spaceCode`, reuse it.
+- If `spaceCode` is unknown, run `+list_spaces` first.
+- If exactly one space is returned, use its `spaceCode`.
+- If multiple spaces are returned and the user intent does not identify one, ask the user which space to use. Do not guess.
 
-# Preview parameter expression
-ae-cli dataops_repo +preview_param_expression --spaceCode "${spaceCode}" --expression '${ws_run_date}'
+```bash
+# List spaces accessible to the current user
+ae-cli dataops_repo +list_spaces
 ```
 
-### Command Quick Reference
-
-| Command | Purpose | Key Flags |
-|---------|---------|-----------|
-| `+list_spaces` | Space list | — |
-| `+list_repos` | Repository list | `--spaceCode` |
-| `+list_catalogs` | Catalog list | `--spaceCode` `--repoCode` |
-| `+list_schemas` | Schema list | `--spaceCode` `--repoCode` `--catalog` |
-| `+list_space_members` | Space members | `--spaceCode` |
-| `+list_support_params` | Space parameters | `--spaceCode` |
-| `+preview_param_expression` | Preview parameter | `--spaceCode` `--expression` |
-| `+list_param_used_flows` | Flows using parameter | `--spaceCode` `--paramKey` |
-
 ---
 
 ## 2. Data Table and View Management
 
-> See `dataops-table` Skill for detailed step-by-step workflows.
-
-### Typical Flow: Search → View → Create
-
-```bash
-# Search tables in data dictionary
-ae-cli dataops_datatable +dict_search_tables --spaceCode "${spaceCode}" --search "user" --maxResults 20
-
-# Get table details (field definitions, DDL, data lineage)
-ae-cli dataops_datatable +get_table_detail --spaceCode "${spaceCode}" --tableName "dwd_user_info"
-
-# Paginated query of registered tables
-ae-cli dataops_datatable +list_tables_by_page --spaceCode "${spaceCode}" --search "user" --pageNum 1 --pageSize 20
-
-# Get hierarchy by repository dimension
-ae-cli dataops_datatable +struct_by_repo --spaceCode "${spaceCode}"
-
-# Create physical table (DEV environment, DDL follows Trino specifications)
-ae-cli dataops_datatable +create_table --spaceCode "${spaceCode}" \
-  --ddl "CREATE TABLE dwd_user_info (user_id VARCHAR, user_name VARCHAR, age INTEGER) WITH (format = 'ORC')" \
-  --repoCode "te_etl" --catalog "hive" --schema "ws_default_dev" --confirmed true
-
-# Create PROD environment (same DDL, change schema to product database)
-ae-cli dataops_datatable +create_table --spaceCode "${spaceCode}" \
-  --ddl "CREATE TABLE dwd_user_info (user_id VARCHAR, user_name VARCHAR, age INTEGER) WITH (format = 'ORC')" \
-  --repoCode "te_etl" --catalog "hive" --schema "ws_default_product" --confirmed true
-
-# Create view
-ae-cli dataops_datatable +create_view --spaceCode "${spaceCode}" \
-  --ddl "CREATE VIEW v_user_info AS SELECT user_id, user_name FROM hive.ws_default_dev.dwd_user_info" \
-  --repoCode "te_etl" --catalog "hive" --schema "ws_default_dev" --confirmed true
-
-# Batch create linked views (async)
-ae-cli dataops_datatable +batch_create --spaceCode "${spaceCode}" \
-  --srcRepoCode "te_etl" --srcCatalog "hive" --srcSchema "ws_default_hidden" \
-  --processType 2 --batchCreationMode 1 --owner "${ownerOpenId}" \
-  --viewInfos '[{"viewName":"v_orders","srcTableName":"ta_event_123","srcTableType":"TABLE"}]' \
-  --confirmed true
-
-# Poll batch creation status (recommend 3-5 second interval)
-ae-cli dataops_datatable +linkview_get_batch_status --spaceCode "${spaceCode}" --batchId "${batchId}"
-```
+Detailed workflow, command flags, examples, and parameter notes live in [`references/dataops-table.md`](references/dataops-table.md).
 
-### Command Quick Reference
-
-| Command | Purpose | Key Flags |
-|---------|---------|-----------|
-| `+dict_search_tables` | Data dictionary search | `--spaceCode` `--search` `--maxResults` |
-| `+get_table_detail` | Table details | `--spaceCode` `--tableName` |
-| `+list_tables_by_page` | Paginated list | `--spaceCode` `--search` `--pageNum` `--pageSize` |
-| `+struct_by_repo` | By repository hierarchy | `--spaceCode` `--search` |
-| `+list_system_tables` | System table list | `--spaceCode` `--projectId`/`--projectName` `--pageNum` `--pageSize` |
-| `+create_table` | Create physical table | `--spaceCode` `--ddl` `--repoCode` `--catalog` `--schema` `--confirmed` |
-| `+create_view` | Create view | `--spaceCode` `--ddl` `--repoCode` `--catalog` `--schema` `--confirmed` |
-| `+batch_create` | Batch create views | `--spaceCode` `--srcRepoCode` `--srcCatalog` `--srcSchema` `--processType` `--batchCreationMode` `--owner` `--viewInfos` `--confirmed` |
-| `+linkview_list_source_tables` | Available source tables | `--spaceCode` `--srcRepoCode` `--srcCatalog` `--srcSchema` |
-| `+linkview_get_batch_status` | Batch status | `--spaceCode` `--batchId` |
-| `+list_batches` | Historical batch records | `--spaceCode` `--status` |
-
-### Parameter Notes
-
-- **schema naming**: DEV environment uses `ws_${spaceCode}_dev`, PROD environment uses `ws_${spaceCode}_product`
-- **processType**: `1`=DEV only | `2`=DEV+PROD
-- **batchCreationMode**: `1`=full sync | `2`=retry failures
-- **Table name rule**: `^[a-z][0-9a-z_]{0,127}$`
-- **DDL specification**: Follow Trino DDL syntax
+Key constraints:
+- Start with `dataops_datatable +dict_search_tables` for visible DataOps catalog discovery.
+- Use `dataops_ide +search_tables` only for raw engine metadata, and `dataops_ide +ide_list_tables` only for known catalog/schema browsing.
+- Create tables/views with `dataops_datatable`, not `dataops_ide`; creation is DEV-only and must be published with `+publish_entity`.
+- DDL follows Trino syntax; current-space view DDL should keep the literal `${env}` placeholder.
 
 ---
 
@@ -229,429 +152,38 @@ ae-cli dataops_datatable +linkview_get_batch_status --spaceCode "${spaceCode}" -
 
 Flow orchestration is divided into two scenario skills: **creation and configuration** and **execution and monitoring**.
 
-### 3a. Creation and Configuration
-
-> See `dataops-flow-create` Skill for detailed 8-step complete process.
-
-**Lifecycle: Create → Configure Nodes → Configure Schedule → DEV Testing → Release to PROD**
-
-```bash
-# Create flow (returns flowCode)
-ae-cli dataops_flow +create_flow --spaceCode "${spaceCode}" \
-  --flowName "Daily ETL Process" --remark "Process user data" --confirmed true
-
-# Create SQL task node (returns taskCode)
-ae-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --taskName "Process User Data" --taskType "TRINO_SQL" --confirmed true
-
-# Create task with upstream dependency
-ae-cli dataops_flow +create_task --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --taskName "Export Results" --taskType "TRINO_SQL" \
-  --preTaskCode "${upstreamTaskCode}" --confirmed true
-
-# Save task definition (SQL task)
-ae-cli dataops_flow +save_task_definition --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --taskCode "${taskCode}" \
-  --taskContentJson '{"sql":"SELECT * FROM dwd_user","repoCode":"te_etl","catalog":"hive","schema":"ws_default_dev"}' \
-  --failRetryTimes 3 --failRetryInterval 30 --timeout 60 --confirmed true
-
-# Validate SQL (recommended before saving)
-ae-cli dataops_flow +validate_task_sql --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --taskCode "${taskCode}" \
-  --sql "SELECT * FROM dwd_user" --repoCode "te_etl" \
-  --catalog "hive" --schema "ws_default_dev"
-
-# Add task dependency (DAG connection)
-ae-cli dataops_flow +add_task_relation --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --preTaskCode "${upstreamTaskCode}" --taskCode "${downstreamTaskCode}" \
-  --confirmed true
-
-# Configure scheduled execution (CRON 6 fields: second minute hour day month weekday)
-ae-cli dataops_flow +save_schedule_config --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --scheduled 1 --scheduleType "CRON" \
-  --cron "0 0 2 * * ?" --confirmed true
-
-# Manual testing in DEV environment
-ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --env "DEV" --confirmed true
-
-# Release to production (PROD schedule takes effect immediately, does not interrupt running instances)
-ae-cli dataops_flow +release_flow --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --confirmed true
-```
-
-**Task Node Types:**
-
-| Type | Description | Typical Use |
-|------|-------------|-------------|
-| TRINO_SQL | Trino SQL query | Data processing, ETL |
-| SHELL | Shell script | System commands, file operations |
-| FLOW_CHECK | Flow check | Check if upstream flow is complete |
-| TASK_CHECK | Task check | Check if specified task is complete |
-| OFFLINE_SYNC | Offline sync | Integration solution data sync (requires syncId) |
-| APP_SYNC | App sync | App table data sync (requires syncId) |
-| PLACE_HOLDER | Placeholder | Flow control, logical grouping |
-
-### 3b. Execution and Monitoring
-
-> See `dataops-flow-monitor` Skill for detailed troubleshooting workflows.
-
-```bash
-# Search flows (get flowCode)
-ae-cli dataops_flow +list_flows --spaceCode "${spaceCode}" --keyword "etl"
-
-# View flow basic information
-ae-cli dataops_flow +get_flow_detail --spaceCode "${spaceCode}" --flowCode "${flowCode}"
-
-# View definition DAG (static structure)
-ae-cli dataops_flow +get_flow_dag --spaceCode "${spaceCode}" --flowCode "${flowCode}"
-
-# View schedule configuration
-ae-cli dataops_flow +get_schedule_config --spaceCode "${spaceCode}" --flowCode "${flowCode}"
-
-# View execution instance list (default PROD environment, sorted by time descending)
-ae-cli dataops_flow +list_flow_instances --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --env "PROD" --size 30
+**Lifecycle: DEV configuration and preview → Release to PROD → PROD manual execution and operations troubleshooting**
 
-# View runtime DAG (with execution status for each node)
-ae-cli dataops_flow +get_execute_dag --spaceCode "${spaceCode}" --executeId "${executeId}"
+Detailed creation/configuration commands live in [`references/dataops-flow-create.md`](references/dataops-flow-create.md). Detailed execution, monitoring, operation instance, task log, and stop commands live in [`references/dataops-flow-monitor.md`](references/dataops-flow-monitor.md).
 
-# View execution record details
-ae-cli dataops_flow +get_execute_record --spaceCode "${spaceCode}" --executeId "${executeId}"
-
-# View task logs (get bsTaskInstanceId and taskCode from DAG)
-ae-cli dataops_flow +get_task_instance_log --bsTaskInstanceId "${instanceId}" \
-  --taskCode "${taskCode}" --limit 100
-
-# View task parameters
-ae-cli dataops_flow +get_task_params --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --taskCode "${taskCode}"
-
-# Manually trigger execution
-ae-cli dataops_flow +execute_flow --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --env "DEV" --confirmed true
-
-# Stop execution (irreversible)
-ae-cli dataops_flow +stop_flow --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --executeId "${executeId}" --confirmed true
-
-# Update flow information
-ae-cli dataops_flow +update_flow --spaceCode "${spaceCode}" \
-  --flowCode "${flowCode}" --flowName "New Name" --remark "New Remark" --confirmed true
-```
-
-### Command Quick Reference (Complete Flow Commands)
-
-| Command | Purpose | Key Flags |
-|---------|---------|-----------|
-| `+create_flow` | Create flow | `--spaceCode` `--flowName` `--remark` `--confirmed` |
-| `+create_task` | Create task node | `--spaceCode` `--flowCode` `--taskName` `--taskType` `--syncId` `--preTaskCode` `--confirmed` |
-| `+save_task_definition` | Save task definition | `--spaceCode` `--flowCode` `--taskCode` `--taskContentJson` `--failRetryTimes` `--failRetryInterval` `--timeout` `--confirmed` |
-| `+validate_task_sql` | Validate SQL | `--spaceCode` `--flowCode` `--taskCode` `--sql` `--repoCode` `--catalog` `--schema` |
-| `+add_task_relation` | Add dependency | `--spaceCode` `--flowCode` `--preTaskCode` `--taskCode` `--confirmed` |
-| `+save_schedule_config` | Configure schedule | `--spaceCode` `--flowCode` `--scheduled` `--scheduleType` `--cron` `--confirmed` |
-| `+get_schedule_config` | View schedule | `--spaceCode` `--flowCode` `--env` |
-| `+list_flows` | Search flows | `--spaceCode` `--keyword` |
-| `+get_flow_detail` | Flow information | `--spaceCode` `--flowCode` `--env` |
-| `+get_flow_dag` | Definition DAG | `--spaceCode` `--flowCode` `--env` |
-| `+update_flow` | Update flow | `--spaceCode` `--flowCode` `--flowName` `--remark` `--confirmed` |
-| `+execute_flow` | Manual execution | `--spaceCode` `--flowCode` `--env` `--confirmed` |
-| `+stop_flow` | Stop execution | `--spaceCode` `--flowCode` `--executeId` `--confirmed` |
-| `+release_flow` | Release to production | `--spaceCode` `--flowCode` `--confirmed` |
-| `+list_flow_instances` | Instance list | `--spaceCode` `--flowCode` `--env` `--size` |
-| `+get_execute_record` | Execution record | `--spaceCode` `--executeId` |
-| `+get_execute_dag` | Runtime DAG | `--spaceCode` `--executeId` |
-| `+get_task_instance_log` | Task logs | `--bsTaskInstanceId` `--taskCode` `--startOffset` `--limit` |
-| `+get_task_params` | Task parameters | `--spaceCode` `--flowCode` `--taskCode` `--env` |
-
-### Parameter Notes
-
-- **Parameter reference**: Reference workspace parameters in tasks using `${paramKey}` (e.g., `${ws_run_date}`)
-- **env**: `DEV` (development, default) | `PROD` (production)
-- **Task status**: `success` / `failure` / `running` / `waiting`
-- **DAG difference**: `+get_flow_dag` returns static definition structure, `+get_execute_dag` returns runtime structure with execution status
-- **taskContentJson format** (varies by taskType):
-  - TRINO_SQL: `{"sql":"SELECT ...","repoCode":"xxx","catalog":"hive","schema":"db"}`
-  - SHELL: `{"script":"#!/bin/bash\n..."}`
+Key constraints:
+- Create and update tasks in DEV, preview/release before PROD execution.
+- `+execute_flow` always runs PROD; it returns `executeId` for early stop.
+- Prefer `flowInstanceId` from operations search for stable inspection and troubleshooting.
+- Reference workspace parameters in task SQL as `${paramKey}`.
 
 ---
 
 ## 4. IDE SQL Queries
 
-> See `dataops-query` Skill for detailed step-by-step workflows.
-
-> **IDE only allows query operations (SELECT), prohibits creating/modifying/deleting data tables. Use `dataops_datatable` for table DDL operations.**
-
-### Typical Flow: Browse Metadata → Search Tables → Execute Queries
-
-```bash
-# List available repositories
-ae-cli dataops_ide +list_repos --spaceCode "${spaceCode}"
-
-# List all catalogs and their schemas in repository
-ae-cli dataops_ide +list_catalogs --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}"
-
-# List tables/views under schema
-ae-cli dataops_ide +list_tables --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
-  --catalog "${catalog}" --schema "${schema}" --isView false --pageNum 1 --pageSize 100
-
-# Get detailed table metadata (column definitions, DDL, partitions, row count)
-ae-cli dataops_ide +get_table_detail --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
-  --catalog "${catalog}" --schema "${schema}" --tableName "${tableName}" \
-  --engineType "TASK_ENGINE_TRINO" --isView false
-
-# Get schema statistics
-ae-cli dataops_ide +get_schema_info --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
-  --catalog "${catalog}" --schema "${schema}"
-
-# Fuzzy search tables across catalog/schema
-ae-cli dataops_ide +search_tables --spaceCode "${spaceCode}" --connType "SPACE" --repoCode "${repoCode}" \
-  --searchKey "user" --size 20
-
-# Fuzzy search column names across tables
-ae-cli dataops_ide +search_columns --spaceCode "${spaceCode}" --repoCode "${repoCode}" \
-  --searchKey "user_id" --tables '[{"catalog":"hive","isView":false,"schema":"ws_default_dev","tableName":"dwd_user"}]' \
-  --engineType "TASK_ENGINE_TRINO" --pageNum 1 --pageSize 100
-
-# Generate SELECT SQL based on table and column names
-ae-cli dataops_ide +generate_sql --spaceCode "${spaceCode}" --repoCode "${repoCode}" \
-  --catalog "hive" --schema "ws_default_dev" --tableName "dwd_user" \
-  --engineType "TASK_ENGINE_TRINO" --selectColumns '["user_id","user_name","age"]'
-```
-
-### SQL Async Query Flow (3 steps: Execute → Poll → Get Results)
-
-```bash
-# Step 1: Execute SQL asynchronously (returns requestId)
-ae-cli dataops_ide +execute_sql --spaceCode "${spaceCode}" --repoCode "te_etl" \
-  --sql "SELECT * FROM hive.ws_default_dev.dwd_user LIMIT 10" \
-  --engineType "TASK_ENGINE_TRINO" --confirmed true
+Detailed metadata browsing, SQL query, async download, and cancel workflows live in [`references/dataops-query.md`](references/dataops-query.md).
 
-# Step 2: Query execution progress (recommend 2-5 second polling)
-# state: QUEUED / RUNNING / FINISHED / FAILED / CANCELLED
-ae-cli dataops_ide +get_query_progress --requestId "${requestId}"
-
-# Step 3: Get results after state=FINISHED
-ae-cli dataops_ide +get_query_result --spaceCode "${spaceCode}" --connType "SPACE" \
-  --repoCode "te_etl" --recordId "${recordId}"
-
-# (Optional) Cancel running query
-ae-cli dataops_ide +cancel_query --requestId "${requestId}" --confirmed true
-```
-
-### Command Quick Reference
-
-| Command | Purpose | Key Flags |
-|---------|---------|-----------|
-| `+list_repos` | List repositories | `--spaceCode` |
-| `+list_catalogs` | List catalog/schema | `--spaceCode` `--connType` `--repoCode` |
-| `+list_tables` | List tables | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` `--isView` `--pageNum` `--pageSize` |
-| `+get_table_detail` | Table metadata | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` `--tableName` `--engineType` `--isView` |
-| `+get_schema_info` | Schema statistics | `--spaceCode` `--connType` `--repoCode` `--catalog` `--schema` |
-| `+search_tables` | Fuzzy search tables | `--spaceCode` `--connType` `--repoCode` `--searchKey` `--size` |
-| `+search_columns` | Fuzzy search columns | `--spaceCode` `--repoCode` `--searchKey` `--tables` (JSON) `--engineType` `--pageNum` `--pageSize` |
-| `+generate_sql` | Generate SELECT | `--spaceCode` `--repoCode` `--catalog` `--schema` `--tableName` `--engineType` `--selectColumns` (JSON) |
-| `+execute_sql` | Execute SQL | `--spaceCode` `--repoCode` `--sql` `--engineType` `--confirmed` |
-| `+get_query_progress` | Query progress | `--requestId` |
-| `+get_query_result` | Get results | `--spaceCode` `--connType` `--repoCode` `--recordId` |
-| `+cancel_query` | Cancel query | `--requestId` `--confirmed` |
-
-### Parameter Notes
-
-- **connType**: `SPACE` (data warehouse for daily queries, default) | `ETL` (ETL engine) | `APP` (app warehouse for external services)
-- **engineType**: `TASK_ENGINE_TRINO` (default, interactive queries) | `TASK_ENGINE_STARROCKS` (real-time analytics, high concurrency)
-- **isView**: `false` for physical tables | `true` for views
+Key constraints:
+- IDE is query-only; create/modify/delete tables with `dataops_datatable`.
+- Prefer `dataops_datatable +dict_search_tables` for table discovery unless raw engine metadata or schema browsing is required.
+- SQL query submission creates a download task; rows are not returned inline.
 
 ---
 
 ## 5. Data Integration
 
-> See `dataops-integration` Skill for detailed step-by-step workflows and JSON configuration templates.
-
-> **Sync solution creation involves complex JSON configurations (sourceConfig/sinkConfig/channelConfig/fieldsMapping), be sure to refer to complete templates in `dataops-integration` Skill.**
-
-### Typical Flow: Create Datasource → Browse Source Tables → Create Sync Solution → Execute
-
-```bash
-# View supported datasource component types
-ae-cli dataops_integration +list_datasource_components
-
-# Test datasource connection
-ae-cli dataops_integration +test_datasource_connect --spaceCode "${spaceCode}" \
-  --componentName "MySQL" \
-  --configValue '{"host":"localhost","port":3306,"username":"root","password":"xxx","database":"test"}'
-
-# Create datasource
-ae-cli dataops_integration +add_datasource --spaceCode "${spaceCode}" \
-  --componentName "MySQL" --dataSourceName "Production MySQL" \
-  --sharedConfig true \
-  --envJsonList '[{"host":"localhost","port":3306,"username":"root","password":"xxx"}]' \
-  --confirmed true
-
-# View all datasources in workspace
-ae-cli dataops_integration +list_space_datasources --spaceCode "${spaceCode}"
-
-# View datasources available for sync (categorized by source/sink)
-ae-cli dataops_integration +list_sync_datasources --spaceCode "${spaceCode}"
-
-# Browse source databases and tables
-ae-cli dataops_integration +list_datasource_databases --spaceCode "${spaceCode}" \
-  --datasourceId "${datasourceId}"
-ae-cli dataops_integration +list_datasource_tables --spaceCode "${spaceCode}" \
-  --datasourceId "${datasourceId}" --database "test"
-
-# Get table structure (column definitions, partitions)
-ae-cli dataops_integration +get_table_structure --spaceCode "${spaceCode}" \
-  --datasourceId "${datasourceId}" --database "test" --tablePath "users"
-
-# Create sync solution (requires complete JSON configuration, see dataops-integration Skill)
-ae-cli dataops_integration +add_sync_solution --spaceCode "${spaceCode}" \
-  --syncName "MySQL to Preset Repository Sync" \
-  --srcComponent "MySQL" --srcDatasourceId "${mysqlDatasourceId}" \
-  --sinkComponent "hive" --sinkDatasourceId "te_etl@TASK_ENGINE_TRINO" \
-  --sourceConfig '...' --sinkConfig '...' --channelConfig '...' --fieldsMapping '...' \
-  --confirmed true
-
-# Execute sync solution
-ae-cli dataops_integration +exec_sync_solution --spaceCode "${spaceCode}" \
-  --syncId "${syncId}" --confirmed true
-
-# View execution history
-ae-cli dataops_integration +list_sync_exec_histories --spaceCode "${spaceCode}" \
-  --syncId "${syncId}" --limit 20
-
-# View execution details
-ae-cli dataops_integration +get_sync_exec_info --spaceCode "${spaceCode}" \
-  --syncId "${syncId}" --taskId "${taskId}"
-
-# Stop sync execution
-ae-cli dataops_integration +stop_sync_solution --spaceCode "${spaceCode}" \
-  --syncId "${syncId}" --taskId "${taskId}" --confirmed true
-```
-
-### JSON Configuration Key Points
-
-When creating sync solutions, the following JSON configurations must be strictly generated according to templates:
-
-| Configuration | Description |
-|---------------|-------------|
-| **sourceConfig** | Source connection information (component, datasource ID, database, table) |
-| **sinkConfig** | Target configuration (preset repository database is empty, non-preset repository database is required) |
-| **channelConfig** | Channel configuration (must include gatewayConfig when involving preset repository) |
-| **fieldsMapping** | Bidirectional field mapping (each field must include manual/partitionKey/primaryKey/shardingKey/sortingKey/upsertKey) |
-
-> See `dataops-integration` Skill for complete JSON templates and conditional required parameters for components like MongoDB.
-
-### Command Quick Reference
-
-| Command | Purpose | Key Flags |
-|---------|---------|-----------|
-| `+list_datasource_components` | Supported components | — |
-| `+list_space_datasources` | Workspace datasources | `--spaceCode` `--componentName` `--dataSourceName` |
-| `+add_datasource` | Create datasource | `--spaceCode` `--componentName` `--dataSourceName` `--sharedConfig` `--envJsonList` `--confirmed` |
-| `+test_datasource_connect` | Test connection | `--spaceCode` `--componentName` `--configValue` |
-| `+modify_datasource` | Modify datasource | `--spaceCode` `--datasourceId` `--confirmed` |
-| `+online_datasource` | Online datasource | `--spaceCode` `--datasourceId` `--confirmed` |
-| `+list_datasource_databases` | List databases | `--spaceCode` `--datasourceId` |
-| `+list_datasource_tables` | List tables | `--spaceCode` `--datasourceId` `--database` |
-| `+get_table_structure` | Table structure | `--spaceCode` `--datasourceId` `--database` `--tablePath` |
-| `+list_sync_datasources` | Sync datasources | `--spaceCode` |
-| `+list_sync_solutions` | Sync solution list | `--spaceCode` |
-| `+get_sync_detail` | Sync solution details | `--spaceCode` `--syncId` |
-| `+add_sync_solution` | Create sync solution | `--spaceCode` `--syncName` `--srcComponent` `--srcDatasourceId` `--sinkComponent` `--sinkDatasourceId` `--sourceConfig` `--sinkConfig` `--channelConfig` `--fieldsMapping` `--confirmed` |
-| `+save_sync_solution` | Update sync solution | `--spaceCode` `--syncId` `--confirmed` |
-| `+exec_sync_solution` | Execute sync | `--spaceCode` `--syncId` `--confirmed` |
-| `+list_sync_exec_histories` | Execution history | `--spaceCode` `--syncId` `--execType` `--limit` |
-| `+get_sync_exec_info` | Execution details | `--spaceCode` `--syncId` `--taskId` |
-| `+stop_sync_solution` | Stop sync | `--spaceCode` `--syncId` `--taskId` `--confirmed` |
-| `+execute_sql` | External DDL execution | `--spaceCode` `--datasourceId` `--database` `--sql` `--confirmed` |
-| `+validate_sql` | Validate SQL | `--spaceCode` `--datasourceId` `--database` `--sql` |
-
----
-
-## 6. Operations and Backfill Management
-
-> See `dataops-operations` Skill for detailed step-by-step workflows.
-
-> **Operations commands use flowInstanceId (not executeId), obtained via `+search_flow_instances`.**
-
-### Typical Flow A: Operations Instance Query and Operations
-
-```bash
-# Search flow instances (supports filtering by status/owner/time)
-ae-cli dataops_operations +search_flow_instances --spaceCode "${spaceCode}" \
-  --pageNum 1 --pageSize 20
-
-# View all task nodes and execution status under instance
-ae-cli dataops_operations +list_flow_task_instances --spaceCode "${spaceCode}" \
-  --flowInstanceId "${flowInstanceId}"
-
-# Get task node definition details
-ae-cli dataops_operations +get_task_definition --spaceCode "${spaceCode}" \
-  --flowInstanceId "${flowInstanceId}" --taskCode "${taskCode}"
-
-# Rerun failed flow instance (continue from failed node)
-ae-cli dataops_operations +rerun_flow_instance --spaceCode "${spaceCode}" \
-  --flowInstanceId "${flowInstanceId}" --confirmed true
-
-# Stop running instance
-ae-cli dataops_operations +stop_flow_instance --spaceCode "${spaceCode}" \
-  --flowInstanceId "${flowInstanceId}" --confirmed true
-
-# Control instance running status (pause/force pause/resume)
-ae-cli dataops_operations +control_flow_instance --spaceCode "${spaceCode}" \
-  --flowInstanceId "${flowInstanceId}" --action "PAUSE" --confirmed true
-```
-
-### Typical Flow B: Backfill Task Management
-
-```bash
-# Query backfill task list
-ae-cli dataops_operations +list_flow_jobs --spaceCode "${spaceCode}" \
-  --pageNum 1 --pageSize 20
-
-# Get backfill task details (configuration, date range, concurrency settings)
-ae-cli dataops_operations +get_flow_job_detail --spaceCode "${spaceCode}" \
-  --jobId "${jobId}"
-
-# Get backfill task execution plan list (status for each date)
-ae-cli dataops_operations +list_flow_job_plans --spaceCode "${spaceCode}" \
-  --jobId "${jobId}"
-
-# Stop running backfill task
-ae-cli dataops_operations +stop_flow_job --spaceCode "${spaceCode}" \
-  --jobId "${jobId}" --confirmed true
-```
-
-### Typical Flow C: Task Node Instance Query
-
-```bash
-# Search task node instances (supports filtering by status/type/time)
-ae-cli dataops_operations +search_task_instances --spaceCode "${spaceCode}" \
-  --pageNum 1 --pageSize 20
-```
+Detailed datasource, metadata browsing, sync solution, execution, and monitoring workflows live in [`references/dataops-integration.md`](references/dataops-integration.md).
 
-### Command Quick Reference
-
-| Command | Purpose | Key Flags |
-|---------|---------|-----------|
-| `+search_flow_instances` | Search flow instances | `--spaceCode` `--pageNum` `--pageSize` |
-| `+list_flow_task_instances` | Instance task nodes | `--spaceCode` `--flowInstanceId` |
-| `+get_task_definition` | Task node definition | `--spaceCode` `--flowInstanceId` `--taskCode` |
-| `+rerun_flow_instance` | Rerun instance | `--spaceCode` `--flowInstanceId` `--confirmed` |
-| `+stop_flow_instance` | Stop instance | `--spaceCode` `--flowInstanceId` `--confirmed` |
-| `+control_flow_instance` | Control instance status | `--spaceCode` `--flowInstanceId` `--action` `--confirmed` |
-| `+search_task_instances` | Search task instances | `--spaceCode` `--pageNum` `--pageSize` |
-| `+list_flow_jobs` | Backfill task list | `--spaceCode` `--pageNum` `--pageSize` |
-| `+get_flow_job_detail` | Backfill task details | `--spaceCode` `--jobId` |
-| `+list_flow_job_plans` | Backfill execution plan | `--spaceCode` `--jobId` |
-| `+stop_flow_job` | Stop backfill task | `--spaceCode` `--jobId` `--confirmed` |
-
-### Parameter Notes
-
-- **action**: `PAUSE` (pause) | `FORCE_PAUSE` (force pause) | `RESUME` (resume)
-- **Instance status**: `RUNNING` / `SUCCESS` / `FAILURE` / `WAITING` / `STOPPED`
-- **flowInstanceId vs executeId**: Operations tools use flowInstanceId, flow tools use executeId, **not interchangeable**
+Key constraints:
+- Generate `sourceConfig`, `sinkConfig`, `channelConfig`, and `fieldsMapping` from the reference templates; do not invent keys.
+- `+save_sync_solution` is not a partial patch: call `+get_sync_detail --withParams true` first, then submit complete configs.
+- Preset repository sync uses `te_etl@TASK_ENGINE_TRINO` and requires gateway configuration.
+- Use `+list_sync_runs` to get `taskId` before stopping a running sync.
 
 ---