@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.96

package/bin/skills/clickzetta-studio-task-manager/SKILL.md

@@ -1,649 +1,156 @@
 ---
 name: clickzetta-studio-task-manager
 description: |
-  Manage ClickZetta Lakehouse Studio tasks, covering task type descriptions (batch sync/multi-table batch sync/
-  real-time sync/multi-table real-time sync/data development), task folder organization, task type differentiation,
-  cz-cli task command family, scheduling configuration, dependency management, and common issue troubleshooting.
-  Implements the "separation of DDL and pipeline management" engineering standard: DDL tasks as drafts,
-  ETL tasks with scheduling, Dynamic Tables with auto-refresh.
+  Manage ClickZetta Lakehouse Studio tasks: task types (batch sync/multi-table batch sync/real-time sync/
+  multi-table real-time sync/data development), folder organization, cz-cli task commands, scheduling,
+  dependency management, and troubleshooting. Implements "separation of DDL and pipeline management".
   Triggered when the user says "create Studio task", "task folder", "task scheduling", "cz-cli task",
-  "task dependency", "task failed", "task status", "full database sync task", "ETL task orchestration",
-  "task management", "separation of DDL and pipeline", "DDL task", "scheduling DAG", "task folder",
-  "Studio task", "batch sync", "real-time sync", "multi-table real-time sync", "data development task",
-  "task types", "which sync to choose", "sync task differences".
-  Keywords: Studio task, task management, cz-cli task, scheduling, DAG, DDL draft, ETL pipeline, task folder, offline sync, realtime sync, CDC, task types
+  "task dependency", "task failed", "task status", "ETL task orchestration", "task management",
+  "DDL task", "scheduling DAG", "Studio task", "task types", "which sync to choose".
+  Keywords: Studio task, task management, cz-cli task, scheduling, DAG, DDL draft, ETL pipeline, task folder, task types
 ---
 
 # ClickZetta Studio Task Management
 
-## Wizard: Clarify Intent
+See [references/engineering-sop.md](references/engineering-sop.md) for the complete new project launch process, incremental iteration guide, and delivery checklist.
+See [references/troubleshooting.md](references/troubleshooting.md) for common issues, MySQL type mapping, scheduling best practices, and multi-environment management.
 
-Upon receiving a task management request, use an interactive question tool (e.g., `question`) to collect intent. If no such tool is available, list options in text:
+## Wizard: Clarify Intent
 
-```
-question({
-  questions: [{
-    question: "What would you like to do?",
-    options: [
-      { label: "Build a new pipeline from scratch", description: "Create folders, DDL tasks, sync tasks, ETL tasks" },
-      { label: "Manage existing tasks", description: "View status, modify config, configure dependencies, rerun, backfill" },
-      { label: "Troubleshoot task issues", description: "Failure diagnosis, dependency check, log analysis → load clickzetta-pipeline-review" },
-      { label: "Standards compliance check", description: "Check if existing tasks follow separation of DDL and pipeline standards" }
-    ]
-  }]
-})
-```
+Upon receiving a task management request, ask what the user wants to do: build a new pipeline from scratch, manage existing tasks (view status, modify config, configure dependencies, rerun, backfill), troubleshoot task issues, or run a standards compliance check.
 
 **If the user has clearly stated what they want to do, proceed directly without asking.**
 
 For **building from scratch**, also collect: business domain/project name, data source type, layering structure.
 
-## Data Pipeline Wizard (Used When Building from Scratch)
-
-Full process: **Requirements Understanding → Data Exploration → Technical Selection → Plan Confirmation → Execution**
-
 ---
 
-### Step 0: Requirements Input
-
-**First ask if the user has a requirements document** (PRD, requirements spec, data warehouse design doc, etc.):
-
-```
-question({
-  questions: [{
-    question: "Before we start, do you have a requirements document or background description?",
-    options: [
-      { label: "Yes, I'll provide it", description: "Paste document content or upload file, I'll extract key information" },
-      { label: "No, I'll describe verbally", description: "I'll guide you through a few key questions" }
-    ]
-  }]
-})
-```
+## Data Pipeline Wizard (Building from Scratch)
 
-**If document provided**: read the document, auto-extract business scenario, data sources, target outputs, freshness requirements, skip to Step 1.
-
-**If no document**, collect the following business requirements (prefer interactive tools; if unavailable, list all in text):
-
-```
-question({
-  questions: [
-    {
-      question: "What business scenario does this pipeline serve?",
-      options: [
-        { label: "BI reports / dashboards", description: "Fixed reports, clear metric system, T+1 or hourly" },
-        { label: "Real-time monitoring / ops dashboard", description: "Minute-level latency, focus on real-time metrics" },
-        { label: "Data science / feature engineering", description: "For model training or inference" },
-        { label: "Data sharing / external output", description: "Provided to other systems or teams" }
-      ]
-    },
-    {
-      question: "Who are the data consumers?",
-      options: [
-        { label: "BI tools (Superset/Tableau, etc.)", description: "Need wide tables or aggregation tables" },
-        { label: "Data analysts (SQL queries)", description: "Need cleaned detail tables" },
-        { label: "Downstream systems / APIs", description: "Need structured output" },
-        { label: "Data scientists (Python/ZettaPark)", description: "Need feature tables or raw detail" }
-      ]
-    },
-    {
-      question: "Data freshness requirements?",
-      options: [
-        { label: "T+1 (available next day)", description: "Batch run at midnight, data ready in the morning" },
-        { label: "Hourly", description: "Updated every hour" },
-        { label: "Minute-level", description: "Near real-time, latency < 10 minutes" },
-        { label: "Second-level real-time", description: "CDC continuous sync, second-level latency" }
-      ]
-    }
-  ]
-})
-```
-
-Also confirm verbally (text follow-up, no menu needed):
-- **Core metric definitions**: if involving GMV, active users, or other business metrics, confirm calculation logic
-- **Project/business domain name**: used for task folder and schema naming (e.g., `ecommerce_dw`)
+### Step 0: Requirements Input
 
----
+Ask whether the user has a requirements document (PRD, design doc — paste or upload). If yes, auto-extract business scenario, data sources, target outputs, freshness requirements. If no, ask three questions: (1) Business scenario (BI reports / real-time monitoring / data science / data sharing)? (2) Data consumers (BI tools / analysts / downstream APIs / data scientists)? (3) Freshness requirements (T+1 / hourly / minute-level / second-level CDC)?
 
-### Step 1: Data Exploration (AI executes autonomously, no user input needed)
+Also confirm: core metric definitions and project/domain name (used for folder and schema naming).
 
-After collecting requirements, immediately explore the current data state:
+### Step 1: Data Exploration (autonomous, no user input needed)
 
 ```sql
--- View related schemas and tables
 SHOW SCHEMAS;
 SHOW TABLES IN <relevant_schema>;
-
--- Check table sizes and row counts
-SELECT table_schema, table_name,
-       ROUND(bytes/1024.0/1024/1024, 2) AS size_gb, row_count
-FROM information_schema.tables
-WHERE table_type = 'MANAGED_TABLE'
+SELECT table_schema, table_name, ROUND(bytes/1024.0/1024/1024, 2) AS size_gb, row_count
+FROM information_schema.tables WHERE table_type = 'MANAGED_TABLE'
 ORDER BY bytes DESC NULLS LAST LIMIT 20;
-
--- Sample to understand field meanings
 SELECT * FROM <schema>.<table> LIMIT 5;
 ```
 
-Also use `cz-cli datasource list` to view configured external data sources.
+Also run `cz-cli datasource list` to view configured external data sources.
 
----
+### Step 2: Technical Selection
 
-### Step 2: Technical Selection (Choose data source type and ingestion method)
+Ask where the data comes from: external database, Kafka, object storage, Lakehouse internal ETL, end-to-end pipeline, or not sure.
 
-Based on requirements and data exploration results, use interactive tools to collect technical choices:
+- External database → ask: real-time CDC or batch offline?
+- Object storage → ask: SQL Pipe (continuous) or Studio batch sync (periodic)?
+- Kafka → ask: SQL Pipe (READ_KAFKA) or Studio real-time sync task?
 
-**Select data source type:**
-```
-question({ questions: [{ question: "Where does the data come from?", options: [
-  { label: "External database", description: "MySQL / PostgreSQL / SQL Server / Oracle, etc." },
-  { label: "Kafka message queue", description: "Kafka Topic → Lakehouse" },
-  { label: "Object storage", description: "OSS / S3 / COS file import" },
-  { label: "Lakehouse internal ETL layering", description: "ODS→DWD→DWS/ADS, SQL tasks + Dynamic Table" },
-  { label: "End-to-end complete pipeline", description: "Data ingestion + layered modeling + aggregation" },
-  { label: "Not sure, explore data first", description: "Look at existing data before recommending an approach" }
-]}]})
-```
-
-**Follow-up (only needed for certain options):**
-
-Selected "External database":
-```
-question({ questions: [{ question: "Sync freshness?", options: [
-  { label: "Real-time sync (second-level)", description: "CDC, based on Binlog/WALs, continuously running" },
-  { label: "Batch offline (hourly/daily)", description: "Periodic full sync, configure Cron" }
-]}]})
-```
-
-Selected "Object storage":
-```
-question({ questions: [{ question: "Ingestion method?", options: [
-  { label: "SQL Pipe (continuous auto-import)", description: "LIST_PURGE or EVENT_NOTIFICATION mode" },
-  { label: "Studio batch sync task", description: "Periodic batch import, configure Cron" }
-]}]})
-```
-
-Selected "Kafka":
-```
-question({ questions: [{ question: "Ingestion method?", options: [
-  { label: "SQL Pipe (READ_KAFKA)", description: "Pure SQL, flexible, recommended for engineers" },
-  { label: "Studio real-time sync task", description: "GUI configuration, supports JSONPath computed columns" }
-]}]})
-```
-
----
-
-### Step 3: Plan Confirmation (Must execute, cannot skip)
+### Step 3: Plan Confirmation (required, cannot skip)
 
-Combining requirements and technical selection, present a complete plan summary to the user for confirmation:
+Present complete plan summary: business scenario, data source, sync method, layering structure, target schema, scheduling. Ask user to confirm or adjust.
 
-```
-question({
-  questions: [{
-    question: "Confirm the following plan to start building:\nBusiness scenario: <scenario>\nData source: <source_name>\nSync method: <batch/real-time/SQL Pipe>\nLayering structure: <ODS/DWD/DWS or Bronze/Silver/Gold>\nTarget schema: <schema>\nScheduling: <Cron or continuous running>\nReady to start?",
-    options: [
-      { label: "Confirmed, start building", description: "Load corresponding skill, begin creating tasks" },
-      { label: "Need adjustments", description: "Re-collect information" }
-    ]
-  }]
-})
-```
-
-After user confirmation, load the corresponding skill per routing table:
+After confirmation, route to the appropriate skill:
 
-**Routing Table**
-
-| Data Source | Freshness/Method | Load Skill |
-|-------------|-----------------|------------|
-| External database | Real-time single-table CDC | `clickzetta-realtime-sync-pipeline` |
-| External database | Real-time multi-table/full database CDC | `clickzetta-cdc-sync-pipeline` |
-| External database | Batch offline | `clickzetta-batch-sync-pipeline` |
+| Data Source | Method | Skill |
+|---|---|---|
+| External DB | Real-time single-table CDC | `clickzetta-realtime-sync-pipeline` |
+| External DB | Real-time multi-table/full DB CDC | `clickzetta-cdc-sync-pipeline` |
+| External DB | Batch offline | `clickzetta-batch-sync-pipeline` |
 | Kafka | SQL Pipe | `clickzetta-kafka-ingest-pipeline` |
 | Kafka | Studio real-time sync | `clickzetta-realtime-sync-pipeline` |
 | Object storage | SQL Pipe | `clickzetta-oss-ingest-pipeline` |
 | Object storage | Studio batch sync | `clickzetta-batch-sync-pipeline` |
-| Lakehouse internal ETL layering | — | `clickzetta-sql-pipeline-manager` |
-| End-to-end complete pipeline / Not sure | — | `clickzetta-dw-modeling` |
-
-> Real-time CDC single vs multi-table: user says "full database" or "multiple tables" → `cdc-sync-pipeline`; "one table" → `realtime-sync-pipeline`; if unclear, ask.
+| Lakehouse internal ETL | — | `clickzetta-sql-pipeline-manager` |
+| End-to-end / Not sure | — | `clickzetta-dw-modeling` |
 
 ---
 
 ## Studio Task Types
 
-Studio provides four major task categories. Choosing the wrong type is the most common engineering mistake:
-
-### Batch Sync (Single-table)
-Periodically full-sync a single source table to Lakehouse.
-
-- **Use cases**: single-table periodic overwrite, low data freshness requirements (daily/hourly batch), resource optimization (no real-time needed)
-- **Run mode**: scheduled (Cron required), full overwrite or append each run
-- **Data sources**: MySQL, PostgreSQL, SQL Server, and other relational databases
-- **Corresponding skill**: `clickzetta-batch-sync-pipeline` (single-table mode)
-
-### Multi-table Batch Sync
-Periodically batch-sync multiple tables or an entire database to Lakehouse.
-
-- **Use cases**:
-  - Full database migration (batch sync all tables, reducing per-table configuration effort)
-  - Sharded table merge (merge multiple sharded tables into a unified target table)
-  - Periodic data calibration (periodic full sync to ensure target matches source)
-- **Run mode**: scheduled (Cron required), supports full database mirror, multi-table mirror, and sharded table merge modes
-- **Data sources**: MySQL, PostgreSQL, SQL Server, etc.
-- **Corresponding skill**: `clickzetta-batch-sync-pipeline` (multi-table mode)
-
-### Real-time Sync (Single-table)
-Continuously sync a single Kafka topic to Lakehouse in real time.
-
-- **Use cases**: Kafka message stream real-time ingestion, second/minute-level latency requirements, single-topic fine-grained sync
-- **Run mode**: continuously running (no Cron needed, runs upon submission)
-- **Data sources**: **Kafka only** (JSON message parsing, supports JSONPath computed columns)
-- **Corresponding skill**: `clickzetta-realtime-sync-pipeline`
-
-### Multi-table Real-time Sync (CDC)
-Sync entire MySQL / PostgreSQL databases or multiple tables to Lakehouse via CDC in real time, with full load + incremental two-phase sync.
-
-- **Use cases**: full database real-time mirror, second-level end-to-end latency, sharded table real-time merge
-- **Run mode**: continuously running (no Cron needed, runs upon submission)
-- **Data sources**:
-
-| Type | Incremental Read Mode | Supported Versions |
-|------|----------------------|-------------------|
-| MySQL (including Aurora MySQL, PolarDB MySQL) | Binlog | 5.6+, 8.x |
-| PostgreSQL (including Aurora PG, PolarDB PG) | WALs | 14+ |
-
-- **Corresponding skill**: `clickzetta-cdc-sync-pipeline`
-
-### Data Development Tasks (SQL / Python / Shell)
-Write and schedule data processing logic in Studio — the core vehicle for data warehouse ETL.
-
-- **SQL tasks**: ODS→DWD cleaning/transformation, data quality checks, ad-hoc data repairs
-- **Python tasks**: custom data processing scripts, external API calls, ML inference
-- **Shell tasks**: system commands, file operations, external tool invocations
-- **Run mode**: scheduled (Cron) or manual trigger
-- **Corresponding skill**: `clickzetta-studio-task-manager` (this skill)
-
-### Four Task Types Quick Comparison
-
-| Task Type | Data Source | Sync Granularity | Run Mode | Freshness |
-|-----------|------------|-----------------|----------|-----------|
-| Batch Sync | Relational DB | Single table | Scheduled | Hourly/daily |
-| Multi-table Batch Sync | Relational DB | Multi-table/full DB | Scheduled | Hourly/daily |
-| Real-time Sync | **Kafka only** | Single topic | Continuously running | Seconds/minutes |
-| Multi-table Real-time Sync | MySQL / PostgreSQL | Multi-table/full DB | Continuously running | Seconds |
-| Data Development | Any (SQL/Python/Shell) | Custom logic | Scheduled or manual | Depends on schedule frequency |
+| Task Type | Data Source | Run Mode | Freshness | Skill |
+|---|---|---|---|---|
+| Batch Sync | Relational DB | Scheduled (Cron) | Hourly/daily | `clickzetta-batch-sync-pipeline` |
+| Multi-table Batch Sync | Relational DB | Scheduled (Cron) | Hourly/daily | `clickzetta-batch-sync-pipeline` |
+| Real-time Sync | **Kafka only** | Continuously running | Seconds/minutes | `clickzetta-realtime-sync-pipeline` |
+| Multi-table Real-time (CDC) | MySQL / PostgreSQL | Continuously running | Seconds | `clickzetta-cdc-sync-pipeline` |
+| Data Development (SQL/Python/Shell) | Any | Scheduled or manual | Depends on schedule | This skill |
 
 ---
 
-## Core Principle: Separation of DDL and Pipeline Management
-
-**Different task types have completely different scheduling strategies.** Confusing task types is the most common engineering mistake.
+## Core Principle: Separation of DDL and Pipeline
 
-| Task Type | Typical Content | Studio Task Type | Scheduling | Status |
-|-----------|----------------|-----------------|------------|--------|
-| **DDL table creation** | CREATE TABLE / CREATE SCHEMA | SQL task | ❌ No Cron, no dependencies | DRAFT |
-| **Data sync tasks** | External source (relational DB/object storage) → ODS | **SINGLE_DI / MULTI_DI / REALTIME** (not SQL tasks) | ✅ Configure Cron (batch) or continuous (real-time) | PUBLISHED |
-| **ETL transformation** | ODS→DWD cleaning SQL (Lakehouse internal) | SQL task | ✅ Configure Cron + depend on upstream sync | PUBLISHED |
-| **Data quality tasks** | Row count checks, NULL rate validation | SQL task | ✅ Configure Cron + depend on ETL | PUBLISHED |
-| **DWS/ADS aggregation** | Metric summaries, report wide tables | ❌ Use Dynamic Table, no task needed | — | — |
+| Task Type | Content | Scheduling | Status |
+|---|---|---|---|
+| **DDL table creation** | CREATE TABLE / SCHEMA | ❌ No Cron, no dependencies | DRAFT |
+| **Data sync** | External source → ODS | ✅ Cron (batch) or continuous (real-time) | PUBLISHED |
+| **ETL transformation** | ODS→DWD cleaning SQL | ✅ Cron + depends on upstream sync | PUBLISHED |
+| **Data quality** | Row count checks, NULL validation | ✅ Cron + depends on ETL | PUBLISHED |
+| **DWS/ADS aggregation** | Metric summaries | ❌ Use Dynamic Table — save DDL as DRAFT task, no scheduling needed | DRAFT |
 
-**Data sync task supported data sources:**
-- Batch sync (SINGLE_DI/MULTI_DI): MySQL, PostgreSQL, Oracle, SQL Server and other relational databases, plus OSS/COS/S3 object storage
-- Single-table real-time sync (REALTIME): Kafka
-- Multi-table real-time sync CDC: MySQL (Binlog, 5.6+/8.x), PostgreSQL (WALs, 14+)
-
-**Other data access methods (not data sync tasks):**
-- Kafka/OSS/S3/COS → can also use SQL Pipe (`READ_KAFKA`/Volume Pipe), both Studio sync tasks and SQL Pipes are valid — choose based on scenario
-- Hive/Databricks/Snowflake Open Catalog → External Catalog federated read-only queries, not data sync
-
-> ⚠️ **DDL tasks must never have Cron**: repeated execution of CREATE TABLE statements causes `SCHEDULE_TASK_HAD_CHILDREN_NODES_EXCEPTION` and other scheduling conflicts. DDL tasks should be demoted to DRAFT immediately after execution.
-
-> ⚠️ **Do not create scheduled tasks for DWS/ADS layer**: Dynamic Tables auto-refresh by the system. Creating additional tasks is redundant computation and wastes resources.
-
-> ⚠️ **Never use SQL tasks as a substitute for data sync tasks**: you cannot use SQL tasks to write `SELECT FROM EXTERNAL` to simulate sync (syntax not supported), nor use JDBC tasks (JDBC can only execute SQL on external databases, cannot sync data to Lakehouse).
+> ⚠️ **DDL tasks must never have Cron** — repeated CREATE TABLE causes `SCHEDULE_TASK_HAD_CHILDREN_NODES_EXCEPTION`.
+> ⚠️ **Do not create scheduled tasks for DWS/ADS** — Dynamic Tables auto-refresh; extra tasks waste resources.
+> ⚠️ **Never use SQL tasks to simulate data sync** — `SELECT FROM EXTERNAL` syntax is not supported.
 
 ---
 
-## Task Folder Organization Standards
-
-Each data warehouse project creates an independent task folder in Studio to manage all task assets uniformly:
+## Task Folder Organization
 
 ```
-<business_domain>_dw/                     ← Project task folder (e.g., shenyu_gateway_dw, ecommerce_dw)
-├── 00_sync_<source>_to_ods               ← Data sync (Cron, runs earliest)
-├── 01_ddl_ods                            ← ODS table creation (DRAFT, no scheduling, run once manually)
-├── 02_ddl_dwd                            ← DWD table creation (DRAFT, no scheduling, run once manually)
-├── 03_ddl_dws_ads                        ← DWS/ADS Dynamic Table creation (DRAFT, no scheduling)
-├── 04_transform_ods_to_dwd               ← ODS→DWD transformation (Cron, depends on 00)
-└── 05_dqc_check                          ← Data quality check (Cron, depends on 04, optional)
+<domain>_dw/
+├── 00_ddl/     ← ALL DDL (CREATE TABLE/VIEW/DT) — DRAFT, run once manually
+├── 01_sync/    ← Data sync tasks — Cron, runs first in dependency chain
+├── 02_ods/     ← ODS ETL (if needed) — DRAFT (view) or Cron (ETL)
+├── 03_dwd/     ← DWD ETL — Cron, depends on 01_sync
+└── 04_dqc/     ← Data quality checks (optional) — Cron, depends on 03_dwd
 ```
 
-> DWS/ADS layer is auto-refreshed by Dynamic Tables — **no task creation needed**.
+DWS/ADS: Dynamic Tables with `refresh_interval` — save DDL as **DRAFT task** (code asset), no scheduling or deployment needed.
 
----
+**Naming conventions:**
+
+| Type | Pattern | Example |
+|---|---|---|
+| Folder | `{domain}_dw` | `retail_dw` |
+| DDL task | `ddl_{layer}_{table}` | `ddl_ods_orders` |
+| ETL task | `{layer}_{table}` | `dwd_fct_orders` |
+| Sync task | `sync_{source}_to_{target}` | `sync_mysql_to_ods` |
 
-## cz-cli task Command Family
+---
 
-### Task Folder Management
+## cz-cli task Commands
 
 ```bash
-# Create task folder
+# Folder management
 cz-cli task folder create <folder_name>
-
-# List all task folders
 cz-cli task folder list
-```
 
-### Task Queries
-
-```bash
-# List all tasks
-cz-cli task list
-
-# Filter by folder
-cz-cli task list --folder <folder_name>
-
-# View task details
+# Query tasks
+cz-cli task list [--folder <folder>]
 cz-cli task get <task_id>
 
-```
+# Create task
+cz-cli task create --name <name> --type SQL --folder <folder> --vcluster default --sql-file ./sql.sql
+cz-cli task create --name <name> --type SINGLE_DI --folder <folder>   # single-table sync
 
-### Task Execution
+# Content and scheduling
+cz-cli task save-content <task_id> --content "<sql>"
+cz-cli task save-cron <task_id> --cron '0 30 2 * * ? *'
+cz-cli task save-config <task_id> --deps replace --dep-tasks '[{"taskId":<id>}]'
 
-```bash
-# Manually trigger task run
+# Deploy and run
+cz-cli task deploy <task_id> [-y]
+cz-cli task execute <task_id>
 cz-cli task run <task_id>
-
-# View task run logs
 cz-cli task logs <task_id>
-
-```
-
-### Task Creation
-
-```bash
-# Create SQL task (ETL/DDL)
-cz-cli task create \
-  --name "04_transform_ods_to_dwd" \
-  --type SQL \
-  --folder <folder_name> \
-  --vcluster default \
-  --sql-file ./transform.sql
-
-# Create data sync task (single-table)
-cz-cli task create \
-  --name "00_sync_mysql_to_ods" \
-  --type SINGLE_DI \
-  --folder <folder_name>
-```
-
-> ⚠️ **Full database sync task (MULTI_DI) capability boundary**: `cz-cli` can create the task framework, but source/target column mapping configuration **must be completed manually in Studio UI**. Recommended SOP:
-> 1. `cz-cli task create --type MULTI_DI` to create task framework
-> 2. Copy the output task link, open in browser
-> 3. Configure source database, target schema, column mapping in Studio UI
-> 4. Click publish to run
-
----
-
-## Scheduling Configuration Best Practices
-
-### Cron Expression Reference
-
-```
-# Daily at 02:00 (data sync)
-0 2 * * *
-
-# Daily at 02:30 (ETL transformation, 30 minutes after sync completes)
-30 2 * * *
-
-# Daily at 03:00 (data quality check)
-0 3 * * *
-
-# Every hour
-0 * * * *
-```
-
-### Dependency Configuration Principles
-
 ```
-Correct dependency chain:
-00_sync (Cron 02:00)
-    ↓ depends on
-04_transform (Cron 02:30)
-    ↓ depends on
-05_dqc (Cron 03:00)
-
-Incorrect dependencies:
-❌ DDL tasks (01/02/03) should not appear in the dependency chain
-❌ Dynamic Tables should not appear in the dependency chain
-```
-
----
-
-## Data Sync Task Type Selection
-
-| Scenario | Task Type | Notes |
-|----------|-----------|-------|
-| MySQL/PG single-table sync to Lakehouse | `SINGLE_DI` | Simple, CLI can fully configure |
-| MySQL/PG full database sync (multi-table mirror) | `MULTI_DI` | CLI creates framework, UI configures mapping |
-| Kafka real-time ingestion | `REALTIME_SYNC` | Continuously running, no Cron needed |
-| File batch import (OSS/S3) | SQL task (COPY INTO) | Use SQL task to execute COPY INTO |
-
----
-
-## Common Issue Troubleshooting
-
-| Issue | Cause | Solution |
-|-------|-------|----------|
-| `SCHEDULE_TASK_HAD_CHILDREN_NODES_EXCEPTION` | DDL task was configured with Cron or dependencies | Clear DDL task scheduling config, demote to DRAFT |
-| Task publish failed, circular dependency | Task A depends on B, B depends on A | Check dependency chain, remove circular dependencies |
-| Sync task keeps failing, no clear error | Column type incompatibility (e.g., MySQL BIT(1) vs Lakehouse BOOLEAN) | Check column type mapping, refer to type mapping table below |
-| Full database sync task cannot run after creation | MULTI_DI task missing column mapping config | Enter Studio UI to configure source/target mapping, then republish |
-| ETL task not triggered on time | Upstream sync task failed, dependency not satisfied | Fix upstream sync task first, then manually trigger ETL |
-| DWS layer data not updated | Mistakenly created scheduled task but Dynamic Table not refreshing | Delete redundant scheduled task, confirm Dynamic Table status is RUNNING |
-| Task run succeeded but data is empty | SQL logic issue (e.g., LEFT JOIN filter condition in wrong position) | Check SQL — LEFT JOIN right-table filter conditions must be in the ON clause |
-
-### MySQL → Lakehouse Column Type Mapping (Common Sync Pitfalls)
-
-| MySQL Type | ❌ Don't Use | ✅ ODS Layer Use | DWD Layer Conversion |
-|-----------|-------------|-----------------|---------------------|
-| `BIT(1)` | `BOOLEAN` | `TINYINT` | `CAST(col AS BOOLEAN)` |
-| `DATETIME` | `DATETIME` | `TIMESTAMP` | Use directly |
-| `ENUM('a','b')` | `ENUM` | `STRING` | Use directly |
-| `TEXT` / `LONGTEXT` | `TEXT` | `STRING` | Use directly |
-| `DECIMAL(p,s)` | `FLOAT` | `DECIMAL(p,s)` | Use directly |
-| `TINYINT(1)` | `BOOLEAN` | `TINYINT` | `CAST(col AS BOOLEAN)` |
-
-> **ODS layer principle: prefer broad types** — sync successfully first, then do precise type conversion in the DWD layer to avoid sync failures due to type incompatibility.
-
----
-
-## Complete Engineering SOP
-
-### Code-as-Asset Principle
-
-**In data pipeline development / data warehouse modeling scenarios, all SQL code should be saved as Studio tasks as manageable code assets.**
-
-- Tasks are the vehicle for code, not just scheduling configurations
-- Even one-time DDL executions should be saved as DRAFT tasks for easy reference, reuse, and multi-environment migration
-- Scenarios that don't need to be saved as tasks: SELECT queries, ad-hoc fix SQL, one-time validation queries
-
-### New Project Launch Process (with Quick Verification Checkpoints)
-
-Agile principle: **verify immediately after each step, know within 30 seconds if it succeeded — don't wait until the full pipeline runs to discover issues.**
-
-```
-1. Create task folder
-   cz-cli task folder create <business_domain>_dw
-
-2. Create ODS layer tables, verify immediately
-   cz-cli task save-content 01_ddl_ods --content "<ods_ddl_sql>"
-   cz-cli task run 01_ddl_ods
-   ✅ Verify: SHOW TABLES IN <ods_schema>  → confirm tables created
-
-3. Create data sync task, trigger once manually, verify immediately
-   - 00_sync: full database or single-table sync to ODS (MULTI_DI requires UI mapping config)
-   cz-cli task execute 00_sync
-   ✅ Verify: SELECT COUNT(*) FROM <ods_schema>.<table>  → compare with source row count
-            SELECT * FROM <ods_schema>.<table> LIMIT 5  → sample check fields
-
-4. Create DWD layer tables, verify immediately
-   cz-cli task save-content 02_ddl_dwd --content "<dwd_ddl_sql>"
-   cz-cli task run 02_ddl_dwd
-   ✅ Verify: SHOW TABLES IN <dwd_schema>  → confirm tables created
-
-5. Generate ETL transformation SQL, manually execute once to verify logic, then configure scheduling
-   cz-cli task save-content 04_transform_ods_to_dwd --content "<etl_sql>"
-   cz-cli task execute 04_transform_ods_to_dwd   ← run manually first
-   ✅ Verify: SELECT COUNT(*) FROM <dwd_schema>.<table>  → row count meets expectations
-            Check key field non-null rate, LEFT JOIN result rows ≥ left table rows
-   After confirmation, configure scheduling:
-   cz-cli task save-cron 04_transform_ods_to_dwd --cron '0 30 2 * * ? *'
-   cz-cli task deploy 04_transform_ods_to_dwd
-
-6. Create DWS/ADS Dynamic Tables, trigger first refresh for verification
-   cz-cli task save-content 03_ddl_dws_ads --content "<dws_ads_ddl_sql>"
-   cz-cli task run 03_ddl_dws_ads
-   REFRESH DYNAMIC TABLE <dws_schema>.<table>
-   ✅ Verify: SHOW DYNAMIC TABLE REFRESH HISTORY <schema>.<table> LIMIT 3
-            → status = SUCCESS, row count matches aggregation logic
-
-7. Optional: data quality check task (Cron + depends on 04)
-   cz-cli task save-content 05_dqc_check --content "<dqc_sql>"
-   cz-cli task save-cron 05_dqc_check --cron '0 0 3 * * ? *'
-   cz-cli task deploy 05_dqc_check
-```
-
-> **Fail-fast principle**: if any step's verification fails, stop immediately and fix — don't continue. If ODS data is wrong, DWD will definitely be wrong too.
-
----
-
-## Incremental Iteration Guide
-
-**When modifying an existing pipeline, follow the incremental process — don't re-run the full pipeline build process.**
-
-When the user says "add a table", "add a field", "add a metric", "change ETL logic", use interactive tools to collect the iteration type:
-
-```
-question({
-  questions: [{
-    question: "What modification do you want to make to the existing pipeline?",
-    options: [
-      { label: "Add sync table", description: "Add a source table to an existing sync task" },
-      { label: "Add field", description: "Source table added a field, ODS/DWD need to follow" },
-      { label: "Add metric/DWS layer", description: "Add aggregation logic or Dynamic Table" },
-      { label: "Modify ETL logic", description: "Cleaning rules, filter conditions, JOIN relationship changes" }
-    ]
-  }]
-})
-```
-
-### Add Sync Table
-
-```
-1. Check lineage, confirm impact scope
-   Load clickzetta-table-lineage, confirm if new table has relationships with existing tables
-
-2. Add table to existing sync task (or create new single-table sync task)
-   cz-cli task content 00_sync  → view existing config
-   Modify as needed and redeploy
-
-3. Manually trigger sync, verify immediately
-   cz-cli task execute 00_sync
-   ✅ SELECT COUNT(*) FROM <ods_schema>.<new_table>
-
-4. If DWD layer processing needed, add ETL SQL to 04_transform task
-   cz-cli task content 04_transform_ods_to_dwd  → view existing SQL
-   Append new table cleaning logic, manually execute to verify, then redeploy
-```
-
-### Add Field (Schema Evolution)
-
-```
-1. Check lineage, identify all affected downstream tasks/DTs
-   Load clickzetta-table-lineage
-
-2. Update layer by layer (upstream to downstream, cannot skip layers)
-   ODS layer: ALTER TABLE <ods_schema>.<table> ADD COLUMN <col> <type>
-   ✅ Verify: DESC TABLE <ods_schema>.<table>  → confirm field added
-
-   DWD layer: update ETL SQL, add cleaning logic for new field
-   Manually execute 04_transform to verify, then redeploy
-   ✅ Verify: SELECT <new_col>, COUNT(*) FROM <dwd_schema>.<table> GROUP BY 1 LIMIT 5
-
-   DWS/ADS layer (if needed): Dynamic Table doesn't support ALTER, use CREATE OR REPLACE to rebuild
-   Immediately REFRESH DYNAMIC TABLE after rebuild
-   ✅ Verify: SHOW DYNAMIC TABLE REFRESH HISTORY LIMIT 3  → status = SUCCESS
-
-3. Update Studio task scripts (keep code assets in sync)
-   cz-cli task save-content <task_name> --content "<updated_sql>"
-```
-
-### Add Metric/DWS Layer
-
-```
-1. Confirm metric definition (confirm calculation logic with user to avoid rework)
-
-2. Check if DWD layer has required fields — if not, follow "Add Field" process first
-
-3. Create new Dynamic Table
-   CREATE OR REPLACE DYNAMIC TABLE <dws_schema>.<new_metric_table>
-     REFRESH INTERVAL <n> <unit> vcluster <gp_cluster>
-   AS SELECT ...;
-   REFRESH DYNAMIC TABLE <dws_schema>.<new_metric_table>
-   ✅ Verify: SELECT COUNT(*), SUM(<metric>) FROM <dws_schema>.<new_metric_table>
-            Compare with known baseline values
-
-4. Save DDL to Studio task
-   cz-cli task save-content 03_ddl_dws_ads --content "<updated_ddl>"
-```
-
-### Modify ETL Logic
-
-```
-1. Check lineage, confirm downstream impact scope
-   Load clickzetta-table-lineage
-
-2. Verify new logic in dev/test environment first (if available)
-
-3. Update ETL SQL
-   cz-cli task content 04_transform_ods_to_dwd  → view existing logic
-   After modification, manually execute to verify:
-   cz-cli task execute 04_transform_ods_to_dwd
-   ✅ Verify: row count comparison, key field sampling, compare with pre-modification results
-
-4. After verification passes, redeploy
-   cz-cli task save-content 04_transform_ods_to_dwd --content "<new_sql>"
-   cz-cli task deploy 04_transform_ods_to_dwd
-
-5. If downstream Dynamic Tables are affected, trigger full refresh
-   SET cz.optimizer.incremental.force.full.refresh = true;
-   REFRESH DYNAMIC TABLE <dws_schema>.<table>;
-   SET cz.optimizer.incremental.force.full.refresh = false;
-```
-
-### Delivery Verification Checklist
-
-- [ ] Row counts at each layer match expectations
-- [ ] Dynamic Table's VCluster exists and `status = RUNNING` (`SHOW VCLUSTERS`)
-- [ ] Dynamic Table refresh history shows SUCCESS
-- [ ] Key field NULL rate within acceptable range
-- [ ] LEFT JOIN result row count ≥ left table row count
-- [ ] All DDL tasks are in DRAFT status
-- [ ] No redundant scheduled tasks for DWS/ADS layer
-- [ ] Scheduling DAG has no circular dependencies
-- [ ] **ETL task dependency chain is complete** (`cz-cli task deps <task>` to verify, `task_dependencies` is not empty)
-- [ ] Key tables and fields have comments (load `clickzetta-manage-comments`)
-
----
-
-## Multi-environment Management (dev → prod)
-
-ClickZetta isolates environments via **Workspace** (dev/staging/prod correspond to different Workspaces). Cross-Workspace pipeline migration currently has limited automation and mainly relies on manual operations.
-
-**When the user raises multi-environment migration needs**, inform them of the following limitations and guide accordingly:
-
-- Data source configurations, schemas, and VCluster names are independent across different Workspaces — each must be confirmed and replaced during migration
-- There is currently no one-click migration tool — recommend contacting **data operations (lh-dba role)** for help planning multi-environment strategy
-- You can use `cz-cli task content <task_id>` to export task scripts, manually adjust, then recreate in the target Workspace
 
-> Multi-environment management is a platform capability evolution direction. At the current stage, it's recommended to use schema naming within a single Workspace to differentiate (e.g., `ecommerce_ods_dev` vs `ecommerce_ods`) to reduce migration complexity.
+> ⚠️ **MULTI_DI (full database sync)**: `cz-cli` creates the task framework only. Source/target column mapping **must be configured in Studio UI** before publishing.