@clickzetta/cz-cli-darwin-x64 0.3.94 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/clone-doc.md

@@ -0,0 +1,101 @@
+# CREATE ....CLONE
+
+>【Preview Release】This feature is currently in public preview.
+
+In Lakehouse, the clone table operation creates an independent copy of a table that is completely separate from the original table and does not affect each other. Clone tables do not take up additional storage space when created because they share the data version of the original table. However, if the original table data changes and affects the data in the clone table, the clone table will need to pay for the storage costs of these changes. Clone tables have similar functions to regular tables, including operations such as querying, copying, and deleting. Lakehouse supports cloning regular tables and dynamic tables, and allows creating clones of clones, i.e., cloning a table that is already a clone.
+
+## Notes
+
+* The clone operation does not copy the permissions of the original table, and the newly cloned table will have independent permission settings.
+* The clone operation does not copy user-set properties, including lifecycle and data time travel retention periods.
+* Real-time written and uncommitted data will not be cloned. Therefore, data that has just been written will not be found in the clone table after the clone is completed.
+* Cloning external tables is not supported.
+
+## Normal Table Clone
+```SQL
+CREATE TABLE [ IF NOT EXISTS ] table_name
+  CLONE <source_object_name> [TIMESTAMP AS OF timestamp_expression]
+```
+* You can use `TIMESTAMP AS OF timestamp_expression` to clone the version of the dynamic table at a specified point in time. As long as the point in time is within the **[data retention period](timetravel.md)** (`data_retention_days`)
+  * `timestamp_expression` can be any of the following:
+    * `'2024-10-18T22:15:12.013Z'`, i.e., a string that can be cast to a timestamp
+    * `cast('2024-10-18 13:36:32 ' as timestamp)`
+    * `'2024-10-18'`, i.e., a date string
+    * `current_timestamp() - interval 12 hours`
+    * `date_sub(current_date(), 1)`
+    * Any other expression that is itself a timestamp or can be cast to a timestamp
+```
+CREATE TABLE normal_table_clone_version CLONE normal_table_base TIMESTAMP AS OF '2024-10-12 19:23:52.329';
+```
+## Dynamic Table Cloning
+```SQL
+CREATE DYNAMIC TABLE dt_name
+ [
+    refreshOption ::=
+    REFRESH [START WITH timestamp_expr] [interval_time] VCLUSTER vcname   
+ ]
+  CLONE <source_dynamic_table> [TIMESTAMP AS OF timestamp_expression]
+```
+* The cloned dynamic table is paused by default. You can use `ALTER DYNAMIC TABLE table_name RESUME;` to start the refresh task.
+* `DYNAMIC TABLE dt_name`: The name of the dynamic table to be created. `dt_name` is a placeholder that you need to replace with the actual table name.
+* `REFRESH`: Specifies the refresh options for the table. When creating a cloned dynamic table, you can specify the refresh options:
+  * If refresh options are specified, the refresh strategy of the cloned table is used.
+  * If no refresh options are specified, the default refresh strategy is used.
+* `START WITH timestamp_expr`: Optional clause that specifies the start time for the table refresh. `timestamp_expr` is a timestamp expression.
+* `interval_time`: Specifies the refresh interval time unit. This is of type `INTERVAL` and can be any of the following:
+    *  INTERVAL 1 year 
+    *  INTERVAL 2 month 
+    *  INTERVAL 3 day 
+    *  INTERVAL 4 hour 
+    *  INTERVAL 5 minute
+    *  INTERVAL 30 second
+* `VCLUSTER vcname`: Specifies the name of the compute cluster. `vcname` is the actual name of the compute cluster.
+* `CLONE <source_dynamic_table>`: Clones from an existing dynamic table `<source_dynamic_table>`.
+* `TIMESTAMP AS OF timestamp_expression`: The version of the cloned dynamic table at the specified point in time. `timestamp_expression` can be one of several timestamp expressions. As long as the point in time is within the [data retention period](timetravel.md) (`data_retention_days`):
+  * `timestamp_expression` can be any of the following:
+    * `'2024-10-18T22:15:12.013Z'`, i.e., a string that can be cast to a timestamp
+    * `cast('2024-10-18 13:36:32 ' as timestamp)`
+    * `'2024-10-18'`, i.e., a date string
+    * `current_timestamp() - interval 12 hours`
+    * `date_sub(current_date(), 1)`
+    * Any other expression that is itself a timestamp or can be cast to a timestamp  
+
+## Permission Requirements
+
+* You need `CREATE TABLE` permission in the current schema to create a cloned table.
+* You need `SELECT` permission on the original table.
+
+## Cloned Table Examples
+
+### Clone a Regular Table
+```SQL
+-- Create a normal table
+CREATE TABLE normal_table_base (id BIGINT, col STRING);
+
+-- Clone the normal table
+CREATE TABLE normal_table_clone CLONE normal_table_base;
+
+-- Clone the base table at a specific version
+CREATE TABLE normal_table_clone_version CLONE normal_table_base TIMESTAMP AS OF '2024-10-12 19:23:52.329';
+```
+### Clone Dynamic Table
+```SQL
+-- Create dynamic table
+CREATE DYNAMIC TABLE dt_table_base
+REFRESH INTERVAL 1 MINUTE
+VCLUSTER DEFAULT
+AS SELECT * FROM normal_table_base;
+
+-- Clone dynamic table
+CREATE DYNAMIC TABLE dt_table_clone CLONE dt_table_base;
+
+-- Specify refresh interval and compute cluster when cloning
+CREATE DYNAMIC TABLE dt_table_clone_refresh
+REFRESH INTERVAL 2 MINUTE
+VCLUSTER DEFAULT
+CLONE dt_table_base;
+
+-- Specify the version of the dynamic table when cloning
+CREATE DYNAMIC TABLE dt_table_clone_version
+CLONE dt_table_base TIMESTAMP AS OF '2024-10-12 20:25:32.247';
+```

package/bin/skills/lakehouse-doc-en/references/cloud_object_storage.md

@@ -0,0 +1 @@
+# Object Storage Guide

package/bin/skills/lakehouse-doc-en/references/cluster-table-guide.md

@@ -0,0 +1,64 @@
+# Bucketing (Clustered Key) and Sorting Key
+
+In the field of big data storage and analysis, the way data is organized has a significant impact on query performance and storage efficiency. This article will provide a detailed introduction to the concepts of Clustered Key and Sorted Key, their usage, and practical application examples to help you better optimize your data organization structure.
+
+## Clustered Key
+
+The Clustered Key is the core of table data distribution. By specifying the column's Hash Key, Lakehouse will perform a hash operation on the data based on these keys and distribute the data into different data buckets. This dispersion helps to avoid data skew and hotspot issues while improving parallel processing capabilities.
+
+### Criteria for Selecting Clustered Keys
+
+- Choose columns with a wide range of values and few duplicate values as Clustered Keys to achieve uniform data distribution.
+- When performing `JOIN` operations, if the join key is consistent with the Clustered Key, it can significantly improve performance.
+- Suitable for scenarios with large amounts of data, the recommended data bucket size is about 128MB to 1GB, depending on the data compression rate and access pattern.
+- If no Clustered Key is specified, 256 buckets are used by default.
+
+### Precautions
+
+- Avoid specifying too small a number of Clustered Keys to prevent generating a large number of small files, which can affect metadata management and I/O operation efficiency.
+- Too many small files can lead to poor data locality, increased task scheduling overhead, and reduced processing efficiency.
+
+## Sorted Key
+
+The Sorted Key is used for the sorting method of file fields. For queries that need to sort the query results, sorting the data by the Sorted Key can improve performance.
+
+### Precautions for Using Sorted Keys
+
+- You can specify ascending (ASC) or descending (DESC) order for the Sorted Key.
+- Although Sorted Keys can improve query performance, sorting a large amount of data during insertion may consume considerable resources.
+
+## Practical Application Examples
+
+### Example 1: Create a Table and Specify Clustered Key and Sorted Key
+```sql
+CREATE TABLE sales_data (
+    sale_id INT,
+    product_id INT,
+    quantity_sold INT,
+    sale_date DATE,
+    ...
+) CLUSTERED BY (product_id)
+SORTED BY (sale_date DESC)
+INTO 50 BUCKETS;
+```
+In this example, a table named `sales_data` is created, and the data will be distributed into 50 buckets based on the Hash value of the `product_id` column. At the same time, the data within each bucket will be sorted in descending order by the `sale_date` column.
+
+### Example 2: Optimizing Data Warehouse Query Performance
+
+Suppose you are dealing with a data warehouse containing a large number of transaction records. You can optimize query performance in the following way:
+```sql
+CREATE TABLE transaction_records (
+    transaction_id INT,
+    customer_id INT,
+    transaction_date DATE,
+    amount DECIMAL(10,2),
+    ...
+) CLUSTERED BY (customer_id)
+SORTED BY (transaction_date ASC)
+INTO 128 BUCKETS;
+```
+In this example, the `transaction_records` table is bucketed by `customer_id`, and the data within each bucket is sorted by `transaction_date`. This design helps improve the efficiency of querying transaction records by customer.
+
+## Conclusion
+
+Using bucket keys and sort keys appropriately can effectively optimize the physical storage structure of data and improve query performance, especially when dealing with large-scale datasets. This method is particularly suitable for data warehousing and big data analysis scenarios, significantly enhancing data processing efficiency and speed. I hope this article helps you better understand and apply bucket keys and sort keys to optimize your data organization methods.

package/bin/skills/lakehouse-doc-en/references/cluster-table.md

@@ -0,0 +1,111 @@
+# Clustered Key and Sorted Key
+
+When creating a table, specifying data distribution and sorting methods using the `CLUSTERED BY` and `SORTED BY` clauses can significantly improve performance in specific query scenarios.
+
+## Syntax
+
+```SQL
+CREATE TABLE table_name (
+    column_definitions
+)
+[CLUSTERED BY (col_name [, col_name ...]) [SORTED BY (col_name [ASC|DESC] [, ...])] INTO num BUCKETS]
+[PARTITIONED BY (col_name data_type [, ...])];
+```
+
+## Parameter Description
+
+| Parameter | Required | Description |
+|------|------|------|
+| `CLUSTERED BY (col_name, ...)` | No | Specifies the clustering key columns. The system computes a hash on these column values and distributes data across buckets. |
+| `SORTED BY (col_name [ASC\|DESC], ...)` | No | Specifies the sorting key columns within each bucket. Data is stored in this order within each bucket. Ascending by default. |
+| `INTO num BUCKETS` | Required with `CLUSTERED BY` | Specifies the number of buckets. It is recommended that each bucket holds approximately 128 MB–1 GB of data. |
+
+## Clustered Key Description
+
+The clustered key determines how data is distributed across buckets. The system computes a hash on the clustered key column values, and data with the same hash value falls into the same bucket.
+
+**Criteria for selecting clustered keys:**
+
+- Choose columns with a wide range of values and few duplicates to achieve even data distribution and avoid data skew.
+- If a column is frequently used as a `JOIN` key in queries, setting it as the clustered key enables Bucket Join, significantly improving JOIN performance.
+- When no clustered key is specified, the system defaults to 256 buckets.
+- Too many buckets result in a large number of small files, impacting metadata management and I/O efficiency. Too few buckets fail to fully utilize parallel processing capabilities.
+
+## Sorted Key Description
+
+The sorted key defines the physical sort order of data within each bucket. For queries that filter or sort by the sorted key, pre-sorted data reduces the scan range and improves query performance.
+
+- Each sort column can be specified as `ASC` (ascending, default) or `DESC` (descending).
+- Sorted keys have some impact on write performance. Sorting during large-scale data writes consumes additional resources.
+
+## Examples
+
+1. Create an orders table clustered by `customer_id` and sorted by `order_date`:
+
+```SQL
+CREATE TABLE orders_clustered (
+    order_id    BIGINT,
+    customer_id INT,
+    product     STRING,
+    amount      DECIMAL(10, 2),
+    order_date  DATE
+)
+CLUSTERED BY (customer_id)
+SORTED BY (order_date DESC)
+INTO 64 BUCKETS;
+```
+
+When querying by `customer_id`, the system only needs to scan the corresponding bucket without a full table scan:
+
+```SQL
+SELECT * FROM orders_clustered WHERE customer_id = 1001;
+```
+
+2. Create a table with a multi-column clustered key, suitable for multi-dimensional JOIN scenarios:
+
+```SQL
+CREATE TABLE transaction_records (
+    transaction_id BIGINT,
+    customer_id    INT,
+    product_id     INT,
+    amount         DECIMAL(10, 2),
+    transaction_date DATE
+)
+CLUSTERED BY (customer_id, product_id)
+SORTED BY (transaction_date ASC)
+INTO 128 BUCKETS;
+```
+
+3. Specify only the clustered key without a sorted key:
+
+```SQL
+CREATE TABLE sales_data (
+    sale_id    INT,
+    product_id INT,
+    region     STRING,
+    amount     DECIMAL(10, 2)
+)
+CLUSTERED BY (product_id)
+INTO 50 BUCKETS;
+```
+
+4. Combine partitioning and clustering (data is clustered within each partition):
+
+```SQL
+CREATE TABLE logs_partitioned (
+    log_id   BIGINT,
+    user_id  INT,
+    action   STRING,
+    log_time TIMESTAMP
+)
+PARTITIONED BY (log_date DATE)
+CLUSTERED BY (user_id)
+INTO 32 BUCKETS;
+```
+
+## Notes
+
+- `SORTED BY` must be used together with `CLUSTERED BY` and cannot be specified alone.
+- It is recommended to estimate the number of buckets based on data volume: aim for approximately 128 MB–1 GB of data per bucket.
+- Clustered keys and sorted keys cannot be modified via `ALTER TABLE` after table creation. The table must be recreated.
+- For partitioned tables, clustering is performed independently within each partition, and the bucket count applies to each partition.

package/bin/skills/lakehouse-doc-en/references/code_approval.md

@@ -0,0 +1,34 @@
+When configuring a workspace, you can enable the "Mandatory Code Review" feature toggle to determine whether code submissions in the current workspace require review. Only code that has passed review can be published to the production environment.
+
+## Prerequisites
+
+Only users with the workspace administrator (workspace_admin) role can enable or disable the mandatory code review process.
+
+## Steps
+
+**1. Enable Code Review**
+
+* Enable the code review process when creating a new workspace.
+* For existing workspaces -> Click to enter the details page -> Edit, and enable the code review feature.
+  ![](/.topwrite/assets/image_1775099278363.png)
+
+**2. Configure the Code Review Approval Flow**
+
+Click Approval -> Approval Flow -> Select the code review flow for the target workspace
+
+Under "Approval", configure the approval roles/users for the code review flow. These roles/users will be the personnel who need to participate in code review when code is submitted in the current workspace.
+
+> Only roles with development permissions can perform approvals
+  ![](/.topwrite/assets/image_1775099287191.png)
+
+**3. Submit Code for Review**
+
+After enabling mandatory code review, any submission action will trigger a "Code Review" approval ticket. Once the approver approves, the code will be published to the production environment. If the approver rejects it, you need to modify and resubmit.
+  ![](/.topwrite/assets/image_1775099296124.png)
+
+**4. Approval**
+
+After code submission, the approver needs to review it.
+
+Click Approval -> Under the Approval tab, find the target task and perform the relevant actions.
+  ![](/.topwrite/assets/image_1775099305006.png)

package/bin/skills/lakehouse-doc-en/references/composite_task.md

@@ -0,0 +1,192 @@
+# Composite Task
+
+A composite task is a special task type that supports encapsulating multiple periodic tasks (such as offline synchronization and SQL tasks) as an independent unit to achieve **unified scheduling configuration, parameter management, and operational control**, while establishing dependencies with other regular nodes. This task type is suited for business scenarios that require batch task flow management and unified scheduling cycle control — for example, incremental computation and cross-task coordination.
+
+Comparing composite tasks with [task groups](task_group.md), the key differences are:
+
+* A task group is a logical management concept that places multiple tasks in a group for unified display and orchestration. A composite task is a concrete task type entity, on equal footing with a regular SQL task. You can add a composite task as an element within a task group.
+* From the outside, a composite task appears as a single unit, but internally it contains one or more subtask nodes (which can be SQL tasks, offline synchronization tasks, or other periodic scheduling task types).
+* A composite task supports a unified scheduling time, and all internal subtask nodes run at the same frequency based on that schedule. This is especially useful for data processing pipelines where all task nodes have the same data freshness requirements. Task groups do not support scheduling time configuration.
+
+## Quick Start: Creating a Composite Task
+
+Creating a composite task is very similar to creating a regular periodic SQL task.
+
+### Creation Entry Points
+
+You can create a composite task through any of the following paths:
+
+* Task Development: Task tab → New Task → Select "Composite Task"
+
+  ![](.topwrite/assets/image_1752130018729.png =420)
+
+* Task Group Details page: Inside a task group → Add New Task → Select "Composite Task"
+  
+  ![](.topwrite/assets/image_1752130033606.png =680)
+
+* Workspace: Top navigation bar → New Task → Select "Composite Task"
+  
+  ![](.topwrite/assets/image_1752130048252.png =680)
+
+### Basic Information
+
+In the composite task creation dialog, fill in the required fields — same as for other regular SQL tasks:
+
+* Task Name: Required.
+* Folder: The folder to place the task in. Required.
+* Task Group: Optionally assign the task to a task group by selecting a specific task group name.
+
+  ![](.topwrite/assets/image_1752130064673.png =680)
+
+## Core Feature Guide
+
+### Subtask Management (Canvas Mode)
+
+Composite tasks use a **canvas** (DAG diagram) to display subtask nodes and support the following operations:
+![](.topwrite/assets/image_1752130101512.png =680)
+
+#### Adding Subtasks
+
+* **Entry point**: Canvas toolbar → "New Subtask" → Select a task type (only periodic tasks are supported: offline synchronization, SQL, etc.; **real-time tasks are not supported**). You can add a subtask by clicking the task type or dragging it onto the canvas.
+
+  ![](.topwrite/assets/image_1752130110716.png =360)
+
+* **Default state**: Newly added subtasks have no dependencies by default. You need to configure dependencies manually — either by drawing connections on the canvas or through the subtask detail configuration.
+
+#### Editing Subtasks
+
+* **How to edit**:
+  * When you add a subtask node, the system automatically opens the subtask detail page so you can write code and configure scheduling settings.
+  * Double-click a subtask → Opens the regular node editing page (reuses existing editing functionality).
+
+* **Restrictions**: To ensure the composite task runs correctly after submission, subtasks cannot be saved in an incomplete state. All required fields (code, cluster, scheduling properties) must be filled in before saving. Make sure this information is complete and saved before exiting the subtask editor.
+
+#### Dependency Orchestration
+
+* **Configure in the subtask**: Click "Scheduling" on the subtask → Set upstream subtask dependencies under "Scheduling Dependencies".
+* **Draw connections on the canvas**: On the composite task's DAG canvas, select a source node → Drag a connection to the target node → This establishes an "upstream → downstream" dependency. You can also remove dependencies by operating on the connection lines.
+
+#### Canvas Tools
+
+* **Auto layout**: Toolbar → "Auto Layout" → Optimizes the DAG display based on dependency relationships.
+* **Zoom/Refresh**: Toolbar → "+" / "-" to adjust canvas size, or "Refresh" to reset the view.
+
+### Scheduling Configuration (Global and Local)
+
+The scheduling strategy for a composite task is managed through a combination of **global configuration** (at the composite task level) and **local configuration** (at the subtask level). Subtasks inherit the composite task's global configuration by default, but can also be configured individually. When a local configuration exists, it takes precedence. Key configuration items:
+
+| Configuration Item | Global Configuration (Composite Task)                                    | Local Configuration (Subtask)                                                                        |
+| ------------------ | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
+| Scheduling Time    | Required; supports Cron expressions. Subtasks cannot set this independently. | None. Subtasks follow the composite task's global scheduling time and run at the same frequency.  |
+| Instance Rerun     | Global setting (rerun count, interval). Subtasks inherit by default.     | Optional override of global settings (only supported for specific task types such as SQL).           |
+| Task Priority      | Global setting (default value).                                          | Optional individual setting (overrides global; only supported for SQL nodes).                        |
+| Self-dependency    | Global setting (controls composite task cycle dependency).               | None. Subtask self-dependency is indirectly achieved through the composite task's global self-dependency. |
+
+**Example**: If the composite task global setting is "3 reruns with a 5-minute interval," and a subtask is individually configured with "5 reruns," that subtask will use "5 reruns with a 5-minute interval."
+
+![](.topwrite/assets/image_1752130126629.png =680)
+
+![](.topwrite/assets/image_1752130133390.png =680)
+
+### Parameter Management (Composite Task → Subtask Propagation)
+
+#### Defining Parameters
+
+* **Entry point**: Composite task detail page → Click "Parameters" → Add parameters in the dialog (e.g., `composite_task_param`):
+
+  ![](.topwrite/assets/image_1752130143435.png =680)
+
+  ![](.topwrite/assets/image_1752130151838.png =680)
+
+* **Scope**: Parameters are only valid within the current composite task. Parameters are isolated between different composite tasks.
+
+#### Referencing Parameters
+
+In a subtask's parameter configuration, if the parameter name matches a parameter defined in the composite task, the system automatically recognizes it and prompts you to select the value source. To use the globally defined composite task parameter, select "Composite Task" as the value source; otherwise, select "Task".
+
+![](.topwrite/assets/image_1752130161305.png =680)
+
+In subtask code, you can reference parameters using `${parameter_name}`. **Example** (SQL subtask):
+
+```
+SELECT * FROM user_log WHERE dt = '${composite_task_param}'  
+```
+
+## Advanced Operations: Saving and Submitting
+
+### Saving a Composite Task
+
+* Composite tasks do not currently support saving all subtask nodes at once. Save each subtask individually to ensure its code, scheduling configuration, and other settings are saved. Then submit the composite task.
+
+### Submitting a Composite Task
+
+* **Purpose**: Submits the composite task and all its internal subtasks to the production environment for periodic scheduled execution, generating a corresponding "submission version."
+
+* **Entry point**: Composite task detail page → "Submit."
+
+  ![](.topwrite/assets/image_1752130174992.png =680)
+
+## Operations and Monitoring
+
+### Composite Task Operations
+
+Composite tasks appear in the Operations Center alongside other periodic tasks and support similar operations such as pause, data backfill, and offline.
+
+![](.topwrite/assets/image_1752130184652.png =680)
+
+| Operation      | Description                                                                                                                                                                                    |
+| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Pause/Resume   | After pausing, the composite task and its subtasks stop scheduling. After resuming, execution continues according to the current configuration.                                                |
+| Data Backfill  | Select a time range → The composite task generates backfill instances as a whole according to scheduling rules (including all subtask instances).                                               |
+| Offline        | After going offline, the task no longer schedules. Check for downstream dependencies — if any exist, you cannot go offline directly and must use the "Offline (including downstream)" feature. |
+
+### Composite Task Instance Operations
+
+In the Instance Operations tab, composite task instances are listed alongside other periodic task instances and support operations such as rerun and mark as success/failure, as shown below:
+
+![](.topwrite/assets/image_1752130199325.png =680)
+
+#### Status Monitoring
+
+* **Overall status**: The composite task instance status is determined by the combined status of its subtasks:
+  * Not Started: all subtasks have not started
+  * Running: at least one subtask is running
+  * Succeeded: all subtasks succeeded
+  * Failed: at least one subtask failed
+* **Drill-down view**: Click the instance ID → Opens the subtask instance DAG page to view runtime details for each subtask (including logs and status).
+
+#### Instance Operations
+
+| Operation              | Supported States                      | Behavior                                                                                                                                          |
+| ---------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Rerun                  | Any terminal state (failed/succeeded) | Reruns the entire composite task. Subtasks generate new instances according to their respective rerun rules. Partial subtask selection is not supported. |
+| Mark Success/Failure   | Not Started / Terminal state          | Forces the composite task and all subtask instance statuses to change to succeeded or failed.                                                     |
+| Terminate              | Running                               | Terminates all running or not-started subtask instances and sets their status to failed.                                                          |
+
+### Monitoring and Alerts
+
+In monitoring and alerts, composite tasks are categorized under periodic scheduling tasks. Existing monitoring items such as "Task Instance Execution Failure" and "Periodic Task Instance Completion Time" apply to composite tasks as well. You can configure monitoring rules for composite tasks the same way you would for regular SQL periodic tasks.
+
+![](.topwrite/assets/image_1752131374833.png =680)
+
+### Triggering Data Quality Rule Checks
+
+When configuring data quality rules, you can select a composite task as the scheduling trigger in the rule execution trigger settings, as shown below. You can select a specific subtask to trigger the check; if "Subtask" is left blank, the check is triggered after the entire composite task completes.
+
+![](.topwrite/assets/image_1752131279717.png =680)
+
+## Notes
+
+1. **Type restrictions**: Subtasks within a composite task only support periodic tasks (offline synchronization, SQL, etc.). Real-time tasks (multi-table real-time synchronization, Continuous Jobs, etc.) cannot be added.
+2. **Parameter scope**: Composite task parameters are only visible within the composite task and cannot be used by other composite tasks. To share parameters across composite tasks, use task group parameters or similar mechanisms.
+3. **Version management**: Composite tasks only support "submission versions" (no saved versions) and do not currently support version rollback.
+4. **Task dependencies**: Within a task group, a composite task exists as a whole unit. Internal subtasks cannot be added individually to the task group, and dependencies are between the composite task as a whole and other tasks. Internal subtasks of a composite task cannot depend on other tasks in the task group, nor can they be depended upon by other tasks in the task group.
+
+---
+
+## Related Documentation
+
+- [Task Parameters](task_param.md) — Concepts and configuration for composite task parameters
+- [Task Parameter Syntax Reference](task_param_reference.md) — Full syntax for time expressions and built-in parameters
+- [Task Group](task_group.md) — Using task group parameters to share parameters across composite tasks
+- [Task Development and Scheduling](task-develop.md) — Development and scheduling configuration for SQL subtasks

package/bin/skills/lakehouse-doc-en/references/comprehensive_comparison.md

@@ -0,0 +1,83 @@
+# Comprehensive Comparison: Singdata Lakehouse vs. Snowflake, Databricks, and BigQuery
+
+This comprehensive comparison delivers an in-depth analysis of Singdata Lakehouse's positioning and advantages against industry-leading competitors Snowflake, Databricks, and BigQuery.
+
+Our evaluation prioritizes performance and cost—the two factors organizations care about most—before examining three additional dimensions critical for modern data analytics solutions, providing you with a clear and thorough foundation for decision-making.
+
+1. Singdata delivers best performance in all 3 scenarios, followed by Snowflake. In real-time analytics testing, Singdata is 2.6x-9.6x faster.
+2. Singdata is 3x-6x lower in cost than Snowflake and 6x-12x lower than Databricks.
+3. From an architectural perspective, Singdata offers an advanced Lakehouse architecture similar to Databricks, combined with the ease of use and maintenance-free features akin to Snowflake. It brings together the best of both worlds.
+
+**Performance & Cost**:
+
+![](.topwrite/assets/new_perf_1.jpeg)
+
+Based on the benchmark performance comparison chart, Singdata demonstrates significant performance advantages across all three tested workloads（For detailed metrics, please refer to the table below.）
+
+1. SSB-FLAT 100GB (Real-time Analytics)
+
+2. TPC-H 100GB(Small-scale Reporting/BI Workloads)
+
+3. TPC-DS 1TB(Complex Queries)
+
+**Exceptional performance translates to reduced costs**:
+
+![](.topwrite/assets/new_cost_1.jpeg)
+
+Following our summary of Performance and Cost factors, we extend our analysis to cover three additional critical dimensions: Architecture, Scalability, and Use Cases. These five evaluation criteria collectively provide a holistic view of each platform's capabilities, limitations, and ideal implementation scenarios, with detailed metrics presented in the comparative tables below. This comprehensive framework enables you to make an informed decision that aligns with your organization's specific data strategy requirements.
+
+1\. **Architecture**
+
+|                                                   | ![](.topwrite/assets/d33edb049c/5fb9f76a0967823749b634044213fed99d77c62b.png =50)
Singdata                            | ![](.topwrite/assets/d33edb049c/76d05b199a886af460a051279cd95b11fd2abc21.png =50)
Snowflake  | ![](.topwrite/assets/d33edb049c/73c2df840d19f5b449bef8ee02d520decc2c5655.png =50)
Databricks                                                     | ![](.topwrite/assets/d33edb049c/c49a1b7f1c44b419e3a8c3b5b324879a025a6aa1.png =50)
BigQuery        |
+| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
+| Storage/Compute Separation                        | Yes                                                                                                                       | Yes                                                                                              | Yes                                                                                                                                                  | Yes                                                                                                   |
+| Supported cloud infrastructure                    | AWS / GCP / Alibaba Cloud / Tencent Cloud / Huawei Cloud                                                                  | AWS, Azure, GCP                                                                                  | AWS, Azure, Google Cloud                                                                                                                             | Google Cloud Only                                                                                     |
+| Deployment Mode                                   | • SaaS
• VPC On Premise                                                                                               | SaaS  – Isolated tenancy available via “VPS” tier                                                | Hybrid of SaaS and PaaS(BYOC)                                                                                                                        | SaaS                                                                                                  |
+| Isolated tenancy – option for dedicated resources | Multi-tenant pooled resources  – Isolated tenancy available via “VPC” tier                                                | Multi-tenant pooled resources  – Isolated tenancy available via “VPS” tier                       | Control plane in Databricks account; Data Plane in customer VPC (optional), Storage in customer VPC. Serverless SQL runs in Multi-tenant environment | Multi-tenant pooled resources  – “VPC Service Controls” can be used to limit connectivity to your VPC |
+| Control vs abstraction of compute                 | Configurable cluster size (1-128 nodes, 256 and 512 in preview)  – Compute types：  • General Purpose    • Analysis Purpos | Configurable cluster size (1-128 nodes, 256 and 512 in preview)  – No control over compute types | Configurable clusters and instance types.  Serverless option runs in Databricks VPC using containers                                                 | Serverless, no control over compute (BigQuery determines how many “slots” each query needs)           |
+
+2\. **Scalability**
+
+|                                                                 | ![](.topwrite/assets/d33edb049c/5fb9f76a0967823749b634044213fed99d77c62b.png =50)
Singdata | ![](.topwrite/assets/d33edb049c/76d05b199a886af460a051279cd95b11fd2abc21.png =50)
Snowflake | ![](.topwrite/assets/d33edb049c/73c2df840d19f5b449bef8ee02d520decc2c5655.png =50)
Databricks | ![](.topwrite/assets/d33edb049c/c49a1b7f1c44b419e3a8c3b5b324879a025a6aa1.png =50)
BigQuery |
+| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
+| Elasticity – Scaling for larger data volumes and faster queries | Yes
Autoscaling cluster with sub-second latency                                            | Yes
Cluster resize                                                                          | Yes
Autoscaling clusters in minutes                                                          | Serverless or slots autoscaling                                                                |
+| Elasticity – Scaling for higher concurrency                     | Yes
Concurrency Scaling with sub-second latency                                            | Automatic Concurrency Scaling                                                                   | Partially supported by SQL warehouses  10 queries per cluster limit.                             | Automatic query execution scaling with multi-slot resource management                          |
+
+3\. **Performance**
+
+![](.topwrite/assets/d33edb049c/fae0d66a1b15b6eb7453738eb4240721c1b69e84.png)
+
+| Benchmark                                               | Duration         | ![](.topwrite/assets/d33edb049c/5fb9f76a0967823749b634044213fed99d77c62b.png =50)
Singdata | ![](.topwrite/assets/d33edb049c/76d05b199a886af460a051279cd95b11fd2abc21.png =50)
Snowflake | ![](.topwrite/assets/d33edb049c/73c2df840d19f5b449bef8ee02d520decc2c5655.png =50)
Databricks | ![](.topwrite/assets/d33edb049c/c49a1b7f1c44b419e3a8c3b5b324879a025a6aa1.png =50)
BigQuery |
+| ------------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
+| **SSB-FLAT 100GB** \[*Real-time Analytics*]             | Duration(Second) | 1.9                                                                                            | 5.0                                                                                             | 12.7                                                                                             | 9.3                                                                                            |
+|                                                         | Compute Config   | XLarge(128 vCore)                                                                              | XLarge(128 vCore)                                                                               | Large                                                                                            | Standard 600 Slot                                                                              |
+| **TPC-H 100GB** \[*Small-scale Reporting/BI Workloads*] | Duration(Second) | 22.7                                                                                           | 28.36                                                                                           | 50.6                                                                                             | 201                                                                                            |
+|                                                         | Compute Config   | XLarge(128 vCore)                                                                              | XLarge(128 vCore)                                                                               | Large                                                                                            | Standard 600 Slot                                                                              |
+| **TPC-DS 1TB** \[*Complex Queries*]                     | Duration(Second) | 208                                                                                            | 314                                                                                             | 386                                                                                              | 1248                                                                                           |
+|                                                         | Compute Config   | XLarge(128 vCore)                                                                              | XLarge(128 vCore)                                                                               | Large                                                                                            | Standard 600 Slot                                                                              |
+
+| Note  For this benchmark, no special tuning was applied. Singdata, Snowflake , Databricks and BigQuery were used with their default settings. |
+| --------------------------------------------------------------------------------------------------------------------------------------------- |
+
+4\. **Cost**
+
+![](.topwrite/assets/d33edb049c/34d17b11eb1bf216dc027b3d552760ab985b0c7f.png)
+
+|                                                         | ![](.topwrite/assets/d33edb049c/5fb9f76a0967823749b634044213fed99d77c62b.png =50)
Singdata                                                                                      | ![](.topwrite/assets/d33edb049c/76d05b199a886af460a051279cd95b11fd2abc21.png =50)
Snowflake                                                                                                | ![](.topwrite/assets/d33edb049c/73c2df840d19f5b449bef8ee02d520decc2c5655.png =50)
Databricks                                                                      | ![](.topwrite/assets/d33edb049c/c49a1b7f1c44b419e3a8c3b5b324879a025a6aa1.png =50)
BigQuery |
+| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
+| Region                                                  | AWS, Singapore                                                                                                                                                                      | AWS, AP Singapore                                                                                                                                                                              | AWS, AP Singapore                                                                                                                                                     | Singapore                                                                                      |
+| Pricing(USD)                                            | [0.0044 CRU/Second](vcluster_size_description.md) XLarge [$1.24/CRU\*h](pricing.md)  Enterprise Edition  $0.0055/Second | [0.0044 Credit/Second](https://docs.snowflake.com/en/user-guide/warehouses-overview) XLarge [$2.50 /credit](https://www.snowflake.com/en/pricing-options/)\*h  Standard Edition  $0.011/Second | [$0.0011 DBU /Second](https://www.databricks.com/product/pricing/product-pricing/instance-types)   SQL Serverless Large $0.88/DBU\*h  Premium Edition  $0.0098/Second | Standard  $0.049 slot/hour  300 Slot: $**0.0039/Second**  600 Slot:  $0.0082/Second            |
+| **SSB-FLAT 100GB** \[*Real-time Analytics*]             | $ 0.010                                                                                                                                                                             | $0.059                                                                                                                                                                                         | $0.12                                                                                                                                                                 | $0.08                                                                                          |
+| **TPC-H 100GB** \[*Small-scale Reporting/BI Workloads*] | $0.12                                                                                                                                                                               | $0.31                                                                                                                                                                                          | $0.50                                                                                                                                                                 | $0.16                                                                                          |
+| **TPC-DS 1TB** \[*Small-scale ETL Workloads*]           | $2.21                                                                                                                                                                               | $5.39                                                                                                                                                                                          | $6.32                                                                                                                                                                 | $10.23                                                                                         |
+
+5\. **Use Cases**
+
+|                       | ![](.topwrite/assets/d33edb049c/5fb9f76a0967823749b634044213fed99d77c62b.png =50)
Singdata | ![](.topwrite/assets/d33edb049c/76d05b199a886af460a051279cd95b11fd2abc21.png =50)
Snowflake | ![](.topwrite/assets/d33edb049c/73c2df840d19f5b449bef8ee02d520decc2c5655.png =50)
Databricks                                                                                       | ![](.topwrite/assets/d33edb049c/c49a1b7f1c44b419e3a8c3b5b324879a025a6aa1.png =50)
BigQuery |
+| --------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
+| Batch Data Process    | Yes,SQL and Zettapark                                                                          | Yes,SQL and Snowpark                                                                            | Yes, Spark SQL Declarative ETL framework simplifying batch and streaming development                                                                                                   | Yes,SQL and Serverless Spark                                                                   |
+| Continous Pipeline    | Streaming Ingestion：Append/Upsert  Table Stream , Dynamic Table                                | Snowpipe
Streaming
Stream+Task, Dynamic Table                                           | **Structured Streaming**: Append/Upsert  **Auto Loader**: Automatically detects and processes new data files  **Delta Live Tables**: Supports streaming tables and streaming pipelines | Yes,SQL and Serverless Spark                                                                   |
+| BI/Ad-hoc             | Yes                                                                                            | Yes                                                                                             | Yes                                                                                                                                                                                    | Yes                                                                                            |
+| Operational Analytics | Yes
Real-time CDC replication  Enable sub-second, high-concurrency analytics at scale      | Yes
Query acceleration: Optimized performance for operational workloads                     | Limited support                                                                                                                                                                        | Yes                                                                                            |
+
+^

package/bin/skills/lakehouse-doc-en/references/comprehensive_guide_to_ingesting_3rd_tools.md

@@ -0,0 +1,11 @@
+# Complete Guide to Importing Data into Singdata Lakehouse
+
+## Data Ingestion: Through Third-Party Tools
+
+#### Resources
+
+[Loading Data into Lakehouse via Airbyte](eltmoderndatastack.md)
+
+[Loading Data Collected by Octoparse into Lakehouse via MySQL Protocol](simpletosimple_bazhuayu_datagpt.md)
+
+[Real-Time Data Synchronization from Oracle Database to Singdata Lakehouse via Bluepipe](bluepipeoraclelakehouse_datasync.md)