@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.100

package/bin/skills/lakehouse-doc-en/references/transform-dt.md

@@ -0,0 +1,291 @@
+# Scenario 1: Incremental Transformation of ETL Tasks Based on Static Partitioning (No Dimension Table or Unchanged Dimension Table)
+
+## Business Background
+
+For traditional full batch processing tasks scheduled daily (T+1 mode), incremental processing optimization is achieved through dynamic partition table technology. After transformation, the following can be achieved:
+
+* Data freshness improvement: supports hourly/minute-level scheduling intervals
+* Resource utilization optimization: reduces redundant computation by approximately 60%-80%
+* O&M efficiency improvement: supports partition-level data repair and backfill
+
+## Scenario Characteristics
+
+Applicable scenarios must meet the following criteria:
+
+1. Business time aligns with natural days: typical day-cut scenarios such as order transactions and log collection
+2. Fixed scheduling window: it is recommended to concentrate processing during the data readiness window (e.g., 00:30-01:00)
+3. Standardized partition field: the source table must include a standard date partition field (e.g., event_day/pt/ds)
+
+## Transformation Approach: Using Partitioned Dynamic Tables
+
+#### 1. Create a Partitioned Dynamic Table
+
+When creating a partitioned dynamic table, define the parameter `SESSION_CONFIGS()['dt.args.event_day']` to specify the partition or partition range of the source table to scan. This parameter is used in the SQL processing logic and is defined via `SESSION_CONFIGS()['dt.args.xx']`, which represents the partition field for querying the source table.
+
+* Parameter format: `SESSION_CONFIGS()['dt.args.xx']`, where `xx` is a custom parameter name that must start with `dt.args.` to avoid conflicts with internal system fields.
+* Equivalent expression: equivalent to `SELECT * FROM source_table WHERE pt = ${bizdate}` in traditional scheduling. For example, `SESSION_CONFIGS()['dt.args.pt']` is equivalent to `pt = ${bizdate}`.
+
+#### 2. Refresh the Dynamic Table
+
+When refreshing the dynamic table, use the following command to specify the partition value:
+
+```SQL
+REFRESH DYNAMIC TABLE target_table PARTITION (pt = ${bizdate});
+```
+
+* Corresponding traditional operation: equivalent to the traditional `INSERT OVERWRITE target_table PARTITION (pt = ${bizdate})`.
+
+## Transformation Implementation Steps
+
+### Converting Offline Tasks to Incremental Tasks
+
+This section guides users on how to convert existing offline tasks into incremental tasks for more efficient data processing. The following is a specific procedure based on a "traditional database," applicable to scenarios where business logic aligns daily and scheduled refreshes occur daily.
+
+#### Step 1: Task Transformation
+
+Original SQL:
+
+```SQL
+CREATE TABLE target PARTITIONED BY (ds);
+
+-- 20250101 schedule
+SQL :=
+INSERT OVERWRITE TABLE target (ds='20250101')
+SELECT * 
+FROM 
+    src_1 AS t1
+LEFT OUTER JOIN 
+    src_2 as t2 WHERE t2.ds = '2025/01/01' AND t2.category = 'A'
+ON t1.id = t2.id;
+
+-- 20250102 schedule
+SQL :=
+INSERT OVERWRITE TABLE target (ds='20250102')
+SELECT * 
+FROM 
+    src_1 AS t1 
+LEFT OUTER JOIN 
+    src_2 as t2 WHERE t2.ds = '2025/01/02' AND t2.category = 'A'
+ON t1.id = t2.id;
+```
+
+First, replace all scheduler-injected parameters `${bizdate}` in the original SQL with `SESSION_CONFIGS()['dt.args.bizdate']`.
+
+1. Locate two consecutive scheduled tasks, as shown above: the 20250101 schedule and the 20250102 schedule.
+
+2. Identify the parts that change, such as the two highlighted sections above.
+
+3. Replace them with `SESSION_CONFIG()['dt.args.xx']`:
+
+   1. It is important to note that **`SESSION_CONFIG()['dt.args.xx']` returns a string result**. If the variable is an int, use SQL for type conversion, e.g., `CAST(SESSION_CONFIG()['dt.args.xx'] AS BIGINT)`.
+   2. And set up the refresh statement:
+   3. ```sql
+      CREATE DYNAMIC TABLE target (columns...)
+      PARTITIONED BY (ds)
+      --INSERT OVERWRITE TABLE target (ds='20250101')
+      AS
+      --SELECT *
+      -- The partition column must appear in the final schema to ensure column count alignment
+      SELECT t1.id, t1.col1, t2.col1, format(t1.ds) -- YYYY-MM-DD ==> YYYYMMDD
+      FROM 
+          src_1 AS t1 
+      LEFT OUTER JOIN 
+       --   src_2 as t2 WHERE t2.ds = '2025/01/02' AND t2.category = 'A'
+          src_2 as t2 WHERE t2.ds = SESSION_CONFIG()['dt.args.date2'] AND t2.category = 'A'
+      ON t1.id = t2.id;
+      ```
+
+#### Step 2: Schedule the Refresh Command
+
+```SQL
+set dt.args.date1 = {$date1};
+set dt.args.date2 = {$date2};
+REFRESH DYNAMIC TABLE target partition(ds = {$date1});
+```
+
+### Incremental Task Data Backfill and O&M
+
+In certain cases, users may need to backfill data into existing partitions.
+
+#### Method 1: Backfill Data to the Source Table
+
+Users can directly backfill data to the source table. This backfilled data will be automatically reflected in the Dynamic Table (DT) through the corresponding REFRESH tasks.
+
+**Procedure**:
+
+1. Insert or update data directly in the source table.
+2. Execute the REFRESH task to sync changes to the DT.
+
+#### Method 2: Use DML Statements to Directly Backfill Data to the DT
+
+Users can directly use DML statements to insert data into a specified partition of the DT. **However, note that if the upstream data has not changed, the next refresh operation will perform a full refresh, which will restore the data to its unmodified state.**
+
+**Procedure**:
+
+1. Use DML statements to insert data into a specific partition of the DT.
+2. Note that directly modifying the DT will trigger a full refresh of that partition on the next refresh. If you do not want a full refresh result, avoid scheduling a REFRESH task for that partition.
+
+**Sample Code**:
+
+```SQL
+set cz.optimizer.incremental.backfill.enabled=true;
+INSERT INTO DYNAMIC TABLE incremental_dt  VALUES (...);
+```
+
+**Notes**:
+
+* Data inserted directly into the DT will participate in the DT's downstream computation. If downstream old partitions do not need this data, do not schedule REFRESH tasks for partitions involving this data.
+* Other unaffected partitions can still undergo incremental refresh.
+
+# Scenario 2: ETL Task Optimization Based on Dynamic Partitioning (No Dimension Table or Unchanged Dimension Table)
+
+## Scenario Characteristics
+
+1. Dynamic data characteristics: source data may have historical partition updates
+2. Time window requirement: the business needs to continuously process data that may change within N days
+3. Dimension table situation: the dimension table is unchanged, no need to preserve historical dimensions, or no dimension table exists
+
+## Converting Offline Tasks to Incremental Tasks
+
+```sql
+CREATE TABLE target PARTITIONED BY (ds);
+
+-- 20250101 schedule
+SQL :=
+INSERT OVERWRITE TABLE target (ds)
+SELECT * 
+FROM 
+    src_1 AS t1 WHERE t1.ds = '2025-01-01'
+LEFT OUTER JOIN 
+    src_2 as t2 WHERE t2.ds = '2025/01/01' AND t2.category = 'A'
+ON t1.id = t2.id;
+
+-- 20250102 schedule
+SQL :=
+INSERT OVERWRITE TABLE target (ds)
+SELECT * 
+FROM 
+    src_1 AS t1 WHERE t1.ds = '2025-01-02'
+LEFT OUTER JOIN 
+    src_2 as t2 WHERE t2.ds = '2025/01/02' AND t2.category = 'A'
+ON t1.id = t2.id;
+```
+
+1. If the user's original job already relies on a scheduling system to fill parameters for static partition scheduling, it is not recommended to rewrite it as dynamic partitioning.
+2. If the user wants old partition data to remain unchanged after a certain stage, it is also not recommended to rewrite it as dynamic partitioning.
+3. When rewriting to dynamic partitioning, the main principle is: all computation is data-driven. That is, each piece of data should interact with other data according to the defined computation logic and be written to the corresponding target partition.
+4. A window filter should be added to the filter conditions based on actual business needs, such as 1 week/1 month/3 months/1 year, to prevent the DT data from continuously growing, which may affect performance over time.
+
+Next, let's try to rewrite the above static partitioning SQL into dynamic partitioning:
+
+```SQL
+CREATE DYNAMIC TABLE target (COLUMNS...)
+PARTITIONED BY (ds)
+--INSERT OVERWRITE TABLE target (ds)
+SELECT *, DATE_FORMAT(STR_TO_DATE(t1.ds, '%Y-%m-%d'), '%Y%m%d')
+FROM 
+ --   src_1 AS t1 WHERE t1.ds = '2025-01-02'
+    src_1 AS t1
+LEFT OUTER JOIN 
+ --   src_2 as t2 WHERE t2.ds = '2025/01/02' AND t2.category = 'A'
+    src_2 as t2
+ON t1.id = t2.id 
+AND t1.ds = DATE_FORMAT(STR_TO_DATE(t2.ds, '%Y/%m/%d'), '%Y-%m-%d')
+WHERE t2.category = 'A';
+```
+
+## Incremental Task Data Backfill and O&M
+
+1. First, backfill the source table data. If the table data comes from an external source, use the data integration tool to re-sync.
+
+2. Historical data backfill:
+
+   1. Impact: The backfilled data will be processed as incremental data. This batch processing will slow down and affect the freshness of the latest data processing.
+
+   2. Recommendations:
+
+      1. Expand computing resources in advance to ensure processing performance, and perform backfill in batches to reduce the amount of computation per batch.
+      2. Alternatively, after the backfill is complete, trigger a full processing run.
+
+3. Dimension table modification:
+
+   1. Impact: Dimension table modifications directly affect incremental computation. If the scope of modification is too large, it may affect processing performance.
+
+   2. Recommendations:
+
+      1. If the impact scope is small, directly modify it, and the incremental computation will automatically detect and process it.
+      2. If the impact scope is large, perform a full processing run after the modification.
+      3. If it affects the code logic (e.g., adding columns requires adding computation logic), you need to modify the DT logic and perform a full recomputation.
+
+# Scenario 3: Daily Partitioning Tasks (Historical Dimensions Need to Be Preserved)
+
+## Business Background
+
+For daily-partitioned ETL scenarios with continuously changing dimension tables, incremental collaborative computation between fact tables and dimension tables is achieved by building a real-time synchronization system for dimension tables and dynamic processing pipelines. After transformation, the following can be achieved:
+
+* Improved dimension data freshness: dimension change latency reduced from T+1 to minute-level
+* Cross-table data consistency guarantee: establish a version alignment mechanism between dimension and fact tables
+* Reduced resource consumption: avoid redundant computation caused by full synchronization of dimension tables
+
+## Scenario Characteristics
+
+Applicable scenarios must meet the following criteria:
+
+1. Hybrid data processing mode: dimension tables must support real-time updates + historical version retention
+2. Version alignment requirement: fact table processing must bind to the dimension snapshot at the corresponding point in time
+
+## Converting Offline Tasks to Incremental Tasks
+
+![](.topwrite/assets/image_1747648578327.png)
+
+## Transformation Implementation Steps
+
+### 1. Real-Time Transformation of Dimension Tables
+
+#### 1. Multi-Source Synchronization Architecture
+
+* **Real-Time Sync Layer**: If the data source is an external system (e.g., MySQL), use the StudioCDC capture tool to achieve real-time multi-table synchronization from the source database. Real-time synchronization requires that the fields be consistent with the source side. Therefore, if the dimension table contains partitions, a new dimension table needs to be created to store daily dimension partition snapshots.
+
+### 2. Incremental Transformation of Processing Tasks
+
+* **Version Binding**: The event time of the fact table is automatically associated with the corresponding day's dimension partition.
+
+The following is an example of incremental transformation of a processing task. Assume t1 is a daily dimension partition table and t2 is a fact table:
+
+```sql
+CREATE DYNAMIC TABLE target (columns...)
+PARTITIONED BY (ds)
+--INSERT OVERWRITE TABLE target (ds='20250101')
+AS
+SELECT *
+-- The partition column must appear in the final schema to ensure column count alignment
+SELECT t1.id, t1.col1, t2.col1, format(t1.ds) -- YYYY-MM-DD ==> YYYYMMDD
+FROM 
+--    src_1 AS t1 WHERE t1.ds = '2025-01-02'
+    src_1 AS t1 WHERE t1.ds = SESSION_CONFIG()['dt.args.date2']
+LEFT OUTER JOIN 
+ --   src_2 as t2 WHERE t2.ds = '2025/01/02' AND t2.category = 'A'
+    src_2 as t2 WHERE t2.ds = SESSION_CONFIG()['dt.args.date3'] AND t2.category = 'A'
+ON t1.id = t2.id;
+```
+
+## Incremental Task Data Backfill and O&M
+
+1. **Historical Data Backfill**: Use the backfill feature to modify input table data day by day, then refresh the Dynamic Table day by day. Incremental computation on partition tables can be refreshed daily without affecting the computation of the current day's latest data.
+
+2. **Modifying Dimension Tables**:
+
+   1. If the impact scope is small, directly modify it, and the incremental computation will automatically detect and process it.
+   2. If the impact scope is large, perform a full processing run after the modification.
+   3. If it affects the code logic (e.g., adding columns requires adding computation logic), you need to modify the DT logic and perform a full recomputation.
+
+# Notes on Incremental Computation Performance in Special Scenarios
+
+In the following scenarios, incremental computation can still execute normally, but the performance optimization effect may be limited. In the worst case, computation performance may be comparable to full computation. It is recommended to conduct actual tests in these scenarios to evaluate the benefits of incremental computation before deciding whether to adopt it.
+
+1. **Frequent Changes to Right Tables in Outer Joins**: Queries contain a large number of Outer Joins, and the right-side table data involved in the Outer Joins is in a state of frequent changes, with each change exceeding 5% of the data volume.
+2. **Large Number of Sorting Operations**: Queries involve a significant amount of data sorting requirements, such as using `ORDER BY` clauses.
+3. **Window Function Sorting and Large Partitions**: Window functions require data sorting (except when RowNumber=1), and the incremental data contains multiple partitions with particularly large data volumes.
+4. **Poor Data Clustering**: Data lacks good clustering characteristics and cannot clearly distinguish cold data from hot data via Join key, Aggregate key, Window partition key, etc.
+
+^

package/bin/skills/lakehouse-doc-en/references/trial-account-quotas-and-limits.md

@@ -0,0 +1,85 @@
+# Trial Account Quotas and Limitations Description
+
+## Introduction to Trial Accounts
+
+Trial accounts are a free experience form provided by Singdata Lakehouse to help new users experience the platform's features and performance at zero cost.
+
+Each account, when newly created, will first become a "trial" status user. You can see the "trial" status identifier on the left side of the management center. After creating a service instance, the account will receive a voucher with a total amount of 200 yuan, valid for 30 days, for product experience.
+
+![](.topwrite/assets/image_1740710821088.png =189)
+
+To ensure the overall operational efficiency of the platform and fair resource allocation, trial users will be subject to certain restrictions during use. The specific limitations are as follows.
+
+##
+
+## Quotas and Usage Limitations Overview
+
+| **Feature**           | **Limitation Description**                                                                                | **Limit Value / Rules**                                                                                                                                                                                                                 | **Quota Upgrade Method**         |
+| --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
+| Global                | Maximum number of users that can be created under a single tenant                                         | Maximum 500                                                                                                                                                                                                                             | Non-changeable                   |
+|                       | Number of instances that can be created under a single tenant                                             | Instances can be opened in any region, with a total not exceeding 1                                                                                                                                                                     | Upgrade to formal account        |
+|                       | Number of workspaces that can be created under a single instance                                          | No more than 100                                                                                                                                                                                                                        | Non-changeable                   |
+| Data Integration      | Offline synchronization tasks:                                                                            | The concurrency limit for a single task is less than 10; the number of tasks running simultaneously does not exceed 2. Task execution time does not exceed 30 minutes.                                                                  | Upgrade to formal account        |
+|                       | Real-time integration tasks                                                                               | The number of tasks running simultaneously does not exceed 2. Task running time does not exceed 2 hours.                                                                                                                                | Upgrade to formal account        |
+|                       | Multi-table real-time synchronization tasks                                                               | During the multi-table real-time synchronization task, offline integration tasks and real-time integration tasks will be used separately, occupying the quotas of offline tasks and real-time tasks respectively.                       | Upgrade to formal account        |
+| Data Development      | Maximum number of lines in SQL/Python/Shell/JDBC task files                                               | Maximum 5000 lines of code                                                                                                                                                                                                              | Non-changeable                   |
+|                       | SQL query result return row limit                                                                         | Default 1000 rows, user adjustable up to a maximum of 10000 rows                                                                                                                                                                        | Non-changeable                   |
+|                       | SQL query result download limit                                                                           | Maximum support for downloading 4GiB data                                                                                                                                                                                               | Immutable                        |
+|                       | Temporary query run history retention time                                                                | Retained for up to 7 days                                                                                                                                                                                                               | Immutable                        |
+|                       | Historical version and commit version retention time                                                      | Permanently retained                                                                                                                                                                                                                    | Immutable                        |
+| Task Scheduling       | Limit on the number of task instances started concurrently by a single tenant                             | Offline sync task instances: 20 SQL task instances: Unlimited Python task instances: 5 Shell task instances: 5 JDBC task instances: 20                                                                                                  | Upgrade to formal account        |
+|                       | Maximum system fallback runtime timeout for a single task instance                                        | Minimum: 1 minute Maximum: 168 hours (7 days)                                                                                                                                                                                           | Immutable                        |
+|                       | Instance auto-retry interval                                                                              | Minimum: 1 minute Maximum: 1 day                                                                                                                                                                                                        | Immutable                        |
+|                       | Scheduling wait time                                                                                      | Minimum: 1 minute Maximum: 168 hours (7 days)                                                                                                                                                                                           | Immutable                        |
+| Operations Center     | Task instance history retention time                                                                      | Periodic instances: 32 days Supplementary instances: 32 days Temporary instances: 8 days                                                                                                                                                | Immutable                        |
+|                       | Query task instance list limit                                                                            | Display results are subject to instance retention time limit, up to 6 months                                                                                                                                                            | Immutable                        |
+|                       | Task operation log query time range limit                                                                 | 6 months                                                                                                                                                                                                                                | Immutable                        |
+|                       | Task operation log history retention time limit                                                           | 6 months                                                                                                                                                                                                                                | Immutable                        |
+| Compute Cluster       | Maximum total specification of compute cluster instances that can be created under a single instance      | Total specification of all VCs does not exceed 28 CRU: of which the default usage in the sys workspace is 9 CRU quota, and the remaining total specification of all other workspaces under the service instance does not exceed 19 CRU. | Upgrade to formal account        |
+|                       | Concurrent Compilation Jobs                                                                               | No more than 2                                                                                                                                                                                                                          | Contact Singdata for adjustments |
+| Data Objects          | Maximum number of data objects (e.g., tables, views, etc.) that can be created under a single instance    | 1000                                                                                                                                                                                                                                    | Upgrade to a formal account      |
+| Stream Writing        | Maximum number of Stream objects that can be created under a single instance                              | 10                                                                                                                                                                                                                                      | Upgrade to a formal account      |
+| Job History           | Limitations on querying job list                                                                          | Time range: maximum 7 days; number of job history records returned: up to 10,000; job history records exceeding the above quota can be queried using SQL in the sys.information\_schema.job\_history view.                              | Cannot be changed                |
+|                       | Daily SMS sending limit                                                                                   | **Verification code**: Up to 1 per minute; up to 20 per hour; up to 40 per natural day. **Alert notifications, etc**.: Up to 50 per natural day.                                                                                        | Cannot be changed                |
+| Monitoring and Alerts | Limit on the number of monitoring rules                                                                   | No more than 5 monitoring rules can be enabled simultaneously                                                                                                                                                                           | Contact Singdata for adjustments |
+|                       | Time range for querying alert events                                                                      | Maximum 7 days                                                                                                                                                                                                                          | Cannot be changed                |
+|                       | Maximum number of notifications in notification policy configuration                                      | Up to 12 times                                                                                                                                                                                                                          | Cannot be changed                |
+| Data Directory        | File upload size limit                                                                                    | Maximum 2 GiB                                                                                                                                                                                                                           | Cannot be changed                |
+| Data Quality          | Time limit for querying validation results                                                                | Validation results are retained for 3 months                                                                                                                                                                                            | Cannot be changed                |
+| Security Center       | Limit on the number of network policies that can be created under a single instance                       | Up to 20                                                                                                                                                                                                                                | Cannot be changed                |
+|                       | Limit on the number of whitelist and blacklist entries that can be recorded under a single network policy | Up to 100,000 IPs                                                                                                                                                                                                                       | Cannot be changed                |
+
+##
+
+## Trial Period and Data Processing
+
+**Limitations**:
+
+The trial period for new accounts is until 30 natural days after the service instance is activated or until the 200 yuan voucher is used up.
+
+**Post-expiration processing**:
+During the trial period, you can apply to upgrade to a formal account at any time, or you can apply to upgrade to a formal account after the trial period ends. After the trial period ends, you will not be able to access or continue using the service instances created during the trial. The resource configurations and data within the service instances will be retained for 7 calendar days, after which the resource configurations and data within the service instances will be cleared and cannot be recovered.
+
+If you need to continue using the service, please contact Singdata business within 7 calendar days after the trial period ends to apply for an upgrade to a formal account.
+
+***
+
+## Handling Exceeding Quotas
+
+When your usage exceeds the quota limit, your operations will receive a system error message indicating the exceeded quota limit. Please clean up unnecessary resources or handle it according to the corresponding quota increase method in the second section "Quota and Usage Limit Overview."
+
+***
+
+## Temporary Disablement or Suspension
+
+If you occupy too many system resources through abnormal usage, Singdata Lakehouse may temporarily disable your trial account or restrict the usage permissions of certain functions to ensure the normal use of other users. If such restrictions are applied, you will be notified through the phone number reserved in your account.
+
+***
+
+## Contact Information
+
+![](.topwrite/assets/20250708-181924.jpeg =611)
+
+* **Email**: <<service@singdata.com>>
+
+^

package/bin/skills/lakehouse-doc-en/references/tutorial_DataGPT.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/tutorial_connect_to_lakehouse.md

@@ -0,0 +1 @@
+# Tutorial: Connect to Lakehouse

package/bin/skills/lakehouse-doc-en/references/tutorial_data_ingestion.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/tutorial_data_transformation.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/tutorial_migration.md

@@ -0,0 +1,49 @@
+# Migration Guide
+
+This section collects hands-on guides for migrating existing data systems to Singdata Lakehouse, covering the most common migration paths: Spark/PySpark, Snowflake, SQL syntax, and more.
+
+---
+
+## Migration Path Overview
+
+| Source System | Recommended Path | Documentation |
+|---------|---------|------|
+| Databricks / PySpark | ZettaPark DataFrame API replacement | [PySpark → ZettaPark Migration in Practice](pyspark-to-zettapark-migration-f1.md) |
+| PySpark RDD (legacy code) | RDD → declarative DataFrame/SQL | [RDD → ZettaPark Migration in Practice](rdd-to-zettapark-migration-weblog.md) |
+| Spark SQL | SQL syntax comparison migration | [Spark SQL Syntax Migration Guide](migration-spark-sql.md) |
+| Spark data engineering projects | Architecture migration best practices | [Spark Data Engineering Migration Best Practices](migrate-spark-data-engineering-best-practices-to-lakehouse.md) |
+| Spark jobs (production) | Smooth migration with minimal changes | [Spark Job Smooth Migration Guide](spark-migration-guide.md) |
+| Snowflake | ETL Pipeline migration | [Snowflake Real-Time ETL Migration](migrate-snowflake-realtime-etl-to-lakehouse.md) |
+| Build Medallion from scratch | Bronze → Silver → Gold modeling | [Building a Medallion Lakehouse from Scratch](medallion-lakehouse-from-scratch.md) |
+
+---
+
+## Choosing a Migration Path
+
+**You have existing PySpark code and want to migrate it directly**
+
+Use the ZettaPark DataFrame API. 90% of your code can be reused as-is, with changes concentrated in 4 areas (import paths, Session creation, `.collect()`, file paths). See [PySpark → ZettaPark Migration in Practice](pyspark-to-zettapark-migration-f1.md) for complete before/after code comparisons and 4 migration notes.
+
+**You have RDD code (Spark 1.x legacy project) and want to migrate to Lakehouse**
+
+See [RDD → ZettaPark Migration in Practice](rdd-to-zettapark-migration-weblog.md). The core change is moving from imperative (`map/reduceByKey/aggregateByKey`) to declarative (`group_by/agg/F.avg()`), resulting in less code and better execution efficiency. Replacing `aggregateByKey` with `F.avg()` yields the greatest reduction in code volume.
+
+**Starting from scratch and want to build a Medallion architecture on Lakehouse**
+
+See [Building a Medallion Lakehouse from Scratch](medallion-lakehouse-from-scratch.md). The guide covers Bronze raw ingestion, Silver cleansing and deduplication, Gold dimensional modeling (including surrogate key generation), and a complete implementation with 22 automated validations.
+
+**Migrating only SQL, not touching the compute layer**
+
+See [Spark SQL Syntax Migration Guide](migration-spark-sql.md) and [Data Type Compatibility Reference](migration-sql-compatibility.md).
+
+**Production Spark jobs requiring minimal downtime**
+
+See [Spark Job Smooth Migration Guide](spark-migration-guide.md), which covers dual-write validation, gradual traffic cutover, and other production migration strategies.
+
+---
+
+## Related Documentation
+
+- [ZettaPark DataFrame API Guide](zettapark-dataframe-guide.md): Complete ZettaPark API reference
+- [Data Type Compatibility Reference](migration-sql-compatibility.md): MySQL/PostgreSQL/Hive/Spark type mappings
+- [Volume Usage Guide](zettapark-volume-guide.md): File storage path format after migration (`vol://schema.vol/path`)

package/bin/skills/lakehouse-doc-en/references/tutorial_virtual_cluster.md

@@ -0,0 +1,149 @@
+# Manage Compute Resources
+
+Singdata Lakehouse uses a storage-compute separation architecture — storage and compute scale independently and elastically. A compute cluster (VCluster) is the compute unit that executes all SQL jobs. It is billed by actual runtime and automatically stops when idle, incurring no charges.
+
+---
+
+## Three Cluster Types
+
+| Type | Use case | Spec unit | Elasticity |
+|------|---------|---------|---------|
+| **General Purpose (GP)** | ETL batch processing, offline jobs | 1–256 CRU, step 1 | Vertical scaling: configure `MIN_VCLUSTER_SIZE` / `MAX_VCLUSTER_SIZE`, auto-adjusts spec based on load |
+| **Analytical (AP)** | BI reports, ad-hoc queries, high-concurrency online queries | 1–256 CRU, must be a power of 2 (1/2/4/8/16…) | Horizontal scaling: configure `MIN_REPLICAS` / `MAX_REPLICAS`, auto-adds/removes instances based on concurrency (up to 10) |
+| **Integration** | Real-time/offline data integration tasks | 0.25–256 CRU | Vertical scaling: configure `MIN_VCLUSTER_SIZE` / `MAX_VCLUSTER_SIZE`, auto-adjusts spec based on load |
+
+> **Selection principle**: Use General Purpose for ETL, Analytical for BI and online queries, Integration for data sync tasks. Multiple clusters can be created in the same workspace to isolate resources for different workloads.
+
+![](/.topwrite/assets/10-vcluster-elasticity.svg)
+
+Singdata Lakehouse compute cluster elasticity operates at three levels:
+
+**GP (General Purpose) — Vertical elasticity**: Single cluster spec auto-scales with load, from 1 CRU up to 256 CRU. Automatically scales up during peak load and scales back down after the peak; automatically stops billing after idle timeout. Suitable for ETL and other scenarios where a single job needs to consume full resources.
+
+**AP (Analytical) — Horizontal elasticity**: When concurrent query count reaches the current instance limit, new instances (Replicas) are automatically added, up to 10 instances, linearly increasing total concurrency capacity. Instances are automatically reduced when concurrency drops; in-flight queries are not affected. Suitable for BI reports and high-concurrency online query scenarios.
+
+**Integration — Fine-grained vertical elasticity**: Minimum spec of 0.25 CRU, precisely matching the actual load of sync tasks, with extremely high resource utilization and costs that scale linearly with task count. Suitable for real-time CDC and offline integration tasks.
+
+All three cluster types support **second-level cold start** (auto-wake on job submission), **auto-stop when idle** (no billing), and **no interruption to in-flight jobs during scaling**.
+
+---
+
+## This Section
+
+| Page | Description |
+|------|------|
+| [Using Lakehouse Compute Clusters](working_with_vclusters.md) | Cluster creation, start/stop, spec adjustment, SQL operations, and best practices for common use cases |
+| [Supporting Multi-Concurrent Queries with Horizontal Elastic Scaling](concurrency_scaling.md) | How multi-instance scaling works for Analytical clusters and hands-on tutorial |
+
+---
+
+## Common Operations
+
+### View Clusters
+
+```sql
+-- View all clusters in the current workspace
+SHOW VCLUSTERS;
+
+-- View only running clusters
+SHOW VCLUSTERS WHERE state = 'RUNNING';
+
+-- View detailed configuration of a specific cluster (spec, status, job count, etc.)
+DESC VCLUSTER my_cluster;
+```
+
+### Switch Clusters
+
+```sql
+-- Switch the current session to a specific cluster
+USE VCLUSTER my_cluster;
+
+-- Confirm the currently active cluster
+SELECT CURRENT_VCLUSTER();
+```
+
+> ⚠️ **Note**: In the Studio interface, it is recommended to switch clusters using the dropdown menu at the top of the page. If you run `USE VCLUSTER` in the SQL editor, you must select that statement together with the subsequent SQL before running.
+
+### Start and Stop
+
+```sql
+-- Start a cluster
+ALTER VCLUSTER my_cluster RESUME;
+
+-- Stop a cluster (waits for current jobs to complete)
+ALTER VCLUSTER my_cluster SUSPEND;
+
+-- Force stop (immediately terminates all running jobs)
+ALTER VCLUSTER my_cluster SUSPEND FORCE;
+```
+
+### Adjust Specifications
+
+```sql
+-- General Purpose: fixed spec
+ALTER VCLUSTER my_gp_cluster SET VCLUSTER_SIZE = 4;
+
+-- General Purpose: enable elastic scaling (min 2 CRU, max 8 CRU)
+ALTER VCLUSTER my_gp_cluster SET MIN_VCLUSTER_SIZE = 2 MAX_VCLUSTER_SIZE = 8;
+
+-- Analytical: adjust instance count range (horizontal scaling)
+ALTER VCLUSTER my_ap_cluster SET MIN_REPLICAS = 1 MAX_REPLICAS = 4;
+
+-- Analytical: adjust max concurrency per instance
+ALTER VCLUSTER my_ap_cluster SET MAX_CONCURRENCY = 16;
+```
+
+### View Cluster Jobs
+
+```sql
+-- View all jobs in a specific cluster
+SHOW JOBS IN VCLUSTER my_cluster;
+
+-- View jobs running longer than 2 minutes
+SHOW JOBS IN VCLUSTER my_cluster WHERE execution_time > INTERVAL 2 MINUTE;
+
+-- View failed jobs
+SHOW JOBS WHERE status = 'FAILED' LIMIT 20;
+```
+
+### Configure Auto-Suspend
+
+```sql
+-- Auto-suspend after 60 seconds idle (recommended for ETL clusters)
+ALTER VCLUSTER etl_cluster SET AUTO_SUSPEND_IN_SECOND = 60;
+
+-- Auto-suspend after 30 minutes idle (recommended for BI query clusters, preserves cache)
+ALTER VCLUSTER bi_cluster SET AUTO_SUSPEND_IN_SECOND = 1800;
+
+-- Never auto-suspend
+ALTER VCLUSTER my_cluster SET AUTO_SUSPEND_IN_SECOND = -1;
+```
+
+### Configure Cache Preloading (Analytical clusters only)
+
+```sql
+-- Preload hot tables into cluster local SSD to accelerate BI queries
+ALTER VCLUSTER bi_cluster SET PRELOAD_TABLES = "dws.daily_sales,dws.user_profile";
+
+-- View preload status
+SHOW PRELOAD CACHED STATUS;
+```
+
+### Set Job Timeout
+
+```sql
+-- Limit single job execution to 10 minutes (prevents runaway queries from consuming resources)
+ALTER VCLUSTER my_cluster SET QUERY_RUNTIME_LIMIT_IN_SECOND = 600;
+```
+
+---
+
+## Related Documentation
+
+| Document | Description |
+|------|------|
+| [Compute Cluster](virtual-cluster.md) | Core concepts, spec selection reference table, Web UI screenshots |
+| [Compute Resource DDL](compute-resource-ddl.md) | Full syntax for CREATE / ALTER / DROP VCLUSTER |
+| [Compute Cluster Cache](vc_cache.md) | How active and passive caching work |
+| [SHOW JOBS](show-jobs.md) | View and filter job records |
+| [Compute Cluster Spec Code Change Description](vcluster_size_description.md) | Old vs. new spec code mapping table |

package/bin/skills/lakehouse-doc-en/references/tutorial_work_with_workspace.md

@@ -0,0 +1 @@
+# Tutorial: Workspace

package/bin/skills/lakehouse-doc-en/references/tutorial_zettapark.md

@@ -0,0 +1 @@
+# Tutorial: ZettaPark