@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.100

package/bin/skills/lakehouse-doc-en/references/dynamicTable-parmaters.md

@@ -0,0 +1,250 @@
+# Parameterized Definition Dynamic Table Creation Syntax
+
+When the dynamic table is a partitioned table, parameters can be passed. Parameters are defined through SESSION\_CONFIGS()\['dt.arg.xx']. SESSION\_CONFIGS() is a built-in system function. For non-partitioned tables, if parameterized definitions are used and the passed parameter values remain unchanged, the system will perform incremental refresh. However, once the parameter values change, the system will perform a full refresh, as this is equivalent to changing the table definition. For partitioned tables, the partition key must be a parameter field, but these parameter fields are not required to be partition columns of the table.
+
+## Creation Statement
+
+The parameterized definition of a Dynamic Table allows users to define parameters through `SESSION_CONFIGS()['dt.arg.xx']`. These parameters are referenced when creating the table, thereby dynamically affecting the content of the table.
+
+* `SESSION_CONFIGS()`：A built-in system function used to read parameters from the configuration.
+* `'dt.arg.xx'`：The name of the DT parameter, which must start with `dt.arg.` to avoid conflicts with internal system fields.
+
+### Example Code
+```sql
+CREATE dynamic table  event_gettime_pt 
+partitioned by(event_day)
+AS SELECT
+  event,
+  process,
+  YEAR(event_time) event_year,
+  MONTH(event_time) event_month,
+  DAY(event_time) event_day
+FROM event_tb_pt
+where day(event_time)=SESSION_CONFIGS()['dt.args.event_day'];
+set dt.args.event_day=19;
+refresh dynamic table event_gettime_pt partition(event_day=19);
+```
+In this example, `dt.args.event_day` is a DT parameter, which is read from the configuration using the `SESSION_CONFIGS()` function. If non-string type parameters are needed, the `CAST` function can be used for type conversion.
+
+## Refresh Statement
+
+The refresh behavior of a parameterized Dynamic Table depends on whether the table is partitioned.
+
+### Non-Partitioned Table Refresh Syntax
+```sql
+REFRESH DYNAMIC TABLE dt;
+```
+* The parameter values of non-partitioned tables remain unchanged and will be incrementally refreshed.
+* The parameter values of non-partitioned tables change, and a full refresh will occur.
+
+### Partitioned Table Refresh Syntax
+```sql
+REFRESH DYNAMIC TABLE dt PARTITION partition_spec;
+```
+When refreshing a partitioned table, the `partition_spec` must be specified in the order of the table's partition hierarchy. This means that if the table is partitioned by multiple fields, these fields need to be specified from the highest level to the lowest level.
+
+### Example Description
+
+Assuming a table is partitioned by `day`, `hour`, and `min` in three levels, the correct way to specify the `partition_spec` is as follows:
+
+**Valid Specification**: You can specify higher-level and some lower-level partitions, but you cannot skip any intermediate level partitions.
+```sql
+set dt.args.day=2024-11-13;
+set dt.args.hour=23;
+REFRESH DYNAMIC TABLE dt PARTITION (day='2024-11-13', hour=23); 
+```
+In this example, `day` and `hour` are specified, while the `min` partition can be ignored.
+
+**Invalid Specification**: Skipping the specification of any intermediate level partitions is not allowed.
+```sql
+set dt.args.day=2024-11-13;
+set dt.args.hour=30;
+REFRESH DYNAMIC TABLE dt PARTITION (day='2024-11-13', min=30); 
+```
+## Precautions
+
+* Parameter and Partition Consistency
+  When performing a refresh operation on a Dynamic Table, you must ensure that the parameter values passed in are consistent with the specified partition values to be refreshed. If there is any inconsistency, the system will report an error during execution.
+
+  * **Error Example**：
+    ```SQL
+    set dt.args.event_day=11; 
+    refresh dynamic table event_gettime_pt partition(event_day=19);
+    ```
+    
+    In the above situation, although the parameter event_day is set to 11, the refresh command attempts to write to partition event_day=19. This will cause the actual calculated partition result to be inconsistent with the specified partition, resulting in errors.
+    
+* **Correct Example**:
+  ```sql
+  set dt.args.event_day=19;
+  refresh dynamic table event_gettime_pt partition(event_day=19);
+  ```
+* Concurrent Refresh Tasks
+  In these commands, the parameter values match the partition values, so they can be executed concurrently without conflicts. As long as there are no conflicts between partitions, the system allows multiple partition refresh tasks to be executed simultaneously.
+```sql
+-- Set parameters for partition event_day=19 and refresh
+set dt.args.event_day=19;
+refresh dynamic table event_gettime_pt partition(event_day=19);
+
+-- Set parameters for partition event_day=20 and refresh
+set dt.args.event_day=20;
+refresh dynamic table event_gettime_pt partition(event_day=20);
+```
+## Full Refresh and Incremental Refresh
+
+* Full Refresh
+  Full refresh occurs in the following situations:
+  * The parameter values of a non-partitioned table change.
+  * The parameter values of a partitioned table change, causing the partition conditions to change.
+    Example code
+```sql
+-- Non-partitioned table
+SET dt.args.x=1;
+REFRESH DYNAMIC TABLE T; -- First time creating DT, parameter x=1, full refresh
+SET dt.args.x=2;
+REFRESH DYNAMIC TABLE T; -- Parameter value changes, full refresh
+-- Partitioned table
+SET dt.args.pt=1;
+SET dt.args.x=1;
+REFRESH DYNAMIC TABLE T PARTITION (pt = 1); -- First time creating DT pt=1, parameter x=1, full refresh
+SET dt.args.x=3;
+REFRESH DYNAMIC TABLE T PARTITION (pt = 1); -- Parameter value changes, full refresh partition pt = 1
+```
+* Incremental Refresh
+  Incremental refresh occurs in the following situations:
+  * The parameter values of the non-partitioned table remain unchanged.
+  * The parameter values of the partitioned table remain unchanged, and the partition conditions have not changed.
+    Example code
+```sql
+-- Non-partitioned table
+SET dt.args.x=1;
+REFRESH DYNAMIC TABLE T; -- Parameter x=1 remains unchanged, incremental refresh
+
+-- Partitioned table
+SET dt.args.pt=1;
+SET dt.args.x=1;
+REFRESH DYNAMIC TABLE T PARTITION (pt = 1); -- Parameter x=1 remains unchanged, incremental refresh for partition pt = 1
+```
+## Scenario Case
+
+**Case 1: Converting Offline Tasks to Incremental Tasks**
+This section will guide users on how to convert existing offline tasks into incremental tasks for more efficient data processing. The following are specific operational steps based on a "traditional database," suitable for scenarios where business logic is aligned and scheduled for daily refresh.
+
+* Step 1: Parameterize the original SQL. The original SQL is as follows
+  ```sql
+  with tmp_channel as (
+    select
+      channel_code,
+      channel_name,
+      channel_type,
+      channel_uid
+    from dim.dim_shop_sales_channel_main
+    where pt = '${bizdate}'
+  ), tmp_bac_misc as (
+    select
+      mini_number,
+      bac_no
+    from dim.dim_customer_bac_misc_df
+    where pt = '${bizdate}'
+  ), tmp_fxiaoke as (
+    select
+      case
+        when record_type in ('dealer__c') then nvl(bac_no, account_no)
+        else account_no
+      end as channel_code,
+      id,
+      account_no
+    from ods.ods_account_obj as a
+    left join tmp_bac_misc on a.account_no = tmp_bac_misc.mini_number
+    where pt = '${bizdate}' and account_no is not null
+  --  and is_deleted = 0
+  --  and life_status not in ('invalid', 'ineffective')
+  )
+  insert overwrite table dim.dim_shop_sales_channel_misc partition(pt='${bizdate}')
+  select
+    tmp_channel.channel_code,
+    channel_name,
+    channel_type,
+    channel_uid,
+    id as fxiaoke_id,
+    account_no as fxiaoke_account_no
+  from tmp_channel
+  left join tmp_fxiaoke on tmp_channel.channel_code = tmp_fxiaoke.channel_code ;
+  ```
+First, all parameters `${bizdate}` passed in by the scheduling engine in the original SQL need to be replaced with `SESSION_CONFIGS()['dt.args.bizdate']`. This step will allow the parameter values to be dynamically passed in through configuration, rather than being hard-coded in the SQL.
+
+**Original SQL Parameter Replacement**:
+Replace all `${bizdate}` with `SESSION_CONFIGS()['dt.args.bizdate']`:
+  ```sql
+  create dynamic table im.dim_shop_sales_channel_misc
+  partitioned by(pt)
+  with tmp_channel as (
+    select
+      channel_code,
+      channel_name,
+      channel_type,
+      channel_uidfrom dim.dim_shop_sales_channel_main
+      where pt = SESSION_CONFIGS()['dt.args.bizdate']
+  ), tmp_bac_misc as (
+    select
+      mini_number,
+      bac_nofrom dim.dim_customer_bac_misc_df
+      where pt = SESSION_CONFIGS()['dt.args.bizdate']
+  ), tmp_fxiaoke as (
+    select
+      case
+        when record_type in ('dealer__c') then nvl(bac_no, account_no)
+        else account_noend as channel_code,
+      id,
+      account_nofrom ods.ods_account_obj as aleft join tmp_bac_misc on a.account_no = tmp_bac_misc.mini_number
+      where pt = SESSION_CONFIGS()['dt.args.bizdate'] and account_no is not null
+  )
+  select
+    tmp_channel.channel_code,
+    channel_name,
+    channel_type,
+    channel_uid,
+    id as fxiaoke_id,
+    account_no as fxiaoke_account_no,
+    pt
+  from tmp_channel
+  left join tmp_fxiaoke on tmp_channel.channel_code = tmp_fxiaoke.channel_code
+  ;
+  ```
+* Step 2: Schedule Refresh Command
+  During each scheduling, the parameter `dt.args.bizdate` needs to be set to a specific date value, and the refresh command should be executed.
+
+  Example of scheduling refresh command
+```sql
+SET dt.args.bizdate=20241130; -- ${bizdate} is replaced by Studio with a specific value each time
+REFRESH DYNAMIC TABLE DT PARTITION (pt ='20241130');
+```
+**Case 2: Incremental Task Data Supplementation, in some cases, users may need to supplement data to existing partitions**.
+
+* Method 1: Supplement data to the source table, users can directly supplement data to the source table. These supplemented data will be automatically reflected in the Dynamic Table (DT) through the corresponding REFRESH task.
+
+  Operation Steps
+
+  1. Directly insert or update data in the source table.
+  2. Execute the REFRESH task to synchronize the changes to the DT.
+* Method 2: Use DML statements to directly supplement data to the DT, users can also use DML statements to insert data into specific partitions of the DT.
+  Operation Steps
+  1. Use DML statements to insert data into specific partitions of the DT.
+  2. Note that directly modifying the DT will result in a full refresh of that partition the next time. If users do not want a full refresh result, they should avoid scheduling the REFRESH task for that partition.
+     Example Code
+  ```sql
+  INSERT INTO DYNAMIC TABLE incremental_dt VALUES (...);
+  ```
+**Notes**
+
+* Data directly inserted into DT will participate in downstream calculations of DT. If the downstream old partitions do not require this data, please do not schedule REFRESH tasks involving these partitions.
+* Other unaffected partitions can still be incrementally refreshed.
+
+**Case Three: Executing incremental tasks in different VCs. For parameterized partitioned DT, refresh tasks for different partitions can be executed simultaneously. Users can allocate different REFRESH tasks to different Virtual Clusters (VC) as needed**.
+Steps
+
+1. Allocate different REFRESH tasks to different VCs based on timeliness requirements and resource needs.
+2. For example, for new partitions with high timeliness requirements, their REFRESH tasks can be executed in large VCs with more resources.
+3. For supplementary tasks of other old partitions, their REFRESH tasks can be executed in smaller VCs with fewer resources.
+
+^

package/bin/skills/lakehouse-doc-en/references/dynamic_table_summary.md

@@ -0,0 +1,366 @@
+# Dynamic Table Overview
+
+> [Preview Release] This feature is currently in an invited preview stage. If you need to use it, please contact our technical support team for assistance.
+
+## What is a Dynamic Table
+
+A Dynamic Table is a data object in Singdata Lakehouse. Unlike a regular table, it dynamically generates data through a defined query statement. During refresh, it automatically retrieves incremental data from the base table and uses incremental algorithms for computation. This approach significantly improves data processing efficiency, making it especially well-suited for large-scale data workloads.
+
+## How Timely Dynamic Table Captures Source Table Changes
+
+Dynamic Table detects changed data based on the metadata commit time of the source table. Changes that have been committed to the source table's metadata can be captured. The following scenarios may affect the timeliness of change data visibility:
+
+* **DML data modification**: After a DML operation on the source table completes successfully, the changed data becomes accessible to Dynamic Table.
+* **Bulk import (Bulkload)**: After a bulk import task on the source table completes successfully, the changed data becomes accessible to Dynamic Table.
+* **Streaming ingestion**: For data written to the source table via the Ingestion Service streaming API, changes are committed by default every 1 minute. Once committed, the changed data becomes accessible to Dynamic Table. Note: SQL queries directly on the source table are real-time; this constraint only applies to the timeliness of incremental changes being visible to Dynamic Table.
+
+## Dynamic Table Use Cases
+
+### Not Suitable for Incremental Scenarios
+
+1. Queries with heavy data sorting requirements, such as those using an ORDER BY clause.
+2. Window functions that require data sorting (except when RowNumber=1), where the incremental data contains multiple very large partitions.
+3. Data that lacks good clustering characteristics and cannot clearly separate cold and hot data via join key, aggregate key, or window partition key.
+4. The supported types for aggregate keys are currently: CHAR, VARCHAR, STRING, TINYINT, SMALLINT, INT, BIGINT, DECIMAL, BOOLEAN, DATE. Other types are not supported.
+
+### Real-time Processing Scenarios
+
+In real-time data processing scenarios, data flows into the system continuously and rapidly. Traditional processing methods such as Full Reload or Full Refresh can be inefficient in terms of performance and resource consumption, especially when handling large-scale data streams. Dynamic Tables use incremental computation, processing only the data that has changed since the last update, which significantly reduces computing resource consumption.
+
+**Advantages of Dynamic Table**:
+
+* **Real-time**: Quickly reflects new data changes in the data warehouse, maintaining high data freshness.
+* **Cost-effective**: By setting a reasonable refresh interval, you can balance performance and cost for optimal resource utilization.
+* **Resource elasticity**: Lakehouse resources can be easily scaled elastically, which is especially advantageous when handling peak data inflows.
+* **On-demand computation**: In the future, Lakehouse will support on-demand activation of computing resources — starting resources only when there is data to compute — further improving efficiency and reducing costs.
+
+**Application Example**:
+
+**Background**: An e-commerce company wants to analyze its sales data in real time to make quick inventory and pricing decisions. Data flows into the system at a high rate, requiring an efficient data processing approach.
+
+**Challenges**:
+
+* Traditional full-data processing methods are inefficient in terms of performance and resource consumption, especially during peak periods.
+* A processing mechanism is needed that can quickly respond to data changes and keep data fresh.
+
+**Solution**:
+
+* Introduce Dynamic Tables, using incremental computation to process only the data that has changed since the last update.
+
+**Advantages of Dynamic Table**:
+
+1. **Real-time**: Dynamic Tables quickly capture and reflect data changes, ensuring decisions are based on the latest sales data.
+
+2. **Cost-effective**: With Dynamic Tables, the company can set reasonable data refresh intervals based on actual needs, avoiding unnecessary waste of computing resources.
+
+3. **Resource elasticity**: During promotional events or holidays when traffic peaks, Lakehouse resources can be scaled on-demand to handle peak data inflows without maintaining high resource configurations long-term.
+
+4. **On-demand computation**: In the future, Lakehouse's on-demand computation feature will further improve efficiency by activating resources only when there is data to compute, thereby reducing costs.
+
+### Fixed Dimension Analysis Queries with High Data Freshness Requirements
+
+In fixed dimension analysis query scenarios, the goal is to provide near-real-time analysis results. Traditional view queries can achieve this, but if large amounts of data transformation are involved, query speed may suffer. To address this, you can materialize the transformed results so that queries return pre-computed data directly, improving query speed. Materialized results can use either traditional tables or dynamic tables.
+
+Traditional tables offer the highest performance because they return pre-transformed data at query time. However, the downside is that you need to periodically schedule full recomputation, which typically takes a long time.
+
+Dynamic Tables combine the advantages of incremental computation. By updating only the records that have changed since the last load, Dynamic Tables reduce per-build time and maintain data freshness by shortening the refresh interval.
+
+### Notes
+
+* When processing large volumes of source data changes, the computation task may approach the load of a full refresh. Although incremental computation has clear efficiency advantages, setting the refresh interval too short may cause task backlog. This is because each refresh operation takes a certain amount of time to complete; if that time exceeds the configured refresh interval, subsequent refresh tasks will queue up.
+  * **Recommendations**:
+    * **Set a reasonable refresh interval**: Based on the frequency of data changes and the time each refresh takes, set a refresh interval that avoids backlog.
+    * **Monitor and adjust**: Continuously monitor data change patterns and system performance, and adjust the refresh interval based on actual conditions to optimize efficiency and resource utilization.
+* When writing operators, refer to the notes on how operators perform incremental refresh to optimize incremental refresh tasks.
+
+### Unsuitable Scenarios
+
+1. Queries involving a large number of Outer Join operations where the right table in the Outer Join changes frequently.
+2. Queries with heavy data sorting operations (such as ORDER BY).
+3. Window operations that require data sorting (except when RowNumber=1), where the incremental data contains multiple extremely large partitions.
+4. Data that lacks good clustering characteristics — for example, where cold and hot data cannot be clearly distinguished via join key, aggregate key, or window partition key.
+
+## Overview of Dynamic Table Working Principle
+
+### How to Obtain Changed Data
+
+* MetaService (the metadata service, a component of Lakehouse) records every historical data version of each table in Lakehouse.
+  * ![](.topwrite/assets/image_1716281294985.png)
+  * Basic concept: A table's **Snapshot (full)** vs. **Delta (changes)**
+
+## Lakehouse Dynamic Table Refresh Mechanism
+
+Lakehouse currently uses a scheduling mechanism to update Dynamic Tables. The following scheduling modes are supported:
+
+1. **Define scheduling properties in the DDL statement**
+2. **Define scheduling in Lakehouse Studio**
+3. **Submit Refresh jobs using a third-party scheduling engine**
+
+|  | Usage | Advantages | Disadvantages |
+| ------------------------ | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------- |
+| Define scheduling properties in DDL | Define the refresh interval in refreshOption. See the [Dynamic Table creation documentation](create-dynamic-table.md) for details. The minimum refresh interval is currently 1 minute. | Simple and easy to use; quickly sets refresh options. No dependency on third-party tools. | Lakehouse does not currently support defining strict upstream/downstream dependencies on Dynamic Tables in DDL. Relies on time-based scheduling in DDL definitions. You can use time intervals to ensure upstream refresh completes before scheduling downstream. |
+| Define scheduling in Lakehouse Studio | Configure scheduling through the visual interface in Lakehouse Studio. See the [task development and scheduling documentation](task-develop.md) for details. The minimum refresh interval is currently 1 minute. | Visual configuration, user-friendly. Supports scheduling dependency configuration to ensure upstream refresh completes before refreshing downstream. Supports single-node run monitoring such as failure alerts and timeout alerts. | |
+| Submit Refresh jobs using a third-party scheduling engine | Download the Lakehouse client and use cron expressions to schedule Refresh tasks, or use the Java JDBC interface to submit Refresh commands programmatically. | More flexible control over job submission and scheduling configuration; no restriction on time intervals. | Requires a third-party scheduling system. |
+
+## Comparison of Dynamic Tables, Materialized Views, and Regular Views
+
+From the Lakehouse implementation perspective, Dynamic Tables evolved from traditional materialized views. Although they share some commonalities, their intended use cases differ significantly.
+
+| | Materialized View | Dynamic Table | Regular View |
+| ----- | ------------------------------------------- | --------------------------- | ------------------------------------------------- |
+| Definition | A special view that pre-computes and stores query results | A high-efficiency tool focused on data processing | A virtual table that stores no data, only the query definition |
+| Performance optimization | Significantly improves query efficiency by reducing redundant computation through pre-stored results. Other SQL can leverage the materialized result for query rewriting. | Can also improve query efficiency, but the query optimizer does not perform query rewriting. | Performance depends on the underlying data table's query efficiency; the query logic is recomputed each time. |
+| Data freshness | Requires immediate data updates to ensure query rewriting accuracy. | Focuses on data processing flexibility; data latency is adjustable. | Data is always the latest each time. |
+| Update mechanism | Uses a scheduled refresh mechanism to keep data current. | Adjusted according to business definitions; real-time updates are not enforced. | Stores no data; no updates needed. |
+| Query optimization | Optimizer automatically identifies and uses pre-stored results. | Does not rely on automatic query rewriting by the optimizer. | |
+| Use case | Suitable for pre-computing and reusing query results. | Suitable for data processing pipelines where data may not be the absolute latest. | Suitable for low-complexity queries without heavy processing. Queries involving large data volumes and complex operators will be slow because they are recomputed each time. |
+| Operations | No complex maintenance scenarios. | Dynamic Tables also support advanced operations such as adding columns and version rollback. | No complex maintenance scenarios. |
+
+## Dynamic Table Refresh Monitoring
+
+### View Refresh History via SQL Commands
+
+You can use SQL commands to monitor the refresh history of Dynamic Tables. To get an overview of the refresh status of all Dynamic Tables, use the following SQL statement:
+
+```SQL
+SHOW DYNAMIC TABLE REFRESH HISTORY [WHERE <condition>];
+```
+
+**Filter refresh history**
+
+You can use the `WHERE` clause to filter by specific fields. For example, to view the refresh history of a Dynamic Table named `my_dy`:
+
+```SQL
+SHOW DYNAMIC TABLE REFRESH HISTORY WHERE name='my_dy';
+```
+
+| Field Name | Description |
+| ---------------- | --------------------------------------------------------------------- |
+| workspace\_name | Workspace name |
+| schema\_name | Schema name |
+| name | Dynamic table name |
+| virtual\_cluster | Computing cluster used |
+| start\_time | Refresh start time |
+| end\_time | Refresh end time |
+| duration | Refresh duration |
+| state | Job status: SETUP \| RESUMING\_CLUSTER \| QUEUED \| RUNNING \| SUCCEED \| FAILED |
+| refresh\_trigger | MANUAL (manually triggered by user, including Studio scheduled refresh) \| LH\_SCHEDULED (scheduled by Lakehouse) |
+| suspended\_reson | Reason for scheduling suspension |
+| refresh\_mode | NO\_DATA \| FULL \| INCREMENTAL |
+| error\_message | Failure message, if the refresh failed |
+| source\_tables | Names of tables used by the Dynamic Table |
+| stats | Information such as incremental refresh row counts |
+| job\_id | Job ID; click to view the job profile |
+
+### View Single Job Refresh Details via Job Profile
+
+In addition to SQL commands, you can view the refresh details of a single job through Job Profile.
+
+![](.topwrite/assets/image_1716280617859.png)
+
+* You can also use the input records in the job profile to determine whether data was read incrementally.
+
+## Cost of Dynamic Tables
+
+#### Computing Cost
+
+Dynamic Table refresh operations rely on computing resources (Virtual Cluster), including:
+
+* **Scheduled refresh**: Automatically executes refresh based on the configured interval.
+* **Manual refresh**: Triggered by the user as needed.
+
+Both types of refresh operations consume computing resources.
+
+#### Storage Cost
+
+Dynamic Tables also require storage space to save their materialized results. Like regular tables, Dynamic Tables support:
+
+* **Time Travel**: Allows you to access data from any point within the past 7 days.
+* **Time Travel retention period**: Defaults to 7 days. After this period, data is no longer accessible via Time Travel and will be physically deleted.
+
+#### Refresh Schedule Settings
+
+**Factors affecting refresh speed**
+
+The speed of incremental refresh mainly depends on two factors:
+
+1. **Volume of source data changes**: The more data the refresh operation needs to process, the longer it takes.
+2. **Fixed overhead**: Some baseline overhead is incurred with each refresh regardless of the volume of changes.
+
+**Business value and refresh frequency**
+
+* If data freshness is not critical to your business, consider reducing the refresh frequency. This reduces the computational overhead from frequent refreshes.
+* Using incremental computation mode improves single-refresh speed because it only processes data that has changed since the last refresh.
+
+**Balancing refresh cost and frequency**
+
+* Refresh costs increase with refresh frequency. You need to balance the business value of data freshness against the resulting computational costs.
+* High-frequency refreshes keep data up-to-date in real time, but accumulated computational costs will also increase.
+
+**Recommendations**
+
+* Evaluate your business needs to determine the specific value of data freshness to your operations.
+* Set a reasonable refresh frequency based on business value to optimize cost-effectiveness.
+* Use incremental computation mode to improve refresh efficiency and reduce unnecessary computational overhead.
+
+## Dynamic Table Limitations
+
+* Incremental refresh limitations: Non-deterministic functions such as `random`, `current_timestamp`, `current_date`, etc., are not supported.
+* Direct modification of Dynamic Table data is not supported — for example, executing `UPDATE`, `DELETE`, or `TRUNCATE` on a Dynamic Table.
+
+## Use Cases for Processing with Dynamic Tables
+
+## Processing Lakehouse Sample Data with Dynamic Tables
+
+Lakehouse provides a dynamic public dataset named `ecommerce_events_multicategorystore_live`, located at `clickzetta_sample_data.clickzetta_sample_data.ecommerce_events_history`. This dataset is updated in real time and can be queried directly.
+
+Real-time dataset availability: Currently, the `ecommerce_events_multicategorystore_live` real-time public dataset is only available in the Shanghai region of Alibaba Cloud. If your account or service is not in that region, you will not be able to query this public dataset.
+
+1. **Write a SQL script to define scheduling and process data using DDL**
+
+```SQL
+CREATE  DYNAMIC TABLE event_type_count
+REFRESH interval 1 minute vcluster default
+as
+SELECT event_type, COUNT(*) AS events_count
+FROM clickzetta_sample_data.ecommerce_events_history.ecommerce_events_multicategorystore_live
+GROUP BY event_type;
+--Initialize DYNAMIC TABLE data
+REFRESH DYNAMIC TABLE event_type_count;
+```
+
+2. View Dynamic Table refresh
+
+**Use a command to view Dynamic Table refresh**
+
+```SQL
+SHOW DYNAMIC TABLE REFRESH HISTORY WHERE name='event_type_count';
+```
+
+**View Dynamic Table refresh history in job history**
+
+![](.topwrite/assets/image_1716280651911.png)
+
+Click into the details to view the input records and see how many incremental rows were retrieved. In the diagnostics section, you can view the execution plan of the incremental SQL.
+
+![](.topwrite/assets/image_1716280668286.png)
+
+3. **After seeing incremental refresh data in the job history, you can view the data changes**
+
+```SQL
+SELECT    *   FROM      event_type_count;
++-------------+--------------+
+| event_type  | events_count |
++-------------+--------------+
+| view        | 91634700     |
+| purchase    | 91630921     |
+| add_to_cart | 91622270     |
++-------------+--------------+
+--View data after incremental refresh completes
+SELECT    *
+FROM      event_type_count;
++-------------+--------------+
+| event_type  | events_count |
++-------------+--------------+
+| purchase    | 91633135     |
+| add_to_cart | 91624515     |
+| view        | 91636913     |
++-------------+--------------+
+```
+
+## Using Studio to Schedule Dynamic Table Refresh Tasks
+
+In this demonstration, we simulate incremental data insertion and show the effects of incremental computation through the following steps:
+
+* **Simulate incremental data insertion**: Use `INSERT INTO` statements to insert simulated data into the specified table, mimicking incremental data updates in a real business scenario.
+* **Use Studio scheduling for refresh**: Use the scheduling feature of Lakehouse Studio to trigger and execute incremental data refresh tasks.
+* **Demonstrate incremental computation effects**: Show how incremental computation efficiently processes newly inserted data and reflects the updates in the final query results.
+
+1. Data preparation
+
+```SQL
+CREATE TABLE event_tb (
+    event STRING,
+    process DOUBLE,
+    event_time TIMESTAMP
+  );
+INSERT INTO event_tb VALUES
+  ('event-0', 20.0, TIMESTAMP '2023-09-20 14:43:13'),
+  ('event-0', 20.0, TIMESTAMP '2023-09-19 11:40:13'),
+  ('event-1', 21.0, TIMESTAMP '2023-09-19 14:30:14'),
+  ('event-1', 22.0, TIMESTAMP '2023-09-20 14:20:15');
+```
+
+2. Data processing
+
+* Create a new SQL script "1. Time Processing dt" to process the prepared data using SQL to create a dynamic table
+
+  ![](.topwrite/assets/image_1716280688236.png)
+
+```SQL
+CREATE dynamic TABLE IF NOT EXISTS event_gettime AS
+SELECT    event,
+          process,
+          YEAR(event_time) event_year,
+          MONTH(event_time) event_month,
+          DAY(event_time) event_day,
+          hour(event_time) event_hour,
+          minute(event_time) event_minute
+FROM      event_tb;
+
+REFRESH   dynamic TABLE event_gettime;
+```
+
+* Create a new SQL script "2. Aggregate dy" to perform aggregation on the data processed in the previous step
+
+  * ```SQL
+    CREATE dynamic TABLE IF NOT EXISTS event_group_minute AS
+    SELECT    event,
+              event_hour,
+              event_minute,
+              SUM(process) process_sum
+    FROM      event_gettime
+    GROUP BY  event,
+              event_hour,
+              event_minute;
+
+    refresh dynamic table event_group_minute;
+    ```
+
+3. Build dependencies and scheduling relationships
+
+* Task one "1. Time Processing dy" — configure to run on a 1-minute schedule
+
+  * ![](.topwrite/assets/image_1716280711741.png)
+
+* Task two "2. Aggregate dy" — configure to run on a 1-minute schedule and set a **dependency on task one "1. Time Processing dy"** in the scheduling dependencies
+
+  * ![](.topwrite/assets/image_1716280739978.png)
+
+* In the data development interface, click to submit each task
+
+  * ![](.topwrite/assets/image_1716280770050.png)
+
+4. Verify incremental refresh
+
+```SQL
+--Manually insert data
+INSERT INTO event_tb VALUES
+  ('event-0', 20.0, TIMESTAMP '2024-01-20 14:43:13');
+ --Check whether event_gettime is incrementally refreshed and how many rows were refreshed
+SHOW DYNAMIC TABLE REFRESH HISTORY WHERE name='event_gettime';
++----------------+-------------+---------------+-----------------+-------------------------+-------------------------+----------------------+---------+-----------------+------------------+--------------+---------------+-------------------------------------------------------------------+------------------------------------------+-------------------+-------------------------------+
+| workspace_name | schema_name |     name      | virtual_cluster |       start_time        |        end_time         |       duration       |  state  | refresh_trigger | suspended_reason | refresh_mode | error_message |                           source_tables                           |                  stats                   | completion_target |            job_id             |
++----------------+-------------+---------------+-----------------+-------------------------+-------------------------+----------------------+---------+-----------------+------------------+--------------+---------------+-------------------------------------------------------------------+------------------------------------------+-------------------+-------------------------------+
+| ql_ws          | public      | event_gettime | DEFAULT         | 2024-05-17 11:33:15.512 | 2024-05-17 11:33:15.839 | 0 00:00:00.327000000 | SUCCEED | MANUAL          | null             | INCREMENTAL  | null          | [{"schema":"public","table_name":"event_tb","workspace":"ql_ws"}] | {"rows_deleted":"0","rows_inserted":"1"} | null              | 202405170333149794gibwyt3dv0g |
++----------------+-------------+---------------+-----------------+-------------------------+-------------------------+----------------------+---------+-----------------+------------------+--------------+---------------+-------------------------------------------------------------------+------------------------------------------+-------------------+-------------------------------+
+```
+
+## Using Dynamic Table Task Nodes for Data Processing
+
+For usage instructions, see [Dynamic Table Tasks](<dynamic_table_task.md>).
+
+## Constraints and Limitations
+
+* Apart from `current_date`, other non-deterministic functions are not supported. Creating a Dynamic Table using non-deterministic functions will result in an error.
+* It is recommended to use GP-type clusters to refresh Dynamic Tables. Reason: During the refresh process, Dynamic Tables automatically perform small file compaction based on built-in policies, and AP-type clusters do not support this operation.