@clickzetta/cz-cli-darwin-arm64 0.3.94 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/ssb-benchmark.md

@@ -0,0 +1,216 @@
+# Overview
+
+Star Schema Benchmark (SSB) is a benchmark used to evaluate the performance of OLAP systems. SSB provides a simplified star schema model dataset based on TPC-H, primarily used to test the performance of multi-table join queries under the star schema model. ClickHouse has flattened the star schema model of SSB into a wide table, transforming it into a single table test set. Refer to the [ClickHouse official documentation](https://clickhouse.com/docs/en/getting-started/example-datasets/star-schema).
+
+This report provides the test results of Singdata Lakehouse and ClickHouse on the SSB single table test set at a 100GB scale. The conclusions are as follows:
+![](.topwrite/assets/ssb.png)
+
+* Among the 13 queries executed on the SSB single table test set with a 100G dataset, ClickHouse's total execution time is 1.48 times that of Singdata Lakehouse, indicating that Singdata Lakehouse performs better.
+
+# Test Environment
+
+* **ClickHouse Test Environment**
+
+| **Configuration Item** | **Configuration Details**                                      |
+| ---------------------- | -------------------------------------------------------------- |
+| Server                 | 1 Alibaba Cloud ECS server (ecs.g7.16xlarge 64 vCPU 256 GiB)   |
+| Network Bandwidth      | 32Gbps                                                         |
+| Software               | ClickHouse 23.3                                                |
+| Storage Service        | ESSD high-efficiency cloud disk 200 GB PL1, single disk IOPS limit 50,000 |
+| Hardware Cost          | 17.164 RMB/hour                                                |
+
+* **Singdata Lakehouse Test Environment**
+
+| **Configuration Item** | **Singdata Lakehouse Configuration Details** |
+| ---------------------- | -------------------------------------------- |
+| Compute Resources      | Large specification compute cluster (64 vCPU equivalent computing power) |
+| Software               | Alibaba Cloud Shanghai Region - Singdata Lakehouse service |
+| Storage Service        | Managed storage, Alibaba Cloud OSS object storage |
+
+# Test Data
+
+The test used the SSB standard dataset, which includes the following tables and record counts:
+
+* **lineorder**: 600 million records
+* **customer**: 3 million records
+* **part**: 1.4 million records
+* **supplier**: 200,000 records
+* **dates**: 2,556 records
+* **lineorder_flat**: 600 million records (flattened wide table)
+
+# Test Procedure
+
+## ClickHouse
+
+The DDL and query methods are consistent with the ClickHouse official test methods. For details, refer to the [ClickHouse official documentation](https://clickhouse.com/docs/en/getting-started/example-datasets/star-schema).
+
+## Singdata Lakehouse
+
+### Create Cluster
+
+* Create a virtual cluster with Large specifications in Lakehouse
+```SQL
+create vcluster if not exists LARGE_CLUSTER vcluster_size='LARGE' vcluster_type='Analytics' AUTO_RESUME=TRUE AUTO_SUSPEND_IN_SECOND=300 min_replicas=1 max_replicas=1;
+```
+### Create Table Statement
+```SQL
+  --Create table statement
+CREATE TABLE demo_examples.ssb_100g.lineorder_flat(
+  `lo_orderkey` int not null,
+  `lo_linenumber` int not null,
+  `lo_custkey` int not null,
+  `lo_partkey` int not null,
+  `lo_suppkey` int not null,
+  `lo_orderdate` date not null,
+  `lo_orderpriority` string not null,
+  `lo_shippriority` int not null,
+  `lo_quantity` int not null,
+  `lo_extendedprice` int not null,
+  `lo_ordtotalprice` int not null,
+  `lo_discount` int not null,
+  `lo_revenue` int not null,
+  `lo_supplycost` int not null,
+  `lo_tax` int not null,
+  `lo_commitdate` date not null,
+  `lo_shipmode` string not null,
+  `c_name` string not null,
+  `c_address` string not null,
+  `c_city` string not null,
+  `c_nation` string not null,
+  `c_region` string not null,
+  `c_phone` string not null,
+  `c_mktsegment` string not null,
+  `s_name` string not null,
+  `s_address` string not null,
+  `s_city` string not null,
+  `s_nation` string not null,
+  `s_region` string not null,
+  `s_phone` string not null,
+  `p_name` string not null,
+  `p_mfgr` string not null,
+  `p_category` string not null,
+  `p_brand` string not null,
+  `p_color` string not null,
+  `p_type` string not null,
+  `p_size` int not null,
+  `p_container` string not null)
+ CLUSTERED BY(lo_orderkey) SORTED BY(lo_orderdate, lo_orderkey) INTO 64 BUCKETS
+  PROPERTIES ('cz.storage.parquet.compression'='lz4');
+
+ --Collect statistics
+ analyze table lineorder_flat compute statistics for all columns;
+```
+### Running Queries
+```SQL
+--Disable result cache
+set cz.sql.enable.shortcut.result.cache=false;
+
+--Q1.1
+SELECT /*Q1.1*/ sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue 
+FROM lineorder_flat 
+WHERE Year(LO_ORDERDATE) = 1993 AND LO_DISCOUNT BETWEEN 1 AND 3 AND LO_QUANTITY < 25;
+
+--Q1.2
+SELECT /*Q1.2*/ sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue 
+FROM lineorder_flat 
+WHERE Year(LO_ORDERDATE) = 1994 and Month(LO_ORDERDATE) = 1 AND LO_DISCOUNT BETWEEN 4 AND 6 AND LO_QUANTITY BETWEEN 26 AND 35;
+
+--Q1.3
+SELECT /*Q1.3*/ sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue 
+FROM lineorder_flat WHERE WeekOfYear(LO_ORDERDATE) = 6 AND Year(LO_ORDERDATE) = 1994 AND LO_DISCOUNT BETWEEN 5 AND 7 AND LO_QUANTITY BETWEEN 26 AND 35;
+
+--Q2.1
+SELECT /*Q2.1*/
+    sum(LO_REVENUE),
+    Year(LO_ORDERDATE) AS year,
+    P_BRAND 
+FROM lineorder_flat 
+WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA' 
+GROUP BY 
+    year,
+    P_BRAND 
+ORDER BY year,P_BRAND;
+
+--Q2.2
+SELECT /*Q2.2*/
+    sum(LO_REVENUE),
+    Year(LO_ORDERDATE) AS year,
+    P_BRAND 
+FROM lineorder_flat 
+WHERE P_BRAND >= 'MFGR#2221' AND P_BRAND <= 'MFGR#2228' AND S_REGION = 'ASIA' GROUP BY year,P_BRAND ORDER BY year,P_BRAND;
+
+--Q2.3
+SELECT /*Q2.3*/ sum(LO_REVENUE),Year(LO_ORDERDATE) AS year,P_BRAND FROM lineorder_flat WHERE P_BRAND = 'MFGR#2239' AND S_REGION = 'EUROPE' GROUP BY year, P_BRAND ORDER BY year,P_BRAND;
+
+--Q3.1
+SELECT /*Q3.1*/ C_NATION,S_NATION,Year(LO_ORDERDATE) AS year,sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE C_REGION = 'ASIA' AND S_REGION = 'ASIA' GROUP BY C_NATION,S_NATION,year HAVING year >= 1992 AND year <= 1997 ORDER BY year ASC,revenue DESC;
+
+--Q3.2
+SELECT /*Q3.2*/ C_CITY,S_CITY,Year(LO_ORDERDATE) AS year,sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE C_NATION = 'UNITED STATES' AND S_NATION = 'UNITED STATES' GROUP BY C_CITY,S_CITY,year HAVING year >= 1992 AND year <= 1997 ORDER BY year ASC,revenue DESC;
+
+--Q3.3
+SELECT /*Q3.3*/
+    C_CITY,
+    S_CITY,
+    Year(LO_ORDERDATE) AS year,
+    sum(LO_REVENUE) AS revenue 
+FROM lineorder_flat 
+WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') 
+AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') 
+GROUP BY C_CITY,S_CITY,year HAVING year >= 1992 AND year <= 1997 
+ORDER BY year ASC,revenue DESC;
+
+--Q3.4
+SELECT /*Q3.4*/
+    C_CITY,
+    S_CITY,
+    Year(LO_ORDERDATE) AS year,
+    sum(LO_REVENUE) AS revenue FROM lineorder_flat 
+WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND Year(LO_ORDERDATE) = 1997 AND Month(LO_ORDERDATE) = 12 GROUP BY C_CITY,S_CITY,year ORDER BY year ASC,revenue DESC;
+
+--Q4.1
+SELECT /*Q4.1*/ Year(LO_ORDERDATE) AS year,
+    C_NATION,
+    sum(LO_REVENUE - LO_SUPPLYCOST) AS profit 
+FROM lineorder_flat WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') GROUP BY year,C_NATION ORDER BY year ASC,C_NATION ASC;
+
+--Q4.2
+SELECT /*Q4.2*/ Year(LO_ORDERDATE) AS year,
+    S_NATION,
+    P_CATEGORY,
+    sum(LO_REVENUE - LO_SUPPLYCOST) AS profit 
+FROM lineorder_flat 
+WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') GROUP BY year,S_NATION,P_CATEGORY HAVING year == 1997 OR year == 1998 ORDER BY year ASC,S_NATION ASC,P_CATEGORY ASC;
+
+SELECT /*Q4.3*/ Year(LO_ORDERDATE) AS year,
+    S_CITY,
+    P_BRAND,
+    sum(LO_REVENUE - LO_SUPPLYCOST) AS profit 
+FROM lineorder_flat 
+WHERE S_NATION = 'UNITED STATES' AND P_CATEGORY = 'MFGR#14' 
+GROUP BY year,S_CITY,P_BRAND 
+HAVING year = 1997 OR year = 1998 
+ORDER BY year ASC,S_CITY ASC,P_BRAND ASC;
+```
+# Test Results
+
+Below are the performance test results of Lakehouse and ClickHouse on 13 queries, measured in milliseconds (ms). Lower values indicate better performance.
+
+ClickHouse used the default LZ4 compression, and Singdata Lakehouse also used LZ4 compression. All queries were pre-warmed once, then executed three times, and the average value was taken as the result.
+
+| Query | Singdata Lakehouse | ClickHouse-23.3 | ClickHouse vs Lakehouse |
+| ----- | ------------------ | --------------- | ----------------------- |
+| Q1.1  | 59                 | 48              | 0.81                    |
+| Q1.2  | 30                 | 15              | 0.5                     |
+| Q1.3  | 28                 | 14              | 0.5                     |
+| Q2.1  | 197                | 301             | 1.53                    |
+| Q2.2  | 183                | 273             | 1.49                    |
+| Q2.3  | 149                | 255             | 1.71                    |
+| Q3.1  | 259                | 398             | 1.54                    |
+| Q3.2  | 200                | 319             | 1.6                     |
+| Q3.3  | 146                | 227             | 1.55                    |
+| Q3.4  | 34                 | 18              | 0.53                    |
+| Q4.1  | 281                | 469             | 1.67                    |
+| Q4.2  | 117                | 160             | 1.37                    |
+| Q4.3  | 101                | 148             | 1.47                    |
+| SUM   | 1784               | 2645            | 1.48                    |

package/bin/skills/lakehouse-doc-en/references/sso-configuration.md

@@ -0,0 +1,160 @@
+# Single Sign-On (Private Preview)
+
+This document is intended to guide you step-by-step through the configuration of Single Sign-On (SSO), ensuring you can successfully enable SSO using either the OAuth 2.0/OIDC or SAML 2.0 protocol.
+
+***
+
+## Basic Concepts
+
+* **Single Sign-On (SSO**): A mechanism that allows users to access multiple systems with a single authentication process, simplifying the login experience and improving security.
+
+* **Federated Authentication**: The Lakehouse (Service Provider, SP) trusts an external Identity Provider (IdP) to perform user authentication.
+
+* **Protocol Overview**:
+
+  * **SAML 2.0**: An XML-based identity assertion protocol that transmits user identity information via an Assertion.
+  * **OAuth 2.0 / OpenID Connect (OIDC**): OAuth provides an authorization framework, while OIDC adds an identity layer using a JWT-formatted ID Token to represent user identity.
+
+## Relationship Between Singdata and the Customer’s IdP
+
+* Register the Lakehouse application in the IdP (provide callback URL/ACS, Entity ID, signing or encryption certificates).
+* Enter the parameters provided by the IdP into the Lakehouse platform.
+* After configuration, two login flows are supported: login from Lakehouse (SP-init) or from the IdP (if IdP-init is supported).
+
+## Limitations
+
+Currently, Lakehouse **does not support** the following features. Please keep these in mind during configuration and use:
+
+1. **No global logout support**. Logging out from the IdP will not affect the login state in Lakehouse. Users must log out from Lakehouse separately.
+
+2. **No multi-IdP configuration**. Only one IdP connection is supported.
+
+3. **SSO is supported only for web login**. JDBC, Python SDK, Java SDK, and similar methods are currently not supported for SSO login.
+
+4. **Identity mapping field limitations**:
+
+   * For SAML 2.0, only Email can be used as the identity mapping field.
+   * For OAuth 2.0 / OpenID Connect (OIDC), only Email or Phone can be used.
+
+5. **Identity mapping field uniqueness requirement**:
+   The field used for identity mapping must be unique across all users in Lakehouse. If multiple users have duplicate Email or Phone information in the chosen mapping field, those users will be unable to log in via SSO.
+
+***
+
+## Protocol Configuration Guide
+
+### Enabling SSO
+
+By default, SSO login is disabled in a Lakehouse account. Users with the Account Administrator role can enable it.
+
+Log in to the Lakehouse account, go to the **Account Center** page, and under the **SSO Login** option, click **Enable SSO Login**.
+
+:-: ![](.topwrite/assets/screenshot-20250814-170309.png =678)
+
+**Note**: Once SSO login is enabled, all users in the account can only log in using SSO, and username/password login for the web will no longer be allowed. Before enabling SSO, ensure that all users who need to log in to Lakehouse:
+
+1. Have a user identity in the IdP.
+2. Have unique identity mapping information (Email or Phone) in Lakehouse that does not duplicate any other user.
+
+***
+
+### OAuth 2.0 / OpenID Connect (OIDC) Protocol
+
+**1. Select the protocol**
+
+After enabling SSO login, select **OAuth 2.0 / OIDC Protocol** in the right-side pop-up. Once selected, you will see the protocol’s callback URL, for example:
+
+:-: ![](.topwrite/assets/image_1755162417316.png =695)
+
+```
+https://uat-api.clickzetta.com/clickzetta-portal/sso/oidc/consume?u={code}
+```
+
+Use this callback URL to register the Lakehouse application in your IdP service and record the Client\_ID and other configuration values for later use.
+
+**Configure Authorization Endpoint**
+
+:-: ![](.topwrite/assets/image_1755162515497.png =546)
+
+**Parameters**:
+
+* **Authorization Endpoint URL** (e.g., `https://idp.example.com/authorize`): Redirects users to the IdP for identity authorization.
+* **Client ID**: Identifies the Lakehouse application in the IdP, obtained after registering the Lakehouse app in your IdP.
+* **Response type**: Only supports `code` mode.
+* **Scope** (e.g., `openid`, `profile`, `email`): Determines the user information that can be requested; multiple scopes are supported. For OpenID, `openid` must be selected. Additional custom scopes can be added by selecting **Other**.
+* **Optional extended parameters**: Allows adding extra fields as required by the IdP (e.g., `prompt=login`).
+
+**Configure Token Endpoint**
+
+:-: ![](.topwrite/assets/image_1755162550809.png =515)
+
+**Parameters**:
+
+* **Token Endpoint URL**: Exchanges the authorization code for tokens. After Lakehouse calls the IdP authorization endpoint and receives the code, it calls this endpoint to exchange the code for tokens.
+* **Client Secret (optional**): Used to verify client identity. If the IdP does not issue a secret, this can be left blank.
+* **Optional extended parameters**: Add as required by the IdP.
+
+**Configure UserInfo Endpoint**
+
+:-: ![](.topwrite/assets/image_1755162592157.png =478)
+
+**Parameters**:
+
+* **UserInfo Endpoint URL**: Used by Lakehouse to retrieve detailed user information.
+
+* **Identity mapping configuration**:
+
+  * **UserInfo field**: The field in the UserInfo response containing the user’s identity. Options: Email, Phone, or custom field.
+  * **Identity mapping field**: The Lakehouse field to match against the UserInfo field. Options: Email or Phone. If using Phone, Lakehouse supports phone numbers with country codes starting with `00`, `+`, or no country code.
+
+* **Test user information completeness**: Checks and lists all users in the account who are missing or have duplicate values in the chosen identity mapping field. Such users will be unable to log in via SSO.
+
+* **Optional extended parameters**: Add as required by the IdP.
+
+**Allow account administrators to also log in with username/password**
+
+If enabled, account administrators can still log in with username and password in case of SSO failures, allowing them to adjust account configurations.
+
+***
+
+### SAML 2.0 Protocol
+
+**Select the protocol**
+
+After enabling SSO login, select **SAML 2.0 Protocol** in the right-side pop-up. Once selected, you will see the callback URL (Assertion Consumer Service URL), for example:
+
+:-: ![](.topwrite/assets/image_1755162706106.png =478)
+
+```
+https://uat-api.clickzetta.com/clickzetta-portal/sso/saml/consume?u={code}
+```
+
+Use this address to register the application in the IdP, and record and save the Entity ID and X.509 certificate returned by the IdP.
+
+:-: **Enter SSO Login URL**
+![](.topwrite/assets/image_1755162752056.png =478)
+
+**Parameters**:
+
+* **SSO Login URL**: The IdP endpoint that receives SAML requests.
+* **Entity ID**: The application identifier in the IdP, obtained after app registration.
+* **X.509 Certificate (optional**): Used for signature verification and data security, obtained after app registration in the IdP.
+* **Issuer (optional**): The entity issuing the SAML assertion, used for trust verification.
+* **Test user information completeness**: For SAML 2.0, only Email is supported as the identity mapping field. The completeness check lists users missing or duplicating email information, who will not be able to log in after SSO is enabled.
+
+**Allow account administrators to also log in with username/password**
+
+If enabled, account administrators can still log in with username and password in case of SSO failures, allowing them to adjust account configurations.
+
+***
+
+## Frequently Asked Questions
+
+**Q: Why does Lakehouse show “Authentication failed” after logging in from the IdP**?
+A: This usually occurs when the mapping field (Email or Phone) is not unique among users, or is missing/incorrectly formatted in the IdP response. If it’s a uniqueness issue, the error message will clearly indicate it. If not, check whether the mapping field configuration is correct (e.g., returning Email but mapping it to Phone).
+
+**Q: Why can’t I log in with username/password even though “Allow account administrators to also log in” is enabled**?
+A: Once SSO is enabled and this option is selected, **only** users with the Account Administrator role can log in using username/password; all other users must log in using SSO.
+
+**Q: Can I use SSO login via SDK (e.g., Python, JDBC**)?
+A: Currently, only web login is supported. Python, JDBC, and similar SDKs only support username/password authentication.

package/bin/skills/lakehouse-doc-en/references/storage_encryption.md

@@ -0,0 +1,81 @@
+# Data Storage Encryption
+
+## Overview
+
+Lakehouse supports server-side encryption (SSE) for table data stored in object storage, using the AES-256 encryption algorithm to ensure your data is securely protected at the storage layer.
+
+You can choose to use cloud provider-managed keys or your own KMS keys for encryption, depending on your security and compliance requirements.
+
+## Encryption Modes
+
+Lakehouse provides two encryption modes:
+
+| Mode         | Description                                             |
+| ---------- | ---------------------------------------------- |
+| Managed Encryption (Recommended)   | Uses managed keys provided by the cloud provider's object storage service for encryption, no additional configuration required                  |
+| Custom KMS Encryption | Uses your own KMS keys for encryption, requires providing KMS ARN (ID) and KMS Region |
+
+> Custom KMS encryption currently supports Alibaba Cloud and AWS platforms. Tencent Cloud only supports managed encryption mode.
+
+## Encryption Scope
+
+* You can enable encryption uniformly at the workspace level. After enabling, newly created data objects within the workspace will be encrypted by default.
+* You can enable encryption for a workspace at any time. Enabling encryption will not affect existing tables in the workspace; it only applies encryption to newly created tables.
+
+> ⚠️ Once a table is encrypted, it cannot be restored to a non-encrypted state. Please confirm before enabling.
+
+## Usage
+
+### Enable Encryption When Creating a Workspace
+
+1. On the Create Workspace page, toggle on the "Storage Encryption" switch.
+
+2. If you wish to use managed encryption, leave the KMS ARN blank.
+
+3. If you wish to use a custom KMS key:
+
+   * Enter your KMS key ARN.
+   * Enter the KMS Region.
+
+### Adjust Encryption Settings When Modifying a Workspace
+
+* If the workspace does not use a custom KMS key, you can freely toggle the encryption switch on or off.
+* If the workspace already uses a custom KMS key, the encryption settings will be locked, and you cannot disable encryption or modify the key configuration.
+
+## Key Management
+
+* Each data object can only use one key during its lifecycle, and the key cannot be changed.
+* You can create multiple keys and select different keys when encrypting data objects in different workspaces.
+* After using a custom KMS key, the KMS ARN and KMS Region become read-only and cannot be modified.
+
+> ⚠️ After encrypting with a custom KMS key, the key configuration cannot be modified and encryption cannot be disabled. Please carefully confirm the key information before configuration.
+
+> ⚠️ Please keep your custom KMS encryption keys secure. Once a key is lost, data encrypted with that key in Lakehouse will be permanently unreadable and unrecoverable.
+
+## KMS Request Costs and Caching Mechanism
+
+Since data warehouse scenarios involve frequent data reads and writes, using custom KMS keys can easily hit the QPS request limits of the cloud provider's KMS service and generate significant KMS API request costs. To address this, Lakehouse employs a key caching mechanism to reduce API calls to KMS, with a cache validity period of **5 minutes**. This means that after a KMS key is revoked or becomes invalid, there is still a window of up to 5 minutes during which encrypted data can be read and written.
+
+> ⚠️ It is recommended to **prioritize managed encryption mode** and avoid using custom KMS keys to prevent hitting KMS request limits that affect read/write performance or generating significant KMS request costs. KMS request costs are typically incurred in the cloud account where your KMS key resides, not in your Lakehouse bill.
+
+## FAQ
+
+**Q: After enabling workspace encryption, will historical data be encrypted?**
+
+No. Enabling encryption only affects data objects created thereafter. Existing data is not affected.
+
+**Q: Can encryption be removed from an already encrypted table?**
+
+No. Table encryption operations are irreversible.
+
+**Q: Can custom KMS key configuration be modified after setup?**
+
+No. Once a custom KMS key is configured, neither the key information nor the encryption toggle can be modified.
+
+**Q: What if a custom KMS key is lost?**
+
+Please ensure you keep your custom KMS encryption keys secure. Once a key is lost, data encrypted with that key in Lakehouse will be permanently unreadable and unrecoverable.
+
+**Q: Does using custom KMS encryption incur additional costs?**
+
+Yes. Each data read/write requires calling the KMS API for encryption/decryption. Although Lakehouse reduces request frequency through a 5-minute key caching mechanism, significant KMS request costs may still be incurred in high-frequency read/write scenarios. These costs are incurred in the cloud account where your KMS key resides. It is recommended to prioritize managed encryption mode to avoid such costs.

package/bin/skills/lakehouse-doc-en/references/streaming_data_pipeline_overview.md

@@ -0,0 +1,46 @@
+# Streaming Data Pipeline Overview
+
+## Incremental Computing Overview
+
+A streaming data pipeline is a set of tasks that continuously collects, processes, and transforms real-time data to produce result data that meets business timeliness requirements. Streaming data processing is the foundation for analysts and business applications to perform real-time insights and real-time decision-making.
+
+Unlike periodic offline processing (batch data pipeline), a streaming data pipeline continuously produces real-time updated results by orchestrating real-time data ingestion tasks and SQL tasks that support incremental data processing. Incremental processing techniques are used throughout to improve efficiency and reduce costs.
+
+The overall processing flow for a Singdata Lakehouse streaming data pipeline is shown below:
+
+![](.topwrite/assets/image_1709520998068.png)
+
+The product features of streaming data processing include:
+
+| Feature | Description |
+| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Real-time Data Loading | **Real-time Data Loading Service**: Provides a Streaming API via SDK/Connector, supporting real-time append/update writes with second-level visibility, as a system-managed service. You can use the SDK or Flink Connector to write data to the target table. **Studio Data Integration Service**: A built-in data integration tool that supports real-time extraction from multiple real-time data sources (such as Kafka, database CDC, etc.) and writes to the Lakehouse target table in real time via the Streaming API. |
+| Incremental Data Processing | **Dynamic Table**: Declaratively defines data processing logic through dynamic tables, supporting any SQL syntax and operators. The system automatically reads incremental changes from the base table and adaptively optimizes processing efficiency using incremental or full-refresh algorithms. You can create one or more dynamic tables with scheduling policies, and the system automatically identifies dependencies and continuously schedules execution, greatly simplifying real-time data processing development. Note: The minimum supported scheduling interval is currently 1 minute. |
+| Change Data Capture | **Table Stream**: Table Stream is a built-in SQL object type. A Table Stream is created on top of a specified table and records the data change (CDC) information for that table. Table Stream supports SQL queries to retrieve change records between two specified data versions — for example, changes from 5 minutes ago to the present. Through Table Stream, downstream SQL ETL jobs can easily read and process table change data, or synchronize change results to external systems. |
+| Continuous Scheduling | The system provides two scheduling modes: **Dynamic Table self-defined scheduling cycle**: When defining via materialized view DDL, you can set the scheduling cycle using the INTERVAL keyword. **Studio task scheduling**: Create SQL tasks in the Web IDE and set the task scheduling cycle. This mode supports scheduling both dynamic tables and SQL ETL jobs that use Table Stream. Note: Using the Studio scheduling system provides better task execution observability and operational alerting capabilities. |
+
+---
+
+## Next Steps
+
+Choose the ingestion and processing approach that matches your data source and scenario:
+
+**Data Ingestion**
+
+| Data Source | Approach | Documentation |
+|--------|------|------|
+| Database CDC (MySQL / PG / Oracle, etc.) | Studio real-time sync task | [Real-time Sync Task](realtime_sync.md) |
+| Full database migration | Studio multi-table real-time sync | [Multi-table Real-time Sync](multitable_realtime_sync.md) |
+| Kafka / AutoMQ message streams | Kafka Pipe or Studio Kafka sync | [Pipe Continuous Ingestion](pipe-introduction.md) |
+| OSS / S3 / COS files | Pipe | [Pipe Continuous Ingestion](pipe-introduction.md) |
+
+**Incremental Data Processing**
+
+- [Dynamic Table](dynamic-table.md) — Declaratively define transformation logic; the system automatically refreshes incrementally
+- [Table Stream](table-stream-title.md) — Capture incremental changes on a table to drive downstream CDC consumption
+- [Studio Workflow](composite_task.md) — Orchestrate multiple tasks into a DAG for unified scheduling and monitoring
+
+**End-to-End Examples**
+
+- [Real-time Data Pipeline: Kafka → Lakehouse → Real-time Analytics](realtime_sync_and_analysis_practice.md)
+- [Build a Streaming Data Pipeline with Dynamic Tables](streaming_pipeline_with_dynamic_table.md)

package/bin/skills/lakehouse-doc-en/references/streaming_data_pipeline_overview1.md

@@ -0,0 +1 @@
+# Incremental Computation

package/bin/skills/lakehouse-doc-en/references/streaming_pipeline_with_dynamic_table.md

@@ -0,0 +1,93 @@
+# Develop Dynamic Tables for Near Real-Time Incremental Processing
+**【Preview Release】This feature is currently in an invited preview stage. To try it out, please contact Singdata Technology through the official website.**
+
+## Tutorial Overview
+
+Through this tutorial, you will learn how to use Lakehouse Dynamic Table to build a complete real-time data ETL processing flow. The scenario design is as follows:
+
+![](https://studio-prod-sh.oss-cn-shanghai.aliyuncs.com/fe-asset/tutorials/resources/tutorial_dt_scenario.jpg?OSSAccessKeyId=LTAI5tBH4MDxrfQw7VTx4w2B\&Expires=2034679003\&Signature=gewhAp%2FJ80Uu%2BFq1sNEH3fwQI54%3D)
+
+This tutorial will implement a streaming data processing pipeline case with minute-level latency through data import tasks, data cleaning and transformation based on dynamic tables, and data aggregation tasks.
+
+> This tutorial leverages Lakehouse Tutorial, providing online tutorial guides and script import functions. After logging into the Lakehouse Web console, you can enter the tutorial through the "Lakehouse Tutorial" entry and follow the online guide to complete the tutorial.
+> ![](<.topwrite/assets/screenshot2025-01-14 19.36.29.png>)
+
+### Prerequisites
+
+Dynamic Table is an object type that supports processing only incremental change data, similar to materialized views. When creating a dynamic table, you need to define the data computation logic. By performing a refresh action, you can trigger the computation logic and update the dynamic table. Compared to traditional ETL tasks, dynamic tables only require the declaration of full semantic computation logic at the time of definition, and dynamic tables can automatically perform incremental computation optimization during refresh. With dynamic tables, Lakehouse can perform near real-time incremental processing for streaming data, providing data preparation for real-time analysis.
+
+### Tutorial Steps
+
+1. **Data Import**: Create the raw table and continuously write user behavior data;
+2. **Develop Dynamic Table Model**: Construct data cleaning and data aggregation processing flows through Dynamic Table;
+3. **Verify Processing Results**: Monitor the refresh execution status of the data pipeline constructed by the dynamic table, and check the changes in the consumption layer data table to verify the real-time data processing results.
+
+Through the above steps, you will learn how to develop streaming processing tasks in Lakehouse using dynamic tables.
+
+## Preparation
+
+First, create a behavior log table and write test data through the INSERT INTO task.
+
+> To simplify the tutorial, this tutorial simulates continuous data writing through periodically scheduled INSERT INTO tasks. In actual production environments, streaming write interfaces are often used for real-time data, and Singdata Lakehouse also provides SDK/Flink Connector and other methods for real-time writing. It is not recommended to use INSERT INTO for production tasks.
+
+This tutorial provides SQL scripts for creating data tables and inserting test data in the "Development" module. Open the \[Tutorial\_Working\_With\_Dynamic\_Table->Step01.Preparation] script file, configure a 1-minute interval scheduling strategy, and submit the deployment to simulate real-time data import.
+
+![](https://studio-prod-sh.oss-cn-shanghai.aliyuncs.com/fe-asset/tutorials/resources/tutorial_dt_insert.gif?OSSAccessKeyId=LTAI5tBH4MDxrfQw7VTx4w2B\&Expires=2034681423\&Signature=m9w14ltjxHYgEUBkADmFSenqmeU%3D)
+
+## Transform Layer Dynamic Table Model Development
+
+In the \[Development] module, open the \[Tutorial\_Working\_With\_Dynamic\_Table->Step02.Transformation\_With\_Dynamic\_Table] sample dynamic table task file. This dynamic table defines the ETL logic for cleaning and transforming the raw table.
+
+Refer to the figure below to configure the "Run Cluster" and "Scheduling Parameters":
+
+![](https://studio-prod-sh.oss-cn-shanghai.aliyuncs.com/fe-asset/tutorials/resources/tutourial_dt_transform.gif?OSSAccessKeyId=LTAI5tBH4MDxrfQw7VTx4w2B\&Expires=2034682100\&Signature=z8F6by8Cae7HFYeTggx0VR7NGxQ%3D)
+
+## Submit Deployment
+
+Click the \[Submit] button to deploy the dynamic table model to the Lakehouse target environment. The system will automatically refresh according to the refresh frequency set by the dynamic table.
+
+## Aggregate Layer Dynamic Table Model Development
+In the [Development] module, open the [Tutorial_Working_With_Dynamic_Table->Step03.Aggregation_With_Dynamic_Table] sample dynamic table task file. This dynamic table defines the ETL logic for aggregating and analyzing the intermediate data model.
+
+Please refer to the following image for scheduling configuration and submitting the run:
+
+![](https://studio-prod-sh.oss-cn-shanghai.aliyuncs.com/fe-asset/tutorials/resources/tutourial_dt_agg.gif?OSSAccessKeyId=LTAI5tBH4MDxrfQw7VTx4w2B\&Expires=2034682137\&Signature=jcNW0LS9R3H2TuyP9xY17097tPk%3D)
+
+## Verify Incremental Update Results
+
+In the [Development] module, open the Step04.Check_Data_Freshness file. Check the data freshness of the automatically refreshed dynamic table by executing the Query.
+![](.topwrite/assets/image_1725463960356.png =754)
+As shown in the image above, with the help of the dynamic table's automatic incremental refresh, real-time data processing can be completed with a delay of about 1 minute.
+
+## Environment Cleanup
+
+In the [Task Operations] module, take the data import task offline from the periodic task list; take the two dynamic table model tasks offline from the dynamic table task list.
+
+## FAQ
+
+**Q1: Data is not updated after the Dynamic Table refreshes. How do I troubleshoot?**
+
+- Run `SHOW DYNAMIC TABLE REFRESH HISTORY` to check the recent refresh status and confirm whether the refresh executed successfully
+- Confirm whether new data has been written to the source table (compare row counts before and after the refresh using `SELECT COUNT(*)`)
+- Check whether the refresh interval is set appropriately — a long `INTERVAL` will cause data delays
+- If `refresh_mode` shows `FULL` instead of `INCREMENTAL`, the SQL contains operators that do not support incremental computation
+
+**Q2: Refresh fails with "incremental refresh not supported". What should I do?**
+
+- Some SQL operators do not support incremental computation, such as window functions with `ORDER BY` or certain subqueries
+- Solution: simplify the SQL logic, or accept full refresh by using `CREATE OR REPLACE DYNAMIC TABLE ... REFRESH_MODE = FULL` to force full mode
+
+**Q3: Which VCluster should I use for refresh execution?**
+
+- Refresh is a compute-intensive operation; it is recommended to use a GENERAL type cluster
+- Do not share the same cluster with query workloads to avoid refresh tasks affecting query response times
+- For high-frequency refreshes (< 1 minute), it is recommended to use a dedicated small-spec cluster (XSMALL)
+
+**Q4: When should I use Dynamic Tables vs. regular Studio scheduled tasks?**
+
+| Scenario | Recommended approach |
+|------|---------|
+| Pure SQL incremental processing with simple dependencies | Dynamic Table (automatically manages dependencies and refresh) |
+| Requires Python processing or calling external APIs | Studio task |
+| Multiple upstream/downstream tasks with complex dependency orchestration | Studio task flow |
+| Requires precise execution timing (e.g. on the hour) | Studio scheduled task |

package/bin/skills/lakehouse-doc-en/references/string_function.md

@@ -0,0 +1 @@
+# String Functions

package/bin/skills/lakehouse-doc-en/references/string_guide.md

@@ -0,0 +1,42 @@
+# String Types
+
+Singdata Lakehouse supports the following string types: CHAR, VARCHAR(n), and STRING.
+
+## Type Comparison
+
+| Type | Length | Description |
+|------|--------|-------------|
+| CHAR(n) | Fixed length n | Lakehouse does not pad with spaces; strings exceeding the length are silently truncated; max 255 characters |
+| VARCHAR(n) | Variable, max n | No space padding; max 1,048,576 characters |
+| STRING | Variable, no limit | Recommended for large text; no length restriction |
+
+## Selection Guide
+
+- Fixed-length codes (e.g., country codes, status codes): use `CHAR`
+- Fields with a length cap (e.g., usernames, email addresses): use `VARCHAR(n)`
+- Long text, JSON strings, fields with no length constraint: use `STRING`
+
+> ⚠️ **Note**: All three types **silently truncate** when the declared length is exceeded — no error is raised. Validate length at the application layer before writing.
+
+> ⚠️ **Note**: Lakehouse `CHAR` does not pad with spaces, which differs from standard SQL behavior. `LENGTH(CAST('abc' AS CHAR(10)))` returns `3`, not `10`.
+
+## Examples
+
+```SQL
+-- CHAR: truncates on overflow, does not pad on underflow
+SELECT CAST('hello' AS CHAR(3));           -- 'hel' (truncated)
+SELECT length(CAST('hi' AS CHAR(5)));      -- 2 (no space padding)
+
+-- VARCHAR: truncates on overflow
+SELECT CAST('hello world' AS VARCHAR(5));  -- 'hello'
+
+-- STRING: no length limit
+SELECT typeof('any length of text');       -- string
+```
+
+## Related Documentation
+
+- [CHAR](char.md)
+- [VARCHAR(n)](varcharlength.md)
+- [STRING](string.md)
+- [Data Types](data-type.md)

package/bin/skills/lakehouse-doc-en/references/struct-function.md

@@ -0,0 +1 @@
+# STRUCT Functions