@clickzetta/cz-cli-darwin-x64 0.3.94 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/create-table-ddl.md

@@ -0,0 +1,390 @@
+# CREATE TABLE
+
+## Description
+
+This statement is used to create a new table. In Lakehouse, a table is the basic unit for storing data. By creating a table, you can organize and manage data according to a specified structure.
+
+**This document covers**: basic table creation syntax → column types → primary key / auto-increment / generated columns / default values → index definitions (Bloomfilter / inverted / vector) → partitioning (PARTITIONED BY) → bucketing (CLUSTERED BY) → table properties (PROPERTIES) → LIKE table creation → CTAS (AS SELECT) → complete examples.
+
+### Basic Table Creation Statement
+
+```SQL
+CREATE TABLE [ IF NOT EXISTS ] table_name
+(
+    column_definition  [column_definition ,...]
+    index_definition_list
+)
+[ PARTITIONED BY (column_name column_type | column_name | transform_function) ]
+[ CLUSTERED BY (column_name,...) 
+    [SORTED BY (column_name [ ASC | DESC ])] 
+    [INTO num_buckets BUCKETS] 
+]
+[ COMMENT 'table_comment' ]
+[PROPERTIES('data_lifecycle'='day_num')];
+
+
+```
+### column\_definition Description
+
+#### Syntax
+```
+column_name column_type 
+{ NOT NULL |
+  PRIMARY KEY|
+  IDENTITY[(seed)]|
+  GENERATED ALWAYS AS ( expr ) |
+  DEFAULT default_expression |
+  COMMENT column_comment |   
+}
+```
+#### Column Type column\_type
+
+* **column\_type**: The data type of the column, supporting integers, floating-point numbers, strings, date/time, booleans, arrays, Map, Struct, JSON, VECTOR, and other types. For the complete list and descriptions, see [Data Types](data-type.md).
+
+* **NOT NULL**: Indicates that this column is not allowed to be NULL. It only supports being specified when creating the table and does not support adding using the ALTER syntax. If you need to remove the NOT NULL constraint, please use the modify table type syntax:
+```
+ALTER TABLE table_name CHANGE COLUMN colum_name data_type
+```
+For example, remove the `not null` constraint for the `int` type
+```
+CREATE TABLE aa_not_null (id int NOT NULL)
+ALTER TABLE aa_not_null CHANGE COLUMN id TYPE int;
+```
+#### Primary Key (PRIMARY KEY)
+
+* **PRIMARY KEY**: Used to ensure the uniqueness of each record in the table. In big data scenarios, since data volumes are typically very large, checking all keys one by one to ensure uniqueness is impractical and inefficient, so primary key constraints are generally not recommended in big data environments. However, Lakehouse still provides support for primary keys to meet data integrity requirements in specific scenarios. In the Lakehouse architecture, for tables with a defined primary key, the system will automatically deduplicate data based on the primary key value during real-time data ingestion, which is especially important for Change Data Capture (CDC) scenarios. For example, you can synchronize MySQL database binlog logs to Lakehouse in real time to ensure data consistency. Once a primary key is set, you need to handle data through the [real-time data interface](java_reference/realtime-upload.md). During CDC real-time writes, the system will automatically deduplicate data based on the primary key to maintain data accuracy and integrity. This can only be specified when creating the table. For more details, refer to the [Primary Key Introduction](primary-key.md).
+```
+CREATE    TABLE pk_table 
+(id int, col string PRIMARY KEY (id));
+
+CREATE    TABLE pk_table 
+(id int PRIMARY KEY, col string);
+--Definition of primary key in bucketed table
+CREATE    TABLE pk_table (
+          id int,
+          col string,
+          cluster_key string,
+          PRIMARY key (id)
+          ) CLUSTERED BY (id, cluster_key) SORTED BY (id) INTO 16 BUCKETS;
+
+--Definition of primary key in partitioned table
+CREATE    TABLE pk_table (
+          id int,
+          col string,
+          pt string,
+          PRIMARY key (id, pt)
+          ) PARTITIONED BY (pt);
+```
+#### Auto-increment Column (IDENTITY\[(seed)])
+
+* **IDENTITY\[(seed**)]: Supports specifying auto-increment. It cannot guarantee that the values in the sequence are continuous (gapless), nor can it guarantee that the sequence values are allocated in a specific order. This is because other concurrent inserts may occur in the table. These limitations are part of the design to improve performance. For specific usage, refer to the [IDENTITY Column documentation](identity-column.md)
+```
+CREATE    TABLE identity_test (id bigint IDENTITY(1), col string);
+```
+#### Generated Columns (GENERATED ALWAYS AS)
+
+* **GENERATED ALWAYS AS (expr)**: Automatically generates the value of a column through the expression `expr`. The expression can include constants and built-in scalar deterministic SQL functions. Non-deterministic functions such as (current\_date, random, current\_timestamp, context functions) or operators are not supported. Aggregate functions, window functions, or table functions are also not supported. Partition columns using generated columns are supported.
+```
+CREATE TABLE t_genet (col1 TIMESTAMP,pt STRING GENERATED ALWAYS AS (date_format(col1, 'yyyy-MM-dd'))) PARTITIONED BY (pt);
+```
+#### Default Values (DEFAULT)
+
+* **DEFAULT default\_expression**: Defines a default value for a newly added column. If the value of this column is not specified in INSERT, UPDATE, or MERGE operations, this default value will be automatically used. For data rows that existed before the column was added, the column will be filled with null. Supports non-deterministic functions such as (current\_date, random, current\_timestamp, context functions) and constant values.
+```
+CREATE TABLE t_default(id INT,col1 STRING DEFAULT current_timestamp());
+```
+### Index Definition (index\_definition\_list)
+
+####  Syntax
+```
+INDEX index_name (col_name) index_type [COMMENT 'xxxxxx'] [PROPERTIES('key'='value')]
+```
+**columns\_difinition**: Defines the field information of the table, the last field must be separated by a comma
+
+**INDEX**: Keyword
+
+**index\_name**: Custom name of the index
+
+**column\_name**: The name of the field that needs to be indexed
+
+**index\_type**: Index type, currently supports [bloomfilter](create-bloomfilter-index.md), [inverted](inverted-index.md), [vector](create-vector-index.md)
+
+**COMMENT**: Specifies the description information of the index
+
+**PROPERTIES**: Specifies the parameters of the INDEX, different indexes support different parameters, refer to the corresponding index documentation for details
+
+### Partitioned By (PARTITIONED BY) {#partitioned-by}
+
+Partitioning is a method of speeding up queries by grouping similar rows together at the time of writing. Using partitioning can achieve data pruning and optimize queries. When querying a table, use the WHERE clause to query the specified partition to avoid full table scans, improve processing efficiency, and reduce computing resources. For details, refer to [Partition Introduction](partition_table.md). Note that a single task currently limits 2048 partitions during writes; exceeding this limit will report an error: `The count of dynamic partitions exceeds the maximum number 2048`. Before inserting, it is recommended to count the number of partitions first, e.g.: `select count(distinct pt) from table`. If you do have that many partitions, you can import in batches, or you can modify this limit by adding the parameter `set cz.sql.table.sink.max.partition.per.thread=10000`. There is no limit on the total number of partitions in Lakehouse; the limit applies per single task. If your data volume is small, it is recommended not to set cluster key and partition key. It is recommended to keep a single partition and cluster key at the hundreds of MB to GB level, for example a parquet format file compressed to 128MB.
+
+Two writing methods are supported.
+The first method declares the partition fields and types when creating the table; just declare the fields in the PARTITIONED BY clause:
+```SQL
+CREATE TABLE prod.db.sample (
+id bigint,
+category string,
+data string
+)
+PARTITIONED BY(category)
+```
+The second type of partition field and type is written in the PARTITIONED BY statement:
+```SQL
+CREATE TABLE prod.db.sample (
+id bigint,
+data string)
+PARTITIONED BY(category string)
+```
+### Bucketed Table (CLUSTERED BY)
+
+* CLUSTERED BY: Specify the Hash Key. Singdata will perform a hash operation on the specified column and distribute the data into various data buckets based on the hash value. To avoid data skew and hotspots, and to improve parallel execution efficiency, it is recommended to choose columns with a large range of values and few duplicate keys as the Hash Key. This usually has a significant effect when performing join operations. It is recommended to use CLUSTERED BY in scenarios with large amounts of data, generally with a bucket size between 128MB and 1GB. If no bucketing is specified, the default is 256 buckets. It is recommended to keep SORTED BY and CLUSTERED BY consistent for better performance. When the SORTED BY clause is specified, row data will be sorted according to the specified columns. For more information, refer to [Bucketing](cluster-table.md)
+```
+--Specify bucketing and sorting
+CREATE TABLE sales_data (
+    sale_id INT,
+    product_id INT,
+    quantity_sold INT,
+    sale_date DATE
+) CLUSTERED BY (product_id)
+SORTED BY (sale_date DESC)
+INTO 50 BUCKETS;
+--Specify bucketing and the number of buckets
+CREATE TABLE sales_data (
+    sale_id INT,
+    product_id INT,
+    quantity_sold INT,
+    sale_date DATE
+) CLUSTERED BY (product_id)
+INTO 50 BUCKETS;
+```
+### SORTED BY
+
+SORTED BY: Specifies the sorting method for fields within a file. SORTED BY in Lakehouse can be used independently, indicating sorting within the file when used alone. Specifying SORTED BY can speed up data retrieval, but since sorting is required during writing, it may increase the time taken for writing.
+```
+CREATE TABLE sales_data (
+    sale_id INT,
+    product_id INT,
+    quantity_sold INT,
+    sale_date DATE
+) 
+SORTED BY (sale_date DESC);
+```
+### PROPERTIES
+
+You can set table-level properties via `PROPERTIES` at table creation time, and also modify them via `ALTER TABLE ... SET PROPERTIES`.
+
+| Parameter | Description | Example Value |
+|------|------|--------|
+| `data_lifecycle` | Data lifecycle (days). Data older than the specified number of days is automatically expired and deleted. Suitable for time-sensitive data such as logs and transaction records. See [Data Lifecycle](data-lifecycle.md) | `'365'` |
+| `data_retention_days` | Time Travel data retention period (days). Determines how long historical version data is retained, affecting the available window for Time Travel, Table Stream, and Dynamic Table. Default is 1 day | `'7'` |
+| `change_tracking` | Enable Table Stream change tracking. **Note: specifying this at table creation time has no effect**; it must be enabled via the ALTER command | `'true'` |
+| `partition.cache.policy.latest.count` | Cache the latest N partitions to the local SSD of the compute cluster. When new partitions are added, the cache of old partitions is automatically invalidated. Suitable for accelerating hot data in time-partitioned tables | `'10'` |
+
+> ⚠️ **Note**: `change_tracking` does not take effect when specified at table creation time; it must be enabled via ALTER:
+> ```sql
+> ALTER TABLE table_name SET PROPERTIES ('change_tracking' = 'true');
+> ```
+
+```sql
+-- Set lifecycle and Time Travel retention period at table creation
+CREATE TABLE historical_prices (
+    ticker_symbol STRING,
+    trading_date  DATE,
+    closing_price DECIMAL(10, 2)
+) PROPERTIES (
+    'data_lifecycle'      = '365',
+    'data_retention_days' = '7'
+);
+
+-- Modify properties of an existing table
+ALTER TABLE historical_prices SET PROPERTIES ('data_lifecycle' = '30');
+ALTER TABLE historical_prices SET PROPERTIES ('partition.cache.policy.latest.count' = '10');
+
+-- View current property settings of a table
+SHOW PROPERTIES IN TABLE historical_prices;
+
+-- Remove a property
+ALTER TABLE historical_prices UNSET PROPERTIES ('data_lifecycle');
+```
+
+### Using LIKE Statement to Create a Table
+```SQL
+CREATE TABLE [ IF NOT EXISTS ] table_name
+LIKE source_table
+[ INCLUDING TBLPROPERTIES ]
+[ COMMENT 'table_comment' ]
+[ TBLPROPERTIES ('key'='value', ...) ];
+```
+When creating a new table using the LIKE statement, the target table will have the same table structure as the source table (column definitions, primary key, unique key, bucketing, sort key), but the data will not be copied. The source table's COMMENT is automatically copied to the new table unless the user explicitly specifies a new COMMENT.
+
+By default, the source table's table properties (TBLPROPERTIES) are not copied. If you need to copy the source table's properties, use the `INCLUDING TBLPROPERTIES` clause. User-specified TBLPROPERTIES will override properties of the same name copied from the source table.
+
+### Creating a Table Using the AS Statement
+
+The CREATE TABLE AS SELECT (CTAS) statement can be used to query the original table synchronously or asynchronously, create a new table based on the query results, and then insert the query results into the new table. It should be noted that tables created in this way do not copy partition information.
+```SQL
+CREATE TABLE [ IF NOT EXISTS ] table_name
+[ AS select_statement ];
+```
+## Example
+
+1. Create a partitioned table
+
+Syntax 1:
+```SQL
+CREATE TABLE table_part (id INT, name STRING)
+PARTITIONED BY (age INT);
+```
+Syntax Two:
+```SQL
+CREATE TABLE table_fpart (id INT, name STRING, dt STRING)
+PARTITIONED BY (dt) COMMENT '11';
+```
+2. Create a product table with an auto-increment column as a unique identifier
+```sql
+CREATE TABLE IF NOT EXISTS products (
+    product_id BIGINT IDENTITY(1) COMMENT 'Product ID',
+    name VARCHAR(255) NOT NULL COMMENT 'Product Name',
+    price DECIMAL(10, 2) NOT NULL COMMENT 'Price'
+) COMMENT 'Product List';
+```
+3. Create a Timestamp Conversion Table with Generated Columns
+
+```sql
+CREATE TABLE IF NOT EXISTS timestamps (
+    event_time TIMESTAMP COMMENT 'Event occurrence time',
+    formatted_date STRING GENERATED ALWAYS AS (date_format(event_time, 'yyyy-MM-dd')) COMMENT 'Formatted date'
+) PARTITIONED BY (formatted_date);
+```
+4. Create an activity record table with a default value of the current timestamp
+
+```sql
+CREATE TABLE IF NOT EXISTS activities (
+    activity_id BIGINT NOT NULL COMMENT 'Activity ID',
+    description VARCHAR(255) COMMENT 'Description',
+    event_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP COMMENT 'Record Time'
+) COMMENT 'Activity Records';
+```
+5. Create a Search Optimized Table with Bloom Filter Index
+```sql
+CREATE TABLE IF NOT EXISTS search_optimized (
+    id BIGINT COMMENT 'ID',
+    description VARCHAR(255) COMMENT 'Keyword',
+    INDEX bloom_index (id) BLOOMFILTER COMMENT 'Bloom Filter Index'
+) COMMENT 'Table for quick lookup';
+```
+6. Create a Text Analysis Table with an Inverted Index
+```sql
+CREATE TABLE IF NOT EXISTS text_analysis (
+    doc_id BIGINT COMMENT 'Document ID',
+    content TEXT COMMENT 'Content',
+    INDEX inverted_content (content) INVERTED PROPERTIES ('analyzer' = 'chinese') COMMENT 'Inverted Index'
+) COMMENT 'Table for text analysis';
+```
+7. Create a Sales Data Table with Bucketing and Sorting
+```sql
+CREATE TABLE IF NOT EXISTS sales_data (
+    sale_id BIGINT COMMENT 'Sale ID',
+    product_id BIGINT COMMENT 'Product ID',
+    quantity_sold INT COMMENT 'Quantity Sold',
+    sale_date DATE COMMENT 'Sale Date'
+) CLUSTERED BY (product_id)
+SORTED BY (sale_date DESC)
+INTO 50 BUCKETS COMMENT 'Sales Data Table';
+```
+8. Create a data retention table with lifecycle management
+```sql
+CREATE TABLE IF NOT EXISTS historical_prices (
+    ticker_symbol VARCHAR(10) COMMENT 'Ticker Symbol',
+    trading_date DATE COMMENT 'Trading Date',
+    closing_price DECIMAL(10, 2) COMMENT 'Closing Price'
+) PROPERTIES (
+    'data_lifecycle' = '365',
+    'data_retention_days' = '1'
+) COMMENT 'Historical Stock Prices Table';
+```
+9. Create a new table similar to the existing table structure
+
+```sql
+CREATE TABLE IF NOT EXISTS new_users LIKE users COMMENT 'New Users Table';
+```
+10. Create a Table Initialized by Query Results
+
+```sql
+CREATE TABLE IF NOT EXISTS recent_sales AS
+SELECT * FROM sales WHERE sale_date >= DATE_SUB(CURRENT_DATE, 3);
+```
+11. Create an Order Details Table Containing Array Types
+
+```sql
+CREATE TABLE IF NOT EXISTS order_details (
+    order_id BIGINT COMMENT 'Order ID',
+    items ARRAY<STRUCT<item_id:BIGINT, quantity:INT, price:DECIMAL(10, 2)>> COMMENT 'Item List'
+) COMMENT 'Order Details';
+```
+12. Create a Customer Feedback Table with JSON Type
+```sql
+CREATE TABLE IF NOT EXISTS customer_feedback (
+    feedback_id BIGINT COMMENT 'Feedback ID',
+    feedback JSON COMMENT 'Feedback Content'
+) COMMENT 'Customer Feedback Table';
+```
+# User Guide
+
+### Partitioning and Bucketing
+
+* **Choosing a Partitioning Strategy**: Select appropriate partition fields based on the query pattern. Generally, columns that can effectively narrow the scan range should be chosen as partition keys. For example, a timestamp column is very suitable as a partition key for date range queries.
+
+* **Setting the Number of Buckets**: The number of buckets should be adjusted based on the expected data volume and hardware resource configuration. Typically, each bucket size should be between 256MB and 1GB. Too many or too few buckets will affect the overall performance of the system. To achieve the best results, it is recommended to test the performance under different configurations and make appropriate adjustments accordingly.
+
+* **Choosing Sorting Fields**: When using the `SORTED BY` clause, select columns that frequently appear in filter conditions as the sorting basis. Good sorting can help speed up point queries and range queries, but it also increases the write cost. Therefore, it is important to make a decision after weighing the pros and cons.
+
+### Indexing
+
+Users can create indexes on multiple columns when creating a table. Indexes can also be added after the table is created. If indexes are added during later use and the table already contains data, all data needs to be rewritten, so the time to create the index depends on the current data volume.
+
+### Best Practices for Table Property Settings
+
+* **Enable Data Lifecycle Management**: By setting the `data_lifecycle` property, the system can automatically clean up historical data that is no longer needed. This is very useful for saving storage, especially when dealing with datasets with a clear retention period, such as logs or transaction records.
+* **Configure Change Tracking**: If your application scenario involves obtaining data changes in the table, you need to enable `change_tracking`. Common scenarios include [TABLE STREAM](tablestream_summary.md).
+
+> ⚠️ **Note**: `CREATE OR REPLACE TABLE` resets associated Streams. Using `OR REPLACE` to rebuild a table clears the table's history, causing all Table Streams based on that table to immediately become stale and unable to retrieve complete change data. To modify the table structure, prefer using `ALTER TABLE`.
+
+* **Set Data Retention Period for Deleted Data**: This parameter defines the length of time that deleted data is retained, which is very important for scenarios that require historical data queries. For example, features like [table stream](tablestream_summary.md), [restore](restore.md), and [dynamic table](dynamic-table-introduce.md) rely on this retention period setting. By default, Lakehouse retains data for one day. Depending on your business needs, you can adjust the `data_retention_days` parameter to extend or shorten the data retention period. Please note that adjusting the data retention period may affect storage costs. Extending the retention period will increase storage requirements, which may increase related costs.
+
+## Related Guides
+
+**Table Creation and Design**
+
+| Document | Description |
+|------|------|
+| [SQL CREATE TABLE Usage Guide](sql_create_table_guide.md) | Complete guide for table creation options, column types, partitioning, sort columns, and property configuration |
+| [Table Design Best Practices](lakehouse_table_design_guide.md) | Comprehensive selection recommendations for partitioning, bucketing, and indexing |
+| [Generated Columns Usage Guide](generated_columns_guide.md) | Definition and usage scenarios for virtual and stored columns |
+| [Data Types](data-type.md) | Descriptions and usage examples for all supported column types |
+
+**Partitioning and Bucketing**
+
+| Document | Description |
+|------|------|
+| [Partitioning and Bucketing](partition_table_guide.md) | Selection and design principles for partitioning and bucketing |
+| [Partitioned Table Usage Guide](notes-and-guidelines-for-partition-tables.md) | Usage patterns for partition design, partition pruning, and dynamic partitions |
+| [Bucketing (CLUSTERED BY)](cluster-table-guide.md) | Hash bucketing to improve Join and aggregation performance |
+| [Primary Key](primary-key.md) | Primary key table design, supporting CDC real-time write deduplication |
+
+**Indexing**
+
+| Document | Description |
+|------|------|
+| [Bloomfilter Index](bloomfilter-summary.md) | Accelerate equality queries |
+| [Inverted Index](inverted-index.md) | Full-text search acceleration, supports Chinese and English tokenization |
+| [Vector Index](vector-search.md) | ANN approximate nearest neighbor retrieval |
+| [Recommended Sort Columns for Tables](auto-index.md) | Accelerate range queries via sort columns |
+
+**Advanced Table Features**
+
+| Document | Description |
+|------|------|
+| [Data Lifecycle](data-lifecycle.md) | `data_lifecycle` parameter, automatically expire and delete historical data |
+| [ALTER TABLE](alter-table.md) | Modify table structure, properties, partitions, and other operations |
+| [Table Stream](tablestream_summary.md) | Capture incremental changes to a table based on `change_tracking` |
+| [Time Travel](timetravel-summary.md) | `data_retention_days` parameter, query and restore historical data |

package/bin/skills/lakehouse-doc-en/references/create-table-stream.md

@@ -0,0 +1,217 @@
+# Table Stream
+
+Table Stream is a real-time data stream used to record data manipulation language (DML) changes made to a table, including insert, update, and delete operations. At the same time, Table Stream also provides metadata about each change, so you can take appropriate actions based on the changed data. Table Stream can be created on Table, Dynamic Table, Materialized View, and External Table (Kafka). It also supports creating Volume Stream on External Volumes to monitor file change events in external object storage (such as OSS).
+
+## Table Stream Syntax
+
+**Syntax for Creating Table Stream on Table, Dynamic Table, and Materialized View**
+
+```SQL
+CREATE [OR REPLACE] TABLE STREAM [IF NOT EXISTS]
+  <name>
+  ON TABLE <table_name>
+  [TIMESTAMP AS OF timestamp_expression]
+  [COMMENT = '<string_literal>']
+  WITH PROPERTIES ('TABLE_STREAM_MODE' = 'APPEND_ONLY|STANDARD','SHOW_INITIAL_ROWS'='TRUE|FALSE');
+```
+
+* `<name>`: The name of the Table Stream.
+
+* `<table_name>`: The base table from which to get the incremental data. Views are not supported.
+
+* `TIMESTAMP AS OF timestamp_expression`: (Optional) Specifies the timestamp expression from which the Table Stream should start receiving updates from the underlying table. If this parameter is omitted, the Table Stream will start receiving updates from the current time.
+
+  * The `timestamp_expression` returns a standard timestamp type expression. The earliest timestamp specified by TIMESTAMP AS OF depends on the [TIME TRAVEL](timetravel.md)(data\_retention\_days) parameter. If the specified version does not exist, an error will be reported. If not specified, the current timestamp version data will be used, for example:
+    * `'2023-11-07 14:49:18'`, a string that can be cast to a timestamp.
+    * `cast('2023-11-07 14:49:18 Asia/Shanghai' as timestamp)`.
+    * `current_timestamp() - interval '12' hours`.
+    * Any other expression that is itself a timestamp or can be cast to a timestamp.
+
+* `COMMENT`: (Optional) Comment for the Table Stream.
+
+* `'TABLE_STREAM_MODE' = 'APPEND_ONLY|STANDARD'`: (Required) Choose one of the two values, APPEND\_ONLY or STANDARD.
+
+  * APPEND\_ONLY records only the INSERT operations of the object. Update and delete operations are not recorded. For example, if 10 rows were initially inserted into the table and then 5 rows were deleted without moving the offset, the Table Stream would still record 10 rows of operations.
+  * STANDARD mode: In STANDARD mode, all DML changes to the source object can be tracked, including inserts, updates, and deletes (including table truncation). This row-level change is provided by joining and processing all delta data changes. The delta changes in the Table Stream refer to the data changes that occur between two transaction points. For example, if a row is inserted and then updated after the Table Stream's offset, the delta change is a new row. If a row is inserted and then deleted after the stream's offset, the delta change is no row. In other words, the delta change reflects the latest state of the source object, not the historical changes.
+
+* SHOW\_INITIAL\_ROWS: Optional parameter that controls the visibility of existing data in the table at the time of Stream creation:
+  * `'FALSE'` (default): Existing data in the table at the time of Stream creation is not visible; only changes that occur **after** Stream creation are captured.
+  * `'TRUE'`: Existing data in the table at the time of Stream creation is exposed as `INSERT` records in the first consumption. After consuming the initial snapshot, subsequent changes produce `UPDATE_BEFORE`/`UPDATE_AFTER`/`DELETE` as normal.
+
+  **Note**: `SHOW_INITIAL_ROWS` only affects whether historical data is visible at the time of Stream creation, and **does not affect the values of `__change_type`**. Regardless of the value of this parameter, in STANDARD mode, UPDATE always produces both `UPDATE_BEFORE` and `UPDATE_AFTER` rows, and DELETE always produces `DELETE` records.
+
+> CAUTION: **Visible Delay for Streaming Write Data**: Data written to the source table through the Ingestion Service (real-time streaming write) requires approximately **1 minute** to be committed before it can be queried in the Table Stream. Data written directly via SQL DML has no such delay and is immediately visible after the task succeeds.
+
+### Notes
+
+* The earliest timestamp specified by TIMESTAMP AS OF depends on the [TIME TRAVEL](timetravel.md)(data\_retention\_days) parameter. If the specified version does not exist, an error will be reported. This parameter defines the length of time that deleted data is retained. By default, Lakehouse retains data for one day. Depending on your business needs, you can extend or shorten the data retention period by adjusting the `data_retention_days` parameter. Please note that adjusting the data retention period may affect storage costs. Extending the retention period will increase storage requirements, which may increase related costs.
+* Data written through real-time uploads can only be read after one minute, and Table Stream can only read committed data. Data written by real-time tasks needs to wait for 1 minute to be confirmed, so Table Stream also needs to wait for 1 minute to see it.
+
+### Usage Examples
+
+Case 1: Create a Table Stream in APPEND\_ONLY mode
+
+```SQL
+-- Clean up the environment
+DROP      TABLE IF EXISTS data_change_test;
+
+DROP      TABLE STREAM IF EXISTS data_change_test_stream;
+
+-- Create test table
+CREATE    TABLE data_change_test (id int, name string);
+
+INSERT    INTO data_change_test
+VALUES    (1, 'apple');
+
+-- Create TABLE stream to capture incremental records inserted from the current time
+CREATE    TABLE STREAM data_change_test_stream ON TABLE data_change_test
+WITH      PROPERTIES ('TABLE_STREAM_MODE' = 'APPEND_ONLY');
+
+-- Insert test data
+INSERT    INTO data_change_test
+VALUES    (2, 'banana');
+
+-- View STREAM data
+SELECT    *
+FROM      data_change_test_stream;
+```
+
+Case 2: Create a Table Stream in STANDARD mode
+
+```SQL
+-- Clean up the environment
+DROP      TABLE IF EXISTS data_change_test;
+
+DROP      TABLE STREAM IF EXISTS data_change_test_stream;
+
+-- Create test table
+CREATE    TABLE data_change_test (id int, name string);
+
+INSERT    INTO data_change_test
+VALUES    (1, 'apple');
+
+-- Create Table Stream in STANDARD mode
+CREATE    TABLE STREAM data_change_test_stream ON TABLE data_change_test
+WITH      PROPERTIES ('TABLE_STREAM_MODE' = 'STANDARD');
+
+-- Insert test data
+INSERT    INTO data_change_test
+VALUES    (2, 'banana');
+
+-- Update test data
+update data_change_test set name = 'orange'
+WHERE     id = 2;
+
+-- Delete test data
+DELETE
+FROM      data_change_test
+WHERE     id = 1;
+
+-- View stream data (STANDARD mode reflects the current state of the source table)
+SELECT    *
+FROM      data_change_test_stream;
+```
+
+## Kafka Table Stream
+
+Supports creating a Table Stream on a Kafka external table for real-time consumption of Topic data.
+
+#### Syntax
+
+```sql
+CREATE TABLE STREAM [IF NOT EXISTS] stream_name 
+ON TABLE external_kafka_table 
+[TIMESTAMP AS OF timestamp_expression]  -- Optional, specifies the consumption start time point
+WITH PROPERTIES (
+  'table_stream_mode' = 'append_only',  -- Only append mode is supported
+  'show_initial_rows' = 'true'|'false'
+);
+```
+
+### Parameter Description
+
+* `TIMESTAMP AS OF`: Specifies the consumption start time point (optional). Supported formats:
+  * **Explicit timestamp**: `'2023-01-01 12:00:00'`
+  * **Time function**: `CURRENT_TIMESTAMP() - INTERVAL '1' HOUR`
+* `show_initial_rows`: Controls initial data loading behavior:
+  * `true`: Loads historical data from the offset specified when the external table was created to the offset specified by the Table Stream
+  * `false`: Starts consuming from the latest offset (`latest`), does not load historical data (default value)
+* `table_stream_mode`: Fixed to `append_only`, only processes new Kafka data (update/delete operations are not supported)
+
+#### Examples
+
+1. Creating a Kafka External Table
+   First, you need to create a [Kafka External Table](kafka-external-table.md). This is the foundation for creating a Table Stream. The following is the syntax for creating a Kafka external table:
+
+Create a Kafka Connection
+
+```SQL
+CREATE STORAGE CONNECTION pipe_kafka
+    TYPE kafka
+    BOOTSTRAP_SERVERS = ['47.99.48.62:9092']
+    SECURITY_PROTOCOL = 'PLAINTEXT';
+```
+
+Create a Kafka External Table
+
+```SQL
+CREATE EXTERNAL TABLE external_table_kafka (   
+ key_column binary,   
+ value_column binary NOT NULL)
+USING kafka
+OPTIONS (    'group_id' = 'external_table_lh',    'topics' = 'test_long')
+CONNECTION pipe_kafka;
+```
+
+* `external_table_kafka` is the name of the external table.
+* `key_column` and `value_column` represent the key and value of the Kafka message respectively, where `value_column` is required.
+* `USING kafka` specifies that Kafka is used as the data source.
+* The `OPTIONS` section contains Kafka consumer configuration, such as the consumer group ID (`group_id`) and the topic to subscribe to (`topics`).
+* `CONNECTION pipe_kafka` specifies the connection configuration to Kafka, which typically includes the Kafka cluster address and other connection parameters.
+
+2. Creating a Table Stream
+
+Based on the Kafka external table, you can create a Table Stream for real-time processing of the Kafka data stream. The following is the syntax for creating a Table Stream:
+
+```
+CREATE TABLE STREAM kafka_table_stream_pipe1
+ ON TABLE external_table_kafka
+WITH PROPERTIES (
+    'table_stream_mode' = 'append_only',
+    'show_initial_rows' = 'true');
+```
+
+## Volume Stream
+
+**CREATE VOLUME STREAM Syntax**
+
+Used to create a Volume Stream on the Directory Table of an External Volume to monitor file creation and deletion events in external object storage.
+
+```
+CREATE [OR REPLACE] STREAM [IF NOT EXISTS] <name>
+  ON VOLUME <volume_name>;
+```
+
+**Parameter Description**
+
+`<name>`: The name of the Volume Stream.
+
+`<volume_name>`: The name of the External Volume to monitor. The Volume must meet the following conditions:
+
+* Directory feature must be enabled, i.e., specify DIRECTORY = (ENABLE = TRUE) when creating the table;
+* Recursive monitoring must be enabled, i.e., specify RECURSIVE = TRUE when creating the table.
+
+**Example**
+
+```
+-- Create a Volume Stream to track file creation and deletion
+CREATE OR REPLACE STREAM str_app_log
+  ON VOLUME vol_app_log;
+
+```
+
+**Notes**
+
+* Before creating a Volume Stream, you must enable DIRECTORY = (ENABLE = TRUE) and RECURSIVE = TRUE.
+* Volume Stream is essentially a Table Stream built on top of a Directory Table, and its consumption mechanism is fully consistent with Table Stream: only DML statements (such as INSERT INTO ... SELECT FROM stream) advance the consumption offset; pure SELECT queries do not advance it.
+* Since object storage events are delivered through message queues with approximately 1 minute of delay, you need to wait about 1 minute after uploading a file before the corresponding change record can be queried in the Volume Stream.

package/bin/skills/lakehouse-doc-en/references/create-user.md

@@ -0,0 +1,59 @@
+## Function
+
+Adds a user to the current workspace, allowing them to access and operate on resources within the workspace.
+
+> ⚠️ **Note**: `CREATE USER` adds a user who already exists in the instance user management system into the current workspace, rather than creating a brand-new account. If the user does not exist in the instance, the command will report an error.
+
+## Syntax
+
+```Plain
+CREATE USER [IF NOT EXISTS] user_name
+    [ DEFAULT_VCLUSTER = vc_name ]
+    [ DEFAULT_SCHEMA = schema_name ]
+    [ COMMENT = 'comment_text' ];
+```
+
+## Parameter Description
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `IF NOT EXISTS` | No | If the user already exists in the workspace, skip the operation without error |
+| `user_name` | Yes | User identifier; must be a username already created in the instance user management system |
+| `DEFAULT_VCLUSTER` | No | Specifies a default compute cluster for the user. When not specified, the workspace's global default cluster is used |
+| `DEFAULT_SCHEMA` | No | Specifies a default Schema for the user. Once set, the user automatically uses this Schema upon login; the `USE` command can switch to a different Schema during the session. Priority: `USE` command > default Schema |
+| `COMMENT` | No | Adds a note or description for the user |
+
+## Usage Examples
+
+1. Add user `uat_test` to the workspace, using the global default compute cluster and Schema:
+
+   ```SQL
+   CREATE USER uat_test;
+   ```
+
+2. Add a user while specifying both a default compute cluster and a default Schema:
+
+   ```SQL
+   CREATE USER uat_test DEFAULT_VCLUSTER=vcluster1 DEFAULT_SCHEMA=schema1;
+   ```
+
+3. Use `IF NOT EXISTS` to avoid errors when the user already exists:
+
+   ```SQL
+   CREATE USER IF NOT EXISTS uat_test;
+   ```
+
+4. Add a user with a comment:
+
+   ```SQL
+   CREATE USER uat_test COMMENT='UAT environment test account';
+   ```
+
+Successful execution returns an empty result set; no error message means the operation succeeded.
+
+## Notes
+
+- Executing this command requires the `workspace_admin` or `user_admin` role.
+- `user_name` must be an account that already exists in the instance user management system (not at the workspace level); otherwise, an error `NotFound: user does not exist` is thrown.
+- After a user is added to a workspace, they have no data permissions by default. Permissions must be granted via the `GRANT` command or by assigning roles.
+- When using `IF NOT EXISTS`, if the user already exists, the command silently succeeds without modifying any existing configuration.