@clickzetta/cz-cli-darwin-arm64 0.3.95 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/pipe-kafka-bestpractice-1.md

@@ -0,0 +1,431 @@
+# Best Practice: Efficiently Ingesting Kafka Data Using Pipe
+
+## Quick Verification of Kafka Parameters
+
+In Singdata Lakehouse, using Kafka Pipe makes it easy to build minute-level near-real-time data integration pipelines. Before you begin, confirm three things:
+
+1. Network connectivity
+2. Kafka bootstrap addresses, ports, and topics
+3. (Optional) Authentication method and related parameters
+
+You can quickly verify all of the above by running a query like `select * from read_kafka()`.
+
+Here is an example of unauthenticated Kafka ingestion:
+
+```SQL
+SELECT *
+FROM read_kafka(
+  'kafka-bootstrap-1:9092,kafka-bootstrap-2:9092,kafka-bootstrap-3:9092', -- bootstrap
+  'topic-name',   -- topic
+  '',             -- reserved
+  'test',         -- kafka group id, for keeping read position
+  '', '', '', '', -- reserved
+  'raw',          -- key format, can only be raw
+  'raw',          -- value format, can only be raw
+  0,
+  MAP(
+    'kafka.security.protocol','PLAINTEXT',
+    'kafka.auto.offset.reset','latest'
+  )
+)
+LIMIT 10;
+```
+
+> The `read_kafka` function has many parameters, but in practice you only need to fill in the bootstrap address, topic, and group id. During the exploration phase, use `test` as the group id.
+
+Here is an example reading Kafka with SASL\_PLAINTEXT authentication (Pipe supports only PLAINTEXT and SASL\_PLAINTEXT):
+
+```SQL
+SELECT *
+FROM read_kafka(
+  'kafka-bootstrap-1:9092,kafka-bootstrap-2:9092,kafka-bootstrap-3:9092', -- bootstrap
+  'topic-name',   -- topic
+  '',             -- reserved
+  'test',         -- kafka group id, for keeping read position
+  '', '', '', '', -- reserved
+  'raw',          -- key format, can only be raw
+  'raw',          -- value format, can only be raw
+  0,
+  MAP(
+    'kafka.security.protocol','SASL_PLAINTEXT',
+    'kafka.sasl.mechanism','PLAIN',
+    'kafka.sasl.username','<username>',
+    'kafka.sasl.password','<password>',
+    'kafka.auto.offset.reset','latest'
+  )
+)
+LIMIT 10;
+```
+
+If the parameters are configured correctly, executing the above SQL will return 10 rows of sample data.
+
+![](/.topwrite/assets/image_1760681545512.png)
+
+# Read Small Batches to Confirm Schema and Create the Target Table
+
+Both the key and value in Kafka data are binary types. Usually you care more about the value content. If it is originally stored as a string, you can cast it to string during SELECT to quickly explore its content.
+
+```sql
+SELECT key::string, value::string
+FROM read_kafka(
+  'kafka-bootstrap-1:9092,kafka-bootstrap-2:9092,kafka-bootstrap-3:9092', -- bootstrap
+  'topic-name',   -- topic
+  '',             -- reserved
+  'test',         -- kafka group id, for keeping read position
+  '', '', '', '', -- reserved
+  'raw',          -- key format, can only be raw
+  'raw',          -- value format, can only be raw
+  0,
+  MAP(
+    'kafka.security.protocol','PLAINTEXT',
+    'kafka.auto.offset.reset','latest'
+  )
+)
+LIMIT 10;
+```
+
+![](/.topwrite/assets/image_1760681595348.png)
+
+Click "Copy" to retrieve sample data and explore it. The value appears to be JSON, but some string fields within the JSON are themselves complete JSON objects — the structure is nested and somewhat complex:
+
+```JSON
+{"event":"{\"instance_id\":1,\"workspace_id\":1057330101457946860,\"session_id\":0,\"job_id\":\"\",\"log_id\":8316782525667550489,\"operator_id\":0,\"operator_type\":\"PT_USER\",\"start_time\":1740590015608,\"end_time\":1740590015608,\"state\":1,\"properties\":\"[]\",\"statements\":\"{\\\"statements\\\":[{\\\"identifier\\\":{\\\"type\\\":\\\"VIRTUAL_CLUSTER\\\",\\\"instanceId\\\":\\\"1\\\",\\\"namespace\\\":[\\\"system_automv_warehouse\\\"],\\\"namespaceId\\\":[\\\"1057330101457946860\\\"],\\\"namespaceType\\\":[],\\\"name\\\":\\\"CZ_MV_DEFAULT\\\",\\\"id\\\":\\\"8013167251718801474\\\",\\\"version\\\":\\\"default_v2\\\",\\\"instanceName\\\":\\\"clickzetta\\\",\\\"accountId\\\":\\\"1\\\",\\\"accountName\\\":\\\"rwyaytaa\\\"},\\\"operations\\\":[{\\\"lsn\\\":0,\\\"type\\\":\\\"ALTER\\\",\\\"alterEntity\\\":{\\\"ifExists\\\":false,\\\"identifier\\\":{\\\"type\\\":\\\"VIRTUAL_CLUSTER\\\",\\\"instanceId\\\":\\\"1\\\",\\\"namespace\\\":[\\\"system_automv_warehouse\\\"],\\\"namespaceId\\\":[\\\"1057330101457946860\\\"],\\\"namespaceType\\\":[],\\\"name\\\":\\\"CZ_MV_DEFAULT\\\",\\\"id\\\":\\\"8013167251718801474\\\",\\\"version\\\":\\\"default_v2\\\",\\\"instanceName\\\":\\\"clickzetta\\\",\\\"accountId\\\":\\\"1\\\",\\\"accountName\\\":\\\"rwyaytaa\\\"},\\\"changeComment\\\":false,\\\"entity\\\":{\\\"identifier\\\":{\\\"type\\\":\\\"VIRTUAL_CLUSTER\\\",\\\"instanceId\\\":\\\"1\\\",\\\"namespace\\\":[\\\"system_automv_warehouse\\\"],\\\"namespaceId\\\":[\\\"1057330101457946860\\\"],\\\"namespaceType\\\":[],\\\"name\\\":\\\"CZ_MV_DEFAULT\\\",\\\"id\\\":\\\"8013167251718801474\\\",\\\"version\\\":\\\"\\\",\\\"instanceName\\\":\\\"clickzetta\\\",\\\"accountId\\\":\\\"1\\\",\\\"accountName\\\":\\\"rwyaytaa\\\"},\\\"creator\\\":\\\"101\\\",\\\"creatorType\\\":\\\"PT_USER\\\",\\\"properties\\\":[],\\\"createTime\\\":\\\"1698056353612\\\",\\\"lastModifyTime\\\":\\\"1740590015607\\\",\\\"state\\\":\\\"ONLINE\\\",\\\"category\\\":\\\"MANAGED\\\",\\\"basicSpecId\\\":0,\\\"flags\\\":\\\"0\\\",\\\"virtualCluster\\\":{\\\"clusterType\\\":\\\"GENERAL\\\",\\\"tag\\\":{},\\\"clusterSize\\\":\\\"SMALL\\\",\\\"autoStopLatencySec\\\":1,\\\"autoStartEnabled\\\":true,\\\"queryProcessTimeLimitSec\\\":259200,\\\"state\\\":\\\"RESUMING\\\",\\\"preState\\\":\\\"SUSPENDED\\\",\\\"errorMsg\\\":\\\"\\\",\\\"workspaceId\\\":\\\"1057330101457946860\\\",\\\"vcId\\\":\\\"8013167251718801474\\\",\\\"stateInfo\\\":\\\"{\\\\\\\"resourceVersion\\\\\\\":\\\\\\\"1740590006831\\\\\\\",\\\\\\\"resumeTaskState\\\\\\\":\\\\\\\"true\\\\\\\"}\\\",\\\"version\\\":\\\"default_v2\\\",\\\"computePoolId\\\":\\\"0\\\",\\\"deployMode\\\":\\\"SERVERLESS\\\"},\\\"comment\\\":\\\"\\\"},\\\"alterProperty\\\":[]}}]}]}\",\"sub_type\":\"\"}","op_type":"CREATE","datasource_id":"17319","database_name":"lakehouse_hz_uat_bak","schema_name":"lakehouse_hz_uat_bak","table_name":"cz_commit_logs_vc","event_ts":1740590015000,"event_seq":"3521832368","server_ts":1740590015924,"server_seq":138789}
+```
+
+Adjust the SELECT statement to use `parse_json` to expand the value field and the `event` within it:
+
+```SQL
+SELECT
+    parse_json(j['event']::string) as event,
+    j['op_type']::string as op_type,
+    j['datasource_id']::string as datasource_id,
+    j['database_name']::string as database_name,
+    j['schema_name']::string as schema_name,
+    j['table_name']::string as table_name,
+    timestamp_millis(j['event_ts']::bigint) as event_ts,
+    j['event_seq']::string as event_seq,
+    timestamp_millis(j['server_ts']::bigint) as server_ts,
+    j['server_seq']::bigint as server_seq
+FROM (
+    SELECT parse_json(value::string) as j
+    FROM read_kafka(
+    'kafka-bootstrap-1:9092,kafka-bootstrap-2:9092,kafka-bootstrap-3:9092', -- bootstrap
+    'topic_name',   -- topic
+    '',             -- reserved
+    'test',         -- kafka group id, for keeping read position
+    '', '', '', '', -- reserved
+    'raw',          -- key format, can only be raw
+    'raw',          -- value format, can only be raw
+    0,
+    MAP(
+        'kafka.security.protocol','PLAINTEXT',
+        'kafka.auto.offset.reset','latest'
+    )
+    )
+    LIMIT 10
+);
+
+```
+
+Running this reveals that `statements` inside `event` is still JSON.
+
+![](/.topwrite/assets/image_1760681703861.png)
+
+Continue adjusting the SELECT statement until all JSON-formatted strings are fully parsed before writing to the table, avoiding redundant `parse_json` calculations in subsequent queries.
+
+```SQL
+SELECT
+    parse_json(j['event']::string) as event,
+    parse_json(parse_json(j['event']::string)['statements']::string) as statements,
+    j['op_type']::string as op_type,
+    j['datasource_id']::string as datasource_id,
+    j['database_name']::string as database_name,
+    j['schema_name']::string as schema_name,
+    j['table_name']::string as table_name,
+    timestamp_millis(j['event_ts']::bigint) as event_ts,
+    j['event_seq']::string as event_seq,
+    timestamp_millis(j['server_ts']::bigint) as server_ts,
+    j['server_seq']::bigint as server_seq
+FROM (
+    SELECT parse_json(value::string) as j
+    FROM read_kafka(
+    'kafka-bootstrap-1:9092,kafka-bootstrap-2:9092,kafka-bootstrap-3:9092', -- bootstrap
+    'topic_name',   -- topic
+    '',             -- reserved
+    'test',         -- kafka group id, for keeping read position
+    '', '', '', '', -- reserved
+    'raw',          -- key format, can only be raw
+    'raw',          -- value format, can only be raw
+    0,
+    MAP(
+        'kafka.security.protocol','PLAINTEXT',
+        'kafka.auto.offset.reset','latest'
+    )
+    )
+    LIMIT 10
+);
+```
+
+Based on the SELECT exploration results, determine the target table structure and create the table:
+
+```SQL
+CREATE TABLE ods_commit_log (
+    event json,
+    statements json,
+    op_type string,
+    datasource_id string,
+    database_name string,
+    schema_name string,
+    table_name string,
+    event_ts timestamp,
+    event_seq string,
+    server_ts timestamp,
+    server_seq bigint,
+    __kafka_timestamp__ timestamp,
+    pt_date string generated always as (date_format(`server_ts`, 'yyyyMMdd')) stored
+) PARTITIONED BY (pt_date)
+PROPERTIES(
+    'data_lifecycle'='14'
+); 
+```
+
+> Kafka data typically consists of append-only log tables. We therefore added an auto-generated column `pt_date` as the partition column based on the business timestamp `server_ts`, combined with the `data_lifecycle` table property to control data volume.
+
+> The table DDL includes the `__kafka_timestamp__` field, which records the time data passed through Kafka. This column has little direct business value, but it can be used to diagnose the cause of data delays. We recommend keeping it.
+
+At this point, you have completed the alignment between the Kafka data and the target table structure.
+
+# Creating the Pipe
+
+Next, use the exploration SELECT statement as a foundation to construct the Pipe DDL. A Pipe is essentially a wrapper around `COPY INTO <table> FROM (SELECT ... FROM READ_KAFKA(...))`, with additional parameters for specifying compute resources, scheduling frequency, and other operational settings.
+
+Given that Kafka data typically arrives as a stream, if you need 1-minute data freshness — meaning the Pipe schedules a `COPY INTO` job every 60 seconds — we recommend allocating a dedicated GP cluster for the Pipe.
+
+```sql
+CREATE VCLUSTER pipe_ods_commit_log 
+    VCLUSTER_TYPE = GENERAL 
+    VCLUSTER_SIZE = 1
+;
+```
+
+> Don't worry about the GP vcluster size for now — you can adjust it later based on actual job execution.
+
+> One reason to allocate a dedicated cluster is that high data freshness requirements (e.g., less than 5 minutes) leave the cluster almost no idle time, making it effectively a permanently resident resource. You generally should not run Pipes on a large cluster, which would waste resources. Conversely, if the Kafka data volume is small, there is no need to allocate a separate cluster for each Pipe. In that case, a group of Pipes can share a single resident cluster to save resources.
+
+The Pipe DDL is as follows:
+
+1. Remove the `LIMIT 10` from the `select ... from read_kafka(...)` clause.
+2. Change the group id from `test` to the production name you intend to use, such as `sub2cz`.
+3. (Optional) Use the `RESET_KAFKA_GROUP_OFFSETS` parameter to specify the starting time point for reading Kafka (otherwise it reads from `latest` per the `kafka.auto.offset.reset` setting). The value must be in millisecond epoch format.
+
+```sql
+CREATE PIPE pipe_ods_commit_log 
+    VIRTUAL_CLUSTER = 'pipe_ods_commit_log' 
+    BATCH_INTERVAL_IN_SECONDS = '60'
+    RESET_KAFKA_GROUP_OFFSETS = '1740931200000' -- epoch in millis of 2025-03-03 00:00:00
+AS COPY INTO ods_commit_log FROM (
+SELECT
+    parse_json(j['event']::string) as event,
+    parse_json(parse_json(j['event']::string)['statements']::string) as statements,
+    j['op_type']::string as op_type,
+    j['datasource_id']::string as datasource_id,
+    j['database_name']::string as database_name,
+    j['schema_name']::string as schema_name,
+    j['table_name']::string as table_name,
+    timestamp_millis(j['event_ts']::bigint) as event_ts,
+    j['event_seq']::string as event_seq,
+    timestamp_millis(j['server_ts']::bigint) as server_ts,
+    j['server_seq']::bigint as server_seq,
+    `timestamp` as __kafka_timestamp__
+FROM (
+    SELECT `timestamp`, parse_json(value::string) as j
+    FROM read_kafka(
+    'kafka-bootstrap-1:9092,kafka-bootstrap-2:9092,kafka-bootstrap-3:9092', -- bootstrap
+    'topic_name',   -- topic
+    '',             -- reserved
+    'sub2cz',       -- kafka group id, for keeping read position
+    '', '', '', '', -- reserved
+    'raw',          -- key format, can only be raw
+    'raw',          -- value format, can only be raw
+    0,
+    MAP(
+        'kafka.security.protocol','PLAINTEXT',
+        'cz.kafka.fetch.retry.enable','true', 
+        'cz.kafka.fetch.retry.times','20',
+        'cz.kafka.fetch.retry.intervalMs','2000'
+    )
+)));
+```
+
+> Here we added three parameters to `read_kafka`: `cz.kafka.fetch.retry.enable`, `cz.kafka.fetch.retry.times`, and `cz.kafka.fetch.retry.intervalMs`. These specify the retry strategy when the function encounters failures reading from Kafka, which helps reduce alerts and manual interventions in production.
+
+After the Pipe is created, it starts running automatically. In Studio, click "Compute" → "Cluster" → "PIPE\_ODS\_COMMIT\_LOG" → "Jobs" to quickly check the execution status of jobs submitted by the Pipe.
+
+When multiple Pipes share a VCluster, you can also use `query_tag` in Studio's "Compute" → "Job History" page or the `job_history` table in InformationSchema to quickly query the job execution of a specific Pipe. For jobs submitted by Pipes, the `query_tag` follows the format `pipe.<workspace_name>.<schema_name>.<pipe_name>`:
+
+```sql
+select query_tag, job_id, status, start_time, end_time, execution_time, input_bytes, output_bytes, rows_produced
+from information_schema.job_history
+where pt_date>='yyyy-MM-dd' -- eg. 2025-03-03
+and query_tag = 'pipe.<workspace_name>.<schema_name>.<pipe_name>' -- eg. 'pipe.quick_start.public.pipe_ods_commit_log'
+order by start_time desc;
+```
+
+# Large-Scale Data and Resource Optimization
+
+Optimizing a Kafka Pipe for stable production operation means using the minimum resource configuration (VCluster) within the required freshness cycle (`BATCH_INTERVAL_IN_SECONDS`) to complete the computation and persistence of incoming data. In production, maintain some computing capacity reserve to ensure the Pipe can catch up when data volume fluctuates, cluster upgrades cause job failovers, etc. We typically recommend maintaining double the redundancy. In plain terms with our example: you need to be able to process all the data that flowed in from Kafka over 1 minute within roughly 30 seconds.
+
+The default Pipe settings work well in most cases. However, if the ingested Kafka data volume is particularly large, adjust parameters based on actual conditions, as indicated by the red annotations in the figure below:
+
+![](/.topwrite/assets/image_1760681949544.png)
+
+To check whether the current Pipe is experiencing backlog, run `desc pipe extended` multiple times and observe whether `timeLag` in the `pipe_latency` row (the gap between the Kafka offset and current time, in milliseconds) is continuously increasing. With 60-second data freshness and double computing redundancy, `timeLag` should fluctuate between 0–90 seconds (without any redundancy, it should fluctuate between 0–120 seconds). If `timeLag` exceeds the upper limit and continues rising over several cycles, the Pipe is building up a backlog.
+
+## Confirm Kafka Message Peak Value (Increase `BATCH_SIZE_PER_KAFKA_PARTITION`)
+
+To prevent the Pipe from reading excessive data at startup and creating oversized jobs, Pipe limits the number of records read from each partition per batch. This is controlled by `BATCH_SIZE_PER_KAFKA_PARTITION`, which defaults to 50,000. When the peak message volume per partition per cycle exceeds this value, manually specify this parameter when creating the Pipe. We typically recommend setting it to 2× the peak value. If the Pipe is already running, you can dynamically adjust it with `alter pipe <pipe_name> set BATCH_SIZE_PER_KAFKA_PARTITION=<event_number>;`.
+
+As shown in the figure, even though the Pipe's per-minute jobs complete in under 30 seconds, `desc pipe extended` may show `timeLag` continuously increasing.
+
+![](/.topwrite/assets/image_1760681973997.png)
+
+Click on the job to enter the job details page. You can see this is a Kafka topic with 10 partitions. Even with `BATCH_SIZE_PER_KAFKA_PARTITION` set to 100,000, not all data from one cycle is read — you need to keep increasing it.
+
+![](/.topwrite/assets/image_1760681981294.png)
+
+Ultimately, by continuously increasing this value and observing `timeLag` (in production you should ideally know the per-partition peak value in advance and set it directly), you determine that at 500,000 the Pipe's consumption speed is fast enough to keep up with Kafka.
+
+> Unlike a direct `select ... read_kafka(...)`, the `copy into` jobs generated by Pipe show the offset information recorded in the Kafka group id managed by the Pipe. This makes it easy to identify how many partitions the Kafka topic has and the actual offset values of each partition.
+
+## Adjusting VCluster Size
+
+By default, the `copy into` jobs started by Pipe have a task count equal to the Kafka topic partition count — each task reads one partition. When the VCluster Size is small, job tasks may need multiple rounds to complete, significantly extending execution time.
+
+Therefore, when you want jobs to finish quickly, ensure the VCluster's allowed task concurrency >= partition count, so all tasks complete in a single round.
+
+In the figure below, a Pipe job using a 128CRU VCluster (1024 cores) consumes a Kafka topic with 1,200 partitions. It runs in two rounds: the first round with 1,024 tasks, and the remaining 176 tasks in the second round, significantly extending execution time. In this case, setting the VCluster to 150CRU (1,200 cores) is appropriate.
+
+![](/.topwrite/assets/image_1760681994525.png)
+
+## Splitting Kafka Data to Utilize More Compute Resources (Setting `COPY_JOB_HINT`)
+
+When Kafka data volume is large but partition count is small, or the computation logic before writing to the table is complex enough that a single task cannot complete within the cycle requirement, consider setting `COPY_JOB_HINT` to further split tasks and combine with more compute resources for acceleration.
+
+`COPY_JOB_HINT` is a composite parameter expressed in JSON format. To manually split Kafka data, use the following two keys together:
+
+1. `"cz.sql.split.kafka.strategy":"size"` — The default is `"simple"`, meaning one task per partition. Change it to `"size"` to split tasks by record count.
+2. `"cz.mapper.kafka.message.size":"200000"` — Specifies how many events constitute one task. The default is 1,000,000 and takes effect when `"cz.sql.split.kafka.strategy"` is `"size"`.
+
+You can adjust the Pipe's task splitting strategy at any time with:
+`alter pipe pipe_ods_commit_log set COPY_JOB_HINT = '{"cz.sql.split.kafka.strategy":"size","cz.mapper.kafka.message.size":"200000"}';`
+
+> `COPY_JOB_HINT` is a composite JSON parameter. When setting it via `alter pipe`, it is an overwrite operation — be careful not to accidentally lose previously set keys. You can retrieve the complete current settings from the `copy_job_hint` field in the properties row of `desc pipe extended` and modify from there.
+
+After adjusting parameters, update the corresponding Pipe DDL to prevent parameter loss from rebuilds or migrations later.
+
+```SQL
+CREATE VCLUSTER pipe_ods_commit_log 
+    VCLUSTER_TYPE = GENERAL 
+    VCLUSTER_SIZE = 150
+;
+
+CREATE PIPE pipe_ods_commit_log
+    VIRTUAL_CLUSTER = 'PIPE_ODS_COMMIT_LOG' 
+    BATCH_INTERVAL_IN_SECONDS = '60'
+    BATCH_SIZE_PER_KAFKA_PARTITION = 2000000
+    RESET_KAFKA_GROUP_OFFSETS = '1740931200000' -- epoch in millis of 2025-03-03 00:00:00
+    COPY_JOB_HINT = '{"cz.sql.split.kafka.strategy":"size","cz.mapper.kafka.message.size":"200000"}' -- to accelerate load
+AS COPY INTO ods_commit_log FROM 
+...
+```
+
+# Modifying Pipe SQL Logic
+
+Pipe allows relatively complex computations between `copy into` and `read_kafka`. When computation logic changes or the target table schema changes (a typical scenario is adding columns), you need to modify the existing Pipe definition.
+
+In short, modifying the Pipe DDL means deleting and rebuilding the Pipe while keeping the `read_kafka` parameters unchanged.
+
+For example, if you need to modify `pipe_ods_commit_log` to remove the `statements` content from the `event` field in the target table (because it is redundantly stored in the `statements` field), proceed as follows:
+
+1. Delete the currently running Pipe
+
+```SQL
+drop pipe pipe_ods_commit_log;
+```
+
+> Dropping or altering an existing Pipe may be blocked by currently running jobs within the Pipe until those jobs complete, which may take some time.
+
+2. Rebuild the Pipe (note: remove the `RESET_KAFKA_GROUP_OFFSETS` parameter)
+
+```sql
+CREATE PIPE pipe_ods_commit_log 
+    VIRTUAL_CLUSTER = 'pipe_ods_commit_log' 
+    BATCH_INTERVAL_IN_SECONDS = '60'
+    BATCH_SIZE_PER_KAFKA_PARTITION = 2000000
+    -- RESET_KAFKA_GROUP_OFFSETS = '1740931200000' -- epoch in millis of 2025-03-03 00:00:00
+    COPY_JOB_HINT = '{"cz.sql.split.kafka.strategy":"size","cz.mapper.kafka.message.size":"200000"}' -- to accelerate load
+AS COPY INTO ods_commit_log FROM (
+SELECT
+    remove_json(parse_json(j['event']::string), '$.statements') as event,
+    parse_json(parse_json(j['event']::string)['statements']::string) as statements,
+    j['op_type']::string as op_type,
+    j['datasource_id']::string as datasource_id,
+    j['database_name']::string as database_name,
+    j['schema_name']::string as schema_name,
+    j['table_name']::string as table_name,
+    timestamp_millis(j['event_ts']::bigint) as event_ts,
+    j['event_seq']::string as event_seq,
+    timestamp_millis(j['server_ts']::bigint) as server_ts,
+    j['server_seq']::bigint as server_seq,
+    `timestamp` as __kafka_timestamp__
+FROM (
+    SELECT `timestamp`, parse_json(value::string) as j
+    FROM read_kafka(
+    'kafka-bootstrap-1:9092,kafka-bootstrap-2:9092,kafka-bootstrap-3:9092', -- bootstrap
+    'topic_name',   -- topic
+    '',             -- reserved
+    'sub2cz',       -- kafka group id, for keeping read position
+    '', '', '', '', -- reserved
+    'raw',          -- key format, can only be raw
+    'raw',          -- value format, can only be raw
+    0,
+    MAP(
+        'kafka.security.protocol','PLAINTEXT',
+        'cz.kafka.fetch.retry.enable','true', 
+        'cz.kafka.fetch.retry.times','20',
+        'cz.kafka.fetch.retry.intervalMs','2000'
+    )
+)));
+```
+
+> Pipe uses Kafka groups to record read offsets. As long as you use the same Kafka cluster, topic, and group id, rebuilding the Pipe will not lose the offset — you get a "resume from checkpoint" effect.
+
+However, `RESET_KAFKA_GROUP_OFFSETS` forcibly overwrites the offset recorded in the group id and should be used with caution.
+
+# Pipe Production Monitoring and Alerting
+
+## Monitoring the Pipe Itself
+
+Studio's data quality module provides latency monitoring capabilities for Pipes.
+
+![](/.topwrite/assets/image_1760682100684.png)
+
+## Monitoring Output Tables
+
+Earlier, we added the `__kafka_timestamp__` field to the Pipe's output table `ods_commit_log`. You can use this field combined with Studio's data quality features for end-to-end latency monitoring, expressed as:
+
+```SQL
+select DATEDIFF(second, max(`__kafka_timestamp__`), now())from ods_commit_logwhere pt_date='${today}';
+```
+
+In Studio, click "Data" → "Data Quality" → "Quality Rules" → "Create Rule".
+
+![](/.topwrite/assets/image_1760682111260.png)
+
+Configure the parameter `today` with value `$[yyyyMMdd]`, select Custom SQL, and fill in the SQL query above. Set other required parameters as shown in the figure and click Save.
+
+In Studio, click "Operations" → "Monitoring Alerts" → "Monitoring Rules" → "Create Rule".
+
+Fill in the required fields as shown in the figure and save.
+
+![](/.topwrite/assets/image_1760682120092.png)

package/bin/skills/lakehouse-doc-en/references/pipe-kafka-table-stream.md

@@ -0,0 +1,182 @@
+# Using Table Stream and Pipe to Import Kafka Data into Lakehouse
+
+## 1. Background Introduction
+
+In the field of big data processing, efficiently importing streaming data from Kafka into a Lakehouse (data lake warehouse) is a common requirement. CloudTech provides powerful Table Stream and Pipe functionalities that simplify and enhance this process. This article will detail how to use Table Stream and Pipe to import Kafka data into a Lakehouse, including the complete process of creating a Kafka external table and a Kafka Table Stream.
+
+## 2. Operational Steps
+
+### Creating a Kafka External Table
+
+Before using Table Stream and Pipe, we need to create an external table integrated with Kafka to access data in Kafka.
+
+```sql
+CREATE STORAGE CONNECTION pipe_kafka
+ TYPE kafka 
+BOOTSTRAP_SERVERS = ['47.00.08.62:9092'] 
+SECURITY_PROTOCOL = 'PLAINTEXT';
+
+CREATE EXTERNAL TABLE external_table_kafka (   
+ key_column binary,   
+ value_column binary NOT NULL)
+USING kafka
+OPTIONS (   'group_id' = 'external_table_lh',    'topics' = 'my_topic')
+CONNECTION pipe_kafka;
+```
+
+### Creating a Table Stream
+
+Create a Table Stream on the Kafka external table to capture real-time data changes in Kafka.
+
+```sql
+CREATE TABLE STREAM kafka_table_stream_pipe1 
+ON TABLE external_table_kafka
+WITH PROPERTIES (
+    'table_stream_mode' = 'append_only'
+);
+```
+
+- `kafka_table_stream_pipe1`: Name of the Table Stream.
+- `ON TABLE external_table_kafka`: Specifies that the Table Stream is created based on the previously created Kafka external table.
+- `table_stream_mode='append_only'`: Sets the mode of the Table Stream to append-only, meaning it will only capture newly added data rows.
+
+After creation, you can verify the data in the Table Stream with the following query:
+
+```sql
+SELECT CAST(value AS STRING) FROM kafka_table_stream_pipe1;
+```
+
+This query converts the `value` field in the Table Stream to a string type and returns it for subsequent processing.
+
+### Creating a Target Table
+
+Next, create a target table to store data imported from Kafka.
+
+```sql
+CREATE TABLE kafak_sink_table_1 (
+    a TIMESTAMP,
+    b STRING
+);
+```
+
+- `kafak_sink_table_1`: Name of the target table.
+- `a TIMESTAMP`: First field for storing timestamp data.
+- `b STRING`: Second field for storing string data.
+
+### Creating a Pipe
+
+Finally, use a Pipe to continuously import data from the Table Stream into the target table.
+
+```sql
+CREATE PIPE kafka_pipe_stream
+VIRTUAL_CLUSTER = 'test_alter'
+AS
+COPY INTO kafak_sink_table_1
+FROM (
+    SELECT CURRENT_TIMESTAMP(), CAST(value AS STRING) FROM kafka_table_stream_pipe1
+);
+```
+
+- `kafka_pipe_stream`: Name of the Pipe.
+- `VIRTUAL_CLUSTER = 'test_alter'`: Specifies the virtual cluster to use.
+- `COPY INTO kafak_sink_table_1`: Copies data into the target table `kafak_sink_table_1`.
+- `SELECT CURRENT_TIMESTAMP(), CAST(value AS STRING) FROM kafka_table_stream_pipe1`: Selects data from the Table Stream, using the current timestamp and the converted `value` field as the two columns for the target table.
+
+Other Configurable Properties:
+- `INITIAL_DELAY_IN_SECONDS`: Initial job scheduling delay (optional, default 0 seconds)
+- `BATCH_INTERVAL_IN_SECONDS`: (Optional) Sets the batch processing interval, default 60 seconds.
+- `BATCH_SIZE_PER_KAFKA_PARTITION`: (Optional) Sets the batch size per Kafka partition, default 500,000 records.
+- `MAX_SKIP_BATCH_COUNT_ON_ERROR`: (Optional) Sets the maximum number of batches to skip on error, default 30.
+- `RESET_KAFKA_GROUP_OFFSETS`: (Optional) Sets the initial offset for Kafka when starting the pipe. Cannot be modified. Possible values: `latest`, `earliest`, `none`, `valid`, `${TIMESTAMP_MILLISECONDS}`
+    - `none`: Default, no action.
+    - `valid`: Checks if the current offset in the group is expired and resets expired partitions to the current earliest.
+    - `earliest`: Resets to the current earliest.
+    - `latest`: Resets to the current latest.
+    - `${TIMESTAMP_MILLISECONDS}`: Resets to the offset corresponding to the millisecond timestamp, e.g., '1737789688000' (2025-01-25 15:21:28).
+
+## 3. Verifying Results
+
+You can verify whether the data has been successfully imported by querying the target table:
+
+```sql
+SELECT * FROM kafak_sink_table_1;
+```
+
+Additionally, check the running status of the Pipe to ensure it is functioning properly:
+
+```sql
+SHOW PIPES;
+```
+
+This command lists all created Pipes and their status information, including whether they are running and the last run time.
+
+## 4. Status Monitoring and Management
+
+### Checking Kafka Consumption Latency
+
+Use the `DESC PIPE` command. For example, the JSON string in `pipe_latency` below:
+- `lastConsumeTimestamp`: The last consumed offset.
+- `offsetLag`: The backlog of Kafka data.
+- `timeLag`: Consumption latency, calculated as the current time minus the last consumed offset. If Kafka consumption is abnormal, the value is -1.
+
+```sql
+DESC PIPE EXTENDED kafka_pipe_stream
++--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|     info_name      |                                                                                                               info_value                                                            |
++--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| name               | kafka_pipe_stream                                                                                                                                                                   |
+| creator            | UAT_TEST                                                                                                                                                                            |
+| created_time       | 2025-03-05 10:40:55.405                                                                                                                                                             |
+| last_modified_time | 2025-03-05 10:40:55.405                                                                                                                                                             |
+| comment            |                                                                                                                                                                                     |
+| properties         | ((virtual_cluster,test_alter))                                                                                                                                                      |
+| copy_statement     | COPY INTO TABLE qingyun.pipe_schema.kafak_sink_table_1 FROM (SELECT `current_timestamp`() AS ```current_timestamp``()`, CAST(kafka_table_stream_pipe1.`value` AS string) AS `value` |
+| pipe_status        | RUNNING                                                                                                                                                                             |
+| output_name        | xxxxxxx.pipe_schema.kafak_sink_table_1                                                                                                                                              |
+| input_name         | kafka_table_stream:xxxxxxx.pipe_schema.kafka_table_stream_pipe1                                                                                                                     |
+| invalid_reason     |                                                                                                                                                                                     |
+| pipe_latency       | {"kafka":{"lags":{"0":0,"1":0,"2":0,"3":0},"lastConsumeTimestamp":-1,"offsetLag":0,"timeLag":-1}}                                                                                   |
++--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+### Viewing Pipe Execution History
+
+Since each Pipe execution is a copy operation, you can view all operations in the job history. Use the `query_tag` in the [Job History](web-job-history.md) to filter, as all Pipe copy jobs are tagged in the format `pipe.``workspace_name``.schema_name.pipe_name`, making it easy to track and manage.
+
+### Stopping and Starting a Pipe
+
+- Pause a Pipe:
+  ```sql
+  ALTER PIPE pipe_name SET PIPE_EXECUTION_PAUSED = true;
+  ```
+
+- Resume a Pipe:
+  ```sql
+  ALTER PIPE pipe_name SET PIPE_EXECUTION_PAUSED = false;
+  ```
+
+### Modifying Pipe Properties
+
+You can modify Pipe properties, but only one property at a time. If multiple properties need to be modified, execute the `ALTER` command multiple times. Below are the modifiable properties and their syntax:
+
+```sql
+ALTER PIPE pipe_name SET 
+   [VIRTUAL_CLUSTER = 'virtual_cluster_name'],
+    [BATCH_INTERVAL_IN_SECONDS=''],
+   [ BATCH_SIZE_PER_KAFKA_PARTITION=''],
+    [MAX_SKIP_BATCH_COUNT_ON_ERROR=''],
+    [COPY_JOB_HINT='']
+```
+
+Examples:
+```sql
+-- Modify compute cluster
+ALTER PIPE pipe_name SET VIRTUAL_CLUSTER = 'default';
+
+-- Set COPY_JOB_HINT
+ALTER PIPE pipe_name SET copy_hints='{"cz.mapper.kafka.message.size": "2000000"}';
+```
+
+**Note**
+- Modifying the COPY statement logic is not supported. If needed, delete the Pipe and recreate it.
+- When modifying the `COPY_JOB_HINT` of a Pipe, the new settings will overwrite existing hints. If your Pipe already has hints (e.g., `{"cz.sql.split.kafka.strategy":"size"}`), you must set all required hints together when adding new ones; otherwise, existing hints will be overwritten. Separate multiple parameters with commas.