@clickzetta/cz-cli-darwin-x64 0.3.91 → 0.3.93
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/cz-cli +0 -0
- package/bin/skills/clickzetta-ai-function/SKILL.md +109 -0
- package/bin/skills/clickzetta-ai-function/eval_cases.jsonl +4 -0
- package/bin/skills/clickzetta-ai-function/references/ai-function-ddl.md +106 -0
- package/bin/skills/clickzetta-batch-sync-pipeline/SKILL.md +124 -124
- package/bin/skills/clickzetta-batch-sync-pipeline/eval_cases.jsonl +5 -5
- package/bin/skills/clickzetta-bi-connect/SKILL.md +79 -78
- package/bin/skills/clickzetta-bi-connect/references/bi-tools.md +56 -56
- package/bin/skills/clickzetta-cdc-sync-pipeline/SKILL.md +386 -382
- package/bin/skills/clickzetta-cdc-sync-pipeline/eval_cases.jsonl +5 -5
- package/bin/skills/clickzetta-data-ingest-pipeline/SKILL.md +73 -212
- package/bin/skills/clickzetta-data-science/SKILL.md +57 -56
- package/bin/skills/clickzetta-data-science/references/bitmap-profile.md +38 -38
- package/bin/skills/clickzetta-data-science/references/data-patterns.md +16 -16
- package/bin/skills/clickzetta-data-science/references/setup.md +28 -28
- package/bin/skills/clickzetta-data-science/references/stats-functions.md +44 -44
- package/bin/skills/clickzetta-data-science/references/write-and-infer.md +22 -22
- package/bin/skills/clickzetta-data-science/references/zettapark-api.md +32 -32
- package/bin/skills/clickzetta-dw-modeling/SKILL.md +1 -1
- package/bin/skills/clickzetta-external-function/SKILL.md +51 -109
- package/bin/skills/clickzetta-external-function/eval_cases.jsonl +4 -4
- package/bin/skills/clickzetta-external-function/references/external-function-ddl.md +39 -77
- package/bin/skills/clickzetta-java-sdk/SKILL.md +49 -48
- package/bin/skills/clickzetta-java-sdk/eval_cases.jsonl +12 -12
- package/bin/skills/clickzetta-java-sdk/references/bulkload.md +34 -34
- package/bin/skills/clickzetta-java-sdk/references/realtime.md +44 -44
- package/bin/skills/clickzetta-kafka-ingest-pipeline/SKILL.md +273 -507
- package/bin/skills/clickzetta-kafka-ingest-pipeline/references/kafka-pipe-syntax.md +197 -231
- package/bin/skills/clickzetta-oss-ingest-pipeline/SKILL.md +231 -304
- package/bin/skills/clickzetta-realtime-sync-pipeline/SKILL.md +180 -179
- package/bin/skills/clickzetta-realtime-sync-pipeline/eval_cases.jsonl +5 -5
- package/bin/skills/clickzetta-semantic-view/SKILL.md +74 -72
- package/bin/skills/clickzetta-semantic-view/eval_cases.jsonl +12 -12
- package/bin/skills/clickzetta-semantic-view/references/semantic-view-reference.md +75 -75
- package/bin/skills/clickzetta-sql-migration/SKILL.md +128 -0
- package/bin/skills/clickzetta-sql-migration/eval_cases.jsonl +10 -0
- package/bin/skills/clickzetta-sql-migration/references/ddl-reference.md +350 -0
- package/bin/skills/clickzetta-sql-migration/references/dml-differences.md +192 -0
- package/bin/skills/clickzetta-sql-migration/references/dml-reference.md +279 -0
- package/bin/skills/{clickzetta-sql-syntax-guide → clickzetta-sql-migration}/references/dql-reference.md +128 -128
- package/bin/skills/clickzetta-sql-migration/references/function-mapping.md +194 -0
- package/bin/skills/clickzetta-sql-migration/references/functions-reference.md +372 -0
- package/bin/skills/clickzetta-sql-migration/references/implicit-type-conversion.md +143 -0
- package/bin/skills/clickzetta-sql-migration/references/migration-databricks.md +260 -0
- package/bin/skills/{clickzetta-sql-syntax-guide → clickzetta-sql-migration}/references/migration-snowflake.md +112 -112
- package/bin/skills/clickzetta-sql-migration/references/vs-snowflake.md +346 -0
- package/bin/skills/clickzetta-sql-migration/references/vs-spark.md +229 -0
- package/bin/skills/clickzetta-studio-task-manager/SKILL.md +326 -329
- package/bin/skills/clickzetta-table-lineage/SKILL.md +57 -55
- package/bin/skills/clickzetta-table-lineage/eval_cases.jsonl +1 -1
- package/bin/skills/clickzetta-table-lineage/references/normalize_func.sql +5 -5
- package/bin/skills/clickzetta-table-lineage/references/table_cost.sql +6 -6
- package/bin/skills/clickzetta-table-lineage/references/table_relation.sql +2 -2
- package/bin/skills/clickzetta-volume-manager/SKILL.md +186 -100
- package/bin/skills/clickzetta-volume-manager/references/volume-ddl.md +153 -52
- package/package.json +1 -1
- package/bin/skills/clickzetta-dynamic-table/best-practices/scheduling-guide.md +0 -135
- package/bin/skills/clickzetta-dynamic-table/dt-creator/references/dt-declaration-strategy.md +0 -185
- package/bin/skills/clickzetta-dynamic-table/dt-creator/references/refresh-history-guide.md +0 -260
- package/bin/skills/clickzetta-dynamic-table/dynamic-table-alter/SKILL.md +0 -191
- package/bin/skills/clickzetta-sql-syntax-guide/SKILL.md +0 -249
- package/bin/skills/clickzetta-sql-syntax-guide/eval_cases.jsonl +0 -3
- package/bin/skills/clickzetta-sql-syntax-guide/references/ddl-reference.md +0 -350
- package/bin/skills/clickzetta-sql-syntax-guide/references/dml-reference.md +0 -279
- package/bin/skills/clickzetta-sql-syntax-guide/references/functions-reference.md +0 -372
- package/bin/skills/clickzetta-sql-syntax-guide/references/migration-databricks.md +0 -260
- package/bin/skills/clickzetta-sql-syntax-guide/references/vs-snowflake.md +0 -346
- package/bin/skills/clickzetta-sql-syntax-guide/references/vs-spark.md +0 -229
- /package/bin/skills/{clickzetta-sql-syntax-guide → clickzetta-sql-migration}/LICENSE +0 -0
|
@@ -1,11 +1,11 @@
|
|
|
1
|
-
# RealtimeStream
|
|
1
|
+
# RealtimeStream Real-Time Write Reference
|
|
2
2
|
|
|
3
|
-
>
|
|
3
|
+
> Best for: Kafka consumption and writes, high-frequency real-time ingestion with second-level queryability, and CDC writes to primary-key tables.
|
|
4
4
|
|
|
5
|
-
## Maven
|
|
5
|
+
## Maven Dependency
|
|
6
6
|
|
|
7
7
|
```xml
|
|
8
|
-
<!--
|
|
8
|
+
<!-- See https://central.sonatype.com/artifact/com.clickzetta/clickzetta-java for the latest version. -->
|
|
9
9
|
<dependency>
|
|
10
10
|
<groupId>com.clickzetta</groupId>
|
|
11
11
|
<artifactId>clickzetta-java</artifactId>
|
|
@@ -18,23 +18,23 @@
|
|
|
18
18
|
</dependency>
|
|
19
19
|
```
|
|
20
20
|
|
|
21
|
-
##
|
|
21
|
+
## Usage Limits
|
|
22
22
|
|
|
23
|
-
-
|
|
24
|
-
-
|
|
25
|
-
-
|
|
23
|
+
- Real-time written data can be queried within seconds.
|
|
24
|
+
- Table Streams and Dynamic Tables need about **1 minute** before they can see the written data.
|
|
25
|
+
- When the table schema changes, stop the task and restart it about **90 minutes** after the schema change.
|
|
26
26
|
|
|
27
|
-
##
|
|
27
|
+
## Operation Modes
|
|
28
28
|
|
|
29
|
-
|
|
|
29
|
+
| Mode | Target table | Available operators |
|
|
30
30
|
|---|---|---|
|
|
31
|
-
| `RealTimeOperate.APPEND_ONLY` |
|
|
32
|
-
| `RealTimeOperate.CDC` |
|
|
31
|
+
| `RealTimeOperate.APPEND_ONLY` | Regular table | `Stream.Operator.INSERT` |
|
|
32
|
+
| `RealTimeOperate.CDC` | Primary-key table | `Stream.Operator.UPSERT`, `Stream.Operator.DELETE_IGNORE` |
|
|
33
33
|
|
|
34
|
-
##
|
|
34
|
+
## Write to a Regular Table: APPEND_ONLY
|
|
35
35
|
|
|
36
36
|
```java
|
|
37
|
-
//
|
|
37
|
+
// Recommended: explicit parameters. Supported in 2.0.0+ and does not depend on URL parsing.
|
|
38
38
|
ClickZettaClient client = ClickZettaClient.newBuilder()
|
|
39
39
|
.service("cn-shanghai-alicloud.api.clickzetta.com")
|
|
40
40
|
.instance("your_instance")
|
|
@@ -53,17 +53,17 @@ RealtimeStream stream = client.newRealtimeStreamBuilder()
|
|
|
53
53
|
.table("events")
|
|
54
54
|
.build();
|
|
55
55
|
|
|
56
|
-
//
|
|
56
|
+
// RealtimeStream uses column names, not indexes.
|
|
57
57
|
Row row = stream.createRow(Stream.Operator.INSERT);
|
|
58
58
|
row.setValue("id", 1);
|
|
59
59
|
row.setValue("event", "{\"type\":\"click\"}");
|
|
60
60
|
stream.apply(row);
|
|
61
61
|
```
|
|
62
62
|
|
|
63
|
-
##
|
|
63
|
+
## Write to a Primary-Key Table: CDC Mode
|
|
64
64
|
|
|
65
65
|
```java
|
|
66
|
-
//
|
|
66
|
+
// Create a primary-key table.
|
|
67
67
|
// CREATE TABLE orders (`txid` STRING PRIMARY KEY, `amount` DOUBLE, `status` STRING);
|
|
68
68
|
|
|
69
69
|
RealtimeStream stream = client.newRealtimeStreamBuilder()
|
|
@@ -73,22 +73,22 @@ RealtimeStream stream = client.newRealtimeStreamBuilder()
|
|
|
73
73
|
.table("orders")
|
|
74
74
|
.build();
|
|
75
75
|
|
|
76
|
-
// UPSERT
|
|
76
|
+
// UPSERT: update an existing row or insert a new row.
|
|
77
77
|
Row row = stream.createRow(Stream.Operator.UPSERT);
|
|
78
78
|
row.setValue("txid", "order-001");
|
|
79
79
|
row.setValue("amount", 299.99);
|
|
80
80
|
row.setValue("status", "paid");
|
|
81
81
|
stream.apply(row);
|
|
82
82
|
|
|
83
|
-
// DELETE_IGNORE
|
|
83
|
+
// DELETE_IGNORE: delete the row and ignore the operation if the target row does not exist.
|
|
84
84
|
Row delRow = stream.createRow(Stream.Operator.DELETE_IGNORE);
|
|
85
85
|
delRow.setValue("txid", "order-001");
|
|
86
86
|
stream.apply(delRow);
|
|
87
87
|
```
|
|
88
88
|
|
|
89
|
-
##
|
|
89
|
+
## Complete Example: Kafka to Lakehouse
|
|
90
90
|
|
|
91
|
-
### KafkaReader
|
|
91
|
+
### KafkaReader Class
|
|
92
92
|
|
|
93
93
|
```java
|
|
94
94
|
import org.apache.kafka.clients.consumer.ConsumerConfig;
|
|
@@ -119,7 +119,7 @@ public class KafkaReader {
|
|
|
119
119
|
}
|
|
120
120
|
```
|
|
121
121
|
|
|
122
|
-
### Kafka2Lakehouse
|
|
122
|
+
### Kafka2Lakehouse Main Class
|
|
123
123
|
|
|
124
124
|
```java
|
|
125
125
|
import com.clickzetta.client.ClickZettaClient;
|
|
@@ -180,33 +180,33 @@ public class Kafka2Lakehouse {
|
|
|
180
180
|
}
|
|
181
181
|
```
|
|
182
182
|
|
|
183
|
-
##
|
|
183
|
+
## Key APIs
|
|
184
184
|
|
|
185
|
-
| API |
|
|
185
|
+
| API | Description |
|
|
186
186
|
|---|---|
|
|
187
|
-
| `realtimeStream.createRow(Stream.Operator.INSERT)` |
|
|
188
|
-
| `realtimeStream.createRow(Stream.Operator.UPSERT)` |
|
|
189
|
-
| `realtimeStream.createRow(Stream.Operator.DELETE_IGNORE)` |
|
|
190
|
-
| `row.setValue(String columnName, Object value)` |
|
|
191
|
-
| `realtimeStream.apply(row)` |
|
|
192
|
-
| `Options.builder().withMutationBufferLinesNum(n)` |
|
|
187
|
+
| `realtimeStream.createRow(Stream.Operator.INSERT)` | Create an insert row for a regular table. |
|
|
188
|
+
| `realtimeStream.createRow(Stream.Operator.UPSERT)` | Create an upsert row for a primary-key table. |
|
|
189
|
+
| `realtimeStream.createRow(Stream.Operator.DELETE_IGNORE)` | Create a delete row for a primary-key table. |
|
|
190
|
+
| `row.setValue(String columnName, Object value)` | Set a value by column name, not by index. |
|
|
191
|
+
| `realtimeStream.apply(row)` | Send the row to the server. |
|
|
192
|
+
| `Options.builder().withMutationBufferLinesNum(n)` | Set the number of buffered rows. The default is 10. |
|
|
193
193
|
|
|
194
|
-
## BulkloadStream vs RealtimeStream
|
|
194
|
+
## BulkloadStream vs RealtimeStream
|
|
195
195
|
|
|
196
|
-
|
|
|
196
|
+
| Dimension | BulkloadStream | RealtimeStream |
|
|
197
197
|
|---|---|---|
|
|
198
|
-
|
|
|
199
|
-
| URL
|
|
200
|
-
| createRow
|
|
201
|
-
|
|
|
202
|
-
|
|
|
203
|
-
|
|
|
198
|
+
| Column value setter | `setValue(int index, value)` | `setValue(String name, value)` |
|
|
199
|
+
| URL parameter | `virtualcluster=` | `vcluster=` |
|
|
200
|
+
| `createRow` argument | No argument | `Stream.Operator.INSERT/UPSERT/DELETE_IGNORE` |
|
|
201
|
+
| Suitable write frequency | Low frequency, >=5 minutes per batch | High frequency, second-level writes |
|
|
202
|
+
| Data visibility latency | Visible after `close()` | Visible after about 1 minute |
|
|
203
|
+
| Primary-key table support | Not supported | Supported in CDC mode |
|
|
204
204
|
|
|
205
|
-
##
|
|
205
|
+
## FAQ
|
|
206
206
|
|
|
207
|
-
|
|
|
207
|
+
| Issue | Cause | Solution |
|
|
208
208
|
|---|---|---|
|
|
209
|
-
|
|
|
210
|
-
|
|
|
211
|
-
|
|
|
212
|
-
|
|
|
209
|
+
| Connection fails | Wrong URL parameter name | RealtimeStream uses `vcluster=`, not `virtualcluster=`. |
|
|
210
|
+
| Column name not found | Column name is misspelled | Column names are case-sensitive and must match the table DDL. |
|
|
211
|
+
| Writes fail after a schema change | The old Stream instance cached the old schema | Stop the task and restart it about 90 minutes after the schema change. |
|
|
212
|
+
| Dynamic Table cannot see the data | Real-time writes have about 1 minute of confirmation latency | Query again after about 1 minute. |
|