@clickzetta/cz-cli-darwin-x64 0.3.87 → 0.3.88

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (27) hide show
  1. package/bin/cz-cli +0 -0
  2. package/bin/skills/clickzetta-dynamic-table/SKILL.md +169 -169
  3. package/bin/skills/clickzetta-dynamic-table/best-practices/dimension-table-join-guide.md +126 -126
  4. package/bin/skills/clickzetta-dynamic-table/best-practices/medallion-and-stream-patterns.md +25 -25
  5. package/bin/skills/clickzetta-dynamic-table/best-practices/non-partitioned-merge-into-warning.md +48 -48
  6. package/bin/skills/clickzetta-dynamic-table/best-practices/performance-optimization.md +51 -51
  7. package/bin/skills/clickzetta-dynamic-table/best-practices/scheduling-guide.md +59 -59
  8. package/bin/skills/clickzetta-dynamic-table/dt-creator/SKILL.md +8 -7
  9. package/bin/skills/clickzetta-dynamic-table/dt-creator/references/dt-declaration-strategy.md +99 -99
  10. package/bin/skills/clickzetta-dynamic-table/dt-creator/references/incremental-config-reference.md +188 -188
  11. package/bin/skills/clickzetta-dynamic-table/dt-creator/references/refresh-history-guide.md +117 -117
  12. package/bin/skills/clickzetta-dynamic-table/dt-creator/references/sql-limitations.md +29 -29
  13. package/bin/skills/clickzetta-dynamic-table/dynamic-table-alter/SKILL.md +80 -79
  14. package/bin/skills/clickzetta-dynamic-table/sql-to-dt/SKILL.md +15 -15
  15. package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-column-validation-rules.md +61 -61
  16. package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-conversion-rules.md +100 -100
  17. package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-placeholder-rules.md +64 -64
  18. package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-refresh-rules.md +32 -32
  19. package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-self-reference-rules.md +21 -21
  20. package/bin/skills/clickzetta-dynamic-table/sql-to-dt/references/sql2dt-workflow.md +71 -71
  21. package/bin/skills/clickzetta-sql-pipeline-manager/SKILL.md +203 -202
  22. package/bin/skills/clickzetta-sql-pipeline-manager/references/dynamic-table.md +62 -62
  23. package/bin/skills/clickzetta-sql-pipeline-manager/references/materialized-view.md +34 -34
  24. package/bin/skills/clickzetta-sql-pipeline-manager/references/pipe.md +61 -61
  25. package/bin/skills/clickzetta-sql-pipeline-manager/references/table-stream.md +41 -41
  26. package/bin/skills/clickzetta-table-stream-pipeline/SKILL.md +103 -101
  27. package/package.json +1 -1
@@ -1,248 +1,249 @@
1
1
  ---
2
2
  name: clickzetta-sql-pipeline-manager
3
3
  description: >
4
- 管理 ClickZetta Lakehouse SQL 数据管道对象,包括动态表(Dynamic Table)、
5
- 物化视图(Materialized View)、表流(Table Stream)和 Pipe。
6
- 覆盖创建、修改、暂停/恢复、删除、查看状态等完整生命周期操作。
7
- 仅涉及 SQL 命令操作,不涉及 Lakehouse Studio 图形化界面。
8
-
9
- 当用户说"创建动态表""创建物化视图""创建 Pipe"、"创建表流"、
10
- "暂停/恢复动态表""查看刷新历史""修改刷新频率"、"接入 Kafka"
11
- "从对象存储持续导入""CDC 变更捕获""增量计算"、"实时 ETL"
12
- "数据管道""pipeline""流式处理"、"动态表刷新失败"、
13
- "帮我设计 ETL""构建数据管道""数据接入方案"
14
- "Medallion Architecture""Bronze Silver Gold""奖章架构"
15
- "湖仓分层""Bronze 层"、"Silver ""Gold "时触发。
4
+ Manage SQL data pipeline objects in ClickZetta Lakehouse, including Dynamic Tables,
5
+ Materialized Views, Table Streams, and Pipes.
6
+ Covers the full lifecycle: create, modify, suspend/resume, drop, and status inspection.
7
+ SQL command operations only — does not cover the Lakehouse Studio GUI.
8
+
9
+ Trigger when the user says "create dynamic table", "create materialized view", "create Pipe",
10
+ "create table stream", "suspend/resume dynamic table", "view refresh history",
11
+ "change refresh interval", "ingest from Kafka", "continuous import from object storage",
12
+ "CDC change capture", "incremental computation", "real-time ETL",
13
+ "data pipeline", "pipeline", "stream processing", "dynamic table refresh failed",
14
+ "help me design ETL", "build a data pipeline", "data ingestion plan",
15
+ "Medallion Architecture", "Bronze Silver Gold", "lakehouse layering",
16
+ "Bronze layer", "Silver layer", "Gold layer".
16
17
  Keywords: SQL pipeline, dynamic table, materialized view, table stream, Pipe, data pipeline
17
18
  ---
18
19
 
19
- # ClickZetta SQL 数据管道管理
20
+ # ClickZetta SQL Data Pipeline Management
20
21
 
21
- ## ⚠️ ClickZetta 与标准 SQL / Snowflake 的关键语法差异
22
+ ## ⚠️ Key Syntax Differences: ClickZetta vs Standard SQL / Snowflake
22
23
 
23
- 这些是最容易写错的地方,必须使用 ClickZetta 特有语法:
24
+ These are the most common mistakes — always use ClickZetta-specific syntax:
24
25
 
25
- | 功能 | ❌ 错误写法(Snowflake/标准SQL | ✅ ClickZetta 正确写法 |
26
+ | Feature | ❌ Wrong (Snowflake/Standard SQL) | ✅ ClickZetta Correct |
26
27
  |---|---|---|
27
- | 动态表计算集群 | `WAREHOUSE = compute_wh` | `vcluster default`(直接跟名称,不带等号) |
28
- | 动态表刷新调度 | `TARGET_LAG = '1 minutes'` | `REFRESH INTERVAL 1 MINUTE vcluster default` |
29
- | Kafka 读取函数 | `TABLE(READ_KAFKA(KAFKA_BROKER => ...))` | `read_kafka('broker', 'topic', '', 'group', '', '', '', '', 'raw', 'raw', 0, MAP(...))` — 位置参数 |
30
- | 物化视图定时刷新 | `REFRESH EVERY 1 HOUR` | `REFRESH INTERVAL 60 MINUTE vcluster default`(与动态表语法相同) |
31
- | 物化视图手动刷新 | `REFRESH MATERIALIZED VIEW` 放在 CREATE | 单独执行 `REFRESH MATERIALIZED VIEW <name>;` |
32
- | 修改动态表 SQL | `ALTER DYNAMIC TABLE ... AS ...` | `CREATE OR REPLACE DYNAMIC TABLE ...`(ALTER 不支持修改 AS 子句) |
33
- | JSON 字段访问 | `$1:field::TYPE` `data:key` | `parse_json(value::string)['field']::TYPE` `data['key']` |
34
- | COPY INTO 导入格式 | `FILE_FORMAT = (TYPE = CSV)` | `USING CSV OPTIONS(...)` |
35
- | COPY INTO 导出格式 | `USING CSV` | `FILE_FORMAT = (TYPE = CSV)` |
28
+ | Dynamic Table compute cluster | `WAREHOUSE = compute_wh` | `vcluster default` (name directly, no equals sign) |
29
+ | Dynamic Table refresh schedule | `TARGET_LAG = '1 minutes'` | `REFRESH INTERVAL 1 MINUTE vcluster default` |
30
+ | Kafka read function | `TABLE(READ_KAFKA(KAFKA_BROKER => ...))` | `read_kafka('broker', 'topic', '', 'group', '', '', '', '', 'raw', 'raw', 0, MAP(...))` — positional args |
31
+ | Materialized View scheduled refresh | `REFRESH EVERY 1 HOUR` | `REFRESH INTERVAL 60 MINUTE vcluster default` (same syntax as Dynamic Table) |
32
+ | Materialized View manual refresh | `REFRESH MATERIALIZED VIEW` inside CREATE | Execute `REFRESH MATERIALIZED VIEW <name>;` separately |
33
+ | Modify Dynamic Table SQL | `ALTER DYNAMIC TABLE ... AS ...` | `CREATE OR REPLACE DYNAMIC TABLE ...` (ALTER does not support modifying the AS clause) |
34
+ | JSON field access | `$1:field::TYPE` or `data:key` | `parse_json(value::string)['field']::TYPE` or `data['key']` |
35
+ | COPY INTO import format | `FILE_FORMAT = (TYPE = CSV)` | `USING CSV OPTIONS(...)` |
36
+ | COPY INTO export format | `USING CSV` | `FILE_FORMAT = (TYPE = CSV)` |
36
37
 
37
38
  ---
38
39
 
39
- ## 向导:明确操作意图
40
+ ## Guide: Clarify the User's Intent
40
41
 
41
- 收到请求后,先判断用户意图,选择对应工作流:
42
+ After receiving a request, determine the user's intent and choose the corresponding workflow:
42
43
 
43
- > 你想做什么?
44
+ > What do you want to do?
44
45
  >
45
- > **A. 设计并创建新的数据管道**(从数据源到各层 DT 的完整 SQL)→ 进入 Pipeline Wizard
46
- > **B. 管理已有管道对象**(修改 DT 刷新间隔、暂停/恢复、查看刷新历史)→ 直接执行对应操作
47
- > **C. 排查管道问题**(DT 刷新失败、Pipe 停止摄入、Stream 积压)→ 进入故障排查流程
46
+ > **A. Design and create a new data pipeline** (complete SQL from data source through all layers) → Enter Pipeline Wizard
47
+ > **B. Manage existing pipeline objects** (modify DT refresh interval, suspend/resume, view refresh history) → Execute the corresponding operation directly
48
+ > **C. Troubleshoot pipeline issues** (DT refresh failure, Pipe stopped ingesting, Stream backlog) → Enter troubleshooting flow
48
49
 
49
- **如果用户已经明确说了要做什么(如"帮我创建一个 Kafka DWD 的管道""暂停这个动态表"),直接执行,不再询问。**
50
+ **If the user has already stated clearly what they want (e.g., "create a pipeline from Kafka to DWD", "suspend this dynamic table"), proceed directly without asking again.**
50
51
 
51
52
  ---
52
53
 
53
- ## Pipeline Wizard(管道设计向导)
54
+ ## Pipeline Wizard
54
55
 
55
- 当用户想设计或构建一个完整的数据管道时,这是最高优先级的模式。触发词包括:
56
- "帮我设计/构建 ETL""完整的数据管道"" Kafka/OSS 接入数据"、"ODS→DWD→DWS""端到端 pipeline"
57
- "Medallion Architecture""Bronze/Silver/Gold""奖章架构"、"湖仓分层"。
56
+ Use this mode when the user wants to design or build a complete data pipeline. Trigger phrases include:
57
+ "help me design/build ETL", "complete data pipeline", "ingest data from Kafka/OSS", "ODS→DWD→DWS", "end-to-end pipeline",
58
+ "Medallion Architecture", "Bronze/Silver/Gold", "lakehouse layering".
58
59
 
59
- ### 层次命名约定
60
+ ### Layer Naming Conventions
60
61
 
61
- 用户可能使用不同的分层命名,含义相同,按用户偏好保留原始命名:
62
+ Users may use different layer naming schemes with the same meaning — preserve the user's preferred naming:
62
63
 
63
- | 用户说的 | 含义 | Schema 命名建议 |
64
+ | User says | Meaning | Suggested Schema names |
64
65
  |---|---|---|
65
66
  | Bronze / Silver / Gold | Medallion Architecture | `bronze` / `silver` / `gold` |
66
- | ODS / DWD / DWS | 国内数仓分层惯例 | `ods` / `dwd` / `dws` |
67
- | Raw / Cleansed / Aggregated | 通用英文描述 | `raw` / `cleansed` / `agg` |
67
+ | ODS / DWD / DWS | Chinese data warehouse convention | `ods` / `dwd` / `dws` |
68
+ | Raw / Cleansed / Aggregated | Generic English description | `raw` / `cleansed` / `agg` |
68
69
 
69
- **不要把 Bronze 映射成 ODSSilver 映射成 DWD 等——保留用户选择的命名,在 SQL 中直接使用对应的 schema 和表名前缀。**
70
+ **Do not map Bronze to ODS, Silver to DWD, etc. preserve the user's chosen naming and use the corresponding schema and table name prefixes in SQL.**
70
71
 
71
- **Schema 命名必须加业务/项目前缀,避免与其他项目冲突。** 如果用户未提供前缀,询问项目名称或业务域名称,然后生成带前缀的 Schema 名:
72
+ **Schema names must include a business/project prefix to avoid conflicts with other projects.** If the user has not provided a prefix, ask for the project or business domain name, then generate prefixed schema names:
72
73
 
73
74
  ```sql
74
- -- ❌ 容易重名,不要这样生成
75
+ -- ❌ Prone to naming conflicts — avoid this
75
76
  CREATE SCHEMA IF NOT EXISTS bronze;
76
77
 
77
- -- ✅ 加项目前缀
78
+ -- ✅ Add a project prefix
78
79
  CREATE SCHEMA IF NOT EXISTS ecommerce_bronze;
79
80
  CREATE SCHEMA IF NOT EXISTS ecommerce_silver;
80
81
  CREATE SCHEMA IF NOT EXISTS ecommerce_gold;
81
82
  ```
82
83
 
83
- ### 需求收集
84
+ ### Requirements Gathering
84
85
 
85
- **如果用户已经提供了足够信息(数据来源、字段、层次需求、项目前缀),直接生成完整 SQL,不要再问。**
86
+ **If the user has already provided sufficient information (data source, fields, layer requirements, project prefix), generate the complete SQL directly without asking further questions.**
86
87
 
87
- 如果信息不完整,优先使用交互式问答工具(如 `question`)收集以下信息并弹出选项菜单;若无此类工具,则用文字一次性列出所有问题:
88
+ If information is incomplete, use an interactive Q&A tool (e.g., `question`) to collect the following and present option menus; if no such tool is available, list all questions in a single text response:
88
89
 
89
90
  ```
90
91
  question({
91
92
  questions: [
92
93
  {
93
- question: "数据来源?",
94
+ question: "Data source?",
94
95
  options: [
95
- { label: "Kafka", description: "提供 broker 地址和 topic 名称" },
96
- { label: "对象存储(OSS/S3/COS", description: "提供 Volume 路径和文件格式" },
97
- { label: "已有 Lakehouse 表(仅 INSERT", description: "Dynamic Table 直接读源表" },
98
- { label: "已有 Lakehouse 表(含 UPDATE/DELETE", description: "需要 Table Stream + Dynamic Table" }
96
+ { label: "Kafka", description: "Provide broker address and topic name" },
97
+ { label: "Object Storage (OSS/S3/COS)", description: "Provide Volume path and file format" },
98
+ { label: "Existing Lakehouse table (INSERT only)", description: "Dynamic Table reads directly from source table" },
99
+ { label: "Existing Lakehouse table (with UPDATE/DELETE)", description: "Requires Table Stream + Dynamic Table" }
99
100
  ]
100
101
  },
101
102
  {
102
- question: "刷新频率?",
103
+ question: "Refresh frequency?",
103
104
  options: [
104
- { label: "实时(秒级)", description: "REFRESH INTERVAL 10~60 SECOND" },
105
- { label: "近实时(分钟级)", description: "REFRESH INTERVAL 1~10 MINUTE" },
106
- { label: "低频(小时/天)", description: "REFRESH INTERVAL 1 HOUR 1 DAY" }
105
+ { label: "Real-time (seconds)", description: "REFRESH INTERVAL 10~60 SECOND" },
106
+ { label: "Near real-time (minutes)", description: "REFRESH INTERVAL 1~10 MINUTE" },
107
+ { label: "Low frequency (hourly/daily)", description: "REFRESH INTERVAL 1 HOUR or 1 DAY" }
107
108
  ]
108
109
  }
109
110
  ]
110
111
  })
111
112
  ```
112
113
 
113
- 还需确认:项目/业务前缀(Schema 命名用)、层次需求(几层、每层做什么)、目标表字段结构。这些可在用户回答后追问,或从上下文推断。
114
+ Also confirm: project/business prefix (for schema naming), layer requirements (how many layers, what each layer does), and target table field structure. These can be asked after the user responds, or inferred from context.
114
115
 
115
- ### 生成完整 SQL
116
+ ### Generate Complete SQL
116
117
 
117
- 收到回答后,生成完整的端到端 SQL,包含以下所有部分:
118
+ After receiving answers, generate complete end-to-end SQL including all of the following:
118
119
 
119
120
  ```
120
- 1. Schema 创建(CREATE SCHEMA IF NOT EXISTS,使用用户指定的层次名称)
121
- 2. 入口层建表(如果是外部摄入)
122
- 3. 数据入口(Pipe Table Stream,根据来源选择)
123
- 4. 中间层动态表(清洗/过滤,REFRESH interval N MINUTE VCLUSTER name
124
- 5. 服务层动态表(聚合/维度,REFRESH interval N MINUTE VCLUSTER name
125
- 6. 各动态表创建后立即执行 REFRESH DYNAMIC TABLE(重置刷新基准)
126
- 7. 验证命令(SHOW + REFRESH HISTORY
127
- 8. 运维操作(SUSPEND/RESUME
121
+ 1. Schema creation (CREATE SCHEMA IF NOT EXISTS, using the user's chosen layer names)
122
+ 2. Ingestion layer table creation (if external ingestion is involved)
123
+ 3. Data entry point (Pipe or Table Stream, based on source type)
124
+ 4. Intermediate layer Dynamic Tables (cleansing/filtering, REFRESH INTERVAL N MINUTE VCLUSTER name)
125
+ 5. Serving layer Dynamic Tables (aggregation/dimensions, REFRESH INTERVAL N MINUTE VCLUSTER name)
126
+ 6. Execute REFRESH DYNAMIC TABLE immediately after each Dynamic Table is created (reset refresh baseline)
127
+ 7. Verification commands (SHOW + REFRESH HISTORY)
128
+ 8. Operations commands (SUSPEND/RESUME)
128
129
  ```
129
130
 
130
- **SQL 生成后,将各段代码保存为 Studio 任务(代码资产化):**
131
+ **After generating SQL, save each segment as a Studio task (code as an asset):**
131
132
 
132
- 数据管道开发场景下,所有 SQL 都应保存为 Studio 任务,作为可管理的代码资产:
133
+ In data pipeline development, all SQL should be saved as Studio tasks as manageable code assets:
133
134
 
134
135
  ```bash
135
- # 建表 DDL → 保存为 DRAFT 任务(不配 Cron
136
+ # DDL SQL save as DRAFT task (no Cron)
136
137
  cz-cli task save-content <ddl_task_name> --content "<ddl_sql>"
137
138
 
138
- # ETL/转换 SQL → 保存为调度任务(配 Cron + 依赖)
139
+ # ETL/transformation SQL → save as scheduled task (with Cron + dependencies)
139
140
  cz-cli task save-content <etl_task_name> --content "<etl_sql>"
140
141
  cz-cli task save-cron <etl_task_name> --cron '0 30 2 * * ? *'
141
142
  cz-cli task deploy <etl_task_name>
142
143
  ```
143
144
 
144
- > Dynamic Table DDL 也应保存为 DRAFT 任务(`03_ddl_dws_ads`),方便后续查阅和多环境迁移。
145
+ > Dynamic Table DDL should also be saved as a DRAFT task (`03_ddl_dws_ads`) for easy reference and multi-environment migration.
145
146
 
146
- **⚠️ DDL 任务 vs 数据流转任务的调度规则(硬性约束,不得违反):**
147
+ **⚠️ DDL tasks vs data flow tasks — scheduling rules (hard constraints, must not be violated):**
147
148
 
148
- | 任务类型 | 判断标准 | 调度配置 | Studio 状态 |
149
+ | Task type | Criteria | Scheduling config | Studio status |
149
150
  |---|---|---|---|
150
- | DDL 任务 | 包含 `CREATE / DROP / ALTER TABLE/SCHEMA` | **禁止配置 Cron,禁止配置依赖** | DRAFT |
151
- | 数据流转任务 | 数据同步、ETL 转换、数据质量检查 | 配置 Cron + 上下游依赖 | PUBLISHED |
152
- | Dynamic Table | DWS/ADS 聚合层 | **不建 Studio 任务**,系统自动刷新 | — |
151
+ | DDL task | Contains `CREATE / DROP / ALTER TABLE/SCHEMA` | **No Cron, no dependencies** | DRAFT |
152
+ | Data flow task | Data sync, ETL transformation, data quality checks | Configure Cron + upstream/downstream dependencies | PUBLISHED |
153
+ | Dynamic Table | DWS/ADS aggregation layer | **No Studio task needed** — system auto-refreshes | — |
153
154
 
154
- > AI 生成 SQL 管道时,如果涉及 Studio 任务编排,必须遵守以上规则。不得为 DDL 语句生成 Cron 调度配置。
155
+ > When AI generates SQL pipelines involving Studio task orchestration, the above rules must be followed. Do not generate Cron scheduling for DDL statements.
155
156
 
156
- **来源入口对象的选择规则:**
157
+ **Sourceentry object selection rules:**
157
158
  - Kafka → `CREATE PIPE ... AS COPY INTO ... FROM (SELECT ... FROM read_kafka('broker', 'topic', '', 'group', '', '', '', '', 'raw', 'raw', 0, MAP(...)))`
158
- - 对象存储(OSS/S3/COS)→ `CREATE PIPE ... VIRTUAL_CLUSTER = 'name' INGEST_MODE = 'LIST_PURGE' AS COPY INTO ... FROM VOLUME <volume_name> USING <format> PURGE=true`
159
- - 已有表 + UPDATE/DELETE → `CREATE TABLE STREAM ... WITH PROPERTIES ('TABLE_STREAM_MODE' = 'STANDARD')`,中间层过滤 `__change_type IN ('INSERT', 'UPDATE_AFTER', 'DELETE')`
160
- - 已有表 + INSERT → Dynamic Table 直接 `FROM` 源表
159
+ - Object storage (OSS/S3/COS) `CREATE PIPE ... VIRTUAL_CLUSTER = 'name' INGEST_MODE = 'LIST_PURGE' AS COPY INTO ... FROM VOLUME <volume_name> USING <format> PURGE=true`
160
+ - Existing table + has UPDATE/DELETE → `CREATE TABLE STREAM ... WITH PROPERTIES ('TABLE_STREAM_MODE' = 'STANDARD')`, intermediate layer filters `__change_type IN ('INSERT', 'UPDATE_AFTER', 'DELETE')`
161
+ - Existing table + INSERT only → Dynamic Table reads directly `FROM` source table
161
162
 
162
- **刷新频率规则:**
163
- - 第一个转换层(Bronze→Silver ODS→DWD)设置用户指定的刷新频率(如 `REFRESH INTERVAL 1 MINUTE vcluster default`)
164
- - 下游层根据业务需求设置各自的刷新频率(如 `REFRESH INTERVAL 5 MINUTE vcluster default`)
163
+ **Refresh frequency rules:**
164
+ - First transformation layer (Bronze→Silver or ODS→DWD): use the user-specified refresh frequency (e.g., `REFRESH INTERVAL 1 MINUTE vcluster default`)
165
+ - Downstream layers: set their own refresh frequency based on business requirements (e.g., `REFRESH INTERVAL 5 MINUTE vcluster default`)
165
166
 
166
167
  ---
167
168
 
168
- ## 对象类型速查
169
+ ## Object Type Quick Reference
169
170
 
170
- | 对象 | 适用场景 | 核心特点 |
171
+ | Object | Use case | Key characteristics |
171
172
  |---|---|---|
172
- | **Dynamic Table** | 实时/近实时增量 ETL | SQL 定义,自动增量刷新,秒/分钟级延迟 |
173
- | **Materialized View** | 固定聚合加速查询 | 预计算存储,手动或定时全量刷新 |
174
- | **Table Stream** | CDC 变更数据捕获 | 捕获 INSERT/UPDATE/DELETE,配合 Dynamic Table 消费 |
175
- | **Pipe** | 持续数据摄入 | Kafka 或对象存储自动持续导入,无需调度 |
173
+ | **Dynamic Table** | Real-time / near real-time incremental ETL | SQL-defined, auto incremental refresh, second/minute-level latency |
174
+ | **Materialized View** | Fixed aggregation to accelerate queries | Pre-computed storage, manual or scheduled full refresh |
175
+ | **Table Stream** | CDC change data capture | Captures INSERT/UPDATE/DELETE, consumed by Dynamic Tables |
176
+ | **Pipe** | Continuous data ingestion | Auto continuous import from Kafka or object storage, no scheduling needed |
176
177
 
177
- ## 决策树
178
+ ## Decision Tree
178
179
 
179
180
  ```
180
- 用户需求
181
- ├── 持续从外部摄入数据(Kafka / OSS / S3
181
+ User requirement
182
+ ├── Continuously ingest from external source (Kafka / OSS / S3)
182
183
  │ └── → Pipe
183
- ├── 对已有表做实时/增量转换
184
- │ ├── 需要感知 UPDATE/DELETE → Table Stream + Dynamic Table
185
- │ └── 只需 INSERT 追加 → Dynamic Table(直接查源表)
186
- ├── 固定聚合,不要求实时
184
+ ├── Real-time / incremental transformation on existing tables
185
+ │ ├── Need to detect UPDATE/DELETE → Table Stream + Dynamic Table
186
+ │ └── INSERT append only → Dynamic Table (reads source table directly)
187
+ ├── Fixed aggregation, real-time not required
187
188
  │ └── → Materialized View
188
- └── 多层 ETLODS→DWD→DWS Bronze→Silver→Gold
189
- └── → 多个 Dynamic Table 级联(各层设置独立 REFRESH interval)
189
+ └── Multi-layer ETL (ODS→DWD→DWS or Bronze→Silver→Gold)
190
+ └── → Multiple cascaded Dynamic Tables (each layer with its own REFRESH INTERVAL)
190
191
  ```
191
192
 
192
- ## 步骤 0:确认连接
193
+ ## Step 0: Confirm Connection
193
194
 
194
- 操作前先确认已连接到 ClickZetta Lakehouse。参考 `clickzetta-lakehouse-connect` skill 获取连接参数。
195
+ Before any operation, confirm you are connected to ClickZetta Lakehouse. Refer to the `clickzetta-lakehouse-connect` skill for connection parameters.
195
196
 
196
- ## 步骤 1:选择对象类型
197
+ ## Step 1: Select Object Type
197
198
 
198
- 根据决策树选择对象类型,阅读对应参考文件:
199
+ Use the decision tree to select the object type, then read the corresponding reference file:
199
200
 
200
- | 对象 | 参考文件 |
201
+ | Object | Reference file |
201
202
  |---|---|
202
203
  | Dynamic Table | [references/dynamic-table.md](references/dynamic-table.md) |
203
204
  | Materialized View | [references/materialized-view.md](references/materialized-view.md) |
204
205
  | Table Stream | [references/table-stream.md](references/table-stream.md) |
205
206
  | Pipe | [references/pipe.md](references/pipe.md) |
206
207
 
207
- ## 步骤 2:生成并执行 SQL
208
+ ## Step 2: Generate and Execute SQL
208
209
 
209
- 阅读对应参考文件后,根据用户提供的参数生成完整可运行 SQL
210
+ After reading the corresponding reference file, generate complete runnable SQL based on the user's parameters.
210
211
 
211
- **必填参数检查:**
212
- - Dynamic Table:`REFRESH INTERVAL N MINUTE vcluster name`、AS 查询
213
- - Table Stream:源表名、MODESTANDARD APPEND_ONLY
214
- - PipeKafka):bootstrap_serverstopicgroup_id、目标表(位置参数语法)
215
- - Pipe(对象存储):Volume 路径、文件格式、目标表、`PURGE=true`(LIST_PURGE 模式)
212
+ **Required parameter checklist:**
213
+ - Dynamic Table: `REFRESH INTERVAL N MINUTE vcluster name`, AS query
214
+ - Table Stream: source table name, MODE (STANDARD or APPEND_ONLY)
215
+ - Pipe (Kafka): bootstrap_servers, topic, group_id, target table (positional parameter syntax)
216
+ - Pipe (object storage): Volume path, file format, target table, `PURGE=true` (LIST_PURGE mode)
216
217
 
217
- 若用户未提供 VCLUSTER,默认使用 `default`(GP 型集群)。
218
+ If the user has not provided a VCLUSTER, default to `default` (GP-type cluster).
218
219
 
219
- ## 步骤 3:验证
220
+ ## Step 3: Verify
220
221
 
221
222
  ```sql
222
- -- 验证动态表
223
+ -- Verify Dynamic Table
223
224
  SHOW TABLES WHERE is_dynamic = true;
224
225
  SHOW DYNAMIC TABLE REFRESH HISTORY <name> LIMIT 5;
225
226
 
226
- -- 验证物化视图
227
+ -- Verify Materialized View
227
228
  SHOW TABLES WHERE is_materialized_view = true;
228
229
 
229
- -- 验证 Table Stream
230
+ -- Verify Table Stream
230
231
  SHOW TABLE STREAMS;
231
- SELECT COUNT(*) FROM <stream_name>; -- 查看待消费变更数
232
+ SELECT COUNT(*) FROM <stream_name>; -- check pending change count
232
233
 
233
- -- 验证 Pipe
234
+ -- Verify Pipe
234
235
  SHOW PIPES;
235
236
  ```
236
237
 
237
238
  ---
238
239
 
239
- ## 典型场景示例
240
+ ## Typical Scenario Examples
240
241
 
241
- ### 场景 AKafka → 动态表(实时 ETL
242
+ ### Scenario A: Kafka → Dynamic Table (Real-time ETL)
242
243
 
243
244
  ```sql
244
- -- Step 1: 创建 Pipe 持续摄入 Kafka 数据到 ODS
245
- -- ⚠️ 注意:ClickZetta 不支持 CREATE OR REPLACE PIPE,需用 CREATE PIPE 或先 DROP CREATE
245
+ -- Step 1: Create Pipe to continuously ingest Kafka data into ODS layer
246
+ -- ⚠️ Note: ClickZetta does not support CREATE OR REPLACE PIPE; use CREATE PIPE or DROP then CREATE
246
247
  CREATE PIPE kafka_orders_pipe
247
248
  VIRTUAL_CLUSTER = 'default'
248
249
  BATCH_INTERVAL_IN_SECONDS = '60'
@@ -261,14 +262,14 @@ COPY INTO ods.orders FROM (
261
262
  'orders', -- topic
262
263
  '', -- reserved
263
264
  'lakehouse_ingest', -- group_id
264
- '', '', '', '', -- 位置参数留空,由 Pipe 管理
265
+ '', '', '', '', -- positional params left empty, managed by Pipe
265
266
  'raw', 'raw', 0,
266
267
  MAP('kafka.security.protocol', 'PLAINTEXT')
267
268
  )
268
269
  )
269
270
  );
270
271
 
271
- -- Step 2: 动态表做 DWD 层清洗(每分钟增量刷新)
272
+ -- Step 2: Dynamic Table for DWD layer cleansing (incremental refresh every minute)
272
273
  CREATE OR REPLACE DYNAMIC TABLE dwd.orders_clean
273
274
  REFRESH INTERVAL 1 MINUTE vcluster default
274
275
  AS
@@ -282,7 +283,7 @@ SELECT
282
283
  FROM ods.orders
283
284
  WHERE amount > 0;
284
285
 
285
- -- Step 3: 动态表做 DWS 层聚合(每 5 分钟刷新)
286
+ -- Step 3: Dynamic Table for DWS layer aggregation (refresh every 5 minutes)
286
287
  CREATE OR REPLACE DYNAMIC TABLE dws.order_hourly
287
288
  REFRESH INTERVAL 5 MINUTE vcluster default
288
289
  AS
@@ -295,15 +296,15 @@ FROM dwd.orders_clean
295
296
  GROUP BY 1, 2;
296
297
  ```
297
298
 
298
- ### 场景 BTable Stream + Dynamic TableCDC UPSERT
299
+ ### Scenario B: Table Stream + Dynamic Table (CDC UPSERT)
299
300
 
300
301
  ```sql
301
- -- Step 1: 在源表上创建 Stream 捕获变更
302
+ -- Step 1: Create Stream on source table to capture changes
302
303
  CREATE TABLE STREAM ods.orders_stream
303
304
  ON TABLE ods.orders
304
305
  WITH PROPERTIES ('TABLE_STREAM_MODE' = 'STANDARD');
305
306
 
306
- -- Step 2: 动态表消费 Stream,过滤出最新状态
307
+ -- Step 2: Dynamic Table consumes Stream, filters for latest state
307
308
  CREATE OR REPLACE DYNAMIC TABLE dwd.orders_latest
308
309
  REFRESH INTERVAL 2 MINUTE vcluster default
309
310
  AS
@@ -312,15 +313,15 @@ FROM ods.orders_stream
312
313
  WHERE __change_type IN ('INSERT', 'UPDATE_AFTER');
313
314
  ```
314
315
 
315
- ### 场景 C:物化视图加速 BI 查询
316
+ ### Scenario C: Materialized View to Accelerate BI Queries
316
317
 
317
318
  ```sql
318
- -- 创建每小时刷新的物化视图
319
- -- ⚠️ 注意:ClickZetta 不支持 CREATE OR REPLACE MATERIALIZED VIEW
320
- -- 方法 1: DROP CREATE(推荐)
319
+ -- Create a materialized view with hourly refresh
320
+ -- ⚠️ Note: ClickZetta does not support CREATE OR REPLACE MATERIALIZED VIEW
321
+ -- Method 1: DROP then CREATE (recommended)
321
322
  DROP MATERIALIZED VIEW IF EXISTS dws.mv_daily_revenue;
322
323
  CREATE MATERIALIZED VIEW dws.mv_daily_revenue
323
- COMMENT '每日收入汇总,供 BI 工具查询'
324
+ COMMENT 'Daily revenue summary for BI tools'
324
325
  REFRESH INTERVAL 60 MINUTE vcluster default
325
326
  AS
326
327
  SELECT
@@ -331,41 +332,41 @@ SELECT
331
332
  FROM dwd.orders_clean
332
333
  GROUP BY 1, 2;
333
334
 
334
- -- 方法 2: 使用 BUILD DEFERRED + DISABLE QUERY REWRITE(复杂,不推荐)
335
+ -- Method 2: Use BUILD DEFERRED + DISABLE QUERY REWRITE (complex, not recommended)
335
336
  -- CREATE OR REPLACE MATERIALIZED VIEW ... BUILD DEFERRED DISABLE QUERY REWRITE AS ...
336
337
 
337
- -- 手动触发刷新
338
+ -- Manually trigger refresh
338
339
  REFRESH MATERIALIZED VIEW dws.mv_daily_revenue;
339
340
 
340
- -- 删除物化视图(⚠️ 注意:必须用 DROP MATERIALIZED VIEW,不能用 DROP TABLE
341
+ -- Drop materialized view (⚠️ must use DROP MATERIALIZED VIEW, not DROP TABLE)
341
342
  DROP MATERIALIZED VIEW dws.mv_daily_revenue;
342
343
  ```
343
344
 
344
- ### 场景 D:运维操作
345
+ ### Scenario D: Operations
345
346
 
346
347
  ```sql
347
- -- 暂停动态表(如集群维护)
348
+ -- Suspend Dynamic Table (e.g., during cluster maintenance)
348
349
  ALTER DYNAMIC TABLE dwd.orders_clean SUSPEND;
349
350
 
350
- -- 恢复
351
+ -- Resume
351
352
  ALTER DYNAMIC TABLE dwd.orders_clean RESUME;
352
353
 
353
- -- 查看刷新历史排查失败
354
+ -- View refresh history to troubleshoot failures
354
355
  SHOW DYNAMIC TABLE REFRESH HISTORY dwd.orders_clean LIMIT 10;
355
356
 
356
- -- 暂停 Pipe
357
+ -- Pause Pipe
357
358
  ALTER PIPE kafka_orders_pipe SET PIPE_EXECUTION_PAUSED = true;
358
359
 
359
- -- 恢复 Pipe
360
+ -- Resume Pipe
360
361
  ALTER PIPE kafka_orders_pipe SET PIPE_EXECUTION_PAUSED = false;
361
362
  ```
362
363
 
363
- ### 场景 E:参数化动态表(按分区刷新)
364
+ ### Scenario E: Parameterized Dynamic Table (Partition-based Refresh)
364
365
 
365
- 通过 `SESSION_CONFIGS()` 函数定义参数化查询,在刷新时传入分区值控制全量或增量刷新范围:
366
+ Use the `SESSION_CONFIGS()` function to define parameterized queries, passing partition values at refresh time to control the refresh scope:
366
367
 
367
368
  ```sql
368
- -- 创建参数化动态表(使用 SESSION_CONFIGS 定义参数)
369
+ -- Create a parameterized Dynamic Table (using SESSION_CONFIGS to define parameters)
369
370
  CREATE OR REPLACE DYNAMIC TABLE dwd.orders_partitioned
370
371
  REFRESH INTERVAL 30 MINUTE vcluster default
371
372
  AS
@@ -373,27 +374,27 @@ SELECT order_id, user_id, amount, status, created_at, DATE(created_at) AS dt
373
374
  FROM ods.orders
374
375
  WHERE dt = SESSION_CONFIGS('target_date', CAST(CURRENT_DATE() AS STRING));
375
376
 
376
- -- 手动触发刷新并传入参数
377
+ -- Manually trigger refresh with parameters
377
378
  REFRESH DYNAMIC TABLE dwd.orders_partitioned
378
379
  WITH PROPERTIES ('target_date' = '2024-06-15');
379
380
  ```
380
381
 
381
- > **适用场景**:传统按天/按小时全量 ETL 任务改造为增量任务时,用 SESSION_CONFIGS 替换调度变量(如 `${bizdate}`),实现参数化分区刷新。
382
+ > **Use case**: When migrating traditional daily/hourly full ETL jobs to incremental jobs, replace scheduling variables (e.g., `${bizdate}`) with SESSION_CONFIGS for parameterized partition refresh.
382
383
 
383
- ### 场景 F:动态表 DML 操作(手动修正数据)
384
+ ### Scenario F: Dynamic Table DML Operations (Manual Data Correction)
384
385
 
385
- ⚠️ **重要**:ClickZetta 动态表**不支持 DML 操作**(INSERT/UPDATE/DELETE)。如需修正数据,有以下方案:
386
+ ⚠️ **Important**: ClickZetta Dynamic Tables **do not support DML operations** (INSERT/UPDATE/DELETE) by default. For data correction, the following options are available:
386
387
 
387
- **方案 1:重建动态表(推荐)**
388
+ **Option 1: Rebuild the Dynamic Table (recommended)**
388
389
  ```sql
389
- -- 1. 在源表中修正数据
390
- -- 2. 等待动态表自动刷新(下一次 REFRESH INTERVAL 会全量刷新)
390
+ -- 1. Correct data in the source table
391
+ -- 2. Wait for the Dynamic Table to auto-refresh (the next REFRESH INTERVAL will trigger a full refresh)
391
392
  ```
392
393
 
393
- **方案 2:使用普通表替代动态表**
394
+ **Option 2: Use a regular table instead of a Dynamic Table**
394
395
  ```sql
395
- -- 对于需要频繁手动修正的场景,建议使用普通表 + 定时调度任务
396
- -- 而不是动态表
396
+ -- For scenarios requiring frequent manual corrections, use a regular table + scheduled Studio task
397
+ -- instead of a Dynamic Table
397
398
  CREATE TABLE dwd.orders_manual (
398
399
  order_id STRING,
399
400
  user_id STRING,
@@ -404,82 +405,82 @@ CREATE TABLE dwd.orders_manual (
404
405
  );
405
406
  ```
406
407
 
407
- > ⚠️ **动态表限制**:
408
- > - 动态表是只读的,不支持 INSERT/UPDATE/DELETE
409
- > - 数据修正应在源表进行,动态表会自动刷新
410
- > - 如需手动控制数据,使用普通表 + Studio 调度任务
408
+ > ⚠️ **Dynamic Table limitations**:
409
+ > - Dynamic Tables are read-only; INSERT/UPDATE/DELETE are not supported
410
+ > - Data corrections should be made in the source table; the Dynamic Table will auto-refresh
411
+ > - For manual data control, use a regular table + Studio scheduled task
411
412
 
412
413
  ---
413
414
 
414
- ## 常见错误
415
+ ## Common Errors
415
416
 
416
- | 错误 | 原因 | 解决方案 |
417
+ | Error | Cause | Solution |
417
418
  |---|---|---|
418
- | `VCluster not available` | 计算集群未启动或名称错误 | 确认 VCLUSTER 名称,检查集群状态 |
419
- | 动态表刷新失败 | SQL 查询报错或源表结构变更 | `SHOW DYNAMIC TABLE REFRESH HISTORY WHERE name = 'xxx'` 查看错误详情 |
420
- | Stream 数据为空 | 已被消费或超出保留周期 | 检查源表 `data_retention_days`,确认是否已消费 |
421
- | Pipe 停止摄入 | Kafka offset 问题或连接断开 | `DESC PIPE EXTENDED` 查看状态,检查 Kafka 连接 |
422
- | `Cannot ALTER AS clause` | 尝试用 ALTER 修改动态表 SQL | 改用 `CREATE OR REPLACE DYNAMIC TABLE` |
423
- | `CREATE OR REPLACE PIPE` 语法报错 | ClickZetta 不支持该语法 | `CREATE PIPE` 或先 `DROP PIPE` `CREATE` |
424
- | `CREATE OR REPLACE MATERIALIZED VIEW` 语法报错 | 仅支持 `REWRITE DISABLED + BUILD DEFER` 模式 | 推荐用 `DROP MATERIALIZED VIEW` + `CREATE MATERIALIZED VIEW` |
425
- | `DROP TABLE` 删除物化视图报错 | 对象类型不匹配 | `DROP MATERIALIZED VIEW`(不是 `DROP TABLE`) |
426
- | 动态表 DML 报错 `not allowed` | 动态表不支持 DML | 在源表修正数据,或使用普通表 + 调度任务 |
427
- | `SET cz.sql.dt.allow.dml` 报错 | 不支持 session statement | 动态表不支持 DML 操作,改用其他方案 |
419
+ | `VCluster not available` | Compute cluster not started or name is wrong | Verify VCLUSTER name, check cluster status |
420
+ | Dynamic Table refresh failed | SQL query error or source table schema changed | Run `SHOW DYNAMIC TABLE REFRESH HISTORY WHERE name = 'xxx'` to view error details |
421
+ | Stream data is empty | Already consumed or past retention period | Check source table `data_retention_days`, confirm whether data was consumed |
422
+ | Pipe stopped ingesting | Kafka offset issue or connection dropped | Run `DESC PIPE EXTENDED` to check status, verify Kafka connection |
423
+ | `Cannot ALTER AS clause` | Attempted to modify Dynamic Table SQL via ALTER | Use `CREATE OR REPLACE DYNAMIC TABLE` instead |
424
+ | `CREATE OR REPLACE PIPE` syntax error | ClickZetta does not support this syntax | Use `CREATE PIPE` or `DROP PIPE` then `CREATE` |
425
+ | `CREATE OR REPLACE MATERIALIZED VIEW` syntax error | Only supports `REWRITE DISABLED + BUILD DEFER` mode | Use `DROP MATERIALIZED VIEW` + `CREATE MATERIALIZED VIEW` |
426
+ | `DROP TABLE` fails on materialized view | Object type mismatch | Use `DROP MATERIALIZED VIEW` (not `DROP TABLE`) |
427
+ | Dynamic Table DML error `not allowed` | Dynamic Tables do not support DML | Correct data in source table, or use a regular table + scheduled task |
428
+ | `SET cz.sql.dt.allow.dml` error | Session statement not supported | Dynamic Tables do not support DML; use an alternative approach |
428
429
 
429
430
  ---
430
431
 
431
- ## 交付验收 Checklist
432
+ ## Delivery Acceptance Checklist
432
433
 
433
- 管道创建完成后,**必须逐项验证**,不得跳过:
434
+ After pipeline creation, **verify each item — do not skip**:
434
435
 
435
436
  ```sql
436
- -- 1. 行数比对:各层行数与预期一致
437
- SELECT COUNT(*) FROM ods.<table>; -- ODS 行数源端
438
- SELECT COUNT(*) FROM dwd.<table>; -- DWD 行数 ≤ ODS(清洗后)
439
- SELECT COUNT(*) FROM dws.<table>; -- DWS 行数符合聚合逻辑
437
+ -- 1. Row count comparison: each layer's row count matches expectations
438
+ SELECT COUNT(*) FROM ods.<table>; -- ODS countsource
439
+ SELECT COUNT(*) FROM dwd.<table>; -- DWD count ≤ ODS (after cleansing)
440
+ SELECT COUNT(*) FROM dws.<table>; -- DWS count matches aggregation logic
440
441
 
441
- -- 2. Dynamic Table 刷新状态
442
+ -- 2. Dynamic Table refresh status
442
443
  SHOW DYNAMIC TABLE REFRESH HISTORY <schema>.<table> LIMIT 5;
443
- -- 确认最近一次 status = SUCCESSrefresh_mode = INCREMENTAL FULL
444
+ -- Confirm latest status = SUCCESS, refresh_mode = INCREMENTAL or FULL
444
445
 
445
- -- 3. 关键字段非空率
446
+ -- 3. Key field non-null rate
446
447
  SELECT
447
448
  COUNT(*) AS total,
448
449
  COUNT(key_field) AS non_null,
449
450
  ROUND(COUNT(key_field) * 100.0 / COUNT(*), 2) AS non_null_pct
450
451
  FROM <schema>.<table>;
451
- -- 核心业务字段非空率应 > 99%
452
+ -- Core business fields should have non-null rate > 99%
452
453
 
453
- -- 4. 主键唯一性(DWD 层事实表)
454
+ -- 4. Primary key uniqueness (DWD fact tables)
454
455
  SELECT key_col, COUNT(*) AS cnt
455
456
  FROM dwd.<table>
456
457
  GROUP BY key_col
457
458
  HAVING cnt > 1
458
459
  LIMIT 10;
459
- -- 结果为空 = 无重复,符合预期
460
+ -- Empty result = no duplicates, as expected
460
461
 
461
- -- 5. Pipe 摄入状态(如有)
462
+ -- 5. Pipe ingestion status (if applicable)
462
463
  SHOW PIPES;
463
- -- status = RUNNINGlast_ingested_timestamp 持续更新
464
+ -- status = RUNNING, last_ingested_timestamp continuously updating
464
465
  ```
465
466
 
466
- **验收标准:**
467
- - [ ] 各层行数与预期一致
468
- - [ ] Dynamic Table 最近刷新状态为 SUCCESS
469
- - [ ] 关键字段非空率 > 99%
470
- - [ ] DWD 层主键无重复
471
- - [ ] Pipe 状态 RUNNING(如有)
472
- - [ ] 所有 DDL 任务为 DRAFT 状态(如涉及 Studio 任务)
473
- - [ ] DWS/ADS 层无冗余 Studio 调度任务
467
+ **Acceptance criteria:**
468
+ - [ ] Row counts at each layer match expectations
469
+ - [ ] Dynamic Table latest refresh status is SUCCESS
470
+ - [ ] Key field non-null rate > 99%
471
+ - [ ] DWD layer primary keys have no duplicates
472
+ - [ ] Pipe status is RUNNING (if applicable)
473
+ - [ ] All DDL tasks are in DRAFT status (if Studio tasks are involved)
474
+ - [ ] No redundant Studio scheduled tasks at DWS/ADS layer
474
475
 
475
476
  ---
476
477
 
477
- ## 参考文档
478
+ ## Reference Documentation
478
479
 
479
- - [增量计算概述](https://www.yunqi.tech/documents/streaming_data_pipeline_overview)
480
+ - [Incremental Computation Overview](https://www.yunqi.tech/documents/streaming_data_pipeline_overview)
480
481
  - [Dynamic Table](https://www.yunqi.tech/documents/dynamic-table)
481
- - [Table Stream 变化数据捕获](https://www.yunqi.tech/documents/table_stream)
482
- - [物化视图](https://www.yunqi.tech/documents/materialized_ddl)
483
- - [Pipe 简介](https://www.yunqi.tech/documents/pipe-summary)
484
- - [使用 Dynamic Table 开展实时 ETL](https://www.yunqi.tech/documents/tutorials-streaming-data-pipeline-with_dynamic-table)
485
- - [LLM 全量文档索引](https://yunqi.tech/llms-full.txt)
482
+ - [Table Stream Change Data Capture](https://www.yunqi.tech/documents/table_stream)
483
+ - [Materialized View](https://www.yunqi.tech/documents/materialized_ddl)
484
+ - [Pipe Overview](https://www.yunqi.tech/documents/pipe-summary)
485
+ - [Real-time ETL with Dynamic Table](https://www.yunqi.tech/documents/tutorials-streaming-data-pipeline-with_dynamic-table)
486
+ - [LLM Full Documentation Index](https://yunqi.tech/llms-full.txt)