@clickzetta/cz-cli-darwin-x64 0.3.40 → 0.3.41

package/bin/skills/clickzetta-app-python-sdk/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: clickzetta-app-python-sdk
+description: |
+  在 Python 应用程序中集成 ClickZetta Lakehouse 的官方 SDK 用法。
+  覆盖 clickzetta-connector-python（SQL 查询、参数绑定、批量插入、异步执行）、
+  clickzetta-ingestion-python（BulkLoad 批量上传，单线程与分布式模式）、
+  clickzetta-ingestion-python-v2（IGS 实时写入，秒级可查，支持主键表 CDC）、
+  SQLAlchemy dialect 集成，以及连接参数说明。
+  当用户说"Python SDK"、"clickzetta-connector-python"、"clickzetta-ingestion-python"、
+  "Python 查询 Lakehouse"、"Python 写入 Lakehouse"、"Python 批量上传"、
+  "BulkLoad Python"、"SQLAlchemy Lakehouse"、"Python 连接 Lakehouse"、
+  "executemany"、"execute_async"、"参数绑定 Python"、
+  "IGS 实时写入"、"实时写入 Python"、"ingestion-python-v2"、
+  "主键表写入 Python"、"CDC 写入"、"UPSERT Python"时触发。
+  Keywords: Python SDK, clickzetta-connector-python, clickzetta-ingestion-python, bulk insert, async query, SQLAlchemy, IGS
+---
+
+# ClickZetta Lakehouse — Python SDK
+
+官方提供三个 Python 包：
+- **`clickzetta-connector-python`** — SQL 查询接口（PEP-249 规范），支持参数绑定、批量插入、异步执行、SQLAlchemy dialect
+- **`clickzetta-ingestion-python`** — 高吞吐批量上传（BulkLoad），数据直传对象存储，不消耗计算资源
+- **`clickzetta-ingestion-python-v2`** — IGS 实时写入，秒级可查，支持主键表 CDC（UPSERT/DELETE）
+
+阅读 [references/connector.md](references/connector.md) 了解 SQL 查询接口，[references/bulkload.md](references/bulkload.md) 了解批量上传，[references/realtime.md](references/realtime.md) 了解 IGS 实时写入。
+
+---
+
+## 安装
+
+```bash
+# SQL 查询接口
+pip install clickzetta-connector-python -U
+
+# 批量上传（按云环境选择）
+pip install "clickzetta-ingestion-python[oss]" -U   # 阿里云
+pip install "clickzetta-ingestion-python[s3]" -U    # AWS
+pip install "clickzetta-ingestion-python[all]" -U   # 全部（安装较慢）
+
+# IGS 实时写入
+pip install clickzetta-ingestion-python-v2
+```
+
+> 注意：旧版 `clickzetta-connector` 已停止维护，请迁移到 `clickzetta-connector-python`。
+
+---
+
+## 连接参数
+
+```python
+from clickzetta import connect
+
+conn = connect(
+    username='your_username',
+    password='your_password',
+    service='api.clickzetta.com',      # region.api.clickzetta.com
+    instance='your_instance',
+    workspace='your_workspace',
+    schema='public',
+    vcluster='default'
+)
+```
+
+| 参数 | 必填 | 说明 |
+|---|---|---|
+| `username` | ✅ | 用户名 |
+| `password` | ✅ | 密码 |
+| `service` | ✅ | 连接地址，格式 `region.api.clickzetta.com` |
+| `instance` | ✅ | 实例名，在 Studio 工作空间 JDBC 连接串中查看 |
+| `workspace` | ✅ | 工作空间名 |
+| `vcluster` | ✅ | 虚拟集群名 |
+| `schema` | ✅ | 默认 schema |
+
+---
+
+## 快速示例
+
+```python
+# 查询
+cursor = conn.cursor()
+cursor.execute('SELECT * FROM orders LIMIT 10')
+results = cursor.fetchall()
+cursor.close()
+conn.close()
+
+# 参数绑定（防 SQL 注入）
+cursor.execute('INSERT INTO test (id, name) VALUES (?, ?)', binding_params=[1, 'test'])
+
+# 批量插入
+data = [(1, 'a'), (2, 'b'), (3, 'c')]
+cursor.executemany('INSERT INTO test (id, name) VALUES (?, ?)', data)
+```
+
+## IGS 实时写入快速示例（ingestion-python-v2）
+
+普通表（APPEND_ONLY）：
+
+```python
+from clickzetta.connector.v0.connection import connect
+from clickzetta.connector.v0.enums import RealtimeOperation
+from clickzetta_ingestion.realtime.realtime_options import RealtimeOptionsBuilder, FlushMode
+from clickzetta_ingestion.realtime.arrow_stream import RowOperator
+
+with connect(**CONN_ARGS) as conn:
+    stream = conn.get_realtime_stream(
+        schema='your_schema',
+        table='your_table',
+        operate=RealtimeOperation.APPEND_ONLY,
+        options=RealtimeOptionsBuilder().with_flush_mode(FlushMode.AUTO_FLUSH_BACKGROUND).build()
+    )
+    row = stream.create_row(RowOperator.INSERT)
+    row.set_value('id', 1)
+    row.set_value('name', 'alice')
+    stream.apply(row)
+    stream.close()
+```
+
+主键表 CDC（UPSERT / DELETE）：
+
+```python
+# 建表：CREATE TABLE users (id STRING NOT NULL PRIMARY KEY, name STRING, age INT);
+
+with connect(**CONN_ARGS) as conn:
+    stream = conn.get_realtime_stream(
+        schema='your_schema',
+        table='users',
+        operate=RealtimeOperation.CDC,           # 主键表必须用 CDC
+        options=RealtimeOptionsBuilder().with_flush_mode(FlushMode.AUTO_FLUSH_SYNC).build()
+    )
+    # UPSERT
+    row = stream.create_row(RowOperator.UPSERT)
+    row.set_value('id', 'u1')
+    row.set_value('name', 'bob')
+    row.set_value('age', 25)
+    stream.apply(row)
+    # DELETE_IGNORE
+    row = stream.create_row(RowOperator.DELETE_IGNORE)
+    row.set_value('id', 'u1')
+    stream.apply(row)
+    stream.close()
+```
+
+---
+
+## 选择指南
+
+| 场景 | 推荐方案 |
+|---|---|
+| 查询 / 小批量写入 | `clickzetta-connector-python` |
+| 大批量数据导入（GB 级，间隔 ≥ 5 分钟） | `clickzetta-ingestion-python` BulkLoad |
+| 高频小批写入（间隔 < 5 分钟，秒级可查） | `clickzetta-ingestion-python-v2` 实时写入 |
+| 主键表写入（UPSERT / DELETE） | `clickzetta-ingestion-python-v2` CDC 模式 |
+| SQLAlchemy / ORM 集成 | `clickzetta-connector-python`（内置 dialect） |

package/bin/skills/clickzetta-app-python-sdk/eval_cases.jsonl

@@ -0,0 +1,12 @@
+{"case_id":"001","type":"should_call","user_input":"clickzetta-connector-python 怎么执行 SQL 查询","expected_skill":"clickzetta-app-python-sdk","expected_output_contains":["cursor","execute"]}
+{"case_id":"002","type":"should_call","user_input":"Python 怎么批量上传数据到 Lakehouse","expected_skill":"clickzetta-app-python-sdk","expected_output_contains":["ingestion","BulkLoad"]}
+{"case_id":"003","type":"should_call","user_input":"executemany 批量插入怎么用","expected_skill":"clickzetta-app-python-sdk","expected_output_contains":["executemany"]}
+{"case_id":"004","type":"should_call","user_input":"Python SDK 怎么做参数绑定","expected_skill":"clickzetta-app-python-sdk","expected_output_contains":["参数绑定"]}
+{"case_id":"005","type":"should_call","user_input":"execute_async 异步执行怎么用","expected_skill":"clickzetta-app-python-sdk","expected_output_contains":["async"]}
+{"case_id":"006","type":"should_call","user_input":"clickzetta-ingestion-python-v2 IGS 实时写入怎么用","expected_skill":"clickzetta-app-python-sdk","expected_output_contains":["IGS","实时"]}
+{"case_id":"007","type":"should_call","user_input":"Python SDK 支持主键表 UPSERT 吗","expected_skill":"clickzetta-app-python-sdk","expected_output_contains":["UPSERT","主键"]}
+{"case_id":"008","type":"should_not_call","user_input":"Java SDK 怎么写入数据","forbidden_skill":"clickzetta-app-python-sdk"}
+{"case_id":"009","type":"should_not_call","user_input":"帮我写一个 Django 应用","forbidden_skill":"clickzetta-app-python-sdk"}
+{"case_id":"010","type":"should_not_call","user_input":"ZettaPark DataFrame 怎么用","forbidden_skill":"clickzetta-app-python-sdk"}
+{"case_id":"011","type":"should_not_call","user_input":"怎么创建 VCluster","forbidden_skill":"clickzetta-app-python-sdk"}
+{"case_id":"012","type":"should_not_call","user_input":"pip install numpy 报错","forbidden_skill":"clickzetta-app-python-sdk"}

package/bin/skills/clickzetta-app-python-sdk/references/bulkload.md

@@ -0,0 +1,196 @@
+# clickzetta-ingestion-python BulkLoad 详细参考
+
+## 安装
+
+```bash
+# 按云环境选择（推荐按需安装，all 安装较慢且可能冲突）
+pip install "clickzetta-ingestion-python[oss]" -U    # 阿里云
+pip install "clickzetta-ingestion-python[s3]" -U     # AWS
+pip install "clickzetta-ingestion-python[cos]" -U    # 腾讯云
+pip install "clickzetta-ingestion-python[gcp]" -U    # Google Cloud
+pip install "clickzetta-ingestion-python[all]" -U    # 全部
+```
+
+## 工作原理
+
+```
+[SDK 写入数据] → [对象存储] → [调用 commit()] → [触发 SQL 导入] → [Lakehouse 表]
+```
+
+- 数据上传阶段不消耗计算资源
+- `commit()` 触发从对象存储到 Lakehouse 表的导入，消耗少量计算资源
+- `commit()` 只能调用一次，commit 后数据可见
+
+## 使用限制
+
+- **不支持主键（pk）表写入**
+- **不适合时间间隔小于 5 分钟的高频写入**
+
+## 单线程写入
+
+### 建表
+
+```sql
+CREATE TABLE public.bulkload_test (
+    i BIGINT,
+    s STRING,
+    d DOUBLE
+);
+```
+
+### 完整示例
+
+```python
+from clickzetta import connect
+
+conn = connect(
+    username='your_username',
+    password='your_password',
+    service='api.clickzetta.com',
+    instance='your_instance',
+    workspace='your_workspace',
+    schema='public',
+    vcluster='default'
+)
+
+bulkload_stream = conn.create_bulkload_stream(schema='public', table='bulkload_test')
+
+writer = bulkload_stream.open_writer(0)  # 单线程传 0
+for index in range(1000000):
+    row = writer.create_row()
+    row.set_value('i', index)      # 按列名设值
+    row.set_value('s', 'Hello')
+    row.set_value('d', 123.456)
+    writer.write(row)
+writer.close()
+
+bulkload_stream.commit()           # 提交，数据可见
+```
+
+## 读取 CSV 写入示例
+
+```python
+from clickzetta import connect
+import csv
+
+conn = connect(
+    username='',
+    password='',
+    service='api.clickzetta.com',
+    instance='',
+    workspace='',
+    schema='public',
+    vcluster='default_ap'
+)
+
+bulkload_stream = conn.create_bulkload_stream(schema='public', table='bulk_order_payments')
+writer = bulkload_stream.open_writer(0)
+
+with open('olist_order_payments_dataset.csv', 'r') as csvfile:
+    reader = csv.reader(csvfile)
+    next(reader)  # 跳过 header
+    for record in reader:
+        row = writer.create_row()
+        row.set_value('order_id', record[0])
+        row.set_value('payment_sequence', int(record[1]))
+        row.set_value('payment_type', record[2])
+        row.set_value('payment_installments', int(record[3]))
+        row.set_value('payment_value', float(record[4]))
+        writer.write(row)  # ⚠️ 必须调用，否则数据不发送到服务端
+
+writer.close()
+bulkload_stream.commit()
+```
+
+## 写入模式
+
+```python
+from clickzetta.bulkload.bulkload_enums import BulkLoadOperation
+
+# APPEND 模式（默认）：新数据追加，不影响旧数据
+bulkload_stream = conn.create_bulkload_stream(schema='public', table='my_table')
+
+# OVERWRITE 模式：清空旧数据，写入新数据
+bulkload_stream = conn.create_bulkload_stream(
+    schema='public',
+    table='my_table',
+    operation=BulkLoadOperation.OVERWRITE
+)
+
+# 分区表 OVERWRITE（只覆盖指定分区）
+bulkload_stream = conn.create_bulkload_stream(
+    schema='public',
+    table='my_partitioned_table',
+    partition_spec='pt=2024-01-01',
+    operation=BulkLoadOperation.OVERWRITE
+)
+```
+
+## 分布式并发写入
+
+适合 GB 级以上数据，多进程并发写入同一 stream，最后统一 commit。
+
+### 控制进程
+
+```python
+import subprocess
+from clickzetta import connect
+
+conn = connect(username='username', password='password',
+               service='api.clickzetta.com', instance='instance',
+               workspace='quickstart_ws', schema='public', vcluster='default')
+
+bulkload_stream = conn.create_bulkload_stream(schema='public', table='bulkload_test')
+stream_id = bulkload_stream.get_stream_id()
+
+# 启动多个写入进程，每个进程用不同的 writer_id
+p1 = subprocess.Popen(['python', 'writer.py', stream_id, '1'])
+p2 = subprocess.Popen(['python', 'writer.py', stream_id, '2'])
+p1.wait()
+p2.wait()
+
+bulkload_stream.commit()  # 所有 writer 完成后统一 commit
+```
+
+### 写入进程
+
+```python
+import sys
+from clickzetta import connect
+
+conn = connect(username='username', password='password',
+               service='api.clickzetta.com', instance='instance',
+               workspace='quickstart_ws', schema='public', vcluster='default')
+
+stream_id = sys.argv[1]
+writer_id = int(sys.argv[2])
+
+# 通过 stream_id 获取已有 stream（不创建新的）
+bulkload_stream = conn.get_bulkload_stream(
+    schema='public', table='bulkload_test', stream_id=stream_id
+)
+
+writer = bulkload_stream.open_writer(writer_id)  # writer_id 必须唯一
+for index in range(1, 1000000):
+    row = writer.create_row()
+    row.set_value('i', index)
+    row.set_value('s', 'Hello')
+    row.set_value('d', 123.456)
+    writer.write(row)
+writer.close()
+# 写入进程不调用 commit，只有控制进程调用
+```
+
+## 关键 API
+
+| API | 说明 |
+|---|---|
+| `conn.create_bulkload_stream(schema, table)` | 创建新的 bulkload stream |
+| `conn.get_bulkload_stream(schema, table, stream_id)` | 获取已有 stream（分布式写入用） |
+| `bulkload_stream.get_stream_id()` | 获取 stream id（传给写入进程） |
+| `bulkload_stream.open_writer(writer_id)` | 创建 writer，id 必须唯一 |
+| `writer.create_row()` | 创建行对象 |
+| `row.set_value(column_name, value)` | 按列名设值 |
+| `writer.write(row)` | 写入行（必须调用） |
+| `writer.close()` | 关闭 writer（写完必须调用） |
+| `bulkload_stream.commit()` | 提交，数据可见（只能调用一次） |

package/bin/skills/clickzetta-app-python-sdk/references/connector.md

@@ -0,0 +1,143 @@
+# clickzetta-connector-python 详细参考
+
+## 安装
+
+```bash
+pip install clickzetta-connector-python -U
+```
+
+## 建立连接
+
+```python
+from clickzetta import connect
+
+conn = connect(
+    username='your_username',
+    password='your_password',
+    service='api.clickzetta.com',
+    instance='your_instance',
+    workspace='your_workspace',
+    schema='public',
+    vcluster='default'
+)
+```
+
+## 基本查询
+
+```python
+cursor = conn.cursor()
+cursor.execute('SELECT * FROM orders LIMIT 10')
+results = cursor.fetchall()
+for row in results:
+    print(row)
+cursor.close()
+conn.close()
+```
+
+## 参数绑定
+
+支持两种风格（PEP-249 规范）：
+
+### qmark 风格（推荐）
+
+```python
+# 单行插入
+cursor.execute('INSERT INTO test (id, name) VALUES (?, ?)', binding_params=[1, 'test'])
+
+# 批量插入（executemany）
+data = [
+    (1, 'test1'),
+    (2, 'test2'),
+    (3, 'test3')
+]
+cursor.executemany('INSERT INTO test (id, name) VALUES (?, ?)', data)
+```
+
+### pyformat 风格
+
+```python
+data = {'id': 1, 'name': 'test'}
+cursor.execute('INSERT INTO test (id, name) VALUES (%(id)s, %(name)s)', data)
+```
+
+## SQL hints（超时控制等）
+
+```python
+my_param = {
+    'hints': {
+        'sdk.job.timeout': 30    # 查询超时秒数
+    }
+}
+cursor.execute('SELECT * FROM large_table', my_param)
+```
+
+## 异步执行（长时间查询）
+
+```python
+import time
+
+cursor.execute_async('SELECT * FROM large_table')
+
+while not cursor.is_job_finished():
+    print("查询执行中...")
+    time.sleep(1)
+
+results = cursor.fetchall()
+```
+
+## 结果保存到 CSV
+
+```python
+import csv
+
+cursor.execute('SELECT * FROM orders LIMIT 1000')
+results = cursor.fetchall()
+
+with open('output.csv', 'w', newline='', encoding='utf-8') as f:
+    writer = csv.writer(f)
+    writer.writerow([col[0] for col in cursor.description])
+    writer.writerows(results)
+
+cursor.close()
+conn.close()
+```
+
+## SQLAlchemy 集成
+
+```python
+from sqlalchemy import create_engine, text
+
+engine = create_engine(
+    "clickzetta://username:password@instance.api.clickzetta.com/workspace"
+    "?schema=public&vcluster=default"
+)
+
+with engine.connect() as conn:
+    result = conn.execute(text("SELECT * FROM orders LIMIT 10"))
+    for row in result:
+        print(row)
+```
+
+### SQLAlchemy + pandas
+
+```python
+import pandas as pd
+from sqlalchemy import create_engine, text
+
+engine = create_engine(
+    "clickzetta://username:password@instance.api.clickzetta.com/workspace"
+    "?schema=public&vcluster=default"
+)
+
+with engine.connect() as conn:
+    result = conn.execute(text("SELECT * FROM orders"))
+    df = pd.DataFrame(result.fetchall(), columns=result.keys())
+
+print(df.head())
+```
+
+## 注意事项
+
+- 不支持 `commit()` 和 `rollback()` 接口
+- 需要 `clickzetta-connector-python >= 0.8.82` 才能使用参数绑定和异步执行
+- 旧版 `clickzetta-connector` 已停止维护，请迁移到 `clickzetta-connector-python`

package/bin/skills/clickzetta-app-python-sdk/references/realtime.md

@@ -0,0 +1,122 @@
+# clickzetta-ingestion-python-v2 实时写入详细参考
+
+## 安装
+
+```bash
+pip install clickzetta-ingestion-python-v2
+```
+
+> 注意：这是独立的包，与 `clickzetta-ingestion-python`（BulkLoad）不同。
+
+## 与 BulkLoad 的区别
+
+| 特性 | BulkLoad (`ingestion-python`) | 实时写入 (`ingestion-python-v2`) |
+|---|---|---|
+| 写入延迟 | 分钟级（commit 后可见） | 秒级可查 |
+| 适用频率 | 间隔 ≥ 5 分钟 | 高频小批（5 分钟以内） |
+| 主键表支持 | ❌ 不支持 | ✅ 支持（CDC 模式） |
+| UPDATE/DELETE | ❌ | ✅ UPSERT / DELETE_IGNORE |
+| 写入原理 | 上传对象存储 → commit 导入 | 直写 Ingestion Service |
+
+## 工作原理
+
+```
+[SDK 写入] → [Ingestion Service] → [混合表（秒级可查）] → [~1分钟后自动 commit] → [普通表（stream/DT 可见）]
+```
+
+- 写入后秒级即可 SELECT 查到数据
+- table stream、materialized view、dynamic table 需等约 1 分钟（commit 后）才能看到
+- commit 后后台合并为普通表，之后可执行 UPDATE/MERGE/DELETE
+
+## 普通表写入（APPEND_ONLY）
+
+```python
+from clickzetta.connector.v0.connection import connect
+from clickzetta.connector.v0.enums import RealtimeOperation
+from clickzetta_ingestion.realtime.realtime_options import RealtimeOptionsBuilder, FlushMode
+from clickzetta_ingestion.realtime.arrow_stream import RowOperator
+
+with connect(
+    username='your_username',
+    password='your_password',
+    service='your_service_endpoint',
+    instance='your_instance',
+    workspace='your_workspace',
+    schema='your_schema',
+    vcluster='default'
+) as conn:
+    stream = conn.get_realtime_stream(
+        schema='your_schema',
+        table='your_table',
+        operate=RealtimeOperation.APPEND_ONLY,
+        options=RealtimeOptionsBuilder()
+            .with_flush_mode(FlushMode.AUTO_FLUSH_BACKGROUND)
+            .build()
+    )
+
+    for i in range(1000):
+        row = stream.create_row(RowOperator.INSERT)
+        row.set_value('id', str(i))
+        row.set_value('name', f'user_{i}')
+        stream.apply(row)
+
+    stream.close()
+```
+
+## 主键表写入（CDC 模式）
+
+主键表**必须**使用 `RealtimeOperation.CDC` + `FlushMode.AUTO_FLUSH_SYNC`。
+
+```python
+from clickzetta.connector.v0.connection import connect
+from clickzetta.connector.v0.enums import RealtimeOperation
+from clickzetta_ingestion.realtime.realtime_options import RealtimeOptionsBuilder, FlushMode
+from clickzetta_ingestion.realtime.arrow_stream import RowOperator
+
+with connect(...) as conn:
+    stream = conn.get_realtime_stream(
+        schema='your_schema',
+        table='your_pk_table',
+        operate=RealtimeOperation.CDC,
+        options=RealtimeOptionsBuilder()
+            .with_flush_mode(FlushMode.AUTO_FLUSH_SYNC)  # PK 表强制同步刷写
+            .build()
+    )
+
+    # UPSERT：存在则更新，不存在则插入
+    row = stream.create_row(RowOperator.UPSERT)
+    row.set_value('id', 'id_1')
+    row.set_value('name', 'alice')
+    row.set_value('age', 30)
+    stream.apply(row)
+
+    # DELETE_IGNORE：删除，目标行不存在时自动忽略
+    row = stream.create_row(RowOperator.DELETE_IGNORE)
+    row.set_value('id', 'id_1')
+    stream.apply(row)
+
+    stream.close()
+```
+
+## 操作类型对照
+
+| Stream 模式 | 可用 RowOperator | 适用表类型 |
+|---|---|---|
+| `APPEND_ONLY` | `INSERT` | 普通表 |
+| `CDC` | `UPSERT`、`DELETE_IGNORE` | 主键表（必须） |
+
+## FlushMode 说明
+
+| 模式 | 说明 | 适用场景 |
+|---|---|---|
+| `AUTO_FLUSH_BACKGROUND` | 异步刷写，高吞吐 | 普通表，对顺序无要求 |
+| `AUTO_FLUSH_SYNC` | 同步刷写，阻塞式 | 主键表（强制），需保证顺序 |
+| `MANUAL_FLUSH` | 手动调用 `stream.flush()` | 精确控制刷写时机 |
+
+> ⚠️ 主键表不支持 `AUTO_FLUSH_BACKGROUND`，会自动重置为 `AUTO_FLUSH_SYNC`。
+
+## 关键注意事项
+
+- 表结构变更前需先停止实时写入任务，变更后约 90 分钟再重启（Flink Connector 的 schema change sink 除外）
+- 分区列必须是 primary key 的子集
+- 避免 `flush()` 过于频繁，会产生大量小文件