@clickzetta/cz-cli-darwin-x64 0.3.39 → 0.3.41

package/bin/skills/clickzetta-zettapark/SKILL.md

@@ -0,0 +1,248 @@
+---
+name: clickzetta-zettapark
+description: |
+  使用 ZettaPark Python 库操作 ClickZetta Lakehouse 数据。ZettaPark 提供类 pandas 的
+  DataFrame API，将 Python 操作翻译为 SQL 在 Lakehouse 中分布式执行。
+  覆盖 Session 创建、DataFrame 构建与转换（filter/select/join/groupBy）、
+  结果收集（collect/to_pandas/show）、写入表（save_as_table）、
+  文件操作（PUT/GET）、执行 SQL 等完整工作流。
+  当用户说"ZettaPark"、"zettapark"、"DataFrame API"、"Python 操作 Lakehouse"、
+  "save_as_table"、"session.table"、"session.sql"、"collect()"、"to_pandas"、
+  "Python 数据工程"、"Python 写入 Lakehouse"、"Python 读取 Lakehouse"、
+  "clickzetta_zettapark_python"时触发。
+  Keywords: ZettaPark, DataFrame, pandas-like, Python, SQL translation, distributed compute
+---
+
+# ClickZetta ZettaPark
+
+ZettaPark 是 ClickZetta Lakehouse 的 Python DataFrame 框架，将 Python 操作翻译为 SQL 在 Lakehouse 中分布式执行，提供类 pandas 的开发体验。
+
+阅读 [references/zettapark-api.md](references/zettapark-api.md) 了解完整 API。
+
+## 安装
+
+> ⚠️ **Python 版本要求**：推荐 **Python 3.12**（最低 3.10，不支持 3.9 及以下）
+
+```bash
+# 方式 1：venv（Python 内置，推荐）
+python3.12 -m venv .venv
+source .venv/bin/activate   # macOS/Linux  |  .venv\Scripts\activate (Windows)
+pip install clickzetta_zettapark_python -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+# 方式 2：pyenv（需要切换 Python 版本时）
+pyenv install 3.12.9 && pyenv local 3.12.9
+python -m venv .venv && source .venv/bin/activate
+pip install clickzetta_zettapark_python -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+# 方式 3：conda（数据科学环境）
+conda create -n lakehouse python=3.12 -y && conda activate lakehouse
+pip install clickzetta_zettapark_python -i https://pypi.tuna.tsinghua.edu.cn/simple
+```
+
+---
+
+## 创建会话
+
+```python
+from clickzetta.zettapark.session import Session
+
+connection_parameters = {
+    "username": "your_username",
+    "password": "your_password",
+    "service": "cn-shanghai-alicloud.api.clickzetta.com",
+    "instance": "your_instance_id",
+    "workspace": "your_workspace",
+    "schema": "public",
+    "vcluster": "default_ap",
+}
+
+session = Session.builder.configs(connection_parameters).create()
+
+# 验证连接
+session.sql("SELECT current_user(), current_workspace()").show()
+```
+
+---
+
+## 核心工作流
+
+### 读取数据
+
+```python
+from clickzetta.zettapark import functions as F
+
+# 从表读取
+df = session.table("orders")
+df = session.table("my_schema.orders")
+
+# 从 SQL 读取
+df = session.sql("SELECT * FROM orders WHERE year = 2024")
+
+# 从 Python 数据创建
+df = session.create_dataframe([[1, "Alice", 100.0], [2, "Bob", 200.0]],
+                               schema=["id", "name", "amount"])
+```
+
+### 转换数据
+
+```python
+# 过滤、选择、新增列
+result = (
+    session.table("orders")
+    .filter(F.col("status") == "completed")
+    .select("order_id", "customer_id", "amount")
+    .with_column("tax", F.col("amount") * 0.1)
+    .sort(F.col("amount").desc())
+    .limit(100)
+)
+```
+
+### 聚合
+
+```python
+summary = (
+    session.table("orders")
+    .group_by("category")
+    .agg(
+        F.sum("amount").as_("total"),
+        F.count("*").as_("cnt"),
+        F.avg("amount").as_("avg_amount"),
+    )
+)
+summary.show()
+```
+
+### JOIN
+
+```python
+orders = session.table("orders")
+customers = session.table("customers")
+
+result = orders.join(
+    customers,
+    orders["customer_id"] == customers["id"],
+    "left"
+).select(
+    orders["order_id"],
+    customers["name"],
+    orders["amount"]
+)
+```
+
+### 写入数据
+
+```python
+# 追加到已有表
+df.write.save_as_table("result_table", mode="append")
+
+# 覆盖写入（自动建表）
+df.write.save_as_table("result_table", mode="overwrite")
+```
+
+### 获取结果
+
+```python
+# 打印预览
+df.show(20)
+
+# 收集为 Row 列表
+rows = df.collect()
+for row in rows:
+    print(row["id"], row["name"])
+
+# 转为 Pandas DataFrame（小数据量）
+pandas_df = df.to_pandas()
+
+# 获取行数
+print(df.count())
+```
+
+---
+
+## 典型场景
+
+### 场景 1：ETL 数据处理
+
+```python
+from clickzetta.zettapark.session import Session
+from clickzetta.zettapark import functions as F
+
+session = Session.builder.configs(config).create()
+
+# 读取原始数据
+raw = session.table("bronze.raw_orders")
+
+# 清洗转换
+cleaned = (
+    raw
+    .filter(F.isnotnull(F.col("order_id")))
+    .filter(F.col("amount") > 0)
+    .with_column("order_date", F.col("created_at").cast("DATE"))
+    .with_column("year_month", F.date_format(F.col("order_date"), "yyyy-MM"))
+    .select("order_id", "customer_id", "amount", "order_date", "year_month")
+)
+
+# 写入 Silver 层
+cleaned.write.save_as_table("silver.orders_cleaned", mode="overwrite")
+
+session.close()
+```
+
+### 场景 2：特征工程（机器学习）
+
+```python
+from clickzetta.zettapark import functions as F
+
+customer = session.table("clickzetta_sample_data.tpch_100g.customer")
+orders = session.table("clickzetta_sample_data.tpch_100g.orders")
+
+# 构建客户消费特征
+customer_features = (
+    orders
+    .group_by("o_custkey")
+    .agg(
+        F.sum("o_totalprice").as_("total_spend"),
+        F.count("*").as_("order_count"),
+        F.avg("o_totalprice").as_("avg_order_value"),
+        F.max("o_orderdate").as_("last_order_date"),
+    )
+    .join(customer, orders["o_custkey"] == customer["c_custkey"])
+    .select("c_custkey", "c_name", "total_spend", "order_count", "avg_order_value")
+)
+
+customer_features.write.save_as_table("ml_features.customer_features", mode="overwrite")
+```
+
+### 场景 3：从本地文件导入
+
+```python
+import json
+import gzip
+from clickzetta.zettapark.session import Session
+
+session = Session.builder.configs(config).create()
+
+# 读取本地 JSON 数据
+data = []
+with gzip.open('data.json.gz', 'rt', encoding='utf-8') as f:
+    for line in f:
+        if line.strip():
+            data.append(json.loads(line))
+
+# 创建 DataFrame 并写入
+df = session.create_dataframe(data)
+df.write.save_as_table("my_table", mode="overwrite")
+
+session.close()
+```
+
+---
+
+## 常见问题
+
+| 问题 | 原因 | 解决方案 |
+|---|---|---|
+| `collect()` 超时 | 数据量过大或集群规格不足 | 增大 `sdk.job.timeout`，或先 `limit()` 测试 |
+| `to_pandas()` 内存溢出 | 结果集过大 | 先聚合/过滤再转 pandas，或分批处理 |
+| 列名冲突（JOIN 后） | 两表有同名列 | 用 `df_left["col"]` 明确指定来源 |
+| `save_as_table` 报错 | 表已存在且 mode 不对 | 使用 `mode="overwrite"` 或 `mode="append"` |

package/bin/skills/clickzetta-zettapark/eval_cases.jsonl

@@ -0,0 +1,12 @@
+{"case_id":"001","type":"should_call","user_input":"用 ZettaPark 读取 orders 表并过滤 amount > 100","expected_skill":"clickzetta-zettapark","expected_output_contains":["session.table","filter"]}
+{"case_id":"002","type":"should_call","user_input":"ZettaPark 怎么安装？需要什么 Python 版本？","expected_skill":"clickzetta-zettapark","expected_output_contains":["pip install","3.12"]}
+{"case_id":"003","type":"should_call","user_input":"怎么用 DataFrame API 做 group by 聚合","expected_skill":"clickzetta-zettapark","expected_output_contains":["group_by","agg"]}
+{"case_id":"004","type":"should_call","user_input":"save_as_table 怎么用？支持哪些写入模式？","expected_skill":"clickzetta-zettapark","expected_output_contains":["save_as_table","overwrite","append"]}
+{"case_id":"005","type":"should_call","user_input":"ZettaPark 怎么把结果转成 pandas DataFrame","expected_skill":"clickzetta-zettapark","expected_output_contains":["to_pandas"]}
+{"case_id":"006","type":"should_call","user_input":"用 session.sql 执行一段 SQL 查询","expected_skill":"clickzetta-zettapark","expected_output_contains":["session.sql"]}
+{"case_id":"007","type":"should_call","user_input":"ZettaPark 怎么 join 两张表","expected_skill":"clickzetta-zettapark","expected_output_contains":["join"]}
+{"case_id":"008","type":"should_not_call","user_input":"帮我写一个 Flask Web 应用","forbidden_skill":"clickzetta-zettapark"}
+{"case_id":"009","type":"should_not_call","user_input":"pandas 怎么读取 CSV 文件","forbidden_skill":"clickzetta-zettapark"}
+{"case_id":"010","type":"should_not_call","user_input":"怎么用 JDBC 连接 Lakehouse","forbidden_skill":"clickzetta-zettapark"}
+{"case_id":"011","type":"should_not_call","user_input":"帮我创建一个 VCluster","forbidden_skill":"clickzetta-zettapark"}
+{"case_id":"012","type":"should_not_call","user_input":"Spark DataFrame 怎么用","forbidden_skill":"clickzetta-zettapark"}

package/bin/skills/clickzetta-zettapark/references/zettapark-api.md

@@ -0,0 +1,283 @@
+# ZettaPark 快速参考
+
+> 来源：https://www.yunqi.tech/documents/ZettaparkQuickStart
+
+## 安装
+
+```bash
+pip install clickzetta_zettapark_python -U -i https://pypi.tuna.tsinghua.edu.cn/simple
+```
+
+---
+
+## 创建会话
+
+```python
+from clickzetta.zettapark.session import Session
+
+connection_parameters = {
+    "username": "your_username",
+    "password": "your_password",
+    "service": "cn-shanghai-alicloud.api.clickzetta.com",
+    "instance": "your_instance_id",
+    "workspace": "your_workspace",
+    "schema": "public",
+    "vcluster": "default_ap",
+}
+
+session = Session.builder.configs(connection_parameters).create()
+```
+
+带 hints（超时、query_tag 等）：
+
+```python
+connection_parameters = {
+    "username": "your_username",
+    "password": "your_password",
+    "service": "cn-shanghai-alicloud.api.clickzetta.com",
+    "instance": "your_instance_id",
+    "workspace": "your_workspace",
+    "schema": "public",
+    "vcluster": "default_ap",
+    "hints": {
+        "sdk.job.timeout": 300,
+        "query_tag": "my_zettapark_app",
+    }
+}
+
+session = Session.builder.configs(connection_parameters).create()
+```
+
+从 JSON 配置文件读取：
+
+```python
+import json
+with open('config.json', 'r') as f:
+    config = json.load(f)
+session = Session.builder.configs(config).create()
+```
+
+验证连接：
+
+```python
+session.sql("SELECT current_user(), current_workspace(), current_vcluster()").show()
+```
+
+关闭会话：
+
+```python
+session.close()
+```
+
+---
+
+## 构建 DataFrame
+
+```python
+# 从表创建
+df = session.table("my_schema.my_table")
+
+# 从 SQL 创建
+df = session.sql("SELECT * FROM orders WHERE year = 2024")
+
+# 从 Python 数据创建
+df = session.create_dataframe([1, 2, 3, 4]).to_df("id")
+df = session.create_dataframe([[1, "Alice"], [2, "Bob"]], schema=["id", "name"])
+
+# 从 Row 对象创建
+from clickzetta.zettapark import Row
+df = session.create_dataframe([Row(id=1, name="Alice"), Row(id=2, name="Bob")])
+
+# 带 Schema 创建
+from clickzetta.zettapark.types import IntegerType, StringType, StructType, StructField
+schema = StructType([StructField("id", IntegerType()), StructField("name", StringType())])
+df = session.create_dataframe([[1, "Alice"], [2, "Bob"]], schema)
+
+# 范围序列
+df = session.range(1, 10, 2).to_df("n")  # 1,3,5,7,9
+```
+
+---
+
+## DataFrame 转换操作
+
+```python
+from clickzetta.zettapark import functions as F
+
+# 过滤行
+df.filter(F.col("age") > 18)
+df.filter(F.col("status") == "active")
+df.where(F.col("amount") > 1000)
+
+# 选择列
+df.select("id", "name", "amount")
+df.select(F.col("id"), F.col("name").as_("user_name"))
+
+# 新增/修改列
+df.with_column("total", F.col("price") * F.col("qty"))
+df.with_column("upper_name", F.upper(F.col("name")))
+
+# 重命名列
+df.rename(F.col("old_name"), "new_name")
+
+# 排序
+df.sort(F.col("amount").desc())
+df.order_by(F.col("created_at").asc())
+
+# 去重
+df.distinct()
+df.drop_duplicates(["user_id"])
+
+# 限制行数
+df.limit(100)
+
+# 删除列
+df.drop("unnecessary_col")
+```
+
+---
+
+## 聚合操作
+
+```python
+from clickzetta.zettapark import functions as F
+
+# 分组聚合
+df.group_by("category").agg(
+    F.sum("amount").as_("total_amount"),
+    F.count("*").as_("order_count"),
+    F.avg("price").as_("avg_price"),
+    F.max("amount").as_("max_amount"),
+    F.min("amount").as_("min_amount"),
+)
+
+# 全局聚合
+df.agg(F.count("*"), F.sum("amount"))
+```
+
+---
+
+## JOIN 操作
+
+```python
+# 内连接
+df_orders.join(df_customers, df_orders["customer_id"] == df_customers["id"])
+
+# 左连接
+df_orders.join(df_customers, df_orders["customer_id"] == df_customers["id"], "left")
+
+# 选择连接后的列（避免列名冲突）
+result = df_orders.join(df_customers, df_orders["customer_id"] == df_customers["id"]) \
+    .select(df_orders["order_id"], df_customers["name"], df_orders["amount"])
+```
+
+---
+
+## 执行与结果获取
+
+```python
+# 打印前 N 行（触发执行）
+df.show()
+df.show(20)
+
+# 收集所有结果为 Row 列表
+rows = df.collect()
+for row in rows:
+    print(row["id"], row["name"])
+
+# 转换为 Pandas DataFrame
+pandas_df = df.to_pandas()
+
+# 获取行数
+count = df.count()
+
+# 获取列名
+print(df.columns)
+
+# 查看 Schema
+df.schema.print_tree()
+```
+
+---
+
+## 写入数据
+
+```python
+# 写入已有表（追加）
+df.write.save_as_table("my_table", mode="append")
+
+# 覆盖写入
+df.write.save_as_table("my_table", mode="overwrite")
+
+# 自动建表并写入（overwrite 会重建表）
+df.write.save_as_table("new_table", mode="overwrite")
+
+# 写入指定 Schema 下的表
+df.write.save_as_table("my_schema.my_table", mode="append")
+```
+
+---
+
+## 执行 SQL
+
+```python
+# 执行 DDL/DML
+session.sql("CREATE TABLE IF NOT EXISTS t (id INT, name STRING)").collect()
+session.sql("INSERT INTO t VALUES (1, 'Alice')").collect()
+
+# 执行查询并获取 DataFrame
+df = session.sql("SELECT * FROM orders WHERE amount > 1000")
+df.show()
+
+# 切换 Schema
+session.use_schema("my_schema")
+```
+
+---
+
+## 文件操作（Volume）
+
+```python
+# 上传文件到 User Volume
+session.file.put("/local/path/data.csv", "volume:user://~/data/")
+
+# 下载文件
+session.file.get("volume:user://~/data/data.csv", "/local/output/")
+
+# 列出 User Volume 文件
+session.sql("LIST USER VOLUME").show()
+session.sql("SHOW USER VOLUME DIRECTORY").show()
+```
+
+---
+
+## 常用 functions 速查
+
+```python
+from clickzetta.zettapark import functions as F
+
+# 字符串
+F.upper(col), F.lower(col), F.concat(col1, col2)
+F.substring(col, 1, 3), F.trim(col), F.length(col)
+
+# 数值
+F.abs(col), F.round(col, 2), F.floor(col), F.ceil(col)
+F.sqrt(col), F.pow(col, 2)
+
+# 日期时间
+F.current_date(), F.current_timestamp()
+F.year(col), F.month(col), F.day(col)
+F.date_add(col, 7), F.datediff(col1, col2)
+
+# 条件
+F.when(F.col("status") == "A", "Active").otherwise("Inactive")
+F.coalesce(col1, col2)  # 第一个非 null 值
+F.isnull(col), F.isnotnull(col)
+
+# 聚合
+F.count("*"), F.sum(col), F.avg(col), F.max(col), F.min(col)
+F.count_distinct(col)
+
+# 类型转换
+F.col("amount").cast(IntegerType())
+```

package/bin/skills/cz-cli-inner/SKILL.md

@@ -32,14 +32,14 @@ cz-cli workspace current                      Show current workspace
 
 cz-cli task list                              List Studio tasks
 cz-cli task create <name> --type <TYPE>       Create task (SQL/PYTHON/SHELL/SPARK/FLOW)
-cz-cli task content <task>                    Get task script and config
-cz-cli task save-content <task> --file <f>    Save task script
+cz-cli task content <task>                    Get task script, config and params (draft)
+cz-cli task save-content <task> --file <f>    Save task script; use --params '{"key":"val","dt":"bizdate","yd":"$[yyyy-MM-dd,-1d]"}' to set params (system params like bizdate/sys_plan_day/sys_biz_datetime etc. auto-detected)
 cz-cli task save-config <task>                Save task non-cron config, like retry, dependency
 cz-cli task save-cron <task>                  Save task schedule config
 cz-cli task deps <task>                       Show task dependencies (draft)
 cz-cli task deploy <task>                     Publish/deploy a task (alias: online)
 cz-cli task undeploy <task>                   Undeploy a task, irreversible (alias: offline)
-cz-cli task execute <task>                    Execute ad-hoc
+cz-cli task execute <task>                    Execute ad-hoc; auto-uses saved manual params as defaults (--param KEY=VAL overrides); warns if unresolved ${placeholders} remain (SQL will fail, Python/Shell silently keeps literal string)
 cz-cli task delete <task>                     Delete draft/offline task
 cz-cli task flow dag <task>                   Get flow DAG
 cz-cli task flow create-node <task>           Add node to flow
@@ -58,7 +58,7 @@ cz-cli runs wait <id>                         Poll until complete
 cz-cli runs logs <id>                         Get execution log
 cz-cli runs deps <task>                       Published dependencies
 cz-cli runs stop <id>                         Stop a running instance
-cz-cli runs refill <task>                     Backfill date range
+cz-cli runs refill <task> --from D --to D    Backfill date range. D accepts YYYY-MM-DD (day boundary) or YYYY-MM-DDTHH:MM:SS (exact datetime for hourly/minutely tasks)
 cz-cli runs rerun <id>                        Rerun failed instance
 cz-cli runs stats                             Run statistics summary
 
@@ -100,6 +100,7 @@ cz-cli datasource sample <name_or_id> <catalog> <object>
 6. **State-changing operations** (deploy/undeploy/execute/delete/refill): confirm intent with user first.
 7. **Multi-environment**: use `--profile <name>` to target a specific environment.
 8. **On `NO_PROFILE` error**: guide user to run `cz-cli setup`.
+9. **补数/回填/重跑历史数据 → `runs refill`**: when user says "补数", "回填", "重跑历史", "backfill", "re-run historical data", use `cz-cli runs refill <task> --from YYYY-MM-DD --to YYYY-MM-DD`. This is under `runs`, NOT `task`.
 
 ## Companion Skills
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@clickzetta/cz-cli-darwin-x64",
-  "version": "0.3.39",
+  "version": "0.3.41",
   "description": "cz-cli binary for macOS x64 (Intel)",
   "os": ["darwin"],
   "cpu": ["x64"],