@clickzetta/cz-cli-darwin-x64 0.3.1

package/bin/skills/lakehouse-doc/references/sql_data_transform_cte.md

@@ -0,0 +1,177 @@
+# 通过CTE进行数据转换
+
+让我们先了解一下使用Lakehouse SQL的[公用表表达式（Common Table Expression, CTE）](WITH.md)进行数据转换的基本概念、优势以及常见的使用场景。
+
+## 基本概念
+
+公用表表达式（CTE）是一种可以在SQL查询中定义临时结果集的表达式。该结果集在查询的执行过程中存在，通常用于简化复杂查询、递归查询或分解查询步骤。CTE通常由[WITH关键字](WITH.md)引入，并可以在随后的`SELECT`、`INSERT`、`UPDATE`和`DELETE`语句中使用。
+
+基本语法：
+
+```sql
+WITH cte_name AS (
+    SELECT column1, column2, ...
+    FROM table_name
+    WHERE condition
+)
+SELECT * FROM cte_name;
+```
+
+## 优势
+
+1. **可读性高**：CTE可以使复杂查询更具可读性和可维护性。通过将复杂的查询逻辑拆解为多个可理解的部分，CTE增强了查询的结构化和清晰度。CTE 使测试复杂查询更简单
+
+   * CTE 是一个可以在单个查询中重用的 `SELECT` 语句。
+   * 复杂的 SQL 查询通常涉及多个子查询。多个子查询会使代码难以阅读。
+   * 使用公共表表达式（CTE）可以使你的查询更易读。
+
+2. **复用性强**：CTE可以在同一查询中多次引用，从而避免重复代码，提高查询的复用性。
+
+3. **步骤化数据转换**：CTE可以将数据转换过程分步骤实现，每个步骤都可以定义为一个独立的CTE，这样便于调试和测试。
+
+## 使用场景
+
+以下是一些利用CTE进行数据转换的常见场景：
+
+### 1. 数据清洗和转换
+
+通过CTE可以分步骤清洗和转换数据，例如去除重复值、修正数据格式：
+
+```sql
+WITH cleaned_data AS (
+    SELECT DISTINCT column1, column2
+    FROM raw_table
+    WHERE column1 IS NOT NULL
+),
+transformed_data AS (
+    SELECT column1, UPPER(column2) AS transformed_column2
+    FROM cleaned_data
+)
+SELECT * FROM transformed_data;
+```
+
+### 2. 分组和聚合
+
+CTE可以用于进行复杂的分组和聚合操作，并在后续查询中引用聚合结果：
+
+```sql
+WITH total_sales AS (
+    SELECT customer_id, SUM(order_amount) AS total_spent
+    FROM orders
+    GROUP BY customer_id
+)
+SELECT customer_id, total_spent
+FROM total_sales
+WHERE total_spent > 1000;
+```
+
+### 3. 数据合并和连接
+
+使用CTE，可以简化多表连接和数据合并操作：
+
+```sql
+WITH customer_orders AS (
+    SELECT c.customer_id, c.customer_name, o.order_id, o.order_date
+    FROM customers c
+    JOIN orders o ON c.customer_id = o.customer_id
+)
+SELECT *
+FROM customer_orders
+WHERE order_date > '2024-01-01';
+```
+
+## 数据模型
+
+TPC-H 数据代表一个汽车零部件销售商的数据仓库，其中记录了订单、构成订单的项目（lineitem）、供应商、客户、销售的零部件（part）、地区、国家和零部件供应商（partsupp）。
+
+云器Lakehouse内置了共享的TPC-H数据，每个用户可以通过加上数据上下文直接使用，比如：
+
+```sql
+SELECT * FROM 
+clickzetta_sample_data.tpch_100g.customer
+LIMIT 10;
+```
+
+## 通过云器Lakehouse SQL CTE进行数据转换
+
+### 如何定义 CTE
+
+```sql
+-- CTE 定义
+WITH
+  supplier_nation_metrics AS ( -- 使用 WITH 关键字定义 CTE 1
+    SELECT
+      n.n_nationkey,
+      SUM(l.l_QUANTITY) AS num_supplied_parts
+    FROM
+      clickzetta_sample_data.tpch_100g.lineitem l
+      JOIN clickzetta_sample_data.tpch_100g.supplier s ON l.l_suppkey = s.s_suppkey
+      JOIN clickzetta_sample_data.tpch_100g.nation n ON s.s_nationkey = n.n_nationkey
+    GROUP BY
+      n.n_nationkey
+  ),
+  buyer_nation_metrics AS ( -- 定义 CTE 2
+    SELECT
+      n.n_nationkey,
+      SUM(l.l_QUANTITY) AS num_purchased_parts
+    FROM
+      clickzetta_sample_data.tpch_100g.lineitem l
+      JOIN clickzetta_sample_data.tpch_100g.orders o ON l.l_orderkey = o.o_orderkey
+      JOIN clickzetta_sample_data.tpch_100g.customer c ON o.o_custkey = c.c_custkey
+      JOIN clickzetta_sample_data.tpch_100g.nation n ON c.c_nationkey = n.n_nationkey
+    GROUP BY
+      n.n_nationkey
+  )
+SELECT -- 最终的 SELECT 语句前不需要逗号
+  n.n_name AS nation_name,
+  s.num_supplied_parts,
+  b.num_purchased_parts
+FROM
+  clickzetta_sample_data.tpch_100g.nation n
+  LEFT JOIN supplier_nation_metrics s ON n.n_nationkey = s.n_nationkey
+  LEFT JOIN buyer_nation_metrics b ON n.n_nationkey = b.n_nationkey
+LIMIT 10;
+```
+
+### 计算因折扣而损失的金额
+
+使用 `lineitem` 表获取订单中项目的价格（不含折扣），并与订单进行比较。先确定比较需要进行的粒度。分步骤思考，即先获取订单中所有项目的不含折扣的价格，然后将其与已计算折扣的 `totalprice` 的订单数据进行比较。
+
+```sql
+WITH lineitem_agg AS (
+    SELECT 
+        l_orderkey,
+        SUM(l_extendedprice) AS total_price_without_discount
+    FROM 
+        clickzetta_sample_data.tpch_100g.lineitem
+    GROUP BY 
+        l_orderkey
+)
+SELECT 
+    o.o_orderkey,
+    o.o_totalprice, 
+    l.total_price_without_discount - o.o_totalprice AS amount_lost_to_discount
+FROM 
+    clickzetta_sample_data.tpch_100g.orders o
+JOIN 
+    lineitem_agg l ON o.o_orderkey = l.l_orderkey
+ORDER BY 
+    o.o_orderkey;
+```
+
+### 不要过度使用 CTE。注意代码的可读性。
+
+1. CTE 可以帮助提高查询的可读性和可重用性。
+
+2. 不要过度使用 CTE；注意查询的大小。
+
+   * 多个临时表的 SQL 查询比带有众多 CTE 的 1000 行 SQL 查询更好。
+   * 保持每个查询中的 CTE 数量较小（取决于查询的大小，但通常 < 5）。
+
+3. 如果对CTE 的性有疑问，请检查你的查询计划。
+
+## 资料
+
+[公用表表达式（Common Table Expression, CTE）](WITH.md)
+
+^

package/bin/skills/lakehouse-doc/references/sql_data_transform_tips.md

@@ -0,0 +1,407 @@
+# 通过SQL进行数据转换的一些提示
+
+## 数据模型
+
+TPC-H 数据代表一个汽车零部件销售商的数据仓库，其中记录了订单、构成订单的项目（lineitem）、供应商、客户、销售的零部件（part）、地区、国家和零部件供应商（partsupp）。
+
+云器Lakehouse内置了共享的TPC-H数据，每个用户可以通过加上数据上下文直接使用，比如：
+
+```sql
+SELECT * FROM 
+clickzetta_sample_data.tpch_100g.customer
+LIMIT 10;
+```
+
+## 前提条件
+
+1. [基础](sql_data_transform_basic.md)
+2. [公共表表达式 (CTE)](sql_data_transform_cte.md)
+3. [窗口函数](sql_data_transform_windows.md)
+4. [嵌套数据类型](sql_data_transfom_NestedDataTypes.md)
+
+## 1. 常见数据处理场景的实用函数
+
+### 1.1. 需要分区中的第一/最后一行时使用 ROW\_NUMBER
+
+```sql
+SELECT 
+    o_custkey, 
+    o_orderdate, 
+    o_totalprice
+FROM (
+    SELECT 
+        o_custkey, 
+        o_orderdate, 
+        o_totalprice,
+        ROW_NUMBER() OVER (PARTITION BY o_custkey ORDER BY o_orderdate DESC) AS rn
+    FROM clickzetta_sample_data.tpch_100g.orders
+) t
+WHERE rn = 1;
+```
+
+### 1.2. STRUCT 数据类型按其键从左到右排序
+
+```sql
+WITH order_struct AS (
+    SELECT 
+        o_orderkey,
+        struct(o_orderdate, o_totalprice, o_orderkey) AS order_info
+    FROM clickzetta_sample_data.tpch_100g.orders
+)
+SELECT 
+    MIN(order_info) AS min_order_date,
+    MAX(order_info) AS max_order_date_price
+FROM order_struct;
+```
+
+### 1.3. 使用 BOOL\_OR 和 BOOL\_AND 分别检查至少一个或所有布尔值是否为真
+
+```sql
+SELECT 
+    o_custkey, 
+    BOOL_OR(o_shippriority > 0) AS has_atleast_one_priority_order,
+    BOOL_AND(o_shippriority > 0) AS has_all_priority_order
+FROM clickzetta_sample_data.tpch_100g.orders
+GROUP BY o_custkey;
+```
+
+### 1.4. 想要选择除少数列之外的所有列时使用 EXCEPT
+
+```sql
+SELECT * EXCEPT(o_orderdate, o_totalprice)
+FROM clickzetta_sample_data.tpch_100g.orders;
+```
+
+### 1.5. 厌倦了在 GROUP BY 中创建长长的列列表时使用 GROUP BY ALL
+
+```sql
+SELECT 
+    o_orderkey, 
+    o_custkey, 
+    o_orderstatus, 
+    SUM(o_totalprice) AS total_price
+FROM clickzetta_sample_data.tpch_100g.orders
+GROUP BY ALL;
+```
+
+### 1.6. ORDER、GROUP字段用SELECT字段的序号代替
+
+```sql
+SELECT 
+    o_orderkey, 
+    o_custkey, 
+    o_orderstatus, 
+    SUM(o_totalprice) AS total_price
+FROM clickzetta_sample_data.tpch_100g.orders
+GROUP BY 1,2,3
+ORDER BY 3,2,1;
+```
+
+### 1.7. 需要仅在满足特定条件时计数行时使用 COUNT IF
+
+```sql
+SELECT 
+    o_custkey, 
+    COUNT_IF(o_totalprice > 100000) AS high_value_orders,
+    COUNT(o_totalprice) as all_orders
+FROM clickzetta_sample_data.tpch_100g.orders
+GROUP BY o_custkey;
+```
+
+### 1.8. 使用 COALESCE 处理空列值与其他列或回退值
+
+```sql
+WITH fake_orders AS (
+    SELECT 1 AS o_orderkey, 100 AS o_totalprice, NULL AS discount
+    UNION ALL
+    SELECT 2 AS o_orderkey, 200 AS o_totalprice, 20 AS discount
+    UNION ALL
+    SELECT 3 AS o_orderkey, 300 AS o_totalprice, NULL AS discount
+)
+SELECT 
+    o_orderkey, 
+    o_totalprice, 
+    discount,
+    COALESCE(discount, o_totalprice * 0.10) AS final_discount
+FROM fake_orders;
+```
+
+### 1.9. 使用 sequence和explode 生成数字/日期行范围
+
+```sql
+SELECT explode(sequence(1, 10)) AS num;
+```
+
+^
+
+```sql
+SELECT    EXPLODE (
+          sequence(
+          to_date('2024-01-01'),
+          to_date('2024-01-10'),
+          interval 1 DAY
+          )
+          ) AS date;
+```
+
+### 1.10. 使用 UNNEST 将 ARRAY/LIST 元素转换为单独的行
+
+```sql
+WITH nested_data AS (
+    SELECT 1 AS id, array(10, 20, 30) AS values
+    UNION ALL
+    SELECT 2 AS id, array(40, 50) AS values
+)
+SELECT 
+    id, 
+    unnest(values) AS flattened_value
+FROM nested_data;
+```
+
+## 2. 另一个表中的存在/不存在的数据获取
+
+### 2.1. 根据另一个表中的数据存在性获取表中的数据时使用 EXISTS
+
+```sql
+SELECT 
+    c_custkey, 
+    c_name
+FROM clickzetta_sample_data.tpch_100g.customer c
+WHERE EXISTS (
+    SELECT o_orderkey
+    FROM clickzetta_sample_data.tpch_100g.orders o
+    WHERE o.o_custkey = c.c_custkey AND o.o_totalprice > 1000
+);
+```
+
+### 2.2. 获取两个表中都存在的数据时使用 INTERSECT
+
+```sql
+SELECT    c_custkey
+FROM      clickzetta_sample_data.tpch_100g.customer
+INTERSECT
+SELECT    o_custkey
+FROM      clickzetta_sample_data.tpch_100g.orders;
+```
+
+### 2.3. 获取表 1 中存在但表 2 中不存在的数据时使用 EXCEPT
+
+```sql
+SELECT c_custkey
+FROM clickzetta_sample_data.tpch_100g.customer
+EXCEPT
+SELECT o_custkey
+FROM clickzetta_sample_data.tpch_100g.orders;
+```
+
+### 2.4. 获取数据差异（即 delta），使用 (A - B) U (B - A)
+
+```sql
+SELECT c_custkey, 'DELETED' as ops FROM ( 
+SELECT c_custkey
+FROM clickzetta_sample_data.tpch_100g.customer
+EXCEPT
+SELECT c_custkey
+FROM clickzetta_sample_data.tpch_100g.customer
+)
+
+UNION ALL
+
+SELECT c_custkey, 'UPSERTED' as ops FROM ( 
+SELECT c_custkey, c_name, c_address
+FROM clickzetta_sample_data.tpch_100g.customer
+EXCEPT
+SELECT c_custkey, c_name, c_address
+FROM clickzetta_sample_data.tpch_100g.customer
+);
+```
+
+## 3. 在 SQL 中CASE 语句
+
+### 3.1. 使用 CASE 语句
+
+```sql
+SELECT 
+    o_orderkey, 
+    o_totalprice,
+    CASE
+        WHEN o_totalprice > 100000 THEN 'Large Order'
+        ELSE 'Regular Order'
+    END AS order_type
+FROM clickzetta_sample_data.tpch_100g.orders
+LIMIT 5;
+```
+
+## 4. 访问关于数据的元数据
+
+### 4.1. 访问在 information\_schema 中存储的元数据
+
+```sql
+-- 查看表的信息
+DESCRIBE TABLE clickzetta_sample_data.tpch_100g.orders;
+```
+
+```sql
+-- 查看所有表
+SHOW TABLES IN clickzetta_sample_data.tpch_100g;
+```
+
+^
+
+## 5. 使用 UPSERTS（即 MERGE INTO）避免数据重复
+
+### 5.1. 使用 UPSERT/MERGE INTO 插入新数据，更新现有数据
+
+```sql
+MERGE INTO dim_customer_scd2 AS target
+USING (
+    VALUES
+        (1, 'Customer#000000001', 'New Address 1', 15, '25-989-741-2988', 711.56, 'BUILDING', 'comment1', '2024-10-18', NULL, TRUE),
+        (2, 'Customer#000000002', 'New Address 2', 18, '12-423-790-3665', 879.49, 'FURNITURE', 'comment2', '2024-10-18', NULL, TRUE),
+        (1501, 'Customer#000001501', 'New Address 1501', 24, '11-345-678-9012', 500.50, 'MACHINERY', 'comment1501', '2024-10-18', NULL, TRUE),
+        (1502, 'Customer#000001502', 'New Address 1502', 21, '22-456-789-0123', 600.75, 'AUTOMOBILE', 'comment1502', '2024-10-18', NULL, TRUE)
+) AS source (c_custkey, c_name, c_address, c_nationkey, c_phone, c_acctbal, c_mktsegment, c_comment, valid_from, valid_to, is_current)
+ON target.c_custkey = source.c_custkey
+WHEN MATCHED AND target.is_current = TRUE THEN
+    UPDATE SET valid_to = source.valid_from, is_current = FALSE
+WHEN NOT MATCHED THEN
+    INSERT (c_custkey, c_name, c_address, c_nationkey, c_phone, c_acctbal, c_mktsegment, c_comment, valid_from, valid_to, is_current)
+    VALUES (source.c_custkey, source.c_name, source.c_address, source.c_nationkey, source.c_phone, source.c_acctbal, source.c_mktsegment, source.c_comment, source.valid_from, source.valid_to, source.is_current);
+```
+
+## 6. 高级 JOIN 类型
+
+### 6.1. 使用JOIN和ROW\_NUMBER获取表 2 中与表 1 行时间最接近的值
+
+```sql
+WITH stock_prices_data AS (
+    SELECT 'APPL' AS ticker, to_timestamp('2001-01-01 00:00:00') AS ts, 1 AS price
+    UNION ALL
+    SELECT 'APPL', to_timestamp('2001-01-01 00:01:00'), 2
+    UNION ALL
+    SELECT 'APPL', to_timestamp('2001-01-01 00:02:00'), 3
+    UNION ALL
+    SELECT 'MSFT', to_timestamp('2001-01-01 00:00:00'), 1
+    UNION ALL
+    SELECT 'MSFT', to_timestamp('2001-01-01 00:01:00'), 2
+    UNION ALL
+    SELECT 'MSFT', to_timestamp('2001-01-01 00:02:00'), 3
+    UNION ALL
+    SELECT 'GOOG', to_timestamp('2001-01-01 00:00:00'), 1
+    UNION ALL
+    SELECT 'GOOG', to_timestamp('2001-01-01 00:01:00'), 2
+    UNION ALL
+    SELECT 'GOOG', to_timestamp('2001-01-01 00:02:00'), 3
+),
+portfolio_holdings_data AS (
+    SELECT 'APPL' AS ticker, to_timestamp('2000-12-31 23:59:30') AS ts, 5.16 AS shares
+    UNION ALL
+    SELECT 'APPL', to_timestamp('2001-01-01 00:00:30'), 2.94
+    UNION ALL
+    SELECT 'APPL', to_timestamp('2001-01-01 00:01:30'), 24.13
+    UNION ALL
+    SELECT 'GOOG', to_timestamp('2000-12-31 23:59:30'), 9.33
+    UNION ALL
+    SELECT 'GOOG', to_timestamp('2001-01-01 00:00:30'), 23.45
+    UNION ALL
+    SELECT 'GOOG', to_timestamp('2001-01-01 00:01:30'), 10.58
+    UNION ALL
+    SELECT 'DATA', to_timestamp('2000-12-31 23:59:30'), 6.65
+    UNION ALL
+    SELECT 'DATA', to_timestamp('2001-01-01 00:00:30'), 17.95
+    UNION ALL
+    SELECT 'DATA', to_timestamp('2001-01-01 00:01:30'), 18.37
+)
+SELECT h.ticker,
+    h.ts AS holdings_ts,
+    p.ts AS stock_price_ts,
+    p.price,
+    h.shares,
+    p.price * h.shares AS value
+FROM portfolio_holdings_data h
+JOIN (
+    SELECT 
+        ticker, 
+        ts, 
+        price,
+        ROW_NUMBER() OVER (PARTITION BY ticker ORDER BY ts DESC) AS rn
+    FROM stock_prices_data
+) p
+ON h.ticker = p.ticker AND h.ts >= p.ts
+WHERE p.rn = 1
+ORDER BY h.ticker, h.ts;
+```
+
+### 6.2. 使用 ANTI JOIN 获取表 1 中存在但表 2 中不存在的行
+
+```sql
+SELECT    c.c_custkey
+FROM      clickzetta_sample_data.tpch_100g.customer c
+LEFT      ANTI JOIN clickzetta_sample_data.tpch_100g.orders o ON c.c_custkey = o.o_custkey
+ORDER BY  c.c_custkey
+LIMIT     5;
+```
+
+### 6.3. 使用 LATERAL JOIN 对表 1 中的每一行连接表 2 中的所有“匹配”行
+
+```sql
+SELECT    o.o_orderkey,
+          o.o_totalprice,
+          l.l_linenumber,
+          l.l_extendedprice
+FROM      clickzetta_sample_data.tpch_100g.orders o
+CROSS     JOIN clickzetta_sample_data.tpch_100g.lineitem l
+WHERE     l.l_orderkey = o.o_orderkey AND      
+          l.l_linenumber <= 2 AND      
+          l.l_extendedprice < (o.o_totalprice / 2)
+ORDER BY  o.o_orderkey,
+          l.l_linenumber;
+```
+
+## 7 业务用例
+
+### 7.1. 使用 CASE和 `GROUP BY `将维度值转换为单独的列
+
+```sql
+SELECT    o_custkey,
+          SUM(
+          CASE
+                    WHEN o_orderstatus = 'F' THEN o_totalprice
+                    ELSE 0
+          END
+          ) AS fulfilled_total,
+          SUM(
+          CASE
+                    WHEN o_orderstatus = 'O' THEN o_totalprice
+                    ELSE 0
+          END
+          ) AS open_total,
+          SUM(
+          CASE
+                    WHEN o_orderstatus = 'P' THEN o_totalprice
+                    ELSE 0
+          END
+          ) AS pending_total
+FROM      clickzetta_sample_data.tpch_100g.orders
+GROUP BY  o_custkey
+ORDER BY  o_custkey;
+```
+
+### 7.2. 使用 CUBE 生成每个可能维度组合的指标
+
+```sql
+SELECT    o_orderpriority,
+          o_orderstatus,
+          EXTRACT(
+          YEAR
+          FROM      o_orderdate
+          ) AS order_year,
+          SUM(o_totalprice) AS total_sales
+FROM      clickzetta_sample_data.tpch_100g.orders
+GROUP BY  CUBE (o_orderpriority, o_orderstatus, order_year)
+ORDER BY  1,
+          2,
+          3;
+```
+
+^