@clickzetta/cz-cli-darwin-x64 0.3.1

package/bin/skills/lakehouse-doc/references/SQL_Join_Guide.md

@@ -0,0 +1,655 @@
+# Lakehouse JOIN使用指南：多技术栈用户迁移手册
+
+## 概述
+
+云器 Lakehouse 提供了完整、高性能的 JOIN 功能，支持从 Spark、Hive、MaxCompute、Snowflake 和传统数据库的无缝迁移。本指南基于真实生产环境经验，为不同技术背景的用户提供专业的迁移指导和最佳实践。
+
+### 🎯 **快速导航**
+- [Spark用户迁移指南](#spark用户迁移指南) - DataFrame API到SQL JOIN的平滑过渡
+- [Hive用户迁移指南](#hive用户迁移指南) - MapReduce到列式存储的性能跃升
+- [MaxCompute用户迁移指南](#maxcompute用户迁移指南) - 阿里云生态的延续与增强
+- [Snowflake用户迁移指南](#snowflake用户迁移指南) - 云原生架构的进一步优化
+- [传统数据库用户迁移指南](#传统数据库用户迁移指南) - OLTP到OLAP的架构升级
+
+---
+
+## JOIN类型与语法
+
+### 完整的JOIN类型支持
+
+云器Lakehouse支持完整的SQL JOIN标准，提供7种JOIN类型：
+
+| JOIN类型 | 功能描述 | 典型应用场景 |
+|----------|----------|-------------|
+| **INNER JOIN** | 返回两表匹配记录 | 标准业务关联查询 |
+| **LEFT [OUTER] JOIN** | 保留左表全部记录 | 主表数据完整性保证 |
+| **RIGHT [OUTER] JOIN** | 保留右表全部记录 | 维度表完整性展示 |
+| **FULL [OUTER] JOIN** | 保留两表全部记录 | 完整数据审计分析 |
+| **SEMI JOIN** | 返回左表中存在匹配的记录 | 数据存在性验证 |
+| **ANTI JOIN** | 返回左表中不存在匹配的记录 | 数据差异化分析 |
+| **CROSS JOIN** | 返回两表笛卡尔积 | 数据组合生成 |
+
+### 基础JOIN语法
+
+```sql
+-- 标准INNER JOIN
+SELECT e.emp_name, d.dept_name
+FROM employees e
+INNER JOIN departments d ON e.dept_id = d.dept_id;
+
+-- LEFT JOIN保证左表完整性
+SELECT e.emp_name, d.dept_name
+FROM employees e
+LEFT JOIN departments d ON e.dept_id = d.dept_id;
+
+-- SEMI JOIN进行存在性检查
+SELECT e.emp_name, e.salary
+FROM employees e
+SEMI JOIN departments d ON e.dept_id = d.dept_id;
+
+-- ANTI JOIN识别孤立数据
+SELECT e.emp_name, e.dept_id
+FROM employees e
+ANTI JOIN departments d ON e.dept_id = d.dept_id;
+```
+
+---
+
+## 性能优化策略
+
+### MAPJOIN广播优化
+
+MAPJOIN 是云器 Lakehouse 的核心优化特性，通过将小表广播到所有计算节点，显著提升JOIN性能。
+
+**优化原理**：
+- 消除昂贵的 Shuffle 操作
+- 缓解数据倾斜问题
+- 提升查询执行速度
+
+```sql
+-- 单表广播优化
+SELECT /*+ MAPJOIN(departments) */ 
+    e.emp_name, d.dept_name, d.budget
+FROM employees e
+JOIN departments d ON e.dept_id = d.dept_id;
+
+-- 多表广播优化
+SELECT /*+ MAPJOIN(employees, departments) */
+    o.order_id, e.emp_name, d.dept_name
+FROM orders o
+JOIN employees e ON o.emp_id = e.emp_id
+JOIN departments d ON e.dept_id = d.dept_id;
+```
+
+**使用建议**：
+- 小表大小建议控制在 1GB 以内
+- 适用于维度表与事实表的关联
+- 优先考虑小表广播，而非大表 JOIN 大表
+
+### SORTMERGEJOIN排序合并优化
+
+适用于大表与大表的 JOIN 场景，特别是数据已按 JOIN 键排序的情况。
+
+```sql
+-- 大表JOIN优化
+SELECT /*+ SORTMERGEJOIN(table1, table2) */
+    t1.customer_id, t2.order_amount
+FROM large_customer_table t1
+JOIN large_order_table t2 ON t1.customer_id = t2.customer_id;
+```
+
+### 查询结构优化
+
+遵循“过滤-连接-聚合”的最佳实践模式：
+
+```sql
+-- 推荐的查询结构
+WITH filtered_facts AS (
+    SELECT fact_id, dimension_id, amount, date_key
+    FROM fact_table
+    WHERE date_key >= '20240101'          -- 先过滤
+      AND amount > 1000
+),
+enriched_data AS (
+    SELECT /*+ MAPJOIN(dim_table) */
+        f.fact_id, f.amount, d.category
+    FROM filtered_facts f
+    JOIN dim_table d ON f.dimension_id = d.dimension_id  -- 后连接
+)
+SELECT 
+    category,
+    COUNT(*) as record_count,
+    SUM(amount) as total_amount                          -- 最后聚合
+FROM enriched_data
+GROUP BY category;
+```
+
+---
+
+## Spark用户迁移指南
+
+### 直接迁移的技能
+
+Spark用户可以无缝使用以下功能：
+
+```sql
+-- DataFrame广播JOIN → MAPJOIN提示
+SELECT /*+ MAPJOIN(small_table) */
+    l.order_id, s.product_name
+FROM large_orders l
+JOIN small_products s ON l.product_id = s.product_id;
+
+-- 窗口函数完全兼容
+SELECT 
+    emp_name,
+    salary,
+    ROW_NUMBER() OVER (PARTITION BY dept_id ORDER BY salary DESC) as rank
+FROM employees;
+
+-- 复杂数据类型处理
+SELECT 
+    user_id,
+    event_array[0] as first_event,           -- 数组索引从0开始
+    size(event_array) as event_count,
+    explode(event_array) as individual_event -- 直接使用，无需LATERAL VIEW
+FROM user_events;
+```
+
+### 语法调整要点
+
+**1. 数组展开语法简化**
+
+```sql
+-- Spark语法（不支持）
+SELECT user_id, tag
+FROM users LATERAL VIEW explode(tags) t AS tag;
+
+-- Lakehouse语法
+SELECT user_id, explode(tags) as tag
+FROM users;
+```
+
+**2. 结构体操作保持一致**
+
+```sql
+-- Spark兼容的结构体操作
+WITH customer_profiles AS (
+    SELECT 
+        customer_id,
+        named_struct(
+            'name', customer_name,
+            'contact', named_struct('email', email, 'phone', phone)
+        ) as profile
+    FROM customers
+)
+SELECT 
+    customer_id,
+    profile.name as customer_name,
+    profile.contact.email as email
+FROM customer_profiles;
+```
+
+### 数据倾斜处理策略
+
+```sql
+-- 盐值JOIN解决数据倾斜
+WITH salted_large AS (
+    SELECT *, 
+           CONCAT(join_key, '_', ABS(HASH(join_key) % 10)) as salted_key
+    FROM large_table
+),
+salted_small AS (
+    SELECT *, 
+           CONCAT(join_key, '_', sequence) as salted_key
+    FROM small_table 
+    CROSS JOIN (SELECT EXPLODE(SEQUENCE(0, 9)) as sequence)
+)
+SELECT l.data, s.info
+FROM salted_large l
+JOIN salted_small s ON l.salted_key = s.salted_key;
+```
+
+---
+
+## Hive用户迁移指南
+
+### 概念延续与性能升级
+
+Hive用户熟悉的分区表和批处理概念在Lakehouse中得到保留和增强：
+
+```sql
+-- 分区裁剪概念保持一致
+SELECT /*+ MAPJOIN(dim_table) */
+    fact.order_id, dim.product_name
+FROM fact_orders fact
+JOIN dim_products dim ON fact.product_id = dim.product_id
+WHERE fact.dt = '2024-06-01';           -- 分区字段过滤
+```
+
+### MapReduce到列式计算的跃升
+
+```sql
+-- Hive多阶段Job → Lakehouse一体化查询
+WITH order_aggregation AS (
+    SELECT 
+        customer_id,
+        SUM(amount) as total_amount,
+        COUNT(*) as order_count
+    FROM orders 
+    WHERE order_date >= '2024-01-01'
+    GROUP BY customer_id
+)
+SELECT /*+ MAPJOIN(customers) */
+    c.customer_name,
+    a.total_amount,
+    a.order_count
+FROM order_aggregation a
+JOIN customers c ON a.customer_id = c.customer_id
+WHERE a.total_amount > 10000;
+```
+
+### MAPJOIN功能增强
+
+```sql
+-- Hive手动控制 → Lakehouse智能优化
+SELECT /*+ MAPJOIN(employees, departments) */  -- 支持多表广播
+    o.order_id, e.emp_name, d.dept_name
+FROM orders o
+JOIN employees e ON o.emp_id = e.emp_id
+JOIN departments d ON e.dept_id = d.dept_id;
+```
+
+---
+
+## MaxCompute用户迁移指南
+
+### 语法兼容性
+
+MaxCompute用户可以直接使用熟悉的语法：
+
+```sql
+-- MAPJOIN语法直接兼容
+SELECT /*+ MAPJOIN(small_table) */
+    large.order_id, small.product_name
+FROM large_orders large
+JOIN small_products small ON large.product_id = small.product_id;
+
+-- 分区查询方式保持一致
+SELECT * FROM orders 
+WHERE order_date = '2024-06-01'
+  AND status = 'completed';
+```
+
+### 函数迁移映射
+
+```sql
+-- 时间函数对应关系
+SELECT 
+    order_id,
+    CURRENT_TIMESTAMP() as process_time,        -- MaxCompute: GETDATE()
+    DATE_FORMAT(order_date, 'yyyy-MM') as month -- 语法兼容
+FROM orders;
+```
+
+### SEMI JOIN优化
+
+```sql
+-- EXISTS查询 → SEMI JOIN性能优化
+SELECT a.customer_id, a.customer_name
+FROM customers a
+SEMI JOIN orders b ON a.customer_id = b.customer_id;
+```
+
+---
+
+## Snowflake用户迁移指南
+
+### 云原生特性对应
+
+```sql
+-- 自动优化 → 显式优化控制
+SELECT /*+ MAPJOIN(customer_dim, product_dim) */
+    cd.customer_name, pd.product_name, sf.sales_amount
+FROM sales_fact sf
+JOIN customer_dim cd ON sf.customer_id = cd.customer_id
+JOIN product_dim pd ON sf.product_id = pd.product_id
+WHERE sf.sale_date >= '2024-06-01';
+```
+
+### 资源管理对比
+
+- **Snowflake WAREHOUSE** → **Lakehouse VCLUSTER**
+- **自动扩缩容** → **弹性计算资源**
+- **按需付费** → **按使用量计费**
+
+### 历史数据查询
+
+<= '2024-06-01 23:59:59';
+```
+
+---
+
+## 传统数据库用户迁移指南
+
+### 思维模式转换
+
+**从OLTP到OLAP的架构升级**
+
+```sql
+-- OLTP思维：单条记录查询
+-- 转换为
+-- OLAP思维：批量分析查询
+
+SELECT /*+ MAPJOIN(customers) */
+    c.customer_segment,
+    COUNT(*) as order_count,
+    AVG(o.order_amount) as avg_amount,
+    SUM(o.order_amount) as total_amount
+FROM customers c
+JOIN orders o ON c.customer_id = o.customer_id
+WHERE o.order_date >
+
+### 索引策略调整
+
+```sql
+-- 传统索引 → 列式存储 + 布隆过滤器
+CREATE BLOOMFILTER INDEX idx_customer_bloom 
+ON TABLE orders(customer_id);
+
+-- 查询自动应用优化
+SELECT /*+ MAPJOIN(customers) */
+    c.customer_name, COUNT(o.order_id) as order_count
+FROM customers c
+JOIN orders o ON c.customer_id = o.customer_id
+GROUP BY c.customer_name;
+```
+
+### 事务处理转换
+
+```sql
+-- 传统事务 → 批量MERGE操作
+MERGE INTO customers c
+USING (
+    SELECT customer_id
+    FROM order_summary 
+    WHERE total_amount > 50000
+) high_value ON c.customer_id = high_value.customer_id
+WHEN MATCHED THEN UPDATE SET status = 'premium';
+```
+
+---
+
+## JOIN条件与语法规范
+
+### 支持的条件语法
+
+```sql
+-- ON条件表达式
+SELECT * FROM table1 t1
+JOIN table2 t2 ON t1.id = t2.id;
+
+-- USING简化语法
+SELECT * FROM table1 t1
+JOIN table2 t2 USING (id);
+
+-- 复合条件
+SELECT * FROM orders o
+JOIN employees e ON o.emp_id = e.emp_id 
+                 AND o.order_date >= e.hire_date;
+```
+
+### 语法限制与替代方案
+
+**JOIN 条件中避免使用子查询**
+
+```sql
+-- 不推荐的写法
+SELECT e.emp_name
+FROM employees e
+JOIN departments d ON e.dept_id = (
+    SELECT dept_id FROM departments WHERE dept_name = 'Engineering'
+);
+
+-- 推荐的替代方案
+SELECT e.emp_name
+FROM employees e
+JOIN departments d ON e.dept_id = d.dept_id
+WHERE d.dept_name = 'Engineering';
+
+-- 或使用CTE
+WITH target_dept AS (
+    SELECT dept_id FROM departments WHERE dept_name = 'Engineering'
+)
+SELECT e.emp_name
+FROM employees e
+JOIN target_dept t ON e.dept_id = t.dept_id;
+```
+
+---
+
+## 数据类型与NULL值处理
+
+### NULL值显示特性
+
+云器Lakehouse中不同数据类型的NULL值具有特定的显示格式：
+
+<- STRING类型NULL正常显示
+|          | nan     | HR        | <- 数值类型NULL显示为"nan"
++----------+---------+-----------+
+*/
+```
+
+**显示规律**：
+- **字符类型**（STRING、VARCHAR）：NULL显示为空
+- **数值类型**（INT、DOUBLE、DECIMAL）：NULL显示为"nan"
+- **时间类型**（DATE、TIMESTAMP）：NULL显示为"NaT"
+- **逻辑判断**：IS NULL和IS NOT NULL在所有类型上正常工作
+
+### NULL值安全处理
+
+```sql
+-- 安全的NULL值处理
+SELECT 
+    emp_name,
+    CASE 
+        WHEN salary IS NULL OR CAST(salary AS STRING) = 'nan' 
+        THEN 0.0 
+        ELSE salary 
+    END as safe_salary,
+    COALESCE(dept_name, 'Unknown Department') as safe_dept_name
+FROM employees e
+LEFT JOIN departments d ON e.dept_id = d.dept_id;
+```
+
+---
+
+## 复杂JOIN场景
+
+### 多表关联分析
+
+```sql
+-- 典型的维度建模查询
+SELECT /*+ MAPJOIN(dim_customer, dim_product, dim_date) */
+    dd.year,
+    dd.quarter,
+    dc.customer_tier,
+    dp.product_line,
+    COUNT(*) as transaction_count,
+    SUM(ft.amount) as total_revenue,
+    AVG(ft.amount) as avg_transaction_value
+FROM fact_transactions ft
+JOIN dim_customer dc ON ft.customer_id = dc.customer_id
+JOIN dim_product dp ON ft.product_id = dp.product_id  
+JOIN dim_date dd ON ft.date_id = dd.date_id
+WHERE dd.year >
+
+### 层次化数据处理
+
+```sql
+-- 组织架构多级查询
+WITH employee_hierarchy AS (
+    -- 第一级：高级管理层
+    SELECT employee_id, employee_name, 1 as level
+    FROM employees WHERE manager_id IS NULL
+    
+    UNION ALL
+    
+    -- 第二级：中层管理
+    SELECT e.employee_id, e.employee_name, 2 as level
+    FROM employees e
+    JOIN employees m ON e.manager_id = m.employee_id
+    WHERE m.manager_id IS NULL
+    
+    UNION ALL
+    
+    -- 第三级：基层员工
+    SELECT e.employee_id, e.employee_name, 3 as level
+    FROM employees e
+    JOIN employees m1 ON e.manager_id = m1.employee_id
+    JOIN employees m2 ON m1.manager_id = m2.employee_id
+    WHERE m2.manager_id IS NULL
+)
+SELECT level, COUNT(*) as employee_count
+FROM employee_hierarchy
+GROUP BY level
+ORDER BY level;
+```
+
+### 时间序列JOIN
+
+```sql
+-- 时间窗口关联分析
+SELECT o.order_id, l.line_item_id, o.order_date, l.ship_date
+FROM orders o
+JOIN line_items l ON o.order_id = l.order_id
+                  AND o.order_date BETWEEN l.ship_date - INTERVAL '5' DAY 
+                                       AND l.ship_date + INTERVAL '5' DAY
+WHERE o.order_date >= '2024-01-01';
+```
+
+---
+
+## 性能调优实践
+
+### 查询优化检查清单
+
+#### 1. JOIN策略选择
+
+```sql
+-- 小表 × 大表：MAPJOIN优先
+SELECT /*+ MAPJOIN(dim_table) */
+    fact.*, dim.dimension_name
+FROM fact_table fact
+JOIN dim_table dim ON fact.dim_id = dim.dim_id;
+
+-- 大表 × 大表：SORTMERGEJOIN
+SELECT /*+ SORTMERGEJOIN(table1, table2) */
+    t1.*, t2.*
+FROM large_table1 t1
+JOIN large_table2 t2 ON t1.join_key = t2.join_key;
+```
+
+#### 2. 谓词下推优化
+
+```sql
+-- 将过滤条件前置
+WITH filtered_base AS (
+    SELECT customer_id, order_id, amount
+    FROM orders
+    WHERE order_date >= '2024-01-01'      -- 过滤前置
+      AND status = 'completed'
+)
+SELECT /*+ MAPJOIN(customers) */
+    c.customer_name, f.amount
+FROM filtered_base f
+JOIN customers c ON f.customer_id = c.customer_id;
+```
+
+#### 3. 列裁剪策略
+
+```sql
+-- 明确指定所需列
+SELECT customer_id, customer_name, order_count
+FROM customer_summary
+WHERE registration_date >= '2024-01-01';
+
+-- 使用EXCEPT排除不必要的列
+SELECT * EXCEPT(internal_notes, created_by, updated_by)
+FROM customer_details;
+```
+
+### 分页查询策略
+
+#### 基于游标的高效分页
+
+```sql
+-- 推荐的分页方式
+SELECT customer_id, order_id, order_date, amount
+FROM orders 
+WHERE customer_id > :last_customer_id  -- 游标位置
+   OR (customer_id = :last_customer_id AND order_id > :last_order_id)
+ORDER BY customer_id, order_id
+LIMIT 1000;
+```
+
+#### 范围分区分页
+
+```sql
+-- 基于业务逻辑的分区处理
+SELECT * FROM orders 
+WHERE order_date >= '2024-06-01'
+  AND order_date < '2024-06-02'  -- 按天分批处理
+ORDER BY order_id;
+```
+
+---
+
+## 最佳实践总结
+
+### 通用优化原则
+
+1. **小表广播优先**：维度表与事实表关联首选MAPJOIN
+2. **过滤条件前置**：在JOIN之前完成数据过滤
+3. **选择精确JOIN类型**：根据业务需求选择最合适的JOIN类型
+4. **避免深度分页**：使用游标分页替代OFFSET分页
+5. **监控执行计划**：关注查询性能和资源使用
+
+### 迁移成功要素
+
+| 技术背景 | 迁移重点 | 预期收益 |
+|----------|----------|----------|
+| **Spark** | MAPJOIN提示语法适配 | 更精细的JOIN优化控制 |
+| **Hive** | 列式存储思维转换 | 查询性能显著提升 |
+| **MaxCompute** | 函数语法映射 | 实时交互体验 |
+| **Snowflake** | 显式优化控制 | 更灵活的性能调优 |
+| **传统数据库** | OLAP思维建立 | 大数据处理能力跃升 |
+
+### 性能优化效果
+
+通过合理应用本指南的优化策略：
+
+- **查询性能**：相比传统数据库提升 10-100 倍
+- **资源效率**：列式存储降低 I/O 开销 60-80%
+- **开发效率**：统一 SQL 接口减少学习成本
+- **运维复杂度**：云原生架构简化管理
+
+---
+
+## 总结
+
+云器Lakehouse的JOIN功能为企业提供了强大而灵活的数据关联能力。通过遵循本指南的迁移策略和最佳实践，不同技术背景的用户都能快速发挥出系统的最大价值，实现从传统数据处理到现代大数据分析的成功转型。
+
+### 关键优势
+
+- **完整兼容性**：支持标准SQL JOIN语法和语义
+- **性能优化**：MAPJOIN、SORTMERGEJOIN 等先进优化技术
+- **平滑迁移**：针对不同技术背景提供专业迁移路径
+- **企业级特性**：NULL 值处理、类型转换等生产就绪功能
+
+### 立即开始
+
+选择适合您技术背景的迁移指南，开始云器Lakehouse的JOIN功能探索之旅。通过实践本指南的建议，您将能够构建高效、可靠的数据分析解决方案，满足企业级数据处理的各种需求。
+
+---
+
+**注意**：本文档基于 Lakehouse 2025 年 6 月的产品文档整理，建议定期查看官方文档获取最新更新。在生产环境中使用前，请务必在测试环境中验证所有操作的正确性和性能影响。