npm - @clickzetta/cz-cli-darwin-arm64 - Versions diffs - 0.3.75 → 0.3.78 - Mend

@clickzetta/cz-cli-darwin-arm64 0.3.75 → 0.3.78

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (1249) hide show

package/bin/skills/lakehouse-doc/references/databricks_yunqi_integration_guide_v2.md DELETED Viewed

@@ -1,811 +0,0 @@
-# Databricks-云器Lakehouse 跨平台数据联邦最佳实践指南
-## 概述
-本指南基于企业级生产环境的成功实施经验，详细阐述如何实现 Databricks 与云器 Lakehouse 之间的跨平台数据联邦。本文档涵盖完整的架构设计、实施方案和运维最佳实践，为企业数据平台建设提供指导。
-## 技术实现原理与特点
-### 什么是跨平台数据联邦
-**数据联邦（Data Federation）** 是一种分布式数据架构模式，允许多个独立的数据系统通过统一的接口进行数据访问和查询，而无需物理移动或复制数据。
-### 核心技术实现特点
-#### **与传统数据集成的根本区别**
-| 特性        | 传统数据集成（ETL/ELT） | 跨平台数据联邦      |
-| --------- | --------------- | ------------ |
-| **数据存储**  | 数据复制到目标系统       | 数据保持在源存储位置   |
-| **数据同步**  | 定期ETL作业同步       | 实时元数据联邦访问    |
-| **存储成本**  | 双倍存储成本          | 单份存储，共享访问    |
-| **数据一致性** | 可能存在延迟和差异       | 访问同一数据源，天然一致 |
-| **实施复杂度** | 需要复杂的管道维护       | 配置元数据连接即可    |
-| **查询性能**  | 本地查询性能优异        | 跨网络查询，需要优化   |
-#### **技术架构核心原理**
-<-------------------|  | Metadata Store |  |
-|  +----------------+  |                     |  +----------------+  |
-+----------------------+                     +---------------------+
-         |                                         |
-         |                                         |
-   Data File Access                           Permission Check
-         |                                         |
-         v                                         v
-+--------------------------------------------------------------+
-|                        AWS S3 Storage                        |
-|  +-------------------------------------------------------+   |
-|  | s3://bucket/external-tables/customer/                 |   |
-|  |  - _delta_log/                                        |   |
-|  |  - part-00000.parquet                                 |   |
-|  |  - part-00001.parquet                                 |   |
-|  |  - ...                                                |   |
-|  +-------------------------------------------------------+   |
-|                                                              |
-|  * Both Databricks and Yunqi access the same files directly  |
-+--------------------------------------------------------------+
-```
-#### **核心技术优势**
-**1. 存储经济性**
-* 数据只存储一份，节省50%+ 存储成本
-* 无需维护复杂的数据同步管道
-* 减少数据传输带宽成本
-**2. 数据一致性**
-* 两个平台访问相同的数据文件
-* 无数据同步延迟问题
-* 天然保证数据一致性
-**3. 架构简洁性**
-* 无需ETL管道开发和维护
-* 配置即用，实施周期短
-* 减少数据管道故障点
-**4. 安全可控性**
-* 统一的权限管理体系
-* 基于存储类型的天然访问隔离
-* 细粒度的表和列级权限控制
-#### **适用场景**
-**推荐场景**：
-* 需要在多个分析平台查询相同数据集
-* 希望减少数据存储和传输成本
-* 要求数据实时一致性
-* 数据主要用于读取分析（OLAP场景）
-* 已采用Delta Lake或类似格式的企业
-**不适用场景**：
-* 需要频繁的跨平台数据写入操作
-* 对查询延迟要求极高（毫秒级）
-* 数据安全要求完全物理隔离
-* 网络环境不稳定或带宽受限
-#### **技术限制与考虑**
-**性能考虑**：
-* 跨网络查询比本地查询延迟略高
-* 大数据量查询需要优化分区策略
-* 并发查询数量受网络带宽限制
-**网络依赖**：
-* 需要稳定的 AWS 区域内网络连接
-* S3访问权限必须正确配置
-* Unity Catalog服务必须可达
-## 环境要求与支持范围
-**支持的技术栈组合**：
-* **推荐配置**：AWS 上的 Databricks + AWS 上的云器 Lakehouse
-* **不支持**：暂不支持其它云平台组合
-**前置技术要求**：
-* Databricks Unity Catalog 已部署并启用 External Data Access
-* Service Principal权限配置完成
-* S3存储访问策略已建立
-* 云器 Lakehouse（AWS 版本）已就绪
-* 稳定的AWS区域内网络连接
-## 架构设计
-### 技术架构图
-```
-+-------------------+    +--------------------+    +---------------------+
-|   Databricks      |    |  Unity Catalog     |    |    Yunqi Lakehouse  |
-|   (AWS)           |    |  (AWS)             |    |    (AWS)            |
-|                   |    |                    |    |                     |
-| +---------------+ |    | +----------------+ |    | +-----------------+ |
-| | External Tbl  | |    | | Metadata       | |    | | Federated Query | |
-| |               | |<-->
-### 数据访问模式
-**企业级数据访问策略**：
-| Databricks 表类型               | 云器访问性 | 生产建议 |
-| ---------------------------- | ----- | ---- |
-| `Managed Iceberg`            | 支持    | 推荐使用 |
-| `Managed Delta`              | 支持    | 推荐使用 |
-| `View` / `Materialized View` | 暂不支持  | 转换使用 |
-| `Streaming Table`            | 暂不支持  | 转换使用 |
-## 环境配置
-### **重要说明：配置必须严格按照以下顺序执行**
-```
-配置依赖关系和顺序：
-1. Account级别：Service Principal创建和配置
-2. Account级别：Unity Catalog Metastore External Data Access启用
-3. Account级别：Storage Credential和External Location配置
-4. Workspace级别：Catalog和Schema创建
-5. Workspace级别：Service Principal权限配置
-6. 端到端测试和验证
-```
-### 1. Databricks 关键配置
-#### 1.1 Service Principal创建和配置
-**步骤1：创建Service Principal（Account Console操作）**
-1. 登录 **Databricks Account Console**（注意：不是workspace）
-2. 导航路径：`Account settings` → `User management` → `Service principals`
-3. 点击 `Add service principal`
-4. 填写配置：
-   * **Display name**: `Lakehouse-Integration-SP`
-   * **Application ID**: 自动生成（**重要：记录此CLIENT\_ID**）
-5. 点击 `Add` 创建
-**步骤2：生成 Secret（Account Console 操作）**
-1. 进入创建的 Service Principal 详情页
-2. 点击 `Secrets` tab
-3. 点击 `Generate secret`
-4. **重要**：立即复制并安全保存 Secret 值（只显示一次）
-5. 记录以下信息用于云器连接：
-   ```
-   CLIENT_ID
-   CLIENT_SECRET
-   ```
-**步骤3：分配 workspace 权限（Account Console 操作）**
-1. 在Account Console中，导航到 `Workspaces`
-2. 选择目标workspace
-3. 点击 `Permissions` tab
-4. 点击 `Add permissions`
-5. 搜索并选择创建的 Service Principal
-6. 分配权限：`Admin`（推荐用于初始配置，后续可调整为User）
-#### 1.2 Unity Catalog Metastore 配置
-**步骤1：检查Metastore状态（Workspace SQL编辑器）**
-```sql
--- 在Databricks workspace中执行
-DESCRIBE METASTORE;
--- 检查输出中的关键信息：
--- Metastore Name: metastore名称
--- Cloud: 应该是 'aws'
--- Region: 应该与云器相同
--- Privileged Model Version: 应该是 '1.0' 或更高
-```
-**步骤2：启用External Data Access（Account Console操作）**
-1. 在 Databricks 工作区左侧的主导航栏中，点击 `Catalog\` 选项进入 Catalog Explorer。
-2. 在 Catalog Explorer 的主界面上方，点击齿轮形状的设置图标 (Manage)。
-3. 在弹出的下拉菜单中，点击 `Metastore\` 选项。
-4. 在 Metastore 页面确保以下选项已启用：
-   ```
-   ✅ External data access: Enabled
-   ```
-**步骤3：Service Principal 权限配置**
-1. 进入 Databricks Workspace Console。
-* 在左侧边栏中，点击 `Catalog` 进入 Catalog 浏览器。&#x20;
-* 在 Catalog 列表中，点击您希望授权的目标 Catalog，例如 `databricks_catalog`。&#x20;
-2\. 打开权限管理
-* 在所选 Catalog 的主页面中，点击 Permissions 标签页。&#x20;
-* 点击 Grant 按钮以打开授权对话框。&#x20;
-3\. 选择主体 (Principal)&#x20;
-* 在弹出的 Grant on <Catalog 名称> 对话框中，于 Principals 字段搜索并选择您的服务主体。
-4\. 分配权限
-* 使用权限预设 (Privilege presets)：为了简化配置，建议使用预设角色。对于需要读写和创建对象的场景，从下拉菜单中选择 Data Editor。此预设会自动授予一系列常用权限，如 `USE CATALOG, USE SCHEMA, SELECT, MODIFY, CREATE TABLE` 等。
-* 授予外部访问权限 (关键步骤)：如果您需要允许外部系统（非 Databricks）通过此服务主体访问数据，请务必勾选页面底部的` EXTERNAL USE SCHEMA` 权限。此权限是允许外部引擎访问此 Catalog 中 Schema 的关键。&#x20;
-5\. 确认授权：检查所选的权限配置无误后，点击 `Confirm` 按钮完成授权。
-### 2. 云器 Lakehouse 环境配置
-#### 2.1 环境准备检查
-**环境要求确认**：
-*   ✅ **必选**：AWS 环境的云器 Lakehouse
-*   ✅ **推荐**：与 Databricks 相同 AWS 区域部署
-#### 步骤1：建立Catalog连接
-```sql
--- 在云器 Lakehouse 中执行
-CREATE CATALOG CONNECTION IF NOT EXISTS databricks_aws_conn
-TYPE DATABRICKS
-HOST = 'https://dbc-91642d78-eab3.cloud.databricks.com/'  -- Databricks 工作空间的URL
-CLIENT_ID = 'your-service-principal-id'
-CLIENT_SECRET = 'your-service-principal-secret'
-ACCESS_REGION = 'us-west-2'
-COMMENT = 'Databricks Unity Catalog 企业级连接';
-```
-#### 步骤2：创建External Catalog
-```sql
--- 在云器 Lakehouse 中执行
-CREATE EXTERNAL CATALOG databricks_catalog
-CONNECTION databricks_aws_conn
-OPTIONS (
-    'catalog' = 'datagpt_catalog'  -- 目标的 Catalog 名称（不是 Metastore 名称）
-);
-```
-#### 步骤3：验证连通性
-```
--- 展示 Catalog 中的 Databricks 的 Schema 信息
-SHOW SCHEMAS IN databricks_catalog;
--- 展示 Databricks default Schema 下的表信息
-SHOW TABLES in databricks_catalog.default;
--- 查询从 Databricks 表的数据
-SELECT * FROM databricks_catalog.<databricks_schema>.<databricks_table> LIMIT 100;
-```
-^
-## 完整实施代码
-### Databricks 端：企业级表设计
-#### 1. 核心业务表创建
-```sql
--- ==== Databricks端实施 ====
--- 1.1 客户主数据表（TABLE_EXTERNAL）
-CREATE TABLE IF NOT EXISTS enterprise_catalog.core_data.customer_master (
-    customer_id INT,
-    customer_name STRING,
-    email STRING,
-    phone STRING,
-    registration_date DATE,
-    customer_tier STRING,
-    total_lifetime_value DOUBLE,
-    status STRING
-)
-USING DELTA
-LOCATION 's3://enterprise-data-lake/core/customer_master/'
-TBLPROPERTIES (
-    'delta.autoOptimize.optimizeWrite' = 'true',
-    'delta.autoOptimize.autoCompact' = 'true'
-)
-COMMENT '企业客户主数据表 - 跨平台核心表';
--- 1.2 订单事实表（TABLE_DELTA_EXTERNAL）
-CREATE TABLE IF NOT EXISTS enterprise_catalog.core_data.order_facts (
-    order_id BIGINT,
-    customer_id INT,
-    product_id INT,
-    product_name STRING,
-    category STRING,
-    quantity INT,
-    unit_price DOUBLE,
-    total_amount DOUBLE,
-    order_timestamp TIMESTAMP,
-    order_date DATE,
-    order_status STRING,
-    payment_method STRING
-)
-USING DELTA
-PARTITIONED BY (order_date)
-LOCATION 's3://enterprise-data-lake/facts/order_facts/'
-TBLPROPERTIES (
-    'delta.autoOptimize.optimizeWrite' = 'true',
-    'delta.autoOptimize.autoCompact' = 'true',
-    'delta.compression' = 'zstd'
-)
-COMMENT '订单事实表 - 按日期分区的高性能分析表';
--- 1.3 产品维度表
-CREATE TABLE IF NOT EXISTS enterprise_catalog.core_data.product_dimension (
-    product_id INT,
-    product_name STRING,
-    category STRING,
-    subcategory STRING,
-    brand STRING,
-    supplier_id INT,
-    cost_price DOUBLE,
-    list_price DOUBLE,
-    product_status STRING,
-    created_date DATE,
-    last_updated TIMESTAMP
-)
-USING DELTA
-LOCATION 's3://enterprise-data-lake/dimensions/product_dimension/'
-COMMENT '产品维度表 - 商品基础信息';
-```
-#### 2. 业务数据初始化
-```sql
--- ==== Databricks端数据初始化 ====
--- 2.1 客户主数据
-INSERT INTO enterprise_catalog.core_data.customer_master VALUES
-(10001, 'Global Corp', 'contact@globalcorp.com', '+1-555-0001', '2023-01-15', 'Enterprise', 125000.00, 'Active'),
-(10002, 'Tech Innovations Ltd', 'info@techinnovations.com', '+1-555-0002', '2023-02-20', 'Enterprise', 89000.00, 'Active'),
-(10003, 'Smart Solutions Inc', 'hello@smartsolutions.com', '+1-555-0003', '2023-03-10', 'Business', 45000.00, 'Active'),
-(10004, 'Digital Dynamics', 'support@digitaldynamics.com', '+1-555-0004', '2023-04-05', 'Business', 67000.00, 'Active'),
-(10005, 'Future Systems', 'sales@futuresystems.com', '+1-555-0005', '2023-05-12', 'Standard', 23000.00, 'Active');
--- 2.2 产品维度数据
-INSERT INTO enterprise_catalog.core_data.product_dimension VALUES
-(20001, 'Enterprise Server Pro', 'Hardware', 'Servers', 'TechBrand', 3001, 2500.00, 4999.99, 'Active', '2023-01-01', '2025-05-26 10:00:00'),
-(20002, 'Cloud Storage License', 'Software', 'Storage', 'CloudTech', 3002, 100.00, 299.99, 'Active', '2023-01-01', '2025-05-26 10:00:00'),
-(20003, 'Analytics Dashboard', 'Software', 'Analytics', 'DataViz', 3003, 50.00, 199.99, 'Active', '2023-02-01', '2025-05-26 10:00:00'),
-(20004, 'Security Suite Enterprise', 'Software', 'Security', 'SecureTech', 3004, 200.00, 599.99, 'Active', '2023-02-01', '2025-05-26 10:00:00'),
-(20005, 'Mobile App Platform', 'Software', 'Development', 'AppBuilder', 3005, 75.00, 249.99, 'Active', '2023-03-01', '2025-05-26 10:00:00');
--- 2.3 订单事实数据（最近30天）
-INSERT INTO enterprise_catalog.core_data.order_facts VALUES
--- 2025-05-26 订单
-(100001, 10001, 20001, 'Enterprise Server Pro', 'Hardware', 2, 4999.99, 9999.98, '2025-05-26 09:30:00', '2025-05-26', 'Completed', 'Wire Transfer'),
-(100002, 10001, 20002, 'Cloud Storage License', 'Software', 10, 299.99, 2999.90, '2025-05-26 10:15:00', '2025-05-26', 'Completed', 'Credit Card'),
-(100003, 10002, 20003, 'Analytics Dashboard', 'Software', 5, 199.99, 999.95, '2025-05-26 11:20:00', '2025-05-26', 'Processing', 'Purchase Order'),
-(100004, 10003, 20004, 'Security Suite Enterprise', 'Software', 3, 599.99, 1799.97, '2025-05-26 14:45:00', '2025-05-26', 'Shipped', 'Credit Card'),
--- 2025-05-25 订单
-(100005, 10004, 20005, 'Mobile App Platform', 'Software', 1, 249.99, 249.99, '2025-05-25 16:30:00', '2025-05-25', 'Completed', 'PayPal'),
-(100006, 10005, 20001, 'Enterprise Server Pro', 'Hardware', 1, 4999.99, 4999.99, '2025-05-25 13:20:00', '2025-05-25', 'Completed', 'Wire Transfer'),
-(100007, 10002, 20002, 'Cloud Storage License', 'Software', 20, 299.99, 5999.80, '2025-05-25 15:45:00', '2025-05-25', 'Completed', 'Purchase Order'),
--- 2025-05-24 订单
-(100008, 10001, 20003, 'Analytics Dashboard', 'Software', 8, 199.99, 1599.92, '2025-05-24 10:10:00', '2025-05-24', 'Completed', 'Credit Card'),
-(100009, 10003, 20005, 'Mobile App Platform', 'Software', 2, 249.99, 499.98, '2025-05-24 12:30:00', '2025-05-24', 'Completed', 'Credit Card'),
-(100010, 10004, 20004, 'Security Suite Enterprise', 'Software', 5, 599.99, 2999.95, '2025-05-24 14:20:00', '2025-05-24', 'Shipped', 'Purchase Order');
-```
-#### 3. 企业级表管理
-```sql
--- ==== Databricks端表管理 ====
--- 检查表配置和性能
-DESCRIBE EXTENDED enterprise_catalog.core_data.customer_master;
-DESCRIBE EXTENDED enterprise_catalog.core_data.order_facts;
-DESCRIBE EXTENDED enterprise_catalog.core_data.product_dimension;
--- 查看企业数据资产
-SHOW TABLES IN enterprise_catalog.core_data;
--- 表性能优化
-OPTIMIZE enterprise_catalog.core_data.order_facts;
-OPTIMIZE enterprise_catalog.core_data.customer_master;
--- 表统计信息更新
-ANALYZE TABLE enterprise_catalog.core_data.order_facts COMPUTE STATISTICS;
-ANALYZE TABLE enterprise_catalog.core_data.customer_master COMPUTE STATISTICS;
-```
-### 云器Lakehouse端：企业级数据分析
-#### 1. 连接状态与数据探索
-```sql
--- ==== 云器Lakehouse端实施 ====
--- 1.1 企业级连接状态检查
-SHOW CONNECTIONS;
-DESCRIBE CONNECTION databricks_aws_conn;
--- 1.2 数据资产发现
-SHOW TABLES IN databricks_business_schema;
--- 1.3 核心业务数据概览
-SELECT
-    'customer_master' as table_name,
-    COUNT(*) as total_records,
-    COUNT(DISTINCT customer_tier) as tier_count
-FROM databricks_business_schema.customer_master
-UNION ALL
-SELECT
-    'order_facts' as table_name,
-    COUNT(*) as total_records,
-    COUNT(DISTINCT order_date) as date_range
-FROM databricks_business_schema.order_facts
-UNION ALL
-SELECT
-    'product_dimension' as table_name,
-    COUNT(*) as total_records,
-    COUNT(DISTINCT category) as category_count
-FROM databricks_business_schema.product_dimension;
-```
-#### 2. 企业级业务分析
-```sql
--- ==== 云器Lakehouse端业务分析 ====
--- 2.1 客户价值分析
-WITH customer_analytics AS (
-    SELECT
-        c.customer_id,
-        c.customer_name,
-        c.customer_tier,
-        c.total_lifetime_value,
-        COUNT(DISTINCT o.order_id) as recent_orders,
-        SUM(o.total_amount) as recent_revenue,
-        AVG(o.total_amount) as avg_order_value,
-        MAX(o.order_date) as last_order_date,
-        COUNT(DISTINCT o.product_id) as product_diversity
-    FROM databricks_business_schema.customer_master c
-    LEFT JOIN databricks_business_schema.order_facts o
-        ON c.customer_id = o.customer_id
-        AND o.order_date >= CURRENT_DATE() - INTERVAL 30 DAY
-    GROUP BY c.customer_id, c.customer_name, c.customer_tier, c.total_lifetime_value
-)
-SELECT
-    customer_tier,
-    COUNT(*) as customer_count,
-    SUM(total_lifetime_value) as total_ltv,
-    AVG(total_lifetime_value) as avg_ltv,
-    SUM(recent_revenue) as recent_30d_revenue,
-    AVG(recent_orders) as avg_recent_orders,
-    AVG(product_diversity) as avg_product_diversity
-FROM customer_analytics
-GROUP BY customer_tier
-ORDER BY total_ltv DESC;
--- 2.2 产品性能分析
-SELECT
-    p.category,
-    p.subcategory,
-    COUNT(DISTINCT p.product_id) as product_count,
-    COUNT(o.order_id) as total_orders,
-    SUM(o.quantity) as total_quantity_sold,
-    SUM(o.total_amount) as total_revenue,
-    AVG(o.unit_price) as avg_selling_price,
-    AVG(p.cost_price) as avg_cost_price,
-    AVG(o.unit_price - p.cost_price) as avg_margin_per_unit
-FROM databricks_business_schema.product_dimension p
-LEFT JOIN databricks_business_schema.order_facts o ON p.product_id = o.product_id
-GROUP BY p.category, p.subcategory
-ORDER BY total_revenue DESC;
--- 2.3 时间趋势分析（利用分区优化）
-SELECT
-    order_date,
-    COUNT(DISTINCT customer_id) as active_customers,
-    COUNT(order_id) as total_orders,
-    SUM(total_amount) as daily_revenue,
-    AVG(total_amount) as avg_order_value,
-    COUNT(DISTINCT product_id) as products_sold
-FROM databricks_business_schema.order_facts
-WHERE order_date >= CURRENT_DATE() - INTERVAL 7 DAY
-GROUP BY order_date
-ORDER BY order_date DESC;
--- 2.4 客户行为深度分析
-WITH customer_behavior AS (
-    SELECT
-        c.customer_id,
-        c.customer_name,
-        c.customer_tier,
-        o.order_date,
-        o.total_amount,
-        p.category,
-        ROW_NUMBER() OVER (PARTITION BY c.customer_id ORDER BY o.order_timestamp) as order_sequence,
-        LAG(o.order_date) OVER (PARTITION BY c.customer_id ORDER BY o.order_timestamp) as prev_order_date,
-        DATEDIFF(o.order_date, LAG(o.order_date) OVER (PARTITION BY c.customer_id ORDER BY o.order_timestamp)) as days_since_last_order
-    FROM databricks_business_schema.customer_master c
-    JOIN databricks_business_schema.order_facts o ON c.customer_id = o.customer_id
-    JOIN databricks_business_schema.product_dimension p ON o.product_id = p.product_id
-    WHERE o.order_date >= CURRENT_DATE() - INTERVAL 90 DAY
-)
-SELECT
-    customer_tier,
-    COUNT(DISTINCT customer_id) as customers,
-    AVG(order_sequence) as avg_orders_per_customer,
-    AVG(days_since_last_order) as avg_days_between_orders,
-    AVG(total_amount) as avg_order_value,
-    COUNT(DISTINCT category) as categories_purchased
-FROM customer_behavior
-WHERE order_sequence > 1  -- 排除首次订单
-GROUP BY customer_tier
-ORDER BY avg_order_value DESC;
-```
-#### 3. 企业级数据质量管理
-```sql
--- ==== 云器Lakehouse端数据质量管理 ====
--- 3.1 数据完整性监控
-SELECT
-    'Data Completeness Check' as check_type,
-    'customer_master' as table_name,
-    COUNT(*) as total_records,
-    COUNT(customer_id) as non_null_ids,
-    COUNT(email) as valid_emails,
-    COUNT(CASE WHEN total_lifetime_value > 0 THEN 1 END) as positive_ltv,
-    ROUND(COUNT(email) * 100.0 / COUNT(*), 2) as email_completeness_pct
-FROM databricks_business_schema.customer_master
-UNION ALL
-SELECT
-    'Data Completeness Check' as check_type,
-    'order_facts' as table_name,
-    COUNT(*) as total_records,
-    COUNT(order_id) as non_null_ids,
-    COUNT(customer_id) as valid_customer_refs,
-    COUNT(CASE WHEN total_amount > 0 THEN 1 END) as positive_amounts,
-    ROUND(COUNT(customer_id) * 100.0 / COUNT(*), 2) as customer_ref_pct
-FROM databricks_business_schema.order_facts;
--- 3.2 数据一致性检查
-SELECT
-    'Referential Integrity' as check_type,
-    COUNT(*) as orphaned_orders
-FROM databricks_business_schema.order_facts o
-LEFT JOIN databricks_business_schema.customer_master c ON o.customer_id = c.customer_id
-WHERE c.customer_id IS NULL
-UNION ALL
-SELECT
-    'Product Reference Check' as check_type,
-    COUNT(*) as missing_product_refs
-FROM databricks_business_schema.order_facts o
-LEFT JOIN databricks_business_schema.product_dimension p ON o.product_id = p.product_id
-WHERE p.product_id IS NULL;
--- 3.3 业务规则验证
-SELECT
-    'Business Rules Validation' as check_type,
-    COUNT(CASE WHEN total_amount != quantity * unit_price THEN 1 END) as amount_calculation_errors,
-    COUNT(CASE WHEN order_date > CURRENT_DATE() THEN 1 END) as future_order_dates,
-    COUNT(CASE WHEN quantity <= 0 THEN 1 END) as invalid_quantities,
-    COUNT(CASE WHEN unit_price <= 0 THEN 1 END) as invalid_prices
-FROM databricks_business_schema.order_facts;
-```
-## 配置检查清单与维护
-### 📋 Databricks配置完成检查清单
-#### ✅ Account 级别配置
-* [ ] Service Principal 已创建并记录 CLIENT_ID 和 CLIENT_SECRET
-* [ ] Service Principal 已分配到目标 workspace（Admin 权限）
-* [ ] Unity Catalog Metastore的External Data Access已启用
-* [ ] IAM Role已创建并配置Trust Policy和Permissions Policy
-* [ ] Storage Credential已配置并关联IAM Role
-* [ ] External Location已创建并测试
-#### ✅ Workspace 级别配置
-* [ ] 专用 Catalog 已创建（enterprise_catalog）
-* [ ] 专用 Schema 已创建（core_data）
-* [ ] Service Principal已获得必要权限：
-#### ✅ 表级别配置
-* [ ] 测试外部表已创建并可访问
-* [ ] 表类型确认为 TABLE_EXTERNAL 或 TABLE_DELTA_EXTERNAL
-* [ ] 表位置指向正确的S3路径
-* [ ] 表权限已正确分配给Service Principal
-#### ✅ 云器 Lakehouse 配置
-* [ ] Catalog Connection已成功创建并测试连接
-* [ ] External Catalog已成功映射
-* [ ] External Schema已成功创建
-* [ ] 端到端数据访问测试通过
-## 问题排查指南
-### 1. 表访问权限问题
-#### 场景：EXTERNAL USE SCHEMA权限缺失
-```
-错误信息：Access denied for external schema access
-```
-**排查步骤**：
-```sql
--- Databricks端 - 检查权限配置
-SHOW GRANTS TO SERVICE_PRINCIPAL 'your-application-id';
--- 查看是否有EXTERNAL USE权限
-SELECT
-    grantee,
-    privilege_type,
-    object_type
-FROM system.information_schema.grants
-WHERE grantee = 'your-application-id'
-AND privilege_type = 'EXTERNAL_USE';
-```
-**解决方案**：
-```sql
--- 重新配置EXTERNAL USE SCHEMA权限
-GRANT EXTERNAL USE SCHEMA ON SCHEMA enterprise_catalog.core_data
-TO SERVICE_PRINCIPAL 'your-application-id';
-```
-#### 场景：外部表访问失败
-```
-错误信息：TABLE_DB_STORAGE cannot be accessed externally
-```
-**根本原因分析**：
-* 表创建时未指定外部存储位置
-* 即使使用`USING DELTA`，缺少`LOCATION`子句会创建内部存储表
-**企业级解决方案**：
-```sql
--- 在Databricks中重新设计表架构
-CREATE TABLE enterprise_catalog.core_data.table_name_v2 (
-    -- 列定义
-    column1 INT,
-    column2 STRING
-)
-USING DELTA
-LOCATION 's3://enterprise-data-lake/core/table_name_v2/';
--- 数据迁移策略
-INSERT INTO enterprise_catalog.core_data.table_name_v2
-SELECT * FROM enterprise_catalog.core_data.table_name_v1;
--- 验证表类型
-SHOW TABLE EXTENDED enterprise_catalog.core_data.table_name_v2;
-```
-### 2. 连接配置问题
-#### 场景：Service Principal认证失败
-```
-错误信息：Authentication failed for service principal
-```
-**企业级排查清单**：
-* [ ] 确认 CLIENT_ID 和 CLIENT_SECRET 正确
-* [ ] 验证Service Principal在Account Console中存在
-* [ ] 检查Service Principal的workspace权限
-* [ ] 确认Unity Catalog External Data Access已启用
-**解决步骤**：
-```sql
--- 1. 在Databricks中验证Service Principal
-SELECT current_user() as current_principal;
--- 2. 检查workspace访问权限（在Account Console中操作）
--- 3. 重新生成Secret（如需要）
--- 在Account Console → Service Principal → Secrets → Generate secret
--- 4. 在云器中更新连接信息
-ALTER CONNECTION databricks_aws_conn
-SET CLIENT_SECRET = 'new-secret-value';
-```
-### 3. 存储访问问题
-#### 场景：S3存储位置无法访问
-```
-错误信息：Access denied to S3 location
-```
-**排查步骤**：
-```sql
--- Databricks端测试存储访问
-LIST 's3://your-bucket/external-tables/';
--- 检查Storage Credential配置
-DESCRIBE STORAGE CREDENTIAL enterprise_s3_credential;
--- 检查External Location配置
-DESCRIBE EXTERNAL LOCATION enterprise_external_tables;
-```
-**解决方案**：
-```json
-// 检查IAM Role Policy配置
-{
-    "Version": "2012-10-17",
-    "Statement": [
-        {
-            "Effect": "Allow",
-            "Action": [
-                "s3:GetObject",
-                "s3:PutObject",
-                "s3:DeleteObject",
-                "s3:ListBucket",
-                "s3:GetBucketLocation"
-            ],
-            "Resource": [
-                "arn:aws:s3:::your-bucket",
-                "arn:aws:s3:::your-bucket/*"
-            ]
-        }
-    ]
-}
-```
-## 企业级部署总结
-### ✅ 核心能力确认
-1. **跨平台数据访问**：外部表实现无缝数据共享
-2. **企业级查询能力**：复杂分析、JOIN、聚合全面支持
-3. **权限管理**：基于Unity Catalog的细粒度权限控制
-4. **数据治理**：完整的元数据同步和血缘管理
-5. **安全控制**：基于存储类型的天然访问隔离
-### ⚠️ 企业级限制与约束
-1. **平台限制**：仅支持 AWS 环境的 Databricks + 云器组合
-2. **表类型要求**：必须使用外部存储表实现跨平台访问
-3. **权限依赖**：需要完整的 Unity Catalog 权限配置，特别是 EXTERNAL USE SCHEMA 权限
-4. **存储依赖**：需要正确配置Storage Credential和External Location
-## 参考资料
-[External Catalog简介](external-catalog-summary.md)
-***
-*注：本指南基于2025年11月的云器Lakehouse版本测试结果，后续版本可能有所变化。请定期检查官方文档以获取最新信息。*