@clickzetta/cz-cli-darwin-x64 0.3.1

package/bin/skills/lakehouse-doc/references/databricks_yunqi_integration_guide_v2.md

@@ -0,0 +1,811 @@
+# Databricks-云器Lakehouse 跨平台数据联邦最佳实践指南
+
+## 概述
+
+本指南基于企业级生产环境的成功实施经验，详细阐述如何实现 Databricks 与云器 Lakehouse 之间的跨平台数据联邦。本文档涵盖完整的架构设计、实施方案和运维最佳实践，为企业数据平台建设提供指导。
+
+## 技术实现原理与特点
+
+### 什么是跨平台数据联邦
+
+**数据联邦（Data Federation）** 是一种分布式数据架构模式，允许多个独立的数据系统通过统一的接口进行数据访问和查询，而无需物理移动或复制数据。
+
+### 核心技术实现特点
+
+#### **与传统数据集成的根本区别**
+
+| 特性        | 传统数据集成（ETL/ELT） | 跨平台数据联邦      |
+| --------- | --------------- | ------------ |
+| **数据存储**  | 数据复制到目标系统       | 数据保持在源存储位置   |
+| **数据同步**  | 定期ETL作业同步       | 实时元数据联邦访问    |
+| **存储成本**  | 双倍存储成本          | 单份存储，共享访问    |
+| **数据一致性** | 可能存在延迟和差异       | 访问同一数据源，天然一致 |
+| **实施复杂度** | 需要复杂的管道维护       | 配置元数据连接即可    |
+| **查询性能**  | 本地查询性能优异        | 跨网络查询，需要优化   |
+
+#### **技术架构核心原理**
+
+<-------------------|  | Metadata Store |  |
+|  +----------------+  |                     |  +----------------+  |
++----------------------+                     +---------------------+
+         |                                         |
+         |                                         |
+   Data File Access                           Permission Check
+         |                                         |
+         v                                         v
++--------------------------------------------------------------+
+|                        AWS S3 Storage                        |
+|  +-------------------------------------------------------+   |
+|  | s3://bucket/external-tables/customer/                 |   |
+|  |  - _delta_log/                                        |   |
+|  |  - part-00000.parquet                                 |   |
+|  |  - part-00001.parquet                                 |   |
+|  |  - ...                                                |   |
+|  +-------------------------------------------------------+   |
+|                                                              |
+|  * Both Databricks and Yunqi access the same files directly  |
++--------------------------------------------------------------+
+```
+
+#### **核心技术优势**
+
+**1. 存储经济性**
+
+* 数据只存储一份，节省50%+ 存储成本
+* 无需维护复杂的数据同步管道
+* 减少数据传输带宽成本
+
+**2. 数据一致性**
+
+* 两个平台访问相同的数据文件
+* 无数据同步延迟问题
+* 天然保证数据一致性
+
+**3. 架构简洁性**
+
+* 无需ETL管道开发和维护
+* 配置即用，实施周期短
+* 减少数据管道故障点
+
+**4. 安全可控性**
+
+* 统一的权限管理体系
+* 基于存储类型的天然访问隔离
+* 细粒度的表和列级权限控制
+
+#### **适用场景**
+
+**推荐场景**：
+
+* 需要在多个分析平台查询相同数据集
+* 希望减少数据存储和传输成本
+* 要求数据实时一致性
+* 数据主要用于读取分析（OLAP场景）
+* 已采用Delta Lake或类似格式的企业
+
+**不适用场景**：
+
+* 需要频繁的跨平台数据写入操作
+* 对查询延迟要求极高（毫秒级）
+* 数据安全要求完全物理隔离
+* 网络环境不稳定或带宽受限
+
+#### **技术限制与考虑**
+
+**性能考虑**：
+
+* 跨网络查询比本地查询延迟略高
+* 大数据量查询需要优化分区策略
+* 并发查询数量受网络带宽限制
+
+**网络依赖**：
+
+* 需要稳定的 AWS 区域内网络连接
+* S3访问权限必须正确配置
+* Unity Catalog服务必须可达
+
+## 环境要求与支持范围
+
+**支持的技术栈组合**：
+
+* **推荐配置**：AWS 上的 Databricks + AWS 上的云器 Lakehouse
+* **不支持**：暂不支持其它云平台组合
+
+**前置技术要求**：
+
+* Databricks Unity Catalog 已部署并启用 External Data Access
+* Service Principal权限配置完成
+* S3存储访问策略已建立
+* 云器 Lakehouse（AWS 版本）已就绪
+* 稳定的AWS区域内网络连接
+
+## 架构设计
+
+### 技术架构图
+
+```
++-------------------+    +--------------------+    +---------------------+
+|   Databricks      |    |  Unity Catalog     |    |    Yunqi Lakehouse  |
+|   (AWS)           |    |  (AWS)             |    |    (AWS)            |
+|                   |    |                    |    |                     |
+| +---------------+ |    | +----------------+ |    | +-----------------+ |
+| | External Tbl  | |    | | Metadata       | |    | | Federated Query | |
+| |               | |<-->
+
+### 数据访问模式
+
+**企业级数据访问策略**：
+
+| Databricks 表类型               | 云器访问性 | 生产建议 |
+| ---------------------------- | ----- | ---- |
+| `Managed Iceberg`            | 支持    | 推荐使用 |
+| `Managed Delta`              | 支持    | 推荐使用 |
+| `View` / `Materialized View` | 暂不支持  | 转换使用 |
+| `Streaming Table`            | 暂不支持  | 转换使用 |
+
+## 环境配置
+
+### **重要说明：配置必须严格按照以下顺序执行**
+
+```
+配置依赖关系和顺序：
+1. Account级别：Service Principal创建和配置
+2. Account级别：Unity Catalog Metastore External Data Access启用
+3. Account级别：Storage Credential和External Location配置
+4. Workspace级别：Catalog和Schema创建
+5. Workspace级别：Service Principal权限配置
+6. 端到端测试和验证
+```
+
+### 1. Databricks 关键配置
+
+#### 1.1 Service Principal创建和配置
+
+**步骤1：创建Service Principal（Account Console操作）**
+
+1. 登录 **Databricks Account Console**（注意：不是workspace）
+2. 导航路径：`Account settings` → `User management` → `Service principals`
+3. 点击 `Add service principal`
+4. 填写配置：
+   * **Display name**: `Lakehouse-Integration-SP`
+   * **Application ID**: 自动生成（**重要：记录此CLIENT\_ID**）
+5. 点击 `Add` 创建
+
+**步骤2：生成 Secret（Account Console 操作）**
+
+1. 进入创建的 Service Principal 详情页
+2. 点击 `Secrets` tab
+3. 点击 `Generate secret`
+4. **重要**：立即复制并安全保存 Secret 值（只显示一次）
+5. 记录以下信息用于云器连接：
+   ```
+   CLIENT_ID
+   CLIENT_SECRET
+   ```
+
+**步骤3：分配 workspace 权限（Account Console 操作）**
+
+1. 在Account Console中，导航到 `Workspaces`
+2. 选择目标workspace
+3. 点击 `Permissions` tab
+4. 点击 `Add permissions`
+5. 搜索并选择创建的 Service Principal
+6. 分配权限：`Admin`（推荐用于初始配置，后续可调整为User）
+
+#### 1.2 Unity Catalog Metastore 配置
+
+**步骤1：检查Metastore状态（Workspace SQL编辑器）**
+
+```sql
+-- 在Databricks workspace中执行
+DESCRIBE METASTORE;
+
+-- 检查输出中的关键信息：
+-- Metastore Name: metastore名称
+-- Cloud: 应该是 'aws'
+-- Region: 应该与云器相同
+-- Privileged Model Version: 应该是 '1.0' 或更高
+```
+
+**步骤2：启用External Data Access（Account Console操作）**
+
+1. 在 Databricks 工作区左侧的主导航栏中，点击 `Catalog\` 选项进入 Catalog Explorer。
+2. 在 Catalog Explorer 的主界面上方，点击齿轮形状的设置图标 (Manage)。
+3. 在弹出的下拉菜单中，点击 `Metastore\` 选项。
+4. 在 Metastore 页面确保以下选项已启用：
+   ```
+   ✅ External data access: Enabled
+   ```
+
+**步骤3：Service Principal 权限配置**
+
+1. 进入 Databricks Workspace Console。
+
+* 在左侧边栏中，点击 `Catalog` 进入 Catalog 浏览器。&#x20;
+* 在 Catalog 列表中，点击您希望授权的目标 Catalog，例如 `databricks_catalog`。&#x20;
+
+2\. 打开权限管理
+
+* 在所选 Catalog 的主页面中，点击 Permissions 标签页。&#x20;
+* 点击 Grant 按钮以打开授权对话框。&#x20;
+
+3\. 选择主体 (Principal)&#x20;
+
+* 在弹出的 Grant on <Catalog 名称> 对话框中，于 Principals 字段搜索并选择您的服务主体。
+
+4\. 分配权限
+
+* 使用权限预设 (Privilege presets)：为了简化配置，建议使用预设角色。对于需要读写和创建对象的场景，从下拉菜单中选择 Data Editor。此预设会自动授予一系列常用权限，如 `USE CATALOG, USE SCHEMA, SELECT, MODIFY, CREATE TABLE` 等。
+* 授予外部访问权限 (关键步骤)：如果您需要允许外部系统（非 Databricks）通过此服务主体访问数据，请务必勾选页面底部的` EXTERNAL USE SCHEMA` 权限。此权限是允许外部引擎访问此 Catalog 中 Schema 的关键。&#x20;
+
+5\. 确认授权：检查所选的权限配置无误后，点击 `Confirm` 按钮完成授权。
+
+### 2. 云器 Lakehouse 环境配置
+
+#### 2.1 环境准备检查
+
+**环境要求确认**：
+
+*   ✅ **必选**：AWS 环境的云器 Lakehouse
+*   ✅ **推荐**：与 Databricks 相同 AWS 区域部署
+
+#### 步骤1：建立Catalog连接
+
+```sql
+-- 在云器 Lakehouse 中执行
+CREATE CATALOG CONNECTION IF NOT EXISTS databricks_aws_conn
+TYPE DATABRICKS
+HOST = 'https://dbc-91642d78-eab3.cloud.databricks.com/'  -- Databricks 工作空间的URL
+CLIENT_ID = 'your-service-principal-id'
+CLIENT_SECRET = 'your-service-principal-secret'
+ACCESS_REGION = 'us-west-2'
+COMMENT = 'Databricks Unity Catalog 企业级连接';
+```
+
+#### 步骤2：创建External Catalog
+
+```sql
+-- 在云器 Lakehouse 中执行
+CREATE EXTERNAL CATALOG databricks_catalog 
+CONNECTION databricks_aws_conn 
+OPTIONS (
+    'catalog' = 'datagpt_catalog'  -- 目标的 Catalog 名称（不是 Metastore 名称）
+);
+```
+
+#### 步骤3：验证连通性
+
+```
+-- 展示 Catalog 中的 Databricks 的 Schema 信息
+SHOW SCHEMAS IN databricks_catalog;
+
+-- 展示 Databricks default Schema 下的表信息
+SHOW TABLES in databricks_catalog.default;
+
+-- 查询从 Databricks 表的数据
+SELECT * FROM databricks_catalog.<databricks_schema>.<databricks_table> LIMIT 100;
+
+```
+
+^
+
+## 完整实施代码
+
+### Databricks 端：企业级表设计
+
+#### 1. 核心业务表创建
+
+```sql
+-- ==== Databricks端实施 ====
+
+-- 1.1 客户主数据表（TABLE_EXTERNAL）
+CREATE TABLE IF NOT EXISTS enterprise_catalog.core_data.customer_master (
+    customer_id INT,
+    customer_name STRING,
+    email STRING,
+    phone STRING,
+    registration_date DATE,
+    customer_tier STRING,
+    total_lifetime_value DOUBLE,
+    status STRING
+) 
+USING DELTA
+LOCATION 's3://enterprise-data-lake/core/customer_master/'
+TBLPROPERTIES (
+    'delta.autoOptimize.optimizeWrite' = 'true',
+    'delta.autoOptimize.autoCompact' = 'true'
+)
+COMMENT '企业客户主数据表 - 跨平台核心表';
+
+-- 1.2 订单事实表（TABLE_DELTA_EXTERNAL）
+CREATE TABLE IF NOT EXISTS enterprise_catalog.core_data.order_facts (
+    order_id BIGINT,
+    customer_id INT,
+    product_id INT,
+    product_name STRING,
+    category STRING,
+    quantity INT,
+    unit_price DOUBLE,
+    total_amount DOUBLE,
+    order_timestamp TIMESTAMP,
+    order_date DATE,
+    order_status STRING,
+    payment_method STRING
+)
+USING DELTA
+PARTITIONED BY (order_date)
+LOCATION 's3://enterprise-data-lake/facts/order_facts/'
+TBLPROPERTIES (
+    'delta.autoOptimize.optimizeWrite' = 'true',
+    'delta.autoOptimize.autoCompact' = 'true',
+    'delta.compression' = 'zstd'
+)
+COMMENT '订单事实表 - 按日期分区的高性能分析表';
+
+-- 1.3 产品维度表
+CREATE TABLE IF NOT EXISTS enterprise_catalog.core_data.product_dimension (
+    product_id INT,
+    product_name STRING,
+    category STRING,
+    subcategory STRING,
+    brand STRING,
+    supplier_id INT,
+    cost_price DOUBLE,
+    list_price DOUBLE,
+    product_status STRING,
+    created_date DATE,
+    last_updated TIMESTAMP
+)
+USING DELTA
+LOCATION 's3://enterprise-data-lake/dimensions/product_dimension/'
+COMMENT '产品维度表 - 商品基础信息';
+```
+
+#### 2. 业务数据初始化
+
+```sql
+-- ==== Databricks端数据初始化 ====
+
+-- 2.1 客户主数据
+INSERT INTO enterprise_catalog.core_data.customer_master VALUES 
+(10001, 'Global Corp', 'contact@globalcorp.com', '+1-555-0001', '2023-01-15', 'Enterprise', 125000.00, 'Active'),
+(10002, 'Tech Innovations Ltd', 'info@techinnovations.com', '+1-555-0002', '2023-02-20', 'Enterprise', 89000.00, 'Active'),
+(10003, 'Smart Solutions Inc', 'hello@smartsolutions.com', '+1-555-0003', '2023-03-10', 'Business', 45000.00, 'Active'),
+(10004, 'Digital Dynamics', 'support@digitaldynamics.com', '+1-555-0004', '2023-04-05', 'Business', 67000.00, 'Active'),
+(10005, 'Future Systems', 'sales@futuresystems.com', '+1-555-0005', '2023-05-12', 'Standard', 23000.00, 'Active');
+
+-- 2.2 产品维度数据
+INSERT INTO enterprise_catalog.core_data.product_dimension VALUES 
+(20001, 'Enterprise Server Pro', 'Hardware', 'Servers', 'TechBrand', 3001, 2500.00, 4999.99, 'Active', '2023-01-01', '2025-05-26 10:00:00'),
+(20002, 'Cloud Storage License', 'Software', 'Storage', 'CloudTech', 3002, 100.00, 299.99, 'Active', '2023-01-01', '2025-05-26 10:00:00'),
+(20003, 'Analytics Dashboard', 'Software', 'Analytics', 'DataViz', 3003, 50.00, 199.99, 'Active', '2023-02-01', '2025-05-26 10:00:00'),
+(20004, 'Security Suite Enterprise', 'Software', 'Security', 'SecureTech', 3004, 200.00, 599.99, 'Active', '2023-02-01', '2025-05-26 10:00:00'),
+(20005, 'Mobile App Platform', 'Software', 'Development', 'AppBuilder', 3005, 75.00, 249.99, 'Active', '2023-03-01', '2025-05-26 10:00:00');
+
+-- 2.3 订单事实数据（最近30天）
+INSERT INTO enterprise_catalog.core_data.order_facts VALUES 
+-- 2025-05-26 订单
+(100001, 10001, 20001, 'Enterprise Server Pro', 'Hardware', 2, 4999.99, 9999.98, '2025-05-26 09:30:00', '2025-05-26', 'Completed', 'Wire Transfer'),
+(100002, 10001, 20002, 'Cloud Storage License', 'Software', 10, 299.99, 2999.90, '2025-05-26 10:15:00', '2025-05-26', 'Completed', 'Credit Card'),
+(100003, 10002, 20003, 'Analytics Dashboard', 'Software', 5, 199.99, 999.95, '2025-05-26 11:20:00', '2025-05-26', 'Processing', 'Purchase Order'),
+(100004, 10003, 20004, 'Security Suite Enterprise', 'Software', 3, 599.99, 1799.97, '2025-05-26 14:45:00', '2025-05-26', 'Shipped', 'Credit Card'),
+
+-- 2025-05-25 订单
+(100005, 10004, 20005, 'Mobile App Platform', 'Software', 1, 249.99, 249.99, '2025-05-25 16:30:00', '2025-05-25', 'Completed', 'PayPal'),
+(100006, 10005, 20001, 'Enterprise Server Pro', 'Hardware', 1, 4999.99, 4999.99, '2025-05-25 13:20:00', '2025-05-25', 'Completed', 'Wire Transfer'),
+(100007, 10002, 20002, 'Cloud Storage License', 'Software', 20, 299.99, 5999.80, '2025-05-25 15:45:00', '2025-05-25', 'Completed', 'Purchase Order'),
+
+-- 2025-05-24 订单
+(100008, 10001, 20003, 'Analytics Dashboard', 'Software', 8, 199.99, 1599.92, '2025-05-24 10:10:00', '2025-05-24', 'Completed', 'Credit Card'),
+(100009, 10003, 20005, 'Mobile App Platform', 'Software', 2, 249.99, 499.98, '2025-05-24 12:30:00', '2025-05-24', 'Completed', 'Credit Card'),
+(100010, 10004, 20004, 'Security Suite Enterprise', 'Software', 5, 599.99, 2999.95, '2025-05-24 14:20:00', '2025-05-24', 'Shipped', 'Purchase Order');
+```
+
+#### 3. 企业级表管理
+
+```sql
+-- ==== Databricks端表管理 ====
+
+-- 检查表配置和性能
+DESCRIBE EXTENDED enterprise_catalog.core_data.customer_master;
+DESCRIBE EXTENDED enterprise_catalog.core_data.order_facts;
+DESCRIBE EXTENDED enterprise_catalog.core_data.product_dimension;
+
+-- 查看企业数据资产
+SHOW TABLES IN enterprise_catalog.core_data;
+
+-- 表性能优化
+OPTIMIZE enterprise_catalog.core_data.order_facts;
+OPTIMIZE enterprise_catalog.core_data.customer_master;
+
+-- 表统计信息更新
+ANALYZE TABLE enterprise_catalog.core_data.order_facts COMPUTE STATISTICS;
+ANALYZE TABLE enterprise_catalog.core_data.customer_master COMPUTE STATISTICS;
+```
+
+### 云器Lakehouse端：企业级数据分析
+
+#### 1. 连接状态与数据探索
+
+```sql
+-- ==== 云器Lakehouse端实施 ====
+
+-- 1.1 企业级连接状态检查
+SHOW CONNECTIONS;
+DESCRIBE CONNECTION databricks_aws_conn;
+
+-- 1.2 数据资产发现
+SHOW TABLES IN databricks_business_schema;
+
+-- 1.3 核心业务数据概览
+SELECT 
+    'customer_master' as table_name,
+    COUNT(*) as total_records,
+    COUNT(DISTINCT customer_tier) as tier_count
+FROM databricks_business_schema.customer_master
+
+UNION ALL
+
+SELECT 
+    'order_facts' as table_name,
+    COUNT(*) as total_records,
+    COUNT(DISTINCT order_date) as date_range
+FROM databricks_business_schema.order_facts
+
+UNION ALL
+
+SELECT 
+    'product_dimension' as table_name,
+    COUNT(*) as total_records,
+    COUNT(DISTINCT category) as category_count
+FROM databricks_business_schema.product_dimension;
+```
+
+#### 2. 企业级业务分析
+
+```sql
+-- ==== 云器Lakehouse端业务分析 ====
+
+-- 2.1 客户价值分析
+WITH customer_analytics AS (
+    SELECT 
+        c.customer_id,
+        c.customer_name,
+        c.customer_tier,
+        c.total_lifetime_value,
+        COUNT(DISTINCT o.order_id) as recent_orders,
+        SUM(o.total_amount) as recent_revenue,
+        AVG(o.total_amount) as avg_order_value,
+        MAX(o.order_date) as last_order_date,
+        COUNT(DISTINCT o.product_id) as product_diversity
+    FROM databricks_business_schema.customer_master c
+    LEFT JOIN databricks_business_schema.order_facts o 
+        ON c.customer_id = o.customer_id 
+        AND o.order_date >= CURRENT_DATE() - INTERVAL 30 DAY
+    GROUP BY c.customer_id, c.customer_name, c.customer_tier, c.total_lifetime_value
+)
+SELECT 
+    customer_tier,
+    COUNT(*) as customer_count,
+    SUM(total_lifetime_value) as total_ltv,
+    AVG(total_lifetime_value) as avg_ltv,
+    SUM(recent_revenue) as recent_30d_revenue,
+    AVG(recent_orders) as avg_recent_orders,
+    AVG(product_diversity) as avg_product_diversity
+FROM customer_analytics
+GROUP BY customer_tier
+ORDER BY total_ltv DESC;
+
+-- 2.2 产品性能分析
+SELECT 
+    p.category,
+    p.subcategory,
+    COUNT(DISTINCT p.product_id) as product_count,
+    COUNT(o.order_id) as total_orders,
+    SUM(o.quantity) as total_quantity_sold,
+    SUM(o.total_amount) as total_revenue,
+    AVG(o.unit_price) as avg_selling_price,
+    AVG(p.cost_price) as avg_cost_price,
+    AVG(o.unit_price - p.cost_price) as avg_margin_per_unit
+FROM databricks_business_schema.product_dimension p
+LEFT JOIN databricks_business_schema.order_facts o ON p.product_id = o.product_id
+GROUP BY p.category, p.subcategory
+ORDER BY total_revenue DESC;
+
+-- 2.3 时间趋势分析（利用分区优化）
+SELECT 
+    order_date,
+    COUNT(DISTINCT customer_id) as active_customers,
+    COUNT(order_id) as total_orders,
+    SUM(total_amount) as daily_revenue,
+    AVG(total_amount) as avg_order_value,
+    COUNT(DISTINCT product_id) as products_sold
+FROM databricks_business_schema.order_facts
+WHERE order_date >= CURRENT_DATE() - INTERVAL 7 DAY
+GROUP BY order_date
+ORDER BY order_date DESC;
+
+-- 2.4 客户行为深度分析
+WITH customer_behavior AS (
+    SELECT 
+        c.customer_id,
+        c.customer_name,
+        c.customer_tier,
+        o.order_date,
+        o.total_amount,
+        p.category,
+        ROW_NUMBER() OVER (PARTITION BY c.customer_id ORDER BY o.order_timestamp) as order_sequence,
+        LAG(o.order_date) OVER (PARTITION BY c.customer_id ORDER BY o.order_timestamp) as prev_order_date,
+        DATEDIFF(o.order_date, LAG(o.order_date) OVER (PARTITION BY c.customer_id ORDER BY o.order_timestamp)) as days_since_last_order
+    FROM databricks_business_schema.customer_master c
+    JOIN databricks_business_schema.order_facts o ON c.customer_id = o.customer_id
+    JOIN databricks_business_schema.product_dimension p ON o.product_id = p.product_id
+    WHERE o.order_date >= CURRENT_DATE() - INTERVAL 90 DAY
+)
+SELECT 
+    customer_tier,
+    COUNT(DISTINCT customer_id) as customers,
+    AVG(order_sequence) as avg_orders_per_customer,
+    AVG(days_since_last_order) as avg_days_between_orders,
+    AVG(total_amount) as avg_order_value,
+    COUNT(DISTINCT category) as categories_purchased
+FROM customer_behavior
+WHERE order_sequence > 1  -- 排除首次订单
+GROUP BY customer_tier
+ORDER BY avg_order_value DESC;
+```
+
+#### 3. 企业级数据质量管理
+
+```sql
+-- ==== 云器Lakehouse端数据质量管理 ====
+
+-- 3.1 数据完整性监控
+SELECT 
+    'Data Completeness Check' as check_type,
+    'customer_master' as table_name,
+    COUNT(*) as total_records,
+    COUNT(customer_id) as non_null_ids,
+    COUNT(email) as valid_emails,
+    COUNT(CASE WHEN total_lifetime_value > 0 THEN 1 END) as positive_ltv,
+    ROUND(COUNT(email) * 100.0 / COUNT(*), 2) as email_completeness_pct
+FROM databricks_business_schema.customer_master
+
+UNION ALL
+
+SELECT 
+    'Data Completeness Check' as check_type,
+    'order_facts' as table_name,
+    COUNT(*) as total_records,
+    COUNT(order_id) as non_null_ids,
+    COUNT(customer_id) as valid_customer_refs,
+    COUNT(CASE WHEN total_amount > 0 THEN 1 END) as positive_amounts,
+    ROUND(COUNT(customer_id) * 100.0 / COUNT(*), 2) as customer_ref_pct
+FROM databricks_business_schema.order_facts;
+
+-- 3.2 数据一致性检查
+SELECT 
+    'Referential Integrity' as check_type,
+    COUNT(*) as orphaned_orders
+FROM databricks_business_schema.order_facts o
+LEFT JOIN databricks_business_schema.customer_master c ON o.customer_id = c.customer_id
+WHERE c.customer_id IS NULL
+
+UNION ALL
+
+SELECT 
+    'Product Reference Check' as check_type,
+    COUNT(*) as missing_product_refs
+FROM databricks_business_schema.order_facts o
+LEFT JOIN databricks_business_schema.product_dimension p ON o.product_id = p.product_id
+WHERE p.product_id IS NULL;
+
+-- 3.3 业务规则验证
+SELECT 
+    'Business Rules Validation' as check_type,
+    COUNT(CASE WHEN total_amount != quantity * unit_price THEN 1 END) as amount_calculation_errors,
+    COUNT(CASE WHEN order_date > CURRENT_DATE() THEN 1 END) as future_order_dates,
+    COUNT(CASE WHEN quantity <= 0 THEN 1 END) as invalid_quantities,
+    COUNT(CASE WHEN unit_price <= 0 THEN 1 END) as invalid_prices
+FROM databricks_business_schema.order_facts;
+```
+
+## 配置检查清单与维护
+
+### 📋 Databricks配置完成检查清单
+
+#### ✅ Account 级别配置
+
+* [ ] Service Principal 已创建并记录 CLIENT_ID 和 CLIENT_SECRET
+* [ ] Service Principal 已分配到目标 workspace（Admin 权限）
+* [ ] Unity Catalog Metastore的External Data Access已启用
+* [ ] IAM Role已创建并配置Trust Policy和Permissions Policy
+* [ ] Storage Credential已配置并关联IAM Role
+* [ ] External Location已创建并测试
+
+#### ✅ Workspace 级别配置
+
+* [ ] 专用 Catalog 已创建（enterprise_catalog）
+* [ ] 专用 Schema 已创建（core_data）
+* [ ] Service Principal已获得必要权限：
+
+#### ✅ 表级别配置
+
+* [ ] 测试外部表已创建并可访问
+* [ ] 表类型确认为 TABLE_EXTERNAL 或 TABLE_DELTA_EXTERNAL
+* [ ] 表位置指向正确的S3路径
+* [ ] 表权限已正确分配给Service Principal
+
+#### ✅ 云器 Lakehouse 配置
+
+* [ ] Catalog Connection已成功创建并测试连接
+* [ ] External Catalog已成功映射
+* [ ] External Schema已成功创建
+* [ ] 端到端数据访问测试通过
+
+## 问题排查指南
+
+### 1. 表访问权限问题
+
+#### 场景：EXTERNAL USE SCHEMA权限缺失
+
+```
+错误信息：Access denied for external schema access
+```
+
+**排查步骤**：
+
+```sql
+-- Databricks端 - 检查权限配置
+SHOW GRANTS TO SERVICE_PRINCIPAL 'your-application-id';
+
+-- 查看是否有EXTERNAL USE权限
+SELECT 
+    grantee,
+    privilege_type,
+    object_type
+FROM system.information_schema.grants 
+WHERE grantee = 'your-application-id'
+AND privilege_type = 'EXTERNAL_USE';
+```
+
+**解决方案**：
+
+```sql
+-- 重新配置EXTERNAL USE SCHEMA权限
+GRANT EXTERNAL USE SCHEMA ON SCHEMA enterprise_catalog.core_data 
+TO SERVICE_PRINCIPAL 'your-application-id';
+```
+
+#### 场景：外部表访问失败
+
+```
+错误信息：TABLE_DB_STORAGE cannot be accessed externally
+```
+
+**根本原因分析**：
+
+* 表创建时未指定外部存储位置
+* 即使使用`USING DELTA`，缺少`LOCATION`子句会创建内部存储表
+
+**企业级解决方案**：
+
+```sql
+-- 在Databricks中重新设计表架构
+CREATE TABLE enterprise_catalog.core_data.table_name_v2 (
+    -- 列定义
+    column1 INT,
+    column2 STRING
+) 
+USING DELTA 
+LOCATION 's3://enterprise-data-lake/core/table_name_v2/';
+
+-- 数据迁移策略
+INSERT INTO enterprise_catalog.core_data.table_name_v2 
+SELECT * FROM enterprise_catalog.core_data.table_name_v1;
+
+-- 验证表类型
+SHOW TABLE EXTENDED enterprise_catalog.core_data.table_name_v2;
+```
+
+### 2. 连接配置问题
+
+#### 场景：Service Principal认证失败
+
+```
+错误信息：Authentication failed for service principal
+```
+
+**企业级排查清单**：
+
+* [ ] 确认 CLIENT_ID 和 CLIENT_SECRET 正确
+* [ ] 验证Service Principal在Account Console中存在
+* [ ] 检查Service Principal的workspace权限
+* [ ] 确认Unity Catalog External Data Access已启用
+
+**解决步骤**：
+
+```sql
+-- 1. 在Databricks中验证Service Principal
+SELECT current_user() as current_principal;
+
+-- 2. 检查workspace访问权限（在Account Console中操作）
+
+-- 3. 重新生成Secret（如需要）
+-- 在Account Console → Service Principal → Secrets → Generate secret
+
+-- 4. 在云器中更新连接信息
+ALTER CONNECTION databricks_aws_conn 
+SET CLIENT_SECRET = 'new-secret-value';
+```
+
+### 3. 存储访问问题
+
+#### 场景：S3存储位置无法访问
+
+```
+错误信息：Access denied to S3 location
+```
+
+**排查步骤**：
+
+```sql
+-- Databricks端测试存储访问
+LIST 's3://your-bucket/external-tables/';
+
+-- 检查Storage Credential配置
+DESCRIBE STORAGE CREDENTIAL enterprise_s3_credential;
+
+-- 检查External Location配置
+DESCRIBE EXTERNAL LOCATION enterprise_external_tables;
+```
+
+**解决方案**：
+
+```json
+// 检查IAM Role Policy配置
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+                "s3:GetObject",
+                "s3:PutObject",
+                "s3:DeleteObject",
+                "s3:ListBucket",
+                "s3:GetBucketLocation"
+            ],
+            "Resource": [
+                "arn:aws:s3:::your-bucket",
+                "arn:aws:s3:::your-bucket/*"
+            ]
+        }
+    ]
+}
+```
+
+## 企业级部署总结
+
+### ✅ 核心能力确认
+
+1. **跨平台数据访问**：外部表实现无缝数据共享
+2. **企业级查询能力**：复杂分析、JOIN、聚合全面支持
+3. **权限管理**：基于Unity Catalog的细粒度权限控制
+4. **数据治理**：完整的元数据同步和血缘管理
+5. **安全控制**：基于存储类型的天然访问隔离
+
+### ⚠️ 企业级限制与约束
+
+1. **平台限制**：仅支持 AWS 环境的 Databricks + 云器组合
+2. **表类型要求**：必须使用外部存储表实现跨平台访问
+3. **权限依赖**：需要完整的 Unity Catalog 权限配置，特别是 EXTERNAL USE SCHEMA 权限
+4. **存储依赖**：需要正确配置Storage Credential和External Location
+
+## 参考资料
+
+[External Catalog简介](external-catalog-summary.md)
+
+***
+
+*注：本指南基于2025年11月的云器Lakehouse版本测试结果，后续版本可能有所变化。请定期检查官方文档以获取最新信息。*