@clickzetta/cz-cli-darwin-x64 0.3.1

package/bin/skills/lakehouse-doc/references/MigrateSnowflakeRealtimeETLPipelinetoClickzettaLakehouse.md

@@ -0,0 +1,865 @@
+# 将Snowflake的实时ETL Pipeline迁移到云器Lakehouse
+
+## 亮点
+
+本文基于云器Lakehouse，快速实现了将Snowflake的实时ETL Pipeline迁移到云器Lakehouse，并发现基于云器Lakehouse的方案具备如下独特优势：
+
+* 全球多云支持。本方案中云器Lakehouse和Snowflake都基于AWS，提供了云体验的一致性。同样本方案也适合在GCP上的迁移。同时云器Lakehouse还支持阿里云、腾讯云、华为云，而不仅仅是海外主流云服务提供商的支持。
+* 迁移成本低。
+  * 云器Lakehouse提供了和Snowflake非常相似的产品概念，这对熟悉Snowflake的用户而言，非常容易理解和上手。
+  * 云器Lakehouse的SQL语法和Snowflake高度兼容，只需要做很少的Tiny修改即可实现代码迁移。
+* 运维成本低
+  * 云器Lakehouse提供了内置的Python运行环境并和SQL等任务实现统一的调度、运维和监控。不需要额外的Airflow服务、Python运行环境，大幅简化了系统架构，降低了运维难度和成本。
+
+## Real-Time Insurance Data ETL Pipeline with Snowflake项目介绍
+
+### 项目概况
+
+如果你熟悉Snowflake的实际操作和应用，那么Github上的[Real-Time Insurance Data ETL Pipeline with Snowflake](https://github.com/ravishankar324/Real-Time-Insurance-claims-Data-ETL-Pipeline) 对你是再熟悉不过了。该项目涉及在Snowflake中创建实时 ETL（提取、转换、加载）数据管道，以处理来自 Kaggle 的保险数据并将其加载到 Snowflake 中。数据管道使用 Apache Airflow 进行管理，涉及几个步骤来清理、规范化和转换数据，然后将其加载到 Snowflake 进行进一步处理。AWS S3 用作数据湖，用于存储原始数据和转换后的数据。
+
+开发完该项目后，可以根据需求安排 Airflow DAG 运行，确保 ETL 流程以所需的频率执行。最终清理、规范化和转换后的数据将可在 Tableau 中进行实时可视化，从而获得最新的见解和报告。项目架构图如下:
+
+:-: ![](.topwrite/assets/image_1737353888502.png =814)
+
+### 使用的工具和服务
+
+* **Python**：用于脚本编写和数据处理。
+* **Pandas**：用于数据操作和分析。
+* **AWS S3**：作为存储原始数据和转换后数据的数据湖。
+* **Snowflake**：用于数据建模和存储。
+* **Apache Airflow**：用于协调 ETL 工作流程。
+* **EC2**：用于托管 Airflow 环境。
+* **Kaggle API**：用于从 Kaggle 提取数据。
+* **Tableau**：用于数据可视化。
+
+### ETL 工作流
+
+1. **数据提取**
+
+   * 创建 Airflow DAG 脚本，使用 Kaggle API 从 Kaggle 保险数据集中提取随机数量的行。
+
+   * 清理、规范化并将数据转换为四个表：
+
+     * `policy_data`
+     * `customers_data`
+     * `vehicles_data`
+     * `claims_data`
+
+2. **数据存储**
+
+   * 将转换后的数据存储在 S3 存储桶中，并针对每种类型的规范化数据 ( `policy_data`、`customers_data`、`vehicles_data`、`claims_data`) 组织到不同的文件夹中。
+
+3. **Snowflake中的数据处理**
+
+   * 创建 Snowflake SQL 工作表来定义数据库模式。
+   * 在 Snowflake 中为每种类型的规范化数据创建暂存表。
+   * 定义 Snowpipes 以自动将数据从 S3 存储桶提取到暂存表。
+   * 为每个暂存表创建流对象以捕获变化。
+   * 创建最终表以合并来自流对象的新数据，确保仅插入不同的或新的行。
+
+4. **使用 Snowflake 流和任务更改数据捕获**
+
+   * 在 Snowflake 中创建任务以自动捕获变更数据。
+   * 当流对象中有新数据可用时，将触发每个任务以将数据加载到最终表中。
+
+5. **Airflow DAG 任务**
+
+   * **任务 1**：检查 Kaggle API 是否可用。
+   * **任务 2**：将转换后的数据上传到 S3 存储桶。
+
+## 迁移方案
+
+### 需要什么
+
+* [云器Lakehouse](https://www.yunqi.tech/)账户
+* 本指南的[Github代码库](https://github.com/yunqiqiliang/clickzetta_quickstart/tree/main/Quickstarts_RealTime_Insurance_claims_Data_ETL_Pipeline)
+* 访问AWS S3的AK信息
+
+### 云器Lakehouse和Snowflake对象概念映射
+
+| [云器Lakehouse概念](Key_Concepts.md) | Snowflake概念         |
+| -------------------------------- | ------------------- |
+| WORSPACE                         | DATABASE            |
+| SCHEMA                           | SCHEMA              |
+| VCLUSTER                         | WAREHOUSE           |
+| STORAGE CONNECTION               | STORAGE INTEGRATION |
+| VOLUME                           | STAGE               |
+| TABLE                            | TABLE               |
+| PIPE                             | SNOWPIPE            |
+| TABLE STREAM                     | STREAM              |
+| STUDIO 任务                        | TASK                |
+| Lakehouse SQL                    | Snowflake SQL       |
+
+### 基于云器Lakehouse的架构
+
+:-: ![](.topwrite/assets/image_1737358887540.png =812)
+
+### 云器Lakehouse方案和Snowflake方案里的工具和服务对比
+
+| 云器Lakehouse方案里工具和服务             | 用途                 | Snowflake方案里的工具和服务    | 用途                 |
+| ------------------------------- | ------------------ | --------------------- | ------------------ |
+| Python                          | 用于脚本编写和数据处理        | Python                | 用于脚本编写和数据处理        |
+| Pandas                          | 用于数据操作和分析          | Pandas                | 用于数据操作和分析          |
+| AWS S3                          | 作为存储原始数据和转换后数据的数据湖 | AWS S3                | 作为存储原始数据和转换后数据的数据湖 |
+| **云器Lakehouse**                 | 用于数据建模和存储          | Snowflake             | 用于数据建模和存储          |
+| **云器Lakehouse Studio IDE**      | 用于协调 ETL 工作流程      | Apache Airflow        | 用于协调 ETL 工作流程      |
+| **云器Lakehouse Studio Python任务** | 用于托管 Airflow 环境    | EC2                   | 用于托管 Airflow 环境    |
+| **云器Lakehouse JDBC Driver**     | 用于连接Tableau        | Tableau  Snowflake连接器 | 用于连接Tableau        |
+| Kaggle API                      | 用于从 Kaggle 提取数据    | Kaggle API            | 用于从 Kaggle 提取数据    |
+| Tableau                         | 用于数据可视化            | Tableau               | 用于数据可视化            |
+
+### 迁移中碰到的语法差异
+
+| 功能        | 云器Lakehouse                                             | Snowflake         |
+| --------- | ------------------------------------------------------- | ----------------- |
+| 代码注释      | —- or —                                                 | /// or //         |
+| stream元数据 | \_\_change\_type字段                                      | METADATA$ACTION   |
+| 创建对象DDL   | 有些对象不支持CREATE OR REPLACE，用CREATE IF NOT EXISTS和ALTER的方式 | CREATE OR REPLACE |
+
+^
+
+## 迁移步骤
+
+### 任务开发
+
+#### 任务树
+
+导航到[Lakehouse Studio](studio_overview.md)的开发->任务，
+
+:-: ![](.topwrite/assets/image_1737359607866.png =766)
+
+单击“+”新建如下目录：
+
+* Quickstarts\_RealTime\_Insurance\_claims\_Data\_ETL\_Pipeline
+
+单击“+”新建如下SQL任务：
+
+* 00\_Setup\_Env
+* 02\_Stages\_or\_Volumes
+* 03\_Tables
+* 04\_Pipes
+* 05\_Streams
+
+以上任务开发好后请点击“运行”完成对象创建。
+
+^
+
+单击“+”新建如下PYTHON任务：
+
+* 01\_Data\_Generate
+* 06\_Tasks(目录)
+  * claims\_cdc\_task
+  * customers\_cdc\_task
+  * policy\_cdc\_task
+  * vehicles\_cdc\_task
+
+以上任务开发好后请点击“运行”先进行测试（后续步骤请参考文档后面的调度和发布指导）。
+
+^
+
+将如下代码复制到对应的任务里，也可以从[GitHub下载文件](https://github.com/yunqiqiliang/clickzetta_quickstart/tree/main/Quickstarts_RealTime_Insurance_claims_Data_ETL_Pipeline)后将内容复制到对应的任务里。
+
+^
+
+#### 任务里的参数设置
+
+^
+
+对于任务（00\_Setup\_Env和01\_Data\_Generate）里是带有参数的，比如：
+
+^
+
+:-: ![](.topwrite/assets/image_1737360457052.png =806)
+
+^
+
+请点击“调度”：
+
+^
+
+:-: ![](.topwrite/assets/image_1737360486722.png =797)
+
+^
+
+然后“加载代码中的参数”并填入实际值：
+
+^
+
+:-: ![](.topwrite/assets/image_1737425374903.png =807)
+
+####
+
+#### 构建云器Lakehouse环境
+
+00\_Setup\_Env:
+
+```SQL
+-- RealTime_Insurance_ETL_VC virtual cluster
+CREATE VCLUSTER IF NOT EXISTS RealTime_Insurance_ETL_VC
+   VCLUSTER_SIZE = XSMALL
+   VCLUSTER_TYPE = GENERAL
+   AUTO_SUSPEND_IN_SECOND = 60
+   AUTO_RESUME = TRUE
+   COMMENT  'RealTime_Insurance_ETL_VC VCLUSTER for test';
+
+-- Use our VCLUSTER for data house
+USE VCLUSTER RealTime_Insurance_ETL_VC;
+
+-- Create and Use SCHEMA
+CREATE SCHEMA IF NOT EXISTS  RealTime_Insurance_ETL_SCH;
+USE SCHEMA RealTime_Insurance_ETL_SCH;
+
+--external Connection for data lake
+--创建数据湖Connection,到数据湖的连接
+CREATE STORAGE CONNECTION if not exists aws_s3_connection_demo
+    TYPE S3
+    ENDPOINT = 's3.us-east-1.amazonaws.com'
+    REGION = 'us-east-1'
+    ACCESS_KEY = ${ACCESS_KEY}
+    SECRET_KEY = ${SECRET_KEY}
+    comments = 'us-east-1 storage connection for etl demo';
+```
+
+#### 将SNOWFLAKE\_ETL.py迁移为云器Lakehouse的内置Python任务Data\_Generate
+
+云器Lakehouse提供了托管的Python运行环境，在云器Lakehouse Studio里开发的Python任务，可以直接运行，也可以通过配置调度信息实现周期性的运行。
+
+01\_Data\_Generate:
+
+```PYTHON
+import subprocess
+import sys
+import warnings
+import contextlib
+import os
+
+# Suppress warnings
+warnings.filterwarnings("ignore", message="A value is trying to be set on a copy of a slice from a DataFrame")
+
+# Suppress stderr
+@contextlib.contextmanager
+def suppress_stderr():
+    with open(os.devnull, 'w') as devnull:
+        old_stderr = sys.stderr
+        sys.stderr = devnull
+        try:
+            yield
+        finally:
+            sys.stderr = old_stderr
+
+with suppress_stderr():
+    # Install kaggle
+    subprocess.run([sys.executable, "-m", "pip", "install", "kaggle", "--target", "/home/system_normal", "-i", "https://pypi.tuna.tsinghua.edu.cn/simple"], stderr=subprocess.DEVNULL)
+    sys.path.append('/home/system_normal')
+
+import pandas as pd
+import boto3
+import random
+import os, json, io
+import zipfile
+from datetime import datetime
+
+def load_random_sample(csv_file, sample_size):
+
+    # Count total rows in the CSV file
+    total_rows = sum(1 for line in open(csv_file, encoding='utf-8')) - 1  # Subtract header row
+
+    # Calculate indices of rows to skip (non-selected)
+    skip_indices = random.sample(range(1, total_rows + 1), total_rows - sample_size)
+
+    # Load DataFrame with random sample of rows
+    df = pd.read_csv(csv_file, skiprows=skip_indices)
+
+    policy_table = df[['policy_id', 'subscription_length', 'region_code', 'segment']].copy()
+    vehicles_table = df[['policy_id', 'vehicle_age', 'fuel_type', 'is_parking_sensors', 'is_parking_camera', 'rear_brakes_type', 'displacement', 'transmission_type', 'steering_type', 'turning_radius', 'gross_weight', 'is_front_fog_lights', 'is_rear_window_wiper', 'is_rear_window_washer', 'is_rear_window_defogger', 'is_brake_assist', 'is_central_locking', 'is_power_steering', 'is_day_night_rear_view_mirror', 'is_speed_alert', 'ncap_rating']].copy()
+    customers_table = df[['policy_id', 'customer_age', 'region_density']].copy()
+    claims_table = df[['policy_id', 'claim_status']].copy()
+
+    vehicles_table.rename(columns={'policy_id': 'vehicle_id'}, inplace=True)
+    customers_table.rename(columns={'policy_id': 'customer_id'}, inplace=True)
+    claims_table.rename(columns={'policy_id': 'claim_id'}, inplace=True)
+
+    return policy_table, vehicles_table, customers_table, claims_table
+
+def upload_df_to_s3():
+    try:
+        with suppress_stderr():
+            # Setup Kaggle API
+
+            # Ensure the directory exists
+            config_dir = '/home/system_normal/tempdata/.config/kaggle'
+            if not os.path.exists(config_dir):
+                os.makedirs(config_dir)
+
+            # Create the kaggle.json file with the given credentials
+            kaggle_json = {
+                "username": ${kaggle_username},
+                "key": ${kaggel_key}
+            }
+            with open(os.path.join(config_dir, 'kaggle.json'), 'w') as f:
+                json.dump(kaggle_json, f)
+
+            # Set the environment variable to the directory containing kaggle.json
+            os.environ['KAGGLE_CONFIG_DIR'] = config_dir
+            from kaggle.api.kaggle_api_extended import KaggleApi
+            # Authenticate the Kaggle API
+            api = KaggleApi()
+            api.authenticate()
+
+            # Define the dataset
+            dataset = 'litvinenko630/insurance-claims'
+
+            # Define the CSV file name
+            csv_file = 'Insurance claims data.csv'
+
+            # Download the entire dataset as a zip file
+            api.dataset_download_files(dataset, path='/home/system_normal/tempdata')
+
+            # Extract the CSV file from the downloaded zip file
+            with zipfile.ZipFile('/home/system_normal/tempdata/insurance-claims.zip', 'r') as zip_ref:
+                zip_ref.extract(csv_file, path='/home/system_normal/tempdata')
+
+            policy_data, vehicles_data, customers_data, claims_data = load_random_sample(f'/home/system_normal/tempdata/{csv_file}', 20)
+            # Convert DataFrame to CSV string
+            policy = policy_data.to_csv(index=False)
+            vehicles = vehicles_data.to_csv(index=False)
+            customers = customers_data.to_csv(index=False)
+            claims = claims_data.to_csv(index=False)
+
+            current_datetime = datetime.now().strftime("%Y%m%d_%H%M%S")
+            
+            # Ensure you have set your AWS credentials in environment variables or replace the following with your credentials
+            s3_client = boto3.client(
+                's3',
+                aws_access_key_id= ${aws_access_key_id},
+                aws_secret_access_key= ${aws_secret_access_key},
+                region_name= ${aws_region_name}
+            )
+
+            # Define S3 bucket and keys with current date and time
+            s3_bucket = 'insurance-data-clickzetta-etl-project'
+            s3_key_policy = f'policy/policy_{current_datetime}.csv'
+            s3_key_vehicles = f'vehicles/vehicles_{current_datetime}.csv'
+            s3_key_customers = f'customers/customers_{current_datetime}.csv'
+            s3_key_claims = f'claims/claims_{current_datetime}.csv'
+
+            # Upload to S3
+            s3_client.put_object(Bucket=s3_bucket, Key=s3_key_policy, Body=policy)
+            s3_client.put_object(Bucket=s3_bucket, Key=s3_key_vehicles, Body=vehicles)
+            s3_client.put_object(Bucket=s3_bucket, Key=s3_key_customers, Body=customers)
+            s3_client.put_object(Bucket=s3_bucket, Key=s3_key_claims, Body=claims)
+            printf("upload_df_to_s3 down:{s3_key_policy},{s3_key_vehicles},{s3_key_customers},{s3_key_claims}")
+    
+    except Exception as e:
+        pass  # Ignore errors
+
+# Run the upload function
+upload_df_to_s3()
+```
+
+#### 创建云器Lakehouse Volumes（对应Snowflake的Stages）
+
+02\_Stages\_or\_Volumes:
+
+^
+
+请将LOCATION 's3://insurance-data-clickzetta-etl-project/policy'里的bucket名称insurance-data-clickzetta-etl-project改为你的bucket名称。
+
+```SQL
+--创建Volume,数据湖存储文件的位置
+CREATE EXTERNAL VOLUME  if not exists policy_data_stage
+  LOCATION 's3://insurance-data-clickzetta-etl-project/policy' 
+  USING connection aws_s3_connection_demo  -- storage Connection
+  DIRECTORY = (
+    enable = TRUE
+  ) 
+  recursive = TRUE;
+
+--同步数据湖Volume的目录到Lakehouse
+ALTER volume policy_data_stage refresh;
+
+--查看云器Lakehouse数据湖Volume上的文件
+SELECT * from directory(volume policy_data_stage);
+
+--********************************************************************--
+--创建Volume,数据湖存储文件的位置
+CREATE EXTERNAL VOLUME  if not exists vehicles_data_stage
+  LOCATION 's3://insurance-data-clickzetta-etl-project/vehicles' 
+  USING connection aws_s3_connection_demo  -- storage Connection
+  DIRECTORY = (
+    enable = TRUE
+  ) 
+  recursive = TRUE;
+
+--同步数据湖Volume的目录到Lakehouse
+ALTER volume vehicles_data_stage refresh;
+
+--查看云器Lakehouse数据湖Volume上的文件
+SELECT * from directory(volume vehicles_data_stage);
+
+--********************************************************************--
+--创建Volume,数据湖存储文件的位置
+CREATE EXTERNAL VOLUME  if not exists customers_data_stage
+  LOCATION 's3://insurance-data-clickzetta-etl-project/customers' 
+  USING connection aws_s3_connection_demo  -- storage Connection
+  DIRECTORY = (
+    enable = TRUE
+  ) 
+  recursive = TRUE;
+
+--同步数据湖Volume的目录到Lakehouse
+ALTER volume customers_data_stage refresh;
+
+--查看云器Lakehouse数据湖Volume上的文件
+SELECT * from directory(volume customers_data_stage);
+
+--********************************************************************--
+--创建Volume,数据湖存储文件的位置
+CREATE EXTERNAL VOLUME  if not exists claims_data_stage
+  LOCATION 's3://insurance-data-clickzetta-etl-project/claims' 
+  USING connection aws_s3_connection_demo  -- storage Connection
+  DIRECTORY = (
+    enable = TRUE
+  ) 
+  recursive = TRUE;
+
+--同步数据湖Volume的目录到Lakehouse
+ALTER volume claims_data_stage refresh;
+
+--查看云器Lakehouse数据湖Volume上的文件
+SELECT * from directory(volume claims_data_stage);
+```
+
+#### 创建云器Lakehouse Tables
+
+03\_Tables:
+
+```SQL
+--- STAGING TABLES ---
+--creating staging tables for each normalized tables created by data pipeline
+CREATE  TABLE IF NOT EXISTS staging_policy(
+    policy_id VARCHAR(15) ,
+    subscription_length FLOAT,
+    region_code VARCHAR(5),
+    segment VARCHAR(10));
+    
+CREATE  TABLE IF NOT EXISTS staging_vehicles (
+    vehicle_id VARCHAR(15) ,
+    vehicle_age FLOAT,
+    fuel_type VARCHAR(10),
+    parking_sensors VARCHAR(5),
+    parking_camera VARCHAR(5),
+    rear_brakes_type VARCHAR(10),
+    displacement INT,
+    trasmission_type VARCHAR(20),
+    steering_type VARCHAR(15),
+    turning_radius FLOAT,
+    gross_weight INT,
+    front_fog_lights VARCHAR(5),
+    rear_window_wiper VARCHAR(5), 
+    rear_window_washer VARCHAR(5),
+    rear_window_defogger VARCHAR(5), 
+    brake_assist VARCHAR(5), 
+    central_locking VARCHAR(5),
+    power_steering VARCHAR(5),      
+    day_night_rear_view_mirror VARCHAR(5),  
+    is_speed_alert VARCHAR(5),
+    ncap_rating INT);
+
+CREATE  TABLE IF NOT EXISTS staging_customers(
+    customer_id VARCHAR(20) ,
+    customer_age INT,
+    region_density INT);
+
+CREATE  TABLE IF NOT EXISTS staging_claims(
+    claim_id VARCHAR(20) ,
+    claim_status INT);
+
+
+
+
+--- FINAL TABLES ---
+--creating final table to store transformed data captured by stream objects 
+CREATE  TABLE IF NOT EXISTS policy(
+    policy_id VARCHAR(15) ,
+    subscription_length FLOAT,
+    region_code VARCHAR(5),
+    segment VARCHAR(10));
+
+SELECT * FROM policy;
+TRUNCATE TABLE policy;
+
+CREATE  TABLE IF NOT EXISTS vehicles (
+    vehicle_id VARCHAR(15) ,
+    vehicle_age FLOAT,
+    fuel_type VARCHAR(10),
+    parking_sensors VARCHAR(5),
+    parking_camera VARCHAR(5),
+    rear_brakes_type VARCHAR(10),
+    displacement INT,
+    trasmission_type VARCHAR(20),
+    steering_type VARCHAR(15),
+    turning_radius FLOAT,
+    gross_weight INT,
+    front_fog_lights VARCHAR(5),
+    rear_window_wiper VARCHAR(5), 
+    rear_window_washer VARCHAR(5),
+    rear_window_defogger VARCHAR(5), 
+    brake_assist VARCHAR(5), 
+    central_locking VARCHAR(5),
+    power_steering VARCHAR(5),      
+    day_night_rear_view_mirror VARCHAR(5),  
+    is_speed_alert VARCHAR(5),
+    ncap_rating INT);
+
+CREATE  TABLE IF NOT EXISTS customers(
+    customer_id VARCHAR(20) ,
+    customer_age INT,
+    region_density INT);
+
+CREATE  TABLE IF NOT EXISTS claims(
+    claim_id VARCHAR(20) ,
+    claim_status INT);
+```
+
+#### 创建云器Lakehouse Pipes
+
+04\_Pipes:
+
+```SQL
+create pipe policy_pipe
+  VIRTUAL_CLUSTER = 'RealTime_Insurance_ETL_VC'
+  --执行获取最新文件使用扫描文件模式
+  INGEST_MODE = 'LIST_PURGE'
+  as
+copy into staging_policy from volume policy_data_stage(
+    policy_id VARCHAR(15) ,
+    subscription_length FLOAT,
+    region_code VARCHAR(5),
+    segment VARCHAR(10))
+using csv OPTIONS(
+  'header'='true'
+)
+--必须添加purge参数导入成功后删除数据 
+purge=true
+;
+--********************************************************************--
+create pipe vehicles_pipe
+  VIRTUAL_CLUSTER = 'RealTime_Insurance_ETL_VC'
+  --执行获取最新文件使用扫描文件模式
+  INGEST_MODE = 'LIST_PURGE'
+  as
+copy into staging_vehicles from volume vehicles_data_stage(
+    vehicle_id VARCHAR(15) ,
+    vehicle_age FLOAT,
+    fuel_type VARCHAR(10),
+    parking_sensors VARCHAR(5),
+    parking_camera VARCHAR(5),
+    rear_brakes_type VARCHAR(10),
+    displacement INT,
+    trasmission_type VARCHAR(20),
+    steering_type VARCHAR(15),
+    turning_radius FLOAT,
+    gross_weight INT,
+    front_fog_lights VARCHAR(5),
+    rear_window_wiper VARCHAR(5), 
+    rear_window_washer VARCHAR(5),
+    rear_window_defogger VARCHAR(5), 
+    brake_assist VARCHAR(5), 
+    central_locking VARCHAR(5),
+    power_steering VARCHAR(5),      
+    day_night_rear_view_mirror VARCHAR(5),  
+    is_speed_alert VARCHAR(5),
+    ncap_rating INT)
+using csv OPTIONS(
+  'header'='true'
+)
+--必须添加purge参数导入成功后删除数据 
+purge=true
+;
+--********************************************************************--
+create pipe customers_pipe
+  VIRTUAL_CLUSTER = 'RealTime_Insurance_ETL_VC'
+  --执行获取最新文件使用扫描文件模式
+  INGEST_MODE = 'LIST_PURGE'
+  as
+copy into staging_customers from volume customers_data_stage(
+    customer_id VARCHAR(20),
+    customer_age INT,
+    region_density INT)
+using csv OPTIONS(
+  'header'='true'
+)
+--必须添加purge参数导入成功后删除数据 
+purge=true
+;
+--********************************************************************--
+create pipe claims_pipe
+  VIRTUAL_CLUSTER = 'RealTime_Insurance_ETL_VC'
+  --执行获取最新文件使用扫描文件模式
+  INGEST_MODE = 'LIST_PURGE'
+  as
+copy into staging_claims from volume claims_data_stage(
+    claim_id VARCHAR(20) ,
+    claim_status INT)
+using csv OPTIONS(
+  'header'='true'
+)
+--必须添加purge参数导入成功后删除数据 
+purge=true
+;
+```
+
+#### 创建云器Lakehouse Table Streams（对应Snowflake的Streams）
+
+05\_Streams:
+
+```SQL
+--********************************************************************--
+--- CREATING TABLE STREAM OBJECTS FOR EACH STAGING TABLES TO CAPTURE NEW DATA
+--  creating TABLE STREAM objects for staging tables
+CREATE TABLE STREAM IF NOT EXISTS STREAM_policy ON TABLE staging_policy
+WITH
+  PROPERTIES ('TABLE_STREAM_MODE' = 'APPEND_ONLY');
+--********************************************************************--
+CREATE TABLE STREAM IF NOT EXISTS STREAM_vehicles ON TABLE staging_vehicles
+WITH
+  PROPERTIES ('TABLE_STREAM_MODE' = 'APPEND_ONLY');
+--********************************************************************--
+CREATE TABLE STREAM IF NOT EXISTS STREAM_customers ON TABLE staging_customers
+WITH
+  PROPERTIES ('TABLE_STREAM_MODE' = 'APPEND_ONLY');
+--********************************************************************--
+CREATE TABLE STREAM IF NOT EXISTS STREAM_claims ON TABLE staging_claims
+WITH
+  PROPERTIES ('TABLE_STREAM_MODE' = 'APPEND_ONLY');
+--********************************************************************--
+-- check total streams
+SHOW TABLE STREAMS;
+```
+
+#### 检查新创建的对象
+
+通过以下命令检查通过SQL命令创建对象是否已成功，以及PIPES的状态。
+
+```SQL
+--Lake Objects
+SHOW CONNECTIONS;
+SHOW VOLUMES;
+
+--House Objects
+SHOW TABLES;
+SHOW PIPES;
+SHOW TABLE STREAMS;
+```
+
+####
+
+```SQL
+SHOW CONNECTIONS;
+```
+
+:-: ![](.topwrite/assets/image_1737425940720.png =803)
+
+```SQL
+SHOW VOLUMES;
+```
+
+:-: ![](.topwrite/assets/image_1737425972728.png =794)
+
+```SQL
+SHOW TABLES;
+```
+
+:-: ![](.topwrite/assets/image_1737425995818.png =788)
+
+```SQL
+SHOW PIPES;
+```
+
+:-: ![](.topwrite/assets/image_1737425892542.png =776)
+
+```SQL
+SHOW TABLE STREAMS;
+```
+
+:-: ![](.topwrite/assets/image_1737426021254.png =766)
+
+#### 创建Transform任务
+
+06\_Tasks\:claims\_cdc\_task
+
+```SQL
+MERGE INTO claims AS a
+    USING stream_claims AS b
+    ON a.claim_id = b.claim_id
+    WHEN NOT MATCHED AND b.__change_type = 'INSERT' THEN
+        INSERT (claim_id, claim_status)
+        VALUES (b.claim_id, b.claim_status);
+```
+
+###
+
+06\_Tasks\:customers\_cdc\_task
+
+```SQL
+MERGE INTO customers AS a
+    USING stream_customers AS b
+    ON a.customer_id = b.customer_id
+    WHEN NOT MATCHED AND b.__change_type = 'INSERT' THEN
+        INSERT (customer_id, customer_age, region_density)
+        VALUES (b.customer_id, b.customer_age, b.region_density);
+```
+
+###
+
+06\_Tasks\:policy\_cdc\_task
+
+```SQL
+MERGE INTO policy AS a
+    USING stream_policy AS b
+    ON a.policy_id = b.policy_id
+    WHEN NOT MATCHED AND b.__change_type = 'INSERT' THEN
+        INSERT (policy_id, subscription_length, region_code, segment)
+        VALUES (b.policy_id, b.subscription_length, b.region_code, b.segment);
+```
+
+###
+
+06\_Tasks\:vehicles\_cdc\_task
+
+```SQL
+MERGE INTO vehicles AS a
+    USING stream_vehicles AS b
+    ON a.vehicle_id = b.vehicle_id
+    WHEN NOT MATCHED AND b.__change_type = 'INSERT' THEN
+        INSERT (vehicle_id, vehicle_age, fuel_type, parking_sensors, parking_camera, rear_brakes_type, displacement, trasmission_type, steering_type, turning_radius, gross_weight, front_fog_lights, rear_window_wiper, rear_window_washer, rear_window_defogger, brake_assist, central_locking, power_steering, day_night_rear_view_mirror, is_speed_alert, ncap_rating)
+        VALUES (b.vehicle_id, b.vehicle_age,b.fuel_type, b.parking_sensors, b.parking_camera, b.rear_brakes_type, b.displacement, b.trasmission_type, b.steering_type, b.turning_radius, b.gross_weight, b.front_fog_lights, b.rear_window_wiper, b.rear_window_washer, b.rear_window_defogger, b.brake_assist, b.central_locking, b.power_steering, b.day_night_rear_view_mirror, b.is_speed_alert, b.ncap_rating);
+```
+
+###
+
+### 生产运行
+
+### 作业调度
+
+导航到[Lakehouse Studio](studio_overview.md)的开发->任务，
+
+配置如下任务的调度参数并提交到生成环境运行。
+
+^
+
+需要配置调度和提交上线的任务如下：
+
+* 01\_Data\_Generate
+* claims\_cdc\_task
+* customers\_cdc\_task
+* policy\_cdc\_task
+* vehicles\_cdc\_task
+
+^
+
+:-: ![](.topwrite/assets/image_1737360326171.png =815)
+
+^
+
+调度参数配置如下：
+
+^
+
+:-: ![](.topwrite/assets/image_1737360361129.png =817)
+
+^
+
+Streams和Pipes创建后会自动运行，无需调度。
+
+^
+
+### 通过Tableau分析云器Lakehouse里的数据
+
+参考[这个文章](TableauConnectToLakehouse.md)将Tableau通过JDBC连接到云器Lakehouse，对Lakehouse里的数据进行数据探查、分析，并创建BI报表。
+
+#### 周期任务运维
+
+导航到[Lakehouse Studio](studio_overview.md)的运维监控->任务运维->周期任务，查看各个周期调度任务的运行状态：
+
+^
+
+:-: ![](.topwrite/assets/image_1737365542778.png =793)
+
+#### 周期任务的任务实例管理
+
+:-: ![](.topwrite/assets/image_1737365603958.png =818)
+
+#### Pipes运维
+
+```SQL
+-- 暂停和启动PIPE
+-- 暂停
+ALTER pipe policy_pipe SET PIPE_EXECUTION_PAUSED = true;
+-- 启动
+ALTER pipe policy_pipe SET PIPE_EXECUTION_PAUSED = false;
+
+-- 查看pipe copy作业执行情况
+-- 七天之内的，延迟半小时
+SELECT * FROM INFORMATION_SCHEMA.JOB_HISTORY WHERE QUERY_TAG="pipe.qiliang_ws.realtime_insurance_etl_sch.policy_pipe";
+-- 实时的
+SHOW JOBS IN VCLUSTER SCD_VC WHERE QUERY_TAG="pipe.qiliang_ws.realtime_insurance_etl_sch.policy_pipe";
+
+SHOW JOBS where length(QUERY_TAG)>10;
+
+-- 查看copy作业导入的历史文件
+select * from load_history('RealTime_Insurance_ETL_SCH.staging_policy');
+select * from load_history('RealTime_Insurance_ETL_SCH.staging_vehicles');
+select * from load_history('RealTime_Insurance_ETL_SCH.staging_customers');
+select * from load_history('RealTime_Insurance_ETL_SCH.staging_claims');
+```
+
+#### 查看staging table、table stream和final table表里的数据
+
+```SQL
+SELECT '01staging' AS table_type, 'staging_policy' AS table_name, COUNT(*) AS row_count FROM staging_policy
+UNION ALL
+SELECT '01staging' AS table_type, 'staging_vehicles' AS table_name, COUNT(*) AS row_count FROM staging_vehicles
+UNION ALL
+SELECT '01staging' AS table_type, 'staging_customers' AS table_name, COUNT(*) AS row_count FROM staging_customers
+UNION ALL
+SELECT '01staging' AS table_type, 'staging_claims' AS table_name, COUNT(*) AS row_count FROM staging_claims
+UNION ALL
+SELECT '02stream' AS table_type, 'STREAM_policy' AS table_name, COUNT(*) AS row_count FROM STREAM_policy
+UNION ALL
+SELECT '02stream' AS table_type, 'STREAM_vehicles' AS table_name, COUNT(*) AS row_count FROM STREAM_vehicles
+UNION ALL
+SELECT '02stream' AS table_type, 'STREAM_customers' AS table_name, COUNT(*) AS row_count FROM STREAM_customers
+UNION ALL
+SELECT '02stream' AS table_type, 'STREAM_claims' AS table_name, COUNT(*) AS row_count FROM STREAM_claims
+UNION ALL
+SELECT '03final' AS table_type, 'policy' AS table_name, COUNT(*) AS row_count FROM policy
+UNION ALL
+SELECT '03final' AS table_type, 'vehicles' AS table_name, COUNT(*) AS row_count FROM vehicles
+UNION ALL
+SELECT '03final' AS table_type, 'customers' AS table_name, COUNT(*) AS row_count FROM customers
+UNION ALL
+SELECT '03final' AS table_type, 'claims' AS table_name, COUNT(*) AS row_count FROM claims
+ORDER BY table_type;
+```
+
+:-: ![](.topwrite/assets/image_1737429058137.png =810)
+
+## 资料
+
+[云器Lakehouse基本概念](Key_Concepts.md)
+
+[云器Lakehouse JDBC Driver](JDBC-Driver.md)
+
+[云器Lakehouse Python任务](Python_Task.md)
+
+[云器云器Lakehouse调度与运维](taskdevelop.md)
+
+^
+^
+^
+^
+^
+^
+^