@clickzetta/cz-cli-darwin-arm64 0.5.16 → 0.5.18

package/bin/skills/lakehouse-doc-en/references/data_sharing_between_accounts_guide.md

@@ -30,7 +30,6 @@ The core working principle of Data Share is:
 
 **Data Sharing Flow**:
 
-:-: ![](.topwrite/assets/image_1747735731291.png =322)
 
 The data provider retains full control and can update or remove shared data at any time. When source data is updated, the consumer immediately obtains the latest data without additional data sync operations.
 
@@ -69,7 +68,6 @@ ALTER SHARE taxi_data_share ADD INSTANCE target_instance_name;
 5. Under **Receiving Instances**, click **Add** and enter the consumer's service instance name.
 6. Click **Confirm** to complete creation.
 
-:-: ![](.topwrite/assets/image_1747742254855.png =699)
 
 #### 4.1.3 Creating a View for Partial Data Sharing
 
@@ -140,7 +138,6 @@ DESC SHARE source_instance_name.taxi_data_share;
 2. In the left navigation, select **Data Management** -> **Data Share**.
 3. Switch to the **Shared with Me** tab to see all received shares.
 
-:-: ![](.topwrite/assets/image_1747742432405.png =757)
 
 ### 5.2 Creating a Local Schema Linked to Shared Data
 
@@ -165,7 +162,6 @@ Where:
 3. In the popup, select the **Source Schema** and specify the schema name to be created locally.
 4. Click **Confirm** to complete the extraction.
 
-:-: ![](.topwrite/assets/image_1747742473685.png =373)
 
 ### 5.3 Using Shared Data
 

package/bin/skills/lakehouse-doc-en/references/data_visualization.md

@@ -21,8 +21,7 @@ After clicking "Chart" to visualize the worksheet results, you need to select th
 
 Hover over the chart to view detailed information for each data point. For example, you can view results as a line chart:
 
-![](.topwrite/assets/image_1742200992399.png =757)
-
+^
 ^
 
 In the "Settings" panel on the right side of the visualization area, configure what to display:
@@ -35,7 +34,7 @@ In the "Settings" panel on the right side of the visualization area, configure w
 
 3. Y-axis field:
 
-![](.topwrite/assets/image_1742201014009.png =347)
+^
 
 The Y-axis supports aggregate functions to derive a single value from multiple data points. The available aggregation methods are:
 
@@ -59,8 +58,7 @@ When there are many X-axis values and you need to view a specific data point on
 
 For example, in the chart below, the accurate value is shown only when hovering over the visualization and the specific timestamp information appears.
 
-![](.topwrite/assets/image_1742201080540.png =387)
-
+^
 ^
 
 ## Use Cases
@@ -75,16 +73,7 @@ For example, in the chart below, the accurate value is shown only when hovering
 select order_date, count(*) as c from big_data_table group by order_date;
 ```
 
-![](.topwrite/assets/20250317-190632.jpeg =617)
-
 ^
-
-![](.topwrite/assets/20250317-190714.jpeg =607)
-
-^
-
-![](.topwrite/assets/20250317-190759.jpeg =608)
-
 ^
 
 **Scenario 2**: To **ignore** time span differences and keep only the specific result data points, cast the result to a string (`order_date::string`):
@@ -93,4 +82,4 @@ select order_date, count(*) as c from big_data_table group by order_date;
 select order_date::string, count(*) as c from big_data_table group by order_date;
 ```
 
-![](.topwrite/assets/20250317-190854.jpeg =654)
+^

package/bin/skills/lakehouse-doc-en/references/dataagent.md

@@ -1,9 +1,11 @@
-## What is Data Agent
+## What is Data Engineering Agent
 
-Data Agent is an AI-powered agent built on top of Singdata Lakehouse and Studio. It covers the full lifecycle of "development, operations, and governance" and implements intelligent platform upgrades through an Agentic AIOps philosophy — transforming data development from "people operating the platform" to "people directing the agent."
+Data Engineering Agent is an AI-powered agent built on top of Singdata Lakehouse and Studio. It covers the full lifecycle of "development, operations, and governance" and implements intelligent platform upgrades through an Agentic AIOps philosophy — transforming data development from "people operating the platform" to "people directing the agent."
 
 Data Agent is not just a tool that makes data teams more productive. It is a **data intelligence collaboration system** that enables everyone in the company to work with data.
 
+^
+
 ## User Value
 
 * **Higher productivity: reclaim 80% of your time for what truly matters**
@@ -42,18 +44,18 @@ For example:
 **High cost of understanding standards** Each business domain has its own layering rules, naming conventions, and field standards, scattered across various documents. Engineers must "catch up" before taking on any new requirement, and even minor oversights get flagged in reviews, keeping rework costs high.
 
 > Example prompt:
-> I need to design a Medallion architecture data warehouse based on this metric requirements spec to support GMV analysis. I've already planned the tables for each layer: [Bronze layer] xxx [Silver layer] xxx [Gold layer] xxx. Based on this table list, please generate a data warehouse modeling standards document.
+> I need to design a Medallion architecture data warehouse based on this metric requirements spec to support GMV analysis. I've already planned the tables for each layer: \[Bronze layer] xxx \[Silver layer] xxx \[Gold layer] xxx. Based on this table list, please generate a data warehouse modeling standards document.
 
-![](/.topwrite/assets/image_1779715425568.png)
+^
 
 ### Scenario 2: Ad-hoc Data Retrieval
 
 **Everything waits in the queue** Exploratory analysis, market research, and other ad-hoc requests are naturally lower priority and get perpetually pushed aside by formal requests. By the time the data finally arrives, the decision window has often closed and the business has already fallen behind the market. The core problem: ad-hoc analysis has no self-service path — it must go through the data team, which simply doesn't have the bandwidth to continuously handle low-priority requests.
 
 > Example prompt:
-> Query brazilianecommerce.olist_orders and count orders by day.
+> Query brazilianecommerce.olist\_orders and count orders by day.
 
-![](/.topwrite/assets/image_1779715858474.png)
+^
 
 ### Scenario 3: Day-to-day Operations
 
@@ -77,4 +79,34 @@ Daily task operations are the most critical routine work on an enterprise data p
 > Please help me analyze which instances failed in the past week.
 > For the task with instance ID xxx, what was the failure reason and which downstream tasks were affected?
 
-![](/.topwrite/assets/image_1779715663695.png)
+^
+
+### Scenario 4: Studio Task Development and Management
+
+Studio tasks are the core scheduling unit of the Lakehouse data pipeline, but traditional management is cumbersome: creating a task requires entering the IDE and configuring step by step, modifying a schedule requires locating the task and opening the schedule panel, and dependency configuration requires manually maintaining task IDs.
+
+Data Agent operates the full lifecycle of Studio tasks directly through natural language:
+
+* **Create tasks**: describe the task logic, and the agent automatically generates SQL or Python code, creates the task, and configures the Virtual Cluster
+* **Schedule configuration**: tell the agent "run at 2 AM every day" and it automatically converts this to a cron expression and applies it
+* **Dependency orchestration**: describe the dependencies between tasks, and the agent automatically configures upstream and downstream dependency chains, avoiding manual task ID lookups
+* **Batch operations**: publish multiple tasks or bulk-update the retry strategy for a category of tasks in a single instruction
+
+> Example prompt:
+> Create a Python task that runs at 3 AM every day, triggers after the ods\_order\_load task completes, and aggregates yesterday's order data into dws\_order\_daily.
+
+### Scenario 5: Data Source Management
+
+Onboarding enterprise data is the first mile of data engineering, involving connection configuration, sync strategy, status monitoring, and more — manual operations are error-prone and hard to track.
+
+Data Agent supports the following data source management operations:
+
+* **Quick onboarding**: describe the database type and connection details, and the agent automatically creates the data source and tests connectivity
+* **Sync configuration**: specify the source and target tables, and the agent selects a full or incremental (CDC) sync strategy based on business needs
+* **Status queries**: ask in a single sentence to get sync delays, recent failure records, and data volume statistics for all data sources
+* **Troubleshooting**: when a sync task fails, the agent automatically pulls error logs, analyzes the root cause, and recommends fix steps
+
+> Example prompt:
+> Help me check which data sources currently have sync delays exceeding 30 minutes, and which ones had sync failures in the last 24 hours.
+
+^

package/bin/skills/lakehouse-doc-en/references/databricks-delta-to-lakehouse-migration.md

@@ -0,0 +1,168 @@
+# Databricks Delta Tables → Lakehouse Migration Guide
+
+Data on Databricks can almost entirely be migrated to Singdata Lakehouse. There are two complementary paths: **federated direct read** (data stays in place, query Databricks tables directly from Lakehouse) and **Studio built-in sync** (move various table types into Lakehouse native format). All 7 table types have been tested and pass, with row counts and field values fully consistent.
+
+Full code on GitHub: [databricks2lakehouse-delta](https://github.com/clickzetta/databricks2lakehouse-delta)
+
+---
+
+## Conclusion First
+
+**Two paths** — selection rule: if federated read is available (External Delta), use federation + CTAS; for everything else, use Studio built-in sync.
+
+| Table Type | Federated Read | Studio Sync | Recommended Path |
+|---|:---:|:---:|---|
+| **External Delta** | ✅ Tested | ✅ Tested | Federated read (no data movement) or CTAS landing |
+| **Managed Delta** | ❌ | ✅ Tested | Studio built-in sync |
+| **Parquet External** | ❌ | ✅ Tested | Studio built-in sync |
+| **CSV External** | ❌ | ✅ Tested | Studio built-in sync |
+| **JSON External** | ❌ | ✅ Tested | Studio built-in sync |
+| **Managed Iceberg** | ❌ | ✅ Tested | Studio built-in sync |
+| **View** | ❌ | ✅ Tested (materialized as table) | Studio built-in sync |
+
+---
+
+## Technical Background
+
+Databricks uses Unity Catalog three-level naming (`catalog.schema.table`), with data stored in two categories:
+- **External tables**: Data files reside in the customer's own S3/ADLS; Databricks only manages metadata
+- **Managed tables**: Data files reside in Databricks managed storage, inaccessible externally
+
+This difference determines the scope of federated reads — only External Delta tables can be directly queried via Lakehouse External Catalog federation. Managed tables must be migrated via Studio sync jobs.
+
+---
+
+![](.topwrite/assets/anim-33-databricks-delta-migration.svg)
+
+---
+
+## Path A: Federated Direct Read (External Delta Only)
+
+Federated read queries Databricks tables directly via Lakehouse External Catalog with **no data movement**, suitable for parallel access during transition or PoC validation.
+
+### Configuration (one-time)
+
+```sql
+-- Step 1: Create a Catalog Connection to Databricks
+CREATE CATALOG CONNECTION IF NOT EXISTS databricks_conn
+TYPE = DATABRICKS_UNITY_CATALOG
+PROPERTIES (
+    'host' = 'https://dbc-xxxx.cloud.databricks.com',
+    'catalog' = 'workspace'
+    -- Authentication parameters configured via Studio UI
+);
+
+-- Step 2: Create External Catalog (federation entry point)
+CREATE EXTERNAL CATALOG IF NOT EXISTS databricks_new_catalog
+USING CONNECTION databricks_conn;
+```
+
+### Federated Query
+
+```sql
+-- Query Databricks tables directly (cross-region/cross-cloud supported, via public internet)
+SELECT * FROM databricks_new_catalog.table_types_demo.customers_external;
+SELECT COUNT(*) FROM databricks_new_catalog.table_types_demo.orders_external;
+
+-- CTAS: land the federated table as a Lakehouse native table (optional)
+CREATE TABLE delta_migration.customers AS
+SELECT * FROM databricks_new_catalog.table_types_demo.customers_external;
+```
+
+### Federated Read Limitations
+
+Federated read **only supports External Delta format**. Test results:
+
+| Table Tested | Format | Result |
+|---|---|---|
+| `customers_external` | External Delta | ✅ 7 rows, schema/types correct |
+| `orders_external` | External Delta | ✅ 8 rows, aggregation correct |
+| `inventory_delta` | External Delta | ✅ 4 rows |
+| `customers_managed` | Managed Delta | ❌ Data in Databricks managed storage, no external access |
+| `products_parquet` | External Parquet | ❌ `unsupported databricks table format [PARQUET]` |
+| `suppliers_csv` | External CSV | ❌ `unsupported databricks table format [CSV]` |
+| `shipments_iceberg` | Managed Iceberg | ❌ S3 400 (data in Databricks managed bucket) |
+| `customer_orders_view` | View | ❌ Not supported |
+
+---
+
+## Path B: Studio Built-in Sync (Handles All Table Types)
+
+Studio has a built-in Databricks data source with visual sync task configuration to move all table types into Lakehouse native format.
+
+### Configuration Steps
+
+1. Studio UI → Data Integration → New Data Source → Select Databricks
+2. Enter Databricks Workspace URL + authentication info (Service Principal)
+3. New sync task → Select source tables → Configure target schema → Execute
+
+### Test Results (All 7 Table Types SUCCEED)
+
+Sync tasks were run against all table types in the `table_types_demo` schema in a real Databricks environment (AWS us-east-1, Unity Catalog). All succeeded:
+
+| Source Table | Type | Result | Duration | Source Rows | Target Rows | Consistent |
+|---|---|:---:|:---:|:---:|:---:|:---:|
+| `customers_managed` | Managed Delta | ✅ | ~88s | 7 | 7 | ✅ |
+| `customers_external` | External Delta | ✅ | ~84s | 7 | 7 | ✅ |
+| `products_parquet` | Parquet | ✅ | ~87s | 5 | 5 | ✅ |
+| `suppliers_csv` | CSV | ✅ | ~93s | 3 | 3 | ✅ |
+| `product_reviews_json` | JSON | ✅ | ~109s | 3 | 3 | ✅ |
+| `shipments_iceberg` | Managed Iceberg | ✅ | ~92s | 5 | 5 | ✅ |
+| `customer_orders_view` | View | ✅ | ~86s | 8 | 8 | ✅ |
+
+> Single task duration is 84–109 seconds (including sync cluster cold start), largely independent of data volume. Estimate total migration time linearly based on table count.
+
+### Data Consistency Verification
+
+**Field-level row-by-row comparison** was performed on 5 tables (Databricks SDK reads source, cz-cli reads target). Findings:
+
+- `products_parquet`, `product_reviews_json`, `shipments_iceberg`, `suppliers_csv`: **All fields fully consistent**
+- `customers_external`, `suppliers_csv`: email/contact_email fields show masked values on the Lakehouse side (e.g., `a***@example.com`)
+
+> Masking is the Lakehouse workspace's **read-time column masking policy** automatically matching by column name. The data itself is migrated intact. To see plaintext values, use a role with appropriate permissions or query in a schema without a masking policy bound.
+
+---
+
+## Delta-Specific Feature Handling
+
+| Feature | Migration Impact |
+|---|---|
+| **Deletion Vectors (DV)** | ✅ Federated read correctly recognizes DVs; deleted rows are not returned. Data state is consistent after sync to Lakehouse |
+| **Change Data Feed (CDF)** | Current data syncs normally. CDF incremental consumption interface (`table_changes()`) has no equivalent in Lakehouse → rebuild incremental pipelines using Lakehouse Table Stream |
+| **Liquid Clustering** | Does not affect data content. After migration to Lakehouse, rebuild layout optimization using Lakehouse `CLUSTER BY` or indexing mechanisms |
+
+---
+
+## Connectivity Notes
+
+- **Cross-region/cross-cloud supported**: Both federated read and Studio sync use the public internet. Tested: Lakehouse AWS Singapore ↔ Databricks us-east-1 cross-region, cross-cloud connectivity confirmed
+- **Only hard limitation**: `COPY INTO`/`Pipe` object storage connections cannot cross cloud providers (e.g., an Alibaba Cloud instance cannot connect to AWS S3), but this affects direct S3 file reads, not reading tables via Databricks External Catalog
+
+---
+
+## Notes
+
+- **Managed tables cannot be federated**: Managed Delta/Iceberg data resides in Databricks managed storage with no external access — must use Studio sync
+- **Parquet/CSV/JSON External tables cannot be federated**: External Catalog currently only supports Delta format; use Studio sync for other formats
+- **Array columns not yet supported for sync**: Columns with `ARRAY<...>` types are not yet supported by Studio sync tasks. Convert them to STRING using `to_json()` on the source side before syncing
+- **Configure sync tasks table by table**: Studio's built-in Databricks data source does not yet support bulk schema-wide sync; configure each table individually. Scripts can be used to batch-generate configurations
+
+## Related Documentation
+
+### Federated Query
+
+- [External Catalog Overview](external-catalog-concept.md): External Catalog federated query principles
+- [Federated Query Guide](SQL_External_Catalog_Guide.md): SQL syntax and usage examples
+
+### Data Ingestion
+
+- [Data Ingestion Overview](streaming_data_pipeline_overview.md): Full ingestion solution landscape
+- [COPY INTO](copy-into.md): Bulk load from object storage
+- [Pipe (Continuous Ingestion)](pipe-overview.md): Continuously monitor object storage for new files
+
+### Other Migration Guides
+
+- [Databricks Notebook → Lakehouse Migration Guide](databricks-notebook-to-studio-migration.md)
+- [Databricks DLT → Lakehouse Migration Guide](databricks-dlt-to-lakehouse-migration.md)
+- [Databricks Jobs → Lakehouse Studio Migration Guide](databricks-jobs-to-studio-migration.md)
+- [Databricks Unity Catalog → Lakehouse Migration Guide](databricks-uc-governance-to-lakehouse-migration.md)

package/bin/skills/lakehouse-doc-en/references/databricks-dlt-to-lakehouse-migration.md

@@ -0,0 +1,331 @@
+# Databricks DLT → Lakehouse Migration Guide: Apparel Retail Streaming Pipeline
+
+If your data pipeline runs on Databricks Delta Live Tables, the core migration effort to Singdata Lakehouse is lower than you might expect — **most PySpark DataFrame code can be reused directly**. DLT uses decorators (`@dlt.table`, `@dlt.expect_or_drop`) to wrap Python functions into pipeline nodes. ZettaPark removes the decorators and replaces them with `df.write.saveAsTable()` — business logic is unchanged, line for line.
+
+This article validates this with a real project: a Databricks DLT-based apparel retail streaming pipeline (Bronze ingestion → Silver SCD2 cleansing → Gold aggregation analytics) fully migrated to Singdata Lakehouse, passing all 16 automated validations.
+
+Full code on GitHub: [databricks2lakehouse-dlt-apparel](https://github.com/clickzetta/databricks2lakehouse-dlt-apparel)
+
+---
+
+## Source Project
+
+[databricks2lakehouse-dlt-apparel](https://github.com/clickzetta/databricks2lakehouse-dlt-apparel) is forked from [jrlasak/databricks_apparel_streaming](https://github.com/jrlasak/databricks_apparel_streaming) (⭐45). The original tech stack is Databricks DLT + Auto Loader + Delta Lake. The project implements a complete data pipeline for an apparel retailer across 4 dimensions (customers, products, stores, transactions), covering streaming ingestion, SCD Type 2 history tracking, data quality constraints, and Gold layer aggregation analytics.
+
+Migrated code is in the `03_lakehouse/` directory, with three available approaches that can be compared file-by-file with `01_source/dlt/`.
+
+## Conclusion First
+
+**Option C (Dynamic Table) is the closest equivalent for DLT pipelines** — declarative definition + automatic scheduled refresh, consistent with DLT's "define and execute" philosophy. If the existing team is comfortable with PySpark DataFrame, Option A (ZettaPark) requires only 5 mechanical substitutions with all business logic fully preserved.
+
+| Option | Files | DLT Equivalent | Characteristics |
+|------|------|---------|------|
+| **A. ZettaPark Python** | `03_lakehouse/*.py` | `@dlt.table` → `df.write.saveAsTable()` | Minimal code changes, retains PySpark skills |
+| **B. Pure SQL** | `03_lakehouse/sql/` | SQL equivalent implementation | SQL-first teams |
+| **C. Dynamic Table (GIC)** | `03_lakehouse/dynamic_tables/` | **Native equivalent of @dlt.table** | Declarative + auto-refresh, closest to DLT semantics |
+
+**Option A change list** (ZettaPark main path):
+
+| Change | Effort | Notes |
+|--------|--------|------|
+| `import dlt` / `from pyspark...` | Very low | `from clickzetta.zettapark import functions as F`, replace package name |
+| `spark` global | Very low | `session = Session.builder.configs({}).create()` |
+| `@dlt.table(name=X)` | Very low | Add `df.write.mode("overwrite").saveAsTable(X)` as last line of function |
+| `@dlt.expect_or_drop("msg","cond")` | Very low | `df.filter(F.col(...)...)`, semantically equivalent |
+| `dlt.read_stream("LIVE.X")` / `dlt.read("LIVE.X")` | Very low | `session.table("X")`, same as PySpark |
+| `dlt.create_auto_cdc_flow(scd_type=2)` | Low | `F.lead().over(Window.partitionBy(key).orderBy(seq))`, ZettaPark Window is identical to PySpark |
+| `F.window("event_time","1 day")` | Very low | `F.to_date(F.col("event_time"))`, ZettaPark has no F.window |
+| Auto Loader → | Low | `session.read.csv("vol://...")` batch; use Pipe for streaming in production |
+
+JOIN logic, aggregation functions (`F.sum/count/avg`), `F.when/coalesce/lead/lag`, `Window` — all require no changes, fully consistent with PySpark API.
+
+---
+
+## Tech Stack Comparison
+
+| | Databricks DLT | ZettaPark (after migration) |
+|---|---|---|
+| Pipeline definition | Python decorator `@dlt.table` | `df.write.mode("overwrite").saveAsTable(X)` |
+| Data quality constraints | `@dlt.expect_or_drop("msg","cond")` | `df.filter(condition)` (same semantics) |
+| Streaming read | `dlt.read_stream("LIVE.X")` | `session.table("X")` (DT auto-incremental) |
+| Static read | `dlt.read("LIVE.X")` | `session.table("X")` |
+| SCD Type 2 | `dlt.create_auto_cdc_flow(stored_as_scd_type=2)` | `F.lead().over(Window.partitionBy(key).orderBy(seq))` |
+| Time window aggregation | `F.window("event_time","1 day")` | `F.to_date(F.col("event_time"))` |
+| File ingestion | Auto Loader (`cloudFiles`) | `session.read.csv("vol://...")` / Pipe (streaming) |
+| DataFrame API | PySpark | **Fully consistent** (F.col/when/lead/Window etc.) |
+
+---
+
+![](.topwrite/assets/anim-30-databricks-dlt-migration.svg)
+
+---
+
+## Project Background
+
+Apparel retailer with 4 data streams + Bronze/Silver/Gold three-layer architecture:
+
+| Data Domain | Raw Events | Row Count | Notes |
+|--------|---------|------|------|
+| Customers | `raw_customers` | 150 | Includes SCD2 update records (30 historical) |
+| Products | `raw_products` | 50 | Includes category/brand/size/color |
+| Stores | `raw_stores` | 5 | 5 stores |
+| Transactions | `raw_sales` | 500 | Includes discount, tax, payment method |
+
+DLT pipeline file structure:
+
+```
+01_bronze.py   — Auto Loader streaming ingestion → 4 Bronze tables
+02A_silver.py  — @dlt.view + @dlt.expect_or_drop (data quality filtering)
+02B_silver.py  — dlt.create_auto_cdc_flow (SCD Type 2 dimension tables)
+02C_silver.py  — Sales transaction cleansing
+03_gold.py     — 4 Gold aggregation tables
+```
+
+---
+
+## Migration Steps
+
+### Step 1: `@dlt.table` / Auto Loader → ZettaPark
+
+DLT uses decorators to declare tables + `spark.readStream` for streaming ingestion. ZettaPark uses `session.read.csv("vol://...")` to load files, with `df.write.saveAsTable()` replacing decorators:
+
+```python
+# Databricks DLT (01_source/dlt/01_bronze.py)
+@dlt.table(name="bronze_sales")
+def bronze_sales():
+    return spark.readStream.format("cloudFiles").option("cloudFiles.format","csv").load(VOL_PATH)
+```
+
+```python
+# ZettaPark (03_lakehouse/01_bronze.py) — minimal rewrite
+from clickzetta.zettapark import Session               # ← import pyspark → clickzetta.zettapark
+session = Session.builder.configs({...}).create()      # ← spark global injection → explicit creation
+
+df = session.read.option("header","true").csv("vol://apparel_bronze.raw_data/sales.csv")
+#   readStream.format("cloudFiles")... → session.read.csv("vol://...")
+df.write.mode("overwrite").saveAsTable("apparel_bronze.raw_sales")
+#   @dlt.table(name=X) → df.write.saveAsTable(X) (last line of function body)
+```
+
+> 💡 **Note**: Auto Loader continuously monitors new files (streaming). The Lakehouse equivalent is **Pipe** (automatically ingests after configuration). Use Pipe instead of COPY INTO in production environments.
+
+### Step 2: `@dlt.expect_or_drop` → `df.filter()`
+
+DLT data quality constraints translate directly to ZettaPark `.filter()` — semantically equivalent, API consistent:
+
+```python
+# Databricks DLT (02A_silver.py)
+@dlt.view(name="customers_cleaned_stream")
+@dlt.expect_or_drop("valid_customer_id", "customer_id IS NOT NULL")
+@dlt.expect("realistic_age", "age >= 18 AND age <= 100")
+def customers_cleaned_stream():
+    return dlt.read_stream(f"LIVE.{BRONZE_CUSTOMERS}")
+```
+
+```python
+# ZettaPark (03_lakehouse/02B_silver.py)
+customers = session.table("apparel_bronze.raw_customers")  # dlt.read_stream → session.table()
+customers = (customers
+    .filter(F.col("customer_id").isNotNull())          # @dlt.expect_or_drop("valid_customer_id",...)
+    .filter((F.col("age") >= 18) & (F.col("age") <= 100))  # @dlt.expect("realistic_age",...)
+)
+```
+
+### Step 3: SCD Type 2 — `dlt.create_auto_cdc_flow` → `LEAD()` Window Function
+
+`create_auto_cdc_flow(stored_as_scd_type=2)` is fundamentally a LEAD window function. ZettaPark implements the equivalent logic directly — `F.lead()` and `Window` APIs are fully consistent between the two:
+
+```python
+# Databricks DLT (02B_silver.py)
+dlt.create_auto_cdc_flow(
+    target=SILVER_CUSTOMERS,
+    source=f"live.{CUSTOMERS_CLEANED_STREAM}",
+    keys=["customer_id"],
+    sequence_by=F.col("last_update_time"),
+    stored_as_scd_type=2,
+)
+```
+
+```python
+# ZettaPark (03_lakehouse/02B_silver.py)
+from clickzetta.zettapark.window import Window   # ← pyspark.sql.window → clickzetta.zettapark.window
+
+w = Window.partitionBy("customer_id").orderBy("last_update_time")
+# F.lead() and Window API are fully consistent, no changes needed
+silver_customers = customers.withColumn(
+    "__end_at", F.lead("last_update_time").over(w)   # SCD2 end timestamp
+).withColumn(
+    "__is_current", F.lead("last_update_time").over(w).isNull()
+).withColumn(
+    "__end_at", F.coalesce(F.col("__end_at"), F.lit("9999-12-31 23:59:59").cast(TimestampType()))
+)
+silver_customers.write.mode("overwrite").saveAsTable("apparel_silver.silver_customers")
+```
+
+**Result**: 150 raw records → 150 rows (including 30 historical) → 120 current snapshot records (`__is_current = TRUE`).
+
+### Step 4: Time Window Aggregation — `F.window()` → `F.to_date()`
+
+`F.window("event_time","1 day")` is not implemented in ZettaPark. Use `F.to_date()` instead (functionally equivalent, results consistent):
+
+```python
+# Databricks DLT (03_gold.py)
+df.groupBy(F.window("event_time","1 day").alias("sale_window"), "store_id","store_name") \
+  .agg(F.round(F.sum("total_amount"),2).alias("total_revenue")) \
+  .select(F.col("sale_window.start").cast("date").alias("sale_date"), ...)
+```
+
+```python
+# ZettaPark (03_lakehouse/03_gold.py)
+facts.withColumn("sale_date", F.to_date(F.col("event_time")))   # F.window → F.to_date
+     .groupBy("sale_date","store_id","store_name")
+     .agg(F.round(F.sum("total_amount"),2).alias("total_revenue"),
+          F.count("transaction_id").alias("total_transactions"), ...)
+```
+
+> 💡 `F.to_date()` is equivalent to truncating to the day, which produces exactly the same grouping result as `F.window("1 day")`.
+
+---
+
+## Pure SQL Alternative
+
+If the team prefers SQL, `03_lakehouse/sql/` provides equivalent SQL scripts:
+
+```bash
+cz-cli sql --file 03_lakehouse/sql/02_silver.sql --profile aws_singapore_prod --sync --write
+cz-cli sql --file 03_lakehouse/sql/03_gold.sql   --profile aws_singapore_prod --sync --write
+```
+
+| DLT Python | SQL Equivalent |
+|---|---|
+| `@dlt.expect_or_drop` | `WHERE condition` |
+| `create_auto_cdc_flow(scd=2)` | `LEAD() OVER (PARTITION BY key ORDER BY seq)` |
+| `F.window("event_time","1 day")` | `DATE_TRUNC('day', event_time)` |
+
+ZettaPark is the recommended primary path (minimal code changes, retains PySpark skills); SQL is suitable for SQL-first teams or rapid validation.
+
+---
+
+## Option C: Dynamic Table (GIC) — Native Equivalent of DLT
+
+Dynamic Table is the most direct equivalent of DLT's `@dlt.table` — declare SQL declaratively, and the platform automatically refreshes on schedule. DLT triggers on new data automatically; Dynamic Table refreshes on a `REFRESH INTERVAL` schedule. Logically equivalent.
+
+```python
+# Databricks DLT — declarative pipeline
+@dlt.table(name="gold_daily_sales_by_store")
+def gold_daily_sales_by_store():
+    return df.groupBy(F.window("event_time","1 day"), "store_id","store_name") \
+             .agg(F.sum("total_amount").alias("total_revenue"))
+```
+
+```sql
+-- Lakehouse Dynamic Table — direct equivalent (03_lakehouse/dynamic_tables/gold_dynamic_tables.sql)
+CREATE OR REPLACE DYNAMIC TABLE apparel_gold.dt_daily_sales_by_store
+REFRESH INTERVAL 10 MINUTE    -- DLT triggers on new data; DT triggers on schedule
+VCLUSTER DEFAULT
+AS
+SELECT CAST(event_time AS DATE) AS sale_date, store_id, store_name,
+    ROUND(SUM(total_amount), 2) AS total_revenue, ...
+FROM apparel_gold.denormalized_sales_facts
+GROUP BY CAST(event_time AS DATE), store_id, store_name;
+
+-- Manual trigger (equivalent to DLT pipeline trigger)
+REFRESH DYNAMIC TABLE apparel_gold.dt_daily_sales_by_store;
+```
+
+| DLT | Dynamic Table | Notes |
+|---|---|---|
+| `@dlt.table(name=X)` | `CREATE DYNAMIC TABLE X ... AS SELECT ...` | Direct equivalent |
+| Auto-refresh on new data | `REFRESH INTERVAL N MINUTE` | Different scheduling method, semantically equivalent |
+| `F.window("1 day")` | `CAST(event_time AS DATE)` | DT uses SQL |
+| `create_auto_cdc_flow(scd=2)` | `LEAD() OVER Window` embedded in DT definition | Standard window function |
+| DLT pipeline DAG | DT dependencies (downstream DT references upstream DT) | Declarative dependencies |
+
+Tested in AWS Singapore: 4 Dynamic Tables created and refreshed, all passed e2e validation: `dt_customers_current` (150 rows / 120 current), `dt_daily_sales_by_store` (367), `dt_product_performance` (50), `dt_customer_lifetime_value` (96).
+
+## About Pipe (Auto Loader Equivalent)
+
+**Pipe** is the Databricks Auto Loader equivalent — continuously monitors object storage (OSS/S3/COS) for new files, automatically triggering COPY INTO ingestion. Tested in AWS Singapore: after uploading a new CSV to S3, Pipe auto-ingested within 15 seconds.
+
+```sql
+-- Step 1: Create External Volume (connected to S3/OSS/COS)
+CREATE EXTERNAL VOLUME apparel_bronze.s3_sales_landing
+    LOCATION 's3://your-bucket/apparel_landing/'
+    USING CONNECTION s3_conn
+    DIRECTORY = (enable=true, auto_refresh=true)
+    RECURSIVE = true;
+
+-- Step 2: Create table (External Volume does not support inferSchema, explicit definition required)
+CREATE TABLE apparel_bronze.raw_sales (
+    transaction_id BIGINT, store_id BIGINT, event_time STRING,
+    customer_id BIGINT, product_id BIGINT, quantity BIGINT,
+    unit_price DOUBLE, total_amount DOUBLE, payment_method STRING,
+    discount_applied DOUBLE, tax_amount DOUBLE
+);
+
+-- Step 3: Create Pipe
+-- DLT: spark.readStream.format("cloudFiles").option("cloudFiles.format","csv").load(PATH)
+CREATE OR REPLACE PIPE apparel_bronze.pipe_sales
+    VIRTUAL_CLUSTER = 'DEFAULT'
+    INGEST_MODE = 'LIST_PURGE'    -- Poll for new files, equivalent to Auto Loader LIST mode
+AS COPY INTO apparel_bronze.raw_sales
+FROM VOLUME apparel_bronze.s3_sales_landing
+USING CSV OPTIONS ('header'='true')
+PURGE = TRUE                      -- Delete landing zone files after processing
+ON_ERROR = CONTINUE;
+```
+
+> 💡 **Pipe vs COPY INTO**: COPY INTO is a one-time batch load; Pipe continuously monitors and auto-triggers when new files arrive — a direct equivalent to Auto Loader. Internal Volumes only support COPY INTO; Pipe requires an External Volume (OSS/S3/COS).
+
+---
+
+## E2E Validation Results
+
+Tested on AWS Singapore instance (`aws_singapore_prod`), **20/20 all passed** (including Dynamic Table validation):
+
+| Check | Expected | Result |
+|--------|--------|------|
+| bronze.raw_customers | 150 | ✅ |
+| bronze.raw_products | 50 | ✅ |
+| bronze.raw_stores | 5 | ✅ |
+| bronze.raw_sales | 500 | ✅ |
+| silver_customers (including history) | 150 | ✅ |
+| silver_products | 50 | ✅ |
+| silver_stores | 5 | ✅ |
+| silver_sales_transactions | 500 | ✅ |
+| silver_customers_current | 120 | ✅ |
+| silver_products_current | 50 | ✅ |
+| silver_stores_current | 5 | ✅ |
+| gold.denormalized_sales_facts | 500 | ✅ |
+| gold.gold_product_performance | 50 | ✅ |
+| Total sales amount | 281,490 | ✅ |
+| Customers with purchases | 96 | ✅ |
+| SCD2 historical records | 30 | ✅ |
+| **dt_customers_current** (Dynamic Table) | 150 | ✅ |
+| **dt_daily_sales_by_store** (Dynamic Table) | ≥100 | ✅ 367 |
+| **dt_product_performance** (Dynamic Table) | 50 | ✅ |
+| **dt_customer_lifetime_value** (Dynamic Table) | 96 | ✅ |
+| SCD2 historical records | 30 | ✅ |
+
+---
+
+## Notes
+
+- **Streaming vs batch**: Auto Loader continuously monitors new files; COPY INTO is a one-time batch. Use Lakehouse **Pipe** instead of Auto Loader in production — automatically ingests new files after configuration, no manual triggering required.
+- **`@dlt.expect` warn semantics**: `expect` (warn level) in DLT counts but does not drop records. When migrating to SQL, you can choose to not filter (ignore entirely) or add a `WHERE` filter (equivalent to upgrading to `expect_or_drop`). Decide based on business requirements.
+- **DT auto-incremental**: Lakehouse Dynamic Tables automatically refresh incrementally after definition (similar to DLT streaming updates), without needing to explicitly write `read_stream`.
+- **SCD2 `__is_current` field**: `LEAD()` returning NULL means there is no subsequent version, i.e., this is the current record. `COALESCE(__end_at, '9999-12-31')` is the standard SCD2 convention, with identical behavior on both sides.
+
+## Related Documentation
+
+### Dynamic Table (GIC)
+
+- [Dynamic Table Overview](dynamic-table-overview.md): Declarative incremental computation principles and architecture
+- [Dynamic Table SQL Reference](dynamic-table-sql.md): Full CREATE DYNAMIC TABLE syntax
+- [Dynamic Table Usage Guide](SQL_DynamicTable_Guide.md): Unified batch-streaming pipeline examples
+
+### Other Migration Guides
+
+- [Databricks Notebook → Lakehouse Migration Guide (Retail Medallion Pipeline)](databricks-notebook-to-studio-migration.md)
+- [dbt-databricks → dbt-clickzetta Migration Guide (Financial Payment Pipeline)](dbt-databricks-to-clickzetta-migration.md)
+- [Spark Migration Guide](spark-migration-guide.md): Spark ecosystem migration overview