@clickzetta/cz-cli-darwin-x64 0.5.16 → 0.5.17

package/bin/skills/lakehouse-doc-en/references/databricks-notebook-to-studio-migration.md

@@ -0,0 +1,350 @@
+# Databricks Notebook → Lakehouse Migration Guide: Retail Data Medallion Pipeline
+
+If your data engineering pipeline runs on Databricks Notebooks, the migration effort to Singdata Lakehouse Studio is lower than you might expect. Databricks' PySpark DataFrame API — `select`, `filter`, `join`, `withColumn`, `when`, Window functions — has identical syntax in ZettaPark. Changes are limited to just 5 mechanical substitutions: import paths, session acquisition method, table path prefix, one API casing difference, and replacing `dbutils.notebook.run()` with Studio task dependencies.
+
+This article validates this with a real project: a Databricks-based retail data Medallion pipeline (Bronze → Silver → Gold three-layer architecture, 14 Notebooks, 81 code cells) fully migrated to Singdata Lakehouse, offering three migration options, all 20 automated validations passing.
+
+Full code on GitHub: [databricks2lakehouse-bootcamp](https://github.com/clickzetta/databricks2lakehouse-bootcamp)
+
+---
+
+## Source Project
+
+[databricks2lakehouse-bootcamp](https://github.com/clickzetta/databricks2lakehouse-bootcamp) is forked from [DataWithBaraa/databricks_bootcamp_2026](https://github.com/DataWithBaraa/databricks_bootcamp_2026) (⭐335). The original tech stack is Databricks + PySpark + Delta Lake + Unity Catalog. The project implements a complete retail data warehouse from dual-source CRM/ERP ingestion to a star schema, covering 18,484 customers, 397 products, and 60,398 sales records, with complete data cleansing, type conversion, code mapping, and dimensional modeling.
+
+Migrated code is in the `03_lakehouse/` directory, comparable file-by-file with `01_source/`.
+
+## Conclusion First
+
+You don't need to rewrite any business logic, or retrain your team. All 5 changes are mechanical substitutions.
+
+| Change | Effort | Notes |
+|--------|--------|------|
+| Import path replacement | Very low | `pyspark.sql` → `clickzetta.zettapark`, global search-replace |
+| Session acquisition method | Very low | `spark` (global injection) → Studio tasks use `clickzetta_dbutils`, local use `Session.builder.configs({})` |
+| Table path prefix | Very low | `workspace.bronze.X` → `bronze.X`, remove catalog prefix |
+| StructField casing | Very low | `field.dataType` → `field.datatype` (ZettaPark API difference) |
+| Orchestration method | Low | `dbutils.notebook.run(nb)` → Studio task dependencies (DAG) |
+
+`select`, `filter`, `join`, `withColumn`, `when`, `coalesce`, `trim`, `regexp_replace`, `to_date`, `cast`, `isNotNull`, Window functions, `ROW_NUMBER()` — these core data engineering operations have identical syntax and require no changes.
+
+---
+
+## Tech Stack Comparison
+
+| | Databricks Notebook | Lakehouse Studio Task |
+|---|---|---|
+| Compute engine | Apache Spark (Databricks) | Singdata Lakehouse |
+| DataFrame API | PySpark (`pyspark.sql`) | ZettaPark (`clickzetta.zettapark`) |
+| Session acquisition | `spark` (Databricks global injection) | `clickzetta_dbutils.get_active_lakehouse_engine()` |
+| Table naming | `workspace.bronze.crm_cust_info` (3-level) | `bronze.crm_cust_info` (2-level) |
+| File path | `/Volumes/workspace/bronze/raw_sources/...` | `vol://bronze.raw_sources/...` |
+| StructField | `field.dataType` | `field.datatype` |
+| SQL execution | `spark.sql(q)` executes immediately | `session.sql(q).collect()` triggers execution |
+| Notebook chaining | `dbutils.notebook.run(nb, timeout_seconds=0)` | Studio task dependencies (`--deps` parameter) |
+| Scheduling orchestration | Databricks Jobs | Studio task DAG |
+
+---
+
+![](.topwrite/assets/anim-28-databricks-notebook-migration.svg)
+
+---
+
+## Project Background
+
+Data comes from a bicycle retailer's dual-source CRM + ERP system, including 6 CSV files:
+
+| Data Source | Table Name | Row Count | Notes |
+|--------|------|------|------|
+| CRM | `crm_cust_info` | 18,494 | Customer info (includes dirty data and inconsistent casing) |
+| CRM | `crm_prd_info` | 397 | Product info (includes category ID encoded in product key) |
+| CRM | `crm_sales_details` | 60,398 | Sales transactions (includes yyyyMMdd format dates) |
+| ERP | `erp_cust_az12` | 18,484 | Customer master data (includes NAS prefix, future dates) |
+| ERP | `erp_loc_a101` | 18,484 | Customer addresses (includes hyphens and country code abbreviations) |
+| ERP | `erp_px_cat_g1v2` | 37 | Product categories (includes YES/NO maintenance flags) |
+
+Medallion architecture in three layers:
+
+- **Bronze**: Raw CSV → 6 Delta tables (no transformation)
+- **Silver**: Cleansing + normalization → 6 wide tables (trim, type conversion, enum mapping, column renaming)
+- **Gold**: Star schema → dim_customers + dim_products + fact_sales
+
+The original project has 14 Databricks Notebooks (81 code cells), chained via `dbutils.notebook.run()`.
+
+---
+
+## Migration Steps
+
+### Step 1: Replace Import Paths
+
+Mechanical global replacement, no logic changes:
+
+```python
+# Databricks
+import pyspark.sql.functions as F
+from pyspark.sql.types import StringType, DateType
+from pyspark.sql.functions import trim, col, length
+from pyspark.sql.window import Window
+
+# ZettaPark (package name only)
+from clickzetta.zettapark import functions as F
+from clickzetta.zettapark.types import StringType, DateType
+from clickzetta.zettapark.functions import trim, col, length
+from clickzetta.zettapark.window import Window
+```
+
+### Step 2: Replace Session Acquisition Method
+
+Databricks injects `spark` into every Notebook without requiring explicit creation. Studio tasks obtain it via `clickzetta_dbutils`, also without needing to manage passwords or connection parameters:
+
+```python
+# Databricks: spark is globally available, use directly
+df = spark.table("workspace.bronze.crm_cust_info")
+
+# Studio task: platform-injected, get session via clickzetta_dbutils
+from clickzetta_dbutils import get_active_lakehouse_engine
+from clickzetta.zettapark.session import Session
+from urllib.parse import urlparse, parse_qs
+
+engine = get_active_lakehouse_engine(schema="quick_start")
+url_str = str(engine.url)
+parsed = urlparse(url_str.replace('clickzetta://', 'https://'))
+params = parse_qs(parsed.query)
+parts = parsed.hostname.split('.', 1)
+
+session = Session.builder.configs({
+    "service":     parts[1],
+    "instance":    parts[0],
+    "magic_token": params['magic_token'][0],
+    "workspace":   parsed.path.lstrip('/'),
+    "schema":      params.get('schema', ['quick_start'])[0],
+    "vcluster":    params.get('virtualcluster', ['DEFAULT'])[0],
+}).getOrCreate()
+
+# After this, session usage is identical to spark
+df = session.table("bronze.crm_cust_info")
+```
+
+> 💡 **Note**: For local development and debugging, explicitly pass credentials with `Session.builder.configs({"instance":..., "password":..., ...}).create()`, without relying on `clickzetta_dbutils`. DataFrame code is identical after either acquisition method.
+
+### Step 3: Remove Catalog Prefix from Table Paths
+
+Unity Catalog uses three-level naming (catalog.schema.table). Lakehouse only requires two levels within a single workspace:
+
+```python
+# Databricks (Unity Catalog three-level)
+df = spark.table("workspace.bronze.crm_cust_info")
+df.write.mode("overwrite").format("delta").saveAsTable("workspace.silver.crm_customers")
+
+# ZettaPark (two-level, remove catalog prefix)
+df = session.table("bronze.crm_cust_info")
+df.write.mode("overwrite").saveAsTable("silver.crm_customers")  # .format("delta") not needed either
+```
+
+### Step 4: Fix StructField Casing
+
+This is an API difference discovered in testing, affecting all code that iterates field types with `df.schema.fields`:
+
+```python
+# Databricks / PySpark
+for field in df.schema.fields:
+    if isinstance(field.dataType, StringType):  # uppercase T
+        df = df.withColumn(field.name, trim(col(field.name)))
+
+# ZettaPark (datatype all lowercase)
+for field in df.schema.fields:
+    if isinstance(field.datatype, StringType):  # lowercase t
+        df = df.withColumn(field.name, trim(col(field.name)))
+```
+
+### Step 5: Replace dbutils.notebook.run() with Studio Task Dependencies
+
+The original project chains 14 Notebooks via `dbutils.notebook.run()`, which maps directly to Studio task DAG:
+
+```python
+# Databricks: orchestration notebook
+notebooks = [
+    "./silver_crm_cust_info",
+    "./silver_crm_prd_info",
+    "./silver_crm_sales_details",
+    "./silver_erp_cust_az12",
+    "./silver_erp_loc_a101",
+    "./silver_erp_px_cat_g1v2"
+]
+for nb in notebooks:
+    dbutils.notebook.run(nb, timeout_seconds=0)
+```
+
+```bash
+# Studio: set task dependencies with cz-cli, one-time configuration, platform handles scheduling
+cz-cli task save-config bootcamp/silver_crm_cust_info \
+    --deps bootcamp/bronze_ingestion --profile aws_singapore_prod
+
+cz-cli task save-config bootcamp/gold_dim_customers \
+    --deps bootcamp/silver_crm_cust_info,bootcamp/silver_erp_cust_az12,bootcamp/silver_erp_loc_a101 \
+    --profile aws_singapore_prod
+```
+
+Execute DAG (equivalent to the original orchestration notebook's chained calls):
+
+```bash
+cz-cli task execute bootcamp/init_lakehouse --profile aws_singapore_prod
+# → automatically triggers bronze → silver (parallel) → gold (in dependency order)
+```
+
+---
+
+## Studio Task DAG After Migration
+
+The original Databricks project used 2 orchestration notebooks chained in series. After migration, the Studio platform automatically manages dependencies and parallelism:
+
+```
+init_lakehouse (SQL task)
+└── bronze_ingestion (Python task)
+    ├── silver_crm_cust_info      ← parallel execution
+    ├── silver_crm_prd_info       ← parallel execution
+    ├── silver_crm_sales_details  ← parallel execution
+    ├── silver_erp_cust_az12      ← parallel execution
+    ├── silver_erp_loc_a101       ← parallel execution
+    └── silver_erp_px_cat_g1v2    ← parallel execution
+        ├── gold_dim_customers  ← depends on silver_crm_cust_info + erp_cust + erp_loc
+        ├── gold_dim_products   ← depends on silver_crm_prd_info + erp_px_cat
+        └── gold_fact_sales     ← depends on dim_customers + dim_products + silver_crm_sales
+```
+
+The original Databricks ran 6 silver notebooks sequentially; Studio runs 6 silver tasks in parallel — same logic, shorter overall runtime.
+
+---
+
+## Fully Compatible Parts
+
+The following code has identical syntax on both sides, validated by testing — no changes needed:
+
+```python
+# String cleansing
+for field in df.schema.fields:
+    if isinstance(field.datatype, StringType):
+        df = df.withColumn(field.name, trim(col(field.name)))
+
+# Enum mapping (conditional replacement)
+df = df.withColumn("cst_marital_status",
+    F.when(F.upper(F.col("cst_marital_status")) == "S", "Single")
+     .when(F.upper(F.col("cst_marital_status")) == "M", "Married")
+     .otherwise("n/a"))
+
+# Composite string parsing (extract category ID from product key)
+df = df.withColumn("cat_id", F.regexp_replace(F.substring(col("prd_key"), 1, 5), "-", "_"))
+df = df.withColumn("prd_key", F.substring(col("prd_key"), 7, F.length(col("prd_key"))))
+
+# Date format conversion (yyyyMMdd integer → DATE)
+df = df.withColumn("sls_order_dt",
+    F.when(
+        (col("sls_order_dt") == 0) | (length(col("sls_order_dt")) != 8), None
+    ).otherwise(F.to_date(col("sls_order_dt").cast("string"), "yyyyMMdd")))
+
+# Conditional price fix (derive from sales/quantity when quantity != 0)
+df = df.withColumn("sls_price",
+    F.when(
+        (col("sls_price").isNull()) | (col("sls_price") <= 0),
+        F.when(col("sls_quantity") != 0, col("sls_sales") / col("sls_quantity")).otherwise(None)
+    ).otherwise(col("sls_price")))
+
+# Prefix cleansing (remove invalid NAS prefix)
+df = df.withColumn("cid",
+    F.when(col("cid").startswith("NAS"), F.substring(col("cid"), 4, F.length(col("cid"))))
+     .otherwise(col("cid")))
+
+# Future date filtering (dirty data cleansing)
+df = df.withColumn("bdate",
+    F.when(col("bdate") > F.current_date(), None).otherwise(col("bdate")))
+
+# Multi-table LEFT JOIN + ROW_NUMBER for dimension table
+df = session.sql("""
+SELECT
+    ROW_NUMBER() OVER (ORDER BY ci.customer_id) AS customer_key,
+    ci.customer_id, ci.customer_number, ci.first_name, ci.last_name,
+    COALESCE(la.country, 'n/a') AS country, ci.marital_status,
+    CASE WHEN ci.gender <> 'n/a' THEN ci.gender ELSE COALESCE(ca.gender, 'n/a') END AS gender,
+    ca.birth_date, ci.created_date
+FROM silver.crm_customers ci
+LEFT JOIN silver.erp_customer_location la ON ci.customer_number = la.customer_number
+LEFT JOIN silver.erp_customers ca ON ci.customer_number = ca.customer_number
+""")
+```
+
+---
+
+## E2E Validation Results
+
+Tested on AWS Singapore instance (`aws_singapore_prod`), 20/20 all passed:
+
+| Check | Expected | Result |
+|--------|--------|------|
+| bronze.crm_cust_info | 18,494 | ✅ |
+| bronze.crm_prd_info | 397 | ✅ |
+| bronze.crm_sales_details | 60,398 | ✅ |
+| bronze.erp_cust_az12 | 18,484 | ✅ |
+| bronze.erp_loc_a101 | 18,484 | ✅ |
+| bronze.erp_px_cat_g1v2 | 37 | ✅ |
+| silver.crm_customers | 18,490 | ✅ |
+| silver.crm_products | 397 | ✅ |
+| silver.crm_sales | 60,398 | ✅ |
+| silver.erp_customers | 18,484 | ✅ |
+| silver.erp_customer_location | 18,484 | ✅ |
+| silver.erp_product_category | 37 | ✅ |
+| gold.dim_customers | 18,490 | ✅ |
+| gold.dim_products | 397 | ✅ |
+| gold.fact_sales | 89,833 | ✅ |
+| Total sales amount | 43,538,800 | ✅ |
+| Rows with negative sales | 5 (raw data) | ✅ |
+| Distinct customer count | 18,484 | ✅ |
+| Distinct product SKUs | 295 | ✅ |
+| Rows with null order date | 24 (raw data format issue) | ✅ |
+
+> Bronze 18,494 → Silver 18,490: 4 records with NULL customer_id were filtered during Silver cleansing, consistent with original Databricks behavior.
+
+---
+
+## Three Migration Options Compared
+
+This project provides three options suited to different team backgrounds and deployment needs:
+
+| Option | File Location | Session Method | Change Volume | Use Case |
+|------|----------|-------------|--------|----------|
+| **A. ZettaPark Local** | `03_lakehouse/{bronze,silver,gold}/` | `Session.builder.configs({}).create()` | ~5% | Local development, CI/CD debugging |
+| **B. Pure SQL** | `03_lakehouse/sql/` | Not needed | Full rewrite (logic unchanged) | SQL-first teams, Studio SQL tasks |
+| **C. Studio Task** | `03_lakehouse/tasks/` | `clickzetta_dbutils` | ~5% | **Production deployment (recommended)** |
+
+All three options produce identical results — row counts, metrics, and data are consistent. Option C is the recommended path for production, directly mapping to the Databricks Notebook + Jobs architecture.
+
+## Notes
+
+- **`field.datatype` casing**: ZettaPark's `StructField` property is `datatype` (all lowercase); PySpark uses `dataType` (camelCase). This is the only non-mechanical API difference. Search for all uses of `schema.fields` and update individually.
+- **`clickzetta_dbutils` template for Studio tasks**: The `get_active_lakehouse_engine()` call returns the current task's connection info; parse the URL to build the Session. This template code is fixed and can be extracted to a shared module for reuse.
+- **Volume path format**: Databricks `/Volumes/catalog/schema/volume/path` → ZettaPark `vol://schema.volume/path`, note the removal of the catalog level.
+- **`.format("delta")` not needed**: ZettaPark's `saveAsTable()` writes in Lakehouse native format by default; no explicit format specification required.
+- **Silver row count difference (18,494 → 18,490)**: The CRM source data contains 4 records with NULL customer_id, filtered during Silver cleansing. This is expected behavior.
+
+## Related Documentation
+
+### ZettaPark DataFrame API
+
+- [ZettaPark Quick Start](zettapark-quick-start.md): Session creation, DataFrame basics
+- [ZettaPark DataFrame API Guide](zettapark-dataframe-guide.md): Full reference for select, filter, join, withColumn, etc.
+- [ZettaPark Common Functions Reference](zettapark-functions-guide.md): Usage of trim, when, regexp_replace, Window, etc.
+- [ZettaPark Data Engineering in Practice](zettapark-etl-guide.md): Complete ETL pipeline examples
+
+### Studio Tasks and Orchestration
+
+- [Lakehouse Studio Introduction](lakehouse-studio-concept.md): Studio platform concepts and architecture
+- [Task Development and Scheduling](task-develop.md): Creating, developing, and scheduling Studio tasks
+- [Task Scheduling Dependencies](task_scheduling_dependency.md): `--deps` parameter for configuring task DAG
+- [Python Task](python-task.md): Creating and running Python tasks in Studio
+- [Studio Task Development and Operations (cz-cli)](cz-cli-studio-tasks.md): Managing Studio tasks via cz-cli command line
+
+### Other Migration Guides
+
+- [PySpark → ZettaPark Migration Guide (Formula 1)](pyspark-to-zettapark-migration-f1.md): PySpark DataFrame API query-by-query migration
+- [Snowpark → ZettaPark Migration Guide](snowflake-snowpark-to-zettapark-migration.md): Snowflake Python DataFrame migration
+- [Spark Migration Guide](spark-migration-guide.md): Spark ecosystem migration overview and common issues
+- [SQL Compatibility Reference](migration-sql-compatibility.md): Cross-platform SQL syntax differences

package/bin/skills/lakehouse-doc-en/references/databricks-uc-governance-to-lakehouse-migration.md

@@ -0,0 +1,327 @@
+# Databricks Unity Catalog → Lakehouse Migration Guide: Permissions and Governance
+
+If your data platform uses Databricks Unity Catalog to manage permissions — RBAC roles, column-level masking, row-level access control, data auditing — the migration effort to Singdata Lakehouse is lower than you might expect. GRANT/REVOKE syntax is fully consistent, role management requires no changes, and the `SET MASK` keyword is identical. Changes are concentrated in 3 areas: removing the catalog prefix (three-level → two-level naming), swapping one API in masking functions (`is_account_group_member` → `array_contains(current_roles(),...)`), and replacing declarative ROW FILTER with explicit security views for row-level security.
+
+This article validates this with a financial payment scenario: a user table with PII fields (email/phone/card), an orders table with region-based isolation, and a balance/accounts table — complete migration of RBAC, column masking, row-level security, and audit logs, passing all 16 automated validations.
+
+Full code on GitHub: [databricks2lakehouse-governance](https://github.com/clickzetta/databricks2lakehouse-governance)
+
+---
+
+## Source Project
+
+Demo data is a financial payment scenario with 3 tables:
+
+| Table | Rows | Sensitive Fields | Access Control Requirements |
+|----|------|---------|------------|
+| `users` | 100 | `email`, `phone`, `ssn_last4` | Masking: non-admin sees masked values |
+| `orders` | 300 | `amount`, `region` | Row-level: analyst only sees North America |
+| `accounts` | 50 | `balance`, `card_number` | Masking: last 4 digits of card visible |
+
+Migrated code is in the `03_lakehouse/sql/` directory (6 SQL files), comparable file-by-file with `01_source/sql/` (original UC SQL).
+
+## Conclusion First
+
+Most permission SQL can be reused directly with **very few or zero changes**.
+
+| Change | Effort | Notes |
+|--------|--------|------|
+| Masking function judgment API | Very low | `is_account_group_member('g')` → `array_contains(current_roles(), 'ws.g')` |
+| Audit table and column names | Low | `system.access.audit` → `sys.information_schema.job_history`, different column names |
+
+**No changes needed**: `CREATE ROLE`, `GRANT SELECT`, `GRANT ALL PRIVILEGES`, `WITH GRANT OPTION`, `SHOW GRANTS`, `SET MASK`, `SET ROW FILTER`, **three-level naming (workspace.schema.table)** — syntax fully consistent. Row-level security migrates with zero changes!
+
+> 💡 Namespacing: UC uses `catalog.schema.table`; Lakehouse uses `workspace.schema.table` — the structure is identical, just replacing the catalog name with the workspace name (a connection parameter change, no SQL code changes needed).
+
+Lakehouse also has additional capabilities that UC lacks: `mask_inner`/`mask_outer` (built-in string masking functions), `AI_MASK` (AI model semantic masking), `CREATE SHARE` (data sharing, SQL DDL simpler than Delta Sharing).
+
+---
+
+## Tech Stack Comparison
+
+| | Databricks Unity Catalog | Singdata Lakehouse |
+|---|---|---|
+| Namespace levels | Three-level: `catalog.schema.table` | Three-level: `workspace.schema.table` (same structure) |
+| Role management | `CREATE ROLE` (metastore-level) | `CREATE ROLE` (workspace-level) |
+| GRANT syntax | `GRANT ... ON TABLE cat.s.t TO ROLE r` | `GRANT ... ON TABLE s.t TO ROLE r` |
+| Column masking judgment | `is_account_group_member('group')` | `array_contains(current_roles(), 'ws.role')` |
+| Column masking application | `ALTER TABLE ... ALTER COLUMN c SET MASK f` | `ALTER TABLE ... ALTER COLUMN c SET MASK f` |
+| Row-level security | `ALTER TABLE ... SET ROW FILTER f ON (col)` | `ALTER TABLE ... SET ROW FILTER f ON (col)` |
+| Data sharing | Delta Sharing (UC + REST API) | `CREATE SHARE` / `GRANT select, read metadata ... TO SHARE` |
+| String masking | Hand-written REGEXP_REPLACE | `mask_inner()` / `mask_outer()` (built-in functions) |
+| AI intelligent masking | — | `AI_MASK()` (Lakehouse exclusive) |
+| Tag / ABAC | UC Tag policies (cross-table tagging) | Not yet supported → use role naming + Schema isolation as substitute |
+| Audit source | `system.access.audit` (account-level) | `sys.information_schema.job_history` (workspace-level) |
+
+---
+
+![](.topwrite/assets/anim-31-databricks-uc-governance-migration.svg)
+
+---
+
+## Project Background
+
+Three tables covering typical financial scenarios:
+
+- `users`: User registration info with PII fields like email/phone/ssn_last4, requiring column-level masking
+- `orders`: Sales orders partitioned by region (North America/Europe/Asia Pacific), requiring row-level isolation
+- `accounts`: Account balances and bank card info; balance and card_number are highly sensitive
+
+The original UC design defines 3 roles:
+- `payments_admin`: Full access to real data
+- `payments_analyst`: Masked data + only North America orders
+- `payments_viewer`: Masked data + only Asia orders
+
+---
+
+## Migration Steps
+
+### Step 1: Remove Catalog Prefix
+
+UC uses three-level naming; the Lakehouse workspace itself is the catalog boundary, requiring only two levels:
+
+```sql
+-- UC (three-level)
+GRANT SELECT ON TABLE payments_catalog.raw.orders TO ROLE payments_analyst;
+
+-- Lakehouse (two-level) — remove catalog prefix
+GRANT SELECT ON TABLE gov_raw.orders TO ROLE payments_analyst;
+```
+
+### Step 2: RBAC — Syntax Fully Consistent
+
+GRANT/REVOKE/SHOW GRANTS syntax requires zero changes:
+
+```sql
+-- Identical on both sides
+CREATE ROLE IF NOT EXISTS payments_analyst;
+CREATE ROLE IF NOT EXISTS payments_admin;
+
+GRANT SELECT ON TABLE gov_raw.orders TO ROLE payments_analyst;
+GRANT ALL PRIVILEGES ON SCHEMA gov_raw TO ROLE payments_admin;
+GRANT SELECT ON ALL TABLES IN SCHEMA gov_raw TO ROLE payments_viewer;
+GRANT SELECT ON TABLE gov_raw.users TO ROLE payments_admin WITH GRANT OPTION;
+
+SHOW GRANTS TO ROLE payments_analyst;
+```
+
+### Step 3: Column Masking — `SET MASK` Identical, One Function API Change
+
+UC uses `is_account_group_member()`; Lakehouse uses `array_contains(current_roles(), 'workspace.role')`:
+
+```python
+# UC masking function
+CREATE FUNCTION payments_catalog.raw.mask_email(email STRING)
+    RETURN CASE
+        WHEN is_account_group_member('payments_admin') THEN email
+        ELSE CONCAT(LEFT(email, 2), '***@***.***')
+    END;
+```
+
+```sql
+-- Lakehouse masking function (only the judgment API changes, logic identical)
+CREATE OR REPLACE FUNCTION gov_raw.mask_email(email STRING)
+    RETURNS STRING
+    RETURN CASE
+        WHEN array_contains(current_roles(), 'quick_start.payments_admin')  -- ← current_roles()
+          OR array_contains(current_roles(), 'workspace_admin')
+        THEN email
+        ELSE CONCAT(LEFT(email, 2), '***@***.***')
+    END;
+
+-- SET MASK application syntax is identical
+ALTER TABLE gov_raw.users ALTER COLUMN email SET MASK gov_raw.mask_email;
+ALTER TABLE gov_raw.users ALTER COLUMN phone SET MASK gov_raw.mask_phone;
+ALTER TABLE gov_raw.accounts ALTER COLUMN card_number SET MASK gov_raw.mask_card;
+```
+
+Tested masking results (seen by non-admin users):
+
+| Field | Original Value | After Masking |
+|------|--------|--------|
+| email | `user001@example.com` | `u***@***.***` |
+| phone | `+12345678901` | `****8901` |
+| card_number | `4866524041574` | `****-****-****-1574` |
+
+### Step 4: Row-Level Security — `SET ROW FILTER` Syntax Fully Consistent
+
+Lakehouse natively supports `SET ROW FILTER`, with syntax **fully consistent with UC, zero changes**:
+
+```sql
+-- UC (Databricks)
+CREATE FUNCTION payments_catalog.raw.filter_orders_by_region(region STRING)
+    RETURN CASE
+        WHEN is_account_group_member('payments_admin') THEN TRUE
+        WHEN is_account_group_member('payments_analyst')
+             AND region = 'North America' THEN TRUE
+        ELSE FALSE
+    END;
+
+ALTER TABLE payments_catalog.raw.orders
+    SET ROW FILTER payments_catalog.raw.filter_orders_by_region ON (region);
+```
+
+```sql
+-- Lakehouse — syntax fully consistent, only change is is_account_group_member in function body
+CREATE OR REPLACE FUNCTION gov_raw.filter_orders_by_role(region STRING)
+RETURNS BOOLEAN
+AS
+    array_contains(current_roles(), 'quick_start.payments_admin')
+    OR array_contains(current_roles(), 'workspace_admin')
+    OR (array_contains(current_roles(), 'quick_start.payments_analyst')
+        AND region = 'North America')
+    OR (array_contains(current_roles(), 'quick_start.payments_viewer')
+        AND region = 'Asia');
+
+-- SET ROW FILTER syntax is identical
+ALTER TABLE gov_raw.orders SET ROW FILTER gov_raw.filter_orders_by_role ON (region);
+
+-- Verify binding
+DESC EXTENDED gov_raw.orders;
+-- Output includes # Row Filter section at the end
+
+-- Remove
+ALTER TABLE gov_raw.orders DROP ROW FILTER;
+```
+
+ROW FILTER takes full effect for SELECT, COUNT, UPDATE, DELETE — the current user (workspace_admin) queries 300 rows; a `payments_analyst` role user only sees the 52 North America rows.
+
+### Step 5: Auditing — Table and Column Name Adaptation
+
+UC uses `system.access.audit` (account-level); Lakehouse uses `sys.information_schema.job_history` (workspace-level):
+
+```sql
+-- UC auditing
+SELECT event_time, user_name, action_name, request_params
+FROM system.access.audit
+WHERE event_time >= current_timestamp() - INTERVAL 7 DAYS
+  AND action_name IN ('createTable','grantPermission','revokePermission');
+```
+
+```sql
+-- Lakehouse auditing (different column names, date uses literal string)
+SELECT
+    job_type,
+    job_creator,           -- UC: user_name
+    LEFT(job_text, 100) AS sql_preview,  -- UC: request_params
+    status,
+    LEFT(start_time, 19) AS time
+FROM sys.information_schema.job_history
+WHERE start_time >= '2026-06-07'   -- must be literal, cannot use NOW() - INTERVAL
+  AND (UPPER(job_text) LIKE 'GRANT%'
+    OR UPPER(job_text) LIKE 'REVOKE%'
+    OR UPPER(job_text) LIKE 'CREATE TABLE%')
+ORDER BY start_time DESC;
+```
+
+---
+
+## Data Sharing
+
+Lakehouse's **Share** is the equivalent of Databricks Delta Sharing — share tables or views with other workspaces or external consumers, with syntax highly similar to UC:
+
+```sql
+-- Step 1: Create Share object
+CREATE SHARE IF NOT EXISTS analytics_share
+    COMMENT 'Analytics data share for partners';
+
+-- Step 2: Add tables/views to Share (dual permissions: select + read metadata)
+GRANT select, read metadata ON VIEW gov_marts.orders_by_region TO SHARE analytics_share;
+GRANT select, read metadata ON TABLE gov_raw.orders TO SHARE analytics_share;
+
+-- Bulk add all tables in a schema
+GRANT SELECT, READ METADATA ON ALL TABLES IN SCHEMA gov_raw TO SHARE analytics_share;
+
+-- Step 3: View Share contents
+DESC SHARE analytics_share;
+
+-- Step 4: Consumer access
+CREATE SCHEMA partner_schema FROM SHARE analytics_share;
+```
+
+`SELECT, READ METADATA` dual permission combination: `select` allows querying data; `read metadata` allows consumers to discover the existence of tables/views. Both are typically granted together. UC Delta Sharing configuration is done via the Databricks UI/REST API; Lakehouse Share uses SQL DDL directly, which is more concise.
+
+---
+
+## Lakehouse-Exclusive Capabilities
+
+**`mask_inner` / `mask_outer`**: Built-in string masking functions not available in UC, usable directly in masking functions without AI connection:
+
+```sql
+-- mask_inner: mask middle characters (preserve first n and last m characters)
+SELECT mask_inner('user001@example.com', 2, 7);
+-- → usXXXXXXXXXXXXX.com (preserve first 2, last 7, mask middle)
+
+-- mask_outer: mask outer characters
+SELECT mask_outer('+12345678901', 1, 4);
+-- → X1234567XXXX
+
+-- More concise in masking functions than hand-written CONCAT+REGEXP_REPLACE
+CREATE OR REPLACE FUNCTION gov_raw.mask_email_v2(email STRING)
+    RETURNS STRING
+    RETURN CASE
+        WHEN array_contains(current_roles(), 'quick_start.payments_admin') THEN email
+        ELSE mask_inner(email, 2, 7)
+    END;
+```
+
+**`AI_MASK`**: AI model-based semantic masking, automatically identifies PII types (names, phones, ID numbers, etc.). UC has no equivalent:
+
+```sql
+-- Requires AI connection configuration (Bailian/Tongyi models etc.)
+SELECT AI_MASK('conn_bailian:qwen3.5-plus',
+               'User Wang Xiaoming, phone: 13800138000',
+               ARRAY('Name', 'Phone'));
+-- → 'User ***, phone: 138****8000'
+```
+
+---
+
+## Validation Results
+
+Tested on AWS Singapore instance (`aws_singapore_prod`), 16/16 all passed:
+
+| Check | Expected | Result |
+|--------|--------|------|
+| gov_raw.users | 100 | ✅ |
+| gov_raw.orders | 300 | ✅ |
+| gov_raw.accounts | 50 | ✅ |
+| analyst has GRANT records | ≥2 | ✅ |
+| admin has GRANT records | ≥1 | ✅ |
+| viewer has GRANT records | ≥1 | ✅ |
+| email is masked | True | ✅ |
+| phone is masked | True | ✅ |
+| card_number is masked | True | ✅ |
+| mask_email function exists | True | ✅ |
+| mask_phone function exists | True | ✅ |
+| mask_card function exists | True | ✅ |
+| orders_by_region admin sees 300 rows | 300 | ✅ |
+| orders_analyst_view only sees North America | 52 | ✅ |
+| analyst_view has only 1 region | 1 | ✅ |
+| job_history has records | ≥1 | ✅ |
+
+---
+
+## Notes
+
+- **`current_roles()` includes workspace prefix**: Lakehouse's `current_roles()` returns role names with workspace prefix (e.g., `quick_start.payments_admin`). When matching, the full prefix is required — `payments_admin` alone will not work.
+- **`SET MASK` becomes invalid after table recreation**: After DROP TABLE + CREATE, column masks are not inherited automatically. `ALTER TABLE ... ALTER COLUMN c SET MASK` must be re-applied.
+- **COPY INTO type inference**: External Volume COPY INTO does not support `inferSchema`. Column types must be explicitly declared when creating tables (especially phone/card should use STRING; otherwise inferred as BIGINT causing data loss).
+- **job_history date condition**: `WHERE start_time >= '2026-06-07'` must be a literal string. `CURRENT_DATE()` or `INTERVAL` expressions are not supported.
+- **Row-level security function judgment API**: The ROW FILTER function body also requires replacing `is_account_group_member('g')` with `array_contains(current_roles(), 'ws.g')`. This is the only change point. `ALTER TABLE ... SET ROW FILTER` / `DROP ROW FILTER` syntax is identical.
+- **Tag / ABAC (honest limitation)**: UC supports fine-grained attribute-based access control (ABAC) via Tags — tagging tables/columns and then writing policies. Lakehouse does not currently support Tag-driven ABAC. The recommended approach is to use role naming conventions (e.g., `dept_finance_analyst`) + Schema isolation to simulate ABAC, or keep tag-based permissions managed on the Databricks side.
+
+## Related Documentation
+
+### Permissions and Security
+
+- [GRANT](GRANT.md): Full GRANT syntax reference
+- [CREATE ROLE](create-role.md): Role creation and management
+- [SET MASK](set-mask.md): Column masking syntax
+- [SHOW GRANTS](show-grants.md): View authorization relationships
+
+### Other Migration Guides
+
+- [Databricks Notebook → Lakehouse Migration Guide](databricks-notebook-to-studio-migration.md)
+- [Databricks DLT → Lakehouse Migration Guide](databricks-dlt-to-lakehouse-migration.md)
+- [dbt-databricks → dbt-clickzetta Migration Guide](dbt-databricks-to-clickzetta-migration.md)