@clickzetta/cz-cli-darwin-x64 0.5.16 → 0.5.17

package/bin/skills/lakehouse-doc-en/references/dbt-databricks-to-clickzetta-migration.md

@@ -0,0 +1,242 @@
+# dbt-databricks → dbt-clickzetta Migration in Practice: Financial Payment Data Pipeline
+
+If your data pipeline is built with dbt + Databricks SQL, migrating to Singdata Lakehouse is far less work than you might expect — as little as changing one line in `profiles.yml`. All core dbt capabilities — CTE models, window functions, SCD Type 2 logic, data quality tests, unit tests, and data contracts — are fully compatible with Lakehouse and require zero changes.
+
+This article validates that claim with a real project: a financial payment data pipeline built on dbt-databricks (staging → intermediate → marts → semantic, 30 models, 36 tests) was fully migrated to dbt-clickzetta and verified with `dbt seed 9/9 + dbt run 30/30 + dbt test 36/36` — all passing.
+
+Full code on GitHub: [dbt-databricks2lakehouse-blueprint](https://github.com/clickzetta/dbt-databricks2lakehouse-blueprint)
+
+---
+
+## Original Project
+
+[dbt-databricks2lakehouse-blueprint](https://github.com/clickzetta/dbt-databricks2lakehouse-blueprint) is forked from [Alex-Teodosiu/dbt-blueprint](https://github.com/Alex-Teodosiu/dbt-blueprint) (⭐72). The original stack is dbt-databricks + Databricks SQL Warehouse. The project simulates a P2P/P2B (peer-to-peer / peer-to-business) payment platform data engineering pipeline, covering five data domains — users, merchants, accounts, bank cards, and transactions — and implements the complete chain from raw events → cleansing → SCD Type 2 dimension tables → fact tables → semantic layer.
+
+The migrated code is in the `03_lakehouse/` directory and can be compared file-by-file with `01_source/dbt_blueprint/`.
+
+## Conclusion First
+
+**The adapter is dbt's database dialect layer** — switching adapters means switching the database; the dbt models themselves do not need to change. Changes are concentrated in `profiles.yml` (connection configuration) and a handful of Databricks-specific syntax items.
+
+| Change | Files | Effort | Notes |
+|--------|-------|--------|-------|
+| Adapter switch | 1 (`profiles.yml`) | Minimal | `type: databricks` → `type: clickzetta`, different connection params |
+| Project profile name | 1 (`dbt_project.yml`) | Minimal | `connection_databricks` → `dbt_blueprint` |
+| `getdate()` | 1 | Minimal | → `current_date()` |
+| Macro target name | 1 | Minimal | `databricks_cluster` → `clickzetta_prod` |
+
+**Parts that need no changes**: all CTE model logic, window functions (LEAD/LAG/ROW_NUMBER), SCD Type 2 logic, `dbt_utils.generate_surrogate_key()`, data quality tests, unit tests, data contracts, `SELECT * EXCEPT (col)`, `col :: type` cast syntax, `DATEDIFF(year, start, end)` — all natively supported by Lakehouse.
+
+---
+
+## Technology Stack Comparison
+
+| | dbt-databricks (original) | dbt-clickzetta (migrated) |
+|---|---|---|
+| dbt adapter | `dbt-databricks` | `dbt-clickzetta` |
+| Compute engine | Databricks SQL Warehouse | Singdata Lakehouse |
+| Connection method | `host` + `http_path` + `token` | `instance` + `workspace` + `service` + `username` + `password` |
+| Target catalog | `catalog: dbt_blueprint` | Not needed (workspace is the catalog) |
+| SQL dialect | Databricks SQL | Lakehouse SQL (ANSI-compatible) |
+| `:: cast` | Supported | **Supported** (same on both sides, no change needed) |
+| `SELECT * EXCEPT` | Supported | **Supported** (same on both sides, no change needed) |
+| `DATEDIFF(year, s, e)` | Supported | **Supported** (three-argument form compatible) |
+| `getdate()` | Supported | Not supported → use `current_date()` |
+| Model logic (CTE/Window/JOIN) | — | **Fully identical** |
+
+---
+
+![](.topwrite/assets/anim-29-dbt-databricks-migration.svg)
+
+---
+
+## Project Background
+
+The data comes from a simulated P2P/P2B payment platform and contains 9 raw event tables:
+
+| Domain | Table | Rows | Description |
+|--------|-------|------|-------------|
+| Users | `user_events` | 196 | User registration/status change events (SCD2 source) |
+| Merchants | `merchant_events` | 30 | Merchant registration/status changes |
+| Merchants | `industry_codes` | 30 | Industry code reference table |
+| Accounts | `account_events` | 80 | Bank account opening/change events |
+| Bank cards | `card_events` | 60 | Card activation/status changes |
+| Transactions | `raw_p2p_captured_events` | 150 | Successful P2P transactions |
+| Transactions | `raw_p2p_failed_events` | 50 | Failed P2P transactions |
+| Transactions | `raw_p2b_captured_events` | 80 | Successful P2B transactions |
+| Transactions | `raw_p2b_failed_events` | 30 | Failed P2B transactions |
+
+dbt four-layer architecture:
+
+```
+staging   → intermediate → marts         → semantic
+(view)       (table)         (table)        (table)
+raw events   SCD2 logic      dim + fact     enriched
+```
+
+---
+
+## Migration Steps
+
+### Step 1: Switch the Adapter (Core Change)
+
+The original `profiles.yml` uses `type: databricks`. The migration only requires switching to `type: clickzetta`, replacing Databricks connection params `host/http_path/token` with Lakehouse params `instance/workspace/service`:
+
+```yaml
+# Original (dbt-databricks)
+connection_databricks:
+  target: dev_local
+  outputs:
+    dev_local:
+      type: databricks
+      catalog: dbt_blueprint
+      schema: default
+      host: dbc-a505ff10-0af5.cloud.databricks.com
+      http_path: /sql/1.0/warehouses/4fa4ca06332da87f
+      token: "{{ env_var('BLUEPRINT_DATABRICKS_TOKEN') }}"
+      threads: 1
+```
+
+```yaml
+# Migrated (dbt-clickzetta)
+dbt_blueprint:
+  target: dev
+  outputs:
+    dev:
+      type: clickzetta
+      instance: "{{ env_var('CZ_INSTANCE') }}"
+      workspace: "{{ env_var('CZ_WORKSPACE') }}"
+      schema: "{{ env_var('CZ_SCHEMA', 'dbt_blueprint_dev') }}"
+      vcluster: "{{ env_var('CZ_VCLUSTER', 'DEFAULT') }}"
+      username: "{{ env_var('CZ_USERNAME') }}"
+      password: "{{ env_var('CZ_PASSWORD') }}"
+      service: "{{ env_var('CZ_SERVICE') }}"
+      threads: 4
+```
+
+Also update the `profile` name in `dbt_project.yml`:
+
+```yaml
+# Original
+profile: 'connection_databricks'
+
+# Migrated
+profile: 'dbt_blueprint'
+```
+
+### Step 2: Replace `getdate()`
+
+`getdate()` is a SQL Server/Databricks function. Lakehouse uses `current_date()`:
+
+```sql
+-- Original (int_users.sql)
+{{ calculate_age('date_of_birth', 'getdate()') }} as age
+
+-- Migrated
+{{ calculate_age('date_of_birth', 'current_date()') }} as age
+```
+
+> 💡 **Tip**: The `:: cast` syntax and the three-argument `DATEDIFF(year, start, end)` form are both **natively supported** in Lakehouse — no changes needed.
+
+### Step 3: Update the Target Name in the `generate_schema_name` Macro
+
+The original macro uses `databricks_cluster` as the prod environment name. Change it to `clickzetta_prod`:
+
+```sql
+-- Original
+{%- if target.name in ["prod", "databricks_cluster"] -%}
+
+-- Migrated
+{%- if target.name in ["prod", "clickzetta_prod"] -%}
+```
+
+---
+
+## Fully Compatible Parts (No Changes Needed)
+
+The following patterns are written identically on both sides and have been tested end-to-end:
+
+```sql
+-- :: cast type conversion — supported on both sides
+userId       :: string    as user_id,
+eventTime    :: timestamp as event_time,
+amount       :: double    as transaction_amount
+
+-- SCD Type 2 window logic
+, users_scd2 as (
+    select
+        user_id, status, event_time as from_event_timestamp,
+        lead(event_time) over w as to_event_timestamp
+    from users
+    window w as (partition by user_id order by event_time)
+)
+
+-- DATEDIFF three-argument form — supported on both sides
+DATEDIFF(year, date_of_birth, current_date())
+
+-- Surrogate key (dbt_utils)
+{{ dbt_utils.generate_surrogate_key(['user_id', 'from_event_timestamp', 'status']) }}
+
+-- SELECT * EXCEPT — supported on both sides
+select * except (age)
+from users
+where age >= 18
+
+-- Data contracts (model contracts)
+config:
+  contract:
+    enforced: true
+columns:
+  - name: transaction_uid
+    data_type: string
+    data_tests:
+      - not_null
+      - unique
+
+-- Unit tests
+unit_tests:
+  - name: test__int_users__scd_logic
+    model: int_users
+    given:
+      - input: ref('stg_user_events')
+        rows: [...]
+    expect:
+      rows: [...]
+```
+
+---
+
+## dbt Test Results
+
+Tested on the AWS Singapore instance (`aws_singapore_prod`), 36/36 all passing:
+
+```
+dbt seed:  9/9  PASS  (196 users, 310 transactions, 80 accounts, 60 cards...)
+dbt run:  30/30 PASS  (9 views + 21 tables)
+dbt test: 36/36 PASS  (21 data tests + 15 unit tests)
+```
+
+Data quality tests include: not_null/unique constraints on source tables, marts-layer data contracts (`contract: enforced: true`), and column type validation on fact_transaction. Unit tests cover SCD2 logic (`int_users__scd_logic`), transaction aggregation (`int_transactions__union_all`), and the age calculation macro (`calculate_age__valid_ages`).
+
+---
+
+## Notes
+
+- **`getdate()` vs `current_date()`**: Databricks' `getdate()` returns the current timestamp. Lakehouse uses `current_date()` (date) or `current_timestamp()` (timestamp) as replacements. The `col :: type` cast syntax and the three-argument `DATEDIFF(year, s, e)` form are both **natively supported** in Lakehouse — no changes needed.
+- **Catalog hierarchy**: Databricks uses three-part naming (`catalog.schema.table`). In dbt with Lakehouse, only `schema.table` is used; the adapter handles the rest automatically. No manual changes to table references in models are needed.
+- **Seeds replacing the ingestion layer**: The original project relies on ingestion tables in the Databricks workspace. After migrating to Lakehouse, seeds CSVs are used as replacements. In production, these can be replaced with Studio data integration or CDC tasks.
+
+## Related Documentation
+
+### dbt Development Guides
+
+- [dbt Quick Start](use-dbt-dev.md): dbt-clickzetta adapter installation and configuration
+- [dbt Incremental Models](dbt-incremental.md): Incremental strategy configuration and unique_key options
+- [dbt Data Quality Checks](dbt-data-quality.md): Data tests, contract validation, and custom tests
+- [dbt Advanced Features](dbt-advanced-features.md): Macros, snapshots, semantic layer, and more
+
+### Other Migration Case Studies
+
+- [Databricks Notebook → Lakehouse Migration (Retail Medallion Pipeline)](databricks-notebook-to-studio-migration.md): PySpark Notebook → ZettaPark/Studio tasks
+- [dbt BigQuery Migration: Retail Data Warehouse Pipeline](dbt-bigquery-to-clickzetta-migration.md): BigQuery → Lakehouse adapter migration
+- [dbt Snowflake Migration: TPC-H Data Warehouse Pipeline](dbt-snowflake-to-clickzetta-migration.md): Snowflake → Lakehouse adapter migration

package/bin/skills/lakehouse-doc-en/references/dynamic-mask.md

@@ -1,12 +1,12 @@
 # Lakehouse Column-Level Security (Dynamic Masking) User Guide
 
-## 1. Overview
+## Overview
 
-Column-level Security provides fine-grained data protection capabilities through Dynamic Data Masking, which dynamically modifies the display of sensitive data (such as partial hiding or character replacement) based on user identity or role. We only store the original data and execute the masking function at runtime during data access. This document introduces how to implement this functionality through SQL interfaces.
+Column-level Security provides fine-grained data protection capabilities through Dynamic Data Masking, which dynamically modifies the display of sensitive data (such as partial hiding or character replacement) based on user identity or role. The system only stores the original data and executes the masking function at runtime during data access. This document introduces how to implement this functionality through the SQL interface.
 
-## 2. Core Syntax
+## Core Syntax
 
-### 2.1 Creating Masking Policy Functions
+### 1 Creating Masking Policy Functions
 
 Refer to the [CREATE FUNCTION (SQL)](create-sql-function.md) syntax.
 
@@ -19,12 +19,12 @@ expression_with_conditional_logic;
 
 **Key Elements**:
 
-- Must return the same data type as the original column.
-- Use security context functions:
-  - `current_user()` to get the current user (note case sensitivity).
-  - `current_roles()` to get an array of user roles.
+* Must return the same data type as the original column.
+* Use security context functions:
+  * `current_user()` to get the current user (note case sensitivity).
+  * `current_roles()` to get an array of user roles.
 
-### 2.2 Binding Policies to Columns
+### 2 Binding Policies to Columns
 
 **When Creating a Table**:
 
@@ -45,13 +45,13 @@ SET MASK schema_name.masking_function;
 
 **Adding a Column with Masking**:
 
-```sql
+```SQL
 ALTER TABLE table_name ADD COLUMN (column_name column_type MASK schema_name.masking_function);
 ```
 
-### 2.3 Removing Policy Binding
+### 3 Removing Policy Binding
 
-```sql
+```SQL
 ALTER TABLE table_name 
 CHANGE COLUMN column_name 
 UNSET MASK;
@@ -59,9 +59,9 @@ UNSET MASK;
 
 ***
 
-## 3. Use Case Examples
+## Use Case Examples
 
-### 3.1 Basic Masking
+### 1 Basic Masking
 
 **Requirement**: Display the first 6 characters of an ID card number, followed by 4 asterisks, and then the last 4 characters.
 
@@ -79,7 +79,7 @@ ALTER TABLE data CHANGE COLUMN idcard SET MASK public.idcard_masking;
 Original Value: 130183199901011234 → Masked: 130183****9010
 ```
 
-### 3.2 Dynamic Masking Based on User
+### 2 Dynamic Masking Based on User
 
 **Requirement**: Only the `UAT_TEST` user should see masked data.
 
@@ -104,7 +104,7 @@ CASE
 END;
 ```
 
-### 3.3 Dynamic Masking Based on Role
+### 3 Dynamic Masking Based on Role
 
 **Requirement**: Users with the `user_admin` role can view the full information.
 
@@ -121,9 +121,9 @@ END;
 
 ***
 
-## 4. Complete Operation Example
+## Complete Operation Example
 
-### 4.1 Initializing the Environment
+### 1 Initializing the Environment
 
 ```sql
 CREATE SCHEMA IF NOT EXISTS security_demo;
@@ -144,7 +144,7 @@ INSERT INTO security_demo.user_data VALUES ('James', '123-45-6789', '123456789')
 SELECT * FROM security_demo.user_data;
 ```
 
-### 4.2 Creating Policy Functions
+### 2 Creating Policy Functions
 
 ```sql
 -- Exemption for privileged roles
@@ -157,7 +157,7 @@ CASE
 END;
 ```
 
-### 4.3 Modifying Masking Policies
+### 3 Modifying Masking Policies
 
 ```sql
 -- Removing the previous policy
@@ -167,7 +167,7 @@ ALTER TABLE security_demo.user_data CHANGE COLUMN ssn UNSET MASK;
 ALTER TABLE security_demo.user_data CHANGE COLUMN ssn SET MASK security_demo.admin_ssn_mask;
 ```
 
-### 4.4 Verifying the Effect
+### 4 Verifying the Effect
 
 **Query by a Regular User**:
 
@@ -185,18 +185,18 @@ SELECT * FROM user_data;
 
 ***
 
-## 5. Management Notes
+## Management Notes
 
-### 5.1 Permission Control
+### 1 Permission Control
 
-- Only roles with `ALTER TABLE` permissions are allowed to modify masking policies.
-- Function creation requires `CREATE FUNCTION` permissions.
+* Only roles with `ALTER TABLE` permissions are allowed to modify masking policies.
+* Function creation requires `CREATE FUNCTION` permissions.
 
-### 5.2 Performance Recommendations
+### 2 Performance Recommendations
 
-- Avoid using complex calculations in masking functions.
-- Use conditional logic cautiously for columns with high query frequency.
+* Avoid using complex calculations in masking functions.
+* Use conditional logic cautiously for columns with high query frequency.
 
-## 6. Limitations
+## Limitations
 
-- Only one masking policy can be bound to a single column. If you want to define multiple masking rules, you can use conditional logic within a single function to apply different policies.
+* Only one masking policy can be bound to a single column. If you want to define multiple masking rules, you can use conditional logic within a single function to apply different policies.

package/bin/skills/lakehouse-doc-en/references/dynamic-table-bestpractice.md

@@ -77,7 +77,7 @@ Lakehouse currently uses a scheduling mechanism to update Dynamic Table. The fol
 
 |                                                              | Usage                                                                                                                                                                                                                                           | Advantages                                                                                                                                                                                                                                | Disadvantages                                                                                                                                                                                                                                                 |
 | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Define scheduling attributes in DDL statements               | Define the refresh interval in refreshOption, refer to the documentation <https://dev-doc.clickzetta.com/zh-CN/create-dynamic-table> for specific usage. Currently, the refresh interval is limited to one minute.                              | Simple and easy to use, can quickly set refresh options. Does not rely on any third-party tools.                                                                                                                                          | Currently, Lakehouse does not support defining strict upstream and downstream dependencies on Dynamic Table. In DDL definition, it relies on time scheduling. You can ensure upstream refresh completion before scheduling downstream through time intervals. |
+| Define scheduling attributes in DDL statements               | Define the refresh interval in refreshOption, refer to the documentation <create-dynamic-table.md> for specific usage. Currently, the refresh interval is limited to one minute.                              | Simple and easy to use, can quickly set refresh options. Does not rely on any third-party tools.                                                                                                                                          | Currently, Lakehouse does not support defining strict upstream and downstream dependencies on Dynamic Table. In DDL definition, it relies on time scheduling. You can ensure upstream refresh completion before scheduling downstream through time intervals. |
 | Define scheduling in Lakehouse Studio                        | You can configure scheduling through a visual interface in Lakehouse Studio, refer to [Task Development Scheduling Documentation](taskdevelop.md) for specific configuration methods. Currently, the refresh interval is limited to one minute. | Visual configuration, user-friendly. Supports scheduling dependency configuration to ensure downstream refresh after upstream refresh completion. Supports single-node operation monitoring, such as failure alerts, timeout alerts, etc. |                                                                                                                                                                                                                                                               |
 | Submitting Refresh Jobs Using Third-Party Scheduling Engines | By downloading the Lakehouse client command, use cron expressions to schedule Refresh tasks. Alternatively, use the Java Jdbc interface to customize the submission of Refresh commands.                                                        | You can more flexibly control job submission and configure scheduling information, with no time interval restrictions                                                                                                                     | Requires reliance on third-party scheduling. Introduces third-party scheduling systems                                                                                                                                                                        |
 

package/bin/skills/lakehouse-doc-en/references/dynamic-table-introduce.md

@@ -75,7 +75,7 @@ Lakehouse currently uses a scheduling mechanism to update Dynamic Tables. The fo
 
 |                          | Usage                                                                                                    | Advantages                                                  | Disadvantages                                                                                   |
 | ------------------------ | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------- |
-| Define scheduling properties in DDL statements             | Define the refresh interval in refreshOption, refer to the documentation https\://dev-doc.clickzetta.com/zh-CN/create-dynamic-table for specific usage. Currently, the refresh interval is limited to one minute. | Simple and easy to use, can quickly set refresh options. Does not rely on any third-party tools. | Currently, Lakehouse does not support defining strict upstream and downstream dependencies on Dynamic Tables. In DDL definitions, it relies on time scheduling. You can ensure the upstream refresh is completed before scheduling the downstream through time intervals. |
+| Define scheduling properties in DDL statements             | Define the refresh interval in refreshOption, refer to the documentation create-dynamic-table.md for specific usage. Currently, the refresh interval is limited to one minute. | Simple and easy to use, can quickly set refresh options. Does not rely on any third-party tools. | Currently, Lakehouse does not support defining strict upstream and downstream dependencies on Dynamic Tables. In DDL definitions, it relies on time scheduling. You can ensure the upstream refresh is completed before scheduling the downstream through time intervals. |
 | Define scheduling in Lakehouse Studio | You can configure scheduling through a visual interface in Lakehouse Studio, refer to the [task development scheduling documentation](taskdevelop.md) for specific configuration methods. Currently, the refresh interval is limited to one minute.                     | Visual configuration, user-friendly. Supports scheduling dependency configuration to ensure the upstream refresh is completed before refreshing the downstream. Supports single-node operation monitoring, such as failure alerts, timeout alerts, etc. |                                                                                       |
 | Submitting Refresh Jobs Using Third-Party Scheduling Engines | By downloading the Lakehouse client command, use cron expressions to schedule Refresh tasks. Alternatively, use the Java Jdbc interface to customize the submission of Refresh commands. | You can more flexibly control job submission and configure scheduling information, with no time interval restrictions | Requires reliance on third-party scheduling. Introduces third-party scheduling systems |
 

package/bin/skills/lakehouse-doc-en/references/dynamic_table_summary.md

@@ -1,6 +1,6 @@
 # Dynamic Table Overview
 
-> [Preview Release] This feature is currently in an invited preview stage. If you need to use it, please contact our technical support team for assistance.
+
 
 ## What is a Dynamic Table
 

package/bin/skills/lakehouse-doc-en/references/eco_integration/streamlit.md

@@ -99,6 +99,6 @@ if st.button('Generate Chart') and sql:
 ## Reference Resources
 
 * [Streamlit Official Documentation](https://docs.streamlit.io/library/get-started)
-* [Singdata Lakehouse Official Documentation](https://docs.singdata.com/lakehouse/)
+* [Singdata Lakehouse Official Documentation](https://singdata.com/documents)
 
 ^

package/bin/skills/lakehouse-doc-en/references/eco_integration/superset.md

@@ -22,7 +22,7 @@ Open the browser and go to http://localhost:8088.
 ### Local Installation
 
 1. Install `clickzetta-sqlalchemy`:
-   `clickzetta-sqlalchemy` needs to be installed in an environment with Python version 3.7 or above.
+   `clickzetta-sqlalchemy` needs to be installed in an environment with Python version 3.10 or above.
 Installation command (ensure the current environment does not need to use clickzetta-sqlalchemy and clickzetta-connector, uninstall them to avoid dependency conflicts):
 ```
 pip uninstall -y clickzetta-sqlalchemy clickzetta-connector && pip install clickzetta-connector -U

package/bin/skills/lakehouse-doc-en/references/ecosystem-all.md

@@ -76,7 +76,6 @@ See [JDBC Driver](jdbc-driver.md) for details.
 |--------|-------------|-----------|
 | Apache Spark | Read and write Singdata Lakehouse tables via the Spark Connector; supports the DataFrame API and spark-sql | [Spark Connector](spark-connector-summary.md) |
 | Apache Flink | Write to Singdata Lakehouse via the Flink Connector; supports CDC scenarios and append-only mode; sink tables only (write) | [Flink Connector](flink-write-connector.md) |
-| Trino | Query Singdata Lakehouse data via Trino federated queries | [Trino Integration Guide](eco_integration/trino.md) |
 
 **Two Flink Connector modes**:
 - `igs-dynamic-table`: supports CDC (insert / update / delete); the target table must have a primary key
@@ -116,6 +115,5 @@ What is your use case?
 │   └── Python applications → Python SDK
 └── Compute engine
     ├── Batch processing / ML → Spark Connector
-    ├── Stream processing / CDC → Flink Connector
-    └── Federated queries → Trino
+    └── Stream processing / CDC → Flink Connector
 ```

package/bin/skills/lakehouse-doc-en/references/ecosystem.md

@@ -0,0 +1,145 @@
+# Ecosystem
+
+Singdata Lakehouse is compatible with mainstream data integration, BI, AI, and development tools, and is deployed on seven public clouds including Alibaba Cloud, Tencent Cloud, and AWS. This document summarizes verified third-party tools and connection solutions organized by category.
+
+If the tool you need is not on the list, that does not mean it is unsupported — Lakehouse provides standard access via JDBC, MySQL protocol, and Python/Java SDKs, and any tool compatible with these protocols can connect directly. If you want to develop a new connector or integration solution based on Lakehouse, feel free to contact our partner team.
+
+## Cloud Platforms (CSP)
+
+Lakehouse is deployed on seven clouds: Alibaba Cloud, Tencent Cloud, AWS, GCP, Huawei Cloud, Baidu AI Cloud, and Volcengine. Alibaba Cloud, Tencent Cloud, and AWS provide complete dedicated documentation (including storage connections, private network connections, and permission configuration); the configuration approach is consistent across all other cloud platforms. BYOS (Bring Your Own Storage) deployment is also supported — data is stored under the user's own cloud account and does not pass through the Singdata platform. See [Supported Cloud Platforms](supported-cloud-platforms.md) and [Private Storage Overview](byos_general.md) for details.
+
+***
+
+## Data Integration
+
+The following data integration tools are compatible with Lakehouse, covering offline batch, real-time CDC, message streaming, and log collection scenarios. Lakehouse also supports [50+ data sources](data-sources.md) (MySQL, Oracle, PostgreSQL, MongoDB, Hive, MaxCompute, etc.) via Studio Data Sync for direct access without third-party tools:
+
+| Tool         | Connection         | Description                                                   | Reference                                                    |
+| ------------ | ------------------ | ------------------------------------------------------------- | ------------------------------------------------------------ |
+| Apache Kafka | Kafka Connector    | Real-time message stream writing to Lakehouse                 | [Kafka Data Source](DataSource_Kafka.md)                     |
+| AutoMQ       | Kafka Protocol     | Next-generation message queue, compatible with Kafka protocol | [AutoMQ Data Source](DataSource_AutoMQ.md)                   |
+| Airbyte      | JDBC               | Open-source ELT platform with a rich connector ecosystem      | [Airbyte Integration Guide](airbyte.md)                      |
+| DataX        | Plugin-based       | Alibaba open-source tool, suitable for batch data sync        | [DataX Integration Guide](eco_integration/datax.md)          |
+| Apache Flink | Flink Connector    | Stream processing engine for real-time writes to Lakehouse    | [Flink Connector](flink-write-connector.md)                  |
+| Apache Spark | Spark Connector    | Large-scale data reads and writes for Lakehouse tables        | [Spark Connector](spark-connector-summary.md)                |
+| Logstash     | Logstash Connector | Import log data into Lakehouse                                | [Logstash Integration Guide](Logstash.md)                    |
+| Bluepipe     | Native integration | Real-time CDC sync from Oracle to Lakehouse                   | [Bluepipe Sync Guide](bluepipe-oracle-lakehouse-datasync.md) |
+
+***
+
+## BI and Visualization
+
+The following BI tools are compatible with Lakehouse. Any BI tool supporting JDBC, ODBC, or MySQL protocol can connect directly and is not limited to the list below:
+
+| Tool            | Connection     | Description                                                      | Reference                                                        |
+| --------------- | -------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------- |
+| FineBI          | JDBC / MySQL   | Leading domestic BI tool                                         | [JDBC Connection](FineBI.md) · [MySQL Protocol](finebi-mysql.md) |
+| Tableau         | JDBC           | Suitable for complex visualizations and exploratory analysis     | [Tableau Connection Guide](tableau-connect-to-lakehouse.md)      |
+| Power BI        | MySQL Protocol | Connect via MySQL protocol                                       | [Power BI Connection Guide](PowerBI.md)                          |
+| Apache Superset | SQLAlchemy     | Open-source, suitable for self-service analytics                 | [Superset Connection Guide](eco_integration/superset.md)         |
+| Metabase        | JDBC           | Open-source, easy to deploy, suitable for small and medium teams | [Metabase Connection Guide](metabase.md)                         |
+| Apache Zeppelin | JDBC           | Notebook-style data exploration                                  | [Zeppelin Connection Guide](eco_integration/Zeppelin.md)         |
+| Rath            | JDBC           | Open-source intelligent analytics with automatic insight support | [Rath Connection Guide](eco_integration/rath.md)                 |
+| Streamlit       | Python SDK     | Rapidly build data apps for data science teams                   | [Streamlit Connection Guide](eco_integration/streamlit.md)       |
+
+***
+
+## Transformation and Compute Engines
+
+The following data transformation tools and compute engines are compatible with Lakehouse:
+
+| Tool         | Connection             | Description                                                              | Reference                                       |
+| ------------ | ---------------------- | ------------------------------------------------------------------------ | ----------------------------------------------- |
+| dbt          | dbt-clickzetta adapter | Data modeling and transformation, supports Dynamic Table materialization | [dbt Integration Guide](eco_integration/dbt.md) |
+| Apache Spark | Spark Connector        | Large-scale batch processing and machine learning                        | [Spark Connector](spark-connector-summary.md)   |
+| Apache Flink | Flink Connector        | Real-time stream processing                                              | [Flink Connector](flink-write-connector.md)     |
+
+The **dbt documentation series** covers all scenarios from quick start to migration practice: jaffle-shop experience, Snowflake/BigQuery migration, incremental processing, real-time pipelines, and data quality testing. See [DBT Practice Series](dbt-practice-series.md).
+
+***
+
+## AI and Machine Learning
+
+The following AI frameworks and platforms are compatible with Lakehouse, supporting vector storage, RAG applications, and AI workflow scenarios:
+
+| Tool            | Integration      | Description                                      | Reference                                                                  |
+| --------------- | ---------------- | ------------------------------------------------ | -------------------------------------------------------------------------- |
+| LangChain       | Python SDK       | Vector storage and RAG application development   | [LangChain Integration](langchain_integration.md)                          |
+| LlamaIndex      | Python SDK       | Data indexing and retrieval                      | [LlamaIndex Integration](llama-index.md)                                   |
+| Dify            | MCP Server / SDK | Vector database + file storage                   | [Dify Integration Overview](dify_yunqilakehouse_integration_overview.md)   |
+| N8N             | MCP Server       | Unified AI workflows                             | [N8N Integration](N8N_AI_Workflow_Integration.md)                          |
+| MindsDB         | JDBC             | ML/LLM modeling and prediction on Lakehouse data | [MindsDB Integration](JDBC_MindsDB_ML_LLM.md)                              |
+| Datus           | MCP Server       | Data engineering agent                           | [Datus Integration](Datus_Lakehouse_Integrated_Guide.md)                   |
+| Zilliz          | Joint solution   | Vector database joint solution                   | [Zilliz Joint Solution](lakehouse-zilliz-make-data-ready-for-bi-and-ai.md) |
+| Unstructured.io | SDK              | Unstructured document parsing and vectorization  | [Unstructured.io Integration](unstructured-io.md)                          |
+
+Lakehouse also provides an [MCP Server](LakehouseMCPServer.md) that can be called by any AI Agent supporting the MCP protocol.
+
+***
+
+## Programmatic Interfaces
+
+Lakehouse provides the following native programming interfaces and SDKs:
+
+| Interface      | Language   | Description                                                 | Reference                                                  |
+| -------------- | ---------- | ----------------------------------------------------------- | ---------------------------------------------------------- |
+| JDBC Driver    | Java / JVM | Standard JDBC interface, compatible with all JVM ecosystems | [JDBC Driver](JDBC-Driver.md)                              |
+| MySQL Protocol | All        | No client dependency, compatible with MySQL ecosystem       | [MySQL Protocol Connection](use-mysql-client.md)           |
+| Python SDK     | Python     | PEP 249 compatible, supports batch/real-time writes         | [Python SDK](python_reference/python-sdk-summary.md)       |
+| Java SDK       | Java       | Supports BulkLoad and real-time stream writes               | [Java SDK Batch Upload](use-java-sdk-upload-data-local.md) |
+| SQLAlchemy     | Python     | Standard Python ORM / SQL toolkit                           | [SQLAlchemy Connection](sqlalchemy.md)                     |
+| cz-cli         | Shell      | Command-line client: SQL + Studio Tasks + AI Agent          | [cz-cli Guide](cz-cli.md)                                  |
+
+***
+
+## SQL Clients and Database Management Tools
+
+These tools connect via JDBC or MySQL protocol, compatible with standard SQL operations:
+
+| Tool            | Connection     | Description                                                                                            | Reference                                                                       |
+| --------------- | -------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- |
+| DBeaver         | JDBC           | Open-source and free, community edition is sufficient, suitable for daily queries and data exploration | [DBeaver Connection Guide](eco_integration/dbeaver-lakehouse.md)                |
+| DataGrip        | JDBC           | JetBrains product with strong code completion and SQL analysis                                         | [DataGrip Connection Guide](eco_integration/datagrip-lakehouse.md)              |
+| SQL Workbench/J | JDBC           | Lightweight, basic SQL execution                                                                       | [SQL Workbench/J Connection Guide](eco_integration/sqlworkbench-j-lakehouse.md) |
+| Navicat         | MySQL Protocol | Visual management with intuitive operations                                                            | [Navicat Connection Guide](navicat-mysql.md)                                    |
+
+***
+
+## Data Lake Formats
+
+Lakehouse is **natively based on Apache Iceberg** — tables are stored in Iceberg format, supporting time travel, partition evolution, schema evolution, and cross-engine access. Delta Lake and Hudi formats are also supported via external tables:
+
+| Format         | Relationship   | Description                                                     | Reference                                                    |
+| -------------- | -------------- | --------------------------------------------------------------- | ------------------------------------------------------------ |
+| Apache Iceberg | Native format  | Underlying format for all Lakehouse tables, cross-engine access | [Spark + Iceberg Analytics](spark-lakehouse-iceberg-rest.md) |
+| Delta Lake     | External table | Open table format from the Databricks ecosystem                 | [Delta Lake External Table](delta-lake.md)                   |
+| Apache Hudi    | External table | Open table format optimized for streaming writes                | [Hudi External Table](external-hudi-table.md)                |
+
+**Federated Queries**: Query Iceberg tables in Hive, Databricks, and Snowflake OpenCatalog directly via External Catalog, without data migration. See [Federated Query](federation-query.md).
+
+***
+
+## Modern Data Stack
+
+The following solution combinations show how to build a complete data platform using Lakehouse and ecosystem tools:
+
+| Solution           | Toolchain                            | Reference                                                                   |
+| ------------------ | ------------------------------------ | --------------------------------------------------------------------------- |
+| ELT-oriented       | Airbyte → Lakehouse → dbt → Metabase | [ELT Modern Data Stack](ELTModernDataStack.md)                              |
+| Analytics-oriented | Lakehouse ← dbt → Superset           | [Analytics Modern Data Stack](analytics-modern-data-stack.md)               |
+| BI + AI            | Lakehouse + Zilliz                   | [BI + AI Joint Solution](lakehouse-zilliz-make-data-ready-for-bi-and-ai.md) |
+
+***
+
+> 💡 **Tip**: The list above contains verified and compatible third-party tools. Lakehouse provides standard access via JDBC, MySQL protocol, and Python/Java SDKs — any tool compatible with these protocols can be used directly. If the tool you need is not on the list, it can still connect normally.
+
+## Quick Navigation
+
+* **Understand product concepts**: [Key Concepts](key-concepts.md) · [Incremental Computing](incremental-computing.md)
+* **Start ingesting data**: [Data Integration](#data-integration) · [50+ Data Source Support](data-sources.md)
+* **Connect BI tools**: [BI and Visualization](#bi-and-visualization)
+* **Data modeling**: [dbt Integration Guide](eco_integration/dbt.md) · [DBT Practice Series](dbt-practice-series.md)
+* **Programmatic access**: [Programmatic Interfaces](#programmatic-interfaces)
+* **AI application development**: [AI and Machine Learning](#ai-and-machine-learning)
+
+^