@clickzetta/cz-cli-darwin-arm64 0.3.95 → 0.3.100

package/bin/skills/lakehouse-doc-en/references/Migrate_Spark_DataEngineeringBestPractices_Project_to_Lakehouse.md

@@ -0,0 +1,300 @@
+# Migrate Spark Data Engineering Best Practices Project to Singdata Lakehouse
+
+## New Solution Highlights
+
+* **Fully Cloud-Based, Focus on Data**: Migrate all offline managed environments to the cloud, eliminating non-data-related tasks such as resource preparation and system maintenance.
+* **Easy to Migrate**: Singdata Lakehouse SQL, Zettapark, and Spark SQL, pySpark are highly compatible, making code migration very low in difficulty and cost.
+* **Reduce Overall Cost and Complexity**: Singdata Lakehouse has built-in workflow services and data quality control services, eliminating the need for separate Airflow and Great Expectations. This greatly simplifies the system architecture, reducing overall cost and complexity.
+* **Openness**: It is also possible to retain Airflow and Great Expectations in the architecture, using Airflow to schedule Python tasks and continuously calling Great Expectations in Python code for data quality checks.
+* **Resource Management and Task Queue Management**: Use different Virtual Clusters to handle ETL, BI, and data quality tasks separately, achieving resource isolation and task queue management.
+
+## Introduction to Spark Data Engineering Best Practices Project
+
+### Background of Spark Data Engineering Best Practices Project
+
+Building data pipelines without sufficient guidance can be overwhelming. It is difficult to determine whether you are following best practices. If you are a data engineer looking to understand the development of your technical skills in building data pipelines but are unsure if you are following industry standards or can compare to data engineers at large tech companies, the [Spark Data Engineering Best Practices](https://github.com/josephmachado/data_engineering_best_practices/tree/main) project on Github is tailored for you. This article reviews best practices for designing data pipelines, understanding industry standards, considering data quality, and writing maintainable and operable code. After completing this project, you will understand the key components and requirements of designing data pipelines and be able to quickly familiarize yourself with any new codebase.
+
+### Scenario of Spark Data Engineering Best Practices Project
+
+Assume we extract customer and order information from upstream sources and create hourly reports of order quantities.
+
+:-: ![](.topwrite/assets/image_1737530842475.png =405)
+
+The components, tools, and environment used are as follows:
+
+* Data Storage: MinIO with Delta Table Format
+* Data Processing: Spark
+* Code Language: PySpark + Spark SQL
+* Data Quality: Great Expectations
+* Workflow: Airflow
+* Runtime Environment: Local Docker (Spark, Airflow)
+
+### Best Practices of Spark Data Engineering Best Practices Project
+
+#### Gradually Transform Data Using Standard Patterns
+
+Following established data processing workflows helps address common potential issues, and you can refer to a wealth of resources. Most industry-standard patterns follow a three-layer data warehouse architecture, as follows:
+
+1. **Raw Layer**: Store data in its original state from the upstream source. This layer sometimes involves changing data types and standardizing column names.
+2. **Transformation Layer**: Transform data from the raw layer based on selected modeling principles to form a set of tables. Common modeling principles include dimensional modeling (Kimball), Data Vault modeling, entity-relationship data models, etc.
+3. **Consumption Layer**: Combine data from the transformation layer to form datasets that directly correspond to end-user use cases. The consumption layer usually involves joining and aggregating transformation layer tables to facilitate end-user analysis of historical performance. **Business-specific metrics** are usually defined in this layer. We should ensure that **a metric is defined in only one place**.
+4. **Interface Layer [Optional]**: Data in the consumption layer usually confirms the naming/data types of the data warehouse. However, data presented to end-users should be user-friendly and easy to understand. The interface layer is usually a **view** that serves as an interface between warehouse tables and their consumers.
+
+:-: ![](.topwrite/assets/image_1737530923177.png =808)
+
+^
+
+In the above diagram, the Bronze Layer, Silver Layer, Gold Layer, and User Interface Layer correspond to the raw layer, transformation layer, consumption layer, and interface layer, respectively. We use dimensional modeling for the Silver Layer. For pipelines/transformation functions/tables, the input is referred to as "upstream," and the output is referred to as "downstream" consumers. In larger companies, multiple teams work at different levels. The data collection team may import data into the Bronze Layer, while other teams can build their own Silver Layer and Gold Layer tables based on their needs.
+
+^
+
+#### Ensure Data Validity Before Providing It to Consumers (Also Known as Data Quality Checks)
+In constructing datasets, it is crucial to clearly define expectations for the data. These expectations can be as simple as ensuring columns meet requirements or as complex as satisfying more intricate business needs. The consequences of downstream consumers using poor data can be disastrous. For example, sending incorrect data to customers or making financial expenditures based on inaccurate data. Correcting the use of poor data often requires a significant amount of time and re-running all affected processes!
+
+To prevent consumers from inadvertently using poor data, we should inspect the data before making it available for use.
+
+This project uses the [Great Expectations](https://www.startdataengineering.com/post/ensuring-data-quality-with-great-expectations/) library to define and run data checks.
+
+#### Using Idempotent Data Pipelines to Avoid Data Duplication
+
+Re-running, or re-executing data pipelines, is a common operation. When re-running data pipelines, we must ensure that the output does not contain duplicate rows. The characteristic of a system that always produces the same output given the same input is called idempotency.
+
+The following demonstrates two techniques to avoid data duplication when re-running data pipelines:
+
+1. **Overwrite based on Run ID**: Used for append-only output data. Ensure that your output data has a run ID as a partition column (e.g., the partition column in our gold table). The run ID indicates the time range to which the created data belongs. When reprocessing data, overwrite based on the given run ID.
+2. **UPSERTS based on Natural Keys**: Used when the pipeline performs inserts and updates on output data using natural keys. This is used when dimension data of existing rows needs to be updated (e.g., SCD2 adopts this method). Duplicates generated by re-running the pipeline will result in updates to existing rows (rather than creating new rows in the output). We use this method to populate the dim\_customer table.
+
+^
+
+## Migration Plan
+
+### New Architecture Based on Singdata Lakehouse
+
+:-: ![](.topwrite/assets/image_1737601829558.png =828)
+
+### Component Comparison of the Two Solutions
+
+:-: ![](.topwrite/assets/image_1737537672374.png =828)
+
+^
+
+| **Component**   | **Spark Solution**                   | **Singdata Lakehouse Service**                                |
+| --------------- | ------------------------------------ | ------------------------------------------------------------ |
+| **Data Storage** | MinIO with Delta Table Format        | Alibaba Cloud OSS (or AWS S3, etc) with Iceberg Table Format  |
+| **Data Processing** | Spark                             | Singdata Lakehouse                                           |
+| **Code Language** | PySpark + Spark SQL                 | ZettaSpark + Lakehouse SQL                                   |
+| **Data Quality** | Great Expectations                   | Singdata Lakehouse DQC (Data Quality)                        |
+| **Workflow**     | Airflow                              | Singdata Lakehouse Workflow                                  |
+| **Runtime Environment** | Local Docker (Spark, Airflow) | Cloud Serverless Service                                     |
+
+### Syntax Differences
+
+| **Function**        | **Spark Syntax**                                                 | **Singdata Lakehouse Syntax**                                        |
+| ------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------- |
+| **Create Table DDL**     | CREATE TABLE ... USING DELTA ... LOCATION '{path}/customer' | CREATE TABLE, no need for USING DELTA and LOCATION '{path}/customer'  |
+| **Create Session** | SparkSession.builder.appName                                | Session.builder.configs(connection\_parameters).create() |
+
+### Features of the New Solution
+
+* **Fully Cloud-based, Data-focused**: Migrate all offline managed environments to the cloud, eliminating non-data-related tasks such as resource preparation and system maintenance.
+* **Easy to Migrate**: Singdata Lakehouse SQL, Zettapark, and Spark SQL, pySpark are highly compatible, making code migration low in difficulty and cost.
+* **Reduced Overall Cost and Complexity**: Singdata Lakehouse has built-in workflow services and data quality control services, eliminating the need for separate Airflow and Great Expectations. This greatly simplifies the system architecture, reducing overall cost and complexity.
+* **Openness**: It is also possible to retain Airflow and Great Expectations in the architecture, using Airflow to schedule Python tasks and continuously calling Great Expectations in Python code for data quality checks.
+* **Resource Management and Task Queue Management**: Use different Virtual Clusters to handle ETL, BI, and data quality jobs separately, achieving resource isolation and task queue management.
+
+## Migration Steps
+
+### Task Code Development
+
+Navigate to [Lakehouse Studio](studio_overview.md) Development -> Tasks, and build the following task tree:
+
+^
+
+:-: ![](.topwrite/assets/image_1737533356377.png =827)
+
+^
+
+Click "+" to create the following directories:
+
+* 01\_QuickStarts\_data\_engineering\_best\_practices
+
+Click "+" to create the following SQL tasks:
+
+* 00\_setup\_env
+
+Click "+" to create the following Python tasks:
+
+* 01\_DDL
+  * create\_bronze\_tables.py
+  * create\_gold\_tables.py
+  * create\_interface\_views.py
+  * create\_silver\_tables.py
+* 02\_Pipeline
+  * sales\_mart.py
+
+Visit the [Github repository](https://github.com/yunqiqiliang/clickzetta_quickstart/tree/main/data_engineering_best_practices) of this migration project and copy the code into the corresponding tasks.
+
+^
+
+### Resource Management and Task Queue Management
+
+In the set\_env task, create three Virtual Clusters to run different tasks:
+```SQL
+-- virtual cluster
+CREATE VCLUSTER IF NOT EXISTS data_engineering_best_practices_vc_etl
+   VCLUSTER_SIZE = XSMALL
+   VCLUSTER_TYPE = GENERAL
+   AUTO_SUSPEND_IN_SECOND = 10
+   AUTO_RESUME = TRUE
+   COMMENT  'data_engineering_best_practices_vc_etl';
+
+CREATE VCLUSTER IF NOT EXISTS data_engineering_best_practices_vc_bi
+   VCLUSTER_SIZE = XSMALL
+   VCLUSTER_TYPE = ANALYTICS
+   AUTO_SUSPEND_IN_SECOND = 300
+   AUTO_RESUME = TRUE
+   COMMENT  'data_engineering_best_practices_vc_bi';
+
+CREATE VCLUSTER IF NOT EXISTS data_engineering_best_practices_vc_dqc
+   VCLUSTER_SIZE = XSMALL
+   VCLUSTER_TYPE = GENERAL
+   AUTO_SUSPEND_IN_SECOND = 10
+   AUTO_RESUME = TRUE
+   COMMENT  'data_engineering_best_practices_vc_dqc';
+```
+### Task Parameter Configuration
+
+#### 01_DDL Task Parameter Configuration
+
+Configure parameters for the following four tasks respectively:
+
+* create_bronze_tables.py
+* create_gold_tables.py
+* create_interface_views.py
+* create_silver_tables.py
+
+Open the task and click "Schedule":
+
+![](.topwrite/assets/image_1737534509318.png =474)
+
+In the pop-up page, click "Load Parameters from Code":
+
+:-: ![](.topwrite/assets/image_1737534572480.png =567)
+
+Set the value for each parameter:
+
+:-: ![](.topwrite/assets/image_1737534633567.png =562)
+
+For how to obtain parameter values, please refer to [this article](https://uat-doc.clickzetta.com/JDBC-Driver).
+
+^
+
+#### 02_Pipeline ETL Task Parameter Configuration
+
+Configure parameters for the following task, the method of parameter configuration is the same as above.
+
+* sales_mart.py
+
+###
+
+### Create Environment
+
+Run setup_env and DDL temporary tasks to create the virtual compute cluster, database Schema, and Tables required for operation.
+Create the virtual compute cluster and database Schema required for operation:
+
+:-: ![](.topwrite/assets/image_1737534798871.png =599)
+
+Create Tables:
+
+:-: ![](.topwrite/assets/image_1737534852030.png =594)
+
+###
+
+### Configure Data Quality Monitoring Rules
+
+Navigate to [Lakehouse Studio](studio_overview.md) -> Data -> Data Quality -> Quality Rules, and create the following rules:
+
+:-: ![](.topwrite/assets/image_1737534933748.png =597)
+
+Take expectation_suite_name as customer for example
+
+:-: ![](.topwrite/assets/image_1737535012940.png =594)
+
+Configure the following rules:
+
+:-: ![](.topwrite/assets/image_1737535163110.png =615)
+
+###
+
+### Workflow Scheduling
+
+Navigate to [Lakehouse Studio](studio_overview.md) -> Development -> Tasks,
+configure the scheduling for the sales_mart.py ETL task:
+
+:-: ![](.topwrite/assets/image_1737535319989.png =578)
+
+To achieve hourly report data refresh.
+
+Then click "Submit" to execute the ETL task periodically:
+
+![](.topwrite/assets/image_1737535401895.png =723)
+
+###
+
+### Task Operations and Maintenance
+
+#### ETL Task Operations and Maintenance
+
+Navigate to [Lakehouse Studio](studio_overview.md) -> Operations Monitoring -> Task Operations and Maintenance, to view task status, execution logs, instance management, etc.
+
+:-: ![](.topwrite/assets/image_1737535489873.png =712)
+
+####
+
+#### Data Quality Control
+
+Navigate to [Lakehouse Studio](studio_overview.md) -> Data -> Data Quality, where you can configure quality rules to monitor whether the output of data tables is normal.
+
+View validation objects:
+
+:-: ![](.topwrite/assets/image_1737535653201.png =729)
+
+View validation results:
+
+:-: ![](.topwrite/assets/image_1737535687027.png =720)
+
+^
+^
+
+#### Monitoring and Alert Configuration
+
+Next, you can navigate to Operations Monitoring -> Monitoring and Alerts, add alert rules to achieve real-time alerts for ETL tasks and data quality control.
+
+^
+
+**Using Built-in Monitoring and Alert Rules**:
+
+Within the system, the following general monitoring and alert rules are built-in. When data quality validation fails, alerts such as emails, SMS, and phone calls will be triggered according to the specific configuration in the rules.
+
+:-: ![](.topwrite/assets/image_1737600310887.png =720)
+
+^
+
+**Using Custom Monitoring and Alert Rules**:
+
+The above built-in monitoring and alert rules apply to the validation results of all global quality monitoring rules. If you want to further refine the control of the monitoring scope and alert strategy, you can also create custom rules. As shown in the figure below, select the monitoring item "Quality Rule Validation Failure" and provide filtering conditions for the restricted scope. For detailed guidelines on monitoring and alert configuration, please refer to the[Monitoring and Alerting](<monitor_and_alert.md>)
+user guide document.
+
+:-: ![](.topwrite/assets/image_1737600502785.png =720)
+
+^
+
+## Reference Materials
+
+* [Task Development and Scheduling](taskdevelop.md)
+* [Python Task Development and Scheduling, Workflow Orchestration](python_task.md)
+* [Zettapark Quick Start](zettaparkquickstart.md)
+* [Task Parameters](task_param.md)
+* [Monitoring and Alerting](<monitor_and_alert.md>)
+
+^

package/bin/skills/lakehouse-doc-en/references/ModelMarket.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/ModernDataStackWithEcosystemTools.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/MonitoringItemSpecification.md

@@ -0,0 +1,44 @@
+# Monitoring Item Specification
+
+This document describes the specific definitions and scope of "Event" and "Metric" monitoring items available when configuring monitoring rules.
+
+## Event-Based Monitoring
+
+| Event Name               | Target Objects                                                           | Scope Description                                                                                      |
+| ------------------ | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
+| Scheduled Task Instance Run Failure         | Includes: Periodically scheduled task instances, backfill task instances, and manual reruns of scheduled instances. Excludes: Manually run temporary instances, real-time sync task instances. | An alert event is generated when a scheduled task instance fails due to various reasons (code logic errors, timeout, system exceptions, etc.). If the scheduled task is configured with automatic retry on error, no alert events are generated during the automatic retry process; only one alert is generated upon final failure. |
+| Scheduled Task Instance Completion Time       | Includes: Periodically scheduled task instances, manually rerun scheduled task instances, etc. Excludes: Manually run temporary instances, backfill task instances.     | An alert event is generated when the completion time of a scheduled task exceeds the expected time point (does not finish running by the specified time).                                                  |
+| Scheduled Task Instance Scheduling Duration (Including Wait Time)  | Includes: Periodically scheduled task instances. Excludes: Manually run temporary instances, backfill task instances, manually rerun scheduled task instances, etc.     | An alert event is generated when the total duration from the planned time to the end time of a scheduled task instance exceeds the threshold. This includes the time the task instance waits for upstream tasks to complete.              |
+| Scheduled Task Run Time Delay         | Includes: Periodically scheduled task instances. Excludes: Manually run temporary instances, backfill task instances, manually rerun scheduled task instances, etc.     | An alert event is generated when the actual start time of a scheduled task instance (when it enters running state) is delayed beyond the threshold compared to the planned time.                                         |
+| Task Instance Execution Duration (Excluding Wait Time)   | All task instances. Includes periodically scheduled task instances, manually run temporary instances, backfill task instances, manually rerun scheduled task instances, etc.   | An alert event is generated when the pure execution time of the instance (excluding the time waiting for upstream completion) exceeds the threshold.                                                      |
+| Dynamic Table Refresh Timeout            | Dynamic Tables                                             | An alert event is generated when a dynamic table refresh operation exceeds the set timeout period.                                                                  |
+| Streaming SQL Task Execution Failure        | Streaming SQL (Continuous Job) tasks                                        | An alert event is generated when the running status of a streaming SQL task changes from "Running" to "Failed".                                                        |
+| Quality Rule Validation Failure           | Quality Rules + Target Table: Select tables configured with quality rules. When a rule validation fails on that table, an alert history is generated.                    | An alert event is generated when the execution result of a data quality validation rule is "Failed".                                                                |
+| Quality Rule Validation Timeout           | Quality Rules                                                           | An alert event is generated when the execution result of a data quality validation rule is "Timeout".                                                                |
+| Multi-Table Real-Time Sync Task Run Failure       | Multi-table real-time sync tasks                                                       | An alert event is generated when the status of a multi-table real-time sync task changes from "Running" to "Failed".                                                         |
+| Multi-Table Real-Time Sync Task Table Enters Blacklist      | Multi-table real-time sync tasks                                                       | An alert event is generated when a table in the task is added to the blacklist due to consecutive sync failures.                                                              |
+| Multi-Table Real-Time Sync Task Single Table Full Data Sync Exception | Multi-table real-time sync tasks                                                       | An alert event is generated when an exception occurs during the full data synchronization of a single table in a multi-table real-time sync task.                                                         |
+| Multi-Table Real-Time Sync Task Single Table Incremental Data Sync Exception | Multi-table real-time sync tasks                                                       | An alert event is generated when an exception occurs during the incremental data synchronization of a single table in a multi-table real-time sync task.                                                         |
+| Full & Incremental Integrated Sync Single Table Full Data Sync Exception  | Full & incremental integrated sync tasks                                                      | An alert event is generated when an exception occurs during the full data synchronization of a single table in a full & incremental integrated sync task.                                                        |
+| Full & Incremental Integrated Sync Single Table Incremental Data Sync Exception  | Full & incremental integrated sync tasks                                                      | An alert event is generated when an exception occurs during the incremental data synchronization of a single table in a full & incremental integrated sync task.                                                        |
+| Full & Incremental Integrated Sync Task Run Failure      | Full & incremental integrated sync tasks                                                      | An alert event is generated when the status of a full & incremental integrated sync task changes from "Running" to "Failed".                                                        |
+| Full & Incremental Integrated Sync Task Target Table Schema Change Failure | Full & incremental integrated sync tasks                                                      | An alert event is generated when automatic schema evolution of the target table encounters an error.                                                 |
+
+## Metric-Based Monitoring
+
+| Metric Name               | Target Objects        | Scope Description                                                         |
+| ------------------ | ----------- | ------------------------------------------------------------ |
+| Dynamic Table Refresh Failure            | Dynamic Tables         | Number of dynamic table refresh operation failures.                                                  |
+| Full & Incremental Integrated Sync Task Delay        | Full & incremental integrated real-time sync tasks | Time delay of data synchronization.                                                    |
+| Full & Incremental Integrated Sync Task Single Table Sync Failure    | Full & incremental integrated real-time sync tasks | Statistics on single table synchronization failure count.                                                   |
+| Multi-Table Real-Time Sync Task Delay         | Multi-table real-time sync tasks    | Overall delay of multi-table real-time synchronization.                                                  |
+| Multi-Table Real-Time Sync Task Job Failover | Multi-table real-time sync tasks    | Number of failover occurrences in multi-table sync tasks.                                               |
+| Multi-Table Real-Time Sync Task Read Position Delay     | Multi-table real-time sync tasks    | Delay of data read position relative to the source.                                               |
+| Multi-Table Real-Time Sync Task Source Database Read Delay  | Multi-table real-time sync tasks    | Difference between source database data generation rate and synchronization rate.                                           |
+| Multi-Table Real-Time Sync Task Data Sync Status Exception   | Multi-table real-time sync tasks    | Monitoring of abnormal conditions in data synchronization status.                                                |
+| Queued Jobs Count              | Single SQL Job     | Sampled every 5 seconds. Counts all SQL jobs whose status is "queued" at each sampling point.                               |
+| Average Job Queue Time           | Single SQL Job     | Sampled every 5 seconds. Calculates the TP90 job queue time within each 5-second window.                                    |
+| General-Purpose / Integration Vcluster Load      | VCluster        | At each sampling point, computes the ratio of compute resources currently processing tasks to the maximum compute resources supported by that vcluster size.                 |
+| Analytics Vcluster Load          | VCluster        | At each sampling point, computes the ratio of current job concurrency to the vcluster's maximum supported concurrency (calculated as max replicas x max concurrency per replica). |
+
+^

package/bin/skills/lakehouse-doc-en/references/N8N_AI_Workflow_Integration.md

@@ -0,0 +1 @@
+# N8N AI Workflow Integration

package/bin/skills/lakehouse-doc-en/references/N8N_Integrated_with_LakehouseMCPServer.md

@@ -0,0 +1,128 @@
+# Integrating Lakehouse MCP Server in N8N
+
+N8N's Agent node supports directly integrating [tools](https://modelcontextprotocol.io/docs/concepts/tools) from external [MCP](https://modelcontextprotocol.io/introduction) servers. This provides a new way to integrate Lakehouse with your N8N AI workflows.
+Lakehouse MCP Server achieves integration with N8N through this approach, allowing the rich tools from Lakehouse MCP Server to be easily invoked in N8N's Agent node.
+
+First, you need to deploy a [Lakehouse MCP Server that supports the SSE protocol](lakehousemcpserver_intro.md).
+
+## Adding MCP Client in N8N Workflow AI Agent Node
+
+The N8N MCP Client Tool node is a [Model Context Protocol (MCP)](https://modelcontextprotocol.io/introduction) client that allows you to use tools exposed by external MCP servers. You can connect the MCP Client Tool node to your model to invoke external tools using the N8N agent. **Currently, N8N only supports the SSE protocol and does not yet support communication with MCP Server via HTTP protocol**.
+
+:-: ![](.topwrite/assets/image_1756887369109.png =795)
+
+In the Agent node's System Message, enter:
+You are a Singdata Lakehouse senior product expert, skilled at using MCP Server Tools to help users answer questions and solve problems, avoiding fabrication. For questions related to Singdata Lakehouse, if there is no suitable MCP Tool, you should first query the product knowledge base through the `get_product_knowledge` MCP Tool to obtain knowledge, and then use the appropriate Tool to execute and obtain results.
+
+## Configuring the AI Agent Node's MCP
+
+Configure the node using the following parameters.
+
+* **SSE Endpoint**: The SSE endpoint of the MCP server you want to connect to.
+
+* **Authentication**: The authentication method used to authenticate with the MCP server. Select **"None"** to attempt connection without authentication.
+
+* **Tools to Include**: Select which tools to expose to the AI agent:
+
+  * **All**: Expose all tools provided by the MCP server.
+  * **Selected**: Activate the **Tools to Include** parameter, where you can select which tools to expose to the AI agent.
+  * **All Except**: Activate the **Tools to Exclude** parameter, where you can select tools to avoid sharing with the AI agent. The AI agent will have access to all unselected MCP server tools.
+
+:-: ![](.topwrite/assets/image_1756887446000.png =614)
+
+Select **Selected** to see the Tools list:
+
+\:-:
+![](.topwrite/assets/image_1756887497226.png =611)
+
+## Testing the Agent Flow
+
+Enter your question in the dialog to operate Lakehouse:
+
+**Question 1: What Lakehouse environments do I have?**
+
+:-: ![](.topwrite/assets/image_1756887670661.png =629)
+
+^
+
+**Answer 1**:
+
+You currently have the following 4 Lakehouse environment configurations:
+
+1. **aliyun\_shanghai\_prod**
+
+   * **Workspace**: quick\_start
+   * **Schema**: mcp\_demo
+   * **Virtual Cluster**: default\_ap
+   * **Status**: Connected (currently active and default)
+
+2. **tencent\_shanghai\_prod**
+
+   * **Workspace**: quick\_start
+   * **Schema**: mcp\_demo
+   * **Virtual Cluster**: default\_ap
+   * **Status**: Connected
+
+3. **tencent\_beijing\_prod**
+
+   * **Workspace**: quick\_start
+   * **Schema**: public
+   * **Virtual Cluster**: default\_ap
+   * **Status**: Connected
+
+4. **tencent\_guangzhou\_prod**
+
+   * **Workspace**: quick\_start
+   * **Schema**: public
+   * **Virtual Cluster**: default\_ap
+   * **Status**: Connected
+
+The currently active environment is **aliyun\_shanghai\_prod**. If you need to switch environments, please let me know your specific requirements!
+
+**Question 2: Help me switch to tencent_shanghai_prod**
+
+**Answer 2**:
+
+Successfully switched the Lakehouse environment to **tencent\_shanghai\_prod**. Here is the verification information:
+
+* **New Connection Status**:
+
+  * **Service Endpoint**: `ap-shanghai-tencentcloud.api.singdata.com`
+  * **Workspace**: `quick_start`
+  * **Schema**: `mcp_demo`
+  * **Instance ID**: `270738` (Internal identifier)
+
+* **Switch Details**:
+
+  * Physical connection has been re-established with no configuration drift.
+  * Processing time: 2.83 seconds.
+
+If you need further verification of the current context, you can execute `get_current_context`.
+Please continue to let me know your subsequent operation needs!
+
+**Question 3: What is the current context?**
+
+**Answer 3**:
+
+Current context information is as follows:
+
+* **Instance ID**: `270738`
+* **Schema**: `mcp_demo`
+* **Username**: `qiliang` (ID: `2162115`)
+* **Virtual Cluster**: `default_ap`
+* **Workspace**: `quick_start`
+* **Connection Time**: `2025-09-03 16:25:45` (UTC+8)
+
+The current environment is **tencent\_shanghai\_prod** (Tencent Cloud Shanghai production environment), with normal status. Please feel free to let me know if you need any other operations!
+
+^
+
+Similarly, you can also enter questions in the dialog for Lakehouse-related knowledge Q&A:
+
+^
+
+:-: ![](.topwrite/assets/image_1756888912810.png =813)
+
+^
+
+As you can see, Lakehouse can be conveniently accessed through N8N workflows!

package/bin/skills/lakehouse-doc-en/references/Notebook.md

@@ -0,0 +1,98 @@
+# Notebook
+
+Notebook is a web-based code editor that can be used for writing code and viewing the results of interactive data analysis. This article introduces the basics of how to use Notebook within the product, including how to navigate the toolbar and how to perform various cell operations.
+
+^
+
+## Creating a Notebook
+
+Click on the sidebar: Analysis -> New Notebook
+
+![](.topwrite/assets/image_1760000673479.png)
+
+A: Left sidebar Directory Tree and Data Tree
+B: Notebook Function Operations Area
+C: Notebook Cells
+
+^
+
+## Left Sidebar Directory Tree and Data Tree
+
+* **Notebook**: The Directory Tree is used to manage the task code of Notebooks under the current workspace in an orderly manner. Creating folders is currently not supported.
+* **Data**: Below the Notebook section, the data directory tree for all data in the user's Region is displayed, presented hierarchically as Workspace -> Schema -> Table/View.
+
+^
+
+## Function Operations Area
+
+| No. | Icon                                                                                                    | Description                                                                                                                                                                                                                                                                                                                                        |
+| -- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| 1  | ![](.topwrite/assets/image_1760000292250.png)                                                         | The global parameters of the current Notebook. If custom parameters are referenced below in the format `${custom_parameter}`, users need to input the parameter values here.Note: Parameters are global. If the same custom parameter name is referenced in different cells, it will be replaced with the same parameter value during execution. |
+| 2  | ![](.topwrite/assets/image_1760000315851.png =162) ![](.topwrite/assets/image_1760000384636.png =194) | Run All Cells or Stop Execution." The name of this button changes dynamically based on the notebook's status. After the user clicks "Run All", the cells will be executed sequentially in order.                                                                                                                                                 |
+| 3  | ![](.topwrite/assets/image_1760000394807.png =191)                                                    | Cluster information for executing SQL cells.                                                                                                                                                                                                                                                                                                     |
+| 4  | ![](.topwrite/assets/image_1760000408009.png =133)                                                    | More Actions: Supports renaming, duplicating, and deleting the Notebook.                                                                                                                                                                                                                                                                         |
+
+^
+
+## Notebook Cells
+
+### Creating a Cell
+
+After creating a new Notebook task, click on different unit types in the operations area to create specific unit cells.
+![](.topwrite/assets/image_1760000698966.png)
+
+^
+
+### Cell Types
+
+Notebook contains two types of cell collections: Code cells and Markdown cells.
+
+^
+
+#### Code Cells
Code cells contain executable code, supporting:
+
+* **SQL Cell**: Executes Singdata SQL queries.
+  If an SQL cell is selected, the user needs to choose the specific Schema information of the Singdata data they want to access. After filtering, in this unit cell, users can directly input table names, and the system will auto-complete the specific three-part identifier code based on the Schema and current workspace information.
+* **Python Cell**: Executes Python data analysis code.
+* **Markdown Cells**: Markdown cells contain Markdown code that is rendered as text and graphics. Use Markdown to document or explain your code.
+  You can add or delete any type of cell in the Notebook to organize your work.
+
+**Limitation**: The output size of a cell is limited to a maximum of 10MB.
+
+^
+
+### Cell Operations
+
+| No. | Icon                                                                                                    | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| -- | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| 1  | ![](.topwrite/assets/image_1760000895389.png =182) | Cell Types.Different cell types support different languages.                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| 2  | ![](.topwrite/assets/image_1760000946839.png =227) | Cell Name. When a user creates a new cell, the system automatically assigns a default cell name. Double-clicking this area allows the user to rename the cell.Notebook enables users to directly reference a cell's output values in Python scripts. This requires users to specify the exact name of the cell containing the output information for data positioning.Cell names only support English letters (both uppercase and lowercase), numbers, and underscores (\_). |
+|    |                                                    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| 3  | ![](.topwrite/assets/image_1760000971875.png =205) | Expand/Collapse Cell Code and Display Results.When the code content in the current Notebook is too long, users can click the arrow on the left side to separately expand or collapse the code and result areas.                                                                                                                                                                                                                                                              |
+| 4  | ![](.topwrite/assets/image_1760001022354.png =220) | Moving Cells.  The execution in a Notebook follows a strict sequential order. If users need to adjust the position of a cell, they can hover the mouse over the cell, select the move handle area, and then drag the cell up or down to reposition it.                                                                                                                                                                                                                       |
+| 5  | ![](.topwrite/assets/image_1760001035621.png =253) | Quickly Add Cells.When hovering over the area above or below the central region of the current cell, new cells can be quickly added above or below the current position.                                                                                                                                                                                                                                                                                                     |
+| 6  | ![](.topwrite/assets/image_1760001050583.png =205) | Run cell.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| 7  | ![](.topwrite/assets/image_1760001074213.png =239) | More Cell Operations. Rename: Click to assign a new name to the cell.Clear Output: Clears the execution results of the current cell. If other cells depend on the output of this cell, their execution may fail. Delete: Deletes the cell.                                                                                                                                                                                                                                      |
+
+##
+
+## Running Cells
+
+### Basic Operations
+
+To run a code cell, click the run button in the upper left corner of the code cell, or click "Run All" in the upper left corner of the entire Notebook area.
+
+* **Run Single**: Only runs the current cell.
+  If the current cell references the output of other cells, please ensure those cells have been successfully run. The execution time should not exceed 30 minutes. If it exceeds 30 minutes, pod clearing scenarios may occur.
+* **Run All**: After clicking, cells are executed sequentially from top to bottom. If one cell fails to execute or is paused, all cells below it will be blocked from execution, meaning the status of all cells below will change to "Paused".
+
+### Context Communication Between Cells
+
+Singdata Notebook uses a communication mechanism based on cell naming. Each cell has a unique name (e.g., cell1, cell2, etc.), which can be directly referenced in other cells.
+
+* **SQL Cell Output**: For multi-segment SQL, only the last segment serves as the output of that cell.
+* **Direct Referencing**: The result of an SQL cell can be directly used as a pandas DataFrame.
+* **No Conversion Needed**: No additional operations like `to_pandas()` are required.
+* **Referencing by Name**: Access results directly via the cell name.
+
+^