@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.100

package/bin/skills/lakehouse-doc-en/references/datalake_privilege.md

@@ -0,0 +1,55 @@
+# Data Lake Permission Management
+
+Data Lake Permission Management is an important part of ensuring data security and compliance. Through proper permission management, you can ensure that users can only access and operate the data and resources they need. Permission management in a data lake mainly involves two aspects: data access permissions and function call permissions. Specifically, it includes the following aspects:
+
+1. Permission points and permission management of Volume objects
+2. Permission points and permission management of Remote Function objects
+
+## Volume Object Permissions
+
+For Volume objects, you can set the following permissions:
+
+* Permissions of the Schema to which the object belongs: CREATE / DROP
+* Permissions of the object itself: READ / WRITE / ALTER
+
+### Example 1: Granting a New User Access to a Volume
+
+Suppose you have a new user named `datalake_user`, and you want to grant them access to the workspace. First, grant the user the workspace\_user role (read-only permission):
+
+```
+GRANT ROLE workspace_user TO USER datalake_user;
+```
+
+Next, if you want to allow the `datalake_user` to read, upload data, and synchronize file metadata to the Lakehouse metadata service for Volume objects, you need to grant the following permissions:
+
+1. Authorize the use of computing resources Virtual Cluster
+2. Grant `datalake_user` READ/WRITE/ALTER permissions on the Volume
+
+The specific steps are as follows:
+
+```SQL
+GRANT USE VCLUSTER ON VCLUSTER DEFAULT TO USER datalake_user;
+```
+
+```SQL
+GRANT READ ON volume xxx TO USER datalake_user;
+GRANT WRITE ON volume xxx TO USER datalake_user;
+GRANT ALTER ON volume xxx TO USER datalake_user;
+```
+
+## Remote Function Object Permissions
+
+For Remote Function objects, you can set the following permissions:
+
+* Permissions of the Schema to which the object belongs: CREATE / DROP
+* Permissions of the object itself: USE
+
+### Example 2: Granting a User Permission to Use a Remote Function
+
+Suppose you want to grant the user `datalake_user` permission to use a Remote Function named fc\_image\_2\_text, you can execute the following command:
+
+```SQL
+GRANT USE ON FUNCTION fc_image_2_text TO USER datalake_user;
+```
+
+^

package/bin/skills/lakehouse-doc-en/references/datalake_query_ingest.md

@@ -0,0 +1,18 @@
+# Data Lake Query
+
+After creating a Volume object, you can directly query data files in CSV, Parquet, and ORC formats from the external storage path through this object without relying on any metadata system.
+
+## Prerequisites:
+
+1. If the data is in the cloud vendor's object storage, please complete the creation of the storage connection and external volume objects first.
+2. Before querying files in the volume in Lakehouse, please confirm whether you have read permission (READ) for the target volume object. If you also need to use the [PUT command](put.md) to upload files, you need to have read and write permissions. You can use `show grants to user <user_name>` to check whether the current user has the above permissions.
+
+## Usage Restrictions:
+
+* Supported external storage includes: Alibaba Cloud OSS, Tencent Cloud COS, and AWS S3.
+* Only CSV, PARQUET, and ORC file formats are supported.
+* Only gzip, zstd, and zlib compression formats are supported.
+
+^
+^
+^

package/bin/skills/lakehouse-doc-en/references/datalake_unstructure_data.md

@@ -0,0 +1,3 @@
+# AI Capabilities to Process Unstructured Data
+
+Please refer to the best practices for unstructured data processing in image analysis: [Best Practices](remotefunctiononacr.md)

package/bin/skills/lakehouse-doc-en/references/datalake_volume.md

@@ -0,0 +1,114 @@
+# Data Lake Storage Management: Volume
+
+## Overview
+
+Lakehouse Volume is an object in Singdata Lakehouse that represents an object storage location. It provides access to object storage, storage, management, and organization of files, and can be used to store and access files in various formats, including structured, semi-structured, and unstructured data. Volumes can be organized and managed under a Lakehouse Schema just like tables, views, and other objects.
+
+Using the Volume feature brings the following benefits:
+
+- **Unified Data Analysis**: Supports calling AI workloads in Singdata Lakehouse to process images, PDFs, and specially formatted unstructured data in object storage, enabling unified processing and analysis together with structured data on the platform.
+- **Unified Permission Management**: Supports using the Singdata Lakehouse platform's permission system for unified permission management of databases, tables, and files in object storage.
+- **Unified Data Governance**: Data in object storage is uniformly managed and governed by the Singdata Lakehouse platform.
+
+## Volume Types
+
+Lakehouse Volumes are classified into the following four types by **creation method**:
+
+| Type | Creation Method | Storage Location | Description |
+|---|---|---|---|
+| **User Volume** | Auto-created | Internal storage | A user-specific personal storage space, each user has one by default |
+| **Table Volume** | Auto-created | Internal storage | The file storage area associated with each table by default, permissions consistent with the table |
+| **External Volume** | `CREATE EXTERNAL VOLUME` | External storage (OSS/COS/S3) | Mounts external object storage, treating object storage as a data lake |
+| **Named Volume** | `CREATE VOLUME` | Internal or external storage | A Volume explicitly created by the user for cross-team resource sharing |
+
+> ⚠️ **Note**: Named Volume is a type of External Volume, emphasizing explicit user creation with a custom name. External Volume can mount external storage (OSS/COS/S3) or use internal storage.
+
+### Type Comparison
+
+| Feature | User Volume | Table Volume | External Volume | Named Volume |
+|---|---|---|---|---|
+| **Creation Method** | Auto-created | Auto-created (one per table) | `CREATE EXTERNAL VOLUME` | `CREATE VOLUME` |
+| **Storage Location** | Internal storage | Internal storage | External storage (OSS/COS/S3) | Internal or external storage |
+| **Permission Management** | User owns by default | Consistent with table permissions | Requires separate authorization | Requires separate authorization |
+| **Storage Cost** | Lakehouse storage billing | Lakehouse storage billing | Cloud provider storage billing | Lakehouse or cloud provider billing |
+| **Typical Scenario** | Upload local files, RAG knowledge base | Table-associated ETL files, batch import/export | Mount existing object storage | Cross-team resource sharing |
+
+## Data Operation Protocols
+
+Volumes support three address formats for referencing files in different scenarios:
+
+| Protocol Type | Address Format | Typical Scenario |
+|---|---|---|
+| External/Named Volume | `volume://volume_name/path_to_file` | Cross-team resource sharing |
+| User Volume | `volume:user://~/path_to_file` | User's personal space |
+| Table Volume | `volume:table://table_name/path_to_file` | Table-associated ETL files |
+
+### Address Format Details
+
+**External/Named Volume Format**: `volume://volume_name/upper.jar`
+
+- `volume_name`: The name of the created Volume
+- `upper.jar`: The target file name
+
+**User Volume Format**: `volume:user://~/upper.jar`
+
+- `user`: Indicates use of the User Volume protocol
+- `~`: Represents the current user, a fixed value
+- `upper.jar`: The target file name
+
+**Table Volume Format**: `volume:table://table_name/upper.jar`
+
+- `table`: Indicates use of the Table Volume protocol
+- `table_name`: The table name, filled in according to the actual situation
+- `upper.jar`: The target file name
+
+## DDL Operations
+
+Supported commands for different Volume types:
+
+| Command | Description | User Volume | Table Volume | External/Named Volume |
+|---|---|---|---|---|
+| `CREATE VOLUME` | Create a Named Volume | No | No | Yes |
+| `CREATE EXTERNAL VOLUME` | Create an External Volume | No | No | Yes |
+| `DROP VOLUME` | Drop a Volume | No | No | Yes |
+| `DESC VOLUME` | Describe Volume properties | No | No | Yes |
+| `SHOW VOLUMES` | List created Volumes | No | No | Yes |
+| `SHOW USER VOLUME DIRECTORY` | List User Volume files | Yes | No | No |
+| `SHOW TABLE VOLUME DIRECTORY` | List Table Volume files | No | Yes | No |
+| `SHOW VOLUME DIRECTORY` | List External/Named Volume files | No | No | Yes |
+| `REMOVE` | Delete files from a Volume | Yes | Yes | Yes |
+| `PUT` | Upload files to a Volume | Yes | Yes | Yes |
+| `GET` | Download files from a Volume | Yes | Yes | Yes |
+
+## Permissions
+
+| Permission | Description |
+|---|---|
+| READ METADATA | Permission to view Volume object metadata |
+| READ VOLUME | Permission to read files and directories under the Volume object. Required when viewing the file list under a Volume, reading Volume files via SQL, and downloading files via the GET command |
+| WRITE VOLUME | Permission to write data to a Volume. Required when uploading files via the PUT command and deleting files via the REMOVE command |
+| ALTER VOLUME | Permission required for the ALTER VOLUME command. For example: `ALTER VOLUME <volume_name> REFRESH` to refresh the file metadata information under the Volume (External Volume only) |
+| ALL | All permissions for the Volume object |
+
+## Cost
+
+- **External Volume**: No additional storage cost on the Lakehouse side; storage costs are charged according to the cloud provider's standard rates.
+- **Named Volume (internal storage)**: Lakehouse storage fees are charged based on the actual storage size.
+- **User Volume / Table Volume**: Lakehouse storage fees are charged based on the actual storage size.
+
+## Constraints and Limitations
+
+- The size of a single uploaded file must not exceed 5 GB.
+- JDBC driver version 1.4.4 or above is required to support local PUT/GET interfaces.
+- External Volume does not support cross-cloud-provider creation: Alibaba Cloud instances can only create OSS Connections, and Tencent Cloud instances can only create COS Connections.
+
+## Related Links
+
+- [Internal Volume Usage](internal_volume.md)
+- [External Volume Usage](external_volume.md)
+- [Alibaba Cloud OSS VOLUME Creation](oss_volume_creation.md)
+- [Tencent Cloud COS VOLUME Creation](cos_volume_creation.md)
+- [Amazon S3 VOLUME Creation](s3_volume_creation.md)
+- [Query Volume Files](structure_data_analysis.md)
+- [Import Data from Volume to Table](from_volume_to_table.md)
+- [Export Data to Volume](from_lakehouse_to_volume.md)

package/bin/skills/lakehouse-doc-en/references/datalake_volume_analytics.md

@@ -0,0 +1,37 @@
+# Data Lake Management and Analytics
+
+Singdata Lakehouse manages file storage uniformly through Volume objects, supporting both internally managed storage (User Volume, Table Volume, Named Volume) and external object storage mounts (OSS / COS / S3). It provides complete capabilities for direct querying, import/export, and permission management.
+
+For enterprises with existing data lakes, you can mount your existing object storage directly — no data migration required. Use `SELECT FROM VOLUME` or `COPY INTO` to query and process data lake files within Lakehouse.
+
+---
+
+## This Section
+
+| Page | Description |
+|------|------|
+| [Data Lake Overview](datalake_overview.md) | Volume architecture, positioning and selection between internal and external Volumes |
+| [Storage Connection](datalake-storage-connection.md) | Create and manage object storage authentication configurations (OSS / COS / S3 / Hive / Kafka) |
+| [Data Lake Volume Objects](datalake_volume_object.md) | Volume object management: creating and operating internal Volumes (User / Table / Named) and external Volumes |
+| [Data Import and Export in Object Storage](data_transfer_datalake.md) | Local file upload/download (PUT / GET), import from Volume to table (COPY INTO TABLE), export from table to Volume (COPY INTO VOLUME) |
+| [Data Lake Volume Query and Analysis](datalake_query_ingest.md) | Directly query CSV / JSON / Parquet files in Volumes, process unstructured data, invoke AI capabilities |
+| [Data Lake Permissions](datalake_privilege.md) | Permission management for Volumes and Storage Connections |
+
+---
+
+## Quick Selection Guide
+
+**I don't have cloud storage and want to temporarily store files**
+→ Use [User Volume](internal_volume.md) — created automatically by the system, no configuration needed, upload files directly with PUT
+
+**I have OSS / S3 / COS and want to query data directly**
+→ Create a [Storage Connection](datalake-storage-connection.md) → Mount an [External Volume](external_volume.md) → `SELECT FROM VOLUME`
+
+**I want to batch import object storage files into a Lakehouse table**
+→ [Import Data from Volume to Table](from_volume_to_table.md) (COPY INTO TABLE)
+
+**I want to export Lakehouse table data to object storage**
+→ [Export Data to Volume](from_lakehouse_to_volume.md) (COPY INTO VOLUME)
+
+**I want to operate Volume files with Python**
+→ [Zettapark Volume and File Operations](zettapark-volume-guide.md)

package/bin/skills/lakehouse-doc-en/references/datalake_volume_anlytics.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/datalake_volume_object.md

@@ -0,0 +1 @@
+# Volume Object Management

package/bin/skills/lakehouse-doc-en/references/datalakeanalysis.md

@@ -0,0 +1,272 @@
+# What is Volume:
+
+Lakehouse Volume is an object in Singdata Lakehouse that represents the location of cloud object storage. It provides access, storage, management, and organization of files in cloud object storage (such as Alibaba Cloud OSS, Tencent Cloud COS), and can be used to store and access files of various formats, including structured, semi-structured, and unstructured data. It can be organized and managed under the Schema of Lakehouse like tables, views, and other objects. Using the Volume feature brings the following benefits:
+
+* Unified data analysis: Supports calling AI workloads in Singdata Lakehouse to process images, PDFs, and special format unstructured data in object storage, and perform unified processing and analysis with structured data in the platform
+* Unified permission management: Supports using the permission system of Singdata Lakehouse platform to perform unified permission management on libraries, tables, and files in object storage
+* Unified data governance: The data in object storage will be managed and governed by the Singdata Lakehouse platform
+
+## Data Lake Overview
+
+[Data Lake Overview](datalake_overview.md)
+
+## How to use Volume (Alibaba Cloud):
+
+### Step0: Preparation
+
+The overall goal of this step is: Allow Lakehouse cluster to access the customer's cloud object storage OSS. To achieve this goal, two types of authentication methods can be used: AcessKey id & AcessKey Secret and role authorization. Among them:
+
+* **AK method**: Just provide the AccessKey ID and AccessKey Secret information of the account with OSS access permission, and use this information to create a storage connection (Storage Connection) object, as shown below:
+
+```
+CREATE STORAGE CONNECTION if not exists hz_conn_ak
+    TYPE oss
+    ENDPOINT = 'oss-cn-hangzhou.aliyuncs.com'
+    access_id = 'LTAI5tMmbq1Ty1xxxxxxxxx'
+    access_key = '0d7Ap1VBuFTzNg7gxxxxxxxxxxxx'
+    comments = 'OSS public endpoint';
+```
+
+* **Role authorization method** (RoleARN): You need to create a role and permission policy: The permission policy represents the access policy of the OSS service, and this policy is authorized to the created role.
+
+The following describes the **role authorization method**:
+
+#### 1. Customer side: Create a permission policy (CzUdfOssAccess) in the Alibaba Cloud RAM console:
+
+* Access the Alibaba Cloud Access Control (RAM) product console
+* Left navigation bar **Permission Management** -> **Permission Policy**, select **Create Permission Policy** in the permission control interface
+* On the **Create Permission Policy** page, select the **Script Editing** tab, and replace the following `[bucket_name_1|2|3]` with the actual OSS bucket name. Note: According to the convention of Alibaba Cloud OSS, the same bucket needs to have two Resource entries: "acs\:oss:\*:\*:\[bucket\_name\_1]" and "acs\:oss:\*:\*:\[bucket\_name\_1]/\*" must exist at the same time to achieve the authorization effect:
+
+```JSON
+{
+    "Version": "1",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+                "oss:GetObject",
+                "oss:ListObjects",
+                "oss:PutObject",
+                "oss:DeleteObject"
+            ],
+            "Resource": [
+                "acs:oss:*:*:[bucket_name_1]",
+                "acs:oss:*:*:[bucket_name_1]/*",
+                "acs:oss:*:*:[bucket_name_2]",
+                "acs:oss:*:*:[bucket_name_2]/*",
+                "acs:oss:*:*:[bucket_name_3]",
+                "acs:oss:*:*:[bucket_name_3]/*"
+            ]
+        }
+    ]
+}
+```
+
+![](.topwrite/assets/en_policy.jpeg)
+
+^
+
+#### 2. Customer side: Create role CzUDFRole in Alibaba Cloud RAM:
+
+* In the left navigation bar of the Alibaba Cloud Access Control (RAM) console, **Identity Management** -> **Role, create role**
+* On the **Create Role** page, select the type as **Alibaba Cloud Account**, fill in the custom role name (such as CzUDFRole) in **Configure Role**, select **Other Cloud Accounts** in **Select Trusted Cloud Account**, and write in: 1384322691904283, click Finish
+
+```Properties
+{
+    "Version": "1",
+    "Statement": [
+        {
+            "Action": "fc:*",
+            "Resource": "*",
+            "Effect": "Allow"
+        },
+        {
+            "Action": "ram:PassRole",
+            "Resource": "*",
+            "Effect": "Allow",
+            "Condition": {
+                "StringEquals": {
+                    "acs:Service": "fc.aliyuncs.com"
+                }
+            }
+        }
+    ]
+}
+```
+
+* After creation, click **Authorize Role**: In **Custom Policy**, authorize the just created policy (CzUdfOssAccess) to this role. On the CzUDFRole role details page, get the RoleARN information of this role: `'acs:ram::1222808864xxxxxxx:role/czudfrole'`
+
+\`\`
+
+![](.topwrite/assets/en_role.jpeg)
+
+^
+
+#### 3. Lakehouse side: Create Connection
+
+* Execute the following command in Studio or Lakehouse JDBC client:
+
+```SQL
+CREATE STORAGE CONNECTION hz_oss_conn_rolearn 
+    TYPE oss 
+    REGION = 'cn-hangzhou' 
+    ROLE_ARN = 'acs:ram::1222808864467016:role/czudfrole' 
+    ENDPOINT = 'oss-cn-hangzhou-internal.aliyuncs.com';
+```
+
+* Execute desc connection to get external ID information: In this example, the external ID is: `O0lQUogDJajHqnAQ`
+
+![](.topwrite/assets/desc_connection_1710725915229.jpeg)
+
+#### 4. Client side: In Alibaba Cloud RAM -> Role -> Trust Policy, modify the **Trust Policy** of CzUDFRole:
+
+```Python
+{
+  "Statement": [
+    {
+      "Action": "sts:AssumeRole",
+      "Condition": {
+        "StringEquals": {
+          "sts:ExternalId": "O0lQUogDJajHqnAQ"
+        }
+      },
+      "Effect": "Allow",
+      "Principal": {
+        "RAM": [
+          "acs:ram::1384322691904283:root"
+        ]
+      }
+    }
+  ],
+  "Version": "1"
+}
+```
+
+^
+
+### Step1: Creation and use of Volume objects:
+
+After the preparation work is completed, you can create a Volume object and access object storage data through it. Take Alibaba Cloud OSS as an example:
+
+#### 1. Create a Volume object
+
+```SQL
+
+CREATE EXTERNAL VOLUME sh_image_volume
+    location 'oss://sh-oss-derek/images'
+    using connection sh_oss_conn_public
+    directory = (
+        enable=true,
+        auto_refresh=true
+    )
+recursive=true;
+```
+
+#### 2. View detailed information of the created Volume object
+
+```SQL
+desc volume sh_image_volume
+```
+
+#### 3. View the files under the created Volume path
+
+```SQL
+show volume directory sh_image_volume;
+```
+
+#### 4. Display the images under the Volume path in Lakehouse Studio
+
+When developing in the Lakehouse Studio interface, you can directly click on the url -> **Preview** to open the image when you get the access url of the image through the [get\_presigned\_url](get_presigned_url.md) function:
+
+![](.topwrite/assets/presigned_url_1710729090982.jpeg)
+
+#### 5. Call UDF such as: fc\_image2text, analyze the dishes and calorie information in the OSS image
+
+For creating Remote Function, please refer to: [Remote Function Usage Document (Alibaba Cloud Version)](remotefunction.md).
+
+The function to get the file access URL is [get\_presigned\_url](get_presigned_url.md)
+
+```SQL
+set cz.sql.remote.udf.enabled = true;
+SELECT relative_path,public.fc_image2text(pre_signed_url) as content
+from
+(
+    select relative_path, get_presigned_url(
+        volume sh_image_volume,
+        relative_path, 7200
+    ) as pre_signed_url
+    from directory(volume sh_image_volume)
+);
+```
+
+^
+
+### Step2: Management of Volume objects:
+
+#### 1. Permission management:
+
+After the preparation work is completed, you need to configure the permissions for specific users to access the object storage data object (Volume). For example: The administrator user grants the user `datalake_user` the permission to create Volume and read Volume data, and also needs to grant the use of Virtual Cluster computing resources to successfully access the data object. The following SQL statement can be executed with administrator privileges:
+
+```SQL
+-- Grant datalake_user READ permission
+GRANT READ ON volume hz_image_volume TO USER datalake_user;
+GRANT USE VCLUSTER ON VCLUSTER DEFAULT TO USER datalake_user;
+```
+
+* View the authorization of datalake\_user
+
+```SQL
+SHOW grants FOR USER datalake_user;
+```
+
+* Revoke the authorization of datalake\_user
+
+```SQL
+REVOKE READ ON volume hz_image_volume FROM USER datalake_user;
+```
+
+#### 2. Other management operations
+
+View the Volume objects under the current schema
+
+```SQL
+show volumes
+```
+
+View detailed information of the specified Volume object
+
+```SQL
+desc volume sh_image_volume
+```
+
+Synchronize the file metadata information in the Volume path to Lakehouse
+
+**Note**: This operation requires specifying the directory attribute enable = true during the volume creation process
+
+```SQL
+ALTER volume sh_image_volume REFRESH;
+
+--Query the volume directory file in the local meta
+SELECT * FROM DIRECTORY (volume hz_image_volume);
+```
+
+Get the access url of the file in the Volume path
+
+```SQL
+SELECT get_presigned_url (volume sh_image_volume, relative_path, 7200) pre_signed_url
+FROM DIRECTORY (volume sh_image_volume);
+```
+
+Delete Volume
+
+```SQL
+drop volume sh_image_volume;
+```
+
+## Usage restrictions:
+
+* The target object storage of the current Volume only supports Alibaba Cloud OSS and Tencent Cloud COS
+* Downloading files from external Volume will generate download costs for the cloud account's object storage. For specific charges, please refer to the billing instructions of the cloud vendor's object storage
+
+^
+^

package/bin/skills/lakehouse-doc-en/references/dataops_practice.md

@@ -0,0 +1,104 @@
+# DataOps Data Security and Stable Production Practices
+
+## Overview
+
+DataOps, by combining best practices from agile development, data management, and data science, can significantly improve the efficiency and quality of data pipelines. In the entire DataOps data development and operation pipeline, data development management and data operations are two extremely critical links.
+
+This practice revolves around these two links, based on NYC TLC Trip Record Data, to build a pipeline for data cleaning and metric statistical analysis, producing data for BI query use. Through this practice, you can learn the following:
+
+* How to conveniently build a scheduled data processing pipeline for unified scheduling and orchestration of data synchronization, SQL tasks, etc.
+* How to use monitoring and alerting rules to promptly monitor the stability of task operations.
+* How to use data quality monitoring rules to monitor the correctness of data output, promptly alert, and prevent the spread of problematic data.
+* How to use comprehensive operational measures to quickly locate the cause of problems.
+* How to improve the efficiency of handling monitoring alerts.
+
+## Pipeline
+
+The entire pipeline includes the following five task nodes:
+
+* 1\_sync\_OSS: Batch synchronize data from the OSS end and write it to the Lakehouse
+* 1\_sync\_Kafka: Batch synchronize data from the Kafka end and write it to the Lakehouse
+* 2\_data\_cleaning: Perform simple cleaning and processing on the data synchronized from the source end, and expand it by associating dimension tables
+* 3\_metric\_calculation\_daily\_summary: Calculate key business metrics
+* 6\_BI\_simple\_query: Use SQL to simulate a simple BI query
+  ![](.topwrite/assets/image_1715856764265.png =414)
+
+## Main Steps
+
+### Task Development Scheduling and Submission for Release
+
+Use the **Task Development** module to orchestrate the entire pipeline: first, perform data synchronization import, then write SQL code for data cleaning, calculate travel data on a daily basis, compute metrics such as average order amount and average number of passengers, and finally provide data for BI report queries.
+
+#### Task Development
+
+In the task development interface, you can create synchronization tasks, SQL tasks, and other task types as needed through the new button shown in the figure below:
+
+![](.topwrite/assets/image_1740383479390.png)
+
+#### Task Debugging and Running
+
+After creating the task, you can code and configure the task. Once these operations are completed, you can use the "Run" function to test whether the code logic is correct.
+
+![](.topwrite/assets/image_1740383498418.png)
+
+#### Task Scheduling Configuration and Submission for Release
+
+After testing is completed, you can set the scheduling properties of the task in the "Configuration" option, including setting the time for scheduled scheduling, upstream dependencies of the scheduling, and other properties. Once the configuration is complete, click the "Submit" button to release the task to the production environment for regular scheduling.
+
+![](.topwrite/assets/image_1740383512199.png)
+
+### Configure Monitoring and Alerts to Monitor Task Operation
+
+After the task is released, use the **Monitoring and Alerts** module to configure monitoring rules to promptly monitor the status or metrics that need to be observed, such as task operation failures, and notify the responsible person through the notification channels configured in the alerts.
+
+#### Monitoring Rule Configuration
+
+Taking this example, configure the monitoring of the task instance's operation failure status, specify the task name, and configure the notification channels (based on notification policies) and personnel, as shown below:
+
+![](.topwrite/assets/image_1740383526310.png)
+
+In the notification policy, you can configure the interval and number of pushes, such as sending once every 10 minutes, up to 6 times, to avoid missing alerts.
+
+### Use Data Quality Monitoring to Ensure Correct Data Output
+
+In addition to monitoring the task scheduling operation status, to avoid logical issues caused by changes in upstream source data or subsequent code changes, it is also necessary to monitor the correctness and reasonableness of the data to ensure that the business side can use the data with confidence. Here, the Data Quality module's functionality will be used:
+
+#### Quality Rule Configuration
+
+In the **Data Quality** module, you can configure data quality monitoring rules to monitor the data in the tables produced by the tasks.
+
+![](.topwrite/assets/image_1740383563023.png)
+
+Taking the avg\_passenger\_count (average number of passengers) field in the dws\_tripdata\_summary table as an example, monitor that its maximum value is greater than 1 (i.e., there is at least 1 passenger on average), which can be configured as follows.
+
+![](.topwrite/assets/image_1740383576780.png)
+
+#### Quality Rule Execution
+
+Quality rules support multiple triggering methods, such as manual triggering, scheduled triggering, and periodic task triggering. Manual triggering can be used to test the correctness of rule configurations. In this instance, the quality rule is configured as "periodic task triggering," which means it is triggered by the table's output task to achieve the purpose of triggering quality checks immediately after the task is completed. It is also configured to use the "strong blocking scheduling" mode, which means that if the quality rule check fails, the instance status of the associated scheduling task will also be set to fail, thereby blocking the scheduling operation of downstream task instances and preventing the spread of data issues.
+
+![](.topwrite/assets/image_1740383594169.png)
+
+### Production Issue Operations
+
+1. After the task scheduling runs, the task instance failure is detected through monitoring, and an IM alert is received.
+
+2. Based on the content of the monitoring alert, you can directly reach the alert object.
+
+3. Investigate the reason for the task instance failure and find code changes from the task's change records.
+
+4. In operations, click the "Edit" button to view task details. In the task version, you can view the task's submission history.
+
+5. Use the version comparison feature to see specific changes. Identify that the issue was introduced by a code logic change, perform a code rollback to fix it, and resubmit.
+
+6. After resubmitting the task, go to the instance details page to perform instance operations and rerun the task instance. **Be sure to select the latest code and configuration**.
+
+7. After the instance rerun is successful, the system **will automatically trigger the rerun of the data quality rule**, and the check will pass.
+
+8. Check the task status monitoring. After the instance status is successfully restored through the rerun, **the alert will automatically close and no longer push, requiring no manual operation**.
+
+## Summary
+
+By using the built-in features of Lakehouse Studio for data development, task scheduling, task operations, data quality, and monitoring alerts, you can be immediately alerted to handle task failures. Through operation auditing to locate changes, using version management for rollback, and rerunning task instances based on the rolled-back code, you can quickly handle issues. At the same time, the linkage between monitoring alert events and task instance status ensures that successful reruns will automatically close alerts without manual intervention, reducing the alert handling process. This can significantly save time in troubleshooting and data production recovery.
+
+These features fully cover the needs of DataOps for data development management and data operations, providing solid support for "testable, monitorable, and rollbackable" data security and stable production.