@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/approval-list.md

@@ -0,0 +1,61 @@
+# Approving Work Orders
+
+Based on the approval flow settings for each resource, as an object permission owner, workspace or catalog owner, or instance administrator, you may receive relevant permission approval notifications and need to approve work orders. This document describes how to manually approve work orders.
+
+## Constraints and Limitations
+
+* Only the approvers designated as approval nodes in the approval flow can approve work orders.
+* Only instance administrators can access and modify approval flows.
+
+## Approval
+
+### My Pending Tasks
+
+When a user applies for permissions, the system automatically creates an approval work order based on the target object's approval flow for review. The applicant can view their application information and approval status under **Approval > Application**.
+
+![](.topwrite/assets/image_1736245807432.png)
+
+When an approver receives an approval notification, they can see pending work orders under **Approval > Approval**. Specific operations are as follows:
+
+* Click **Approval > Approval** in the left menu to enter the "Pending My Approval" tab.
+
+* The approval page supports both card view and list view.
+  ![](.topwrite/assets/image_1736245843874.png)
+
+* Filter conditions: Set keywords or work order ID, applicant, data object type, application time, sort order, and other search criteria to query pending approval work orders that meet the conditions. When multiple search conditions are set, the intersection of all conditions is used for the query.
+
+**1. View Work Order**
+
+*   **Card view**: Click a specific work order card on the left to view its application details on the right.
+*   **List view**: Click the ID column in the list to open a pop-up with the current approval work order details.
+
+Details include:
+
+* Application object, requested permission points, authorized user, authorization time, application reason, approval history
+
+**2. Approve Work Order**
+
+On the "Pending My Approval" page, approvers can perform the following approval operations:
+
+*   **Approve**
+    *   Click the **Approve** button in the **Operation** column of a work order in the list, or the **Approve** button on the work order details page, or the **Approve** button in the upper right corner of the card view. In the pop-up window, click **Confirm** to approve the application.
+    *   Select one or more items, then click the **Approve** button at the bottom of the list. In the pop-up window, click **Confirm** to approve applications in bulk.
+
+*   **Reject**
+    *   Click the **Reject** button in the **Operation** column of a work order in the list, or the **Reject** button on the work order details page, or the **Reject** button in the upper right corner of the card view. In the pop-up window, enter the rejection reason and click **Confirm** to reject the application.
+    *   Select one or more items, then click the **Reject** button at the bottom of the list. In the pop-up window, click **Confirm** to reject applications in bulk.
+
+*   **Add Signer**: Indicates that after the current approver approves, additional signers are still needed to participate in approval.
+    *   Click the **Add Signer** button in the **Operation** column of a work order in the list, or the **Add Signer** button on the work order details page, or the **Add Signer** button in the upper right corner of the card view. In the **Add Signer** pop-up window, configure the signer information and click **Confirm** to complete the add-signer approval for the application.
+    *   Select one or more items, then click the **Add Signer** button at the bottom of the list. In the pop-up window, configure the signer information and click **Confirm** to complete batch add-signer.
+
+**Note:**
+
+*   When the approver clicks the add-signer operation, the system automatically treats the current approver as having performed the approval;
+*   When a work order contains multiple approval items, the approval operation applies to all items, meaning you can only approve all or reject all at once.
+
+### Completed
+
+Click the "Completed" tab under the Approval function to view all approval work orders that you have processed and those awaiting your action within the current instance.
+
+![](https://tq2pllvokz.feishu.cn/space/api/box/stream/download/asynccode/?code=YmU2MjVkZTA4OWU3NDlkYWZmOGI1ZTZlOTFlZmY4MWVfZHAwaE94eWpkUzB2S2NWSGppcFd3NWsxekw3QmlFN3ZfVG9rZW46UWl5QWJ1SmZnb0pkZ1h4NDJUWmNGMUpJbjZmXzE3Mjk4MjU3MDQ6MTcyOTgyOTMwNF9WNA)

package/bin/skills/lakehouse-doc-en/references/array_function.md

@@ -0,0 +1 @@
+# ARRAY Functions

package/bin/skills/lakehouse-doc-en/references/array_size.md

@@ -0,0 +1,31 @@
+### SIZE
+
+####  Description
+1. The `SIZE(array)` function is used to calculate and return the number of elements in the array.
+2. The `SIZE(map)` function is used to calculate and return the number of key-value pairs in the map.
+
+#### Syntax
+```
+SIZE(array)
+SIZE(map)
+```
+#### Parameter Description
+- `array`: Array type `array<T>`, representing the array whose elements need to be counted.
+- `map`: Map type `map<K, V>`, representing the map whose key-value pairs need to be counted.
+
+#### Return Type
+Integer (int)
+
+#### Usage Example
+1. Count the number of elements in an array:
+```
+SELECT SIZE(array(1, 2, 3)); -- Result: 3
+```
+
+2. Calculate the number of key-value pairs in the mapping:
+```
+SELECT SIZE(map("a", 1, "b", 2)); -- Result: 2
+```
+#### Notes
+- When the passed parameter is not an array or map, the function will return an error.
+- If the array or map is empty, the function will return 0.

package/bin/skills/lakehouse-doc-en/references/authentication.md

@@ -0,0 +1,53 @@
+# Identity Verification
+
+Identity verification is the process of confirming your real identity to validate the ownership of your Singdata account. It is divided into two types: individual verification and enterprise verification.
+
+| Type         | Individual Verification     | Enterprise Verification                  |
+| ------------ | --------------------------- | ---------------------------------------- |
+| Applicable to | Individual                 | Enterprise, sole proprietor, government, public institution, school, organization, etc. |
+| Account Ownership | Individual             | Enterprise                              |
+| Verification Method | Personal ID + facial recognition | Business license + legal representative facial recognition |
+| Estimated Time | 10 minutes                | 10 minutes                               |
+
+## Permissions
+
+For accounts with multiple users, identity verification can only be performed by users with the account administrator role. The first user who creates the account is granted the account administrator role by default. Non-administrator users can view the verification status but cannot perform verification.
+
+## How to Check Verification Status
+
+Log in to the Singdata platform and go to the account home page to view the current identity verification status:
+![](.topwrite/assets/image_1747638304936.png)
+
+## Enterprise Identity Verification
+
+Go to Account Home -> Select "Verification Upgrade" -> Select "Enterprise Verification" to enter the enterprise identity verification page. The verification process consists of two steps: fill in enterprise basic information -> legal representative completes facial recognition, as shown below:
+![](.topwrite/assets/image_1747638456789.png)
+![](.topwrite/assets/image_1747638493466.png)
+![](.topwrite/assets/image_1747638545329.png)
+
+* Required enterprise information includes: business license, company name, unified social credit code, legal representative name, legal representative ID number, bank name, bank account number.
+* Business license must be in JPG/JPEG/PNG/PDF format and file size less than 10 MB.
+* Company name, unified social credit code, and legal representative name must match the information on the business license.
+
+After confirming the information is correct, click "Next". When the enterprise information verification passes, the legal representative facial recognition step begins. The system generates a facial recognition QR code. The legal representative scans the code with a mobile device and completes facial recognition on the mobile device, as shown below:
+![](.topwrite/assets/image_1747639041666.png)
+
+## Individual Identity Verification
+
+Go to Account Home -> Select "Verification Upgrade" -> Select "Individual Verification" to enter the individual identity verification page. The verification process consists of two steps: fill in personal identity information -> complete individual facial recognition, as shown below:
+![](.topwrite/assets/image_1747639086741.png)
+![](.topwrite/assets/image_1747639102052.png)
+![](.topwrite/assets/image_1747639142744.png)
+
+* Required personal information includes: name, ID number.
+
+After confirming the name and ID number match and are correct, click "Next" to enter the individual facial recognition step. Scan the QR code with a mobile device and complete facial verification on the mobile device, as shown below:
+![](.topwrite/assets/image_1747639179450.png)
+
+## Additional Notes
+
+1. Facial recognition can be completed using any mobile device that supports QR code scanning. Alipay or WeChat is recommended.
+
+2. Each account is limited to 3 verification attempts per day. If you encounter issues during verification, please contact 400-6767-862.
+
+3. Identity verification information does not support online changes, including: individual verified accounts cannot be changed to enterprise verified accounts; enterprise information for completed enterprise verification cannot be changed. If needed, please contact your business representative.

package/bin/skills/lakehouse-doc-en/references/authority-management.md

@@ -0,0 +1,157 @@
+# User Management
+
+Users are the identity principals that perform operations in Singdata Lakehouse. Lakehouse uses a two-tier user model: instance-level user management and workspace-level user management.
+
+## User Tiers
+
+### Instance-Level Users
+
+Instance-level users are created in the global account and represent independent identities in the system. Creating, deleting, enabling, and disabling instance-level users is done in the **Account Console → User Management** page.
+
+- Multiple users can be created under one account; usernames must be unique.
+- Instance-level users are granted the `instance_user` role by default and have no data or feature permissions.
+- A user must be added to a workspace before they can be granted roles and permissions within that workspace.
+
+### Workspace-Level Users
+
+A workspace-level user is an instance-level user that has been added to a specific workspace. Only users who have been added to a workspace can be granted roles and permissions within it.
+
+- Use `CREATE USER` to add an instance user to a workspace.
+- Use `DROP USER` to remove a user from a workspace.
+- A user can be a member of multiple workspaces simultaneously.
+
+## User Types
+
+| Type | Description | Login Method |
+|------|-------------|-------------|
+| Regular User | Actual personnel within the organization; performs daily data queries, analysis, and management | Web console login, JDBC connection |
+| Service User | A special user for automated processes or system-level calls | JDBC connection only; cannot log in via the web console |
+
+Service users include **system service users** (created by default when Lakehouse is initialized) and **custom service users** (created by users themselves).
+
+---
+
+## In This Chapter
+
+| Page | Description |
+|------|-------------|
+| [CREATE USER](create-user.md) | Add an instance user to the current workspace; set default cluster and schema |
+| [ALTER USER](alter-user.md) | Modify a user's default cluster, default schema, and other properties |
+| [DROP USER](drop-user.md) | Remove a user from the current workspace (does not delete the instance user account) |
+| [SHOW USERS](show-users.md) | List all users in the current workspace |
+
+---
+
+## User Lifecycle Management
+
+### 1. Create an Instance User
+
+Create a new user in the "User Management" page of the account console, setting the username, password, phone number, email, and other information.
+
+### 2. Add the User to a Workspace
+
+```SQL
+CREATE USER user_name [DEFAULT_VCLUSTER = vc_name] [DEFAULT_SCHEMA = schema_name] [COMMENT = 'comment_text'];
+```
+
+### 3. Grant the User a Role or Permissions
+
+```SQL
+GRANT ROLE workspace_dev TO USER user_name;
+GRANT SELECT ON TABLE public.my_table TO USER user_name;
+```
+
+### 4. Modify User Properties
+
+```SQL
+ALTER USER user_name SET DEFAULT_VCLUSTER = new_vc DEFAULT_SCHEMA = new_schema COMMENT = 'updated comment';
+```
+
+### 5. View User List and Permissions
+
+```SQL
+SHOW USERS;
+SHOW GRANTS TO USER user_name;
+```
+
+### 6. Remove a User from the Workspace
+
+```SQL
+DROP USER [IF EXISTS] user_name;
+```
+
+> ⚠️ **Note**: `DROP USER` only removes the user from the current workspace. It does not delete the user's account and password from the instance user management system.
+
+---
+
+## Common Operations
+
+### Add a User to a Workspace
+
+```SQL
+-- Add an instance user to the current workspace
+CREATE USER alice;
+
+-- Add with a default cluster and schema
+CREATE USER bob
+  DEFAULT_VCLUSTER = analytics_cluster
+  DEFAULT_SCHEMA = public;
+```
+
+### Modify User Configuration
+
+```SQL
+-- Change the default cluster
+ALTER USER bob SET DEFAULT_VCLUSTER = etl_cluster;
+
+-- Change the default schema
+ALTER USER bob SET DEFAULT_SCHEMA = dwd;
+```
+
+### View Users
+
+```SQL
+-- List all users
+SHOW USERS;
+```
+
+### Remove a User from the Workspace
+
+```SQL
+-- Remove a user (instance account is preserved)
+DROP USER IF EXISTS alice;
+```
+
+---
+
+## Access Management Best Practices
+
+1. **Use RBAC**: Manage permissions through roles rather than granting them directly to users.
+2. **Follow the principle of least privilege**: Grant users only the minimum permissions needed to do their work.
+3. **Review permissions regularly**: Use `SHOW GRANTS TO USER` to periodically audit user permissions.
+4. **Use WITH GRANT OPTION with caution**: A user granted this option can re-grant the permission to others.
+
+---
+
+## Notes
+
+- `CREATE USER` does not create a new account — it only adds an existing instance user to the workspace. Instance users are created in the console.
+- `DROP USER` only removes workspace access; it does not delete the instance user's account and password.
+- Before removing a user, run `SHOW GRANTS TO USER user_name` to confirm their permissions have been handled.
+
+---
+
+## Related Documentation
+
+| Document | Description |
+|----------|-------------|
+| [SQL Commands Overview](sql-commands.md) | Categorized navigation for all SQL commands |
+| [Roles and Privileges](role-privilege-manage.md) | Grant roles and permissions to users |
+| [CREATE USER](create-user.md) | Full syntax for adding an instance user to a workspace |
+| [ALTER USER](alter-user.md) | Full syntax for modifying user properties |
+| [DROP USER](drop-user.md) | Full syntax for removing a user from a workspace |
+| [SHOW USERS](show-users.md) | View the workspace user list |
+| [GRANT](grant-privileges.md) | Grant permissions to users or roles |
+| [REVOKE](revoke-privileges.md) | Revoke permissions from users or roles |
+| [SHOW GRANTS](show-grants.md) | View the permission list for a user or role |
+| [Roles](roles.md) | Create and manage roles |

package/bin/skills/lakehouse-doc-en/references/authoritymanagement.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/auto-index.md

@@ -0,0 +1,52 @@
+# Reasons for Recommendation
+
+The recommended columns can be used as the sort key for the table
+
+The selected columns frequently appear in filter statements. Setting these columns as the sort key for the table can speed up query execution.
+
+### How to Enable
+
+* You can set the property on the workspace, which will analyze all tables under the workspace
+
+```Plain
+ALTER WORKSPACE workspace_name SET properties (auto_index='day[,150,5,100]');
+
+For example:
+ALTER WORKSPACE quick_start SET properties (auto_index='day'); 
+ALTER WORKSPACE quick_start SET properties (auto_index='day,150,5,100');
+
+--The first parameter is required, specifying whether to collect daily or monthly. Monthly collection is done on the 1st of each month at 6 PM.
+--The second parameter specifies the duration of jobs to be used (in minutes), default is 150.
+--The third parameter specifies how many times a job needs to be repeated to be used, default is 5.
+--The fourth parameter specifies the maximum number of jobs used per column, default is 100.
+```
+
+* Automatic Collection
+
+Automatically collects the top 10 tables used in filter statements for each schema
+
+### How to Query
+
+```Plain
+SELECT * FROM information_schema.sortkey_candidates;
+
+--The returned results are as follows:
+instance_id,workspace_id,workspace_name,schema_id,schema_name,table_id,table_name,col,statement,ratio,insert_time,p_date
+11111111111,855911111111,aaaaaaaaaaa,23671111111111,ddddddddd,84465111,aaaaaaaaaaa.ddddddddd.member,brandid,alter table aaaaaaaaaaa.ddddddddd.member set properties("hint.sort.columns"="brandid"), 2.21%,05:49.1,2024/11/7
+11111111111,855911111111,aaaaaaaaaaa,23671111111111,ddddddddd,84465111,aaaaaaaaaa.ddddddddd.member,birthday,alter table aaaaaaaaaa.ddddddddd.member set properties("hint.sort.columns"="birthday"),0.01%,05:49.1,2024/11/7
+11111111111,855911111111,aaaaaaaaaaa,23671111111111,ddddddddd,84465111,aaaaaaaaaaa.ddddddddd.member,id,alter table aaaaaaaaaaa.ddddddddd.member set properties("hint.sort.columns"="id"),0.01%,05:49.1,2024/11/7
+
+--It can be seen that for the aaaaaaaaaaa.ddddddddd.member table, the improvement effect of brandid is the highest, so set it as the sortkey alter table aaaaaaaaaaa.ddddddddd.member set properties("hint.sort.columns"="brandid"), this setting will
+```
+
+The included columns are `instance_id, workspace_id, workspace_name, schema_id, schema_name, table_id, table_name, col, statement, ratio, insert_time, p_date`. Among them:
+
+* `p_date` is the partition column, formatted as yyyy-mm-dd.
+* `insert_time` is the time when the collection results are inserted.
+* `col` is the recommended column, currently a single column.
+* `statement` is the SQL statement on how to set the recommended column as the sorting column.
+* `ratio` is the estimated improvement effect, in percentage.
+
+### Usage Recommendations
+
+It is recommended to use the [analyze statement](analyze-table.md) to collect stats for the table columns, which will improve the accuracy of the recommendations.

package/bin/skills/lakehouse-doc-en/references/aws_storage_connection.md

@@ -0,0 +1,112 @@
+# Create Amazon Cloud Storage Connection
+
+The goal of this step is to allow the Lakehouse cluster to access Amazon Cloud (AWS) object storage S3. To achieve this goal, two authentication methods provided by AWS's **Identity and Access Management (IAM)** product can be used: **Access Keys** and **Role Authorization**.
+
+## Based on Access Keys
+
+```SQL
+CREATE STORAGE CONNECTION aws_bj_conn
+    TYPE S3
+    ACCESS_KEY_ID = 'AKIAQNBSBP6EIJE33***'
+    SECRET_ACCESS_KEY = '7kfheDrmq***************************'
+    ENDPOINT = 's3.cn-north-1.amazonaws.com.cn'
+    REGION = 'cn-north-1';
+```
+
+### Parameters:
+
+* `TYPE`: The object storage type, for AWS, fill in S3 (case insensitive)
+* `ACCESS_KEY_ID / SECRET_ACCESS_KEY`: The access key for AWS, refer to: [Access Keys](https://docs.aws.amazon.com/zh_cn/IAM/latest/UserGuide/id_credentials_access-keys.html) for how to obtain it
+* `ENDPOINT`: The service address for S3. AWS China is divided into Beijing and Ningxia regions. The service address for S3 in the Beijing region is `s3.cn-north-1.amazonaws.com.cn`, and for the Ningxia region is `s3.cn-northwest-1.amazonaws.com.cn`. Refer to: [China Region Endpoints](https://docs.amazonaws.cn/aws/latest/userguide/endpoints-arns.html) to find the endpoints for the Beijing and Ningxia regions -> Amazon S3 corresponding endpoints
+* `REGION`: AWS China is divided into Beijing and Ningxia regions, the region values are: Beijing region `cn-north-1`, Ningxia region `cn-northwest-1`. Refer to: [China Region Endpoints](https://docs.amazonaws.cn/aws/latest/userguide/endpoints-arns.html). For the international site, refer to: [Amazon Simple Storage Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/s3.html)
+
+## Role-Based Authorization
+
+You need to create a **permission policy** and a **role** in AWS IAM under the account that owns the target S3 bucket. The permission policy defines the rules for accessing AWS S3 data, and this policy is attached to the created role. Singdata Lakehouse achieves read and write operations on S3 data by assuming this role.
+
+### STEP1: Create a Permission Policy (LakehouseAccess) on AWS:
+
+* Log in to the AWS cloud platform and enter the **Identity and Access Management (IAM)** product console
+* In the IAM page's left navigation bar, go to **Account Settings**, in the **Security Token Service (STS)** section's **Endpoints** list, find the region corresponding to the current Singdata Lakehouse instance. If the **STS Status** is not enabled, please enable it.
+* In the IAM page's left navigation bar, go to **Policies**, in the **Policies** interface, select **Create Policy**, and choose the JSON method in the policy editor
+* Add the policy that allows Singdata Lakehouse to access the S3 bucket and directory. Below is a sample policy, please replace `<bucket>` and `<prefix>` with the actual bucket and path prefix names.
+
+> ⚠️ **Note**: Please fill in the `"s3:prefix"` item with: `["*"]` or `["<path>/*"]` to grant access permissions to all prefixes in the specified bucket or path in the bucket respectively.
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+              "s3:PutObject",
+              "s3:GetObject",
+              "s3:GetObjectVersion",
+              "s3:DeleteObject",
+              "s3:DeleteObjectVersion"
+            ],
+            "Resource": "arn:aws:s3:::<bucket>/<prefix>/*"
+        },
+        {
+            "Effect": "Allow",
+            "Action": [
+                "s3:ListBucket",
+                "s3:GetBucketLocation"
+            ],
+            "Resource": "arn:aws:s3:::<bucket>",
+            "Condition": {
+                "StringLike": {
+                    "s3:prefix": [
+                        "<prefix>/*"
+                    ]
+                }
+            }
+        }
+    ]
+}
+```
+
+* Select **Next**, enter the policy name (e.g., `LakehouseAccess`) and description (optional)
+* Click **Create Policy** to complete the policy creation
+
+### STEP2: Create Role (LakehouseVolumeRole) on AWS:
+
+* Log in to the AWS cloud platform and go to the **Identity and Access Management (IAM)** product console
+* In the IAM page, navigate to **Roles** -> **Create Role** -> **AWS Account**, select **Another AWS Account**, and enter `028022243208` (China site) / `014617434350` (international site) in the Account ID
+
+> ⚠️ **Note**: To prevent the ROLE\_ARN from being obtained by third parties for unauthorized data access, you can check **Options** and select **Require external ID (best practice for third parties assuming this role)**. After checking, you can fill in `000000` as a placeholder in the **EXTERNAL ID** field to be filled in later. `EXTERNAL ID` serves as an additional layer of verification, ensuring that access is only allowed when the request includes the preset `EXTERNAL ID`. This means that even if a third party knows some other access information (such as the role ARN), they cannot access the resources without the correct `EXTERNAL ID`.
+
+![](.topwrite/assets/image_1728551322654.png)
+
+* Select **Next**, on the Add permissions page, choose the policy created in STEP1 `LakehouseAccess`, then select **Next**
+* Fill in the Role name (e.g., `LakehouseVolumeRole`) and description, click **Create Role** to complete the role creation
+* In the role details page, obtain the value of **Role ARN** to use when creating the STORAGE CONNECTION
+
+![](.topwrite/assets/image_1728551348143.png)
+
+^
+
+### STEP3: Create STORAGE CONNECTION on Singdata Lakehouse Side:
+
+* Execute the following commands in Studio or Lakehouse JDBC client:
+
+```SQL
+CREATE STORAGE CONNECTION aws_bj_conn_arn
+  TYPE S3
+  REGION = 'cn-north-1'
+  ROLE_ARN = 'arn:aws-cn:iam::02802*******:role/LakehouseVolumeRole';
+```
+
+* During the process of creating a storage connection, Lakehouse will generate this EXTERNAL ID. You can configure this EXTERNAL ID into the Trust Policy of the AWS IAM role (`LakehouseVolumeRole`) created in STEP2 to achieve additional access control:
+
+```SQL
+-- View EXTERNAL ID 
+DESC CONNECTION aws_bj_conn_arn;
+```
+
+![](.topwrite/assets/20240626-215920.jpeg)
+
+^
+
+* In the AWS IAM console, navigate to **Roles** in the left sidebar, find the role created in STEP2 and go to the role details page. In **Trust relationships**, replace the value of `sts:ExternalId` `000000` with `EXTERNAL_ID` from the DESC result. Click **Update** to complete the role policy update.

package/bin/skills/lakehouse-doc-en/references/awscloud_privatelink.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/backfilling_data.md

@@ -0,0 +1,50 @@
+# Backfilling Data
+
+Backfilling data can be done by supplementing historical or future data for a period of time, writing the data to the corresponding time partition. If scheduling parameters are configured in the code, they will be automatically replaced with specific values based on the business time selected for data backfilling, and the corresponding time data will be written to the specified partition in combination with the business code. The specific partition written to and the code logic executed are related to the code defined in the task.
+
+## Function Usage Scenarios
+
+* When a new task is created and you want to process historical data, you can use the data backfilling function to select a historical time period to execute the task;
+* If there is a problem with the data or the task is abnormal during a certain period, you can use data backfilling to regenerate and execute instances of the historical abnormal data or task to achieve the purpose of repairing historical data.
+
+## Product Operation
+
+### Step 1: Create a Backfilling Data Task
+
+1. Click on the left navigation **Operations Monitoring -> Task Operations**, **select the Periodic Task** Tab. Choose any of the periodic tasks under management, click on Operation - Data Supplement, and enter the data supplement operation interface.
+2. Configure the corresponding operations for the data supplement task
+
+* Task Name: System default name, user input is not supported for now
+
+* Select the task nodes included in the data supplement task
+
+  * Include current node: Users can choose whether to check the current task to participate in the data supplement based on actual needs.
+  * Include downstream nodes: Users can check the **downstream task links** that need data backfilling based on actual needs. It supports three options: no selection, select all, and custom selection range.
+
+* Task scheduling time range: When creating a data supplement task, it supports adding multiple planned execution time ranges to solve the scenario of wanting to supplement multiple non-continuous times. Currently, it supports adding up to 4 planned execution time ranges. Users are not allowed to select overlapping times.
+
+* Concurrency settings: Indicates that when supplementing data for multiple consecutive periods, if the concurrency number is met, multiple period instances can run simultaneously, which can improve the efficiency of data backfilling but will also consume more resources.
+
+  * After enabling concurrent execution, the system will by default attach self-dependency to task instances allocated within the same concurrency group.
+
+### Step 2: View the Backfilling Data Task
+
+1. Click on the left navigation **Operations Monitoring -> Task Operations**, **select the Data Supplement Task** Tab.
+
+2. Basic Information: Includes concurrency group, target task ID, task name, workspace, whether it includes downstream, status, execution order, task scheduling time range, submission time, submitter, etc.
+
+3. Statistics Chart:
+
+* Global Statistics: You can see the total number of all times included in the current data supplement task and give the execution status based on the day granularity.
+
+* Effect Chart by Granularity: Provides data statistics at year/month/day granularity. Clicking on the statistics chart will link to the instance list below.
+
+  * **Year View**: Hovering over the date shows the execution status of that day;
+
+  * **Month View**: You can see the total number of instances to be executed on that day in the month view, and hovering over the calendar shows the execution status summary;
+
+  * **Day View**: Supports switching between day/1-hour granularity, showing the average execution time of instances under different aggregation granularities.
+
+4. Instance Statistics List: By default, it displays the information of all execution instances under the current data supplement task according to the planned time. Clicking can jump to the instance execution details page to view dependency relationships and other details.
+
+^

package/bin/skills/lakehouse-doc-en/references/batch-load-parquet-file-into-lakehouse.md

@@ -0,0 +1,79 @@
+# Data Import into Lakehouse: Batch Loading via COPY Command
+
+This guide will help you import large amounts of data from public URL Parquet files (such as New York City Taxi and For-Hire Vehicle Data) into Singdata Lakehouse using scripts and the Singdata Lakehouse SQLLine command-line tool.
+
+1. download_raw_data.sh: Download Public URL Parquet files to local. This article uses NYC Taxi Data as an example, containing over 450 files and 60GB of data (Parquet format).
+
+2. initialize_database.sh: Create Singdata Lakehouse schema and tables via Lakehouse SQLLine commands.
+
+3. Import taxi and FHV data: Convert Parquet files to CSV files using R, then import data from local CSV files into Singdata Lakehouse tables using the Lakehouse SQLLine COPY command, supporting batch import of multiple files.
+
+# Operation Guide
+
+Script download address: <https://github.com/yunqiqiliang/nyc-taxi-data-clickzetta>
+
+## 1. Install [Singdata SQLLine](https://doc.clickzetta.com/zh-CN/connect-with-cli)
+
+## 2. Install [R](https://www.r-project.org/)
+
+From [CRAN](https://cloud.r-project.org/)
+
+Note that R used to be optional for this repo, but is required starting with the 2022 file format change. The scripts use R to convert Parquet files to CSV before loading into Postgres. There are other ways to convert from Parquet to CSV that wouldn't require R, but I found that R's `arrow` package was faster than some of the other CLI tools I tried
+
+## 3. Download raw data
+
+`./download_raw_data.sh`
+
+When done, modify download_raw_data.sh and run again to download data from 202212:
+wget -i setup_files/raw_data_urls_new\.txt -P data/ -w 2
+
+## 4. Initialize database and set up schema
+
+`./initialize_database.sh`
+
+## 5. Import taxi and FHV data
+
+`./import_yellow_taxi_trip_data.sh`
+
+`./import_green_taxi_trip_data.sh`
+
+`./import_fhv_taxi_trip_data.sh`
+
+`./import_fhvhv_trip_data.sh`
+
+Note that the full import process might take several hours or possibly even over a day, depending on computing power.
+
+# Schema
+
+* `trips` table contains all yellow and green taxi trips. Each trip has a `cab_type_id`, which references the `cab_types` table and refers to one of `yellow` or `green`
+* `fhv_trips` table contains all for-hire vehicle trip records, including ride-hailing apps Uber, Lyft, Via, and Juno
+* `fhv_bases` maps `fhv_trips` to base names and "doing business as" labels, which include ride-hailing app names
+* `nyct2010` table contains NYC census tracts plus the Newark Airport. It also maps census tracts to NYC's official neighborhood tabulation areas
+* `taxi_zones` table contains the TLC's official taxi zone boundaries. Starting in July 2016, the TLC no longer provides pickup and dropoff coordinates. Instead, each trip comes with taxi zone pickup and dropoff location IDs
+* `central_park_weather_observations` has summary weather data by date
+
+# Load Script Example
+
+[Full Script](https://github.com/clickzetta/nyc-taxi-data-clickzetta/blob/master/readme.md)
+
+```
+#!/bin/bash
+
+fhv_schema="(dispatching_base_num, pickup_datetime, dropoff_datetime, pickup_location_id, dropoff_location_id, legacy_shared_ride_flag, affiliated_base_num)"
+
+for parquet_filename in data/fhv_tripdata*.parquet; do
+  echo "`date`: converting ${parquet_filename} to csv"
+  ./setup_files/convert_parquet_to_csv.R ${parquet_filename}
+
+  csv_filename=${parquet_filename/.parquet/.csv}
+
+  sh ../sqlline_cz/sqlline properties ../sqlline_cz/clickzetta.properties -e "set copy.csv.with.header=false;set copy.csv.skip.header=true; copy fhv_trips_staging from '${csv_filename}';"
+
+  echo "`date`: finished raw load for ${csv_filename}"
+
+  rm -f $csv_filename
+  echo "`date`: deleted ${csv_filename}"
+done;
+```
+
+^