@clickzetta/cz-cli-darwin-x64 0.3.94 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/refactor-elt-practice.md

@@ -0,0 +1,241 @@
+# Redefining ELT: A New Approach to ELT with Singdata Lakehouse
+
+## Problems and Challenges
+
+In many data projects, especially those closely related to ML and AI, development often requires a mix of Python and SQL. For example, Python code is developed via Jupyter Notebook, while SQL code is developed using data management and development tools like DBeaver or VS Code. This approach frequently leads to issues such as a lack of version management and the absence of a unified scheduling service, making development and operations complex and difficult.
+
+As modern data stacks evolve, next-generation BI and AI analysis products such as Metabase and MindsDB are also rapidly advancing. How a data platform tightly integrates into the modern data stack to provide a complete solution has become critically important.
+
+For the growing volume of semi-structured data and vector data (such as CSV and JSON), relying on client-side parsing severely impacts overall processing speed. This demands that the data platform be capable of high-performance processing of structured data in tables as well as JSON data and vector data simultaneously. For example, the hourly GitHub event archive dataset provided by GH Archive is delivered in JSON format.
+
+In data engineering tasks involving data cleaning and transformation, many intermediate tables often need to be developed before reaching the final target table. Improving efficiency in this area has become an important topic for boosting data engineer productivity.
+
+## A New ELT Approach with Singdata Lakehouse
+
+The author attempts to use Singdata Lakehouse to find new solutions to the above problems from the following three perspectives.
+
+![](.topwrite/assets/image_1734348544459.png)
+
+### Unified Data Task Development and Scheduling
+
+Simplify the data task development and operations environment through Singdata Lakehouse, achieving unified task version management:
+
+* [Python Task Development](python-task.md): From Jupyter Notebook development to Python task development in Singdata Lakehouse Studio.
+
+* [SQL Task Development](task-develop.md): From DBeaver SQL task development to SQL task development in Singdata Lakehouse Studio.
+
+* [ML/AI Task Development](jdbc_task.md): From MindsDB Web IDE to JDBC task development in Singdata Lakehouse Studio.
+
+* [Task Scheduling and Operations](task-instance-maintenance.md): From Airflow to task scheduling and operations in Singdata Lakehouse Studio.
+
+  ![](.topwrite/assets/image_1734348698701.png =460)
+
+### [Multi-Data-Type Support](data-type.md)
+
+Singdata Lakehouse supports efficient storage and access for the following data types:
+
+* Common basic data types in tables
+
+* Common complex data types in tables
+
+* JSON
+
+* Vector
+
+* Full-text inverted index
+
+  ![](.topwrite/assets/image_1734348746004.png =380)
+
+### [Dynamic Table](dynamic-table.md)
+
+The Dynamic Table provided by Singdata has the following typical features:
+
+* Declarative Programming: Use declarative SQL to define pipeline results without having to consider intermediate table logic, reducing complexity.
+* Transparent Orchestration: Manage refresh orchestration and scheduling by chaining together Dynamic Tables and regular tables.
+* Incremental Processing: Suitable for incremental workloads with excellent Dynamic Table performance.
+* Easy Switching: A single command transitions from batch processing to streaming processing, balancing cost and freshness.
+* Observability: Dynamic Tables can be managed through the Singdata Lakehouse Web Console Studio, improving observability.
+
+  ![](.topwrite/assets/image_1734348809529.png =450)
+
+^
+
+## Application Case: GHArchive Data ELT Implementation
+
+![](.topwrite/assets/image_1734348838607.png)
+
+### Data Source Introduction: [GHArchive](http://www.gharchive.org/)
+
+Open-source developers worldwide are engaged in millions of projects covering tasks such as writing code and documentation, fixing bugs, and submitting work. The [GH Archive](http://www.gharchive.org/) project records public GitHub event data, providing over 13 years of GitHub event data archives starting from February 12, 2011, and making the data more accessible for further analysis.
+
+GitHub has over 15 event types, such as new commits (Push), forking a repository (Fork), creating new Issues, commenting, and adding members to projects. These events are aggregated into hourly archive files accessible via any HTTP client. Each archive file encodes events as reported by the GitHub API in JSON format. Users can download the raw data for self-processing, such as writing custom aggregation scripts or importing it into a database.
+
+Example GitHub event data:
+
+```
+{"id":"44809399421","type":"CreateEvent","actor":{"id":81469924,"login":"temperature48","display_login":"temperature48","gravatar_id":"","url":"https://api.github.com/users/temperature48","avatar_url":"https://avatars.githubusercontent.com/u/81469924?"},"repo":{"id":903968841,"name":"temperature48/SmashVat","url":"https://api.github.com/repos/temperature48/SmashVat"},"payload":{"ref":null,"ref_type":"repository","master_branch":"master","description":null,"pusher_type":"user"},"public":true,"created_at":"2024-12-16T02:00:00Z"}
+```
+
+^
+
+### Development Task: Data Extraction (**E**xtract)
+
+Next, we develop a Python task through Singdata Lakehouse Studio to extract data from the GH Archive website into data lake storage (this solution uses Alibaba Cloud OSS).
+
+![](.topwrite/assets/image_1734348912514.png)
+
+^
+
+```py
+import os,io
+import subprocess
+from datetime import datetime, timedelta
+import oss2
+
+# Alibaba Cloud OSS configuration
+ACCESS_KEY_ID = '${ak}'
+ACCESS_KEY_SECRET = '${sk}'
+BUCKET_NAME = 'OSS Bucket Name'
+ENDPOINT = 'Alibaba Cloud OSS Endpoint'
+ROOT_PATH = 'ghachive, replace with the actual root directory under the bucket'
+
+# Get current time in UTC+8
+# beijing_time = datetime.now() 
+beijing_time = datetime.strptime('${datetime}', "%Y-%m-%d %H:%M:%S")
+
+# Get file time; Beijing time offset by 9 hours (8 hours time zone + 1 hour for gharchive data file delay)
+ny_time = beijing_time - timedelta(hours=9)
+
+# Format time
+year = ny_time.strftime('%Y')
+month = ny_time.strftime('%m')
+day = ny_time.strftime('%d')
+hour = ny_time.strftime('%H')
+
+# Print converted time
+print(f"Converted to data file Time and -9 hour: {year}-{month}-{day} {hour}:00:00")
+
+# Check if hour is in '0x' format, and if so remove leading zero
+if hour.startswith('0') and len(hour) > 1:
+    # Remove leading '0'
+    hour = hour[1:]
+
+try:
+    # Build wget command
+    url = f"https://data.gharchive.org/{year}-{month}-{day}-{hour}.json.gz"
+    cmd = ["wget", "-qO-", url]
+    print(f"wget cmd: {cmd}")
+
+    # Execute wget command and capture output
+    wget_output = subprocess.check_output(cmd)
+    print(f"Wget file done...")
+
+    # Convert output to in-memory file object
+    file_obj = io.BytesIO(wget_output)
+except Exception as e:
+    print(f"An error occurred: {e}")
+    file_obj = None
+    raise
+
+if file_obj:
+    try:
+        # Initialize Alibaba Cloud OSS
+        auth = oss2.Auth(ACCESS_KEY_ID, ACCESS_KEY_SECRET)
+        bucket = oss2.Bucket(auth, ENDPOINT, BUCKET_NAME)
+
+        # Upload file to OSS
+        oss_path = f"{ROOT_PATH}/{year}/{month}/{day}/{year}-{month}-{day}-{hour}.json.gz"
+        print(f"osspath: {oss_path}")
+        bucket.put_object(oss_path, file_obj)
+        print(f"Put file to oss done...")
+    except Exception as e:
+        print(f"An error occurred: {e}")
+    finally:
+        # Close in-memory file object
+        file_obj.close()
+```
+
+^
+
+### Development Task: Data Loading (**L**oad)
+
+Singdata Lakehouse Studio supports loading data from data lakes into Lakehouse tables via offline sync tasks, storing it in optimized formats and enabling fine-grained permission management.
+
+![](.topwrite/assets/image_1734348942887.png)
+
+^
+
+### Development Task: Data Cleaning and Transformation (**T**ransform)
+
+Singdata Lakehouse Dynamic Tables support incremental refresh optimization by processing only changed data. Compared to traditional ETL tasks, Dynamic Tables eliminate the need for full data computation and do not require specifying incremental logic (such as partition alignment or using max(system/event_time)). Users simply declare the business logic, and the Dynamic Table automatically performs incremental computation optimization.
+
+This article uses Dynamic Tables for data cleaning and transformation.
+
+![](.topwrite/assets/image_1734348956194.png)
+
+^
+
+#### Ad Hoc Data Analysis
+
+Singdata Lakehouse has a built-in, convenient SQL query interface where you can freely write SQL code, use variables, select clusters, and perform flexible ad hoc data analysis. It is also convenient for debugging.
+
+![](.topwrite/assets/image_1734348969154.png)
+
+^
+
+#### Development Task: AI/ML Enhanced Analysis
+
+Based on the JDBC task nodes provided by Lakehouse Studio, you can conveniently connect to MindsDB for enhanced data analysis.
+
+![](.topwrite/assets/image_1734348988810.png)
+
+^
+
+### Incremental Sync Task Scheduling
+
+Configure scheduling and dependency relationships for the ELT tasks just developed, ensuring tasks run on an hourly cycle. Submit and publish the tasks so that GitHub event data can be synced hourly from GHArchive (with a one-hour lag) and cleaned and transformed.
+
+![](.topwrite/assets/image_1734349005883.png)
+
+^
+
+### Backfilling Full Data
+
+For data backfill tasks in Singdata Lakehouse Studio, you can fully reuse the incremental tasks. Simply configure and execute the backfill task with the start and end dates for the full data range to backfill historical data.
+
+Taking GHArchive data as an example, its website (http://www.gharchive.org/) has data starting from February 12, 2011. Configure the backfill task as shown below and run it to achieve full data sync. Backfill tasks can run concurrently with incremental data sync tasks, and there is no need to develop a new full sync task.
+
+![](.topwrite/assets/image_1734349021555.png)
+
+^
+^
+
+### Task Operations
+
+Singdata Lakehouse Studio also provides comprehensive operations interfaces, such as displaying upstream and downstream DAG relationships for tasks, running status, and performing operations like reruns.
+
+^
+
+![](.topwrite/assets/image_1734349033926.png)
+
+^
+^
+
+## Summary
+
+This article discusses the problems and challenges of traditional development approaches in data projects, especially those closely tied to ML and AI, and how to implement a new ELT approach for GHArchive using Singdata Lakehouse -- enabling unified development, scheduling, and operations, significantly streamlining the product components in the solution, and helping to greatly improve efficiency and reduce management costs. Key takeaways include:
+
+* **Unified Data Task Development and Scheduling**: Simplify development and operations environments; achieve consistent task version management covering Python, SQL, and ML/AI task development, as well as unified task scheduling orchestration and operations control.
+* **Multi-Data-Type Support**: Efficient storage and access for common basic data types, complex data types, JSON, Vector, full-text inverted indexes, and more.
+* **Dynamic Table**: Features declarative programming, transparent orchestration, incremental processing, easy switching, and observability.
+* **Application Case**: Using GHArchive data to demonstrate data extraction, loading, cleaning and transformation, ad hoc data analysis, AI/ML enhanced analysis, incremental sync task scheduling, full data backfilling, and task operations.
+
+## References
+
+* [Singdata Lakehouse Dynamic Table](dynamic_table_summary.md)
+* [Singdata Lakehouse Studio Task Development and Scheduling](task-develop.md)
+* [Data Backfill Tasks](backfilling_data.md)
+* [Singdata Lakehouse JSON Data Type](json.md)
+* [Singdata Lakehouse Vector Data Type](vector-type.md)
+
+^

package/bin/skills/lakehouse-doc-en/references/refresh-dynamic-table.md

@@ -0,0 +1,35 @@
+# REFRESH DYNAMIC TABLE
+
+## Overview
+
+Manually triggers an immediate refresh of a dynamic table, forcing one incremental or full refresh before the next scheduled cycle. The system automatically selects the optimal strategy (incremental or full).
+
+## Syntax
+
+```Plain
+REFRESH DYNAMIC TABLE [<schema_name>.]<dt_name>
+```
+
+## Parameters
+
+- `<schema_name>`: Optional. Specifies the schema name; defaults to the current schema if omitted.
+- `<dt_name>`: The name of the dynamic table to refresh.
+
+## Examples
+
+```sql
+REFRESH DYNAMIC TABLE dt_hourly_stats;
+
+REFRESH DYNAMIC TABLE doc_test.dt_hourly_stats;
+```
+
+## Notes
+
+- A manual refresh triggers one additional refresh only and does not affect the existing automatic scheduling cycle.
+- The system automatically determines whether to perform an incremental or full refresh; you cannot specify this manually.
+
+## Related Documentation
+
+- [REFRESH MATERIALIZED VIEW](refresh-materialized-view.md) — refresh a materialized view
+- [CREATE DYNAMIC TABLE](create-dynamic-table.md)
+- [ALTER DYNAMIC TABLE](alter-dynamic-table.md) — pause/resume/modify the refresh interval

package/bin/skills/lakehouse-doc-en/references/refresh-history.md

@@ -0,0 +1,54 @@
+# Description
+
+Used to return the job history of all ongoing and historical refreshes. Currently, only the most recent 10,000 records from the past 7 days will be displayed. The results can show how many incremental refreshes there are, whether it is an incremental refresh, etc.
+
+# Syntax
+```SQL
+ SHOW DYNAMIC TABLE REFRESH HISTORY [where <expr>] [LIMIT num];
+```
+# Parameter Description
+
+* `WHERE <expr>`: (Optional) Allows users to filter based on the fields displayed by the `SHOW JOBS` command. Users can use expressions to filter the results for more precise data retrieval.
+* `LIMIT num`: (Optional) Limits the number of job records returned, ranging from 1-10000.
+
+# Return Results
+
+| Field              | Description                                                                 |
+| ------------------ | --------------------------------------------------------------------------- |
+| workspace\_name    | Workspace name                                                              |
+| schema\_name       | Schema name                                                                 |
+| name               | Dynamic table name                                                          |
+| virtual\_cluster   | Computing cluster used                                                      |
+| start\_time        | Refresh start time, timestamp type                                          |
+| end\_time          | Refresh end time, timestamp type                                            |
+| duration           | Refresh duration, interval type                                             |
+| state              | Job status                                                                  |
+| refresh\_trigger   | MANUAL (manually triggered by user, including studio scheduling) SYSTEM\_SCHEDULED (scheduled by lakehouse) |
+| suspended\_reson   | Reserved field with no special meaning                                      |
+| refresh\_mode      | NO\_DATA FULL INCREMENTAL                                                   |
+| error\_message     | Information on refresh failure                                              |
+| source\_tables     | Records the base table names used by the dynamic table                      |
+| stats              | Information on incremental refresh counts                                   |
+| completion\_target | Reserved field with no special meaning                                      |
+| job\_id            | Job ID, clicking on the job ID allows viewing of the job profile            |
+
+#  Example
+
+* Filter the corresponding refresh history based on the dynamic table name
+```SQL
+ SHOW DYNAMIC TABLE REFRESH HISTORY  where name='dau';
+```
+* Filter based on time consumption, filtering out UI elements that take more than one second
+```SQL
+ SHOW DYNAMIC TABLE REFRESH HISTORY  where name='dau' and duration>interval 1 second;
+```
+* Filter based on computing cluster
+```SQL
+ SHOW DYNAMIC TABLE REFRESH HISTORY  where name='dau' and duration>interval 1 second and virtual_cluster='DEFAULT';
+```
+* Filter by start time
+```SQL
+ SHOW DYNAMIC TABLE REFRESH HISTORY  where name='dau' and duration>interval 1 second and virtual_cluster='DEFAULT'  and start_time>timestamp'2024-06-12 12:47:07.881';
+```
+
+^

package/bin/skills/lakehouse-doc-en/references/refresh-materialized-view.md

@@ -0,0 +1,35 @@
+# REFRESH MATERIALIZED VIEW
+
+## Overview
+
+Manually triggers a full recomputation of a materialized view, completely rebuilding the data from the source tables and resetting the materialized view status to valid. A materialized view is automatically invalidated after DML changes to its source tables. Running this command restores its ability to participate in query rewriting.
+
+## Syntax
+
+```Plain
+REFRESH MATERIALIZED VIEW [<schema_name>.]<mv_name>
+```
+
+## Parameters
+
+- `<schema_name>`: Optional. Specifies the schema name; defaults to the current schema if omitted.
+- `<mv_name>`: The name of the materialized view to refresh.
+
+## Examples
+
+```sql
+REFRESH MATERIALIZED VIEW mv_sales_summary;
+
+REFRESH MATERIALIZED VIEW doc_test.mv_sales_summary;
+```
+
+## Notes
+
+- This performs a full recomputation. When source tables are large, this consumes significant compute resources. It is recommended to run this during off-peak hours.
+- A materialized view is automatically invalidated after INSERT, UPDATE, or DELETE operations on its source tables. While invalidated, it can still be queried directly but does not participate in query rewriting.
+
+## Related Documentation
+
+- [REFRESH DYNAMIC TABLE](refresh-dynamic-table.md) — refresh a dynamic table
+- [CREATE MATERIALIZED VIEW](create-materialized-view.md)
+- [ALTER MATERIALIZED VIEW](alter-materialized-view.md) — pause/resume automatic refresh scheduling

package/bin/skills/lakehouse-doc-en/references/regexp-function.md

@@ -0,0 +1 @@
+# Regex Functions

package/bin/skills/lakehouse-doc-en/references/regexp-statement.md

@@ -0,0 +1,75 @@
+# List of Functions Supported by Regular Expressions
+| Function | Description | 
+| -------------------- | ------------------------------------------------------- |
+| RLIKE                | Used to check if a string matches a specified regular expression pattern. |
+| REGEXP_EXTRACT      | Extracts matching text from an input string (str) using a specified regular expression (regexp). |
+| REGEXP_EXTRACT_ALL | Used to extract all substrings from a string that match a regular expression. |
+| REGEXP_REPLACE      | Finds all substrings in string str that match the regular expression regexp and replaces them with the specified string rep. |
+
+# Instructions
+
+The regular expression engine used by Lakehouse is [re2](https://github.com/google/re2):
+
+* **POSIX Character Classes**: Allows the use of character classes like `[[:alnum:]]`.
+* **Backslash Sequences**: Includes `\d` (digit), `\D` (non-digit), `\s` (whitespace), `\S` (non-whitespace), `\w` (word character), `\W` (non-word character), `\b` (word boundary), and `\B` (non-word boundary).
+* **POSIX Wildcards**: Such as `.` (matches any single character) and `*` (matches zero or more occurrences of the preceding element).
+* **Unicode Support**: All regular expression functions support Unicode characters.
+
+# Handling Escape Characters
+When using string literals in SQL statements, if they contain backslash sequences or metacharacters (such as `\d`, `\s`, `*`, `?`), these escape characters will be automatically escaped. For example, the output of `SELECT '\\' AS col;` will be `\`. For values actually stored in the table as `\\`, no escaping will occur; only string literals undergo this escaping.
+
+Therefore, when writing regular expressions that need to match these special sequences, additional escape characters need to be added. For example:
+```SQL
+-- In string literals, backslash sequences will be automatically escaped
+SELECT '\\\\' as col1, '\\' as col2;
++------+------+
+| col1 | col2 |
++------+------+
+| \\   | \    |
++------+------+
+-- If \\ is stored in the table, there will be no automatic escaping
+CREATE TABLE test_regex(col STRING);
+INSERT INTO test_regex VALUES('\\');
+SELECT * FROM test_regex;
++-----+
+| col |
++-----+
+| \\  |
++-----+
+```
+When using regular expressions for matching, it is necessary to escape strings that contain backslash sequences or metacharacters:
+```SQL
+SELECT col, RLIKE(col, '\\\\') FROM test_regex;
++-----+---------------------+
+| col | RLIKE(col, '\\\\') |
++-----+---------------------+
+| \\  | TRUE                |
++-----+---------------------+
+```
+# Usage of String Prefix r
+
+To avoid manually handling escape characters when it is uncertain when they are needed, Lakehouse supports adding the `r` prefix to strings, indicating that escape characters in the [string](<STRING.md>) will not be escaped and can be directly input into the regular expression for execution. This way, users can write regular expressions as usual without worrying about SQL engine translation issues. For example:
+```SQL
+-- Use the r prefix to match data in the table
+SELECT col, RLIKE(col, r'\\') FROM test_regex;
++-----+---------------------+
+| col | RLIKE(col, '\\\\') |
++-----+---------------------+
+| \\  | TRUE                |
++-----+---------------------+
+-- Use the r prefix to match string constants
+SELECT RLIKE(r'%SystemDrive%\Users\John', r'%SystemDrive%\\Users.*') as res;
++------+
+| res  |
++------+
+| TRUE |
++------+
+-- Regular expression matching without using the r prefix
+SELECT r'%SystemDrive%\Users\John' rlike '%SystemDrive\\\\\Users.*';
++------+
+| res  |
++------+
+| true |
++------+
+```
+By using the `r` prefix, Lakehouse simplifies the use of regular expressions, making text matching and data processing more intuitive and efficient.

package/bin/skills/lakehouse-doc-en/references/register_account.md

@@ -0,0 +1,77 @@
+# Register Account
+
+This guide will help you complete the registration of a Singdata account, activate a service instance, and complete the instance initialization process, helping you smoothly start your journey with Singdata.
+
+Start using Singdata
+
+## Account Registration
+
+Before using Singdata services, you need to register a Singdata account.
+
+1. **Visit the account registration address**: [accounts.clickzetta.com](https://accounts.clickzetta.com/register), click the "Register" button, and follow the prompts on the registration page to fill in the information.
+
+:-: ![](.topwrite/assets/image_1736314854109.png =734)
+
+2. :-: **Enter your phone number**: On the Singdata registration page, please enter a phone number that can receive SMS normally, then click the "Get Verification Code" button. The system will send a verification code via SMS to your phone. Please enter the 4-digit verification code you received into the "Verification Code" input box and click "Next". ![](.topwrite/assets/image_1739260609977.png =762)
+
+^
+
+3. **Create a user**: In this step, you need to create the first user for your account. Please fill in "Username", "Password", and "Confirm Password". The username length must be between 5-32 characters, supporting English letters, numbers, and underscores, and must start with a letter. The password length must be between 8-32 characters and must include uppercase letters, lowercase letters, numbers, and special symbols.
+
+4. **Complete registration**: After clicking the "Complete" button, you will see a registration success prompt and receive the account login address. Please save this address properly for future login. The user created during registration will automatically become the administrator of the account. Click the "Go to Login" button in the prompt window and log in to your account using the username and password you registered with.
+
+## Log in to the Account
+
+After you have registered an account, you can log in using the account and the username and password under this account.
+
+1. **Use the account login address**: Log in using the URL of your .accounts.clickzetta.com. Enter the correct username and password in the login prompt box and click the "Login" button. If you forget your username or password, click the "Forgot Username?" or "Forgot Password?" button and use the phone number you registered with to verify and retrieve it.
+
+:-: ![](.topwrite/assets/image_1736314906745.png =776)
+
+2. **Use the general login address**: Visit [accounts.clickzetta.com](https://accounts.clickzetta.com/login) to log in. Enter your 8-digit account name (displayed upon registration), click "Login", and enter the account login page. Enter the correct username and password in the login prompt box and click the "Login" button. If you forget your account name, click "Retrieve Account" on the account login page and use the phone number you registered with to verify and retrieve it.
+
+## Create a Lakehouse Service Instance
+
+After logging into your Singdata account, you will enter the management center homepage. You can activate a Lakehouse service instance in the management center. Please follow these steps:
+
+* Find the Lakehouse product card on the "Home" page of the management center and click the "Activate for Free" button. ![](.topwrite/assets/20250211-191828.jpeg)
+* In the pop-up window, select the cloud service provider and activation region, and click the "Confirm" button to complete the creation of the Lakehouse service instance. Currently, Singdata Lakehouse supports activating Lakehouse service instances in Alibaba Cloud Shanghai, Tencent Cloud Shanghai, and Amazon Cloud (China) Beijing regions.
+
+:-: ![](.topwrite/assets/create_instance.png =616)
+
+* Please note that currently, each account is limited to creating one Lakehouse service instance.
+
+## Instance Initialization
+
+After creating a Lakehouse service instance, you can find and enter the service instance in "My Subscriptions" in the management center. The system will automatically initialize, including:
+
+1. Creating the first workspace named "quick\_start".
+2. Creating the first virtual compute cluster (Virtual Cluster) in the "quick\_start" workspace, which is in a stopped state by default.
+3. Creating system service users `sysservice_clickzetta` and `sysservice_auto_mv`.
+4. Setting the user who created the Lakehouse service instance as the administrator role of the service instance and the workspace administrator (workspace\_admin) role of the "quick\_start" workspace. At this point, you will have all operational permissions for the Lakehouse service instance and the "quick\_start" workspace. Subsequently, you can add more users and assign corresponding permissions through the user and permission management functions.
+
+Now, you have completed the preparation for using Singdata Lakehouse. You can refer to other quick start documents to learn how to quickly get started with Lakehouse. For more detailed operational guidance, please refer to the "[Operation Guide](studio_manual.md)" section.
+
+## Activate DataGPT Service
+
+# DataGPT Service Now Available in Alibaba Cloud East China 2 (Shanghai) Region
+
+DataGPT service is now available in the Alibaba Cloud East China 2 (Shanghai) region. The system will be integrated by default with the Lakehouse instance in the same region, East China 2 (Shanghai) Alibaba Cloud. For Lakehouse instances not activated in Alibaba Cloud - East China 2 (Shanghai), you can manually add them directly on the DataGPT data source page without needing to select any options.
+
+* Find the DataGPT product card on the "Home" page of the management center and click the "Activate for Free" button. 
+
+       ![](.topwrite/assets/20250211-192212.jpeg =731)
+
+* In the pop-up window, the **Cloud Service Provider** Alibaba Cloud and **Region** East China 2 (Shanghai) will be specified by default. The system provides the option "**Activate Alibaba Cloud - East China 2 (Shanghai) Lakehouse instance as the default data source**":
+* * **Checked (Recommended for new users**): The Alibaba Cloud - East China 2 (Shanghai) Lakehouse will be automatically activated as the default data source, no manual configuration required.
+  * **Unchecked**: The system will not automatically activate the Lakehouse instance in the East China 2 (Shanghai) region. You can manually add it on the data source management page after the service is activated. Please note that in this case, **DataGPT will not include preset sample data**.
+
+       ![](.topwrite/assets/20250211-192304.jpeg =618)
+
+^
+
+* Click "Activate", and after a short wait, you can enter the usage interface
+
+       ![](.topwrite/assets/20250211-193618.jpeg =787)
+
+^

package/bin/skills/lakehouse-doc-en/references/release-note-0414-1.1.md

@@ -0,0 +1,56 @@
+This release introduces a series of new features, enhancements, and fixes. Please note that these updates will be rolled out gradually to the following regions. Deployment will complete within one to two weeks of the release date, with timing depending on your region.
+
+* **China Regions**
+
+  * Alibaba Cloud (Shanghai)
+  * Tencent Cloud (Shanghai/Beijing/Guangzhou)
+  * AWS (Beijing)
+
+* **International Regions**
+
+  * Alibaba Cloud (Singapore)
+  * AWS (Singapore)
+
+## New Features
+
+* [Web Search](https://www.singdata.com/documents/web_search) (**Web Search**):
+
+  * Added the WebSearch tool, enabling the Agent to invoke external search engines during data analysis to retrieve real-time information (currently only available to select customers; whitelist activation required).
+* [Scheduled Tasks](https://www.singdata.com/documents/scheduled_task):
+
+  * Supports conversational creation of scheduled analysis tasks via ASK AI (e.g., "Execute at 6 PM daily to check DataGPT usage"). The system automatically performs the analysis at the configured time and delivers results via email.
+* [Table Rendering Optimizations](https://www.singdata.com/documents/table_rendering):
+
+  * Supports describing table styles using natural language, automatically generating data tables with complex layouts including grouping, aggregation, color marking, fixed headers, and column sorting.
+* [Row-Level Permissions](https://www.singdata.com/documents/row_level_permission):
+
+  * Added row-level permission management. Administrators can define permission rules and apply them to specific users, controlling the range of data rows accessible to each user.
+
+## Improvements
+
+* **Session Memory Instructions**:
+
+  * Optimized in-session context memory, making the Agent more focused within a single session and improving multi-turn conversation coherence and accuracy.
+* **Skills Mechanism**:
+
+  * Introduced the Skills mechanism to address declining Agent performance and rising costs caused by an increasing number of tools and features, improving overall analysis quality and response efficiency.
+* **Chart Computation Optimization**:
+
+  * Optimized chart computation tool effects, revised chart title generation logic, and improved `save chart` compatibility with model hallucination issues.
+* **Dashboard Optimizations**:
+
+  * Improved dashboard creation/editing effects and enhanced the stability of the dashboard list retrieval tool.
+
+## **Bug Fixes**
+
+* Fixed a bug where document Q&A content was missing.
+
+* Fixed an issue where JDBC connections did not support custom parameters.
+
+* Fixed a JDBC URL whitespace issue.
+
+* Fixed a bug where chart computation time was not displayed.
+
+* Fixed a dashboard list tool bug.
+
+^

package/bin/skills/lakehouse-doc-en/references/release-note-2026-05-19.md

@@ -0,0 +1,64 @@
+^
+
+**Product Upgrade Notice**: Starting with this release, DataGPT is officially renamed to **Analytics Agent**. Product capabilities remain unchanged; only the product name has been upgraded. Subsequent documentation and UI will gradually switch to the new name.
+
+This release introduces a series of new features, enhancements, and fixes. Please note that these updates will be rolled out gradually to the following regions. Deployment will complete within one to two weeks of the release date, with timing depending on your region.
+
+* **China Regions**
+
+  * Alibaba Cloud (Shanghai)
+  * Tencent Cloud (Shanghai/Beijing/Guangzhou)
+  * AWS (Beijing)
+
+* **International Regions**
+
+  * Alibaba Cloud (Singapore)
+  * AWS (Singapore)
+
+## New Features
+
+* [Row-Level Permissions](https://www.singdata.com/documents/row_level_permission) Enhancements:
+  * When configuring row-level permissions, administrators can now dynamically obtain permission scopes from other data tables using expressions (e.g., automatically retrieving a user's assigned store list from a store table), in addition to manually entering fixed values. Suitable for scenarios where permission values change frequently or need to follow business data updates automatically.
+* **Dashboard Version Management**:
+  * Each modification to a dashboard chart via ASK AI conversation automatically generates a version. Users can view thumbnail previews and modification history of historical versions in the version panel, with support for one-click preview and restore. Solves the pain point of "can't undo after breaking things" — revert at any time if unsatisfied.
+* **Open API**:
+  * Supports initiating data analysis requests and retrieving results via API, making it easy to embed Analytics Agent's analysis capabilities into internal enterprise systems, workflows, or custom applications for automated data Q&A.
+* **Automatic Chart Error Correction**:
+  * When AI generates charts, the system automatically detects rendering errors and attempts to fix them, reducing cases of "chart display anomalies" — users can get correct visualization results without manual retries.
+* **Scheduled Task Email: Chinese/English Bilingual and Multiple Recipients**:
+  * Scheduled analysis report emails now support Chinese/English language adaptation and can be sent to multiple recipients simultaneously.
+* **Chart Auto-Refresh Configuration**:
+  * Dashboard charts now support setting an auto-refresh interval (default: 24 hours). The system automatically updates chart data at the configured time without manual refresh.
+
+## Improvements
+
+* **GPT-5.5 Model Support**:
+  * Compatible with the GPT-5.5 model, enhancing Agent reasoning capability and answer quality.
+
+* **Response Speed Optimization**:
+  * Removed the summary tool and disabled streaming summary output, reducing unnecessary processing steps and improving overall response speed.
+  * Large dataset creation no longer requires waiting for page load completion; the system processes it automatically in the background, with creation progress viewable on the page at any time.
+
+* **Knowledge Search Parallelization**:
+  * Knowledge search now supports parallel execution of multiple keywords, significantly reducing knowledge retrieval latency and improving Agent knowledge citation efficiency.
+
+* **ML Analysis Capability Integration**:
+  * Consolidated ML-related tools into a single Skill, optimizing the invocation effectiveness and stability of machine learning analysis.
+
+## Bug Fixes
+
+* Fixed an issue where dataset attribute fields could be missing.
+
+* Fixed an issue where `replace` in subqueries was not taking effect and Chinese column names were not properly quoted.
+
+* Fixed validation errors caused by missing table names when referencing columns in metric validation.
+
+* Fixed an issue where the knowledge editing tool was outputting results in Chinese.
+
+* Fixed ECharts formatter syntax errors causing chart rendering failures.
+
+* Fixed an issue where KG Service unavailability affected frontend operations.
+
+* Fixed time format compatibility — now supports the `yyyy-MM-dd HH:mm:ss` format.
+
+^

package/bin/skills/lakehouse-doc-en/references/release-notes.md

@@ -0,0 +1 @@
+# Release Notes History

package/bin/skills/lakehouse-doc-en/references/releasenotes.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/releasenotesupdata.md

@@ -0,0 +1 @@
+^