@clickzetta/cz-cli-darwin-arm64 0.3.95 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/analytics_cluster_best_practices.md

@@ -0,0 +1,367 @@
+# Analytical Cluster Planning, Configuration, and Best Practices
+
+## Analytical Scenario Requirements
+
+![](.topwrite/assets/real_time_analytics_system.png)
+
+•**Data Freshness: Real-time collection and processing of rapidly changing data**
+
+•**Low Latency Queries: Provide second-level, sub-second query latency to ensure SLA**
+
+•**Support for Complex Queries: Meet the analytical requirements of different business logics and data scales**
+
+•**High Concurrency: Multi-concurrency for end users**
+
+•**Historical Detail Queries: Support real-time historical comparison, historical retrieval, and exploratory analysis at the detail level**
+
+## Singdata Lakehouse Product Capabilities in Analytical Scenarios
+
+![](.topwrite/assets/cz_platform_arch_ap.png)
+
+| **Category**   | **Product Capability**                                                                                     | **Applicable Scenarios**                                                                                                                         |
+| -------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| Storage       | Columnar Storage                                                                                         | Most ETL, OLAP analytical scenarios                                                                                                                 |
+|          | Local Cache                                                                                  | Local Cache is provided by the local SSD of the computing resource Virtual Cluster, accelerating data I/O                                                                                   |
+|          | Cache Strategy: LRU, Preload Cache                                                                    | LRU: Automatically cache hot data based on Query identification Preload Cache: Incrementally cache new data proactively based on a fixed range of data objects set by analysts, ensuring SLA                                         |
+| Data Layout | Partition                                                                                           | Large-scale tables with continuous time-based writes                                                                                                                                |
+|          | Bucketing + System Compaction Service automatically re-clustering to maintain new data bucketing aggregation                                                   | Commonly used in ETL+BI scenarios for filtering based on cluster key conditions                                                                                                    |
+|          | SORT                                                                                         | Improve the efficiency of sort key condition filtering, commonly used in BI analysis scenarios                                                                                                     |
+|          | Lightweight Index: bloomfilter index                                                                    | Specify key point check filtering, can be used in combination with Cluster key, commonly used in BI analysis scenarios                                                                                         |
+| Import/Export    | COPY command                                                                                       | Batch export and import based on file method                                                                                                                    |
+|          | Real-time Ingestion Service                                                                      | Provides real-time write API, supports Append/Upsert real-time writing, aimed at scenarios requiring data freshness                                                                                        |
+| Compute       | CZ SQL Engine provides two different load-optimized execution modes: ETL DAG execution optimization, MPP+Pipeline analysis execution optimization mode                          | Differentiated use through different types of Virtual Clusters: General Purpose, Analytics |
+| User Interface     | Studio Web-UI                                                                                | Ad-hoc analysis, data processing tasks                                                                                                                  |
+|          | JDBC                                                                                         | Connects to a wide range of BI/client tool ecosystems                                                                                                                  |
+|          | SQL Alchemy/Java SDK/Go SDK                                                                  | Python, Java, Go programming interfaces                                                                                                               |
+
+# Reasonable Planning and Use of Virtual Cluster
+<Notes>
+
+This chapter will introduce the product features and planning methods for using Singdata Lakehouse's elastic computing resources.
+
+## Introduction to Virtual Cluster Features
+<Notes>
+## Compute Cluster (Virtual Cluster, VC) <a name="compute-cluster"></a>
+
+The Compute Cluster (Virtual Cluster, VC) is a compute resource cluster service provided by Singdata Lakehouse. The compute cluster provides the necessary resources such as CPU, memory, and local temporary storage (SSD medium) to execute query analysis in the Lakehouse. These three resources are bundled into a compute scale unit called the Lakehouse Compute Unit. CRU serves as an abstract, standardized measurement unit for the size and performance of compute resources.
+
+![](.topwrite/assets/image_1718616076626.png)
+
+Singdata Lakehouse measures the actual usage duration of different specifications of compute resources (measured in seconds), and the different specifications of CUR and their unit prices are as follows:
+
+| **Lakehouse Virtual Cluster Specification** | **CRU** | **MAX\_CONCURRENCY Default Value** |
+| ------------------------------------------- | ------- | ---------------------------------- |
+| X-Small                                     | 1       | 8                                  |
+| Small                                       | 2       | 8                                  |
+| Medium                                      | 4       | 8                                  |
+| Large                                       | 8       | 8                                  |
+| X-Large                                     | 16      | 8                                  |
+| 2X-Large                                    | 32      | 8                                  |
+| 3X-Large                                    | 64      | 8                                  |
+| 4X-Large                                    | 128     | 8                                  |
+| 5X-Large                                    | 256     | 8                                  |
+
+Singdata Lakehouse supports elastic scaling of compute resources in two dimensions:
+
+* Vertical: Adjust the number of CRUs by modifying the size of the Virtual Cluster to increase or decrease the computing power and performance of the specified cluster.
+* Horizontal: Adjust the concurrency query capability by modifying the number of cluster replicas to expand or reduce the number of Cluster replicas.
+
+## Plan the Number and Size of Compute Resources Based on Load
+
+### Best Practices for Using Compute Resources
+
+* Achieve load isolation through multiple compute resources
+
+  * Use multiple independent Virtual Clusters to support different workloads. Create different compute resources and allocate them to different users or applications based on different needs such as periodic ETL, online business reports, and analyst data analysis to avoid SLA degradation caused by resource contention among different businesses or personnel.
+
+* Set automatic pause duration reasonably
+
+  * The compute resources of Lakehouse have the characteristics of automatically pausing when idle, automatically resuming when a job is initiated, and billing based on the actual resource usage duration (not billing during the pause phase). This can be set through the AUTO\_SUSPEND\_IN\_SECOND parameter, with a default of automatically pausing after 600 seconds of no queries.
+
+  * Recommendations for different scenarios (considering data cache and resource resume time in two dimensions)
+
+    * For low-frequency Ad-hoc queries or ETL jobs, set AUTO\_SUSPEND\_IN\_SECOND to around 1 minute.
+    * For high-frequency analysis scenarios, set AUTO\_SUSPEND\_IN\_SECOND to 10 minutes or more to keep the data relied on by the analysis always in the cache and avoid the latency degradation caused by frequent VC start and stop.
+
+* Choose the resource type of the Virtual Cluster based on the load type
+
+  * **General Engine (VCLUSTER\_TYPE=GENERAL)** can complete all the tasks done by the analysis engine when the requirements for concurrency and job timeliness in analysis scenarios are not strong. It is optimized for batch processing workloads, prioritizes throughput, supports resource contention & sharing, and is suitable for ELT, data import (such as external table reading and writing), and Ad-hoc queries.
+  * **Analytics Engine (VCLUSTER\_TYPE=ANALYTICS)** supports an analysis engine with a specified number of job concurrency. It is suitable for scenarios with strong guarantees on query latency and concurrency capabilities, such as reporting, dashboards, and critical business data products.
+
+* Use elastic concurrency capabilities to support critical data applications
+* The analytical Virtual Cluster supports multiple Replica features, with the default number of Replicas being 1. This can be modified during or after creation. Each Replica supports a specific query concurrency (determined by the MAX\_CONCURRENCY parameter, which defaults to 8). The total concurrency supported by each Virtual Cluster is: Replica Number \* MAX\_CONCURRENCY
+![](.topwrite/assets/image_1718616850891.png)
+  * Concurrency settings: Increasing concurrency may reduce the query performance of a single query when the size of the Virtual Cluster is fixed. In most cases, it is recommended that the concurrency limit does not exceed 100. The concurrency value is recommended to be a multiple of 8. An example is as follows:
+```SQL
+--Create an analytical VC, enable elastic concurrency (min_replicas=1 max_replicas=3)
+create vcluster if not exists your_ap_vc vcluster_size='XSMALL' vcluster_type='Analytics'  AUTO_RESUME=TRUE AUTO_SUSPEND_IN_SECOND=300 min_replicas=1 max_replicas=3 ;
+—Modify the replicas parameter to increase the number of horizontal scaling replicas to enhance elastic concurrency
+alter vcluster your_ap_vc set max_replicas=6;
+--Modify the concurrency size of a single replica, suitable for fixed report scenarios with simple calculation logic and small data scale
+alter vcluster your_ap_vc set MAX_CONCURRENCY=32 ;
+--Modify the specification size of the Virtual Cluster, affecting the specification concurrency of all elastic concurrency created replicas to improve the query performance of a single query
+alter vcluster your_ap_vc set vcluster_type='MEDIUM';
+
+3. Query and configure MAX_CONCURRENCY
+- Query MAX_CONCURRENCY
+Use the following statement to query the configuration parameters of the specified VC, where you can find the current MAX_CONCURRENCY configuration value of the VC cluster:
+DESC VCluster <name>
+- Configure MAX_CONCURRENCY
+Use the following statement to configure the MAX_CONCURRENCY of the specified VC. For analytical VC clusters, the MAX_CONCURRENCY configuration applies to all instances within it.
+ALTER VCluster <name> SET MAX_CONCURRENCY = {desired positive integer}
+```
+### Lakehouse Virtual Cluster Capacity Design Recommendations for Different Scenarios
+
+The main business characteristics of the current Singdata Lakehouse supported business scenarios are as follows:
+
+| **Scenario/Workload** | **Scenario Requirements**                     | **Resource Requirements** | **Data Scale and Query Latency Requirements**          | **Query Characteristics** | **Concurrency Characteristics**    |
+| --------------------- | --------------------------------------------- | ------------------------- | ------------------------------------------------------ | ------------------------- | ---------------------------------- |
+| Fixed Reports         | High concurrency, stable low latency          | Small + Elastic Scaling   | • Small, query GB scale • TP95 <3 seconds              | Simple logic, fixed pattern on processed data | For a wide range of frontline personnel, 10\~100 or more |
+| Ad-hoc                | Detailed data supports interactive exploration and analysis | Medium                    | • Medium, 100GB\~10TB • Most queries expected within 10 seconds, tolerant of complex query latency | Relatively complex logic | For professional analysts, fewer (related to organization size) |
+| ETL                   | Stable output as planned, cost-sensitive, elastic demand under burst situations (e.g., data replenishment, temporary output) | Large                     | • Large, TB\~PB+ level • Hourly level, SLA requirement for output completion time | Complex logic, processing type: many inputs and outputs | Low, multi-tasking for single user, accepts job queuing |
+
+### Virtual Cluster Resource Specification Reference
+
+This document plans to provide sample resource specification designs from the dimensions of business load type, execution frequency, job concurrency, data processing scale, and SLA requirements for reference.
+
+* **Load Type**: Includes ETL, Ad-hoc, Reporting, real-time analysis (Operational Analytics, etc.) with different data warehouse workload characteristics
+* **Execution Frequency**: Execution cycle of related loads
+* **Concurrency**: Number of jobs to be executed simultaneously
+* **Data Processing Scale**: Average data volume scanned per job
+* **Job Latency SLA**: Expected task delivery time in business
+
+Example:
+
+| Business Scenario                | Load Type             | Execution Frequency | Job Concurrency | Data Processing Scale | VCluster Type | Job Latency SLA         | VCluster Name              | VCluster Size |
+| -------------------------------- | --------------------- | ------------------- | --------------- | --------------------- | -------------- | ----------------------- | -------------------------- | ------------- |
+| ETL Scheduling Jobs              | Near real-time offline processing | Hourly               | 1                 | 1 TB                 | General Purpose         | 15 Min                    | etl\_vc\_hourly            | Medium        |
+|                                  | T+1 Offline Processing | Daily                | 1                 | 10 TB                | General Purpose         | 4 Hours                   | etl\_vc\_daily             | X-Large       |
+| Tableau/FineBI                   | Ad-Hoc Analytics      | Ad hoc               | 8                 | 1 TB                 | General Purpose         | <1 MinTP90:<5s            | analytics\_vc              | X-Large       |
+| Data Application Products     | Applications     | On demand | 8    | 100 GB | Analytical     | <3 seconds         | application\_vc            | Medium        |
+|                           |                  | On demand | 96   | 100 MB | Analytical     | <3 seconds         | application\_vc            | Medium        |
+| ClickZetta Web-UI<Data Development and Testing> | Ad-Hoc Analytics | Ad hoc    | 8    | 3 TB   | General-purpose | < 1 MinTP90:<15 s | analytics\_vc(shared with Analyst BI exploration) | X-Large       |
+
+Note: The above examples are for reference only. In actual business environments, resource planning should be based on the actual business results, as they are influenced by various factors such as query complexity, data model (partitioning, bucketing), and cache hit rate.
+
+## Configuration of Virtual Computing Resources Based on Concurrency Requirements
+
+Singdata Lakehouse analytical virtual computing clusters (Virtual Cluster) are suitable for scenarios with high query latency requirements such as ad-hoc, multi-concurrent fixed reports, and real-time analysis. Analytical virtual computing clusters offer the following configuration options:
+
+| **Configuration Options** | **Description**                                                                                           |
+| -------- | ------------------------------------------------------------------------------------------------ |
+| Specification       | The size of the computing resource instance within the virtual cluster. The specification ranges from XS to 5XLarge, with each level increase doubling the computing resources.                                                  |
+| Type       | General-purpose or analytical, here we use analytical.                                                                               |
+| Maximum Concurrency    | The number of concurrent queries supported by a single computing resource instance. When the maximum concurrency of a single instance is exceeded, new queries will queue or be allocated to other computing resource instances within the virtual cluster.                                           |
+| Minimum Instances    | The number of cluster instances when the virtual cluster initially starts, default is 1.                                                                            |
+| Maximum Instances    | The maximum number of instances that the virtual cluster can use. When the maximum instances > minimum instances, it means that new computing instances will be automatically scaled out based on whether the number of concurrent requests received by the virtual cluster exceeds the total maximum concurrency of all computing instances, until the current number of instances = maximum instances. |
+| Current Instances   | The number of instances currently used by the virtual cluster.                                                                                   |
+
+The following diagram describes the relationship between the parameters related to the analytical virtual cluster:
+
+![](.topwrite/assets/image_1718616481037.png)
+
+Vertical Scaling: The instance specification is an option for vertical scaling of the virtual cluster. Adjusting the instance specification can provide more computing power for a single job to improve job performance.
+
+Horizontal Scaling: Elastic capability aimed at improving cluster concurrency. Analytical clusters are designed for horizontal scaling scenarios, providing a multi-instance architecture within a single cluster. Each analytical cluster sets the initial number of instances and the upper limit of horizontal scaling by setting the minimum and maximum instances.
+
+In actual business scenarios, data products facing users often have SLA requirements, such as P99 query latency under 100 concurrency being less than 2 seconds. At the same time, it is hoped that computing resources are fully utilized to minimize resource costs while meeting business needs. As shown in the figure below:
+
+![](.topwrite/assets/image_1718616492933.png)
+
+In specific business scenarios, we hope to find the minimum resource requirements that meet the SLA performance requirements. This article, from a practical perspective, combines example scenarios to determine reasonable resource specification configurations through the following steps.
+
+### Scenario Example
+The data processed by the data platform is packaged as data products (such as Dashboards), supporting up to 20 concurrent user queries externally. Each user request for a data product generates an average of 8 data platform queries (an average of 8 metrics per Dashboard), which translates to 160 concurrent queries on the data platform. The requirement is that P99 query latency should be less than 1.5 seconds.
+
+Data queries have the following characteristics:
+
+* Dataset size for Dashboard queries: around 1GB.
+* Characteristics of Dashboard queries: mainly point queries, filtering, and aggregation, without JOINs.
+
+This document uses the SSB FLAG 1G dataset and query statements to simulate the above scenario, to verify the cluster configuration and performance under concurrent queries.
+
+### Step 1: Find the single instance specification size that balances performance and cost
+
+1. Run queries using clusters of different sizes (this document tests several sizes from Large to XSMALL) to find the smallest resource specification that does not significantly degrade performance.
+
+As shown in the figure below, the SSBF-1G benchmark maintains over 80% of the performance of the Large specification even at the XSMALL specification. Here, the XSMALL specification is chosen to support data products.
+
+Note: Based on test results, it is recommended to use Large size for 100G data scale by default, Medium for 10G, and Small or XSmall for 1G.
+
+![](.topwrite/assets/image_1718616512495.png)
+
+### Step 2: Test the query SLA and QPS under different levels of concurrency for a single instance
+
+As shown in the figure below, using an XSMALL specification cluster, at 8 concurrent requests, the QPS is high, and the P95/P99 query performance is good.
+
+![](.topwrite/assets/image_1718616524873.png)
+
+### Step 3: Find the maximum number of instances that meet both the SLA and actual business concurrency needs through horizontal scaling
+
+As introduced earlier, the number of concurrent queries a cluster can support = maximum number of instances \* maximum concurrency per instance. In principle, once the single instance specification size is determined, horizontally scaled instances should have the same query SLA and QPS.
+
+Based on the test data, this document uses an XSMALL specification, with a maximum of 10 instances in a virtual cluster, to meet the need for 160 concurrent queries. The query response time P95=1111 milliseconds, P99=1280.65 milliseconds.
+
+![](.topwrite/assets/image_1718616534247.png)
+
+### Summary
+
+This section uses a small-scale dataset with simple concurrent query scenarios to illustrate how to reasonably set the virtual compute cluster specifications and related parameters of Singdata Lakehouse to meet SLA requirements.
+
+To simplify this reference process, some empirical values are provided below based on test information for reference.
+
+| Load Type                        | Data Scale  | VC Instance SIZE | Single Instance QPS Reference | Scenario Example                                                                                                                                                             |
+| --------------------------- | ----- | -------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Fixed Reports/Serving Queries (mainly simple filtering or aggregation)   | 1GB   | XSmall   | 25       | Demand Example: 1. Result data scale is 10GB 2. Query analysis is simple filtering or aggregation 3. SLA requires support for 200 concurrent queries 4. Average response time: 1 second Cluster configuration example: 1. Analytical VC, SIZE choose Medium specification 2. Based on a single Medium having 20 QPS for simple queries on 10GB data, it is estimated that 10 instances can provide 200 QPS, so the maximum number of instances is set to 10. |
+|                             | 10GB  | Medium   | 20       |                                                                                                                                                                  |
+|                             | 100GB | Large    | <10      |                                                                                                                                                                  |
+| Ad-hoc/BI dynamic report generation (with associative analysis, complex calculations) | 1GB   | Small    | 15       |                                                                                                                                                                  |
+|                             | 1GB   | Medium   | 25       |                                                                                                                                                                  |
+|                             | 100GB | XLarge   | <10      |                                                                                                                                                                  |
+
+# Improve the Stability of Key Business Queries Using Preload Cache Mechanism
+
+## **Preload Working Mechanism and Applicable Scenarios**
+
+![](.topwrite/assets/image_1718616561238.png)
+
+Suitable Usage Scenarios
+
+• Cache existing data with computing resources proactively to avoid latency on the first access to historical data
+
+• Trigger cache upon real-time data write-in to reduce query latency jitter after new data is written to disk
+
+## Usage Method
+
+Set the data object range for PRELOAD for specified computing resources, supporting enumeration and wildcard methods.
+```SQL
+-- Set the data objects to PRELOAD when creating the compute cluster
+
+create vcluster if not exists <name>
+
+auto_preload_in_second=600
+
+PRELOAD_TABLES="s1.t1,s1.t2";
+
+-- Modify the compute cluster to add or modify PRELOAD data objects
+
+ALTER VCLUSTER <name> SET PRELOAD_TABLES='schema01.tb01,schema01.tb02,schema01.tb03';
+```
+View the current Preload configuration of computing resources:
+
+![](.topwrite/assets/image_1718616600431.png)
+
+## **Preload eliminates query jitter for new data in real-time analysis scenarios**
+
+
+
+| Uncached file ratio | TPCH 100G (0%) | TPCH 100G (1%) | TPCH 100G (5%) | TPCH 100G (10%) | TPCH 100G (20%) | TPCH 100G (30%) |
+| ----------- | -------------- | -------------- | -------------- | --------------- | --------------- | --------------- |
+| Query latency | 9300 ms        | 15418ms        | 20311ms        | 19027ms         | 18815ms         | 20841ms         |
+
+
+
+# Computing Resources and Job Monitoring
+
+### Computing Resource Status
+
+#### Monitoring via **Web-UI**
+
+![](.topwrite/assets/image_1718616622179.png)
+
+#### Monitoring via **SQL API**
+```SQL
+desc vcluster quning_ap_vc;
+```
+| name                              | QUNING\_AP\_VC                                                          |
+| --------------------------------- | ----------------------------------------------------------------------- |
+| creator                           | demo\_project                                                           |
+| created\_time                     | 2024-01-28 23:15:13.625                                                 |
+| last\_modified\_time              | 2024-03-27 09:59:48.212                                                 |
+| comment                           |                                                                         |
+| vcluster\_size                    | XSMALL                                                                  |
+| vcluster\_type                    | ANALYTICS                                                               |
+| state                             | RUNNING                                                                 |
+| scaling\_policy                   | STANDARD                                                                |
+| min\_replicas                     | 1                                                                       |
+| max\_replicas                     | 4                                                                       |
+| preload\_tables                   | qn\_demo.lineitem,upserted\_ssb\_1g.lineorder\_flat,leanken\_preload.\* |
+| current\_replicas                 | 1                                                                       |
+| max\_concurrency\_per\_replica    | 8                                                                       |
+| auto\_resume                      | true                                                                    |
+| auto\_suspend\_in\_second         | 1800                                                                    |
+| running\_jobs                     | 0                                                                       |
+| queued\_jobs                      | 0                                                                       |
+| query\_runtime\_limit\_in\_second | 259200                                                                  |
+| error\_message                    |                                                                         |
+
+### Job Monitoring
+
+Scenario: Job failure, long-running jobs, monitoring jobs for specific users/VC, job runtime and cost statistics, monitoring specific TAGs.
+
+#### Monitoring through **Web-UI**
+
+![](.topwrite/assets/image_1718616681911.png)
+
+Add filter conditions: such as filtering specific jobs/job groups by TAG
+```SQL
+-- Set query_tag at the session level
+set query_tag='aa';
+select 1;
+```
+![](.topwrite/assets/image_1718616691133.png)
+
+#### Monitor Recent Job Execution Success/Queue/Long Running Status via SHOW JOBS Command
+
+The SHOW JOBS command retrieves ongoing and recently completed jobs within the workspace's computing resources.
+```SQL
+SHOW  JOBS  [IN VCLUSTER vc_name] [WHERE where_condition ] [LIMIT num];
+```
+Note:
+
+List the most recently executed jobs, including both completed and running jobs. Users can use this command to list jobs they have permission for. By default, it displays tasks submitted in the last 7 days, with a maximum query limit of 10,000 entries.
+```SQL
+-- View the recent job history within the Workspace
+show jobs limit 100;
+
+-- Filter by conditions: view jobs with specified computing resources, submitted by a specific user, and with execution time exceeding a certain duration
+show jobs where vcluster_name='quning_ap_vc'  
+and creator='demo_project' and (end_time - start_time) > interval 0.1 second  
+limit 100;
+```
+
+| name                              | QUNING\_AP\_VC                                                          |
+| --------------------------------- | ----------------------------------------------------------------------- |
+| creator                           | demo\_project                                                           |
+| created\_time                     | 2024-01-28 23:15:13.625                                                 |
+| last\_modified\_time              | 2024-03-27 09:59:48.212                                                 |
+| comment                           |                                                                         |
+| vcluster\_size                    | XSMALL                                                                  |
+| vcluster\_type                    | ANALYTICS                                                               |
+| state                             | RUNNING                                                                 |
+| scaling\_policy                   | STANDARD                                                                |
+| min\_replicas                     | 1                                                                       |
+| max\_replicas                     | 4                                                                       |
+| preload\_tables                   | qn\_demo.lineitem,upserted\_ssb\_1g.lineorder\_flat,leanken\_preload.\* |
+| current\_replicas                 | 1                                                                       |
+| max\_concurrency\_per\_replica    | 8                                                                       |
+| auto\_resume                      | true                                                                    |
+| auto\_suspend\_in\_second         | 1800                                                                    |
+| running\_jobs                     | 0                                                                       |
+| queued\_jobs                      | 0                                                                       |
+| query\_runtime\_limit\_in\_second | 259200                                                                  |
+| error\_message                    |                                                                         |
+
+#### View Historical **Execution Jobs** through INFORMATION\_SCHEMA
+
+The data dictionary view of INFORMATION\_SCHEMA has a delay of about 10 minutes, which is suitable for analyzing long-term historical job data. For online real-time job analysis, use the SHOW JOBS command to obtain it.
+```SQL
+-- View job history in the current workspace
+select * from information_schema.job_history limit 100;
+
+-- View job history in the current Lakehouse service instance
+select * from sys.information_schema.job_history limit 100;
+```
+
+![](.topwrite/assets/image_1718616725992.png)
+

package/bin/skills/lakehouse-doc-en/references/analyze-table.md

@@ -0,0 +1,51 @@
+# Description
+
+The ANALYZE TABLE statement is used to collect statistics for a specific table or all tables within a specified schema. Statistics include table size, row count, etc., which the query optimizer uses to generate the best query plan. Although these statistics may become outdated as data changes, the query optimizer can still use this outdated information to generate effective query plans. In Lakehouse, the statistics obtained by executing the desc command may be delayed, so executing the ANALYZE TABLE statement can obtain the latest statistics.
+
+# Syntax
+```SQL
+-- Collect statistics for a specified table
+ANALYZE TABLE table_name
+   COMPUTE STATISTICS [NOSCAN | FOR COLUMNS col1 , col2... | FOR ALL COLUMNS ]
+
+-- Collect statistics for all tables in a specified schema
+ANALYZE TABLES [IN schema_name] COMPUTE STATISTICS [NOSCAN]
+```
+# Parameters
+
+Required parameters:
+
+* table_name: The name of the table for which to collect statistics. Supports specifying tables, dynamic tables, materialized views
+
+Optional parameters:
+
+* NOSCAN: Only collect the size of the table (in bytes), without scanning the entire table
+* FOR COLUMNS col1, col2, ...: Collect column statistics for each specified column
+* FOR ALL COLUMNS: Collect statistics for all columns
+* IN schema_name: Specify the name of the schema to execute, if not specified, the current schema is used by default
+
+# Notes
+- When executing the ANALYZE TABLE statement, the table may be scanned, which may affect query performance. It is recommended to perform this operation when the system load is low
+- When using the [real-time interface](<java_reference/realtime-upload.md>) in append mode, the ANALYZE TABLE statement will not count uncommitted data. Therefore, there may be cases where the result of select count(*) from table is less.
+
+#  Example
+
+1. Collect statistics for a specified table (without scanning the table), only collect the size of the table (in bytes), without scanning the entire table:
+```SQL
+ANALYZE TABLE sales COMPUTE STATISTICS NOSCAN;
+```
+2. Collect column statistics for the specified table:
+```SQL
+ANALYZE TABLE customers COMPUTE STATISTICS FOR COLUMNS customer_id, customer_name;
+```
+3. Collect statistics for all tables under the specified schema:
+```SQL
+ANALYZE TABLES IN sales_schema COMPUTE STATISTICS;
+```
+
+4. Collect statistics for all columns of the specified table:
+
+```SQL
+ANALYZE TABLE orders COMPUTE STATISTICS FOR ALL COLUMNS;
+```
+。

package/bin/skills/lakehouse-doc-en/references/answer-accuracy-improve.md

@@ -0,0 +1,179 @@
+# Analytics Agent Answer Accuracy Improvement - Best Practices
+
+In response to the most frequently asked question: What methods are available to improve answer accuracy? Which scenarios are suitable for each method? How to manage them?
+
+## 1. What Capabilities Improve Accuracy?
+
+Analytics Agent (formerly known as DataGPT) provides **4 core capabilities** to give the model more business information input and improve answer accuracy:
+
+### 1. Table/Column Schema Annotations
+
+![](/.topwrite/assets/image_1779160426141.png)
+
+| Configuration Item              | Purpose                                                                         | Example                                           |
+| ------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------- |
+| Table Description               | Tells the model what this table represents                                      | "Daily sales transaction records for all stores in Hong Kong region" |
+| Column Alias                    | Helps the model understand different names for the same field                   | Column name rev\_amt → alias "revenue", "income"            |
+| Column Description              | Explains the business meaning and value rules of a column                       | "payment\_type: payment method, values include cash/card/octopus" |
+| Hidden Column                   | When a table has many fields, some columns are irrelevant for business analysis (e.g., system primary keys, ETL timestamps, internal codes). Hiding them prevents them from participating in AI Q&A and analysis, reducing the chance of the model selecting the wrong column | Applicable scenarios:
Tables with many technical fields that interfere with column selection; or sensitive fields that should not be referenced by AI |
+| Virtual Column                  | When schema changes require AI to adapt its understanding, create virtual columns through SQL expressions without modifying the underlying data table | Use case: When raw fields need transformation to help AI understand business meaning |
+
+### 2. Knowledge (Knowledge Base / Analysis Documents)
+
+Business background knowledge provided in file form, injected into conversation context via RAG retrieval.
+
+Content suitable for the knowledge base:
+
+* Business rule descriptions
+* Data dictionary / code mapping tables
+* Industry terminology explanations
+* Analysis methodology / calculation formula specifications
+* FAQ frequently asked questions
+* Content recalled only in specific scenarios
+
+**Core Value**: Helps the model understand the "business logic behind the data," not just the data structure.
+
+### 3. Metrics / Answer Builder
+
+Predefined calculation formulas supporting alias matching, ensuring consistent metric definitions. Specifically:
+
+* Metrics: Used to define aggregation calculations (e.g., SUM, AVG)
+* Answer Builder: Used to define multi-table JOINs and complex filter conditions
+
+| Element       | Description                                                                                           | Screenshot                                            |
+| ------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
+| Metric Name   | e.g., "Average Transaction Value"                                                                     | ![](/.topwrite/assets/image_1779160566375.png =215)   |
+| Calculation   | SUM(revenue) / COUNT(DISTINCT order\_id)                                                              | ![](/.topwrite/assets/image_1779160586259.png =206)   |
+| Alias         | Helps the model understand different names for the same concept                                       | ![](/.topwrite/assets/image_1779160603309.png =204)   |
+| Auto-Generation | Basic aggregate metrics (e.g., SUM, COUNT) can now be auto-generated for selected tables. Context-aware metric insights and auto-generation will be available in future releases, enabling smarter metric recommendations | ![](/.topwrite/assets/image_1779160824613.png =218)   |
+
+**Core Value**: Ensures key business metrics are calculated consistently and accurately, preventing different SQL from being generated each time
+
+### 4. Domain Prompt
+
+System prompt at the analysis domain level, equivalent to giving the model a "role setting" and "behavioral guidelines."
+
+* Role setting: "You are a data analyst for the Hong Kong food & beverage industry"
+* Answer guidelines: "Monetary amounts default to HKD, date format uses YYYY-MM-DD"
+* Business constraints: "When calculating same-store growth, only include stores that have been open for at least 12 months"
+* Output preferences: "For trend-related questions, prioritize using line charts"
+
+**Core Value**: Set global rules to avoid repeating prerequisites in every conversation
+
+***
+
+## 2. Which Capability for Which Scenario? (Scenario → Capability Mapping)
+
+| Scenario/Problem Type                       | Recommended Capability            | Why                                       |
+| -------------------------------------------- | --------------------------------- | ----------------------------------------- |
+| Model selects wrong table or column          | Table/Column Schema Annotations   | Unclear descriptions cause matching errors |
+| Filter values are wrong (e.g., using full name but data stores abbreviations) | Column Value Index                 | Establish fuzzy matching                  |
+| Calculation formula is wrong (e.g., "gross profit" algorithm is incorrect) | Metrics                          | Predefined exact formulas                  |
+| Complex calculation logic (multi-table JOIN + conditional filtering) | Answer Builder                   | Provides SQL templates                     |
+| Model does not understand business terms (e.g., "table turnover rate") | Knowledge Base                   | Provides terminology documentation         |
+| Background knowledge needed to answer (e.g., "why did March revenue decline") | Knowledge Base                   | Provides business context documentation    |
+| Need to repeat prerequisites every time      | Domain Prompt                     | Set once, effective globally               |
+| Same metric has multiple names               | Metric Alias / Column Alias       | Support matching with multiple expressions  |
+
+### Decision Flowchart
+
+```
+
+User question is inaccurate
+
+↓
+
+Is it "wrong table/column selection"? → Yes → Improve Schema Annotations + Column Value Index
+
+↓ No
+
+Is it "wrong calculation logic"? → Yes → Create Metrics or Answer Builder
+
+↓ No
+
+Is it "not understanding business concepts"? → Yes → Add Knowledge Base documents
+
+↓ No
+
+Is it "need to repeat prerequisites every time"? → Yes → Configure Domain Prompt
+
+↓ No
+
+Investigate using a combination of the above methods
+
+```
+
+***
+
+## 3. How to Use? (Knowledge Base Management)
+
+### 3.1 Knowledge Base Organization
+
+The new knowledge base uses a hierarchical folder structure (similar to Lark/Feishu knowledge base):
+![](/.topwrite/assets/image_1779161523552.png)
+
+### 3.2 Content Creation Methods
+
+| Method         | Applicable Scenarios                           |
+| -------------- | ----------------------------------------------- |
+| Create directly | Manually write business rules, terminology explanations, etc. |
+| Local upload   | Upload existing PDF/Word/Excel/Markdown documents |
+| Cloud import   | Batch import existing documents from OSS/S3     |
+
+### 3.3 Associate with Analysis Domains (Critical!)
+
+Knowledge base content must be associated with an analysis domain for conversations under that domain to retrieve it:
+
+* Can associate at the folder level → child files automatically inherit
+* A single file can be associated with multiple analysis domains
+* Unassociated files will not be retrieved by any conversation
+
+**Best Practices**:
+
+* Create folders by business theme, associate at the folder level
+* General knowledge (e.g., company glossary) associates with all analysis domains
+* Specific business knowledge associates only with the corresponding domain
+
+### 3.4 What Content is Suitable for Knowledge Base?
+
+✅ Content suitable for knowledge base:
+
+| Content Type     | Example                                                    |
+| ---------------- | ---------------------------------------------------------- |
+| Business rules   | Table turnover rate = tables served / total tables; Average transaction value = revenue / customer count |
+| Code mapping     | Store code A01 = Causeway Bay store, A02 = Mong Kok store |
+| Industry terms   | Definitions of "dine-in", "takeaway", "table turnover", "preparation time" |
+| Data definitions | "Valid orders" exclude refunded orders and test orders     |
+| Analysis methods | Same-store growth only counts stores open for 12+ months   |
+| Common FAQ       | "Why don't the report numbers match finance?" → Because the report excludes tax |
+| Background info  | In Q1 2026, the company adjusted stores: 3 new openings, 2 closures |
+
+❌ Content NOT suitable for knowledge base:
+
+| Content Type                | What to Use Instead           |
+| --------------------------- | ----------------------------- |
+| Exact calculation formulas  | → Use the Metrics feature     |
+| Complex SQL templates       | → Use the Answer Builder      |
+| Column aliases/descriptions | → Use Schema Annotations      |
+| Global answer guidelines    | → Use Domain Prompt           |
+
+### 3.5 Management Recommendations
+
+| Recommendation          | Description                                           |
+| ----------------------- | ----------------------------------------------------- |
+| Organize by theme folder | Facilitates management and batch domain association  |
+| Keep documents concise  | Focus each file on one topic; avoid being overly broad |
+| Update regularly        | Sync knowledge base when business rules change        |
+| Naming conventions      | File name reflects content, e.g., "table-turnover-rate-rules.md" |
+| Verify before rollout   | Test a few questions after adding knowledge before broad use |
+
+## 4. Summary: Synergy of the 4 Capabilities
+
+In one sentence:
+
+* Schema Annotations help the model "find the right data"
+* Metrics help the model "calculate the right numbers"
+* Knowledge Base helps the model "understand the business"
+* Domain Prompt helps the model "follow the rules"
+
+Use all four together for the highest accuracy.

package/bin/skills/lakehouse-doc-en/references/application-list.md

@@ -0,0 +1,26 @@
+# Application List
+
+Through the application record function, you can view all the work orders you have submitted, including application content, application status, etc. You can also revoke or copy work orders.
+
+## Prerequisites
+
+You must have already applied for resource permissions. For related operation instructions, see **Permission Application**.
+
+## Operation Steps
+
+Click **Approval > Application** in the left menu to enter the **Application Records** page.
+
+![](.topwrite/assets/image_1736245885982.png)
+
+You can perform the following operations:
+![](.topwrite/assets/image_1736245926683.png)
+
+* **View work order details**: Click a **Work Order ID** in the list. In the pop-up side panel, you can view details of the work order, including application details, requested resources, approval status, etc.
+
+* **Revoke**: For work orders currently under approval, you can perform a revocation.
+
+  * Click the **Operation** column on a list item or the **Revoke** button on the work order details page. In the confirmation dialog, click **Confirm** to revoke the corresponding resource application. Revocation will withdraw all in-progress resource applications within the work order; resources that have already been approved will not be affected.
+
+* **Copy**: Click the **Operation** column on a list item or the **Copy** button on the work order details page to copy all current approval information. You can modify the relevant information as needed and then submit the application.
+
+^