@clickzetta/cz-cli-darwin-arm64 0.3.95 → 0.3.100

package/bin/skills/lakehouse-doc-en/references/intro-supported-features.md

@@ -0,0 +1,48 @@
+# Clickzetta Lakehouse: High-Performance Integrated Online and Offline Service
+
+## Overview
+
+Clickzetta Lakehouse is a high-performance integrated data management platform designed to simplify enterprise data management architecture. With features such as real-time and offline import, processing, and analysis, users can easily handle and analyze large-scale data. Additionally, Clickzetta Lakehouse adopts a cloud-native architecture, characterized by intelligence and openness, providing users with rich interfaces and comprehensive product solutions.
+
+## Integrated Real-Time and Offline
+
+Clickzetta Lakehouse supports integrated real-time and offline capabilities, enabling users to process both real-time and offline data on the same platform. Here are some use cases:
+
+1. **Real-Time Data Import**: Users can import real-time data streams into Clickzetta Lakehouse for real-time analysis. For example, e-commerce platforms can monitor order data in real-time to quickly respond to market changes.
+2. **Offline Batch Import**: Users can batch import historical data into Clickzetta Lakehouse for offline analysis. For instance, financial institutions can import historical transaction data for risk management and compliance reporting.
+3. **Real-Time Data Processing**: Clickzetta Lakehouse supports real-time data processing based on materialized views, allowing users to generate reports and dashboards in real-time. For example, logistics companies can track the status of goods in real-time and generate real-time reports for management decision-making.
+4. **Offline Analysis**: Users can perform high-performance online analysis on offline data without moving the data. For example, retailers can analyze historical sales data to develop more effective marketing strategies.
+
+## Cloud-Native Architecture
+
+Clickzetta Lakehouse adopts a cloud-native architecture with the following features:
+
+1. **Serverless**: Serverless architecture allows for on-demand use and release of computing resources, with pay-as-you-go pricing. Users do not need to worry about underlying resource management and maintenance, allowing them to focus on data analysis and business innovation.
+2. **Storage-Compute Separation**: Uses object storage as the persistent storage layer, with independent scaling of compute resources. Users can flexibly adjust computing resources based on business needs, reducing costs.
+3. **Multi-Cluster Architecture**: Supports rapid creation/automatic suspension/destruction/elastic scaling of multiple virtual computing resources based on load, avoiding resource contention among different businesses; supports vertical and horizontal scaling of a single virtual computing resource to meet different scenarios such as single-job computing power enhancement and multi-concurrency assurance.
+
+## Intelligent Features
+
+Clickzetta Lakehouse provides the following intelligent features:
+
+1. **Real-Time Write Service**: Independent real-time write service that automatically scales resources based on business load, without user management.
+2. **Automatic Storage Optimization**: Automatically performs data consolidation, index optimization, and cache optimization to improve data storage efficiency and query performance.
+3. **Intelligent Optimization**: Uses algorithms to analyze job running characteristics and perform intelligent optimization, continuously improving performance and cost.
+4. **Autopilot**: Elastic computing resources sense load for automatic start/stop and elastic concurrency, reducing resource usage costs while ensuring query quality under critical business concurrency.
+
+## Openness
+
+Clickzetta Lakehouse has the following openness features:
+
+1. **Rich User Interfaces**: Provides tools such as JDBC, SDK, and CLI, integrating with a wide range of ecosystem tools, making it easy for users to integrate into existing environments.
+2. **Ecosystem Engine Integration**: Supports users in accessing Clickzetta Lakehouse data using open-source ecosystem standard storage formats and metadata access interfaces, and using open-source computing engines (such as Spark) for data preparation and algorithm analysis scenarios.
+
+## Comprehensive Product Solutions
+
+Clickzetta Lakehouse natively integrates data integration, data development and operations, and data asset management tools, allowing users to develop data applications through Clickzetta Lakehouse without complex integration work. Users can use Clickzetta Lakehouse in the following ways:
+
+1. **Data Integration**: Use Clickzetta Lakehouse's data integration tools to import data from various data sources into the platform.
+2. **Data Development and Operations**: Utilize Clickzetta Lakehouse's computing capabilities to develop data processing and analysis tasks, achieving data processing, transformation, and analysis.
+3. **Data Asset Management**: Use Clickzetta Lakehouse's data asset management tools for metadata management, data quality management, and data lifecycle management.
+
+In summary, Clickzetta Lakehouse provides users with a high-performance, easy-to-use, and flexible integrated data management platform, helping users easily handle various data processing and analysis needs.

package/bin/skills/lakehouse-doc-en/references/inverted-index.md

@@ -0,0 +1,350 @@
+# Inverted Index
+**[Preview Release] This feature is currently in public preview.**
+
+Introduction to the Inverted Index Principle
+1. Basic Concepts
+The inverted index consists of two parts:
+* **Dictionary**: A list that stores all the unique words (or phrases) that have appeared in the entire document collection.
+* **Posting List**: For each word in the dictionary, there is a corresponding posting list that records the document IDs of all documents containing that word and the positions where the word appears in the documents.
+2. Construction Process
+    -  **Tokenization**: The process of splitting the document content into words or phrases.
+    -  **Normalization**: Processing the tokenized results, such as converting to lowercase, removing stop words, stemming, etc.
+    -  **Building the Dictionary**: Adding the processed words to the dictionary and assigning a unique ID to each word.
+    -  **Building the Posting List**: For each document, recording the IDs and positions of all words in the document, and associating this information with the word IDs in the dictionary to form the posting list.
+3. Query Process
+When a user submits a query request, the search engine performs the following steps:
+
+    -  **Query Parsing**: Breaking down the user's query into words or phrases.
+    -  **Dictionary Lookup**: Looking up the IDs of these words or phrases in the dictionary.
+    -  **Retrieving the Posting List**: Retrieving the document IDs of all documents containing these words from the posting list based on the word IDs.
+    -  **Merging Results**: Merging the retrieval results of different words according to the logic in the query (such as AND, OR, NOT, etc.) to determine the final document list.
+
+4. Application Scenarios
+The inverted index is mainly used in the field of full-text search, such as search engines, document retrieval systems, etc. It can quickly respond to user queries and provide efficient retrieval services.
+
+### Feature Introduction
+
+Each row of the Table corresponds to a document in Elasticsearch, and each column corresponds to a field in the document. Therefore, using the inverted index can quickly locate the rows containing the keyword, achieving the purpose of accelerating the WHERE clause. At the same time, as mentioned above, the dictionary and posting list will also occupy Lakehouse storage, so you will see that the data size is higher than that of tables without an inverted index.
+
+1. Supports full-text search for string types, supporting match_all, match_any, match_phrase, match_phrase_prefix, match_regexp functions. The performance will be faster than like. Currently, Lakehouse uses the inverted index for like in some scenarios, but it is recommended to use full-text search functions.
+
+2. Accelerates ordinary equality and range queries
+   * Supports fast filtering for string and numeric types with =, !=, >, >=, <, <=
+
+3. Supports comprehensive logical combinations
+   * Supports AND, OR, NOT pushdown
+
+4. Flexible and fast index management
+
+   * Creating an Inverted Index: Supports defining an inverted index when creating a table, and adding an inverted index to an existing table. If the index is created on an existing table, the inverted index needs to be added to the existing data. You can execute the build index command.
+   * Building an Inverted Index: The inverted index needs to be added to the existing data. Building the index will start an SQL task and consume computing resources.
+   * Listing All Inverted Indexes: View all inverted indexes on a table.
+   * Viewing Inverted Index Details: View details of the index, including size, etc.
+   * Deleting an Inverted Index: Executing drop index will succeed immediately, deleting the index metadata information. However, the index storage information will not be deleted immediately.
+
+5. Currently supported data types for the inverted index:
+
+* Numeric types (TINYINT, SMALLINT, INT, FLOAT, DOUBLE)
+* Date types (DATE, TIMESTAMP)
+* String types (STRING, VARCHAR, CHAR) When the inverted index field is of string type, you need to specify tokenization. The currently supported tokenization types are:
+  * **keyword**: This type of field will not be tokenized. The entire string is treated as a single root word and stored in the inverted index. For searches, the complete field value needs to be provided for exact matching.
+  * **english**: A tokenizer designed for English, recognizing continuous ASCII letters and numbers, and converting the text to lowercase. Suitable for processing English content, ignoring non-alphanumeric characters.
+  * **chinese**: A Chinese text tokenization plugin that recognizes Chinese and English characters, filters out punctuation, and converts English letters to lowercase. Suitable for processing mixed Chinese and English text.
+  * **unicode**: A tokenizer based on the Unicode text segmentation algorithm, capable of recognizing text boundaries in multiple languages, effectively splitting text into words, and converting letters to lowercase. Suitable for processing text in a multilingual environment.
+    You can use the tokenize(input[, option]) function to test tokenization. An example of using tokenize to test tokenization is as follows:
+```SQL
+--Use keyword tokenizer
+SELECT TOKENIZE('Lakehouse inverted index',map('analyzer', 'keyword')) as toke;
++-------------------------------------+
+|                toke                 |
++-------------------------------------+
+| ["Lakehouse inverted index"]        |
++-------------------------------------+
+--Use chinese tokenizer, not supported yet
+SELECT TOKENIZE('Lakehouse inverted index',map('analyzer', 'chinese')) as toke;
++----------------------------------+
+|              toke                |
++----------------------------------+
+| ["lakehouse","inverted","index"] |
++----------------------------------+
+--Use unicode tokenizer
+SELECT TOKENIZE('Lakehouse inverted index',map('analyzer', 'unicode')) as toke;
++----------------------------------+
+|              toke                |
++----------------------------------+
+| ["lakehouse","inverted","index"] |
++----------------------------------+
+--Use english tokenizer
+SELECT TOKENIZE('Lakehouse inverted index',map('analyzer', 'english')) as toke;
++----------------------------------+
+|               toke               |
++----------------------------------+
+| ["lakehouse","inverted","index"] |
++----------------------------------+
+```
+### Scenarios Where Inverted Index Cannot Be Optimized
+* In most cases, inverted index does not significantly improve the performance of queries with execution times in sub-seconds.
+* External tables are not supported
+* Does not support forced conversion of table columns, for example
+```SQL
+-- Can be queried with acceleration, data type in the table is string
+where string_col='10086';
+-- Convert the value to be matched
+where string_col=cast(10086 as string);
+
+-- Cannot be queried with acceleration, because the table column is forcibly converted
+where cast(string_col as int)=10086 ;
+```
+### Using Inverted Index Query
+
+#### Inverted Index Functions
+
+The function requires the tokenization to be consistent with the table; otherwise, the inverted index cannot be used to accelerate the query. The analyzer supports the auto parameter, which can automatically map the tokenization in the table when the auto parameter is added.
+
+
+| Function Name                                  | Function                                                    | Example                                                                                                                           |
+| ---------------------------------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| tokenize(input\[, option])                     | Tokenize, returns an array                                   | tokenize('a b', map('analyzer', 'english')) = \["a", "b"]                                                                         |
+| match\_all(input, query\[, option])            | Match all, first tokenize the query, then tokenize the input. The tokenized result of the input must contain all the tokenized results of the query. Returns bool | match\_all('a b c', 'b a', map('analyzer', 'english')) = true                                                                     |
+| match\_any(input, query\[, option])            | Match any, the tokenized result of the input only needs to contain any element of the tokenized result of the query. Returns bool | match\_any('a b c', 'd a', map('analyzer', 'english')) = true                                                                     |
+| match\_phrase(input, query\[, option])         | Match phrase, based on match\_all, the order of the matched results must be consistent and continuous with the order of the tokenized results of the query | match\_phrase('a b c', 'a b', map('analyzer', 'english')) = truematch\_phrase('a b c', 'a c', map('analyzer', 'english')) = false |
+| match\_phrase\_prefix(input, query\[, option]) | Match phrase prefix, the tokenization result of input, the first n-1 elements follow the same matching rule as match\_phrase, and the last element follows the prefix matching rule | match\_phrase\_prefix('a b cd', 'b c', map('analyzer', 'english')) = true                                                         |
+| match\_regexp(input, query\[, option])         | Match regular expression, the tokenization result of input, using the regular expression of query, any match is sufficient                 | match\_regexp('a b cd', 'c.\*', map('analyzer', 'english')) = true                                                                |
+
+#### Specific Cases
+
+Create an inverted index table
+```SQL
+create table bulkload_data(id int,data string,INDEX data_index (data) INVERTED PROPERTIES('analyzer'='unicode'));
+
+* Full-text search
+
+```SQL
+--Match All
+select * from bulkload_data where match_all(data,'Elfriede Heaney',map('analyzer', 'unicode'));
++------------------------------------------------------------------------------------------------------------------------------------------+
+|                                                                   data                                                                   |
++------------------------------------------------------------------------------------------------------------------------------------------+
+| {"address":"Apt. 423 78018 Wisozk Meadow, West Marge, WV 16958","name":"Elfriede Heaney","email":"jamar.schoen@gmail.com"}               |
+| {"address":"Suite 654 89305 Dan Drive, Haiview, AZ 55461","name":"Elfriede Heaney","email":"kristofer.upton@yahoo.com"}                  |
+| {"address":"1133 Cartwright Orchard, Port Jonathon, UT 71589-4026","name":"Elfriede Heaney","email":"douglass.nitzsche@yahoo.com"}       |
+| {"address":"115 Avery Mountains, New Elfriede, TN 83686-3466","name":"Clarence Heaney","email":"emmanuel.lockman@yahoo.com"}             |
+| {"address":"Suite 342 631 Konopelski Hollow, East Chingview, UT 79212","name":"Elfriede Heaney DDS","email":"mikel.keebler@hotmail.com"} |
+| {"address":"415 Elfriede Row, New Adriene, SC 90250","name":"Jefferson Heaney","email":"len.price@yahoo.com"}                            |
++------------------------------------------------------------------------------------------------------------------------------------------+
+--Match Phrase
+select * from bulkload_data where match_phrase(data,'Elfriede Heaney',map('analyzer', 'unicode'));
++------------------------------------------------------------------------------------------------------------------------------------------+
+|                                                                   data                                                                   |
++------------------------------------------------------------------------------------------------------------------------------------------+
+| {"address":"Apt. 423 78018 Wisozk Meadow, West Marge, WV 16958","name":"Elfriede Heaney","email":"jamar.schoen@gmail.com"}               |
+| {"address":"Suite 654 89305 Dan Drive, Haiview, AZ 55461","name":"Elfriede Heaney","email":"kristofer.upton@yahoo.com"}                  |
+| {"address":"1133 Cartwright Orchard, Port Jonathon, UT 71589-4026","name":"Elfriede Heaney","email":"douglass.nitzsche@yahoo.com"}       |
+| {"address":"Suite 342 631 Konopelski Hollow, East Chingview, UT 79212","name":"Elfriede Heaney DDS","email":"mikel.keebler@hotmail.com"} |
++------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+* Accelerate ordinary value queries
+select * from bulkload_data where data='{"address":"Apt. 423 78018 Wisozk Meadow, West Marge, WV 16958","name":"Elfriede Heaney","email":"jamar.schoen@gmail.com"}'
++----------------------------------------------------------------------------------------------------------------------------+
+|                                                            data                                                            |
++----------------------------------------------------------------------------------------------------------------------------+
+| {"address":"Apt. 423 78018 Wisozk Meadow, West Marge, WV 16958","name":"Elfriede Heaney","email":"jamar.schoen@gmail.com"} |
++----------------------------------------------------------------------------------------------------------------------------+
+```
+
+### Inverted Index Billing
+
+* Storage Resources: Inverted indexes will create additional inverted index files outside of the data files. The storage fees for index files and data files will be collected based on the unified storage size.
+
+## Create Inverted Index
+
+### Define Index When Creating Table
+
+#### Syntax
+
+```SQL
+CREATE TABLE table_name(
+  columns_difinition,
+  INDEX index_name (column_name) INVERTED [COMMENT '']  PROPERTIES('analyzer'='english|chinese|keyword|unicode'),
+  INDEX index_name (column_name) INVERTED [COMMENT '']  PROPERTIES('analyzer'='english|chinese|keyword|unicode'),
+...
+  INDEX index_name (column_name) INVERTED [COMMENT '']  PROPERTIES('analyzer'='english|chinese|keyword|unicode')
+);
+```
+
+**columns\_difinition**: Defines the field information of the table, the last field must be separated by a comma
+
+**INDEX**: Keyword
+
+**index\_name**: Custom name for the index
+
+**column\_name**: Name of the field to add the index to
+
+**INVERTED**: Keyword, indicates an inverted index
+
+**COMMENT**: Specifies the description information of the index
+
+**PROPERTIES**: Specifies the parameters of the INDEX, currently supports specifying tokenization. Numerical and date types do not need to specify properties, if it is a string type, tokenization must be specified
+
+* Tokenization parameters: 'analyzer'='english|chinese | keyword|unicode', used to specify the text tokenization strategy, suitable for different languages and needs.
+  * **keyword**: This type of field will not be tokenized. The entire string is treated as a single root word and stored in the inverted index. When searching, the complete field value needs to be provided for an exact match.
+  * **english**: A tokenizer designed specifically for English, recognizing only consecutive ASCII letters and numbers, and converting text to lowercase. Suitable for processing English content, ignoring non-alphanumeric characters.
+  * **chinese**: Chinese text tokenization plugin, recognizing both Chinese and English characters, filtering out punctuation, and converting English letters to lowercase. Suitable for processing mixed Chinese and English text.
+  * **unicode**: A tokenizer based on the Unicode text segmentation algorithm, capable of recognizing text boundaries in multiple languages, effectively splitting text into words, and converting letters to lowercase. Suitable for text processing in a multilingual environment.
+
+#### **Example**
+
+```SQL
+CREATE TABLE inverted_index_test(
+  id int,
+  name string,
+  INDEX id_index (id) INVERTED  ,
+  INDEX name_index (name) INVERTED  PROPERTIES('analyzer'='keyword')
+);
+```
+
+### Adding Inverted Index to Existing Table
+
+#### Syntax
+
+```SQL
+CREATE  INVERTED INDEX [IF NOT EXISTS] index_name ON TABLE 
+[schema].table_name(col_name)
+[COMMENT 'comment']
+PROPERTIES('analyzer'='english|chinese|keyword|unicode')
+```
+
+**INVERTED**: Index type, inverted index
+
+**index\_name**: Table name, located under schema, index name under schema must be unique
+
+**col\_name**: Column name, only supports single column
+
+**PROPERTIES**: Specifies the parameters of the INDEX, currently supports specifying tokenization. Numerical and date types do not need to specify properties, if it is a string type, tokenization must be specified
+
+#### Description
+
+Executing CREATE INDEX is only effective for new data. To index existing data, please use the BUILD INDEX command.
+
+#### **Example**
+
+```SQL
+CREATE TABLE inverted_index_test(
+  id int,
+  name string
+);
+
+
+CREATE  INVERTED INDEX  id_index ON TABLE 
+public.inverted_index_test(name)
+PROPERTIES('analyzer'='unicode')
+```
+
+## Using Inverted Index Query
+
+### Building Index
+
+Add inverted index to existing data
+
+### Syntax
+-- Syntax 1, by default adds an inverted index to the entire table's existing data
+BUILD INDEX index_name ON [schema].table_name;
+-- Syntax 2, can specify partition, can specify one or more, supports =, !=, >, >=, <, <=
+BUILD INDEX index_name ON table_name WHERE partition_name1 = '1' and partition_name2 = '2';
+```
+
+* index\_name: Specify the name of the inverted index to be added
+* Support for specifying partition construction: One or more can be specified
+
+### Description
+
+Executing BULD INDEX is a synchronous task, and the execution process will consume computing resources. You can check the progress through Job Profile.
+
+When the data volume of the partition table is large, it is recommended to create indexes sequentially by partition.
+
+### Example
+
+```SQL
+BUILD INDEX bulkload_data_index ON public.bulkload_data ;
+```
+
+## List All Inverted Indexes on a Table
+
+The command is used to list the inverted indexes that have been created on the specified table.
+
+### Syntax
+SHOW INDEX FROM [schema].table_name;
+```
+
+### Case
+ show index from public.bulkload_data;
++---------------------+------------+
+|     index_name      | index_type |
++---------------------+------------+
+| bulkload_data_index | inverted   |
++---------------------+------------+
+```
+
+## View Inverted Index Details
+
+This command is used to list the details of the inverted indexes that have been created in the specified table. Adding the keyword `extended` allows you to view the size of the inverted index.
+
+### Syntax
+
+```SQL
+DESC INDEX [EXTENDED]  index_name;
+```
+
+### Example
+```
+desc index bulkload_data_index;
++--------------------+-------------------------+
+|     info_name      |       info_value        |
++--------------------+-------------------------+
+| name               | bulkload_data_index     |
+| creator            | system_admin            |
+| created_time       | 2024-05-27 16:11:23.928 |
+| last_modified_time | 2024-05-27 16:11:23.928 |
+| comment            |                         |
+| index_type         | inverted                |
+| table_name         | bulkload_data           |
+| table_column       | data                    |
++--------------------+-------------------------+
+
+desc index extended  bulkload_data_index;
++--------------------------+--------------------------+
+|        info_name         |        info_value        |
++--------------------------+--------------------------+
+| name                     | bulkload_data_index      |
+| creator                  | system_admin             |
+| created_time             | 2024-05-27 16:11:23.928  |
+| last_modified_time       | 2024-05-27 16:11:23.928  |
+| comment                  |                          |
+| properties               | (("analyzer","unicode")) |
+| index_type               | inverted                 |
+| table_name               | bulkload_data            |
+| table_column             | data                     |
+| index_size_in_data_file  | 0                        |
+| index_size_in_index_file | 0                        |
+| total_index_size         | 0                        |
++--------------------------+--------------------------+
+```
+
+## Delete Inverted Index
+
+### Syntax
+
+```SQL
+DROP INDEX [IF EXISTS] index_name;
+```
+
+Parameter Description:
+
+* `DROP INDEX`: Keyword to delete an index.
+* `IF EXISTS`: Optional parameter, no error is reported if the specified index does not exist.
+* `index_name`: The name of the index to be deleted.
+
+### Description
+
+Executing drop index will succeed immediately and will delete the index metadata. However, it will not immediately delete the index storage information.

package/bin/skills/lakehouse-doc-en/references/inverted_idx_bm25_param.md

@@ -0,0 +1,251 @@
+# Lakehouse Inverted Index BM25 Parameter Tuning
+
+BM25 (Best Matching 25) is one of the most important relevance scoring algorithms in modern full-text search. The Lakehouse inverted index supports BM25 and allows users to adjust key parameters to optimize search effectiveness for different scenarios.
+
+Supported parameters:
+
+`k1` - Term frequency saturation control
+
+* **Value range**: `[0.0, 3.0]`, recommended range `[1.0, 2.0]`
+
+* **Default value**: `1.2`
+
+* **Purpose**: Controls the impact of term frequency on relevance scoring
+
+* **Effect**:
+
+  * **Larger k1**: Greater impact of term frequency on scoring, high-frequency-term documents are more likely to receive high scores
+  * **Smaller k1**: Faster term frequency saturation, reducing the excessive influence of high-frequency terms
+
+`b` - Document length normalization
+
+* **Value range**: `[0.0, 1.0]`
+
+* **Default value**: `0.75`
+
+* **Purpose**: Controls the impact of document length on relevance scoring
+
+* **Effect**:
+
+  * **b = 0**: Completely ignore document length, short and long documents treated equally
+  * **b = 1**: Full length normalization, short documents gain significant advantage
+  * **b = 0.75**: Balanced length influence, suitable for most scenarios
+
+In the Lakehouse, add the following statement before SQL queries:
+
+```
+SET cz.storage.parquet.inverted.index.similarity.bm25={"k1": 1.2, "b": 0.75} 
+```
+
+to control search effectiveness.
+
+## Example:
+
+### Step 1: Data Preparation
+
+```
+-- 1. CREATE TEST TABLE
+
+CREATE TABLE bm25_demo_table (
+    id INT,
+    title STRING,
+    content_en STRING,
+    content_cn STRING,
+    doc_type STRING
+);
+
+-- 2. Create index and insert data
+CREATE INVERTED INDEX idx_content_cn_score ON TABLE bm25_demo_table(content_cn) 
+     PROPERTIES("analyzer"="chinese", "support_score"="true")
+;
+CREATE INVERTED INDEX idx_content_en_score ON TABLE bm25_demo_table(content_en) 
+     PROPERTIES("analyzer"="english", "support_score"="true")
+;
+
+INSERT INTO bm25_demo_table VALUES
+-- Short document series: high keyword density
+(1, 'AI Intro', 'AI technology.', 'Artificial intelligence technology.', 'short'),
+(2, 'AI Applications', 'AI AI applications in business.', 'Artificial intelligence artificial intelligence applications in business.', 'short'),
+
+-- Medium document series: medium keyword density  
+(3, 'AI Development History', 'The history of AI technology spans decades. AI researchers developed machine learning algorithms. Modern AI systems use deep learning techniques.', 'The development history of artificial intelligence technology spans decades. Artificial intelligence researchers developed machine learning algorithms. Modern artificial intelligence systems use deep learning techniques.', 'medium'),
+(4, 'AI Real-world Applications', 'AI technology revolutionizes industries. Companies implement AI solutions for automation. AI chatbots improve customer service efficiency.', 'Artificial intelligence technology revolutionizes industries. Companies implement artificial intelligence solutions for automation. Artificial intelligence chatbots improve customer service efficiency.', 'medium'),
+
+-- Long document series: low density but multiple occurrences
+(5, 'AI Technology Overview', 'Artificial intelligence represents one of the most transformative technologies of our time. AI systems can process vast amounts of data, recognize patterns, and make predictions with remarkable accuracy. The field of AI encompasses machine learning, deep learning, natural language processing, and computer vision. Modern AI applications span across healthcare, finance, transportation, and entertainment industries. As AI technology continues to evolve, researchers are exploring new frontiers in artificial general intelligence and quantum computing integration with AI systems.', 'Artificial intelligence represents one of the most transformative technologies of our time. Artificial intelligence systems can process vast amounts of data, recognize patterns, and make predictions with remarkable accuracy. The field of artificial intelligence encompasses machine learning, deep learning, natural language processing, and computer vision. Modern artificial intelligence applications span across healthcare, finance, transportation, and entertainment industries. As artificial intelligence technology continues to evolve, researchers are exploring new frontiers in artificial general intelligence and quantum computing integration with artificial intelligence systems.', 'long'),
+
+-- Distractor document: does not contain target keywords
+(6, 'Blockchain Technology', 'Blockchain technology provides decentralized solutions. Cryptocurrency mining requires significant computational power. Smart contracts automate business processes.', 'Blockchain technology provides decentralized solutions. Cryptocurrency mining requires significant computational power. Smart contracts automate business processes.', 'control'),
+
+-- High-frequency keyword document
+(7, 'AI Intensive Discussion', 'AI AI AI is everywhere. AI development, AI research, AI implementation, AI optimization, AI performance, AI scalability, AI security, AI ethics, AI governance, AI regulation.', 'Artificial intelligence artificial intelligence artificial intelligence is everywhere. Artificial intelligence development, artificial intelligence research, artificial intelligence implementation, artificial intelligence optimization, artificial intelligence performance, artificial intelligence scalability, artificial intelligence security, artificial intelligence ethics, artificial intelligence governance, artificial intelligence regulation.', 'high_freq'),
+
+-- Low-frequency but precise match
+(8, 'Precise AI Definition', 'The definition of AI varies among experts.', 'The definition of artificial intelligence varies among experts.', 'precise')
+;
+```
+
+> ⚠️ **Note**: If data is inserted before index creation, you need to BUILD INDEX for the index to take effect:
+>
+> ```
+> BUILD INDEX idx_content_cn_score ON bm25_demo_table;
+> BUILD INDEX idx_content_en_score ON bm25_demo_table;
+> ```
+
+We have constructed a test set containing different types of documents:
+
+* **Short documents** (2-20 characters): High keyword density
+* **Medium documents** (100-200 characters): Medium keyword density
+* **Long documents** (500+ characters): Low keyword density but multiple occurrences
+* **High-frequency document**: Many repeated keywords
+* **Precise match**: Small number but precise keyword matches
+
+### Step 2: Search Verification:
+
+#### English Search Test
+
+**Case 1: Default parameters** `(using defaults)` - **Balanced configuration**
+
+Search keyword: `"AI"`
+
+```
+SELECT score (),
+    id,
+    title,
+    doc_type,
+    LENGTH(content_en) AS len,
+    REGEXP_COUNT (content_en, 'AI') AS ai_count,
+    SUBSTRING(content_en, 1, 50) AS preview
+FROM bm25_demo_table
+WHERE match_any(content_en, 'AI')
+ORDER BY score () DESC, id
+LIMIT 50;
+```
+
+| score()    | id | title                | doc\_type  | len | ai\_count | preview                                            |
+| ---------- | -- | -------------------- | ---------- | --- | --------- | -------------------------------------------------- |
+| 0.16484952 | 7  | AI Intensive Discussion | high\_freq | 174 | 13        | AI AI AI is everywhere. AI development, AI researc |
+| 0.14495456 | 2  | AI Applications      | short      | 31  | 2         | AI AI applications in business.                    |
+| 0.13709007 | 4  | AI Real-world Applications | medium     | 138 | 3         | AI technology revolutionizes industries. Companies |
+| 0.13152356 | 1  | AI Intro             | short      | 14  | 1         | AI technology.                                     |
+| 0.13141003 | 3  | AI Development History | medium     | 145 | 3         | The history of AI technology spans decades. AI res |
+| 0.1138232  | 8  | Precise AI Definition | precise    | 42  | 1         | The definition of AI varies among experts.         |
+| 0.10628956 | 5  | AI Technology Overview | long       | 580 | 5         | Artificial intelligence represents one of the most |
+
+**Key observations**:
+* High-frequency-term document ranks first (as expected)
+* Short documents gain significant length advantage
+* Long documents, despite having more matches, are penalized by length
+
+**Case 2**: `b=0.0` - **Ignore document length**
+
+```
+SET cz.storage.parquet.inverted.index.similarity.bm25={"k1": 1.2, "b": 0.0};
+SELECT score (),
+    id,
+    title,
+    doc_type,
+    LENGTH(content_en) AS len,
+    REGEXP_COUNT (content_en, 'AI') AS ai_count,
+    SUBSTRING(content_en, 1, 50) AS preview
+FROM bm25_demo_table
+WHERE match_any(content_en, 'AI')
+ORDER BY score () DESC, id
+LIMIT 10;
+```
+
+| score()        | id    | title                   | doc\_type  | len     | ai\_count | preview                                                |
+| -------------- | ----- | ----------------------- | ---------- | ------- | --------- | ------------------------------------------------------ |
+| 0.16691414     | 7     | AI Intensive Discussion | high\_freq | 174     | 13        | AI AI AI is everywhere. AI development, AI researc     |
+| **0.14703354** | **5** | **AI Technology Overview** | **long**   | **580** | **5**     | **Artificial intelligence represents one of the most** |
+| 0.13022971     | 3     | AI Development History  | medium     | 145     | 3         | The history of AI technology spans decades. AI res     |
+| 0.13022971     | 4     | AI Real-world Applications | medium     | 138     | 3         | AI technology revolutionizes industries. Companies     |
+| 0.11395099     | 2     | AI Applications         | short      | 31      | 2         | AI AI applications in business.                        |
+| 0.08287345     | 1     | AI Intro                | short      | 14      | 1         | AI technology.                                         |
+| 0.08287345     | 8     | Precise AI Definition   | precise    | 42      | 1         | The definition of AI varies among experts.             |
+
+**Key observations**:
+* **Long documents benefit significantly**: jumping from last place to second place
+* **Short documents lose their advantage**: length advantage is completely eliminated
+* **Pure term-frequency ranking**: ranking is entirely based on content relevance
+
+^
+
+### Chinese Search Test
+
+**Case 1: Default parameters** `(using defaults)` - **Balanced configuration**
+
+```
+SET cz.storage.parquet.inverted.index.similarity.bm25={"k1": 1.2, "b": 0.75};
+SELECT score (),
+    id,
+    title,
+    doc_type,
+    LENGTH(content_en) AS len,
+    REGEXP_COUNT (content_en, 'Artificial intelligence') AS ai_count,
+    SUBSTRING(content_en, 1, 50) AS preview
+FROM bm25_demo_table
+WHERE match_any(content_en, 'Artificial intelligence')
+ORDER BY score () DESC, id
+LIMIT 10;
+```
+
+| score()     | id | title                | doc\_type  | len | ai\_count | preview                                            |
+| ----------- | -- | -------------------- | ---------- | --- | --------- | -------------------------------------------------- |
+| 0.16505295  | 7  | AI Intensive Discussion | high\_freq | 89  | 13        | Artificial intelligence artificial intelligence artificial intelligence is everywhere. Artificial intelligence development, artificial intelligence research, artificial intelligence implementation, artificial intelligence optimization, artificial intelligence performance |
+| 0.13975275  | 2  | AI Applications      | short      | 16  | 2         | Artificial intelligence artificial intelligence applications in business. |
+| 0.13214059  | 4  | AI Real-world Applications | medium     | 54  | 3         | Artificial intelligence technology revolutionizes industries. Companies implement artificial intelligence solutions for automation. Artificial intelligence chatbots improve customer service efficiency |
+| 0.1313231   | 1  | AI Intro             | short      | 7   | 1         | Artificial intelligence technology. |
+| 0.12937927  | 3  | AI Development History | medium     | 51  | 3         | The development history of artificial intelligence technology spans decades. Artificial intelligence researchers developed machine learning algorithms. Modern artificial intelligence systems use deep learning techniques |
+| 0.12602468  | 5  | AI Technology Overview | long       | 161 | 7         | Artificial intelligence represents one of the most transformative technologies of our time. Artificial intelligence systems can process vast amounts of data, recognize patterns, and make predictions with remarkable accuracy |
+| 0.113299355 | 8  | Precise AI Definition | precise    | 16  | 1         | The definition of artificial intelligence varies among experts. |
+
+## Parameter Tuning Strategy
+
+### Scenario-Based Parameter Recommendations
+
+#### 1. News Search Scenario
+
+```sql
+-- Optimized configuration: balance term frequency and document length
+SET cz.storage.parquet.inverted.index.similarity.bm25={"k1": 1.2, "b": 0.8};
+```
+
+* **Applicable scenarios**: News articles, blog content
+* **Optimization goal**: Give more attention to short documents, preventing long articles from burying important short news
+* **Effect**: Improve the visibility of short news
+
+#### 2. Academic Literature Search
+
+```sql
+-- Optimized configuration: reduce length penalty, emphasize content quality
+SET cz.storage.parquet.inverted.index.similarity.bm25={"k1": 1.0, "b": 0.3};
+```
+
+* **Applicable scenarios**: Papers, research reports, technical documentation
+* **Optimization goal**: Long documents are not overly penalized, content richness matters more
+* **Effect**: Detailed document rankings improve
+
+#### 3. Product Search Scenario
+
+```sql
+-- Optimized configuration: strengthen exact matching
+SET cz.storage.parquet.inverted.index.similarity.bm25={"k1": 0.8, "b": 0.5};
+```
+
+* **Applicable scenarios**: E-commerce search, product catalogs
+* **Optimization goal**: Exact matches take priority over high-frequency matches
+* **Effect**: Reduce the impact of term frequency stacking on rankings
+
+#### 4. Social Media Search
+
+```sql
+-- Optimized configuration: prioritize short content
+SET cz.storage.parquet.inverted.index.similarity.bm25={"k1": 1.5, "b": 0.9};
+```
+
+* **Applicable scenarios**: Microblogs, comments, short video descriptions
+* **Optimization goal**: Short content first, quickly locate key information
+* **Effect**: Short content rankings significantly improved
+
+^