@clickzetta/cz-cli-darwin-x64 0.3.94 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/AI_Gateway.md

@@ -0,0 +1,253 @@
+## 1. Product Overview
+
+### 1.1 What is Model Management
+
+Model Management (AI Gateway) is an enterprise-grade "unified model invocation and governance hub" that integrates rate limiting, operational monitoring, permission isolation, and cost management capabilities by default. It supports aggregating "cloud service provider hosted models" and "enterprise self-owned external models," enabling standardized model invocation, visualized resource governance, and granular cost attribution across multiple scenarios.
+
+### 1.2 Core Values
+
+| Value Dimension  | Description                                                      |
+| ----- | --------------------------------------------------------- |
+| Unified Entry Point  | No need to switch between multiple platforms; one-stop management of hosted/external models, reducing cross-platform collaboration costs                           |
+| Risk Control  | Role-based permission isolation (RBAC) + dynamic rate limiting + API KEY security management, preventing resource abuse and data leaks          |
+| Cost Transparency  | Supports tenant/Endpoint-level quota control, multi-dimensional usage statistics (Token / call count), and cost allocation reports for precise cost attribution |
+| Low-Barrier Invocation | Covers three major scenarios: SQL embedding, DataGPT visualization, and OpenAPI development, meeting the needs of both technical and non-technical users       |
+| Traceability  | Retains model invocation logs, usage details, and permission change records to meet compliance and audit requirements                             |
+
+### 1.3 Core Concepts
+
+| Concept                    | Definition                                                                    | Applicable Scenarios                   |
+| --------------------- | ----------------------------------------------------------------------- | ---------------------- |
+| AI Gateway            | The unified entry point for model invocation and governance, responsible for permission verification, rate limiting, usage statistics, and request forwarding                                      | All model invocation and management scenarios            |
+| Model Endpoint | The unique access identifier of a model, with a unique name within the tenant (Chinese characters not supported); can be invoked in multiple scenarios after registration                                     | An Endpoint must be created before a model can be used |
+| Managed Model                  | Preset models provided by cloud service providers (e.g., Alibaba Cloud Qwen, OpenAI GPT-4o), with Base URL/API KEY managed uniformly by AI Gateway | Enterprises without self-owned models that need to quickly enable general AI capabilities  |
+| External Model                  | Enterprise self-owned cloud vendor MaaS models (e.g., self-built Volcengine large models), requiring manual configuration of Base URL/API KEY                      | Scenarios requiring private deployment or self-owned models, such as finance and government   |
+
+## 2. Prerequisites
+
+### 2.1 Preconditions
+
+1. The cloud service provider Studio service must be activated, and you must have access to the "AI - Model Management" module (when empty, only Instance Admin can see the "New Model" entry);
+2. If using external models: you must obtain the cloud vendor-provided **Base URL** (must comply with the OpenAI API specification) and **API KEY** (must have model invocation permissions) in advance;
+3. If associating business data: you must obtain Lakehouse table read permissions (for SQL invocation scenarios) and AI Function execution permissions (for function invocation scenarios) in advance.
+
+### 2.2 Roles and Permissions Matrix (RBAC)
+
+| Role                 | Core Permissions                                                                  | Target Users            | Typical Operations          |
+| ------------------ | --------------------------------------------------------------------- | --------------- | --------------- |
+| Account Admin      | 1. Initialize Gateway Admin (auto-synced on first use); 2. Handle billing and service activation                         | Enterprise cloud account admin (low-frequency operations)  | Configure initial admin on first enterprise access  |
+| Gateway Admin      | 1. Configure tenant-level quotas and rate limiting policies; 2. Manage all Endpoints/API KEYs; 3. Add/remove admins; 4. View all usage and cost allocation data | Enterprise IT admin, model owner | Enterprise budget allocation, global permission control |
+| Endpoint Admin     | 1. Manage specified Endpoints (view usage, modify config, delete); 2. Grant others permission for that Endpoint                   | Business line owner (e.g., e-commerce, finance)  | Business line model permission allocation, usage monitoring  |
+| ENDPOINT\_VIEWER   | Read-only access to model metadata; cannot invoke                                                          | Operations analyst           | Model invocation development, data analysis generation   |
+| ENDPOINT\_USER     | Can only invoke models                                                                | Development engineer           | Model invocation development, data analysis generation   |
+| ENDPOINT\_OPERATOR | Invoke + modify model configuration                                                           | Development engineer           | Model invocation development, data analysis generation   |
+
+## 3. Core Operation Guide
+
+### 3.1 Endpoint Lifecycle Management (Model Access and Monitoring)
+
+#### 3.1.1 Managed Model Management (Quickly Enable Preset Models)
+
+**Business Value**: No configuration required; directly use cloud service provider preset models, suitable for quickly validating business requirements (e.g., temporary data analysis, Demo development).
+**Steps**:
+
+1. Go to "Model Management -> Endpoint" page, the system displays the list of managed models configured within the tenant by default;
+2. Core operations (varies by role permissions):
+   * Steps:
+     * Go to "Model Management -> Endpoint" page;
+     * Locate the target model in the Endpoint list and perform the following operations:
+       * New Endpoint (Gateway Admin only): Click "New" on the right to create an Endpoint. ![](.topwrite/assets/1.png)
+       * Copy Endpoint: Click "Copy" to duplicate an existing Endpoint, modify its content, and generate a new Endpoint. ![](.topwrite/assets/ScreenShot_2026-03-10_191249_127.png)
+       * Test Connectivity: Click "Test Connection" to trigger a real large model invocation to verify connectivity; this operation consumes a very small amount of Tokens. ![](.topwrite/assets/image_1769764093331.png)
+       * View Details: Click the Endpoint name to view model version, invocation volume, authorized users, and change history. ![](.topwrite/assets/image%20(1).png)
+       * Delete Model (Gateway Admin and Endpoint Admin only): Click "Delete" -> Confirm in the popup by clicking "OK" (deletion is irreversible).
+
+#### 3.1.2 External Model Access (Enterprise Self-Owned Model Integration)
+
+**Business Value**: Supports enterprises in connecting privately deployed or self-owned copyrighted models to AI Gateway for unified management (e.g., financial risk control models, e-commerce recommendation models).
+Access is divided into two scenarios to meet different model specification requirements:
+**Scenario 1: Standard Vendor Access (e.g., Volcengine, Anthropic)**
+Advantage: Models comply with cloud vendor standard API specifications; no custom URL required.
+
+1. Go to "Model Management -> Endpoint -> New -> External Model";
+2. Fill in the configuration:
+   * Enter "Endpoint Name" (unique within the tenant, e.g., "finance-risk-model");
+   * Select "Vendor" (system auto-fills the standard Base URL);
+   * Select "Model Category" (LLM/Embedding/Image Generation);
+   * Enter "Model Version" (e.g., "risk-v2.1");
+   * Fill in "API KEY ID" (API Key information is only required when invoking external models; you can create it in the API KEY interface and view the specific ID from the generated API Key);
+3. AI Gateway Configuration:
+   * Set "Access Rate Limit" (e.g., TPM=1000, RPM=50);
+   * "Access Quota" (e.g., 100,000 Tokens per month);
+4. Click "Test Connection" to verify connectivity (see Chapter 4 for troubleshooting);
+5. After a successful test, click "OK"; the Endpoint will be added to the list.
+   ![](.topwrite/assets/image%20(2).png)
+
+**Scenario 2: Custom Access (OpenAI-Compatible)**
+Advantage: For enterprise self-developed models and third-party non-standard vendor models.
+
+1. Go to "Model Management -> Endpoint -> New -> External Model -> Custom";
+2. Fill in the configuration:
+   * Enter "Endpoint Name" (e.g., "custom-ocr-model");
+   * Fill in "Model Vendor" (e.g., "Enterprise Self-Developed");
+   * Select "Model Category" (e.g., "Image Generation");
+   * Enter "Model Version" (custom, e.g., "20260120-beta");
+   * Fill in "Base URL" (e.g., "https://internal-ocr.example.com/v1/chat/completions");
+   * Fill in "API KEY ID" (API Key information is only required when invoking external models; you can create it in the API KEY interface and view the specific ID from the generated API Key);
+3. Follow steps 3-5 of "Standard Vendor Access" for the remaining steps.
+   ![](.topwrite/assets/image%20(3).png)
+
+#### 3.1.3 Model Monitoring (Real-Time Operational Status)
+
+**Business Value**: Monitor model invocation success rate, latency, and error rate in real time to promptly detect anomalies (e.g., model downtime, rate limit triggered).
+**Steps**:
+
+1. Go to "Model Management -> Endpoint", select the target Endpoint, click "More -> Monitoring";
+   ![](.topwrite/assets/image%20(4).png)
+2. Configure query conditions:
+   * Time Range: Default is the last 1 day, supports 1 minute to 60 days (data granularity varies by range, e.g., 1 minute per point within 1 day, 1 hour per point within 30 days);
+   * Metric Dimensions: Default display includes "Request Count, Success Rate, End-to-End Latency, Total Token Usage";
+3. View Data:
+   * Overview Panel: Displays core KPIs (Total Tokens, Success Rate, Error Rate, Average Latency);
+   * Trend Charts: Line charts show metric changes (e.g., a sudden drop in success rate requires checking model connectivity);
+   * Error Details: Click "Error Rate" to view error types (e.g., rate limiting, insufficient permissions);
+
+### 3.2 API KEY Management (Secure Access Credentials)
+
+**Business Value**: The API KEY is the sole credential for external model invocation; strict management prevents resource abuse and data leaks.
+**Steps (Gateway Admin only)**:
+
+1. Go to "Model Management -> API KEY" page;
+2. Create a New API KEY:
+   * Click "New KEY" -> Enter a name (should be associated with a business scenario, e.g., "ecommerce-recommend-key");
+   * Click "Confirm"; the system generates the KEY;
+3. Daily Management:
+   * Disable/Enable: Click "Disable" (invocation is blocked after disabling, suitable for temporary suspension); click "Enable" when needed;
+   * Delete: Must first stop associated invocations (e.g., SQL/OpenAPI invocations), click "Delete" -> Confirm (KEY becomes invalid immediately);
+   * Filter: Quickly locate target KEYs by "Status (Normal / Disabled)" and "Owner";
+4. Security Notes:
+   * Avoid plaintext storage (e.g., in code repositories, configuration files);
+   * Rotate regularly (recommended every 3 months);
+   * If leaked, delete and recreate immediately, and update all associated invocation configurations;
+     ![](.topwrite/assets/image%20(5).png)
+
+### 3.3 Permission Management (Granular Resource Control)
+
+#### 3.3.1 Gateway Admin Management (Global Permission Control)
+
+**Business Value**: Ensures that only authorized personnel within the enterprise are responsible for global configuration, avoiding permission chaos.
+**Steps (Current Gateway Admin only)**:
+
+1. Go to "Model Management -> Permissions -> Gateway Admin";
+2. Add Admin:
+   * Click "Add" -> Select target users (must be registered within the tenant);
+   * Click "Confirm"; the user immediately obtains full Gateway Admin permissions;
+3. Remove Admin:
+   * Locate the target user in the list and click "Remove";
+   * System validation: (1) The currently logged-in user cannot be removed; (2) At least 1 Admin must remain after removal;
+4. View Records: The page displays the "Username, Addition Time, Operator" for all Admins;
+   ![](.topwrite/assets/image%20(6).png)
+
+#### 3.3.2 Endpoint Permission Management (Business Line Permission Isolation)
+
+**Business Value**: Allocate Endpoint permissions by business line to prevent cross-business resource abuse (e.g., an e-commerce Endpoint can only be invoked by the e-commerce team).
+**Steps (Endpoint Admin only)**:
+
+1. Go to "Model Management -> Permissions -> Endpoint Permissions";
+2. Authorization:
+   * Click "Add" -> Select target user -> Choose permission type (refer to 2.2 Roles and Permissions);
+   * Select the Endpoints to authorize (multiple selection supported);
+   * Click "Confirm"; the user obtains the corresponding permissions;
+3. Revoke Permissions:
+   * Find the target user and Endpoint in the "Authorized List";
+   * Click "Remove" -> Confirm; permissions take effect immediately;
+     ![](.topwrite/assets/image%20(7).png)
+
+### 3.4 Quota and Usage Management (Cost Control)
+
+#### 3.4.1 Tenant Quota Configuration (Enterprise-Level Budget Control)
+
+**Business Value**: Set an enterprise monthly total Token upper limit to avoid budget overruns (e.g., 5 million Tokens per month covering all business lines).
+**Steps (Gateway Admin only)**:
+
+1. Go to "Model Management -> Usage -> Tenant Quota";
+2. First-Time Configuration:
+   * If empty (prompt "No quota has been set for the current tenant"), click "Set Tenant Quota";
+   * Fill in configuration: Granularity defaults to "Tenant Level", Cycle defaults to "Monthly", Token Quota (e.g., 5000000, i.e., 5 million/month);
+   * Click "Save" and navigate to the quota page to view "Total Quota, Used, Remaining, Daily Usage Trend";
+3. Adjust Quota: Click "Edit Quota" -> Modify the value -> Save (takes effect immediately);
+   ![](.topwrite/assets/image%20(8).png)
+
+#### 3.4.2 Endpoint Quota Configuration (Business Line Budget Allocation)
+
+**Business Value**: Allocate independent quotas for individual Endpoints (e.g., 1 million Tokens per month for the e-commerce recommendation model) to achieve business line cost isolation.
+**Steps (Gateway Admin/Endpoint Admin)**:
+
+1. Go to "Model Management -> Endpoint", select the target Endpoint, click "Edit";
+2. Locate the "AI Gateway -> Access Quota" module;
+3. Fill in configuration: Granularity defaults to "Endpoint Level", Cycle "Monthly", Token Quota (e.g., 1000000);
+4. Key Rule: The actual available quota = the smaller of the tenant quota and the Endpoint quota (e.g., if the tenant has 500,000 remaining and the Endpoint quota is 1,000,000, the actual available is 500,000);
+5. Click "Save" and view quota usage on the Endpoint details page;
+   ![](.topwrite/assets/image%20(9).png)
+
+#### 3.4.3 Usage Statistics Query (Multi-Dimensional Cost Attribution)
+
+**Business Value**: Query usage by Endpoint / User / API KEY dimensions to identify high-cost resources (e.g., abnormal invocation volume by a specific user).
+**Steps (varies by role permissions)**:
+
+1. Go to "Model Management -> Usage -> Usage Statistics";
+2. Configure query conditions:
+   * Time Range: Default is the last 1 month, supports switching to "Last 1 Week / Custom";
+   * Statistical Dimensions:
+     * Regular Users: Only view "their associated Endpoints / Users";
+     * Gateway Admin: View "Endpoint / User / API KEY / Account / Account Details";
+3. View Data:
+   * List View: Displays "Statistical Dimension, Date, Call Count, Input Tokens, Output Tokens, Total Usage";
+   * Chart View: Click "Switch Chart" to view usage trends via line/bar charts;
+4. Download Details (Gateway Admin only):
+   * Click "Download Details" -> Select year and month (supports the last 1 year) -> Confirm;
+     ![](.topwrite/assets/image%20(10).png)
+
+### 3.5 Multi-Scenario Model Invocation (Adapting to Different User Needs)
+
+#### 3.5.1 SQL Invocation (Data Analysts / Development Engineers)
+
+**Business Value**: Invoke models directly in SQL, enabling integrated "data query + AI analysis" (e.g., order text embedding, user review sentiment analysis).
+You can invoke directly in SQL, for example:
+
+```Plain
+select ai_complete('endpoint\:lis\_aliyun\_qwen\_max','What is the capital of China?')
+```
+
+#### 3.5.2 DataGPT Invocation (All Roles)
+
+DataGPT enables AI model invocation through standardized interfaces, allowing rapid model integration and replacement. You simply replace the existing model configuration on the DataGPT model settings page with our dedicated invocation address (Base URL) and access credential (API Key) to switch the model invocation pipeline, leveraging AI Gateway for stable, governable AI capability invocation.
+
+#### 3.5.3 OpenAPI Standard Invocation (Development Scenarios)
+
+* Invocation Address: The "Access Address" in the Endpoint list;
+* API KEY: Obtain from "API KEY Management" (only Gateway Admin can create);
+* Invocation Method:
+  * Go to "Model Management -> Endpoint" page, click the Endpoint name to enter the details page;
+  * At the bottom of the details page, the "Endpoint Invocation Examples" section shows example code for OpenAI-python and REST API-cURL invocation methods.
+
+## 4. Key Notes
+
+1. **Permission Control**: Confirm your role permissions before operating (e.g., regular users cannot delete Endpoints) to avoid operation failures due to insufficient permissions; if you need extended permissions, contact the corresponding Admin (Gateway Admin handles global permissions, Endpoint Admin handles business line permissions).
+
+2. **Data Retention**:
+   * Account detail data: Only retained for the last 1 year; download monthly backups for long-term storage (Gateway Admin operation);
+   * Usage monitoring data: Retained for up to 60 days; regularly take screenshots and archive key metrics (e.g., monthly quota usage).
+
+3. **Model Deletion Risks**:
+   * Endpoints cannot be recovered after deletion. Before deletion, you must: (1) Stop all associated invocations (SQL/OpenAPI/DataGPT); (2) Notify all authorized users;
+   * After a built-in managed model is deleted, contact technical support if you need to use it again (self-recovery is not supported).
+
+4. **API KEY Security**:
+   * Avoid leakage (e.g., plaintext sharing, storage in code repositories);
+   * Rotate regularly (recommended every 3 months);
+   * If leaked, immediately delete and recreate in "API KEY Management" and update all associated configurations.
+
+5. **Quota Effective Rules**: The actual available Endpoint quota = the smaller of the tenant quota and the Endpoint quota. When setting, avoid the Endpoint quota far exceeding the tenant quota (e.g., tenant 1 million/month, Endpoint set to 2 million, actual only 1 million available, resulting in configuration waste).
+
+6. **Compatibility Notes**: External models must comply with the OpenAI API specification, otherwise they cannot be connected (e.g., models that do not support the "messages" parameter require vendor adaptation); when editing model versions, confirm whether downstream invocation systems support the new version (to avoid compatibility issues).

package/bin/skills/lakehouse-doc-en/references/AI_Gateway_Guide.md

@@ -0,0 +1 @@
+^

package/bin/skills/lakehouse-doc-en/references/AI_eco.md

@@ -0,0 +1,34 @@
+# AI Ecosystem
+
+Singdata Lakehouse is deeply integrated with mainstream AI development frameworks and workflow platforms, providing vector storage, full-text search, file storage, and data query capabilities as the data infrastructure for AI applications.
+
+---
+
+## Contents
+
+| Integration | Description |
+|-------------|-------------|
+| [Dify Integration](dify_yunqilakehouse_integration_overview.md) | Use Lakehouse as Dify's vector database and file storage service, supporting RAG knowledge base construction and unstructured data management |
+| [N8N Integration](n8n_ai_workflow_integration.md) | Use Lakehouse nodes in N8N visual workflows, supporting data queries, table data comparison, and multi-database workflow automation |
+| [LangChain Integration](langchain_integration.md) | LangChain Singdata plugin that brings Lakehouse capabilities into LangChain AI applications, supporting vector retrieval and RAG application development |
+| [Unstructured Data ETL Pipeline](unstructured-io.md) | Use Unstructured to convert PDFs, emails, images, and other unstructured data into JSON, write it to Lakehouse, and build a vector index to feed RAG applications |
+| [Datus Data Engineering Agent](datus_lakehouse_integrated_guide.md) | An open-source data engineering agent that provides domain-aware intelligent assistance for analysts and business users, integrating with Lakehouse via MCP Server |
+
+---
+
+## Quick Selection Guide
+
+**I'm building an AI application with Dify and need knowledge base storage**
+→ [Dify Integration](dify_yunqilakehouse_integration_overview.md): configure Lakehouse as a vector database or file storage
+
+**I'm building an automated workflow with N8N and need to operate a database**
+→ [N8N Integration](n8n_ai_workflow_integration.md): install the Lakehouse N8N node, supporting queries and data comparison
+
+**I'm developing an AI application with LangChain and need vector retrieval**
+→ [LangChain Integration](langchain_integration.md): use the LangChain Singdata plugin to connect to Lakehouse
+
+**I have large volumes of PDFs, images, and other unstructured files and need to build RAG**
+→ [Unstructured Data ETL Pipeline](unstructured-io.md): complete pipeline from Unstructured to a Lakehouse vector table
+
+**I want to use an AI Agent to automate data engineering tasks**
+→ [Datus Integration](datus_lakehouse_integrated_guide.md): let the agent operate Lakehouse directly via MCP Server

package/bin/skills/lakehouse-doc-en/references/AI_function_in_SQL.md

@@ -0,0 +1,355 @@
+# AI Functions
+
+Singdata Lakehouse AI Functions are a set of AI-enhanced functions built directly into the SQL engine. Without leaving the data platform or writing Python scripts, you can call large language models (LLMs) and embedding models directly in standard SQL queries to intelligently process text, images, audio, and more.
+
+For a detailed introduction, see [AI Functions Overview](ai_functions_overview.md).
+
+---
+
+## Model Access Methods
+
+The first parameter of every AI Function is `model`, which supports two access methods:
+
+**Method 1: API Gateway Endpoint (recommended)**
+
+A platform administrator pre-configures model services in the API Gateway, and you reference them using the `endpoint:` prefix:
+
+```sql
+'endpoint:qwen3-max-preview'
+'endpoint:qwen3.5-plus'
+'endpoint:text-embedding-v4'
+'endpoint:qwen3-asr-flash'
+```
+
+**Method 2: API Connection object**
+
+You create a connection object yourself using `CREATE API CONNECTION`, which is suitable for custom service addresses or private deployment scenarios:
+
+```sql
+CREATE API CONNECTION conn_bailian
+    TYPE ai_function
+    PROVIDER = 'bailian'
+    BASE_URL = 'https://dashscope.aliyuncs.com/api/v1'
+    API_KEY = 'sk-xxxxxxxxxxxxxxxxxxxxxxxx';
+
+-- Reference format: <connection_name>:<model_name>
+SELECT AI_SENTIMENT('conn_bailian:qwen3.5-plus', 'This product is great!');
+```
+
+---
+
+## Function Reference
+
+### Text Understanding and Generation
+
+#### AI_COMPLETE
+
+A general-purpose LLM completion function that supports custom prompts. Use it for complex reasoning, code generation, and other custom scenarios that specialized functions cannot cover.
+
+```sql
+ai_complete(<model>, <prompt> [, json '{}'])
+-- Image mode
+ai_complete(<model>, (<prompt> AS prompt, <image_url> AS image) [, json '{}'])
+```
+
+**Return value:** STRING — the response text generated by the model.
+
+**Usage notes:**
+- When a task can be handled by a specialized function (such as `AI_TRANSLATE` or `AI_SENTIMENT`), prefer the specialized function for more stable results and lower cost.
+- Image mode requires a multimodal-capable model (e.g., `doubao-seed-2-0-pro-260215`); passing an image to a text-only model will not raise an error but the image content will be ignored.
+- The qwen3 series models have thinking mode enabled by default. For batch processing, it is recommended to disable it via options: `json '{"model.params":{"enable_thinking":false}}'`
+- Returns NULL when `prompt` is NULL or an empty string.
+
+Full documentation: [AI_COMPLETE](ai_complete.md)
+
+---
+
+#### AI_SUMMARIZE
+
+Generates a concise summary of the input text. Supports controlling summary length via `max_words`.
+
+```sql
+AI_SUMMARIZE(model, content [, max_words])
+```
+
+**Return value:** STRING — a summary of the input text.
+
+**Usage notes:**
+- `max_words` defaults to 50. It is a target value, not a hard limit; actual output may vary by ±20%.
+- When `max_words=0`, the original text is returned without calling the model.
+- A negative `max_words` value raises an error.
+- Returns NULL for NULL input; returns an empty string for empty string input.
+- The output language automatically follows the input language.
+
+Full documentation: [AI_SUMMARIZE](ai_summarize.md)
+
+---
+
+#### AI_TRANSLATE
+
+Translates input text into a specified language. The source language is detected automatically, and 20+ languages are supported.
+
+```sql
+AI_TRANSLATE(model, content, to_lang)
+```
+
+**Return value:** STRING — the translated text.
+
+**Usage notes:**
+- `to_lang` must be a valid ISO-639 language code (e.g., `'zh'`, `'en'`, `'ja'`). Full language names are not supported.
+- Manual specification of the source language is not supported; it is always detected automatically.
+- If the input and target languages are the same, the original text is returned unchanged.
+- Can be combined with `AI_SUMMARIZE` (summarize first, then translate).
+
+Full documentation: [AI_TRANSLATE](ai_translate.md)
+
+---
+
+#### AI_FIX_GRAMMAR
+
+Automatically corrects grammar, spelling, and punctuation errors in input text. Supports Chinese, English, and mixed-language text.
+
+```sql
+AI_FIX_GRAMMAR(<model>, <content>)
+```
+
+**Return value:** STRING — the corrected text. If the input contains no grammar errors, the original text is returned unchanged.
+
+**Usage notes:**
+- Returns NULL for NULL input; returns an empty string for empty string input.
+- Mixed Chinese-English text is intelligently unified to the dominant language.
+- The focus is on grammar correction; in rare cases the meaning may be slightly altered. Manual spot-checking is recommended for semantically sensitive scenarios.
+- It is recommended to use this function to clean text before calling `AI_SENTIMENT` or `AI_SUMMARIZE`.
+
+Full documentation: [AI_FIX_GRAMMAR](ai_fix_grammar.md)
+
+---
+
+### Text Analysis and Classification
+
+#### AI_CLASSIFY
+
+Classifies text or images into user-defined categories without writing a prompt. Supports 29+ languages and cross-language classification.
+
+```sql
+AI_CLASSIFY(model, content, labels [, options])
+```
+
+**Return value:** STRING — the name of the best-matching category (a plain string, not JSON).
+
+**Usage notes:**
+- `labels` is an ARRAY(STRING). 2–10 categories are recommended, and category names should have descriptive semantics.
+- For image input, use the `GET_PRESIGNED_URL(USER VOLUME, path, expiry) AS image` syntax.
+- For batch classification, setting `enable_thinking:false` is recommended to speed up responses.
+- Returns NULL for NULL input; returns an empty string `""` for empty string input.
+- Labels can all be in English and will still classify correctly even when the input is in another language.
+
+Full documentation: [AI_CLASSIFY](ai_classify.md)
+
+---
+
+#### AI_SENTIMENT
+
+Performs sentiment analysis on input text. Supports multiple languages including Chinese, English, and Japanese.
+
+```sql
+AI_SENTIMENT(<model>, <text>)
+```
+
+**Return value:** STRING — one of the following values:
+
+| Value | Meaning |
+|-------|---------|
+| `positive` | Positive sentiment |
+| `negative` | Negative sentiment |
+| `neutral` | No clear sentiment |
+| `mixed` | Both positive and negative sentiment present |
+
+**Usage notes:**
+- Returns NULL for NULL or empty string input.
+- Can recognize complex semantics such as sarcasm/irony, double negatives, and emoji sentiment.
+- `mixed` means the text contains both positive and negative evaluations; `neutral` means the text carries no sentiment.
+- Results are non-deterministic; the same input may produce slightly different results across multiple executions.
+
+Full documentation: [AI_SENTIMENT](ai_sentiment.md)
+
+---
+
+#### AI_EXTRACT
+
+Extracts structured JSON data from unstructured text or images according to specified fields, without writing a prompt.
+
+```sql
+AI_EXTRACT(model, content, schema [, options])
+```
+
+**Return value:** STRING (JSON format) — contains all fields defined in `schema` and their extracted values. All field values are of string type.
+
+**Usage notes:**
+- `schema` format is `JSON'{"key":"field description", ...}'`. Values support simple labels or question-style descriptions.
+- No more than 10 fields are recommended; up to approximately 20 are supported.
+- If a field cannot be found in the text, its value is an empty string `""`.
+- Returns NULL for NULL or empty string input.
+- The return value is a JSON string and can be further parsed with `json_extract_string`.
+
+Full documentation: [AI_EXTRACT](ai_extract.md)
+
+---
+
+#### AI_MASK
+
+Identifies and masks PII sensitive information in text, replacing it with a `[MASKED]` placeholder. Labels are user-defined.
+
+```sql
+AI_MASK(<model>, <content>, <labels>)
+```
+
+**Return value:** STRING — the masked text. If the text does not contain information matching the specified labels, the original text is returned unchanged.
+
+**Usage notes:**
+- `labels` is an ARRAY(STRING). The count must be between 1 and 20; an empty array is not allowed.
+- More specific labels yield more accurate results (e.g., "national ID number" is more precise than "document").
+- Matching the label language to the text language produces better results.
+- All masked information is uniformly replaced with `[MASKED]`, regardless of type.
+- AI masking is LLM-based and may have missed or false detections. Manual spot-checking is recommended for compliance scenarios.
+
+Full documentation: [AI_MASK](ai_mask.md)
+
+---
+
+### Vector and Semantic Search
+
+#### AI_EMBEDDING
+
+Converts text into high-dimensional embedding vectors for downstream tasks such as semantic retrieval, recommendations, and clustering.
+
+```sql
+AI_EMBEDDING(<model>, <input> [, <model_parameters>])
+```
+
+**Return value:** ARRAY\<FLOAT\> — the embedding vector of the input text.
+
+**Usage notes:**
+- An embedding model (e.g., `text-embedding-v4`) must be used; chat completion models cannot be used.
+- Results are deterministic; the same input always returns the same vector.
+- In retrieval scenarios, use `"input":"document"` when indexing documents and `"input":"query"` for user queries.
+- Returns NULL for NULL input; raises an error for empty string input — filter before calling.
+- Input length limit is 8,192 tokens; a maximum of 10 items can be processed per call.
+
+Full documentation: [AI_EMBEDDING](ai_embedding.md)
+
+---
+
+#### AI_SIMILARITY
+
+Calculates the cosine similarity between two pieces of text based on an embedding model, returning a score in [0, 1].
+
+```sql
+AI_SIMILARITY(<model>, <text1>, <text2> [, <options>])
+```
+
+**Return value:** FLOAT — cosine similarity, which in practice typically falls in the [0, 1] range.
+
+| Range | Meaning |
+|-------|---------|
+| 1.0 | Semantically identical |
+| > 0.7 | Highly similar |
+| 0.3 ~ 0.7 | Somewhat related |
+| < 0.3 | Largely unrelated |
+
+**Usage notes:**
+- An embedding model must be used; LLMs cannot be used.
+- Results are deterministic, and the function is symmetric (`a,b` and `b,a` return the same result).
+- Returns 0 when either parameter is NULL.
+- Supports cross-language similarity calculation (e.g., comparing Chinese and English semantics).
+- In batch CROSS JOIN scenarios, token consumption = rows² × average token count. Evaluate your quota in advance.
+
+Full documentation: [AI_SIMILARITY](ai_similarity.md)
+
+---
+
+### Multimodal Processing
+
+#### AI_TRANSCRIBE
+
+Transcribes audio files in a Volume to plain text (ASR). Supports multiple languages including Chinese and English.
+
+```sql
+AI_TRANSCRIBE(<model>, <audio_url> [, <options>])
+```
+
+**Return value:** STRING — the plain-text transcription of the audio content, without timestamps or speaker information.
+
+**Usage notes:**
+- `audio_url` must be a URL starting with `http://` or `https://`, typically obtained via `GET_PRESIGNED_URL()`.
+- Supports WAV, MP3, FLAC, and M4A formats. 16kHz mono WAV is recommended.
+- A presigned URL validity of 36,000 seconds (10 hours) is recommended to avoid URL expiration during batch processing.
+- Returns plain text that can be used directly as input to functions like `AI_CLASSIFY` or `AI_EXTRACT`.
+- Silent or near-silent files may produce a small amount of hallucinated text; downstream length filtering is recommended.
+
+Full documentation: [AI_TRANSCRIBE](ai_transcribe.md)
+
+---
+
+## Common `options` Parameter
+
+Functions such as `AI_CLASSIFY`, `AI_EXTRACT`, `AI_SIMILARITY`, and `AI_TRANSCRIBE` support an optional `options` JSON parameter:
+
+| Parameter key | Type | Description |
+|---------------|------|-------------|
+| `response.timeout` | STRING | Per-request timeout in seconds, e.g., `"300"` |
+| `task.concurrency` | STRING | Concurrency for batch processing, e.g., `"12"` |
+| `model.params` | JSON | Parameters passed through to the model, e.g., `{"enable_thinking": false}` |
+
+```sql
+JSON'{"model.params":{"enable_thinking":false},"response.timeout":"300","task.concurrency":"12"}'
+```
+
+---
+
+## Prerequisites
+
+1. A model endpoint has been configured in the API Gateway, or a connection object has been created via `CREATE API CONNECTION`.
+2. The current user has permission to call the relevant endpoint or connection.
+3. For image and audio processing scenarios, files must be uploaded to a Volume and an access URL obtained via `GET_PRESIGNED_URL()`.
+
+---
+
+## Related Documentation
+
+- [AI Functions Overview](ai_functions_overview.md)
+- [AI_COMPLETE](ai_complete.md)
+- [AI_CLASSIFY](ai_classify.md)
+- [AI_EXTRACT](ai_extract.md)
+- [AI_SENTIMENT](ai_sentiment.md)
+- [AI_SUMMARIZE](ai_summarize.md)
+- [AI_TRANSLATE](ai_translate.md)
+- [AI_FIX_GRAMMAR](ai_fix_grammar.md)
+- [AI_MASK](ai_mask.md)
+- [AI_EMBEDDING](ai_embedding.md)
+- [AI_SIMILARITY](ai_similarity.md)
+- [AI_TRANSCRIBE](ai_transcribe.md)
+
+---
+
+## Customizing AI Functions via External Functions
+
+In addition to the built-in AI Functions, Singdata Lakehouse also supports developing custom AI functions through the **External Function** framework. This is suitable for scenarios that require connecting to private models, offline inference packages, or custom business logic.
+
+External functions (also known as Remote Functions) are a special type of user-defined function (UDF) that allows you to define function logic in Python or Java, with the core computation offloaded to an external remote service (Alibaba Cloud Function Compute FC and Tencent Cloud SCF are supported). During execution, the following can be called:
+
+- **Online services**: Online services provided externally via API, such as large language model APIs and online AI API services provided by cloud platforms.
+- **Offline features**: Offline service packages that bundle specific function code, dependency libraries, models, and data files, such as image recognition models downloaded from Hugging Face.
+
+Singdata Lakehouse stores the connection and access information for external function compute services in metadata by creating an API Connection. External functions call the external compute service via HTTP protocol, process data, and return results.
+
+![](.topwrite/assets/external-function-arch.svg)
+
+After receiving prior user authorization, the Lakehouse platform automatically deploys the function to the function compute service under the customer's account when the external function is created. When you use an external function in a SQL query, the external function handles the secure connection to the external compute service, data processing, and returns the query results.
+
+**Main process for creating external functions**: [Usage Process: External Function](remotefunction-best-practice.md)
+
+**Development guides**:
+
+- [External Function (Python3)](remotefunction-dev-guide-python3.md)
+- [External Function (Java)](external-function-dev-guide-java.md)