@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.100

package/bin/skills/lakehouse-doc-en/references/compute-overview.md

@@ -0,0 +1,142 @@
+# Compute Resources
+
+Compute resources are the **compute layer in Lakehouse's storage-compute separation architecture** — data is stored in object storage, and VClusters (Virtual Compute Clusters) provide CPU and memory on demand. Multiple clusters can read from and write to the same data simultaneously without interfering with each other.
+
+Think of a VCluster as an "on-demand compute engine": start it when you need it, stop it when you're done, and pay only for the minutes it actually runs. Unlike traditional databases, compute and storage are completely decoupled here — resizing a cluster requires no data migration, and different teams can use independent clusters to access the same table without competing for resources.
+
+![](/.topwrite/assets/18-compute.png)
+
+## Choosing a Cluster Type
+
+| Type | Use cases | Key characteristics |
+|------|-----------|---------------------|
+| **General-purpose (GENERAL)** | ETL data processing, Dynamic Table refresh, offline batch jobs | Jobs share resources with fair scheduling and elastic scaling. Dynamic Table refresh requires this type (supports automatic small-file compaction). |
+| **Analytics (ANALYTICS)** | BI queries, ad-hoc analysis, high-concurrency queries | Supports horizontal scale-out with multiple instances; result caching accelerates repeated queries. |
+| **Integration (INTEGRATION)** | Real-time / offline data sync jobs | Optimized for integration tasks; multiple sync jobs share a single cluster. |
+
+**Selection rules:**
+
+- ETL jobs → General-purpose: lower cost, no resource contention with queries.
+- BI reports / ad-hoc queries → Analytics: result caching, multi-instance concurrency.
+- Data sync jobs → Integration: dedicated resources, no impact on other workloads.
+- Dynamic Table refresh → General-purpose: Analytics clusters do not support automatic small-file compaction during refresh.
+
+> ⚠️ **Workload isolation principle**: Do not share the same cluster between ETL jobs and BI queries. Large jobs will consume all resources and cause queries to queue.
+
+## Core Mechanisms
+
+**CRU (Compute Resource Unit)**: The unit of measurement for compute resources, abstracting away differences between cloud platforms and CPU architectures. Billed by actual runtime; no charges when suspended. Runtimes under 1 minute are billed as 1 minute.
+
+**Auto suspend/resume**: A cluster can automatically suspend when idle (billing stops) and automatically wake up when a new job is submitted. Recommended settings:
+- ETL job clusters: auto-suspend after 60 seconds to release resources quickly.
+- BI query clusters: auto-suspend after 1800 seconds (30 minutes) to leverage result caching for repeated queries.
+
+**Horizontal scale-out (Analytics only)**: When concurrent queries exceed the capacity of a single instance, additional replicas start automatically to share the load and scale back down once queries complete.
+
+## Quick Example
+
+```sql
+-- Create a General-purpose cluster (for ETL)
+CREATE VCLUSTER etl_cluster
+    VCLUSTER_SIZE = 1
+    VCLUSTER_TYPE = GENERAL
+    AUTO_SUSPEND_IN_SECOND = 60;
+
+-- Create an Analytics cluster (for BI, up to 2 concurrent instances)
+CREATE VCLUSTER bi_cluster
+    VCLUSTER_SIZE = 2
+    VCLUSTER_TYPE = ANALYTICS
+    MIN_REPLICAS = 1
+    MAX_REPLICAS = 2
+    AUTO_SUSPEND_IN_SECOND = 1800;
+
+-- Switch the current session to a cluster
+USE VCLUSTER bi_cluster;
+
+-- View all clusters and their status
+SHOW VCLUSTERS;
++-----------+----------+---------------+---------+
+| name      | type     | size          | state   |
++-----------+----------+---------------+---------+
+| etl_cluster | GENERAL | S_1CRU       | RUNNING |
+| bi_cluster  | ANALYTICS | S_2CRU     | SUSPENDED |
++-----------+----------+---------------+---------+
+```
+
+## Common Issues
+
+### Issue 1: Dynamic Table using an Analytics cluster
+
+**Problem**: Dynamic Table refresh is scheduled on an Analytics cluster.
+
+**Symptom**: The number of data files keeps growing, queries get progressively slower, and small-file count explodes.
+
+**Solution**: Dynamic Table refresh must use a General-purpose cluster. Analytics clusters do not support automatic small-file compaction during the refresh process.
+
+### Issue 2: ETL and BI sharing the same cluster
+
+**Problem**: Large-batch ETL jobs and BI queries share the same cluster.
+
+**Symptom**: BI queries queue up and reports are delayed; ETL jobs also slow down because BI queries consume resources.
+
+**Solution**: Split clusters by workload type — give ETL and BI each their own independent cluster so they don't interfere with each other.
+
+### Issue 3: Auto-suspend time too short causes frequent cold starts
+
+**Problem**: The BI cluster's auto-suspend is set to 60 seconds.
+
+**Symptom**: Every query has to wait for a cold start (typically 10–30 seconds), resulting in a poor user experience. Additionally, since each start is billed as at least 1 minute, costs actually increase.
+
+**Solution**: Set the BI cluster's auto-suspend time to 30 minutes or more to take advantage of result caching for repeated queries.
+
+## Cost Impact
+
+### Compute Cost
+
+- Billed by CRU × runtime; **no charges while suspended**.
+- Larger sizes (more CRUs) cost more per unit time but complete large jobs faster.
+- When horizontal scale-out is enabled on an Analytics cluster, the replica count changes dynamically — monitor peak costs.
+
+### Storage Cost
+
+- VClusters themselves incur no storage costs; data is stored in object storage.
+- The `PRELOAD_TABLES` feature on Analytics clusters preloads data into local SSD cache (temporary storage, no additional charge).
+
+> 💡 For detailed billing rules, see the [Billing Documentation](billing.md).
+
+## Lifecycle Management
+
+```
+Create cluster → Auto-start → Execute jobs → Auto-suspend when idle → Resize → Drop cluster
+      ↓               ↓             ↓                  ↓                  ↓           ↓
+  Specify type   Wake on job    Consume CRU       Stop billing      Scale up/down  Suspend first
+```
+
+```sql
+-- Resize (hot change, does not affect running jobs)
+ALTER VCLUSTER etl_cluster SET VCLUSTER_SIZE = 4;
+
+-- Manually suspend
+ALTER VCLUSTER etl_cluster SUSPEND;
+
+-- Manually resume
+ALTER VCLUSTER etl_cluster RESUME;
+
+-- Drop the cluster
+DROP VCLUSTER etl_cluster;
+```
+
+## In This Section
+
+| Page | Description |
+|------|-------------|
+| [VCluster (Compute Cluster)](om-vcluster.md) | Cluster type details, size selection, horizontal scale-out mechanism |
+
+## Related Documentation
+
+| Document | Description |
+|----------|-------------|
+| [Managing Compute Clusters](tutorial_virtual_cluster.md) | Web UI operations and management |
+| [Horizontal Elastic Scaling](concurrency_scaling.md) | How concurrency scaling works for Analytics clusters |
+| [Specification Reference](vcluster_size_description.md) | CRU specification code reference table |
+| [Create Compute Cluster](create_cluster.md) | Complete DDL syntax |

package/bin/skills/lakehouse-doc-en/references/compute-resource-ddl.md

@@ -0,0 +1,128 @@
+# Compute Resource DDL
+
+Compute cluster (VCluster) DDL commands are used to create, configure, start/stop, and drop compute clusters. All SQL that requires computation (SELECT, DML) runs on a compute cluster.
+
+---
+
+## In This Chapter
+
+| Page | Description |
+|------|-------------|
+| [CREATE VCLUSTER](create_cluster.md) | Create a general-purpose, analytics, or sync compute cluster |
+| [ALTER VCLUSTER](alter-vcluster.md) | Start, stop, or modify cluster properties |
+| [DROP VCLUSTER](drop-vcluster.md) | Drop a compute cluster |
+| [USE VCLUSTER](use-vcluster.md) | Switch the cluster used by the current session |
+| [DESC VCLUSTER](desc-vcluster.md) | View detailed cluster configuration and status |
+| [SHOW VCLUSTERS](show-vclusters.md) | List all clusters in the workspace |
+
+---
+
+## Common Operations
+
+### Create a Cluster
+
+```SQL
+-- General-purpose (GP): suited for ETL batch processing
+CREATE VCLUSTER IF NOT EXISTS my_etl_cluster
+  VCLUSTER_TYPE = GENERAL
+  VCLUSTER_SIZE = 2
+  AUTO_SUSPEND_IN_SECOND = 60
+  AUTO_RESUME = TRUE;
+
+-- General-purpose (GP) with auto-scaling enabled
+CREATE VCLUSTER IF NOT EXISTS my_etl_cluster
+  VCLUSTER_TYPE = GENERAL
+  MIN_VCLUSTER_SIZE = 1
+  MAX_VCLUSTER_SIZE = 8
+  AUTO_SUSPEND_IN_SECOND = 60
+  AUTO_RESUME = TRUE;
+
+-- Analytics (AP): suited for BI queries and high-concurrency online queries
+CREATE VCLUSTER IF NOT EXISTS my_bi_cluster
+  VCLUSTER_TYPE = ANALYTICS
+  VCLUSTER_SIZE = 2
+  MIN_REPLICAS = 1
+  MAX_REPLICAS = 4
+  MAX_CONCURRENCY = 8
+  AUTO_SUSPEND_IN_SECOND = 120
+  AUTO_RESUME = TRUE;
+```
+
+### Start and Stop
+
+```SQL
+-- Start a cluster
+ALTER VCLUSTER IF EXISTS my_cluster RESUME;
+
+-- Stop a cluster (waits for current jobs to complete)
+ALTER VCLUSTER IF EXISTS my_cluster SUSPEND;
+
+-- Force stop (immediately terminates all running jobs)
+ALTER VCLUSTER IF EXISTS my_cluster SUSPEND FORCE;
+```
+
+### Modify Cluster Configuration
+
+```SQL
+-- General-purpose: adjust auto-scaling range
+ALTER VCLUSTER my_etl_cluster SET MIN_VCLUSTER_SIZE = 1 MAX_VCLUSTER_SIZE = 4;
+
+-- Analytics: adjust replica count range
+ALTER VCLUSTER my_bi_cluster SET MIN_REPLICAS = 1 MAX_REPLICAS = 4;
+
+-- Analytics: adjust max concurrency per replica
+ALTER VCLUSTER my_bi_cluster SET MAX_CONCURRENCY = 16;
+
+-- Set auto-suspend time (seconds)
+ALTER VCLUSTER my_cluster SET AUTO_SUSPEND_IN_SECOND = 300;
+
+-- Set job timeout (seconds)
+ALTER VCLUSTER my_cluster SET QUERY_RUNTIME_LIMIT_IN_SECOND = 600;
+```
+
+### View Clusters
+
+```SQL
+-- List all clusters
+SHOW VCLUSTERS;
+
+-- List only running clusters
+SHOW VCLUSTERS WHERE state = 'RUNNING';
+
+-- View detailed cluster configuration
+DESC VCLUSTER my_cluster;
+```
+
+### View Jobs
+
+```SQL
+-- View failed jobs
+SHOW JOBS WHERE status = 'FAILED' LIMIT 20;
+```
+
+### Switch Cluster
+
+```SQL
+-- Switch the current session to a specific cluster
+USE VCLUSTER my_cluster;
+```
+
+### Drop a Cluster
+
+```SQL
+DROP VCLUSTER IF EXISTS my_cluster;
+
+-- Force drop (does not wait for running jobs to complete)
+DROP VCLUSTER IF EXISTS my_cluster FORCE;
+```
+
+---
+
+## Related Documentation
+
+| Document | Description |
+|----------|-------------|
+| [Managing Compute Resources](tutorial_virtual_cluster.md) | Cluster type selection guide and common operation walkthrough |
+| [Compute Cluster Size Code Changes](vcluster_size_description.md) | Mapping between old and new size codes |
+| [Compute Cluster Cache](vc_cache.md) | How active and passive caching work |
+| [SHOW JOBS](show-jobs.md) | Full syntax for viewing and filtering job records |

package/bin/skills/lakehouse-doc-en/references/concepts.md

@@ -0,0 +1,209 @@
+# Singdata Lakehouse Product Introduction
+
+Singdata Lakehouse is a fully managed, cloud-native lakehouse data platform equipped with a newly designed vectorized SQL compute engine, delivering industry-leading performance on [TPC-DS 10TB](tpcds-benchmark.md), [TPC-H 100GB](tpch-benchmark.md), and [SSB 100GB](ssb-benchmark.md) benchmarks. The platform features a proprietary [Generic Incremental Computation (GIC)](incremental-computing.md) model that unifies batch and stream processing into a single incremental computation paradigm — Dynamic Tables defined with standard SQL automatically detect upstream changes and incrementally refresh.
+
+Lakehouse unification solved the fragmentation between data storage and processing, but it did not solve the new challenges of the AI era: vector databases, LLM calls, and unstructured data processing remain outside the data platform, forcing AI teams and data teams to use two separate toolsets on the same data. Singdata defines this direction as **[AI Lakehouse](lakehouse-ai.md)** — GIC-driven incremental refresh allows AI processing results to flow back into the data processing pipeline in real time, rather than sitting in an external system; AI capabilities become native to the data platform, completing the full cycle within the same platform as data processing.
+
+The platform is delivered as SaaS, currently available on Alibaba Cloud (East China Shanghai), Tencent Cloud (East China Shanghai, North China Beijing, South China Guangzhou), Amazon Cloud (Beijing), and more — on-demand activation, no infrastructure management required. See [Supported Cloud Platforms and Regions](supported-cloud-platforms.md).
+
+## Architecture Overview
+
+For years, data platforms served only two types of users: **humans** (data analysts and engineers operating through [Studio](lakehouse-studio-101.md)) and **applications** (business systems fetching data via JDBC/API). That assumption is breaking down.
+
+With the rise of AI coding tools like ChatGPT, Claude, Cursor, and Codex, more and more enterprises are letting AI Agents directly participate in data work — automatically generating ETL tasks, analyzing data quality, diagnosing slow queries, and answering business questions. This is not a future trend; it is happening now.
+
+But AI Agents participating in data work is fundamentally different from humans doing so: **humans can read UIs, consult documentation, and rely on experience; Agents can only operate through programmatic interfaces. They don't know which table and field "GMV" refers to. They cannot accept "the data is from yesterday." Their concurrent requests can surge to hundreds per second**. This cannot be solved by simply adding a CLI tool to an old platform — it requires the data platform to be redesigned at four levels:
+
+* **Programmable interfaces**: Traditional platform Web UIs are designed for humans; having an Agent operate them is like asking someone to click on a screenshot — inefficient and unreliable. Singdata provides [cz-cli](setup_cz_cli.md) for AI Agents: deterministic command formats, structured output, and complete data warehouse operation coverage, enabling tools like Codex, Claude Code, Cursor, and Kiro to complete full workflows of table creation, SQL writing, task submission, and log inspection.
+* **Real-time data**: When Agents execute tasks, they need current state — is inventory sufficient? Has the order been paid? If data is yesterday's batch snapshot, the Agent's decisions are built on stale information and cannot be trusted. GIC-driven Dynamic Tables refresh the entire warehouse processing pipeline in minutes, so Agents query current state, not yesterday's snapshot.
+* **Understandable semantics**: Agents facing raw table structures can only guess at field meanings — does "order\_amt" include tax? What is the definition of "active\_user"? [Semantic Views](semantic-view-overview.md) build a business semantic layer on top of physical tables. Agents understand business concepts like "GMV" and "active users" through Semantic Views, generating SQL with accurate metric definitions.
+* **Elastically scalable compute**: Human access to data platforms has limited concurrency, but AI Agents can simultaneously issue tens or even hundreds of concurrent queries. With storage-compute separation, analytical clusters support automatic horizontal scaling based on concurrent load, automatically scaling back when pressure subsides, with scale-out latency measured in seconds — no impact on online services.
+
+Today, Singdata Lakehouse serves three types of users, each equally important as a first-class citizen:
+
+* **Human**: Operates through the [Studio](lakehouse-studio-101.md) Web interface, covering data development, scheduling, analysis Notebooks, and [DataGPT](lakehousedatagpt-tour.md) conversational queries
+* **App**: Connects via JDBC, Python SDK, MySQL protocol, REST API, using Lakehouse as the data backend for applications
+* **AI Agent**: Connects via [cz-cli](setup_cz_cli.md), autonomously completing data warehouse development, operations, and data consumption
+
+![](/.topwrite/assets/01-architecture_1779606768355.svg)
+
+Vertically, data is persisted from the bottom-most object storage layer, unified through metadata and federation layers, organized into various tables and pipelines in the data object layer, driven by the SQL compute engine for processing, and finally exposed to all three user types through the access layer.
+
+The following three sections explain Singdata's underlying architectural choices for storage, compute, and data freshness; programmable interfaces, semantic capabilities, multimodal storage, and other AI-layer capabilities are covered in the AI Lakehouse section.
+
+***
+
+## Technical Foundations
+
+### Lakehouse Unification
+
+Most enterprise data architectures face the same dilemma: raw data lives in a data lake (low cost, flexible formats, but slow queries and weak governance), while processed data lives in a data warehouse (high performance, strong governance, but high cost and closed formats). Two systems mean two storage layers, two permission systems, two metadata stores, and endless data sync pipelines — data moves from lake to warehouse and back, with delays and errors as constants. The proliferation of AI applications makes the fragmentation worse: BI consumes structured data from the warehouse, AI applications need raw files from the lake, and the same business data must be maintained in two copies.
+
+Singdata Lakehouse unifies structured tables, semi-structured data, and unstructured files on a single platform: sharing the same SQL engine, metadata service, and permission system. Data is stored only once; BI and AI read from the same table. The SQL compute engine features a proprietary [Generic Incremental Computation (GIC)](incremental-computing.md) model that unifies batch and stream processing into a single incremental computation paradigm — Dynamic Tables automatically detect upstream changes and incrementally refresh.
+
+GIC is the underlying engine driving Dynamic Tables. Traditional incremental computation approaches degrade to full recomputation when encountering JOINs, nested subqueries, or UPDATE/DELETE; GIC decomposes any standard SQL into operator-level incremental plans, with Filter, Join, Aggregate, and Window all supporting incremental execution. GIC has three core properties: **generality** (no restriction on query complexity), **cost-awareness** (dynamically selects incremental or full execution plan at each refresh based on data statistics, automatically falling back to full recomputation when incremental data volume is too large), and **semantic consistency** (based on MVCC version management, incremental results are strictly consistent with full recomputation, guaranteeing Exactly-Once semantics).
+
+The storage layer is built on Apache Iceberg. Singdata engineers are the first contributors to the [Apache Iceberg C++ implementation (iceberg-cpp)](https://github.com/apache/iceberg-cpp), leading the core development of this official C++ library from scratch to production-ready. Singdata engineers serve as Apache Iceberg Committers and are PMC members of Apache Arrow, Apache ORC, and Apache Parquet. Singdata Lakehouse is one of the earliest lakehouse products to fully support Iceberg V3 (Deletion Vectors, Row Lineage, VARIANT type). Deletion Vectors dramatically reduce write amplification in UPDATE/DELETE-intensive scenarios (such as CDC writes and GDPR deletions), eliminating the need to rewrite entire data files; Row Lineage makes the origin of every row traceable, supporting fine-grained auditing; the VARIANT type natively supports semi-structured data, enabling queries on nested JSON without predefined schemas.
+
+![](/.topwrite/assets/00-lakehouse-unified.svg)
+
+**What this means for you**: BI reports and AI applications no longer each maintain a separate copy of data — both read from the same table, with permissions and metadata managed in a single system. Historical data in Hive, Delta Lake, and Hudi is directly queryable via external tables; existing data assets can be onboarded without migration. A single SQL can simultaneously process raw files in object storage and processed wide tables, aligning BI and AI data definitions for the first time.
+
+### Storage-Compute Separation
+
+Traditional data warehouses couple storage and compute: storage and compute are bound to the same machines, with ETL, BI queries, and AI inference sharing the same resources — one large job saturates the CPU and all other queries queue up; scaling compute requires scaling storage simultaneously, with unavoidable cost waste.
+
+Singdata Lakehouse persists data to object storage (OSS / S3 / COS), with compute handled by independent [VClusters](compute-overview.md) — completely decoupled. **Data exists only once; compute can scale to any number of instances** — multiple VClusters simultaneously read and write the same data without interference, each independently scaling and billed separately. When there are no queries, compute clusters pause and stop billing, but data remains online and queryable at any time.
+
+The compute layer is split into three cluster types by workload. GP and AP scale differently because their workload characteristics differ: ETL batch processing has high per-job resource requirements but low concurrency, making vertical scaling (larger nodes) more effective; BI queries have low per-job resource requirements but high concurrency, requiring horizontal scaling (more instances) to linearly increase throughput.
+
+* [General Purpose (GP)](compute-overview.md): ETL batch processing, Dynamic Table incremental refresh, AI\_COMPLETE / Embedding and other AI compute tasks. Jobs share resources with fair scheduling; supports vertical elastic scaling
+* [Analytical (AP)](compute-overview.md): BI reports, ad-hoc queries, high-concurrency online analytics, and latency-sensitive AI inference scenarios. Jobs have dedicated resources; supports multi-instance horizontal scaling with built-in result cache acceleration
+* [Integration](compute-overview.md): Offline batch sync and CDC real-time sync share one cluster, completely isolated from query traffic
+
+The proliferation of AI Agents places higher demands on elastic scaling. AI Agents can simultaneously issue tens or even hundreds of concurrent queries, far exceeding the concurrency limits of human operations. With storage-compute separation, analytical clusters support automatic horizontal instance scaling based on concurrent load, automatically scaling back when pressure subsides, with scale-out latency measured in seconds — no data migration involved.
+
+![](/.topwrite/assets/02-compute-separation_1779606809686.svg)
+
+**What this means for you**: Compute and storage are billed and scaled independently: when business data volume grows, only compute needs to scale; during query off-peak hours, compute clusters pause and stop billing while data remains online. Batch jobs and BI queries use separate independent clusters without competing for resources — a single large job cannot overwhelm all queries. When AI Agents begin concurrent access to the data platform, analytical clusters automatically scale out horizontally to handle peak load and scale back when pressure subsides — the entire process requires no manual intervention and no data migration.
+
+### Unified Real-Time and Batch
+
+In traditional data architectures, real-time pipelines use Flink and offline pipelines use Spark — two codebases, two operations teams, and the same business metric must be written twice in two places, with results that frequently diverge. The more fundamental problem: even if the ODS layer has real-time CDC, downstream DWD and DWS layers are still T+1 batch processing — the freshness of the entire pipeline is determined by its slowest node. When AI Agents execute tasks, they need current state; overnight batch snapshots cause Agents to make decisions based on stale information.
+
+Singdata Lakehouse uses [Dynamic Tables](dynamic-table-introduce.md) to enable incremental refresh throughout the entire processing pipeline. Most incremental computation approaches degrade to full recomputation when encountering JOINs or UPDATE/DELETE; GIC's design goal is generality — decomposing the execution of any SQL into operator-level incremental plans, with Filter, Join, Aggregate, and Window all supporting incremental execution, each operator processing only the changed portion from upstream. Define a SQL in a Dynamic Table, and GIC automatically detects upstream table changes, computing only the incremental portion and refreshing the result. ODS → DWD → DWS → ADS — every layer is a Dynamic Table, every layer automatically triggers after upstream refresh, with end-to-end freshness measured in minutes. Whether upstream is scheduled batch writes or CDC real-time streams, downstream processing logic is identical — one codebase covers both scenarios.
+
+The cost of traditional full recomputation is proportional to total data volume; Dynamic Tables process only the incremental data since the last refresh, with refresh time proportional to the volume of changes — minute-level refresh costs are not in the same order of magnitude as full recomputation. GIC does not statically bind incremental plans; instead, it dynamically selects the execution approach at each refresh based on data statistics — automatically falling back to full recomputation when incremental data volume is too large, always optimizing cost without manual intervention.
+
+Dynamic Tables support three scheduling modes, switchable without modifying the SQL: **real-time trigger** (computes immediately on upstream data changes, second-level latency, suitable for risk control and monitoring), **periodic scheduling** (batch processing at minute/hour intervals, suitable for most near-real-time scenarios), and **DAG dependency trigger** (downstream automatically triggers after upstream Dynamic Table refresh completes; configure only leaf nodes and the entire pipeline chains automatically).
+
+![](/.topwrite/assets/03-unified-pipeline_1779606830577.svg)
+
+**What this means for you**: No more maintaining a dual-pipeline Lambda architecture with Flink + data warehouse; warehouse layering logic is written once and reused for both batch and real-time scenarios; AI Agents query processing results fresh to the minute; switching upstream data sources from batch to real-time CDC requires no changes to downstream Dynamic Tables.
+
+***
+
+## AI Lakehouse
+
+The three technical foundations solve the data platform's own problems, but there is a larger fragmentation that remains unresolved: **the fragmentation between the data platform and AI systems**. Data must be repeatedly moved between two systems, vector databases and data warehouses each maintain separate permissions and metadata, and AI processing results cannot flow directly back into the data processing pipeline.
+
+Singdata AI Lakehouse's answer: rather than building an AI system alongside the data platform, make AI capabilities native to the data platform. GIC is the key — precisely because incremental refresh is cheap enough, AI processing results can flow back into the data processing pipeline in real time, rather than sitting in an external system waiting for the next batch run. AI capabilities span four layers:
+
+**Data Layer: Multimodal Storage + AI Processing**
+
+The same table natively supports three data types and their corresponding indexes, with BI and AI reading from the same data:
+
+![](/.topwrite/assets/22-multimodal_1779617786266.svg)
+
+| Data type                             | Index               | Typical query                                             | Use case                                        |
+| ------------------------------------- | ------------------- | --------------------------------------------------------- | ----------------------------------------------- |
+| Scalar (numbers, strings, timestamps) | Bloomfilter index   | WHERE col = value · WHERE col IN (...)                    | BI reports, exact filtering                     |
+| Text (articles, logs, reviews)        | Inverted index      | match\_any(col, 'keyword') · multi\_match(c1, c2, 'term') | Full-text search, log analysis                  |
+| Vector (Embeddings)                   | Vector index (HNSW) | L2\_DISTANCE(vec, query\_vec) < threshold LIMIT K         | Semantic search, RAG retrieval, recommendations |
+
+A single SQL can simultaneously filter scalar fields, match keywords, and retrieve semantically similar vectors, fusing rankings with the RRF algorithm — no need to separately maintain Elasticsearch or a vector database. See [Full-Text + Vector Hybrid Search Best Practices](rrf-fulltext-vector-hybrid-search-best-practices.md).
+
+[AI Functions](ai_function_best_practice.md) embed LLM capabilities into the SQL engine, performing OCR, summarization, classification, and vectorization on unstructured data (contracts, images, tickets) in [Volumes](volume-overview.md), writing results directly into the table's vector or text columns for unified consumption by BI and AI.
+
+**Understanding Layer: AI Reads Business Semantics**
+
+[Semantic Views](semantic-view-overview.md) build a business semantic layer on top of physical tables, centralizing metric definitions and entity relationships. AI understands business concepts like "active users" and "GMV" through Semantic Views, rather than guessing at field meanings from raw table structures. [Data Analytics Agent (DataGPT)](lakehousedatagpt-tour.md) implements natural language data queries based on Semantic Views — users describe business questions, the Agent generates SQL, executes queries, and interprets results.
+
+**Operations Layer: AI Autonomously Completes Warehouse Development**
+
+[cz-cli](setup_cz_cli.md) is the standard interface for AI Agents to operate Lakehouse, encapsulating table creation, SQL writing, task submission, log inspection, and performance diagnostics as structured commands. **The Data Engineering Agent built into** [Studio](lakehouse-studio-101.md) participates directly in the development interface — understanding table structures and business context, automatically generating ETL SQL, debugging data quality issues, and explaining slow query causes. AI coding tools like Codex, Claude Code, Cursor, and Kiro can complete full data warehouse development and operations workflows through cz-cli.
+
+**Governance Layer: Unified Management of AI Model Resources**
+
+[AI Gateway (Model Management)](ai_gateway.md) is the enterprise's unified LLM access point, aggregating Alibaba Cloud Qwen, OpenAI, self-hosted models, and other sources. It provides RBAC permission isolation, call rate limiting, token usage statistics, and multi-tenant cost allocation — AI Functions and various Agents share the same model governance mechanism.
+
+**Overall value**: Data is stored only once; BI reports and AI applications read from the same table with unified permissions and metadata management; AI processing results flow back into the data processing pipeline in real time via GIC incremental refresh, no longer sitting in external systems waiting for batch runs; from data ingestion, processing, AI handling to analytics consumption, the full pipeline closes within a single platform — data teams need not switch tools or maintain multiple systems.
+
+***
+
+## Lakehouse Studio
+
+Singdata chose an integrated approach from the very beginning of product design — bringing the complete lifecycle of data development into a single platform, allowing ingestion, development, scheduling, operations, and governance to share the same metadata and permission system. [Studio](lakehouse-studio-101.md) was built in parallel with the compute engine, not as an add-on module.
+
+Studio's main modules:
+
+* **Data Sync**: Offline batch sync and CDC real-time sync for 40+ data sources, visually configured with no code required
+* **Task Development and Scheduling**: Unified management of SQL / Python / Shell tasks, DAG dependency orchestration, supports backfill and rerun
+* **Operations Monitoring**: Instance logs, run alerts, resource usage — troubleshoot without leaving the platform
+* **Data Catalog**: Metadata management, data lineage, data quality rules — governance and development in the same interface
+* **Analysis Notebook**: Interactive SQL and Python analysis, results directly connectable to BI
+
+Quick start: [Lakehouse Studio Getting Started Guide](lakehouse-studio-101.md)
+
+***
+
+## Core Object Relationships
+
+Relationships between data objects:
+
+![](/.topwrite/assets/04-object-relationship_1779606899090.svg)
+
+* [Dynamic Table](dynamic-table.md): Define a SQL; the system automatically detects upstream table changes and incrementally refreshes. Suitable for building ODS→DWD→ADS data pipelines. Does not depend on Table Stream.
+* [Table Stream](table_stream.md): An independent CDC capture mechanism that records every row's insert/update/delete changes for downstream custom consumption. Suitable for scenarios requiring fine-grained control over MERGE logic.
+* Both start from a Table and are two parallel paths — choose one based on your scenario.
+
+Dynamic Table's incremental computation is driven by Singdata's proprietary GIC (Generic Incremental Computation) model, supporting second-level trigger, minute-level periodic, and DAG dependency scheduling modes. Defined with standard SQL, no stream processing framework required. See [Incremental Computing Mechanism and Dynamic Tables](incremental-computing.md).
+
+***
+
+## Typical Use Cases
+
+### Enterprise Knowledge Base and RAG System
+
+Store unstructured documents (contracts, research reports, tickets, product manuals) in a [Volume](volume-overview.md), batch-vectorize them with `AI_EMBEDDING()` and write into the same table, while building an inverted index for exact keyword matching. At retrieval time, a single SQL simultaneously retrieves semantically similar content (vector index) and exact matches for proper nouns (inverted index), fuses rankings with the RRF algorithm, then calls `AI_COMPLETE()` to generate answers. The entire RAG pipeline closes within Lakehouse — no need to separately maintain a vector database and Elasticsearch.
+
+Reference: [Full-Text + Vector Hybrid Search Best Practices](rrf-fulltext-vector-hybrid-search-best-practices.md)
+
+### Real-Time Data Backend for AI Applications
+
+AI applications executing tasks depend on real-time business state: is inventory sufficient? Has the order been paid? Has the user profile been updated? Use CDC real-time sync to write business database changes into Lakehouse; Dynamic Tables refresh DWD/DWS layers in minutes; [Semantic Views](semantic-view-overview.md) unify metric definitions. AI applications query Semantic Views via REST API or JDBC, receiving results that are business-readable and fresh to the minute — not overnight batch snapshots.
+
+Reference: [Semantic Views](semantic-view-overview.md)
+
+### AI Agent-Driven Autonomous Data Engineering
+
+Using AI coding tools like Claude Code, Cursor, or Codex, operate Lakehouse directly through [cz-cli](setup_cz_cli.md): describe requirements, and the Agent automatically creates tables, writes ETL SQL, submits scheduling tasks, inspects run logs, and identifies performance bottlenecks. Data warehouse development shifts from "humans write code" to "humans describe goals, Agents execute."
+
+Reference: [cz-cli Configuration Guide](setup_cz_cli.md)
+
+### Replacing Spark or Traditional Big Data Architectures
+
+Migrate existing Spark ETL jobs to Lakehouse, replace Spark Streaming with Dynamic Tables, and replace PySpark scripts with SQL tasks. No Spark cluster to maintain, compute is pay-per-use, dramatically reducing total cost.
+
+Reference: [Spark SQL Migration Guide](migration-spark-sql.md)
+
+### Real-Time Data Platform
+
+Use multi-table real-time sync (CDC) to write business database changes into Lakehouse in real time; use Dynamic Tables to build ODS→DWD→ADS layered data pipelines with minute-level refresh, replacing the dual-pipeline Lambda architecture of Flink + data warehouse.
+
+Reference: [Real-Time Pipeline Selection Guide](realtime-pipeline-selection-guide.md)
+
+### Connected Vehicle / IoT Data Platform
+
+Massive device data continuously written via Kafka; Dynamic Tables aggregate device status and alert metrics in real time; analytical clusters support high-concurrency online queries — one architecture covers real-time ingestion, processing, and analytics end-to-end. Device anomaly data is processed by AI Functions for pattern recognition, with results written back to structured tables for real-time consumption by operations Agents.
+
+***
+
+## Ecosystem Compatibility
+
+Singdata Lakehouse natively supports Apache Iceberg format and supports direct read/write of Delta Lake, Hudi, and Paimon as external tables — existing data lake assets can be onboarded without migration. On the compute ecosystem side, [Spark Connector](spark-connector-summary.md), [Flink Connector](flink-write-connector.md), and Trino Connector are provided for smooth integration with existing big data pipelines. On the BI side, FineBI, Tableau, Superset, Metabase, and PowerBI have all completed integration certification.
+
+## Start Here
+
+| Your goal                                   | Recommended starting point                                                                             |
+| ------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
+| Understand all platform concept definitions | [Key Concepts](key-concepts.md)                                                                        |
+| Hands-on with your first complete workflow  | [Quick Start](lakehouse-quick-experience_guide.md)                                                     |
+| Understand the full AI Lakehouse picture    | [AI Lakehouse](lakehouse-ai.md)                                                                        |
+| Build RAG or vector search                  | [Full-Text + Vector Hybrid Search Best Practices](rrf-fulltext-vector-hybrid-search-best-practices.md) |
+| Understand AI data analytics capabilities   | [Data Analytics Agent Tour](lakehousedatagpt-tour.md)                                                  |
+| Let AI Agents operate the data warehouse    | [cz-cli Configuration Guide](setup_cz_cli.md)                                                          |
+| Learn common SQL commands                   | [SQL Reference](sql-reference.md)                                                                      |
+| Design data models and object relationships | [Object Model Design](object_model_design.md)                                                          |
+
+^

package/bin/skills/lakehouse-doc-en/references/concurrency_scaling.md

@@ -0,0 +1,56 @@
+# Support Multiple Concurrent Queries with Horizontal Elastic Scaling
+
+## Overview
+
+Through this tutorial, you will learn how to use the horizontal elastic scaling capability of the Lakehouse virtual compute cluster to support dynamically changing multiple concurrent queries from clients.
+
+> This tutorial leverages the Lakehouse Tutorial intelligent assistant, providing online tutorial guides and script import functions. After logging into the Lakehouse Web console, you can access the tutorial through the "Lakehouse Tutorial" entry in the Lakehouse Tutorial and follow the online guide to complete the tutorial.
+> ![](.topwrite/assets/image_1736854951691.png)
+
+### Import Scripts
+
+Open the "Lakehouse Tutorial" on the console's Tutorial page and select the "Support Multiple Concurrent Queries with Horizontal Elastic Scaling" course. Follow the prompts on the page to import the script files needed for this course.
+
+View the "Tutorial\_Working\_With\_Concurrency\_Scaling" directory in the "Development Module".
+
+![](.topwrite/assets/image_1725287877169.png =436)
+
+### Basic Knowledge
+
+A virtual compute cluster (Virtual Cluster, abbreviated as VC or cluster) is a computing resource object provided by Singdata Lakehouse for data processing and analysis. Virtual compute clusters provide the CPU, memory, and local temporary storage (SSD medium) resources needed to execute SQL jobs in Lakehouse. Clusters feature rapid creation/destruction, scaling up/down, pausing/resuming, and are charged based on resource specifications and usage duration, with no charges incurred after pausing or deletion. Virtual compute clusters offer two types of clusters, general-purpose and analytical, to meet the isolation and optimization needs of different workloads for ETL and analysis scenarios.
+
+![](.topwrite/assets/image_1714992266569.png =700)
+
+### Tutorial Steps
+
+1. **Environment Preparation**: Create a compute cluster for testing.
+2. **Initiate Queries**: Use the Studio Web environment to create Python tasks to perform continuous queries on Lakehouse with different concurrency levels, and observe the execution log results of the Python tasks to understand the cluster's rapid scaling capabilities under different concurrent requests.
+3. **Clean Up Environment**: Delete the compute cluster used for testing.
+
+Through the above steps, you will be able to understand how to configure and use the elastic concurrency feature of virtual clusters and understand the performance of elastic concurrency.
+
+## Preparation
+
+First, create an analytical compute cluster and enable and set the elastic concurrency feature through SQL commands.
+
+This tutorial creates a cluster and sets the elastic scaling policy by running the \[Tutorial\_Working\_With\_Concurrency\_Scaling->Step01.Preparation] SQL script task in the "Development" module.
+![](.topwrite/assets/image_1725288058473.png)
+
+## Initiate Concurrent Queries Using Python Program
+
+Open the \[Tutorial\_Working\_With\_Concurrency\_Scaling->Step02.Run\_Concurrent\_Queries] Python concurrent task template in the \[Development] module. You need to modify the connect connection configuration parameters before you can connect to Lakehouse and execute queries.
+
+![](https://studio-prod-sh.oss-cn-shanghai.aliyuncs.com/fe-asset/tutorials/resources/tu_concurrent_connection.png?OSSAccessKeyId=LTAI5tBH4MDxrfQw7VTx4w2B\&Expires=1880895431\&Signature=B2eODbnltYz%2FzjifxINWWzcf%2Bzw%3D)
+
+After modifying the connection information, please click to run the task and view the task execution log.
+
+![](https://studio-prod-sh.oss-cn-shanghai.aliyuncs.com/fe-asset/tutorials/resources/tu_concurrent_report.png?OSSAccessKeyId=LTAI5tBH4MDxrfQw7VTx4w2B\&Expires=1880982939\&Signature=%2B1YAL8o1IDjFdlaIpBgxB0Sa%2B90%3D)
+By observing the printed performance report, you can see that the Reporting\_VC cluster scales horizontally with millisecond-level latency by dynamically increasing the number of replicas as the client's concurrent requests increase. Dynamic scaling maintains the query SLA for continuous concurrent requests.
+
+While executing the task, you can also view the cluster's concurrent requests and elastic scaling status through the cluster monitoring page.
+![](https://studio-prod-sh.oss-cn-shanghai.aliyuncs.com/fe-asset/tutorials/resources/python_concurrency_scaling.gif?OSSAccessKeyId=LTAI5tBH4MDxrfQw7VTx4w2B\&Expires=2034444050\&Signature=wnBuZv1bereCq1etNW66OrF6UsA%3D)
+
+## Environment Cleanup
+
+Open the "Development" module \[Tutorial\_Working\_With\_Concurrency\_Scaling->Step03.Clean\_Up] SQL script file, execute the script to delete the test cluster for this tutorial.
+![](.topwrite/assets/image_1725288753867.png)

package/bin/skills/lakehouse-doc-en/references/conditional_function.md

@@ -0,0 +1 @@
+# Conditional Functions

package/bin/skills/lakehouse-doc-en/references/config-datasource.md

@@ -0,0 +1,82 @@
+# Data Source Management
+
+## Introduction
+
+This document aims to guide users on how to configure and manage data sources to achieve efficient data synchronization tasks in the Lakehouse product. Through this document, you will learn how to add different types of data sources and connect to data sources within a VPC via SSH Tunnel.
+
+## Steps
+
+1. **Log in and Access the Data Source Management Page**
+
+   First, log in to the Lakehouse product console. On the console homepage, select the "Management" module and click to enter the "Data Source" page. You will see a list of data sources showing all currently configured data sources.
+
+   ![](.topwrite/assets/image_1740137071389.png =640)
+
+2. **Create a New Data Source**
+
+   On the data source list page, click the "New Data Source" button in the upper right corner. At this point, you need to select the data source type. Lakehouse supports multiple data source types, including but not limited to MySQL, Oracle, SQL Server, etc.
+
+   ![](.topwrite/assets/image_1740137096346.png =640)
+
+
+3. **Fill in the Data Source Definition and Test Connectivity**
+
+   Based on the data source type you selected, fill in the corresponding connection information. For example, for a MySQL data source, you need to provide the database address, port, username, password, etc. After filling in, click the "Test Connection" button to ensure the information provided is correct.
+
+   ![](.topwrite/assets/image_1740137122891.png =640)
+
+## Connect to Data Sources within a VPC via SSH Tunnel
+
+### Prerequisites
+
+1. **VPC Data Source Access Restrictions**
+
+   Ensure that your VPC data source (such as Alibaba Cloud RDS for MySQL database) only provides internal network access. This means the data source is not publicly accessible, thereby enhancing data security.
+
+2. **SSH Tunnel Server Preparation**
+
+   Prepare an ECS server located within the VPC that has public network access capabilities and supports SSH login. Additionally, ensure that the data synchronization service can access this ECS server via the public network.
+
+3. **VPC Network Access Configuration**
+
+   Ensure that the ECS server within the VPC can access the data source within the VPC via internal network IP or domain name. This will facilitate the data synchronization service to connect to the data source smoothly when establishing the SSH Tunnel.
+
+### Configure Data Source Connection Parameters
+
+On the new data source page, enable the "Connect via SSH Tunnel" option. Fill in the following information as prompted:
+
+* **SSH Tunnel Server Public Address**: Enter the public IP address or domain name of the ECS server within the VPC.
+* **SSH Port**: Enter the port number of the SSH service on the ECS server (default is 22).
+* **SSH Username**: Enter the username for SSH login.
+* **SSH Password**: Enter the corresponding SSH login password.
+
+  ![](.topwrite/assets/image_1740137146102.png =640)
+
+
+After completing the above configuration, click the "Test Connection" button. If the connection is successful, you will see a green checkmark prompt. This means that the data synchronization service can establish a network connection with the data source within the VPC via SSH Tunnel when the task starts.
+
+## Usage Examples
+
+### Example 1: Connect to Alibaba Cloud RDS for MySQL Database
+
+1. On the data source type selection page, select "MySQL" as the data source type.
+2. On the data source definition page, fill in the following information:
+   * Hostname: Enter the internal IP address of Alibaba Cloud RDS for MySQL.
+   * Port: Enter the port number of the MySQL database (default is 3306).
+   * Username: Enter the database access username.
+   * Password: Enter the corresponding user password.
+3. Enable the "Connect via SSH Tunnel" option and fill in the relevant information of the ECS server within the VPC.
+4. Click "Test Connection". After confirming the connection is successful, click the "Save" button to complete the data source configuration.
+
+### Example 2: Connect to Oracle Database
+
+1. On the data source type selection page, select "Oracle" as the data source type.
+2. On the data source definition page, fill in the following information:
+   * Hostname: Enter the internal IP address of the Oracle database.
+   * Port: Enter the port number of the Oracle database (default is 1521).
+   * Username: Enter the database access username.
+   * Password: Enter the corresponding user password.
+3. Enable the "Connect via SSH Tunnel" option and fill in the relevant information of the ECS server within the VPC.
+4. Click "Test Connection". After confirming the connection is successful, click the "Save" button to complete the data source configuration.
+
+^