@clickzetta/cz-cli-darwin-arm64 0.5.15 → 0.5.17

package/bin/skills/lakehouse-doc-en/references/lakehouse-multi-cloud-acceleration.md

@@ -0,0 +1,274 @@
+# Multi-Cloud Unified Data Lake Acceleration
+
+The core concept of "data lake acceleration" is **no data migration**—directly querying and processing files in existing object storage using Singdata Lakehouse's Serverless compute, replacing traditional Spark/Hive ETL and Presto/Trino ad hoc queries.
+
+Alibaba Cloud OSS, Tencent Cloud COS, and AWS S3—the three mainstream object storage services—are unified through a single approach: **Volume mount → Pipe continuous ingestion → Dynamic Table incremental aggregation**. Aside from different parameter names when creating Storage Connections, all other SQL syntax is identical.
+
+---
+
+## Why a Multi-Cloud Unified Approach
+
+- Enterprise data is spread across multiple cloud providers and needs a unified query entry point
+- Different cloud object storage APIs differ, but Lakehouse data processing logic should remain consistent
+- Reduces the mental overhead and operational costs of switching between cloud environments
+
+Singdata Lakehouse's abstraction layer solves this perfectly: **the same SQL, with just a Connection parameter change, runs on different clouds**.
+
+---
+
+## SQL Commands Involved
+
+| Command / Function | Purpose | Multi-Cloud Differences |
+|------------|------|---------|
+| `CREATE STORAGE CONNECTION` | Establish object storage authentication channel | **The only step with differences** (different parameter names) |
+| `CREATE EXTERNAL VOLUME` | Mount object storage path | Syntax fully unified (only change protocol prefix) |
+| `COPY INTO VOLUME` | Export data to Volume | Fully unified |
+| `SELECT FROM VOLUME` | Directly query Volume files | Fully unified |
+| `DIRECTORY()` | List files in a Volume | Fully unified |
+| `COPY INTO` | Import data from Volume to table | Fully unified |
+| `CREATE PIPE` | Create continuous ingestion pipeline | Fully unified |
+| `ALTER PIPE` | Pause/resume Pipe | Fully unified |
+| `load_history()` | View historical load records | Fully unified |
+| `CREATE DYNAMIC TABLE` | Create incremental refresh aggregation table | Fully unified |
+| `REFRESH DYNAMIC TABLE` | Manually trigger refresh | Fully unified |
+
+---
+
+## Core Architecture
+
+![](.topwrite/assets/anim-15-multi-cloud-acceleration.svg)
+
+---
+
+## Cross-Cloud Comparison
+
+The exact same Volume + Pipe test cases were executed on Alibaba Cloud Shanghai (`f8866243`) and Tencent Cloud Shanghai (`0c3c358d`):
+
+### Configuration Differences
+
+| Configuration | Alibaba Cloud OSS | Tencent Cloud COS |
+|---|---|---|
+| **Connection type** | `TYPE OSS` | `TYPE COS` |
+| **Auth parameter names** | `access_id` / `access_key` (lowercase) | `ACCESS_KEY` / `SECRET_KEY` (uppercase) |
+| **Endpoint** | `ENDPOINT = 'oss-cn-shanghai.aliyuncs.com'` | No ENDPOINT required |
+| **Region parameter** | Embedded in ENDPOINT | `REGION = 'ap-shanghai'` |
+| **APP_ID** | Not applicable | `APP_ID = '1253896122'` |
+| **Volume location syntax** | `LOCATION 'oss://bucket/path/'` | `LOCATION 'cos://bucket-appid/path/'` |
+| **Recommended Endpoint** | `oss-cn-shanghai-internal.aliyuncs.com` (internal) | Auto-resolved, no configuration needed |
+
+> ⚠️ **Note**: Alibaba Cloud must use `access_id` / `access_key` or `ACCESS_KEY_ID` / `ACCESS_KEY_SECRET`. Do not use `ACCESS_KEY` / `SECRET_KEY` (missing `_ID` / `_SECRET` suffix). Tencent Cloud is the opposite—must use `ACCESS_KEY` / `SECRET_KEY`.
+
+### Feature Consistency
+
+| Test Item | Alibaba Cloud OSS | Tencent Cloud COS | Conclusion |
+|---|---|---|---|
+| Storage Connection creation | ✅ | ✅ | Different parameter names, otherwise identical |
+| External Volume creation | ✅ | ✅ | **Syntax fully identical** |
+| COPY INTO VOLUME export CSV | ✅ | ✅ | Identical |
+| COPY INTO VOLUME export Parquet | ✅ | ✅ | Identical |
+| DIRECTORY() file listing | ✅ | ✅ | Identical |
+| SELECT FROM VOLUME (CSV) | ✅ f0-f4 column names | ✅ f0-f4 column names | Identical |
+| SELECT FROM VOLUME (Parquet) | ✅ preserves column names | ✅ preserves column names | Identical |
+| COPY INTO TABLE from Volume | ✅ | ✅ | Identical |
+| PIPE LIST_PURGE creation | ✅ | ✅ | Identical |
+| PIPE load trigger | ✅ ~30s | ✅ ~30s | Identical |
+| PIPE PURGE delete source files | ✅ | ✅ | Identical |
+| PIPE load_history dedup | ✅ | ✅ | Identical |
+| PIPE pause/resume | ✅ | ✅ | Identical |
+| Dynamic Table incremental refresh | ✅ | ✅ | Identical |
+
+**Key finding**: Aside from different parameter names when creating the Connection, all other 12 test items are **completely identical**. Volume, Pipe, and Dynamic Table SQL syntax has no differences.
+
+---
+
+## Unified Implementation
+
+### Step 1: Create Storage Connection (the only step with differences)
+
+```sql
+-- ============ Alibaba Cloud OSS ============
+CREATE STORAGE CONNECTION IF NOT EXISTS my_storage_conn
+    TYPE OSS
+    access_id  = '<AccessKey ID>'
+    access_key = '<AccessKey Secret>'
+    ENDPOINT   = 'oss-cn-shanghai-internal.aliyuncs.com';
+
+-- ============ Tencent Cloud COS ============
+CREATE STORAGE CONNECTION IF NOT EXISTS my_storage_conn
+    TYPE COS
+    ACCESS_KEY = '<SecretId>'
+    SECRET_KEY = '<SecretKey>'
+    REGION     = 'ap-shanghai'
+    APP_ID     = '<APP_ID>';
+
+-- ============ AWS S3 ============
+CREATE STORAGE CONNECTION IF NOT EXISTS my_storage_conn
+    TYPE S3
+    ACCESS_KEY = '<Access Key ID>'
+    SECRET_KEY = '<Secret Access Key>'
+    REGION     = 'us-east-1';
+```
+
+> 💡 **Same-region internal network acceleration**: Alibaba Cloud uses `oss-cn-shanghai-internal.aliyuncs.com` (internal endpoint). AWS S3 and Tencent Cloud COS automatically use internal routing via the Region parameter. Internal network transfers have no egress fees and lower latency.
+
+### Step 2: Create External Volume (unified syntax across all three clouds)
+
+```sql
+-- ✅ Unified syntax for all three clouds: only change the protocol prefix in LOCATION
+CREATE EXTERNAL VOLUME my_data_vol
+    LOCATION 'oss://my-bucket/data/'           -- Alibaba Cloud: oss://
+    -- LOCATION 'cos://my-bucket-appid/data/'  -- Tencent Cloud: cos://
+    -- LOCATION 's3://my-bucket/data/'         -- AWS: s3://
+    USING CONNECTION my_storage_conn
+    DIRECTORY = (ENABLE = TRUE, AUTO_REFRESH = FALSE)
+    RECURSIVE = TRUE
+    COMMENT 'Multi-cloud unified data lake acceleration Volume';
+```
+
+### Step 3: Data Import/Export (fully unified)
+
+```sql
+-- Export to Volume (unified across all three clouds)
+COPY INTO VOLUME my_data_vol
+    SUBDIRECTORY 'export/'
+FROM TABLE source_table
+    FILE_FORMAT = (TYPE = PARQUET);
+
+-- Directly query from Volume (unified across all three clouds)
+SELECT * FROM VOLUME my_data_vol
+    USING PARQUET
+    FILES('export/part00001.parquet');
+
+-- Import to table (unified across all three clouds)
+COPY INTO target_table
+FROM VOLUME my_data_vol
+    USING PARQUET
+    SUBDIRECTORY 'export/';
+```
+
+### Step 4: Pipe Continuous Ingestion (fully unified)
+
+```sql
+-- Create dedicated Volume for Pipe
+CREATE EXTERNAL VOLUME pipe_vol
+    LOCATION 'oss://my-bucket/incoming/'   -- change only the protocol prefix across clouds
+    USING CONNECTION my_storage_conn
+    DIRECTORY = (ENABLE = TRUE, AUTO_REFRESH = TRUE)
+    RECURSIVE = TRUE;
+
+-- Create Pipe (unified across all three clouds)
+CREATE PIPE my_pipe
+    INGEST_MODE = 'LIST_PURGE'
+    VIRTUAL_CLUSTER = 'DEFAULT'
+    COMMENT 'Multi-cloud unified continuous ingestion pipeline'
+AS
+COPY INTO target_table
+FROM VOLUME pipe_vol
+USING CSV PURGE = TRUE;
+```
+
+### Step 5: Dynamic Table Incremental Aggregation (fully unified)
+
+```sql
+CREATE OR REPLACE DYNAMIC TABLE summary_table
+    REFRESH INTERVAL 1 DAY vcluster DEFAULT
+    COMMENT 'Multi-cloud unified aggregated metrics'
+AS
+SELECT category, COUNT(*) AS cnt, SUM(amount) AS total
+FROM target_table
+GROUP BY category;
+```
+
+---
+
+## Best Practices for Multi-Cloud Unification
+
+### 1. Code Reuse Strategy
+
+```
+Project directory structure:
+├── connections/
+│   ├── aliyun_oss.sql       ← only this file differs
+│   ├── tencent_cos.sql      ← only this file differs
+│   └── aws_s3.sql           ← only this file differs
+├── volumes/
+│   └── create_volumes.sql   ← universal across all three clouds (protocol prefix can be replaced via variable)
+├── tables/
+│   └── ddl.sql              ← fully unified
+├── pipes/
+│   └── create_pipes.sql     ← fully unified
+└── dynamic_tables/
+    └── aggregates.sql       ← fully unified
+```
+
+**Only the Connection creation SQL needs to be written per cloud.** The remaining 90% of the code can be reused directly.
+
+### 2. Naming Convention Recommendations
+
+| Object | Convention | Example |
+|---|---|---|
+| Storage Connection | `<cloud>_<purpose>` | `oss_prod_conn`, `cos_archive_conn` |
+| External Volume | `<source_system>_vol` | `orders_vol`, `logs_vol` |
+| Pipe | `<source_system>_pipe` | `orders_pipe`, `logs_pipe` |
+
+Do not embed cloud provider information in Volume/Pipe names—these objects may appear in multi-cloud reuse scenarios.
+
+### 3. Cost Optimization
+
+| Strategy | Description |
+|---|---|
+| Use internal Endpoints | Alibaba Cloud: `*-internal.aliyuncs.com`, no egress fees, lower latency |
+| T+1 refresh frequency | Most analytics scenarios do not need minute-level refresh; `1 DAY` is sufficient |
+| PURGE=true | LIST_PURGE mode auto-deletes source files, prevents OSS/COS storage accumulation |
+| GP type Virtual Cluster | Use `DEFAULT` (GENERAL type), Serverless on-demand billing |
+| Keep file size at 128-256MB | Large CSV/Parquet files are 3-5x more efficient than many small files |
+
+### 4. Security Recommendations
+
+- Do not use root account AK/SK; create sub-accounts with minimum permissions (Bucket read + specific directory write)
+- Internal Endpoints can bind VPC policies to restrict access sources
+- AK/SK in Storage Connection is not visible in `SHOW STORAGE CONNECTIONS` (masked)
+- When using AWS S3, prefer IAM Role (`ROLE_ARN`) over long-lived AK/SK
+
+---
+
+## FAQ
+
+### Q: How is cross-cloud data transfer latency calculated?
+
+Volume does not migrate data—files always remain in the original object storage. Query network latency depends on the link from the Lakehouse instance's region to the object storage. **Same-region internal Endpoint is recommended** (such as `oss-cn-shanghai-internal.aliyuncs.com`), with latency typically under 10ms.
+
+### Q: Can you access Tencent Cloud COS from an Alibaba Cloud instance?
+
+**Not supported.** Storage Connection must be in the same cloud provider as the Lakehouse instance. For cross-cloud queries, the following alternatives are available:
+
+1. Create a Lakehouse instance in the target cloud, use External Catalog for federated queries
+2. Sync data cross-cloud to Lakehouse internal tables via data integration (Studio Sync)
+3. Use Private Link to bridge the network, then access via External Schema
+
+### Q: Is EVENT_NOTIFICATION mode for Pipe supported on all three clouds?
+
+| Cloud | LIST_PURGE | EVENT_NOTIFICATION |
+|---|---|---|
+| Alibaba Cloud OSS | ✅ | ✅ (requires MNS queue configuration) |
+| Tencent Cloud COS | ✅ | ❌ |
+| AWS S3 | ✅ | ✅ (requires SQS queue configuration) |
+
+`EVENT_NOTIFICATION` mode supports Alibaba Cloud OSS and AWS S3, with lower latency (near real-time) and without deleting source files. Tencent Cloud COS does not support it yet.
+
+### Q: Why does COPY INTO VOLUME require SUBDIRECTORY?
+
+`COPY INTO VOLUME` without the `SUBDIRECTORY` clause will throw `Syntax error at or near 'FROM'`. This is a mandatory SQL parser requirement, unrelated to the cloud platform. To export to the Volume root path, use `SUBDIRECTORY '/'`.
+
+---
+
+## Related Documents
+
+- [Volume + Pipe Data Lake Acceleration Practice](lakehouse-volume-pipe-acceleration-guide.md) — Alibaba Cloud OSS end-to-end walkthrough
+- [Medallion Pure SQL DT Architecture](lakehouse-medallion-sql-dt-guide.md) — Bronze→Silver→Gold three-layer modeling
+- [Create Storage Connection](create-storage-connection.md) — Complete syntax for all three clouds
+- [Create External Volume](create-external-volume.md) — Parameter details
+- [Pipe Overview](pipe-overview.md) — LIST_PURGE vs EVENT_NOTIFICATION
+- [Alibaba Cloud OSS Connection](aliyun_storage_connection.md) — OSS-specific configuration
+- [Tencent Cloud COS Connection](cos_storage_connection.md) — COS-specific configuration
+- [AWS S3 Connection](aws_storage_connection.md) — S3-specific configuration

package/bin/skills/lakehouse-doc-en/references/lakehouse-multimodal-ai-pipeline.md

@@ -0,0 +1,198 @@
+# AI Gateway in Action: Analyzing Images in SQL to Build a Multimodal AI Pipeline
+
+The image mode of `AI_COMPLETE` allows SQL to handle not just text, but also directly analyze image content—product image classification, document OCR recognition, monitoring screenshot anomaly detection, all completed in a single SQL query. Combined with **Volume storage + presigned URLs**, it forms a fully managed pipeline from image upload to AI analysis.
+
+This article uses the Kimi K2.6 multimodal model as an example to demonstrate how to build an end-to-end multimodal AI pipeline in Singdata Lakehouse.
+
+---
+
+## Pipeline Architecture
+
+![](.topwrite/assets/12-multimodal-ai-pipeline.svg)
+
+Three key steps:
+1. **`PUT`** uploads images to User Volume (internal storage, user-isolated)
+2. **`GET_PRESIGNED_URL`** generates a time-limited OSS signed URL so the LLM can access it
+3. **`AI_COMPLETE` image mode** sends the image URL + Prompt to get analysis results
+
+---
+
+## Upload Images to Volume
+
+```sql
+-- Upload a single image to User Volume
+PUT '/path/to/product.jpg' TO USER VOLUME FILE 'images/product.jpg';
+
+-- Upload to a subdirectory
+PUT '/path/to/screenshot.png' TO USER VOLUME FILE 'screenshots/error_001.png';
+```
+
+> User Volume is private to the user; other users cannot access it. For team-shared images, use an External Volume mounting OSS/COS/S3.
+
+---
+
+## Generate Presigned URL
+
+`GET_PRESIGNED_URL` generates a time-limited OSS signed URL for files in the Volume, which the LLM uses to download the image.
+
+```sql
+-- ⚠️ Must enable external mode first, otherwise an internal URL is generated (LLM cannot access)
+SET cz.sql.function.get.presigned.url.force.external = true;
+
+-- Generate a presigned URL valid for 1 hour
+SELECT GET_PRESIGNED_URL(USER VOLUME, 'images/product.jpg', 3600) AS url;
+```
+
+Example return value:
+```
+http://cz-lh-sh-system-prod.oss-cn-shanghai.aliyuncs.com/.../product.jpg
+  ?Expires=1780392774&OSSAccessKeyId=...&Signature=...
+```
+
+Parameter descriptions:
+- First parameter: Volume type and name (`USER VOLUME` / `VOLUME my_vol`)
+- Second parameter: File path within the Volume
+- Third parameter: Validity period (seconds), recommended 3600 (1 hour), maximum 7 days
+
+---
+
+## AI_COMPLETE Image Mode
+
+Image mode uses the **row value syntax** `(prompt AS prompt, image_url AS image)` to pass both the Prompt and image URL together:
+
+```sql
+SET cz.sql.function.get.presigned.url.force.external = true;
+
+SELECT AI_COMPLETE('ai_gateway_conn:moonshotai/kimi-k2.6',
+    ('Describe this image in one sentence' AS prompt,
+     GET_PRESIGNED_URL(USER VOLUME, 'ai_test/test_icon.png', 3600) AS image)
+);
+```
+
+Output (1.3 seconds):
+
+> The image is completely black, showing nothing visible.
+
+### Syntax Details
+
+| Parameter | Format | Description |
+|---|---|---|
+| model | `'ai_gateway_conn:moonshotai/kimi-k2.6'` | Must use a model that supports multimodal capabilities |
+| prompt | `'Describe this image' AS prompt` | Text prompt; `AS prompt` cannot be omitted |
+| image | `GET_PRESIGNED_URL(...) AS image` | Image URL; `AS image` cannot be omitted |
+
+### Supported Models
+
+Image mode requires a model with multimodal (vision) support, such as `moonshotai/kimi-k2.6` (confirmed working, 1.3s response). Text-only models will not throw an error when passed an image URL but cannot recognize image content.
+
+---
+
+## In Practice: Batch Image Analysis
+
+### Scenario: Automatic Product Image Classification
+
+Suppose an e-commerce system uploads thousands of product images to OSS daily and needs to automatically identify the product type in each image.
+
+```sql
+-- 1. Create table to store analysis results
+CREATE TABLE product_image_analysis (
+    image_path  STRING  COMMENT 'Image path',
+    category    STRING  COMMENT 'AI-recognized category',
+    description STRING  COMMENT 'AI description',
+    analyzed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP()
+);
+
+-- 2. Read image list from External Volume, analyze in batch
+SET cz.sql.function.get.presigned.url.force.external = true;
+
+INSERT INTO product_image_analysis (image_path, category, description)
+SELECT
+    relative_path AS image_path,
+    -- Classification
+    AI_COMPLETE('ai_gateway_conn:moonshotai/kimi-k2.6',
+        ('Classify the product into one of the following categories: electronics/clothing/food/furniture. Reply with the category name only.'
+         AS prompt,
+         GET_PRESIGNED_URL(VOLUME product_images_vol, relative_path, 3600) AS image)
+    ) AS category,
+    -- Description
+    AI_COMPLETE('ai_gateway_conn:moonshotai/kimi-k2.6',
+        ('Describe this product in one sentence' AS prompt,
+         GET_PRESIGNED_URL(VOLUME product_images_vol, relative_path, 3600) AS image)
+    ) AS description
+FROM DIRECTORY(VOLUME product_images_vol)
+WHERE relative_path LIKE '%.jpg'
+    AND last_modified_time > CURRENT_TIMESTAMP() - INTERVAL 1 DAY;
+```
+
+### Query Results
+
+```sql
+SELECT category, COUNT(*) AS cnt
+FROM product_image_analysis
+GROUP BY category
+ORDER BY cnt DESC;
+```
+
+### Dynamic Table for Automatic Incremental Analysis
+
+Wrap the above logic in a Dynamic Table so new images are automatically analyzed after upload:
+
+```sql
+SET cz.sql.function.get.presigned.url.force.external = true;
+
+CREATE OR REPLACE DYNAMIC TABLE gold.product_image_tags
+REFRESH INTERVAL 1 HOUR vcluster DEFAULT
+COMMENT 'Product image AI tags — auto incremental update'
+AS
+SELECT
+    relative_path AS image_path,
+    AI_COMPLETE('ai_gateway_conn:moonshotai/kimi-k2.6',
+        ('Classify the product as electronics/clothing/food/furniture, reply with category name only.'
+         AS prompt,
+         GET_PRESIGNED_URL(VOLUME product_images_vol, relative_path, 3600) AS image)
+    ) AS category,
+    last_modified_time
+FROM DIRECTORY(VOLUME product_images_vol)
+WHERE relative_path LIKE '%.jpg';
+```
+
+> ⚠️ **Note**: DT re-calls AI_COMPLETE for all rows on every REFRESH, which can be costly. It is recommended to use a regular table + scheduled INSERT instead, analyzing only newly added images.
+
+---
+
+## Applicable Scenarios
+
+| Scenario | Implementation |
+|---|---|
+| Automatic product image classification | AI_COMPLETE image mode + classification Prompt |
+| Document OCR extraction | "Extract all text from the image" |
+| Monitoring screenshot anomaly detection | "Determine if there is an anomaly in the image, reply normal/abnormal only" |
+| Social media image content moderation | "Determine if the image content violates rules, reply safe/unsafe only" |
+| Invoice/document information extraction | "Extract the amount, date, and company name from the image" |
+| Preliminary medical imaging screening | "Determine if there is an abnormal area in the image, reply yes/no only" |
+
+---
+
+## Notes
+
+| Note | Description |
+|---|---|
+| **Presigned URL must use force.external** | `SET cz.sql.function.get.presigned.url.force.external = true`, otherwise generates internal URL |
+| **Image formats** | Supports PNG, JPG, JPEG. SVG has a null MIME type, causing `Invalid image MIME type` error |
+| **Model selection** | Must use a multimodal (vision) model. Text-only models won't error but cannot recognize image content |
+| **URL validity** | Presigned URL default recommendation is 3600 seconds (1 hour), maximum 7 days |
+| **DT cost** | AI_COMPLETE in DT fully recomputes on every REFRESH; use regular table + manual INSERT instead |
+| **Concurrency limits** | For large-scale concurrent image analysis, watch AI Gateway TPM/RPM quota limits |
+| **File size** | Images recommended to be < 10MB; oversized images may time out or consume too many tokens |
+| **User Volume isolation** | User Volume is private; other users cannot see images you upload |
+
+---
+
+## Related Documents
+
+- [AI_COMPLETE documentation](ai_complete.md) — Image mode parameters, options configuration
+- [AI-Enhanced Data Analysis](lakehouse-ai-sql-analysis.md) — AI_COMPLETE text mode in practice
+- [Volume Overview](volume-overview.md) — User Volume and External Volume concepts
+- [GET_PRESIGNED_URL](put_get_volume.md) — Presigned URL generation
+- [AI Gateway Product Overview](AI_Gateway.md) — Endpoint management, quota control
+- [Multi-Cloud Unified Data Lake Acceleration](lakehouse-multi-cloud-acceleration.md) — Volume + Pipe cross-cloud approach