@clickzetta/cz-cli-darwin-arm64 0.3.94 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/vector_data_process_guide.md

@@ -0,0 +1,952 @@
+# Lakehouse VECTOR Data Processing User Guide
+
+## Native Support and Practice for Vector Data
+
+***
+
+## Introduction
+
+### Vector Data Processing Scenarios You May Encounter
+
+#### Scenario 1: Document Similarity Search System
+
+**Current Common Practice in Hive/Spark**:
+
+```sql
+-- Storing document vectors in Hive
+CREATE TABLE document_embeddings (
+    doc_id STRING,
+    title STRING, 
+    content STRING,
+    embedding ARRAY<DOUBLE>  -- Store vectors generated by LLM
+) PARTITIONED BY (dt STRING);
+
+-- Similarity calculation requires complex UDF
+CREATE FUNCTION cosine_similarity_udf(vec1 ARRAY<DOUBLE>, vec2 ARRAY<DOUBLE>) 
+RETURNS DOUBLE
+LANGUAGE PYTHON AS $
+import numpy as np
+def cosine_similarity_udf(vec1, vec2):
+    return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))
+$;
+
+-- Query requires manual dimension check and UDF call
+SELECT doc_id, title,
+    cosine_similarity_udf(embedding, array(0.1, 0.2, 0.3, ...)) as score
+FROM document_embeddings 
+WHERE size(embedding) = 1536  -- Manual dimension check
+    AND dt >= '2024-01-01'
+ORDER BY score DESC 
+LIMIT 10;
+```
+
+**Technical Challenges**:
+
+* **Complex implementation**: Requires custom UDFs to implement distance calculation functions
+* **No dedicated index**: Array types cannot have similarity-specific indexes
+* **Type safety issues**: Array length is not fixed, prone to dimension mismatch errors
+* **Performance limitations**: Lack of specialized optimization for large-scale vector computation
+
+#### Scenario 2: Recommendation System User Feature Matching
+
+**Typical Implementation in Spark**:
+
+```python
+# User vector similarity calculation in Spark
+from pyspark.sql.functions import *
+from pyspark.ml.linalg import Vectors
+
+# Requires custom function to implement similarity calculation
+def cosine_similarity_udf(vec1, vec2):
+    from pyspark.ml.linalg import Vectors
+    import numpy as np
+    v1 = np.array(vec1)
+    v2 = np.array(vec2)
+    return float(np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2)))
+
+# Register UDF
+cosine_udf = udf(cosine_similarity_udf, DoubleType())
+
+# Query similar users
+similar_users = user_features.crossJoin(user_features.alias("other")) \
+    .filter(col("user_id") != col("other.user_id")) \
+    .withColumn("similarity", cosine_udf(col("features"), col("other.features"))) \
+    .filter(col("similarity") > 0.8) \
+    .orderBy(desc("similarity"))
+```
+
+**Common Technical Pain Points**:
+
+* **Complex architecture**: Vector computation is separated from data storage, increasing system complexity
+* **Development cost**: Requires implementing and maintaining custom similarity calculation logic
+* **Performance bottleneck**: Lack of dedicated acceleration mechanisms for large-scale vector computation
+* **Error handling**: No compile-time check for vector dimensions
+
+### Why Vector Data Type is Needed
+
+Based on the technical challenges in the above scenarios, the design goals of the VECTOR data type are:
+
+**Solving Type Safety Issues**:
+
+* Provide fixed-dimension vector type definition
+* Compile-time dimension matching check
+* Avoid runtime dimension mismatch errors
+
+**Simplifying Development Process**:
+
+* Built-in common vector distance calculation functions
+* No need to write and maintain custom UDFs
+* Fully compatible with standard SQL syntax
+
+**Improving Query Performance**:
+
+* Support for dedicated vector indexes
+* Optimized vector computation
+* Support for efficient similarity search
+
+### How to Use This Guide
+
+| Reader Role         | Recommended Focus                              | Expected Takeaway                          |
+| ------------------- | ---------------------------------------------- | ------------------------------------------ |
+| Data Engineer       | Type Definition -> Function Usage -> Pitfall Guide | Master correct VECTOR usage methods        |
+| System Architect    | Application Scenarios -> Performance Optimization -> Migration Strategy | Obtain architecture design and migration planning references |
+| AI Application Developer | Quick Start -> Application Patterns -> Query Limitations | Understand best practices in AI scenarios |
+
+***
+
+## Quick Start
+
+### Basic Usage Flow
+
+```sql
+-- 1. Create a table with a vector column
+CREATE TABLE document_embeddings (
+    id BIGINT,
+    content STRING,
+    embedding VECTOR(FLOAT, 1536)  -- 1536-dimensional float vector
+);
+
+-- 2. Insert vector data
+INSERT INTO document_embeddings VALUES 
+(1, 'Artificial Intelligence Technology Development', vector(0.1, 0.2, 0.3, ...));  -- Requires providing complete 1536-dimensional data
+
+-- 3. Create a vector index
+CREATE VECTOR INDEX embedding_idx ON TABLE document_embeddings(embedding) 
+PROPERTIES(
+    "distance.function" = "cosine_distance",
+    "scalar.type" = "f32"
+);
+
+-- 4. Build index (for existing data)
+BUILD INDEX embedding_idx ON document_embeddings;
+
+-- 5. Execute similarity query (recommended pattern)
+SELECT id, content FROM document_embeddings 
+WHERE COSINE_DISTANCE(embedding, vector(0.1, 0.2, 0.3, ...)) < 0.8
+ORDER BY id LIMIT 10;
+```
+
+***
+
+## Technical Comparison: ARRAY vs VECTOR
+
+### Traditional ARRAY Implementation
+
+```sql
+-- Using ARRAY to store vectors
+CREATE TABLE docs_array (
+    content STRING,
+    embedding ARRAY<DOUBLE>
+);
+
+-- Requires custom function to implement similarity calculation
+CREATE FUNCTION cosine_similarity(vec1 ARRAY<DOUBLE>, vec2 ARRAY<DOUBLE>) 
+RETURNS DOUBLE
+LANGUAGE PYTHON AS $$
+def cosine_similarity(vec1, vec2):
+    import numpy as np
+    return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))
+$$;
+
+-- Query requires function call and manual dimension check
+SELECT doc_id, cosine_similarity(embedding, array(0.1, 0.2, ...)) as score
+FROM docs_array
+WHERE array_size(embedding) = 1536  -- Requires manual dimension check
+ORDER BY score DESC;
+```
+
+### VECTOR Native Implementation
+
+```sql
+-- Using VECTOR type
+CREATE TABLE docs_vector (
+    content STRING,
+    embedding VECTOR(FLOAT, 1536),  -- Type and dimension explicitly defined
+    
+    INDEX embedding_idx (embedding) USING VECTOR PROPERTIES (
+        "distance.function" = "cosine_distance"
+    )
+);
+
+-- Using built-in functions, dimension check is automatic
+SELECT doc_id FROM docs_vector 
+WHERE COSINE_DISTANCE(embedding, vector(0.1, 0.2, ...)) < 0.8
+ORDER BY doc_id LIMIT 10;
+```
+
+### Feature Comparison
+
+| Feature               | ARRAY Solution         | VECTOR Solution        | Impact on Users                         |
+| --------------------- | ---------------------- | ---------------------- | --------------------------------------- |
+| Type Safety           | Runtime check          | Compile-time check     | Reduces production environment errors   |
+| Dimension Control     | Manual verification    | Type definition guaranteed | Avoids dimension mismatch issues    |
+| Distance Calculation  | Custom UDF             | Built-in functions     | Reduces code development and maintenance |
+| Index Support         | No dedicated index     | Vector index           | Improves large-scale query performance  |
+| Development Complexity| Requires implementation logic | Directly use built-in functions | Lowers technical barrier     |
+| Maintenance Cost      | Requires maintaining UDF code | Native database support | Reduces operational workload        |
+
+### Migration Comparison from Spark MLlib
+
+| Spark MLlib API          | Singdata VECTOR                 | Migration Complexity | Description                     |
+| ------------------------ | ------------------------------- | -------------------- | ------------------------------- |
+| `Vectors.dense(array)`   | `vector(v1, v2, ...)`           | Easy                 | Direct function replacement     |
+| `Vectors.norm(v, 2)`     | `L2_DISTANCE(v, vector(0,...))` | Medium               | Requires providing zero vector  |
+| Custom cosine UDF        | `COSINE_DISTANCE(v1, v2)`       | Easy                 | Remove UDF, use built-in function |
+| Custom normalize UDF     | `L2_NORMALIZE(v)`               | Easy                 | Remove UDF, use built-in function |
+| `Vectors.sqdist(v1, v2)` | `POW(L2_DISTANCE(v1, v2), 2)`   | Easy                 | Combine built-in functions       |
+
+***
+
+## VECTOR Data Type Specification
+
+### Type Definition Syntax
+
+```sql
+VECTOR(scalar_type, dimension)
+```
+
+### Supported Scalar Types
+
+| Scalar Type | Storage Size | Value Range             | Applicable Scenario                        | Index Support |
+| ----------- | ------------ | ----------------------- | ------------------------------------------ | ------------- |
+| FLOAT       | 4 bytes/dim  | 32-bit floating point   | General vector storage, e.g. embeddings    | f32           |
+| INT         | 4 bytes/dim  | 32-bit integer          | Integer feature vectors, e.g. counts       | Not supported |
+| TINYINT     | 1 byte/dim   | -128 to 127             | Compressed storage, e.g. binary vectors    | i8            |
+
+> **Suggestion**: FLOAT type is recommended for production environments, offering the best compatibility and performance
+
+### Vector Creation Methods
+
+```sql
+-- Direct vector creation
+SELECT vector(1.0, 2.0, 3.0);              -- Create 3-dimensional float vector
+SELECT vector(1, 2, 3);                     -- Create 3-dimensional integer vector
+
+-- Conversion from other types
+SELECT cast(array(1,2,3) as VECTOR(INT, 3));     -- Array to vector
+SELECT cast('[1,2,3]' as VECTOR(INT, 3));        -- JSON string to vector
+
+-- Usage in table definition
+CREATE TABLE feature_vectors (
+    id BIGINT,
+    dense_features VECTOR(FLOAT, 128),       -- Dense features
+    sparse_features VECTOR(INT, 1000)        -- Sparse features
+);
+```
+
+***
+
+## Vector Functions
+
+### Distance Calculation Functions
+
+#### L2\_DISTANCE - Euclidean Distance
+
+```sql
+SELECT L2_DISTANCE(vector(1,2,3), vector(4,5,6)) as euclidean_distance;
+-- Result: 5.196152210235596
+```
+
+**Formula**: sqrt(sum((ai - bi)^2))
+**Applicable Scenarios**: Spatial data analysis, clustering algorithms, image feature matching
+
+#### COSINE\_DISTANCE - Cosine Distance
+
+```sql
+SELECT COSINE_DISTANCE(vector(1,2,3), vector(4,5,6)) as cosine_distance;
+-- Result: 0.025368213653564453
+```
+
+**Formula**: 1 - (a.b)/(|a|*|b|)
+**Applicable Scenarios**: Text semantic similarity, recommendation systems, information retrieval
+
+#### DOT\_PRODUCT - Dot Product
+
+```sql
+SELECT DOT_PRODUCT(vector(1,2,3), vector(4,5,6)) as dot_product;
+-- Result: 32.0
+```
+
+**Formula**: sum(ai * bi)
+**Applicable Scenarios**: Correlation calculation, neural network weight computation
+
+### Vector Processing Functions
+
+#### L2\_NORMALIZE - Vector Normalization
+
+```sql
+-- IMPORTANT: Must use float vectors
+SELECT L2_NORMALIZE(vector(3.0, 4.0)) as normalized_vector;
+-- Result: [0.6, 0.8]
+
+-- Incorrect example: Integer vectors may cause precision issues
+SELECT L2_NORMALIZE(vector(3, 4)) as normalized_vector;
+-- Possible result: [0, 0]
+```
+
+**Function**: Normalize vector to unit length (L2 norm = 1)
+**Applicable Scenarios**: Preprocessing for cosine similarity calculation
+
+### Function Usage Notes
+
+* All distance functions require input vectors to have the same dimension; otherwise, a compile-time error will be raised
+* Functions support mixed computation of vector type and array type
+* Normalization function returns a vector type of the same dimension
+* **L2\_NORMALIZE function is sensitive to data type; float vectors are recommended**
+
+***
+
+## Type Conversion and Compatibility
+
+### Automatic Type Conversion
+
+```sql
+-- Vectors can be implicitly converted to arrays for array operations
+SELECT array_size(vector(1,2,3,4)) as vector_size;
+-- Result: 4
+
+-- Arrays can be used in vector functions
+SELECT L2_DISTANCE(array(1,2,3), vector(1,2,3)) as distance;
+-- Result: 0.0
+```
+
+### Explicit Type Conversion
+
+```sql
+-- Array to vector (dimensions must match)
+SELECT cast(array(1.0, 2.0, 3.0) as VECTOR(FLOAT, 3));
+-- Result: [1.0, 2.0, 3.0]
+
+-- JSON string to vector
+SELECT cast('[1,2,3]' as VECTOR(INT, 3));
+-- Result: [1, 2, 3]
+
+-- Returns NULL when dimensions don't match
+SELECT cast(array(1,2) as VECTOR(INT, 3));  -- Returns NULL
+```
+
+### SQL Operation Support
+
+Vector type supports standard SQL operations:
+
+```sql
+-- Sorting
+SELECT * FROM vectors ORDER BY feature_vector;
+
+-- Grouping
+SELECT feature_vector, COUNT(*) FROM vectors GROUP BY feature_vector;
+
+-- Comparison
+SELECT * FROM vectors WHERE feature_vector = vector(1,2,3);
+
+-- Deduplication
+SELECT DISTINCT feature_vector FROM vectors;
+```
+
+***
+
+## Vector Index
+
+### Index Configuration Parameters
+
+| Parameter Name       | Available Values                                                      | Default Value      | Description                                  |
+| -------------------- | --------------------------------------------------------------------- | ------------------ | -------------------------------------------- |
+| distance.function    | cosine\_distance, l2\_distance, jaccard\_distance, hamming\_distance  | cosine\_distance   | Distance function type                       |
+| scalar.type          | f32, f16, i8, b1                                                      | f32                | Storage type of vector elements in the index |
+| M                    | Positive integer                                                      | 16                 | Maximum number of connections for HNSW algorithm |
+| ef\_construction     | Positive integer                                                      | 200                | Candidate set size during index construction |
+
+> ⚠️ **Note**: Parameter names must use the full form, e.g. `cosine_distance` instead of `cosine`
+
+### Index Creation and Management
+
+```sql
+-- Create vector index (recommended syntax)
+CREATE VECTOR INDEX embedding_idx ON TABLE table_name(vector_column) 
+PROPERTIES(
+    "distance.function" = "cosine_distance",  -- Full function name
+    "scalar.type" = "f32",                   -- Standard type name
+    "M" = 16,
+    "ef_construction" = 200
+);
+
+-- Build index for existing data
+BUILD INDEX embedding_idx ON table_name;
+
+-- View index status
+SHOW INDEXES ON table_name;
+
+-- Drop index
+DROP INDEX embedding_idx ON table_name;
+```
+
+### Index Performance Tuning
+
+Performance characteristics of different parameter configurations:
+
+| Configuration Scenario | M Value | ef\_construction | scalar.type | Characteristics                             |
+| ---------------------- | ------- | ---------------- | ----------- | ------------------------------------------- |
+| Fast Retrieval         | 8       | 64               | f16         | Lower construction time and memory usage    |
+| Balanced               | 16      | 200              | f32         | Balanced precision and performance          |
+| High Precision         | 32      | 400              | f32         | Higher retrieval precision, higher resource consumption |
+
+***
+
+## Mainstream AI Model Adaptation
+
+### Vector Dimensions of Common Embedding Models
+
+| Model Provider | Model Name                 | Dimension | Recommended Configuration |
+| -------------- | -------------------------- | --------- | ------------------------- |
+| OpenAI         | text-embedding-3-small     | 1536      | VECTOR(FLOAT, 1536)       |
+| OpenAI         | text-embedding-3-large     | 3072      | VECTOR(FLOAT, 3072)       |
+| Google         | Universal Sentence Encoder | 512       | VECTOR(FLOAT, 512)        |
+| Baidu          | ERNIE-Embed                | 768       | VECTOR(FLOAT, 768)        |
+| Zhipu          | GLM-Embedding              | 1024      | VECTOR(FLOAT, 1024)       |
+
+### Multimodal Model Support
+
+| Model Type          | Typical Dimension | Application Scenario                    |
+| ------------------- | ----------------- | --------------------------------------- |
+| CLIP                | 512/768           | Image-text matching, multimodal retrieval |
+| BLIP                | 768               | Image Q&A, visual understanding         |
+| Vision Transformer  | 768/1024          | Image classification, feature extraction |
+
+***
+
+## Application Scenario Design
+
+### Document Retrieval System
+
+```sql
+-- Knowledge base table design
+CREATE TABLE knowledge_base (
+    doc_id BIGINT PRIMARY KEY,
+    title STRING,
+    content STRING,
+    embedding VECTOR(FLOAT, 1536),
+    created_at TIMESTAMP,
+    
+    INDEX doc_embedding_idx (embedding) USING VECTOR PROPERTIES (
+        "scalar.type" = "f32",
+        "distance.function" = "cosine_distance"
+    )
+);
+
+-- Semantic retrieval query (recommended pattern)
+SELECT doc_id, title FROM knowledge_base 
+WHERE COSINE_DISTANCE(embedding, ?) < 0.7
+ORDER BY doc_id LIMIT 10;
+```
+
+### Recommendation System
+
+```sql
+-- User profile table
+CREATE TABLE user_profiles (
+    user_id BIGINT PRIMARY KEY,
+    behavior_vector VECTOR(FLOAT, 128),
+    preference_vector VECTOR(FLOAT, 64),
+    last_updated TIMESTAMP,
+    
+    INDEX behavior_idx (behavior_vector) USING VECTOR PROPERTIES (
+        "distance.function" = "cosine_distance"
+    )
+);
+
+-- Item feature table
+CREATE TABLE item_features (
+    item_id BIGINT PRIMARY KEY,
+    content_vector VECTOR(FLOAT, 256),
+    category_vector VECTOR(INT, 50),
+    
+    INDEX content_idx (content_vector) USING VECTOR PROPERTIES (
+        "distance.function" = "cosine_distance"
+    )
+);
+
+-- Find similar users (recommended query pattern)
+SELECT target.user_id as similar_user FROM user_profiles source
+CROSS JOIN user_profiles target
+WHERE source.user_id = ?
+    AND target.user_id != source.user_id
+    AND COSINE_DISTANCE(source.behavior_vector, target.behavior_vector) < 0.6
+ORDER BY target.user_id LIMIT 20;
+```
+
+### Image Feature Matching
+
+```sql
+-- Image feature library
+CREATE TABLE image_features (
+    image_id BIGINT PRIMARY KEY,
+    image_path STRING,
+    visual_features VECTOR(FLOAT, 2048),
+    semantic_features VECTOR(FLOAT, 512),
+    upload_time TIMESTAMP,
+    
+    INDEX visual_idx (visual_features) USING VECTOR PROPERTIES (
+        "distance.function" = "l2_distance",
+        "scalar.type" = "f16"
+    ),
+    INDEX semantic_idx (semantic_features) USING VECTOR PROPERTIES (
+        "distance.function" = "cosine_distance"
+    )
+);
+
+-- Similar image retrieval
+SELECT image_id, image_path FROM image_features
+WHERE L2_DISTANCE(visual_features, ?) < 100.0
+ORDER BY image_id LIMIT 50;
+```
+
+***
+
+## Important Pitfall Avoidance Guide
+
+### Vector Query Limitations
+
+Through actual testing in the UAT environment, two different error patterns were discovered:
+
+#### 1. Dimension Mismatch Error (Expected Behavior)
+
+```sql
+-- Incorrect example: Using a 5-dimensional vector to query 128-dimensional data
+SELECT id FROM embeddings_table 
+WHERE COSINE_DISTANCE(embedding_128d, vector(0.1, 0.2, 0.3, 0.4, 0.5)) < 0.8;
+
+-- Error message:
+-- function 'COSINE_DISTANCE' cannot be resolved, 
+-- expect 'vector shall have the same dimension'
+```
+
+**Solution**: Ensure the query vector has exactly the same dimension as the vector in the table
+
+#### 2. Generated Column Internal Error (Current Limitation)
+
+```sql
+-- Problematic query: Returning distance calculation in SELECT
+SELECT id, content,
+    COSINE_DISTANCE(embedding, vector(...)) as distance
+FROM embeddings_table 
+ORDER BY distance LIMIT 10;
+
+-- Possible error:
+-- Generated column 4292967295(generated_field_4292967295) is not filled for VECTOR
+```
+
+**Currently Available Query Patterns**:
+
+```sql
+-- Recommended: Vector filtering in WHERE clause
+SELECT id, content FROM embeddings_table 
+WHERE COSINE_DISTANCE(embedding, vector(...)) < 0.8
+ORDER BY id LIMIT 10;
+
+-- Available: Simple vector functions
+SELECT COSINE_DISTANCE(embedding, embedding) as self_distance
+FROM embeddings_table;
+
+-- Avoid: Complex SELECT returning distance calculation with sorting
+```
+
+### Function Usage Pitfalls
+
+#### L2\_NORMALIZE Function Type Requirements
+
+```sql
+-- Correct: Using float vectors
+SELECT L2_NORMALIZE(vector(3.0, 4.0)) as normalized;
+-- Result: [0.6, 0.8]
+
+-- Incorrect: Using integer vectors
+SELECT L2_NORMALIZE(vector(3, 4)) as normalized;
+-- Possible result: [0, 0] (precision loss)
+```
+
+#### Index Parameter Specifications
+
+```sql
+-- Correct: Using full parameter names
+CREATE VECTOR INDEX idx ON TABLE t(col) PROPERTIES(
+    "distance.function" = "cosine_distance",  -- Full function name
+    "scalar.type" = "f32"                    -- Standard type name
+);
+
+-- Incorrect: Using abbreviated names
+CREATE VECTOR INDEX idx ON TABLE t(col) PROPERTIES(
+    "distance.function" = "cosine",          -- Will cause error
+    "scalar.type" = "float"                  -- Will cause error
+);
+```
+
+### Best Practice Recommendations
+
+#### Query Pattern Selection
+
+| Usage Scenario             | Recommended Pattern                                | Pattern to Avoid                              |
+| -------------------------- | -------------------------------------------------- | --------------------------------------------- |
+| Similarity Filtering       | `WHERE COSINE_DISTANCE(...) < threshold`            | `ORDER BY COSINE_DISTANCE(...)`               |
+| Top-K Retrieval            | WHERE filtering + simple sorting                    | Distance calculation + sorting                 |
+| Batch Computation          | Step-by-step queries                                | Complex nested queries                         |
+
+#### Performance Optimization Strategies
+
+```sql
+-- Recommended: Filter query utilizing index
+SELECT doc_id, title FROM knowledge_base 
+WHERE COSINE_DISTANCE(embedding, ?) < 0.7
+ORDER BY doc_id LIMIT 10;
+
+-- Optional: Step-by-step Top-K retrieval
+-- Step 1: Filter candidates
+CREATE VIEW candidates AS 
+SELECT doc_id, title FROM knowledge_base 
+WHERE COSINE_DISTANCE(embedding, ?) < 0.8;
+
+-- Step 2: Precise sorting (if needed)
+SELECT doc_id, title FROM candidates ORDER BY doc_id LIMIT 10;
+```
+
+***
+
+## Migration Strategy
+
+### Migration Value Assessment
+
+Before deciding to migrate to the VECTOR type, it is recommended to evaluate your usage scenarios:
+
+#### High-Value Migration Scenarios
+
+* **Frequent vector similarity calculations**: Such as recommendation systems, document retrieval, image matching
+* **Large-scale vector data**: Vector data queries exceeding millions of records
+* **Complex UDF maintenance**: Currently maintaining multiple custom vector computation functions
+* **High performance requirements**: Applications with strict query response time requirements
+
+#### Optional Migration Scenarios
+
+* **Occasional vector calculations**: Vector computation is not a core business process
+* **Small-scale data**: Vector data volume is small, performance pressure is not significant
+* **Stable existing systems**: Current system runs stably, change risk outweighs benefit
+
+### Migration Recommendations for Different Users
+
+#### Spark User Migration Path
+
+**If you primarily use Spark MLlib for vector computation**:
+
+```python
+# Original Spark code
+from pyspark.ml.linalg import Vectors, VectorUDT
+from pyspark.sql.functions import udf
+
+# Custom similarity function
+def cosine_similarity(v1, v2):
+    # Complex implementation logic
+    pass
+
+# Register UDF
+cosine_udf = udf(cosine_similarity, DoubleType())
+```
+
+**After Migrating to Singdata VECTOR**:
+
+```sql
+-- Directly use built-in functions and recommended query patterns
+SELECT user_id FROM user_features
+WHERE COSINE_DISTANCE(user_vector, target_vector) < 0.8;
+```
+
+**Migration Benefits**:
+
+* Remove custom UDF code
+* Leverage vector indexes to accelerate queries
+* Unified vector computation in SQL
+
+#### Hive User Migration Path
+
+**If you use ARRAY to store vectors in Hive**:
+
+```sql
+-- Original Hive table structure
+CREATE TABLE user_embeddings (
+    user_id BIGINT,
+    features ARRAY<DOUBLE>
+) PARTITIONED BY (dt STRING);
+
+-- Requires complex UDF
+SELECT user_id, my_cosine_udf(features, target_array) as score
+FROM user_embeddings 
+WHERE size(features) = 128;  -- Manual dimension check
+```
+
+**Migrating to VECTOR Type**:
+
+```sql
+-- New table structure
+CREATE TABLE user_embeddings_v2 (
+    user_id BIGINT,
+    features VECTOR(FLOAT, 128)  -- Type safe
+) PARTITIONED BY (dt STRING);
+
+-- Use built-in functions and recommended patterns
+SELECT user_id FROM user_embeddings_v2
+WHERE COSINE_DISTANCE(features, vector(...)) < 0.8;  -- No manual dimension check needed
+```
+
+### Evaluating Existing Systems
+
+#### Data Scale Assessment
+
+```sql
+-- Statistics on the scale and dimensions of existing vector data
+SELECT 
+    table_name,
+    column_name,
+    COUNT(*) as record_count,
+    AVG(size(array_column)) as avg_dimension,
+    MIN(size(array_column)) as min_dimension,
+    MAX(size(array_column)) as max_dimension,
+    COUNT(DISTINCT size(array_column)) as dimension_variants
+FROM (
+    SELECT 'user_features' as table_name, 'embedding' as column_name, embedding as array_column FROM user_features
+    UNION ALL
+    SELECT 'product_vectors' as table_name, 'features' as column_name, features as array_column FROM product_vectors
+    -- Add other tables that need evaluation
+) all_vectors
+GROUP BY table_name, column_name;
+```
+
+#### UDF Complexity Assessment
+
+```sql
+-- Identify currently used vector-related UDFs
+SHOW FUNCTIONS LIKE '*similarity*';
+SHOW FUNCTIONS LIKE '*distance*';
+SHOW FUNCTIONS LIKE '*cosine*';
+SHOW FUNCTIONS LIKE '*euclidean*';
+```
+
+#### Query Frequency Assessment
+
+```sql
+-- Analyze the frequency of vector computation queries (if query logs are available)
+SELECT 
+    DATE(query_time) as query_date,
+    COUNT(*) as vector_query_count
+FROM query_logs 
+WHERE query_text LIKE '%cosine%' 
+    OR query_text LIKE '%similarity%'
+    OR query_text LIKE '%distance%'
+GROUP BY DATE(query_time)
+ORDER BY query_date DESC
+LIMIT 30;
+```
+
+### Parallel Construction of New Tables
+
+```sql
+-- Create new vector table
+CREATE TABLE embeddings_new (
+    id BIGINT,
+    content STRING,
+    embedding VECTOR(FLOAT, 768),
+    migrated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP(),
+    
+    INDEX embedding_idx (embedding) USING VECTOR PROPERTIES (
+        "distance.function" = "cosine_distance"
+    )
+);
+
+-- Data migration
+INSERT INTO embeddings_new (id, content, embedding)
+SELECT 
+    id,
+    content,
+    CASE 
+        WHEN size(embedding_array) = 768 
+        THEN cast(embedding_array as VECTOR(FLOAT, 768))
+        ELSE NULL
+    END as embedding
+FROM embeddings_old
+WHERE size(embedding_array) = 768;
+
+-- Build index
+BUILD INDEX embedding_idx ON embeddings_new;
+```
+
+### Verifying Migration Results
+
+```sql
+-- Compare query result consistency (using recommended query patterns)
+WITH old_results AS (
+    SELECT id FROM embeddings_old 
+    WHERE your_cosine_udf(embedding_array, ?) > 0.8
+    ORDER BY id LIMIT 10
+),
+new_results AS (
+    SELECT id FROM embeddings_new 
+    WHERE COSINE_DISTANCE(embedding, ?) < 0.2  -- Note the distance vs. similarity conversion
+    ORDER BY id LIMIT 10
+)
+SELECT 
+    COALESCE(o.id, n.id) as id,
+    CASE WHEN o.id IS NOT NULL THEN 'OLD' ELSE NULL END as in_old,
+    CASE WHEN n.id IS NOT NULL THEN 'NEW' ELSE NULL END as in_new
+FROM old_results o
+FULL OUTER JOIN new_results n ON o.id = n.id
+ORDER BY id;
+```
+
+***
+
+## Performance Optimization
+
+### Query Optimization
+
+```sql
+-- Recommended: Use threshold filtering to improve query efficiency
+SELECT id, content FROM embeddings_table 
+WHERE COSINE_DISTANCE(embedding, ?) < 0.8  -- Leverage vector index filtering
+ORDER BY id LIMIT 10;
+
+-- Optional: Step-by-step processing for complex queries
+CREATE VIEW filtered_candidates AS
+SELECT id, content FROM embeddings_table 
+WHERE COSINE_DISTANCE(embedding, ?) < 0.8;
+
+SELECT id, content FROM filtered_candidates 
+ORDER BY id LIMIT 10;
+```
+
+### Storage Optimization
+
+```sql
+-- Use half-precision storage for large-scale data
+CREATE TABLE large_embeddings (
+    id BIGINT,
+    embedding VECTOR(FLOAT, 1536),
+    
+    INDEX embedding_idx (embedding) USING VECTOR PROPERTIES (
+        "scalar.type" = "f16",  -- Half-precision storage saves memory
+        "distance.function" = "cosine_distance"
+    )
+);
+```
+
+### Index Tuning
+
+Choose appropriate index parameters based on application requirements:
+
+* **Real-time query scenarios**: Smaller M value and ef\_construction, use f16 type
+* **Batch analysis scenarios**: Larger M value and ef\_construction, use f32 type
+* **Storage optimization scenarios**: Use f16 or i8 type to reduce memory footprint
+
+***
+
+## Important Notes
+
+### Functional Limitations
+
+* Vector dimension is determined at creation time and does not support dynamic adjustment
+* Vectors cannot be used as partition keys
+* JACCARD\_DISTANCE and HAMMING\_DISTANCE functions have special type requirements
+* **The current version has limitations on complex SELECT queries; it is recommended to use WHERE filtering patterns**
+
+### Index Management
+
+* CREATE INDEX only applies to newly added data; existing data requires BUILD INDEX
+* The performance of vector indexes is closely related to memory caching
+* It is recommended to use a dedicated Virtual Cluster for vector queries
+
+### Troubleshooting
+
+| Issue                        | Possible Cause                     | Solution                                       |
+| ---------------------------- | ---------------------------------- | ---------------------------------------------- |
+| Poor query performance       | Index not in effect                | Check index status, execute BUILD INDEX        |
+| Dimension mismatch error     | Inconsistent vector dimensions     | Check VECTOR type definition and input data dimensions |
+| Generated column error       | Complex SELECT query limitation    | Use WHERE filtering pattern instead            |
+| L2\_NORMALIZE returns zero vector | Using integer vectors           | Switch to float vectors: vector(3.0, 4.0)      |
+| Index parameter error        | Non-standard parameter names       | Use full parameter names: cosine\_distance     |
+| Out of memory                | Vector index consumes too much memory | Adjust index parameters or increase cluster resources |
+
+***
+
+## Summary
+
+### Core Value of VECTOR Data Type
+
+**Type Safety**: Avoid runtime dimension mismatch errors through fixed dimension definition
+
+**Development Simplification**: Provide built-in vector distance calculation functions, reducing custom code
+
+**Performance Optimization**: Support dedicated vector indexes, improving large-scale vector query efficiency
+
+**Standard Compatibility**: Compatible with SQL syntax, supporting sorting, grouping, and other standard operations
+
+### Applicable Scenarios
+
+**Recommended use of VECTOR**:
+
+* Frequent vector similarity calculations needed
+* Fixed and clear vector dimensions
+* Desire to leverage vector indexes to improve query performance
+* Newly developed AI application projects
+
+**Continue using ARRAY**:
+
+* Primarily performing array operations rather than vector computation
+* Frequently changing vector dimensions
+* High cost of retrofitting existing systems
+* Need for complex vector sorting queries (current limitation)
+
+### Implementation Recommendations
+
+**Evaluation Phase**: Analyze the scale and dimensional characteristics of existing vector data
+
+**Pilot Validation**: Select small-scale scenarios to validate VECTOR effectiveness
+
+**Gradual Migration**: Use parallel construction approach for smooth migration of existing systems
+
+**Continuous Optimization**: Adjust index parameters and query strategies based on actual usage
+
+### Query Pattern Recommendations
+
+**Prioritize**:
+
+* Vector filtering in WHERE clause: `WHERE COSINE_DISTANCE(...) < threshold`
+* Simple vector function calculations: `COSINE_DISTANCE(v1, v2)`
+
+**Use with Caution**:
+
+* Complex SELECT returning distance calculation with sorting
+* Nested queries on large vectors
+
+**Alternative Approaches**:
+
+* Step-by-step queries: Filter first, then sort
+* Pre-computation: Batch-compute distances and store
+
+***
+
+**Note**: This document is based on the Lakehouse product documentation as of June 2025. It is recommended to check the official documentation regularly for the latest updates. Before using in a production environment, be sure to verify the correctness and performance impact of all operations in a test environment.