ragx-cli 0.2.0__tar.gz → 0.3.2__tar.gz

ragx_cli-0.3.2/.agents/skills/zvec/SKILL.md

@@ -0,0 +1,219 @@
+---
+name: zvec
+description: |
+  Zvec vector database development assistant. Use this skill when users need to develop vector search applications based on zvec,
+  build RAG systems, implement semantic search, or handle vector data storage and querying.
+  Suitable for Python and Node.js development environments, providing complete technical guidance from basic concepts to advanced usage.
+  Proactively use this skill when users mention vector databases, similarity search, embedding storage, HNSW/IVF indexes,
+  hybrid search, multi-vector queries, or zvec API usage.
+---
+
+## Usage Instructions
+
+### Before starting, understand the following:
+
+1. **Development Language**: Python or Node.js?
+   - Python: use `pip install zvec`
+   - Node.js: use `npm install @zvec/zvec`
+
+2. **Use Cases**:
+   - RAG document retrieval system
+   - Semantic search
+   - Multimodal search (image + text)
+   - Hybrid search (keywords + semantic)
+
+3. **Data Scale**:
+   - < 100k: use FLAT index (exact search)
+   - 100k-10M: use HNSW index (recommended default)
+   - > 10M: use IVF index (memory optimized)
+
+### Decision Workflow
+
+- User needs vector search functionality
+  - Choose development language (Python/Node.js)
+  - Determine use case
+    - RAG system → use single-vector search + document chunk management
+    - E-commerce search → use hybrid search (vector + filter)
+    - Multimodal → use multi-vector search + weighted ranking
+  - Design Schema (vector fields + scalar fields)
+  - Select index type (HNSW/FLAT/IVF)
+  - Implement data synchronization strategy
+
+### Default Recommendations
+
+- Use `create_and_open()` / `ZVecCreateAndOpen()` to create Collection
+- Use cosine similarity (COSINE) as default distance metric
+- Use FP32 type for dense vectors
+- Create `InvertIndexParam` index for filter fields
+
+### Validation Checklist
+
+- Vector dimensions match Schema definition
+- Scalar field types are correct
+- Filter condition syntax is correct
+- Call `optimize()` after large batch writes
+
+## Quick Start
+
+**Python:**
+
+```python
+import zvec
+
+# Create Collection
+schema = zvec.CollectionSchema(
+    name="my_collection",
+    fields=[
+        zvec.FieldSchema(name="title", data_type=zvec.DataType.STRING),
+    ],
+    vectors=[
+        zvec.VectorSchema(
+            name="embedding",
+            data_type=zvec.DataType.VECTOR_FP32,
+            dimension=768,
+            index_param=zvec.HnswIndexParam(
+                metric_type=zvec.MetricType.COSINE
+            ),
+        ),
+    ],
+)
+
+collection = zvec.create_and_open("./my_data", schema)
+
+# Insert document
+collection.upsert(zvec.Doc(
+    id="doc_1",
+    vectors={"embedding": [0.1] * 768},
+    fields={"title": "Hello World"},
+))
+
+# Search
+results = collection.query(
+    vectors=zvec.VectorQuery(
+        field_name="embedding",
+        vector=[0.1] * 768,
+    ),
+    topk=10,
+)
+```
+
+**Node.js:**
+
+```typescript
+import { ZVecCreateAndOpen, ZVecCollectionSchema, ZVecFieldSchema, ZVecVectorSchema, ZVecDataType, ZVecHnswIndexParams, ZVecMetricType } from "@zvec/zvec";
+
+const schema = new ZVecCollectionSchema({
+  name: "my_collection",
+  fields: [new ZVecFieldSchema({ name: "title", dataType: ZVecDataType.STRING })],
+  vectors: [new ZVecVectorSchema({
+    name: "embedding",
+    dataType: ZVecDataType.VECTOR_FP32,
+    dimension: 768,
+    indexParams: new ZVecHnswIndexParams({ metricType: ZVecMetricType.COSINE }),
+  })],
+});
+
+const collection = ZVecCreateAndOpen("./my_data", schema);
+```
+
+## Core Concepts
+
+### Data Model
+
+**Collection**
+- Similar to a table in relational databases, a container for storing, organizing, and querying data
+- Each Collection has a Schema defining its structure
+- Each Collection is independently persisted in a dedicated directory on disk
+
+**Document**
+- Basic unit of data storage, similar to a row in a relational table
+- Contains three core components:
+  - `id`: unique string identifier
+  - `vectors`: named vector collection (supports dense and sparse vectors)
+  - `fields`: named scalar field collection
+
+**Schema**
+- Dynamic Schema: scalar fields and vectors can be added or removed at any time
+- Strong type system: each field must declare a DataType
+
+### Vector Types
+
+**Dense Vector**
+- Fixed-length real-valued embeddings
+- Types: `VECTOR_FP16`, `VECTOR_FP32`, `VECTOR_INT8`
+- Suitable for: semantic understanding, context capture
+
+**Sparse Vector**
+- High-dimensional representation with only a few non-zero dimensions
+- Types: `SPARSE_VECTOR_FP32`, `SPARSE_VECTOR_FP16`
+- Suitable for: keyword matching, BM25 scoring
+
+### Index Types
+
+| Index Type | Characteristics | Use Case |
+|---------|------|---------|
+| **FLAT** | Brute force search, exact results | Small scale data (<100k) |
+| **HNSW** | Approximate nearest neighbor, graph structure | Large scale data (recommended default) |
+| **IVF** | Inverted file index | Very large scale data |
+
+## Available Topics
+
+### Python
+
+- [Quick Start](./quick-start/python.md) - Quick start with Zvec Python API
+- [Collection Management](./collection-management/python.md) - Create, open, and manage Collections
+- [Data Operations](./data-operations/python.md) - Insert, update, and delete documents
+- [Vector Search](./vector-search/python.md) - Single-vector, multi-vector, and hybrid search
+- [RAG System](./rag-system/python.md) - Build document retrieval system
+- [Hybrid Search](./hybrid-search/python.md) - Vector similarity + scalar filtering
+- [Multimodal Search](./multimodal-search/python.md) - Image + text joint search
+
+### Node.js
+
+- [Quick Start](./quick-start/typescript.md) - Quick start with Zvec Node.js API
+- [Collection Management](./collection-management/typescript.md) - Create, open, and manage Collections
+- [Data Operations](./data-operations/typescript.md) - Insert, update, and delete documents
+- [Vector Search](./vector-search/typescript.md) - Single-vector, multi-vector, and hybrid search
+- [RAG System](./rag-system/typescript.md) - Build document retrieval system
+- [Hybrid Search](./hybrid-search/typescript.md) - Vector similarity + scalar filtering
+- [Multimodal Search](./multimodal-search/typescript.md) - Image + text joint search
+
+### General
+
+- [Configuration](./configuration.md) - Global configuration and initialization
+- [Data Model](./data-model.md) - Zvec data model overview
+- [Embedding](./embedding.md) - Text embedding functions (Python only)
+- [Reranker](./reranker.md) - Result reranking functions (Python only)
+- [API Cheatsheet](./api-cheatsheet.md) - Python & Node.js API quick reference
+- [Troubleshooting](./troubleshooting.md) - Common issues and solutions
+
+## Available Topics
+
+### Python
+
+- [Collection Management](./collection-management/python.md)
+- [Data Operations](./data-operations/python.md)
+- [Hybrid Search](./hybrid-search/python.md)
+- [Multimodal Search](./multimodal-search/python.md)
+- [Quick Start](./quick-start/python.md)
+- [Rag System](./rag-system/python.md)
+- [Vector Search](./vector-search/python.md)
+
+### Node.js
+
+- [Collection Management](./collection-management/typescript.md)
+- [Data Operations](./data-operations/typescript.md)
+- [Hybrid Search](./hybrid-search/typescript.md)
+- [Multimodal Search](./multimodal-search/typescript.md)
+- [Quick Start](./quick-start/typescript.md)
+- [Rag System](./rag-system/typescript.md)
+- [Vector Search](./vector-search/typescript.md)
+
+### General
+
+- [Configuration](./configuration.md)
+- [Data Model](./data-model.md)
+- [Embedding](./embedding.md)
+- [Reranker](./reranker.md)
+- [Api Cheatsheet](./api-cheatsheet.md)
+- [Troubleshooting](./troubleshooting.md)

ragx_cli-0.3.2/.agents/skills/zvec/api-cheatsheet.md

@@ -0,0 +1,354 @@
+# Zvec API Cheatsheet
+
+## Python API
+
+### Installation
+```bash
+pip install zvec
+```
+
+### Global Configuration
+```python
+import zvec
+
+# Initialize with defaults
+zvec.init()
+
+# Initialize with custom settings
+zvec.init(
+    log_type=zvec.LogType.CONSOLE,      # or LogType.FILE
+    log_level=zvec.LogLevel.WARN,       # DEBUG, INFO, WARN, ERROR, FATAL
+    query_threads=4,
+    optimize_threads=2,
+    memory_limit_mb=2048,
+)
+```
+
+### Core Classes
+```python
+import zvec
+```
+
+### Data Types
+
+**Scalar Types:**
+- `zvec.DataType.STRING`
+- `zvec.DataType.BOOL`
+- `zvec.DataType.INT32`, `INT64`, `UINT32`, `UINT64`
+- `zvec.DataType.FLOAT`, `DOUBLE`
+- Arrays: `ARRAY_STRING`, `ARRAY_BOOL`, `ARRAY_INT32`, etc.
+
+**Vector Types:**
+- `zvec.DataType.VECTOR_FP16`, `VECTOR_FP32`, `VECTOR_INT8`
+- `zvec.DataType.SPARSE_VECTOR_FP32`, `SPARSE_VECTOR_FP16`
+
+**Distance Metrics:**
+- `zvec.MetricType.COSINE`
+- `zvec.MetricType.IP`
+- `zvec.MetricType.L2`
+
+### Index Parameters
+
+```python
+# HNSW
+zvec.HnswIndexParam(
+    metric_type=zvec.MetricType.COSINE,
+    M=16,
+    ef_construction=100,
+)
+
+# FLAT
+zvec.FlatIndexParam(metric_type=zvec.MetricType.COSINE)
+
+# IVF
+zvec.IVFIndexParam(
+    metric_type=zvec.MetricType.COSINE,
+    nlist=100,
+    nprobe=10,
+)
+
+# Scalar index
+zvec.InvertIndexParam(enable_range_optimization=True)
+```
+
+### Collection Operations
+
+```python
+# Create/Open
+collection = zvec.create_and_open(path, schema, option)
+collection = zvec.open(path, option)
+
+# Properties
+collection.path
+collection.schema
+collection.stats
+collection.option
+
+# Data Write
+collection.insert(doc)
+collection.insert([doc1, doc2])  # batch
+collection.upsert(doc)
+collection.update(ids, vectors, fields)
+
+# Data Query
+doc = collection.fetch(id)
+results = collection.query(vectors, topk, filter)
+
+# Data Delete
+collection.delete(ids)
+collection.delete_by_filter(filter)
+
+# Schema Modify
+collection.add_column(field, expression="default_value")
+collection.drop_column(name)
+collection.alter_column(old_name="old", new_name="new")
+collection.alter_column(field_schema=updated_field)
+collection.create_index(field_name, index_param)
+collection.drop_index(field_name)
+
+# Maintenance
+collection.optimize()
+collection.flush()
+collection.destroy()
+```
+
+### Document Structure
+
+```python
+doc = zvec.Doc(
+    id="unique_id",
+    vectors={
+        "dense_vec": [0.1, 0.2, ...],
+        "sparse_vec": {42: 1.25, 1337: 0.8},
+    },
+    fields={
+        "string_field": "value",
+        "int_field": 123,
+        "bool_field": True,
+    },
+)
+```
+
+### Query Structure
+
+```python
+# Vector query
+zvec.VectorQuery(
+    field_name="vector_name",
+    vector=[0.1, 0.2, ...],
+    param=zvec.HnswQueryParam(ef=100),
+)
+
+# Reranker
+zvec.WeightedReRanker(
+    topn=10,
+    metric=zvec.MetricType.IP,
+    weights={"vec1": 1.0, "vec2": 0.8},
+)
+
+zvec.RRFReRanker(topn=10, rank_constant=60)
+```
+
+---
+
+## Node.js API
+
+### Installation
+```bash
+npm install @zvec/zvec
+```
+
+### Global Configuration
+```typescript
+import { ZVecInitialize, ZVecLogType, ZVecLogLevel } from "@zvec/zvec";
+
+// Initialize with defaults
+ZVecInitialize();
+
+// Initialize with custom settings
+ZVecInitialize({
+  logType: ZVecLogType.CONSOLE,       // or ZVecLogType.FILE
+  logLevel: ZVecLogLevel.WARN,        // DEBUG, INFO, WARN, ERROR, FATAL
+  queryThreads: 4,
+  optimizeThreads: 2,
+  memoryLimitMb: 2048,
+});
+```
+
+### Core Imports
+
+```typescript
+import {
+  ZVecInitialize,
+  ZVecCollectionSchema,
+  ZVecFieldSchema,
+  ZVecVectorSchema,
+  ZVecDataType,
+  ZVecHnswIndexParams,
+  ZVecMetricType,
+  ZVecCreateAndOpen,
+  ZVecOpen,
+} from "@zvec/zvec";
+```
+
+### Data Type Enums
+
+**Scalar Types:**
+- `ZVecDataType.STRING`
+- `ZVecDataType.BOOL`
+- `ZVecDataType.INT32`, `INT64`, `UINT32`, `UINT64`
+- `ZVecDataType.FLOAT`, `DOUBLE`
+
+**Vector Types:**
+- `ZVecDataType.VECTOR_FP16`, `VECTOR_FP32`, `VECTOR_INT8`
+- `ZVecDataType.SPARSE_VECTOR_FP32`, `SPARSE_VECTOR_FP16`
+
+**Distance Metrics:**
+- `ZVecMetricType.COSINE`
+- `ZVecMetricType.IP`
+- `ZVecMetricType.L2`
+
+### Index Parameters
+
+```typescript
+// HNSW
+new ZVecHnswIndexParams({
+  metricType: ZVecMetricType.COSINE,
+  M: 16,
+  efConstruction: 100,
+});
+
+// FLAT
+new ZVecFlatIndexParams({ metricType: ZVecMetricType.COSINE });
+
+// IVF
+new ZVecIVFIndexParams({
+  metricType: ZVecMetricType.COSINE,
+  nlist: 100,
+  nprobe: 10,
+});
+
+// Scalar index
+new ZVecInvertIndexParams({ enableRangeOptimization: true });
+```
+
+### Collection Operations
+
+```typescript
+// Create/Open
+const collection = ZVecCreateAndOpen(path, schema, option);
+const collection = ZVecOpen(path, option);
+
+// Properties
+collection.path;
+collection.schema;
+collection.stats;
+collection.option;
+
+// Data Write
+collection.upsertSync(doc);
+collection.upsertSync([doc1, doc2]);  // batch
+collection.updateSync({ ids, vectors, fields });
+
+// Data Query
+const doc = collection.fetchSync(id);
+const results = collection.querySync({ vectors, topk, filter });
+
+// Data Delete
+collection.deleteSync({ ids });
+collection.deleteByFilterSync(filter);
+
+// Schema Modify
+collection.addColumnSync({ field, expression: "default_value" });
+collection.dropColumnSync(name);
+collection.alterColumnSync({ oldName: "old", newName: "new" });
+collection.alterColumnSync({ field: updatedField });
+collection.createIndexSync({ fieldName, indexParams });
+collection.dropIndexSync(fieldName);
+
+// Maintenance
+collection.optimizeSync();
+collection.flushSync();
+collection.destroySync();
+```
+
+### Document Structure
+
+```typescript
+interface ZVecDocInput {
+  id: string;
+  vectors: {
+    [name: string]: number[] | { [index: number]: number };
+  };
+  fields: {
+    [name: string]: any;
+  };
+}
+
+// Example
+const doc: ZVecDocInput = {
+  id: "unique_id",
+  vectors: {
+    dense_vec: [0.1, 0.2, 0.3],
+    sparse_vec: { 42: 1.25, 1337: 0.8 },
+  },
+  fields: {
+    string_field: "value",
+    int_field: 123,
+    bool_field: true,
+  },
+};
+```
+
+### Query Structure
+
+```typescript
+// Vector query
+interface ZVecQuery {
+  fieldName: string;
+  vector?: number[] | { [index: number]: number };
+  id?: string;
+  params?: ZVecHnswQueryParams | ZVecIVFQueryParams;
+}
+
+// Example
+const query: ZVecQuery = {
+  fieldName: "vector_name",
+  vector: [0.1, 0.2, ...],
+  params: new ZVecHnswQueryParams({ ef: 100 }),
+};
+```
+
+---
+
+## Filter Syntax (Common)
+
+```
+# Comparison
+field == value
+field != value
+field > value
+field >= value
+field < value
+field <= value
+
+# Range
+field BETWEEN min AND max
+
+# Logical operations
+condition1 AND condition2
+condition1 OR condition2
+NOT condition
+
+# Array contains
+'value' IN array_field
+
+# String matching
+field LIKE 'prefix%'
+field LIKE '%suffix'
+field LIKE '%contains%'
+
+# Null check
+field IS NULL
+field IS NOT NULL
+```