@ia-ccun/code-agent-cli 0.0.52-beta.5 → 0.0.52-beta.7

package/config/agent/skills/milvus-skill/SKILL.md

@@ -0,0 +1,259 @@
+---
+name: milvus
+description: Operate Milvus vector database with pymilvus Python SDK. Use when the user wants to connect to Milvus, create collections, insert vectors, perform similarity search, hybrid search, full-text search, manage indexes, partitions, databases, or RBAC via Python code.
+license: Apache-2.0
+compatibility: Requires Python 3.8+ and pymilvus (pip install pymilvus). Runs on macOS and Linux.
+metadata:
+  author: zilliztech
+  version: "1.0.0"
+allowed-tools: Bash Read Write
+---
+
+# Milvus Vector Database Skill
+
+Operate [Milvus](https://milvus.io/) vector databases directly through Python code using the `pymilvus` SDK. Covers the full lifecycle — connecting, schema design, collection management, vector CRUD, search, hybrid search, full-text search, indexing, partitions, databases, and RBAC.
+
+## When to Use
+
+Use this skill when the user wants to:
+- Connect to a Milvus instance (local, standalone, cluster, or Milvus Lite)
+- Create collections with custom schemas
+- Insert, upsert, search, query, get, or delete vectors
+- Perform hybrid search with reranking
+- Perform full-text search (BM25)
+- Manage indexes, partitions, databases
+- Set up users, roles, and access control (RBAC)
+- Build RAG pipelines, semantic search, or recommendation systems with Milvus
+- Iterate over large result sets with search/query iterators
+
+## Requirements
+
+- Python 3.8+
+- `pymilvus` (`pip install pymilvus`)
+- A running Milvus instance, or use Milvus Lite (embedded, file-based) for development
+
+## Capabilities Overview
+
+| Area | What You Can Do |
+|------|----------------|
+| **Connection** | Connect to Milvus Lite, Standalone, Cluster, or Zilliz Cloud |
+| **Collections** | Create (quick or custom schema), list, describe, drop, rename, truncate, load, release |
+| **Vectors** | Insert, upsert, search, hybrid search, query, get, delete |
+| **Full-Text Search** | BM25-based keyword search with sparse vectors |
+| **Iterators** | Paginated search and query over large datasets |
+| **Indexes** | Create (AUTOINDEX, HNSW, IVF_FLAT, etc.), list, describe, drop |
+| **Partitions** | Create, list, load, release, drop |
+| **Databases** | Create, list, switch, drop |
+| **RBAC** | Users, roles, privileges management |
+
+---
+
+## Connection
+
+> **IMPORTANT: Before writing any connection code, you MUST ask the user for their connection details.** Ask:
+> 1. **Deployment type** — Milvus Lite (local file), Standalone/Cluster (self-hosted), or Zilliz Cloud (managed)?
+> 2. **URI** — For self-hosted: host and port (e.g., `http://localhost:19530`). For Zilliz Cloud: the endpoint URL.
+> 3. **Authentication** — Token, API key, or username/password if required.
+> 4. **Database name** — If not using the default database.
+>
+> **Never assume or hardcode connection parameters.** Use Milvus Lite (`uri="./milvus.db"`) only if the user explicitly wants local/embedded mode for development.
+
+```python
+from pymilvus import MilvusClient
+
+# Milvus Lite (embedded, file-based — great for dev/test)
+client = MilvusClient(uri="./milvus.db")
+
+# Standalone / Cluster Milvus (ask user for actual host:port and credentials)
+client = MilvusClient(uri="<USER_URI>", token="<USER_TOKEN>")
+
+# Zilliz Cloud (ask user for endpoint and API key)
+client = MilvusClient(uri="<USER_ZILLIZ_ENDPOINT>", token="<USER_API_KEY>")
+```
+
+**Parameters:**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `uri` | str | `"./file.db"` for Milvus Lite, `"http://host:19530"` for server |
+| `token` | str | API key or `"username:password"` |
+| `user` | str | Username (alternative to token) |
+| `password` | str | Password (alternative to token) |
+| `db_name` | str | Target database (default: `"default"`) |
+| `timeout` | float | Operation timeout in seconds |
+
+### Async Client
+
+```python
+from pymilvus import AsyncMilvusClient
+
+async with AsyncMilvusClient(uri="<USER_URI>") as client:
+    results = await client.search(collection_name="my_collection", data=[query_vector], limit=10)
+```
+
+---
+
+## Collection Management
+
+### Quick Create (auto schema + auto index + auto load)
+
+```python
+client.create_collection(
+    collection_name="my_collection",
+    dimension=768,
+    metric_type="COSINE"  # Optional: "COSINE" (default), "L2", "IP"
+)
+```
+
+This automatically creates an `id` field (INT64, primary key, auto_id), a `vector` field (FLOAT_VECTOR), AUTOINDEX, and auto-loads the collection.
+
+### Custom Schema Create
+
+```python
+from pymilvus import DataType
+
+schema = client.create_schema(auto_id=False, enable_dynamic_field=True)
+schema.add_field("id", DataType.INT64, is_primary=True)
+schema.add_field("text", DataType.VARCHAR, max_length=512)
+schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=768)
+
+index_params = client.prepare_index_params()
+index_params.add_index(field_name="embedding", index_type="AUTOINDEX", metric_type="COSINE")
+
+client.create_collection(collection_name="my_collection", schema=schema, index_params=index_params)
+```
+
+**See [references/collection.md](references/collection.md) for data types, add_field parameters, and all collection operations.**
+
+### Other Collection Operations
+
+```python
+client.list_collections()
+client.describe_collection(collection_name="my_collection")
+client.has_collection(collection_name="my_collection")
+client.rename_collection(old_name="old", new_name="new")
+client.drop_collection(collection_name="my_collection")
+client.truncate_collection(collection_name="my_collection")
+client.load_collection(collection_name="my_collection")
+client.release_collection(collection_name="my_collection")
+client.get_load_state(collection_name="my_collection")
+client.get_collection_stats(collection_name="my_collection")
+```
+
+- Quick create is best for prototyping; use custom schema for production.
+- A collection must be **loaded** before search or query.
+- Use `enable_dynamic_field=True` to allow inserting fields not defined in the schema.
+
+---
+
+## Vector Operations
+
+**See [references/vector.md](references/vector.md) for hybrid search, full-text search, iterators, filter syntax, and detailed examples.**
+
+### Insert / Upsert
+
+```python
+# Vectors must come from an embedding model — never use fake/placeholder vectors
+from pymilvus import model
+
+embedding_fn = model.dense.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")
+
+docs = ["AI advances in 2024", "ML basics for beginners"]
+vectors = embedding_fn.encode_documents(docs)
+
+data = [
+    {"id": 1, "text": docs[0], "embedding": vectors[0]},
+    {"id": 2, "text": docs[1], "embedding": vectors[1]},
+]
+client.insert(collection_name="my_collection", data=data)
+client.upsert(collection_name="my_collection", data=data)
+```
+
+### Search (vector similarity)
+
+```python
+# Use the same embedding model to encode the query
+query_vectors = embedding_fn.encode_queries(["What is artificial intelligence?"])
+
+results = client.search(
+    collection_name="my_collection",
+    data=query_vectors,
+    anns_field="embedding",
+    limit=10,
+    output_fields=["text", "id"],
+    filter='age > 20 and status == "active"',
+    search_params={"metric_type": "COSINE", "params": {"nprobe": 10}}
+)
+```
+
+### Query / Get / Delete
+
+```python
+# Query by filter
+client.query(collection_name="my_collection", filter='id in [1, 2, 3]', output_fields=["text"], limit=100)
+
+# Get by primary key
+client.get(collection_name="my_collection", ids=[1, 2, 3], output_fields=["text"])
+
+# Delete
+client.delete(collection_name="my_collection", ids=[1, 2, 3])
+client.delete(collection_name="my_collection", filter='status == "obsolete"')
+```
+
+- **Never use fake or placeholder vectors** (e.g., `[0.1, 0.2, ...]`). Always generate vectors from an embedding model.
+- Use `pip install "pymilvus[model]"` for built-in embedding functions, or use any embedding model (OpenAI, Cohere, etc.).
+- Vector dimension in search must match the collection schema exactly.
+- The query embedding model must be the **same model** used to generate the stored vectors.
+- For large inserts, batch data into chunks (e.g., 1000 rows per batch).
+- For large result sets, use iterators — see [references/vector.md](references/vector.md).
+
+---
+
+## Index Management
+
+**See [references/index.md](references/index.md) for index types, metric types, and parameters.**
+
+```python
+index_params = client.prepare_index_params()
+index_params.add_index(
+    field_name="embedding",
+    index_type="HNSW",
+    metric_type="COSINE",
+    params={"M": 16, "efConstruction": 256}
+)
+client.create_index(collection_name="my_collection", index_params=index_params)
+
+client.list_indexes(collection_name="my_collection")
+client.describe_index(collection_name="my_collection", index_name="my_index")
+client.drop_index(collection_name="my_collection", index_name="my_index")
+```
+
+- `AUTOINDEX` is recommended for most use cases.
+- An index is required before loading a collection.
+
+---
+
+## Additional Features
+
+| Feature | Reference |
+|---------|-----------|
+| Partition Management | [references/partition.md](references/partition.md) |
+| Database Management | [references/database.md](references/database.md) |
+| User & Role Management (RBAC) | [references/user-role.md](references/user-role.md) |
+| Common Patterns (RAG, Semantic Search) | [references/patterns.md](references/patterns.md) |
+
+---
+
+## General Guidance
+
+- **Always ask the user for connection details** (URI, token/credentials) before writing connection code. Never assume or hardcode connection parameters.
+- **Never generate fake or placeholder vectors.** Always use an embedding model to produce real vectors. Suggest `pip install "pymilvus[model]"` for built-in embedding functions.
+- For quick prototyping, use **Milvus Lite** (`uri="./file.db"`) — no server needed, but only if the user explicitly requests local/embedded mode.
+- A collection must be **loaded into memory** before search/query.
+- The vector dimension in search data must **exactly match** the collection schema.
+- The query embedding model must be the **same model** used to generate the stored vectors.
+- Before any destructive operation (drop collection, drop database, delete vectors), always confirm with the user.
+- Use `enable_dynamic_field=True` when the schema may evolve.
+- Prefer `AUTOINDEX` unless the user has specific performance requirements.
+- Use `truncate_collection` to clear all data without dropping the collection.
+- For large datasets, use iterators (`search_iterator`, `query_iterator`) instead of increasing limit.

package/config/agent/skills/milvus-skill/references/collection.md

@@ -0,0 +1,86 @@
+# Collection Management — Detailed Reference
+
+## Supported Data Types
+
+### Scalar Types
+
+| DataType | Notes |
+|----------|-------|
+| `DataType.BOOL` | Boolean |
+| `DataType.INT8` / `INT16` / `INT32` / `INT64` | Integers |
+| `DataType.FLOAT` / `DOUBLE` | Floating point |
+| `DataType.VARCHAR` | String (requires `max_length`) |
+| `DataType.JSON` | JSON object |
+| `DataType.ARRAY` | Array (requires `element_type`, `max_capacity`) |
+
+### Vector Types
+
+| DataType | Notes |
+|----------|-------|
+| `DataType.FLOAT_VECTOR` | Float32 vector (requires `dim`) |
+| `DataType.FLOAT16_VECTOR` | Float16 vector (requires `dim`) |
+| `DataType.BFLOAT16_VECTOR` | BFloat16 vector (requires `dim`) |
+| `DataType.BINARY_VECTOR` | Binary vector (requires `dim`) |
+| `DataType.SPARSE_FLOAT_VECTOR` | Sparse vector (no `dim` needed) |
+| `DataType.INT8_VECTOR` | Int8 vector (requires `dim`) |
+
+## add_field Parameters
+
+```python
+schema.add_field(
+    field_name="my_field",
+    datatype=DataType.VARCHAR,
+    is_primary=False,
+    auto_id=False,
+    max_length=256,          # Required for VARCHAR
+    dim=768,                 # Required for vector types (except sparse)
+    element_type=DataType.INT64,  # Required for ARRAY
+    max_capacity=100,        # Required for ARRAY
+    nullable=False,
+    default_value=None,
+    is_partition_key=False,
+    description=""
+)
+```
+
+## All Collection Operations
+
+```python
+# List all collections
+collections = client.list_collections()
+
+# Describe a collection
+info = client.describe_collection(collection_name="my_collection")
+
+# Check if collection exists
+exists = client.has_collection(collection_name="my_collection")
+
+# Rename a collection
+client.rename_collection(old_name="old_name", new_name="new_name")
+
+# Drop a collection
+client.drop_collection(collection_name="my_collection")
+
+# Truncate a collection (delete all data, keep schema and index)
+client.truncate_collection(collection_name="my_collection")
+
+# Load collection into memory (required before search/query)
+client.load_collection(collection_name="my_collection")
+
+# Release collection from memory
+client.release_collection(collection_name="my_collection")
+
+# Get load state
+state = client.get_load_state(collection_name="my_collection")
+
+# Get collection statistics
+stats = client.get_collection_stats(collection_name="my_collection")
+```
+
+## Guidance
+
+- Quick create is best for prototyping; use custom schema for production.
+- A collection must be **loaded** before search or query operations.
+- Before dropping a collection, confirm with the user — this deletes all data.
+- Use `enable_dynamic_field=True` to allow inserting fields not defined in the schema.
+- Use `truncate_collection` to clear all data while preserving the collection structure.

package/config/agent/skills/milvus-skill/references/database.md

@@ -0,0 +1,24 @@
+# Database Management
+
+```python
+# Create a database
+client.create_database(db_name="my_database")
+
+# List all databases
+databases = client.list_databases()
+# Returns: ["default", "my_database"]
+
+# Switch to a database
+client.using_database(db_name="my_database")
+
+# Drop a database (must drop all collections first)
+client.drop_database(db_name="my_database")
+
+# Or connect to a specific database at init (use the user's actual URI and credentials)
+client = MilvusClient(uri="<USER_URI>", token="<USER_TOKEN>", db_name="my_database")
+```
+
+## Guidance
+
+- Every Milvus instance has a `"default"` database.
+- Before dropping a database, all collections in it must be dropped first.

package/config/agent/skills/milvus-skill/references/index.md

@@ -0,0 +1,69 @@
+# Index Management — Detailed Reference
+
+## Create Index
+
+```python
+index_params = client.prepare_index_params()
+
+# Vector index
+index_params.add_index(
+    field_name="embedding",
+    index_type="HNSW",               # See index types table below
+    metric_type="COSINE",            # "COSINE", "L2", "IP"
+    params={"M": 16, "efConstruction": 256}
+)
+
+# Optional: scalar index
+index_params.add_index(
+    field_name="text",
+    index_type=""                    # Auto-select for scalars
+)
+
+client.create_index(
+    collection_name="my_collection",
+    index_params=index_params
+)
+```
+
+## Common Index Types
+
+| Index Type | For | Key Params | Notes |
+|------------|-----|------------|-------|
+| `AUTOINDEX` | Dense vectors | Auto-tuned | Recommended for most cases |
+| `FLAT` | Dense vectors | None | Brute force, 100% recall |
+| `IVF_FLAT` | Dense vectors | `nlist` | Good balance |
+| `IVF_SQ8` | Dense vectors | `nlist` | Compressed, less memory |
+| `HNSW` | Dense vectors | `M`, `efConstruction` | High recall, more memory |
+| `DISKANN` | Dense vectors | None | Disk-based, large datasets |
+| `SPARSE_INVERTED_INDEX` | Sparse vectors | `drop_ratio_build` | For sparse vectors |
+| `SPARSE_WAND` | Sparse vectors | `drop_ratio_build` | Faster sparse search |
+
+## Metric Types
+
+| Metric | Description | Use With |
+|--------|-------------|----------|
+| `"COSINE"` | Cosine similarity (larger = more similar) | Dense vectors |
+| `"L2"` | Euclidean distance (smaller = more similar) | Dense vectors |
+| `"IP"` | Inner product (larger = more similar) | Dense & Sparse vectors |
+| `"BM25"` | BM25 relevance scoring | Full-text search (sparse vectors from built-in tokenizer) |
+
+## Other Index Operations
+
+```python
+# List indexes
+indexes = client.list_indexes(collection_name="my_collection")
+
+# Describe an index
+info = client.describe_index(collection_name="my_collection", index_name="my_index")
+
+# Drop an index
+client.drop_index(collection_name="my_collection", index_name="my_index")
+```
+
+## Guidance
+
+- `AUTOINDEX` is recommended for most use cases.
+- An index is required before loading a collection.
+- After creating an index, load the collection before searching.
+- Sparse vectors only support `"IP"` metric type.
+- For full-text search, use `"BM25"` metric type with `SPARSE_INVERTED_INDEX` or `SPARSE_WAND`.

package/config/agent/skills/milvus-skill/references/partition.md

@@ -0,0 +1,29 @@
+# Partition Management
+
+```python
+# Create a partition
+client.create_partition(collection_name="my_collection", partition_name="partition_A")
+
+# List partitions
+partitions = client.list_partitions(collection_name="my_collection")
+# Returns: ["_default", "partition_A"]
+
+# Check if partition exists
+exists = client.has_partition(collection_name="my_collection", partition_name="partition_A")
+
+# Load specific partitions
+client.load_partitions(collection_name="my_collection", partition_names=["partition_A"])
+
+# Release specific partitions
+client.release_partitions(collection_name="my_collection", partition_names=["partition_A"])
+
+# Drop a partition
+client.drop_partition(collection_name="my_collection", partition_name="partition_A")
+```
+
+## Guidance
+
+- Every collection has a `_default` partition.
+- Use `is_partition_key=True` on a field to enable automatic partitioning by field value.
+- A partition must be loaded before search.
+- Before dropping a partition, confirm with the user — all data in it will be deleted.

package/config/agent/skills/milvus-skill/references/patterns.md

@@ -0,0 +1,150 @@
+# Common Patterns
+
+> **Note:** All patterns below use `<USER_URI>` and `<USER_TOKEN>` as connection placeholders. Always ask the user for their actual connection details before writing code. For local development, use Milvus Lite (`uri="./milvus.db"`) only if the user explicitly requests it.
+
+## RAG Pipeline Pattern
+
+```python
+from pymilvus import MilvusClient, DataType, model
+
+# 1. Connect (ask user for URI and credentials)
+client = MilvusClient(uri="<USER_URI>", token="<USER_TOKEN>")
+
+# 2. Set up embedding model
+embedding_fn = model.dense.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")
+
+# 3. Create collection (dim must match embedding model output)
+schema = client.create_schema(auto_id=True, enable_dynamic_field=True)
+schema.add_field("id", DataType.INT64, is_primary=True)
+schema.add_field("text", DataType.VARCHAR, max_length=2048)
+schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=384)  # all-MiniLM-L6-v2 outputs 384-dim
+schema.add_field("source", DataType.VARCHAR, max_length=256)
+
+index_params = client.prepare_index_params()
+index_params.add_index(field_name="embedding", index_type="AUTOINDEX", metric_type="COSINE")
+
+client.create_collection(collection_name="knowledge_base", schema=schema, index_params=index_params)
+
+# 4. Insert documents — generate real vectors from text chunks
+chunks = ["Milvus is a vector database...", "RAG combines retrieval and generation..."]
+vectors = embedding_fn.encode_documents(chunks)
+
+client.insert("knowledge_base", data=[
+    {"text": chunk, "embedding": vec, "source": "doc1.pdf"}
+    for chunk, vec in zip(chunks, vectors)
+])
+
+# 5. Retrieve relevant context — use the SAME embedding model
+query = "What is a vector database?"
+query_vectors = embedding_fn.encode_queries([query])
+
+results = client.search(
+    collection_name="knowledge_base",
+    data=query_vectors,
+    limit=5,
+    output_fields=["text", "source"],
+    search_params={"metric_type": "COSINE"}
+)
+```
+
+## Quick Semantic Search Pattern
+
+```python
+from pymilvus import MilvusClient, model
+
+# Simplest possible setup (Milvus Lite for local dev)
+client = MilvusClient(uri="./search.db")
+embedding_fn = model.dense.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")
+
+# Prepare data — vectors come from embedding model
+texts = ["first document", "second document", "third document"]
+vectors = embedding_fn.encode_documents(texts)
+
+client.create_collection(collection_name="docs", dimension=384)
+client.insert("docs", data=[
+    {"id": i, "vector": vec, "text": txt}
+    for i, (vec, txt) in enumerate(zip(vectors, texts))
+])
+
+# Search — encode query with the same model
+query_vectors = embedding_fn.encode_queries(["search query"])
+results = client.search("docs", data=query_vectors, limit=10, output_fields=["text"])
+```
+
+## Hybrid Search Pattern (Dense + Sparse)
+
+```python
+from pymilvus import MilvusClient, DataType, AnnSearchRequest, RRFRanker
+
+# Ask user for connection details
+client = MilvusClient(uri="<USER_URI>", token="<USER_TOKEN>")
+
+# Schema with both dense and sparse vectors
+schema = client.create_schema(auto_id=True)
+schema.add_field("id", DataType.INT64, is_primary=True)
+schema.add_field("text", DataType.VARCHAR, max_length=2048)
+schema.add_field("dense_embedding", DataType.FLOAT_VECTOR, dim=768)
+schema.add_field("sparse_embedding", DataType.SPARSE_FLOAT_VECTOR)
+
+index_params = client.prepare_index_params()
+index_params.add_index(field_name="dense_embedding", index_type="AUTOINDEX", metric_type="COSINE")
+index_params.add_index(field_name="sparse_embedding", index_type="SPARSE_INVERTED_INDEX", metric_type="IP")
+
+client.create_collection(collection_name="hybrid_col", schema=schema, index_params=index_params)
+
+# Search with both vectors and fuse results
+# dense_query_vector and sparse_query_vector come from your respective embedding models
+req1 = AnnSearchRequest(data=[dense_query_vector], anns_field="dense_embedding",
+                         param={"metric_type": "COSINE"}, limit=20)
+req2 = AnnSearchRequest(data=[sparse_query_vector], anns_field="sparse_embedding",
+                         param={"metric_type": "IP"}, limit=20)
+
+results = client.hybrid_search(
+    collection_name="hybrid_col",
+    reqs=[req1, req2],
+    ranker=RRFRanker(k=60),
+    limit=10,
+    output_fields=["text"]
+)
+```
+
+## Full-Text Search Pattern (BM25)
+
+```python
+from pymilvus import MilvusClient, DataType, Function, FunctionType
+
+# Ask user for connection details
+client = MilvusClient(uri="<USER_URI>", token="<USER_TOKEN>")
+
+schema = client.create_schema(auto_id=True)
+schema.add_field("id", DataType.INT64, is_primary=True)
+schema.add_field("title", DataType.VARCHAR, max_length=512)
+schema.add_field("body", DataType.VARCHAR, max_length=4096, enable_analyzer=True)
+schema.add_field("body_sparse", DataType.SPARSE_FLOAT_VECTOR)
+
+schema.add_function(Function(
+    name="body_bm25",
+    input_field_names=["body"],
+    output_field_names=["body_sparse"],
+    function_type=FunctionType.BM25,
+))
+
+index_params = client.prepare_index_params()
+index_params.add_index(field_name="body_sparse", index_type="AUTOINDEX", metric_type="BM25")
+
+client.create_collection(collection_name="articles", schema=schema, index_params=index_params)
+
+# Insert — only provide text, sparse vector is auto-generated by BM25 function
+client.insert("articles", data=[
+    {"title": "Intro to ML", "body": "Machine learning is a subset of artificial intelligence..."},
+])
+
+# Search with raw text — no embedding model needed, Milvus handles BM25 tokenization
+results = client.search(
+    collection_name="articles",
+    data=["machine learning fundamentals"],
+    anns_field="body_sparse",
+    limit=10,
+    output_fields=["title", "body"]
+)
+```