vecspan 0.1.0__tar.gz

vecspan-0.1.0/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.4
+Name: vecspan
+Version: 0.1.0
+Summary: One Python API for Milvus, Qdrant, and other vector databases.
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: milvus
+Requires-Dist: pymilvus>=2.5.0; extra == "milvus"
+Provides-Extra: qdrant
+Requires-Dist: qdrant-client>=1.12.0; extra == "qdrant"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+
+# vecspan
+
+vecspan 是一个同步 Python 客户端。业务代码只依赖一套集合、写入、过滤和检索接口，Milvus 和 Qdrant 放在适配器后面。向量由调用方传入，这个库不负责生成 embedding。
+
+## 安装
+
+```bash
+pip install vecspan
+pip install "vecspan[milvus]"
+pip install "vecspan[qdrant]"
+```
+
+只装其中一个可选依赖时，另一个后端不会被导入。
+
+## 连接
+
+```python
+from vecspan import VectorClient
+
+client = VectorClient.connect("memory://")
+client = VectorClient.connect("milvus://localhost:19530")
+client = VectorClient.connect("milvus://user:password@localhost:19530/default")
+client = VectorClient.connect("qdrant://localhost:6333", api_key="...")
+client = VectorClient.connect("qdrant+https://example.com", api_key="...")
+```
+
+`memory://` 把数据放在当前进程里，用来做单测和本地开发。第三方后端可以用 `register_backend(scheme)` 注册，工厂签名是 `factory(parsed_url, **kwargs)`。
+
+## 集合和写入
+
+主键在创建时选定，之后不再改变。`int` 的可移植范围是 `0 .. 2^63-1`。`uuid` 只接受 UUID，因为 Qdrant 的 point id 不能是任意字符串。
+
+```python
+from vecspan import (
+    CollectionSpec,
+    Distance,
+    Filter,
+    PayloadField,
+    PayloadType,
+    Point,
+    VectorField,
+)
+
+client.create_collection(CollectionSpec(
+    name="docs",
+    id_type="uuid",
+    vectors=[VectorField("embedding", size=768, distance=Distance.COSINE)],
+    payload_fields=[
+        PayloadField("source", PayloadType.KEYWORD, indexed=True),
+        PayloadField("year", PayloadType.INT, indexed=True),
+    ],
+))
+
+client.upsert("docs", [Point(
+    id="550e8400-e29b-41d4-a716-446655440000",
+    vectors={"embedding": [0.1] * 768},
+    payload={"source": "wiki", "year": 2020},
+)])
+```
+
+一个 point 可以带多个命名向量。写入分成四个语义：
+
+| 方法 | 语义 |
+| --- | --- |
+| `upsert(name, points)` | 按 id 新增，或整条替换向量和 payload |
+| `update_payload(name, ids, payload)` | 合并 payload，未提到的字段保留 |
+| `update_vectors(name, ids, vectors)` | 只替换给出的向量字段 |
+| `delete(name, ids=...)` 或 `delete(name, filter=...)` | 按 id 或按过滤条件删除 |
+
+`update_payload` 在 Qdrant 上是合并写。Milvus 在客户端支持 `partial_update` 时走局部更新，否则先读出整行再写回。后一种做法在读和写之间有并发窗口。
+
+未在 `payload_fields` 里声明的字段仍然可以写入。声明过的字段会按类型检查；标了 `indexed=True` 的字段会建标量索引。
+
+## 读取
+
+```python
+client.get("docs", [point_id])
+client.count("docs", filter=Filter.eq("source", "wiki"))
+page = client.scroll("docs", limit=100)
+page = client.scroll("docs", limit=100, offset=page.next_offset)
+
+query = (Filter.eq("source", "wiki") & Filter.gte("year", 2020)) | Filter.is_null("lang")
+hits = client.search("docs", vector=query_vector, filter=query, limit=10)
+```
+
+`scroll` 的 `offset` 只对同一次连接里的同一个集合有意义。过滤条件支持 `eq`、`gt` / `gte` / `lt` / `lte`、`in_`、`is_null`，以及 `&`、`|`、`~`。
+
+集合有多个向量字段时，`search` 必须传 `vector_name`。只有一个向量字段时可以省略。
+
+`SearchHit.score` 是后端自己的分数，不会在库之间换算。Cosine 和 Dot 越大越近，L2 越小越近。memory 和 Qdrant 的 L2 是欧氏距离，Milvus 的 L2 是平方欧氏距离，数值不要跨库比较。
+
+HNSW 参数 `m` 和 `ef_construct` 写在 `VectorField` 上，默认 `16` 和 `200`。检索时可以用 `backend_options` 覆盖搜索宽度：
+
+```python
+client.search("docs", vector=query_vector, backend_options={"ef": 128})
+```
+
+`hnsw_ef` 和 `ef` 是同一个参数。Milvus 另外接受 `nprobe`。
+
+## 范围
+
+v1 包含：
+
+- 集合的创建、删除、是否存在、描述、列表
+- `upsert`、`update_payload`、`update_vectors`、`delete`
+- `get`、`count`、`scroll`、`search`
+- 过滤表达式，以及默认的 HNSW 索引
+- `MemoryBackend`，以及同一套契约测试
+
+v1 不包含稀疏向量、混合检索、分区、别名、多副本和权限。客户端是同步的。
+
+## 测试
+
+```bash
+pip install pytest
+pytest
+```
+
+连上真实数据库时再跑集成测试：
+
+```bash
+VECSPAN_MILVUS_URL=milvus://localhost:19530 pytest -m integration
+VECSPAN_QDRANT_URL=qdrant://localhost:6333 pytest -m integration
+```

vecspan-0.1.0/README.md

@@ -0,0 +1,124 @@
+# vecspan
+
+vecspan 是一个同步 Python 客户端。业务代码只依赖一套集合、写入、过滤和检索接口，Milvus 和 Qdrant 放在适配器后面。向量由调用方传入，这个库不负责生成 embedding。
+
+## 安装
+
+```bash
+pip install vecspan
+pip install "vecspan[milvus]"
+pip install "vecspan[qdrant]"
+```
+
+只装其中一个可选依赖时，另一个后端不会被导入。
+
+## 连接
+
+```python
+from vecspan import VectorClient
+
+client = VectorClient.connect("memory://")
+client = VectorClient.connect("milvus://localhost:19530")
+client = VectorClient.connect("milvus://user:password@localhost:19530/default")
+client = VectorClient.connect("qdrant://localhost:6333", api_key="...")
+client = VectorClient.connect("qdrant+https://example.com", api_key="...")
+```
+
+`memory://` 把数据放在当前进程里，用来做单测和本地开发。第三方后端可以用 `register_backend(scheme)` 注册，工厂签名是 `factory(parsed_url, **kwargs)`。
+
+## 集合和写入
+
+主键在创建时选定，之后不再改变。`int` 的可移植范围是 `0 .. 2^63-1`。`uuid` 只接受 UUID，因为 Qdrant 的 point id 不能是任意字符串。
+
+```python
+from vecspan import (
+    CollectionSpec,
+    Distance,
+    Filter,
+    PayloadField,
+    PayloadType,
+    Point,
+    VectorField,
+)
+
+client.create_collection(CollectionSpec(
+    name="docs",
+    id_type="uuid",
+    vectors=[VectorField("embedding", size=768, distance=Distance.COSINE)],
+    payload_fields=[
+        PayloadField("source", PayloadType.KEYWORD, indexed=True),
+        PayloadField("year", PayloadType.INT, indexed=True),
+    ],
+))
+
+client.upsert("docs", [Point(
+    id="550e8400-e29b-41d4-a716-446655440000",
+    vectors={"embedding": [0.1] * 768},
+    payload={"source": "wiki", "year": 2020},
+)])
+```
+
+一个 point 可以带多个命名向量。写入分成四个语义：
+
+| 方法 | 语义 |
+| --- | --- |
+| `upsert(name, points)` | 按 id 新增，或整条替换向量和 payload |
+| `update_payload(name, ids, payload)` | 合并 payload，未提到的字段保留 |
+| `update_vectors(name, ids, vectors)` | 只替换给出的向量字段 |
+| `delete(name, ids=...)` 或 `delete(name, filter=...)` | 按 id 或按过滤条件删除 |
+
+`update_payload` 在 Qdrant 上是合并写。Milvus 在客户端支持 `partial_update` 时走局部更新，否则先读出整行再写回。后一种做法在读和写之间有并发窗口。
+
+未在 `payload_fields` 里声明的字段仍然可以写入。声明过的字段会按类型检查；标了 `indexed=True` 的字段会建标量索引。
+
+## 读取
+
+```python
+client.get("docs", [point_id])
+client.count("docs", filter=Filter.eq("source", "wiki"))
+page = client.scroll("docs", limit=100)
+page = client.scroll("docs", limit=100, offset=page.next_offset)
+
+query = (Filter.eq("source", "wiki") & Filter.gte("year", 2020)) | Filter.is_null("lang")
+hits = client.search("docs", vector=query_vector, filter=query, limit=10)
+```
+
+`scroll` 的 `offset` 只对同一次连接里的同一个集合有意义。过滤条件支持 `eq`、`gt` / `gte` / `lt` / `lte`、`in_`、`is_null`，以及 `&`、`|`、`~`。
+
+集合有多个向量字段时，`search` 必须传 `vector_name`。只有一个向量字段时可以省略。
+
+`SearchHit.score` 是后端自己的分数，不会在库之间换算。Cosine 和 Dot 越大越近，L2 越小越近。memory 和 Qdrant 的 L2 是欧氏距离，Milvus 的 L2 是平方欧氏距离，数值不要跨库比较。
+
+HNSW 参数 `m` 和 `ef_construct` 写在 `VectorField` 上，默认 `16` 和 `200`。检索时可以用 `backend_options` 覆盖搜索宽度：
+
+```python
+client.search("docs", vector=query_vector, backend_options={"ef": 128})
+```
+
+`hnsw_ef` 和 `ef` 是同一个参数。Milvus 另外接受 `nprobe`。
+
+## 范围
+
+v1 包含：
+
+- 集合的创建、删除、是否存在、描述、列表
+- `upsert`、`update_payload`、`update_vectors`、`delete`
+- `get`、`count`、`scroll`、`search`
+- 过滤表达式，以及默认的 HNSW 索引
+- `MemoryBackend`，以及同一套契约测试
+
+v1 不包含稀疏向量、混合检索、分区、别名、多副本和权限。客户端是同步的。
+
+## 测试
+
+```bash
+pip install pytest
+pytest
+```
+
+连上真实数据库时再跑集成测试：
+
+```bash
+VECSPAN_MILVUS_URL=milvus://localhost:19530 pytest -m integration
+VECSPAN_QDRANT_URL=qdrant://localhost:6333 pytest -m integration
+```

vecspan-0.1.0/pyproject.toml

@@ -0,0 +1,28 @@
+[project]
+name = "vecspan"
+version = "0.1.0"
+description = "One Python API for Milvus, Qdrant, and other vector databases."
+readme = "README.md"
+requires-python = ">=3.10"
+
+[project.optional-dependencies]
+milvus = ["pymilvus>=2.5.0"]
+qdrant = ["qdrant-client>=1.12.0"]
+dev = ["pytest>=8"]
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+vecspan = ["py.typed"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]
+markers = [
+    "integration: tests that talk to a live Milvus or Qdrant",
+]

vecspan-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

vecspan-0.1.0/src/vecspan/__init__.py

@@ -0,0 +1,50 @@
+"""One Python API for Milvus, Qdrant, and other vector databases."""
+
+from vecspan.capabilities import Capability
+from vecspan.client import VectorClient
+from vecspan.errors import (
+    BackendError,
+    CollectionAlreadyExistsError,
+    CollectionNotFoundError,
+    UnsupportedOperation,
+    ValidationError,
+    VecspanError,
+)
+from vecspan.filters import Filter
+from vecspan.registry import register_backend
+from vecspan.types import (
+    CollectionSpec,
+    Distance,
+    IdType,
+    PayloadField,
+    PayloadType,
+    Point,
+    ScrollPage,
+    SearchHit,
+    VectorField,
+)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "BackendError",
+    "Capability",
+    "CollectionAlreadyExistsError",
+    "CollectionNotFoundError",
+    "CollectionSpec",
+    "Distance",
+    "Filter",
+    "IdType",
+    "PayloadField",
+    "PayloadType",
+    "Point",
+    "ScrollPage",
+    "SearchHit",
+    "UnsupportedOperation",
+    "ValidationError",
+    "VecspanError",
+    "VectorClient",
+    "VectorField",
+    "__version__",
+    "register_backend",
+]

vecspan-0.1.0/src/vecspan/backends/__init__.py

@@ -0,0 +1,6 @@
+"""Built-in backends. Milvus and Qdrant stay lazy so importing vecspan is cheap."""
+
+from vecspan.backends.base import VectorBackend
+from vecspan.backends.memory import MemoryBackend
+
+__all__ = ["MemoryBackend", "VectorBackend"]

vecspan-0.1.0/src/vecspan/backends/base.py

@@ -0,0 +1,107 @@
+"""Backend interface implemented by Milvus, Qdrant, and the in-memory store."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import Sequence
+
+from vecspan.capabilities import ALL_CAPABILITIES, Capability
+from vecspan.filters import Filter
+from vecspan.types import CollectionSpec, Point, ScrollPage, SearchHit
+
+
+class VectorBackend(ABC):
+    """Storage operations behind `VectorClient`.
+
+    `VectorClient` validates arguments and checks capabilities before it calls
+    these methods. A backend still raises `VecspanError` for conditions only
+    the database can see, such as a missing collection or a missing point id.
+    """
+
+    name = "backend"
+    capabilities: frozenset[Capability] = ALL_CAPABILITIES
+
+    @abstractmethod
+    def create_collection(self, spec: CollectionSpec) -> None:
+        """Create a collection from `spec`."""
+
+    @abstractmethod
+    def drop_collection(self, name: str) -> None:
+        """Drop a collection."""
+
+    @abstractmethod
+    def has_collection(self, name: str) -> bool:
+        """Return whether the collection exists."""
+
+    @abstractmethod
+    def describe_collection(self, name: str) -> CollectionSpec:
+        """Return the schema stored for a collection."""
+
+    @abstractmethod
+    def list_collections(self) -> list[str]:
+        """Return collection names visible to callers."""
+
+    @abstractmethod
+    def upsert(self, name: str, points: Sequence[Point]) -> None:
+        """Insert points or replace them completely when the id already exists."""
+
+    @abstractmethod
+    def update_payload(self, name: str, ids: Sequence[int | str], payload: dict) -> None:
+        """Merge `payload` into existing points. Missing ids are an error."""
+
+    @abstractmethod
+    def update_vectors(
+        self,
+        name: str,
+        ids: Sequence[int | str],
+        vectors: Sequence[dict[str, list[float]]],
+    ) -> None:
+        """Replace the given vector fields on existing points and leave the rest."""
+
+    @abstractmethod
+    def delete(
+        self,
+        name: str,
+        *,
+        ids: Sequence[int | str] | None = None,
+        filter: Filter | None = None,
+    ) -> None:
+        """Delete by id or by filter. Exactly one of the two is set."""
+
+    @abstractmethod
+    def get(self, name: str, ids: Sequence[int | str], *, with_vectors: bool = False) -> list[Point]:
+        """Return the points that exist, in the same order as `ids`."""
+
+    @abstractmethod
+    def count(self, name: str, *, filter: Filter | None = None) -> int:
+        """Count points, optionally restricted by a filter."""
+
+    @abstractmethod
+    def scroll(
+        self,
+        name: str,
+        *,
+        filter: Filter | None = None,
+        limit: int = 100,
+        offset: int | str | None = None,
+        with_vectors: bool = False,
+    ) -> ScrollPage:
+        """Read a page of points. `offset` comes from a previous page."""
+
+    @abstractmethod
+    def search(
+        self,
+        name: str,
+        vector: Sequence[float],
+        *,
+        vector_name: str,
+        filter: Filter | None = None,
+        limit: int = 10,
+        with_vectors: bool = False,
+        backend_options: dict | None = None,
+    ) -> list[SearchHit]:
+        """Return the nearest points for one named vector field."""
+
+    def close(self) -> None:
+        """Release the underlying connection. The default does nothing."""
+        return None