pyvastbase 0.2.2__tar.gz

pyvastbase-0.2.2/CLAUDE.md

@@ -0,0 +1,111 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Language
+
+回复时请使用简体中文（Simplified Chinese）。
+
+## Project Overview
+
+pyvastbase is a Python SDK for Vastbase V3 vector database — a PyMilvus-style API wrapping Vastbase (PostgreSQL fork with vector extensions). Single dependency: `psycopg[binary]`.
+
+## Commands
+
+```bash
+# Install dev dependencies
+pip install -e ".[dev]"
+
+# Run all tests (unit; integration skipped without VASTBASE_HOST)
+pytest
+
+# Run integration tests (requires real Vastbase instance)
+VASTBASE_HOST=192.168.253.128 VASTBASE_PORT=18000 VASTBASE_USER=zzj \
+  VASTBASE_PASSWORD=Aa123456 VASTBASE_DATABASE=milvus pytest -m integration
+
+# Run a single test file
+pytest tests/test_collection.py
+
+# Run a single test function
+pytest tests/test_collection.py::test_insert
+
+# Run with coverage
+pytest --cov=pyvastbase --cov-report=term-missing
+
+# Format
+black . --line-length 100
+isort . --profile black
+
+# Type check
+mypy .
+```
+
+## Architecture
+
+Two-layer design — every module has a synchronous implementation and a parallel `async_impl/` counterpart:
+
+| Layer | Files | DB Driver |
+|-------|-------|-----------|
+| Sync | `core/`, `collection.py`, `utility.py`, `index/`, `search/`, `orm/` | psycopg 3 |
+| Async | `async_impl/` (collection, connections, index_builder, search_builder, utility) | psycopg 3 async |
+
+### Core modules (`core/`)
+
+- [constants.py](core/constants.py) — Enums for IndexType, DistanceType, QuantizerType; version tuples; default index/query params; vector operator mapping
+- [connections.py](core/connections.py) — `ConnectionConfig` dataclass, `Connections` singleton (thread-safe alias-based connection registry), `VastbaseConnection` wrapper with optional connection pool and version caching
+- [schema.py](core/schema.py) — `DataType` class (scalar + vector type constants including FLOAT16_VECTOR, INT8_VECTOR, SPARSE_FLOAT_VECTOR), `FieldSchema` dataclass, `CollectionSchema` that generates CREATE TABLE SQL
+- [exceptions.py](core/exceptions.py) — 19 exception classes rooted at `VastbaseException` (18 concrete + 1 base), each carrying domain-specific attributes (e.g., `VectorDimensionError.expected/.actual`, `CollectionNotExistsError.collection_name`)
+- [mutation_result.py](core/mutation_result.py) — `MutationResult` with typed fields (insert_count, upsert_count, delete_count, primary_keys) and `MutationResultWrapper` providing backward-compatible list-like access
+
+### Key files
+
+- [collection.py](collection.py) (74KB) — Sync `Collection` class: full CRUD, insert/upsert/delete, vector/hybrid search, query/get, index lifecycle, pagination (`SearchIterator`, `QueryIterator`)
+- [utility.py](utility.py) (73KB) — 50+ module-level functions: connection management, server info, user/DB/privilege management, collection lifecycle, flush/compact, search highlight, buffer inspection
+- [client.py](client.py) — `VastbaseClient` (aliased as `MilvusClient`): pymilvus-compatible high-level entry point. 40+ implemented methods delegating to Collection/utility, plus ~30 NotSupportedError stubs for PG-incompatible features (partition, role, replica, etc.). New pymilvus compat methods: `hybrid_search`, `query_iterator`, `search_iterator`, `describe_index`, `describe_database`, `describe_user`, `update_password`, `list_aliases`, `describe_alias`, `alter_index_properties`, `drop_index_properties`, `alter_collection_properties`, `drop_collection_properties`, `add_collection_field`, `flush_all`, `get_flush_all_state`
+- [operations/](operations/) — Pure SQL generation layer used by both sync and async paths. Each function returns `(sql, params)` tuple. Modules: admin (user/privilege/database SQL), ddl (drop/truncate/refresh/alter_table/add_column), index (drop/list/get/describe/alter_index), insert, delete, query, result, schema_reflect, search_result
+- [orm/](orm/) — Django-style ORM: `Entity` base class, `Model` decorator, `Field`/`VectorField` descriptors, field validators. Generates `CollectionSchema` from decorated classes.
+
+### Supporting modules
+
+- [index/index_builder.py](index/index_builder.py) — `IndexParams` with 3 factory methods (graph_index, hybrid_index, fulltext_index), `IndexBuilder` with matching fluent methods + convenience `create_vector_index`/`create_fulltext_index`
+- [search/search_builder.py](search/search_builder.py) — `SearchParams`, `SearchResult`/`SearchResults`, `SearchQueryBuilder`, `FulltextSearchBuilder`
+- [utils/](utils/) — Internal SQL generation, vector serialization, distance calculation, filter expression building, result parsing
+
+### Version detection & feature gating
+
+Vastbase exposes version via `SELECT vb_version()`, which returns both a Build number and a patch level (e.g., `version:V3.0 (Build 9)` + `patch:1`). Feature gating for 3.0.9+ features checks **both** `build >= 9 AND patch_level >= 1`. Version is cached per connection (one query, reused). New vector types (int8vector, halfvector, sparsevector), BM25 highlight, and auto-quantization all gate on this dual check. See [core/connections.py:219](core/connections.py#L219) `require_version()`.
+
+### Test conventions
+
+- Unit tests: `tests/test_*.py`, run with plain `pytest`
+- Integration tests: marked `@pytest.mark.integration`, auto-skipped unless `VASTBASE_HOST` env var is set
+- Fixtures in `tests/conftest.py` provide `vastbase_conn` (async) and `vastbase_sync_conn` (sync) connection fixtures that auto-cleanup
+- `tests/conftest_integration.py` is alternative config using `VASTBASE_HOST/PORT/USER/PASSWORD/DATABASE` env vars
+- Async test mode: `asyncio_mode = "auto"` in pyproject.toml
+
+### Module-level config
+
+- `configure(**kwargs)` sets global defaults: `timeout`, `pool_size`, `max_overflow`, `default_connection`
+- `get_config()` returns a copy of current settings
+
+### Code style
+
+- Python >= 3.9, black (line-length=100), isort (black profile), mypy
+- Type annotations on all public function signatures
+- dataclass/NamedTuple for immutable data objects
+- Sparse vectors use dict format: `{"indices": [0, 5, 100], "values": [0.5, 0.8, 0.3]}`
+- Async module renamed to `async_impl` (avoids Python keyword collision)
+
+## Agent skills
+
+### Issue tracker
+
+Issues live as local markdown files under `.scratch/<feature-slug>/`. See `docs/agents/issue-tracker.md`.
+
+### Triage labels
+
+Default canonical labels: `needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`. See `docs/agents/triage-labels.md`.
+
+### Domain docs
+
+Single-context layout: `CONTEXT.md` + `docs/adr/` at repo root. See `docs/agents/domain.md`.

pyvastbase-0.2.2/CONTEXT.md

@@ -0,0 +1,79 @@
+# Domain Glossary
+
+## Collection
+
+A named set of data organized by a schema. Maps to a PostgreSQL table. Composed of scalar fields and vector fields. Created, queried, and mutated via the `Collection` module.
+
+## DataType
+
+An IntEnum holding the type of a field. Members carry both the Python-facing enum value and the PostgreSQL column type string via the `.pg_type` property. Example: `DataType.FLOAT_VECTOR` (value=101) maps to `floatvector`. The IntEnum design supports pymilvus-compatible `if dtype == DataType.INT64` comparisons while retaining PG type mapping for DDL generation. Aliases exist for backward compatibility (`DataType.INT64 == DataType.INT64`).
+
+## MetricType
+
+An alias for `DistanceType`. Used in pymilvus-style API: `MetricType.L2`, `MetricType.COSINE`, `MetricType.IP`. Delegates directly to the corresponding `DistanceType` enum values.
+
+## Vector Type
+
+A data type that holds multi-dimensional numerical data for similarity search. Four registered types: `floatvector` (4-byte float), `int8vector` (1-byte signed int), `halfvector` (2-byte fp16), `sparsevector` (sparse float). Each type carries metadata: dimension range, distance operators, index operator classes, SQL literal format, minimum Vastbase version. All type metadata is registered in a `VectorTypeRegistry` — the single source of truth queried by schema validation, SQL generation, and version gating.
+
+## Index
+
+A database-side structure that accelerates vector similarity search. Types include: `graph_index` (HNSW), `hybridann` (vector-scalar hybrid), `ivfflat`, `ivfpq`, `diskann`, `fulltext` (BM25). Each index type has its own dimension limits and configuration parameters. Note: `GRAPH_INDEX` is the SQL-level operator class name for HNSW indexes in Vastbase; `IndexType.HNSW` is the pymilvus-compatible alias. Both `GRAPH_INDEX` and `HNSW` refer to the same underlying HNSW implementation.
+
+## Distance Metric
+
+The mathematical measure used to compare vector similarity: L2 (Euclidean `<->`), Cosine (`<=>`), IP (Inner Product `<#>`). Stored as metadata on each vector type entry.
+
+## Executor
+
+An abstraction over the database driver that executes SQL and returns rows. The **sole seam** to the database — all SQL execution crosses this interface, not the underlying VastbaseConnection directly. Implementations: `SyncExecutor` (psycopg 3, returns `list[dict]`), `AsyncExecutor` (psycopg 3 async, returns `list[dict]`), `FakeExecutor` (sync in-memory test double), `FakeAsyncExecutor` (async in-memory test double). Utility functions access an Executor via `@with_executor` (sync) or `@async_with_executor` (async) decorators; Collection delegates to CollectionCore (sync) or uses `_executor.execute()` directly (async). Only two operations (`create_database`, `drop_database`) bypass this seam to toggle psycopg's autocommit mode — they receive both `_executor` and `_conn` from the decorator. AsyncCollection properties (`num_entities`, `index_params_async`) use `_executor.execute()` directly, not through CollectionCore (Core is sync-only). VastbaseConnection methods `execute/fetch_one/fetch_all` are private (`_execute/_fetch_one/_fetch_all`) — callers must cross the Executor seam. AsyncVastbaseConnection follows the same pattern.
+
+## Transaction
+
+A unit of work that groups multiple SQL executions. Managed as a context manager by the `Executor`. On exit without commit, the transaction is rolled back automatically.
+
+## Adapter
+
+A concrete module that satisfies an interface at a seam. In this codebase, `SyncCollection` and `AsyncCollection` are adapters — thin modules (~50 lines each) that provide connection management and an `Executor` implementation, delegating all business logic to `CollectionCore`.
+
+## Schema
+
+The structural definition of a Collection: field names, data types, vector dimensions, primary keys, and nullability constraints. Represented by `CollectionSchema` (a list of `FieldSchema` objects). Generates `CREATE TABLE` SQL.
+
+## Entity
+
+A single record in a Collection. Represented as a `dict[str, Any]` where keys are field names and values are typed data (scalars, vectors, or JSON). The ORM layer wraps entities in `Model`-decorated classes.
+
+## Mutation
+
+A write operation that changes Collection data: insert, upsert, or delete. Returns a `MutationResult` carrying affected primary keys and counts.
+
+## hybrid_search (SDK-layer multi-way recall)
+
+Performs multi-way vector recall with reranking across multiple `AnnSearchRequest` objects. Unlike pymilvus (where Milvus server executes multi-vector ANN + reranking in a single gRPC call), pyvastbase implements this at the SDK layer: issues multiple independent ANN search SQL queries (one per vector field), then merges results using a reranking strategy. Returns `SearchResult` — same type as single-field `search()`, with `distance` holding the merged score. Reranking strategies: `RRFRanker` (reciprocal rank fusion), `WeightedRRFRanker` (weighted RRF), `WeightedRanker` (weighted distance with arctan normalization per metric type). Execution model: parallel by default (asyncio.gather / thread pool), serial optional. Two distinct `hybrid_*` methods coexist with different semantics — `hybrid_search` (multi-vector recall + rerank) vs `hybrid_ann_search` (single-vector + scalar filtering via HybridANN index).
+
+## hybrid_ann_search (Vastbase-native vector + scalar filtering)
+
+Single-vector-field ANN search with scalar field filtering, using Vastbase's HybridANN index. Not multi-way recall — queries one vector field and applies WHERE-style conditions on scalar fields. The Vastbase-native counterpart to pymilvus filtered search, not to pymilvus `hybrid_search`. Confirmed distinct from `hybrid_search` by database inspection: Vastbase V3.0 Build 9 has no native multi-way recall functions (no `rrf`, `fusion`, `rerank` functions registered in `pg_proc`).
+
+## VastbaseClient
+
+High-level pymilvus `MilvusClient`-compatible entry point. Delegates all operations to existing `Collection` and `utility` module functions. Aliased as `MilvusClient` for import compatibility. Connects via `uri` (PostgreSQL connection string) or individual host/port parameters.
+
+**Implemented methods** (delegating to Collection/utility with `using` parameter):
+- Collection CRUD: `create_collection`, `drop_collection`, `has_collection`, `list_collections`, `describe_collection`, `rename_collection`, `truncate_collection`
+- Data: `insert`, `upsert`, `delete`, `search`, `query`, `get`
+- High-level query: `hybrid_search`, `query_iterator`, `search_iterator`
+- Index: `create_index`, `drop_index`, `list_indexes`, `describe_index`, `alter_index_properties`, `drop_index_properties`
+- Collection property: `alter_collection_properties`, `drop_collection_properties`, `add_collection_field`
+- Alias: `create_alias`, `drop_alias`, `alter_alias`, `list_aliases`, `describe_alias`
+- User/DB: `create_user`, `drop_user`, `reset_password`, `update_password`, `list_users`, `describe_user`, `create_database`, `drop_database`, `list_databases`, `describe_database`
+- Flush/Load: `flush`, `flush_all`, `get_flush_all_state`, `compact`, `load_collection`, `release_collection`, `get_load_state`, `get_collection_stats`
+- Server: `get_server_version`
+- Connection: `close`
+
+**NotSupportedError stubs** (PG architecture fundamentally different): partition, role, privilege_group, resource_group, replica, bulk_insert, snapshot, analyzer, optimize, collection_function, segments, compaction_plans
+
+## Connection Provider
+
+A module that supplies database connections by alias. The `ConnectionProvider` protocol defines a single method `get_connection(alias) -> VastbaseConnection`. The default implementation (`VastbaseConnections`) holds a registry of named connection configurations. Consumers (Collection, utility functions) accept an optional `_connection_provider` parameter, defaulting to the global singleton instance. VastbaseConnection is an internal implementation detail of SyncExecutor/AsyncExecutor — callers should not use `_execute/_fetch_one/_fetch_all` directly; they cross the Executor seam instead.

pyvastbase-0.2.2/MANIFEST.in

@@ -0,0 +1,22 @@
+include *.py
+include *.md
+include *.toml
+include MANIFEST.in
+graft core
+graft utils
+graft orm
+graft index
+graft search
+graft async_impl
+graft operations
+graft tests
+prune pyvastbase.egg-info
+recursive-exclude pyvastbase.egg-info *
+prune dist
+prune .pytest_cache
+prune .ruff_cache
+prune .coverage
+prune .claude
+prune .scratch
+global-exclude *.pyc
+global-exclude __pycache__