embedflow 0.3.0__tar.gz → 0.4.0__tar.gz

{embedflow-0.3.0 → embedflow-0.4.0}/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## v0.4.0 — Milvus backend
+
+- Added a read-only Milvus backend for existing dense `FLOAT_VECTOR`
+  collections.
+- Added collection/database/partition selection, HNSW/IVF search parameters,
+  metric and schema auditing, and Milvus-backed document text resolution.
+- Added deterministic standalone Docker fixtures, examples, and optional
+  `pymilvus` packaging.
+
 ## v0.3.0 — Pinecone backend
 
 - Added a read-only Pinecone backend for existing dense indexes.

{embedflow-0.3.0 → embedflow-0.4.0}/CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 title: "EmbedFlow: Upgrading Legacy Embeddings Without Full Upfront Re-Embedding"
 message: "If EmbedFlow contributes to your work, please cite this software release."
 type: software
-version: 0.3.0
+version: 0.4.0
 date-released: 2026-09-06
 repository-code: "https://github.com/arnsri33/embedflow"
 url: "https://github.com/arnsri33/embedflow"

{embedflow-0.3.0 → embedflow-0.4.0}/MANIFEST.in

@@ -1,6 +1,6 @@
 include README.md README_PYPI.md LICENSE CHANGELOG.md CONTRIBUTING.md SECURITY.md CITATION.cff requirements.txt requirements-dev.txt
 recursive-include docs *
-recursive-include examples *.md *.yaml *.json *.jsonl *.py *.sql *.sh
+recursive-include examples *.md *.yaml *.yaml.example *.json *.jsonl *.py *.sql *.sh
 recursive-include scripts *.sh *.py
 recursive-include frozen *.md *.sha256
 prune .github

{embedflow-0.3.0 → embedflow-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: embedflow
-Version: 0.3.0
+Version: 0.4.0
 Summary: Progressive embedding-model migration over existing vector indexes.
 Author: Arnav Srivastav
 License-Expression: AGPL-3.0-only
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://embedflow.org
 Project-URL: Repository, https://github.com/arnsri33/embedflow
 Project-URL: Documentation, https://github.com/arnsri33/embedflow#readme
 Project-URL: Issues, https://github.com/arnsri33/embedflow/issues
-Keywords: embeddings,vector-search,rag,information-retrieval,faiss,qdrant,pgvector,pinecone
+Keywords: embeddings,vector-search,rag,information-retrieval,faiss,qdrant,pgvector,pinecone,milvus
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
@@ -31,6 +31,8 @@ Provides-Extra: pgvector
 Requires-Dist: psycopg[binary]>=3.2; extra == "pgvector"
 Provides-Extra: pinecone
 Requires-Dist: pinecone>=6.0; extra == "pinecone"
+Provides-Extra: milvus
+Requires-Dist: pymilvus<4,>=2.5.5; extra == "milvus"
 Provides-Extra: models
 Requires-Dist: huggingface-hub<1.0,>=0.34; extra == "models"
 Requires-Dist: transformers<5.0,>=4.45; extra == "models"
@@ -45,6 +47,7 @@ Requires-Dist: faiss-cpu>=1.8.0; extra == "all"
 Requires-Dist: qdrant-client>=1.9; extra == "all"
 Requires-Dist: psycopg[binary]>=3.2; extra == "all"
 Requires-Dist: pinecone>=6.0; extra == "all"
+Requires-Dist: pymilvus<4,>=2.5.5; extra == "all"
 Requires-Dist: huggingface-hub<1.0,>=0.34; extra == "all"
 Requires-Dist: transformers<5.0,>=4.45; extra == "all"
 Requires-Dist: sentence-transformers>=3.0; extra == "all"
@@ -65,7 +68,7 @@ Dynamic: license-file
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, FAISS,
-Qdrant, pgvector, Pinecone, a CLI, and FastAPI.
+Qdrant, pgvector, Pinecone, Milvus, a CLI, and FastAPI.
 
 The full project README and architecture diagram are on
 <https://github.com/arnsri33/embedflow>.
@@ -94,6 +97,12 @@ For an existing Pinecone dense index:
 python -m pip install "embedflow[pinecone]"
 ```
 
+For an existing Milvus collection:
+
+```bash
+python -m pip install "embedflow[milvus]"
+```
+
 Qdrant and model-runtime extras are documented in the
 [installation guide](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -196,11 +205,13 @@ for definitions and reproduction details.
 | Qdrant | Supported |
 | pgvector | Supported |
 | Pinecone | Supported |
+| Milvus | Supported |
 
 See the [FAISS guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/faiss.md),
 [Qdrant guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/qdrant.md),
 and [pgvector guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md),
-and [Pinecone guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md).
+and [Pinecone guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md),
+and [Milvus guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/milvus.md).
 
 ## CLI
 
@@ -220,7 +231,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.3.0 is an alpha release for research and early real-world
+EmbedFlow v0.4.0 is an alpha release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.3.0 → embedflow-0.4.0}/README.md

@@ -7,7 +7,7 @@
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, and serving
-through FAISS, Qdrant, pgvector, Pinecone, a CLI, and FastAPI.
+through FAISS, Qdrant, pgvector, Pinecone, Milvus, a CLI, and FastAPI.
 
 [Quickstart](#try-it) · [Documentation](#documentation) · [Research](#research)
 
@@ -58,6 +58,12 @@ For an existing Pinecone dense index:
 python -m pip install "embedflow[pinecone]"
 ```
 
+For an existing Milvus collection:
+
+```bash
+python -m pip install "embedflow[milvus]"
+```
+
 Qdrant and model-runtime extras are documented in
 [`docs/installation.md`](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -197,6 +203,7 @@ is measured separately and is `UNKNOWN` until an exact reference is supplied.
 | Qdrant | Supported |
 | pgvector | Supported |
 | Pinecone | Supported |
+| Milvus | Supported |
 
 Backend-specific setup and examples:
 
@@ -204,6 +211,7 @@ Backend-specific setup and examples:
 - [Qdrant](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/qdrant.md)
 - [pgvector](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md)
 - [Pinecone](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md)
+- [Milvus](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/milvus.md)
 - [Adding a backend](https://github.com/arnsri33/embedflow/blob/main/CONTRIBUTING.md)
 
 ## CLI
@@ -238,7 +246,7 @@ OpenAPI documentation; see
 
 ## Status
 
-EmbedFlow v0.3.0 is an alpha release for research and early real-world
+EmbedFlow v0.4.0 is an alpha release for research and early real-world
 testing.
 
 - T2-v1 reports an empirical finite-tail diagnostic.

{embedflow-0.3.0 → embedflow-0.4.0}/README_PYPI.md

@@ -5,7 +5,7 @@
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, FAISS,
-Qdrant, pgvector, Pinecone, a CLI, and FastAPI.
+Qdrant, pgvector, Pinecone, Milvus, a CLI, and FastAPI.
 
 The full project README and architecture diagram are on
 <https://github.com/arnsri33/embedflow>.
@@ -34,6 +34,12 @@ For an existing Pinecone dense index:
 python -m pip install "embedflow[pinecone]"
 ```
 
+For an existing Milvus collection:
+
+```bash
+python -m pip install "embedflow[milvus]"
+```
+
 Qdrant and model-runtime extras are documented in the
 [installation guide](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -136,11 +142,13 @@ for definitions and reproduction details.
 | Qdrant | Supported |
 | pgvector | Supported |
 | Pinecone | Supported |
+| Milvus | Supported |
 
 See the [FAISS guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/faiss.md),
 [Qdrant guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/qdrant.md),
 and [pgvector guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md),
-and [Pinecone guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md).
+and [Pinecone guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md),
+and [Milvus guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/milvus.md).
 
 ## CLI
 
@@ -160,7 +168,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.3.0 is an alpha release for research and early real-world
+EmbedFlow v0.4.0 is an alpha release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.3.0 → embedflow-0.4.0}/docs/configuration.md

@@ -14,7 +14,7 @@ target:
   device: cuda
 
 index:
-  backend: faiss                 # faiss, qdrant, pgvector, or pinecone
+  backend: faiss                 # faiss, qdrant, pgvector, pinecone, or milvus
   path: ./legacy.index
   ids: ./legacy.index.ids.json   # FAISS sidecar
   metric: cosine
@@ -24,6 +24,8 @@ index:
   #                 hnsw_ef_search, ivfflat_probes
   # Pinecone fields: host (preferred) or index_name, api_key_env, namespace,
   #                  text_metadata_field
+  # Milvus fields: uri, token_env, database, collection, id_field, vector_field,
+  #                text_field, partition_names, search_params, auto_load
 
 documents:
   path: ./documents.jsonl
@@ -95,6 +97,15 @@ EMBEDFLOW_PINECONE_INDEX_NAME
 EMBEDFLOW_PINECONE_NAMESPACE
 EMBEDFLOW_PINECONE_TEXT_METADATA_FIELD
 EMBEDFLOW_PINECONE_API_KEY_ENV
+EMBEDFLOW_MILVUS_URI
+EMBEDFLOW_MILVUS_TOKEN_ENV
+EMBEDFLOW_MILVUS_DATABASE
+EMBEDFLOW_MILVUS_COLLECTION
+EMBEDFLOW_MILVUS_ID_FIELD
+EMBEDFLOW_MILVUS_VECTOR_FIELD
+EMBEDFLOW_MILVUS_TEXT_FIELD
+EMBEDFLOW_MILVUS_PARTITIONS
+EMBEDFLOW_MILVUS_AUTO_LOAD
 EMBEDFLOW_INDEX_NPROBE
 EMBEDFLOW_DOCUMENTS_PATH
 EMBEDFLOW_CACHE_PATH

{embedflow-0.3.0 → embedflow-0.4.0}/docs/installation.md

@@ -32,6 +32,7 @@ python -m pip install -e .
 | `qdrant` | Qdrant client and adapter |
 | `pgvector` | Psycopg 3 binary driver and pgvector adapter |
 | `pinecone` | Official Pinecone Python SDK and adapter |
+| `milvus` | Official pymilvus SDK and adapter |
 | `models` | PyTorch, Transformers, Sentence Transformers, and Hub client |
 | `dashboard` | FastAPI, Uvicorn, and Pydantic |
 | `dev` | Pytest, Ruff, and build tooling |
@@ -66,6 +67,12 @@ Install the optional adapter with `python -m pip install "embedflow[pinecone]"`.
 Set `PINECONE_API_KEY` in the environment and configure an existing dense
 index host; see [`integrations/pinecone.md`](integrations/pinecone.md).
 
+## Milvus
+
+Install the optional adapter with `python -m pip install "embedflow[milvus]"`.
+Configure an existing collection URI, database, and vector field; see
+[`integrations/milvus.md`](integrations/milvus.md).
+
 ## CPU and GPU
 
 The deterministic demo runs on CPU. Real model serving accepts `--device cpu`

embedflow-0.4.0/docs/integrations/milvus.md

@@ -0,0 +1,165 @@
+# Milvus
+
+EmbedFlow can use an existing Milvus collection as a read-only source
+candidate index. A source query is executed by Milvus, the returned documents
+are reranked by the target model, and target vectors are stored in EmbedFlow's
+local cache. The adapter does not copy or rebuild the source collection.
+
+## Install and connect
+
+Install the optional official SDK (the adapter is tested with pymilvus 2.5.5
+through 3.0.1):
+
+```bash
+python -m pip install "embedflow[milvus]"
+```
+
+For a local standalone server, the URI is normally
+`http://127.0.0.1:19530`. Cloud/Milvus-compatible endpoints can use the same
+`MilvusClient` URI and token semantics. Keep credentials out of YAML:
+
+```bash
+export EMBEDFLOW_MILVUS_TOKEN='user:password-or-cloud-token'
+```
+
+An unauthenticated local server may omit the variable. EmbedFlow never puts a
+token in status, telemetry, generated configuration, or error messages.
+
+Example configuration for a collection that stores its own text:
+
+```yaml
+source:
+  model: sentence-transformers/all-MiniLM-L6-v2
+  dimension: 384
+  device: cuda
+target:
+  model: Qwen/Qwen3-Embedding-0.6B
+  device: cuda
+index:
+  backend: milvus
+  uri: http://127.0.0.1:19530
+  token_env: EMBEDFLOW_MILVUS_TOKEN
+  database: default
+  collection: documents
+  id_field: id
+  vector_field: embedding
+  text_field: content
+  metric: cosine
+  partition_names: []
+  auto_load: false
+documents:
+  path: ./documents.jsonl
+  id_field: id
+  text_field: text
+```
+
+`uri`, `database`, `collection`, `id_field`, `vector_field`, and
+`text_field` are quoted as ordinary SDK arguments; they are not interpolated
+into SQL. `vector_field` should identify a dense `FLOAT_VECTOR` field. The
+first release does not claim support for `SPARSE_FLOAT_VECTOR`, `BINARY_VECTOR`,
+`FLOAT16_VECTOR`, or `BFLOAT16_VECTOR`. If a collection has multiple dense
+vector fields, configure `vector_field` explicitly.
+
+## Text and IDs
+
+There are two supported text layouts:
+
+* Set `text_field` when the Milvus collection has `id`, `embedding`, and a
+  string text field. Search requests ask Milvus for only that field.
+* Keep `text_field` unset and provide the normal EmbedFlow JSONL document
+  store. Candidate retrieval then asks Milvus for IDs only and resolves text
+  through the shared document-store contract.
+
+Milvus `INT64` and `VARCHAR` primary keys are supported. IDs are exposed to
+EmbedFlow as strings, so an INT64 value `42` is represented as `"42"` at the
+application boundary while a VARCHAR value `"42"` remains a string. No numeric
+coercion is performed for VARCHAR IDs.
+
+## Metrics and indexes
+
+Canonical metric names are `cosine`, `inner_product`/`dot`, and
+`l2`/`euclidean`; they map to Milvus `COSINE`, `IP`, and `L2`. Milvus returns
+COSINE/IP similarities (larger is better) and L2 distances (smaller is
+better). EmbedFlow negates L2 distances only, preserving the shared
+higher-is-better `SearchHit.score` convention.
+
+The adapter does not create indexes. Existing HNSW and IVF_FLAT indexes are
+queried when present. Optional native search parameters can be supplied as:
+
+```yaml
+index:
+  search_params:
+    ef: 64       # HNSW; raised to at least top-k when required by Milvus
+    # nprobe: 16 # IVF_FLAT
+```
+
+Native Milvus form (`metric_type` plus `params`) is also accepted. Unknown
+keys are rejected. `ef` is never sent below the requested top-k because HNSW
+servers reject that request.
+
+## Loading, databases, and partitions
+
+EmbedFlow checks collection load state. `auto_load: false` (the conservative
+default) returns an actionable error if the collection is not loaded. Set
+`auto_load: true` only when explicitly permitting EmbedFlow to load this
+collection; EmbedFlow never releases or unloads a collection.
+
+`database` selects the Milvus database. `partition_names` restricts every
+search and text lookup to the listed partitions. A missing partition fails at
+connection time when the server exposes partition metadata. No arbitrary
+Milvus filter expression is exposed because the shared `VectorIndex` contract
+has no portable filter field.
+
+## Commands and audit
+
+```bash
+embedflow doctor --config embedflow.yaml
+embedflow audit-index --config embedflow.yaml
+embedflow analyze --config embedflow.yaml --queries ./probe_queries.jsonl
+embedflow serve --config embedflow.yaml --device cuda
+embedflow search --config embedflow.yaml "your query"
+embedflow status --config embedflow.yaml
+```
+
+`audit-index` reports URI (redacted), database, collection, fields, dense
+vector type, dimension, metric, index type, load state, partitions, row
+count, and a small retrieval/text probe. It intentionally keeps ANN fidelity,
+T2-v1 compatibility, and candidate quality as separate `UNKNOWN`/empirical
+questions; a reachable collection is not proof of migration compatibility.
+
+Normal `doctor`, `audit-index`, `analyze`, `serve`, `search`, `status`, and
+`prewarm` operations are read-only against the configured collection. Fixture
+creation and inserts are confined to `scripts/milvus_fixture.py`, tests, and
+the example setup.
+
+## Local example
+
+The repository includes a small standalone deployment and deterministic
+fixture:
+
+```bash
+cd examples/milvus
+docker compose up -d
+python -m pip install "embedflow[milvus,dashboard]"
+./run_demo.sh
+```
+
+The fixture utility is deliberately explicit and refuses to overwrite an
+existing collection. It can create HNSW, IVF_FLAT, or FLAT test indexes. No
+model weights or credentials are committed.
+
+## Troubleshooting and limitations
+
+* `Milvus support requires ...`: install the `[milvus]` extra in the active
+  environment.
+* `collection ... was not found`: check `uri`, `database`, and collection name.
+* `collection ... is not loaded`: load it administratively or opt in to
+  `auto_load: true`.
+* Dimension/metric errors mean the configured source model contract does not
+  match the existing vector field/index; EmbedFlow does not rewrite it.
+* The adapter uses one synchronous `MilvusClient`; concurrent calls are
+  serialized by a lock. A pool is intentionally not introduced for this
+  release.
+* Milvus-compatible URI/token semantics should work with Zilliz Cloud where
+  supported by the SDK, but Zilliz Cloud has not been independently validated
+  for this release.

{embedflow-0.3.0 → embedflow-0.4.0}/docs/limitations.md

@@ -1,6 +1,6 @@
 # Limitations and release scope
 
-EmbedFlow v0.3.0 is an alpha release for research and early real-world
+EmbedFlow v0.4.0 is an alpha release for research and early real-world
 testing. The serving path is designed to make migration experiments concrete;
 production rollout still requires application-specific validation.
 

{embedflow-0.3.0 → embedflow-0.4.0}/docs/releasing.md

@@ -18,8 +18,8 @@ python -m twine check dist/*
 Inspect both archives before uploading:
 
 ```bash
-unzip -l dist/embedflow-0.3.0-py3-none-any.whl
-tar -tzf dist/embedflow-0.3.0.tar.gz
+unzip -l dist/embedflow-0.4.0-py3-none-any.whl
+tar -tzf dist/embedflow-0.4.0.tar.gz
 sha256sum dist/*
 ```
 
@@ -32,7 +32,7 @@ Test the wheel outside the source tree:
 ```bash
 python -m venv /tmp/embedflow-wheel-test
 /tmp/embedflow-wheel-test/bin/python -m pip install --upgrade pip
-/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.3.0-py3-none-any.whl
+/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.4.0-py3-none-any.whl
 cd /tmp
 /tmp/embedflow-wheel-test/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-wheel-test/bin/embedflow --help
@@ -65,7 +65,7 @@ python -m venv /tmp/embedflow-testpypi
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  embedflow==0.3.0
+  embedflow==0.4.0
 cd /tmp
 /tmp/embedflow-testpypi/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-testpypi/bin/embedflow --help
@@ -79,11 +79,11 @@ Test optional integrations in a second clean environment:
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  "embedflow[faiss,dashboard,pinecone]==0.3.0"
+  "embedflow[faiss,dashboard,pinecone,milvus]==0.4.0"
 ```
 
 If the same filename already exists on TestPyPI, use a pre-release such as
-`0.3.0rc1` for the TestPyPI-only trial. Keep production `0.3.0` unchanged.
+`0.4.0rc1` for the TestPyPI-only trial. Keep production `0.4.0` unchanged.
 
 ## Trusted Publishing configuration
 
@@ -115,7 +115,7 @@ above keeps the test step explicit.
 3. Run the final release gate and review the generated report.
 4. Configure the PyPI pending publisher and protected `pypi` environment.
 5. Create a Git tag and GitHub Release for the exact package version, for
-   example `v0.3.0`.
+   example `v0.4.0`.
 6. Approve the `pypi` environment when the release workflow is ready.
 7. Verify the files and metadata on PyPI.
 8. Install from production PyPI in a directory outside this checkout.

{embedflow-0.3.0 → embedflow-0.4.0}/embedflow/__init__.py

@@ -1,12 +1,12 @@
 """EmbedFlow: progressive embedding-model migration for existing indexes."""
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 
 from .config import EmbedFlowConfig, load_config
 
 
 def migrate(*args, **kwargs):
-    """Start progressive migration over an existing FAISS, Qdrant, pgvector, or Pinecone index.
+    """Start progressive migration over an existing FAISS, Qdrant, pgvector, Pinecone, or Milvus index.
 
     Imported lazily to keep the lightweight configuration package free of
     model-serving dependencies at import time.  See ``embedflow.migration``