embedflow 0.4.0__tar.gz → 0.6.0__tar.gz

{embedflow-0.4.0 → embedflow-0.6.0}/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## v0.6.0 — Migration planner
+
+- Added an advisory `embedflow plan` command and Python API that combine
+  source-index preflight, registry evidence, representative probes, frozen
+  T2-v1 diagnostics, candidate-depth selection, cache planning, economics, and
+  staged rollout guidance without routing traffic or mutating the source.
+- Added structured JSON/YAML plan artifacts with explicit warnings and
+  measured/user-supplied/modeled/unknown provenance.
+
+## v0.5.0 — Weaviate backend
+
+- Added a read-only Weaviate v4 backend for existing externally-vectorized
+  dense collections, including named-vector selection and UUID IDs.
+- Added HTTP/gRPC connection settings, property-backed or external text
+  resolution, metric normalization, collection auditing, and safe client
+  shutdown.
+- Added the optional `weaviate-client` extra, Docker fixture, example, and
+  live 10,000-object validation harness.
+
 ## v0.4.0 — Milvus backend
 
 - Added a read-only Milvus backend for existing dense `FLOAT_VECTOR`

{embedflow-0.4.0 → embedflow-0.6.0}/CITATION.cff

@@ -2,8 +2,8 @@ cff-version: 1.2.0
 title: "EmbedFlow: Upgrading Legacy Embeddings Without Full Upfront Re-Embedding"
 message: "If EmbedFlow contributes to your work, please cite this software release."
 type: software
-version: 0.4.0
-date-released: 2026-09-06
+version: 0.6.0
+date-released: 2026-09-13
 repository-code: "https://github.com/arnsri33/embedflow"
 url: "https://github.com/arnsri33/embedflow"
 license: AGPL-3.0-only

{embedflow-0.4.0 → embedflow-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: embedflow
-Version: 0.4.0
+Version: 0.6.0
 Summary: Progressive embedding-model migration over existing vector indexes.
 Author: Arnav Srivastav
 License-Expression: AGPL-3.0-only
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://embedflow.org
 Project-URL: Repository, https://github.com/arnsri33/embedflow
 Project-URL: Documentation, https://github.com/arnsri33/embedflow#readme
 Project-URL: Issues, https://github.com/arnsri33/embedflow/issues
-Keywords: embeddings,vector-search,rag,information-retrieval,faiss,qdrant,pgvector,pinecone,milvus
+Keywords: embeddings,vector-search,rag,information-retrieval,faiss,qdrant,pgvector,pinecone,milvus,weaviate
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
@@ -33,6 +33,8 @@ Provides-Extra: pinecone
 Requires-Dist: pinecone>=6.0; extra == "pinecone"
 Provides-Extra: milvus
 Requires-Dist: pymilvus<4,>=2.5.5; extra == "milvus"
+Provides-Extra: weaviate
+Requires-Dist: weaviate-client<5,>=4.16; extra == "weaviate"
 Provides-Extra: models
 Requires-Dist: huggingface-hub<1.0,>=0.34; extra == "models"
 Requires-Dist: transformers<5.0,>=4.45; extra == "models"
@@ -48,6 +50,7 @@ Requires-Dist: qdrant-client>=1.9; extra == "all"
 Requires-Dist: psycopg[binary]>=3.2; extra == "all"
 Requires-Dist: pinecone>=6.0; extra == "all"
 Requires-Dist: pymilvus<4,>=2.5.5; extra == "all"
+Requires-Dist: weaviate-client<5,>=4.16; extra == "all"
 Requires-Dist: huggingface-hub<1.0,>=0.34; extra == "all"
 Requires-Dist: transformers<5.0,>=4.45; extra == "all"
 Requires-Dist: sentence-transformers>=3.0; extra == "all"
@@ -68,7 +71,7 @@ Dynamic: license-file
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, FAISS,
-Qdrant, pgvector, Pinecone, Milvus, a CLI, and FastAPI.
+Qdrant, pgvector, Pinecone, Milvus, Weaviate, a CLI, and FastAPI.
 
 The full project README and architecture diagram are on
 <https://github.com/arnsri33/embedflow>.
@@ -103,6 +106,12 @@ For an existing Milvus collection:
 python -m pip install "embedflow[milvus]"
 ```
 
+For an existing Weaviate v4 collection:
+
+```bash
+python -m pip install "embedflow[weaviate]"
+```
+
 Qdrant and model-runtime extras are documented in the
 [installation guide](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -180,6 +189,21 @@ Selected core records (nDCG@10, `G(50)`):
 See the [registry documentation](https://github.com/arnsri33/embedflow/blob/main/docs/registry.md)
 for matching levels, contract fingerprints, and provenance.
 
+## Advisory migration plans
+
+Use representative query probes to produce a bounded migration recommendation
+before serving:
+
+```bash
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl --format json --output migration-plan.json
+```
+
+The planner reports preflight, registry evidence, frozen T2-v1 finite-tail
+behavior, candidate K, cache/economics projections, and staged rollout
+guidance. It never routes traffic or mutates the source index; `SAFE` is not a
+qrels-based retrieval-quality guarantee. See the [planner guide](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
+
 ## Research
 
 For candidate depth `K`, EmbedFlow measures:
@@ -206,18 +230,21 @@ for definitions and reproduction details.
 | pgvector | Supported |
 | Pinecone | Supported |
 | Milvus | Supported |
+| Weaviate | Supported |
 
 See the [FAISS guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/faiss.md),
 [Qdrant guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/qdrant.md),
 and [pgvector guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md),
 and [Pinecone guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md),
 and [Milvus guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/milvus.md).
+See the [Weaviate guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/weaviate.md).
 
 ## CLI
 
 ```bash
 embedflow --help
 embedflow analyze --help
+embedflow plan --help
 embedflow serve --config ./embedflow.yaml
 embedflow status --config ./embedflow.yaml
 embedflow registry list
@@ -231,7 +258,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.4.0 is an alpha release for research and early real-world
+EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.4.0 → embedflow-0.6.0}/README.md

@@ -7,7 +7,7 @@
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, and serving
-through FAISS, Qdrant, pgvector, Pinecone, Milvus, a CLI, and FastAPI.
+through FAISS, Qdrant, pgvector, Pinecone, Milvus, Weaviate, a CLI, and FastAPI.
 
 [Quickstart](#try-it) · [Documentation](#documentation) · [Research](#research)
 
@@ -64,6 +64,12 @@ For an existing Milvus collection:
 python -m pip install "embedflow[milvus]"
 ```
 
+For an existing Weaviate v4 collection (HTTP and gRPC endpoints required):
+
+```bash
+python -m pip install "embedflow[weaviate]"
+```
+
 Qdrant and model-runtime extras are documented in
 [`docs/installation.md`](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -136,6 +142,20 @@ session = embedflow.migrate(
 results = session.search("what causes auroras?", top_k=10)
 ```
 
+## Plan a migration
+
+Build a conservative, evidence-aware recommendation before serving. The
+planner reuses backend preflight, registry matching, and frozen T2-v1; it is
+advisory and never routes traffic or mutates the source index.
+
+```bash
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl --format json --output migration-plan.json
+```
+
+`SAFE` is an empirical finite-tail signal, not a retrieval-quality guarantee.
+See [`docs/planner.md`](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
+
 Search responses expose `COLD`, `PARTIAL`, or `WARM`, cache hits and misses,
 synchronous work, queued work, and stage timings. Once the candidate vectors
 are warm, target scoring over that candidate set is deterministic.
@@ -204,6 +224,7 @@ is measured separately and is `UNKNOWN` until an exact reference is supplied.
 | pgvector | Supported |
 | Pinecone | Supported |
 | Milvus | Supported |
+| Weaviate | Supported |
 
 Backend-specific setup and examples:
 
@@ -212,6 +233,7 @@ Backend-specific setup and examples:
 - [pgvector](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md)
 - [Pinecone](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md)
 - [Milvus](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/milvus.md)
+- [Weaviate](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/weaviate.md)
 - [Adding a backend](https://github.com/arnsri33/embedflow/blob/main/CONTRIBUTING.md)
 
 ## CLI
@@ -219,6 +241,7 @@ Backend-specific setup and examples:
 ```bash
 embedflow --help
 embedflow analyze --help
+embedflow plan --help
 embedflow serve --config ./embedflow.yaml
 embedflow status --config ./embedflow.yaml
 embedflow registry list
@@ -246,7 +269,7 @@ OpenAPI documentation; see
 
 ## Status
 
-EmbedFlow v0.4.0 is an alpha release for research and early real-world
+EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
 testing.
 
 - T2-v1 reports an empirical finite-tail diagnostic.

{embedflow-0.4.0 → embedflow-0.6.0}/README_PYPI.md

@@ -5,7 +5,7 @@
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, FAISS,
-Qdrant, pgvector, Pinecone, Milvus, a CLI, and FastAPI.
+Qdrant, pgvector, Pinecone, Milvus, Weaviate, a CLI, and FastAPI.
 
 The full project README and architecture diagram are on
 <https://github.com/arnsri33/embedflow>.
@@ -40,6 +40,12 @@ For an existing Milvus collection:
 python -m pip install "embedflow[milvus]"
 ```
 
+For an existing Weaviate v4 collection:
+
+```bash
+python -m pip install "embedflow[weaviate]"
+```
+
 Qdrant and model-runtime extras are documented in the
 [installation guide](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -117,6 +123,21 @@ Selected core records (nDCG@10, `G(50)`):
 See the [registry documentation](https://github.com/arnsri33/embedflow/blob/main/docs/registry.md)
 for matching levels, contract fingerprints, and provenance.
 
+## Advisory migration plans
+
+Use representative query probes to produce a bounded migration recommendation
+before serving:
+
+```bash
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl --format json --output migration-plan.json
+```
+
+The planner reports preflight, registry evidence, frozen T2-v1 finite-tail
+behavior, candidate K, cache/economics projections, and staged rollout
+guidance. It never routes traffic or mutates the source index; `SAFE` is not a
+qrels-based retrieval-quality guarantee. See the [planner guide](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
+
 ## Research
 
 For candidate depth `K`, EmbedFlow measures:
@@ -143,18 +164,21 @@ for definitions and reproduction details.
 | pgvector | Supported |
 | Pinecone | Supported |
 | Milvus | Supported |
+| Weaviate | Supported |
 
 See the [FAISS guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/faiss.md),
 [Qdrant guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/qdrant.md),
 and [pgvector guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md),
 and [Pinecone guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md),
 and [Milvus guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/milvus.md).
+See the [Weaviate guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/weaviate.md).
 
 ## CLI
 
 ```bash
 embedflow --help
 embedflow analyze --help
+embedflow plan --help
 embedflow serve --config ./embedflow.yaml
 embedflow status --config ./embedflow.yaml
 embedflow registry list
@@ -168,7 +192,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.4.0 is an alpha release for research and early real-world
+EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.4.0 → embedflow-0.6.0}/docs/api.md

@@ -48,6 +48,11 @@ The response includes the result list and migration fields such as:
 request. A partial response scores the available target vectors; it can differ
 from the fully warm ranking.
 
+The advisory migration planner is exposed through the Python API and the
+`embedflow plan` CLI. It is intentionally not a synchronous FastAPI endpoint:
+probe analysis may load models and perform bounded candidate work, so operators
+should generate a plan artifact out of band and publish/read it as needed.
+
 `/status` includes safe backend metadata. For pgvector this names the
 schema/table and vector contract; for Pinecone it names the host/index,
 namespace, dimension, metric, and safe vector counts. Neither backend returns

{embedflow-0.4.0 → embedflow-0.6.0}/docs/cli.md

@@ -9,6 +9,7 @@ list. The commands below are the main public entry points.
 embedflow init --config ./embedflow.yaml
 embedflow analyze --config ./embedflow.yaml --output-dir ./analysis
 embedflow evaluate --config ./experiment.yaml --output-dir ./results
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
 ```
 
 `analyze` is the no-target-index workflow. It uses probe queries and frozen
@@ -16,6 +17,15 @@ T2-v1 logic. `evaluate` is the labelled workflow; it computes source quality,
 native target quality, target-within-source-candidates quality, candidate gap,
 containment, and migration depth when the required inputs are available.
 
+`plan` is the advisory migration planner. It runs source preflight and registry
+matching, optionally samples a JSONL probe set, and reports a candidate-depth
+recommendation, cache/economics projections, warnings, and staged rollout
+guidance. Use `--format json` (or `yaml`) for a structured artifact. A plan
+with no probes reports `T2-v1: NOT_RUN`; `SAFE` is a finite-tail diagnostic,
+not a qrels-based retrieval-quality guarantee. A `DEFER`/`EXPAND_PROBE` result
+is still a successful analytical command and exits zero; invalid configuration
+or failed preflight exits nonzero.
+
 ## Serving and operations
 
 ```bash

{embedflow-0.4.0 → embedflow-0.6.0}/docs/configuration.md

@@ -14,7 +14,7 @@ target:
   device: cuda
 
 index:
-  backend: faiss                 # faiss, qdrant, pgvector, pinecone, or milvus
+  backend: faiss                 # faiss, qdrant, pgvector, pinecone, milvus, or weaviate
   path: ./legacy.index
   ids: ./legacy.index.ids.json   # FAISS sidecar
   metric: cosine
@@ -26,6 +26,9 @@ index:
   #                  text_metadata_field
   # Milvus fields: uri, token_env, database, collection, id_field, vector_field,
   #                text_field, partition_names, search_params, auto_load
+  # Weaviate fields: uri (or http_host/http_port), grpc_host/grpc_port,
+  #                  secure, collection, vector_name, text_property,
+  #                  tenant, api_key_env
 
 documents:
   path: ./documents.jsonl
@@ -55,6 +58,25 @@ economics:
   gpu_price_per_hour: null
   target_docs_per_second: null
 
+# Optional advisory planner settings. CLI flags override these values.
+planner:
+  max_probes: 250
+  seed: 42
+  k_grid: [20, 50, 100, 200, 500]
+  max_candidates: null
+  max_target_encodes: null
+  max_sync_misses: null
+  background_batch_size: null
+  gpu_hourly_cost: null
+  target_docs_per_second: null
+  queries_per_second: null
+  daily_queries: null
+  cache_hit_rate: null
+  latency_budget_ms: null
+  access_trace: null
+  corpus_name: null
+  corpus_fingerprint: null
+
 state_path: ./embedflow_state.json
 ```
 
@@ -106,6 +128,18 @@ EMBEDFLOW_MILVUS_VECTOR_FIELD
 EMBEDFLOW_MILVUS_TEXT_FIELD
 EMBEDFLOW_MILVUS_PARTITIONS
 EMBEDFLOW_MILVUS_AUTO_LOAD
+EMBEDFLOW_WEAVIATE_URI
+EMBEDFLOW_WEAVIATE_HTTP_HOST
+EMBEDFLOW_WEAVIATE_HTTP_PORT
+EMBEDFLOW_WEAVIATE_GRPC_HOST
+EMBEDFLOW_WEAVIATE_GRPC_PORT
+EMBEDFLOW_WEAVIATE_SECURE
+EMBEDFLOW_WEAVIATE_GRPC_SECURE
+EMBEDFLOW_WEAVIATE_API_KEY_ENV
+EMBEDFLOW_WEAVIATE_COLLECTION
+EMBEDFLOW_WEAVIATE_VECTOR_NAME
+EMBEDFLOW_WEAVIATE_TEXT_PROPERTY
+EMBEDFLOW_WEAVIATE_TENANT
 EMBEDFLOW_INDEX_NPROBE
 EMBEDFLOW_DOCUMENTS_PATH
 EMBEDFLOW_CACHE_PATH
@@ -115,6 +149,21 @@ EMBEDFLOW_CANDIDATE_DEPTH
 EMBEDFLOW_MAX_SYNC_MISSES
 EMBEDFLOW_BACKGROUND_BATCH_SIZE
 EMBEDFLOW_PROBE_KMAX
+EMBEDFLOW_PLANNER_MAX_PROBES
+EMBEDFLOW_PLANNER_SEED
+EMBEDFLOW_PLANNER_MAX_CANDIDATES
+EMBEDFLOW_PLANNER_MAX_TARGET_ENCODINGS
+EMBEDFLOW_PLANNER_MAX_SYNC_MISSES
+EMBEDFLOW_PLANNER_BACKGROUND_BATCH_SIZE
+EMBEDFLOW_PLANNER_GPU_HOURLY_COST
+EMBEDFLOW_PLANNER_TARGET_DOCS_PER_SECOND
+EMBEDFLOW_PLANNER_QPS
+EMBEDFLOW_PLANNER_DAILY_QUERIES
+EMBEDFLOW_PLANNER_CACHE_HIT_RATE
+EMBEDFLOW_PLANNER_LATENCY_BUDGET_MS
+EMBEDFLOW_PLANNER_ACCESS_TRACE
+EMBEDFLOW_PLANNER_CORPUS_NAME
+EMBEDFLOW_PLANNER_CORPUS_FINGERPRINT
 ```
 
 ## Input files

{embedflow-0.4.0 → embedflow-0.6.0}/docs/installation.md

@@ -33,6 +33,7 @@ python -m pip install -e .
 | `pgvector` | Psycopg 3 binary driver and pgvector adapter |
 | `pinecone` | Official Pinecone Python SDK and adapter |
 | `milvus` | Official pymilvus SDK and adapter |
+| `weaviate` | Official Weaviate Python v4 SDK and adapter |
 | `models` | PyTorch, Transformers, Sentence Transformers, and Hub client |
 | `dashboard` | FastAPI, Uvicorn, and Pydantic |
 | `dev` | Pytest, Ruff, and build tooling |
@@ -73,6 +74,12 @@ Install the optional adapter with `python -m pip install "embedflow[milvus]"`.
 Configure an existing collection URI, database, and vector field; see
 [`integrations/milvus.md`](integrations/milvus.md).
 
+## Weaviate
+
+Install the current v4 client with `python -m pip install "embedflow[weaviate]"`.
+Configure an existing externally-vectorized collection, including both the
+HTTP and gRPC endpoints; see [`integrations/weaviate.md`](integrations/weaviate.md).
+
 ## CPU and GPU
 
 The deterministic demo runs on CPU. Real model serving accepts `--device cpu`

embedflow-0.6.0/docs/integrations/weaviate.md

@@ -0,0 +1,114 @@
+# Weaviate
+
+EmbedFlow's Weaviate backend (tested with `weaviate-client` 4.23.1 and
+Weaviate 1.30.6) reads an existing, externally vectorized dense collection.
+It retrieves candidates with the v4 `near_vector` API, reranks them with the
+target model, and progressively materializes target vectors in EmbedFlow's
+local cache. It does not create, update, delete, batch-import, or re-vectorize
+objects in the source collection.
+
+## Install and connect
+
+```bash
+python -m pip install "embedflow[weaviate]"
+```
+
+The local v4 client needs both HTTP and gRPC. A standard Docker deployment
+exposes HTTP 8080 and gRPC 50051:
+
+```bash
+cd examples/weaviate
+docker compose up -d
+```
+
+Configuration uses a URI or explicit host/port fields:
+
+```yaml
+index:
+  backend: weaviate
+  uri: http://127.0.0.1:8080
+  http_port: 8080
+  grpc_host: 127.0.0.1
+  grpc_port: 50051
+  collection: Documents
+  vector_name: default
+  text_property: content
+  metric: cosine
+  api_key_env: WEAVIATE_API_KEY
+```
+
+For a custom or cloud-compatible endpoint set `http_host`, `http_port`,
+`grpc_host`, `grpc_port`, `secure`, and `grpc_secure` as appropriate. Keep the
+credential out of YAML:
+
+```bash
+export WEAVIATE_API_KEY='your-key'
+embedflow doctor --config ./embedflow.yaml
+```
+
+Cloud connections use the same v4 URI/token conventions. This release has
+validated local standalone Weaviate, not an independently operated Weaviate
+Cloud cluster.
+
+## Collection contract
+
+The selected collection must contain a dense, externally supplied vector. Set
+`vector_name` for named-vector collections; if more than one vector exists and
+the name is omitted, EmbedFlow fails rather than guessing. `cosine`, `dot` (or
+`inner_product`), and `l2`/`euclidean` are supported. Weaviate's distance is
+converted to EmbedFlow's larger-is-better score convention (`-distance`), so
+rank order remains the server's rank order.
+
+Weaviate objects use UUIDs. EmbedFlow preserves their canonical string form;
+it does not generate IDs during reads. The source model dimension must match
+the existing vector index. Weaviate versions that do not expose dimensionality
+in collection configuration report the value as unverified and the server
+still validates each query vector.
+
+For text, choose one of these modes:
+
+* `text_property: content` requests only that property with each candidate and
+  validates that it is a string.
+* Provide the normal EmbedFlow JSONL `documents` store. Candidate queries then
+  request no Weaviate properties and the shared document store resolves text.
+
+The adapter requests no stored vector payload. `tenant` can bind a configured
+multi-tenant collection to one tenant; an omitted tenant never selects one
+implicitly.
+
+## Audit and operations
+
+```bash
+embedflow audit-index --config ./embedflow.yaml
+embedflow analyze --config ./embedflow.yaml
+embedflow serve --config ./embedflow.yaml --device cuda
+```
+
+`doctor` and `audit-index` report readiness, collection/vector configuration,
+metric, object count, tenant, and text resolution. Reachability is reported
+separately from candidate compatibility and ANN fidelity; a healthy server is
+not evidence that a model migration is scientifically safe. `client.close()`
+is called when a one-shot command or serving session shuts down.
+
+## Read-only and troubleshooting
+
+The production adapter uses only collection lookup/config, `near_vector`,
+batched property lookup, iteration, and aggregate counts. Fixture creation and
+batch writes live only in `scripts/weaviate_fixture.py` and tests. Do not point
+fixture setup at a user collection.
+
+* `Weaviate support requires ...`: install the optional extra in the active
+  environment.
+* HTTP works but startup says gRPC is unavailable: expose 50051 and set
+  `grpc_host`/`grpc_port` correctly; v4 search uses gRPC.
+* `vector_name ... was not found` or multiple-vector errors: inspect the
+  collection schema and configure the exact name.
+* Dimension or metric mismatch: use the source model contract and the
+  collection's configured distance metric.
+* Missing/invalid text: fix `text_property` or use an external document store.
+
+The adapter does not expose arbitrary Weaviate filters or index-tuning writes.
+HNSW configuration is introspected read-only. Integrated Weaviate vectorizers
+are not treated as the EmbedFlow source model unless their contract is
+independently verified; this release focuses on externally vectorized dense
+collections.

{embedflow-0.4.0 → embedflow-0.6.0}/docs/limitations.md

@@ -1,6 +1,6 @@
 # Limitations and release scope
 
-EmbedFlow v0.4.0 is an alpha release for research and early real-world
+EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
 testing. The serving path is designed to make migration experiments concrete;
 production rollout still requires application-specific validation.
 

embedflow-0.6.0/docs/planner.md

@@ -0,0 +1,111 @@
+# Migration planner
+
+`embedflow plan` is a bounded, advisory analysis for moving from an existing
+source vector index to a target embedding model. It composes the existing
+backend audit, model contracts, retained registry evidence, probe retrieval,
+frozen T2-v1 diagnostic, cache behavior, and economics helpers. It never
+routes traffic and does not create, update, or delete source-index data.
+
+## Quickstart
+
+Install EmbedFlow with the optional dependency for the configured backend, then
+provide representative query probes as JSONL:
+
+```json
+{"id":"q1","query":"how do I reset my password?"}
+{"id":"q2","query":"what is the vacation policy?"}
+```
+
+Run a terminal plan or save a machine-readable artifact:
+
+```bash
+embedflow plan --config embedflow.yaml --queries probes.jsonl
+embedflow plan --config embedflow.yaml --queries probes.jsonl \
+  --k-grid 20,50,100,200,500 --format json --output migration-plan.json
+```
+
+For a deterministic, network-free walkthrough, run `examples/planner/run_demo.sh`.
+The demo uses the same planner API and tiny hash encoders; it is not a quality
+benchmark.
+
+## What the result means
+
+The result is a versioned (`schema_version: 1`) object with source/target
+contracts, preflight checks, evidence, candidate-depth diagnostics, cache and
+rollout guidance, economics, and structured warnings. The Python API is:
+
+```python
+from embedflow.planner import MigrationPlanner
+
+result = MigrationPlanner("embedflow.yaml").plan("probes.jsonl")
+artifact = result.to_dict()
+```
+
+Recommendation states are deliberately small:
+
+- `PROCEED`: probe evidence and preflight are strong enough for an operator-controlled shadow/canary sequence.
+- `PROCEED_WITH_CAUTION`: probes are acceptable, but evidence coverage or contract/ANN provenance is limited.
+- `EXPAND_PROBE`: T2-v1 asks for a deeper pool or the evidence is too small.
+- `DEFER`: finite-tail behavior is `UNSAFE_OR_UNCERTAIN`; do not canary based on this plan.
+- `BLOCKED`: structural/backend preflight failed; no candidate K is certified.
+
+These are recommendations, not automatic deployment actions. The suggested
+rollout is Validate → Shadow → small canary → expanded canary → target-primary
+path, with the source remaining authoritative until the operator approves each
+step. Stop conditions include source/target errors, p95 latency, queue depth,
+cache hit rate, and any native/qrels evaluation regressions.
+
+## Evidence and T2-v1
+
+Registry matching preserves the existing hierarchy: `EXACT REGISTRY MATCH`,
+`PRIOR EVIDENCE AVAILABLE` (same contracts, different/unknown corpus),
+`RELATED EVIDENCE ONLY`, and `NO REGISTRY MATCH`. Prior or related evidence can
+seed context but cannot override current probe results or certify a new corpus.
+
+The planner calls the existing frozen T2-v1 implementation unchanged. T2-v1 is
+a no-qrels finite-tail diagnostic of candidate behavior. `SAFE` does **not**
+prove a candidate gap, nDCG, recall, or zero quality loss. Without actual qrels
+or native-target evaluation, `candidate_gap` is `null`/`UNKNOWN`. ANN fidelity
+is also `UNKNOWN` unless an exact/reference comparison is supplied by the
+backend audit workflow.
+
+No probes is a valid preflight/economics mode, but T2 is `NOT_RUN`, confidence
+is `INSUFFICIENT`, and no K is presented as safe. Probe JSONL rows may use
+`query` (or the existing `text` alias) and an optional `id`; empty, malformed,
+or duplicate IDs are rejected. `--max-probes` and `--seed` provide deterministic
+bounded sampling. Candidate documents are deduplicated by canonical ID and
+encoded in bounded batches; `--max-target-encodes` can impose a hard cap.
+
+## Cache, performance, and economics
+
+Planning uses an ephemeral target-cache/state directory when it opens a runtime,
+so the normal serving cache is not polluted. It recommends conservative sync
+miss and background batch settings and defaults to traffic-driven progressive
+warming when no access trace is supplied. An access trace can be supplied with
+`--access-trace` as rows such as `{"document_id":"abc","count":192}` to
+model hot-document coverage.
+
+Performance values retain provenance (`measured`, `user_supplied`, `modeled`,
+`registry`, or `unknown`). `--profile` performs only a small local encode/search
+sample with one unmeasured warmup pass excluded; it is diagnostic and is not a
+formal benchmark. Economics accepts `--target-docs-per-second` and
+`--gpu-hourly-cost`; absent inputs remain `UNKNOWN`. Raw vector storage is
+`documents × target dimension × dtype bytes` and excludes ANN overhead,
+metadata, replicas, backups, and database overhead. Progressive percentages
+are materialization scenarios, not claims that a percentage is sufficient.
+
+## Configuration and privacy
+
+An optional `planner:` section mirrors the CLI options (`max_probes`, `seed`,
+`k_grid`, work caps, cache hints, throughput/cost inputs, and corpus identity).
+CLI values take precedence. `EMBEDFLOW_PLANNER_*` environment overrides follow
+the normal config mechanism. Probe text is never copied into plan artifacts or
+telemetry by default; only counts, IDs used for diagnostics, and aggregates are
+reported. Backend credentials are handled by the existing backend adapters and
+are redacted from errors and structured output.
+
+The planner does not provide qrel evaluation, traffic routing, rollback
+automation, distributed scheduling, ANN tuning, integrated-vectorizer contract
+verification, or production cost guarantees. Use `embedflow evaluate` with
+qrels/native target rankings when empirical retrieval-quality claims are
+required.