embedflow 0.5.0__tar.gz → 0.7.0__tar.gz

{embedflow-0.5.0 → embedflow-0.7.0}/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## v0.7.0 — Source-authoritative Shadow Mode
+
+- Added bounded, deterministic, source-authoritative Shadow Mode for observing
+  target reranking on sampled traffic without delaying or changing source
+  responses.
+- Added failure/timeout isolation, queue backpressure, optional asynchronous
+  target-cache materialization, target-coverage and ranking diagnostics, and
+  privacy-conscious SQLite telemetry.
+- Added `embedflow shadow report`, API/status integration, documentation, and a
+  deterministic offline demonstration. Shadow reports remain operational
+  diagnostics and do not claim qrel-based retrieval quality.
+
+## v0.6.0 — Migration planner
+
+- Added an advisory `embedflow plan` command and Python API that combine
+  source-index preflight, registry evidence, representative probes, frozen
+  T2-v1 diagnostics, candidate-depth selection, cache planning, economics, and
+  staged rollout guidance without routing traffic or mutating the source.
+- Added structured JSON/YAML plan artifacts with explicit warnings and
+  measured/user-supplied/modeled/unknown provenance.
+
 ## v0.5.0 — Weaviate backend
 
 - Added a read-only Weaviate v4 backend for existing externally-vectorized

{embedflow-0.5.0 → embedflow-0.7.0}/CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 title: "EmbedFlow: Upgrading Legacy Embeddings Without Full Upfront Re-Embedding"
 message: "If EmbedFlow contributes to your work, please cite this software release."
 type: software
-version: 0.5.0
+version: 0.7.0
 date-released: 2026-09-13
 repository-code: "https://github.com/arnsri33/embedflow"
 url: "https://github.com/arnsri33/embedflow"

{embedflow-0.5.0 → embedflow-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: embedflow
-Version: 0.5.0
+Version: 0.7.0
 Summary: Progressive embedding-model migration over existing vector indexes.
 Author: Arnav Srivastav
 License-Expression: AGPL-3.0-only
@@ -189,6 +189,41 @@ Selected core records (nDCG@10, `G(50)`):
 See the [registry documentation](https://github.com/arnsri33/embedflow/blob/main/docs/registry.md)
 for matching levels, contract fingerprints, and provenance.
 
+## Advisory migration plans
+
+Use representative query probes to produce a bounded migration recommendation
+before serving:
+
+```bash
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl --format json --output migration-plan.json
+```
+
+The planner reports preflight, registry evidence, frozen T2-v1 finite-tail
+behavior, candidate K, cache/economics projections, and staged rollout
+guidance. It never routes traffic or mutates the source index; `SAFE` is not a
+qrels-based retrieval-quality guarantee. See the [planner guide](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
+
+### Source-authoritative Shadow Mode
+
+Run the target migration path beside sampled traffic while always returning the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h --format json
+```
+
+Shadow work is asynchronous, bounded, privacy-conscious, and failure isolated.
+Reports contain operational and ranking-disagreement diagnostics—not qrel-based
+quality guarantees—and recommendations never route canary traffic. See the
+[Shadow Mode guide](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
 ## Research
 
 For candidate depth `K`, EmbedFlow measures:
@@ -229,6 +264,7 @@ See the [Weaviate guide](https://github.com/arnsri33/embedflow/blob/main/docs/in
 ```bash
 embedflow --help
 embedflow analyze --help
+embedflow plan --help
 embedflow serve --config ./embedflow.yaml
 embedflow status --config ./embedflow.yaml
 embedflow registry list
@@ -242,7 +278,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.5.0 is an alpha release for research and early real-world
+EmbedFlow v0.7.0 is a pre-1.0 release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.5.0 → embedflow-0.7.0}/README.md

@@ -142,6 +142,38 @@ session = embedflow.migrate(
 results = session.search("what causes auroras?", top_k=10)
 ```
 
+## Plan a migration
+
+Build a conservative, evidence-aware recommendation before serving. The
+planner reuses backend preflight, registry matching, and frozen T2-v1; it is
+advisory and never routes traffic or mutates the source index.
+
+```bash
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl --format json --output migration-plan.json
+```
+
+`SAFE` is an empirical finite-tail signal, not a retrieval-quality guarantee.
+See [`docs/planner.md`](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
+
+Observe the reviewed migration path on real traffic without changing the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h
+```
+
+Shadow Mode is source-authoritative, bounded, and advisory. It records cache,
+coverage, latency, and ranking-disagreement diagnostics; it does not claim
+retrieval-quality preservation without qrels and never routes canary traffic.
+See [`docs/shadow-mode.md`](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
 Search responses expose `COLD`, `PARTIAL`, or `WARM`, cache hits and misses,
 synchronous work, queued work, and stage timings. Once the candidate vectors
 are warm, target scoring over that candidate set is deterministic.
@@ -227,6 +259,7 @@ Backend-specific setup and examples:
 ```bash
 embedflow --help
 embedflow analyze --help
+embedflow plan --help
 embedflow serve --config ./embedflow.yaml
 embedflow status --config ./embedflow.yaml
 embedflow registry list
@@ -248,13 +281,14 @@ OpenAPI documentation; see
 - [CLI reference](https://github.com/arnsri33/embedflow/blob/main/docs/cli.md)
 - [API](https://github.com/arnsri33/embedflow/blob/main/docs/api.md)
 - [Economics](https://github.com/arnsri33/embedflow/blob/main/docs/economics.md)
+- [Shadow Mode](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md)
 - [Limitations](https://github.com/arnsri33/embedflow/blob/main/docs/limitations.md)
 - [Contributing](https://github.com/arnsri33/embedflow/blob/main/CONTRIBUTING.md)
 - [Security](https://github.com/arnsri33/embedflow/blob/main/SECURITY.md)
 
 ## Status
 
-EmbedFlow v0.5.0 is an alpha release for research and early real-world
+EmbedFlow v0.7.0 is a pre-1.0 release for research and early real-world
 testing.
 
 - T2-v1 reports an empirical finite-tail diagnostic.

{embedflow-0.5.0 → embedflow-0.7.0}/README_PYPI.md

@@ -123,6 +123,41 @@ Selected core records (nDCG@10, `G(50)`):
 See the [registry documentation](https://github.com/arnsri33/embedflow/blob/main/docs/registry.md)
 for matching levels, contract fingerprints, and provenance.
 
+## Advisory migration plans
+
+Use representative query probes to produce a bounded migration recommendation
+before serving:
+
+```bash
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl --format json --output migration-plan.json
+```
+
+The planner reports preflight, registry evidence, frozen T2-v1 finite-tail
+behavior, candidate K, cache/economics projections, and staged rollout
+guidance. It never routes traffic or mutates the source index; `SAFE` is not a
+qrels-based retrieval-quality guarantee. See the [planner guide](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
+
+### Source-authoritative Shadow Mode
+
+Run the target migration path beside sampled traffic while always returning the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h --format json
+```
+
+Shadow work is asynchronous, bounded, privacy-conscious, and failure isolated.
+Reports contain operational and ranking-disagreement diagnostics—not qrel-based
+quality guarantees—and recommendations never route canary traffic. See the
+[Shadow Mode guide](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
 ## Research
 
 For candidate depth `K`, EmbedFlow measures:
@@ -163,6 +198,7 @@ See the [Weaviate guide](https://github.com/arnsri33/embedflow/blob/main/docs/in
 ```bash
 embedflow --help
 embedflow analyze --help
+embedflow plan --help
 embedflow serve --config ./embedflow.yaml
 embedflow status --config ./embedflow.yaml
 embedflow registry list
@@ -176,7 +212,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.5.0 is an alpha release for research and early real-world
+EmbedFlow v0.7.0 is a pre-1.0 release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.5.0 → embedflow-0.7.0}/docs/api.md

@@ -21,6 +21,7 @@ Interactive OpenAPI documentation is available at
 | GET | `/metrics` | Aggregated latency and queue metrics |
 | GET | `/plan` | Current migration plan |
 | GET | `/economics` | Configured economics estimate |
+| GET | `/shadow/report` | Bounded Shadow Mode diagnostics |
 
 ## Search
 
@@ -48,6 +49,18 @@ The response includes the result list and migration fields such as:
 request. A partial response scores the available target vectors; it can differ
 from the fully warm ranking.
 
+When `runtime.mode=shadow`, `/search` returns the source-only result and marks
+`migration.source_authoritative=true`; target work is scheduled in the
+background. `/shadow/report`, `/status`, and `/metrics` expose aggregate shadow
+observations without raw query/document text or vectors. Shadow failures and
+timeouts are isolated from the response. Ranking overlap is diagnostic and is
+not a qrels-based quality claim.
+
+The advisory migration planner is exposed through the Python API and the
+`embedflow plan` CLI. It is intentionally not a synchronous FastAPI endpoint:
+probe analysis may load models and perform bounded candidate work, so operators
+should generate a plan artifact out of band and publish/read it as needed.
+
 `/status` includes safe backend metadata. For pgvector this names the
 schema/table and vector contract; for Pinecone it names the host/index,
 namespace, dimension, metric, and safe vector counts. Neither backend returns

{embedflow-0.5.0 → embedflow-0.7.0}/docs/cli.md

@@ -9,6 +9,8 @@ list. The commands below are the main public entry points.
 embedflow init --config ./embedflow.yaml
 embedflow analyze --config ./embedflow.yaml --output-dir ./analysis
 embedflow evaluate --config ./experiment.yaml --output-dir ./results
+embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow shadow report --config ./embedflow.yaml --since 24h
 ```
 
 `analyze` is the no-target-index workflow. It uses probe queries and frozen
@@ -16,6 +18,15 @@ T2-v1 logic. `evaluate` is the labelled workflow; it computes source quality,
 native target quality, target-within-source-candidates quality, candidate gap,
 containment, and migration depth when the required inputs are available.
 
+`plan` is the advisory migration planner. It runs source preflight and registry
+matching, optionally samples a JSONL probe set, and reports a candidate-depth
+recommendation, cache/economics projections, warnings, and staged rollout
+guidance. Use `--format json` (or `yaml`) for a structured artifact. A plan
+with no probes reports `T2-v1: NOT_RUN`; `SAFE` is a finite-tail diagnostic,
+not a qrels-based retrieval-quality guarantee. A `DEFER`/`EXPAND_PROBE` result
+is still a successful analytical command and exits zero; invalid configuration
+or failed preflight exits nonzero.
+
 ## Serving and operations
 
 ```bash
@@ -32,6 +43,12 @@ materialization throughput. `audit-index` checks the source index against an
 exact/reference configuration where supported. `prewarm` schedules target
 document work; it does not change source-index results.
 
+Use `embedflow serve --mode shadow` to opt into source-authoritative Shadow
+Mode for one process. `embedflow shadow report` reads the bounded telemetry
+store and supports `--format text|json|yaml`, `--since`, `--output`, and
+`--quiet`. A valid `DEFER`/`EXPAND_K` report is still an analytical success;
+only invalid configuration or initialization returns a non-zero exit code.
+
 ## Registry and profiles
 
 ```bash

{embedflow-0.5.0 → embedflow-0.7.0}/docs/configuration.md

@@ -58,9 +58,57 @@ economics:
   gpu_price_per_hour: null
   target_docs_per_second: null
 
+# Optional advisory planner settings. CLI flags override these values.
+planner:
+  max_probes: 250
+  seed: 42
+  k_grid: [20, 50, 100, 200, 500]
+  max_candidates: null
+  max_target_encodes: null
+  max_sync_misses: null
+  background_batch_size: null
+  gpu_hourly_cost: null
+  target_docs_per_second: null
+  queries_per_second: null
+  daily_queries: null
+  cache_hit_rate: null
+  latency_budget_ms: null
+  access_trace: null
+  corpus_name: null
+  corpus_fingerprint: null
+
+# Source-authoritative observation mode.  ``runtime.mode: migration`` (the
+# default) leaves Shadow Mode inactive.  ``mode: shadow`` is an explicit opt-in.
+runtime:
+  mode: migration             # migration, normal, source, or shadow
+
+shadow:
+  enabled: true
+  sample_rate: 0.10
+  sample_seed: 42
+  candidate_k: 100
+  materialize: true
+  max_inflight: 32
+  queue_capacity: 1000
+  timeout_ms: 10000
+  shutdown_grace_ms: 1000
+  telemetry:
+    enabled: true
+    path: ./.embedflow/shadow.sqlite3
+    retain_query_records: false
+    retain_query_text: false
+    max_records: 10000
+    retention_days: null
+    report_k: 10
+    min_target_coverage_for_ranking: 1.0
+
 state_path: ./embedflow_state.json
 ```
 
+Shadow Mode always returns the source-authoritative result before target work
+finishes. See [`shadow-mode.md`](shadow-mode.md) for queue, timeout, privacy,
+materialization, and report semantics.
+
 ## Model contracts
 
 Model configuration can include `revision`, `dimension`, `max_length`,
@@ -130,6 +178,39 @@ EMBEDFLOW_CANDIDATE_DEPTH
 EMBEDFLOW_MAX_SYNC_MISSES
 EMBEDFLOW_BACKGROUND_BATCH_SIZE
 EMBEDFLOW_PROBE_KMAX
+EMBEDFLOW_PLANNER_MAX_PROBES
+EMBEDFLOW_PLANNER_SEED
+EMBEDFLOW_PLANNER_MAX_CANDIDATES
+EMBEDFLOW_PLANNER_MAX_TARGET_ENCODINGS
+EMBEDFLOW_PLANNER_MAX_SYNC_MISSES
+EMBEDFLOW_PLANNER_BACKGROUND_BATCH_SIZE
+EMBEDFLOW_PLANNER_GPU_HOURLY_COST
+EMBEDFLOW_PLANNER_TARGET_DOCS_PER_SECOND
+EMBEDFLOW_PLANNER_QPS
+EMBEDFLOW_PLANNER_DAILY_QUERIES
+EMBEDFLOW_PLANNER_CACHE_HIT_RATE
+EMBEDFLOW_PLANNER_LATENCY_BUDGET_MS
+EMBEDFLOW_PLANNER_ACCESS_TRACE
+EMBEDFLOW_PLANNER_CORPUS_NAME
+EMBEDFLOW_PLANNER_CORPUS_FINGERPRINT
+EMBEDFLOW_RUNTIME_MODE
+EMBEDFLOW_SHADOW_ENABLED
+EMBEDFLOW_SHADOW_SAMPLE_RATE
+EMBEDFLOW_SHADOW_SAMPLE_SEED
+EMBEDFLOW_SHADOW_CANDIDATE_K
+EMBEDFLOW_SHADOW_MATERIALIZE
+EMBEDFLOW_SHADOW_MAX_INFLIGHT
+EMBEDFLOW_SHADOW_QUEUE_CAPACITY
+EMBEDFLOW_SHADOW_TIMEOUT_MS
+EMBEDFLOW_SHADOW_SHUTDOWN_GRACE_MS
+EMBEDFLOW_SHADOW_TELEMETRY_ENABLED
+EMBEDFLOW_SHADOW_TELEMETRY_PATH
+EMBEDFLOW_SHADOW_RETAIN_QUERY_RECORDS
+EMBEDFLOW_SHADOW_RETAIN_QUERY_TEXT
+EMBEDFLOW_SHADOW_MAX_RECORDS
+EMBEDFLOW_SHADOW_REPORT_K
+EMBEDFLOW_SHADOW_RETENTION_DAYS
+EMBEDFLOW_SHADOW_MIN_TARGET_COVERAGE
 ```
 
 ## Input files

{embedflow-0.5.0 → embedflow-0.7.0}/docs/limitations.md

@@ -1,6 +1,6 @@
 # Limitations and release scope
 
-EmbedFlow v0.5.0 is an alpha release for research and early real-world
+EmbedFlow v0.7.0 is a pre-1.0 release for research and early real-world
 testing. The serving path is designed to make migration experiments concrete;
 production rollout still requires application-specific validation.
 

embedflow-0.7.0/docs/planner.md

@@ -0,0 +1,111 @@
+# Migration planner
+
+`embedflow plan` is a bounded, advisory analysis for moving from an existing
+source vector index to a target embedding model. It composes the existing
+backend audit, model contracts, retained registry evidence, probe retrieval,
+frozen T2-v1 diagnostic, cache behavior, and economics helpers. It never
+routes traffic and does not create, update, or delete source-index data.
+
+## Quickstart
+
+Install EmbedFlow with the optional dependency for the configured backend, then
+provide representative query probes as JSONL:
+
+```json
+{"id":"q1","query":"how do I reset my password?"}
+{"id":"q2","query":"what is the vacation policy?"}
+```
+
+Run a terminal plan or save a machine-readable artifact:
+
+```bash
+embedflow plan --config embedflow.yaml --queries probes.jsonl
+embedflow plan --config embedflow.yaml --queries probes.jsonl \
+  --k-grid 20,50,100,200,500 --format json --output migration-plan.json
+```
+
+For a deterministic, network-free walkthrough, run `examples/planner/run_demo.sh`.
+The demo uses the same planner API and tiny hash encoders; it is not a quality
+benchmark.
+
+## What the result means
+
+The result is a versioned (`schema_version: 1`) object with source/target
+contracts, preflight checks, evidence, candidate-depth diagnostics, cache and
+rollout guidance, economics, and structured warnings. The Python API is:
+
+```python
+from embedflow.planner import MigrationPlanner
+
+result = MigrationPlanner("embedflow.yaml").plan("probes.jsonl")
+artifact = result.to_dict()
+```
+
+Recommendation states are deliberately small:
+
+- `PROCEED`: probe evidence and preflight are strong enough for an operator-controlled shadow/canary sequence.
+- `PROCEED_WITH_CAUTION`: probes are acceptable, but evidence coverage or contract/ANN provenance is limited.
+- `EXPAND_PROBE`: T2-v1 asks for a deeper pool or the evidence is too small.
+- `DEFER`: finite-tail behavior is `UNSAFE_OR_UNCERTAIN`; do not canary based on this plan.
+- `BLOCKED`: structural/backend preflight failed; no candidate K is certified.
+
+These are recommendations, not automatic deployment actions. The suggested
+rollout is Validate → Shadow → small canary → expanded canary → target-primary
+path, with the source remaining authoritative until the operator approves each
+step. Stop conditions include source/target errors, p95 latency, queue depth,
+cache hit rate, and any native/qrels evaluation regressions.
+
+## Evidence and T2-v1
+
+Registry matching preserves the existing hierarchy: `EXACT REGISTRY MATCH`,
+`PRIOR EVIDENCE AVAILABLE` (same contracts, different/unknown corpus),
+`RELATED EVIDENCE ONLY`, and `NO REGISTRY MATCH`. Prior or related evidence can
+seed context but cannot override current probe results or certify a new corpus.
+
+The planner calls the existing frozen T2-v1 implementation unchanged. T2-v1 is
+a no-qrels finite-tail diagnostic of candidate behavior. `SAFE` does **not**
+prove a candidate gap, nDCG, recall, or zero quality loss. Without actual qrels
+or native-target evaluation, `candidate_gap` is `null`/`UNKNOWN`. ANN fidelity
+is also `UNKNOWN` unless an exact/reference comparison is supplied by the
+backend audit workflow.
+
+No probes is a valid preflight/economics mode, but T2 is `NOT_RUN`, confidence
+is `INSUFFICIENT`, and no K is presented as safe. Probe JSONL rows may use
+`query` (or the existing `text` alias) and an optional `id`; empty, malformed,
+or duplicate IDs are rejected. `--max-probes` and `--seed` provide deterministic
+bounded sampling. Candidate documents are deduplicated by canonical ID and
+encoded in bounded batches; `--max-target-encodes` can impose a hard cap.
+
+## Cache, performance, and economics
+
+Planning uses an ephemeral target-cache/state directory when it opens a runtime,
+so the normal serving cache is not polluted. It recommends conservative sync
+miss and background batch settings and defaults to traffic-driven progressive
+warming when no access trace is supplied. An access trace can be supplied with
+`--access-trace` as rows such as `{"document_id":"abc","count":192}` to
+model hot-document coverage.
+
+Performance values retain provenance (`measured`, `user_supplied`, `modeled`,
+`registry`, or `unknown`). `--profile` performs only a small local encode/search
+sample with one unmeasured warmup pass excluded; it is diagnostic and is not a
+formal benchmark. Economics accepts `--target-docs-per-second` and
+`--gpu-hourly-cost`; absent inputs remain `UNKNOWN`. Raw vector storage is
+`documents × target dimension × dtype bytes` and excludes ANN overhead,
+metadata, replicas, backups, and database overhead. Progressive percentages
+are materialization scenarios, not claims that a percentage is sufficient.
+
+## Configuration and privacy
+
+An optional `planner:` section mirrors the CLI options (`max_probes`, `seed`,
+`k_grid`, work caps, cache hints, throughput/cost inputs, and corpus identity).
+CLI values take precedence. `EMBEDFLOW_PLANNER_*` environment overrides follow
+the normal config mechanism. Probe text is never copied into plan artifacts or
+telemetry by default; only counts, IDs used for diagnostics, and aggregates are
+reported. Backend credentials are handled by the existing backend adapters and
+are redacted from errors and structured output.
+
+The planner does not provide qrel evaluation, traffic routing, rollback
+automation, distributed scheduling, ANN tuning, integrated-vectorizer contract
+verification, or production cost guarantees. Use `embedflow evaluate` with
+qrels/native target rankings when empirical retrieval-quality claims are
+required.

{embedflow-0.5.0 → embedflow-0.7.0}/docs/releasing.md

@@ -18,8 +18,8 @@ python -m twine check dist/*
 Inspect both archives before uploading:
 
 ```bash
-unzip -l dist/embedflow-0.5.0-py3-none-any.whl
-tar -tzf dist/embedflow-0.5.0.tar.gz
+unzip -l dist/embedflow-0.7.0-py3-none-any.whl
+tar -tzf dist/embedflow-0.7.0.tar.gz
 sha256sum dist/*
 ```
 
@@ -32,7 +32,7 @@ Test the wheel outside the source tree:
 ```bash
 python -m venv /tmp/embedflow-wheel-test
 /tmp/embedflow-wheel-test/bin/python -m pip install --upgrade pip
-/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.5.0-py3-none-any.whl
+/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.7.0-py3-none-any.whl
 cd /tmp
 /tmp/embedflow-wheel-test/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-wheel-test/bin/embedflow --help
@@ -65,7 +65,7 @@ python -m venv /tmp/embedflow-testpypi
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  embedflow==0.5.0
+  embedflow==0.7.0
 cd /tmp
 /tmp/embedflow-testpypi/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-testpypi/bin/embedflow --help
@@ -79,11 +79,11 @@ Test optional integrations in a second clean environment:
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  "embedflow[faiss,dashboard,pinecone,milvus,weaviate]==0.5.0"
+  "embedflow[faiss,dashboard,pinecone,milvus,weaviate]==0.7.0"
 ```
 
-If the same filename already exists on TestPyPI, use a pre-release such as
-`0.5.0rc1` for the TestPyPI-only trial. Keep production `0.5.0` unchanged.
+If the same filename already exists on TestPyPI, use a pre-release for the
+TestPyPI-only trial. Never overwrite the production version.
 
 ## Trusted Publishing configuration
 
@@ -115,7 +115,7 @@ above keeps the test step explicit.
 3. Run the final release gate and review the generated report.
 4. Configure the PyPI pending publisher and protected `pypi` environment.
 5. Create a Git tag and GitHub Release for the exact package version, for
-   example `v0.5.0`.
+   example `v0.7.0`.
 6. Approve the `pypi` environment when the release workflow is ready.
 7. Verify the files and metadata on PyPI.
 8. Install from production PyPI in a directory outside this checkout.

embedflow-0.7.0/docs/shadow-mode.md

@@ -0,0 +1,104 @@
+# Shadow Mode
+
+Shadow Mode runs the configured target migration path beside real traffic while
+the existing source result remains authoritative. It is an observation and
+cache-warming tool, not a traffic router:
+
+```yaml
+runtime:
+  mode: shadow
+
+shadow:
+  enabled: true
+  sample_rate: 0.10
+  sample_seed: 42
+  candidate_k: 100
+  materialize: true
+  max_inflight: 32
+  queue_capacity: 1000
+  timeout_ms: 10000
+  telemetry:
+    enabled: true
+    path: ./.embedflow/shadow.sqlite3
+    retain_query_records: false
+    retain_query_text: false
+```
+
+Install the optional backend/model extras required by the selected
+configuration, then start the ordinary service:
+
+```bash
+pip install "embedflow[dashboard]"
+embedflow serve --config embedflow.yaml
+# or override the file for one process:
+embedflow serve --config embedflow.yaml --mode shadow
+```
+
+The request path performs source encoding, source ANN retrieval, and document
+resolution exactly as source-only serving. It returns that result without
+waiting for target encoding, cache misses, reranking, telemetry, or the
+materialization worker. Shadow jobs use a bounded queue and worker pool. A full
+queue drops only shadow work; a target/model/cache/telemetry failure or timeout
+is recorded and cannot change the primary response.
+
+If the target model cannot be loaded during startup, source/shadow serving
+still opens with target work marked unavailable; sampled requests record the
+isolated target-encoding failure. Normal migration mode remains fail-fast for
+the same startup error.
+
+`candidate_k` is explicit. The planner may suggest a value, but Shadow Mode
+does not silently select one. `materialize: false` reads already cached target
+vectors without changing cache or queue state. With `materialize: true`, cold
+candidate IDs are deduplicated in the existing persistent materialization queue
+and warmed asynchronously; synchronous target misses remain zero in Shadow
+Mode. A comparison with incomplete target vectors is reported as `partial`,
+with target coverage shown separately.
+
+## Reports
+
+Reports use the privacy-preserving SQLite telemetry store (by default beside
+the configured cache):
+
+```bash
+embedflow shadow report --config embedflow.yaml --since 24h
+embedflow shadow report --config embedflow.yaml --since 24h --format json --output shadow-report.json
+```
+
+Use `/shadow/report` or the `shadow` section of `/status` and `/metrics` in the
+FastAPI service. JSON/YAML stdout contains only the requested artifact; progress
+and errors belong on stderr. `--since` accepts seconds or `s`, `m`, `h`, `d`,
+and `w` suffixes.
+
+Reports contain sampled/completed/partial/failed/timed-out/dropped counts,
+cache and materialization accounting, source and shadow latency distributions,
+target coverage, top-1 agreement, and top-k overlap. Shadow latency is not
+user-facing latency because it is off the primary critical path. Ranking
+disagreement and overlap are diagnostics, not recall, nDCG, or quality loss;
+without qrels/native-target evaluation no quality guarantee is made.
+
+T2-v1 remains the frozen window-level diagnostic. EmbedFlow does not label an
+individual production query `T2 SAFE`. A report recommendation is operational
+guidance only:
+
+- `CONTINUE_SHADOW` means evidence or coverage is still limited.
+- `EXPAND_K` means the accumulated T2 window requests a larger candidate pool.
+- `INVESTIGATE` means failures, timeouts, or uncertain T2 behavior need review.
+- `READY_FOR_CANARY_EVALUATION` means an operator may consider a separately
+  designed canary; it never routes traffic automatically.
+
+Raw query text, candidate text, source vectors, target vectors, and credentials
+are not persisted in telemetry. Query IDs may be retained only when explicitly
+enabled, and retention is bounded by `max_records`. The telemetry database is
+segmented by source/target contract and candidate-K fingerprint so unrelated
+migrations are not mixed.
+
+On shutdown EmbedFlow stops accepting new shadow work, drops queued jobs when
+necessary, and waits only the configured bounded grace period. A corrupted or
+unwritable telemetry file degrades observability; it does not take source
+serving down. Source indexes are read-only during Shadow Mode. The planner and
+Shadow Mode are separate: generate a plan first, then copy its recommended K
+explicitly into a reviewed Shadow configuration.
+
+Shadow Mode is advisory and experimental operational instrumentation. It does
+not provide autonomous rollout, rollback, qrel evaluation, or ANN-fidelity
+proof.