embedflow 0.6.0__tar.gz → 0.8.0__tar.gz

{embedflow-0.6.0 → embedflow-0.8.0}/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## v0.8.0 — Traffic-aware target-vector prewarming
+
+- Added bounded `prewarm plan`, `prewarm run`, and `prewarm status` workflows
+  that select uncached target documents by observed Shadow Mode candidate
+  occurrences.
+- Added privacy-safe aggregate document popularity telemetry, content/model
+  fingerprint checks, deterministic tie-breaking, hard document/storage/time
+  budgets, atomic plan/run artifacts, and resumable materialization through the
+  existing target cache and worker.
+- Observed candidate-occurrence coverage remains an operational warming signal;
+  it is not retrieval quality or recall evidence.
+
+## v0.7.0 — Source-authoritative Shadow Mode
+
+- Added bounded, deterministic, source-authoritative Shadow Mode for observing
+  target reranking on sampled traffic without delaying or changing source
+  responses.
+- Added failure/timeout isolation, queue backpressure, optional asynchronous
+  target-cache materialization, target-coverage and ranking diagnostics, and
+  privacy-conscious SQLite telemetry.
+- Added `embedflow shadow report`, API/status integration, documentation, and a
+  deterministic offline demonstration. Shadow reports remain operational
+  diagnostics and do not claim qrel-based retrieval quality.
+
 ## v0.6.0 — Migration planner
 
 - Added an advisory `embedflow plan` command and Python API that combine

{embedflow-0.6.0 → embedflow-0.8.0}/CITATION.cff

@@ -2,8 +2,8 @@ cff-version: 1.2.0
 title: "EmbedFlow: Upgrading Legacy Embeddings Without Full Upfront Re-Embedding"
 message: "If EmbedFlow contributes to your work, please cite this software release."
 type: software
-version: 0.6.0
-date-released: 2026-09-13
+version: 0.8.0
+date-released: 2026-09-15
 repository-code: "https://github.com/arnsri33/embedflow"
 url: "https://github.com/arnsri33/embedflow"
 license: AGPL-3.0-only

{embedflow-0.6.0 → embedflow-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: embedflow
-Version: 0.6.0
+Version: 0.8.0
 Summary: Progressive embedding-model migration over existing vector indexes.
 Author: Arnav Srivastav
 License-Expression: AGPL-3.0-only
@@ -204,6 +204,37 @@ behavior, candidate K, cache/economics projections, and staged rollout
 guidance. It never routes traffic or mutates the source index; `SAFE` is not a
 qrels-based retrieval-quality guarantee. See the [planner guide](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
 
+### Source-authoritative Shadow Mode
+
+Run the target migration path beside sampled traffic while always returning the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h --format json
+```
+
+Shadow work is asynchronous, bounded, privacy-conscious, and failure isolated.
+Reports contain operational and ranking-disagreement diagnostics—not qrel-based
+quality guarantees—and recommendations never route canary traffic. See the
+[Shadow Mode guide](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
+After collecting Shadow traffic, prioritize uncached target vectors by observed
+candidate popularity:
+
+```bash
+embedflow prewarm plan --config ./embedflow.yaml --since 24h --max-docs 50000 --output prewarm-plan.json
+embedflow prewarm run --config ./embedflow.yaml --plan prewarm-plan.json
+```
+
+Observed candidate-occurrence coverage is an operational warming signal, not
+retrieval quality or recall. See the [prewarming guide](https://github.com/arnsri33/embedflow/blob/main/docs/prewarming.md).
+
 ## Research
 
 For candidate depth `K`, EmbedFlow measures:
@@ -258,7 +289,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
+EmbedFlow v0.8.0 is a pre-1.0 release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.6.0 → embedflow-0.8.0}/README.md

@@ -156,6 +156,35 @@ embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl --forma
 `SAFE` is an empirical finite-tail signal, not a retrieval-quality guarantee.
 See [`docs/planner.md`](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
 
+Observe the reviewed migration path on real traffic without changing the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h
+```
+
+Shadow Mode is source-authoritative, bounded, and advisory. It records cache,
+coverage, latency, and ranking-disagreement diagnostics; it does not claim
+retrieval-quality preservation without qrels and never routes canary traffic.
+See [`docs/shadow-mode.md`](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
+After collecting Shadow traffic, prioritize uncached target vectors by observed
+candidate popularity:
+
+```bash
+embedflow prewarm plan --config ./embedflow.yaml --since 24h --max-docs 50000 --output prewarm-plan.json
+embedflow prewarm run --config ./embedflow.yaml --plan prewarm-plan.json
+```
+
+Observed candidate-occurrence coverage is an operational warming signal, not
+retrieval quality or recall. See [`docs/prewarming.md`](https://github.com/arnsri33/embedflow/blob/main/docs/prewarming.md).
+
 Search responses expose `COLD`, `PARTIAL`, or `WARM`, cache hits and misses,
 synchronous work, queued work, and stage timings. Once the candidate vectors
 are warm, target scoring over that candidate set is deterministic.
@@ -263,13 +292,14 @@ OpenAPI documentation; see
 - [CLI reference](https://github.com/arnsri33/embedflow/blob/main/docs/cli.md)
 - [API](https://github.com/arnsri33/embedflow/blob/main/docs/api.md)
 - [Economics](https://github.com/arnsri33/embedflow/blob/main/docs/economics.md)
+- [Shadow Mode](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md)
 - [Limitations](https://github.com/arnsri33/embedflow/blob/main/docs/limitations.md)
 - [Contributing](https://github.com/arnsri33/embedflow/blob/main/CONTRIBUTING.md)
 - [Security](https://github.com/arnsri33/embedflow/blob/main/SECURITY.md)
 
 ## Status
 
-EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
+EmbedFlow v0.8.0 is a pre-1.0 release for research and early real-world
 testing.
 
 - T2-v1 reports an empirical finite-tail diagnostic.

{embedflow-0.6.0 → embedflow-0.8.0}/README_PYPI.md

@@ -138,6 +138,37 @@ behavior, candidate K, cache/economics projections, and staged rollout
 guidance. It never routes traffic or mutates the source index; `SAFE` is not a
 qrels-based retrieval-quality guarantee. See the [planner guide](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
 
+### Source-authoritative Shadow Mode
+
+Run the target migration path beside sampled traffic while always returning the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h --format json
+```
+
+Shadow work is asynchronous, bounded, privacy-conscious, and failure isolated.
+Reports contain operational and ranking-disagreement diagnostics—not qrel-based
+quality guarantees—and recommendations never route canary traffic. See the
+[Shadow Mode guide](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
+After collecting Shadow traffic, prioritize uncached target vectors by observed
+candidate popularity:
+
+```bash
+embedflow prewarm plan --config ./embedflow.yaml --since 24h --max-docs 50000 --output prewarm-plan.json
+embedflow prewarm run --config ./embedflow.yaml --plan prewarm-plan.json
+```
+
+Observed candidate-occurrence coverage is an operational warming signal, not
+retrieval quality or recall. See the [prewarming guide](https://github.com/arnsri33/embedflow/blob/main/docs/prewarming.md).
+
 ## Research
 
 For candidate depth `K`, EmbedFlow measures:
@@ -192,7 +223,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
+EmbedFlow v0.8.0 is a pre-1.0 release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.6.0 → embedflow-0.8.0}/docs/api.md

@@ -21,6 +21,7 @@ Interactive OpenAPI documentation is available at
 | GET | `/metrics` | Aggregated latency and queue metrics |
 | GET | `/plan` | Current migration plan |
 | GET | `/economics` | Configured economics estimate |
+| GET | `/shadow/report` | Bounded Shadow Mode diagnostics |
 
 ## Search
 
@@ -48,6 +49,13 @@ The response includes the result list and migration fields such as:
 request. A partial response scores the available target vectors; it can differ
 from the fully warm ranking.
 
+When `runtime.mode=shadow`, `/search` returns the source-only result and marks
+`migration.source_authoritative=true`; target work is scheduled in the
+background. `/shadow/report`, `/status`, and `/metrics` expose aggregate shadow
+observations without raw query/document text or vectors. Shadow failures and
+timeouts are isolated from the response. Ranking overlap is diagnostic and is
+not a qrels-based quality claim.
+
 The advisory migration planner is exposed through the Python API and the
 `embedflow plan` CLI. It is intentionally not a synchronous FastAPI endpoint:
 probe analysis may load models and perform bounded candidate work, so operators

{embedflow-0.6.0 → embedflow-0.8.0}/docs/cli.md

@@ -10,6 +10,10 @@ embedflow init --config ./embedflow.yaml
 embedflow analyze --config ./embedflow.yaml --output-dir ./analysis
 embedflow evaluate --config ./experiment.yaml --output-dir ./results
 embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow shadow report --config ./embedflow.yaml --since 24h
+embedflow prewarm plan --config ./embedflow.yaml --since 24h --max-docs 50000 --output prewarm-plan.json
+embedflow prewarm run --config ./embedflow.yaml --plan prewarm-plan.json
+embedflow prewarm status --config ./embedflow.yaml
 ```
 
 `analyze` is the no-target-index workflow. It uses probe queries and frozen
@@ -40,7 +44,19 @@ embedflow export-target --config ./embedflow.yaml --output-index ./target.index
 `status` reports cache coverage, hit/miss counters, queue depth, and
 materialization throughput. `audit-index` checks the source index against an
 exact/reference configuration where supported. `prewarm` schedules target
-document work; it does not change source-index results.
+document work; it does not change source-index results. `prewarm plan` ranks
+uncached IDs by privacy-safe Shadow candidate occurrence counts and is always
+bounded by `--max-docs` (or the configured `prewarm.max_docs`). `prewarm run`
+refuses mismatched/tampered fingerprints, skips vectors that became warm, and
+uses the existing durable materializer. Observed candidate-occurrence coverage
+is not retrieval quality or recall. Use `--docs-per-second` and
+`--gpu-hourly-cost` only for explicitly labeled modeled economics.
+
+Use `embedflow serve --mode shadow` to opt into source-authoritative Shadow
+Mode for one process. `embedflow shadow report` reads the bounded telemetry
+store and supports `--format text|json|yaml`, `--since`, `--output`, and
+`--quiet`. A valid `DEFER`/`EXPAND_K` report is still an analytical success;
+only invalid configuration or initialization returns a non-zero exit code.
 
 ## Registry and profiles
 

{embedflow-0.6.0 → embedflow-0.8.0}/docs/configuration.md

@@ -77,9 +77,50 @@ planner:
   corpus_name: null
   corpus_fingerprint: null
 
+# Optional bounded traffic-aware target-cache prewarming defaults. CLI flags
+# override these values for one invocation; there is no implicit full-corpus
+# prewarm operation.
+prewarm:
+  max_docs: 1000
+  strategy: traffic_hotset
+  target_observed_coverage: null
+  max_storage_gb: null
+  max_runtime_seconds: null
+  batch_size: null
+  max_retries: null
+
+# Source-authoritative observation mode.  ``runtime.mode: migration`` (the
+# default) leaves Shadow Mode inactive.  ``mode: shadow`` is an explicit opt-in.
+runtime:
+  mode: migration             # migration, normal, source, or shadow
+
+shadow:
+  enabled: true
+  sample_rate: 0.10
+  sample_seed: 42
+  candidate_k: 100
+  materialize: true
+  max_inflight: 32
+  queue_capacity: 1000
+  timeout_ms: 10000
+  shutdown_grace_ms: 1000
+  telemetry:
+    enabled: true
+    path: ./.embedflow/shadow.sqlite3
+    retain_query_records: false
+    retain_query_text: false
+    max_records: 10000
+    retention_days: null
+    report_k: 10
+    min_target_coverage_for_ranking: 1.0
+
 state_path: ./embedflow_state.json
 ```
 
+Shadow Mode always returns the source-authoritative result before target work
+finishes. See [`shadow-mode.md`](shadow-mode.md) for queue, timeout, privacy,
+materialization, and report semantics.
+
 ## Model contracts
 
 Model configuration can include `revision`, `dimension`, `max_length`,
@@ -164,6 +205,24 @@ EMBEDFLOW_PLANNER_LATENCY_BUDGET_MS
 EMBEDFLOW_PLANNER_ACCESS_TRACE
 EMBEDFLOW_PLANNER_CORPUS_NAME
 EMBEDFLOW_PLANNER_CORPUS_FINGERPRINT
+EMBEDFLOW_RUNTIME_MODE
+EMBEDFLOW_SHADOW_ENABLED
+EMBEDFLOW_SHADOW_SAMPLE_RATE
+EMBEDFLOW_SHADOW_SAMPLE_SEED
+EMBEDFLOW_SHADOW_CANDIDATE_K
+EMBEDFLOW_SHADOW_MATERIALIZE
+EMBEDFLOW_SHADOW_MAX_INFLIGHT
+EMBEDFLOW_SHADOW_QUEUE_CAPACITY
+EMBEDFLOW_SHADOW_TIMEOUT_MS
+EMBEDFLOW_SHADOW_SHUTDOWN_GRACE_MS
+EMBEDFLOW_SHADOW_TELEMETRY_ENABLED
+EMBEDFLOW_SHADOW_TELEMETRY_PATH
+EMBEDFLOW_SHADOW_RETAIN_QUERY_RECORDS
+EMBEDFLOW_SHADOW_RETAIN_QUERY_TEXT
+EMBEDFLOW_SHADOW_MAX_RECORDS
+EMBEDFLOW_SHADOW_REPORT_K
+EMBEDFLOW_SHADOW_RETENTION_DAYS
+EMBEDFLOW_SHADOW_MIN_TARGET_COVERAGE
 ```
 
 ## Input files

{embedflow-0.6.0 → embedflow-0.8.0}/docs/limitations.md

@@ -1,6 +1,6 @@
 # Limitations and release scope
 
-EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
+EmbedFlow v0.8.0 is a pre-1.0 release for research and early real-world
 testing. The serving path is designed to make migration experiments concrete;
 production rollout still requires application-specific validation.
 

embedflow-0.8.0/docs/prewarming.md

@@ -0,0 +1,84 @@
+# Traffic-aware prewarming
+
+Traffic-aware prewarming uses privacy-safe Shadow Mode aggregates to select the
+uncached target documents that occurred most often in the source candidate
+pool. It is a bounded operational warming tool in the existing migration
+workflow:
+
+```text
+PLAN -> SHADOW -> PREWARM -> CANARY -> MIGRATE
+```
+
+It never writes the source index. `prewarm plan` only reads Shadow telemetry
+and the target cache. `prewarm run` delegates document lookup, target encoding,
+and cache writes to the existing materialization worker.
+
+## Plan and run
+
+```bash
+embedflow prewarm plan \
+  --config embedflow.yaml --since 24h --max-docs 50000 \
+  --output prewarm-plan.json
+embedflow prewarm run --config embedflow.yaml --plan prewarm-plan.json
+embedflow prewarm status --config embedflow.yaml
+```
+
+Plans are schema-versioned (`schema_version: 1`), deterministically ranked by
+`candidate_occurrences DESC, document_id ASC`, and fingerprinted with the
+source/target contracts, source-index identity, candidate K, and Shadow
+configuration. Execution refuses a stale or tampered plan. A plan is always
+bounded by `max_docs`; there is no implicit full-corpus operation.
+
+Optional configuration defaults are available under `prewarm`:
+
+```yaml
+prewarm:
+  max_docs: 1000
+  strategy: traffic_hotset
+  batch_size: 32
+  max_retries: 3
+```
+
+`--target-observed-coverage`, `--max-storage-gb`, and `--max-runtime` (when a
+real measured/user-supplied `--docs-per-second` is supplied) further reduce
+the hard selection cap. `--gpu-hourly-cost` (or the existing planner/economics
+configuration value) adds a user-supplied modeled cost. Runtime, cost, and
+storage values are explicitly marked as modeled/unknown; no cloud price or
+throughput is guessed.
+
+## Coverage and privacy
+
+The plan reports **observed candidate-occurrence coverage**:
+
+```text
+occurrences with a valid target vector / all source candidate occurrences
+```
+
+This is not retrieval coverage, recall, nDCG, or a guarantee of migration
+quality. It only describes the selected Shadow window and candidate K. Ties are
+stable, and current target-cache entries count as warm only when their target
+model and (when document text is available) content fingerprints match. Legacy
+cache rows created before content bindings were introduced remain usable but
+are treated as content-unknown.
+
+Shadow aggregation stores document IDs and counters, not raw query text,
+document text, vectors, or credentials. Treat the generated plan/ID list as
+application data because selected document IDs are needed to execute it.
+
+## Resume and idempotence
+
+The runner stores a plan-specific queue/state beside the target cache and uses
+the target cache as the execution source of truth. Re-running a completed plan
+skips already-warm vectors. Interrupts leave completed vectors valid and a
+subsequent run can continue the remaining IDs. Documents that disappear or
+change are resolved again at run time; stale content is not silently reused.
+
+Two concurrent runners use the existing durable queue/cache deduplication. The
+queue is at-least-once under process crashes, so operators should inspect
+`prewarm status` and the run report for failures.
+
+The source FAISS/Qdrant/pgvector/Pinecone/Milvus/Weaviate index is never
+created, altered, rebuilt, upserted, or deleted by this feature. The API for
+prewarming is currently Python/core plus CLI; dashboard execution controls are
+intentionally not added to avoid turning an advisory plan into autonomous
+traffic routing.

{embedflow-0.6.0 → embedflow-0.8.0}/docs/releasing.md

@@ -18,8 +18,8 @@ python -m twine check dist/*
 Inspect both archives before uploading:
 
 ```bash
-unzip -l dist/embedflow-0.6.0-py3-none-any.whl
-tar -tzf dist/embedflow-0.6.0.tar.gz
+unzip -l dist/embedflow-0.8.0-py3-none-any.whl
+tar -tzf dist/embedflow-0.8.0.tar.gz
 sha256sum dist/*
 ```
 
@@ -32,7 +32,7 @@ Test the wheel outside the source tree:
 ```bash
 python -m venv /tmp/embedflow-wheel-test
 /tmp/embedflow-wheel-test/bin/python -m pip install --upgrade pip
-/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.6.0-py3-none-any.whl
+/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.8.0-py3-none-any.whl
 cd /tmp
 /tmp/embedflow-wheel-test/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-wheel-test/bin/embedflow --help
@@ -65,7 +65,7 @@ python -m venv /tmp/embedflow-testpypi
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  embedflow==0.6.0
+  embedflow==0.8.0
 cd /tmp
 /tmp/embedflow-testpypi/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-testpypi/bin/embedflow --help
@@ -79,11 +79,11 @@ Test optional integrations in a second clean environment:
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  "embedflow[faiss,dashboard,pinecone,milvus,weaviate]==0.6.0"
+  "embedflow[faiss,dashboard,pinecone,milvus,weaviate]==0.8.0"
 ```
 
-If the same filename already exists on TestPyPI, use a pre-release such as
-`0.6.0rc1` for the TestPyPI-only trial. Keep production `0.6.0` unchanged.
+If the same filename already exists on TestPyPI, use a pre-release for the
+TestPyPI-only trial. Never overwrite the production version.
 
 ## Trusted Publishing configuration
 
@@ -115,7 +115,7 @@ above keeps the test step explicit.
 3. Run the final release gate and review the generated report.
 4. Configure the PyPI pending publisher and protected `pypi` environment.
 5. Create a Git tag and GitHub Release for the exact package version, for
-   example `v0.6.0`.
+   example `v0.8.0`.
 6. Approve the `pypi` environment when the release workflow is ready.
 7. Verify the files and metadata on PyPI.
 8. Install from production PyPI in a directory outside this checkout.

embedflow-0.8.0/docs/shadow-mode.md

@@ -0,0 +1,104 @@
+# Shadow Mode
+
+Shadow Mode runs the configured target migration path beside real traffic while
+the existing source result remains authoritative. It is an observation and
+cache-warming tool, not a traffic router:
+
+```yaml
+runtime:
+  mode: shadow
+
+shadow:
+  enabled: true
+  sample_rate: 0.10
+  sample_seed: 42
+  candidate_k: 100
+  materialize: true
+  max_inflight: 32
+  queue_capacity: 1000
+  timeout_ms: 10000
+  telemetry:
+    enabled: true
+    path: ./.embedflow/shadow.sqlite3
+    retain_query_records: false
+    retain_query_text: false
+```
+
+Install the optional backend/model extras required by the selected
+configuration, then start the ordinary service:
+
+```bash
+pip install "embedflow[dashboard]"
+embedflow serve --config embedflow.yaml
+# or override the file for one process:
+embedflow serve --config embedflow.yaml --mode shadow
+```
+
+The request path performs source encoding, source ANN retrieval, and document
+resolution exactly as source-only serving. It returns that result without
+waiting for target encoding, cache misses, reranking, telemetry, or the
+materialization worker. Shadow jobs use a bounded queue and worker pool. A full
+queue drops only shadow work; a target/model/cache/telemetry failure or timeout
+is recorded and cannot change the primary response.
+
+If the target model cannot be loaded during startup, source/shadow serving
+still opens with target work marked unavailable; sampled requests record the
+isolated target-encoding failure. Normal migration mode remains fail-fast for
+the same startup error.
+
+`candidate_k` is explicit. The planner may suggest a value, but Shadow Mode
+does not silently select one. `materialize: false` reads already cached target
+vectors without changing cache or queue state. With `materialize: true`, cold
+candidate IDs are deduplicated in the existing persistent materialization queue
+and warmed asynchronously; synchronous target misses remain zero in Shadow
+Mode. A comparison with incomplete target vectors is reported as `partial`,
+with target coverage shown separately.
+
+## Reports
+
+Reports use the privacy-preserving SQLite telemetry store (by default beside
+the configured cache):
+
+```bash
+embedflow shadow report --config embedflow.yaml --since 24h
+embedflow shadow report --config embedflow.yaml --since 24h --format json --output shadow-report.json
+```
+
+Use `/shadow/report` or the `shadow` section of `/status` and `/metrics` in the
+FastAPI service. JSON/YAML stdout contains only the requested artifact; progress
+and errors belong on stderr. `--since` accepts seconds or `s`, `m`, `h`, `d`,
+and `w` suffixes.
+
+Reports contain sampled/completed/partial/failed/timed-out/dropped counts,
+cache and materialization accounting, source and shadow latency distributions,
+target coverage, top-1 agreement, and top-k overlap. Shadow latency is not
+user-facing latency because it is off the primary critical path. Ranking
+disagreement and overlap are diagnostics, not recall, nDCG, or quality loss;
+without qrels/native-target evaluation no quality guarantee is made.
+
+T2-v1 remains the frozen window-level diagnostic. EmbedFlow does not label an
+individual production query `T2 SAFE`. A report recommendation is operational
+guidance only:
+
+- `CONTINUE_SHADOW` means evidence or coverage is still limited.
+- `EXPAND_K` means the accumulated T2 window requests a larger candidate pool.
+- `INVESTIGATE` means failures, timeouts, or uncertain T2 behavior need review.
+- `READY_FOR_CANARY_EVALUATION` means an operator may consider a separately
+  designed canary; it never routes traffic automatically.
+
+Raw query text, candidate text, source vectors, target vectors, and credentials
+are not persisted in telemetry. Query IDs may be retained only when explicitly
+enabled, and retention is bounded by `max_records`. The telemetry database is
+segmented by source/target contract and candidate-K fingerprint so unrelated
+migrations are not mixed.
+
+On shutdown EmbedFlow stops accepting new shadow work, drops queued jobs when
+necessary, and waits only the configured bounded grace period. A corrupted or
+unwritable telemetry file degrades observability; it does not take source
+serving down. Source indexes are read-only during Shadow Mode. The planner and
+Shadow Mode are separate: generate a plan first, then copy its recommended K
+explicitly into a reviewed Shadow configuration.
+
+Shadow Mode is advisory and experimental operational instrumentation. It does
+not provide autonomous rollout, rollback, qrel evaluation, or ANN-fidelity
+proof.

embedflow-0.8.0/embedflow/__init__.py

@@ -0,0 +1,59 @@
+"""EmbedFlow: progressive embedding-model migration for existing indexes."""
+
+__version__ = "0.8.0"
+
+from .config import (
+    EmbedFlowConfig,
+    PlannerConfig,
+    PrewarmConfig,
+    RuntimeConfig,
+    ShadowConfig,
+    ShadowTelemetryConfig,
+    load_config,
+)
+from .prewarm import PrewarmPlan, PrewarmPlanner, PrewarmRunner, TrafficHotsetPlanner, load_prewarm_plan
+
+
+def plan(*args, **kwargs):
+    """Generate an advisory migration plan without changing source traffic."""
+    from .planner import plan_migration
+    return plan_migration(*args, **kwargs)
+
+
+def migrate(*args, **kwargs):
+    """Start progressive migration over an existing FAISS, Qdrant, pgvector, Pinecone, Milvus, or Weaviate index.
+
+    Imported lazily to keep the lightweight configuration package free of
+    model-serving dependencies at import time.  See ``embedflow.migration``
+    for the ``MigrationSession`` type.
+    """
+    from .migration.facade import migrate as _migrate
+    return _migrate(*args, **kwargs)
+
+
+def analyze_migration(*args, **kwargs):
+    """Run the leakage-safe no-target-index analysis programmatically."""
+    from .analysis import analyze_migration as _analyze_migration
+    return _analyze_migration(*args, **kwargs)
+
+
+def prewarm_plan(telemetry, cache, **kwargs):
+    """Build a bounded traffic-hotset plan through the reusable API.
+
+    Planner-constructor options and ``plan`` options may be supplied together;
+    recognized plan options are routed to :meth:`PrewarmPlanner.plan`.
+    """
+    plan_keys = {"since_seconds", "start", "end", "max_docs", "target_observed_coverage",
+                 "max_storage_gb", "max_runtime_seconds", "docs_per_second", "gpu_hourly_cost", "strategy"}
+    plan_kwargs = {key: kwargs.pop(key) for key in tuple(kwargs) if key in plan_keys}
+    return PrewarmPlanner(telemetry, cache, **kwargs).plan(**plan_kwargs)
+
+
+def prewarm_run(plan, *args, **kwargs):
+    """Execute a validated prewarm plan through the existing materializer."""
+    run_keys = {"max_runtime_seconds", "progress"}
+    run_kwargs = {key: kwargs.pop(key) for key in tuple(kwargs) if key in run_keys}
+    return PrewarmRunner(*args, **kwargs).run(plan, **run_kwargs)
+
+
+__all__ = ["EmbedFlowConfig", "PlannerConfig", "PrewarmConfig", "RuntimeConfig", "ShadowConfig", "ShadowTelemetryConfig", "load_config", "migrate", "plan", "analyze_migration", "prewarm_plan", "prewarm_run", "PrewarmPlan", "PrewarmPlanner", "TrafficHotsetPlanner", "PrewarmRunner", "load_prewarm_plan", "__version__"]

{embedflow-0.6.0 → embedflow-0.8.0}/embedflow/cache/base.py

@@ -8,15 +8,15 @@ import numpy as np
 
 class TargetVectorCache(ABC):
     @abstractmethod
-    def get(self, document_ids: list[str]) -> dict[str, np.ndarray]:
+    def get(self, document_ids: list[str], content_fingerprints: Any = None) -> dict[str, np.ndarray]:
         raise NotImplementedError
 
     @abstractmethod
-    def put(self, document_ids: list[str], vectors: np.ndarray) -> None:
+    def put(self, document_ids: list[str], vectors: np.ndarray, content_fingerprints: Any = None) -> None:
         raise NotImplementedError
 
     @abstractmethod
-    def contains(self, document_ids: list[str]) -> set[str]:
+    def contains(self, document_ids: list[str], content_fingerprints: Any = None) -> set[str]:
         raise NotImplementedError
 
     @abstractmethod