embedflow 0.6.0__tar.gz → 0.7.0__tar.gz

{embedflow-0.6.0 → embedflow-0.7.0}/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## v0.7.0 — Source-authoritative Shadow Mode
+
+- Added bounded, deterministic, source-authoritative Shadow Mode for observing
+  target reranking on sampled traffic without delaying or changing source
+  responses.
+- Added failure/timeout isolation, queue backpressure, optional asynchronous
+  target-cache materialization, target-coverage and ranking diagnostics, and
+  privacy-conscious SQLite telemetry.
+- Added `embedflow shadow report`, API/status integration, documentation, and a
+  deterministic offline demonstration. Shadow reports remain operational
+  diagnostics and do not claim qrel-based retrieval quality.
+
 ## v0.6.0 — Migration planner
 
 - Added an advisory `embedflow plan` command and Python API that combine

{embedflow-0.6.0 → embedflow-0.7.0}/CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 title: "EmbedFlow: Upgrading Legacy Embeddings Without Full Upfront Re-Embedding"
 message: "If EmbedFlow contributes to your work, please cite this software release."
 type: software
-version: 0.6.0
+version: 0.7.0
 date-released: 2026-09-13
 repository-code: "https://github.com/arnsri33/embedflow"
 url: "https://github.com/arnsri33/embedflow"

{embedflow-0.6.0 → embedflow-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: embedflow
-Version: 0.6.0
+Version: 0.7.0
 Summary: Progressive embedding-model migration over existing vector indexes.
 Author: Arnav Srivastav
 License-Expression: AGPL-3.0-only
@@ -204,6 +204,26 @@ behavior, candidate K, cache/economics projections, and staged rollout
 guidance. It never routes traffic or mutates the source index; `SAFE` is not a
 qrels-based retrieval-quality guarantee. See the [planner guide](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
 
+### Source-authoritative Shadow Mode
+
+Run the target migration path beside sampled traffic while always returning the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h --format json
+```
+
+Shadow work is asynchronous, bounded, privacy-conscious, and failure isolated.
+Reports contain operational and ranking-disagreement diagnostics—not qrel-based
+quality guarantees—and recommendations never route canary traffic. See the
+[Shadow Mode guide](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
 ## Research
 
 For candidate depth `K`, EmbedFlow measures:
@@ -258,7 +278,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
+EmbedFlow v0.7.0 is a pre-1.0 release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.6.0 → embedflow-0.7.0}/README.md

@@ -156,6 +156,24 @@ embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl --forma
 `SAFE` is an empirical finite-tail signal, not a retrieval-quality guarantee.
 See [`docs/planner.md`](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
 
+Observe the reviewed migration path on real traffic without changing the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h
+```
+
+Shadow Mode is source-authoritative, bounded, and advisory. It records cache,
+coverage, latency, and ranking-disagreement diagnostics; it does not claim
+retrieval-quality preservation without qrels and never routes canary traffic.
+See [`docs/shadow-mode.md`](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
 Search responses expose `COLD`, `PARTIAL`, or `WARM`, cache hits and misses,
 synchronous work, queued work, and stage timings. Once the candidate vectors
 are warm, target scoring over that candidate set is deterministic.
@@ -263,13 +281,14 @@ OpenAPI documentation; see
 - [CLI reference](https://github.com/arnsri33/embedflow/blob/main/docs/cli.md)
 - [API](https://github.com/arnsri33/embedflow/blob/main/docs/api.md)
 - [Economics](https://github.com/arnsri33/embedflow/blob/main/docs/economics.md)
+- [Shadow Mode](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md)
 - [Limitations](https://github.com/arnsri33/embedflow/blob/main/docs/limitations.md)
 - [Contributing](https://github.com/arnsri33/embedflow/blob/main/CONTRIBUTING.md)
 - [Security](https://github.com/arnsri33/embedflow/blob/main/SECURITY.md)
 
 ## Status
 
-EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
+EmbedFlow v0.7.0 is a pre-1.0 release for research and early real-world
 testing.
 
 - T2-v1 reports an empirical finite-tail diagnostic.

{embedflow-0.6.0 → embedflow-0.7.0}/README_PYPI.md

@@ -138,6 +138,26 @@ behavior, candidate K, cache/economics projections, and staged rollout
 guidance. It never routes traffic or mutates the source index; `SAFE` is not a
 qrels-based retrieval-quality guarantee. See the [planner guide](https://github.com/arnsri33/embedflow/blob/main/docs/planner.md).
 
+### Source-authoritative Shadow Mode
+
+Run the target migration path beside sampled traffic while always returning the
+source result:
+
+```yaml
+runtime: {mode: shadow}
+shadow: {enabled: true, sample_rate: 0.10, candidate_k: 100, materialize: true}
+```
+
+```bash
+embedflow serve --config ./embedflow.yaml
+embedflow shadow report --config ./embedflow.yaml --since 24h --format json
+```
+
+Shadow work is asynchronous, bounded, privacy-conscious, and failure isolated.
+Reports contain operational and ranking-disagreement diagnostics—not qrel-based
+quality guarantees—and recommendations never route canary traffic. See the
+[Shadow Mode guide](https://github.com/arnsri33/embedflow/blob/main/docs/shadow-mode.md).
+
 ## Research
 
 For candidate depth `K`, EmbedFlow measures:
@@ -192,7 +212,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
+EmbedFlow v0.7.0 is a pre-1.0 release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.6.0 → embedflow-0.7.0}/docs/api.md

@@ -21,6 +21,7 @@ Interactive OpenAPI documentation is available at
 | GET | `/metrics` | Aggregated latency and queue metrics |
 | GET | `/plan` | Current migration plan |
 | GET | `/economics` | Configured economics estimate |
+| GET | `/shadow/report` | Bounded Shadow Mode diagnostics |
 
 ## Search
 
@@ -48,6 +49,13 @@ The response includes the result list and migration fields such as:
 request. A partial response scores the available target vectors; it can differ
 from the fully warm ranking.
 
+When `runtime.mode=shadow`, `/search` returns the source-only result and marks
+`migration.source_authoritative=true`; target work is scheduled in the
+background. `/shadow/report`, `/status`, and `/metrics` expose aggregate shadow
+observations without raw query/document text or vectors. Shadow failures and
+timeouts are isolated from the response. Ranking overlap is diagnostic and is
+not a qrels-based quality claim.
+
 The advisory migration planner is exposed through the Python API and the
 `embedflow plan` CLI. It is intentionally not a synchronous FastAPI endpoint:
 probe analysis may load models and perform bounded candidate work, so operators

{embedflow-0.6.0 → embedflow-0.7.0}/docs/cli.md

@@ -10,6 +10,7 @@ embedflow init --config ./embedflow.yaml
 embedflow analyze --config ./embedflow.yaml --output-dir ./analysis
 embedflow evaluate --config ./experiment.yaml --output-dir ./results
 embedflow plan --config ./embedflow.yaml --queries ./probe_queries.jsonl
+embedflow shadow report --config ./embedflow.yaml --since 24h
 ```
 
 `analyze` is the no-target-index workflow. It uses probe queries and frozen
@@ -42,6 +43,12 @@ materialization throughput. `audit-index` checks the source index against an
 exact/reference configuration where supported. `prewarm` schedules target
 document work; it does not change source-index results.
 
+Use `embedflow serve --mode shadow` to opt into source-authoritative Shadow
+Mode for one process. `embedflow shadow report` reads the bounded telemetry
+store and supports `--format text|json|yaml`, `--since`, `--output`, and
+`--quiet`. A valid `DEFER`/`EXPAND_K` report is still an analytical success;
+only invalid configuration or initialization returns a non-zero exit code.
+
 ## Registry and profiles
 
 ```bash

{embedflow-0.6.0 → embedflow-0.7.0}/docs/configuration.md

@@ -77,9 +77,38 @@ planner:
   corpus_name: null
   corpus_fingerprint: null
 
+# Source-authoritative observation mode.  ``runtime.mode: migration`` (the
+# default) leaves Shadow Mode inactive.  ``mode: shadow`` is an explicit opt-in.
+runtime:
+  mode: migration             # migration, normal, source, or shadow
+
+shadow:
+  enabled: true
+  sample_rate: 0.10
+  sample_seed: 42
+  candidate_k: 100
+  materialize: true
+  max_inflight: 32
+  queue_capacity: 1000
+  timeout_ms: 10000
+  shutdown_grace_ms: 1000
+  telemetry:
+    enabled: true
+    path: ./.embedflow/shadow.sqlite3
+    retain_query_records: false
+    retain_query_text: false
+    max_records: 10000
+    retention_days: null
+    report_k: 10
+    min_target_coverage_for_ranking: 1.0
+
 state_path: ./embedflow_state.json
 ```
 
+Shadow Mode always returns the source-authoritative result before target work
+finishes. See [`shadow-mode.md`](shadow-mode.md) for queue, timeout, privacy,
+materialization, and report semantics.
+
 ## Model contracts
 
 Model configuration can include `revision`, `dimension`, `max_length`,
@@ -164,6 +193,24 @@ EMBEDFLOW_PLANNER_LATENCY_BUDGET_MS
 EMBEDFLOW_PLANNER_ACCESS_TRACE
 EMBEDFLOW_PLANNER_CORPUS_NAME
 EMBEDFLOW_PLANNER_CORPUS_FINGERPRINT
+EMBEDFLOW_RUNTIME_MODE
+EMBEDFLOW_SHADOW_ENABLED
+EMBEDFLOW_SHADOW_SAMPLE_RATE
+EMBEDFLOW_SHADOW_SAMPLE_SEED
+EMBEDFLOW_SHADOW_CANDIDATE_K
+EMBEDFLOW_SHADOW_MATERIALIZE
+EMBEDFLOW_SHADOW_MAX_INFLIGHT
+EMBEDFLOW_SHADOW_QUEUE_CAPACITY
+EMBEDFLOW_SHADOW_TIMEOUT_MS
+EMBEDFLOW_SHADOW_SHUTDOWN_GRACE_MS
+EMBEDFLOW_SHADOW_TELEMETRY_ENABLED
+EMBEDFLOW_SHADOW_TELEMETRY_PATH
+EMBEDFLOW_SHADOW_RETAIN_QUERY_RECORDS
+EMBEDFLOW_SHADOW_RETAIN_QUERY_TEXT
+EMBEDFLOW_SHADOW_MAX_RECORDS
+EMBEDFLOW_SHADOW_REPORT_K
+EMBEDFLOW_SHADOW_RETENTION_DAYS
+EMBEDFLOW_SHADOW_MIN_TARGET_COVERAGE
 ```
 
 ## Input files

{embedflow-0.6.0 → embedflow-0.7.0}/docs/limitations.md

@@ -1,6 +1,6 @@
 # Limitations and release scope
 
-EmbedFlow v0.6.0 is a pre-1.0 release for research and early real-world
+EmbedFlow v0.7.0 is a pre-1.0 release for research and early real-world
 testing. The serving path is designed to make migration experiments concrete;
 production rollout still requires application-specific validation.
 

{embedflow-0.6.0 → embedflow-0.7.0}/docs/releasing.md

@@ -18,8 +18,8 @@ python -m twine check dist/*
 Inspect both archives before uploading:
 
 ```bash
-unzip -l dist/embedflow-0.6.0-py3-none-any.whl
-tar -tzf dist/embedflow-0.6.0.tar.gz
+unzip -l dist/embedflow-0.7.0-py3-none-any.whl
+tar -tzf dist/embedflow-0.7.0.tar.gz
 sha256sum dist/*
 ```
 
@@ -32,7 +32,7 @@ Test the wheel outside the source tree:
 ```bash
 python -m venv /tmp/embedflow-wheel-test
 /tmp/embedflow-wheel-test/bin/python -m pip install --upgrade pip
-/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.6.0-py3-none-any.whl
+/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.7.0-py3-none-any.whl
 cd /tmp
 /tmp/embedflow-wheel-test/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-wheel-test/bin/embedflow --help
@@ -65,7 +65,7 @@ python -m venv /tmp/embedflow-testpypi
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  embedflow==0.6.0
+  embedflow==0.7.0
 cd /tmp
 /tmp/embedflow-testpypi/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-testpypi/bin/embedflow --help
@@ -79,11 +79,11 @@ Test optional integrations in a second clean environment:
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  "embedflow[faiss,dashboard,pinecone,milvus,weaviate]==0.6.0"
+  "embedflow[faiss,dashboard,pinecone,milvus,weaviate]==0.7.0"
 ```
 
-If the same filename already exists on TestPyPI, use a pre-release such as
-`0.6.0rc1` for the TestPyPI-only trial. Keep production `0.6.0` unchanged.
+If the same filename already exists on TestPyPI, use a pre-release for the
+TestPyPI-only trial. Never overwrite the production version.
 
 ## Trusted Publishing configuration
 
@@ -115,7 +115,7 @@ above keeps the test step explicit.
 3. Run the final release gate and review the generated report.
 4. Configure the PyPI pending publisher and protected `pypi` environment.
 5. Create a Git tag and GitHub Release for the exact package version, for
-   example `v0.6.0`.
+   example `v0.7.0`.
 6. Approve the `pypi` environment when the release workflow is ready.
 7. Verify the files and metadata on PyPI.
 8. Install from production PyPI in a directory outside this checkout.

embedflow-0.7.0/docs/shadow-mode.md

@@ -0,0 +1,104 @@
+# Shadow Mode
+
+Shadow Mode runs the configured target migration path beside real traffic while
+the existing source result remains authoritative. It is an observation and
+cache-warming tool, not a traffic router:
+
+```yaml
+runtime:
+  mode: shadow
+
+shadow:
+  enabled: true
+  sample_rate: 0.10
+  sample_seed: 42
+  candidate_k: 100
+  materialize: true
+  max_inflight: 32
+  queue_capacity: 1000
+  timeout_ms: 10000
+  telemetry:
+    enabled: true
+    path: ./.embedflow/shadow.sqlite3
+    retain_query_records: false
+    retain_query_text: false
+```
+
+Install the optional backend/model extras required by the selected
+configuration, then start the ordinary service:
+
+```bash
+pip install "embedflow[dashboard]"
+embedflow serve --config embedflow.yaml
+# or override the file for one process:
+embedflow serve --config embedflow.yaml --mode shadow
+```
+
+The request path performs source encoding, source ANN retrieval, and document
+resolution exactly as source-only serving. It returns that result without
+waiting for target encoding, cache misses, reranking, telemetry, or the
+materialization worker. Shadow jobs use a bounded queue and worker pool. A full
+queue drops only shadow work; a target/model/cache/telemetry failure or timeout
+is recorded and cannot change the primary response.
+
+If the target model cannot be loaded during startup, source/shadow serving
+still opens with target work marked unavailable; sampled requests record the
+isolated target-encoding failure. Normal migration mode remains fail-fast for
+the same startup error.
+
+`candidate_k` is explicit. The planner may suggest a value, but Shadow Mode
+does not silently select one. `materialize: false` reads already cached target
+vectors without changing cache or queue state. With `materialize: true`, cold
+candidate IDs are deduplicated in the existing persistent materialization queue
+and warmed asynchronously; synchronous target misses remain zero in Shadow
+Mode. A comparison with incomplete target vectors is reported as `partial`,
+with target coverage shown separately.
+
+## Reports
+
+Reports use the privacy-preserving SQLite telemetry store (by default beside
+the configured cache):
+
+```bash
+embedflow shadow report --config embedflow.yaml --since 24h
+embedflow shadow report --config embedflow.yaml --since 24h --format json --output shadow-report.json
+```
+
+Use `/shadow/report` or the `shadow` section of `/status` and `/metrics` in the
+FastAPI service. JSON/YAML stdout contains only the requested artifact; progress
+and errors belong on stderr. `--since` accepts seconds or `s`, `m`, `h`, `d`,
+and `w` suffixes.
+
+Reports contain sampled/completed/partial/failed/timed-out/dropped counts,
+cache and materialization accounting, source and shadow latency distributions,
+target coverage, top-1 agreement, and top-k overlap. Shadow latency is not
+user-facing latency because it is off the primary critical path. Ranking
+disagreement and overlap are diagnostics, not recall, nDCG, or quality loss;
+without qrels/native-target evaluation no quality guarantee is made.
+
+T2-v1 remains the frozen window-level diagnostic. EmbedFlow does not label an
+individual production query `T2 SAFE`. A report recommendation is operational
+guidance only:
+
+- `CONTINUE_SHADOW` means evidence or coverage is still limited.
+- `EXPAND_K` means the accumulated T2 window requests a larger candidate pool.
+- `INVESTIGATE` means failures, timeouts, or uncertain T2 behavior need review.
+- `READY_FOR_CANARY_EVALUATION` means an operator may consider a separately
+  designed canary; it never routes traffic automatically.
+
+Raw query text, candidate text, source vectors, target vectors, and credentials
+are not persisted in telemetry. Query IDs may be retained only when explicitly
+enabled, and retention is bounded by `max_records`. The telemetry database is
+segmented by source/target contract and candidate-K fingerprint so unrelated
+migrations are not mixed.
+
+On shutdown EmbedFlow stops accepting new shadow work, drops queued jobs when
+necessary, and waits only the configured bounded grace period. A corrupted or
+unwritable telemetry file degrades observability; it does not take source
+serving down. Source indexes are read-only during Shadow Mode. The planner and
+Shadow Mode are separate: generate a plan first, then copy its recommended K
+explicitly into a reviewed Shadow configuration.
+
+Shadow Mode is advisory and experimental operational instrumentation. It does
+not provide autonomous rollout, rollback, qrel evaluation, or ANN-fidelity
+proof.

{embedflow-0.6.0 → embedflow-0.7.0}/embedflow/__init__.py

@@ -1,8 +1,8 @@
 """EmbedFlow: progressive embedding-model migration for existing indexes."""
 
-__version__ = "0.6.0"
+__version__ = "0.7.0"
 
-from .config import EmbedFlowConfig, PlannerConfig, load_config
+from .config import EmbedFlowConfig, PlannerConfig, RuntimeConfig, ShadowConfig, ShadowTelemetryConfig, load_config
 
 
 def plan(*args, **kwargs):
@@ -28,4 +28,4 @@ def analyze_migration(*args, **kwargs):
     return _analyze_migration(*args, **kwargs)
 
 
-__all__ = ["EmbedFlowConfig", "PlannerConfig", "load_config", "migrate", "plan", "analyze_migration", "__version__"]
+__all__ = ["EmbedFlowConfig", "PlannerConfig", "RuntimeConfig", "ShadowConfig", "ShadowTelemetryConfig", "load_config", "migrate", "plan", "analyze_migration", "__version__"]

{embedflow-0.6.0 → embedflow-0.7.0}/embedflow/cache/persistent_cache.py

@@ -141,6 +141,29 @@ class SQLiteVectorCache(TargetVectorCache):
             self._db.commit()
             return values
 
+    def peek(self, document_ids: list[str]) -> dict[str, np.ndarray]:
+        """Read cached vectors without changing hit/miss telemetry.
+
+        Shadow analysis with ``materialize=false`` must not make ordinary
+        serving cache statistics look like migration traffic.  This method is
+        intentionally a read-only fast path; callers that need accounting
+        should continue to use :meth:`get`.
+        """
+        ids = [str(x) for x in document_ids]
+        if not ids:
+            return {}
+        with self._lock:
+            self._ensure_open()
+            rows: list[tuple[Any, ...]] = []
+            for start in range(0, len(ids), self._LOOKUP_BATCH_SIZE):
+                chunk = ids[start:start + self._LOOKUP_BATCH_SIZE]
+                placeholders = ",".join("?" for _ in chunk)
+                rows.extend(self._db.execute(
+                    f"SELECT document_id,model_fingerprint,dimension,dtype,vector,checksum,created_at,accessed_at "
+                    f"FROM target_vectors WHERE model_fingerprint=? AND document_id IN ({placeholders})",
+                    [self.model_fingerprint, *chunk]).fetchall())
+            return {str(row[0]): self._decode(row) for row in rows}
+
     def put(self, document_ids: list[str], vectors: np.ndarray) -> None:
         ids = [str(x) for x in document_ids]
         values = np.asarray(vectors, dtype=self.dtype)