embedflow 0.2.0__tar.gz → 0.3.0__tar.gz

{embedflow-0.2.0 → embedflow-0.3.0}/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## v0.3.0 — Pinecone backend
+
+- Added a read-only Pinecone backend for existing dense indexes.
+- Added host targeting, index-name resolution, namespace-aware retrieval, and
+  metadata or external-document text resolution.
+- Added Pinecone status/audit integration, optional dependency packaging, and
+  unit/integration smoke fixtures.
+
 ## v0.2.0 — pgvector backend
 
 - Added a read-only pgvector backend for existing PostgreSQL vector tables.

{embedflow-0.2.0 → embedflow-0.3.0}/CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 title: "EmbedFlow: Upgrading Legacy Embeddings Without Full Upfront Re-Embedding"
 message: "If EmbedFlow contributes to your work, please cite this software release."
 type: software
-version: 0.2.0
+version: 0.3.0
 date-released: 2026-09-06
 repository-code: "https://github.com/arnsri33/embedflow"
 url: "https://github.com/arnsri33/embedflow"

{embedflow-0.2.0 → embedflow-0.3.0}/CONTRIBUTING.md

@@ -11,7 +11,7 @@ python -m pip install -e '.[dev]'
 ```
 
 Optional integrations can be installed with `.[faiss]`, `.[qdrant]`,
-`.[models]`, or `.[dashboard]`.
+`.[pgvector]`, `.[pinecone]`, `.[models]`, or `.[dashboard]`.
 
 ## Checks before opening a pull request
 

{embedflow-0.2.0 → embedflow-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: embedflow
-Version: 0.2.0
+Version: 0.3.0
 Summary: Progressive embedding-model migration over existing vector indexes.
 Author: Arnav Srivastav
 License-Expression: AGPL-3.0-only
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://embedflow.org
 Project-URL: Repository, https://github.com/arnsri33/embedflow
 Project-URL: Documentation, https://github.com/arnsri33/embedflow#readme
 Project-URL: Issues, https://github.com/arnsri33/embedflow/issues
-Keywords: embeddings,vector-search,rag,information-retrieval,faiss,qdrant,pgvector
+Keywords: embeddings,vector-search,rag,information-retrieval,faiss,qdrant,pgvector,pinecone
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
@@ -29,6 +29,8 @@ Provides-Extra: qdrant
 Requires-Dist: qdrant-client>=1.9; extra == "qdrant"
 Provides-Extra: pgvector
 Requires-Dist: psycopg[binary]>=3.2; extra == "pgvector"
+Provides-Extra: pinecone
+Requires-Dist: pinecone>=6.0; extra == "pinecone"
 Provides-Extra: models
 Requires-Dist: huggingface-hub<1.0,>=0.34; extra == "models"
 Requires-Dist: transformers<5.0,>=4.45; extra == "models"
@@ -42,6 +44,7 @@ Provides-Extra: all
 Requires-Dist: faiss-cpu>=1.8.0; extra == "all"
 Requires-Dist: qdrant-client>=1.9; extra == "all"
 Requires-Dist: psycopg[binary]>=3.2; extra == "all"
+Requires-Dist: pinecone>=6.0; extra == "all"
 Requires-Dist: huggingface-hub<1.0,>=0.34; extra == "all"
 Requires-Dist: transformers<5.0,>=4.45; extra == "all"
 Requires-Dist: sentence-transformers>=3.0; extra == "all"
@@ -62,7 +65,7 @@ Dynamic: license-file
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, FAISS,
-Qdrant, pgvector, a CLI, and FastAPI.
+Qdrant, pgvector, Pinecone, a CLI, and FastAPI.
 
 The full project README and architecture diagram are on
 <https://github.com/arnsri33/embedflow>.
@@ -85,6 +88,12 @@ For an existing PostgreSQL/pgvector table:
 python -m pip install "embedflow[pgvector]"
 ```
 
+For an existing Pinecone dense index:
+
+```bash
+python -m pip install "embedflow[pinecone]"
+```
+
 Qdrant and model-runtime extras are documented in the
 [installation guide](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -186,10 +195,12 @@ for definitions and reproduction details.
 | FAISS | Supported |
 | Qdrant | Supported |
 | pgvector | Supported |
+| Pinecone | Supported |
 
 See the [FAISS guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/faiss.md),
 [Qdrant guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/qdrant.md),
-and [pgvector guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md).
+and [pgvector guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md),
+and [Pinecone guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md).
 
 ## CLI
 
@@ -209,7 +220,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.2.0 is an alpha release for research and early real-world
+EmbedFlow v0.3.0 is an alpha release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.2.0 → embedflow-0.3.0}/README.md

@@ -7,7 +7,7 @@
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, and serving
-through FAISS, Qdrant, pgvector, a CLI, and FastAPI.
+through FAISS, Qdrant, pgvector, Pinecone, a CLI, and FastAPI.
 
 [Quickstart](#try-it) · [Documentation](#documentation) · [Research](#research)
 
@@ -52,6 +52,12 @@ For FAISS and the dashboard, add the optional integrations:
 python -m pip install "embedflow[faiss,dashboard]"
 ```
 
+For an existing Pinecone dense index:
+
+```bash
+python -m pip install "embedflow[pinecone]"
+```
+
 Qdrant and model-runtime extras are documented in
 [`docs/installation.md`](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -190,12 +196,14 @@ is measured separately and is `UNKNOWN` until an exact reference is supplied.
 | FAISS | Supported |
 | Qdrant | Supported |
 | pgvector | Supported |
+| Pinecone | Supported |
 
 Backend-specific setup and examples:
 
 - [FAISS](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/faiss.md)
 - [Qdrant](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/qdrant.md)
 - [pgvector](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md)
+- [Pinecone](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md)
 - [Adding a backend](https://github.com/arnsri33/embedflow/blob/main/CONTRIBUTING.md)
 
 ## CLI
@@ -230,7 +238,7 @@ OpenAPI documentation; see
 
 ## Status
 
-EmbedFlow v0.2.0 is an alpha release for research and early real-world
+EmbedFlow v0.3.0 is an alpha release for research and early real-world
 testing.
 
 - T2-v1 reports an empirical finite-tail diagnostic.

{embedflow-0.2.0 → embedflow-0.3.0}/README_PYPI.md

@@ -5,7 +5,7 @@
 EmbedFlow lets a new embedding model serve over candidates from an existing
 vector index while target document vectors are materialized progressively. It
 supports migration analysis, persistent caching, background work, FAISS,
-Qdrant, pgvector, a CLI, and FastAPI.
+Qdrant, pgvector, Pinecone, a CLI, and FastAPI.
 
 The full project README and architecture diagram are on
 <https://github.com/arnsri33/embedflow>.
@@ -28,6 +28,12 @@ For an existing PostgreSQL/pgvector table:
 python -m pip install "embedflow[pgvector]"
 ```
 
+For an existing Pinecone dense index:
+
+```bash
+python -m pip install "embedflow[pinecone]"
+```
+
 Qdrant and model-runtime extras are documented in the
 [installation guide](https://github.com/arnsri33/embedflow/blob/main/docs/installation.md).
 For model-backed analysis, install `embedflow[faiss,models,dashboard]`.
@@ -129,10 +135,12 @@ for definitions and reproduction details.
 | FAISS | Supported |
 | Qdrant | Supported |
 | pgvector | Supported |
+| Pinecone | Supported |
 
 See the [FAISS guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/faiss.md),
 [Qdrant guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/qdrant.md),
-and [pgvector guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md).
+and [pgvector guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pgvector.md),
+and [Pinecone guide](https://github.com/arnsri33/embedflow/blob/main/docs/integrations/pinecone.md).
 
 ## CLI
 
@@ -152,7 +160,7 @@ cover the remaining commands and endpoints.
 
 ## Status
 
-EmbedFlow v0.2.0 is an alpha release for research and early real-world
+EmbedFlow v0.3.0 is an alpha release for research and early real-world
 testing. T2-v1 is an empirical finite-tail diagnostic, partial rankings can
 differ from fully warm target reranking, and ANN fidelity needs a reference
 comparison to audit.

{embedflow-0.2.0 → embedflow-0.3.0}/docs/api.md

@@ -49,7 +49,9 @@ request. A partial response scores the available target vectors; it can differ
 from the fully warm ranking.
 
 `/status` includes safe backend metadata. For pgvector this names the
-schema/table and vector contract without returning the DSN or credentials;
+schema/table and vector contract; for Pinecone it names the host/index,
+namespace, dimension, metric, and safe vector counts. Neither backend returns
+the DSN, API key, or credentials;
 `/health` remains a compact liveness response for probes and load balancers.
 
 ## Errors

{embedflow-0.2.0 → embedflow-0.3.0}/docs/configuration.md

@@ -14,7 +14,7 @@ target:
   device: cuda
 
 index:
-  backend: faiss                 # faiss, qdrant, or pgvector
+  backend: faiss                 # faiss, qdrant, pgvector, or pinecone
   path: ./legacy.index
   ids: ./legacy.index.ids.json   # FAISS sidecar
   metric: cosine
@@ -22,6 +22,8 @@ index:
   # Qdrant fields: url, collection, vector_name, api_key_env
   # pgvector fields: dsn_env, schema, table, id_column, vector_column, text_column
   #                 hnsw_ef_search, ivfflat_probes
+  # Pinecone fields: host (preferred) or index_name, api_key_env, namespace,
+  #                  text_metadata_field
 
 documents:
   path: ./documents.jsonl
@@ -88,6 +90,11 @@ EMBEDFLOW_PGVECTOR_VECTOR_COLUMN
 EMBEDFLOW_PGVECTOR_TEXT_COLUMN
 EMBEDFLOW_PGVECTOR_HNSW_EF_SEARCH
 EMBEDFLOW_PGVECTOR_IVFFLAT_PROBES
+EMBEDFLOW_PINECONE_HOST
+EMBEDFLOW_PINECONE_INDEX_NAME
+EMBEDFLOW_PINECONE_NAMESPACE
+EMBEDFLOW_PINECONE_TEXT_METADATA_FIELD
+EMBEDFLOW_PINECONE_API_KEY_ENV
 EMBEDFLOW_INDEX_NPROBE
 EMBEDFLOW_DOCUMENTS_PATH
 EMBEDFLOW_CACHE_PATH

{embedflow-0.2.0 → embedflow-0.3.0}/docs/installation.md

@@ -31,6 +31,7 @@ python -m pip install -e .
 | `faiss` | FAISS source-index adapter |
 | `qdrant` | Qdrant client and adapter |
 | `pgvector` | Psycopg 3 binary driver and pgvector adapter |
+| `pinecone` | Official Pinecone Python SDK and adapter |
 | `models` | PyTorch, Transformers, Sentence Transformers, and Hub client |
 | `dashboard` | FastAPI, Uvicorn, and Pydantic |
 | `dev` | Pytest, Ruff, and build tooling |
@@ -59,6 +60,12 @@ Install the optional adapter with `python -m pip install "embedflow[pgvector]"`.
 The adapter connects to an existing table and reads the DSN from the
 environment; see [`integrations/pgvector.md`](integrations/pgvector.md).
 
+## Pinecone
+
+Install the optional adapter with `python -m pip install "embedflow[pinecone]"`.
+Set `PINECONE_API_KEY` in the environment and configure an existing dense
+index host; see [`integrations/pinecone.md`](integrations/pinecone.md).
+
 ## CPU and GPU
 
 The deterministic demo runs on CPU. Real model serving accepts `--device cpu`

embedflow-0.3.0/docs/integrations/pinecone.md

@@ -0,0 +1,157 @@
+# Pinecone
+
+EmbedFlow can use an existing Pinecone dense index as its source candidate
+layer. It queries that index, reranks the returned documents with the target
+embedding model, and materializes target vectors into EmbedFlow's local cache.
+The source index is read-only during normal operation.
+
+## Install
+
+```bash
+python -m pip install "embedflow[pinecone]"
+```
+
+The extra installs the official `pinecone` Python SDK. The base package does
+not import or require it.
+
+## Configure an existing index
+
+Set the API key in the environment, never in YAML:
+
+```bash
+export PINECONE_API_KEY='...'
+```
+
+Use the data-plane host copied from the Pinecone console:
+
+```yaml
+source:
+  model: sentence-transformers/all-MiniLM-L6-v2
+  dimension: 384
+  device: cuda
+
+target:
+  model: Qwen/Qwen3-Embedding-0.6B
+  device: cuda
+
+index:
+  backend: pinecone
+  api_key_env: PINECONE_API_KEY
+  host: my-index-xxxxx.svc.aped-xxxx.pinecone.io
+  namespace: production
+  metric: cosine
+  text_metadata_field: text
+
+documents:
+  # Omit this file when text_metadata_field is present in Pinecone metadata.
+  # If present, it is used as the external text resolver instead.
+  path: ./documents.jsonl
+  id_field: id
+  text_field: text
+```
+
+`host` takes precedence when both `host` and `index_name` are supplied. If a
+host is unavailable, `index_name` can resolve through the control plane; host
+targeting is recommended for production data-plane traffic. The configured
+namespace is passed on every query and fetch. An empty namespace selects
+Pinecone's default namespace.
+
+The source index must be a dense vector index whose dimension and metric match
+the source model contract. Pinecone IDs remain strings, including numeric-
+looking IDs such as `"42"`, UUID-looking values, Unicode, and punctuation.
+
+## Document text
+
+EmbedFlow needs candidate text for target-model encoding. There are two modes:
+
+* Set `text_metadata_field` to read text from Pinecone metadata. Candidate
+  queries request metadata but never request vector values. Missing or
+  non-string text is reported with the candidate ID and field name.
+* Provide the normal JSONL `documents` store when Pinecone contains IDs and
+  vectors only. The external store remains the source of text and avoids
+  duplicating a corpus in Pinecone.
+
+## Commands
+
+```bash
+embedflow doctor --config embedflow.yaml
+embedflow audit-index --config embedflow.yaml
+embedflow analyze --config embedflow.yaml
+embedflow serve --config embedflow.yaml
+embedflow search --config embedflow.yaml "what causes auroras?"
+embedflow status --config embedflow.yaml
+```
+
+`audit-index` separates backend health from retrieval fidelity. It checks SDK
+and credentials, reachability, namespace stats, dimension, metric, dense
+vector type, a small candidate query, and text resolution where possible. A
+successful API call is not an ANN recall or T2-v1 compatibility guarantee.
+
+## Read-only behavior
+
+`doctor`, `audit-index`, `analyze`, `serve`, `search`, `status`, and `prewarm`
+only query the configured index. EmbedFlow does not create or delete indexes,
+upsert or delete vectors, alter metadata, or change namespaces. Test fixtures
+may create temporary indexes explicitly, but production initialization never
+does.
+
+## Metrics and limits
+
+`cosine`, `dot`/`inner_product` (Pinecone `dotproduct`), and
+`l2`/`euclidean` are accepted. Cosine and dot-product scores already rank
+higher-is-better. Pinecone's Euclidean score is squared distance, so EmbedFlow
+negates it to preserve the shared higher-is-better convention. Query `top_k`
+is validated in the range 1–10,000, matching the Pinecone API limit.
+
+## Integrated embedding indexes
+
+Pinecone indexes created with hosted/integrated embedding manage the source
+embedding contract outside EmbedFlow. Unless the source model, dimension, and
+metric can be verified, EmbedFlow reports that contract as unknown; it does not
+assume Pinecone's hosted model is the configured source encoder. The first
+adapter release is intended for standard dense-vector indexes.
+
+## Eventual consistency and tests
+
+Pinecone is eventually consistent after writes. Any test setup that creates a
+temporary fixture must poll `describe_index_stats` with a bounded timeout before
+querying it. EmbedFlow's production adapter performs no writes and therefore
+does not need a write-read delay.
+
+## Troubleshooting
+
+* `Pinecone support requires ...`: install `embedflow[pinecone]` in the active
+  environment.
+* `Environment variable PINECONE_API_KEY is not set.`: export the variable
+  named by `api_key_env`.
+* `Unable to reach configured Pinecone index`: check the host, project, and
+  network policy. Credentials are not printed in this error.
+* Dimension or metric mismatch: compare the source model contract with the
+  existing index configuration; EmbedFlow will not rewrite the index.
+* Missing metadata text: set the correct `text_metadata_field` or provide an
+  external JSONL document store.
+
+## Local smoke fixture
+
+The repository includes a non-destructive smoke helper for an existing index:
+
+```bash
+export PINECONE_API_KEY='...'
+python scripts/pinecone_smoke.py \
+  --host "$PINECONE_INDEX_HOST" \
+  --namespace production
+```
+
+It only describes stats and, when given a query vector, performs a query. It
+never upserts or deletes records. A remote create/query/delete integration is
+opt-in and requires both `EMBEDFLOW_PINECONE_INTEGRATION_TEST=1` and
+`EMBEDFLOW_PINECONE_CREATE_TEST_INDEX=1`; it uses a unique disposable index and
+deletes it in teardown.
+
+## Limitations
+
+The adapter uses one synchronous SDK client. The SDK call itself is safe for
+the modest concurrent serving loads tested by EmbedFlow, but no connection pool
+is introduced. Metadata filters are not exposed because the shared
+`VectorIndex` contract has no portable filter field. Pinecone index provisioning
+and namespace administration remain outside EmbedFlow.

{embedflow-0.2.0 → embedflow-0.3.0}/docs/limitations.md

@@ -1,6 +1,6 @@
 # Limitations and release scope
 
-EmbedFlow v0.2.0 is an alpha release for research and early real-world
+EmbedFlow v0.3.0 is an alpha release for research and early real-world
 testing. The serving path is designed to make migration experiments concrete;
 production rollout still requires application-specific validation.
 

{embedflow-0.2.0 → embedflow-0.3.0}/docs/releasing.md

@@ -18,8 +18,8 @@ python -m twine check dist/*
 Inspect both archives before uploading:
 
 ```bash
-unzip -l dist/embedflow-0.2.0-py3-none-any.whl
-tar -tzf dist/embedflow-0.2.0.tar.gz
+unzip -l dist/embedflow-0.3.0-py3-none-any.whl
+tar -tzf dist/embedflow-0.3.0.tar.gz
 sha256sum dist/*
 ```
 
@@ -32,7 +32,7 @@ Test the wheel outside the source tree:
 ```bash
 python -m venv /tmp/embedflow-wheel-test
 /tmp/embedflow-wheel-test/bin/python -m pip install --upgrade pip
-/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.2.0-py3-none-any.whl
+/tmp/embedflow-wheel-test/bin/python -m pip install dist/embedflow-0.3.0-py3-none-any.whl
 cd /tmp
 /tmp/embedflow-wheel-test/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-wheel-test/bin/embedflow --help
@@ -65,7 +65,7 @@ python -m venv /tmp/embedflow-testpypi
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  embedflow==0.2.0
+  embedflow==0.3.0
 cd /tmp
 /tmp/embedflow-testpypi/bin/python -c "import embedflow; print(embedflow.__version__)"
 /tmp/embedflow-testpypi/bin/embedflow --help
@@ -79,11 +79,11 @@ Test optional integrations in a second clean environment:
 /tmp/embedflow-testpypi/bin/python -m pip install \
   --index-url https://test.pypi.org/simple/ \
   --extra-index-url https://pypi.org/simple/ \
-  "embedflow[faiss,dashboard]==0.2.0"
+  "embedflow[faiss,dashboard,pinecone]==0.3.0"
 ```
 
 If the same filename already exists on TestPyPI, use a pre-release such as
-`0.2.0rc1` for the TestPyPI-only trial. Keep production `0.2.0` unchanged.
+`0.3.0rc1` for the TestPyPI-only trial. Keep production `0.3.0` unchanged.
 
 ## Trusted Publishing configuration
 
@@ -115,7 +115,7 @@ above keeps the test step explicit.
 3. Run the final release gate and review the generated report.
 4. Configure the PyPI pending publisher and protected `pypi` environment.
 5. Create a Git tag and GitHub Release for the exact package version, for
-   example `v0.2.0`.
+   example `v0.3.0`.
 6. Approve the `pypi` environment when the release workflow is ready.
 7. Verify the files and metadata on PyPI.
 8. Install from production PyPI in a directory outside this checkout.

{embedflow-0.2.0 → embedflow-0.3.0}/embedflow/__init__.py

@@ -1,12 +1,12 @@
 """EmbedFlow: progressive embedding-model migration for existing indexes."""
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 from .config import EmbedFlowConfig, load_config
 
 
 def migrate(*args, **kwargs):
-    """Start progressive migration over an existing FAISS, Qdrant, or pgvector index.
+    """Start progressive migration over an existing FAISS, Qdrant, pgvector, or Pinecone index.
 
     Imported lazily to keep the lightweight configuration package free of
     model-serving dependencies at import time.  See ``embedflow.migration``

embedflow-0.3.0/embedflow/api.py

@@ -0,0 +1,10 @@
+"""Public compatibility exports for EmbedFlow's existing FastAPI app.
+
+The implementation remains in :mod:`embedflow.serving.api`; this module keeps
+the short ``embedflow.api`` import path lightweight and does not create an app
+or import FastAPI until :func:`create_app` is called.
+"""
+
+from .serving.api import create_app, dashboard_html
+
+__all__ = ["create_app", "dashboard_html"]