passageway-embedder-openai 0.3.0__tar.gz

passageway_embedder_openai-0.3.0/.gitignore

@@ -0,0 +1,11 @@
+docs/superpowers/
+.superpowers/
+.env
+.DS_Store
+.venv/
+__pycache__/
+.pytest_cache/
+.models/
+dist/
+site/
+.playwright-mcp/

passageway_embedder_openai-0.3.0/PKG-INFO

@@ -0,0 +1,73 @@
+Metadata-Version: 2.5
+Name: passageway-embedder-openai
+Version: 0.3.0
+Summary: passageway embedder for OpenAI-compatible embedding services: Ollama, LM Studio, vLLM, OpenAI, Voyage
+License-Expression: PolyForm-Noncommercial-1.0.0
+Requires-Python: >=3.12
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: truststore>=0.10
+Description-Content-Type: text/markdown
+
+# passageway-embedder-openai
+
+A [passageway](https://github.com/getpassageway/passageway#readme) embedder for services that speak the OpenAI embeddings API: **Ollama**, **LM Studio**, **vLLM**, **OpenAI** itself, and **Voyage** (whose API differs in one field). It speaks the [embedder protocol](https://passageway.marceltov.de/reference/embedder-protocol/), so passageway runs it as `kind = "command"` and the core does not change.
+
+**Where your text goes:** every indexed passage and every question is sent to the service at `BASE_URL`. With a local Ollama or LM Studio that is your own machine; with OpenAI or Voyage it is a third party, under their terms. passageway's default stays local; this embedder is your explicit choice.
+
+## Configure
+
+```toml
+[embedder]
+kind = "command"
+command = ["/path/to/python", "-m", "passageway_embedder_openai"]
+# Ollama: nomic-embed-text wants these prefixes; DIMENSIONS lets passageway start before Ollama does
+env = { BASE_URL = "http://localhost:11434/v1", MODEL = "nomic-embed-text", DIMENSIONS = "768",
+        QUERY_PREFIX = "search_query: ", PASSAGE_PREFIX = "search_document: " }
+# min_similarity = ...                      # measure it for this model: passageway eval --sweep
+timeout = 60                                # the core's limit per request; keep it above TIMEOUT below
+
+# OpenAI
+# env = { MODEL = "text-embedding-3-small" }
+# secrets = { API_KEY = "env:OPENAI_API_KEY" }
+
+# Voyage
+# env = { PROVIDER = "voyage", MODEL = "voyage-3" }
+# secrets = { API_KEY = "env:VOYAGE_API_KEY" }
+```
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `MODEL` | required | The embedding model the service serves |
+| `BASE_URL` | `https://api.openai.com/v1`, or Voyage's for `PROVIDER = "voyage"` | The service; `POST {BASE_URL}/embeddings` is called (a trailing `/embeddings` is dropped). No credentials in the URL |
+| `API_KEY` | | Sent as `Authorization: Bearer`. Required for api.openai.com and api.voyageai.com, usually not for a local server. Put it in `secrets`, never in `env` |
+| `PROVIDER` | `openai` | `openai` for every OpenAI-compatible service, `voyage` for Voyage, which wants `input_type` (`query` or `document`) with each request |
+| `QUERY_PREFIX`, `PASSAGE_PREFIX` | | Text put in front of every question or passage; e5-style models served locally want `query: ` and `passage: ` |
+| `DIMENSIONS` | | The vector length. Set, it answers `info` without a probe, so passageway can start while the service is still down; it is also sent as `dimensions` (`output_dimension` for Voyage) for models that can shorten their vectors. The first answer is checked against it |
+| `TIMEOUT` | `50` | Seconds for one request including its retries. Keep the core's `timeout` for this embedder above it, or the core kills the process before this program can report what went wrong |
+| `NAME` | `<provider>:<model>@<host>[:port][#prefixes]` | The embedder's name as the index records it. A prefix changes every vector, so a short hash of the prefixes is part of it; set your own when the model behind a name changes |
+
+`min_similarity` belongs to the model: similarity scales differ, so measure it with [`passageway eval --sweep`](https://passageway.marceltov.de/guides/evaluate/) before relying on it.
+
+## Behaviour
+
+- **The dimension** is learned at start by embedding one word (`passageway`), unless `DIMENSIONS` is set. Without `DIMENSIONS`, a service that is down when passageway starts keeps passageway in its error state until passageway is restarted; with it, passageway starts and the first `recall` succeeds once the service is up.
+- **Retries:** 429 and 5xx answers and connection errors are retried with a growing wait (1 s, doubling up to 60 s; a longer `Retry-After` is honoured) for as long as `TIMEOUT` allows: `TIMEOUT` alone bounds a request, retries included, so raise it to wait out a longer rate limit. A request that timed out while being answered is not repeated, because a paid service may already have charged for it. Any other HTTP error, and giving up, is an error reply to the core for that request; the process stays up.
+- **The key** is never printed: it is masked in every error, also when a gateway quotes the header back.
+- **Tested against a fake** of the API in this repository, not against any live service; Voyage is written from its public API documentation. Recent Ollama versions serve the OpenAI-compatible `/v1/embeddings`; older ones only `/api/embed`, which this embedder does not speak.
+- **Batches:** the core sends at most `batch_size` texts per request (32 by default); the services accept far more, so raise it for a cloud model if you like.
+
+## Install
+
+```bash
+uv tool install \
+  --with 'passageway-embedder-openai @ git+https://github.com/getpassageway/passageway#subdirectory=embedders/openai' \
+  git+https://github.com/getpassageway/passageway
+```
+
+and use that environment's interpreter in `command` (`$(uv tool dir)/passageway/bin/python`), as for a connector.
+
+## Test
+
+```bash
+uv run pytest embedders/openai        # offline, against an in-test embeddings API
+```

passageway_embedder_openai-0.3.0/README.md

@@ -0,0 +1,63 @@
+# passageway-embedder-openai
+
+A [passageway](https://github.com/getpassageway/passageway#readme) embedder for services that speak the OpenAI embeddings API: **Ollama**, **LM Studio**, **vLLM**, **OpenAI** itself, and **Voyage** (whose API differs in one field). It speaks the [embedder protocol](https://passageway.marceltov.de/reference/embedder-protocol/), so passageway runs it as `kind = "command"` and the core does not change.
+
+**Where your text goes:** every indexed passage and every question is sent to the service at `BASE_URL`. With a local Ollama or LM Studio that is your own machine; with OpenAI or Voyage it is a third party, under their terms. passageway's default stays local; this embedder is your explicit choice.
+
+## Configure
+
+```toml
+[embedder]
+kind = "command"
+command = ["/path/to/python", "-m", "passageway_embedder_openai"]
+# Ollama: nomic-embed-text wants these prefixes; DIMENSIONS lets passageway start before Ollama does
+env = { BASE_URL = "http://localhost:11434/v1", MODEL = "nomic-embed-text", DIMENSIONS = "768",
+        QUERY_PREFIX = "search_query: ", PASSAGE_PREFIX = "search_document: " }
+# min_similarity = ...                      # measure it for this model: passageway eval --sweep
+timeout = 60                                # the core's limit per request; keep it above TIMEOUT below
+
+# OpenAI
+# env = { MODEL = "text-embedding-3-small" }
+# secrets = { API_KEY = "env:OPENAI_API_KEY" }
+
+# Voyage
+# env = { PROVIDER = "voyage", MODEL = "voyage-3" }
+# secrets = { API_KEY = "env:VOYAGE_API_KEY" }
+```
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `MODEL` | required | The embedding model the service serves |
+| `BASE_URL` | `https://api.openai.com/v1`, or Voyage's for `PROVIDER = "voyage"` | The service; `POST {BASE_URL}/embeddings` is called (a trailing `/embeddings` is dropped). No credentials in the URL |
+| `API_KEY` | | Sent as `Authorization: Bearer`. Required for api.openai.com and api.voyageai.com, usually not for a local server. Put it in `secrets`, never in `env` |
+| `PROVIDER` | `openai` | `openai` for every OpenAI-compatible service, `voyage` for Voyage, which wants `input_type` (`query` or `document`) with each request |
+| `QUERY_PREFIX`, `PASSAGE_PREFIX` | | Text put in front of every question or passage; e5-style models served locally want `query: ` and `passage: ` |
+| `DIMENSIONS` | | The vector length. Set, it answers `info` without a probe, so passageway can start while the service is still down; it is also sent as `dimensions` (`output_dimension` for Voyage) for models that can shorten their vectors. The first answer is checked against it |
+| `TIMEOUT` | `50` | Seconds for one request including its retries. Keep the core's `timeout` for this embedder above it, or the core kills the process before this program can report what went wrong |
+| `NAME` | `<provider>:<model>@<host>[:port][#prefixes]` | The embedder's name as the index records it. A prefix changes every vector, so a short hash of the prefixes is part of it; set your own when the model behind a name changes |
+
+`min_similarity` belongs to the model: similarity scales differ, so measure it with [`passageway eval --sweep`](https://passageway.marceltov.de/guides/evaluate/) before relying on it.
+
+## Behaviour
+
+- **The dimension** is learned at start by embedding one word (`passageway`), unless `DIMENSIONS` is set. Without `DIMENSIONS`, a service that is down when passageway starts keeps passageway in its error state until passageway is restarted; with it, passageway starts and the first `recall` succeeds once the service is up.
+- **Retries:** 429 and 5xx answers and connection errors are retried with a growing wait (1 s, doubling up to 60 s; a longer `Retry-After` is honoured) for as long as `TIMEOUT` allows: `TIMEOUT` alone bounds a request, retries included, so raise it to wait out a longer rate limit. A request that timed out while being answered is not repeated, because a paid service may already have charged for it. Any other HTTP error, and giving up, is an error reply to the core for that request; the process stays up.
+- **The key** is never printed: it is masked in every error, also when a gateway quotes the header back.
+- **Tested against a fake** of the API in this repository, not against any live service; Voyage is written from its public API documentation. Recent Ollama versions serve the OpenAI-compatible `/v1/embeddings`; older ones only `/api/embed`, which this embedder does not speak.
+- **Batches:** the core sends at most `batch_size` texts per request (32 by default); the services accept far more, so raise it for a cloud model if you like.
+
+## Install
+
+```bash
+uv tool install \
+  --with 'passageway-embedder-openai @ git+https://github.com/getpassageway/passageway#subdirectory=embedders/openai' \
+  git+https://github.com/getpassageway/passageway
+```
+
+and use that environment's interpreter in `command` (`$(uv tool dir)/passageway/bin/python`), as for a connector.
+
+## Test
+
+```bash
+uv run pytest embedders/openai        # offline, against an in-test embeddings API
+```

passageway_embedder_openai-0.3.0/passageway_embedder_openai/__init__.py

@@ -0,0 +1 @@
+

passageway_embedder_openai-0.3.0/passageway_embedder_openai/__main__.py

@@ -0,0 +1,3 @@
+from passageway_embedder_openai.embedder import main
+
+main()

passageway_embedder_openai-0.3.0/passageway_embedder_openai/embedder.py

@@ -0,0 +1,227 @@
+"""passageway embedder for OpenAI-compatible embedding services (Ollama, LM Studio, vLLM, OpenAI) and Voyage.
+
+Speaks the embedder protocol (https://passageway.marceltov.de/reference/embedder-protocol/) on stdin/stdout; every setting
+comes from the environment and is documented in README.md.
+"""
+
+import hashlib
+import json
+import math
+import os
+import ssl
+import sys
+import time
+from collections.abc import Mapping
+from dataclasses import dataclass
+from email.utils import parsedate_to_datetime
+from typing import TextIO
+from urllib.parse import urlparse
+
+import httpx
+import truststore
+
+PROVIDERS = {"openai": "https://api.openai.com/v1", "voyage": "https://api.voyageai.com/v1"}
+CLOUD_HOSTS = {"api.openai.com", "api.voyageai.com"}
+PROBE = "passageway"  # the word embedded once at `info` to learn the dimension
+MAX_BACKOFF = 60  # the longest wait the embedder picks itself; a Retry-After may ask for more, up to TIMEOUT
+RETRY_STATUS = (429, 500, 502, 503, 504)
+RETRY_ERRORS = (httpx.ConnectError, httpx.ConnectTimeout, httpx.RemoteProtocolError, httpx.PoolTimeout)  # never a read timeout:
+# a request that timed out while being answered may already have been charged for
+
+
+class SettingsError(Exception):
+    pass
+
+
+@dataclass(frozen=True)
+class Settings:
+    provider: str
+    base_url: str
+    model: str
+    api_key: str
+    query_prefix: str
+    passage_prefix: str
+    dimensions: int | None
+    timeout: float
+    name: str
+
+
+def parse_settings(env: Mapping[str, str]) -> Settings:
+    provider = env.get("PROVIDER", "").strip() or "openai"
+    if provider not in PROVIDERS:
+        raise SettingsError(f"PROVIDER must be one of {', '.join(PROVIDERS)}, got {provider!r}")
+    model = env.get("MODEL", "").strip()
+    if not model:
+        raise SettingsError("MODEL is required: the embedding model the service serves, e.g. nomic-embed-text")
+    base_url = (env.get("BASE_URL", "").strip() or PROVIDERS[provider]).rstrip("/")
+    if base_url.endswith("/embeddings"):
+        base_url = base_url[: -len("/embeddings")]  # the endpoint is added here
+    parsed = urlparse(base_url)
+    if parsed.scheme not in ("http", "https") or not parsed.hostname:
+        raise SettingsError(f"BASE_URL must be an http(s) URL such as http://localhost:11434/v1, got {base_url!r}")
+    if parsed.username or parsed.password:
+        raise SettingsError("BASE_URL must not carry credentials: put the key in API_KEY (as a secret reference)")
+    api_key = env.get("API_KEY", "").strip()
+    if parsed.hostname in CLOUD_HOSTS and not api_key:
+        raise SettingsError(f"API_KEY is required for {parsed.hostname} (as a secret reference, never a value)")
+    if not api_key.isascii():
+        raise SettingsError("API_KEY holds a character that cannot go into an HTTP header; check for a pasted quote or space")
+    raw = env.get("DIMENSIONS", "").strip()
+    dimensions = None
+    if raw:
+        if not raw.isdecimal() or int(raw) < 1:
+            raise SettingsError(f"DIMENSIONS must be a whole number from 1, got {raw!r}")
+        dimensions = int(raw)
+    try:
+        timeout = float(env.get("TIMEOUT", "").strip() or 50)
+    except ValueError:
+        raise SettingsError(f"TIMEOUT must be a number of seconds, got {env.get('TIMEOUT')!r}") from None
+    if not math.isfinite(timeout) or timeout <= 0:
+        raise SettingsError("TIMEOUT must be a number of seconds greater than 0")
+    query_prefix, passage_prefix = env.get("QUERY_PREFIX", ""), env.get("PASSAGE_PREFIX", "")
+    name = env.get("NAME", "").strip()
+    if not name:
+        host = parsed.hostname + (f":{parsed.port}" if parsed.port else "")
+        name = f"{provider}:{model}@{host}"
+        if query_prefix or passage_prefix:  # a prefix changes every vector, so it is part of the identity
+            name += "#" + hashlib.sha256(f"{query_prefix}\0{passage_prefix}".encode()).hexdigest()[:6]
+    return Settings(provider, base_url, model, api_key, query_prefix, passage_prefix, dimensions, timeout, name)
+
+
+class ApiError(Exception):
+    pass
+
+
+def _retry_after(header: str | None, default: float) -> float:
+    if header:
+        try:
+            value = float(header)
+        except ValueError:
+            try:
+                value = (parsedate_to_datetime(header) - parsedate_to_datetime(time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime()))).total_seconds()
+            except (TypeError, ValueError):
+                value = default
+        if math.isfinite(value):
+            return max(value, 0.0)  # how long is up to the deadline
+    return max(default, 0.0)
+
+
+class Embedder:
+    def __init__(self, settings: Settings, *, sleep=time.sleep):
+        self.s = settings
+        self._sleep = sleep
+        headers = {"Authorization": f"Bearer {settings.api_key}"} if settings.api_key else {}
+        self._http = httpx.Client(headers=headers, timeout=settings.timeout, follow_redirects=False,
+                                  verify=truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT))
+        self._dim = settings.dimensions
+
+    def _mask(self, text: str) -> str:
+        return text.replace(self.s.api_key, "***") if self.s.api_key else text
+
+    def _post(self, texts: list[str], kind: str) -> list[list[float]]:
+        body: dict = {"model": self.s.model, "input": texts}
+        if self.s.dimensions:
+            body["output_dimension" if self.s.provider == "voyage" else "dimensions"] = self.s.dimensions
+        if self.s.provider == "voyage":
+            body["input_type"] = "query" if kind == "query" else "document"
+        start, waited, attempt = time.monotonic(), 0.0, 0  # the whole request, retries included, stays within TIMEOUT
+        while True:
+            # the slept seconds count even when the sleep does not pass them on the clock (an injected one)
+            remaining = self.s.timeout - max(time.monotonic() - start, waited)
+            if remaining <= 0:
+                break
+            wait = min(2 ** attempt, MAX_BACKOFF)
+            attempt += 1
+            try:
+                response = self._http.post(f"{self.s.base_url}/embeddings", json=body, timeout=remaining)
+            except RETRY_ERRORS as e:
+                response, reason = None, f"{self.s.base_url}: {type(e).__name__}: {e}"
+            except httpx.HTTPError as e:  # a read timeout or another error: not repeated
+                raise ApiError(f"{self.s.base_url}: {type(e).__name__}: {e}") from None
+            if response is not None:
+                if response.status_code in RETRY_STATUS:
+                    reason = f"HTTP {response.status_code} from {self.s.base_url}"
+                    wait = max(wait, _retry_after(response.headers.get("Retry-After"), wait))
+                elif 300 <= response.status_code < 400:
+                    raise ApiError(f"{self.s.base_url} answered with a redirect to {response.headers.get('Location', '?')}: "
+                                   "set BASE_URL to the final address")
+                elif response.status_code >= 400:
+                    detail = ""
+                    try:
+                        detail = response.json().get("error", {}).get("message", "")
+                    except (ValueError, AttributeError):
+                        pass
+                    raise ApiError(f"HTTP {response.status_code} from {self.s.base_url}: {detail or response.text[:200]}")
+                else:
+                    return self._vectors(response, len(texts))
+            if wait >= self.s.timeout - max(time.monotonic() - start, waited):
+                break
+            self._sleep(wait)
+            waited += wait
+        raise ApiError(f"giving up after {attempt} attempt(s) within {self.s.timeout:g} s: {reason}")
+
+    def _vectors(self, response: httpx.Response, expected: int) -> list[list[float]]:
+        try:
+            data = response.json()["data"]
+            rows = sorted(data, key=lambda d: d["index"])
+            vectors = [row["embedding"] for row in rows]
+        except (ValueError, KeyError, TypeError, RecursionError):
+            raise ApiError(f"the service did not answer with {{data: [{{index, embedding}}]}}: {response.text[:200]}") from None
+        if len(vectors) != expected:
+            raise ApiError(f"the service returned {len(vectors)} vectors for {expected} texts")
+        for v in vectors:
+            if not (isinstance(v, list) and v and all(isinstance(x, (int, float)) and not isinstance(x, bool)
+                                                       and math.isfinite(x) for x in v)):
+                raise ApiError("the service returned an embedding that is not a non-empty list of finite numbers")
+        if self._dim is not None and len(vectors[0]) != self._dim:
+            raise ApiError(f"the service returned vectors with {len(vectors[0])} dimensions, but DIMENSIONS (or the probe) "
+                           f"says {self._dim}: the model does not honour the setting")
+        return vectors
+
+    def embed(self, kind: str, texts: list[str]) -> list[list[float]]:
+        if not texts:
+            return []
+        prefix = self.s.query_prefix if kind == "query" else self.s.passage_prefix
+        return self._post([prefix + t for t in texts], kind)
+
+    def info(self) -> dict:
+        if self._dim is None:
+            self._dim = len(self._post([PROBE], "query")[0])
+        return {"name": self.s.name, "dim": self._dim, "protocol": "1"}
+
+    # -- the protocol loop -------------------------------------------------------------------
+
+    @classmethod
+    def run(cls, env: Mapping[str, str], stdin: TextIO, stdout: TextIO, stderr: TextIO, *, sleep=time.sleep) -> int:
+        try:
+            settings = parse_settings(env)
+        except SettingsError as e:
+            print(f"openai embedder: {e}", file=stderr)
+            return 2
+        embedder = cls(settings, sleep=sleep)
+        for line in stdin:
+            if not line.strip():
+                continue
+            try:
+                request = json.loads(line)
+                if not isinstance(request, dict):
+                    raise ValueError("a request is a JSON object")
+                if request.get("op") == "info":
+                    reply = embedder.info()
+                elif request.get("op") == "embed":
+                    kind, texts = request.get("kind"), request.get("texts")
+                    if kind not in ("query", "passage") or not isinstance(texts, list) or not all(isinstance(t, str) for t in texts):
+                        raise ValueError("embed needs kind query|passage and texts as a list of strings")
+                    reply = {"vectors": embedder.embed(kind, texts)}
+                else:
+                    reply = {"error": f"unknown op {request.get('op')!r}"}
+            except (ApiError, httpx.HTTPError) as e:
+                reply = {"error": embedder._mask(str(e))}
+            except (ValueError, KeyError, TypeError, RecursionError) as e:
+                reply = {"error": f"bad request: {e}"}
+            print(json.dumps(reply), file=stdout, flush=True)
+        return 0
+
+
+def main() -> None:
+    sys.exit(Embedder.run(os.environ, sys.stdin, sys.stdout, sys.stderr))

passageway_embedder_openai-0.3.0/pyproject.toml

@@ -0,0 +1,21 @@
+[build-system]
+requires = ["hatchling==1.33.0"]
+build-backend = "hatchling.build"
+
+[project]
+name = "passageway-embedder-openai"
+version = "0.3.0"
+description = "passageway embedder for OpenAI-compatible embedding services: Ollama, LM Studio, vLLM, OpenAI, Voyage"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "PolyForm-Noncommercial-1.0.0"
+dependencies = ["httpx>=0.28.1", "truststore>=0.10"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["passageway_embedder_openai"]
+
+[dependency-groups]
+dev = ["pytest>=8.3", "passageway"]
+
+[tool.uv.sources]
+passageway = { workspace = true }

passageway_embedder_openai-0.3.0/tests/conftest.py

@@ -0,0 +1,12 @@
+import pytest
+
+from fake_openai import FakeApi, serve
+
+
+@pytest.fixture
+def api_server():
+    api = FakeApi()
+    server = serve(api)
+    yield f"http://127.0.0.1:{server.server_port}/v1", api
+    server.shutdown()
+    server.server_close()

passageway_embedder_openai-0.3.0/tests/fake_openai.py

@@ -0,0 +1,72 @@
+"""An in-test embeddings API: the JSON shapes of POST /v1/embeddings as OpenAI, Ollama and Voyage answer them."""
+
+import hashlib
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+
+def vector(text: str, dim: int) -> list[float]:
+    """Deterministic, and different for different texts: hashed words into buckets (not normalised on purpose)."""
+    out = [0.0] * dim
+    for word in text.lower().split():
+        out[int.from_bytes(hashlib.blake2b(word.encode(), digest_size=4).digest(), "big") % dim] += 1.0
+    return out
+
+
+class FakeApi:
+    def __init__(self):
+        self.dim = 16
+        self.requests: list[dict] = []  # {"path", "headers", "body"} per request
+        self.status_errors: list[int] = []  # answer the next requests with these statuses
+        self.retry_after = "0"
+        self.models = {"nomic-embed-text", "text-embedding-3-small", "voyage-3"}
+        self.ignore_dimensions = False  # like a service that does not know the field
+        self.echo_auth = False  # answer 401 with the Authorization header quoted, like some gateways do
+        self.redirect_to = None  # answer 301 to this address
+        self.bad_body = None  # answer 200 with this raw body
+
+    def respond(self, body: dict, headers: dict) -> tuple[int, dict]:
+        if self.echo_auth:
+            return 401, {"error": {"message": f"bad key {headers.get('Authorization', '')}"}}
+        if body.get("model") not in self.models:
+            return 404, {"error": {"message": f"model '{body.get('model')}' not found", "type": "invalid_request_error"}}
+        texts = body["input"] if isinstance(body["input"], list) else [body["input"]]
+        dim = int(body.get("dimensions") or self.dim) if not self.ignore_dimensions else self.dim
+        data = [{"object": "embedding", "index": i, "embedding": vector(t, dim)} for i, t in enumerate(texts)]
+        data.reverse()  # out of order, as the API may answer: the client must sort by index
+        return 200, {"object": "list", "data": data, "model": body["model"], "usage": {"prompt_tokens": 1, "total_tokens": 1}}
+
+
+def serve(api: FakeApi):
+    class Handler(BaseHTTPRequestHandler):
+        def do_POST(self):
+            raw = self.rfile.read(int(self.headers.get("Content-Length", "0")))
+            body = json.loads(raw) if raw else {}
+            api.requests.append({"path": self.path, "headers": dict(self.headers), "body": body})
+            if api.status_errors:
+                self.send_response(api.status_errors.pop(0))
+                self.send_header("Retry-After", api.retry_after)
+                self.end_headers()
+                return
+            if api.redirect_to:
+                self.send_response(301)
+                self.send_header("Location", api.redirect_to)
+                self.end_headers()
+                return
+            if api.bad_body is not None:
+                status, payload = 200, api.bad_body
+            else:
+                status, reply = api.respond(body, dict(self.headers))
+                payload = json.dumps(reply).encode()
+            self.send_response(status)
+            self.send_header("Content-Type", "application/json")
+            self.end_headers()
+            self.wfile.write(payload)
+
+        def log_message(self, *args):
+            pass
+
+    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+    threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True).start()
+    return server

passageway_embedder_openai-0.3.0/tests/test_openai_embedder.py

@@ -0,0 +1,300 @@
+import io
+import json
+import re
+import sys
+from pathlib import Path
+
+import pytest
+
+from passageway_embedder_openai.embedder import Embedder, SettingsError
+
+README = Path(__file__).resolve().parents[1] / "README.md"
+
+
+def run(env, requests):
+    stdin = io.StringIO("".join(json.dumps(r) + "\n" for r in requests))
+    out, err = io.StringIO(), io.StringIO()
+    code = Embedder.run(env, stdin, out, err, sleep=lambda s: None)
+    return code, [json.loads(line) for line in out.getvalue().splitlines()], err.getvalue()
+
+
+@pytest.fixture
+def env(api_server):
+    url, _ = api_server
+    return {"BASE_URL": url, "MODEL": "nomic-embed-text"}
+
+
+def test_info_learns_the_dimension_by_embedding_a_probe_and_names_model_and_host(env, api_server):
+    _, api = api_server
+    code, replies, _ = run(env, [{"op": "info"}])
+    assert code == 0 and replies[0]["dim"] == 16 and replies[0]["protocol"] == "1"
+    assert replies[0]["name"].startswith("openai:nomic-embed-text@127.0.0.1:")
+    assert len(api.requests) == 1 and api.requests[0]["body"]["input"] == ["passageway"]
+
+
+def test_embed_posts_the_texts_and_returns_the_vectors_in_order(env, api_server):
+    _, api = api_server
+    code, replies, _ = run(env, [{"op": "embed", "kind": "passage", "texts": ["alpha beta", "gamma"]}])
+    assert code == 0 and len(replies[0]["vectors"]) == 2
+    assert api.requests[-1]["path"] == "/v1/embeddings" and api.requests[-1]["body"] == {"model": "nomic-embed-text", "input": ["alpha beta", "gamma"]}
+    from fake_openai import vector
+    assert replies[0]["vectors"] == [vector("alpha beta", 16), vector("gamma", 16)]  # sorted by index, as sent
+
+
+def test_the_api_key_goes_in_the_authorization_header_and_nowhere_else(env, api_server):
+    _, api = api_server
+    run({**env, "API_KEY": "sk-secret"}, [{"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert api.requests[-1]["headers"]["Authorization"] == "Bearer sk-secret"
+    run(env, [{"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert "Authorization" not in api.requests[-1]["headers"]  # a local server needs none
+
+
+def test_prefixes_are_added_per_kind(env, api_server):
+    _, api = api_server
+    run({**env, "QUERY_PREFIX": "query: ", "PASSAGE_PREFIX": "passage: "},
+        [{"op": "embed", "kind": "query", "texts": ["q"]}, {"op": "embed", "kind": "passage", "texts": ["p"]}])
+    assert [r["body"]["input"] for r in api.requests[-2:]] == [["query: q"], ["passage: p"]]
+
+
+def test_dimensions_is_sent_and_answers_info_without_a_probe(env, api_server):
+    _, api = api_server
+    code, replies, _ = run({**env, "DIMENSIONS": "8"}, [{"op": "info"}, {"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert code == 0 and replies[0]["dim"] == 8 and len(replies[1]["vectors"][0]) == 8
+    assert len(api.requests) == 1 and api.requests[0]["body"]["dimensions"] == 8
+
+
+def test_voyage_sends_input_type_and_defaults_to_its_own_url(env, api_server):
+    _, api = api_server
+    run({**env, "PROVIDER": "voyage", "MODEL": "voyage-3"},
+        [{"op": "embed", "kind": "query", "texts": ["q"]}, {"op": "embed", "kind": "passage", "texts": ["p"]}])
+    assert [r["body"]["input_type"] for r in api.requests[-2:]] == ["query", "document"]
+    _, replies, _ = run({**env, "PROVIDER": "voyage", "MODEL": "voyage-3"}, [{"op": "info"}])
+    assert replies[0]["name"].startswith("voyage:voyage-3@")
+    from passageway_embedder_openai.embedder import parse_settings
+    assert parse_settings({"PROVIDER": "voyage", "MODEL": "voyage-3", "API_KEY": "k"}).base_url == "https://api.voyageai.com/v1"
+    assert parse_settings({"MODEL": "text-embedding-3-small", "API_KEY": "k"}).base_url == "https://api.openai.com/v1"
+
+
+def test_a_name_override(env):
+    _, replies, _ = run({**env, "NAME": "my-ollama-nomic"}, [{"op": "info"}])
+    assert replies[0]["name"] == "my-ollama-nomic"
+
+
+def test_a_failing_request_is_an_error_reply_and_the_process_goes_on(env, api_server):
+    _, api = api_server
+    code, replies, _ = run({**env, "MODEL": "no-such-model"},
+                           [{"op": "embed", "kind": "query", "texts": ["x"]}, {"op": "info"}])
+    assert code == 0 and "error" in replies[0] and "no-such-model" in replies[0]["error"] and "404" in replies[0]["error"]
+    assert "error" in replies[1]
+
+
+def test_the_key_never_appears_in_an_error(env, api_server):
+    _, api = api_server
+    api.status_errors = [401]
+    _, replies, err = run({**env, "API_KEY": "sk-secret"}, [{"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert "sk-secret" not in json.dumps(replies) + err and "401" in replies[0]["error"]
+
+
+def test_429_and_5xx_are_retried_with_backoff_and_retry_after(env, api_server):
+    _, api = api_server
+    api.status_errors = [429, 503, 502]
+    api.retry_after = "0"
+    code, replies, _ = run(env, [{"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert code == 0 and "vectors" in replies[0] and len(api.requests) == 4
+
+
+def test_giving_up_at_the_deadline_is_an_error_reply(env, api_server):
+    _, api = api_server
+    api.status_errors = [503] * 1000
+    _, replies, _ = run({**env, "TIMEOUT": "300"}, [{"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert "error" in replies[0] and "503" in replies[0]["error"] and "300 s" in replies[0]["error"]
+
+
+def test_more_than_five_429s_are_waited_out_within_a_long_timeout(env, api_server):
+    _, api = api_server
+    api.status_errors = [429] * 8
+    waited = []
+    stdin = io.StringIO('{"op": "embed", "kind": "query", "texts": ["x"]}\n')
+    out = io.StringIO()
+    Embedder.run({**env, "TIMEOUT": "1000"}, stdin, out, io.StringIO(), sleep=waited.append)
+    assert "vectors" in json.loads(out.getvalue()) and len(api.requests) == 9 and sum(waited) > 120
+
+
+def test_a_retry_after_beyond_30_seconds_is_honoured_within_timeout(env, api_server):
+    _, api = api_server
+    api.status_errors, api.retry_after = [429], "200"
+    waited = []
+    stdin = io.StringIO('{"op": "embed", "kind": "query", "texts": ["x"]}\n')
+    out = io.StringIO()
+    Embedder.run({**env, "TIMEOUT": "1000"}, stdin, out, io.StringIO(), sleep=waited.append)
+    assert "vectors" in json.loads(out.getvalue()) and waited == [200]
+
+
+def test_a_server_that_is_not_there_is_an_error_reply_not_a_crash():
+    _, replies, _ = run({"BASE_URL": "http://127.0.0.1:9/v1", "MODEL": "m"}, [{"op": "info"}])
+    assert "error" in replies[0] and "127.0.0.1:9" in replies[0]["error"]
+
+
+@pytest.mark.parametrize("env,message", [
+    ({}, "MODEL"),
+    ({"MODEL": "m", "BASE_URL": "ftp://x"}, "BASE_URL"),
+    ({"MODEL": "m", "PROVIDER": "cohere"}, "PROVIDER"),
+    ({"MODEL": "m", "BASE_URL": "http://localhost:11434/v1", "DIMENSIONS": "big"}, "DIMENSIONS"),
+    ({"MODEL": "m", "BASE_URL": "http://localhost:11434/v1", "DIMENSIONS": "0"}, "DIMENSIONS"),
+    ({"MODEL": "m", "BASE_URL": "http://localhost:11434/v1", "TIMEOUT": "soon"}, "TIMEOUT"),
+    ({"MODEL": "m", "PROVIDER": "voyage"}, "API_KEY"),  # a cloud provider needs a key
+    ({"MODEL": "m"}, "API_KEY"),  # api.openai.com too
+])
+def test_bad_settings_exit_2_with_a_message_and_no_output(env, message):
+    stdin, out, err = io.StringIO('{"op": "info"}\n'), io.StringIO(), io.StringIO()
+    assert Embedder.run(env, stdin, out, err) == 2
+    assert out.getvalue() == "" and message in err.getvalue()
+
+
+def test_an_empty_text_list_is_an_empty_reply_without_a_request(env, api_server):
+    _, api = api_server
+    _, replies, _ = run(env, [{"op": "embed", "kind": "query", "texts": []}])
+    assert replies == [{"vectors": []}] and api.requests == []
+
+
+def test_every_setting_is_in_the_readme():
+    code = (Path(__file__).resolve().parents[1] / "passageway_embedder_openai" / "embedder.py").read_text()
+    names = sorted(set(re.findall(r'env\.get\(\s*"([A-Z_]+)"', code)))
+    assert len(names) >= 8, names
+    readme = README.read_text()
+    for name in names:
+        assert f"`{name}`" in readme, f"{name} is read by the embedder but missing from its README"
+    assert "third party" in readme or "third-party" in readme  # the privacy sentence
+
+
+def test_the_contract_check_passes_as_a_real_subprocess(api_server):
+    url, _ = api_server
+    from passageway.embed_command import CommandEmbedder
+    from passageway.testing.embedder_contract import check_embedder
+    embedder = CommandEmbedder("openai", [sys.executable, "-m", "passageway_embedder_openai"],
+                               env={"BASE_URL": url, "MODEL": "nomic-embed-text"})
+    try:
+        check_embedder(embedder)
+        assert embedder.dim == 16
+    finally:
+        embedder.close()
+
+
+# ---- review findings ------------------------------------------------------------------------------
+
+def test_dimensions_is_checked_against_what_the_service_really_returns(env, api_server):
+    _, api = api_server
+    api.ignore_dimensions = True
+    _, replies, _ = run({**env, "DIMENSIONS": "8"}, [{"op": "info"}, {"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert replies[0]["dim"] == 8 and "error" in replies[1] and "8" in replies[1]["error"] and "16" in replies[1]["error"]
+
+
+def test_voyage_sends_output_dimension_not_dimensions(env, api_server):
+    _, api = api_server
+    run({**env, "PROVIDER": "voyage", "MODEL": "voyage-3", "DIMENSIONS": "8"}, [{"op": "embed", "kind": "query", "texts": ["x"]}])
+    body = api.requests[-1]["body"]
+    assert body.get("output_dimension") == 8 and "dimensions" not in body
+
+
+def test_the_whole_request_including_retries_stays_within_timeout(env, api_server):
+    _, api = api_server
+    api.status_errors = [503] * 5
+    api.retry_after = "100"
+    waited = []
+    stdin = io.StringIO('{"op": "embed", "kind": "query", "texts": ["x"]}\n')
+    out, err = io.StringIO(), io.StringIO()
+    Embedder.run({**env, "TIMEOUT": "2"}, stdin, out, err, sleep=lambda s: waited.append(s))
+    reply = json.loads(out.getvalue())
+    assert "error" in reply and "503" in reply["error"]
+    assert sum(waited) <= 2 and len(api.requests) <= 3  # it gave up within the budget, not after five long waits
+
+
+def test_retry_after_is_honoured_and_clamped(env, api_server):
+    _, api = api_server
+    api.status_errors = [429]
+    api.retry_after = "3"
+    waited = []
+    stdin = io.StringIO('{"op": "embed", "kind": "query", "texts": ["x"]}\n')
+    Embedder.run({**env, "TIMEOUT": "30"}, stdin, io.StringIO(), io.StringIO(), sleep=lambda s: waited.append(s))
+    assert waited == [3.0]
+    for value in ("-5", "nan", "Wed, 21 Oct 2015 07:28:00 GMT"):
+        api.status_errors = [429]
+        api.retry_after = value
+        waited.clear()
+        stdin = io.StringIO('{"op": "embed", "kind": "query", "texts": ["x"]}\n')
+        out = io.StringIO()
+        Embedder.run({**env, "TIMEOUT": "30"}, stdin, out, io.StringIO(), sleep=lambda s: waited.append(s))
+        assert "vectors" in json.loads(out.getvalue()) and len(waited) == 1 and 0 <= waited[0] <= 30
+
+
+def test_a_connection_error_is_retried_but_a_read_timeout_is_not(env, api_server, monkeypatch):
+    import httpx
+    calls = []
+
+    class Client:
+        def __init__(self, *a, **k): pass
+        def post(self, url, json, timeout=None):
+            calls.append(json)
+            raise (httpx.ConnectError("refused") if len(calls) < 3 else httpx.ReadTimeout("slow"))
+    monkeypatch.setattr(httpx, "Client", Client)
+    _, replies, _ = run(env, [{"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert "error" in replies[0] and "ReadTimeout" in replies[0]["error"] and len(calls) == 3  # 2 retries, then no retry
+
+
+def test_a_key_quoted_by_the_service_is_masked(env, api_server):
+    _, api = api_server
+    api.echo_auth = True
+    _, replies, _ = run({**env, "API_KEY": "sk-secret"}, [{"op": "embed", "kind": "query", "texts": ["x"]}])
+    assert "sk-secret" not in json.dumps(replies) and "***" in replies[0]["error"]
+
+
+def test_credentials_in_the_base_url_are_refused():
+    out, err = io.StringIO(), io.StringIO()
+    assert Embedder.run({"MODEL": "m", "BASE_URL": "http://user:pw@localhost:11434/v1"}, io.StringIO('{"op":"info"}\n'), out, err) == 2
+    assert "API_KEY" in err.getvalue() and "pw" not in err.getvalue().replace("API_KEY", "")
+
+
+@pytest.mark.parametrize("env_extra,message", [
+    ({"DIMENSIONS": "²"}, "DIMENSIONS"), ({"TIMEOUT": "nan"}, "TIMEOUT"), ({"TIMEOUT": "inf"}, "TIMEOUT"),
+    ({"API_KEY": "sk-’"}, "API_KEY"),
+])
+def test_more_settings_that_must_be_refused_cleanly(env_extra, message):
+    out, err = io.StringIO(), io.StringIO()
+    env = {"MODEL": "m", "BASE_URL": "http://localhost:11434/v1", **env_extra}
+    assert Embedder.run(env, io.StringIO('{"op":"info"}\n'), out, err) == 2 and message in err.getvalue()
+
+
+def test_a_redirect_and_a_base_url_ending_in_embeddings_give_clear_answers(env, api_server):
+    url, api = api_server
+    api.redirect_to = "https://elsewhere.example/v1/embeddings"
+    _, replies, _ = run(env, [{"op": "info"}])
+    assert "redirect" in replies[0]["error"] and "elsewhere.example" in replies[0]["error"]
+    api.redirect_to = None
+    from passageway_embedder_openai.embedder import parse_settings
+    assert parse_settings({**env, "BASE_URL": url + "/embeddings"}).base_url == url
+
+
+@pytest.mark.parametrize("body", [b'{"data": [{"index": 0, "embedding": "abc"}]}', b'{"data": [{"index": 0, "embedding": null}]}',
+                                  b'{"data": [{"index": 0, "embedding": [NaN, 1.0]}]}', b'{"data": [{"index": 0, "embedding": []}]}',
+                                  b"[" * 100000])
+def test_a_service_answer_that_is_not_vectors_is_an_error_reply_not_a_crash(env, api_server, body):
+    _, api = api_server
+    api.bad_body = body
+    code, replies, _ = run(env, [{"op": "embed", "kind": "query", "texts": ["x"]}, {"op": "info"}])
+    assert code == 0 and len(replies) == 2 and all("error" in r for r in replies)
+
+
+@pytest.mark.parametrize("request_", ['{"op": "embed", "kind": "query", "texts": "abc"}', '{"op": "embed", "kind": "other", "texts": ["a"]}',
+                                      "[" * 100000])
+def test_a_bad_request_line_is_an_error_reply_and_the_process_goes_on(env, request_):
+    stdin = io.StringIO(request_ + '\n{"op": "info"}\n')
+    out = io.StringIO()
+    assert Embedder.run(env, stdin, out, io.StringIO(), sleep=lambda s: None) == 0
+    replies = [json.loads(line) for line in out.getvalue().splitlines()]
+    assert "error" in replies[0] and "dim" in replies[1]
+
+
+def test_the_name_changes_with_the_prefixes_and_carries_the_port(env):
+    plain = run(env, [{"op": "info"}])[1][0]["name"]
+    prefixed = run({**env, "QUERY_PREFIX": "search_query: "}, [{"op": "info"}])[1][0]["name"]
+    assert plain != prefixed and plain.startswith("openai:nomic-embed-text@127.0.0.1:") and prefixed.startswith(plain)