digline-openai 0.1.0__tar.gz

digline_openai-0.1.0/.gitignore

@@ -0,0 +1,41 @@
+# Python bytecode and build output
+__pycache__/
+*.py[cod]
+build/
+dist/
+*.egg-info/
+
+# Environment. Recreated by `uv sync`; it pins absolute paths, so it must
+# never be committed.
+.venv/
+.env
+
+# Tool caches. Each already drops its own `.gitignore`; listed here so a
+# fresh clone is clean before the tools have run once.
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+
+# IDE. Excluded because the project files carry machine-specific SDK paths
+# (`digline.iml` names the interpreter by absolute path). Drop these two lines
+# to version the shared part, and keep ignoring `.idea/workspace.xml`.
+.idea/
+*.iml
+
+# Local Claude Code settings. `.claude/settings.json`, if it appears, is shared
+# and stays versioned. `CLAUDE.local.md` is the personal working agreement —
+# how I want to be worked with — as against `CLAUDE.md`, which is the project.
+.claude/settings.local.json
+CLAUDE.local.md
+
+# macOS
+.DS_Store
+
+# Working material that stays local and is not part of the package.
+private/
+
+# NOT ignored: `.digline/`. Decision 2 — baselines are versioned, run
+# artifacts are not — and the split is enforced one level down, by the
+# `.gitignore` the store itself writes into `.digline/` (`*/runs/`).
+# Ignoring `.digline/` here would take the baselines out of git with it.
+to-publish/

digline_openai-0.1.0/PKG-INFO

@@ -0,0 +1,166 @@
+Metadata-Version: 2.5
+Name: digline-openai
+Version: 0.1.0
+Summary: OpenAI target and judges for digline, at any OpenAI-compatible endpoint.
+Author-email: Alessandro Prandini <alessandro.prandini@ict-group.it>
+License-Expression: Apache-2.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: digline>=0.1.3
+Requires-Dist: openai>=1.40
+Description-Content-Type: text/markdown
+
+# digline-openai
+
+An [OpenAI](https://openai.com) target **and judges** for
+[digline](https://pypi.org/project/digline/): a prompt file goes in, a priced
+`Response` comes out — at any OpenAI-compatible endpoint.
+
+```sh
+pip install digline-openai
+```
+
+## One argument, three providers
+
+The wire protocol is the same everywhere, so `base_url` is the only thing that
+changes. **OpenAI** — the key is read by the SDK from `OPENAI_API_KEY`, and this
+package never touches your environment:
+
+```python
+from digline_openai import OpenAITarget
+
+target = OpenAITarget("prompts/answer.md", model="gpt-5", max_tokens=1024)
+```
+
+**Azure OpenAI** — your resource's v1 endpoint, with the key passed explicitly
+because Azure names its variable something else:
+
+```python
+import os
+from digline_openai import OpenAITarget
+
+target = OpenAITarget(
+    "prompts/answer.md",
+    model="gpt-4.1",
+    max_tokens=1024,
+    base_url="https://my-resource.openai.azure.com/openai/v1",
+    api_key=os.environ["AZURE_OPENAI_API_KEY"],
+)
+```
+
+**Ollama** — no key at all, and a model that costs nothing because you are the
+one hosting it:
+
+```python
+from digline_openai import OpenAITarget, free
+
+target = OpenAITarget(
+    "prompts/answer.md",
+    model="llama3.2",
+    max_tokens=1024,
+    base_url="http://localhost:11434/v1",
+    pricing=free("llama3.2"),
+)
+```
+
+The same target covers OpenRouter, Groq, Together and a vLLM in your own VPC. Nothing here is a gateway or an abstraction layer: it is the `openai` SDK
+with its own `base_url` argument, which is what that argument is for.
+
+## The judge runs in your perimeter too
+
+A plugin is a target **and** a judge ([ADR
+0004](https://github.com/digline/digline/blob/main/docs/adr/0004-every-plugin-is-a-target-and-a-judge.md)).
+The point is not convenience: what a judge is sent is the model's *output*, so a
+judge that lives at somebody else's API takes the payload out of the perimeter
+it was generated in, and no setting in the suite would say so.
+
+```python
+from digline.core import LlmRubric
+from digline_openai import OpenAIJudge
+
+judge = OpenAIJudge(model="gpt-5-mini")
+rubric = LlmRubric(
+    rubric="The answer is one sentence and cites the passage it came from.",
+    judge=judge,
+    threshold=0.8,
+    tolerance=0.05,
+)
+```
+
+`Faithfulness` asks a judge to decompose rather than to score — how many claims
+the output makes, how many the context supports — so it takes the other one:
+
+```python
+from digline.core import Faithfulness
+from digline_openai import OpenAIClaimJudge
+
+faithful = Faithfulness(
+    judge=OpenAIClaimJudge(model="gpt-5-mini"),
+    threshold=0.9,
+    tolerance=0.05,
+)
+```
+
+Both take the same `base_url` and `api_key` as the target, so judging an Ollama
+run on that same Ollama is one argument:
+
+```python
+from digline_openai import OpenAIJudge, free
+
+local = OpenAIJudge(
+    model="llama3.2",
+    base_url="http://localhost:11434/v1",
+    pricing=free("llama3.2"),
+)
+```
+
+## What judging cost
+
+The target's cost lands on the `Response` and in the run. A judge's does not —
+it is counted on the judge, and it is not reset:
+
+```python
+from digline_openai import OpenAIJudge
+
+judge = OpenAIJudge(model="gpt-5-mini")
+print(f"{judge.calls} judgements, {judge.spent_usd:.4f} USD, {judge.latency_ms:.0f} ms")
+```
+
+A suite with `samples=5` and `Repeated(n=3)` makes fifteen judging calls per
+case, so this is not a rounding error. For a per-run figure, read it before and
+after and subtract. It is in-process only today, and ADR 0004 §3 says what it
+would take to put it in the report.
+
+## The details that bite
+
+**Cached tokens.** OpenAI counts cached prompt tokens *inside* `prompt_tokens`.
+They are subtracted before pricing, so the discounted half is not also billed at
+the full rate — the opposite convention to Anthropic, and getting it wrong is
+invisible in the direction of good news.
+
+**`max_tokens` vs `max_completion_tokens`.** The official API rejects
+`max_tokens` for GPT-5 and the o-series; most compatible servers accept it and
+silently ignore `max_completion_tokens`, which generates without a cap and bills
+for it. So: `max_completion_tokens` when `base_url` is unset, `max_tokens`
+otherwise. Override with `token_param="max_tokens"` when your server disagrees.
+
+**JSON.** The judges ask for `{"type": "json_object"}` where it is supported. A
+provider that refuses it is retried once without it, the fallback is remembered,
+and the reply is parsed leniently either way — a fenced block or a sentence in
+front of the object both read correctly.
+
+**Prices.** `OPENAI_PRICING` carries the day it was copied and is one argument
+to replace; an unknown model raises at `preflight` rather than costing nothing.
+`free("llama3.2")` is how you say a self-hosted model really is free, out loud.
+
+**Keys.** Passed explicitly or resolved by the SDK from the environment — this
+package contains no `os.environ` and no `getenv`, and a test enforces it. Only
+when `base_url` is custom *and* the SDK found nothing does the client fall back
+to the obviously-fake `digline-no-key`, for local servers that ignore it.
+
+Apache-2.0. Docs: [digline/digline](https://github.com/digline/digline).

digline_openai-0.1.0/README.md

@@ -0,0 +1,149 @@
+# digline-openai
+
+An [OpenAI](https://openai.com) target **and judges** for
+[digline](https://pypi.org/project/digline/): a prompt file goes in, a priced
+`Response` comes out — at any OpenAI-compatible endpoint.
+
+```sh
+pip install digline-openai
+```
+
+## One argument, three providers
+
+The wire protocol is the same everywhere, so `base_url` is the only thing that
+changes. **OpenAI** — the key is read by the SDK from `OPENAI_API_KEY`, and this
+package never touches your environment:
+
+```python
+from digline_openai import OpenAITarget
+
+target = OpenAITarget("prompts/answer.md", model="gpt-5", max_tokens=1024)
+```
+
+**Azure OpenAI** — your resource's v1 endpoint, with the key passed explicitly
+because Azure names its variable something else:
+
+```python
+import os
+from digline_openai import OpenAITarget
+
+target = OpenAITarget(
+    "prompts/answer.md",
+    model="gpt-4.1",
+    max_tokens=1024,
+    base_url="https://my-resource.openai.azure.com/openai/v1",
+    api_key=os.environ["AZURE_OPENAI_API_KEY"],
+)
+```
+
+**Ollama** — no key at all, and a model that costs nothing because you are the
+one hosting it:
+
+```python
+from digline_openai import OpenAITarget, free
+
+target = OpenAITarget(
+    "prompts/answer.md",
+    model="llama3.2",
+    max_tokens=1024,
+    base_url="http://localhost:11434/v1",
+    pricing=free("llama3.2"),
+)
+```
+
+The same target covers OpenRouter, Groq, Together and a vLLM in your own VPC. Nothing here is a gateway or an abstraction layer: it is the `openai` SDK
+with its own `base_url` argument, which is what that argument is for.
+
+## The judge runs in your perimeter too
+
+A plugin is a target **and** a judge ([ADR
+0004](https://github.com/digline/digline/blob/main/docs/adr/0004-every-plugin-is-a-target-and-a-judge.md)).
+The point is not convenience: what a judge is sent is the model's *output*, so a
+judge that lives at somebody else's API takes the payload out of the perimeter
+it was generated in, and no setting in the suite would say so.
+
+```python
+from digline.core import LlmRubric
+from digline_openai import OpenAIJudge
+
+judge = OpenAIJudge(model="gpt-5-mini")
+rubric = LlmRubric(
+    rubric="The answer is one sentence and cites the passage it came from.",
+    judge=judge,
+    threshold=0.8,
+    tolerance=0.05,
+)
+```
+
+`Faithfulness` asks a judge to decompose rather than to score — how many claims
+the output makes, how many the context supports — so it takes the other one:
+
+```python
+from digline.core import Faithfulness
+from digline_openai import OpenAIClaimJudge
+
+faithful = Faithfulness(
+    judge=OpenAIClaimJudge(model="gpt-5-mini"),
+    threshold=0.9,
+    tolerance=0.05,
+)
+```
+
+Both take the same `base_url` and `api_key` as the target, so judging an Ollama
+run on that same Ollama is one argument:
+
+```python
+from digline_openai import OpenAIJudge, free
+
+local = OpenAIJudge(
+    model="llama3.2",
+    base_url="http://localhost:11434/v1",
+    pricing=free("llama3.2"),
+)
+```
+
+## What judging cost
+
+The target's cost lands on the `Response` and in the run. A judge's does not —
+it is counted on the judge, and it is not reset:
+
+```python
+from digline_openai import OpenAIJudge
+
+judge = OpenAIJudge(model="gpt-5-mini")
+print(f"{judge.calls} judgements, {judge.spent_usd:.4f} USD, {judge.latency_ms:.0f} ms")
+```
+
+A suite with `samples=5` and `Repeated(n=3)` makes fifteen judging calls per
+case, so this is not a rounding error. For a per-run figure, read it before and
+after and subtract. It is in-process only today, and ADR 0004 §3 says what it
+would take to put it in the report.
+
+## The details that bite
+
+**Cached tokens.** OpenAI counts cached prompt tokens *inside* `prompt_tokens`.
+They are subtracted before pricing, so the discounted half is not also billed at
+the full rate — the opposite convention to Anthropic, and getting it wrong is
+invisible in the direction of good news.
+
+**`max_tokens` vs `max_completion_tokens`.** The official API rejects
+`max_tokens` for GPT-5 and the o-series; most compatible servers accept it and
+silently ignore `max_completion_tokens`, which generates without a cap and bills
+for it. So: `max_completion_tokens` when `base_url` is unset, `max_tokens`
+otherwise. Override with `token_param="max_tokens"` when your server disagrees.
+
+**JSON.** The judges ask for `{"type": "json_object"}` where it is supported. A
+provider that refuses it is retried once without it, the fallback is remembered,
+and the reply is parsed leniently either way — a fenced block or a sentence in
+front of the object both read correctly.
+
+**Prices.** `OPENAI_PRICING` carries the day it was copied and is one argument
+to replace; an unknown model raises at `preflight` rather than costing nothing.
+`free("llama3.2")` is how you say a self-hosted model really is free, out loud.
+
+**Keys.** Passed explicitly or resolved by the SDK from the environment — this
+package contains no `os.environ` and no `getenv`, and a test enforces it. Only
+when `base_url` is custom *and* the SDK found nothing does the client fall back
+to the obviously-fake `digline-no-key`, for local servers that ignore it.
+
+Apache-2.0. Docs: [digline/digline](https://github.com/digline/digline).

digline_openai-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[project]
+name = "digline-openai"
+version = "0.1.0"
+description = "OpenAI target and judges for digline, at any OpenAI-compatible endpoint."
+readme = "README.md"
+requires-python = ">=3.12"
+license = "Apache-2.0"
+authors = [
+    { name = "Alessandro Prandini", email = "alessandro.prandini@ict-group.it" },
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Testing",
+    "Typing :: Typed",
+]
+dependencies = [
+    # Pinned to a version that exists, like the Anthropic plugin: unpinned was
+    # fine while the two were built together in one workspace; published, it
+    # would let a resolver pick something older than the API this plugin is
+    # written against — `JudgeBase` and its two subclasses are 0.1.3.
+    "digline>=0.1.3",
+    # The SDK and nothing else. No LiteLLM, no gateway, no abstraction over the
+    # abstraction: `base_url` is what makes one SDK cover every compatible
+    # provider, and a layer on top would only add a second thing to be wrong.
+    "openai>=1.40",
+]
+
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/digline_openai"]
+
+[tool.uv.sources]
+digline = { workspace = true }

digline_openai-0.1.0/src/digline_openai/__init__.py

@@ -0,0 +1,25 @@
+"""OpenAI target and judges for digline, at any OpenAI-compatible endpoint.
+
+Installed beside digline, never inside it: `pip install digline` must not pull
+somebody's HTTP client along with it.
+
+A plugin is a target **and** a judge (ADR 0004), so a suite can generate and
+judge in one perimeter, with one key and one price list — including when that
+perimeter is an Azure deployment, a vLLM in a VPC or an Ollama on a laptop.
+"""
+
+from digline_openai.client import NO_KEY, OpenAIChat
+from digline_openai.judge import OpenAIClaimJudge, OpenAIJudge
+from digline_openai.pricing import OPENAI_PRICING, PRICES_READ_ON, free
+from digline_openai.target import OpenAITarget
+
+__all__ = [
+    "NO_KEY",
+    "OPENAI_PRICING",
+    "PRICES_READ_ON",
+    "OpenAIChat",
+    "OpenAIClaimJudge",
+    "OpenAIJudge",
+    "OpenAITarget",
+    "free",
+]

digline_openai-0.1.0/src/digline_openai/client.py

@@ -0,0 +1,229 @@
+"""The client half: one chat call, shared by the target and the two judges.
+
+The SDK is imported on first use, not at module scope. A suite that only wants
+to be *loaded* — `digline list`, a preflight, a test — should not need `openai`
+installed, and lazy import is what makes that true.
+
+**No key is read here.** There is no `os.environ` and no `getenv` in this
+package, and the test suite enforces it. What the SDK reads on its own is the
+SDK's business; see `build_client` for the one place where an absent key is
+tolerated, and why it is only reachable behind a custom `base_url`.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping, Sequence
+from typing import Any, Literal
+
+from digline.targets import Pricing, Usage
+
+__all__ = ["NO_KEY", "OpenAIChat", "TokenParam", "build_client", "usage_of"]
+
+#: Passed as the key when — and only when — a custom `base_url` is set and the
+#: SDK found nothing in the environment. Ollama and most self-hosted servers do
+#: not look at it; anything that does rejects it with an authentication error,
+#: which is the right error. It is written to be unmistakably not a credential:
+#: whoever finds it in a log has found a placeholder, not a leak.
+NO_KEY = "digline-no-key"
+
+#: Which argument carries the output cap. See `OpenAIChat.complete`.
+TokenParam = Literal["auto", "max_tokens", "max_completion_tokens"]
+
+
+def build_client(base_url: str | None, api_key: str | None) -> Any:
+    """The SDK client, with the key resolution rule of ADR 0004 §5.
+
+    Three cases, in this order, and the order is the whole trick:
+
+    1. a key was passed — it is used;
+    2. no key and the **official** endpoint — `None` goes to the SDK, which
+       resolves `OPENAI_API_KEY` itself and raises its own message if there is
+       none. That message is clearer than anything this package could write,
+       and it keeps the environment out of our source;
+    3. no key, a **custom** `base_url`, and the SDK found nothing — the client
+       is rebuilt with `NO_KEY`, because a local server has no key to give.
+
+    Case 3 is reached only *after* the SDK has looked, so pointing at OpenRouter
+    with `OPENAI_API_KEY` set still authenticates with the real key. This
+    function never learns whether one exists.
+    """
+    import openai
+
+    kwargs: dict[str, Any] = {}
+    if base_url is not None:
+        kwargs["base_url"] = base_url
+    if api_key is not None:
+        kwargs["api_key"] = api_key
+    try:
+        return openai.OpenAI(**kwargs)
+    except openai.OpenAIError:
+        if api_key is not None or base_url is None:
+            raise
+        return openai.OpenAI(base_url=base_url, api_key=NO_KEY)
+
+
+def usage_of(reply: Any, model: str, pricing: Pricing) -> Usage:
+    """Tokens out of a chat completion, with the cached ones subtracted.
+
+    OpenAI counts cached prompt tokens **inside** `prompt_tokens` — the opposite
+    of Anthropic, where a cache write is not in `input_tokens` at all (friction
+    25). Adding the two straight would bill the cached half twice, at the full
+    rate and again at the discounted one, so the cached count comes off the
+    input before the `Usage` is built.
+
+    A provider that reports no usage at all is refused, unless the model is
+    priced at zero anyway — which is to say unless you told us, with `free()`,
+    that this one costs nothing. Anything else would report a run as cheaper
+    than it was, and that is the failure that reads as good news.
+    """
+    usage = getattr(reply, "usage", None)
+    if usage is None:
+        if _is_free(model, pricing):
+            return Usage(input_tokens=0, output_tokens=0)
+        raise ValueError(
+            f"the provider returned no usage for model {model!r}, so this call "
+            "cannot be priced. If it is a model you host and it costs nothing, "
+            "say so: `pricing=free(...)`"
+        )
+    prompt_tokens = int(getattr(usage, "prompt_tokens", 0) or 0)
+    details = getattr(usage, "prompt_tokens_details", None)
+    cached = int(getattr(details, "cached_tokens", 0) or 0)
+    return Usage(
+        input_tokens=max(prompt_tokens - cached, 0),
+        output_tokens=int(getattr(usage, "completion_tokens", 0) or 0),
+        cache_read_tokens=cached,
+        # No cache-write charge on this API: there is nothing to count, which
+        # is why the price list leaves the rate at `None` rather than at zero.
+        cache_write_tokens=0,
+    )
+
+
+def _is_free(model: str, pricing: Pricing) -> bool:
+    price = pricing.per_model.get(model)
+    return price is not None and not any(
+        (
+            price.input_per_mtok,
+            price.output_per_mtok,
+            price.cache_read_per_mtok,
+            price.cache_write_per_mtok,
+        )
+    )
+
+
+class OpenAIChat:
+    """A lazily built client and the one call every OpenAI-compatible server has.
+
+    Held by the target and by both judges, so the key resolution, the token
+    argument and the `response_format` fallback are decided once for the whole
+    package rather than three times.
+    """
+
+    def __init__(
+        self,
+        *,
+        base_url: str | None = None,
+        api_key: str | None = None,
+        client: Any = None,
+    ) -> None:
+        self.base_url = base_url
+        self._api_key = api_key
+        self._injected = client
+        #: Set once, the first time a provider refuses `response_format`. The
+        #: fallback is remembered rather than rediscovered: one wasted call per
+        #: process, not one per judgement.
+        self.json_mode_refused = False
+
+    def __repr__(self) -> str:
+        """The endpoint, never the key.
+
+        Explicit because the default `repr` is only safe by accident, and a
+        `repr` is what ends up in a pytest failure, a log line and a traceback.
+        """
+        return f"{type(self).__name__}(base_url={self.base_url!r})"
+
+    def client(self) -> Any:
+        if self._injected is None:
+            self._injected = build_client(self.base_url, self._api_key)
+        return self._injected
+
+    def token_argument(self, token_param: TokenParam) -> str:
+        """Which of the two names carries the output cap.
+
+        `"auto"` is `max_completion_tokens` on the official endpoint and
+        `max_tokens` everywhere else, and the asymmetry is not ours: the
+        official API **rejects** `max_tokens` for the GPT-5 and o-series
+        models, while most compatible servers accept `max_tokens` and quietly
+        *ignore* `max_completion_tokens` — which does not fail, it just
+        generates without a cap and bills for it.
+
+        A guess in one direction is an error you see; in the other, a cost you
+        do not. Pass the name explicitly when your server disagrees.
+        """
+        if token_param != "auto":
+            return token_param
+        return "max_tokens" if self.base_url is not None else "max_completion_tokens"
+
+    def complete(
+        self,
+        *,
+        model: str,
+        messages: Sequence[Mapping[str, Any]],
+        max_tokens: int,
+        pricing: Pricing,
+        temperature: float | None = None,
+        response_format: Mapping[str, Any] | None = None,
+        token_param: TokenParam = "auto",
+        extra_body: Mapping[str, Any] | None = None,
+    ) -> tuple[str, Usage]:
+        """One chat completion: the text and what it cost in tokens.
+
+        `response_format` is sent when asked for and **never required** (ADR
+        0004 §4). A provider that rejects it — Ollama does, some vLLM builds do
+        — is retried once without it, and the fallback is remembered. That
+        means one call's worth of latency the first time, and a judge that
+        works on an endpoint nobody tested it against.
+        """
+        request: dict[str, Any] = {
+            "model": model,
+            "messages": list(messages),
+            self.token_argument(token_param): max_tokens,
+        }
+        if temperature is not None:
+            request["temperature"] = temperature
+        if extra_body:
+            request.update(extra_body)
+
+        wants_json = response_format is not None and not self.json_mode_refused
+        if wants_json and response_format is not None:
+            request["response_format"] = dict(response_format)
+
+        try:
+            reply = self.client().chat.completions.create(**request)
+        except Exception:
+            # Retried only when `response_format` was in the request: without
+            # it there is nothing to fall back to, and swallowing the exception
+            # would hide an auth or a rate-limit error behind a second identical
+            # failure. With it, the second attempt either works or raises the
+            # error that was really there.
+            if not wants_json:
+                raise
+            self.json_mode_refused = True
+            del request["response_format"]
+            reply = self.client().chat.completions.create(**request)
+
+        return _text_of(reply), usage_of(reply, model, pricing)
+
+
+def _text_of(reply: Any) -> str:
+    choices: Any = getattr(reply, "choices", None) or []
+    if not choices:
+        raise ValueError(
+            "the provider returned no choices: there is no output to judge or "
+            "to assert on"
+        )
+    message: Any = getattr(choices[0], "message", None)
+    # `None` rather than missing when the model produced nothing — a refusal, or
+    # a cap hit before the first token. Empty text is an output the assertions
+    # can fail; an exception here would make it an `error` instead, which says
+    # the run could not be judged rather than that the model said nothing.
+    return str(getattr(message, "content", None) or "")