agentprdiff 0.2.4__tar.gz → 0.3.0__tar.gz

{agentprdiff-0.2.4 → agentprdiff-0.3.0}/.gitignore

@@ -31,3 +31,7 @@ env/
 .agentprdiff/runs/
 .agentprdiff/cache/
 .pypirc
+
+# Engine worktree nested inside Studio. Not committed — engine source
+# already lives at the repo root.
+studio/.claude/

{agentprdiff-0.2.4 → agentprdiff-0.3.0}/CHANGELOG.md

@@ -8,6 +8,58 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.0] — 2026-05-17
+
+Adds a model-override hook on the adapters package. This unblocks
+tooling that wants to run a single suite against multiple models — e.g.
+the Studio's multi-model benchmark — without forcing user-written agent
+code to take a `model` parameter.
+
+The change is purely additive: no existing call site has to change, and
+the 155-test suite passes unchanged. The override is module-level and
+process-wide, read at call time inside the patched `create()` function.
+
+### Added
+
+- `agentprdiff.adapters.set_default_model(model: str | None) -> None`:
+  set a process-wide override that rewrites `kwargs["model"]` on every
+  subsequent `instrument_client`-patched `create()` call. Pass `None` to
+  clear.
+- `agentprdiff.adapters.get_default_model() -> str | None`: read the
+  current override.
+- The OpenAI adapter's sync + async patched-create paths and the
+  Anthropic adapter's patched-create now consult the override at call
+  time and rewrite kwargs before delegating to the SDK. No-op when no
+  override is active.
+
+### Notes
+
+- The override is process-wide. Sequencing `set_default_model("A"); run();
+  set_default_model("B"); run(); set_default_model(None)` works as
+  expected. Concurrent runs in the same process share the override —
+  isolate per-leg benchmarks in fresh subprocesses if you need
+  concurrency.
+- Only rewrites when the caller actually passed a `model=` kwarg, so the
+  SDK's "no model provided" error path still surfaces correctly.
+- The override is exported from `agentprdiff.adapters` so callers don't
+  need to reach into submodules.
+
+## [0.2.5] — 2026-04-30
+
+Infrastructure-only release. Code is identical to 0.2.4 — this exists
+solely to publish via PyPI's Trusted Publishing flow (OIDC from GitHub
+Actions) so the project page shows verified details for the source
+repository. No API token is used to upload this release.
+
+### Internal
+
+- Added `.github/workflows/release.yml` that builds sdist + wheel and
+  publishes via `pypa/gh-action-pypi-publish` with OIDC. Triggered on
+  GitHub `release: published` and `workflow_dispatch`. Uses the `pypi`
+  GitHub environment for tag-restricted deploys (`v*`).
+- First release published through the configured PyPI Trusted Publisher
+  for `vnageshwaran-de/agentprdiff` → `release.yml` → `pypi` env.
+
 ## [0.2.4] — 2026-04-29
 
 Metadata-only release. Code is identical to 0.2.3 — this exists solely

{agentprdiff-0.2.4 → agentprdiff-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentprdiff
-Version: 0.2.4
+Version: 0.3.0
 Summary: Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
 Project-URL: Homepage, https://agentprdiff.dev
 Project-URL: Documentation, https://agentprdiff.dev
@@ -42,7 +42,7 @@ Description-Content-Type: text/markdown
 
 **Guard your LLM agents in CI.** Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
 
-📚 **[Documentation: agentprdiff.dev](https://agentprdiff.dev/)**  ·  ⚡ [Quickstart](https://agentprdiff.dev/quickstart/)  ·  🤖 [AI-agent adoption](https://agentprdiff.dev/quickstart/#path-a--let-an-ai-agent-adopt-the-package-for-you-recommended)  ·  📦 [PyPI](https://pypi.org/project/agentprdiff/)
+📚 **[Documentation: agentprdiff.dev](https://agentprdiff.dev/)**  ·  ⚡ [Quickstart](https://agentprdiff.dev/quickstart/)  ·  🤖 [AI-agent adoption](https://agentprdiff.dev/quickstart/#path-a)  ·  📦 [PyPI](https://pypi.org/project/agentprdiff/)
 
 > You upgraded Claude. You tweaked a system prompt. You swapped `gpt-4o` for `gpt-4o-mini` in the cheap path. Which of your agent's behaviors just changed? `agentprdiff` tells you — before the PR merges.
 
@@ -50,6 +50,15 @@ Description-Content-Type: text/markdown
 pip install agentprdiff
 ```
 
+> **Don't have Python 3.10+ yet?** Step-by-step install instructions for
+> [macOS, Windows, and Linux](https://agentprdiff.dev/installation/#install-python-310-first-if-you-dont-have-it).
+>
+> **Multiple Python versions on your machine?** If `pip install` reports
+> `No matching distribution found` even after installing Python 3.10+,
+> use `python3.12 -m pip install agentprdiff` (substitute your installed
+> 3.10+ binary). Sidesteps `$PATH` confusion when Homebrew's Python and
+> the system Python coexist. Full troubleshooting: [Installation guide](https://agentprdiff.dev/installation/).
+
 [![CI](https://github.com/vnageshwaran-de/agentprdiff/actions/workflows/ci.yml/badge.svg)](https://github.com/vnageshwaran-de/agentprdiff/actions/workflows/ci.yml)
 [![PyPI](https://img.shields.io/pypi/v/agentprdiff.svg)](https://pypi.org/project/agentprdiff/)
 [![Python](https://img.shields.io/pypi/pyversions/agentprdiff.svg)](https://pypi.org/project/agentprdiff/)

{agentprdiff-0.2.4 → agentprdiff-0.3.0}/README.md

@@ -2,7 +2,7 @@
 
 **Guard your LLM agents in CI.** Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
 
-📚 **[Documentation: agentprdiff.dev](https://agentprdiff.dev/)**  ·  ⚡ [Quickstart](https://agentprdiff.dev/quickstart/)  ·  🤖 [AI-agent adoption](https://agentprdiff.dev/quickstart/#path-a--let-an-ai-agent-adopt-the-package-for-you-recommended)  ·  📦 [PyPI](https://pypi.org/project/agentprdiff/)
+📚 **[Documentation: agentprdiff.dev](https://agentprdiff.dev/)**  ·  ⚡ [Quickstart](https://agentprdiff.dev/quickstart/)  ·  🤖 [AI-agent adoption](https://agentprdiff.dev/quickstart/#path-a)  ·  📦 [PyPI](https://pypi.org/project/agentprdiff/)
 
 > You upgraded Claude. You tweaked a system prompt. You swapped `gpt-4o` for `gpt-4o-mini` in the cheap path. Which of your agent's behaviors just changed? `agentprdiff` tells you — before the PR merges.
 
@@ -10,6 +10,15 @@
 pip install agentprdiff
 ```
 
+> **Don't have Python 3.10+ yet?** Step-by-step install instructions for
+> [macOS, Windows, and Linux](https://agentprdiff.dev/installation/#install-python-310-first-if-you-dont-have-it).
+>
+> **Multiple Python versions on your machine?** If `pip install` reports
+> `No matching distribution found` even after installing Python 3.10+,
+> use `python3.12 -m pip install agentprdiff` (substitute your installed
+> 3.10+ binary). Sidesteps `$PATH` confusion when Homebrew's Python and
+> the system Python coexist. Full troubleshooting: [Installation guide](https://agentprdiff.dev/installation/).
+
 [![CI](https://github.com/vnageshwaran-de/agentprdiff/actions/workflows/ci.yml/badge.svg)](https://github.com/vnageshwaran-de/agentprdiff/actions/workflows/ci.yml)
 [![PyPI](https://img.shields.io/pypi/v/agentprdiff.svg)](https://pypi.org/project/agentprdiff/)
 [![Python](https://img.shields.io/pypi/pyversions/agentprdiff.svg)](https://pypi.org/project/agentprdiff/)

{agentprdiff-0.2.4 → agentprdiff-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentprdiff"
-version = "0.2.4"
+version = "0.3.0"
 description = "Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change."
 readme = "README.md"
 requires-python = ">=3.10"

{agentprdiff-0.2.4 → agentprdiff-0.3.0}/src/agentprdiff/__init__.py

@@ -67,7 +67,7 @@ from .differ import AssertionChange, TraceDelta, diff_traces
 from .runner import CaseReport, Runner, RunReport
 from .store import BaselineStore
 
-__version__ = "0.2.4"
+__version__ = "0.2.5"
 
 __all__ = [
     # core

agentprdiff-0.3.0/src/agentprdiff/adapters/__init__.py

@@ -0,0 +1,86 @@
+"""SDK adapters for agentprdiff.
+
+The adapters take an agent that uses a real LLM SDK (OpenAI, Anthropic, or any
+OpenAI-compatible provider like Groq, Gemini's openai-compat endpoint,
+OpenRouter, Ollama, or vLLM) and capture every model call as an `LLMCall` on a
+`Trace` — without forcing the user to rewrite their agent loop.
+
+The pattern is::
+
+    from agentprdiff.adapters.openai import instrument_client, instrument_tools
+
+    def my_agent(query: str):
+        client = OpenAI(...)
+        with instrument_client(client) as trace:
+            tools = instrument_tools(TOOL_MAP, trace)
+            # ... user's existing tool-calling loop, untouched ...
+            return final_text, trace
+
+Submodules are imported lazily so the base `agentprdiff` install doesn't pull
+in `openai` / `anthropic` unless the user opts in via the extras::
+
+    pip install "agentprdiff[openai]"
+    pip install "agentprdiff[anthropic]"
+
+See `docs/adapters.md` for the full reference.
+"""
+
+from __future__ import annotations
+
+# Re-export pricing helpers — these are pure-Python and have no SDK dependency.
+from .pricing import (
+    DEFAULT_PRICES,
+    PriceTable,
+    estimate_cost_usd,
+    register_prices,
+)
+
+# ---------------------------------------------------------------------------
+# Optional global model override.
+#
+# When set via ``set_default_model("gpt-4o-mini")``, every subsequent
+# ``instrument_client``-patched ``create()`` call rewrites the ``model``
+# keyword argument before delegating to the underlying SDK. Pass ``None``
+# to clear.
+#
+# Module-level + process-wide on purpose — this is a knob for tooling
+# (Studio's multi-model benchmark, or anyone doing batch comparisons
+# from a notebook), not for production agent code. Production agents
+# should keep passing ``model=`` explicitly.
+#
+# Reads happen at call time (the patched_create looks up the current
+# value each invocation), so a single-threaded sequence of:
+#
+#     set_default_model("gpt-4o-mini"); run_suite(); \
+#     set_default_model("claude-haiku-4-5"); run_suite(); \
+#     set_default_model(None)
+#
+# does exactly what you'd expect. Concurrent runs in the same process
+# share the override — if you need per-task isolation, run each in a
+# fresh subprocess.
+
+_DEFAULT_MODEL_OVERRIDE: str | None = None
+
+
+def set_default_model(model: str | None) -> None:
+    """Override the model on every subsequent patched ``create()`` call.
+
+    Pass ``None`` to clear. See module docstring for the semantics.
+    """
+    global _DEFAULT_MODEL_OVERRIDE
+    _DEFAULT_MODEL_OVERRIDE = model
+
+
+def get_default_model() -> str | None:
+    """Read the current model override (``None`` if not set)."""
+    return _DEFAULT_MODEL_OVERRIDE
+
+
+__all__ = [
+    "DEFAULT_PRICES",
+    "PriceTable",
+    "estimate_cost_usd",
+    "register_prices",
+    "set_default_model",
+    "get_default_model",
+]

{agentprdiff-0.2.4 → agentprdiff-0.3.0}/src/agentprdiff/adapters/anthropic.py

@@ -106,6 +106,15 @@ def instrument_client(
     instance_attr_value = vars(messages_attr).get("create")
 
     def patched_create(*args: Any, **kwargs: Any) -> Any:
+        # Honor the global model override if set — same contract as the
+        # OpenAI adapter. Lookup happens at call time so the value reflects
+        # whatever set_default_model() last set.
+        from . import get_default_model
+
+        override = get_default_model()
+        if override is not None and "model" in kwargs:
+            kwargs = dict(kwargs)
+            kwargs["model"] = override
         start = time.perf_counter()
         try:
             response = original_create(*args, **kwargs)

{agentprdiff-0.2.4 → agentprdiff-0.3.0}/src/agentprdiff/adapters/openai.py

@@ -311,6 +311,30 @@ def _record_failure(
     )
 
 
+def _apply_model_override(kwargs: dict[str, Any]) -> dict[str, Any]:
+    """If a global model override is set, return a new kwargs dict with
+    ``model`` rewritten. Returns the original dict when no override is active.
+
+    The lookup happens at call time so multi-leg benchmarks can flip the
+    override between runs in the same process.
+    """
+    # Import at call time to avoid an import cycle (adapters/__init__.py
+    # itself imports from .pricing, which is fine; this module imports from
+    # adapters at package level but we want the *live* value).
+    from . import get_default_model
+
+    override = get_default_model()
+    if override is None:
+        return kwargs
+    # Only rewrite if the caller actually passed a model — preserves the
+    # SDK's own error path when model is missing.
+    if "model" not in kwargs:
+        return kwargs
+    new_kwargs = dict(kwargs)
+    new_kwargs["model"] = override
+    return new_kwargs
+
+
 def _make_sync_patched_create(
     original_create: Callable[..., Any],
     *,
@@ -319,6 +343,7 @@ def _make_sync_patched_create(
     prices: PriceTable | None,
 ) -> Callable[..., Any]:
     def patched_create(*args: Any, **kwargs: Any) -> Any:
+        kwargs = _apply_model_override(kwargs)
         start = time.perf_counter()
         try:
             response = original_create(*args, **kwargs)
@@ -350,6 +375,7 @@ def _make_async_patched_create(
     prices: PriceTable | None,
 ) -> Callable[..., Awaitable[Any]]:
     async def patched_create(*args: Any, **kwargs: Any) -> Any:
+        kwargs = _apply_model_override(kwargs)
         start = time.perf_counter()
         try:
             response = await original_create(*args, **kwargs)

agentprdiff-0.3.0/studio/README.md

@@ -0,0 +1,108 @@
+# agentprdiff Studio
+
+Web UI on top of the [agentprdiff](https://github.com/vnageshwaran-de/agentprdiff) engine.
+
+The CLI (`pip install agentprdiff`, write `suite.py`, `agentprdiff check`) still works exactly the same. Studio is a parallel surface for non-dev users (PMs, QA, vibecoders) who want to trigger runs, watch progress live, diff cases, and approve baselines in a browser.
+
+## Quick start (Docker)
+
+```bash
+# SQLite, single container, zero config:
+docker compose up --build
+
+# Open the UI:
+open http://localhost:8080
+```
+
+That spins up one container, persists state to a `studio-data` volume, and serves both the API (under `/api/*`) and the SPA on port 8080.
+
+For shared deployments swap to Postgres:
+
+```bash
+docker compose --profile postgres up --build
+# also export STUDIO_DATABASE_URL=postgresql+asyncpg://studio:studio@db:5432/studio
+# (or set it in studio.environment in compose.yml).
+```
+
+## Run modes
+
+| Mode | Storage | Best for |
+|---|---|---|
+| Default (SQLite) | `studio-data` volume → `/data/studio.db` | Solo / small team, single host |
+| `--profile postgres` | Postgres container + `studio-pg` volume | Multiple users, durable history |
+
+Plus the orthogonal **intake modes** for projects you create inside Studio:
+
+* **git** — clone a remote repo, walk it for suites, run via subprocess+venv.
+* **zip** — upload an archive, same execution path as git.
+* **http** — point Studio at a deployed endpoint, author suites as JSON, runs go in-process via httpx (no Python execution, baselines stored in the DB).
+
+## Local development (without Docker)
+
+```bash
+# Backend (Python).
+pip install -e .                # the engine, from the repo root
+pip install -e studio/backend
+uvicorn agentprdiff_studio.main:app --reload --port 8080
+
+# Frontend (separate terminal).
+cd studio/frontend
+npm install
+npm run dev                     # Vite on :5173, proxies /api → :8080
+open http://localhost:5173
+```
+
+In dev the frontend lives on Vite. In Docker / prod the FastAPI app serves the built `dist/` on the same port as the API.
+
+## Configuration
+
+Everything is env-driven. The most useful knobs:
+
+| Env var | Default | Notes |
+|---|---|---|
+| `STUDIO_DATA_DIR` | `./.studio-data` | DB + cloned repos + uploaded zips |
+| `STUDIO_DATABASE_URL` | `sqlite+aiosqlite:///<data>/studio.db` | Set to `postgresql+asyncpg://…` for Postgres |
+| `STUDIO_FRONTEND_DIR` | unset (dev) / `/opt/studio/frontend` (Docker) | If set + exists, FastAPI serves the SPA there |
+| `STUDIO_SECRET_KEY` | generated and persisted to `<data>/.secret_key` | Fernet key for encrypting secrets at rest |
+| `STUDIO_ENGINE_REQ` | `agentprdiff>=0.2.5` | What pip installs into per-project venvs |
+| `STUDIO_RUN_WALLTIME_SECONDS` | `300` | Per-run hard wall-time |
+| `STUDIO_RUN_MEMORY_MB` | `1024` | Per-run memory cap (POSIX rlimit) |
+| `STUDIO_CORS_ORIGINS` | `["*"]` | Pass as JSON array via env if tightening |
+
+## What's in the image
+
+* Engine (`agentprdiff`) installed from PyPI (or a path, via `STUDIO_ENGINE_REQ`).
+* Studio backend (FastAPI + SQLAlchemy + httpx + GitPython).
+* Built SPA (Vite output) at `/opt/studio/frontend`.
+* `git`, `build-essential`, `tini` for clean signal handling.
+
+The image runs as `uvicorn agentprdiff_studio.main:app --host 0.0.0.0 --port 8080` under `tini`. A healthcheck hits `/api/health` every 30s.
+
+## Persistence
+
+All durable state lives under `/data` inside the container, mounted via the named volume `studio-data`:
+
+```
+/data
+├── studio.db              # SQLite (default mode)
+├── .secret_key            # Fernet key (created on first boot)
+└── projects/<id>/
+    ├── repo/              # git intake
+    │   └── .studio-venv/  # per-project venv (provisioned on first run)
+    └── upload/            # zip intake
+```
+
+For Postgres deployments the DB lives in `studio-pg`; the `studio-data` volume still holds the per-project workspaces and venvs.
+
+## Where things land in the repo
+
+```
+studio/
+├── Dockerfile                  multi-stage: node build → python runtime
+├── docker-compose.yml          default + postgres profile
+├── .dockerignore               keeps the build context small
+├── backend/
+│   └── src/agentprdiff_studio/ FastAPI app + executor + DB layer
+└── frontend/
+    └── src/                    Vite + React + TS SPA
+```

agentprdiff-0.3.0/studio/backend/README.md

@@ -0,0 +1,34 @@
+# agentprdiff-studio (backend)
+
+Web server for **agentprdiff Studio** — a browser UI on top of the [agentprdiff](https://github.com/vnageshwaran-de/agentprdiff) engine.
+
+The CLI workflow (`pip install agentprdiff`, write `suite.py`, `agentprdiff check`) is unchanged. Studio is a parallel surface for non-dev users (PMs, QA, vibecoders) who want to trigger runs, review diffs, and approve baselines in a browser.
+
+## Status
+
+**M1** — backend skeleton + git intake + executor. No UI yet; everything is driven by `curl`.
+
+## Run it locally
+
+```bash
+cd studio/backend
+pip install -e ../..       # install the engine from the monorepo
+pip install -e ".[dev]"    # install studio backend + dev tools
+uvicorn agentprdiff_studio.main:app --reload --port 8080
+```
+
+Default config writes to `./.studio-data/` (SQLite + cloned project workspaces). Override with env vars — see `src/agentprdiff_studio/settings.py`.
+
+## API (M1)
+
+```
+POST /api/projects                       create + git-clone
+POST /api/projects/{id}/sync             re-pull + rediscover suites
+GET  /api/projects/{id}                  detail
+GET  /api/projects/{id}/suites           list discovered suites
+POST /api/runs                           {project_id, suite_id, command}
+GET  /api/runs/{id}                      status + case summary
+GET  /api/runs/{id}/cases                full per-case results
+```
+
+Zip / HTTP intake, secrets, SSE, and the React UI come in M2–M5.

agentprdiff-0.2.4/src/agentprdiff/adapters/__init__.py

@@ -1,43 +0,0 @@
-"""SDK adapters for agentprdiff.
-
-The adapters take an agent that uses a real LLM SDK (OpenAI, Anthropic, or any
-OpenAI-compatible provider like Groq, Gemini's openai-compat endpoint,
-OpenRouter, Ollama, or vLLM) and capture every model call as an `LLMCall` on a
-`Trace` — without forcing the user to rewrite their agent loop.
-
-The pattern is::
-
-    from agentprdiff.adapters.openai import instrument_client, instrument_tools
-
-    def my_agent(query: str):
-        client = OpenAI(...)
-        with instrument_client(client) as trace:
-            tools = instrument_tools(TOOL_MAP, trace)
-            # ... user's existing tool-calling loop, untouched ...
-            return final_text, trace
-
-Submodules are imported lazily so the base `agentprdiff` install doesn't pull
-in `openai` / `anthropic` unless the user opts in via the extras::
-
-    pip install "agentprdiff[openai]"
-    pip install "agentprdiff[anthropic]"
-
-See `docs/adapters.md` for the full reference.
-"""
-
-from __future__ import annotations
-
-# Re-export pricing helpers — these are pure-Python and have no SDK dependency.
-from .pricing import (
-    DEFAULT_PRICES,
-    PriceTable,
-    estimate_cost_usd,
-    register_prices,
-)
-
-__all__ = [
-    "DEFAULT_PRICES",
-    "PriceTable",
-    "estimate_cost_usd",
-    "register_prices",
-]