evalcore 2.4.2__tar.gz → 2.5.0__tar.gz

{evalcore-2.4.2 → evalcore-2.5.0}/CHANGELOG.md

@@ -6,6 +6,74 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [2.5.0] - 2026-08-11
+
+A suite can declare the modules it needs imported.
+
+### Added
+- `plugins:` on a suite: a list of module paths the runner imports before it
+  looks up any `type`, so a suite that names a custom adapter or grader
+  resolves it without a flag at the call site. Registration is an import side
+  effect and nothing in the engine imports a consumer's module on its own, so
+  until now every entry point had to remember `--plugins my.graders` (CLI) or a
+  bare `import my.graders` (Python API) - and the two could disagree. A suite
+  is now self-contained: the same file runs from the CLI, from a consumer's own
+  `run_eval.py`, and from a test with nothing to remember.
+
+  `--plugins` is unchanged and still the way to add a module without editing
+  the suite, which would change `suite_hash`.
+
+  The import happens when a **run** starts, never in `load_suite`. Parsing,
+  hashing, diffing or reporting on a suite executes no consumer code, so a
+  suite you have not decided to run is still only data. `compare` and `report`
+  therefore do not import a suite's plugins - they do not need the registries.
+
+  A module that cannot be imported raises `ConfigError` naming it, rather than
+  the unknown-`type` error one lookup later.
+
+### Fixed
+- `examples/quickstart/run_eval.py` called `JsonlOutboxExporter.export()`,
+  removed when 2.2.0 named the exporter seam, so `just example-api` had been
+  failing with `AttributeError` since. It now exports score rows only, which is
+  the one grain the store has: a scorecard is a read-time aggregation over
+  those rows, so exporting it too would persist something derived that could
+  disagree with them.
+
+### Changed
+- `examples/quickstart` declares its own `plugins:` and no longer needs
+  `--plugins` on the command line, nor the `import ... # noqa: F401` that three
+  of its entry points carried to force registration. `graders.py` no longer
+  imports `adapter.py` for the side effect either. The example is the same
+  eval; it just stops demonstrating the workaround.
+
+## [2.4.3] - 2026-08-11
+
+Live Anthropic judges work on current Claude models again.
+
+### Fixed
+- The Anthropic judge and pairwise clients no longer send `temperature=0`.
+  `temperature` (with `top_p`/`top_k`) was removed from the Claude request
+  surface at Opus 4.7, and sending it at all is a 400 there and on every model
+  after it - so a judge or a pairwise comparison pinned to `claude-opus-4-7`,
+  `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` or `claude-fable-5`
+  failed every call. The OpenAI clients still send it; that API still takes it.
+
+### Changed
+- Live judge `max_tokens` defaults are now 8192 (from 1024 on the rubric
+  judge, 512 on pairwise). Thinking is on by default from Opus 5 and Sonnet 5
+  onward and `max_tokens` bounds thinking plus reply together, so a
+  1024-token budget could be spent on reasoning before the forced tool call
+  landed - which surfaced as error scores rather than as an error. It is a
+  ceiling, not a spend: a model that does not think generates the same handful
+  of tokens it did before.
+
+**Upgrading:** a judge on a model that still accepts `temperature` (Sonnet
+4.6, Opus 4.6, the 4.5 line and older) now samples at the API default instead
+of 0, so its scores are no longer pinned run to run - expect more variance in
+a rubric dimension or a win-rate than before, and re-baseline if a gate sits
+close to its threshold. A judge on a thinking model also now bills thinking
+tokens on every call. Pass `max_tokens=` to a client to keep the old budget.
+
 ## [2.4.2] - 2026-08-09
 
 Every `llm_as_judge` row now describes what it measures.
@@ -312,7 +380,9 @@ by semantic versioning: a breaking change to either means a 2.0.
   rating + ranking with judge agreement, Markdown/HTML reporters, JSON +
   column-store outbox, and content-hash provenance.
 
-[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.4.2...HEAD
+[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.5.0...HEAD
+[2.5.0]: https://github.com/scottpmiller/evalcore/compare/2.4.3...2.5.0
+[2.4.3]: https://github.com/scottpmiller/evalcore/compare/2.4.2...2.4.3
 [2.4.2]: https://github.com/scottpmiller/evalcore/compare/2.4.1...2.4.2
 [2.4.1]: https://github.com/scottpmiller/evalcore/compare/2.4.0...2.4.1
 [2.4.0]: https://github.com/scottpmiller/evalcore/compare/2.3.0...2.4.0

{evalcore-2.4.2 → evalcore-2.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: evalcore
-Version: 2.4.2
+Version: 2.5.0
 Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
 Project-URL: Homepage, https://github.com/scottpmiller/evalcore
 Project-URL: Repository, https://github.com/scottpmiller/evalcore
@@ -214,8 +214,15 @@ class MyAdapter(http.HTTPAdapter):
 ```
 
 Constructor kwargs come from the suite's `adapter:` mapping (everything
-except `type`). Load the module at run time with `--plugins my_service.graders`
-(CLI) or a plain `import` (Python API) - registration happens on import.
+except `type`, which is popped to select the class).
+
+Registration happens on import, so something has to import the module. List
+it in the suite's `plugins:` and the runner does it for you, which keeps the
+suite self-contained - the same file then works from the CLI, the Python API,
+and another harness with nothing to remember at the call site. `--plugins
+my_service.graders` (CLI) and a plain `import` (Python API) both still work,
+and are the way to add a module without editing the suite (which would change
+`suite_hash`). Suites are imported when a run starts, never by `load_suite`.
 
 An adapter need not be HTTP-backed: it can grade *what a deployed system
 already did* by reading from an observability store - turning an aggregated
@@ -327,7 +334,7 @@ degrade to `null`. `compare`'s guardrails and a `win_metric` with
 `generation_cost` or `tool_error_rate` alongside quality judges.
 
 The judge runs live (`AnthropicJudgeClient` forced tool call, or
-`OpenAIJudgeClient` `json_schema` - both temperature 0, needing the `judge`
+`OpenAIJudgeClient` `json_schema` at temperature 0 - needing the `judge`
 extra plus `ANTHROPIC_API_KEY`/`OPENAI_API_KEY`) or offline
 (`ReplayJudgeClient`), chosen by the run mode like the adapter. Each
 dimension becomes a metric `<name>.<key>` plus a `<name>.overall` mean.
@@ -403,6 +410,8 @@ dataset: datasets/support_reply/v1
 dataset_version: v1
 mode_default: http           # 'replay' to default offline
 replay_fixtures: fixtures/replay.yaml
+plugins:                     # modules to import so custom `type`s resolve
+  - my_service.graders       # (see step 2/4); omit if you use only built-ins
 adapter: {...}               # step 2
 graders: [...]               # step 4
 variants: {...}              # step 3
@@ -438,16 +447,17 @@ reported informationally.
 
 ## 7. Running it
 
-**CLI** (plug-ins first, so custom types register):
+**CLI** (a suite's `plugins:` registers its custom types; add `--plugins
+mod1,mod2` before the subcommand for anything the suite does not declare):
 
 ```bash
 # one variant -> scorecard (optionally saved)
-evalcore --plugins my_service.graders run \
+evalcore run \
     --suite my_service/suite.yaml --variant candidate --mode replay \
     --out candidate.scorecard.json --revision "$GIT_SHA"
 
 # the CI workhorse: run baseline+candidate, compare, exit 1 on 'fail'
-evalcore --plugins my_service.graders gate \
+evalcore gate \
     --suite my_service/suite.yaml --mode replay \
     --export outbox.jsonl --revision "$GIT_SHA"
 

{evalcore-2.4.2 → evalcore-2.5.0}/README.md

@@ -182,8 +182,15 @@ class MyAdapter(http.HTTPAdapter):
 ```
 
 Constructor kwargs come from the suite's `adapter:` mapping (everything
-except `type`). Load the module at run time with `--plugins my_service.graders`
-(CLI) or a plain `import` (Python API) - registration happens on import.
+except `type`, which is popped to select the class).
+
+Registration happens on import, so something has to import the module. List
+it in the suite's `plugins:` and the runner does it for you, which keeps the
+suite self-contained - the same file then works from the CLI, the Python API,
+and another harness with nothing to remember at the call site. `--plugins
+my_service.graders` (CLI) and a plain `import` (Python API) both still work,
+and are the way to add a module without editing the suite (which would change
+`suite_hash`). Suites are imported when a run starts, never by `load_suite`.
 
 An adapter need not be HTTP-backed: it can grade *what a deployed system
 already did* by reading from an observability store - turning an aggregated
@@ -295,7 +302,7 @@ degrade to `null`. `compare`'s guardrails and a `win_metric` with
 `generation_cost` or `tool_error_rate` alongside quality judges.
 
 The judge runs live (`AnthropicJudgeClient` forced tool call, or
-`OpenAIJudgeClient` `json_schema` - both temperature 0, needing the `judge`
+`OpenAIJudgeClient` `json_schema` at temperature 0 - needing the `judge`
 extra plus `ANTHROPIC_API_KEY`/`OPENAI_API_KEY`) or offline
 (`ReplayJudgeClient`), chosen by the run mode like the adapter. Each
 dimension becomes a metric `<name>.<key>` plus a `<name>.overall` mean.
@@ -371,6 +378,8 @@ dataset: datasets/support_reply/v1
 dataset_version: v1
 mode_default: http           # 'replay' to default offline
 replay_fixtures: fixtures/replay.yaml
+plugins:                     # modules to import so custom `type`s resolve
+  - my_service.graders       # (see step 2/4); omit if you use only built-ins
 adapter: {...}               # step 2
 graders: [...]               # step 4
 variants: {...}              # step 3
@@ -406,16 +415,17 @@ reported informationally.
 
 ## 7. Running it
 
-**CLI** (plug-ins first, so custom types register):
+**CLI** (a suite's `plugins:` registers its custom types; add `--plugins
+mod1,mod2` before the subcommand for anything the suite does not declare):
 
 ```bash
 # one variant -> scorecard (optionally saved)
-evalcore --plugins my_service.graders run \
+evalcore run \
     --suite my_service/suite.yaml --variant candidate --mode replay \
     --out candidate.scorecard.json --revision "$GIT_SHA"
 
 # the CI workhorse: run baseline+candidate, compare, exit 1 on 'fail'
-evalcore --plugins my_service.graders gate \
+evalcore gate \
     --suite my_service/suite.yaml --mode replay \
     --export outbox.jsonl --revision "$GIT_SHA"
 

{evalcore-2.4.2 → evalcore-2.5.0}/docs/design.md

@@ -116,6 +116,15 @@ A consumer adds an eval tree (its own repo, or a directory like
 4. **Suite + threshold config** - guardrail metrics, win metric + dead band,
    N samples, which triggers run which suite.
 
+A consumer that writes custom graders or adapters adds one more line: a
+`plugins:` list in the suite naming the modules to import, so the registry
+holds its `type`s before they are looked up. Registration is an import side
+effect, and nothing in the engine imports a consumer's module on its own. The
+import happens when a **run** starts, not when a suite is loaded - `load_suite`
+parses and hashes YAML and executes nothing, so reading, hashing, or reporting
+on a suite is free of side effects, while starting a run is already the point
+where the suite's adapter gets to make network calls.
+
 The engine supplies runner, comparison, gate, store, reporters, and the CLI.
 That ratio - four data files vs. a whole engine - is the genericity test: if
 onboarding a consumer ever requires editing `src/evalcore/`, that's an

{evalcore-2.4.2 → evalcore-2.5.0}/examples/quickstart/graders.py

@@ -17,10 +17,6 @@ import re
 from evalcore import models
 from evalcore.graders import base
 
-# Registering the adapter alongside the graders means one plug-in module wires
-# up the whole consumer (adapter + custom graders).
-from examples.quickstart import adapter  # noqa: F401
-
 _ACK = re.compile(
     r"\b(sorry|understand|happy to help|you're right|apolog)", re.IGNORECASE
 )

{evalcore-2.4.2 → evalcore-2.5.0}/examples/quickstart/run_eval.py

@@ -23,7 +23,6 @@ import os
 import pathlib
 import sys
 
-import examples.quickstart.graders  # noqa: F401 - registers plug-ins
 from evalcore import compare, loader, report, runner, store
 
 HERE = pathlib.Path(__file__).resolve().parent
@@ -91,17 +90,14 @@ def main(argv: list[str] | None = None) -> int:
         raise RuntimeError('run JSON round-trip mismatch')
 
     # 6. Flatten to self-describing rows and append the outbox a column-store
-    # shipper would drain: scorecard metrics AND per-sample score rows.
-    exporter = store.JsonlOutboxExporter(out / 'outbox.jsonl')
-    metric_rows = exporter.export(baseline) + exporter.export(candidate)
-    score_exporter = store.JsonlOutboxExporter(out / 'scores.jsonl')
-    n_scores = score_exporter.export_scores(
-        baseline_run
-    ) + score_exporter.export_scores(candidate_run)
-    print(f'\noutbox: {metric_rows} metric rows -> {exporter.outbox_path}')
-    print(
-        f'scores: {n_scores} per-sample rows -> {score_exporter.outbox_path}'
+    # shipper would drain. One grain only - per-sample score rows. A run's
+    # scorecard is a read-time aggregation over these, so exporting it too
+    # would store something derived that could disagree with them.
+    exporter = store.JsonlOutboxExporter(out / 'scores.jsonl')
+    n_scores = exporter.export_scores(baseline_run) + exporter.export_scores(
+        candidate_run
     )
+    print(f'\nscores: {n_scores} per-sample rows -> {exporter.outbox_path}')
 
     # 7. Gate semantics: non-zero exit on a failing verdict.
     print(f'\ngate verdict: {result.verdict} ({result.summary})')

{evalcore-2.4.2 → evalcore-2.5.0}/examples/quickstart/suite.yaml

@@ -10,8 +10,7 @@
 #   - a head-to-head pairwise config
 #
 # Run it:
-#   evalcore --plugins examples.quickstart.graders gate \
-#       --suite examples/quickstart/suite.yaml --mode replay
+#   evalcore gate --suite examples/quickstart/suite.yaml --mode replay
 #
 # Paths are relative to this file.
 
@@ -22,6 +21,14 @@ dataset_version: v1
 mode_default: replay
 replay_fixtures: fixtures/support_reply_replay.yaml
 
+# This consumer's own modules. The runner imports them before it looks up any
+# `type` below, which is what makes `canned_support` and the two custom graders
+# resolvable - so this suite runs from the CLI, from run_eval.py, or from a
+# test with nothing to remember at the call site.
+plugins:
+  - examples.quickstart.adapter
+  - examples.quickstart.graders
+
 # The custom offline stub adapter (examples/quickstart/adapter.py). In replay
 # mode the engine ignores this and reads replay_fixtures instead; in any other
 # mode it builds this adapter (which needs no network).

{evalcore-2.4.2 → evalcore-2.5.0}/examples/quickstart/tests/test_quickstart.py

@@ -9,7 +9,6 @@ the custom adapter and the custom graders.
 import pathlib
 import unittest
 
-import examples.quickstart.graders  # noqa: F401 - registers adapter + graders
 from evalcore import compare, loader, runner
 
 SUITE_PATH = pathlib.Path(__file__).resolve().parents[1] / 'suite.yaml'

{evalcore-2.4.2 → evalcore-2.5.0}/justfile

@@ -23,7 +23,7 @@ lint:
 
 # Run the quickstart suite offline against recorded fixtures.
 example:
-    uv run evalcore --plugins examples.quickstart.graders gate --suite examples/quickstart/suite.yaml --mode replay
+    uv run evalcore gate --suite examples/quickstart/suite.yaml --mode replay
 
 # Run the quickstart suite through the Python API (no CLI), offline.
 example-api:
@@ -31,4 +31,4 @@ example-api:
 
 # Head-to-head A-vs-B win-rate over the quickstart suite (offline).
 example-pairwise:
-    uv run evalcore --plugins examples.quickstart.graders pairwise --suite examples/quickstart/suite.yaml --a baseline --b candidate --mode replay
+    uv run evalcore pairwise --suite examples/quickstart/suite.yaml --a baseline --b candidate --mode replay

{evalcore-2.4.2 → evalcore-2.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "evalcore"
-version = "2.4.2"
+version = "2.5.0"
 description = "A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs."
 readme = "README.md"
 requires-python = ">=3.11"

{evalcore-2.4.2 → evalcore-2.5.0}/src/evalcore/cli.py

@@ -26,28 +26,25 @@ adapters register before the run.
 import argparse
 import asyncio
 import datetime
-import importlib
-import os
 import pathlib
 import sys
 
 from evalcore import compare as compare_mod
-from evalcore import loader, rating, report, reporters, runner, store
+from evalcore import loader, plugins, rating, report, reporters, runner, store
 from evalcore import pairwise as pairwise_mod
 from evalcore import sweep as sweep_mod
 
 
 def _load_plugins(spec: str | None) -> None:
-    if not spec:
-        return
-    # Consumers run the CLI from their repo root; the console script
-    # (unlike `python -m`) does not put the cwd on sys.path, so add it
-    # or `--plugins my_pkg.graders` could never import.
-    cwd = os.getcwd()
-    if cwd not in sys.path:
-        sys.path.insert(0, cwd)
-    for name in filter(None, spec.split(',')):
-        importlib.import_module(name.strip())
+    """Import the `--plugins` modules, and make cwd importable for both.
+
+    A suite's own `plugins:` list is imported by the runner, not here - but it
+    needs the same working-directory path fix, and this runs first on every
+    executing command, so it is done unconditionally rather than only when the
+    flag is passed.
+    """
+    plugins.allow_cwd_imports()
+    plugins.load((spec or '').split(','))
 
 
 def _now() -> str:

{evalcore-2.4.2 → evalcore-2.5.0}/src/evalcore/graders/judge.py

@@ -104,7 +104,7 @@ class AnthropicJudgeClient:
         self,
         model: str,
         api_key_env: str = 'ANTHROPIC_API_KEY',
-        max_tokens: int = 1024,
+        max_tokens: int = 8192,
         timeout: float = 30.0,
     ):
         self.model = model
@@ -156,7 +156,6 @@ class AnthropicJudgeClient:
             response = await client.messages.create(
                 model=self.model,
                 max_tokens=self.max_tokens,
-                temperature=0,
                 timeout=self.timeout,
                 system=system,
                 tools=[tool],
@@ -179,7 +178,7 @@ class OpenAIJudgeClient:
         self,
         model: str,
         api_key_env: str = 'OPENAI_API_KEY',
-        max_tokens: int = 1024,
+        max_tokens: int = 8192,
         timeout: float = 30.0,
     ):
         # Accept a 'provider:model' id (e.g. 'openai:gpt-4o'); SDK wants bare.
@@ -326,7 +325,7 @@ class RubricJudge:
         judge_version: str = 'v1',
         replay_path: str | None = None,
         client: JudgeClient | None = None,
-        max_tokens: int = 1024,
+        max_tokens: int = 8192,
     ):
         self.name = name
         self.content_ref = content_ref

{evalcore-2.4.2 → evalcore-2.5.0}/src/evalcore/loader.py

@@ -56,6 +56,11 @@ class SuiteConfig(pydantic.BaseModel):
     replay_fixtures: str | None = None
     adapter: dict
     graders: list[dict] = pydantic.Field(default_factory=list)
+    # Consumer modules to import before a run, so their registered custom
+    # adapters/graders exist by the time a `type` here is looked up. Imported
+    # by the runner, NOT by load_suite - loading a suite stays side-effect
+    # free. See evalcore.plugins.
+    plugins: list[str] = pydantic.Field(default_factory=list)
     variants: dict[str, dict] = pydantic.Field(default_factory=dict)
     n_samples: int = 1
     # Max concurrent (case, sample) invocations. Above 1, the adapter and

{evalcore-2.4.2 → evalcore-2.5.0}/src/evalcore/pairwise.py

@@ -70,7 +70,7 @@ class AnthropicPairwiseClient:
         self,
         model: str,
         api_key_env: str = 'ANTHROPIC_API_KEY',
-        max_tokens: int = 512,
+        max_tokens: int = 8192,
         timeout: float = 30.0,
     ):
         self.model = model
@@ -88,7 +88,6 @@ class AnthropicPairwiseClient:
             response = await client.messages.create(
                 model=self.model,
                 max_tokens=self.max_tokens,
-                temperature=0,
                 timeout=self.timeout,
                 system=system,
                 tools=[tool],
@@ -111,7 +110,7 @@ class OpenAIPairwiseClient:
         self,
         model: str,
         api_key_env: str = 'OPENAI_API_KEY',
-        max_tokens: int = 512,
+        max_tokens: int = 8192,
         timeout: float = 30.0,
     ):
         self.model = model.split(':', 1)[1] if ':' in model else model

evalcore-2.5.0/src/evalcore/plugins.py

@@ -0,0 +1,79 @@
+"""Import consumer modules so their registrations take effect.
+
+A custom adapter or grader reaches the engine through a registry keyed by the
+``type`` string a suite names, and the registry entry is written by the
+``@register`` decorator *at import time*. Nothing in evalcore imports a
+consumer's module, so a suite naming ``type: my_grader`` fails as an unknown
+type until something has imported the module that defines it.
+
+Two things do that importing: a suite's own ``plugins:`` list, and the CLI's
+``--plugins`` flag. Prefer the suite key. It keeps a suite self-contained -
+the same file then works from the CLI, from the Python API, and from another
+consumer's harness, with nothing to remember at the call site. The flag stays
+for a suite that does not declare its own, and for adding a module to a run
+without editing the suite (which would change ``suite_hash``).
+
+Importing is deliberately **not** part of loading a suite.
+:func:`~evalcore.loader.load_suite` parses and hashes YAML and executes
+nothing, so reading a suite, rendering a report, or diffing two of them stays
+free of side effects. The import happens when a run starts - which is already
+the point where the suite's adapter gets to make network calls and spend
+money, so it is the honest place to also let it run code.
+"""
+
+import importlib
+import os
+import sys
+
+from evalcore import errors
+
+#: Names already imported. ``sys.modules`` is the real cache - this only keeps
+#: a repeated run in one process from re-attempting an import that failed, and
+#: lets :func:`load` report what it actually did.
+_loaded: set[str] = set()
+
+
+def allow_cwd_imports() -> None:
+    """Put the working directory on ``sys.path``.
+
+    The console script (unlike ``python -m``) does not, so a
+    ``plugins: [evals.graders]`` in a suite run from a repo root would never
+    resolve. Only the CLI calls this: a library has no business editing the
+    path of the process that imported it.
+    """
+    cwd = os.getcwd()
+    if cwd not in sys.path:
+        sys.path.insert(0, cwd)
+
+
+def load(names: list[str] | None) -> list[str]:
+    """Import each module in ``names`` once; return those newly imported.
+
+    Args:
+        names: Module paths (``evals.graders``). Blanks are skipped, so a
+            comma-split CLI string can be passed straight in.
+
+    Returns:
+        The names imported by this call, in order. Already-imported names are
+        omitted rather than repeated.
+
+    Raises:
+        ConfigError: If a module cannot be imported. The name is quoted in the
+            message, because the failure is nearly always a typo in the suite
+            or a module that is not importable from the working directory.
+
+    """
+    fresh: list[str] = []
+    for raw in names or ():
+        name = raw.strip()
+        if not name or name in _loaded:
+            continue
+        try:
+            importlib.import_module(name)
+        except ImportError as exc:
+            raise errors.ConfigError(
+                f'plugin module {name!r} could not be imported: {exc}'
+            ) from exc
+        _loaded.add(name)
+        fresh.append(name)
+    return fresh

{evalcore-2.4.2 → evalcore-2.5.0}/src/evalcore/runner.py

@@ -18,7 +18,7 @@ import statistics
 import time
 import uuid
 
-from evalcore import compare, loader, models, store
+from evalcore import compare, loader, models, plugins, store
 from evalcore import retry as retry_mod
 from evalcore.adapters import base as adapters_base
 from evalcore.graders import base as graders_base
@@ -169,6 +169,11 @@ async def run_suite(
     and their invocations skipped, so an interrupted run continues instead of
     restarting. Without ``resume`` an existing checkpoint is overwritten.
     """
+    # Before anything reads the registries: a suite's own plugin modules have
+    # to be imported for its custom `type`s to resolve. This is the executing
+    # path, so it is where the suite gets to run code - load_suite does not.
+    plugins.load(suite.plugins)
+
     if variant_name not in suite.variants:
         raise KeyError(
             f'variant {variant_name!r} not in suite '

evalcore-2.5.0/tests/test_plugins.py

@@ -0,0 +1,160 @@
+"""Suite-declared plugin modules (``plugins:``) and the CLI flag.
+
+The registry is written by ``@register`` at import time, so the only question
+these tests answer is *who* does the importing and *when*. The when matters as
+much as the who: loading a suite must not run a consumer's code, and starting a
+run must.
+"""
+
+import pathlib
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+from evalcore import errors, loader, models, plugins, runner
+
+
+def _suite(**over):
+    base = {
+        'project': 'p',
+        'suite': 's',
+        'dataset': 'd',
+        'dataset_version': 'v1',
+        'mode_default': 'live',
+        'adapter': {'type': '_plugin_probe'},
+        'variants': {'baseline': {}},
+    }
+    base.update(over)
+    return loader.SuiteConfig.model_validate(base)
+
+
+class LoadTests(unittest.TestCase):
+    def setUp(self):
+        # The module-level cache is process state; keep tests independent of
+        # each other and of whatever imported first.
+        self._saved = set(plugins._loaded)
+        plugins._loaded.clear()
+        self.addCleanup(
+            lambda: (
+                plugins._loaded.clear(),
+                plugins._loaded.update(self._saved),
+            )
+        )
+
+    def test_imports_once_and_reports_only_fresh_names(self):
+        with mock.patch.object(plugins.importlib, 'import_module') as imp:
+            first = plugins.load(['json', 'json', ' csv '])
+            second = plugins.load(['json'])
+        self.assertEqual(first, ['json', 'csv'])
+        self.assertEqual(second, [])
+        self.assertEqual(
+            [c.args[0] for c in imp.call_args_list], ['json', 'csv']
+        )
+
+    def test_blanks_and_none_are_skipped(self):
+        # A comma-split CLI string with no value is the common case:
+        # ''.split(',') is [''], not [].
+        self.assertEqual(plugins.load(None), [])
+        self.assertEqual(plugins.load(''.split(',')), [])
+        self.assertEqual(plugins.load(['', '  ']), [])
+
+    def test_unimportable_module_raises_config_error_naming_it(self):
+        with self.assertRaises(errors.ConfigError) as ctx:
+            plugins.load(['evalcore_no_such_plugin_module'])
+        self.assertIn('evalcore_no_such_plugin_module', str(ctx.exception))
+        # ConfigError is also a ValueError, so existing handlers keep working.
+        self.assertIsInstance(ctx.exception, ValueError)
+
+    def test_failed_import_is_not_cached_as_loaded(self):
+        for _ in range(2):
+            with self.assertRaises(errors.ConfigError):
+                plugins.load(['evalcore_no_such_plugin_module'])
+        self.assertNotIn('evalcore_no_such_plugin_module', plugins._loaded)
+
+    def test_allow_cwd_imports_adds_cwd_once(self):
+        saved = list(sys.path)
+        self.addCleanup(lambda: sys.path.__setitem__(slice(None), saved))
+        cwd = str(pathlib.Path.cwd())
+        # The test runner already has cwd on the path, so drop it first or the
+        # insert branch never runs and this asserts nothing.
+        sys.path[:] = [p for p in sys.path if p != cwd]
+
+        plugins.allow_cwd_imports()
+        self.assertEqual(sys.path[0], cwd)
+        plugins.allow_cwd_imports()
+        self.assertEqual(sys.path.count(cwd), 1)
+
+
+class SuiteIntegrationTests(unittest.TestCase):
+    def test_load_suite_accepts_plugins_and_defaults_to_empty(self):
+        self.assertEqual(_suite().plugins, [])
+        self.assertEqual(_suite(plugins=['a.b']).plugins, ['a.b'])
+
+    def test_load_suite_does_not_import_them(self):
+        """Reading a suite must stay side-effect free.
+
+        A bad module name would raise if load_suite imported it; parsing and
+        hashing a suite has to work on a file you have not decided to run.
+        """
+        body = (
+            'project: p\nsuite: s\ndataset: d\n'
+            'adapter: {type: http}\n'
+            'plugins: [evalcore_no_such_plugin_module]\n'
+        )
+        with tempfile.TemporaryDirectory() as tmp:
+            path = pathlib.Path(tmp) / 'suite.yaml'
+            path.write_text(body, encoding='utf-8')
+            cfg = loader.load_suite(str(path))
+        self.assertEqual(cfg.plugins, ['evalcore_no_such_plugin_module'])
+        self.assertNotIn('evalcore_no_such_plugin_module', sys.modules)
+
+    def test_run_suite_imports_them_before_building_the_adapter(self):
+        """The registration has to land before the `type` lookup.
+
+        The probe module registers `_plugin_probe`, which the suite names. If
+        the import happened after build_adapter - or not at all - this raises
+        an unknown-adapter ConfigError instead of running.
+        """
+        module = 'evalcore_plugin_probe'
+        source = (
+            'from evalcore import models\n'
+            'from evalcore.adapters import base\n'
+            '\n'
+            '@base.register("_plugin_probe")\n'
+            'class Probe:\n'
+            '    async def invoke(self, case, variant):\n'
+            '        return models.Output(fields={"text": "ok"})\n'
+        )
+        with tempfile.TemporaryDirectory() as tmp:
+            (pathlib.Path(tmp) / f'{module}.py').write_text(
+                source, encoding='utf-8'
+            )
+            sys.path.insert(0, tmp)
+            self.addCleanup(lambda: sys.path.remove(tmp))
+            self.addCleanup(lambda: sys.modules.pop(module, None))
+            plugins._loaded.discard(module)
+            self.addCleanup(lambda: plugins._loaded.discard(module))
+
+            # No dataset dir needed; the point here is the registry lookup.
+            with mock.patch.object(
+                loader, 'load_cases', return_value=[models.Case(id='c1')]
+            ):
+                run = runner.run_suite_sync(
+                    _suite(plugins=[module]), 'baseline'
+                )
+
+        self.assertEqual(run.results[0].output.fields['text'], 'ok')
+
+    def test_run_suite_surfaces_a_bad_plugin_before_anything_else(self):
+        suite = _suite(
+            plugins=['evalcore_no_such_plugin_module'],
+            variants={'baseline': {}},
+        )
+        with self.assertRaises(errors.ConfigError) as ctx:
+            runner.run_suite_sync(suite, 'baseline')
+        self.assertIn('evalcore_no_such_plugin_module', str(ctx.exception))
+
+
+if __name__ == '__main__':
+    unittest.main()

{evalcore-2.4.2 → evalcore-2.5.0}/uv.lock

@@ -188,7 +188,7 @@ wheels = [
 
 [[package]]
 name = "evalcore"
-version = "2.4.2"
+version = "2.5.0"
 source = { editable = "." }
 dependencies = [
     { name = "pydantic" },