evalcore 2.4.3__tar.gz → 2.6.0__tar.gz

{evalcore-2.4.3 → evalcore-2.6.0}/CHANGELOG.md

@@ -6,6 +6,97 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [2.6.0] - 2026-09-12
+
+A metric says which way is good and what its values sit on.
+
+### Added
+- `MetricRange` and a `value_range` field on `Score` and `MetricValue`.
+  Nothing about a float says what it is on: `0.86` is 86% if the metric runs
+  0..1 and 4.3 out of 5 if it runs 1..5. A consumer holding only the number
+  guesses from magnitude, which is how a run whose costs happened to stay
+  under a dollar gets rendered as percentages, and how the same metric gets
+  classified differently depending on the window you look at.
+- Built-in graders declare their own ranges. Deterministic checks are 0..1;
+  `classification` separates its confusion-matrix metrics (0..1) from
+  `support_*` and `errors` (0..unbounded); judge metrics are 0..1, being
+  points over the scale, except `disagreement`, a spread in raw points,
+  which is 0..scale-1.
+- `range:` on a `numeric` field spec, as `{min, max}` or a two-element list.
+  That grader surfaces whatever the adapter put in the field, so it is the
+  one place the engine cannot know. It is a separate key from the `min`/`max`
+  bounds beside it, which are a pass/fail threshold rather than a domain: a
+  cost that must stay under a dollar can still cost five.
+- `maximum=None` is a positive statement - unbounded above, so not a
+  fraction of anything - and is different from carrying no range at all,
+  which says only that nobody declared one. Nothing is inferred from
+  observed values.
+- `direction` and `higher_is_better` on `MetricDelta`. A delta's sign and its
+  meaning are different questions - `f1` rising is an improvement, and
+  `false_negative_rate` rising is a regression - and until now the engine only
+  knew the difference for the single win metric, via `win_higher_is_better`.
+  Every other metric came out of `compare()` as a bare number, so anything
+  downstream that wanted to rank, colour or summarize deltas had to keep its
+  own list of which metrics are inverted, or get it wrong.
+- `thresholds.metrics`, an optional map declaring polarity per metric:
+  `lower_is_better`, `higher_is_better`, or `neutral` for one that moves
+  without either direction being a result.
+- **Polarity is inferred from the guardrails when it is not declared**, which
+  is what makes this useful without editing a single existing suite. A `max`
+  or `must_not_increase` rule is only ever written about a metric you want
+  low, and `min`/`must_not_decrease` about one you want high, so a gated
+  metric has already stated its direction. Precedence, strongest first: an
+  explicit `metrics:` entry, then `win_higher_is_better` for the win metric,
+  then the guardrails. A metric none of them mention defaults to
+  higher-is-better; a metric fenced in on both sides by a band resolves to
+  `neutral` rather than falling through to that default, since a band says
+  neither direction is the good one.
+
+### Changed
+- `_evaluate_win` derives the win verdict through the same call that fills in
+  each `MetricDelta.direction`, so `Comparison.win` and the win metric's own
+  delta row cannot disagree about the same number.
+
+## [2.5.0] - 2026-08-11
+
+A suite can declare the modules it needs imported.
+
+### Added
+- `plugins:` on a suite: a list of module paths the runner imports before it
+  looks up any `type`, so a suite that names a custom adapter or grader
+  resolves it without a flag at the call site. Registration is an import side
+  effect and nothing in the engine imports a consumer's module on its own, so
+  until now every entry point had to remember `--plugins my.graders` (CLI) or a
+  bare `import my.graders` (Python API) - and the two could disagree. A suite
+  is now self-contained: the same file runs from the CLI, from a consumer's own
+  `run_eval.py`, and from a test with nothing to remember.
+
+  `--plugins` is unchanged and still the way to add a module without editing
+  the suite, which would change `suite_hash`.
+
+  The import happens when a **run** starts, never in `load_suite`. Parsing,
+  hashing, diffing or reporting on a suite executes no consumer code, so a
+  suite you have not decided to run is still only data. `compare` and `report`
+  therefore do not import a suite's plugins - they do not need the registries.
+
+  A module that cannot be imported raises `ConfigError` naming it, rather than
+  the unknown-`type` error one lookup later.
+
+### Fixed
+- `examples/quickstart/run_eval.py` called `JsonlOutboxExporter.export()`,
+  removed when 2.2.0 named the exporter seam, so `just example-api` had been
+  failing with `AttributeError` since. It now exports score rows only, which is
+  the one grain the store has: a scorecard is a read-time aggregation over
+  those rows, so exporting it too would persist something derived that could
+  disagree with them.
+
+### Changed
+- `examples/quickstart` declares its own `plugins:` and no longer needs
+  `--plugins` on the command line, nor the `import ... # noqa: F401` that three
+  of its entry points carried to force registration. `graders.py` no longer
+  imports `adapter.py` for the side effect either. The example is the same
+  eval; it just stops demonstrating the workaround.
+
 ## [2.4.3] - 2026-08-11
 
 Live Anthropic judges work on current Claude models again.
@@ -340,7 +431,9 @@ by semantic versioning: a breaking change to either means a 2.0.
   rating + ranking with judge agreement, Markdown/HTML reporters, JSON +
   column-store outbox, and content-hash provenance.
 
-[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.4.3...HEAD
+[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.6.0...HEAD
+[2.6.0]: https://github.com/scottpmiller/evalcore/compare/2.5.0...2.6.0
+[2.5.0]: https://github.com/scottpmiller/evalcore/compare/2.4.3...2.5.0
 [2.4.3]: https://github.com/scottpmiller/evalcore/compare/2.4.2...2.4.3
 [2.4.2]: https://github.com/scottpmiller/evalcore/compare/2.4.1...2.4.2
 [2.4.1]: https://github.com/scottpmiller/evalcore/compare/2.4.0...2.4.1

{evalcore-2.4.3 → evalcore-2.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalcore
-Version: 2.4.3
+Version: 2.6.0
 Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
 Project-URL: Homepage, https://github.com/scottpmiller/evalcore
 Project-URL: Repository, https://github.com/scottpmiller/evalcore
@@ -214,8 +214,15 @@ class MyAdapter(http.HTTPAdapter):
 ```
 
 Constructor kwargs come from the suite's `adapter:` mapping (everything
-except `type`). Load the module at run time with `--plugins my_service.graders`
-(CLI) or a plain `import` (Python API) - registration happens on import.
+except `type`, which is popped to select the class).
+
+Registration happens on import, so something has to import the module. List
+it in the suite's `plugins:` and the runner does it for you, which keeps the
+suite self-contained - the same file then works from the CLI, the Python API,
+and another harness with nothing to remember at the call site. `--plugins
+my_service.graders` (CLI) and a plain `import` (Python API) both still work,
+and are the way to add a module without editing the suite (which would change
+`suite_hash`). Suites are imported when a run starts, never by `load_suite`.
 
 An adapter need not be HTTP-backed: it can grade *what a deployed system
 already did* by reading from an observability store - turning an aggregated
@@ -403,6 +410,8 @@ dataset: datasets/support_reply/v1
 dataset_version: v1
 mode_default: http           # 'replay' to default offline
 replay_fixtures: fixtures/replay.yaml
+plugins:                     # modules to import so custom `type`s resolve
+  - my_service.graders       # (see step 2/4); omit if you use only built-ins
 adapter: {...}               # step 2
 graders: [...]               # step 4
 variants: {...}              # step 3
@@ -420,6 +429,9 @@ thresholds:
   win_min_delta: 0.02              # dead band: |delta| <= this -> neutral
   on_regression: warn              # or 'fail' to hard-gate the win metric
   variants: {baseline: baseline, candidate: candidate}   # gate defaults
+  metrics:                         # optional: which way is good, per metric
+    generation_cost: lower_is_better
+    total_tool_calls: neutral      # moves, but neither way is a result
   guardrails:                      # hard constraints on the CANDIDATE
     - metric: false_negative_rate
       max: 0.10                    # absolute ceiling
@@ -432,22 +444,67 @@ thresholds:
 
 Guardrail rules compose: `max`, `min`, `must_not_increase`,
 `must_not_decrease`. A guardrail whose metric is missing on the candidate
-fails closed. Pick guardrails for the failures that must never ship, and
+fails closed.
+
+Every metric also carries a `value_range`, because nothing about a float says
+what it is on: `0.86` is 86% if the metric runs 0..1 and 4.3 out of 5 if it
+runs 1..5. The grader declares it - deterministic checks are 0..1,
+`classification` puts `support_*` and `errors` on 0..unbounded while its
+confusion-matrix metrics are 0..1, and judge metrics are 0..1 except
+`disagreement`, a spread in raw points, which is 0..scale-1. A custom grader
+passes `value_range=` on the `Score`s it returns.
+
+`numeric` is the exception, since it surfaces whatever the adapter put in the
+field, so declare it there:
+
+```yaml
+graders:
+  - type: numeric
+    fields:
+      - {ref: output.cost, range: {min: 0, max: null}}
+      - {ref: output.quality, range: [1, 5]}
+      # `range` and `max` are different things - see below
+      - {ref: output.latency, range: {min: 0}, max: 2.5}
+```
+
+That last field is the distinction worth knowing. `min`/`max` on a numeric
+field are a **pass/fail threshold** - "fail this case over 2.5 seconds" - and
+`range` is what the number could be at all. A cost that must stay under a
+dollar can still cost five, so the two keys stay separate.
+
+`maximum: null` is a statement, not a gap - unbounded above, so not a fraction
+of anything, render it as it is. A metric with no range at all is different
+again: nobody declared one. The engine never infers a range from the values it
+sees, because "everything stayed under 1, so it must be a 0..1 metric" is what
+turns a run whose costs happened to stay cheap into percentages.
+
+Every `MetricDelta` carries `direction` (`improved`/`regressed`/`neutral`)
+alongside `delta`, because the sign of a number does not say what it means: a
+rise in `f1` is an improvement and a rise in `false_negative_rate` is a
+regression. Direction comes from the first of these that speaks to the metric:
+an explicit `metrics:` entry, `win_higher_is_better` for the win metric, or the
+guardrails themselves - a `max` or `must_not_increase` rule means you want the
+metric low, a `min` or `must_not_decrease` rule means high. So a suite that
+gates a metric already declares its direction and needs no `metrics:` block at
+all. A metric nothing speaks to is assumed higher-is-better; one fenced in on
+both sides by a band, or declared `neutral`, reports `direction: neutral` and
+`higher_is_better: null` rather than guessing. Pick guardrails for the failures that must never ship, and
 one win metric for the improvement you're hunting; everything else is
 reported informationally.
 
 ## 7. Running it
 
-**CLI** (plug-ins first, so custom types register):
+**CLI** (a suite's `plugins:` registers its custom types; add `--plugins
+mod1,mod2` before the subcommand for anything the suite does not declare):
 
 ```bash
 # one variant -> scorecard (optionally saved)
-evalcore --plugins my_service.graders run \
+evalcore run \
     --suite my_service/suite.yaml --variant candidate --mode replay \
     --out candidate.scorecard.json --revision "$GIT_SHA"
 
 # the CI workhorse: run baseline+candidate, compare, exit 1 on 'fail'
-evalcore --plugins my_service.graders gate \
+evalcore gate \
     --suite my_service/suite.yaml --mode replay \
     --export outbox.jsonl --revision "$GIT_SHA"
 

{evalcore-2.4.3 → evalcore-2.6.0}/README.md

@@ -182,8 +182,15 @@ class MyAdapter(http.HTTPAdapter):
 ```
 
 Constructor kwargs come from the suite's `adapter:` mapping (everything
-except `type`). Load the module at run time with `--plugins my_service.graders`
-(CLI) or a plain `import` (Python API) - registration happens on import.
+except `type`, which is popped to select the class).
+
+Registration happens on import, so something has to import the module. List
+it in the suite's `plugins:` and the runner does it for you, which keeps the
+suite self-contained - the same file then works from the CLI, the Python API,
+and another harness with nothing to remember at the call site. `--plugins
+my_service.graders` (CLI) and a plain `import` (Python API) both still work,
+and are the way to add a module without editing the suite (which would change
+`suite_hash`). Suites are imported when a run starts, never by `load_suite`.
 
 An adapter need not be HTTP-backed: it can grade *what a deployed system
 already did* by reading from an observability store - turning an aggregated
@@ -371,6 +378,8 @@ dataset: datasets/support_reply/v1
 dataset_version: v1
 mode_default: http           # 'replay' to default offline
 replay_fixtures: fixtures/replay.yaml
+plugins:                     # modules to import so custom `type`s resolve
+  - my_service.graders       # (see step 2/4); omit if you use only built-ins
 adapter: {...}               # step 2
 graders: [...]               # step 4
 variants: {...}              # step 3
@@ -388,6 +397,9 @@ thresholds:
   win_min_delta: 0.02              # dead band: |delta| <= this -> neutral
   on_regression: warn              # or 'fail' to hard-gate the win metric
   variants: {baseline: baseline, candidate: candidate}   # gate defaults
+  metrics:                         # optional: which way is good, per metric
+    generation_cost: lower_is_better
+    total_tool_calls: neutral      # moves, but neither way is a result
   guardrails:                      # hard constraints on the CANDIDATE
     - metric: false_negative_rate
       max: 0.10                    # absolute ceiling
@@ -400,22 +412,67 @@ thresholds:
 
 Guardrail rules compose: `max`, `min`, `must_not_increase`,
 `must_not_decrease`. A guardrail whose metric is missing on the candidate
-fails closed. Pick guardrails for the failures that must never ship, and
+fails closed.
+
+Every metric also carries a `value_range`, because nothing about a float says
+what it is on: `0.86` is 86% if the metric runs 0..1 and 4.3 out of 5 if it
+runs 1..5. The grader declares it - deterministic checks are 0..1,
+`classification` puts `support_*` and `errors` on 0..unbounded while its
+confusion-matrix metrics are 0..1, and judge metrics are 0..1 except
+`disagreement`, a spread in raw points, which is 0..scale-1. A custom grader
+passes `value_range=` on the `Score`s it returns.
+
+`numeric` is the exception, since it surfaces whatever the adapter put in the
+field, so declare it there:
+
+```yaml
+graders:
+  - type: numeric
+    fields:
+      - {ref: output.cost, range: {min: 0, max: null}}
+      - {ref: output.quality, range: [1, 5]}
+      # `range` and `max` are different things - see below
+      - {ref: output.latency, range: {min: 0}, max: 2.5}
+```
+
+That last field is the distinction worth knowing. `min`/`max` on a numeric
+field are a **pass/fail threshold** - "fail this case over 2.5 seconds" - and
+`range` is what the number could be at all. A cost that must stay under a
+dollar can still cost five, so the two keys stay separate.
+
+`maximum: null` is a statement, not a gap - unbounded above, so not a fraction
+of anything, render it as it is. A metric with no range at all is different
+again: nobody declared one. The engine never infers a range from the values it
+sees, because "everything stayed under 1, so it must be a 0..1 metric" is what
+turns a run whose costs happened to stay cheap into percentages.
+
+Every `MetricDelta` carries `direction` (`improved`/`regressed`/`neutral`)
+alongside `delta`, because the sign of a number does not say what it means: a
+rise in `f1` is an improvement and a rise in `false_negative_rate` is a
+regression. Direction comes from the first of these that speaks to the metric:
+an explicit `metrics:` entry, `win_higher_is_better` for the win metric, or the
+guardrails themselves - a `max` or `must_not_increase` rule means you want the
+metric low, a `min` or `must_not_decrease` rule means high. So a suite that
+gates a metric already declares its direction and needs no `metrics:` block at
+all. A metric nothing speaks to is assumed higher-is-better; one fenced in on
+both sides by a band, or declared `neutral`, reports `direction: neutral` and
+`higher_is_better: null` rather than guessing. Pick guardrails for the failures that must never ship, and
 one win metric for the improvement you're hunting; everything else is
 reported informationally.
 
 ## 7. Running it
 
-**CLI** (plug-ins first, so custom types register):
+**CLI** (a suite's `plugins:` registers its custom types; add `--plugins
+mod1,mod2` before the subcommand for anything the suite does not declare):
 
 ```bash
 # one variant -> scorecard (optionally saved)
-evalcore --plugins my_service.graders run \
+evalcore run \
     --suite my_service/suite.yaml --variant candidate --mode replay \
     --out candidate.scorecard.json --revision "$GIT_SHA"
 
 # the CI workhorse: run baseline+candidate, compare, exit 1 on 'fail'
-evalcore --plugins my_service.graders gate \
+evalcore gate \
     --suite my_service/suite.yaml --mode replay \
     --export outbox.jsonl --revision "$GIT_SHA"
 

{evalcore-2.4.3 → evalcore-2.6.0}/docs/design.md

@@ -116,6 +116,15 @@ A consumer adds an eval tree (its own repo, or a directory like
 4. **Suite + threshold config** - guardrail metrics, win metric + dead band,
    N samples, which triggers run which suite.
 
+A consumer that writes custom graders or adapters adds one more line: a
+`plugins:` list in the suite naming the modules to import, so the registry
+holds its `type`s before they are looked up. Registration is an import side
+effect, and nothing in the engine imports a consumer's module on its own. The
+import happens when a **run** starts, not when a suite is loaded - `load_suite`
+parses and hashes YAML and executes nothing, so reading, hashing, or reporting
+on a suite is free of side effects, while starting a run is already the point
+where the suite's adapter gets to make network calls.
+
 The engine supplies runner, comparison, gate, store, reporters, and the CLI.
 That ratio - four data files vs. a whole engine - is the genericity test: if
 onboarding a consumer ever requires editing `src/evalcore/`, that's an

{evalcore-2.4.3 → evalcore-2.6.0}/examples/quickstart/graders.py

@@ -17,10 +17,6 @@ import re
 from evalcore import models
 from evalcore.graders import base
 
-# Registering the adapter alongside the graders means one plug-in module wires
-# up the whole consumer (adapter + custom graders).
-from examples.quickstart import adapter  # noqa: F401
-
 _ACK = re.compile(
     r"\b(sorry|understand|happy to help|you're right|apolog)", re.IGNORECASE
 )
@@ -51,6 +47,10 @@ class AcknowledgesCustomer:
                 grader=self.name,
                 metric=self.name,
                 value=1.0 if ok else 0.0,
+                # 1/0 per case, so the run mean is a share of cases. A
+                # grader knows this about its own output; nothing
+                # downstream can work it out from the numbers alone.
+                value_range=models.MetricRange(minimum=0.0, maximum=1.0),
                 passed=ok,
                 detail=detail,
                 case_id=case.id,
@@ -87,6 +87,7 @@ class DistinctReplyRate:
                 grader=self.name,
                 metric=self.name,
                 value=rate,
+                value_range=models.MetricRange(minimum=0.0, maximum=1.0),
                 detail=f'{distinct} distinct across {len(by_case)} cases',
                 kind='aggregate',
             )

{evalcore-2.4.3 → evalcore-2.6.0}/examples/quickstart/run_eval.py

@@ -23,7 +23,6 @@ import os
 import pathlib
 import sys
 
-import examples.quickstart.graders  # noqa: F401 - registers plug-ins
 from evalcore import compare, loader, report, runner, store
 
 HERE = pathlib.Path(__file__).resolve().parent
@@ -91,17 +90,14 @@ def main(argv: list[str] | None = None) -> int:
         raise RuntimeError('run JSON round-trip mismatch')
 
     # 6. Flatten to self-describing rows and append the outbox a column-store
-    # shipper would drain: scorecard metrics AND per-sample score rows.
-    exporter = store.JsonlOutboxExporter(out / 'outbox.jsonl')
-    metric_rows = exporter.export(baseline) + exporter.export(candidate)
-    score_exporter = store.JsonlOutboxExporter(out / 'scores.jsonl')
-    n_scores = score_exporter.export_scores(
-        baseline_run
-    ) + score_exporter.export_scores(candidate_run)
-    print(f'\noutbox: {metric_rows} metric rows -> {exporter.outbox_path}')
-    print(
-        f'scores: {n_scores} per-sample rows -> {score_exporter.outbox_path}'
+    # shipper would drain. One grain only - per-sample score rows. A run's
+    # scorecard is a read-time aggregation over these, so exporting it too
+    # would store something derived that could disagree with them.
+    exporter = store.JsonlOutboxExporter(out / 'scores.jsonl')
+    n_scores = exporter.export_scores(baseline_run) + exporter.export_scores(
+        candidate_run
     )
+    print(f'\nscores: {n_scores} per-sample rows -> {exporter.outbox_path}')
 
     # 7. Gate semantics: non-zero exit on a failing verdict.
     print(f'\ngate verdict: {result.verdict} ({result.summary})')

{evalcore-2.4.3 → evalcore-2.6.0}/examples/quickstart/suite.yaml

@@ -10,8 +10,7 @@
 #   - a head-to-head pairwise config
 #
 # Run it:
-#   evalcore --plugins examples.quickstart.graders gate \
-#       --suite examples/quickstart/suite.yaml --mode replay
+#   evalcore gate --suite examples/quickstart/suite.yaml --mode replay
 #
 # Paths are relative to this file.
 
@@ -22,6 +21,14 @@ dataset_version: v1
 mode_default: replay
 replay_fixtures: fixtures/support_reply_replay.yaml
 
+# This consumer's own modules. The runner imports them before it looks up any
+# `type` below, which is what makes `canned_support` and the two custom graders
+# resolvable - so this suite runs from the CLI, from run_eval.py, or from a
+# test with nothing to remember at the call site.
+plugins:
+  - examples.quickstart.adapter
+  - examples.quickstart.graders
+
 # The custom offline stub adapter (examples/quickstart/adapter.py). In replay
 # mode the engine ignores this and reads replay_fixtures instead; in any other
 # mode it builds this adapter (which needs no network).

{evalcore-2.4.3 → evalcore-2.6.0}/examples/quickstart/tests/test_quickstart.py

@@ -9,7 +9,6 @@ the custom adapter and the custom graders.
 import pathlib
 import unittest
 
-import examples.quickstart.graders  # noqa: F401 - registers adapter + graders
 from evalcore import compare, loader, runner
 
 SUITE_PATH = pathlib.Path(__file__).resolve().parents[1] / 'suite.yaml'

{evalcore-2.4.3 → evalcore-2.6.0}/justfile

@@ -23,7 +23,7 @@ lint:
 
 # Run the quickstart suite offline against recorded fixtures.
 example:
-    uv run evalcore --plugins examples.quickstart.graders gate --suite examples/quickstart/suite.yaml --mode replay
+    uv run evalcore gate --suite examples/quickstart/suite.yaml --mode replay
 
 # Run the quickstart suite through the Python API (no CLI), offline.
 example-api:
@@ -31,4 +31,4 @@ example-api:
 
 # Head-to-head A-vs-B win-rate over the quickstart suite (offline).
 example-pairwise:
-    uv run evalcore --plugins examples.quickstart.graders pairwise --suite examples/quickstart/suite.yaml --a baseline --b candidate --mode replay
+    uv run evalcore pairwise --suite examples/quickstart/suite.yaml --a baseline --b candidate --mode replay

{evalcore-2.4.3 → evalcore-2.6.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "evalcore"
-version = "2.4.3"
+version = "2.6.0"
 description = "A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs."
 readme = "README.md"
 requires-python = ">=3.11"

{evalcore-2.4.3 → evalcore-2.6.0}/src/evalcore/cli.py

@@ -26,28 +26,25 @@ adapters register before the run.
 import argparse
 import asyncio
 import datetime
-import importlib
-import os
 import pathlib
 import sys
 
 from evalcore import compare as compare_mod
-from evalcore import loader, rating, report, reporters, runner, store
+from evalcore import loader, plugins, rating, report, reporters, runner, store
 from evalcore import pairwise as pairwise_mod
 from evalcore import sweep as sweep_mod
 
 
 def _load_plugins(spec: str | None) -> None:
-    if not spec:
-        return
-    # Consumers run the CLI from their repo root; the console script
-    # (unlike `python -m`) does not put the cwd on sys.path, so add it
-    # or `--plugins my_pkg.graders` could never import.
-    cwd = os.getcwd()
-    if cwd not in sys.path:
-        sys.path.insert(0, cwd)
-    for name in filter(None, spec.split(',')):
-        importlib.import_module(name.strip())
+    """Import the `--plugins` modules, and make cwd importable for both.
+
+    A suite's own `plugins:` list is imported by the runner, not here - but it
+    needs the same working-directory path fix, and this runs first on every
+    executing command, so it is done unconditionally rather than only when the
+    flag is passed.
+    """
+    plugins.allow_cwd_imports()
+    plugins.load((spec or '').split(','))
 
 
 def _now() -> str: