oryxeval 26.11.6__tar.gz

oryxeval-26.11.6/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Oryx Intelligence LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

oryxeval-26.11.6/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: oryxeval
+Version: 26.11.6
+Summary: Cheap, durable LLM evals: free and local, every arm cached so re-runs pay only for what changed, with confidence intervals and a verdict.
+Author-email: Oryx Intelligence LLC <dev@oryxintel.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/oryxintel/oryxeval
+Project-URL: Documentation, https://docs.oryxflow.dev/docs/llm-evals/
+Project-URL: Source, https://github.com/oryxintel/oryxeval
+Project-URL: Changelog, https://github.com/oryxintel/oryxeval/blob/main/CHANGELOG.md
+Keywords: llm,evals,llm-evaluation,prompt-engineering,pydantic-evals,llm-as-a-judge,a-b-testing,confidence-intervals,caching,oryxflow
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: oryxflow>=26.11.6
+Requires-Dist: pydantic-evals>=2
+Requires-Dist: scipy
+Requires-Dist: typer
+Requires-Dist: tenacity
+Requires-Dist: jinja2
+Requires-Dist: pyyaml
+Requires-Dist: pandas
+Requires-Dist: numpy
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# oryxeval
+
+**Cheap, durable LLM evals. Free, local, and you never pay twice for the same answer.**
+
+Every arm of an eval matrix (prompt x model x ...) is a cached cell, so a re-run pays only for what
+changed. You get a confidence interval and a plain-English verdict on whether the change is
+actually better. [pydantic-evals](https://pydantic.dev/docs/ai/evals/) does the scoring; oryxeval
+adds the matrix, the cache, the intervals and the verdict. Built on
+[oryxflow](https://github.com/oryxintel/oryxflow). Python 3.10+.
+
+```
+pip install oryxeval
+```
+
+## How it's different
+
+- **Free, and it runs where your code runs.** A Python library: no platform, no account, no
+  per-trace or per-score metering, no retention window. Your only bill is your model provider's,
+  and the cache keeps that small. Results are files on your disk.
+- **The cache is keyed on your code, not on the API request.** Most eval tools cache raw provider
+  responses. oryxeval caches the whole arm: the function under test, the cases it read and the
+  settings. So an agent that makes five model calls and a tool call per case is cached as one
+  unit, an edited function re-runs on its own, and nothing expires after a couple of weeks.
+- **Re-score for free.** Model outputs and scores are cached separately. Change a scorer or a
+  judge rubric and only the scoring re-runs, over outputs you already paid for.
+- **A verdict, not a dashboard.** Paired differences with intervals, a guardrail that must not
+  get worse, a judge-vs-human agreement check, and a last line that says "inside noise" when it
+  is.
+- **It tests your production code as-is.** The thing under test is an ordinary async Python
+  function that calls your real code. No YAML config, no rewriting it into a framework's
+  pipeline format.
+
+## The problem: prompt changes ship on vibes
+
+- **Noise passes for a win.** The same prompt can score 69% and then 64% an hour later. Without an
+  interval, a 5-point "improvement" is a coin flip you're about to ship.
+- **The headline metric hides the regression.** A prompt that answers *everything* scores 100% on
+  "did it answer", and also makes things up. A prompt that stays silent on hard cases looks great
+  on quality.
+- **Nobody checked the judge.** An LLM judge with no human labels behind it is an opinion with a
+  percentage printed next to it.
+- **Every re-run re-bills everything.** Tweak one prompt, or fix a typo in a judge rubric, and a
+  hand-rolled script calls the model for every case and every arm again. So people run fewer
+  evals, on fewer cases, and trust them less.
+- **Stale results look fresh.** A hand-rolled cache keyed on the wrong thing quietly serves
+  last week's outputs for this week's prompt.
+
+## What else you get
+
+- **Guardrails and coverage built in.** Name the metric that must not get worse (with a budget) and
+  a winner that breaks it is reported as *not a clean win*. A quality rate on a filtered subset is
+  shown next to how much it covers.
+- **A bill before you spend.** Every run prints the projected call count (and cost, if you give
+  it a per-call price) and asks first. One keystroke spends a single call to check credentials
+  before committing to the rest.
+- **Baselines from git.** Compare your working tree against the prompt as it was at any commit,
+  without copying files around.
+- **Where to look next.** Slice the best arm by any metadata column, worst first, and write a
+  side-by-side of every case where the arms disagree, straight from the stored outputs.
+
+## Quick example
+
+Write the async function you want to compare. Any keyword holding a list is an axis: two prompt
+versions make two arms.
+
+```python
+import oryxeval as ev
+from pydantic import BaseModel
+
+
+class Turn(BaseModel):
+    question: str
+    phrasing: str = 'plain'
+
+
+async def run_turn(inputs, prompt_version):
+    return await my_assistant(inputs.question, prompt=prompt_version)   # your code
+
+
+r = ev.sweep(run_turn,
+             cases=ev.load_cases('cases.csv', inputs=Turn),
+             prompt_version=['prod', 'preship'],
+             repeats=3,
+             metric=ev.Metric('accuracy', 'correct'),
+             guardrail=ev.Metric('false positives', 'false_positive',
+                                 higher_is_better=False, budget=0.05),
+             slices=('phrasing',))
+r.verdict()
+```
+
+Before anything is billed:
+
+```text
+run_turn · 20 cases × 2 arms × 3 reps = 120 calls
+  cached 0 · new 120   (+1 preflight call per new arm)
+
+Run 120 new calls for run_turn? (y/n/c=check one call first)
+```
+
+Then the verdict, with intervals. Run it again and it costs nothing:
+
+```text
+run_turn · 20 cases × 2 arms × 3 reps = 120 calls
+  cached 120 · new 0
+```
+
+## Let your coding agent run it
+
+The [oryxeval Claude Code plugin](https://github.com/oryxintel/oryxeval-claude-plugin) takes an
+agent from "I changed this prompt" to a verdict in four commands: plan the eval from your diff,
+scaffold it, harvest real cases from the repo, and run it. It always shows the bill and waits
+before spending.
+
+## Docs
+
+- [LLM evals quickstart](https://docs.oryxflow.dev/docs/llm-evals-quickstart/): a complete eval
+  in one file, with what each line does and what it prints.
+- [The guide](https://docs.oryxflow.dev/docs/llm-evals/): guardrails, judges, baselines, slices
+  and the method behind the verdict.
+
+## Coming from `oryxflow.evals`
+
+oryxeval was split out of `oryxflow.evals`. To keep an existing eval's cache, change only its
+import lines and no function body: the steps are in [CHANGELOG.md](CHANGELOG.md) (`26.11.6`).
+
+## License
+
+MIT

oryxeval-26.11.6/README.md

@@ -0,0 +1,128 @@
+# oryxeval
+
+**Cheap, durable LLM evals. Free, local, and you never pay twice for the same answer.**
+
+Every arm of an eval matrix (prompt x model x ...) is a cached cell, so a re-run pays only for what
+changed. You get a confidence interval and a plain-English verdict on whether the change is
+actually better. [pydantic-evals](https://pydantic.dev/docs/ai/evals/) does the scoring; oryxeval
+adds the matrix, the cache, the intervals and the verdict. Built on
+[oryxflow](https://github.com/oryxintel/oryxflow). Python 3.10+.
+
+```
+pip install oryxeval
+```
+
+## How it's different
+
+- **Free, and it runs where your code runs.** A Python library: no platform, no account, no
+  per-trace or per-score metering, no retention window. Your only bill is your model provider's,
+  and the cache keeps that small. Results are files on your disk.
+- **The cache is keyed on your code, not on the API request.** Most eval tools cache raw provider
+  responses. oryxeval caches the whole arm: the function under test, the cases it read and the
+  settings. So an agent that makes five model calls and a tool call per case is cached as one
+  unit, an edited function re-runs on its own, and nothing expires after a couple of weeks.
+- **Re-score for free.** Model outputs and scores are cached separately. Change a scorer or a
+  judge rubric and only the scoring re-runs, over outputs you already paid for.
+- **A verdict, not a dashboard.** Paired differences with intervals, a guardrail that must not
+  get worse, a judge-vs-human agreement check, and a last line that says "inside noise" when it
+  is.
+- **It tests your production code as-is.** The thing under test is an ordinary async Python
+  function that calls your real code. No YAML config, no rewriting it into a framework's
+  pipeline format.
+
+## The problem: prompt changes ship on vibes
+
+- **Noise passes for a win.** The same prompt can score 69% and then 64% an hour later. Without an
+  interval, a 5-point "improvement" is a coin flip you're about to ship.
+- **The headline metric hides the regression.** A prompt that answers *everything* scores 100% on
+  "did it answer", and also makes things up. A prompt that stays silent on hard cases looks great
+  on quality.
+- **Nobody checked the judge.** An LLM judge with no human labels behind it is an opinion with a
+  percentage printed next to it.
+- **Every re-run re-bills everything.** Tweak one prompt, or fix a typo in a judge rubric, and a
+  hand-rolled script calls the model for every case and every arm again. So people run fewer
+  evals, on fewer cases, and trust them less.
+- **Stale results look fresh.** A hand-rolled cache keyed on the wrong thing quietly serves
+  last week's outputs for this week's prompt.
+
+## What else you get
+
+- **Guardrails and coverage built in.** Name the metric that must not get worse (with a budget) and
+  a winner that breaks it is reported as *not a clean win*. A quality rate on a filtered subset is
+  shown next to how much it covers.
+- **A bill before you spend.** Every run prints the projected call count (and cost, if you give
+  it a per-call price) and asks first. One keystroke spends a single call to check credentials
+  before committing to the rest.
+- **Baselines from git.** Compare your working tree against the prompt as it was at any commit,
+  without copying files around.
+- **Where to look next.** Slice the best arm by any metadata column, worst first, and write a
+  side-by-side of every case where the arms disagree, straight from the stored outputs.
+
+## Quick example
+
+Write the async function you want to compare. Any keyword holding a list is an axis: two prompt
+versions make two arms.
+
+```python
+import oryxeval as ev
+from pydantic import BaseModel
+
+
+class Turn(BaseModel):
+    question: str
+    phrasing: str = 'plain'
+
+
+async def run_turn(inputs, prompt_version):
+    return await my_assistant(inputs.question, prompt=prompt_version)   # your code
+
+
+r = ev.sweep(run_turn,
+             cases=ev.load_cases('cases.csv', inputs=Turn),
+             prompt_version=['prod', 'preship'],
+             repeats=3,
+             metric=ev.Metric('accuracy', 'correct'),
+             guardrail=ev.Metric('false positives', 'false_positive',
+                                 higher_is_better=False, budget=0.05),
+             slices=('phrasing',))
+r.verdict()
+```
+
+Before anything is billed:
+
+```text
+run_turn · 20 cases × 2 arms × 3 reps = 120 calls
+  cached 0 · new 120   (+1 preflight call per new arm)
+
+Run 120 new calls for run_turn? (y/n/c=check one call first)
+```
+
+Then the verdict, with intervals. Run it again and it costs nothing:
+
+```text
+run_turn · 20 cases × 2 arms × 3 reps = 120 calls
+  cached 120 · new 0
+```
+
+## Let your coding agent run it
+
+The [oryxeval Claude Code plugin](https://github.com/oryxintel/oryxeval-claude-plugin) takes an
+agent from "I changed this prompt" to a verdict in four commands: plan the eval from your diff,
+scaffold it, harvest real cases from the repo, and run it. It always shows the bill and waits
+before spending.
+
+## Docs
+
+- [LLM evals quickstart](https://docs.oryxflow.dev/docs/llm-evals-quickstart/): a complete eval
+  in one file, with what each line does and what it prints.
+- [The guide](https://docs.oryxflow.dev/docs/llm-evals/): guardrails, judges, baselines, slices
+  and the method behind the verdict.
+
+## Coming from `oryxflow.evals`
+
+oryxeval was split out of `oryxflow.evals`. To keep an existing eval's cache, change only its
+import lines and no function body: the steps are in [CHANGELOG.md](CHANGELOG.md) (`26.11.6`).
+
+## License
+
+MIT

oryxeval-26.11.6/oryxeval/__init__.py

@@ -0,0 +1,107 @@
+"""LLM evaluation harness: one cached cell per arm of an eval matrix.
+
+Install it with::
+
+    pip install oryxeval
+
+It is a separate package from oryxflow, which it builds on. Import it explicitly::
+
+    import oryxeval as ev
+
+SEAM: the per-case DataFrame is the interface. Everything downstream of a run --
+``Metric``, the slices, the intervals, the verdict, the report -- reads only that
+frame and never touches a pydantic-evals object. The two override points on
+``TaskEval`` are the exit: ``_evaluate()`` returns a report, so swap it for a
+different runner, and ``_to_frame(report)`` returns the rows, so swap it for a
+different report shape. Nothing else has to change. The property that buys is
+available today rather than later: every metric, verdict and report test is
+written against a hand-built DataFrame, so everything downstream of a run is
+testable without pydantic-evals present.
+
+``cli`` and ``build_app`` are resolved on first use rather than at import, so
+``import oryxeval`` does not pay for Typer -- a sweep run from a notebook
+never needs a command line.
+"""
+
+import importlib
+
+from importlib.metadata import PackageNotFoundError, version as _version
+
+try:
+    __version__ = _version("oryxeval")
+except PackageNotFoundError:
+    __version__ = "unknown"
+
+from oryxflow import (Parameter, IntParameter, FloatParameter, BoolParameter,
+                      ChoiceParameter, ListParameter, DictParameter, DateParameter,
+                      EnumParameter, hash_files)
+
+from oryxeval.baseline import (
+    BaselineError,
+    PromptArm,
+    Variant,
+    VariantError,
+    git_sha,
+    git_tree,
+)
+from oryxeval.cases import RESERVED_METADATA, CaseList, load_cases
+from oryxeval.metric import Metric
+from oryxeval.stats import (DeltaCI, JudgeAlignment, RateCI, cluster_labels,
+                            corrected_rate, delta_ci, judge_alignment,
+                            rate_ci)
+from oryxeval.sweep import EvalResult, sweep
+from oryxeval.task import EvalFailureRateError, TaskEval
+
+__all__ = [
+    '__version__',
+    'BoolParameter',
+    'ChoiceParameter',
+    'DateParameter',
+    'DictParameter',
+    'EnumParameter',
+    'FloatParameter',
+    'IntParameter',
+    'ListParameter',
+    'Parameter',
+    'hash_files',
+    'BaselineError',
+    'CaseList',
+    'DeltaCI',
+    'EvalFailureRateError',
+    'EvalResult',
+    'JudgeAlignment',
+    'Metric',
+    'PromptArm',
+    'RESERVED_METADATA',
+    'RateCI',
+    'TaskEval',
+    'Variant',
+    'VariantError',
+    'build_app',
+    'cli',
+    'cluster_labels',
+    'corrected_rate',
+    'delta_ci',
+    'git_sha',
+    'git_tree',
+    'judge_alignment',
+    'load_cases',
+    'rate_ci',
+    'sweep',
+]
+
+_LAZY = ('cli', 'build_app')
+
+
+def __getattr__(name):
+    """Import the Typer-backed names only when one of them is actually used."""
+    if name in _LAZY:
+        module = importlib.import_module('oryxeval.cli')
+        value = module if name == 'cli' else getattr(module, name)
+        globals()[name] = value
+        return value
+    raise AttributeError("module {!r} has no attribute {!r}".format(__name__, name))
+
+
+def __dir__():
+    return sorted(set(list(globals()) + list(__all__)))

oryxeval-26.11.6/oryxeval/baseline.py

@@ -0,0 +1,260 @@
+"""The two ways to build a comparison arm, deliberately distinct.
+
+A baseline is *something that existed*, so it is checked out of git: byte-exact by
+construction, immune to the live files moving under it, and -- the reason this module
+does not simply ship a string-replacement helper -- able to restore a section that was
+DELETED. A probe is a rewrite that has not shipped anywhere, so there is no ref to check
+out and an anchored ``(old -> new)`` replacement is all there is.
+
+Both fail loudly. A comparison arm that silently becomes a no-op reports the old version
+as having scored well, which is indistinguishable from a real result and therefore the
+worst failure a comparison can have.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import subprocess
+from dataclasses import dataclass
+from pathlib import Path
+
+
+class BaselineError(RuntimeError):
+    """A baseline arm could not be built, or was built empty.
+
+    A ``RuntimeError``, because an empty or wrong baseline is not a recoverable
+    condition: whatever the arm went on to measure would not be the old version.
+    """
+
+
+class VariantError(ValueError):
+    """A variant's anchor text was not found in the rendered text.
+
+    A ``ValueError``, because the arguments no longer describe the text they are
+    applied to -- the anchor has been edited away, and applying the variant anyway
+    would report the unchanged prompt's numbers as the variant's.
+    """
+
+
+def _git(args, repo=None, binary=False):
+    """Run a git command, returning stdout (bytes when ``binary``). Raise on failure."""
+    cwd = str(repo) if repo is not None else os.getcwd()
+    proc = subprocess.run(['git'] + list(args), cwd=cwd,
+                          stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    if proc.returncode != 0:
+        raise BaselineError("git {} failed in {}: {}".format(
+            ' '.join(args), cwd, proc.stderr.decode('utf-8', 'replace').strip()))
+    return proc.stdout if binary else proc.stdout.decode('utf-8', 'replace')
+
+
+def git_sha(ref, repo=None):
+    """Resolve ``ref`` to its full commit SHA.
+
+    An arm built from a git ref must report *this* as its ``code_version()``, never the
+    ref string: ``HEAD~1`` names different content every time something is committed, and
+    a cache key that moves under a stable-looking name reads stale numbers as fresh ones.
+    A resolved SHA is immutable, so the arm's cache key is stable forever.
+    """
+    try:
+        out = _git(['rev-parse', '--verify', '{}^{{commit}}'.format(ref)], repo=repo)
+    except BaselineError:
+        out = _git(['rev-parse', '--verify', str(ref)], repo=repo)
+    sha = out.strip()
+    if not sha:
+        raise BaselineError("ref '{}' did not resolve to a commit".format(ref))
+    return sha
+
+
+def _as_list(paths):
+    if paths is None:
+        return []
+    if isinstance(paths, (str, bytes, os.PathLike)):
+        return [paths]
+    return list(paths)
+
+
+def _ls_tree(ref, paths, repo=None):
+    """Filenames under ``paths`` at ``ref``, recursively, git-style forward slashes."""
+    args = ['ls-tree', '-r', '-z', '--name-only', str(ref), '--']
+    args += [str(p).replace('\\', '/') for p in paths]
+    out = _git(args, repo=repo, binary=True).decode('utf-8', 'surrogateescape')
+    return [n for n in out.split('\0') if n]
+
+
+def _default_dest(sha, repo=None):
+    base = Path(repo) if repo is not None else Path.cwd()
+    root = base / '.oryxflow' / 'baseline'
+    root.mkdir(parents=True, exist_ok=True)
+    # The materialized tree is generated, never source: ignore it wherever it lands,
+    # so a user whose repo predates this never has to notice it.
+    ignore = root / '.gitignore'
+    if not ignore.exists():
+        ignore.write_text('*\n')
+    return root / sha[:12]
+
+
+def git_tree(ref, paths, dest=None, expect_absent=None, repo=None):
+    """Materialize files from a git ref into a directory, for a BASELINE arm.
+
+    Use this for a prompt/template tree as it was before a change: it is
+    byte-exact by construction, cannot decay as the live files move, and -- the
+    reason it exists -- it restores DELETED files, which a string-replacement
+    variant cannot do at all.
+
+    ``ref`` is anything git resolves ('abc123^', 'v2.1', 'main~3'). Because a
+    resolved ref is immutable, an arm built from one has a cache key that is
+    stable forever.
+
+    Raises RuntimeError when the ref yields no files under ``paths`` -- a
+    silently empty baseline reads as "the old version already scored well",
+    which is the worst possible failure for a comparison.
+
+    ``expect_absent=`` names files that must NOT exist at this ref; their
+    presence means the ref is wrong and the arm is measuring nothing.
+
+    Returns the directory the files were written to. Point your own loader at it --
+    how your templates are loaded is yours to know, so nothing here is patched or
+    installed on your behalf. The default location is keyed by the resolved SHA,
+    beside the eval and gitignored, so a ref is materialized once and reused.
+    Pass ``repo=`` to run git somewhere other than the working directory, and use
+    ``git_sha(ref)`` for the arm's ``code_version()``.
+    """
+    paths = _as_list(paths)
+    if not paths:
+        raise BaselineError(
+            "git_tree(ref={!r}) needs at least one path; materializing nothing would "
+            "give an empty baseline".format(ref))
+    sha = git_sha(ref, repo=repo)
+
+    for name in _as_list(expect_absent):
+        found = _ls_tree(sha, [name], repo=repo)
+        if found:
+            raise BaselineError(
+                "ref '{}' ({}) still has {} -- expect_absent said it should not exist "
+                "there, so this is not the ref you meant and the arm would measure "
+                "nothing.".format(ref, sha[:10], ', '.join(sorted(found))))
+
+    names = _ls_tree(sha, paths, repo=repo)
+    if not names:
+        raise BaselineError(
+            "ref '{}' ({}) has no files under {} -- an empty baseline reads as 'the old "
+            "version already scored well', so it is refused rather than measured.".format(
+                ref, sha[:10], ', '.join(repr(str(p)) for p in paths)))
+
+    dest = Path(dest) if dest is not None else _default_dest(sha, repo=repo)
+    dest.mkdir(parents=True, exist_ok=True)
+    for name in names:
+        target = dest.joinpath(*name.split('/'))
+        blob = _git(['show', '{}:{}'.format(sha, name)], repo=repo, binary=True)
+        if target.exists() and target.read_bytes() == blob:
+            continue
+        target.parent.mkdir(parents=True, exist_ok=True)
+        target.write_bytes(blob)
+    manifest = dest / '_oryxflow_baseline.json'
+    manifest.write_text(json.dumps(
+        {'ref': str(ref), 'sha': sha, 'paths': [str(p) for p in paths], 'files': names},
+        indent=2, sort_keys=True))
+    return dest
+
+
+@dataclass
+class Variant:
+    """An exact (old -> new) replacement applied to already-rendered text, for a
+    PROBE arm -- a rewrite that has not shipped, so there is no ref to check out.
+
+    Raises when ``old`` is not present: a variant whose anchor text has been
+    edited away silently becomes a no-op, and then reports the unchanged prompt's
+    numbers as the variant's.
+
+    NOT for baselines. Use ``git_tree`` there: a replacement cannot restore a
+    deleted section, and cannot put one back in its original position.
+    """
+    old: str
+    new: str
+
+    def apply(self, text):
+        """Return ``text`` with ``old`` replaced by ``new``. Raise ``VariantError``
+        (a ``ValueError``) when ``old`` is absent, rather than returning the text
+        unchanged."""
+        if self.old not in text:
+            raise VariantError(
+                "variant anchor not found in the rendered text (anchor {!r}...): the "
+                "text it was written against has changed, so this variant would be a "
+                "no-op and its arm would report the unchanged prompt's numbers.".format(
+                    self.old[:80]))
+        return text.replace(self.old, self.new)
+
+    def __call__(self, text):
+        return self.apply(text)
+
+    def code_version(self):
+        """A token that changes when this replacement changes, for an arm's cache key."""
+        h = hashlib.md5()
+        h.update(self.old.encode('utf-8'))
+        h.update(b'\0')
+        h.update(self.new.encode('utf-8'))
+        return 'variant-' + h.hexdigest()[:10]
+
+
+@dataclass
+class PromptArm:
+    """Where one arm reads its prompt files, AND the cache key that matches them.
+
+    The two have to agree, and nothing else makes them. Written by hand it is two
+    places -- a ``prompt_root(arm)`` that returns a directory and a
+    ``code_version()`` that returns a token -- and the failure when they drift is
+    silent and bad: a baseline arm reading a checked-out tree but keyed on a hash
+    of the LIVE files re-runs when a live prompt is edited (merely wasteful) and
+    does NOT re-run when you point it at a different ref (stale, and reported as
+    fresh). One object owns both, so it cannot happen.
+
+        ARMS = {
+            'live':    ev.PromptArm(globs=['prompts/*.jinja2'], root=ROOT),
+            'baseline': ev.PromptArm(ref='v2.1', paths=['prompts/'], root=ROOT),
+        }
+
+        # in the agent:  ARMS[arm].dir()
+        # in the task:   def code_version(self): return ARMS[self.arm].code_version()
+
+    ``ref=None`` means the working tree, keyed on the CONTENT of ``globs``. A
+    ``ref`` means that commit, materialized with :py:func:`git_tree` and keyed on
+    the resolved sha -- immutable, so a baseline arm's cell stays valid for good.
+
+    ``subdir`` is applied to both, so the two arms hand the loader the same
+    layout and it never has to know which one it got.
+    """
+    ref: str = None
+    paths: tuple = ()
+    globs: tuple = ()
+    root: str = None
+    subdir: str = ''
+    expect_absent: tuple = ()
+    dest: str = None
+
+    def __post_init__(self):
+        if self.ref is None and not self.globs:
+            raise ValueError(
+                'PromptArm: the working-tree arm needs globs= (the files whose CONTENT '
+                'is its cache key) -- without them the arm is keyed on nothing and an '
+                'edited prompt is served from cache')
+        if self.ref is not None and not self.paths:
+            raise ValueError(
+                'PromptArm: a ref arm needs paths= (what to materialize from {!r})'.format(
+                    self.ref))
+
+    def dir(self):
+        """The directory this arm reads its prompts from."""
+        base = Path(self.root) if self.root is not None else Path.cwd()
+        if self.ref is not None:
+            base = git_tree(self.ref, self.paths, dest=self.dest,
+                            expect_absent=self.expect_absent, repo=self.root)
+        return base.joinpath(*self.subdir.split('/')) if self.subdir else base
+
+    def code_version(self):
+        """The cache token for the files :py:meth:`dir` will hand out."""
+        if self.ref is not None:
+            return git_sha(self.ref, repo=self.root)
+        from oryxflow.utils import hash_files
+        return hash_files(*self.globs, root=self.root)