repeatability-evals 0.1.0__tar.gz → 0.2.0__tar.gz

{repeatability_evals-0.1.0 → repeatability_evals-0.2.0}/.gitignore

@@ -4,3 +4,4 @@ __pycache__/
 .pytest_cache/
 .ruff_cache/
 dist/
+examples/demo_report.md

repeatability_evals-0.2.0/PKG-INFO

@@ -0,0 +1,155 @@
+Metadata-Version: 2.5
+Name: repeatability-evals
+Version: 0.2.0
+Summary: Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.
+Project-URL: Homepage, https://github.com/krishna-gogineni-765/repeatability-evals
+Project-URL: Issues, https://github.com/krishna-gogineni-765/repeatability-evals/issues
+Author: Krishna Gogineni
+License: MIT
+License-File: LICENSE
+Keywords: agents,consistency,evaluation,llm,llm-as-judge,non-determinism,rag,reliability,repeatability,retrieval
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# repeatability-evals
+
+**Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.**
+
+Most evals ask *"is the output correct?"* and need gold labels to answer. This library asks
+two questions that are extremely important in enterprise settinga and need **no gold labels**:
+
+1. **Is my system consistent/repeatable with itself?** Same input, N runs: do the answers, the retrieved
+   documents, the tool-call traces agree?
+2. **Did anything user facing change when I changed something in the backend?** Model upgrade, prompt edit, new index,
+   framework bump: which inputs behave differently, *beyond the system's own run-to-run noise*,
+   and *how* (facts? numbers? format? refusals?).
+
+## Why it matters
+
+| If you are... | You can... |
+|---|---|
+| **a platform or model provider** shipping an update | run it offline on a regression set and see which customer-style workflows would shift before you release |
+| **an enterprise on a vendor's LLM or agent platform** | run it yourself, on your own data and cluster, to decide whether an upstream upgrade is safe for the workflows you built. Nothing leaves your environment, and `include_text=False` keeps raw text out of reports |
+| **building an agent, RAG or harness product** | track whether retrieval and answers stay stable across runs, and catch silent drift from non-deterministic infrastructure |
+| **without labelled data** | still get a signal: consistency needs only outputs, not answers |
+
+
+## Usage pattern 1: See what changed
+
+```python
+from repeatability_evals import compare_versions, RubricJudge
+
+# {input_id: [output of run 1, run 2, run 3, ...]} for each version. No labels.
+report = compare_versions(v1_runs, v2_runs, RubricJudge(my_llm), names=("v1", "v2"))
+print(report.verdict)            # no_meaningful_change | changed | inconclusive
+report.save("report.md")         # also .html / .json
+```
+
+```
+# v1 vs v2: consistency report
+CHANGED: the candidate's outputs differ from the baseline's by more than run-to-run noise explains.
+120 inputs compared; 34% changed beyond the noise floor (per-input drift > 0.1).
+
+| v1 vs itself (noise floor)               | 0.925 |
+| v2 vs itself                             | 0.948 |
+| v1 vs v2                                 | 0.850 |
+| drift (expected - cross; 0 = no change)  | +0.087 (95% CI +0.060 to +0.116) |
+
+## What changed
+| format            | +0.200 |
+| specifics         | +0.115 |
+| supporting_detail | +0.100 |
+| core_facts        | +0.057 |
+Changed inputs by main driver: 21 mainly in `format`, 20 mainly in `specifics`.
+
+### `q000`: drift +0.62 (cross 0.38 vs expected 1.00)
+- Main driver: `specifics`
+- Judge: Retention period differs.
+- v1: Retention for gold is 30 days.
+- v2: - Retention for gold is 90 days. - Applies to all sites.
+```
+
+*(Real output of `examples/compare_versions_demo.py`, which uses simulated runs and a scripted
+stand-in for the LLM judge so it runs without an API key.)*
+
+The key idea is the **noise floor**. A non-deterministic system never matches itself perfectly,
+so comparing v2 to v1 naively overstates change. We measure how much v1 differs from *itself*
+and v2 from *itself*, and report only the **extra** disagreement across versions (`drift`).
+Two equally noisy systems show no drift; a real change does. The report also says *what kind*
+of change it is, since format drift and changed numbers deserve different decisions.
+
+From the shell, with no code and no model (offline n-gram similarity), or with your own LLM:
+
+```bash
+pip install repeatability-evals
+repeatability-evals compare v1.json v2.json --out report.md
+repeatability-evals compare v1.json v2.json --judge mypkg.llm:complete --rubric --out report.html
+```
+
+Input is JSON: `{"input id": ["run 1", "run 2", ...]}`. The exit code is `0` (no meaningful
+change), `1` (inconclusive) or `2` (changed), so it can gate a CI pipeline.
+
+## Usage pattern 2: Measure self-consistency
+
+```python
+from repeatability_evals import PairwiseConsistency, JaccardSimilarity, NgramJaccard, RubricJudge
+
+PairwiseConsistency(JaccardSimilarity()).evaluate(chunk_ids_per_run)                 # retrieved IDs
+PairwiseConsistency(NgramJaccard(3)).evaluate(chunk_texts_per_run)                   # robust to re-chunking
+PairwiseConsistency(RubricJudge(my_llm)).evaluate(answers, context={"query": q})     # answers
+```
+
+Over a dataset, `evaluate_groups({input_id: runs, ...}, metric)` returns the mean, a bootstrap
+confidence interval and the least consistent inputs.
+
+## Design
+
+The library never knows whether you are comparing answers, chunk IDs or agent traces. It
+compares *items*; you pick the similarity metric that makes sense for what you are comparing.
+
+* **`Similarity`** scores two items in `[0, 1]`: sets (`JaccardSimilarity`, `DiceSimilarity`,
+  `OverlapCoefficient`), text (`ExactMatch`, `NgramJaccard`, `RougeL`, `EditSimilarity`),
+  rankings (`RankBiasedOverlap`), embeddings (`EmbeddingCosine`, bring your own embed
+  function) and LLM judges (`RubricJudge`, `LLMJudge`). Subclass it, or pass a plain function.
+* **`PairSampler`** picks which of the N(N-1)/2 pairs to compare. The default `CycleSampler`
+  costs N comparisons and touches every run twice; `CrossSampler` does the same across versions.
+* **Metrics**: `PairwiseConsistency` (mean similarity), `ClusterConsistency` (largest
+  "same answer" cluster, plus entropy), `TotalAgreement` (all N identical) and `compare_versions`.
+
+Everything has an async twin (`aevaluate`, `acompare_versions`) with bounded concurrency. The
+core has **no dependencies**; you bring the model, as any `str -> str` function.
+
+### The judge
+
+`RubricJudge` scores five dimensions of agreement in one call: `core_facts`, `specifics`,
+`supporting_detail`, `stance` (answer vs hedge vs refuse) and `format`. The headline is a
+weighted mean (core facts count most) and the per-dimension means explain *what* varies. Pass
+your own `Dimension`s to change them. `LLMJudge` is the single 1-5 semantic-equivalence score.
+`symmetric=True` judges both orders to cancel position bias. **Validate any judge against human
+ratings on your own data before relying on it.**
+
+### Sampling cost
+
+All pairs for 10 runs is 45 judge calls; the default sampler uses 10. On synthetic data the
+per-input error versus all pairs was about 0.03 with no bias (`benchmarks/sampling_error.py`);
+check it on your data, or use `AllPairs()` for the exact value.
+
+### Writing your own similarity
+
+```python
+from repeatability_evals import Similarity, PairwiseConsistency
+
+class SameFirstSentence(Similarity):
+    name = "same_first_sentence"
+    def compute(self, a, b, *, context=None):
+        return float(a.split(".")[0] == b.split(".")[0])
+
+PairwiseConsistency(SameFirstSentence())      # or PairwiseConsistency(lambda a, b: ...)
+```
+

repeatability_evals-0.2.0/README.md

@@ -0,0 +1,136 @@
+# repeatability-evals
+
+**Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.**
+
+Most evals ask *"is the output correct?"* and need gold labels to answer. This library asks
+two questions that are extremely important in enterprise settinga and need **no gold labels**:
+
+1. **Is my system consistent/repeatable with itself?** Same input, N runs: do the answers, the retrieved
+   documents, the tool-call traces agree?
+2. **Did anything user facing change when I changed something in the backend?** Model upgrade, prompt edit, new index,
+   framework bump: which inputs behave differently, *beyond the system's own run-to-run noise*,
+   and *how* (facts? numbers? format? refusals?).
+
+## Why it matters
+
+| If you are... | You can... |
+|---|---|
+| **a platform or model provider** shipping an update | run it offline on a regression set and see which customer-style workflows would shift before you release |
+| **an enterprise on a vendor's LLM or agent platform** | run it yourself, on your own data and cluster, to decide whether an upstream upgrade is safe for the workflows you built. Nothing leaves your environment, and `include_text=False` keeps raw text out of reports |
+| **building an agent, RAG or harness product** | track whether retrieval and answers stay stable across runs, and catch silent drift from non-deterministic infrastructure |
+| **without labelled data** | still get a signal: consistency needs only outputs, not answers |
+
+
+## Usage pattern 1: See what changed
+
+```python
+from repeatability_evals import compare_versions, RubricJudge
+
+# {input_id: [output of run 1, run 2, run 3, ...]} for each version. No labels.
+report = compare_versions(v1_runs, v2_runs, RubricJudge(my_llm), names=("v1", "v2"))
+print(report.verdict)            # no_meaningful_change | changed | inconclusive
+report.save("report.md")         # also .html / .json
+```
+
+```
+# v1 vs v2: consistency report
+CHANGED: the candidate's outputs differ from the baseline's by more than run-to-run noise explains.
+120 inputs compared; 34% changed beyond the noise floor (per-input drift > 0.1).
+
+| v1 vs itself (noise floor)               | 0.925 |
+| v2 vs itself                             | 0.948 |
+| v1 vs v2                                 | 0.850 |
+| drift (expected - cross; 0 = no change)  | +0.087 (95% CI +0.060 to +0.116) |
+
+## What changed
+| format            | +0.200 |
+| specifics         | +0.115 |
+| supporting_detail | +0.100 |
+| core_facts        | +0.057 |
+Changed inputs by main driver: 21 mainly in `format`, 20 mainly in `specifics`.
+
+### `q000`: drift +0.62 (cross 0.38 vs expected 1.00)
+- Main driver: `specifics`
+- Judge: Retention period differs.
+- v1: Retention for gold is 30 days.
+- v2: - Retention for gold is 90 days. - Applies to all sites.
+```
+
+*(Real output of `examples/compare_versions_demo.py`, which uses simulated runs and a scripted
+stand-in for the LLM judge so it runs without an API key.)*
+
+The key idea is the **noise floor**. A non-deterministic system never matches itself perfectly,
+so comparing v2 to v1 naively overstates change. We measure how much v1 differs from *itself*
+and v2 from *itself*, and report only the **extra** disagreement across versions (`drift`).
+Two equally noisy systems show no drift; a real change does. The report also says *what kind*
+of change it is, since format drift and changed numbers deserve different decisions.
+
+From the shell, with no code and no model (offline n-gram similarity), or with your own LLM:
+
+```bash
+pip install repeatability-evals
+repeatability-evals compare v1.json v2.json --out report.md
+repeatability-evals compare v1.json v2.json --judge mypkg.llm:complete --rubric --out report.html
+```
+
+Input is JSON: `{"input id": ["run 1", "run 2", ...]}`. The exit code is `0` (no meaningful
+change), `1` (inconclusive) or `2` (changed), so it can gate a CI pipeline.
+
+## Usage pattern 2: Measure self-consistency
+
+```python
+from repeatability_evals import PairwiseConsistency, JaccardSimilarity, NgramJaccard, RubricJudge
+
+PairwiseConsistency(JaccardSimilarity()).evaluate(chunk_ids_per_run)                 # retrieved IDs
+PairwiseConsistency(NgramJaccard(3)).evaluate(chunk_texts_per_run)                   # robust to re-chunking
+PairwiseConsistency(RubricJudge(my_llm)).evaluate(answers, context={"query": q})     # answers
+```
+
+Over a dataset, `evaluate_groups({input_id: runs, ...}, metric)` returns the mean, a bootstrap
+confidence interval and the least consistent inputs.
+
+## Design
+
+The library never knows whether you are comparing answers, chunk IDs or agent traces. It
+compares *items*; you pick the similarity metric that makes sense for what you are comparing.
+
+* **`Similarity`** scores two items in `[0, 1]`: sets (`JaccardSimilarity`, `DiceSimilarity`,
+  `OverlapCoefficient`), text (`ExactMatch`, `NgramJaccard`, `RougeL`, `EditSimilarity`),
+  rankings (`RankBiasedOverlap`), embeddings (`EmbeddingCosine`, bring your own embed
+  function) and LLM judges (`RubricJudge`, `LLMJudge`). Subclass it, or pass a plain function.
+* **`PairSampler`** picks which of the N(N-1)/2 pairs to compare. The default `CycleSampler`
+  costs N comparisons and touches every run twice; `CrossSampler` does the same across versions.
+* **Metrics**: `PairwiseConsistency` (mean similarity), `ClusterConsistency` (largest
+  "same answer" cluster, plus entropy), `TotalAgreement` (all N identical) and `compare_versions`.
+
+Everything has an async twin (`aevaluate`, `acompare_versions`) with bounded concurrency. The
+core has **no dependencies**; you bring the model, as any `str -> str` function.
+
+### The judge
+
+`RubricJudge` scores five dimensions of agreement in one call: `core_facts`, `specifics`,
+`supporting_detail`, `stance` (answer vs hedge vs refuse) and `format`. The headline is a
+weighted mean (core facts count most) and the per-dimension means explain *what* varies. Pass
+your own `Dimension`s to change them. `LLMJudge` is the single 1-5 semantic-equivalence score.
+`symmetric=True` judges both orders to cancel position bias. **Validate any judge against human
+ratings on your own data before relying on it.**
+
+### Sampling cost
+
+All pairs for 10 runs is 45 judge calls; the default sampler uses 10. On synthetic data the
+per-input error versus all pairs was about 0.03 with no bias (`benchmarks/sampling_error.py`);
+check it on your data, or use `AllPairs()` for the exact value.
+
+### Writing your own similarity
+
+```python
+from repeatability_evals import Similarity, PairwiseConsistency
+
+class SameFirstSentence(Similarity):
+    name = "same_first_sentence"
+    def compute(self, a, b, *, context=None):
+        return float(a.split(".")[0] == b.split(".")[0])
+
+PairwiseConsistency(SameFirstSentence())      # or PairwiseConsistency(lambda a, b: ...)
+```
+

repeatability_evals-0.2.0/examples/compare_versions_demo.py

@@ -0,0 +1,57 @@
+"""Simulated demo: compare two versions of a Q&A system with no gold labels.
+
+No API key needed. The "system" and the "judge" are scripted stand-ins so the demo is
+deterministic; in real use, `judge` is a call to your LLM and the runs come from your system.
+Run:  python examples/compare_versions_demo.py
+"""
+import json
+import random
+import re
+
+from repeatability_evals import DEFAULT_DIMENSIONS, RubricJudge, compare_versions
+
+rng = random.Random(7)
+POLICIES = ["gold", "silver", "bronze", "archive", "dr", "edge"]
+
+
+def answer(policy, days, bullets, noisy):
+    """One run of a pretend system. `noisy` runs sometimes reword (same meaning)."""
+    lead = rng.choice(["Retention for", "The retention period of"]) if noisy else "Retention for"
+    if noisy and rng.random() < 0.08:  # occasional flaky run: a wrong number
+        days += 1
+    body = f"{lead} {policy} is {days} days."
+    return f"- {body}\n- Applies to all sites." if bullets else body
+
+
+def version(days_for, bullets_for):
+    runs = {}
+    for q in range(120):
+        p = POLICIES[q % len(POLICIES)]
+        runs[f"q{q:03d}"] = [answer(p, days_for(q), bullets_for(q), True) for _ in range(4)]
+    return runs
+
+
+v1 = version(lambda q: 30, lambda q: False)
+# v2 changes the retention number on ~12% of questions and switches ~20% to bulleted format.
+v2 = version(lambda q: 90 if q % 8 == 0 else 30, lambda q: q % 5 == 0)
+
+
+def judge(prompt):
+    """Scripted stand-in for an LLM: scores each dimension by simple rules."""
+    a = prompt.split("Response A:")[1].split("Response B:")[0]
+    b = prompt.split("Response B:")[1].split("Score each dimension")[0]
+    num = lambda t: re.findall(r"\d+", t)  # noqa: E731
+    bullets = lambda t: t.lstrip().startswith("-")  # noqa: E731
+    scores = {d.name: 5 for d in DEFAULT_DIMENSIONS}
+    if num(a)[:1] != num(b)[:1]:
+        scores["specifics"], scores["core_facts"] = 1, 3
+    if bullets(a) != bullets(b):
+        scores["format"], scores["supporting_detail"] = 1, 3
+    why = "Retention period differs." if scores["specifics"] == 1 else (
+        "Same facts; formatting differs." if scores["format"] == 1 else "Equivalent.")
+    return json.dumps({"reason": why, "scores": scores})
+
+
+report = compare_versions(v1, v2, RubricJudge(judge), names=("v1", "v2"))
+print(report.to_markdown(top=2))
+report.save("examples/demo_report.md")

{repeatability_evals-0.1.0 → repeatability_evals-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "repeatability-evals"
-version = "0.1.0"
+version = "0.2.0"
 description = "Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows."
 readme = "README.md"
 license = { text = "MIT" }
@@ -35,6 +35,9 @@ src = ["src", "tests"]
 select = ["E", "F", "I", "B", "UP"]
 ignore = ["UP006", "UP007", "UP035", "UP045"]  # keep typing.Optional/List for py3.9 readability
 
+[project.scripts]
+repeatability-evals = "repeatability_evals.cli:main"
+
 [project.urls]
 Homepage = "https://github.com/krishna-gogineni-765/repeatability-evals"
 Issues = "https://github.com/krishna-gogineni-765/repeatability-evals/issues"

{repeatability_evals-0.1.0 → repeatability_evals-0.2.0}/src/repeatability_evals/__init__.py

@@ -1,8 +1,9 @@
 """repeatability-evals: consistency evals for non-deterministic LLM systems."""
+from .compare import InputComparison, VersionComparison, acompare_versions, compare_versions
 from .dataset import DatasetResult, aevaluate_groups, evaluate_groups
 from .metrics import ClusterConsistency, ConsistencyMetric, PairwiseConsistency, TotalAgreement
 from .prompts import DEFAULT_DIMENSIONS, SEMANTIC_EQUIVALENCE_PROMPT, Dimension, build_rubric_prompt
-from .sampling import AllPairs, AnchorPairs, CycleSampler, PairSampler, RandomPairs, cycle_pairs
+from .sampling import AllPairs, AnchorPairs, CrossSampler, CycleSampler, PairSampler, RandomPairs, cycle_pairs
 from .similarity import (
     DiceSimilarity,
     EditSimilarity,
@@ -26,6 +27,7 @@ from .types import ConsistencyResult, PairScore, SimilarityResult
 __version__ = "0.1.0"
 
 __all__ = [
+    "compare_versions", "acompare_versions", "VersionComparison", "InputComparison", "CrossSampler",
     "ConsistencyMetric", "PairwiseConsistency", "ClusterConsistency", "TotalAgreement",
     "evaluate_groups", "aevaluate_groups", "DatasetResult", "bootstrap_ci",
     "Similarity", "FunctionSimilarity", "as_similarity",

repeatability_evals-0.2.0/src/repeatability_evals/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

repeatability_evals-0.2.0/src/repeatability_evals/cli.py

@@ -0,0 +1,98 @@
+"""Command line: compare two versions of a system from saved outputs, no gold labels.
+
+    repeatability-evals compare baseline.json candidate.json --out report.md
+
+Input files are JSON: ``{"<input id>": ["run 1 output", "run 2 output", ...]}`` or
+``{"<input id>": {"query": "...", "outputs": [...]}}``. By default outputs are compared with
+n-gram overlap (no model needed, runs fully offline). For semantic comparison pass
+``--judge package.module:function`` where the function takes a prompt string and returns the
+model's reply (it may be ``async``); add ``--rubric`` for the multi-dimension judge.
+"""
+from __future__ import annotations
+
+import argparse
+import asyncio
+import importlib
+import inspect
+import json
+import sys
+from typing import Any, Callable, Dict, List, Optional, Tuple
+
+from .compare import acompare_versions, compare_versions
+from .similarity import EditSimilarity, ExactMatch, LLMJudge, NgramJaccard, RougeL, RubricJudge
+
+
+def load_runs(path: str) -> Tuple[Dict[str, List[Any]], Dict[str, Dict[str, Any]]]:
+    with open(path, encoding="utf-8") as f:
+        data = json.load(f)
+    if not isinstance(data, dict):
+        raise SystemExit(f"{path}: expected a JSON object mapping input id to runs")
+    runs: Dict[str, List[Any]] = {}
+    contexts: Dict[str, Dict[str, Any]] = {}
+    for key, val in data.items():
+        if isinstance(val, dict):
+            runs[key] = list(val.get("outputs", []))
+            if val.get("query") is not None:
+                contexts[key] = {"query": val["query"]}
+        else:
+            runs[key] = list(val)
+    return runs, contexts
+
+
+def _import_callable(spec: str) -> Callable[..., Any]:
+    mod, _, attr = spec.partition(":")
+    if not attr:
+        raise SystemExit("--judge must look like package.module:function")
+    sys.path.insert(0, ".")
+    return getattr(importlib.import_module(mod), attr)
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    ap = argparse.ArgumentParser(prog="repeatability-evals", description=__doc__.split("\n")[0])
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    c = sub.add_parser("compare", help="compare a candidate version against a baseline")
+    c.add_argument("baseline")
+    c.add_argument("candidate")
+    c.add_argument("--similarity", choices=["ngram", "rouge", "edit", "exact"], default="ngram",
+                   help="offline similarity used when no --judge is given (default: ngram)")
+    c.add_argument("--judge", metavar="MODULE:FUNC", help="LLM completion function (prompt -> reply)")
+    c.add_argument("--rubric", action="store_true", help="use the 5-dimension RubricJudge")
+    c.add_argument("--symmetric", action="store_true", help="judge both orders (2x cost)")
+    c.add_argument("--tolerance", type=float, default=0.1, help="per-input drift counted as changed")
+    c.add_argument("--acceptable-drift", type=float, default=0.05, help="mean drift you would ship")
+    c.add_argument("--concurrency", type=int, default=8)
+    c.add_argument("--no-text", action="store_true", help="omit raw outputs from the report")
+    c.add_argument("--names", nargs=2, metavar=("BASE", "CAND"), default=["baseline", "candidate"])
+    c.add_argument("--out", help="write report here (.md, .html or .json); default: print markdown")
+    args = ap.parse_args(argv)
+
+    base, ctx_a = load_runs(args.baseline)
+    cand, ctx_b = load_runs(args.candidate)
+    contexts = {**ctx_b, **ctx_a}
+    judge_fn = _import_callable(args.judge) if args.judge else None
+    if judge_fn is not None:
+        cls = RubricJudge if args.rubric else LLMJudge
+        kw: Dict[str, Any] = {"symmetric": args.symmetric}
+        sim = (cls(allm=judge_fn, **kw) if inspect.iscoroutinefunction(judge_fn)
+               else cls(judge_fn, **kw))
+    else:
+        sim = {"ngram": NgramJaccard(2), "rouge": RougeL(), "edit": EditSimilarity(),
+               "exact": ExactMatch()}[args.similarity]
+    kwargs: Dict[str, Any] = dict(contexts=contexts, tolerance=args.tolerance,
+                                  acceptable_drift=args.acceptable_drift, names=tuple(args.names),
+                                  include_examples=not args.no_text)
+    if judge_fn is not None:
+        report = asyncio.run(acompare_versions(base, cand, sim, concurrency=args.concurrency, **kwargs))
+    else:
+        report = compare_versions(base, cand, sim, **kwargs)
+    if args.out:
+        report.save(args.out, include_text=not args.no_text)
+        print(f"{report.verdict}: drift {report.drift:+.3f}, "
+              f"{report.share_changed:.0%} of {report.n_inputs} inputs changed -> {args.out}")
+    else:
+        print(report.to_markdown(include_text=not args.no_text))
+    return {"no_meaningful_change": 0, "inconclusive": 1, "changed": 2}[report.verdict]
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())