repeatability-evals 0.2.0__tar.gz → 0.3.0__tar.gz

repeatability_evals-0.3.0/CHANGELOG.md

@@ -0,0 +1,50 @@
+# Changelog
+
+## 0.3.0
+
+### Fixed
+- **Dataset means no longer count single-run groups.** Groups with fewer than 2 items were scored
+  1.0 and averaged in, inflating `evaluate_groups` results. They are now excluded and listed in
+  `DatasetResult.degenerate`.
+- **`acompare_versions(concurrency=N)` is a real global cap.** Previously each input and each metric
+  had its own semaphore, so `concurrency=4` could issue ~24 simultaneous judge calls.
+- **Judge prompts are filled in one pass.** A response containing `{item_b}` could pull the other
+  response into its own slot. Responses are now also enclosed in tags with an instruction to
+  treat them as data (`render_prompt` is public).
+- **Score parsing ignores scale text.** `"Score (1-5): 4"` parsed as 1 and `"2 out of 5"` could
+  parse as 5. JSON is now found even after prose containing braces, and rubric scores may be
+  nested under `"scores"` or top level.
+- `EmbeddingCosine` cache was not thread-safe under the async metrics (possible `KeyError`).
+- `TotalAgreement` crashed on unhashable items (dicts, lists); they are compared as canonical JSON.
+- `cycle_pairs` with a truncated budget now always covers `min(N, 2 * budget)` items (odd N missed
+  one).
+- `__version__` is read from package metadata (0.2.0 on PyPI reported 0.1.0).
+
+### Added
+- **Noise-aware per-input flags in `compare_versions`.** An input now counts as changed only if
+  its drift exceeds `tolerance` *and* a permutation test on its pair scores (cross-version pairs
+  less alike than within-version pairs) gives `p_value <= alpha` (default 0.2; `alpha=1.0`
+  restores the old drift-only rule). On simulated noisy-but-unchanged systems (30% of runs
+  differ, 4 runs per side) the share of inputs flagged by chance drops from ~21% to ~5%, and in
+  `examples/compare_versions_demo.py` the flagged set now matches the 36 truly changed inputs
+  exactly (previously 5 false positives). The dataset-level verdict is unchanged in method.
+  `InputComparison.p_value` and CLI `--alpha` expose it.
+- `on_error="skip"` on `PairwiseConsistency`, `evaluate_groups`, `aevaluate_groups`,
+  `compare_versions`, `acompare_versions` and the CLI (`--on-error skip`): failing comparisons are
+  dropped and reported instead of aborting a long run. `AllComparisonsFailed` when nothing is left.
+- Judges: `retries` (default 2) for unparsable replies, `skip_identical` (default on: identical
+  outputs score 1.0 with no model call) and an LRU `cache_size` (default 10,000). Symmetric async
+  judging runs both orders concurrently.
+- `Similarity.prepare` / `aprepare` hook, called with every item of a group; `EmbeddingCosine` uses
+  it to embed a group in one batch, and accepts an async `aembed`.
+- `call_limit(n)` context manager and `aevaluate_groups(max_calls=...)` for global rate limiting.
+- `FunctionSimilarity` passes `context` to functions that accept it.
+- CLI reads JSON Lines (`{"id", "output", "query"}` per line).
+- HTML reports are now structured pages (tables, headings, light/dark) instead of preformatted
+  markdown.
+
+### Changed
+- CLI exit codes: `0` no meaningful change, `3` inconclusive, `4` changed (were 0/1/2, which
+  collided with Python and argparse error codes).
+- Judge prompts wrap the input and responses in `<input>`, `<response_a>`, `<response_b>` tags.
+  The rubric wording is unchanged.

{repeatability_evals-0.2.0 → repeatability_evals-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: repeatability-evals
-Version: 0.2.0
+Version: 0.3.0
 Summary: Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.
 Project-URL: Homepage, https://github.com/krishna-gogineni-765/repeatability-evals
 Project-URL: Issues, https://github.com/krishna-gogineni-765/repeatability-evals/issues
@@ -22,7 +22,7 @@ Description-Content-Type: text/markdown
 **Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.**
 
 Most evals ask *"is the output correct?"* and need gold labels to answer. This library asks
-two questions that are extremely important in enterprise settinga and need **no gold labels**:
+two questions that are extremely important in enterprise settings and need **no gold labels**:
 
 1. **Is my system consistent/repeatable with itself?** Same input, N runs: do the answers, the retrieved
    documents, the tool-call traces agree?
@@ -54,7 +54,7 @@ report.save("report.md")         # also .html / .json
 ```
 # v1 vs v2: consistency report
 CHANGED: the candidate's outputs differ from the baseline's by more than run-to-run noise explains.
-120 inputs compared; 34% changed beyond the noise floor (per-input drift > 0.1).
+120 inputs compared; 30% changed beyond the noise floor.
 
 | v1 vs itself (noise floor)               | 0.925 |
 | v2 vs itself                             | 0.948 |
@@ -66,13 +66,13 @@ CHANGED: the candidate's outputs differ from the baseline's by more than run-to-
 | specifics         | +0.115 |
 | supporting_detail | +0.100 |
 | core_facts        | +0.057 |
-Changed inputs by main driver: 21 mainly in `format`, 20 mainly in `specifics`.
+Changed inputs by main driver: 21 mainly in `format`, 15 mainly in `specifics`.
 
-### `q000`: drift +0.62 (cross 0.38 vs expected 1.00)
+### `q000`: drift +0.62 (cross 0.38 vs expected 1.00, p = 0.001)
 - Main driver: `specifics`
 - Judge: Retention period differs.
 - v1: Retention for gold is 30 days.
-- v2: - Retention for gold is 90 days. - Applies to all sites.
+- v2: - The retention period of gold is 90 days. - Applies to all sites.
 ```
 
 *(Real output of `examples/compare_versions_demo.py`, which uses simulated runs and a scripted
@@ -81,8 +81,12 @@ stand-in for the LLM judge so it runs without an API key.)*
 The key idea is the **noise floor**. A non-deterministic system never matches itself perfectly,
 so comparing v2 to v1 naively overstates change. We measure how much v1 differs from *itself*
 and v2 from *itself*, and report only the **extra** disagreement across versions (`drift`).
-Two equally noisy systems show no drift; a real change does. The report also says *what kind*
-of change it is, since format drift and changed numbers deserve different decisions.
+Two equally noisy systems show no drift; a real change does. Each input is also screened with a
+permutation test (are its cross-version pairs less alike than its within-version pairs?), so a
+noisy system does not get a long list of inputs flagged by chance. In the demo above, 36 inputs
+were changed by construction and exactly those 36 are flagged. The report also says *what kind*
+of change it is, since format drift and changed numbers deserve different decisions. Use 3 or
+more runs per side, 5 if the system is noisy.
 
 From the shell, with no code and no model (offline n-gram similarity), or with your own LLM:
 
@@ -92,8 +96,10 @@ repeatability-evals compare v1.json v2.json --out report.md
 repeatability-evals compare v1.json v2.json --judge mypkg.llm:complete --rubric --out report.html
 ```
 
-Input is JSON: `{"input id": ["run 1", "run 2", ...]}`. The exit code is `0` (no meaningful
-change), `1` (inconclusive) or `2` (changed), so it can gate a CI pipeline.
+Input is JSON, `{"input id": ["run 1", "run 2", ...]}`, or JSON Lines with one run per line,
+`{"id": "...", "output": "...", "query": "..."}`, which is the shape most request logs already
+have. The exit code is `0` (no meaningful change), `3` (inconclusive) or `4` (changed), so it can
+gate a CI pipeline.
 
 ## Usage pattern 2: Measure self-consistency
 
@@ -134,6 +140,23 @@ your own `Dimension`s to change them. `LLMJudge` is the single 1-5 semantic-equi
 `symmetric=True` judges both orders to cancel position bias. **Validate any judge against human
 ratings on your own data before relying on it.**
 
+### Running it at scale
+
+LLM-judged evaluations over thousands of inputs should not fall over on one bad reply, and
+should not pay for comparisons whose answer is obvious:
+
+* **Failures**: judges re-ask when a reply has no parsable score (`retries=2`). Pass
+  `on_error="skip"` to metrics, `evaluate_groups` or `compare_versions` to drop a comparison that
+  still fails (timeout, API error) and keep going; skipped pairs and inputs are reported.
+* **Cost**: identical outputs, which deterministic systems produce constantly, score 1.0 without a
+  model call (`skip_identical=True`), and repeated comparisons are cached. `EmbeddingCosine`
+  embeds each group in one batch.
+* **Rate limits**: `concurrency` on `acompare_versions` (and `max_calls` on `aevaluate_groups`)
+  caps simultaneous judge calls across the whole run.
+* **Prompt safety**: outputs are inserted into the judge prompt in one pass, inside tags, and the
+  judge is told to treat them as data, so placeholder-like text in a response cannot change the
+  prompt's structure. (No prompt fully rules out injection; review judge reasons on outliers.)
+
 ### Sampling cost
 
 All pairs for 10 runs is 45 judge calls; the default sampler uses 10. On synthetic data the

{repeatability_evals-0.2.0 → repeatability_evals-0.3.0}/README.md

@@ -3,7 +3,7 @@
 **Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.**
 
 Most evals ask *"is the output correct?"* and need gold labels to answer. This library asks
-two questions that are extremely important in enterprise settinga and need **no gold labels**:
+two questions that are extremely important in enterprise settings and need **no gold labels**:
 
 1. **Is my system consistent/repeatable with itself?** Same input, N runs: do the answers, the retrieved
    documents, the tool-call traces agree?
@@ -35,7 +35,7 @@ report.save("report.md")         # also .html / .json
 ```
 # v1 vs v2: consistency report
 CHANGED: the candidate's outputs differ from the baseline's by more than run-to-run noise explains.
-120 inputs compared; 34% changed beyond the noise floor (per-input drift > 0.1).
+120 inputs compared; 30% changed beyond the noise floor.
 
 | v1 vs itself (noise floor)               | 0.925 |
 | v2 vs itself                             | 0.948 |
@@ -47,13 +47,13 @@ CHANGED: the candidate's outputs differ from the baseline's by more than run-to-
 | specifics         | +0.115 |
 | supporting_detail | +0.100 |
 | core_facts        | +0.057 |
-Changed inputs by main driver: 21 mainly in `format`, 20 mainly in `specifics`.
+Changed inputs by main driver: 21 mainly in `format`, 15 mainly in `specifics`.
 
-### `q000`: drift +0.62 (cross 0.38 vs expected 1.00)
+### `q000`: drift +0.62 (cross 0.38 vs expected 1.00, p = 0.001)
 - Main driver: `specifics`
 - Judge: Retention period differs.
 - v1: Retention for gold is 30 days.
-- v2: - Retention for gold is 90 days. - Applies to all sites.
+- v2: - The retention period of gold is 90 days. - Applies to all sites.
 ```
 
 *(Real output of `examples/compare_versions_demo.py`, which uses simulated runs and a scripted
@@ -62,8 +62,12 @@ stand-in for the LLM judge so it runs without an API key.)*
 The key idea is the **noise floor**. A non-deterministic system never matches itself perfectly,
 so comparing v2 to v1 naively overstates change. We measure how much v1 differs from *itself*
 and v2 from *itself*, and report only the **extra** disagreement across versions (`drift`).
-Two equally noisy systems show no drift; a real change does. The report also says *what kind*
-of change it is, since format drift and changed numbers deserve different decisions.
+Two equally noisy systems show no drift; a real change does. Each input is also screened with a
+permutation test (are its cross-version pairs less alike than its within-version pairs?), so a
+noisy system does not get a long list of inputs flagged by chance. In the demo above, 36 inputs
+were changed by construction and exactly those 36 are flagged. The report also says *what kind*
+of change it is, since format drift and changed numbers deserve different decisions. Use 3 or
+more runs per side, 5 if the system is noisy.
 
 From the shell, with no code and no model (offline n-gram similarity), or with your own LLM:
 
@@ -73,8 +77,10 @@ repeatability-evals compare v1.json v2.json --out report.md
 repeatability-evals compare v1.json v2.json --judge mypkg.llm:complete --rubric --out report.html
 ```
 
-Input is JSON: `{"input id": ["run 1", "run 2", ...]}`. The exit code is `0` (no meaningful
-change), `1` (inconclusive) or `2` (changed), so it can gate a CI pipeline.
+Input is JSON, `{"input id": ["run 1", "run 2", ...]}`, or JSON Lines with one run per line,
+`{"id": "...", "output": "...", "query": "..."}`, which is the shape most request logs already
+have. The exit code is `0` (no meaningful change), `3` (inconclusive) or `4` (changed), so it can
+gate a CI pipeline.
 
 ## Usage pattern 2: Measure self-consistency
 
@@ -115,6 +121,23 @@ your own `Dimension`s to change them. `LLMJudge` is the single 1-5 semantic-equi
 `symmetric=True` judges both orders to cancel position bias. **Validate any judge against human
 ratings on your own data before relying on it.**
 
+### Running it at scale
+
+LLM-judged evaluations over thousands of inputs should not fall over on one bad reply, and
+should not pay for comparisons whose answer is obvious:
+
+* **Failures**: judges re-ask when a reply has no parsable score (`retries=2`). Pass
+  `on_error="skip"` to metrics, `evaluate_groups` or `compare_versions` to drop a comparison that
+  still fails (timeout, API error) and keep going; skipped pairs and inputs are reported.
+* **Cost**: identical outputs, which deterministic systems produce constantly, score 1.0 without a
+  model call (`skip_identical=True`), and repeated comparisons are cached. `EmbeddingCosine`
+  embeds each group in one batch.
+* **Rate limits**: `concurrency` on `acompare_versions` (and `max_calls` on `aevaluate_groups`)
+  caps simultaneous judge calls across the whole run.
+* **Prompt safety**: outputs are inserted into the judge prompt in one pass, inside tags, and the
+  judge is told to treat them as data, so placeholder-like text in a response cannot change the
+  prompt's structure. (No prompt fully rules out injection; review judge reasons on outliers.)
+
 ### Sampling cost
 
 All pairs for 10 runs is 45 judge calls; the default sampler uses 10. On synthetic data the

{repeatability_evals-0.2.0 → repeatability_evals-0.3.0}/examples/compare_versions_demo.py

@@ -38,8 +38,8 @@ v2 = version(lambda q: 90 if q % 8 == 0 else 30, lambda q: q % 5 == 0)
 
 def judge(prompt):
     """Scripted stand-in for an LLM: scores each dimension by simple rules."""
-    a = prompt.split("Response A:")[1].split("Response B:")[0]
-    b = prompt.split("Response B:")[1].split("Score each dimension")[0]
+    a = prompt.split("<response_a>")[1].split("</response_a>")[0]
+    b = prompt.split("<response_b>")[1].split("</response_b>")[0]
     num = lambda t: re.findall(r"\d+", t)  # noqa: E731
     bullets = lambda t: t.lstrip().startswith("-")  # noqa: E731
     scores = {d.name: 5 for d in DEFAULT_DIMENSIONS}

{repeatability_evals-0.2.0 → repeatability_evals-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "repeatability-evals"
-version = "0.2.0"
+version = "0.3.0"
 description = "Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows."
 readme = "README.md"
 license = { text = "MIT" }

{repeatability_evals-0.2.0 → repeatability_evals-0.3.0}/src/repeatability_evals/__init__.py

@@ -1,7 +1,14 @@
 """repeatability-evals: consistency evals for non-deterministic LLM systems."""
 from .compare import InputComparison, VersionComparison, acompare_versions, compare_versions
 from .dataset import DatasetResult, aevaluate_groups, evaluate_groups
-from .metrics import ClusterConsistency, ConsistencyMetric, PairwiseConsistency, TotalAgreement
+from .metrics import (
+    AllComparisonsFailed,
+    ClusterConsistency,
+    ConsistencyMetric,
+    PairwiseConsistency,
+    TotalAgreement,
+    call_limit,
+)
 from .prompts import DEFAULT_DIMENSIONS, SEMANTIC_EQUIVALENCE_PROMPT, Dimension, build_rubric_prompt
 from .sampling import AllPairs, AnchorPairs, CrossSampler, CycleSampler, PairSampler, RandomPairs, cycle_pairs
 from .similarity import (
@@ -20,13 +27,21 @@ from .similarity import (
     RubricJudge,
     Similarity,
     as_similarity,
+    render_prompt,
 )
 from .stats import bootstrap_ci
 from .types import ConsistencyResult, PairScore, SimilarityResult
 
-__version__ = "0.1.0"
+try:
+    from importlib.metadata import PackageNotFoundError
+    from importlib.metadata import version as _version
+
+    __version__ = _version("repeatability-evals")
+except PackageNotFoundError:  # running from a source checkout that is not installed
+    __version__ = "0+unknown"
 
 __all__ = [
+    "call_limit", "AllComparisonsFailed", "render_prompt",
     "compare_versions", "acompare_versions", "VersionComparison", "InputComparison", "CrossSampler",
     "ConsistencyMetric", "PairwiseConsistency", "ClusterConsistency", "TotalAgreement",
     "evaluate_groups", "aevaluate_groups", "DatasetResult", "bootstrap_ci",

{repeatability_evals-0.2.0 → repeatability_evals-0.3.0}/src/repeatability_evals/cli.py

@@ -3,10 +3,15 @@
     repeatability-evals compare baseline.json candidate.json --out report.md
 
 Input files are JSON: ``{"<input id>": ["run 1 output", "run 2 output", ...]}`` or
-``{"<input id>": {"query": "...", "outputs": [...]}}``. By default outputs are compared with
+``{"<input id>": {"query": "...", "outputs": [...]}}``, or JSON Lines (``.jsonl``) with one run
+per line: ``{"id": "...", "output": "...", "query": "..."}``, the shape most request logs already
+have. By default outputs are compared with
 n-gram overlap (no model needed, runs fully offline). For semantic comparison pass
 ``--judge package.module:function`` where the function takes a prompt string and returns the
 model's reply (it may be ``async``); add ``--rubric`` for the multi-dimension judge.
+
+Exit status: 0 no meaningful change, 3 inconclusive, 4 changed (1 and 2 are left to Python and
+argparse for errors), so the command can gate a CI pipeline.
 """
 from __future__ import annotations
 
@@ -21,8 +26,34 @@ from typing import Any, Callable, Dict, List, Optional, Tuple
 from .compare import acompare_versions, compare_versions
 from .similarity import EditSimilarity, ExactMatch, LLMJudge, NgramJaccard, RougeL, RubricJudge
 
+EXIT_CODES = {"no_meaningful_change": 0, "inconclusive": 3, "changed": 4}
+
+
+def _load_jsonl(path: str) -> Tuple[Dict[str, List[Any]], Dict[str, Dict[str, Any]]]:
+    runs: Dict[str, List[Any]] = {}
+    contexts: Dict[str, Dict[str, Any]] = {}
+    with open(path, encoding="utf-8") as f:
+        for n, line in enumerate(f, 1):
+            if not line.strip():
+                continue
+            try:
+                row = json.loads(line)
+            except ValueError as e:
+                raise SystemExit(f"{path}:{n}: invalid JSON ({e})") from e
+            if not isinstance(row, dict) or "id" not in row or not ("output" in row or "outputs" in row):
+                raise SystemExit(f'{path}:{n}: each line needs "id" and "output" (or "outputs")')
+            key = str(row["id"])
+            outs = row["outputs"] if "outputs" in row else [row["output"]]
+            runs.setdefault(key, []).extend(outs)
+            if row.get("query") is not None:
+                contexts.setdefault(key, {"query": row["query"]})
+    return runs, contexts
+
 
 def load_runs(path: str) -> Tuple[Dict[str, List[Any]], Dict[str, Dict[str, Any]]]:
+    """Read runs from ``.json`` (object keyed by input id) or ``.jsonl`` (one run per line)."""
+    if path.endswith(".jsonl"):
+        return _load_jsonl(path)
     with open(path, encoding="utf-8") as f:
         data = json.load(f)
     if not isinstance(data, dict):
@@ -34,8 +65,10 @@ def load_runs(path: str) -> Tuple[Dict[str, List[Any]], Dict[str, Dict[str, Any]
             runs[key] = list(val.get("outputs", []))
             if val.get("query") is not None:
                 contexts[key] = {"query": val["query"]}
-        else:
+        elif isinstance(val, list):
             runs[key] = list(val)
+        else:
+            raise SystemExit(f"{path}: input {key!r} must map to a list of runs or an object")
     return runs, contexts
 
 
@@ -59,9 +92,13 @@ def main(argv: Optional[List[str]] = None) -> int:
     c.add_argument("--rubric", action="store_true", help="use the 5-dimension RubricJudge")
     c.add_argument("--symmetric", action="store_true", help="judge both orders (2x cost)")
     c.add_argument("--tolerance", type=float, default=0.1, help="per-input drift counted as changed")
+    c.add_argument("--alpha", type=float, default=0.2,
+                   help="per-input significance level for 'changed' (1.0 = drift threshold only)")
     c.add_argument("--acceptable-drift", type=float, default=0.05, help="mean drift you would ship")
     c.add_argument("--concurrency", type=int, default=8)
     c.add_argument("--no-text", action="store_true", help="omit raw outputs from the report")
+    c.add_argument("--on-error", choices=["raise", "skip"], default="raise",
+                   help="skip comparisons that fail (bad judge reply, timeout) instead of stopping")
     c.add_argument("--names", nargs=2, metavar=("BASE", "CAND"), default=["baseline", "candidate"])
     c.add_argument("--out", help="write report here (.md, .html or .json); default: print markdown")
     args = ap.parse_args(argv)
@@ -78,9 +115,9 @@ def main(argv: Optional[List[str]] = None) -> int:
     else:
         sim = {"ngram": NgramJaccard(2), "rouge": RougeL(), "edit": EditSimilarity(),
                "exact": ExactMatch()}[args.similarity]
-    kwargs: Dict[str, Any] = dict(contexts=contexts, tolerance=args.tolerance,
+    kwargs: Dict[str, Any] = dict(contexts=contexts, tolerance=args.tolerance, alpha=args.alpha,
                                   acceptable_drift=args.acceptable_drift, names=tuple(args.names),
-                                  include_examples=not args.no_text)
+                                  include_examples=not args.no_text, on_error=args.on_error)
     if judge_fn is not None:
         report = asyncio.run(acompare_versions(base, cand, sim, concurrency=args.concurrency, **kwargs))
     else:
@@ -91,7 +128,9 @@ def main(argv: Optional[List[str]] = None) -> int:
               f"{report.share_changed:.0%} of {report.n_inputs} inputs changed -> {args.out}")
     else:
         print(report.to_markdown(include_text=not args.no_text))
-    return {"no_meaningful_change": 0, "inconclusive": 1, "changed": 2}[report.verdict]
+    if report.failed_inputs:
+        print(f"warning: {len(report.failed_inputs)} inputs failed and were skipped", file=sys.stderr)
+    return EXIT_CODES[report.verdict]
 
 
 if __name__ == "__main__":