repeatability-evals 0.1.0__tar.gz → 0.3.0__tar.gz

{repeatability_evals-0.1.0 → repeatability_evals-0.3.0}/.gitignore

@@ -4,3 +4,4 @@ __pycache__/
 .pytest_cache/
 .ruff_cache/
 dist/
+examples/demo_report.md

repeatability_evals-0.3.0/CHANGELOG.md

@@ -0,0 +1,50 @@
+# Changelog
+
+## 0.3.0
+
+### Fixed
+- **Dataset means no longer count single-run groups.** Groups with fewer than 2 items were scored
+  1.0 and averaged in, inflating `evaluate_groups` results. They are now excluded and listed in
+  `DatasetResult.degenerate`.
+- **`acompare_versions(concurrency=N)` is a real global cap.** Previously each input and each metric
+  had its own semaphore, so `concurrency=4` could issue ~24 simultaneous judge calls.
+- **Judge prompts are filled in one pass.** A response containing `{item_b}` could pull the other
+  response into its own slot. Responses are now also enclosed in tags with an instruction to
+  treat them as data (`render_prompt` is public).
+- **Score parsing ignores scale text.** `"Score (1-5): 4"` parsed as 1 and `"2 out of 5"` could
+  parse as 5. JSON is now found even after prose containing braces, and rubric scores may be
+  nested under `"scores"` or top level.
+- `EmbeddingCosine` cache was not thread-safe under the async metrics (possible `KeyError`).
+- `TotalAgreement` crashed on unhashable items (dicts, lists); they are compared as canonical JSON.
+- `cycle_pairs` with a truncated budget now always covers `min(N, 2 * budget)` items (odd N missed
+  one).
+- `__version__` is read from package metadata (0.2.0 on PyPI reported 0.1.0).
+
+### Added
+- **Noise-aware per-input flags in `compare_versions`.** An input now counts as changed only if
+  its drift exceeds `tolerance` *and* a permutation test on its pair scores (cross-version pairs
+  less alike than within-version pairs) gives `p_value <= alpha` (default 0.2; `alpha=1.0`
+  restores the old drift-only rule). On simulated noisy-but-unchanged systems (30% of runs
+  differ, 4 runs per side) the share of inputs flagged by chance drops from ~21% to ~5%, and in
+  `examples/compare_versions_demo.py` the flagged set now matches the 36 truly changed inputs
+  exactly (previously 5 false positives). The dataset-level verdict is unchanged in method.
+  `InputComparison.p_value` and CLI `--alpha` expose it.
+- `on_error="skip"` on `PairwiseConsistency`, `evaluate_groups`, `aevaluate_groups`,
+  `compare_versions`, `acompare_versions` and the CLI (`--on-error skip`): failing comparisons are
+  dropped and reported instead of aborting a long run. `AllComparisonsFailed` when nothing is left.
+- Judges: `retries` (default 2) for unparsable replies, `skip_identical` (default on: identical
+  outputs score 1.0 with no model call) and an LRU `cache_size` (default 10,000). Symmetric async
+  judging runs both orders concurrently.
+- `Similarity.prepare` / `aprepare` hook, called with every item of a group; `EmbeddingCosine` uses
+  it to embed a group in one batch, and accepts an async `aembed`.
+- `call_limit(n)` context manager and `aevaluate_groups(max_calls=...)` for global rate limiting.
+- `FunctionSimilarity` passes `context` to functions that accept it.
+- CLI reads JSON Lines (`{"id", "output", "query"}` per line).
+- HTML reports are now structured pages (tables, headings, light/dark) instead of preformatted
+  markdown.
+
+### Changed
+- CLI exit codes: `0` no meaningful change, `3` inconclusive, `4` changed (were 0/1/2, which
+  collided with Python and argparse error codes).
+- Judge prompts wrap the input and responses in `<input>`, `<response_a>`, `<response_b>` tags.
+  The rubric wording is unchanged.

repeatability_evals-0.3.0/PKG-INFO

@@ -0,0 +1,178 @@
+Metadata-Version: 2.5
+Name: repeatability-evals
+Version: 0.3.0
+Summary: Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.
+Project-URL: Homepage, https://github.com/krishna-gogineni-765/repeatability-evals
+Project-URL: Issues, https://github.com/krishna-gogineni-765/repeatability-evals/issues
+Author: Krishna Gogineni
+License: MIT
+License-File: LICENSE
+Keywords: agents,consistency,evaluation,llm,llm-as-judge,non-determinism,rag,reliability,repeatability,retrieval
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# repeatability-evals
+
+**Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.**
+
+Most evals ask *"is the output correct?"* and need gold labels to answer. This library asks
+two questions that are extremely important in enterprise settings and need **no gold labels**:
+
+1. **Is my system consistent/repeatable with itself?** Same input, N runs: do the answers, the retrieved
+   documents, the tool-call traces agree?
+2. **Did anything user facing change when I changed something in the backend?** Model upgrade, prompt edit, new index,
+   framework bump: which inputs behave differently, *beyond the system's own run-to-run noise*,
+   and *how* (facts? numbers? format? refusals?).
+
+## Why it matters
+
+| If you are... | You can... |
+|---|---|
+| **a platform or model provider** shipping an update | run it offline on a regression set and see which customer-style workflows would shift before you release |
+| **an enterprise on a vendor's LLM or agent platform** | run it yourself, on your own data and cluster, to decide whether an upstream upgrade is safe for the workflows you built. Nothing leaves your environment, and `include_text=False` keeps raw text out of reports |
+| **building an agent, RAG or harness product** | track whether retrieval and answers stay stable across runs, and catch silent drift from non-deterministic infrastructure |
+| **without labelled data** | still get a signal: consistency needs only outputs, not answers |
+
+
+## Usage pattern 1: See what changed
+
+```python
+from repeatability_evals import compare_versions, RubricJudge
+
+# {input_id: [output of run 1, run 2, run 3, ...]} for each version. No labels.
+report = compare_versions(v1_runs, v2_runs, RubricJudge(my_llm), names=("v1", "v2"))
+print(report.verdict)            # no_meaningful_change | changed | inconclusive
+report.save("report.md")         # also .html / .json
+```
+
+```
+# v1 vs v2: consistency report
+CHANGED: the candidate's outputs differ from the baseline's by more than run-to-run noise explains.
+120 inputs compared; 30% changed beyond the noise floor.
+
+| v1 vs itself (noise floor)               | 0.925 |
+| v2 vs itself                             | 0.948 |
+| v1 vs v2                                 | 0.850 |
+| drift (expected - cross; 0 = no change)  | +0.087 (95% CI +0.060 to +0.116) |
+
+## What changed
+| format            | +0.200 |
+| specifics         | +0.115 |
+| supporting_detail | +0.100 |
+| core_facts        | +0.057 |
+Changed inputs by main driver: 21 mainly in `format`, 15 mainly in `specifics`.
+
+### `q000`: drift +0.62 (cross 0.38 vs expected 1.00, p = 0.001)
+- Main driver: `specifics`
+- Judge: Retention period differs.
+- v1: Retention for gold is 30 days.
+- v2: - The retention period of gold is 90 days. - Applies to all sites.
+```
+
+*(Real output of `examples/compare_versions_demo.py`, which uses simulated runs and a scripted
+stand-in for the LLM judge so it runs without an API key.)*
+
+The key idea is the **noise floor**. A non-deterministic system never matches itself perfectly,
+so comparing v2 to v1 naively overstates change. We measure how much v1 differs from *itself*
+and v2 from *itself*, and report only the **extra** disagreement across versions (`drift`).
+Two equally noisy systems show no drift; a real change does. Each input is also screened with a
+permutation test (are its cross-version pairs less alike than its within-version pairs?), so a
+noisy system does not get a long list of inputs flagged by chance. In the demo above, 36 inputs
+were changed by construction and exactly those 36 are flagged. The report also says *what kind*
+of change it is, since format drift and changed numbers deserve different decisions. Use 3 or
+more runs per side, 5 if the system is noisy.
+
+From the shell, with no code and no model (offline n-gram similarity), or with your own LLM:
+
+```bash
+pip install repeatability-evals
+repeatability-evals compare v1.json v2.json --out report.md
+repeatability-evals compare v1.json v2.json --judge mypkg.llm:complete --rubric --out report.html
+```
+
+Input is JSON, `{"input id": ["run 1", "run 2", ...]}`, or JSON Lines with one run per line,
+`{"id": "...", "output": "...", "query": "..."}`, which is the shape most request logs already
+have. The exit code is `0` (no meaningful change), `3` (inconclusive) or `4` (changed), so it can
+gate a CI pipeline.
+
+## Usage pattern 2: Measure self-consistency
+
+```python
+from repeatability_evals import PairwiseConsistency, JaccardSimilarity, NgramJaccard, RubricJudge
+
+PairwiseConsistency(JaccardSimilarity()).evaluate(chunk_ids_per_run)                 # retrieved IDs
+PairwiseConsistency(NgramJaccard(3)).evaluate(chunk_texts_per_run)                   # robust to re-chunking
+PairwiseConsistency(RubricJudge(my_llm)).evaluate(answers, context={"query": q})     # answers
+```
+
+Over a dataset, `evaluate_groups({input_id: runs, ...}, metric)` returns the mean, a bootstrap
+confidence interval and the least consistent inputs.
+
+## Design
+
+The library never knows whether you are comparing answers, chunk IDs or agent traces. It
+compares *items*; you pick the similarity metric that makes sense for what you are comparing.
+
+* **`Similarity`** scores two items in `[0, 1]`: sets (`JaccardSimilarity`, `DiceSimilarity`,
+  `OverlapCoefficient`), text (`ExactMatch`, `NgramJaccard`, `RougeL`, `EditSimilarity`),
+  rankings (`RankBiasedOverlap`), embeddings (`EmbeddingCosine`, bring your own embed
+  function) and LLM judges (`RubricJudge`, `LLMJudge`). Subclass it, or pass a plain function.
+* **`PairSampler`** picks which of the N(N-1)/2 pairs to compare. The default `CycleSampler`
+  costs N comparisons and touches every run twice; `CrossSampler` does the same across versions.
+* **Metrics**: `PairwiseConsistency` (mean similarity), `ClusterConsistency` (largest
+  "same answer" cluster, plus entropy), `TotalAgreement` (all N identical) and `compare_versions`.
+
+Everything has an async twin (`aevaluate`, `acompare_versions`) with bounded concurrency. The
+core has **no dependencies**; you bring the model, as any `str -> str` function.
+
+### The judge
+
+`RubricJudge` scores five dimensions of agreement in one call: `core_facts`, `specifics`,
+`supporting_detail`, `stance` (answer vs hedge vs refuse) and `format`. The headline is a
+weighted mean (core facts count most) and the per-dimension means explain *what* varies. Pass
+your own `Dimension`s to change them. `LLMJudge` is the single 1-5 semantic-equivalence score.
+`symmetric=True` judges both orders to cancel position bias. **Validate any judge against human
+ratings on your own data before relying on it.**
+
+### Running it at scale
+
+LLM-judged evaluations over thousands of inputs should not fall over on one bad reply, and
+should not pay for comparisons whose answer is obvious:
+
+* **Failures**: judges re-ask when a reply has no parsable score (`retries=2`). Pass
+  `on_error="skip"` to metrics, `evaluate_groups` or `compare_versions` to drop a comparison that
+  still fails (timeout, API error) and keep going; skipped pairs and inputs are reported.
+* **Cost**: identical outputs, which deterministic systems produce constantly, score 1.0 without a
+  model call (`skip_identical=True`), and repeated comparisons are cached. `EmbeddingCosine`
+  embeds each group in one batch.
+* **Rate limits**: `concurrency` on `acompare_versions` (and `max_calls` on `aevaluate_groups`)
+  caps simultaneous judge calls across the whole run.
+* **Prompt safety**: outputs are inserted into the judge prompt in one pass, inside tags, and the
+  judge is told to treat them as data, so placeholder-like text in a response cannot change the
+  prompt's structure. (No prompt fully rules out injection; review judge reasons on outliers.)
+
+### Sampling cost
+
+All pairs for 10 runs is 45 judge calls; the default sampler uses 10. On synthetic data the
+per-input error versus all pairs was about 0.03 with no bias (`benchmarks/sampling_error.py`);
+check it on your data, or use `AllPairs()` for the exact value.
+
+### Writing your own similarity
+
+```python
+from repeatability_evals import Similarity, PairwiseConsistency
+
+class SameFirstSentence(Similarity):
+    name = "same_first_sentence"
+    def compute(self, a, b, *, context=None):
+        return float(a.split(".")[0] == b.split(".")[0])
+
+PairwiseConsistency(SameFirstSentence())      # or PairwiseConsistency(lambda a, b: ...)
+```
+

repeatability_evals-0.3.0/README.md

@@ -0,0 +1,159 @@
+# repeatability-evals
+
+**Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows.**
+
+Most evals ask *"is the output correct?"* and need gold labels to answer. This library asks
+two questions that are extremely important in enterprise settings and need **no gold labels**:
+
+1. **Is my system consistent/repeatable with itself?** Same input, N runs: do the answers, the retrieved
+   documents, the tool-call traces agree?
+2. **Did anything user facing change when I changed something in the backend?** Model upgrade, prompt edit, new index,
+   framework bump: which inputs behave differently, *beyond the system's own run-to-run noise*,
+   and *how* (facts? numbers? format? refusals?).
+
+## Why it matters
+
+| If you are... | You can... |
+|---|---|
+| **a platform or model provider** shipping an update | run it offline on a regression set and see which customer-style workflows would shift before you release |
+| **an enterprise on a vendor's LLM or agent platform** | run it yourself, on your own data and cluster, to decide whether an upstream upgrade is safe for the workflows you built. Nothing leaves your environment, and `include_text=False` keeps raw text out of reports |
+| **building an agent, RAG or harness product** | track whether retrieval and answers stay stable across runs, and catch silent drift from non-deterministic infrastructure |
+| **without labelled data** | still get a signal: consistency needs only outputs, not answers |
+
+
+## Usage pattern 1: See what changed
+
+```python
+from repeatability_evals import compare_versions, RubricJudge
+
+# {input_id: [output of run 1, run 2, run 3, ...]} for each version. No labels.
+report = compare_versions(v1_runs, v2_runs, RubricJudge(my_llm), names=("v1", "v2"))
+print(report.verdict)            # no_meaningful_change | changed | inconclusive
+report.save("report.md")         # also .html / .json
+```
+
+```
+# v1 vs v2: consistency report
+CHANGED: the candidate's outputs differ from the baseline's by more than run-to-run noise explains.
+120 inputs compared; 30% changed beyond the noise floor.
+
+| v1 vs itself (noise floor)               | 0.925 |
+| v2 vs itself                             | 0.948 |
+| v1 vs v2                                 | 0.850 |
+| drift (expected - cross; 0 = no change)  | +0.087 (95% CI +0.060 to +0.116) |
+
+## What changed
+| format            | +0.200 |
+| specifics         | +0.115 |
+| supporting_detail | +0.100 |
+| core_facts        | +0.057 |
+Changed inputs by main driver: 21 mainly in `format`, 15 mainly in `specifics`.
+
+### `q000`: drift +0.62 (cross 0.38 vs expected 1.00, p = 0.001)
+- Main driver: `specifics`
+- Judge: Retention period differs.
+- v1: Retention for gold is 30 days.
+- v2: - The retention period of gold is 90 days. - Applies to all sites.
+```
+
+*(Real output of `examples/compare_versions_demo.py`, which uses simulated runs and a scripted
+stand-in for the LLM judge so it runs without an API key.)*
+
+The key idea is the **noise floor**. A non-deterministic system never matches itself perfectly,
+so comparing v2 to v1 naively overstates change. We measure how much v1 differs from *itself*
+and v2 from *itself*, and report only the **extra** disagreement across versions (`drift`).
+Two equally noisy systems show no drift; a real change does. Each input is also screened with a
+permutation test (are its cross-version pairs less alike than its within-version pairs?), so a
+noisy system does not get a long list of inputs flagged by chance. In the demo above, 36 inputs
+were changed by construction and exactly those 36 are flagged. The report also says *what kind*
+of change it is, since format drift and changed numbers deserve different decisions. Use 3 or
+more runs per side, 5 if the system is noisy.
+
+From the shell, with no code and no model (offline n-gram similarity), or with your own LLM:
+
+```bash
+pip install repeatability-evals
+repeatability-evals compare v1.json v2.json --out report.md
+repeatability-evals compare v1.json v2.json --judge mypkg.llm:complete --rubric --out report.html
+```
+
+Input is JSON, `{"input id": ["run 1", "run 2", ...]}`, or JSON Lines with one run per line,
+`{"id": "...", "output": "...", "query": "..."}`, which is the shape most request logs already
+have. The exit code is `0` (no meaningful change), `3` (inconclusive) or `4` (changed), so it can
+gate a CI pipeline.
+
+## Usage pattern 2: Measure self-consistency
+
+```python
+from repeatability_evals import PairwiseConsistency, JaccardSimilarity, NgramJaccard, RubricJudge
+
+PairwiseConsistency(JaccardSimilarity()).evaluate(chunk_ids_per_run)                 # retrieved IDs
+PairwiseConsistency(NgramJaccard(3)).evaluate(chunk_texts_per_run)                   # robust to re-chunking
+PairwiseConsistency(RubricJudge(my_llm)).evaluate(answers, context={"query": q})     # answers
+```
+
+Over a dataset, `evaluate_groups({input_id: runs, ...}, metric)` returns the mean, a bootstrap
+confidence interval and the least consistent inputs.
+
+## Design
+
+The library never knows whether you are comparing answers, chunk IDs or agent traces. It
+compares *items*; you pick the similarity metric that makes sense for what you are comparing.
+
+* **`Similarity`** scores two items in `[0, 1]`: sets (`JaccardSimilarity`, `DiceSimilarity`,
+  `OverlapCoefficient`), text (`ExactMatch`, `NgramJaccard`, `RougeL`, `EditSimilarity`),
+  rankings (`RankBiasedOverlap`), embeddings (`EmbeddingCosine`, bring your own embed
+  function) and LLM judges (`RubricJudge`, `LLMJudge`). Subclass it, or pass a plain function.
+* **`PairSampler`** picks which of the N(N-1)/2 pairs to compare. The default `CycleSampler`
+  costs N comparisons and touches every run twice; `CrossSampler` does the same across versions.
+* **Metrics**: `PairwiseConsistency` (mean similarity), `ClusterConsistency` (largest
+  "same answer" cluster, plus entropy), `TotalAgreement` (all N identical) and `compare_versions`.
+
+Everything has an async twin (`aevaluate`, `acompare_versions`) with bounded concurrency. The
+core has **no dependencies**; you bring the model, as any `str -> str` function.
+
+### The judge
+
+`RubricJudge` scores five dimensions of agreement in one call: `core_facts`, `specifics`,
+`supporting_detail`, `stance` (answer vs hedge vs refuse) and `format`. The headline is a
+weighted mean (core facts count most) and the per-dimension means explain *what* varies. Pass
+your own `Dimension`s to change them. `LLMJudge` is the single 1-5 semantic-equivalence score.
+`symmetric=True` judges both orders to cancel position bias. **Validate any judge against human
+ratings on your own data before relying on it.**
+
+### Running it at scale
+
+LLM-judged evaluations over thousands of inputs should not fall over on one bad reply, and
+should not pay for comparisons whose answer is obvious:
+
+* **Failures**: judges re-ask when a reply has no parsable score (`retries=2`). Pass
+  `on_error="skip"` to metrics, `evaluate_groups` or `compare_versions` to drop a comparison that
+  still fails (timeout, API error) and keep going; skipped pairs and inputs are reported.
+* **Cost**: identical outputs, which deterministic systems produce constantly, score 1.0 without a
+  model call (`skip_identical=True`), and repeated comparisons are cached. `EmbeddingCosine`
+  embeds each group in one batch.
+* **Rate limits**: `concurrency` on `acompare_versions` (and `max_calls` on `aevaluate_groups`)
+  caps simultaneous judge calls across the whole run.
+* **Prompt safety**: outputs are inserted into the judge prompt in one pass, inside tags, and the
+  judge is told to treat them as data, so placeholder-like text in a response cannot change the
+  prompt's structure. (No prompt fully rules out injection; review judge reasons on outliers.)
+
+### Sampling cost
+
+All pairs for 10 runs is 45 judge calls; the default sampler uses 10. On synthetic data the
+per-input error versus all pairs was about 0.03 with no bias (`benchmarks/sampling_error.py`);
+check it on your data, or use `AllPairs()` for the exact value.
+
+### Writing your own similarity
+
+```python
+from repeatability_evals import Similarity, PairwiseConsistency
+
+class SameFirstSentence(Similarity):
+    name = "same_first_sentence"
+    def compute(self, a, b, *, context=None):
+        return float(a.split(".")[0] == b.split(".")[0])
+
+PairwiseConsistency(SameFirstSentence())      # or PairwiseConsistency(lambda a, b: ...)
+```
+

repeatability_evals-0.3.0/examples/compare_versions_demo.py

@@ -0,0 +1,57 @@
+"""Simulated demo: compare two versions of a Q&A system with no gold labels.
+
+No API key needed. The "system" and the "judge" are scripted stand-ins so the demo is
+deterministic; in real use, `judge` is a call to your LLM and the runs come from your system.
+Run:  python examples/compare_versions_demo.py
+"""
+import json
+import random
+import re
+
+from repeatability_evals import DEFAULT_DIMENSIONS, RubricJudge, compare_versions
+
+rng = random.Random(7)
+POLICIES = ["gold", "silver", "bronze", "archive", "dr", "edge"]
+
+
+def answer(policy, days, bullets, noisy):
+    """One run of a pretend system. `noisy` runs sometimes reword (same meaning)."""
+    lead = rng.choice(["Retention for", "The retention period of"]) if noisy else "Retention for"
+    if noisy and rng.random() < 0.08:  # occasional flaky run: a wrong number
+        days += 1
+    body = f"{lead} {policy} is {days} days."
+    return f"- {body}\n- Applies to all sites." if bullets else body
+
+
+def version(days_for, bullets_for):
+    runs = {}
+    for q in range(120):
+        p = POLICIES[q % len(POLICIES)]
+        runs[f"q{q:03d}"] = [answer(p, days_for(q), bullets_for(q), True) for _ in range(4)]
+    return runs
+
+
+v1 = version(lambda q: 30, lambda q: False)
+# v2 changes the retention number on ~12% of questions and switches ~20% to bulleted format.
+v2 = version(lambda q: 90 if q % 8 == 0 else 30, lambda q: q % 5 == 0)
+
+
+def judge(prompt):
+    """Scripted stand-in for an LLM: scores each dimension by simple rules."""
+    a = prompt.split("<response_a>")[1].split("</response_a>")[0]
+    b = prompt.split("<response_b>")[1].split("</response_b>")[0]
+    num = lambda t: re.findall(r"\d+", t)  # noqa: E731
+    bullets = lambda t: t.lstrip().startswith("-")  # noqa: E731
+    scores = {d.name: 5 for d in DEFAULT_DIMENSIONS}
+    if num(a)[:1] != num(b)[:1]:
+        scores["specifics"], scores["core_facts"] = 1, 3
+    if bullets(a) != bullets(b):
+        scores["format"], scores["supporting_detail"] = 1, 3
+    why = "Retention period differs." if scores["specifics"] == 1 else (
+        "Same facts; formatting differs." if scores["format"] == 1 else "Equivalent.")
+    return json.dumps({"reason": why, "scores": scores})
+
+
+report = compare_versions(v1, v2, RubricJudge(judge), names=("v1", "v2"))
+print(report.to_markdown(top=2))
+report.save("examples/demo_report.md")

{repeatability_evals-0.1.0 → repeatability_evals-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "repeatability-evals"
-version = "0.1.0"
+version = "0.3.0"
 description = "Consistency evals for non-deterministic systems: LLM generation, agentic retrieval, agent workflows."
 readme = "README.md"
 license = { text = "MIT" }
@@ -35,6 +35,9 @@ src = ["src", "tests"]
 select = ["E", "F", "I", "B", "UP"]
 ignore = ["UP006", "UP007", "UP035", "UP045"]  # keep typing.Optional/List for py3.9 readability
 
+[project.scripts]
+repeatability-evals = "repeatability_evals.cli:main"
+
 [project.urls]
 Homepage = "https://github.com/krishna-gogineni-765/repeatability-evals"
 Issues = "https://github.com/krishna-gogineni-765/repeatability-evals/issues"

{repeatability_evals-0.1.0 → repeatability_evals-0.3.0}/src/repeatability_evals/__init__.py

@@ -1,8 +1,16 @@
 """repeatability-evals: consistency evals for non-deterministic LLM systems."""
+from .compare import InputComparison, VersionComparison, acompare_versions, compare_versions
 from .dataset import DatasetResult, aevaluate_groups, evaluate_groups
-from .metrics import ClusterConsistency, ConsistencyMetric, PairwiseConsistency, TotalAgreement
+from .metrics import (
+    AllComparisonsFailed,
+    ClusterConsistency,
+    ConsistencyMetric,
+    PairwiseConsistency,
+    TotalAgreement,
+    call_limit,
+)
 from .prompts import DEFAULT_DIMENSIONS, SEMANTIC_EQUIVALENCE_PROMPT, Dimension, build_rubric_prompt
-from .sampling import AllPairs, AnchorPairs, CycleSampler, PairSampler, RandomPairs, cycle_pairs
+from .sampling import AllPairs, AnchorPairs, CrossSampler, CycleSampler, PairSampler, RandomPairs, cycle_pairs
 from .similarity import (
     DiceSimilarity,
     EditSimilarity,
@@ -19,13 +27,22 @@ from .similarity import (
     RubricJudge,
     Similarity,
     as_similarity,
+    render_prompt,
 )
 from .stats import bootstrap_ci
 from .types import ConsistencyResult, PairScore, SimilarityResult
 
-__version__ = "0.1.0"
+try:
+    from importlib.metadata import PackageNotFoundError
+    from importlib.metadata import version as _version
+
+    __version__ = _version("repeatability-evals")
+except PackageNotFoundError:  # running from a source checkout that is not installed
+    __version__ = "0+unknown"
 
 __all__ = [
+    "call_limit", "AllComparisonsFailed", "render_prompt",
+    "compare_versions", "acompare_versions", "VersionComparison", "InputComparison", "CrossSampler",
     "ConsistencyMetric", "PairwiseConsistency", "ClusterConsistency", "TotalAgreement",
     "evaluate_groups", "aevaluate_groups", "DatasetResult", "bootstrap_ci",
     "Similarity", "FunctionSimilarity", "as_similarity",

repeatability_evals-0.3.0/src/repeatability_evals/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())