probatio-llm 0.1.0__tar.gz

probatio_llm-0.1.0/.gitignore

@@ -0,0 +1,35 @@
+# Byte-compiled / cache
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Build artefacts
+build/
+dist/
+*.egg-info/
+.eggs/
+
+# Tooling caches
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.hypothesis/
+
+# Coverage
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+
+# Editors and OS
+.idea/
+.vscode/
+.DS_Store
+
+# NOTE: .probatio/ is deliberately NOT ignored. It holds committed baselines and
+# judge validation records, which CI must read.

probatio_llm-0.1.0/CHANGELOG.md

@@ -0,0 +1,147 @@
+# Changelog
+
+Notable changes to Probatio, newest first. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[semantic versioning](https://semver.org/spec/v2.0.0.html).
+
+The `Added` list below is in the order of the specification's architecture section, which is also
+the order in which the phases recorded in [`PROGRESS.md`](PROGRESS.md) built it.
+
+This file states no measurement. Every figure Probatio has produced about itself is printed in
+[`README.md`](README.md) and under [`docs/`](docs/), and indexed against the committed file that
+produced it in [`docs/PROVENANCE.md`](docs/PROVENANCE.md); repeating a number here would be a
+second copy with no test behind it.
+
+## [0.1.0] — 2026-09-06
+
+First release. Probatio is a pytest plugin for regression-testing LLM applications, distributed as
+`probatio-llm` and imported as `probatio`. Its two headline features are metamorphic relations and
+flakiness statistics; everything else on this list exists so that those two are usable from a
+normal test suite.
+
+### Added
+
+- **Foundations.** `ProbatioError` and the seven subclasses that name the offending case and, where
+  one exists, the command that fixes it: `ProbatioConfigError`, `MissingCassetteError`,
+  `StaleCassetteError`, `MissingVariantsError`, `BaselineDriftError`, `BudgetExceededError`,
+  `JudgeOutputError`. `stable_hash`, the canonical-JSON content hash behind every persisted
+  identity in the tool — cassette keys, prompt hashes, variant provenance, label files — stable
+  across processes and platforms.
+- **Cases.** `LLMCase`, a pydantic model with an id, a string-or-mapping input, an optional system
+  prompt and parameters, its assertions, an optional budget, a snapshot mode and tags; and
+  `load_cases`, which loads a directory of YAML files in filename order and reports a malformed
+  file by name and field rather than by traceback.
+- **Providers.** The `Provider` protocol and the `Completion` it returns, carrying text, token
+  counts, latency and a cost that is `None` when the provider cannot price the call rather than a
+  zero a ceiling would silently pass. Four adapters: `FakeProvider` and `ScriptedProvider` for
+  tests, `AnthropicProvider` over the SDK behind the `anthropic` extra, and `ClaudeCLIProvider`,
+  which runs the Claude Code CLI as a subprocess in a fresh empty directory so a developer on a
+  Claude plan can record without an API key. `--probatio-timeout` sets the subprocess ceiling.
+- **Assertions.** Five checks over one `AssertionResult`: `schema_valid` (JSON Schema, with fenced
+  blocks stripped), `contains` and `not_contains`, `similarity` against a reference, and `judge`.
+  Similarity ships a pure-Python trigram-cosine backend and a `SimilarityBackend` protocol for
+  pluggable embeddings; nothing is downloaded, and no backend but the default is ever installed or
+  invoked by a test run.
+- **Judge and validation.** `Judge` grades a case against a markdown rubric resolved from
+  `rubrics/`; the prompt template is a module constant whose hash is part of every judge cassette
+  key. A reply that is not strict JSON is asked again a bounded number of times and then returned
+  as a failed assertion, never raised. `cohens_kappa` and the validation record it writes;
+  `probatio validate-judge` measures a judge against human labels, from two existing columns or by
+  running the judge over the rows, and exits non-zero below `--min-kappa`. A judge with no
+  validation record on disk is reported as unenforceable rather than trusted.
+- **Snapshots.** Per-case baselines under `.probatio/baseline/`, in `scores` or `output` mode,
+  keyed on a hash of the prompt so a case whose input changed is reported as a new baseline rather
+  than as drift. `--update-baseline` rewrites them; `--baseline-dir` moves them.
+- **Budgets.** Per-case `max_cost_usd` and `max_latency_ms`, suite-wide `--max-cost` and
+  `--max-latency`, and `PriceTable` loaded from `--probatio-prices`. **The unenforceable rule:** a
+  cost ceiling evaluated where any call has an unknown cost is reported under warnings and not
+  under passed checks, because a ceiling that cannot be evaluated has not been met. No prices ship
+  with the package.
+- **Cassettes.** `CassetteProvider` wraps any provider in `replay`, `record` or `off` mode, with
+  `replay` the default, so a suite runs from committed tapes and makes no model call. Tapes are
+  keyed on prompt, system, model, parameters and — for a judge call — the template hash, so an
+  edited prompt or rubric is detected as a stale tape rather than replayed as if nothing changed.
+  Missing and stale tapes are both raised and carried into the report. `--cassette-dir` moves the
+  store; `probatio import-cassettes` builds one from a JSONL of interactions recorded elsewhere.
+- **Metamorphic relations.** The `Relation` base and its registry, and four decorators:
+  `@paraphrase_invariant`, `@order_invariant`, `@distractor_robust` and `@format_jitter`. Each
+  generates variants of a case that should not change its verdict; the plugin runs them and reports
+  a per-relation violation rate with the individual flips, and prints *not applicable* rather than
+  a rate for a relation whose case does not carry the field it needs. Paraphrases are not generated
+  at test time: `probatio freeze-variants` asks a model once, a human reviews and edits the
+  written files, and the suite reads the frozen, committed variants.
+- **Flakiness statistics.** `--runs N` repeats each case in place and reports a per-case pass rate
+  with a hand-implemented Wilson 95% interval, a majority verdict, a suite stability score, and the
+  number of cases whose interval falls below their declared floor. `@flaky_tolerant(p=…)` declares
+  that floor per case; a case that declares none is held to a floor of 1.0.
+- **Reporters.** One `RunReport` rendered four ways: the terminal section pytest prints at the end
+  of every run, a GitHub-flavoured markdown report written to `$GITHUB_STEP_SUMMARY` or
+  `--probatio-report`, JUnit XML at `--probatio-junit` carrying pass rates, interval bounds, costs
+  and per-relation violation rates as properties, and the full report as JSON at
+  `--probatio-results`. A quantity that was not measured is omitted, never written as a zero.
+- **The pytest surface.** The `probatio`, `provider` and `judge_provider` fixtures; the
+  `probatio_relation` and `flaky_tolerant` markers; and the option group that selects the provider
+  and model, the judge provider and model, the cassette mode and store, the baseline directory,
+  the price table, the ceilings, the timeout and the three report paths. A non-fake provider with
+  no model named is refused before the session starts.
+- **The `probatio` console script.** `validate-judge`, `freeze-variants` and `import-cassettes`.
+  These are the only entry points that call a live model, and only when a human runs them.
+- **Packaging.** `probatio-llm` on Python 3.11 and newer, with runtime dependencies of exactly
+  `pytest`, `pydantic`, `jsonschema` and `PyYAML`; an `anthropic` extra for the SDK adapter and a
+  `dev` extra for the gate. Ships `py.typed`: the package is typed and `mypy --strict` is part of
+  the gate. The sdist carries the package, this file, the README and the licence, and neither
+  `tests/` nor `examples/`.
+- **Examples.** `examples/demo_suite/`, the executable design specification, written before the
+  implementation and byte-frozen since — every decorator, every assertion type and a deliberate
+  failing case, runnable offline. `examples/consilium/`, a dogfood suite over a real
+  retrieval-augmented medical Q&A system, offline from committed tapes, with a live variant used
+  to record them.
+- **Documentation.** `README.md` with a prior-art comparison against DeepEval, promptfoo, Giskard,
+  Ragas, Braintrust and LangSmith; `docs/DESIGN.md`, `docs/assertions.md`, `docs/relations.md`,
+  `docs/stability.md`, `docs/providers.md`, `docs/EVALUATION.md`, `docs/CASE_STUDY.md`; and
+  `docs/PROVENANCE.md`, which indexes every number in the documentation to the committed file that
+  produced it and is enforced by a test that re-derives each one.
+- **Continuous integration.** The five-part quality gate — pytest, coverage of `src/probatio`,
+  ruff, `mypy --strict`, and byte-identity of the frozen demo suite — on the floor version and the
+  one above it, with no credentials and no network access to a model provider.
+
+### Known limitations
+
+Stated in full, with the evidence, in [`docs/CASE_STUDY.md`](docs/CASE_STUDY.md) §5 and
+[`docs/EVALUATION.md`](docs/EVALUATION.md); summarised here because they bound what this release
+has shown.
+
+- **The judge in this repository is not validated on disk, deliberately.** Its two measured kappas
+  disagree with each other, and neither is high; `docs/EVALUATION.md` §4 gives both and says why no
+  validation record was written. Every judge verdict in the case study therefore carries
+  `unenforceable=True`, and the relation rates that depend on those verdicts inherit it. They are
+  reported because an unenforceable result is still a measurement — of an unvalidated instrument.
+- **A suite cannot yet declare a minimum kappa that fails the build.**
+  `probatio validate-judge --min-kappa` exits non-zero from the command line, but there is no
+  equivalent of a budget overrun inside a pytest run.
+- **`--runs N` repeats inside one pytest item, so function-scoped fixtures are shared across the
+  runs.** That is deliberate — it measures the model's nondeterminism, not state leakage — but a
+  suite wanting a fresh fixture per run has to build one inside its own callable
+  (`docs/stability.md`).
+- **The case study's live system under test is not the system it borrows its questions from.** It
+  makes one grounded call with the corpus notes already in hand; the real pipeline plans,
+  retrieves, runs agents and repairs. Retrieval is not modelled at all, so nothing here speaks to
+  retrieval quality.
+- **The case study is a small sample, run once, on one model family.** Fifteen cases, one run
+  each, two Anthropic models reached through the same adapter. No interval in that document is a
+  confidence interval, and the flip pattern most in need of repetition is the one repetition would
+  have resolved.
+- **Route A of the case study is retrospective.** The regression it replays was found and
+  documented by the system's own evaluation before the suite was written, and the suite's
+  assertions were derived from that system's golden labels and phrase list.
+- **Costs reported through `ClaudeCLIProvider` are notional.** They are the API price the CLI
+  computes for calls made on a subscription that was not billed that money;
+  `docs/providers.md` says so wherever the figure is used.
+- **Anthropic is the only live provider.** There is no OpenAI adapter and none is planned; another
+  provider is a one-class extension, documented in `docs/providers.md`.
+- **Similarity's default backend is lexical.** Trigram cosine is deterministic and needs no
+  download, which is why it is the default; it is not a semantic measure, and
+  `docs/assertions.md` publishes the band it produced on real answers instead of recommending a
+  threshold.
+
+[0.1.0]: https://github.com/Lexieli666/probatio/releases/tag/v0.1.0

probatio_llm-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yichen Li
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

probatio_llm-0.1.0/PKG-INFO

@@ -0,0 +1,326 @@
+Metadata-Version: 2.5
+Name: probatio-llm
+Version: 0.1.0
+Summary: A pytest plugin for regression-testing LLM applications: metamorphic relations and flakiness statistics.
+Project-URL: Homepage, https://github.com/Lexieli666/probatio
+Project-URL: Repository, https://github.com/Lexieli666/probatio
+Project-URL: Changelog, https://github.com/Lexieli666/probatio/blob/main/CHANGELOG.md
+Author-email: Yichen Li <lexieli@seas.upenn.edu>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: llm,metamorphic-testing,pytest,regression-testing,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: jsonschema>=4
+Requires-Dist: pydantic>=2
+Requires-Dist: pytest>=8
+Requires-Dist: pyyaml>=6
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.40; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: coverage; extra == 'dev'
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Requires-Dist: types-jsonschema; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Probatio
+
+Probatio is a **pytest plugin for regression-testing LLM applications**. You write cases in YAML,
+one test function per suite, and keep running the suite you already have. It adds two disciplines
+the software-testing literature has and LLM developer tooling mostly does not: **metamorphic
+relations**, decorators that assert a semantics-preserving change to an input does not change the
+verdict, reported as a per-relation violation rate; and **flakiness statistics**, repeated
+execution with a per-case pass rate, a Wilson 95% interval and a suite stability score, instead of
+running a nondeterministic system once and printing a tick. Everything else it ships — JSON-schema
+validity, contains and not-contains assertions, similarity, a rubric judge with Cohen's κ against
+human labels, snapshot baselines, cost and latency ceilings, record-replay cassettes, and
+terminal, GitHub-summary, JUnit-XML and JSON reporters — is there so those two are usable on a
+real suite. The judge shipped here has **measured agreement** with human labels, not validated
+agreement; §*Case study* says what was measured and what it came to.
+
+## Quick start
+
+```bash
+pip install probatio-llm
+```
+
+```yaml
+# examples/demo_suite/cases/06-copd-spirometry.yaml
+id: copd-spirometry
+input: "In one sentence, what test confirms a diagnosis of COPD?"
+assertions:
+  - {type: contains, any: ["spirometry"]}
+```
+
+```python
+# examples/demo_suite/test_demo.py
+CASES = load_cases(HERE / "cases")
+
+
+@pytest.mark.parametrize("case", CASES, ids=lambda c: c.id)
+@format_jitter(field="input.question")
+def test_case(case, probatio, provider):
+    probatio.check(case, sut=lambda c: answer(c, provider))
+```
+
+```bash
+pytest examples/demo_suite --runs 5
+```
+
+That is shortened from [`examples/demo_suite/`](examples/demo_suite/), the executable
+specification this tool was built against and which has not been edited since it was written. Every
+line above is verbatim from it except one: the real test parametrizes over a filtered subset of
+`CASES` rather than over `CASES` itself, because the demo routes its expected-fail and flaky cases
+to their own test functions. What is left out is the rest of that case — a second assertion, a
+budget, its tags — and the three other relations the real `test_case` carries. Note what the shown
+case does *not* exercise: its `input` is a bare string, so `@format_jitter(field="input.question")`
+reports **not applicable** on it rather than `0.00`, and
+[`examples/demo_suite/cases/01-htn-definition.yaml`](examples/demo_suite/cases/01-htn-definition.yaml)
+is the dict-input case, with a `question` and a list of `documents`, that the field relations
+actually apply to.
+
+## Metamorphic relations
+
+A relation names a transformation that should preserve a case's meaning and asserts the verdict
+does not move. Four ship as decorators — `@order_invariant`, `@paraphrase_invariant`,
+`@distractor_robust`, `@format_jitter` — and each reports a violation rate **alongside** the case,
+never folded into its pass or fail. A violation counts in either direction: a variant that passes
+where the original failed breaks the invariant just as much.
+
+From `pytest examples/demo_suite --runs 5`, run at commit `4c2114e` and recorded verbatim in
+[`PROGRESS.md`](PROGRESS.md):
+
+```
+relations:
+  relation              cases  n/a  violations  mean rate  worst case           worst rate
+  --------------------  -----  ---  ----------  ---------  -------------------  ----------
+  distractor_robust     7      1    0/70        0.00       gerd-alarm-features  0.00
+  format_jitter         7      1    15/105      0.14       htn-definition       0.33
+  order_invariant       6      2    0/50        0.00       gerd-alarm-features  0.00
+  paraphrase_invariant  2      0    5/30        0.17       htn-definition       0.33
+```
+
+The `n/a` column is the point of the design: a relation a case is out of scope for — reordering a
+one-element document list, rewording a case whose input is a bare string — reports *not
+applicable*, never `0.00`. Paraphrases are generated once by `probatio freeze-variants`, reviewed
+by a human and committed; they are never sampled during a test run.
+
+## Flakiness statistics
+
+`--runs N` runs each case N times and reports what happened, rather than what happened once. Same
+run, same commit:
+
+```
+cases:
+  case                   verdict  assertions  pass rate  95% Wilson    floor  cost       latency ms  snapshot
+  ---------------------  -------  ----------  ---------  ------------  -----  ---------  ----------  ---------
+  htn-definition         pass     4/4         1.00       [0.57, 1.00]  1.00   $0.000500  100         unchanged
+  htn-first-line         pass     3/3         1.00       [0.57, 1.00]  1.00   $0.000500  100         unchanged
+  t2d-screening-json     pass     2/2         1.00       [0.57, 1.00]  1.00   $0.000500  100         -
+  t2d-metformin          pass     4/4         1.00       [0.57, 1.00]  1.00   $0.000500  100         unchanged
+  gerd-alarm-features    pass     3/3         1.00       [0.57, 1.00]  1.00   $0.000500  100         -
+  copd-spirometry        pass     2/2         1.00       [0.57, 1.00]  1.00   $0.000500  100         -
+  red-flag-chest-pain    pass     3/3         1.00       [0.57, 1.00]  1.00   $0.000500  100         -
+  insomnia-first-line    pass     4/4         1.00       [0.57, 1.00]  1.00   $0.000500  100         -
+  htn-definition         pass     4/4         1.00       [0.57, 1.00]  1.00   $0.000500  100         unchanged
+  t2d-metformin          pass     4/4         1.00       [0.57, 1.00]  1.00   $0.000500  100         unchanged
+  anxiety-expected-fail  FAIL     1/3         0.00       [0.00, 0.43]  1.00   $0.000500  100         -
+  flu-antivirals-flaky   pass     2/2         0.80       [0.38, 0.96]  0.80   $0.000500  100         -
+```
+
+```
+stability score: 0.90 over 12 repeated case(s)
+cases whose Wilson lower bound is below their floor: 12 of 12
+cost: $0.031500 (no --max-cost ceiling)
+```
+
+`flu-antivirals-flaky` declares `@flaky_tolerant(p=0.8, n=5)`, passes four of its five runs, and
+passes the case at the bar its author wrote down; `anxiety-expected-fail` fails every run and is
+in the suite so the report has a failure to show. The stability score is the mean of those twelve
+pass rates.
+
+**"12 of 12" is not twelve problems.** It counts the repeated cases whose Wilson *lower* bound
+falls under their floor, and at n = 5 the lower bound of a perfect five-for-five is 0.57, so no
+case can clear a floor of 1.0 however well it does — the line is a distance from the suite's own
+floors, and it becomes informative when `--runs` is large enough for a bound to reach one
+([`docs/stability.md`](docs/stability.md)). Nothing in it decides a verdict: the observed pass
+rate does that.
+
+The run also names what it could not check:
+
+```
+warnings (7):
+  - htn-definition: 1 judge verdict(s) from a rubric with no validation record (run: probatio validate-judge --labels <csv> --rubric <name>)
+  - htn-first-line: 1 judge verdict(s) from a rubric with no validation record (run: probatio validate-judge --labels <csv> --rubric <name>)
+```
+
+— and five more like them, one per judged case. A judge with no validation record on disk is not a
+validated judge, and the tool says so about every suite, including its own.
+
+## Prior art
+
+The field is crowded and this table is the honest version of where Probatio sits. Every claim in
+it is what that tool's own documentation says, **checked against each tool's documentation on
+2026-09-05**; "not documented" means the page that would carry the feature was read and does not
+carry it, not that the feature is absent.
+
+| tool | what it is (their words) | pytest | repeated runs | interval on a per-case pass rate | metamorphic relations | JUnit XML |
+|---|---|---|---|---|---|---|
+| **DeepEval** | "plugs into Pytest, so `deepeval test run` collects and runs your eval files the same way pytest would"; a large catalog of "research-backed LLM-as-a-Judge" metrics | yes, native (`assert_test`) | not documented | none documented; a **flaky metric** flag instead | none documented | not documented |
+| **promptfoo** | "open-source CLI and library for evaluating and red-teaming LLM apps" | no | **yes**, `--repeat <number>` | none documented | none documented | **yes**, `--output junit.xml` |
+| **Giskard** | "enterprise platform and open-source library for LLM evaluation and security" | yes; behavioural tests "pass or fail under pytest" | not documented | none documented | the word does not appear; its scan is adversarial generation | not documented |
+| **Ragas** | "move from 'vibe checks' to systematic evaluation loops"; RAG metrics | not documented | not documented | none documented | none documented | not documented |
+| **Braintrust** | "the active observability platform for instrumenting, understanding, and improving agents" | not documented | not documented | none documented | none documented | not documented |
+| **LangSmith** | "a framework for measuring quality throughout the application lifecycle" | **yes**, `@pytest.mark.langsmith` | **yes**, `num_repetitions` | mean and standard deviation; no interval documented | none documented | not documented |
+| **Probatio** | this repository | yes, a plugin | yes, `--runs N` | Wilson 95% per case, plus a suite stability score and a floor marker | four, as decorators, with frozen variants | yes, with the statistics as `<properties>` |
+
+Pages read on 2026-09-05: `deepeval.com/docs/getting-started` and `/docs/metrics-introduction`;
+`promptfoo.dev/docs/intro` and `/docs/usage/command-line`; `docs.giskard.ai`;
+`docs.ragas.io/en/stable`; `braintrust.dev/docs` and `/docs/guides/evals`;
+`docs.langchain.com/langsmith/evaluation-concepts`, `/langsmith/pytest`, `/langsmith/repetition`
+and `/langsmith/evaluate-llm-application`.
+
+**What is not a differentiator, said plainly.** Being pytest-native is not one: DeepEval, Giskard
+and LangSmith all are, and DeepEval owns that ground with a far larger metric catalog. Running a
+case repeatedly is not one: promptfoo has `--repeat` and LangSmith has `num_repetitions`. JUnit
+output is not one: promptfoo writes `junit.xml`. If you want a metric catalog, use DeepEval; if
+you want a red-teaming engine, use promptfoo or Giskard; if you want hosted eval tracing, use
+Braintrust or LangSmith. Probatio is meant to sit beside them.
+
+**What is.** A confidence interval on a per-case pass rate, a suite-level stability score, and a
+floor marker (`@flaky_tolerant(p, n)`) that holds a case to a number somebody wrote down — none of
+the six documents any of these. And metamorphic relations as stated invariants, declared as
+decorators on the test: none of the six documents those either, and Giskard's scan is adversarial
+generation, which searches for inputs that break a system rather than checking a property the
+system is supposed to have. The one-sentence contrast that matters: **DeepEval's answer to a flaky
+metric is a flag that makes its failure non-deciding; Probatio's is to run the case n times and
+report the rate, the interval and the floor.**
+
+Metamorphic testing of LLMs exists as research, and Probatio claims no new relation — its four are
+packaging, with reviewable frozen variants, of transformations these papers name. LLMorph
+`[LLMORPH]` and MTF `[MTF]` are the two research frameworks in this space; the NLP catalogue
+`[MR-CATALOG-NLP]` collected **191** metamorphic relations in a literature review, of which 36
+were implemented, and it is the source `order_invariant` and `paraphrase_invariant` cite. Chen et
+al. `[CHEN-MT-SURVEY]` and Segura et al. `[SEGURA-MT-SURVEY]` are the surveys for the definition
+of a relation and for violation-as-oracle. Full entries, with authors, venues, years and DOIs
+verified against Crossref on 2026-09-05, are in [`docs/relations.md`](docs/relations.md).
+
+## Case study
+
+Probatio was dogfooded on
+[Consilium-Health](https://github.com/Lexieli666/consilium-health), the author's own
+public multi-agent retrieval-grounded
+question-answering project, along two routes. Both are written up with their limitations in
+[`docs/CASE_STUDY.md`](docs/CASE_STUDY.md), and every number below is re-derived from a committed
+artefact by `tests/test_docs_case_study.py`.
+
+**Route A — a retrospective catch, and the headline claim.** Fifteen of Consilium's golden items
+replay through Probatio from its published traces, against two recorded configurations. On
+`baseline_llm`, a single call with no retrieval and no agents, all fifteen pass. On the
+multi-agent `full` pipeline, **five of the six red-flag cases fail** — each on a `contains`
+assertion carrying Consilium's own thirty-eight escalation phrases, and on nothing else — while
+**none fails on `baseline_llm`**. That is a safety regression Consilium had already documented in
+its own `docs/FAILURE_CASES.md`, reproduced as a failing `pytest` run from committed tapes with no
+model called. It is labelled retrospective in the case study, because it is: the suite was written
+after the regression was known.
+
+**Route B — prospective in form.** One flag changed, `claude-opus-5` to
+`claude-haiku-4-5-20251001`, on a live single-call system under test, with nothing else moved.
+The suite reported that **4 of 15 verdicts moved**, all of them on the judge; that the judge marked
+**a third of the cases** (5 of 15) unfaithful to their sources, naming the sentence in each; that
+eight snapshots drifted, three of them on cases whose every assertion still passed; that
+`paraphrase_invariant` went from 0.04 to 0.30 under rewordings a human had confirmed do not change
+the question; and that the cheaper recording cost **$0.987356 against $6.037457** in notional API
+price, a six-fold difference. It is prospective in form — suite first, change second, outcome
+unknown in advance — and it is on a system under test built for the purpose, not on a shipping
+application. The case study says so in those words.
+
+Dogfooding also found three things wrong with Probatio itself — two defects and one gap that nine
+phases of unit tests had not reached — all fixed during Phase 12 and listed with the commit that
+fixed each in [`docs/EVALUATION.md`](docs/EVALUATION.md) §7.
+
+## Design positions
+
+**Similarity is for paraphrase-stable content; a judge is for claims.** Reach for `similarity`
+when what you are protecting is wording that should stay roughly the same, and for a `judge` when
+it is whether a claim is true, grounded or complete. The docs take that position rather than
+listing options, because the substitution fails in a way a threshold cannot fix: a trigram cosine
+scores a sentence that *contradicts* the reference at 0.223 while the same claim in different
+words scores 0.453, so the ordering is wrong, not the number
+([`docs/assertions.md`](docs/assertions.md), rationale in [`docs/DESIGN.md`](docs/DESIGN.md)).
+
+**An unenforceable ceiling is never a pass.** A `max_cost_usd` ceiling evaluated where any call's
+cost is unknown cannot be checked, so it is reported as unenforceable, listed under warnings, and
+excluded from passing checks — never quietly satisfied by treating unknown as zero. The same holds
+for a case that recorded no provider calls at all: a check that cannot fail is not a check. This
+is the failure mode a testing tool must not have, and it is why `Completion.cost_usd` is `None`
+rather than `0.0` when nobody priced the call (`docs/DESIGN.md`, Phases 6 and 11).
+
+**Frozen variants.** Paraphrases are generated once by `probatio freeze-variants`, written to
+`variants/<case_id>.yaml` with their provenance, reviewed, and committed. Three reasons in order:
+the suite stays deterministic; the model is not part of the oracle at test time; and the variants
+are in the diff, so a reviewer can throw out a rewording that changed the meaning. That last one is
+not hypothetical — **2 of 45** frozen paraphrases were deleted at review in this project, each
+noted in its file's header ([`docs/EVALUATION.md`](docs/EVALUATION.md) §3).
+
+**A plugin, not a platform.** Teams do not migrate eval platforms mid-quarter; they add a
+dependency to the tests they already run. So Probatio is four runtime dependencies, one entry
+point, a `Provider` protocol of one method, and reporters that write into CI you already have —
+JUnit XML and a GitHub job summary — rather than a dashboard to log into. No hosted service, no
+async, no metric catalog (`docs/DESIGN.md`, Phase 0).
+
+## Install
+
+The distribution is named `probatio-llm`; the import name is `probatio`.
+
+```bash
+pip install probatio-llm                  # core: pytest, pydantic, jsonschema, PyYAML
+pip install "probatio-llm[anthropic]"     # plus the Anthropic SDK adapter
+pip install "probatio-llm[dev]"           # ruff, mypy, coverage, build, twine
+```
+
+Python 3.11 or newer. The `claude-cli` provider needs no extra and no API key — it runs the Claude
+Code CLI as a subprocess, so a developer on a Claude plan can record cassettes and freeze variants;
+the cost it reports is *notional* API price, not money billed, and
+[`docs/providers.md`](docs/providers.md) says so wherever that figure is used.
+
+> The unrelated PyPI placeholder distribution `probatio` also installs a `probatio/` import
+> package. Do not install both into the same environment.
+
+## Status and roadmap
+
+Pre-1.0. Phase-by-phase state, with the gate result and the run log for each phase, is in
+[`PROGRESS.md`](PROGRESS.md); every judgement call made without asking is numbered in
+[`DECISIONS.md`](DECISIONS.md); anything abandoned is in [`BLOCKERS.md`](BLOCKERS.md). Every number
+in this file and under `docs/` is indexed in [`docs/PROVENANCE.md`](docs/PROVENANCE.md) with the
+committed file it came from and the command that regenerates it.
+
+Next, in the order they would be built:
+
+- **Per-run fixture isolation.** `--runs N` repeats inside one pytest item, so function-scoped
+  fixtures are shared across the runs. That is deliberate — it measures the model's
+  nondeterminism, not state leakage — but a suite that wants a fresh fixture per run has to build
+  it inside its own callable today (`docs/stability.md`).
+- **A minimum-κ gate.** `probatio validate-judge --min-kappa` already exits non-zero, but a suite
+  cannot yet declare a floor that fails the build the way a budget overrun does. The judge in this
+  repository scored κ 0.600 on one blind forty-row sample and 0.253 on another, which is the
+  argument for the gate and for reporting the number rather than the adjective.
+- **The variance study.** The `--runs` engine is also an instrument: run a benchmark subset n
+  times and measure how often rankings flip. Seeded as an optional phase in `PROGRESS.md`, and
+  the one thing that would settle whether the judge or the answer is the unstable half of this
+  project's own relation rates (`docs/EVALUATION.md` §2).
+
+## License
+
+MIT. See [`LICENSE`](LICENSE).