physverify 0.4.1__tar.gz → 0.4.2__tar.gz

physverify-0.4.2/PKG-INFO

@@ -0,0 +1,89 @@
+Metadata-Version: 2.4
+Name: physverify
+Version: 0.4.2
+Summary: Verification checks for physics simulation code: manufactured solutions, order-of-accuracy studies and a seeded-bug benchmark.
+Author: Ram Chandra Giri
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/rcgiri-physics/physverify
+Project-URL: Documentation, https://rcgiri-physics.github.io/physverify/
+Project-URL: Issues, https://github.com/rcgiri-physics/physverify/issues
+Project-URL: Changelog, https://github.com/rcgiri-physics/physverify/blob/main/CHANGELOG.md
+Keywords: verification,method of manufactured solutions,numerical methods,physics,simulation,testing
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Physics
+Classifier: Framework :: Pytest
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Education
+Classifier: Topic :: Software Development :: Testing
+Classifier: Development Status :: 3 - Alpha
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Provides-Extra: symbolic
+Requires-Dist: sympy>=1.12; extra == "symbolic"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-cov>=6.1; extra == "dev"
+Requires-Dist: hypothesis>=6; extra == "dev"
+Requires-Dist: sympy>=1.12; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
+Requires-Dist: mkdocs-material<10,>=9.5; extra == "docs"
+Dynamic: license-file
+
+# physverify
+
+[![tests](https://github.com/rcgiri-physics/physverify/actions/workflows/ci.yml/badge.svg)](https://github.com/rcgiri-physics/physverify/actions/workflows/ci.yml) [![PyPI](https://img.shields.io/pypi/v/physverify)](https://pypi.org/project/physverify/) [![Python](https://img.shields.io/pypi/pyversions/physverify)](https://pypi.org/project/physverify/) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23534512.svg)](https://doi.org/10.5281/zenodo.23534512) [![Docs](https://img.shields.io/badge/docs-site-blue)](https://rcgiri-physics.github.io/physverify/)
+
+**Checks that a simulation solves the equations it claims, at the order of accuracy it claims, not just that it runs.**
+
+physverify is actively evolving (v0.4.2), welcoming contributions and test cases from the computational physics community. Scope: numerical solutions of ODEs and PDEs (time integration, convergence, conservation), plus a few Monte Carlo checks. It needs only NumPy.
+
+![The check loop: your solver and an independent description of the physics go in; physverify runs the solver at several resolutions, measures the error and reports a verdict with a reason](https://raw.githubusercontent.com/rcgiri-physics/physverify/main/docs/images/check-loop.svg)
+
+```bash
+pip install physverify
+physverify demo
+```
+
+The demo checks a correct solver (all checks pass), then the same solver with the damping 10% too large (all fail):
+
+![Error against step size: the correct solver follows the slope-2 line, the broken one levels off](https://raw.githubusercontent.com/rcgiri-physics/physverify/main/docs/images/convergence.svg)
+
+## Use it on your own code
+
+- **An ODE solver:** `physverify init verify_solver.py --solver mysolver --order 4` writes a spec; fill in the equations (from the physics, not from your code) and run `physverify run verify_solver.py`.
+- **A Runge-Kutta table:** `check_tableau(A, b, c, claimed=3)` from `physverify.butcher` checks its order, stage order and stability from the coefficients alone.
+- **Errors you already have:** `check_order(hs, errors, expected=2)` gives PASS, FAIL or INCONCLUSIVE with a reason.
+
+## Run it in CI
+
+A failed check fails the job, like a broken unit test ([CI guide](https://rcgiri-physics.github.io/physverify/ci/): options, pre-commit, pytest):
+
+```yaml
+      - uses: rcgiri-physics/physverify@v0.4.2
+        with:
+          specs: tests/verify_*.py
+```
+
+[Getting started](https://rcgiri-physics.github.io/physverify/getting-started/) · [Recipes](https://rcgiri-physics.github.io/physverify/recipes/) · [What a PASS means](https://rcgiri-physics.github.io/physverify/assumptions/) · [Troubleshooting](https://rcgiri-physics.github.io/physverify/troubleshooting/) · [API](https://rcgiri-physics.github.io/physverify/api/)
+
+## Does it catch bugs?
+
+Of 37 bugs seeded into five small solvers, it flagged 35 (plain "compare with one answer" tests: 27), with no false alarms. The bugs and checks were written by the same person, so this shows the tool works as designed, not how it does on other people's code. On public physics repositories it found real bugs, for example Runge-Kutta tables with a wrong order label; the counts are not a bug rate. Details: [results](https://rcgiri-physics.github.io/physverify/results/), and [results/README.md](https://github.com/rcgiri-physics/physverify/blob/main/results/README.md) to regenerate every table.
+
+## Limits
+
+Works best for ODEs; PDEs only in 1D so far. Your code needs a way to add a source term, or a small wrapper that can itself be wrong. A PASS means these checks found nothing wrong; it does not prove the simulation correct, and it is only as independent as the reference you wrote: see [What a PASS means](https://rcgiri-physics.github.io/physverify/assumptions/).
+
+## Contributing, citing, licence
+
+Help is welcome: a trial report on your own solver, seeded bugs written by someone other than the author, adapters, documentation. See [CONTRIBUTING.md](https://github.com/rcgiri-physics/physverify/blob/main/CONTRIBUTING.md). To cite, use [doi:10.5281/zenodo.23534512](https://doi.org/10.5281/zenodo.23534512) or [CITATION.cff](https://github.com/rcgiri-physics/physverify/blob/main/CITATION.cff). Changes: [CHANGELOG.md](https://github.com/rcgiri-physics/physverify/blob/main/CHANGELOG.md); plans: [ROADMAP.md](https://github.com/rcgiri-physics/physverify/blob/main/ROADMAP.md). MIT licence, copyright (c) 2026 Ram Chandra Giri.

physverify-0.4.2/README.md

@@ -0,0 +1,48 @@
+# physverify
+
+[![tests](https://github.com/rcgiri-physics/physverify/actions/workflows/ci.yml/badge.svg)](https://github.com/rcgiri-physics/physverify/actions/workflows/ci.yml) [![PyPI](https://img.shields.io/pypi/v/physverify)](https://pypi.org/project/physverify/) [![Python](https://img.shields.io/pypi/pyversions/physverify)](https://pypi.org/project/physverify/) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23534512.svg)](https://doi.org/10.5281/zenodo.23534512) [![Docs](https://img.shields.io/badge/docs-site-blue)](https://rcgiri-physics.github.io/physverify/)
+
+**Checks that a simulation solves the equations it claims, at the order of accuracy it claims, not just that it runs.**
+
+physverify is actively evolving (v0.4.2), welcoming contributions and test cases from the computational physics community. Scope: numerical solutions of ODEs and PDEs (time integration, convergence, conservation), plus a few Monte Carlo checks. It needs only NumPy.
+
+![The check loop: your solver and an independent description of the physics go in; physverify runs the solver at several resolutions, measures the error and reports a verdict with a reason](https://raw.githubusercontent.com/rcgiri-physics/physverify/main/docs/images/check-loop.svg)
+
+```bash
+pip install physverify
+physverify demo
+```
+
+The demo checks a correct solver (all checks pass), then the same solver with the damping 10% too large (all fail):
+
+![Error against step size: the correct solver follows the slope-2 line, the broken one levels off](https://raw.githubusercontent.com/rcgiri-physics/physverify/main/docs/images/convergence.svg)
+
+## Use it on your own code
+
+- **An ODE solver:** `physverify init verify_solver.py --solver mysolver --order 4` writes a spec; fill in the equations (from the physics, not from your code) and run `physverify run verify_solver.py`.
+- **A Runge-Kutta table:** `check_tableau(A, b, c, claimed=3)` from `physverify.butcher` checks its order, stage order and stability from the coefficients alone.
+- **Errors you already have:** `check_order(hs, errors, expected=2)` gives PASS, FAIL or INCONCLUSIVE with a reason.
+
+## Run it in CI
+
+A failed check fails the job, like a broken unit test ([CI guide](https://rcgiri-physics.github.io/physverify/ci/): options, pre-commit, pytest):
+
+```yaml
+      - uses: rcgiri-physics/physverify@v0.4.2
+        with:
+          specs: tests/verify_*.py
+```
+
+[Getting started](https://rcgiri-physics.github.io/physverify/getting-started/) · [Recipes](https://rcgiri-physics.github.io/physverify/recipes/) · [What a PASS means](https://rcgiri-physics.github.io/physverify/assumptions/) · [Troubleshooting](https://rcgiri-physics.github.io/physverify/troubleshooting/) · [API](https://rcgiri-physics.github.io/physverify/api/)
+
+## Does it catch bugs?
+
+Of 37 bugs seeded into five small solvers, it flagged 35 (plain "compare with one answer" tests: 27), with no false alarms. The bugs and checks were written by the same person, so this shows the tool works as designed, not how it does on other people's code. On public physics repositories it found real bugs, for example Runge-Kutta tables with a wrong order label; the counts are not a bug rate. Details: [results](https://rcgiri-physics.github.io/physverify/results/), and [results/README.md](https://github.com/rcgiri-physics/physverify/blob/main/results/README.md) to regenerate every table.
+
+## Limits
+
+Works best for ODEs; PDEs only in 1D so far. Your code needs a way to add a source term, or a small wrapper that can itself be wrong. A PASS means these checks found nothing wrong; it does not prove the simulation correct, and it is only as independent as the reference you wrote: see [What a PASS means](https://rcgiri-physics.github.io/physverify/assumptions/).
+
+## Contributing, citing, licence
+
+Help is welcome: a trial report on your own solver, seeded bugs written by someone other than the author, adapters, documentation. See [CONTRIBUTING.md](https://github.com/rcgiri-physics/physverify/blob/main/CONTRIBUTING.md). To cite, use [doi:10.5281/zenodo.23534512](https://doi.org/10.5281/zenodo.23534512) or [CITATION.cff](https://github.com/rcgiri-physics/physverify/blob/main/CITATION.cff). Changes: [CHANGELOG.md](https://github.com/rcgiri-physics/physverify/blob/main/CHANGELOG.md); plans: [ROADMAP.md](https://github.com/rcgiri-physics/physverify/blob/main/ROADMAP.md). MIT licence, copyright (c) 2026 Ram Chandra Giri.

{physverify-0.4.1 → physverify-0.4.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "physverify"
-version = "0.4.1"
+version = "0.4.2"
 description = "Verification checks for physics simulation code: manufactured solutions, order-of-accuracy studies and a seeded-bug benchmark."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -32,13 +32,14 @@ dependencies = ["numpy>=1.24"]
 
 [project.urls]
 Homepage = "https://github.com/rcgiri-physics/physverify"
-Documentation = "https://github.com/rcgiri-physics/physverify/blob/main/docs/technical.md"
+Documentation = "https://rcgiri-physics.github.io/physverify/"
 Issues = "https://github.com/rcgiri-physics/physverify/issues"
 Changelog = "https://github.com/rcgiri-physics/physverify/blob/main/CHANGELOG.md"
 
 [project.optional-dependencies]
 symbolic = ["sympy>=1.12"]
-dev = ["pytest>=8", "hypothesis>=6", "sympy>=1.12"]
+dev = ["pytest>=8", "pytest-cov>=6.1", "hypothesis>=6", "sympy>=1.12"]
+docs = ["mkdocs>=1.6,<2", "mkdocs-material>=9.5,<10"]
 
 [project.scripts]
 physverify = "physverify.cli:main"

{physverify-0.4.1 → physverify-0.4.2}/src/physverify/__init__.py

@@ -6,7 +6,7 @@ from .order import check_order, fit_order, pairwise_orders
 from .order import check_exact
 from .studies import invariant_study, ode_checks, patch_study, reference_study
 
-__version__ = "0.4.1"
+__version__ = "0.4.2"
 
 __all__ = [
     "Check",

{physverify-0.4.1 → physverify-0.4.2}/src/physverify/checks.py

@@ -42,6 +42,7 @@ class OrderResult(Result):
     fitted: float = float("nan")
     hs: tuple = ()
     errors: tuple = ()
+    acceptance: str = ""  # the rule a PASS had to meet, in words (reported in the JSON output)
 
 
 @dataclass

{physverify-0.4.1 → physverify-0.4.2}/src/physverify/cli.py

@@ -1,6 +1,7 @@
 """Command line.
 
     physverify run SPEC.py [SPEC.py ...] [--json FILE|-] [--html FILE] [--markdown FILE] [--quiet]
+    physverify demo        (a correct and a broken solver, no files needed)
     physverify init [SPEC.py] [--solver MODULE] [--function NAME] [--dim N] [--order P]
     physverify matrix [OUT.md]
     physverify compare [OUT.md]
@@ -42,7 +43,7 @@ def expand_specs(patterns) -> list[str]:
 
 
 def _cmd_run(args) -> int:
-    from .report import load_spec, run_checks, to_html, to_json, to_markdown, to_text
+    from .report import NOTE, load_spec, run_checks, to_html, to_json, to_markdown, to_text
 
     try:
         args.specs = expand_specs(args.specs)
@@ -56,6 +57,9 @@ def _cmd_run(args) -> int:
         except Exception as exc:
             print(f"physverify: cannot load {spec}: {type(exc).__name__}: {exc}", file=sys.stderr)
             return 2
+        if not checks:
+            print(f"physverify: {spec}: checks() returned no checks, so nothing was tested", file=sys.stderr)
+            return 2
         outcomes += run_checks(checks, args.budget)
     source = ", ".join(str(Path(s)) for s in args.specs)
     if args.json == "-":
@@ -63,6 +67,7 @@ def _cmd_run(args) -> int:
     else:
         if not args.quiet:
             print(to_text(outcomes))
+            print(NOTE)
         if args.json:
             Path(args.json).write_text(to_json(outcomes, source), encoding="utf-8")
     if args.html:
@@ -87,6 +92,12 @@ def _cmd_init(args) -> int:
     return 0
 
 
+def _cmd_demo(args) -> int:
+    from .demo import main as demo_main
+
+    return demo_main()
+
+
 def _cmd_matrix(args) -> int:
     from . import matrix
 
@@ -149,6 +160,9 @@ def build_parser() -> argparse.ArgumentParser:
                           " and a check still running at twice the budget is reported INCONCLUSIVE")
     run.set_defaults(func=_cmd_run)
 
+    d = sub.add_parser("demo", help="run the checks on a correct and on a broken example solver (no files needed)")
+    d.set_defaults(func=_cmd_demo)
+
     i = sub.add_parser("init", help="write a starter spec for a fixed-step ODE solver")
     i.add_argument("spec", nargs="?", default="verify_solver.py", help="spec file to write (default verify_solver.py)")
     i.add_argument("--solver", default="solver", help="module that holds the solver, importable from the spec's folder (default solver)")

physverify-0.4.2/src/physverify/demo.py

@@ -0,0 +1,75 @@
+"""`physverify demo`: the damped-oscillator example, shipped in the package.
+
+It runs the checks on a correct midpoint solver, then on a copy whose damping
+is 10% too large, so a first run needs only `pip install physverify`.
+The same solver and spec, as files to copy, are in examples/ in the repository.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+from .studies import ode_checks
+from .mms import ManufacturedODE
+
+OMEGA, GAMMA = 3.0, 0.4
+
+
+def make_solver(damping_factor: float = 1.0):
+    """Midpoint method for x'' = -omega^2 x - 2 gamma x', with an additive source term."""
+
+    def rhs(t, y):
+        damping = 2.0 * GAMMA * damping_factor
+        return np.array([y[1], -(OMEGA**2) * y[0] - damping * y[1]])
+
+    def integrate(source, y0, t_end, n_steps, trajectory=False):
+        h = t_end / n_steps
+        y = np.array(y0, dtype=float)
+        states = [y]
+        for i in range(n_steps):
+            t = i * h
+            k1 = rhs(t, y) + source(t)
+            mid = y + 0.5 * h * k1
+            y = y + h * (rhs(t + 0.5 * h, mid) + source(t + 0.5 * h))
+            states.append(y)
+        return np.array(states) if trajectory else y
+
+    return integrate
+
+
+def reference_rhs(t, y):
+    """The equations, written from the physics, not copied from the solver."""
+    return np.array([y[1], -(OMEGA**2) * y[0] - 2.0 * GAMMA * y[1]])
+
+
+def checks(damping_factor: float = 1.0):
+    integrate = make_solver(damping_factor)
+    mms = ManufacturedODE(lambda t: np.array([np.sin(1.3 * t) + 0.2, 1.04 * np.cos(1.3 * t)]), reference_rhs)
+    return ode_checks(
+        "midpoint damped oscillator",
+        integrate,
+        rhs_ref=reference_rhs,
+        mms=mms,
+        y0_free=np.array([1.0, 0.0]),
+        t_end=1.0,
+        ns=(10, 20, 40, 80, 160),
+        order=2.0,
+        patch=(np.array([0.5, -0.3]), np.array([0.8, 1.1])),
+        integrate_trajectory=lambda s, y0, t, n: integrate(s, y0, t, n, trajectory=True),
+    )
+
+
+def main() -> int:
+    """Exit 0 when the correct solver passes and the broken one fails, as they should."""
+    from .report import NOTE, run_checks, to_text
+
+    print("1. A correct midpoint solver for a damped oscillator:\n")
+    good = run_checks(checks())
+    print(to_text(good))
+    print("\n2. The same solver with the damping 10% too large (a typical slip):\n")
+    bad = run_checks(checks(damping_factor=1.1))
+    print(to_text(bad))
+    ok = not any(o.result.failed for o in good) and any(o.result.failed for o in bad)
+    print(f"\n{NOTE}")
+    print("\nTo check your own solver, start a spec with:  physverify init verify_solver.py --solver mysolver --order 2")
+    return 0 if ok else 1

{physverify-0.4.1 → physverify-0.4.2}/src/physverify/matrix.py

@@ -2,17 +2,23 @@
 
     python -m physverify.matrix results/detection_matrix.md
 
+This also writes detection_matrix.json (every check's full status, message and
+run time, with how and where it was made) and run_time.md next to it.
+
 The bugs were written by the author of this package, so the catch rate below is
 a measure of these bugs, not of bugs in general.
 """
 
 from __future__ import annotations
 
+import json
 import sys
+import time
+from collections import Counter
 from dataclasses import dataclass, field
 from pathlib import Path
 
-from .checks import FAMILIES, Check, Result, run_check
+from .checks import FAIL, FAMILIES, Check, Result, run_check
 from .problems import heat1d, kepler, pendulum, projectile, sod
 
 PROBLEMS = {m.NAME: m for m in (projectile, pendulum, kepler, heat1d, sod)}
@@ -23,6 +29,7 @@ class Row:
     problem: str
     bug: str | None  # None is the bug-free baseline
     results: list = field(default_factory=list)  # list of (Check, Result)
+    seconds: list = field(default_factory=list)  # run time of each check, when measured
 
     def flagged(self) -> dict:
         """family -> labels of the checks that did not pass"""
@@ -32,6 +39,10 @@ class Row:
                 out.setdefault(check.family, []).append(check.label)
         return out
 
+    def flagged_by_fail(self) -> bool:
+        """At least one check gave FAIL, a clean detection (not only INCONCLUSIVE or ERROR)."""
+        return any(result.status == FAIL for _, result in self.results)
+
     def families_present(self) -> set:
         return {check.family for check, _ in self.results}
 
@@ -40,14 +51,23 @@ def evaluate(module, bugs=frozenset()) -> list:
     return [(check, run_check(check)) for check in module.checks(bugs)]
 
 
+def evaluate_timed(problem, bug) -> Row:
+    row = Row(problem, bug)
+    for check in PROBLEMS[problem].checks(frozenset([bug]) if bug else frozenset()):
+        start = time.perf_counter()
+        row.results.append((check, run_check(check)))
+        row.seconds.append(time.perf_counter() - start)
+    return row
+
+
 def build_rows(problems=None) -> list:
     rows = []
     for name, module in PROBLEMS.items():
         if problems and name not in problems:
             continue
-        rows.append(Row(name, None, evaluate(module)))
+        rows.append(evaluate_timed(name, None))
         for bug in module.BUGS:
-            rows.append(Row(name, bug, evaluate(module, frozenset([bug]))))
+            rows.append(evaluate_timed(name, bug))
     return rows
 
 
@@ -74,6 +94,7 @@ def to_markdown(rows) -> str:
     caught = [r for r in bug_rows if r.flagged()]
     missed = [r for r in bug_rows if not r.flagged()]
     false_alarms = [r for r in rows if r.bug is None and r.flagged()]
+    by_fail = [r for r in caught if r.flagged_by_fail()]
     by_family = {f: sum(1 for r in bug_rows if f in r.flagged()) for f in FAMILIES}
     only_one = {f: sum(1 for r in bug_rows if set(r.flagged()) == {f}) for f in FAMILIES}
     summary = [
@@ -81,6 +102,11 @@ def to_markdown(rows) -> str:
         f"Bugs seeded: {len(bug_rows)}. Flagged by at least one check: {len(caught)}. Missed by all: {len(missed)}"
         + (" (" + ", ".join(f"{r.problem}/{r.bug}" for r in missed) + ")" if missed else "")
         + ".",
+        f"Of the {len(caught)} flagged: {len(by_fail)} by at least one FAIL, {len(caught) - len(by_fail)} only by"
+        " INCONCLUSIVE or ERROR"
+        + (" (" + ", ".join(f"{r.problem}/{r.bug}" for r in caught if not r.flagged_by_fail()) + ")"
+           if len(by_fail) < len(caught) else "")
+        + ".",
         f"Bug-free baselines with a false alarm: {len(false_alarms)} of {len(rows) - len(bug_rows)}.",
         "Bugs flagged per family: " + ", ".join(f"{f} {n}" for f, n in by_family.items()) + ".",
         "Bugs flagged by that family alone: " + ", ".join(f"{f} {n}" for f, n in only_one.items()) + ".",
@@ -88,6 +114,73 @@ def to_markdown(rows) -> str:
     return head + "\n".join(lines) + "\n" + "\n".join(summary) + "\n"
 
 
+def provenance() -> dict:
+    """How and where a result file was made; kept out of the Markdown so the committed tables stay comparable."""
+    import datetime
+    import platform
+    import subprocess
+
+    import numpy
+
+    from . import __version__
+
+    try:
+        commit = subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True, timeout=10,
+                                cwd=Path(__file__).resolve().parent).stdout.strip() or None
+    except (OSError, subprocess.SubprocessError):
+        commit = None
+    return {
+        "physverify": __version__,
+        "git_commit": commit,
+        "python": platform.python_version(),
+        "numpy": numpy.__version__,
+        "platform": platform.platform(),
+        "date": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
+    }
+
+
+def to_json(rows) -> str:
+    """One record per (problem, bug, check) with the full status, message, prediction and run time."""
+    records = []
+    for row in rows:
+        predicted = sorted(PROBLEMS[row.problem].BUGS[row.bug].expected) if row.bug else []
+        seconds = row.seconds or [None] * len(row.results)
+        for (check, result), sec in zip(row.results, seconds):
+            records.append({
+                "problem": row.problem,
+                "bug": row.bug,
+                "family": check.family,
+                "check": check.label,
+                "status": result.status,
+                "message": result.message,
+                "predicted_to_flag": check.family in predicted,
+                "seconds": None if sec is None else round(sec, 4),
+            })
+    return json.dumps({"provenance": provenance(), "records": records}, indent=1) + "\n"
+
+
+def run_time(rows) -> str:
+    """Total run time per check family and problem, over the baseline and every bug (not deterministic)."""
+    totals: dict = {}
+    for row in rows:
+        for (check, _), sec in zip(row.results, row.seconds):
+            totals.setdefault(row.problem, {}).setdefault(check.family, 0.0)
+            totals[row.problem][check.family] += sec
+    lines = [
+        "# Run time per check family",
+        "",
+        "Seconds, summed over the baseline and every seeded bug of each problem, on the machine recorded in",
+        "detection_matrix.json (`provenance`). Generated by `python scripts/reproduce_all.py`; it varies between runs.",
+        "",
+        "| Problem | " + " | ".join(FAMILIES) + " | Total |",
+        "|---|" + "---|" * (len(FAMILIES) + 1),
+    ]
+    for problem, fam in totals.items():
+        cells = " | ".join(f"{fam[f]:.2f}" if f in fam else "n/a" for f in FAMILIES)
+        lines.append(f"| {problem} | {cells} | {sum(fam.values()):.2f} |")
+    return "\n".join(lines) + "\n"
+
+
 def details(rows) -> str:
     """Per-check messages, for reading why a cell is what it is."""
     out = []
@@ -99,13 +192,28 @@ def details(rows) -> str:
     return "\n".join(out)
 
 
+REPO = "https://github.com/rcgiri-physics/physverify/blob/main/"
+
+
+def bug_switches(problem: str, bug: str) -> list:
+    """(path, line number, code) of every line that switches the bug on: `"bug" in bugs`."""
+    root = Path(__file__).resolve().parents[2]
+    files = [Path(PROBLEMS[problem].__file__), Path(__file__).resolve().parent / "problems" / "_common.py"]
+    out = []
+    for f in files:
+        for n, line in enumerate(f.read_text(encoding="utf-8").splitlines(), 1):
+            if f'"{bug}" in bugs' in line:
+                out.append((f.resolve().relative_to(root).as_posix(), n, line.strip()))
+    return out
+
+
 def catalogue(rows) -> str:
     """One entry per seeded bug: what is changed, the expected symptom, predicted and observed families."""
     out = [
         "# Seeded-bug catalogue",
         "",
         "Generated by `python -m physverify.matrix results/detection_matrix.md`. Each bug is switched on by",
-        "name in the problem's solver (search the source for the bug's name to see the exact change).",
+        "name in the problem's solver; the lines that switch it on are linked under each bug.",
         "Predictions and symptoms were written from reasoning before the checks were run.",
         "",
     ]
@@ -121,8 +229,14 @@ def catalogue(rows) -> str:
             f"- Expected symptom: {info.symptom or '(not recorded)'}",
             f"- Predicted to be flagged by: {', '.join(sorted(info.expected)) or 'nothing (blind spot)'}",
             f"- Flagged by: {', '.join(flagged) or 'nothing'}",
-            "",
+            "- Outcome of its " + str(len(row.results)) + " checks: " + ", ".join(
+                f"{n} {s}" for s, n in sorted(Counter(r.status for _, r in row.results).items())),
         ]
+        switches = bug_switches(row.problem, row.bug)
+        if switches:
+            out.append("- Where it is switched on:")
+            out += [f"    - [{path}:{n}]({REPO}{path}#L{n}): `{code}`" for path, n, code in switches]
+        out.append("")
     return "\n".join(out)
 
 
@@ -153,6 +267,8 @@ def main(argv=None) -> int:
         path.with_name(path.stem + "_details.md").write_text(details(rows), encoding="utf-8")
         path.with_name("heat1d_by_check.md").write_text(by_check_markdown(rows, "heat1d"), encoding="utf-8")
         path.with_name("bug_catalogue.md").write_text(catalogue(rows), encoding="utf-8")
+        path.with_suffix(".json").write_text(to_json(rows), encoding="utf-8")
+        path.with_name("run_time.md").write_text(run_time(rows), encoding="utf-8")
     return 0
 
 

{physverify-0.4.1 → physverify-0.4.2}/src/physverify/order.py

@@ -54,7 +54,8 @@ def check_exact(hs, errors, *, floor: float = 1e-12, scale: float = 1.0) -> Orde
     errs = np.asarray(errors, dtype=float)
     order = np.argsort(-hs)
     hs, errs = hs[order], errs[order]
-    base = dict(expected=float("inf"), hs=tuple(hs), errors=tuple(errs))
+    base = dict(expected=float("inf"), hs=tuple(hs), errors=tuple(errs),
+                acceptance=f"every error at most {floor:.2g} (round-off)")
     if not np.all(np.isfinite(errs)):
         return OrderResult(FAIL, "non-finite error: the solution diverged or produced NaN", **base)
     worst = float(errs.max())
@@ -100,9 +101,19 @@ def check_order(
     floor = effective_floor(floor, scale)
     hs = np.asarray(hs, dtype=float)
     errs = np.asarray(errors, dtype=float)
+    if hs.shape != errs.shape:
+        return OrderResult(INCONCLUSIVE, f"{hs.size} step sizes but {errs.size} errors; they must match",
+                           expected=float(expected), hs=tuple(hs), errors=tuple(errs))
     order = np.argsort(-hs)  # coarse to fine
     hs, errs = hs[order], errs[order]
-    base = dict(expected=float(expected), hs=tuple(hs), errors=tuple(errs))
+    base = dict(expected=float(expected), hs=tuple(hs), errors=tuple(errs),
+                acceptance=f"fitted order within [{expected - tol_low:g}, {expected + tol_high:g}] on errors above {floor:.2g}")
+
+    if not np.all(np.isfinite(hs) & (hs > 0)):
+        return OrderResult(INCONCLUSIVE, "step sizes must be positive and finite; check how h is computed", **base)
+    if len(np.unique(hs)) < 2:
+        return OrderResult(INCONCLUSIVE, "all step sizes are the same, so no order can be measured; use at least two"
+                           " different resolutions", **base)
 
     if not np.all(np.isfinite(errs)):
         return OrderResult(FAIL, "non-finite error: the solution diverged or produced NaN", **base)

{physverify-0.4.1 → physverify-0.4.2}/src/physverify/report.py

@@ -54,6 +54,17 @@ def load_spec(path):
     return checks
 
 
+SCHEMA_VERSION = 1  # of the --json output; see docs/api.md
+NOTE = "A PASS means these checks found nothing wrong; it does not prove the simulation correct."
+CLAIMS = {  # what each family tests, for the JSON output
+    "mms": "the code converges at the claimed order to a manufactured solution of the reference equations",
+    "reference": "the code converges at the claimed order to an independently computed reference solution",
+    "invariant": "the drift of a conserved quantity shrinks as the resolution is refined",
+    "limit": "the code reproduces a known limiting or special case",
+    "patch": "the code reproduces a solution it should represent exactly, to round-off",
+}
+
+
 def _number(x):
     x = float(x)
     return x if math.isfinite(x) else None
@@ -67,12 +78,15 @@ def outcome_to_dict(o: Outcome) -> dict:
         "ok": o.result.ok,
         "message": o.result.message,
         "seconds": round(o.seconds, 4),
+        "claim": CLAIMS.get(o.check.family, ""),
     }
     if isinstance(o.result, OrderResult):
         d["expected_order"] = _number(o.result.expected)
         d["observed_order"] = _number(o.result.fitted)
         d["h"] = [_number(h) for h in o.result.hs]
         d["error"] = [_number(e) for e in o.result.errors]
+        if o.result.acceptance:
+            d["acceptance"] = o.result.acceptance
     return d
 
 
@@ -90,10 +104,12 @@ def to_json(outcomes, source: str = "") -> str:
     from . import __version__
 
     payload = {
+        "schema_version": SCHEMA_VERSION,
         "physverify": __version__,
         "source": source,
         "summary": summary(outcomes),
         "checks": [outcome_to_dict(o) for o in outcomes],
+        "note": NOTE,
     }
     return json.dumps(payload, indent=2)
 
@@ -101,7 +117,8 @@ def to_json(outcomes, source: str = "") -> str:
 def to_text(outcomes) -> str:
     lines = [f"{o.check.family:10s} {o.result.status:12s} {o.check.label}: {o.result.message}" for o in outcomes]
     s = summary(outcomes)
-    lines.append(f"{s['ok']} of {s['total']} checks passed")
+    skipped = f", {s['skipped']} skipped" if s["skipped"] else ""
+    lines.append(f"{s['ok']} of {s['total']} checks passed{skipped}")
     return "\n".join(lines)
 
 
@@ -125,6 +142,7 @@ def to_markdown(outcomes, source: str = "") -> str:
         lines.append(
             f"| {cell(o.check.family)} | {cell(o.result.status)} | {cell(o.check.label)} | {observed} | {cell(o.result.message)} |"
         )
+    lines += ["", f"_{NOTE}_"]
     return "\n".join(lines) + "\n"
 
 
@@ -210,5 +228,5 @@ def to_html(outcomes, title: str = "physverify report", source: str = "") -> str
         '<div class="wrap"><table><thead><tr><th>Status</th><th>Family</th><th>Check</th><th>Message</th>'
         "<th>Observed order</th><th>Convergence</th></tr></thead><tbody>"
         + "".join(rows)
-        + "</tbody></table></div></body></html>\n"
+        + f'</tbody></table></div><p class="meta">{html.escape(NOTE)}</p></body></html>\n'
     )

{physverify-0.4.1 → physverify-0.4.2}/src/physverify/scaffold.py

@@ -24,6 +24,14 @@ taking n_steps fixed steps from t = 0 and adding source(t), an array of
 length {dim}, to the right-hand side wherever the solver evaluates it. If the
 solver has no such hook, write a small wrapper in this file that adds one and
 pass the wrapper to ode_checks instead.
+
+Before trusting a PASS:
+- reference_rhs written from the physics, not copied from the solver?
+- units and signs stated, and matching the solver's conventions?
+- initial and boundary conditions checked separately?
+- the test reaches the states where bugs hide (large amplitudes, boundaries)?
+- a second, independent check (a conserved quantity or a known limit)?
+See https://rcgiri-physics.github.io/physverify/assumptions/
 """
 
 import numpy as np