dynamic-systems-testing-framework 1.0.1__tar.gz

dynamic_systems_testing_framework-1.0.1/.gitignore

@@ -0,0 +1,89 @@
+# Python
+__pycache__/
+*.py[codz]
+*.egg-info/
+dist/
+build/
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+.env
+
+# uv
+# uv.lock is committed for reproducibility
+
+# Dymola artifacts
+*.mat
+dslog.txt
+dsin.txt
+dsfinal.txt
+buildlog.txt
+dymosim
+dymosim.exe
+dymosim.exp
+dymosim.lib
+dymosim.dll
+dsmodel.c
+dsmodel_att.h
+dymola.log
+status
+success
+failure
+*.bif-*
+
+# Keep test fixture results (override above exclusions)
+!tests/fixtures/results/**/*.mat
+!tests/fixtures/results/**/dslog.txt
+!tests/fixtures/results/**/dsin.txt
+!tests/fixtures/results/**/dsfinal.txt
+!tests/fixtures/results_openmodelica/**/*.mat
+
+# Simulation output (generated by the tool)
+testing_output/
+
+# Generated reports (dashboard.html, interactive plots, comparison_data.json)
+reports/
+
+# Reference-FMUs binaries (fetched by scripts/fetch_reference_fmus.py)
+examples/fmu/reference-fmus-binaries/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Coverage / testing
+.coverage
+.coverage.*
+htmlcov/
+.pytest_cache/
+.tox/
+.nox/
+
+# Translations (gettext .mo — NOT Modelica .mo files)
+*.pot
+
+# Cursor
+.cursorignore
+.cursorindexingignore
+# Local agent configuration and working notes. Machine-local: the paths
+# inside resolve only on the machine that wrote them.
+.claude/
+
+# Test fixtures named like Dymola artifacts are real, committed test data —
+# the artifact patterns above (dslog.txt, *.mat, ...) must not swallow them.
+# (results/Dymola/dslog.txt + dsres.mat were already force-added; this keeps
+# the results_additional/ dslog fixture tracked too. Big generated fixtures —
+# dsfinal.txt, dsin.txt, dsmodel.c — stay ignored: no test references them.)
+!tests/fixtures/**/dslog.txt
+
+# Local working-session notes and agent state — never published.
+# The project's durable record is docs/decisions.md and docs/ideas.md.
+.soul/

dynamic_systems_testing_framework-1.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ORNL-Modelica
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dynamic_systems_testing_framework-1.0.1/PKG-INFO

@@ -0,0 +1,266 @@
+Metadata-Version: 2.5
+Name: dynamic-systems-testing-framework
+Version: 1.0.1
+Summary: Dynamic Systems Testing Framework — regression & unit testing for time-dependent system behavior across Modelica (Dymola, OpenModelica), FMU (FMPy), Julia (ModelingToolkit), and Python backends
+Project-URL: Homepage, https://github.com/ORNL-Modelica/DynamicSystemsTestingFramework
+Project-URL: Repository, https://github.com/ORNL-Modelica/DynamicSystemsTestingFramework
+Project-URL: Documentation, https://github.com/ORNL-Modelica/DynamicSystemsTestingFramework/blob/main/docs/usage.md
+Project-URL: Issues, https://github.com/ORNL-Modelica/DynamicSystemsTestingFramework/issues
+Author: Scott Greenwood
+License-Expression: MIT
+License-File: LICENSE
+Keywords: dymola,fmi,fmu,julia,modelica,modelingtoolkit,openmodelica,regression-testing,simulation,testing
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: jinja2>=3.1
+Requires-Dist: numpy>=1.20
+Requires-Dist: psutil>=5.9
+Provides-Extra: author
+Requires-Dist: nicegui>=3.16; extra == 'author'
+Requires-Dist: regex>=2026.9.10; extra == 'author'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.16; extra == 'dev'
+Requires-Dist: scipy>=1.10; extra == 'dev'
+Requires-Dist: types-regex>=2026.9.10; extra == 'dev'
+Provides-Extra: fmpy
+Requires-Dist: fmpy>=0.3; extra == 'fmpy'
+Provides-Extra: om
+Requires-Dist: ompython>=3.5; extra == 'om'
+Description-Content-Type: text/markdown
+
+# Dynamic Systems Testing Framework (DSTF)
+
+[![quality](https://github.com/ORNL-Modelica/DynamicSystemsTestingFramework/actions/workflows/quality.yml/badge.svg)](https://github.com/ORNL-Modelica/DynamicSystemsTestingFramework/actions/workflows/quality.yml)
+
+Regression and unit testing for time-dependent system behavior. Discovers tests, runs simulations, compares against versioned baselines, and reports — across **Modelica (Dymola, OpenModelica), FMU (FMPy), Julia/ModelingToolkit, and arbitrary Python**.
+
+Library-agnostic: tests can be discovered from in-model annotations (`UnitTests` components), declarative `test_spec.json`, or both. Reference results are partitioned per backend / OS so the same library can be regression-tested across backends without ref drift.
+
+Criteria are written in **`dstf author`**, a local browser tool over the same figures the report draws: edit a criterion beside the plot it scores — drag a handle, or Ctrl+click to add a row that settles on the data — give a tolerance a lower and an upper side, put one criterion on many elements of an array, queue tests on simulator workers that stay alive, and watch a running test's trace as Dymola or OpenModelica writes it. A default and a dark theme cover the tool, the dashboard and the report pages.
+
+![DSTF dashboard](https://raw.githubusercontent.com/ORNL-Modelica/DynamicSystemsTestingFramework/main/docs/img/dashboard.png)
+
+*The unified dashboard — live progress during a run, a static report after. Each report is stamped with the backend, tool + MSL version, and OS that produced it; the table gives per-test verdicts, worst-NRMSE, and phase timing.*
+
+## What's in the box
+
+- **5 backends**, each with persistent-worker support where it pays off:
+
+  | Backend | Source kind | Persistent | Notes |
+  |---|---|---|---|
+  | Dymola | Modelica `.mo` | ✓ default | Python interface; `--batch` selects the `.mos` path. Records translation statistics, so a change in a model's state count is a structural warning |
+  | OpenModelica | Modelica `.mo` | ✓ default | OMPython ZMQ; `--batch` selects the `omc -s` path. Records timing statistics and the model's state count (from the init file omc generates), so a change in it is the structural warning it is on Dymola — against a baseline that records the count, which one accepted before this does at its next accept; system sizes and Jacobian counts are Dymola's alone |
+  | FMPy | Pre-built FMU | per-test thread | `fmpy.simulate_fmu` in-process |
+  | Julia / MTK | `.jl` ModelingToolkit | ✓ default | stdin-JSON pipe; Dyad untested |
+  | Python | Any `simulate(stop_time, tol) -> dict` | subprocess-per-test | scipy, pandas, CSV loader, HTTP — anything |
+
+- **7 comparison modes**, composable via a `MetricTree`:
+
+  | Mode | What it scores |
+  |---|---|
+  | `nrmse` | RMSE over signal range — the default |
+  | `tube` | Pass if signal stays inside a (constant or time-varying) envelope around ref |
+  | `points` | Per-point checks at declared times with `time_tolerance` x-box and abs/rel y-tolerance |
+  | `range` | Scalar `min_value` / `max_value` bounds — baseline-free |
+  | `event-timing` | Match Modelica solver events against a declared list within per-event tolerance |
+  | `dominant-frequency` | FFT peak matching against a declared peaks table — windowable |
+  | `scalar` | One number — a reduction of the signal (`max`/`min`/`mean`/`sample_mean`/`final`/`integral`) or a stored statistic by dotted path — vs a baseline scalar or declared bounds |
+
+  Combinators: `and`, `or`, `k-of-n`, `warn` (relax to advisory), `weighted`. Leaves can scope to a time window and target named baselines (`primary`, soft-checks, companions).
+
+- **Read-only reports** — per-test Plotly HTML drawn from figures Python built: the trajectory with its criterion geometry (tube band, range bounds, point boxes, event windows, FFT spectrum), overlay picker, error overlays, and the metric tree with the CLI's verdict and score text per leaf. Plus a live dashboard during runs and JUnit XML for CI. The report shows, the CLI decides.
+
+- **An authoring tool** — `dstf author` (the `[author]` extra) serves a Suite page and a Test page on `127.0.0.1`. Criteria are edited inline under the plot they score, staged, and written to `test_spec.json` on Save:
+
+  | Feature | What it does |
+  |---|---|
+  | Two-sided tolerance | A tolerance is one number or `{"lower": a, "upper": b}`; on the plot, an arrow outside each side sets it |
+  | On-plot editing | Drag a ring, an arrow or a bound line; Ctrl+click adds a row that settles on the data; a field's ⌖ arms it for the next click; a placed value is snapped to a readable step |
+  | Simulation queue | Queue the selected tests: first in, first out, on workers that stay alive between runs; Remove a waiting test, Cancel a running one |
+  | Live results | A running test's trace is drawn as Dymola or OpenModelica writes its result file; a cancelled or timed-out run is shown, never scored or accepted |
+  | Criteria over an array | One criterion on the elements `1:5:20` of an array; identical criteria fold into one row, edited as one |
+  | Themes | `"theme": "dark"` in `testing.json` for the dashboard, the report pages and the tool; the tool's ⋮ menu switches a browser tab live |
+
+- **Three baseline roles** per test, partitioned on disk: **primary** (regression anchor, hard-fail), **soft_checks** (warn-wrapped cross-regression imports), **companions** (plot-only overlays, including cross-library).
+
+## Install
+
+End users:
+
+```bash
+# Not on PyPI yet — these work from the first published release.
+uv tool install dynamic-systems-testing-framework   # then run plain: dstf ...
+# or: pipx install dynamic-systems-testing-framework
+```
+
+Developers:
+
+```bash
+uv sync --extra dev --locked                             # core + pytest, from uv.lock
+uv sync --extra dev --extra author --extra fmpy --extra om --locked   # plus the authoring tool, FMPy + OpenModelica extras
+uv run dstf --help
+```
+
+`uv sync` installs the project editable and pins every dependency to `uv.lock`,
+the same lock CI uses; `--locked` fails instead of silently re-locking when the
+lock has drifted from `pyproject.toml`. It prunes anything you did not ask for
+on that command line — being in the lock does not save it — so name every extra
+at once.
+
+For the Julia backend, install Julia 1.11+ and instantiate the demo project once:
+
+```bash
+curl -fsSL https://install.julialang.org | sh -s -- -y --default-channel 1.11
+cd examples/julia/JuliaMtkTestingLib && julia --project=. -e 'using Pkg; Pkg.instantiate()'
+```
+
+The Python examples (`SimpleRamp`, `PointsCheck`) need scipy, which the `dev`
+extra installs. Reach them through `uv run dstf ...`: the Python backend takes
+the interpreter from `testing.json`'s `backends: {"Python": [...]}` entry and
+falls back to `PATH` — and the shipped example names `python`, so either route
+ends at the first `python` on `PATH`, which `uv run` points at `.venv`. Run the
+same examples from a tool-installed `dstf` and that is whatever comes first on
+your PATH — a conda env, often — which needs its own scipy.
+
+## First 5 minutes
+
+The repo ships demo libraries for every backend. Easiest first run is the Modelica demo:
+
+```bash
+# From the repo root:
+uv run dstf --config examples/modelica/ModelicaTestingLib/Resources/ReferenceResults/testing.json run
+```
+
+What this does:
+
+1. Reads `testing.json` to find the demo library, the backend (Dymola or OpenModelica — first one whose binary exists wins), the reference root, and any per-test recognizers.
+2. Discovers the demo tests via the bundled `UnitTests` recognizer plus declarative recognizers in the JSON.
+3. Runs each in a persistent simulator worker, partitioning output under `testing_output/` by backend + OS.
+4. Compares against `Resources/ReferenceResults/<Backend>/<os>/ref_NNNN.json` baselines.
+5. Prints a console report with pass/fail per test.
+
+To see the interactive HTML reporter:
+
+```bash
+uv run dstf --config examples/modelica/ModelicaTestingLib/Resources/ReferenceResults/testing.json run --report
+```
+
+The dashboard URL is printed as a run starts (`Live progress: file://…`) and the page is opened in a browser tab; it reloads itself every 5s during the run. `--report` writes the per-test pages under `<work_dir>/reports/` and opens the dashboard again when they are written. `--no-browser` (on `run` and `compare`) opens no tab and still prints the link.
+
+The examples ship baselines for these `<Backend>/<os>` partitions: Modelica — `Dymola/linux`, `Dymola/windows` and `OpenModelica/linux` (its array model, `HeatRodTest`, on the two Linux ones); FMU — `FMPy/linux` and `FMPy/windows`; Julia — `Julia/linux`; Python — `Python/linux`. A first run on an OS a backend has no partition for (OpenModelica, Julia or Python on Windows, say) reports every test `NO_REF` until `run --accept` stores its baselines there. A companion in the examples is another source's curve of the SAME model, drawn over the test's plot when its box is ticked on the report page, and each ships in the partitions named here and no other: in `OpenModelica/linux`, the Modelica `FrequencyTest`, `RangeCheckTest` and `MultiFrequencyTest` carry the Julia example's baseline of that model; in `Julia/linux`, the Julia `Frequency`, `RangeCheck` and `MultiFrequency` carry the OpenModelica one; and `MetricTreeTest` carries its closed-form solution in `Dymola/linux` and `OpenModelica/linux`. A Dymola run of the example therefore shows no Julia companion, and `Dymola/windows` ships no companion at all.
+
+To repeat with another backend's demo library, swap the config path:
+
+```bash
+# Julia/MTK, Python, and prebuilt FMUs through FMPy
+uv run dstf --config examples/julia/JuliaMtkTestingLib/Resources/ReferenceResults/testing.json run
+uv run dstf --config examples/python/PythonTestingLib/Resources/ReferenceResults/testing.json run
+uv run dstf --config examples/fmu/testing.json run    # needs reference-fmus-binaries/
+```
+
+## Configuration cheat-sheet
+
+`testing.json` is the single entry point. The full field reference lives in `docs/usage.md`; the headline shape:
+
+```json
+{
+  "source_path": "../../MyLib",
+  "backends": {
+    "Dymola":       ["C:\\Program Files\\Dymola 2026x\\bin64\\Dymola.exe"],
+    "OpenModelica": ["/usr/bin/omc"]
+  },
+  "test_spec": "test_spec.json",
+  "dependencies": ["/path/to/SomeDep"],
+  "tolerance": 1e-4,
+  "diagnostic_variables": ["CPUtime", "EventCounter"]
+}
+```
+
+The first `backends` entry whose binary exists wins (no need for per-OS config files). `backend: "Dymola"` (or any named entry) forces a specific one.
+
+## Common commands
+
+```bash
+# Discover tests without running
+uv run dstf --config testing.json discover
+
+# Run, compare, accept all current results as new baselines
+uv run dstf --config testing.json run --accept
+
+# Interactive review by category
+uv run dstf --config testing.json run -i failed
+
+# Filter (glob, comma list, or @file)
+uv run dstf --config testing.json run --filter "Foo.A,Foo.B"
+uv run dstf --config testing.json run --filter @rerun.txt
+
+# Parallel + interactive HTML report (--batch-size N splits a Dymola --batch run into small batches)
+uv run dstf --config testing.json run --parallel 4 --report
+
+# Rerun only previously-failed tests, merge with last full run, regen report
+uv run dstf --config testing.json run --rerun failed,sim-failed --merge --report
+
+# Compare without re-simulating (uses last results)
+uv run dstf --config testing.json compare
+
+# Open the authoring tool in a browser (needs the [author] extra)
+uv run dstf --config testing.json author
+
+# Apply an RFC 6902 JSON-Patch envelope against test_spec.json — single- or
+# multi-model form, from any source
+uv run dstf --config testing.json spec-update spec_patch.json
+
+# Manage the other baseline roles
+uv run dstf --config testing.json companion add <model> <name> <path>
+uv run dstf --config testing.json soft-check list <model>
+uv run dstf --config testing.json import-baseline <model> <name> <source>   # soft_checks only
+
+# Manifest / schema
+uv run dstf --config testing.json manifest dump
+uv run dstf --config testing.json manifest cleanup [--orphans] [--apply]   # dry-run by default
+uv run dstf --config testing.json export-schema --output schema.json
+
+# Diagnose Dymola Python interface discovery
+uv run dstf check-dymola
+```
+
+`uv run dstf <command> --help` for each subcommand's full flag set.
+
+## Where things live
+
+| Doc | What's in it |
+|---|---|
+| `docs/usage.md` | The deep user guide — every command, every config field, every comparison mode |
+| `docs/architecture.md` | Six-layer plug-in pipeline + current-state map to code |
+| `docs/extensibility.md` | Plug-in contracts for Source / Discovery / Backend / Dataset / Metric / Combinator |
+| `docs/vision.md` | Where the framework is going (target user base, non-Modelica futures) |
+| `docs/decisions.md` | Authoritative decision log |
+| `docs/ideas.md` | Backlog (unimplemented features, with explicit deferred-by-design entries) |
+| `examples/{modelica,fmu,julia,python}/` | Working demo libraries — one per backend |
+
+## CI integration
+
+```yaml
+- run: |
+    uv run dstf --config ./testing.json run --report-format junit --output test-results.xml
+- uses: actions/upload-artifact@v4
+  with:
+    name: test-results
+    path: test-results.xml
+```
+
+For HTML reports as build artifacts, also pass `--report` and upload `dashboard.html` and the `reports/` directory from the work dir (`--work-dir` sets it).
+
+## Status
+
+v1.0. 5 backends in production, 4 demo libraries, and a comprehensive pytest suite (a Playwright subset drives the report pages and the authoring tool in a real browser). The report is read-only and mutates nothing: every number it draws comes from a figure Python built or a string the server rendered. Criteria are authored in `dstf author`, driven by the same figures and the same scorer as the CLI. Nothing is published to PyPI yet.

dynamic_systems_testing_framework-1.0.1/README.md

@@ -0,0 +1,226 @@
+# Dynamic Systems Testing Framework (DSTF)
+
+[![quality](https://github.com/ORNL-Modelica/DynamicSystemsTestingFramework/actions/workflows/quality.yml/badge.svg)](https://github.com/ORNL-Modelica/DynamicSystemsTestingFramework/actions/workflows/quality.yml)
+
+Regression and unit testing for time-dependent system behavior. Discovers tests, runs simulations, compares against versioned baselines, and reports — across **Modelica (Dymola, OpenModelica), FMU (FMPy), Julia/ModelingToolkit, and arbitrary Python**.
+
+Library-agnostic: tests can be discovered from in-model annotations (`UnitTests` components), declarative `test_spec.json`, or both. Reference results are partitioned per backend / OS so the same library can be regression-tested across backends without ref drift.
+
+Criteria are written in **`dstf author`**, a local browser tool over the same figures the report draws: edit a criterion beside the plot it scores — drag a handle, or Ctrl+click to add a row that settles on the data — give a tolerance a lower and an upper side, put one criterion on many elements of an array, queue tests on simulator workers that stay alive, and watch a running test's trace as Dymola or OpenModelica writes it. A default and a dark theme cover the tool, the dashboard and the report pages.
+
+![DSTF dashboard](https://raw.githubusercontent.com/ORNL-Modelica/DynamicSystemsTestingFramework/main/docs/img/dashboard.png)
+
+*The unified dashboard — live progress during a run, a static report after. Each report is stamped with the backend, tool + MSL version, and OS that produced it; the table gives per-test verdicts, worst-NRMSE, and phase timing.*
+
+## What's in the box
+
+- **5 backends**, each with persistent-worker support where it pays off:
+
+  | Backend | Source kind | Persistent | Notes |
+  |---|---|---|---|
+  | Dymola | Modelica `.mo` | ✓ default | Python interface; `--batch` selects the `.mos` path. Records translation statistics, so a change in a model's state count is a structural warning |
+  | OpenModelica | Modelica `.mo` | ✓ default | OMPython ZMQ; `--batch` selects the `omc -s` path. Records timing statistics and the model's state count (from the init file omc generates), so a change in it is the structural warning it is on Dymola — against a baseline that records the count, which one accepted before this does at its next accept; system sizes and Jacobian counts are Dymola's alone |
+  | FMPy | Pre-built FMU | per-test thread | `fmpy.simulate_fmu` in-process |
+  | Julia / MTK | `.jl` ModelingToolkit | ✓ default | stdin-JSON pipe; Dyad untested |
+  | Python | Any `simulate(stop_time, tol) -> dict` | subprocess-per-test | scipy, pandas, CSV loader, HTTP — anything |
+
+- **7 comparison modes**, composable via a `MetricTree`:
+
+  | Mode | What it scores |
+  |---|---|
+  | `nrmse` | RMSE over signal range — the default |
+  | `tube` | Pass if signal stays inside a (constant or time-varying) envelope around ref |
+  | `points` | Per-point checks at declared times with `time_tolerance` x-box and abs/rel y-tolerance |
+  | `range` | Scalar `min_value` / `max_value` bounds — baseline-free |
+  | `event-timing` | Match Modelica solver events against a declared list within per-event tolerance |
+  | `dominant-frequency` | FFT peak matching against a declared peaks table — windowable |
+  | `scalar` | One number — a reduction of the signal (`max`/`min`/`mean`/`sample_mean`/`final`/`integral`) or a stored statistic by dotted path — vs a baseline scalar or declared bounds |
+
+  Combinators: `and`, `or`, `k-of-n`, `warn` (relax to advisory), `weighted`. Leaves can scope to a time window and target named baselines (`primary`, soft-checks, companions).
+
+- **Read-only reports** — per-test Plotly HTML drawn from figures Python built: the trajectory with its criterion geometry (tube band, range bounds, point boxes, event windows, FFT spectrum), overlay picker, error overlays, and the metric tree with the CLI's verdict and score text per leaf. Plus a live dashboard during runs and JUnit XML for CI. The report shows, the CLI decides.
+
+- **An authoring tool** — `dstf author` (the `[author]` extra) serves a Suite page and a Test page on `127.0.0.1`. Criteria are edited inline under the plot they score, staged, and written to `test_spec.json` on Save:
+
+  | Feature | What it does |
+  |---|---|
+  | Two-sided tolerance | A tolerance is one number or `{"lower": a, "upper": b}`; on the plot, an arrow outside each side sets it |
+  | On-plot editing | Drag a ring, an arrow or a bound line; Ctrl+click adds a row that settles on the data; a field's ⌖ arms it for the next click; a placed value is snapped to a readable step |
+  | Simulation queue | Queue the selected tests: first in, first out, on workers that stay alive between runs; Remove a waiting test, Cancel a running one |
+  | Live results | A running test's trace is drawn as Dymola or OpenModelica writes its result file; a cancelled or timed-out run is shown, never scored or accepted |
+  | Criteria over an array | One criterion on the elements `1:5:20` of an array; identical criteria fold into one row, edited as one |
+  | Themes | `"theme": "dark"` in `testing.json` for the dashboard, the report pages and the tool; the tool's ⋮ menu switches a browser tab live |
+
+- **Three baseline roles** per test, partitioned on disk: **primary** (regression anchor, hard-fail), **soft_checks** (warn-wrapped cross-regression imports), **companions** (plot-only overlays, including cross-library).
+
+## Install
+
+End users:
+
+```bash
+# Not on PyPI yet — these work from the first published release.
+uv tool install dynamic-systems-testing-framework   # then run plain: dstf ...
+# or: pipx install dynamic-systems-testing-framework
+```
+
+Developers:
+
+```bash
+uv sync --extra dev --locked                             # core + pytest, from uv.lock
+uv sync --extra dev --extra author --extra fmpy --extra om --locked   # plus the authoring tool, FMPy + OpenModelica extras
+uv run dstf --help
+```
+
+`uv sync` installs the project editable and pins every dependency to `uv.lock`,
+the same lock CI uses; `--locked` fails instead of silently re-locking when the
+lock has drifted from `pyproject.toml`. It prunes anything you did not ask for
+on that command line — being in the lock does not save it — so name every extra
+at once.
+
+For the Julia backend, install Julia 1.11+ and instantiate the demo project once:
+
+```bash
+curl -fsSL https://install.julialang.org | sh -s -- -y --default-channel 1.11
+cd examples/julia/JuliaMtkTestingLib && julia --project=. -e 'using Pkg; Pkg.instantiate()'
+```
+
+The Python examples (`SimpleRamp`, `PointsCheck`) need scipy, which the `dev`
+extra installs. Reach them through `uv run dstf ...`: the Python backend takes
+the interpreter from `testing.json`'s `backends: {"Python": [...]}` entry and
+falls back to `PATH` — and the shipped example names `python`, so either route
+ends at the first `python` on `PATH`, which `uv run` points at `.venv`. Run the
+same examples from a tool-installed `dstf` and that is whatever comes first on
+your PATH — a conda env, often — which needs its own scipy.
+
+## First 5 minutes
+
+The repo ships demo libraries for every backend. Easiest first run is the Modelica demo:
+
+```bash
+# From the repo root:
+uv run dstf --config examples/modelica/ModelicaTestingLib/Resources/ReferenceResults/testing.json run
+```
+
+What this does:
+
+1. Reads `testing.json` to find the demo library, the backend (Dymola or OpenModelica — first one whose binary exists wins), the reference root, and any per-test recognizers.
+2. Discovers the demo tests via the bundled `UnitTests` recognizer plus declarative recognizers in the JSON.
+3. Runs each in a persistent simulator worker, partitioning output under `testing_output/` by backend + OS.
+4. Compares against `Resources/ReferenceResults/<Backend>/<os>/ref_NNNN.json` baselines.
+5. Prints a console report with pass/fail per test.
+
+To see the interactive HTML reporter:
+
+```bash
+uv run dstf --config examples/modelica/ModelicaTestingLib/Resources/ReferenceResults/testing.json run --report
+```
+
+The dashboard URL is printed as a run starts (`Live progress: file://…`) and the page is opened in a browser tab; it reloads itself every 5s during the run. `--report` writes the per-test pages under `<work_dir>/reports/` and opens the dashboard again when they are written. `--no-browser` (on `run` and `compare`) opens no tab and still prints the link.
+
+The examples ship baselines for these `<Backend>/<os>` partitions: Modelica — `Dymola/linux`, `Dymola/windows` and `OpenModelica/linux` (its array model, `HeatRodTest`, on the two Linux ones); FMU — `FMPy/linux` and `FMPy/windows`; Julia — `Julia/linux`; Python — `Python/linux`. A first run on an OS a backend has no partition for (OpenModelica, Julia or Python on Windows, say) reports every test `NO_REF` until `run --accept` stores its baselines there. A companion in the examples is another source's curve of the SAME model, drawn over the test's plot when its box is ticked on the report page, and each ships in the partitions named here and no other: in `OpenModelica/linux`, the Modelica `FrequencyTest`, `RangeCheckTest` and `MultiFrequencyTest` carry the Julia example's baseline of that model; in `Julia/linux`, the Julia `Frequency`, `RangeCheck` and `MultiFrequency` carry the OpenModelica one; and `MetricTreeTest` carries its closed-form solution in `Dymola/linux` and `OpenModelica/linux`. A Dymola run of the example therefore shows no Julia companion, and `Dymola/windows` ships no companion at all.
+
+To repeat with another backend's demo library, swap the config path:
+
+```bash
+# Julia/MTK, Python, and prebuilt FMUs through FMPy
+uv run dstf --config examples/julia/JuliaMtkTestingLib/Resources/ReferenceResults/testing.json run
+uv run dstf --config examples/python/PythonTestingLib/Resources/ReferenceResults/testing.json run
+uv run dstf --config examples/fmu/testing.json run    # needs reference-fmus-binaries/
+```
+
+## Configuration cheat-sheet
+
+`testing.json` is the single entry point. The full field reference lives in `docs/usage.md`; the headline shape:
+
+```json
+{
+  "source_path": "../../MyLib",
+  "backends": {
+    "Dymola":       ["C:\\Program Files\\Dymola 2026x\\bin64\\Dymola.exe"],
+    "OpenModelica": ["/usr/bin/omc"]
+  },
+  "test_spec": "test_spec.json",
+  "dependencies": ["/path/to/SomeDep"],
+  "tolerance": 1e-4,
+  "diagnostic_variables": ["CPUtime", "EventCounter"]
+}
+```
+
+The first `backends` entry whose binary exists wins (no need for per-OS config files). `backend: "Dymola"` (or any named entry) forces a specific one.
+
+## Common commands
+
+```bash
+# Discover tests without running
+uv run dstf --config testing.json discover
+
+# Run, compare, accept all current results as new baselines
+uv run dstf --config testing.json run --accept
+
+# Interactive review by category
+uv run dstf --config testing.json run -i failed
+
+# Filter (glob, comma list, or @file)
+uv run dstf --config testing.json run --filter "Foo.A,Foo.B"
+uv run dstf --config testing.json run --filter @rerun.txt
+
+# Parallel + interactive HTML report (--batch-size N splits a Dymola --batch run into small batches)
+uv run dstf --config testing.json run --parallel 4 --report
+
+# Rerun only previously-failed tests, merge with last full run, regen report
+uv run dstf --config testing.json run --rerun failed,sim-failed --merge --report
+
+# Compare without re-simulating (uses last results)
+uv run dstf --config testing.json compare
+
+# Open the authoring tool in a browser (needs the [author] extra)
+uv run dstf --config testing.json author
+
+# Apply an RFC 6902 JSON-Patch envelope against test_spec.json — single- or
+# multi-model form, from any source
+uv run dstf --config testing.json spec-update spec_patch.json
+
+# Manage the other baseline roles
+uv run dstf --config testing.json companion add <model> <name> <path>
+uv run dstf --config testing.json soft-check list <model>
+uv run dstf --config testing.json import-baseline <model> <name> <source>   # soft_checks only
+
+# Manifest / schema
+uv run dstf --config testing.json manifest dump
+uv run dstf --config testing.json manifest cleanup [--orphans] [--apply]   # dry-run by default
+uv run dstf --config testing.json export-schema --output schema.json
+
+# Diagnose Dymola Python interface discovery
+uv run dstf check-dymola
+```
+
+`uv run dstf <command> --help` for each subcommand's full flag set.
+
+## Where things live
+
+| Doc | What's in it |
+|---|---|
+| `docs/usage.md` | The deep user guide — every command, every config field, every comparison mode |
+| `docs/architecture.md` | Six-layer plug-in pipeline + current-state map to code |
+| `docs/extensibility.md` | Plug-in contracts for Source / Discovery / Backend / Dataset / Metric / Combinator |
+| `docs/vision.md` | Where the framework is going (target user base, non-Modelica futures) |
+| `docs/decisions.md` | Authoritative decision log |
+| `docs/ideas.md` | Backlog (unimplemented features, with explicit deferred-by-design entries) |
+| `examples/{modelica,fmu,julia,python}/` | Working demo libraries — one per backend |
+
+## CI integration
+
+```yaml
+- run: |
+    uv run dstf --config ./testing.json run --report-format junit --output test-results.xml
+- uses: actions/upload-artifact@v4
+  with:
+    name: test-results
+    path: test-results.xml
+```
+
+For HTML reports as build artifacts, also pass `--report` and upload `dashboard.html` and the `reports/` directory from the work dir (`--work-dir` sets it).
+
+## Status
+
+v1.0. 5 backends in production, 4 demo libraries, and a comprehensive pytest suite (a Playwright subset drives the report pages and the authoring tool in a real browser). The report is read-only and mutates nothing: every number it draws comes from a figure Python built or a string the server rendered. Criteria are authored in `dstf author`, driven by the same figures and the same scorer as the CLI. Nothing is published to PyPI yet.