vnnlib-test-solver 2.0.0__tar.gz → 2.0.1__tar.gz

vnnlib_test_solver-2.0.1/CHANGELOG.md

@@ -0,0 +1,16 @@
+# Version 2.0.1
+
+* Widened the `vnnlib` requirement from `==1.0.2` to `>=1.0.2,<2`, so that the package can be installed alongside a newer 1.x release of that library.
+
+# Version 2.0.0
+
+* Breaking changes:
+    - Renamed the command from `vnnlib-test-solver` to `vnnlibTestSolver`, and the configuration file it looks for in the working directory to `vnnlibTestSolver.toml`. Neither old spelling is accepted. The distribution is still installed as `vnnlib-test-solver`.
+    - Removed the default configuration. A run with no `--config`, no `VNNLIB_TEST_SOLVER_CONFIG` and no `vnnlibTestSolver.toml` now exits `2` instead of answering `unknown` and reporting a full set of capabilities. This applies to `--name` and `--version` as well as to `verify`.
+    - Restricted the `real` exemption to queries that use `real` exclusively, per the definition of a real-valued query in section 3.5. A query mixing `real` with another element type is no longer exempt from the section 5.3.2 error conditions, including for its `real` variables; a query written entirely in `real` is unaffected.
+* Removed the warning issued when `[capabilities]` reports the `real` element type without a matching `[soundness]` claim. Reporting `float32` or `float64` unclaimed still warns.
+* First version published to PyPI.
+
+# Version 1.0.0
+
+* Initial release.

vnnlib_test_solver-2.0.1/CONTRIBUTING.md

@@ -0,0 +1,52 @@
+# Contributing to VNNLIB-Test-Solver
+
+## Setting up the project
+
+1. Clone the repository and create a virtual environment:
+   ```bash
+   git clone https://github.com/VNNLIB/VNNLIB-Test-Solver.git
+   cd VNNLIB-Test-Solver
+   python -m venv .venv
+   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+   ```
+
+2. Install the package in editable mode along with the development dependencies:
+   ```bash
+   pip install -e ".[dev]"
+   ```
+
+## Building and Testing
+
+Run the checks:
+```bash
+pytest                  # the test suite
+pytest -m contract      # the published command line surface, on its own
+ruff check .            # lint
+ruff format --check .   # formatting
+mypy                    # type checking
+```
+
+Three conventions to know before adding a test:
+
+- Tests run the installed `vnnlibTestSolver` as a subprocess rather than importing the package, so an editable install is what makes them exercise your changes.
+- `stdout` is compared as raw bytes against the files in `tests/golden/`. When one fails, establish which of the two is wrong rather than regenerating it.
+- `vnnlib` is imported only in `querymodel.py`, which converts the parser's objects into plain strings, integers and tuples. It imports from `vnnlib.query` and falls back to the top level names, so it works either side of that library's namespace migration.
+
+The **Build-and-Test** workflow runs on every pull request: the suite on Ubuntu, macOS and Windows against Python 3.9, 3.11 and 3.13; lint and type checking; and an install into a container with no build toolchain.
+
+## Making a release
+
+1. Update the version in `pyproject.toml` and `src/vnnlib_test_solver/__init__.py`.
+
+2. Update `README.md` with a new entry to the compatibility table.
+
+3. Update `CHANGELOG.md` with the new version and changes.
+
+4. Create a new Release on GitHub:
+   - Tag the version (e.g., `v2.0.1`).
+   - Provide a title and description.
+   - Publish the release.
+
+5. The **Build-and-Deploy** GitHub workflow will automatically trigger:
+   - It builds the source distribution and the wheel, and checks both before uploading.
+   - It uploads them to PyPI (or TestPyPI if the workflow is run manually with that target).

vnnlib_test_solver-2.0.1/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.5
+Name: vnnlib-test-solver
+Version: 2.0.1
+Summary: A configurable test solver implementing the VNN-LIB standard's command-line interface.
+Project-URL: Homepage, https://github.com/VNNLIB/VNNLIB-Test-Solver
+Project-URL: Repository, https://github.com/VNNLIB/VNNLIB-Test-Solver
+Project-URL: Changelog, https://github.com/VNNLIB/VNNLIB-Test-Solver/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/VNNLIB/VNNLIB-Test-Solver/issues
+Author: VNNLIB
+License-Expression: MIT
+License-File: LICENSE
+Keywords: conformance,neural-network,testing,verification,vnnlib
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Requires-Dist: tomli>=2.0; python_version < '3.11'
+Requires-Dist: vnnlib<2,>=1.0.2
+Provides-Extra: dev
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: serialise
+Requires-Dist: onnx; extra == 'serialise'
+Description-Content-Type: text/markdown
+
+# VNN-LIB Test Solver
+
+[![Build-and-Test](https://github.com/VNNLIB/VNNLIB-Test-Solver/actions/workflows/buildAndTest.yml/badge.svg)](https://github.com/VNNLIB/VNNLIB-Test-Solver/actions/workflows/buildAndTest.yml)
+
+A configurable test solver for the [VNN-LIB format](https://www.vnnlib.org/). It implements the command line interface of chapter 5 of the standard and answers entirely from a TOML configuration file, performing no verification of its own.
+
+It exists so that tools driving VNN-LIB solvers can be tested against a solver whose responses they choose.
+
+## Features
+
+Features include:
+- Results: answer `sat`, `unsat`, `unknown` or `timed-out` from `verify`, selected by the query file's name.
+- Assignments: print a satisfying assignment in the command line format of section 5.3.1.
+- Capabilities: answer all eleven mandatory `supports` capabilities of section 5.4.
+- Identity: answer the `--name` and `--version` global options from configuration.
+- Error conditions: trigger any of the three error conditions of section 5.3.2 on demand, such as a `--network` mapping naming too few or too many model files.
+- Timeouts: wait a configured number of seconds, so a caller's `--timeout` handling can be exercised.
+- Misbehaviour: emit unparseable output, an exit code disagreeing with the answer, noise on `stderr`, or an abnormal termination.
+
+`--serialise-assignments` is accepted but writes no files, whatever the configuration reports for that capability.
+
+## Installation
+
+Install the latest stable version via PyPI:
+```bash
+pip install vnnlib-test-solver
+```
+
+To drive it from another project's test suite, declare it among that project's test dependencies rather than its runtime ones:
+```toml
+[project.optional-dependencies]
+test = ["pytest", "vnnlib-test-solver>=2.0.1"]
+```
+
+Requires Python 3.9 or later. The package is pure Python, but its `vnnlib` parser dependency is distributed only as prebuilt wheels, for Linux `x86_64`, macOS and Windows. Linux `aarch64` has neither a wheel nor a source distribution, so `pip install` cannot satisfy the dependency there.
+
+## Basic Usage
+
+The solver answers only from a configuration file. A run that resolves none exits `2`.
+
+```toml
+# my-solver.toml
+[solver]
+name = "My Test Solver"
+version = "1.0.0"
+
+[[rules]]
+match = "*.vnnlib"
+result = "sat"
+```
+
+```bash
+vnnlibTestSolver --config my-solver.toml verify query.vnnlib
+```
+
+Configuration is resolved from `--config`, then the `VNNLIB_TEST_SOLVER_CONFIG` environment variable, then a `vnnlibTestSolver.toml` in the working directory.
+
+See [docs/CONFIGURATION.md](https://github.com/VNNLIB/VNNLIB-Test-Solver/blob/main/docs/CONFIGURATION.md) for every configuration key and command line option, and [examples/](https://github.com/VNNLIB/VNNLIB-Test-Solver/tree/main/examples/) for runnable configurations.
+
+## Version compatibility
+
+| VNNLIB-Test-Solver version | VNNLIB version | vnnlib version |
+| --- | --- | --- |
+| v2.0.1 | v2.0 | >=1.0.2,<2 |
+| v2.0.0 | v2.0 | 1.0.2 |
+| v1.0.0 | v2.0 | 1.0.2 |

vnnlib_test_solver-2.0.1/README.md

@@ -0,0 +1,66 @@
+# VNN-LIB Test Solver
+
+[![Build-and-Test](https://github.com/VNNLIB/VNNLIB-Test-Solver/actions/workflows/buildAndTest.yml/badge.svg)](https://github.com/VNNLIB/VNNLIB-Test-Solver/actions/workflows/buildAndTest.yml)
+
+A configurable test solver for the [VNN-LIB format](https://www.vnnlib.org/). It implements the command line interface of chapter 5 of the standard and answers entirely from a TOML configuration file, performing no verification of its own.
+
+It exists so that tools driving VNN-LIB solvers can be tested against a solver whose responses they choose.
+
+## Features
+
+Features include:
+- Results: answer `sat`, `unsat`, `unknown` or `timed-out` from `verify`, selected by the query file's name.
+- Assignments: print a satisfying assignment in the command line format of section 5.3.1.
+- Capabilities: answer all eleven mandatory `supports` capabilities of section 5.4.
+- Identity: answer the `--name` and `--version` global options from configuration.
+- Error conditions: trigger any of the three error conditions of section 5.3.2 on demand, such as a `--network` mapping naming too few or too many model files.
+- Timeouts: wait a configured number of seconds, so a caller's `--timeout` handling can be exercised.
+- Misbehaviour: emit unparseable output, an exit code disagreeing with the answer, noise on `stderr`, or an abnormal termination.
+
+`--serialise-assignments` is accepted but writes no files, whatever the configuration reports for that capability.
+
+## Installation
+
+Install the latest stable version via PyPI:
+```bash
+pip install vnnlib-test-solver
+```
+
+To drive it from another project's test suite, declare it among that project's test dependencies rather than its runtime ones:
+```toml
+[project.optional-dependencies]
+test = ["pytest", "vnnlib-test-solver>=2.0.1"]
+```
+
+Requires Python 3.9 or later. The package is pure Python, but its `vnnlib` parser dependency is distributed only as prebuilt wheels, for Linux `x86_64`, macOS and Windows. Linux `aarch64` has neither a wheel nor a source distribution, so `pip install` cannot satisfy the dependency there.
+
+## Basic Usage
+
+The solver answers only from a configuration file. A run that resolves none exits `2`.
+
+```toml
+# my-solver.toml
+[solver]
+name = "My Test Solver"
+version = "1.0.0"
+
+[[rules]]
+match = "*.vnnlib"
+result = "sat"
+```
+
+```bash
+vnnlibTestSolver --config my-solver.toml verify query.vnnlib
+```
+
+Configuration is resolved from `--config`, then the `VNNLIB_TEST_SOLVER_CONFIG` environment variable, then a `vnnlibTestSolver.toml` in the working directory.
+
+See [docs/CONFIGURATION.md](https://github.com/VNNLIB/VNNLIB-Test-Solver/blob/main/docs/CONFIGURATION.md) for every configuration key and command line option, and [examples/](https://github.com/VNNLIB/VNNLIB-Test-Solver/tree/main/examples/) for runnable configurations.
+
+## Version compatibility
+
+| VNNLIB-Test-Solver version | VNNLIB version | vnnlib version |
+| --- | --- | --- |
+| v2.0.1 | v2.0 | >=1.0.2,<2 |
+| v2.0.0 | v2.0 | 1.0.2 |
+| v1.0.0 | v2.0 | 1.0.2 |

vnnlib_test_solver-2.0.1/docs/CONFIGURATION.md

@@ -0,0 +1,261 @@
+# Configuration reference
+
+Every key `vnnlibTestSolver` accepts, and what it does. The [readme](../README.md) covers installing the package and answering a first query, and [`examples/`](../examples/) holds runnable configurations.
+
+The query file supplies structure, the configuration supplies answers.
+
+## Configuration file
+
+| Table | Required | Purpose |
+|-------|----------|---------|
+| `[solver]` | yes | The name and version the solver reports |
+| `[[rules]]` | for `verify` | One entry per answer, selected by the query file's name |
+| `[capabilities]` | for `supports` | The eleven capability responses |
+| `[soundness]` | no | The element types the solver claims to analyse soundly |
+| `[injection]` | no | Misbehaviour applied to whichever command runs |
+
+An unknown key is refused when the configuration loads, and the message names the permitted keys.
+
+## `[solver]`
+
+| Key | Type | Notes |
+|-----|------|-------|
+| `name` | string | Answers `--name`. May contain spaces, but not a line break |
+| `version` | string | Answers `--version` |
+
+Both are required and neither may be empty.
+
+## `[[rules]]`
+
+| Key | Type | Notes |
+|-----|------|-------|
+| `match` | string | Glob matched against the query file's name. Required |
+| `result` | string | `sat`, `unsat`, `unknown` or `timed-out`. Required |
+| `delay_seconds` | integer | Seconds to wait before answering, 0 to 2147483647. Omitted means no delay |
+| `assignments` | table | One array of values per declared variable |
+| `model_element_types` | table | The element types the supplied model files expose |
+| `stderr` | string | Text written to standard error before the answer |
+| `exit_code` | integer | Process exit status, 0 to 255 |
+| `raw_stdout` | string | Replaces standard output entirely |
+| `crash` | boolean | Terminate abnormally instead of exiting |
+
+The last four are the injection controls, described under [`[injection]`](#injection).
+
+### Matching
+
+The first rule whose `match` pattern matches wins. Patterns are compared against the file's **name**, not the path it was given by, and matching is case sensitive on every platform. A query that no rule matches is an error naming the file and every pattern tried.
+
+```toml
+[[rules]]
+match = "sat-*.vnnlib"
+result = "sat"
+
+[[rules]]
+match = "*"
+result = "unknown"
+```
+
+### Delays
+
+`delay_seconds` is how long the solver waits before answering. A delay longer than `--timeout` waits out the timeout and answers `timed-out`; a delay equal to the timeout answers normally; an absent `--timeout` is not a timeout of zero.
+
+### Assignments
+
+A rule answering `sat` may supply the assignment to print, as one array per variable, row-major, keyed by the name the query declares:
+
+```toml
+[[rules]]
+match = "example.vnnlib"
+result = "sat"
+
+[rules.assignments]
+A = [0.5, 0.3, 0.4, 0.2]
+B = [-1]
+Y = [0.0]
+```
+
+Against a query declaring `A float32 [2,2]`, `B int32 [1]` and `Y float32 [1]`, that prints:
+
+```text
+sat
+A float32 [2,2]
+0.5
+0.3
+0.4
+0.2
+B int32 [1]
+-1
+Y float32 [1]
+0.0
+```
+
+Only the values are configured. Each variable's name, element type, dimensions and position come from the query file. Three rules follow:
+
+- A rule covers **every** declared variable or supplies none at all.
+- An array must hold exactly as many values as the declared shape has positions.
+- A name the query does not declare is refused.
+
+Variables appear in the order the query declares them: for each network in turn, its inputs, then its hidden nodes, then its outputs. A network declared `(equal-to f)` still declares its own variables and they still appear.
+
+Values print as the **shortest representation that round-trips to the same value**, which is identical on every platform. Scientific notation is normalised, so `1e10` prints as `10000000000.0` and `1.5e-8` as `1.5e-08`. `inf`, `-inf` and `nan` pass through.
+
+Assignments are printed for `sat` only, and the three rules above are checked only then: a wrong-length array is refused beside `result = "sat"` and passes unremarked beside `result = "unsat"`.
+
+### Model element types
+
+`model_element_types` states what the supplied model files expose, keyed by variable name, since nothing here opens a model file. A variable the table does not name is taken to match; a name the query does not declare is a configuration error.
+
+```toml
+[rules.model_element_types]
+B = "float64"
+```
+
+## `[capabilities]`
+
+`supports` answers one capability at a time. Each key is the flag name without its leading dashes.
+
+| Key | Value | Response |
+|-----|-------|----------|
+| `onnx-opset-versions` | two strings | Two lines: minimum, then maximum |
+| `vnnlib-versions` | two strings | Two lines: minimum, then maximum |
+| `onnx-element-types` | list of element types | One per line |
+| `onnx-operators` | array of tables | One line each: name, then its element types |
+| `hidden-node-theories` | list of identifiers | One per line |
+| `multiple-input-output-theories` | list of identifiers | One per line |
+| `multiple-network-theories` | list of identifiers | One per line |
+| `multiple-node-comparison-theories` | list of identifiers | One per line |
+| `arithmetic-complexity-theories` | list of identifiers | One per line |
+| `optimised-disjunctive-reasoning` | boolean | One line: `true` or `false` |
+| `serialise-assignments` | boolean | One line: `true` or `false` |
+
+A capability the file does not mention is an error, not an empty answer. An explicitly empty list is the answer "none", and prints nothing.
+
+Only the exact flag spellings are accepted: `--onnx-e` is refused rather than resolved to `--onnx-element-types`.
+
+### Theory identifiers
+
+Validated when the configuration loads, against the section 4.1 sets:
+
+| Key | Legal identifiers |
+|-----|-------------------|
+| `hidden-node-theories` | `NH`, `H` |
+| `multiple-input-output-theories` | `SIO`, `MIO` |
+| `multiple-network-theories` | `SNET`, `MNET`, `MENET`, `MINET` |
+| `multiple-node-comparison-theories` | `SNC`, `MNC` |
+| `arithmetic-complexity-theories` | `BND`, `LIN`, `OUTC`, `POLY` |
+
+### Element types
+
+The 21 names the standard defines, accepted wherever an element type is given:
+
+`real`, `bool`, `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float16`, `float32`, `float64`, `bfloat16`, `complex64`, `complex128`, `float4e2m1`, `float8e4m3fn`, `float8e4m3fnuz`, `float8e5m2`, `float8e5m2fnuz`
+
+### Operators
+
+```toml
+[[capabilities.onnx-operators]]
+name = "Gemm"
+element-types = ["float32", "float64"]
+
+[[capabilities.onnx-operators]]
+name = "Relu"
+```
+
+Omitting `element-types` means every element type the solver reports. The line is printed as configured rather than expanded.
+
+## `[soundness]`
+
+| Key | Type | Notes |
+|-----|------|-------|
+| `sound-for` | list of element types | The types the solver claims to analyse soundly |
+
+Reporting a type under `[capabilities]` without claiming it here is allowed, and warns on `stderr` each run. `real` is exempt from that warning.
+
+With no `[soundness]` table, `verify` skips the soundness condition, but the warning above still fires for any reported type other than `real`. An explicitly empty `sound-for = []` is a claim, and refuses every query that is not real-valued.
+
+## `[injection]`
+
+The same four keys a rule carries, applied to whichever command runs. This is the only way to reach `supports`, which has no query file and so matches no rule.
+
+| Key | Type | Effect |
+|-----|------|--------|
+| `stderr` | string | Written before the answer and before any delay |
+| `exit_code` | integer | Exit status, 0 to 255 |
+| `raw_stdout` | string | Replaces standard output entirely; `""` suppresses it |
+| `crash` | boolean | Terminates abnormally after output is flushed; by signal on Unix-like systems |
+
+Where a rule and this table both set the same key, the rule wins **for that key alone** rather than replacing the whole table.
+
+`crash` and `exit_code` cannot be set together, because a process that dies abnormally reports no exit status. An `exit_code` outside 0 to 255 is refused when the configuration loads, since 256 would reach the caller as 0.
+
+**None of this applies to a run the solver refuses.** A configuration mistake, a usage mistake and each of the standard's own error conditions keep their own exit code and write nothing to standard output.
+
+## Command line
+
+```bash
+vnnlibTestSolver [--config <path>] (--name | --version)
+vnnlibTestSolver [--config <path>] verify <filepath> [--network <name>=<filepath>] [--timeout <seconds>] [--serialise-assignments <folder>]
+vnnlibTestSolver [--config <path>] supports <capability>
+```
+
+`--timeout` takes whole seconds, 0 to 2147483647. `--network` is given once per declared network, and mapping the same network twice is refused. A network declared `(equal-to h)` takes no model file of its own, and supplying one for it is an error.
+
+`--serialise-assignments` is accepted but writes no files, and says so on `stderr`. That is independent of what `serialise-assignments` reports under `[capabilities]`: a configuration may report `true` and still no files appear.
+
+## Error conditions
+
+Section 5.3.2 requires `verify` to error in three circumstances. All three write nothing to `stdout`, exit `1`, and carry messages distinct from one another.
+
+| Condition | Driven by |
+|-----------|-----------|
+| Too few or too many model files | The `--network` arguments against the query's declarations |
+| Model element types disagreeing with the query | `model_element_types` |
+| No sound analysis over a declared type | `[soundness] sound-for` |
+
+**The last two are checked only when the run opens the query file**, which happens in exactly three circumstances: a `--network` argument is given, a rule supplies `assignments`, or a rule supplies `model_element_types`. A bare `verify <path>` with none of the three answers from the rule alone and never opens the file. Opening the query also enforces the model mapping, so reaching either condition needs a complete `--network` mapping.
+
+### The `real` exemption
+
+The standard exempts a real-valued query from the last two conditions. Section 3.5 defines a real-valued query as one that uses the `real` element type **exclusively**, so the exemption belongs to the whole query, not to one declaration.
+
+A query mixing `real` with any other type carries none of it, including for its `real` variables: `sound-for = ["float32"]` still refuses such a query, naming its `real` declaration. A query in which every declaration is `real` is exempt in full and needs no claim at all, even under `sound-for = []`.
+
+Mixing element types is not itself rejected.
+
+## Configuration discovery
+
+Resolved in this order, stopping at the first source that applies:
+
+1. `--config <path>`
+2. The `VNNLIB_TEST_SOLVER_CONFIG` environment variable
+3. A `vnnlibTestSolver.toml` in the working directory
+
+**There is no fourth source and no default.** If none resolves, the run exits `2` with nothing on `stdout` and a message naming all three. That applies to `--name` and `--version` as much as to `verify`.
+
+A path that is set but does not point to an existing file is an error, never a fall-through to the next source. Setting `VNNLIB_TEST_SOLVER_DEBUG=1` prints the file a run used to `stderr`.
+
+## Exit codes
+
+The standard defines none, so these are this package's own convention.
+
+| Code | Meaning |
+|------|---------|
+| `0` | A result was delivered, `unsat` and `timed-out` included |
+| `1` | The invocation was wrong: an unknown option, a malformed `--network` pair, a missing query file, an unknown capability |
+| `2` | The configuration was wrong: missing, malformed, or unable to answer the query |
+
+A rule or `[injection]` may set any code from 0 to 255, so these three are a convention rather than a guarantee. A configured run can pair a non-zero exit with a good answer, or an empty `stdout` with a zero exit, so do not infer either stream from the other.
+
+## Output guarantees
+
+- The result is the **first line of `stdout`**, and `stdout` carries nothing else the standard does not define. Warnings, errors and the configuration-source note go to `stderr`. Only `raw_stdout` overrides this.
+- Line endings on `stdout` are **`LF` on every platform**, Windows included, so a recorded response compares byte for byte across machines.
+- Output is **deterministic**: the same configuration and query produce byte-identical `stdout` every time, and lists print in the order the configuration wrote them.
+
+## Departures from chapter 5
+
+Serialised assignments, section 5.3.1's optional `.pb` `TensorProto` output, are not written. Section 5.3 requires the option only of a solver reporting support for it, and section 5.3.1 makes the command-line format the default every solver provides.
+
+The assignment output matches section 5.3.1.1's worked example except for four lines. That example prints `Real` for `H`, `Y`, `C` and `Z`, which its own declarations give as `float32`, and declares `A` and `C` identically while printing them differently. This package prints the declared type. Reported as [VNNLIB-Standard#181](https://github.com/VNNLIB/VNNLIB-Standard/issues/181).
+
+Two names are spelled two ways in chapter 5, and both spellings are accepted: `capabilities` for `supports`, and `--assignments` for `--serialise-assignments`. The second of each pair is the documented form. Reported as [VNNLIB-Standard#179](https://github.com/VNNLIB/VNNLIB-Standard/issues/179).

vnnlib_test_solver-2.0.1/examples/README.md

@@ -0,0 +1,40 @@
+# Example configurations
+
+Four configurations, each runnable as written from the root of this repository. The [configuration reference](../docs/CONFIGURATION.md) explains every key they use.
+
+## `results.toml` - four answers, chosen by file name
+
+```bash
+vnnlibTestSolver --config examples/results.toml verify proved.vnnlib
+vnnlibTestSolver --config examples/results.toml verify refuted.vnnlib
+vnnlibTestSolver --config examples/results.toml verify anything-else.vnnlib
+vnnlibTestSolver --config examples/results.toml verify slow.vnnlib --timeout 1
+```
+
+None of those query files needs to exist, because a rule is selected by the name alone. The last command waits one second and answers `timed-out`, because its rule claims to think for thirty.
+
+## `capabilities.toml` - the eleven capability responses
+
+```bash
+vnnlibTestSolver --config examples/capabilities.toml supports --onnx-element-types
+vnnlibTestSolver --config examples/capabilities.toml supports --onnx-operators
+vnnlibTestSolver --config examples/capabilities.toml --name
+```
+
+## `worked-example.toml` - the standard's own assignment example
+
+```bash
+vnnlibTestSolver --config examples/worked-example.toml verify tests/fixtures/worked_example.vnnlib --network f=examples/f.onnx --network g=examples/g.onnx
+```
+
+This one does need its query file: the variable names, types, dimensions and order come from the query, and only the values come from configuration. The two `.onnx` files are placeholders; the mapping only establishes that the right networks were named.
+
+## `misbehaving.toml` - a solver that answers wrongly on purpose
+
+```bash
+vnnlibTestSolver --config examples/misbehaving.toml verify crash.vnnlib
+vnnlibTestSolver --config examples/misbehaving.toml verify garbage.vnnlib
+vnnlibTestSolver --config examples/misbehaving.toml verify noisy.vnnlib
+```
+
+A correct answer followed by abnormal termination; a first line that is not a result; and noise on standard error beside an exit code that disagrees with the answer.

vnnlib_test_solver-2.0.1/examples/capabilities.toml

@@ -0,0 +1,29 @@
+# All eleven mandatory capabilities, in the four response shapes of section 5.4.
+
+[solver]
+name = "Example Solver"
+version = "1.0.0"
+
+[capabilities]
+onnx-opset-versions = ["13", "21"]
+onnx-element-types = ["real"]
+vnnlib-versions = ["2.0", "2.0"]
+hidden-node-theories = ["NH"]
+multiple-input-output-theories = ["SIO"]
+multiple-network-theories = ["SNET"]
+multiple-node-comparison-theories = ["SNC"]
+arithmetic-complexity-theories = ["BND", "LIN"]
+optimised-disjunctive-reasoning = false
+serialise-assignments = false
+
+# An operator with no element types supports every type reported above, not none.
+[[capabilities.onnx-operators]]
+name = "Gemm"
+element-types = ["real"]
+
+[[capabilities.onnx-operators]]
+name = "Relu"
+
+# Reporting a type above without claiming it here warns on stderr (section 5.4.1).
+[soundness]
+sound-for = ["real"]

vnnlib_test_solver-2.0.1/examples/misbehaving.toml

@@ -0,0 +1,24 @@
+# A solver that answers wrongly on purpose. None of these controls applies to a
+# run the solver refuses.
+
+[solver]
+name = "Example Solver"
+version = "1.0.0"
+
+# Answers, flushes, then terminates abnormally. Setting exit_code beside it is refused.
+[[rules]]
+match = "crash.vnnlib"
+result = "sat"
+crash = true
+
+# Replaces stdout entirely; "" suppresses it.
+[[rules]]
+match = "garbage.vnnlib"
+result = "sat"
+raw_stdout = "not-a-result-line\n"
+
+[[rules]]
+match = "noisy.vnnlib"
+result = "unsat"
+stderr = "solver: exploring branch 1 of 4"
+exit_code = 7

vnnlib_test_solver-2.0.1/examples/results.toml

@@ -0,0 +1,24 @@
+# Four answers from one binary, chosen by the query file's name.
+
+[solver]
+name = "Example Solver"
+version = "1.0.0"
+
+[[rules]]
+match = "proved.vnnlib"
+result = "sat"
+
+[[rules]]
+match = "refuted.vnnlib"
+result = "unsat"
+
+# A delay longer than --timeout answers timed-out instead.
+[[rules]]
+match = "slow.vnnlib"
+result = "sat"
+delay_seconds = 30
+
+# Without a catch-all, a query no rule matches is an error rather than an unknown.
+[[rules]]
+match = "*"
+result = "unknown"

vnnlib_test_solver-2.0.1/examples/worked-example.toml

@@ -0,0 +1,19 @@
+# The worked example of section 5.3.1. Only the values are configured: each
+# variable's name, element type, dimensions and position come from the query file.
+
+[solver]
+name = "Example Solver"
+version = "1.0.0"
+
+[[rules]]
+match = "worked_example.vnnlib"
+result = "sat"
+
+# One array per declared variable, row-major. A rule covers every variable or none.
+[rules.assignments]
+A = [0.5, 0.3, 0.4, 0.2]
+B = [-1]
+H = [0.5, 0.3]
+Y = [0.1]
+C = [1.0, 0.3, 0.4, 0.2]
+Z = [0.0]