testerkit 0.4.0__py3-none-any.whl

testerkit/_docs/tutorial/01-first-test.md

@@ -0,0 +1,113 @@
+# Step 1: Run Something
+
+**Goal:** Install TesterKit and run your first test against a mock instrument.
+
+## What You'll Build
+
+A tiny project with one mock instrument and a passing measurement test — no hardware, and no station or part YAML yet. This is the **bench-bringup** scaffold: the smallest thing that records a real measurement.
+
+## Install
+
+```bash
+pip install testerkit
+```
+
+That installs the `testerkit` CLI and the pytest plugin — your tests are ordinary pytest functions; the plugin adds the hardware-test pieces.
+
+## Scaffold a project
+
+```bash
+testerkit init my_project --tier=bringup
+cd my_project
+```
+
+The `bringup` tier is the smallest scaffold: mock instrument fixtures in a `conftest.py`, one smoke test, and one sidecar — no station, catalog, or part YAML. It creates:
+
+```
+my_project/
+├── testerkit.yaml          # project config
+├── pyproject.toml
+├── tests/
+│   ├── conftest.py      # mock dmm / psu fixtures
+│   ├── test_smoke.py    # measurement tests
+│   └── test_smoke.yaml  # sidecar limits
+└── reports/
+```
+
+## Run it
+
+```bash
+pytest -v
+```
+
+Expected output:
+
+```
+tests/test_smoke.py::test_rail_inline PASSED
+tests/test_smoke.py::test_rail_sidecar PASSED
+tests/test_smoke.py::test_current_draw PASSED
+```
+
+Three measurements recorded against mock instruments, each checked against a limit.
+
+## What's in the scaffold
+
+The `conftest.py` defines instrument fixtures with `MagicMock` standing in for a real driver:
+
+```python
+# tests/conftest.py
+from unittest.mock import MagicMock
+import pytest
+
+
+@pytest.fixture
+def dmm() -> MagicMock:
+    """Bench DMM. Replace MagicMock with a real driver."""
+    inst = MagicMock()
+    inst.measure_dc_voltage.return_value = 3.3
+    return inst
+```
+
+`test_smoke.py` ships three measurement tests; here's the first — it takes the `dmm` fixture and records a measurement:
+
+```python
+# tests/test_smoke.py
+def test_rail_inline(dmm, verify) -> None:
+    verify(
+        "v_rail",
+        float(dmm.measure_dc_voltage()),
+        limit={"low": 3.2, "high": 3.4, "nominal": 3.3, "unit": "V"},
+    )
+```
+
+`verify` is a fixture the TesterKit plugin provides (installed with `testerkit`): it records the measurement and checks it against the limit, failing the test if the value is out of band. You'll meet `verify` and limits properly in [Step 3](03-fixtures.md) and [Step 4](04-limits.md) — for now, you've run a test that captures a real measurement. Swap the `MagicMock` for a [PyVISA](https://pyvisa.readthedocs.io/) or [PyMeasure](https://pymeasure.readthedocs.io/) driver when you move to the bench; the test body doesn't change.
+
+## About conftest.py
+
+Right now the instruments come from `conftest.py` fixtures — the same pattern you'd use in any pytest project. TesterKit doesn't need its own configuration to get started.
+
+Later steps introduce a [station YAML](../concepts/configuration/stations.md) — one file that declares the bench's instruments. When it exists, TesterKit auto-registers an instrument-role fixture for each instrument it declares (`dmm`, `psu`, …), and you delete the matching `conftest.py` fixtures. The test bodies stay the same.
+
+## Results
+
+Each measurement is recorded to TesterKit's **run store** — the value `verify` captured, not just a pass/fail. Viewing and querying runs comes in later steps; the point for now is that the test recorded a real measurement.
+
+## Troubleshooting
+
+**"pytest: command not found"** — make sure `testerkit` installed into the active environment, and if you use a virtualenv, that it's activated.
+
+**"No tests collected"** — check the test file name starts with `test_` and each function starts with `test_`.
+
+**"fixture 'dmm' not found"** — the fixture lives in `tests/conftest.py`, which `testerkit init --tier=bringup` creates. Later steps lift the fixture into a station YAML, where the role fixture is auto-registered.
+
+## What You Learned
+
+- Install TesterKit with `pip install testerkit`
+- Scaffold the smallest project with `testerkit init --tier=bringup`
+- Run measurement tests against mock instruments with `pytest`
+
+## Continue
+
+Next, run the same tests in TesterKit's mock mode and control the returned values from config.
+
+← [Quick Start](quickstart.md)  |  [Step 2: Mock Instruments →](02-mock-instruments.md)

testerkit/_docs/tutorial/02-mock-instruments.md

@@ -0,0 +1,88 @@
+# Step 2: Running Without Hardware
+
+**Goal:** Run your tests without real instruments by wrapping your driver classes with TesterKit's `Mock` helper.
+
+In step 1 you wrote vanilla pytest tests against `psu` and `dmm` fixtures defined in `conftest.py`. This step shows the smallest change that lets the same tests run on a laptop with no hardware attached.
+
+## The conftest pattern
+
+Your `conftest.py` already returns driver instances. Wrap them in `testerkit.instruments.mocks.Mock` when the `--mock-instruments` flag is set:
+
+```python
+# tests/conftest.py
+import pytest
+from drivers import DMM, PSU
+from testerkit import Mock
+
+
+@pytest.fixture(scope="session")
+def psu(mock_instruments) -> PSU:
+    if mock_instruments:
+        return Mock(PSU, measure_voltage=5.0, measure_current=0.042)
+    return PSU(resource="TCPIP::192.168.1.101::INSTR")
+
+
+@pytest.fixture(scope="session")
+def dmm(mock_instruments) -> DMM:
+    if mock_instruments:
+        return Mock(DMM, measure_dc_voltage=3.31)
+    return DMM(resource="TCPIP::192.168.1.102::INSTR")
+```
+
+`mock_instruments` is a fixture TesterKit provides — it returns `True` whenever `--mock-instruments` is on the command line or `TESTERKIT_MOCK_INSTRUMENTS=1` is set.
+
+`Mock(DMM, measure_dc_voltage=3.31)` returns a stand-in `DMM` — every method call does nothing and returns `None` unless you give it a return value. Pass a literal for a constant reading, a `dict` to map an argument value to a reading, or a function for dynamic behavior.
+
+This is exactly what [`examples/01-vanilla`](https://github.com/pragmatest-dev/testerkit/tree/main/examples/01-vanilla) and [`examples/02-verify`](https://github.com/pragmatest-dev/testerkit/tree/main/examples/02-verify) ship.
+
+## Running with mocks
+
+```bash
+pytest tests/ --mock-instruments -v
+```
+
+Same test code as step 1, no hardware required.
+
+```bash
+# Or via env var
+TESTERKIT_MOCK_INSTRUMENTS=1 pytest tests/ -v
+```
+
+## Mock cheat sheet
+
+```python
+# Constant return
+dmm = Mock(DMM, measure_dc_voltage=3.31)
+
+# Map by argument — different return per query string
+dmm = Mock(DMM, query={"MEAS:VOLT:DC?": "3.300", "*IDN?": "Keysight,34461A,..."})
+
+# Callable — for noise or sweeps
+import random
+dmm = Mock(DMM, measure_dc_voltage=lambda: 3.3 + random.gauss(0, 0.005))
+```
+
+Every method you don't configure does nothing and returns `None`. Reading an attribute you never configured raises an `AttributeError` instead — so a missing mock value fails loudly rather than silently returning nothing.
+
+## Mocks vs real hardware
+
+| You run | `mock_instruments` is | Test code |
+|---|---|---|
+| `pytest tests/` | `False` | identical |
+| `pytest tests/ --mock-instruments` | `True` | identical |
+
+The point of the wrap-in-conftest pattern: **the test code is the same on a laptop and on the bench**. Tests don't know which mode they're in.
+
+## What you learned
+
+- `--mock-instruments` flag + the `mock_instruments` fixture
+- `Mock(DriverClass, method=return_value, ...)` wraps any driver class
+- The conftest fixture decides real vs mock — tests don't change
+
+In later steps you'll move this real-vs-mock choice out of `conftest.py` and into station YAML (step 7), so one setting serves a whole bench of tests. For now, conftest is enough.
+
+## Continue
+
+Now let's adopt three of [TesterKit's per-test fixtures](../reference/pytest/fixtures.md) — `context`, `verify`, `measure` — to start recording measurements with limits.
+
+← [Step 1: Run Something](01-first-test.md)  |  [Step 3: pytest-native tests →](03-fixtures.md)

testerkit/_docs/tutorial/03-fixtures.md

@@ -0,0 +1,169 @@
+# Step 3: pytest-native tests
+
+**Goal:** Adopt TesterKit's per-test fixtures so measurements get recorded with full [traceability](../how-to/execution/traceability.md).
+
+In step 2, your tests called driver methods and used `assert` for pass/fail. TesterKit's `measure` and `verify` fixtures slot in alongside that, recording each measurement to the run record (see [data stores](../concepts/data/data-stores.md)) without changing how your test reads.
+
+You don't need any new YAML for this step. Keep the `conftest.py` from step 2 — the `psu` / `dmm` fixtures still work.
+
+## The fixtures you add
+
+All three are available on every test run — no station, no sidecar, no sweep required. `measure` and `verify` record measurements; `context` exposes what's active for this test — the run, UUT, station, and any sweep values.
+
+| Fixture  | What it gives the test                                 | Verbs                                            |
+|----------|--------------------------------------------------------|--------------------------------------------------|
+| `measure`| Records a measurement row — no pass/fail check          | `measure(name, value, limit=None, characteristic=None)` |
+| `verify` | Records the row, resolves a limit, raises on FAIL      | `verify(name, value, limit=..., characteristic=...)` (`characteristic` = a named measurable property on the part spec — covered in step 6 / [concepts/capabilities](../concepts/configuration/capabilities.md)) |
+| `context`| What's active — the run, UUT, station, and (if parametrized) sweep values | `get_param`, `changed`, `last`, `observe`, `.part`, `.station`, `.run` |
+
+These are the three you'll reach for most. The TesterKit plugin for pytest provides more — hardware accessors (`pins`, `instruments`, `uut`), config accessors (`part`, `station_config`), and special-purpose fixtures (`vectors`, `sync`) — see the [TesterKit fixtures reference](../reference/pytest/fixtures.md) for the full set.
+
+## From assert to measure
+
+Take the test from step 2:
+
+```python
+def test_output_voltage(psu, dmm):
+    psu.set_voltage(5.0)
+    psu.enable_output()
+    v = dmm.measure_dc_voltage()
+    assert 3.2 <= v <= 3.4
+```
+
+Add `measure` and record the measurement explicitly:
+
+```python
+def test_output_voltage(psu, dmm, measure):
+    psu.set_voltage(5.0)
+    psu.enable_output()
+    v = dmm.measure_dc_voltage()
+    measure("output_voltage", v, limit={"low": 3.2, "high": 3.4, "unit": "V"})
+    assert 3.2 <= v <= 3.4
+```
+
+Same control flow, but now there's a measurement recorded in the run record — value, units, limits, and outcome — visible to `testerkit runs`, the operator UI, and any downstream analysis.
+
+## Skip the assert with `verify`
+
+`verify` is `measure` + `assert` in one call. Pass / fail is decided by the limit; an out-of-range value raises `AssertionError`:
+
+```python
+def test_output_voltage(psu, dmm, verify):
+    psu.set_voltage(5.0)
+    psu.enable_output()
+    verify("output_voltage", dmm.measure_dc_voltage(),
+           limit={"low": 3.2, "high": 3.4, "unit": "V"})
+```
+
+For one-off tests, passing `limit=` inline is fine. The cleaner home for limits is the part spec or the sidecar YAML — both arrive in later steps.
+
+## Classes group related tests
+
+Grouping related tests in a plain pytest class is the standard way to structure a TesterKit test:
+
+```python
+class TestPowerUp:
+    def test_input_voltage(self, psu, verify):
+        psu.set_voltage(5.0)
+        psu.enable_output()
+        verify("input_voltage", psu.measure_voltage(),
+               limit={"low": 4.5, "high": 5.5, "unit": "V"})
+
+    def test_output_voltage(self, dmm, verify):
+        verify("output_voltage", dmm.measure_dc_voltage(),
+               limit={"low": 3.2, "high": 3.4, "unit": "V"})
+```
+
+Methods run in source order. Each emits its own [step](../concepts/execution/step-hierarchy.md) events; the class container's [outcome](../reference/data/models.md#enum-outcome) rolls up from the worst child outcome.
+
+If a downstream test should skip when an upstream one fails, use `@pytest.mark.dependency(depends=["test_input_voltage"])` from the [`pytest-dependency`](https://pytest-dependency.readthedocs.io/) plugin — pytest's ecosystem, not a TesterKit addition.
+
+## Parametrize is first-class
+
+`@pytest.mark.parametrize` works the way it always does. Add the `context` fixture if you want those parametrize values recorded with the measurement:
+
+```python
+import pytest
+@pytest.mark.parametrize("vin", [4.5, 5.0, 5.5])
+def test_output_voltage(vin, psu, dmm, verify):
+    psu.set_voltage(vin)
+    psu.enable_output()
+    verify("output_voltage", dmm.measure_dc_voltage(),
+           limit={"low": 3.2, "high": 3.4, "unit": "V"})
+```
+
+Each parametrized `vin` value is recorded as an **input** named `vin` on the measurement (its role is `input` — see [traceability](../how-to/execution/traceability.md)), so you can later query "how did output_voltage track vin?" — inputs are addressable by name and role — without adding extra code to the test. Sweeping from YAML instead of inline arrives in step 5.
+
+TesterKit also adds a native sweep marker, `@pytest.mark.testerkit_sweeps`, that records the same inputs and supports range expanders (`linspace`, `arange`, `logspace`):
+
+```python
+import pytest
+
+@pytest.mark.testerkit_sweeps([{"vin": [4.5, 5.0, 5.5]}])
+def test_output_voltage(vin, psu, dmm, verify):
+    ...
+```
+
+Use `@pytest.mark.parametrize` when you want pytest's per-row `pytest.param(..., id="...")` metadata; use `@pytest.mark.testerkit_sweeps` when you want range expanders (`linspace` / `arange` / `logspace`) or want the sweep to match how you'll define it in YAML (step 5). See [`testerkit_sweeps`](../reference/pytest/markers.md#testerkit_sweeps) and the [TesterKit markers reference](../reference/pytest/markers.md) for all seven `testerkit_*` markers.
+
+## Multiple measurements per test
+
+Each `verify` or `measure` call records one measurement. Call them as many times as you need:
+
+```python
+def test_power_analysis(psu, dmm, verify):
+    verify("input_voltage",  psu.measure_voltage(),
+           limit={"low": 4.5, "high": 5.5, "unit": "V"})
+    verify("input_current",  psu.measure_current(),
+           limit={"high": 0.5, "unit": "A"})
+    verify("output_voltage", dmm.measure_dc_voltage(),
+           limit={"low": 3.2, "high": 3.4, "unit": "V"})
+```
+
+## Recording many samples
+
+`measure` records one row per name within a step. For a stream of samples under a single name — a stability capture or a scope trace — use a channel (`stream`), covered in [Step 10](10-live-monitoring.md) and [Step 11](11-waveforms-and-evidence.md).
+
+## Running the tests
+
+Nothing new on the command line — same `pytest` invocation from step 2:
+
+```bash
+pytest tests/ --mock-instruments -v
+```
+
+If you want to see the recorded measurements, list runs from the CLI:
+
+```bash
+testerkit runs
+testerkit show <run_id>
+```
+
+## What a measurement records
+
+Read a run back and each measurement gives you:
+
+| Field | Description |
+|--------|-------------|
+| `measurement_name` | name passed to `verify` / `measure` |
+| `measurement_value` | the measured value |
+| `measurement_unit` | unit (from `limit.unit`) |
+| `measurement_outcome` | `passed` / `failed` / `done` / `skipped` / `errored` |
+| `limit_low`, `limit_high`, `limit_nominal`, `limit_comparator` | the active limit |
+| `measurement_timestamp` | when it was recorded |
+| `vector_index` | which sweep variant (NULL for non-parametrized tests) |
+
+Read these with `testerkit runs` / the operator UI, or the [Query API](../reference/data/query-api.md).
+
+## What you learned
+
+- `measure(name, value, limit={"low": ..., "high": ..., "unit": "V"})` records a measurement explicitly
+- `verify(name, value, limit=...)` does the same plus pass/fail + raise on FAIL
+- Pytest classes group related tests; methods run in source order
+- Parametrize works as it always does; values are recorded as inputs (role `input`)
+
+## Continue
+
+So far you've been passing `limit=` inline on every `verify` call. Step 4 separates the limit shape from the test code.
+
+← [Step 2: Mock Instruments](02-mock-instruments.md)  |  [Step 4: Add Limits →](04-limits.md)

testerkit/_docs/tutorial/04-limits.md

@@ -0,0 +1,139 @@
+# Step 4: Add Limits
+
+**Goal:** Decide pass/fail for a measurement.
+
+In step 3 your tests called `verify(name, value, limit=...)`. The pass/fail decision happens because a **limit** is attached. This step covers the limit shape and the two ways to attach a limit from code: inline on the call, or via the `testerkit_limits` marker on the test function. Both end up as the same limit.
+
+Step 5 moves limits out of code into a YAML file next to the test — keep that destination in mind, but don't reach for YAML yet.
+
+## The limit shape
+
+A limit is a plain dict — same shape whether it lives inline, on a marker, or in YAML:
+
+```python
+limit = {
+    "low": 3.135,    # Minimum acceptable value
+    "high": 3.465,   # Maximum acceptable value
+    "nominal": 3.3,  # Expected value (optional)
+    "unit": "V",     # Unit of measure
+}
+```
+
+Both `verify(name, value, limit=...)` and `measure(name, value, limit=...)` accept this dict directly. It's validated against the [`Limit`](../reference/data/models.md#model-limit) model. If you'd rather construct it explicitly — for IDE type-checking or a shared constant — `Limit` is re-exported from the top-level package:
+
+```python
+from testerkit import Limit
+
+V_RAIL = Limit(low=3.135, high=3.465, unit="V")
+```
+
+The dict form is what tutorials and examples use; reach for `Limit(...)` when you want the model object.
+
+## How a measurement is checked
+
+`measure(...)` records a [`Measurement`](../reference/data/models.md#model-measurement) row with the value, units, and limit. `verify(...)` does the same plus raises `AssertionError` on FAIL. Either way, the row carries an `Outcome`:
+
+| Outcome | String value | Meaning |
+|---------|--------------|---------|
+| `Outcome.PASSED` | `"passed"` | Value within limits |
+| `Outcome.FAILED` | `"failed"` | Value outside limits |
+| `Outcome.SKIPPED` | `"skipped"` | Test was skipped |
+| `Outcome.ERRORED` | `"errored"` | Test encountered an error |
+| `Outcome.ABORTED` | `"aborted"` | Run aborted by operator |
+| `Outcome.TERMINATED` | `"terminated"` | Run terminated (keyboard interrupt, signal) |
+| `Outcome.DONE` | `"done"` | Recorded without a limit, or container with no measurements |
+
+A parent step takes the **worst** outcome of its children, ranked `skipped < done < passed < failed < errored < terminated < aborted` (so a parent with one skipped child and one passing child still resolves to `passed`).
+
+## Inline limit on the call
+
+The simplest form — pass `limit=` directly to `verify` or `measure`:
+
+```python
+def test_output_voltage(dmm, verify):
+    verify(
+        "output_voltage",
+        dmm.measure_dc_voltage(),
+        limit={"low": 3.135, "high": 3.465, "unit": "V"},
+    )
+```
+
+Inline limits are fine for one-off tests. They clutter the test body when limits get long or vary per test.
+
+## Limit via marker
+
+`testerkit_limits` pulls the limit out of the body and pins it at the top of the test:
+
+```python
+import pytest
+
+@pytest.mark.testerkit_limits(
+    output_voltage={"low": 3.135, "high": 3.465, "unit": "V"},
+)
+def test_output_voltage(dmm, verify):
+    verify("output_voltage", dmm.measure_dc_voltage())
+```
+
+The marker accepts one keyword per measurement name. `verify("output_voltage", ...)` resolves the limit from the marker without an explicit `limit=`. You can apply `@pytest.mark.testerkit_limits` at function, class, or module level — class scope applies to every method on the class.
+
+## Comparators
+
+By default, limits use `GELE` (greater-or-equal to low, less-or-equal to high): `low <= value <= high`. Other comparators are available when the test needs a different shape:
+
+```python
+# Upper limit only
+limit = {"high": 1.0, "unit": "A", "comparator": "LE"}      # value <= 1.0
+
+# Lower limit only
+limit = {"low": 0.0, "unit": "V", "comparator": "GE"}       # value >= 0.0
+
+# Must equal nominal
+limit = {"nominal": 5.0, "unit": "V", "comparator": "EQ"}
+```
+
+Full list:
+
+| Comparator | Pass condition |
+|------------|----------------|
+| `GELE` | `low <= value <= high` (default) |
+| `GELT` | `low <= value < high` |
+| `GTLE` | `low < value <= high` |
+| `GTLT` | `low < value < high` |
+| `EQ` | `value == nominal` |
+| `NE` | `value != nominal` |
+| `GE` | `value >= low` |
+| `GT` | `value > low` |
+| `LE` | `value <= high` |
+| `LT` | `value < high` |
+
+## Recording without judging
+
+`measure` records a value without comparing it to a limit — pass no `limit=` and the row carries `Outcome.DONE`:
+
+```python
+def test_voltage(dmm, measure):
+    measure("output_voltage", dmm.measure_dc_voltage())
+```
+
+`verify` requires a limit: calling it with none (no inline `limit=`, no marker, no sidecar, no part spec) raises `MissingLimitError`. For a wide characterization sweep where you want the same `verify()` test code to record values without judging, set `verify_requires_limit: false` on a [profile](../how-to/execution/profiles.md) — `verify` then falls back to `measure` semantics for that session.
+
+## What's missing — and what step 5 fixes
+
+Inline limits and markers live in the test code. That means a non-developer can't change them, condition-dependent limits get awkward, and limits can't be reused across multiple test files. Step 5 introduces the **[sidecar YAML](05-configuration.md)** — a file next to the test that carries limits (and sweeps, mocks, retries, prompts) without changing the test code.
+
+For [condition-indexed bands](../how-to/execution/limits.md#condition-indexed-bands) (different bands at different temperatures or loads) jump to [Test limits](../how-to/execution/limits.md#condition-indexed-bands) when you need it.
+
+## What you learned
+
+- The limit dict — `low`, `high`, `nominal`, `unit`, `comparator`
+- Inline limits via `verify(..., limit={...})` or `measure(..., limit={...})`
+- The `testerkit_limits` marker for attaching limits at class or function level
+- The `Outcome` ladder and what each value means
+- The `Comparator` enum for non-`GELE` checks
+- Recording without judging via `measure(no limit)` or a record-only profile
+
+## Continue
+
+Move the limits out of code and into a YAML file next to your test.
+
+← [Step 3: pytest-native tests](03-fixtures.md)  |  [Step 5: Test Configuration →](05-configuration.md)