testerkit 0.4.0__py3-none-any.whl

testerkit/_docs/concepts/execution/outcomes.md

@@ -0,0 +1,199 @@
+# Outcomes
+
+Every measurement, vector, step, and run carries an `Outcome` — one of seven values — or `None` if no verdict was ever rendered. This page explains what each value means, how a worse outcome on a child rolls up to the parent, and where each value gets stamped.
+
+For the column-by-column shape of how outcomes land in parquet, see [parquet schema → outcome values](../../reference/data/parquet-schema.md). For the level hierarchy (measurement → vector → step → run) the cascade walks, see [step hierarchy](step-hierarchy.md).
+
+## The severity ladder
+
+Outcomes are ordered by severity. When a parent has multiple children, the parent's outcome is the worst child's outcome — and once a parent reaches a given level, a less-severe later child doesn't weaken it.
+
+| Severity | Value | One-liner |
+|---:|---|---|
+| 7 | `ABORTED` | Process died before cleanup; rig state is unknown. |
+| 6 | `TERMINATED` | Operator stopped the run; cleanup ran; rig is safe. |
+| 5 | `ERRORED` | Code blew up (not an assertion). |
+| 4 | `FAILED` | A verdict ran and was violated. |
+| 3 | `PASSED` | A verdict ran and was satisfied. |
+| 2 | `DONE` | Code ran cleanly with no verdict — "I logged data". |
+| 1 | `SKIPPED` | Explicit skip; the body didn't run. |
+| — | `None` | Never judged at all. Treated as severity `-1` by the cascade. |
+
+### Cascade rule
+
+The cascade returns the higher-severity of the two. **Ties favor the current outcome** — once a parent reaches FAILED, an incoming FAILED leaves it unchanged. This matters when interpreting timestamps: the first FAILED stamp is the one that survives.
+
+`None` participates with severity `-1`, so any real outcome wins against `None`.
+
+## Verdict intent — what separates PASSED from DONE
+
+The difference between PASSED and DONE on a step that ran cleanly: did the body *try* to judge?
+
+A step has **verdict intent** if either fires during the test:
+
+- A passing `assert` ran in the test body.
+- A measurement with a limit was recorded — the limit is the thing that gets judged.
+
+At step end, the plugin picks PASSED if the step had any verdict intent, DONE if not.
+
+So:
+
+- Test body exits cleanly **with** verdict intent → **PASSED**.
+- Test body exits cleanly **without** verdict intent → **DONE**.
+- Test body raises `AssertionError` → **FAILED**.
+- Test body raises anything else → **ERRORED**.
+- Test body raises `pytest.skip.Exception` → **SKIPPED**.
+
+## What each outcome means
+
+### `PASSED` — a verdict ran and was satisfied
+
+- **Measurement-level**: value was checked against a limit and was in range (`value in limit` returned `True`).
+- **Step-level**: the test body exited cleanly AND verdict intent fired at least once.
+- **Run-level**: rolls up from PASSED steps with nothing worse anywhere.
+
+### `FAILED` — a verdict ran and was violated
+
+- **Measurement-level**: value was checked and `value in limit` returned `False`.
+- **Step-level**: the test body raised `AssertionError` (rewritten or bare), OR a contained measurement landed FAILED and cascaded up.
+- **Run-level**: rolls up from any FAILED step.
+
+### `DONE` — clean run, no verdict
+
+The "I logged data" outcome. Not a "good" outcome and not a "bad" one — judgment never happened.
+
+- **Measurement-level**: a value was recorded with no `low`/`high`/`nominal` (no limit to check against).
+- **Step-level**: the body exited cleanly AND no verdict intent fired.
+- **Run-level**: rolls up from DONE steps with nothing worse.
+
+### `SKIPPED` — explicit skip
+
+- **Step-level**: `pytest.skip(...)`, `@pytest.mark.skip`, `@pytest.mark.skipif`, or a setup-phase skip exception. The test body either didn't run or stopped early.
+- **Run-level**: cascade rollup where the only contained outcomes were SKIPPED.
+- **Vector-level**: `VectorBuilder.skip(...)` on the `TesterKitClient` builder path explicitly stamps SKIPPED. Not produced by the runtime cascade.
+
+### `ERRORED` — unhandled exception
+
+Two distinct paths land here, and they're not interchangeable:
+
+- **Step-level**: the test body (or setup / teardown) raised any non-`AssertionError`, non-skip exception. A `ValueError`, `RuntimeError`, `pyvisa.VisaIOError`, etc. **No `Measurement` row is recorded for the broken call** — the step is ERRORED, not the (non-existent) measurement.
+- **Measurement-level**: the row exists, with `value=None`. Happens when:
+  - `verify("vout", instr.measure_voltage())` was called and `measure_voltage()` *returned* None silently (broken driver, mock not configured, swallowed timeout).
+
+Exceptions do not produce ERRORED measurements — they produce a step-level ERRORED with no measurement row.
+
+- **Run-level**: rolls up from any ERRORED step or measurement.
+
+### `TERMINATED` — operator stopped cleanly
+
+The operator stopped the run (Ctrl-C or SIGTERM). Cleanup ran to completion — fixture teardowns finished, instruments went to a safe state, and results were saved.
+
+The rig **is** in a known state. The run was stopped on purpose, with cleanup. Downstream tooling and operator runbooks can read TERMINATED as "intentional stop; rig safe."
+
+### `ABORTED` — process died before cleanup
+
+The runs daemon was asked to write a run that never saw a `RunEnded` event. The teardown chain didn't complete — the run never finalized.
+
+- The process was killed mid-flight (SIGKILL, segfault, OOM kill, host shutdown).
+- An exception bypassed teardown before finalization could run.
+
+The rig state is **unknown**. When a run is killed before it finalizes, TesterKit records the unfinished run as ABORTED automatically — so a process that dies mid-run is never silently lost. A non-pytest runner can also mark a run ABORTED through the results API; see the [client reference](../../reference/runtime/client.md).
+
+ABORTED on a parquet row means the run never closed cleanly — downstream tooling and operator runbooks should treat the rig as "physically inspect required."
+
+### `None` — never judged, never finalized
+
+The row exists (it was collected, or a step was opened) but no outcome was ever set.
+
+- A pytest test that pytest collected but never ran — earlier failure aborted the session, or `--exitfirst` cut things short.
+- A vector that ran but recorded nothing and didn't raise.
+
+A finalized run whose row has no outcome simply never reached a verdict. The operator UI shows these as "Never Ran" — see [step manifest](step-manifest.md#never-ran).
+
+## Where each outcome gets stamped
+
+The tables below list what triggers each outcome at each level, worst → least severe.
+
+### Measurement level
+
+A measurement has no outcome until its value is checked against a limit.
+
+| Outcome | Triggering conditions |
+|---|---|
+| `ABORTED` | *(never produced at this level)* |
+| `TERMINATED` | *(never produced at this level)* |
+| `ERRORED` | A `None` value reached the check — typically a driver returned `None` silently (e.g. `verify("vout", None)`) |
+| `FAILED` | Value checked against its limit and out of range |
+| `PASSED` | Value checked against its limit and in range |
+| `DONE` | Value recorded with no limit to judge against — e.g. `measure(...)` with no limit |
+| `SKIPPED` | *(never produced at this level)* |
+| `None` | Default; row constructed but limit check not invoked |
+
+An exception in a called function (e.g. driver raises a VISA timeout) does **not** produce an ERRORED measurement — it produces no measurement record at all, and the enclosing step lands ERRORED instead.
+
+### Vector level
+
+`TestVector.outcome` defaults to `None`.
+
+| Outcome | Triggering conditions |
+|---|---|
+| `ABORTED` | *(never produced at this level)* |
+| `TERMINATED` | *(never produced at this level)* |
+| `ERRORED` | Rolls up from an ERRORED measurement; or the vector body raised a non-`AssertionError` exception |
+| `FAILED` | Rolls up from a FAILED measurement; or the vector body raised `AssertionError`; or `VectorBuilder.fail(...)` was called (results-API path) |
+| `PASSED` | Rolls up from a PASSED measurement |
+| `DONE` | Rolls up from a recorded-but-unjudged measurement |
+| `SKIPPED` | `VectorBuilder.skip(...)` explicit call (`TesterKitClient` builder path only — not produced by the runtime cascade) |
+| `None` | Default; vector ran but recorded nothing and didn't raise |
+
+### Step level
+
+A step's outcome rolls up from its measurements and from how the test body ended; once a worse outcome is set, a less-severe later result doesn't weaken it.
+
+| Outcome | Triggering conditions |
+|---|---|
+| `ABORTED` | *(never produced at this level — see Run level)* |
+| `TERMINATED` | Operator stopped the run (Ctrl-C or SIGTERM) |
+| `ERRORED` | Test body raised any non-`AssertionError`, non-skip exception; setup or teardown raised any non-skip exception; or rolls up from an ERRORED measurement or vector |
+| `FAILED` | Test body raised `AssertionError`; or rolls up from a FAILED measurement or vector; or `StepBuilder.fail(...)` was called (results-API path) |
+| `PASSED` | Test body exited cleanly AND verdict intent was recorded for this step; or rolls up from a PASSED measurement with nothing worse |
+| `DONE` | Test body exited cleanly with **no** verdict intent (no asserts ran, no measurements with limits) |
+| `SKIPPED` | Test body raised `pytest.skip.Exception` (`pytest.skip(...)`, `@pytest.mark.skip`, `skipif`); setup-phase skip exception (e.g. `skipif` from a fixture); `StepBuilder.skip(...)` explicit call |
+| `None` | Default; the step opened but never ran (e.g. an upstream failure stopped the session) |
+
+### Run level
+
+A run's outcome rolls up through every measurement and step; it's written to the run's saved record when the run ends.
+
+| Outcome | Triggering conditions |
+|---|---|
+| `ABORTED` | The run was killed before it finalized; TesterKit records the unfinished run as ABORTED automatically. A non-pytest runner can also mark a run ABORTED via the results API. |
+| `TERMINATED` | Rolls up from any step that landed TERMINATED |
+| `ERRORED` | Rolls up from any ERRORED step or measurement |
+| `FAILED` | Rolls up from any FAILED step or measurement |
+| `PASSED` | Rolls up from PASSED step(s) with nothing worse anywhere |
+| `DONE` | Rolls up from step(s) that ran cleanly without verdict intent |
+| `SKIPPED` | Cascade rollup where the only contained outcomes were SKIPPED |
+| `None` | Default; a run that ended without any outcome being set. A run killed before it finalizes is recorded as ABORTED instead (see top row). |
+
+### Multi-UUT site orchestrator
+
+In a multi-UUT run, each site (one DUT) gets its own result; the session's overall result is the worst across sites.
+
+| Site result | Triggering condition |
+|---|---|
+| `errored` | The site's run never finished cleanly (killed, orphaned) |
+| `failed` | The site's run finished with a failure |
+| `passed` | The site's run finished clean |
+
+Each DUT still records its own detailed outcome from the per-run tables above. For the operational guide, see [multi-UUT testing](../../how-to/execution/multi-uut-testing.md).
+
+## See also
+
+- [Step hierarchy](step-hierarchy.md) — the measurement / vector / step / run levels the cascade walks
+- [Step manifest](step-manifest.md) — how `outcome IS NULL` rows show as "Never Ran" in finalized runs
+- [Event log](../data/event-log.md) — `RunEnded` / `StepEnded` events that carry the cascade-derived outcome
+- [Multi-UUT testing](../../how-to/execution/multi-uut-testing.md) — site orchestrator outcomes in practice
+- [Limits](../../how-to/execution/limits.md) — how a measurement gets a limit attached (the trigger for measurement-level PASSED/FAILED and for step-level verdict intent)
+- [Models reference](../../reference/data/models.md) — `Outcome` enum source-of-truth and field tables for `Measurement` / `TestVector` / `TestStep` / `TestRun`
+- [Parquet schema](../../reference/data/parquet-schema.md) — column-level definitions of `run_outcome`, `step_outcome`, `measurement_outcome`

testerkit/_docs/concepts/execution/step-hierarchy.md

@@ -0,0 +1,170 @@
+# Step Hierarchy — runs, steps, vectors, measurements
+
+This page is the single reference for TesterKit's run-data hierarchy: what each level represents, how the levels nest, and how they're identified in the event log and the materialized tables. Pair it with [Outcomes](outcomes.md) for what each level's verdict means and where it gets set, and [Step Manifest](step-manifest.md) for the planned-vs-executed reconciliation.
+
+## The hierarchy
+
+```
+TestRun                              ← one per pytest session
+└── Step                             ← class container (one per class; one per outer iteration if the class is swept)
+    └── Step                         ← test method (one per pytest item)
+        └── TestVector               ← one per inner iteration (1 for normal swept tests; N for `vectors`-fixture tests)
+            └── Measurement          ← one per `measure` / `verify` call
+```
+
+Each level emits its own event in the run log. Each level rolls its outcome up to the next level via the severity-max ladder (see [Outcomes](outcomes.md)). `verify` and `measure` are pytest [fixtures](../../reference/pytest/fixtures.md); `vectors` is the [self-loop fixture](../../how-to/execution/vector-expansion.md).
+
+## What each level is
+
+### TestRun
+
+One run = one pytest session. Wraps a session_id, run_id, uut_serial_number, station, fixture, operator — all of it is the run-level context.
+
+Events: `RunStarted` at session start, `RunEnded` at session end. The session also emits `SessionStarted` / `SessionEnded`, but those are session-scoped (could span multiple runs in a multi-site harness).
+
+### Step — the unit of "one thing the test did"
+
+A step is a named, ordered unit. Two kinds, but they share one event type and one shape:
+
+1. **Container step (class container).** Synthesized by the pytest plugin when execution enters a test class. Methods inside the class push beneath it on the step stack. When the class is swept (class-level `@pytest.mark.testerkit_sweeps`), one container step is emitted **per outer iteration** — each with its own `vector_index` and `inputs={outer_param: value}`.
+
+2. **Method step.** One per pytest-collected item. The test function's body is the step's work.
+
+Whether a step is a class container or a method isn't a separate flag — it's implied by the nesting: a step is a container when at least one other step's `step_path` starts with it as a prefix.
+
+Events: `StepStarted` when the step opens, `StepEnded` when it closes. `step_path` encodes the full hierarchy.
+
+Identity: `(step_path, vector_index)` is unique per executed step instance within a run. For a method run as 3 parametrize variants, you get 3 `StepStarted` events with the same `step_path` (e.g., `"TestPower/test_voltage"`) and distinct `vector_index` 0/1/2.
+
+### TestVector — one inner iteration
+
+For normal swept tests (one pytest item per variant), each step has exactly one TestVector. The vector carries the sweep parameter values (`vin=5.0`) and ends up as that step's `inputs`.
+
+For tests using the `vectors` fixture, the test body iterates the matrix itself. Each iteration appends a new TestVector to the step. The step has one outer identity, but N internal vectors and N measurements.
+
+### Measurement — one recorded value
+
+A `measure("vin_voltage", 3.30)` or a `verify(...)` call. Carries the value, units, limit, characteristic_id, and uut_pin / instrument_resource / fixture_connection traceability fields.
+
+Events: `MeasurementRecorded`. Carries the full effective `inputs` dict — outer step params **merged with** the current vector's inner params — so analytics queries can filter on either dimension without joining back to the step.
+
+## Identity fields
+
+| Field | Where it's set | Used for |
+|---|---|---|
+| `step_path` | Built from the chain of enclosing steps (e.g., `TestPower/test_voltage`) | Hierarchical identity; parent derivable as `step_path.rsplit("/", 1)[0]` |
+| `step_index` | Assigned to each step before the run, when tests are collected | Sequence-relative ordering within a parent bucket |
+| `vector_index` | Pre-assigned at collection time for swept items; 0 for plain steps | Distinguishes sweep variants of the same logical step |
+| `step_name` | The function or class name | Display |
+| `inputs` (on `StepStarted`) | Outer sweep params from `callspec.params` | Step row's commanded conditions |
+| `inputs` (on `MeasurementRecorded`) | Outer step inputs + active vector params | Full per-row sweep context |
+
+`(step_path, vector_index)` is the per-step-instance identity. For container steps it's `(class_name, outer_iteration_index)`; for method steps it's `(class_name/method_name, per-method-counter)`.
+
+## Worked example — swept class with method-level inner sweep
+
+```python
+@pytest.mark.testerkit_sweeps([{"voltage": [1, 2, 3]}])      # class-level → outer
+class TestPower:
+    def test_warmup(self, voltage, measure):
+        measure("vin_warmup", voltage)
+
+    @pytest.mark.testerkit_sweeps([{"current": [4, 5, 6]}])  # method-level → inner
+    def test_load(self, voltage, current, measure):
+        measure("vout_load", voltage * 1.1)
+
+    def test_cooldown(self, voltage, measure):
+        measure("vin_cooldown", 0)
+```
+
+Event stream (condition-first):
+
+```
+StepStarted ("TestPower",            vi=0, inputs={voltage:1})
+  StepStarted ("test_warmup",        vi=0, inputs={voltage:1})
+  MeasurementRecorded("vin_warmup",                inputs={voltage:1})
+  StepEnded   ("test_warmup",        vi=0)
+  StepStarted ("test_load",          vi=0, inputs={voltage:1, current:4})
+  MeasurementRecorded("vout_load",                 inputs={voltage:1, current:4})
+  StepEnded   ("test_load",          vi=0)
+  StepStarted ("test_load",          vi=1, inputs={voltage:1, current:5})
+  ...
+  StepStarted ("test_cooldown",      vi=0, inputs={voltage:1})
+  ...
+StepEnded   ("TestPower",            vi=0, outcome=<rolled-up>)
+StepStarted ("TestPower",            vi=1, inputs={voltage:2}, ...)
+...
+```
+
+3 container iterations × (3 methods, with `test_load` unrolling to 3 inner variants) = 15 method `StepStarted` events under 3 container iterations.
+
+`step_index` for the container is 0 (root-level), for each method is 0/1/2 within the `TestPower` class bucket — so `(step_index=1, vector_index=0)` uniquely points to `test_load[voltage=1, current=4]`.
+
+## `vectors` fixture — one step, many in-body vectors (Mode 2)
+
+When the method uses the `vectors` fixture, pytest sees ONE item per outer iteration. The step has ONE outer identity (matching the outer-iteration position) and N in-body vector iterations, each bracketed by `VectorStarted` / `VectorEnded` events:
+
+```python
+@pytest.mark.testerkit_sweeps([{"voltage": [1, 2, 3]}])
+class TestPower:
+    @pytest.mark.testerkit_sweeps([{"current": [4, 5, 6]}])
+    def test_load(self, voltage, vectors, measure):
+        for v in vectors:
+            measure("vout", voltage * v["current"])
+```
+
+Event stream:
+
+```
+StepStarted   ("test_load", vi=0, inputs={voltage:1})         # outer position
+VectorStarted ("test_load", vi=0, inputs={voltage:1, current:4})
+MeasurementRecorded("vout", inputs={voltage:1, current:4})
+VectorEnded   ("test_load", vi=0, outcome=passed)
+VectorStarted ("test_load", vi=1, inputs={voltage:1, current:5})
+MeasurementRecorded("vout", inputs={voltage:1, current:5})
+VectorEnded   ("test_load", vi=1, outcome=passed)
+VectorStarted ("test_load", vi=2, inputs={voltage:1, current:6})
+MeasurementRecorded("vout", inputs={voltage:1, current:6})
+VectorEnded   ("test_load", vi=2, outcome=passed)
+StepEnded     ("test_load", vi=0, outcome=<rolled-up>)
+```
+
+`StepStarted.vector_index` and `StepEnded.vector_index` agree (the outer iteration position). Each `VectorStarted`/`VectorEnded` pair is the in-body analog of a Mode-1 step boundary — it brackets one iteration's work and carries the full effective `inputs` for that iteration. In the materialized parquet, each `VectorStarted`/`VectorEnded` pair produces one `record_type = 'vector'` row; `record_type = 'step'` rows are NOT emitted for Mode-2 in-body iterations (only for the enclosing step itself).
+
+## Outcome rollup chain
+
+```
+measurement.outcome
+   ↓ escalate (severity-max ladder)
+TestVector.outcome  (per inner iteration)
+   ↓ escalate
+StepEnded.outcome   (the method's aggregate verdict)
+   ↓ escalate (the container takes the worst verdict among its children)
+container.outcome   (the class iteration's verdict, on its StepEnded)
+   ↓ escalate
+TestRun.outcome     (the run's overall verdict, on RunEnded)
+```
+
+Severity ladder: `ABORTED > TERMINATED > ERRORED > FAILED > PASSED > DONE > SKIPPED`. Worst wins at every level. Full mapping and producer sites in [Outcomes](outcomes.md).
+
+## Materialized record identity
+
+The at-rest per-run parquet contains three `record_type` values: `run`, `step`, and `vector` (querying also gives you a `measurement` record type, expanded from the measurements inside each vector). Container steps and method steps share the `step` record type — tell them apart by `step_path` depth. The parent of any step is derivable as `step_path.rsplit("/", 1)[0] if "/" in step_path else ""`:
+
+- no `/` in `step_path` → root-level (run-level test functions or class containers)
+- one `/` in `step_path` → method directly under a class container (e.g. `TestPower/test_voltage`)
+- two or more `/` → nested step (via `harness.step()` self-loops)
+
+`vector` records appear only for Mode-2 in-body iterations (`vectors` fixture). They key on `(step_path, vector_index, retry)` and sit below their enclosing `step` row. Mode-1 steps (parametrize / single) have no `vector` rows — the `step` row already carries the vector data.
+
+`MAX(severity)` over `step` rows sharing a `step_path` aggregates "did this class ever fail in this run" across its iterations. See the [results storage reference](../../reference/data/parquet-schema.md) for the full column schema.
+
+
+## See also
+
+**Related quadrants:**
+
+- [How-to → Execution](../../how-to/execution/index.md) — how-to entry point for this category
+- [Reference](../../reference/index.md) — reference entry point for this category
+- [Integration](../../integration/index.md) — integration entry point for this category
+- [Tutorial](../../tutorial/index.md) — tutorial entry point for this category

testerkit/_docs/concepts/execution/step-manifest.md

@@ -0,0 +1,115 @@
+# Step Results & the Step Manifest
+
+The step manifest is the full list of planned steps for a run — including the ones that never ran. Step results give a complete view of every planned test step (early abort, `--maxfail`, skip markers).
+
+## Why record planned steps
+
+Without explicit step records, you only know about steps that actually executed. If a run aborts after step 3 of 10, the Parquet file would only have 3 rows of evidence. There's no record that 7 other steps were planned but never ran.
+
+That matters for:
+
+- **Yield analysis** — A 3/3 pass result is not the same as a 3/10 partial result.
+- **Coverage tracking** — Which steps are consistently skipped or never reached?
+- **Compliance** — Auditors need to know the full test plan, not just what ran.
+
+## StepsDiscovered event
+
+The `StepsDiscovered` event fires after instruments connect but before any steps execute. It carries the complete list of pytest-collected items:
+
+```python
+class StepsDiscovered(EventBase):
+    event_type: Literal["test.steps_discovered"] = "test.steps_discovered"
+    items: list[dict[str, str | int | None]] = Field(default_factory=list)
+```
+
+Each item in `items` contains the pytest identity and collection-time manifest data:
+
+| Field | Description |
+|-------|-------------|
+| `node_id` | The step's pytest address — file plus test name, e.g. `tests/test_power.py::test_voltage` |
+| `file` | Source file path |
+| `module` | Python module name |
+| `class_name` | Test class name (if any) |
+| `function` | Test function name |
+| `markers` | pytest markers on the item |
+| `step_path` | Hierarchical step identifier, matching executed step events |
+| `step_index` | Position within the parent sequence |
+| `vector_index` | 0-based position within the sweep expansion |
+| `vector_count_planned` | Total vectors collected for this logical step — drives placeholder row synthesis for unrun vectors |
+
+## How it flows
+
+When the run ends, TesterKit writes one row per planned step — executed steps with their real outcome and timing, and a placeholder row for any step that never started.
+
+## Storage
+
+There is **one parquet file per run**. Run, step, and vector records share the same file; the [`record_type`](../../reference/data/parquet-schema.md) column says which kind each row is. Measurements are stored inside each vector row:
+
+```
+<data_dir>/runs/{date}/
+└── {timestamp}_{serial}.parquet          # All rows for one run
+   ├── record_type='run'                  # exactly one row, run-level metadata
+   ├── record_type='step'                 # one row per (step_path, vector_index, retry)
+   └── record_type='vector'               # one row per execution; nests the measurements list
+```
+
+Key step-row columns (full list in [Parquet schema](../../reference/data/parquet-schema.md)):
+
+- `step_name`, `step_path`, `step_index`, `step_node_id`
+- `step_started_at`, `step_ended_at`, `step_vector_count`
+- `step_outcome` (rollup), `vector_outcome` (per vector), `run_outcome` (run-wide)
+- Run context repeated on every step row (so a step row is self-contained): `run_id`, `uut_serial_number`, `station_id`, `session_id`
+
+## Steps that never ran {#never-ran}
+
+A run can finish before every planned step executes — an early abort, `--maxfail`, or a skip. TesterKit still records those steps so the run shows the full plan, not just what ran. Each unrun step gets a placeholder row with a blank outcome and no start/end time (query hint: `step_outcome IS NULL`).
+
+- `step_outcome`: NULL
+- Timing fields: NULL
+- Step identity columns: populated from the collected item
+
+Every run thus has a complete picture — executed steps with real data, plus placeholder rows for the rest.
+
+## Querying step results
+
+With DuckDB:
+
+```sql
+-- Step summary for one run
+SELECT step_name, step_outcome, step_started_at, step_ended_at
+FROM read_parquet('data/runs/**/*.parquet')
+WHERE record_type = 'step'
+  AND run_id = 'abc123'
+ORDER BY step_index;
+
+-- Find steps that are frequently skipped or never run
+SELECT step_name,
+       COUNT(*) AS total,
+       SUM(CASE WHEN step_outcome IS NULL THEN 1 ELSE 0 END) AS never_ran
+FROM read_parquet('data/runs/**/*.parquet')
+WHERE record_type = 'step'
+GROUP BY step_name
+HAVING never_ran > 0;
+```
+
+From Python (via `RunStore`):
+
+```python
+from testerkit.data.run_store import RunStore
+
+steps = RunStore().get_steps("abc123")
+for step in steps:
+    print(f"{step['step_name']}: {step['outcome']}")
+```
+
+From the event store:
+
+```python
+store.events(event_type="test.steps_discovered", session_id=sid)
+```
+
+## See also
+
+- [Event log](../data/event-log.md) — how events get to Parquet
+- [Parquet schema](../../reference/data/parquet-schema.md) — full column list
+- [Data stores](../data/data-stores.md) — EventStore, ChannelStore, FileStore, RunStore

testerkit/_docs/concepts/index.md

@@ -0,0 +1,40 @@
+# Concepts
+
+Why TesterKit is built the way it is. Concepts are grouped by what you're trying to understand. Each group has its own index; jump to the page you care about, or read top-to-bottom for the full mental model.
+
+## Overview
+
+[Overview index →](overview/index.md). The platform's mental model from above — what TesterKit is, what it isn't, and why pytest sits underneath.
+
+- [Architecture](overview/architecture.md) — system-level view of parts, stations, fixtures, and runs
+- [Platform vs framework](overview/platform-vs-framework.md) — what the platform owns vs what the runner owns
+- [pytest](overview/pytest.md) — why a hardware-test platform rides on pytest instead of its own runner
+- [AI integration](overview/ai-integration.md) — what the MCP surface buys you and where it draws the line
+
+## Configuration
+
+[Configuration index →](configuration/index.md). The UUT-to-instrument model — the YAML entities you author once and re-use across runs.
+
+- [Parts](configuration/parts.md) — what you're testing
+- [Stations](configuration/stations.md) — where you test
+- [Capabilities](configuration/capabilities.md) — what instruments can do, how matching pairs them with part characteristics
+- [Fixtures](configuration/fixtures.md) — pin-to-instrument mapping
+
+## Execution
+
+[Execution index →](execution/index.md). How a test run unfolds — the step model, the outcome ladder, what each step records.
+
+- [Step hierarchy](execution/step-hierarchy.md) — how test classes, methods, and vectors nest
+- [Step manifest](execution/step-manifest.md) — what each step records
+- [Outcomes](execution/outcomes.md) — skipped / done / passed / failed / errored / terminated / aborted severity ladder
+
+## Data
+
+[Data index →](data/index.md). Where the run data lives and how the platform stays consistent across processes.
+
+- [Event log](data/event-log.md) — the durable record of every run
+- [Event sourcing](data/event-sourcing.md) — append-only event log as the source of truth
+- [Sessions](data/sessions.md) — connect-to-disconnect observation windows for instrument use
+- [Three verbs](data/three-verbs.md) — `observe` / `verify` / `stream`; routing a value to the right store by shape
+- [Data stores](data/data-stores.md) — EventStore, ChannelStore, FileStore, RunStore; on-disk layout, data_dir resolution, schema-evolution contract
+- [Flight streaming](data/flight-streaming.md) — cross-process data access via Arrow Flight

testerkit/_docs/concepts/overview/ai-integration.md

@@ -0,0 +1,57 @@
+# AI integration
+
+The platform exposes its operations as **typed tool calls** an AI client can drive — over an MCP server (the open protocol AI assistants use to call external tools) for clients that speak it, or via the CLI for any agent with a terminal: discovery, capability matching, run launching, results query, config validation, datasheet extraction, test scaffolding. It does not embed an LLM client itself — the user brings their own assistant (Claude Code, Cursor, Cline, Claude Desktop, Copilot CLI, …), and TesterKit is the typed surface the assistant drives.
+
+The cost of any structured platform is the upfront encoding work — getting existing instruments and parts encoded as catalog and spec YAML. AI integration is the platform's answer to that cost. An agent reads a datasheet PDF, drafts the catalog YAML, lands it as a file you can diff and review. The structured approach becomes worth adopting because the encoding work shrinks from hours to minutes.
+
+The rest of this page documents the integration's boundary: what changes for the user, why it's safe to lean on without losing visibility, what it deliberately doesn't do, and the adoption ramp.
+
+## What changes for the user
+
+| You're doing | Without AI | With AI |
+|---|---|---|
+| Onboarding a new instrument | Read datasheet, write capability YAML by hand from scratch (hours) | The `testerkit-datasheets` skill produces a reviewable draft; you correct (minutes) |
+| Starting a new part | Read datasheet, write spec YAML, decide instruments, wire station, scaffold test | The `testerkit-datasheets` skill drafts all of it, with approval gates at every phase |
+| Routine test authoring | Look up the spec in the YAML, write the `verify()` calls | (largely unchanged — you stay in pytest) |
+| Run + analyze | (unchanged — pytest runs, results write to parquet) | (unchanged — pytest runs, results write to parquet) |
+
+The agent's output is **a starting point you review**, not a finished artifact. Every YAML lands as a file in your repo. Every test is a `def test_*()` you can edit. If the agent gets a capability wrong, you spot it in the diff and push back. If you delete the AI from the loop tomorrow, the project keeps working — the files don't depend on the AI ever running again.
+
+## What keeps the integration safe to lean on
+
+Three properties of the platform make the AI surface safe to use without losing visibility:
+
+1. **Everything is a file.** Parts, stations, fixtures, profiles, sequences, results — all YAML or Parquet. An AI editing a part spec produces a diff you can review the same way you'd review a colleague's PR. There's no opaque database for the AI to mutate.
+
+2. **Tool calls, not LLM calls.** TesterKit does not embed an OpenAI / Anthropic / Google client. The AI tooling drives TesterKit from outside — over MCP for clients that speak it, via the CLI for agents with a terminal. You bring your own AI client; TesterKit exposes the operations.
+
+3. **Operator-in-the-loop by design.** The `testerkit-datasheets` skill (see [skills reference](../../reference/overview/skills.md)) is a gated, multi-phase pipeline that STOPS at an approval checkpoint after every phase — part spec, instrument match, station wiring, test scaffold. "Here is the part spec I extracted, ok to save?" — you say yes or you edit first.
+
+This matters most when the AI gets it wrong. A misread accuracy spec or a wrongly assigned pin reads as a YAML diff or a `verify()` line — you spot it, push back, iterate. Compare to a workflow where the AI commits to a database: the same mistake disappears under "the system says so."
+
+## Anti-goals
+
+A few things AI integration in TesterKit deliberately does **not** try to do:
+
+- **Run the test.** Test execution is pytest. The AI can scaffold a test file, but the test runs the same way it would without the AI.
+- **Decide pass/fail.** Limit checking, traceability, capability matching — all in code. The AI proposes; the deterministic platform decides.
+- **Hide the prompt.** All shipped skills ([skills reference](../../reference/overview/skills.md)) are plain `SKILL.md` markdown files you can read, audit, fork, or ignore.
+- **Lock you to one model.** The `testerkit-datasheets` skill's sub-agent prompts each name a recommended **tier** (e.g. high-capability reasoning for schema-correct YAML emission) but no hard-coded model name. Pick whichever your client supports.
+
+## Adoption ramp
+
+Three adoption levels — pick whichever matches the user's trust level today:
+
+1. **Just tool calls.** Either register the MCP server (`testerkit setup claude-code` and friends) for clients that speak MCP, or have your agent invoke the CLI directly (`testerkit runs`, `testerkit show`, `testerkit discover`, `testerkit metrics`, ...). Drive operations conversationally without any skill matching. "Add a 3.3V output rail to the part spec" — the agent calls `testerkit_project(action="save", ...)` (or edits the YAML file directly and runs `testerkit validate`) and shows the diff.
+
+2. **The `testerkit-datasheets` skill as a starting draft.** Point it at a new part's datasheet PDF. Treat the YAML files it produces as a first draft, then hand-edit. Often faster than starting from a blank file.
+
+3. **Full datasheet → tests pipeline.** Run the `testerkit-datasheets` skill end-to-end — instrument catalog entries, part spec, station wiring, and test scaffold, in one gated pipeline. Operator approval gates at every phase. See [how-to/datasheet-to-test](../../how-to/catalog/datasheet-to-test.md) for the walkthrough.
+
+## See also
+
+- [How-to: AI-assisted test development via MCP](../../how-to/overview/mcp-integration.md) — registering the MCP server with each supported AI client
+- [How-to: datasheet-to-test walkthrough](../../how-to/catalog/datasheet-to-test.md) — the `testerkit-datasheets` skill end-to-end
+- [Reference: skills](../../reference/overview/skills.md) — the 11 Agent Skills, MCP tools, and CLI
+- [Reference: MCP server + HTTP API](../../reference/runtime/api.md) — the operations AI clients call over MCP
+- [Reference: CLI](../../reference/cli.md) — the operations AI clients call via the terminal