pycontrol-core 3.0.0b2__tar.gz → 3.0.0b4__tar.gz

{pycontrol_core-3.0.0b2 → pycontrol_core-3.0.0b4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pycontrol-core
-Version: 3.0.0b2
+Version: 3.0.0b4
 Summary: Sans-IO host library for the pyControl behavioural-experiment framework.
 Project-URL: Homepage, https://github.com/pyControl/core
 Author: pyControl contributors
@@ -91,11 +91,15 @@ uv sync --extra test --extra lint
 Run checks:
 
 ```bash
-uv run pytest
+uv run python tools/run_tests.py -q
 uv run ruff check src tests
+uv run ruff format --check src tests
 uv run mypy
 ```
 
+The test runner enforces a process deadline and reports thread traces if a test
+hangs. See [Testing](docs/testing.md) for focused checks and hardware limits.
+
 ## Create A Workspace
 
 Most lab-authored files live in a workspace, not inside this package.
@@ -110,13 +114,9 @@ The workspace contains:
 
 | Path | Purpose |
 | --- | --- |
-| `tasks/` | Task files uploaded to the board. |
+| `tasks/` | Task files uploaded to the board, each with its host-side addons (task and experiment handlers, GUI plotters and controls) in its task folder ([task-folders.md](docs/task-folders.md)). |
 | `hardware_definitions/` | Rig wiring files uploaded as `hardware_definition.py`. |
 | `devices/` | Workspace device classes uploaded when referenced. |
-| `addons/task_handlers/` | Host-side task automation. |
-| `addons/experiment_handlers/` | Whole-experiment automation. |
-| `addons/plotters/` | Optional GUI plotters. |
-| `addons/controls/` | Optional GUI control panels. |
 | `experiments/*.json` | Multi-subject experiment definitions. |
 | `settings.json` / `setups.json` | Workspace settings and setup registry. |
 | `data/` | Default output folder for native `.pycontrol.zip` run bundles. |
@@ -126,7 +126,7 @@ For a full walkthrough, use [getting-started.md](docs/getting-started.md).
 ## Run A Task
 
 ```bash
-pycontrol run --setup BoxA --task examples/blinker \
+pycontrol run --setup BoxA --task examples/blinker/blinker-pattern-h7 \
   --subject mouse42 --duration 10
 ```
 
@@ -197,6 +197,9 @@ See [public-api.md](docs/public-api.md) for the supported import surface.
 Start with [Understanding one run](docs/core-walkthrough.md), then use the
 references below. [Release acceptance](docs/release-acceptance.md) distinguishes
 host verification from the hardware evidence required for lab deployment.
+Ownership and completion rules are detailed in
+[Threading and state](docs/threading-and-state.md). The tradeoffs to
+preserve are recorded in [Maintainability decisions](docs/maintainability.md).
 
 Orientation and operation:
 

{pycontrol_core-3.0.0b2 → pycontrol_core-3.0.0b4}/README.md

@@ -49,11 +49,15 @@ uv sync --extra test --extra lint
 Run checks:
 
 ```bash
-uv run pytest
+uv run python tools/run_tests.py -q
 uv run ruff check src tests
+uv run ruff format --check src tests
 uv run mypy
 ```
 
+The test runner enforces a process deadline and reports thread traces if a test
+hangs. See [Testing](docs/testing.md) for focused checks and hardware limits.
+
 ## Create A Workspace
 
 Most lab-authored files live in a workspace, not inside this package.
@@ -68,13 +72,9 @@ The workspace contains:
 
 | Path | Purpose |
 | --- | --- |
-| `tasks/` | Task files uploaded to the board. |
+| `tasks/` | Task files uploaded to the board, each with its host-side addons (task and experiment handlers, GUI plotters and controls) in its task folder ([task-folders.md](docs/task-folders.md)). |
 | `hardware_definitions/` | Rig wiring files uploaded as `hardware_definition.py`. |
 | `devices/` | Workspace device classes uploaded when referenced. |
-| `addons/task_handlers/` | Host-side task automation. |
-| `addons/experiment_handlers/` | Whole-experiment automation. |
-| `addons/plotters/` | Optional GUI plotters. |
-| `addons/controls/` | Optional GUI control panels. |
 | `experiments/*.json` | Multi-subject experiment definitions. |
 | `settings.json` / `setups.json` | Workspace settings and setup registry. |
 | `data/` | Default output folder for native `.pycontrol.zip` run bundles. |
@@ -84,7 +84,7 @@ For a full walkthrough, use [getting-started.md](docs/getting-started.md).
 ## Run A Task
 
 ```bash
-pycontrol run --setup BoxA --task examples/blinker \
+pycontrol run --setup BoxA --task examples/blinker/blinker-pattern-h7 \
   --subject mouse42 --duration 10
 ```
 
@@ -155,6 +155,9 @@ See [public-api.md](docs/public-api.md) for the supported import surface.
 Start with [Understanding one run](docs/core-walkthrough.md), then use the
 references below. [Release acceptance](docs/release-acceptance.md) distinguishes
 host verification from the hardware evidence required for lab deployment.
+Ownership and completion rules are detailed in
+[Threading and state](docs/threading-and-state.md). The tradeoffs to
+preserve are recorded in [Maintainability decisions](docs/maintainability.md).
 
 Orientation and operation:
 

{pycontrol_core-3.0.0b2 → pycontrol_core-3.0.0b4}/docs/addon-settings.md

@@ -6,20 +6,22 @@ Use addon settings for values that should not live in a task file or experiment
 
 ## Supported Addon Types
 
-Settings can be declared by:
+Settings can be declared by the task handlers, experiment handlers and task
+plotters in a [task folder](task-folders.md).
 
-- task handlers in `addons/task_handlers/*.py`
-- experiment handlers in `addons/experiment_handlers/*.py`
-- task plotters in `addons/plotters/*.py`
-
-The settings are stored separately by addon type and file stem:
+The settings are stored in the workspace `settings.json`, never in the task
+folder, separately by addon type and keyed by the addon's path relative to
+`tasks/` (without `.py`), so same-named addons in two folders keep separate
+settings:
 
 ```python
-settings.addons.task_handlers["run_notify"]
-settings.addons.experiment_handlers["training_summary"]
-settings.addons.plotters["SequencePlotter"]
+settings.addons.task_handlers["sequence/run_notify"]
+settings.addons.experiment_handlers["training/training_summary"]
+settings.addons.plotters["sequence/sequence_plotter"]
 ```
 
+The GUI's built-in plot keeps the key `settings.addons.plotters["DefaultPlotter"]`.
+
 ## Declaring Settings
 
 Add a `SETTINGS` tuple to the addon module. Each entry should be made with `BoolSetting`, `IntSetting`, or `StringSetting`.
@@ -83,6 +85,12 @@ StringSetting("bot_token", "Bot token", secret=True)
 
 Secret fields are masked in the Settings dialog, but they are still saved in the workspace `settings.json`.
 
+A run bundle records the settings each task handler and experiment handler
+was configured with when it loaded, minus the secret ones, whose names it
+lists under `redacted_settings`. Which names are secret is read from the
+`SETTINGS` of the loaded code (module level, or on a class the module
+defines); when that cannot be read, every value of the addon is redacted.
+
 All setting helpers accept `hint="..."`. When provided, the Settings dialog shows
 the hint as a tooltip when the user hovers over the setting label.
 
@@ -91,14 +99,14 @@ the hint as a tooltip when the user hovers over the setting label.
 At runtime, the addon receives two attributes:
 
 ```python
-self.addon_settings   # dict for this addon file stem
+self.addon_settings   # dict for this addon, keyed by its path relative to tasks/
 self.workspace_settings   # full workspace Settings object
 ```
 
-For a task handler file named `addons/task_handlers/run_notify.py`, settings are passed from:
+For a task handler file `tasks/sequence/run_notify.py`, settings are passed from:
 
 ```python
-self.workspace_settings.addons.task_handlers["run_notify"]
+self.workspace_settings.addons.task_handlers["sequence/run_notify"]
 ```
 
 and are usually read through:
@@ -112,14 +120,14 @@ enabled = self.addon_settings.get("slack_enabled", False)
 Task-addon settings are configured for `pycontrol run`, `pycontrol
 experiment`, and GUI-run sessions before callbacks run.
 
-For an experiment handler file named `addons/experiment_handlers/training_summary.py`, settings are passed from:
+For an experiment handler file `tasks/training/training_summary.py`, settings are passed from:
 
 ```python
-self.workspace_settings.addons.experiment_handlers["training_summary"]
+self.workspace_settings.addons.experiment_handlers["training/training_summary"]
 ```
 
 Task plotters use the same stored settings through
-`settings.addons.plotters[stem]`. In the GUI, `PlotterHost` passes that
+`settings.addons.plotters["<folder>/<stem>"]`. In the GUI, `PlotterHost` passes that
 dict to custom plotter constructors as `addon_settings` when the constructor
 accepts that keyword argument.
 
@@ -169,7 +177,7 @@ class RunNotify(TaskHandler):
         self._notify.close()
 ```
 
-Select this addon by either naming it after the task, or adding this to the task file:
+Put the file in the task's folder and select it in the task file:
 
 ```python
 v.handler = "run_notify"
@@ -197,19 +205,18 @@ class TelegramExperimentNotify(ExperimentHandler):
         print(f"Would notify {chat_id} that {context.experiment_name} finished.")
 ```
 
-Select this addon with the experiment setup `Experiment handler` picker.
+Put the file in the folder of the experiment's task and select it with the
+experiment setup `Experiment handler` picker.
 
 ## Settings Dialog Discovery
 
-The Settings dialog scans addon folders when it opens:
-
-```text
-addons/task_handlers/*.py
-addons/experiment_handlers/*.py
-addons/plotters/*.py
-```
-
-Files whose names start with `_` are skipped. Files with no valid `SETTINGS` are still valid addons, but they do not get a Settings dialog group.
+When it opens, the Settings dialog lists the task handlers, experiment
+handlers and plotters of every task folder. They are found statically, by a
+class deriving from `TaskHandler`, `ExperimentHandler` or `TaskPlotter`
+(`pycontrol.workspace.classify_task_folder`), then imported to read
+`SETTINGS`. Files whose names start with `_` are helpers and are skipped.
+Files with no valid `SETTINGS` are still valid addons, but they do not get a
+Settings dialog group.
 
 Each configurable file page has a `Use defaults` button. It resets only the
 currently selected addon or plotter settings group to the defaults declared

{pycontrol_core-3.0.0b2 → pycontrol_core-3.0.0b4}/docs/architecture.md

@@ -2,12 +2,12 @@
 
 This document describes the implemented `pycontrol-core` architecture. It is the
 engine shared by the CLI, `pycontrol-gui`, tests, and downstream Python tools.
-For frozen compatibility contracts, see `contracts.md`, `wire-protocol.md`,
-`data-format.md`, and `threading-and-state.md`. For a guided, layered
-learning path through this architecture — from a one-paragraph mental model
-down to per-subsystem deep dives — see [core-walkthrough.md](core-walkthrough.md).
+Start with [core-walkthrough.md](core-walkthrough.md) for the reading order,
+ownership, and normal and failed runs. This page is the
+package map. [contracts.md](contracts.md) and [public-api.md](public-api.md)
+distinguish frozen compatibility contracts from provisional frontend APIs.
 
-The shortest mental model is:
+The acquisition path is:
 
 ```text
 Pyboard <-> transport bytes <-> sans-IO protocol <-> BoardSession
@@ -35,23 +35,27 @@ they are not installed into `pycontrol-core`.
 flowchart TD
     Frontends["Frontends: pycontrol CLI, pycontrol-gui, Python APIs"]
     Workspace["workspace/ + config/: discover tasks, setups, experiments, settings"]
-    Session["session/ BoardSession: FSM, REPL setup, run commands"]
+    Run["experiment/ Run: prepare, start, finish"]
+    Session["session/ BoardSession: lifecycle, REPL setup, run commands"]
     Pump["session/pump.py SessionPump: read run bytes"]
     Bus["bus/ SubscriberRegistry + typed events"]
     Protocol["protocol/: sans-IO decoder/encoder"]
-    Transport["transport/: Serial, Mock, Replay, Recording"]
+    Transport["transport/: Serial, Mock, Replay"]
     Board["Pyboard + vendored MicroPython framework"]
     Recording["recording/: NativeEventLogger, analog writer, console"]
     Provenance["provenance/: collect, hash, stage, finalize"]
     Bundles["bundles/: run and protocol ZIP formats"]
-    Experiment["experiment/: Supervisor, hooks"]
+    Experiment["experiment/: Supervisor + experiment hooks"]
     Addons["addons/: task/experiment APIs, controls specs, loaders"]
     Analysis["analysis/: read run bundles into Python data structures"]
     Firmware["firmware/: on-board framework, devices, profiles"]
 
     Frontends --> Workspace
-    Frontends --> Session
-    Workspace --> Session
+    Frontends --> Run
+    Frontends --> Experiment
+    Experiment --> Run
+    Workspace --> Run
+    Run --> Session
     Session --> Transport
     Transport <--> Board
     Transport --> Protocol
@@ -61,8 +65,8 @@ flowchart TD
     Session --> Bus
     Bus --> Recording
     Bus --> Addons
-    Bus --> Experiment
-    Recording --> Provenance
+    Run --> Provenance
+    Recording --> Bundles
     Provenance --> Bundles
     Bundles --> Analysis
     Firmware -. synced to .-> Board
@@ -79,9 +83,9 @@ flowchart TD
 | `provenance/` | File identity, hashing, run staging, protocol references, and run-bundle finalization. |
 | `bundles/` | Run bundle and protocol bundle ZIP readers/writers, manifests, and validation. |
 | `analysis/` | Post-run helpers for reading bundle event tables and session dataframes. |
-| `experiment/` | Headless multi-rig `Supervisor`, rig outcomes, and experiment hooks. |
+| `experiment/` | Shared single-run `Run`/`RunPlan`, task loading and connection helpers, headless `Supervisor`, rig outcomes and experiment hooks. |
 | `addons/` | Task handlers, experiment handlers, task-controls specs, addon loaders, and settings collection. |
-| `workspace/` | Workspace discovery, path resolution, data-directory resolution, addon lookup, and hwdef inspection. |
+| `workspace/` | Workspace discovery, path resolution, data-directory resolution, task detection, task folders and addon lookup, and hwdef inspection. |
 | `config/` | Pydantic models and load/save helpers for `settings.json`, `setups.json`, and experiments. |
 | `firmware/` | Vendored on-board pyControl framework, board profiles, breakouts, and device drivers. |
 | `integrations/` | Optional notification/storage primitives used by trusted workspace addons. |
@@ -97,8 +101,8 @@ services (`FirmwareService`, `StateMachineSetupService`,
 the stored results (`identity`, `firmware`, `device_sync`, `last_erase`).
 Services never reach back into the session; the one upward call is the link's
 `on_repl_reset` hook, which tells the session to forget its loaded task after a
-recovery soft-reset. The stable surface is the documented public API and
-contracts, not the internal file layout.
+recovery soft-reset. The compatibility tier of each documented surface is listed in
+[public-api.md](public-api.md); internal file layout is not a contract.
 
 ## Single-Run Pipeline
 
@@ -140,8 +144,9 @@ sequenceDiagram
     Run-->>Driver: RunResult(bundle_path | staging_dir + error)
 ```
 
-`Run.finish()` is the only code that turns a recording into a bundle and it
-never fails silently: a missing bundle is an ERROR log, a
+`Run.finish()` is the shared driver entry point that closes and packages a
+recording; lower-level bundle/recovery helpers implement those operations.
+It never reports a packaging failure as success: a missing bundle is an ERROR log, a
 `RunObserver.on_data_at_risk(staging_dir, error)` call, and a non-`ok`
 `RunResult` naming the staging directory. Drivers map that onto their own
 surface (CLI exit 1 + stderr, `RigOutcome.error`, GUI `error` signal).
@@ -150,6 +155,10 @@ The recorder opens explicitly before framework startup. It stays open through
 task-handler end hooks; `Run.finish()` closes it after confirmed completion.
 A recorder error is sticky and makes the result fail even when cleanup succeeds.
 Task/transport failures also produce failed results when partial data is saved.
+An uncertain startup retains its original error and attempted-start metadata,
+then keeps staging instead of packaging it. A completion timeout leaves the
+recorder and resource store owned by `Run` for a later retry. The GUI establishes
+that ownership before calling `Run.start()`.
 
 The default completed artifact is a `.pycontrol.zip` native run bundle. During a
 run, `NativeEventLogger` writes the native event table and analog payloads into a
@@ -175,9 +184,18 @@ Experiment-addon hooks are sequenced by `ExperimentHookSequencer`
 `experiment_cancelled` and builds every `ExperimentHookContext` from a
 `{subject_id: RigOutcome}` mapping. `Supervisor` and the GUI's experiment run
 view both drive the same sequencer (the GUI's `SubjectOutcome` *is* a
-`RigOutcome`), so the two drivers cannot drift on hook order or context shape.
+`RigOutcome`), so both drivers share the once-only hooks and context construction.
 *When* a rig counts as started or finished stays with each driver.
 
+Each experiment run (a GUI run of an experiment, a `Supervisor.run()` call)
+has one fresh experiment handler instance and one `ExperimentRunContext`
+(`addons/run_context.py`), made by `ExperimentRunContext.create` before any
+rig is prepared: a unique `experiment_run_id` plus the frozen JSON
+`shared_data` the handler returns from `shared_run_data()`. The same object
+goes into every rig's `RunPlan.experiment_run` (bound to the task handler by
+`Run.start()` before `prepare_run`) and every hook context, so rigs never wait
+for experiment callbacks to learn which run they belong to.
+
 ```mermaid
 flowchart LR
     Exp["Experiment config"] --> Supervisor["Supervisor"]
@@ -194,7 +212,11 @@ flowchart LR
 driver calls it directly any more.
 
 Each rig records independently. Cross-rig coordination is experiment-hook
-sequencing, not a merged event stream.
+sequencing, not a merged event stream. Duplicate setup assignments are rejected
+before sessions open. The Supervisor uses a finite join grace after its duration
+or stop condition; slow preparation/finalization can consume that grace. A rig
+reported as timed out may still have a daemon thread alive when the CLI exits.
+The workload/timeout decision remains a release gate.
 
 ## Protocol, Transport, And REPL
 
@@ -218,33 +240,20 @@ fast commands such as `set_variable`, `get_variable`, `trigger_event`, and
 
 ## BoardSession Lifecycle
 
-`BoardSession` is the only mutable owner of a live board connection. Each
-instance should have one command-thread owner at a time. During a run it also
-starts a pump thread that reads bytes and dispatches bus events.
-
-```text
-DISCONNECTED -> CONNECTING -> CONNECTED -> READY -> RUNNING -> STOPPED
-any non-closed state -> CLOSED
-```
-
-Valid transitions:
+`BoardSession` alone owns lifecycle state. A normal run follows
+`DISCONNECTED → CONNECTING → CONNECTED → READY → RUNNING → STOPPED`.
+A new task setup moves STOPPED back to READY. The command owner confirms pump
+completion and join before STOPPED; receiving an end marker or bus notification
+alone does not make that transition.
 
-| From | Operation | To |
-| --- | --- | --- |
-| `DISCONNECTED` | `connect()` begins | `CONNECTING` |
-| `CONNECTING` | `connect()` succeeds | `CONNECTED` |
-| `CONNECTING` | `connect()` fails | `DISCONNECTED` |
-| `CONNECTED` | `setup_state_machine()` | `READY` |
-| `STOPPED` | `setup_state_machine()` | `READY` |
-| `READY` | `start()` | `RUNNING` |
-| `RUNNING` | `stop()` or board end-of-run marker | `STOPPED` |
-| any non-closed state | `close()` | `CLOSED` |
+Uncertain startup or reader failure makes the session unusable: request stop
+best-effort, complete callbacks, and invalidate to CLOSED. `close()` also reaches
+CLOSED, but raises and remains retryable while callbacks are alive. After
+callbacks finish, transport-disposal errors are logged once and disposal is
+terminal. CLOSED is no proof of physical board stop or OS handle release.
 
-Invalid transitions raise `BoardSessionError`. Hardware-definition upload does
-not create a state; it syncs board files used by task setup.
-
-See `threading-and-state.md` for the authoritative threading and transition
-contract.
+Use [threading-and-state.md](threading-and-state.md) for the authoritative state
+transitions, callback contexts and shutdown barriers.
 
 ## Event Bus And Subscribers
 
@@ -283,7 +292,8 @@ The bundle contains:
 - `meta.json` with run identity, board identity, software versions, variables,
   setup/experiment metadata, and provenance references.
 - Native `events.tsv` with typed rows for info, state, event, variable,
-  threshold, print (including operator notes), log, status, warning, and error.
+  threshold, print (including operator notes), log, status, warning, error and
+  runtime resource capture.
 - Analog payloads.
 - Snapshots and hashes for task, hardware definition, devices, addons, and
   relevant workspace settings.
@@ -381,26 +391,18 @@ addon loading around those core surfaces. A future web or service frontend
 should be able to use the same core surfaces without changing protocol,
 transport, session, or recording code.
 
-## Stability
-
-Stable contracts:
-
-- Wire protocol and deployed firmware frame semantics.
-- Session state names and valid lifecycle transitions.
-- Bus event dataclass names and callback names.
-- Native run bundle layout.
-- Documented public import surfaces.
-
-Implementation details:
+## Compatibility and release readiness
 
-- Internal package/file layout under `session/services/`, `cli/commands/`, and
-  provenance helpers.
-- Exact lock scopes inside `BoardSession`.
-- CLI command parser internals and GUI orchestration.
-- Test fixtures and private helper classes.
+The beta freezes addon-author APIs, bus events/callbacks, wire and data formats,
+and the lifecycle rules in [contracts.md](contracts.md). The documented session,
+run, workspace and other frontend Python APIs are **provisional**, as listed in
+[public-api.md](public-api.md). Internal helpers, lock scopes and file layout may
+change while preserving the relevant behavior. Record compatibility changes in
+the contract documents, changelog and behavioral tests.
 
-When changing an implementation detail, preserve the contracts above or update
-the relevant contract document and tests in the same change.
+A compatibility freeze is not stable-release approval. Physical F4/H7 acceptance,
+a clean release version/dependency-floor pairing, and a Supervisor workload/
+timeout decision remain required; see [release-acceptance.md](release-acceptance.md).
 
 ## Non-Goals