mesofield 0.3.2b0__py3-none-any.whl

docs/user_guide.md

@@ -0,0 +1,172 @@
+# User Guide
+
+This guide is for **experimenters** — people running acquisitions on a
+configured rig. If you're writing a new device class or subclassing
+`Procedure`, see the [Developer Guide](developer_guide.md) instead.
+
+## Overview
+
+A mesofield experiment is described by two files:
+
+| File | Owns | Usually edited by |
+|------|------|-------------------|
+| `hardware.yaml` | What devices exist on this rig and how to talk to them | Rig maintainer (one-time per machine) |
+| `experiment.json` | Subjects, sessions, protocol, duration | Experimenter (per study / per day) |
+
+The `mesofield` CLI loads both, brings up the GUI, and orchestrates the
+acquisition.
+
+## Launching an acquisition
+
+The CLI installs as both a console script and a Python module entry
+point; either form works:
+
+```bash
+mesofield launch path/to/experiment.json
+# equivalent
+python -m mesofield launch path/to/experiment.json
+```
+
+Either form opens the main acquisition window with hardware initialised
+and the parameters from `experiment.json` populated in the form.
+
+## Experiment configuration (`experiment.json`)
+
+```json
+{
+    "Configuration": {
+        "experimenter": "you",
+        "protocol": "HFSA",
+        "experiment_directory": "/where/mesofield/writes_outputs",
+        "hardware_config_file": "path/to/hardware.yaml",
+        "duration": 1000
+    },
+    "Subjects": {
+        "STREHAB07": {
+            "sex": "F",
+            "session": "01",
+            "task": "mesoscope"
+        }
+    },
+    "DisplayKeys": [
+        "subject",
+        "session",
+        "task",
+        "experimenter",
+        "protocol",
+        "duration",
+        "start_on_trigger",
+        "led_pattern"
+    ]
+}
+```
+
+**Field notes:**
+
+- `experiment_directory` and `hardware_config_file` are optional — both
+  default to siblings of the JSON file's parent directory.
+- `duration` is in seconds. The MDA sequence builds
+  `duration × camera.fps` frames.
+- `Subjects` keys become BIDS `sub-<key>` directories under
+  `experiment_directory/data/`. `session` and `task` become `ses-<id>`
+  and `task-<id>`.
+- `DisplayKeys` decides which fields appear in the editable form in the
+  GUI. Edits persist back to `experiment.json` when the run completes
+  (or via the **Save** button).
+- Anything you add to the JSON outside of these reserved keys is
+  preserved on save.
+
+## The acquisition window
+
+The window has three regions:
+
+1. **Live Viewer (top-left)** — per-camera snap / live / progress
+   panels. The mesoscope view sits next to the pupil view by default.
+2. **Configuration form (top-right)** — the `DisplayKeys` you declared,
+   plus a subject selector, **Record**, **Add Note**, and dynamic
+   hardware controls (LED test, NIDAQ pulse, etc.) for whatever your
+   `hardware.yaml` requested.
+3. **Encoder / processor plots (bottom)** — live traces of any frame
+   processor with `plot=True` and any encoder / serial device with
+   `start_live_view` enabled.
+
+The **Toggle Console** action in the toolbar opens an embedded IPython
+shell with the live `procedure` bound — handy for inspecting state
+mid-run.
+
+## Notes during a run
+
+Click **Add Note** at any time. Notes are timestamped and saved to
+`data/sub-<id>/ses-<id>/notes.json` when the run completes.
+
+## What ends up on disk
+
+After a run, your experiment directory looks like:
+
+```
+<experiment_dir>/
+    experiment.json               # updated with any DisplayKeys edits
+    hardware.yaml
+    data/
+        sub-<id>/
+            ses-<id>/
+                manifest.json     # AcquisitionManifest — the contract
+                notes.json
+                <task>/
+                    *_meso.ome.tiff
+                    *_meso_frame_metadata.json
+                    *_pupil.mp4
+                    *_pupil_frame_metadata.json
+                    *_wheel.csv
+```
+
+The `manifest.json` is a typed `AcquisitionManifest` (from
+`mesokit-schema`) describing every producer, its output path, its
+metadata sidecar, and any calibration constants. Downstream analysis
+tools read the manifest instead of globbing.
+
+## Embedded IPython console
+
+Toolbar → **Toggle Console**. The kernel pre-binds:
+
+- `self` — the main window (`MainWindow`)
+- `procedure` — the active [`Procedure`](api/generated/mesofield.base)
+- `data` — the [`mesofield.data`](api/generated/mesofield.data) package
+
+Common one-liners:
+
+```python
+procedure.config.items()                    # all configuration values
+procedure.config.set("duration", 600)       # change the run length
+procedure.hardware.cameras                  # list configured cameras
+procedure.hardware.primary                  # the camera that drives MDA
+procedure.events.procedure_started.connect(my_callback)
+```
+
+## Logging
+
+All application logs flow through one `loguru` logger and land in:
+
+```
+logs/mesofield.log
+```
+
+- Rotates **daily at midnight**.
+- The console shows colourised logs at `INFO`; the file captures
+  everything down to `DEBUG`.
+- Uncaught exceptions are routed through the same hook so crashes leave
+  a trail.
+- Chatty third-party libraries (`matplotlib`, `asyncio`, `traitlets`)
+  are pinned at `WARNING` or above.
+
+To change the location or verbosity, see
+[`mesofield/utils/_logger.py`](api/generated/mesofield.utils).
+
+## System requirements
+
+Mesofield is tested on Windows 10/11. For multi-camera acquisition with
+large files we recommend:
+
+- ≥ 32 GB RAM
+- 12th-gen Intel i7 or equivalent
+- Fast local storage (NVMe SSD) for the experiment directory

examples/teensy_pulse_generator.py

@@ -0,0 +1,320 @@
+"""Example custom device: a commandable Teensy 4.0 sync pulse generator.
+
+Targets the ``sync-pulse-generator v3-teensy40`` firmware.  Protocol is
+line-based and ``\\n``-terminated; the device USB-CDC speed is fixed at
+115200.
+
+Device -> host telemetry
+========================
+
+Pulse events (one line per rising LED edge)::
+
+    LED,<device_us>,<event_id>
+
+Banners, status, sync replies, and errors all start with ``#``::
+
+    # sync-pulse-generator v3-teensy40 ts=micros_at_rising_edge
+    # status state=idle seq_len=1 run_count=0 run_count_limit=0 epoch=0 next_id=0
+    # pattern len=1 (20000,480000)
+    # sync token=<t> device_us=<micros>
+    # running | # stopped | # auto_stop | # epoch=<n> | # pong
+    # err=<reason> | # dropped=<n> | # pattern_set
+
+Host -> device commands (case-insensitive)
+==========================================
+
+::
+
+    STATUS                              -> '# status ...'
+    PING                                -> '# pong'
+    SYNC <token>                        -> '# sync token=<token> device_us=...'
+
+    PATTERN SIMPLE <period_us> <width_us>      (idle only)
+    PATTERN SEQ <w1> <g1> <w2> <g2> ...        (idle only, up to 16 pairs)
+    PATTERN SHOW                         -> '# pattern ...'
+
+    RUN                                  free run until STOP
+    RUN DURATION <us>                    auto-stop after wall-clock duration
+    RUN COUNT <n>                        auto-stop after N pulses
+    STOP                                 force LED LOW, return to idle
+
+    RESET                                STOP + zero event_id, increment epoch
+
+Limits enforced by the firmware: width >= 100 us, gap >= 100 us,
+each interval <= 60_000_000 us, sequence length <= 16 pairs.
+
+Hardware YAML
+=============
+
+::
+
+    teensy:
+      type: teensy_pulses
+      port: COM7              # /dev/ttyACM0 on Linux
+      baudrate: 115200
+      development_mode: false # true -> no port opened, send_line no-ops
+"""
+
+from __future__ import annotations
+
+import re
+import time
+from typing import Iterable, List, Optional, Sequence, Tuple
+
+from mesofield import DeviceRegistry
+from mesofield.devices.base import BaseSerialDevice
+
+
+# Pulse line:    LED,<device_us>,<event_id>
+_PULSE_RE = re.compile(r"^LED,(\d+),(\d+)$")
+# Status line:   # status state=<s> seq_len=<n> run_count=<c> ...
+_STATUS_RE = re.compile(r"(\w+)=(\S+)")
+# Sync reply:    # sync token=<t> device_us=<u>
+_SYNC_RE = re.compile(r"^sync\s+token=(\S+)\s+device_us=(\d+)\s*$")
+
+
+@DeviceRegistry.register("teensy_pulses")
+class TeensyPulseGenerator(BaseSerialDevice):
+    """Sync pulse generator driver matching firmware v3-teensy40.
+
+    Each rising LED edge becomes a row::
+
+        timestamp,device_us,event_id,epoch
+
+    where ``timestamp`` is the host wall-clock at line receipt and
+    ``device_us`` is the Teensy's ``micros()`` clock latched in the
+    rising-edge ISR.  ``epoch`` increments on every ``RESET`` so
+    ``event_id`` collisions can be disambiguated across runs.
+    """
+
+    device_type = "stimulator"
+    file_type = "csv"
+    bids_type = "events"
+
+    # Firmware-enforced limits (see sync-pulse-generator v3 source).
+    MIN_WIDTH_US: int = 100
+    MIN_GAP_US: int = 100
+    MAX_INTERVAL_US: int = 60_000_000
+    MAX_SEQ_LEN: int = 16
+
+    def __init__(self, cfg=None, **kwargs):
+        super().__init__(cfg, **kwargs)
+        # Firmware state mirrored from '#'-prefixed lines.
+        self.firmware_banner: Optional[str] = None
+        self.last_status: dict = {}
+        self.last_pattern: List[Tuple[int, int]] = []
+        self.last_sync: Optional[Tuple[str, int]] = None  # (token, device_us)
+        self.last_dropped: int = 0
+        self.epoch: int = 0
+        self.is_running_firmware: bool = False
+
+    # ------------------------------------------------------------------
+    # High-level command API (callable from a Procedure or the GUI).
+    # All methods are fire-and-forget; replies arrive asynchronously
+    # through ``parse_line`` and update the ``last_*`` attributes.
+    # ------------------------------------------------------------------
+
+    def ping(self) -> None:
+        self.send_line("PING")
+
+    def query_status(self) -> None:
+        self.send_line("STATUS")
+
+    def query_pattern(self) -> None:
+        self.send_line("PATTERN SHOW")
+
+    def sync(self, token: Optional[str] = None) -> str:
+        """Send ``SYNC <token>``; the reply lands in :attr:`last_sync`."""
+        if token is None:
+            token = f"host{int(time.time() * 1000)}"
+        self.send_line(f"SYNC {token}")
+        return token
+
+    # -- pattern --------------------------------------------------------
+    def set_pattern_simple(self, period_us: int, width_us: int) -> None:
+        """Repeating single pulse: ``period_us`` cycle, ``width_us`` high.
+
+        Firmware accepts only when device is IDLE.  Validates locally
+        against the firmware's documented limits to fail fast.
+        """
+        period_us = int(period_us)
+        width_us = int(width_us)
+        if width_us < self.MIN_WIDTH_US:
+            raise ValueError(f"width_us must be >= {self.MIN_WIDTH_US}")
+        if period_us < width_us + self.MIN_GAP_US:
+            raise ValueError(
+                f"period_us must be >= width_us + {self.MIN_GAP_US}"
+            )
+        if period_us > self.MAX_INTERVAL_US:
+            raise ValueError(f"period_us must be <= {self.MAX_INTERVAL_US}")
+        self.send_line(f"PATTERN SIMPLE {period_us} {width_us}")
+
+    def set_pattern_sequence(
+        self, steps: Sequence[Tuple[int, int]]
+    ) -> None:
+        """Set a (width_us, gap_us) sequence (up to 16 pairs)."""
+        steps = list(steps)
+        if not steps:
+            raise ValueError("steps must contain at least one (width,gap) pair")
+        if len(steps) > self.MAX_SEQ_LEN:
+            raise ValueError(f"at most {self.MAX_SEQ_LEN} pairs supported")
+        for w, g in steps:
+            if w < self.MIN_WIDTH_US or g < self.MIN_GAP_US:
+                raise ValueError("width/gap below firmware minimum (100 us)")
+            if w > self.MAX_INTERVAL_US or g > self.MAX_INTERVAL_US:
+                raise ValueError(
+                    f"width/gap above firmware maximum ({self.MAX_INTERVAL_US} us)"
+                )
+        flat: Iterable[int] = (v for pair in steps for v in pair)
+        self.send_line("PATTERN SEQ " + " ".join(str(int(v)) for v in flat))
+
+    def set_frequency(self, hz: float, duty: float = 0.5) -> None:
+        """Convenience wrapper around :meth:`set_pattern_simple`.
+
+        ``duty`` is the fractional high time (0 < duty < 1).
+        """
+        if hz <= 0:
+            raise ValueError(f"frequency must be > 0, got {hz}")
+        if not 0.0 < duty < 1.0:
+            raise ValueError(f"duty must be in (0,1), got {duty}")
+        period_us = int(round(1_000_000 / hz))
+        width_us = max(self.MIN_WIDTH_US, int(round(period_us * duty)))
+        self.set_pattern_simple(period_us, width_us)
+
+    # -- run/stop -------------------------------------------------------
+    def run(self) -> None:
+        """Free-run until :meth:`stop_pulses` (or RESET)."""
+        self.send_line("RUN")
+
+    def run_for_duration(self, duration_us: int) -> None:
+        if duration_us <= 0:
+            raise ValueError("duration_us must be > 0")
+        self.send_line(f"RUN DURATION {int(duration_us)}")
+
+    def run_for_count(self, count: int) -> None:
+        if count <= 0:
+            raise ValueError("count must be > 0")
+        self.send_line(f"RUN COUNT {int(count)}")
+
+    def stop_pulses(self) -> None:
+        self.send_line("STOP")
+
+    def reset(self) -> None:
+        """``STOP`` + zero ``event_id`` + increment ``epoch`` on the device."""
+        self.send_line("RESET")
+
+    # ------------------------------------------------------------------
+    # BaseSerialDevice hooks
+    # ------------------------------------------------------------------
+
+    def setup_serial(self) -> None:
+        """Drain the boot banner so the first run starts from a clean slate."""
+        # The firmware emits its banner + status + pattern as soon as
+        # USB-CDC enumerates.  Give it a moment, then ask for a fresh
+        # snapshot so ``self.last_status``/``last_pattern`` are populated
+        # before the experiment starts.
+        time.sleep(0.1)
+        self.send_line("STATUS")
+        self.send_line("PATTERN SHOW")
+
+    def parse_line(self, line: bytes) -> Optional[Tuple[dict, Optional[float]]]:
+        text = line.decode("utf-8", errors="replace").strip()
+        if not text:
+            return None
+
+        # ---------- pulse event ----------
+        m = _PULSE_RE.match(text)
+        if m is not None:
+            device_us = int(m.group(1))
+            event_id = int(m.group(2))
+            payload = {
+                "device_us": device_us,
+                "event_id": event_id,
+                "epoch": self.epoch,
+                "device_id": self.device_id,
+            }
+            # ts=None -> BaseDataProducer.record stamps with host time.
+            return payload, None
+
+        # ---------- '#'-prefixed control/telemetry lines ----------
+        if text.startswith("#"):
+            self._handle_meta_line(text[1:].strip())
+            return None
+
+        self.logger.debug("unrecognised teensy line: %r", text)
+        return None
+
+    # ------------------------------------------------------------------
+    # Internal: '#' line dispatch
+    # ------------------------------------------------------------------
+
+    def _handle_meta_line(self, body: str) -> None:
+        """Update mirrored firmware state from a ``#``-prefixed line."""
+        if not body:
+            return
+
+        head = body.split(None, 1)[0].lower()
+
+        if head.startswith("sync-pulse-generator"):
+            self.firmware_banner = body
+            self.logger.info("teensy banner: %s", body)
+            return
+
+        if head == "status":
+            self.last_status = dict(_STATUS_RE.findall(body))
+            self.is_running_firmware = self.last_status.get("state") == "running"
+            try:
+                self.epoch = int(self.last_status.get("epoch", self.epoch))
+            except ValueError:
+                pass
+            self.logger.debug("teensy status=%s", self.last_status)
+            return
+
+        if head == "pattern":
+            self.last_pattern = [
+                (int(w), int(g))
+                for w, g in re.findall(r"\((\d+),(\d+)\)", body)
+            ]
+            self.logger.debug("teensy pattern=%s", self.last_pattern)
+            return
+
+        m = _SYNC_RE.match(body)
+        if m is not None:
+            self.last_sync = (m.group(1), int(m.group(2)))
+            self.logger.debug("teensy sync token=%s device_us=%s",
+                              *self.last_sync)
+            return
+
+        if body == "running":
+            self.is_running_firmware = True
+            return
+        if body in ("stopped", "auto_stop"):
+            self.is_running_firmware = False
+            return
+        if body == "pong":
+            return
+        if body == "pattern_set":
+            return
+
+        if body.startswith("epoch="):
+            try:
+                self.epoch = int(body.split("=", 1)[1])
+            except ValueError:
+                pass
+            return
+
+        if body.startswith("dropped="):
+            try:
+                self.last_dropped += int(body.split("=", 1)[1])
+            except ValueError:
+                pass
+            self.logger.warning("teensy dropped pulses (total=%d)",
+                                self.last_dropped)
+            return
+
+        if body.startswith("err="):
+            self.logger.warning("teensy firmware error: %s", body[4:])
+            return
+
+        # Unknown '#' line -> log at debug.
+        self.logger.debug("teensy meta: %s", body)

experiments/pipeline_demo/experiment.json

@@ -0,0 +1,24 @@
+{
+    "Configuration": {
+        "experimenter": "pipeline-test",
+        "protocol": "PIPELINE_DEMO",
+        "duration": 2,
+        "start_on_trigger": false
+    },
+    "procedure_file": "procedure.py",
+    "procedure_class": "PipelineDemoProcedure",
+    "Subjects": {
+        "DEMO01": {
+            "session": "01",
+            "task": "demo"
+        }
+    },
+    "DisplayKeys": [
+        "subject",
+        "session",
+        "task",
+        "experimenter",
+        "protocol",
+        "duration"
+    ]
+}

experiments/pipeline_demo/hardware.yaml

@@ -0,0 +1,23 @@
+# Headless demo rig for the mesokit-schema pipeline test.
+#
+# Two synthetic devices, no real serial or pymmcore hardware:
+#   - mock_wheel: BaseSerialDevice subclass producing CSV samples
+#   - mock_camera: BaseDataProducer subclass writing a real OME-TIFF
+#                  stack + a _frame_metadata.json sidecar
+#
+# The wheel is primary; its signals.finished would normally trigger
+# cleanup, but since the mock devices run forever, the procedure uses
+# a wall-clock `duration` cap (see experiment.json) to call cleanup.
+memory_buffer_size: 1000
+
+
+camera:
+  type: mock_camera       # registered by procedure.py at import time
+  primary: true
+  width: 32
+  height: 32
+  frame_interval_ms: 100
+  output:
+    suffix: meso
+    file_type: ome.tiff
+    bids_type: func

experiments/pipeline_demo/procedure.py

@@ -0,0 +1,50 @@
+"""Headless demo procedure for the mesokit-schema pipeline test.
+
+Reuses the legacy SampleProcedure's wall-clock duration cap so the run
+terminates cleanly without any GUI or real hardware. The
+AcquisitionManifest is written by the base `Procedure._cleanup_procedure`
+hook -- subclasses do not have to import or know about mesokit-schema.
+
+Registers two synthetic device types so `hardware.yaml` can reference
+them without touching real hardware:
+
+  - mock_wheel  : MockEncoderDevice (CSV samples)
+  - mock_camera : MockFrameProducer (OME-TIFF + frame metadata JSON)
+"""
+
+from __future__ import annotations
+
+import threading
+
+from mesofield import DeviceRegistry
+from mesofield.base import Procedure
+from mesofield.devices.mocks import MockFrameProducer
+from mesofield.devices.mocks import MockEncoderDevice
+
+
+DeviceRegistry._registry.setdefault("mock_wheel", MockEncoderDevice)
+DeviceRegistry._registry.setdefault("mock_camera", MockFrameProducer)
+
+
+class PipelineDemoProcedure(Procedure):
+    """Mock-encoder-only procedure with duration-gated cleanup."""
+
+    def on_started(self) -> None:
+        duration = self.config.get("duration")
+        if duration:
+            self.logger.info(f"Duration cap armed: {duration}s")
+            self._duration_timer = threading.Timer(float(duration), self.cleanup)
+            self._duration_timer.daemon = True
+            self._duration_timer.start()
+
+    def on_finished(self) -> None:
+        super().on_finished()
+        timer = getattr(self, "_duration_timer", None)
+        if timer is not None:
+            timer.cancel()
+            self._duration_timer = None
+
+def main():
+    """Run the procedure."""
+    procedure = PipelineDemoProcedure("/Users/jakegronemeyer/dev/mesofield/experiments/pipeline_demo/experiment.json")
+    procedure.run()

experiments/two_cam_demo/experiment.json

@@ -0,0 +1,24 @@
+{
+    "Configuration": {
+        "experimenter": "demo",
+        "protocol": "TWO_CAM_DEMO",
+        "duration": 10,
+        "start_on_trigger": false
+    },
+    "procedure_file": "procedure.py",
+    "procedure_class": "TwoCamDemoProcedure",
+    "Subjects": {
+        "DEMO": {
+            "session": "01",
+            "task": "freeview"
+        }
+    },
+    "DisplayKeys": [
+        "subject",
+        "session",
+        "task",
+        "experimenter",
+        "protocol",
+        "duration"
+    ]
+}

experiments/two_cam_demo/hardware.yaml

@@ -0,0 +1,58 @@
+# Two mock cameras + one mock wheel encoder.
+#
+# Three producers all writing real files (real OME-TIFFs, real CSV) into a
+# BIDS layout under data/sub-DEMO/ses-01/. No GUI required; no real hardware
+# required; the procedure runs to its `duration` cap and writes the
+# AcquisitionManifest at the session root.
+#
+# Layout that gets produced:
+#   data/sub-DEMO/ses-01/
+#     manifest.json                                          ← contract
+#     <ts>_sub-DEMO_ses-01_task-freeview_configuration.csv
+#     <ts>_sub-DEMO_ses-01_task-freeview_timestamps.csv
+#     <ts>_sub-DEMO_ses-01_task-freeview_notes.txt
+#     func/
+#       <ts>_sub-DEMO_ses-01_task-freeview_meso.ome.tiff
+#       <ts>_sub-DEMO_ses-01_task-freeview_meso.ome.tiff_frame_metadata.json
+#     behav/
+#       <ts>_sub-DEMO_ses-01_task-freeview_pupil.ome.tiff
+#       <ts>_sub-DEMO_ses-01_task-freeview_pupil.ome.tiff_frame_metadata.json
+#     beh/
+#       <ts>_sub-DEMO_ses-01_task-freeview_wheel.csv
+#       <ts>_sub-DEMO_ses-01_task-freeview_dataqueue.csv
+memory_buffer_size: 4000
+
+# Primary mesoscope: 128x128 @ 10 fps. Drives session-level timing.
+mesoscope:
+  type: opencv_camera
+  device_index: 0
+  primary: true
+  frame_interval_ms: 100
+  cv_backend: "mpv4"
+  backend: "opencv"
+  output:
+    suffix: meso
+    file_type: mp4
+    bids_type: func
+
+# Pupil camera: 64x64 @ 20 fps.
+pupil:
+  type: mock_camera
+  width: 64
+  height: 64
+  frame_interval_ms: 50
+  output:
+    suffix: pupil
+    file_type: ome.tiff
+    bids_type: behav
+
+# Wheel encoder: 50 Hz.
+wheel:
+  type: mock_wheel
+  sample_interval_ms: 20
+  cpr: 2400
+  diameter_mm: 80
+  output:
+    suffix: wheel
+    file_type: csv
+    bids_type: beh