plesty-lib 0.3.4__tar.gz → 0.3.5.dev1__tar.gz

{plesty_lib-0.3.4 → plesty_lib-0.3.5.dev1}/CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## 0.3.5.dev1 — 2026-08-17
+
+Development pre-release for field-testing the run convention (#34); the
+final release of this line is 0.4.0 — the API diff against 0.3.4 (the
+`pattern=` arguments and `data_dir` attributes of the run sources are
+gone, `Experiment(run_root=)` defaults to the mount) is a minor bump.
+
+- `plesty.lib.monitor.Viz` (#34): the `monitor` / `render` command line over
+  an experiment's panels. The experiment registers one function
+  (`@viz.panels`, `Run` → panels) and calls `viz.main()`; run discovery,
+  `--run-root/--mount/--data-dir/--run`, live vs `--replay`, `--record`,
+  offscreen `render` with self-stop, and the text catalog
+  (`monitor/assets/viz.yaml`) are the lib's. `Run(..., data_dir=)` supplies
+  the host spelling for runs that did not record it.
+- **Run layout: one `records.jsonl` per run instead of one `data/step_NNNN.json`
+  per step** (#34). Each completed step appends one fsynced line (its
+  `ResultDocument` + `index`; blobs still go to `data/`), and readers tail
+  the file by byte offset — writing and following a long run on a network
+  share no longer costs a file open and a directory listing per step. New:
+  `plesty.lib.data.append_record` / `read_records` / `record_value`;
+  `RunSource`/`ReplaySource` lose their `pattern` argument and gain `unread`
+  and `legacy`; the journal's `step_completed` carries `recorded` instead of
+  a `result` path; a resumed run also counts steps with a committed record
+  as done. Runs in the old layout are still read (`RunSource.legacy`).
+- The run convention is spelled out once and exported (#34):
+  `plesty.lib.experiment.runs` — `run_id`/`parse_run_id`, `list_runs`,
+  `latest_run`, and `Run` (plan, config, journal status, records, and
+  `local()` translation of device-written paths onto this machine).
+- One environment convention for the shared disk: `PLESTY_DATA_DIR` (as the
+  acquiring host sees it) and `PLESTY_DATA_MOUNT` (as this machine sees it).
+  `Experiment` reads both (or `data_dir=`/`data_mount=`), defaults
+  `run_root` to the mount when set, exposes `self.run_id`/`self.run_dir`
+  before `setup()` and `self.raw_dir` = `<data_dir>/<run_id>/raw` for the
+  writing devices, and journals all three with `run_started`. `run_root`
+  still defaults to `runs` without a mount.
+
 ## 0.3.4 — 2026-08-14
 
 - The server never cancels a receive (#33): the old

{plesty_lib-0.3.4 → plesty_lib-0.3.5.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: plesty-lib
-Version: 0.3.4
+Version: 0.3.5.dev1
 Summary: A library for the Plesty ecosystem.
 Author: Plesty Development Team
 Author-email: Yunshuang Yuan <yunshuang.yuan@fkp.uni-hannover.de>

{plesty_lib-0.3.4 → plesty_lib-0.3.5.dev1}/docs/experiment.md

@@ -17,23 +17,29 @@ instances, never raw hardware.
 | `Step` | The smallest unit of work: a stable `id`, an operation name `op`, and its `params`. A step either completes (result persisted and journaled) or is re-run on resume. |
 | `Plan` | The frozen, ordered schedule of steps plus the run configuration. Written to disk at run start, never mutated; a SHA-256 `content_hash()` identifies the schedule. |
 | `Journal` | A crash-safe, append-only JSONL event log (`run_started`, `step_started`, `step_completed`, `step_failed`, `run_completed`/`_canceled`/`_aborted`). Replaying it derives the run state and the completed-step set. |
+| `records.jsonl` | One line per completed step: its result document (`plesty.lib.data.append_record` / `read_records`). One file, appended and fsynced per step, tailed by offset — not one small file per step, which is slow to write and slow to follow on a network share. |
 | `Experiment` | The async ABC tying it together: validate → journal → execute with retries → persist results → resume. |
+| `Run` | Read access to a stored run — plan, config, journal status, records, and the translation of device-written paths onto this machine (`plesty.lib.experiment.runs`, which also spells out the naming below and finds runs: `list_runs`, `latest_run`). |
 
 Every run owns a directory:
 
 ```text
-<run_root>/<run_id>/
-├── plan.json        # frozen schedule + config + content hash
-├── journal.jsonl    # append-only event log
-└── data/            # per-step results: raw blob + JSON metadata
-    ├── step_0000.json
-    └── step_0000.npy
+<run_root>/<run_id>/            # run_id = <name>_<YYYYmmdd-HHMMSS>
+├── plan.json                   # frozen schedule + config + content hash
+├── journal.jsonl               # append-only event log
+├── records.jsonl               # one line per completed step: its result document
+└── data/                       # blobs the records reference (arrays, images), if any
+    └── step_0003.npy
 ```
 
-Step results are persisted with
-[`save_result`](data_schemas.md) — the raw blob stays in its native format,
-described by a typed `ResultDocument` with provenance (`step_id`, `op`,
-`params`).
+Each record line is a typed [`ResultDocument`](data_schemas.md) with
+provenance (`step_id`, `op`, `params`) plus the step `index`; a JSON value
+result is inlined, an array or image stays a raw blob in `data/` that the
+line references. The line is the commit — it is appended (and fsynced) only
+after the blob is on disk, and a resumed run treats a committed record as a
+completed step even if the crash came before its journal event. Runs
+written by plesty-lib ≤ 0.3.4 (one `data/step_*.json` per step) are still
+read by `RunSource`, `records()`, and `Run`.
 
 ## Writing an experiment
 
@@ -78,6 +84,52 @@ Step methods may be sync or async; results are awaited automatically. Each
 step is retried (`max_retries`, `retry_sleep` constructor arguments) before
 the run aborts.
 
+## Data on a shared disk
+
+Blobs a device writes itself (spectrometer frames, camera images) never
+cross the network: the acquiring host saves them to a shared disk and the
+step record carries only the path. Two machines spell that share
+differently, so it is named once, by environment:
+
+| Variable | Meaning |
+|---|---|
+| `PLESTY_DATA_DIR` | the share as the acquiring host sees it (e.g. `G:\RAWDATA`) |
+| `PLESTY_DATA_MOUNT` | the same share as this machine sees it (e.g. `/mnt/group/RAWDATA`) |
+
+The experiment reads both (or takes `data_dir=` / `data_mount=` at
+construction). With the mount set, `run_root` defaults to it, so runs land
+on the share as well. Before `setup()` is awaited the run id is known and
+`self.raw_dir` names `<data_dir>/<run_id>/raw` in the host's spelling — one
+folder per run, the same one on resume — which `setup()` routes the writing
+devices to:
+
+```python
+async def setup(self) -> None:
+    await super().setup()
+    if self.raw_dir is not None:
+        self.devices.set_data_path(self.raw_dir, devices=["spec"])
+```
+
+`run_started` journals `data_dir`, `data_mount`, and `raw_dir`, so a reader
+needs at most its own mount:
+
+```python
+from plesty.lib.experiment import Run, latest_run
+
+run = Run(latest_run("/mnt/group/RAWDATA", name="pol_pl"))
+run.config["powermeter"]                 # the frozen configuration
+run.status()["state"]                    # "running", "completed", ...
+for record in run.records():
+    run.local(record["data_file"])       # G:\RAWDATA\... → /mnt/group/RAWDATA/...
+```
+
+`Run.local` uses, in order, an explicit `mount=`, `PLESTY_DATA_MOUNT`, and
+the mount journaled by the machine that ran the experiment (only when that
+directory exists here); with neither a data root nor a mount the path is
+returned as recorded — the case of reading on the acquiring host itself.
+`run.source(mapper)` returns a `RunSource` (live) or, with
+`Run(..., replay_rows=n)`, a `ReplaySource` — see [Monitors](monitor.md).
+
 ## Crash, cancel, resume
 
 `run()` returns the run id. If the run is interrupted — `Ctrl-C`, a crash,

{plesty_lib-0.3.4 → plesty_lib-0.3.5.dev1}/docs/monitor.md

@@ -82,16 +82,19 @@ clock belongs to the caller: a GUI timer, a test loop, a notebook cell.
 
 | Source | Follows | Typical use |
 |---|---|---|
-| `RunSource` | `<run_dir>/data/step_*.json` | any running experiment |
+| `RunSource` | `<run_dir>/records.jsonl`, tailed by offset | any running experiment |
 | `TelemetrySource` | `TelemetryEvent`s of devices | live device readings |
 | `PushSource` | whatever the producer emits | tests, notebooks |
 
 `RunSource` is the generic live view of *any* PLESTY experiment: the
-[experiment framework](experiment.md) writes one result document per
-completed step, and that document is the commit record of the write, so
-following the directory needs no cooperation from the experiment and cannot
-disturb it. It also means a viewer started mid-run still sees every earlier
-row.
+[experiment framework](experiment.md) appends one record line per completed
+step, and that line is the commit record of the write, so following the file
+needs no cooperation from the experiment and cannot disturb it. Each poll
+reads only the bytes appended since the last one, so following a run of
+thousands of steps on a network share stays cheap. It also means a viewer
+started mid-run still sees every earlier row. A `Run` from
+`plesty.lib.experiment` builds the source for you (`run.source(mapper)`),
+live or replayed.
 
 ```python
 from plesty.lib.monitor import RunSource
@@ -137,6 +140,57 @@ infrastructure. Reusable views live in their own package
 (`plesty-common-monitors`); rig-specific ones live next to the experiment
 that needs them.
 
+## Watching a run: `Viz`
+
+Everything about *watching* is the same for every experiment — which run
+(the newest, or `--run`), where the share is mounted, live or replayed, in a
+window or rendered offscreen to a video, recorded or not. `Viz` owns all of
+it; the experiment writes one function that says which record key means
+what, and that is its whole viz module:
+
+```python
+# plesty/pol_pl/viz.py
+import sys
+
+from plesty.common_monitors import SeriesMonitor, SpectrumMonitor, WaterfallMonitor
+from plesty.common_monitors.series import series_mapper
+from plesty.common_monitors.spectrum import spectrum_mapper
+from plesty.lib.monitor import Viz
+from plesty.lib.ui import MonitorPanel
+
+viz = Viz("Polarization PL", experiment="pol_pl")
+
+@viz.panels
+def panels(run):
+    spectra = spectrum_mapper("data_file", locate=run.local, row_key="hwp_deg")
+    yield MonitorPanel(SpectrumMonitor(run.source(spectra), name="spectrum"), weight=3)
+    yield MonitorPanel(WaterfallMonitor(run.source(spectra), name="map", y_label="HWP angle (deg)"), weight=3)
+    if run.config.get("powermeter"):
+        power = run.source(series_mapper("hwp_deg", "power_w", y_scale=1e6))
+        yield MonitorPanel(SeriesMonitor(power, name="power", y_label="Power (uW)"), weight=2)
+
+if __name__ == "__main__":
+    sys.exit(viz.main())
+```
+
+```bash
+python -m plesty.pol_pl.viz monitor                       # follow the newest run live
+python -m plesty.pol_pl.viz monitor --replay 2 --record demo.mp4
+python -m plesty.pol_pl.viz render --run pol_pl_20260805-091909   # <run-id>.mp4, no window
+```
+
+The function receives a [`Run`](experiment.md#data-on-a-shared-disk):
+`run.source(mapper)` is already a `RunSource` or a `ReplaySource` as the
+subcommand decided — call it **once per view**: a source hands each record
+out once, so two views polling one source would split the rows between them
+(a mapper, by contrast, is freely shared); `run.local` translates the acquiring host's paths onto
+this machine, `run.config` is the frozen configuration. Runs are looked for
+under `--run-root` (default `PLESTY_DATA_MOUNT`, else `runs`); `--mount`
+and `--data-dir` override the share's two spellings for runs that did not
+record them. `render` sets Qt's offscreen platform before the toolkit is
+imported and stops by itself once every source is exhausted; neither
+`--help` nor `resolve_run` needs the `gui` extra.
+
 ## Testing a monitor
 
 `NullRenderer` records draw calls instead of drawing them, so the whole path

{plesty_lib-0.3.4 → plesty_lib-0.3.5.dev1}/gl-secret-detection-report.json

@@ -19,19 +19,19 @@
       "version": "8.30.1"
     },
     "type": "secret_detection",
-    "start_time": "2026-08-14T16:45:05",
-    "end_time": "2026-08-14T16:45:05",
+    "start_time": "2026-08-17T12:45:23",
+    "end_time": "2026-08-17T12:45:24",
     "status": "success",
     "observability": {
       "events": [
         {
           "event": "collect_secrets_analyzer_scan_metrics_from_pipeline",
-          "time_s": 0.490144528,
+          "time_s": 0.469797198,
           "exit_code": 0,
           "git_strategy": "FetchShallow",
-          "repo_size_kb": 662,
+          "repo_size_kb": 685,
           "commit_count": 1,
-          "bytes_scanned": 24,
+          "bytes_scanned": 314,
           "pipeline_type": "Tag"
         },
         {

{plesty_lib-0.3.4 → plesty_lib-0.3.5.dev1}/plesty/lib/data/__init__.py

@@ -15,7 +15,16 @@ from .types import (
     cast_basic_type,
 )
 from .table import TableHeader, PlestyTable2D, PlestyTable3D
-from .io import ResultDocument, save_result, load_result, load_document, convert_to_hdf5
+from .io import (
+    ResultDocument,
+    append_record,
+    convert_to_hdf5,
+    load_document,
+    load_result,
+    read_records,
+    record_value,
+    save_result,
+)
 
 __all__ = [
     "PlestyArray",
@@ -24,6 +33,9 @@ __all__ = [
     "load_result",
     "load_document",
     "convert_to_hdf5",
+    "append_record",
+    "read_records",
+    "record_value",
     "Units",
     "TableHeader",
     "PlestyTable2D",

{plesty_lib-0.3.4 → plesty_lib-0.3.5.dev1}/plesty/lib/data/io.py

@@ -21,7 +21,15 @@ reference link to the blob:
 
 Writes are crash-safe: the blob is written first (atomically, via a temporary
 sibling file and :func:`os.replace`), then the JSON document. The JSON
-document is the commit record — a result exists once its ``.json`` file does.
+document is the commit record — a result exists once its document does.
+
+Two layouts carry the documents. :func:`save_result` writes one ``.json``
+file per result — the right shape for standalone results. An experiment run
+writes all of its step documents to **one** append-only ``records.jsonl``
+(:func:`append_record` / :func:`read_records`): opening, syncing, and later
+listing thousands of small files on a network share is what made a run slow
+to write and slow to follow, whereas one line appended and fsynced costs one
+write and is read back by offset. Blobs still go to ``data/`` beside it.
 """
 
 import json
@@ -160,32 +168,20 @@ def _sniff_suffix(payload: bytes) -> str:
     return "bin"
 
 
-def save_result(
-    result: Any,
-    path_stem: str | Path,
-    provenance: Optional[dict[str, Any]] = None,
-) -> Path:
-    """Persist a measurement result as raw blob + JSON metadata document.
+#: Name of a run's single record file (one result document per line).
+RECORDS_FILE = "records.jsonl"
+#: Directory under a run holding the blobs its records reference.
+BLOB_SUBDIR = "data"
 
-    Args:
-        result:     The value to persist — a :class:`PlestyArray`, encoded
-                    ``bytes`` (e.g. an image), or any JSON-serializable value.
-        path_stem:  Destination path without suffix; the JSON document is
-                    written to ``<stem>.json`` and any blob next to it.
-        provenance: Optional context (step id, operation, parameters,
-                    device identity) recorded verbatim in the document.
 
-    Returns:
-        The path of the JSON metadata document.
+def _persist(result: Any, stem: Path, provenance: Optional[dict[str, Any]]) -> ResultDocument:
+    """Write *result*'s blob (if it has one) next to *stem*; return its document.
 
-    Raises:
-        TypeError: If the result is neither a PlestyArray, bytes, nor
-            JSON-serializable.
+    The document's ``blob`` is the bare file name; callers writing the
+    document elsewhere than beside the blob re-base it.
     """
-    stem = Path(path_stem)
-    stem.parent.mkdir(parents=True, exist_ok=True)
-
     if isinstance(result, PlestyArray):
+        stem.parent.mkdir(parents=True, exist_ok=True)
         blob_path = stem.with_suffix(".npy")
         buffer = BytesIO()
         np.save(buffer, np.asarray(result))
@@ -200,6 +196,7 @@ def save_result(
             meta={k: getattr(result, k, None) for k in PlestyArray._META_KEYS},
         )
     elif isinstance(result, (bytes, bytearray)):
+        stem.parent.mkdir(parents=True, exist_ok=True)
         payload = bytes(result)
         suffix = _sniff_suffix(payload)
         blob_path = stem.with_suffix(f".{suffix}")
@@ -209,12 +206,142 @@ def save_result(
         )
     else:
         document = ResultDocument(type="value", provenance=provenance or {}, value=result)
+    return document
 
+
+def save_result(
+    result: Any,
+    path_stem: str | Path,
+    provenance: Optional[dict[str, Any]] = None,
+) -> Path:
+    """Persist a measurement result as raw blob + JSON metadata document.
+
+    Args:
+        result:     The value to persist — a :class:`PlestyArray`, encoded
+                    ``bytes`` (e.g. an image), or any JSON-serializable value.
+        path_stem:  Destination path without suffix; the JSON document is
+                    written to ``<stem>.json`` and any blob next to it.
+        provenance: Optional context (step id, operation, parameters,
+                    device identity) recorded verbatim in the document.
+
+    Returns:
+        The path of the JSON metadata document.
+
+    Raises:
+        TypeError: If the result is neither a PlestyArray, bytes, nor
+            JSON-serializable.
+    """
+    stem = Path(path_stem)
+    document = _persist(result, stem, provenance)
+    stem.parent.mkdir(parents=True, exist_ok=True)
     target = stem.with_suffix(".json")
     _atomic_write_bytes(target, json.dumps(document.to_dict(), default=str, indent=2).encode())
     return target
 
 
+def append_record(
+    run_dir: str | Path,
+    index: int,
+    result: Any,
+    provenance: Optional[dict[str, Any]] = None,
+) -> dict[str, Any]:
+    """Append one step result to the run's ``records.jsonl``.
+
+    A blob-backed result (array, bytes) is written to
+    ``<run_dir>/data/step_<index>.<fmt>`` first, atomically; the document
+    line — the commit record — is appended and fsynced afterwards, so a
+    reader never sees a document whose blob is missing.
+
+    Args:
+        run_dir:    The run directory.
+        index:      Position of the step in the plan; stamped onto the line
+                    and used for the blob file name.
+        result:     The value to persist (see :func:`save_result`).
+        provenance: Context recorded verbatim in the document.
+
+    Returns:
+        The line that was written, as a dictionary — the document fields
+        plus ``index``; ``blob`` (when present) is relative to *run_dir*.
+
+    Raises:
+        TypeError: If the result is neither a PlestyArray, bytes, nor
+            JSON-serializable.
+    """
+    root = Path(run_dir)
+    document = _persist(result, root / BLOB_SUBDIR / f"step_{index:04d}", provenance)
+    line: dict[str, Any] = {"index": index, **document.to_dict()}
+    if document.blob is not None:
+        line["blob"] = f"{BLOB_SUBDIR}/{document.blob}"
+    root.mkdir(parents=True, exist_ok=True)
+    with open(root / RECORDS_FILE, "a", encoding="utf-8") as stream:
+        stream.write(json.dumps(line, default=str) + "\n")
+        stream.flush()
+        os.fsync(stream.fileno())
+    return line
+
+
+def read_records(run_dir: str | Path, offset: int = 0) -> tuple[list[dict[str, Any]], int]:
+    """Read the record lines appended to ``records.jsonl`` since *offset*.
+
+    Only complete lines are returned; a line still being written (no
+    trailing newline yet) is left for the next call, which is what makes
+    tailing a run in progress safe. Lines that fail to parse are skipped.
+
+    Args:
+        run_dir: The run directory.
+        offset:  Byte position to read from — pass back the returned offset
+                 to read only what appeared since.
+
+    Returns:
+        The parsed lines (see :func:`append_record`) and the offset just
+        after the last complete line; ``([], offset)`` while the file does
+        not exist yet.
+    """
+    path = Path(run_dir) / RECORDS_FILE
+    try:
+        with open(path, "rb") as stream:
+            stream.seek(offset)
+            chunk = stream.read()
+    except FileNotFoundError:
+        return [], offset
+    end = chunk.rfind(b"\n")
+    if end < 0:
+        return [], offset
+    lines: list[dict[str, Any]] = []
+    for raw in chunk[: end + 1].splitlines():
+        try:
+            lines.append(json.loads(raw))
+        except json.JSONDecodeError:
+            continue
+    return lines, offset + end + 1
+
+
+def record_value(line: dict[str, Any], run_dir: str | Path) -> Any:
+    """Return the value a ``records.jsonl`` line stands for.
+
+    Args:
+        line:    A line as returned by :func:`read_records`.
+        run_dir: The run directory the line's ``blob`` is relative to.
+
+    Returns:
+        The inlined value, or the :class:`PlestyArray` / raw ``bytes`` loaded
+        from the referenced blob.
+
+    Raises:
+        ValueError: If the line declares an unknown result type.
+    """
+    document = ResultDocument.from_dict(line)
+    if document.type == "value":
+        return document.value
+    blob = Path(run_dir) / (document.blob or "")
+    if document.type == "array":
+        meta = dict(document.meta or {})
+        if meta.get("range") is not None:
+            meta["range"] = tuple(meta["range"])
+        return PlestyArray(np.load(blob), **meta)
+    return blob.read_bytes()
+
+
 def load_document(path: str | Path) -> ResultDocument:
     """Load only the metadata document of a persisted result — no blob I/O.
 

{plesty_lib-0.3.4 → plesty_lib-0.3.5.dev1}/plesty/lib/experiment/__init__.py

@@ -7,6 +7,7 @@
 
 from .base_experiment import Experiment, InvalidOperationError, PlanMismatchError
 from .journal import Journal
+from .runs import Run, latest_run, list_runs, parse_run_id, run_id
 from .schedule import Plan, Step
 
 __all__ = [
@@ -15,5 +16,10 @@ __all__ = [
     "PlanMismatchError",
     "Journal",
     "Plan",
+    "Run",
     "Step",
+    "latest_run",
+    "list_runs",
+    "parse_run_id",
+    "run_id",
 ]