euler-train 2.6.0__tar.gz → 2.7.0__tar.gz

{euler_train-2.6.0 → euler_train-2.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-train
-Version: 2.6.0
+Version: 2.7.0
 Summary: Lightweight file-based experiment logger for PyTorch
 Requires-Python: >=3.9
 Requires-Dist: numpy

{euler_train-2.6.0 → euler_train-2.7.0}/README.md

@@ -49,6 +49,57 @@ with euler_train.init(dir="runs/exp02", config=cfg) as run:
     ...  # if an exception is raised, meta.json records status="crashed" + traceback
 ```
 
+## Euler View stream mode
+
+`euler_train` can dual-write runs to Euler View while still keeping the normal file-based run directory as the primary source of truth. Streaming is best-effort: local files are always written first, and stream events are sent alongside them when a stream consumer is configured.
+
+Pass the stream config via `stream=...` on `euler_train.init()`:
+
+```python
+import os
+import euler_train
+
+run = euler_train.init(
+    dir="runs/experiment_01",
+    config=cfg,
+    stream={
+        "base_url": os.environ["EULER_VIEW_BASE_URL"],
+        "model_id": 42,
+        "api_token": os.environ["EULER_VIEW_API_TOKEN"],
+        "stream_attach_token": os.environ.get("EULER_VIEW_STREAM_ATTACH_TOKEN"),
+    },
+)
+```
+
+What each field is for:
+
+- `base_url`: Euler View server base URL, for example `https://view.example.com`.
+- `model_id`: Euler View model that should receive this run.
+- `api_token` or `access_token`: credential used to open a stream session. This is the normal choice when the package should negotiate its own short-lived ingest token.
+- `stream_attach_token`: optional opaque launch-attachment token. When present, the producer sends it during the session handshake and Euler View attaches the stream to that launch directly. This is the preferred path for interactive or otherwise non-SLURM launches. Internally Euler View currently maps this token to `model_launches.id`, but the package treats it as an opaque token.
+- `stream_token`: optional pre-issued ingest token. If you pass this, the package skips the session handshake entirely and posts events directly to the ingest endpoint.
+- `datasource_id`: optional Euler View datasource hint for the run record.
+- `euler_train_dir` and `run_dir`: optional path overrides. These are auto-derived in normal usage.
+- `session_expires_in_sec`: optional requested session lifetime for the short-lived ingest token.
+
+How the handshake works:
+
+1. `euler_train` creates the local run directory and writes `meta.json`, `config.json`, and the other standard files as usual.
+2. When the first stream flush happens, the package requests a session from Euler View at `/api/models/{model_id}/runs/{run_id}/stream/session`.
+3. If `stream_attach_token` is configured, it is sent as `streamAttachToken` and used as the explicit launch attachment key.
+4. If no `stream_attach_token` is configured, the package falls back to the SLURM metadata already captured in `meta.json["slurm"]["job_id"]`.
+5. Euler View returns a short-lived ingest token plus the canonical `streamAttachToken`; the package caches that returned token and reuses it on later session refreshes.
+6. Subsequent events are sent to `/api/model-run-stream/ingest`.
+
+What has to be configured where:
+
+- In the training script or job wrapper: pass `stream={...}` to `euler_train.init(...)`.
+- In Euler View / the launcher: if the run should attach to a specific launch regardless of SLURM availability, provide a `stream_attach_token` to the runtime and forward it into the `stream` config.
+- Under plain SLURM launches: no extra attachment config is required as long as Euler View can uniquely match the run by `slurm.job_id`.
+- If neither `stream_attach_token` nor a unique SLURM match is available, run streaming still works, but launch-to-run linking in Euler View may remain unresolved.
+
+For new integrations, prefer `stream_attach_token`. The older `launch_id` / `launchId` config keys are still accepted as compatibility aliases, but new code should treat the attachment value as an opaque stream token rather than a launch ID.
+
 ## Directory structure
 
 Each `euler_train.init(dir=...)` call creates a timestamped subdirectory under `{dir}/runs/`:
@@ -67,6 +118,7 @@ Each `euler_train.init(dir=...)` call creates a timestamped subdirectory under `
         │   └── epoch_{N}.pt
         └── outputs/
             └── epoch_{N}_step_{M}/
+                ├── manifest.json
                 └── {output_type}/
                     ├── pred/
                     ├── gt/
@@ -80,7 +132,7 @@ The run ID and directory are available as `run.run_id` and `run.dir`.
 
 ## API reference
 
-### `euler_train.init(dir, config=None, meta=None, output_formats=None, output_visualization=None, run_id=None, datasets=None, run_name=None, evaluations=None, mode=None) → Run`
+### `euler_train.init(dir, config=None, meta=None, output_formats=None, output_visualization=None, run_id=None, datasets=None, run_name=None, evaluations=None, mode=None, stream=None, metric_naming=None) → Run`
 
 Creates the run directory and writes `meta.json`, `config.json`, `code_ref.json`, and `run_environment.json`. On resume (`run_id` provided), only `meta.json` and `config.json` are updated. `meta.json` also maintains an `updated_at` map with last-write timestamps for tracked artifacts.
 
@@ -96,6 +148,8 @@ Creates the run directory and writes `meta.json`, `config.json`, `code_ref.json`
 | `run_name` | `str \| None` | Optional human-readable run label stored in `meta.json`. |
 | `evaluations` | `dict[str, dict] \| None` | Optional evaluation key → entry map. See [Evaluations](#evaluations). |
 | `mode` | `str \| None` | Optional process label such as `"train"`, `"val"`, or `"eval"`. When set, lifecycle and crash details are also written under `meta.json["modes"][mode]`. |
+| `stream` | `dict \| EulerViewStreamConfig \| consumer \| sequence \| None` | Optional output stream config / consumer. A mapping with `base_url`, `model_id`, and `api_token` (or `access_token`) enables best-effort streaming to Euler View. Prefer `stream_attach_token` for explicit launch attachment; otherwise the package falls back to `meta.json["slurm"]["job_id"]` when available. |
+| `metric_naming` | `dict[str, Any] \| None` | Optional structured metric naming declaration stored in `meta.json["metric_naming"]` and included in the stream init event. |
 
 ---
 
@@ -112,7 +166,7 @@ run.log({"rgb.psnr": 28.3, "depth.mae": 0.03}, step=100, epoch=1, mode="val")
 
 ---
 
-### `run.save_outputs(*, epoch=None, step=None, **output_types)`
+### `run.save_outputs(*, epoch=None, step=None, metadata=None, **output_types)`
 
 Saves arrays/images to `outputs/epoch_{N}_step_{M}/{output_type}/{slot}/`.
 
@@ -125,10 +179,11 @@ Each output type is a dict with these slot keys:
 | `input` | Model input |
 | `aux` | Dict of named auxiliary outputs (each becomes a subdirectory) |
 
-Values can be:
+Slot values can be:
 - A single numpy array, torch tensor, or PIL Image
 - A list of the above (saved as `0000.ext`, `0001.ext`, ...)
 - A 4D numpy/torch array (split along dim 0 as a batch)
+- A dict mapping custom string IDs to items (saved as `{id}.ext`)
 
 ```python
 run.save_outputs(
@@ -142,6 +197,61 @@ run.save_outputs(
 )
 ```
 
+#### Named sample IDs
+
+When slot values are dicts with string keys, files are saved with the key as the basename instead of sequential indices. This is useful for matching outputs back to specific input samples:
+
+```python
+run.save_outputs(
+    epoch=1, step=500,
+    rgb=dict(pred={"scene_042": img_a, "scene_117": img_b}),
+)
+# produces: rgb/pred/scene_042.png, rgb/pred/scene_117.png
+```
+
+#### Source metadata
+
+The optional `metadata` parameter attaches dataset provenance and other context to the output manifest. When provided, the `dataset` key is required and must be a non-empty string:
+
+```python
+run.save_outputs(
+    epoch=1, step=500,
+    metadata={"dataset": "vkitti2", "split": "val", "scene": "0001"},
+    rgb=dict(pred={"frame_042": img_a}),
+)
+```
+
+#### Output manifest
+
+Each `save_outputs` call writes a `manifest.json` alongside the saved files, recording what was written and how. The manifest is also streamed via the `output_snapshot` event when streaming is active.
+
+```json
+{
+  "version": 1,
+  "epoch": 1,
+  "step": 500,
+  "metadata": {"dataset": "vkitti2", "split": "val"},
+  "output_types": {
+    "rgb": {
+      "pred": {
+        "id_mode": "named",
+        "files": [
+          {"sample_id": "scene_042", "filename": "scene_042.png", "format": "png"}
+        ]
+      },
+      "gt": {
+        "id_mode": "indexed",
+        "files": [
+          {"sample_id": 0, "filename": "0000.png", "format": "png"}
+        ]
+      }
+    }
+  },
+  "format_overrides": {},
+  "visualization_overrides": {}
+}
+```
+
 Torch tensors in `(C,H,W)` or `(B,C,H,W)` layout are automatically transposed to channels-last before saving.
 
 Pass `None` for any slot or output type to skip it.

{euler_train-2.6.0 → euler_train-2.7.0}/euler_train/stream.py

@@ -56,6 +56,8 @@ class EulerViewStreamConfig:
     api_token: str | None = None
     access_token: str | None = None
     stream_token: str | None = None
+    stream_attach_token: str | None = None
+    launch_id: str | None = None
     datasource_id: int | None = None
     euler_train_dir: str | None = None
     run_dir: str | None = None
@@ -84,6 +86,24 @@ class EulerViewStreamConfig:
             self.stream_token,
             field_name="stream.stream_token",
         )
+        self.stream_attach_token = _normalize_optional_text(
+            self.stream_attach_token,
+            field_name="stream.stream_attach_token",
+        )
+        self.launch_id = _normalize_optional_text(
+            self.launch_id,
+            field_name="stream.launch_id",
+        )
+        if (
+            self.stream_attach_token is not None
+            and self.launch_id is not None
+            and self.stream_attach_token != self.launch_id
+        ):
+            raise ValueError(
+                "stream.stream_attach_token and stream.launch_id must match when both are provided",
+            )
+        if self.stream_attach_token is None:
+            self.stream_attach_token = self.launch_id
         self.euler_train_dir = _normalize_optional_text(
             self.euler_train_dir,
             field_name="stream.euler_train_dir",
@@ -191,6 +211,7 @@ class EulerViewStreamConsumer:
         self._context: StreamContext | None = None
         self._pending: list[dict[str, Any]] = []
         self._stream_token: str | None = config.stream_token
+        self._resolved_stream_attach_token: str | None = config.stream_attach_token
         self._session_expires_at = float("inf") if config.stream_token else 0.0
         self._ingest_url = f"{self.config.base_url}/api/model-run-stream/ingest"
         self._last_flush_at = 0.0
@@ -299,6 +320,14 @@ class EulerViewStreamConsumer:
             "runDir": self.config.run_dir or str(self._context.run_dir),
             "eulerTrainDir": self.config.euler_train_dir or str(self._context.runs_dir),
         }
+        stream_attach_token = (
+            self.config.stream_attach_token or self._resolved_stream_attach_token
+        )
+        if stream_attach_token is not None:
+            session_body["streamAttachToken"] = stream_attach_token
+        slurm_job_id = _extract_slurm_job_id(self._context.meta)
+        if slurm_job_id is not None:
+            session_body["slurmJobId"] = slurm_job_id
         if self.config.datasource_id is not None:
             session_body["datasourceId"] = self.config.datasource_id
         if self.config.session_expires_in_sec is not None:
@@ -328,6 +357,10 @@ class EulerViewStreamConsumer:
         else:
             self._ingest_url = f"{self.config.base_url}/api/model-run-stream/ingest"
 
+        resolved_stream_attach_token = _extract_session_stream_attach_token(payload)
+        if resolved_stream_attach_token is not None:
+            self._resolved_stream_attach_token = resolved_stream_attach_token
+
         self._session_expires_at = _parse_timestamp(payload.get("expiresAt"))
         self._next_retry_at = 0.0
         return True
@@ -448,6 +481,12 @@ def _config_from_mapping(raw: Mapping[str, Any]) -> EulerViewStreamConfig:
         access_token=_mapping_get(raw, "access_token", "accessToken"),
         api_token=_mapping_get(raw, "api_token", "apiToken"),
         stream_token=_mapping_get(raw, "stream_token", "streamToken"),
+        stream_attach_token=_mapping_get(
+            raw,
+            "stream_attach_token",
+            "streamAttachToken",
+        ),
+        launch_id=_mapping_get(raw, "launch_id", "launchId"),
         datasource_id=_mapping_get(raw, "datasource_id", "datasourceId"),
         euler_train_dir=_mapping_get(raw, "euler_train_dir", "eulerTrainDir"),
         run_dir=_mapping_get(raw, "run_dir", "runDir"),
@@ -484,6 +523,45 @@ def _mapping_get(
     return default
 
 
+def _extract_slurm_job_id(meta: Mapping[str, Any] | None) -> int | None:
+    if not isinstance(meta, Mapping):
+        return None
+    slurm = meta.get("slurm")
+    if not isinstance(slurm, Mapping):
+        return None
+    raw = slurm.get("job_id")
+    if isinstance(raw, bool):
+        return None
+    if isinstance(raw, int):
+        return raw if raw > 0 else None
+    if isinstance(raw, float):
+        return int(raw) if raw.is_integer() and raw > 0 else None
+    if isinstance(raw, str):
+        text = raw.strip()
+        if not text:
+            return None
+        try:
+            value = int(text, 10)
+        except ValueError:
+            return None
+        return value if value > 0 else None
+    return None
+
+
+def _extract_session_stream_attach_token(payload: Mapping[str, Any]) -> str | None:
+    run = payload.get("run")
+    if not isinstance(run, Mapping):
+        return None
+    stream_attach_token = run.get("streamAttachToken")
+    if not isinstance(stream_attach_token, str):
+        legacy_launch_id = run.get("launchId")
+        if not isinstance(legacy_launch_id, str):
+            return None
+        stream_attach_token = legacy_launch_id
+    normalized = stream_attach_token.strip()
+    return normalized or None
+
+
 def _normalize_optional_text(
     value: Any,
     *,

{euler_train-2.6.0 → euler_train-2.7.0}/euler_train.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euler-train
-Version: 2.6.0
+Version: 2.7.0
 Summary: Lightweight file-based experiment logger for PyTorch
 Requires-Python: >=3.9
 Requires-Dist: numpy

{euler_train-2.6.0 → euler_train-2.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "euler-train"
-version = "2.6.0"
+version = "2.7.0"
 description = "Lightweight file-based experiment logger for PyTorch"
 requires-python = ">=3.9"
 dependencies = ["numpy", "euler-dataset-contract"]