expr-tracker 0.2.7__tar.gz → 0.3.0__tar.gz

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: expr_tracker
-Version: 0.2.7
+Version: 0.3.0
 Summary: Local-first experiment tracking with queryable history and expression-based alerts on your training metrics
 Project-URL: Homepage, https://hspk.github.io/expr_tracker/
 Project-URL: Documentation, https://hspk.github.io/expr_tracker/

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/docs/design.md

@@ -106,6 +106,13 @@ Two differences from a log call:
 - the metrics are built here from floats and ints, so they skip `RecordCodec`.
   That encoder costs 5 µs, which a per-sub-step call cannot afford.
 
+Metrics are named `<measurement>/<span path>` — `time_ms/step/forward`, not
+`step/forward/time_ms`. A tracker UI groups by the segment before the first
+slash, so putting the path first would file every timing under the thing it
+timed, mixing a span called `train` in with `train/loss` and scattering a deep
+tree across the metric namespace. Measurement-first keeps all timings in one
+group and leaves the caller's names alone.
+
 A span never commits a step: durations ride along with whatever `log()` commits,
 so timing a region adds no row. The tree, with timestamps and attributes, goes to
 `spans[.stream][.rankN].jsonl` through a second `JsonlWriter`, enqueued rather

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/docs/getting-started.md

@@ -48,15 +48,34 @@ et.finish()
 This writes:
 
 ```
-./tracker/jsonl/mnist/baseline/
-├── metrics.jsonl      # one JSON object per step
-├── metrics.meta.json  # index sidecar, for fast resume and seeks
-├── config.json
-├── summary.json
-└── artifacts.jsonl    # lineage: what this run produced and consumed
+./tracker/jsonl/            # the root, from dir=
+└── mnist/                  # the project
+    ├── baseline/           # this run
+    │   ├── metrics.jsonl      # one JSON object per step
+    │   ├── metrics.meta.json  # index sidecar, for fast resume and seeks
+    │   ├── config.json
+    │   ├── summary.json
+    │   └── artifacts.jsonl    # lineage: what this run produced and consumed
+    └── artifacts/          # shared by every run of the project
 ```
 
-Change the location with `et.init(dir="/data/runs")`.
+`dir` is a **root**, so a run lands in `<dir>/<project>/<name>`. That is what
+lets a project's runs share and deduplicate artifacts, and what lets a resume
+find its files from the project and name alone. The path is logged at `init()`,
+and `et.get_run().dir` returns it.
+
+```python
+et.init(project="mnist", name="baseline", dir="/data/runs")
+# -> /data/runs/mnist/baseline
+
+et.init(project="mnist", name="baseline", run_dir="/data/runs/exp-42")
+# -> /data/runs/exp-42, exactly
+```
+
+Use `run_dir` when something else already chose the path — a scheduler's output
+directory, say. Artifacts then live in `<run_dir>/artifacts` and are no longer
+shared with the project's other runs, which is the price of the flat layout.
+Passing both `dir` and `run_dir` is an error.
 
 ## Reading it back
 

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/docs/guide/artifacts.md

@@ -33,6 +33,17 @@ assert b.version == 0                          # same bytes, same version
 et.use_artifact("model:best").version == 0
 ```
 
+## Where they live
+
+The store sits at `<dir>/<project>/artifacts`, beside the project's runs rather
+than inside any one of them. That is what makes sharing and deduplication
+possible: a later run asking for `model:best` finds what an earlier one logged.
+
+Naming a run's directory with `run_dir` moves the store to
+`<run_dir>/artifacts`. The run becomes self-contained, and gives up sharing with
+the project's other runs — nothing dedupes against them, and `use_artifact`
+cannot see them.
+
 ## Storage modes
 
 ```python

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/docs/guide/spans.md

@@ -20,15 +20,29 @@ The step's row then carries:
 
 ```jsonc
 {"_step": 42, "_time": ...,
- "forward/duration_ms": 31.2,  "forward/count": 1,
- "forward/attention/duration_ms": 18.4, "forward/attention/count": 1,
- "forward/mlp/duration_ms": 9.1, "forward/mlp/count": 1,
- "backward/duration_ms": 44.7, "backward/count": 1,
+ "time_ms/forward": 31.2,  "count/forward": 1,
+ "time_ms/forward/attention": 18.4, "count/forward/attention": 1,
+ "time_ms/forward/mlp": 9.1, "count/forward/mlp": 1,
+ "time_ms/backward": 44.7, "count/backward": 1,
  "loss": 0.31}
 ```
 
-Nested names join with `/`, so `forward/norm` and `backward/norm` stay distinct,
-and the alert language reads them directly.
+## How a span metric is named
+
+```
+<what was measured>/<the span's path>
+```
+
+Nested span names join with `/`, so `forward/norm` and `backward/norm` stay
+distinct, and the alert language reads them directly.
+
+The measurement comes **first**, not last, because a tracker UI groups metrics
+by the segment before the first slash. Naming them `forward/time_ms` would file
+every timing under whatever it timed — a span called `train` would put its
+duration in the same group as `train/loss`, and a deep tree would scatter
+timings across the whole metric namespace. This way every timing shares one
+`time_ms` group, every count a `count` group, and your own metric names stay
+yours.
 
 ## Forms
 
@@ -56,7 +70,7 @@ for layer in layers:            # 32 layers
 ```
 
 ```jsonc
-{"layer/duration_ms": 412.8, "layer/count": 32}
+{"time_ms/layer": 412.8, "count/layer": 32}
 ```
 
 The total is usually what you want; divide by the count for the mean.
@@ -92,7 +106,7 @@ already handles it:
 ```python
 et.init(..., alert_rules=[
     "mean(data/load_ms[50]) > 200 => warning: data loading is slowing down",
-    "forward/duration_ms > 3 * mean(forward/duration_ms[100]) => error: slow step",
+    "time_ms/forward > 3 * mean(time_ms/forward[100]) => error: slow step",
 ])
 ```
 
@@ -215,9 +229,9 @@ with et.span("forward", plugins=[CpuTime(), TorchMemory()]):
 
 ```python
 et.history(1)[0]
-# {"forward/duration_ms": 41.2, "forward/count": 1,
-#  "forward/cpu_percent": 101.2, "forward/cpu_time_ms": 41.4,
-#  "forward/gpu_mem_peak_mb": 456.1, "forward/gpu_mem_delta_mb": 8.1, ...}
+# {"time_ms/forward": 41.2, "count/forward": 1,
+#  "cpu_percent/forward": 101.2, "cpu_time_ms/forward": 41.4,
+#  "gpu_mem_peak_mb/forward": 456.1, "gpu_mem_delta_mb/forward": 8.1, ...}
 ```
 
 Like `print_fn`, plugins are inherited by children, can be overridden per span,
@@ -227,8 +241,10 @@ code being measured, and a plugin that fails on every span is logged only once.
 Plugin metrics also reach `spans.jsonl` and show up in the `et trace` viewer
 under a span's *Arguments*.
 
-`duration_ms` and `count` belong to the span. A plugin returning either has that
-key dropped with a warning, so nothing can quietly redefine what a span measured.
+`time_ms` and `count` belong to the span. A plugin returning either has that key
+dropped with a warning, so nothing can quietly redefine what a span measured. Any
+other key becomes its own group: a plugin returning `gpu_mem_peak_mb` writes
+`gpu_mem_peak_mb/<path>`.
 
 ### Built-ins
 

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/docs/reference/api.md

@@ -16,6 +16,7 @@ et.init(
     name: str | None = None,
     entity: str | None = None,
     dir: str | None = None,
+    run_dir: str | None = None,
     notes: str | None = None,
     tags: list[str] | None = None,
     resume: bool | str | None = "allow",
@@ -33,9 +34,15 @@ et.init(
 ```
 
 Starts a run and publishes it as the process-wide current run. `name` defaults to a
-timestamp. `dir` defaults to `./tracker/jsonl`. Extra keyword arguments are
+timestamp. Extra keyword arguments are
 [history options](configuration.md#history-options).
 
+`dir` is a root and defaults to `./tracker/jsonl`, so the run lands in
+`<dir>/<project>/<name>` and artifacts in `<dir>/<project>/artifacts`. `run_dir`
+names this run's directory outright, putting its artifacts in
+`<run_dir>/artifacts`; passing both is an error. The resolved path is logged at
+`init()` and available as `Run.dir`.
+
 `resume` is forwarded to wandb and trackio. The local history always continues an
 existing run directory, so re-running with the same project and name picks up the
 step cursor whatever `resume` says; use a new name to start clean.
@@ -174,8 +181,10 @@ span.metrics                                    # what the plugins measured
 ```
 
 `et.span` is also an async context manager and a decorator. A closed span adds
-`<path>/duration_ms`, `<path>/count` and one key per plugin metric to the open
-row, and appends the full record to `spans.jsonl`.
+`time_ms/<path>`, `count/<path>` and one key per plugin metric to the open row,
+and appends the full record to `spans.jsonl`. The measurement leads the name so
+that a UI groups every timing together instead of filing each one under whatever
+it timed; `Span.duration_ms` is the same number on the object.
 
 `print_fn(line)` announces the span's start and end, indented two spaces per level.
 `plugins` measure a resource across the span; both are inherited by child spans

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/examples/profile_step.py

@@ -65,15 +65,17 @@ def main(argv=None) -> Path:
         et.log({"train/loss": 1.0 / (step + 1)})
 
     # Timings are ordinary metrics, so they query like any other
-    rows = et.history(-1, metrics=["step/duration_ms", "step/data/read/duration_ms"])
-    total = sum(row["step/duration_ms"] for row in rows) / len(rows)
-    waiting = sum(row["step/data/read/duration_ms"] for row in rows) / len(rows)
+    # A span's timing is named "<measurement>/<span path>", so every timing in
+    # the run shares one "time_ms" group instead of polluting the metric tree
+    rows = et.history(-1, metrics=["time_ms/step", "time_ms/step/data/read"])
+    total = sum(row["time_ms/step"] for row in rows) / len(rows)
+    waiting = sum(row["time_ms/step/data/read"] for row in rows) / len(rows)
     print(f"\nmean step {total:.2f}ms, of which {waiting:.2f}ms waiting on IO")
     print(f"the data loader is {100 * waiting / total:.0f}% of the step")
 
     last = et.history(1)[0]
     print("\nwhat one step recorded:")
-    for key in sorted(k for k in last if k.endswith(("duration_ms", "cpu_percent"))):
+    for key in sorted(k for k in last if k.startswith(("time_ms/", "cpu_percent/"))):
         print(f"  {key:<38} {last[key]:.2f}")
 
     et.finish()
@@ -82,9 +84,7 @@ def main(argv=None) -> Path:
     print(f"\n{spans} spans -> {output}")
     print("Open it at https://ui.perfetto.dev")
     print("Timings are metrics, so they alert too:")
-    print(
-        '  et.init(..., alert_rules=["step/duration_ms > 500 => warning: slow step"])'
-    )
+    print('  alert_rules=["time_ms/step > 500 => warning: slow step"]')
     return output
 
 

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/src/expr_tracker/history/naming.py

@@ -8,7 +8,9 @@ from __future__ import annotations
 
 import os
 import re
+from pathlib import Path
 
+DEFAULT_ROOT = "./tracker/jsonl"
 STREAM_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]*$")
 RANK_PATTERN = re.compile(r"rank\d+")
 
@@ -72,3 +74,37 @@ def parse_stream(filename: str) -> str | None:
     if parts and RANK_PATTERN.fullmatch(parts[-1]):
         parts = parts[:-1]
     return parts[0] if parts else None
+
+
+def resolve_log_dir(
+    project: str, name: str, dir: str | None = None, run_dir: str | None = None
+) -> Path:
+    """Where this run's files go.
+
+    ``dir`` is a root holding many projects, so a run lands in
+    ``<dir>/<project>/<name>``. That layout is what lets artifacts be shared and
+    deduplicated across a project's runs, and what lets a resume find its files
+    from the project and name alone. ``run_dir`` opts out of it and names the
+    directory outright, for when something else already chose the path.
+    """
+    if run_dir is not None:
+        if dir is not None:
+            raise ValueError(
+                "Pass either dir (a root, giving <dir>/<project>/<name>) or "
+                "run_dir (this run's directory), not both"
+            )
+        return Path(run_dir)
+    return Path(dir or DEFAULT_ROOT) / project / name
+
+
+def resolve_artifact_root(
+    project: str, dir: str | None = None, run_dir: str | None = None
+) -> Path:
+    """Where artifacts live: beside the project's runs, or inside a lone run.
+
+    A project-level store is what makes deduplication across runs possible, so
+    ``run_dir`` gives up that sharing in exchange for a self-contained directory.
+    """
+    if run_dir is not None:
+        return Path(run_dir) / "artifacts"
+    return Path(dir or DEFAULT_ROOT) / project / "artifacts"

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/src/expr_tracker/history/store.py

@@ -27,6 +27,7 @@ from .codec import RESERVED_KEYS, RecordCodec, encode_line
 from .frame import project, to_output
 from .naming import (
     metrics_filename,
+    resolve_log_dir,
     sidecar_filename,
     spans_filename,
     validate_stream,
@@ -184,6 +185,7 @@ class HistoryStore:
         name: str | None = None,
         config: dict | None = None,
         dir: str | None = None,
+        run_dir: str | None = None,
         on_commit: Callable[[dict], None] | None = None,
         **options,
     ) -> HistoryStore:
@@ -193,7 +195,7 @@ class HistoryStore:
             name = time.strftime("run-%Y%m%d-%H%M%S")
             logger.warning(f"No run name provided, using generated name: {name}")
         self.project, self.name = project, name
-        self.log_dir = Path(dir or "./tracker/jsonl") / project / name
+        self.log_dir = resolve_log_dir(project, name, dir, run_dir)
         self.log_dir.mkdir(parents=True, exist_ok=True)
         self.stream = opts.stream
         self.config_fp = self.log_dir / sidecar_filename("config", opts.stream, "json")
@@ -221,6 +223,10 @@ class HistoryStore:
         self._resume(opts.alert_window)
         self._write_config(config)
         self._register_atexit()
+        # Say where the files went: dir is a root, so the run is two levels below
+        # what the caller passed, and guessing that is nobody's idea of fun
+        resumed = f", resuming at step {self._next_step}" if self._next_step else ""
+        logger.info(f"Run {self.name!r} -> {self.log_dir}{resumed}")
         return self
 
     def _close_previous(self):

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/src/expr_tracker/run.py

@@ -10,14 +10,13 @@ import os
 import threading
 import time
 from collections.abc import Sequence
-from pathlib import Path
 from typing import Any, Literal
 
 from loguru import logger
 
 from .artifacts import Artifact, ArtifactStore, coerce_artifact
 from .history import HistoryStore, current_rank, resolve_commit
-from .history.naming import sidecar_filename
+from .history.naming import resolve_artifact_root, sidecar_filename
 from .summary import Summary
 
 _lock = threading.RLock()
@@ -63,6 +62,7 @@ class Run:
         name: str | None = None,
         entity: str | None = None,
         dir: str | None = None,
+        run_dir: str | None = None,
         notes: str | None = None,
         tags: list[str] | None = None,
         resume: bool | str | None = "allow",
@@ -97,6 +97,7 @@ class Run:
             name=name,
             config=self.config or None,
             dir=dir,
+            run_dir=run_dir,
             print_to_screen=print_to_screen,
             on_commit=self._on_commit,
             stream=stream,
@@ -113,7 +114,7 @@ class Run:
         atexit.register(self._save_summary_at_exit)
         self._closers.append(lambda: atexit.unregister(self._save_summary_at_exit))
         self.artifacts = ArtifactStore(
-            root=Path(dir or "./tracker/jsonl") / project / "artifacts"
+            root=resolve_artifact_root(project, dir, run_dir)
         )
 
         self.rank = current_rank()

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/src/expr_tracker/spans.py

@@ -23,9 +23,9 @@ from loguru import logger
 # and each asyncio task gets its own copy, which is exactly span nesting.
 _STACK: ContextVar[tuple[Span, ...]] = ContextVar("et_span_stack", default=())
 
-DURATION_SUFFIX = "duration_ms"
-COUNT_SUFFIX = "count"
-RESERVED_METRICS = frozenset({DURATION_SUFFIX, COUNT_SUFFIX})
+TIME_METRIC = "time_ms"
+COUNT_METRIC = "count"
+RESERVED_METRICS = frozenset({TIME_METRIC, COUNT_METRIC})
 # Spaces, not a tab: a terminal measures tab stops from the start of the line, so
 # behind a log prefix the first level collapses to whatever is left of the stop.
 # Two spaces render the same width wherever the line begins.
@@ -45,6 +45,19 @@ def _warn_once(*key: str) -> None:
     logger.warning(key[-1])
 
 
+def span_metric(metric: str, path: str) -> str:
+    """Name a span's measurement: what was measured first, then what it measured.
+
+    ``time_ms/step/forward`` rather than ``step/forward/time_ms``, because a
+    tracker UI groups metrics by the segment before the first slash. With the
+    path first, every timing lands in the group of whatever it timed, next to
+    that name's real metrics -- a span called ``train`` would file its duration
+    beside ``train/loss``. With the measurement first, all timings share one
+    group and the metric namespace stays the caller's.
+    """
+    return f"{metric}/{path}"
+
+
 def _safely(plugin: Any, hook: str, span: Span):
     """Call a plugin hook; a broken plugin must not break the measured code."""
     method = getattr(plugin, hook, None)
@@ -233,9 +246,12 @@ class Span:
         if self._store is None:
             return
         metrics = {
-            f"{self.path}/{DURATION_SUFFIX}": self.duration_ms,
-            f"{self.path}/{COUNT_SUFFIX}": 1,
-            **{f"{self.path}/{key}": value for key, value in self.metrics.items()},
+            span_metric(TIME_METRIC, self.path): self.duration_ms,
+            span_metric(COUNT_METRIC, self.path): 1,
+            **{
+                span_metric(key, self.path): value
+                for key, value in self.metrics.items()
+            },
         }
         record = {
             "name": self.path,

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/src/expr_tracker/tracker.py

@@ -23,6 +23,7 @@ def init(
     name: str | None = None,
     entity: str | None = None,
     dir: str | None = None,
+    run_dir: str | None = None,
     notes: str | None = None,
     tags: list[str] | None = None,
     resume: bool | Literal["allow", "never", "must", "auto"] | None = "allow",
@@ -36,6 +37,10 @@ def init(
 ) -> Run:
     """Initialise the tracker. Local jsonl history is always on, whatever ``backends``.
 
+    ``dir`` is a root holding many projects, so the run lands in
+    ``<dir>/<project>/<name>``. Pass ``run_dir`` instead to name the directory
+    outright.
+
     Extra keyword arguments are forwarded to the history store (``cache_bytes``,
     ``alert_window``, ``max_open_seconds``, ``step_policy``, ``buffer_size``, ...).
     """
@@ -48,6 +53,7 @@ def init(
             name=name,
             entity=entity,
             dir=dir,
+            run_dir=run_dir,
             notes=notes,
             tags=tags,
             resume=resume,

{expr_tracker-0.2.7 → expr_tracker-0.3.0}/tests/test_examples.py

@@ -310,14 +310,14 @@ def test_profiling_records_the_span_tree_as_metrics(profile_step, tmp_path):
     output = profile_step.main(argv(**{"--steps": 5, "--dir": tmp_path}))
     row = read_history(output.parent, 1)[0]
     for key in (
-        "step/duration_ms",
-        "step/data/read/duration_ms",
-        "step/data/collate/duration_ms",
-        "step/forward/duration_ms",
-        "step/backward/duration_ms",
+        "time_ms/step",
+        "time_ms/step/data/read",
+        "time_ms/step/data/collate",
+        "time_ms/step/forward",
+        "time_ms/step/backward",
     ):
         assert key in row, key
-    assert row["step/duration_ms"] >= row["step/forward/duration_ms"]
+    assert row["time_ms/step"] >= row["time_ms/step/forward"]
 
 
 def test_profiling_attaches_the_cpu_plugin_to_every_span(profile_step, tmp_path):
@@ -325,8 +325,8 @@ def test_profiling_attaches_the_cpu_plugin_to_every_span(profile_step, tmp_path)
 
     output = profile_step.main(argv(**{"--steps": 5, "--dir": tmp_path}))
     row = read_history(output.parent, 1)[0]
-    assert row["step/forward/cpu_percent"] > 50  # a spin loop
-    assert row["step/data/read/cpu_percent"] < 50  # a sleep
+    assert row["cpu_percent/step/forward"] > 50  # a spin loop
+    assert row["cpu_percent/step/data/read"] < 50  # a sleep
 
 
 def test_profiling_writes_a_loadable_trace(profile_step, tmp_path):