expr-tracker 0.3.5__tar.gz → 0.4.0__tar.gz

{expr_tracker-0.3.5 → expr_tracker-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: expr_tracker
-Version: 0.3.5
+Version: 0.4.0
 Summary: Local-first experiment tracking with queryable history and expression-based alerts on your training metrics
 Project-URL: Homepage, https://hspk.github.io/expr_tracker/
 Project-URL: Documentation, https://hspk.github.io/expr_tracker/

{expr_tracker-0.3.5 → expr_tracker-0.4.0}/docs/architecture.md

@@ -58,7 +58,7 @@ Run.log ──> Summary.observe(data)          # last value per metric
         └──> HistoryStore.log
                  │  encode once (jsonable_encoder)
                  │  merge into the open row for the current step
-                 ▼  commit when the step advances / commit=True / finish / timeout
+                 ▼  commit when the step advances / commit=True / finish / flush(commit_open=True)
              HistoryStore._emit
                  ├── _store_row(record, line)         (holds the store lock)
                  │      ├── cache.append((row, step, line))  # ordinal allocated here
@@ -125,12 +125,13 @@ plain functions over a sequence, so they hold no lock and are tested directly.
 
 ### Rows versus steps
 
-A step normally occupies one physical row, but a `max_open_seconds` timeout followed
-by more data for that step writes a *patch line*, so one step can span several rows.
+A step normally occupies one physical row, but an explicit commit followed by
+more data for that step writes a *patch line*, so one step can span several rows.
 The two are kept strictly separate in the read path:
 
-* `JsonlReader.tail_rows(n)` returns **physical rows**; `JsonlReader.tail(n)` returns
-  **merged steps** and widens its own read until it has one whole step to spare.
+* `JsonlReader.tail_rows(n)` returns **physical rows**; `JsonlReader.tail(n)` selects
+  the most recently written **steps** and merges all their patches. Unique-step
+  files use reverse reads; arbitrary patch histories require scanning older data.
 * `HistoryStore._collect_tail(steps)` works in rows, then `_take_steps` merges and
   trims by *step*, so `get(n)` never returns a half-merged oldest row.
 * The cache stores each row's step next to its bytes, so `_view_tail` snapshots the
@@ -156,16 +157,38 @@ Disk fallback only happens once rows have been evicted or the run was resumed;
 
 | Lock / thread | Owner | Protects |
 | --- | --- | --- |
-| `HistoryStore._lock` (RLock) | store | open row, cache, series, row ordinal, writer enqueue |
+| `HistoryStore._lifecycle_lock` | store | reinitialization and shutdown |
+| `HistoryStore._write_lock` (RLock) | store | accepted write transactions and row admission |
+| `HistoryStore._lock` (RLock) | store | open row, cache, series and row ordinal |
+| notification condition | store | ordered commit tickets, delivery ownership and completion |
 | `JsonlWriter._lock` (RLock) | writer | buffer, index, meta fields |
 | `JsonlWriter._write_lock` | writer | batch swap + file append (ordering) |
-| open-row `threading.Timer` | store | commits a stale open row; carries a generation token |
-| buffer `threading.Timer` | writer | flushes records that sat in memory too long |
+| reusable writer worker | writer | asynchronous span writes and timed buffer flushing; drained at shutdown |
 | `CompiledRule.lock` | engine | one rule's state machine transitions |
 | dispatch worker thread | dispatcher | drains the send queue; drained at exit |
 | watchdog thread | engine | evaluates time-based rules when no logs arrive |
 
-Lock order is always `_write_lock` → `_lock`; nothing acquires them the other way.
+Store writes enter the transaction lock before the store-state lock. Shutdown
+rejects new writes, waits for accepted metric/span transactions, then flushes and
+closes the writers before closing downstream sinks. Writer workers never acquire
+store locks. Open rows have no timeout and no background commit thread.
+Admission waits outside the store-state lock, so queue backpressure does not
+block history reads. Rejected records do not consume cache row ordinals or reach
+commit callbacks.
+Read snapshots acquire the metric writer lock before the store-state lock, the
+same order used when writer error handlers inspect history.
+
+Commit notifications are staged in write order, then delivered after releasing
+transaction/state locks. Committing callers wait for their own tickets; span-only
+writes do not wait for unrelated callbacks. Delivery is serial and supports
+recursive logging/flush, while lifecycle operations are rejected inside callbacks.
+Reinitialization drains the old run's notifications before replacing its metadata.
+Alert-series updates occur during ordered delivery, not admission, preventing
+queued future rows from changing an earlier notification's scalar/window results.
+
+Run/store/span/plugin objects are process-owned. Fork resets process-global
+locks and the active run/stack in the child; inherited objects are rejected before
+acquiring their locks, and inherited exit hooks do not flush parent data.
 
 ## Extension points
 

{expr_tracker-0.3.5 → expr_tracker-0.4.0}/docs/design.md

@@ -22,7 +22,7 @@ Three write stages:
 
 ```
 log() ──merge──> open row (current step, in memory)
-                   │ step advances / commit=True / flush / finish / max_open_seconds
+                   │ step advances / commit=True / flush(commit_open=True) / finish
                    ▼
                 write buffer (adaptive: buffer_size / buffer_interval / max_buffer_seconds)
                    │
@@ -45,10 +45,9 @@ log(metrics: dict, step: int | None = None, commit: bool | None = None)
 - `step_policy="monotonic"` (default) drops backward writes with a warning;
   `"allow"` accepts them and marks `meta.sorted = false`.
 - Writing the same key twice in one step is last-wins, warned once per key per run.
-- `max_open_seconds` (default 60s) commits an idle open row so a crash cannot lose
-  it. If more data arrives for that step afterwards it is written as a **patch line**
-  and `meta.has_duplicate_steps` is set; the reader merges by step, so correctness is
-  unaffected.
+- An open row has no time limit: elapsed time never commits it or advances the
+  step. If more data arrives for an explicitly committed step, it is written as a
+  **patch line** and `meta.has_duplicate_steps` is set; the reader merges by step.
 
 **The writer guarantees one line per step in ascending order; the reader assumes
 none of that** — it must cope with older files, stray writers and patch lines.
@@ -102,9 +101,11 @@ commit rules exist in one place regardless of which API produced the metrics.
 Two differences from a log call:
 
 - the merge **accumulates** rather than replaces, because the same span can run
-  many times in one step;
-- the metrics are built here from floats and ints, so they skip `RecordCodec`.
-  That encoder costs 5 µs, which a per-sub-step call cannot afford.
+  many times in one step: durations and counters sum, memory peaks take the
+  maximum, and utilization uses a duration-weighted mean;
+- built-in durations and counts are already floats and ints, so they skip
+  `RecordCodec`. Optional plugin measurements and attributes are encoded
+  independently, keeping bare spans cheap without accepting unserializable data.
 
 Metrics are named `<measurement>/<span path>` — `time_ms/step/forward`, not
 `step/forward/time_ms`. A tracker UI groups by the segment before the first
@@ -116,12 +117,16 @@ group and leaves the caller's names alone.
 A span never commits a step: durations ride along with whatever `log()` commits,
 so timing a region adds no row. The tree, with timestamps and attributes, goes to
 `spans[.stream][.rankN].jsonl` through a second `JsonlWriter`, enqueued rather
-than appended so a span does not pay for a flush decision of its own.
+than committed as metric rows. That writer's reusable background worker flushes completed
+spans even if no further spans or metrics arrive, without advancing the step.
 
 The committed history row is also the fan-out boundary. Summary, alerts and
 remote backends consume the exact row emitted by `HistoryStore`, rather than the
 individual `log()` calls that assembled it. Span and plugin metrics therefore
 reach every sink without a second metric path.
+Commit tickets preserve that order while callbacks execute outside the transaction
+lock. A slow sink blocks its committing callers, not unrelated span transactions;
+shutdown and reinitialization drain notifications before closing or replacing sinks.
 
 Plugins hang off the same two hooks. `start` runs before the clock starts and
 `end` after it stops, so a plugin never inflates the duration it reports on; what
@@ -130,12 +135,12 @@ means a plugin needs no knowledge of the store, the codec or the accumulate rule
 A plugin that raises is logged and dropped: a measurement must not break the
 thing it measures.
 
-Peak-memory plugins have to fight the fact that `torch.cuda.max_memory_allocated`
-is one global high-water mark. Resetting it per span is the only way to attribute
-a peak, but a child's reset destroys its parent's history. `TorchMemory` keeps a
-per-thread stack and, as each span closes, hands its parent back the peak the
-parent had reached before that child opened, so `max(counter, carried)` is
-correct at every level.
+Peak-memory plugins have to account for `torch.cuda.max_memory_allocated` being a
+global high-water mark. `TorchMemory` coordinates active spans per device and
+preserves every existing span's peak before a new span resets the counter.
+CPU measurements are paired by span identity rather than a thread stack.
+NVML samplers update bounded per-active-span aggregates, so long spans retain
+their full sampling window without keeping every historical sample.
 
 GPU utilisation cannot be attributed by reading NVML at a span boundary: the
 counter is a rolling average over NVML's own window, so for a 3 ms span it
@@ -451,8 +456,8 @@ Read these before changing the modules they mention.
    dispatch does not advance the state.
 9. **Delivery failures are only logged** (`fail_silently`); configuration errors
    raise at configuration time.
-10. **Open-row timers carry a generation token**: once the step advances, an old
-    timer must not commit the new open row.
+10. **Elapsed time never commits an open row**: a training step can run arbitrarily
+    long. Writer timers only flush buffered records; they do not advance the step.
 
 ---
 

{expr_tracker-0.3.5 → expr_tracker-0.4.0}/docs/guide/artifacts.md

@@ -28,6 +28,9 @@ recorded against it.
 
 Publishers sharing a store serialize version allocation, file copying, and index
 updates with a filesystem lock, including when they run in separate processes.
+On POSIX, filenames represented using Python's filesystem-surrogate characters
+round-trip through artifact indexes and retain distinct digests from literal
+backslash-escape names.
 
 ```python
 a = et.log_artifact("ckpt.pt", name="model")   # v0

expr_tracker-0.4.0/docs/guide/logging.md

@@ -0,0 +1,180 @@
+# Logging metrics
+
+## One step, one line
+
+`log()` merges metrics into an **open row** for the current step. The row is written
+when it is committed, so several `log()` calls for one step become one line:
+
+```python
+et.log({"loss": 1.0}, step=5)
+et.log({"acc": 0.9}, step=5)     # merged into the same row
+et.log({"loss": 0.9}, step=6)    # the step advanced -> step 5 is committed
+```
+
+This matters because training and evaluation usually run at different points in the
+loop but describe the same step.
+
+## Commit semantics
+
+`log(data, step=None, commit=None)` mirrors `wandb.log`.
+
+| Call | `commit` default | Behaviour |
+| --- | --- | --- |
+| `log(d)` | `True` | Merge, commit, advance the step |
+| `log(d, commit=False)` | — | Merge only |
+| `log(d, step=N)` | `False` | Commit when the step advances |
+| `log(d, step=N, commit=True)` | — | Commit immediately |
+
+An open row is also committed by `finish()`, by the process exit hook, and by
+`run.history.flush(commit_open=True)`. It never commits just because time passes,
+so a long training step stays on one row. A plain `run.history.flush()` only writes
+already committed rows; it does not commit the open row.
+
+`finish()` waits for accepted writes and their commit callbacks before closing
+downstream sinks. Concurrent `run.finish()` callers wait for the same shutdown
+completion; a failed shutdown is surfaced to waiters rather than reported as
+success. Call it outside commit/print and asynchronous alert-delivery callbacks;
+trying to finish or reinitialize from such a callback raises `RuntimeError` to
+avoid reentrant shutdown.
+Lifecycle calls are also rejected from writer diagnostic callbacks, history
+writer workers, unfinished write transactions, or an already-running lifecycle
+operation; use the caller thread after the logging operation returns.
+Writer diagnostics are emitted after writer locks are released and accounting is
+consistent, allowing error sinks to inspect history without locking out writers.
+
+Commit callbacks run outside the write transaction lock. A slow backend or
+metric-print callback therefore does not block another thread recording an
+uncommitted span. Committing callers still wait for their own notifications,
+which are delivered serially in commit order; recursive logging and flush are
+supported. No asynchronous callback worker or unbounded work queue is created.
+
+Delivery may occur on another committing caller's thread. Context variables
+come from the originating commit, but callbacks must not assume thread identity.
+`flush()` and `finish()` wait for outstanding notifications before returning.
+Alert-series windows advance with notification delivery, so a newer queued
+commit cannot leak into an earlier alert. Ordinary history queries can already
+see those accepted rows. Recursive logging advances the live alert series; when
+an outer callback resumes, it sees the nested delivered rows, not a frozen snapshot.
+
+```python
+et.log({"train/loss": 1.0}, commit=False)
+et.log({"train/grad_norm": 2.0}, commit=False)
+et.log({"lr": 3e-4})              # commits all three as one step
+```
+
+Once an explicit step is open, subsequent calls without `step=` (including
+completed spans) reuse it:
+
+```python
+et.log({"train/loss": 1.0}, step=100)
+et.log({"lr": 3e-4})              # commits step 100; the next step is 101
+```
+
+## Timestamps and liveness
+
+Every row's `_time` is a Unix timestamp in seconds, captured when its open row
+was first created. Later logs merged into that row do not replace the timestamp,
+and committing or flushing it does not change it.
+
+`no_data(duration)` uses the actual time the latest metric row was accepted,
+not its `_time`, so completing a long training step does not immediately look
+stale. Uncommitted updates, trace-only events and rejected rows do not refresh
+this commit clock. `age(metric)` uses the sample timestamp from the row instead.
+On resume, an invalid or non-finite timestamp is skipped for alert-window
+backfill with a warning, rather than preventing later valid samples from being
+restored. The original history rows remain available for inspection.
+
+## Step ordering
+
+Steps accept Python integers and integer-index values such as NumPy integer
+scalars. They are normalized to native integers before encoding. Strings and
+floating-point steps are rejected before payload consumption or open-row mutation,
+so an invalid step cannot poison later logs.
+
+By default steps must not go backwards. A backward step is dropped with a warning and
+reaches no sink — not the file, not the summary, not the alert rules, not wandb:
+
+```python
+et.log({"loss": 1.0}, step=10)
+et.log({"loss": 2.0}, step=3)     # dropped, with a warning
+```
+
+Pass `step_policy="allow"` to keep out-of-order writes. The file then contains
+repeated steps, and the reader merges them on the way out.
+
+!!! note
+    With `step_policy="allow"`, `history(n)` returns the *n* most recently **written**
+    steps rather than the *n* highest-numbered ones. `history(-1)` and `step_range`
+    are always step-ordered. A positive `n` combined with `step_range` selects the
+    highest matching steps. Live and offline queries use the same selection rules.
+
+Patch lines are merged in full even when separated by other steps or split across
+cache and disk. Reads pin an append-prefix boundary and do not flush unrelated
+span records or wait for commit callbacks. A damaged or unreadable metadata
+sidecar produces a warning and a scan of the readable data file.
+
+## What you can log
+
+Anything JSON-serialisable, plus the things ML code actually holds:
+
+```python
+et.log({
+    "loss": np.float32(0.5),          # numpy scalars -> python scalars
+    "grads": np.array([1.0, 2.0]),    # arrays -> lists
+    "config": PydanticModel(),        # pydantic models -> dicts
+    "when": datetime.now(),           # datetime -> ISO string
+    "path": Path("ckpt.pt"),          # Path -> str
+    "stage": Stage.TRAIN,             # Enum -> its value
+})
+```
+
+`NaN` and `Infinity` are preserved as-is, because a NaN loss is a signal and
+`isnan()` in an alert rule depends on it. Python and pandas read this back; strict
+JSON parsers such as `jq` do not.
+
+A value that cannot be encoded falls back to `repr()` with a one-time warning, per
+field — one bad value never costs you the rest of the row.
+JSON integers too large for floating-point alert buffers remain unchanged in
+history. They register their metric name but are skipped by numeric series with
+a warning; neighboring metrics and their alerts continue to work.
+Filesystem-surrogate characters in strings and paths are written as JSON escapes
+rather than invalid UTF-8, so they do not discard neighboring metrics. Ordinary
+Unicode remains readable UTF-8. Config replacement is atomic: a failed write
+preserves the previous config file.
+
+Metric names are free-form. `/` is conventional for grouping (`train/loss`), and the
+alert language accepts `train.loss` as a shorthand for it.
+
+## Reserved keys
+
+`_step` and `_time` are written by the tracker. If you log them they are ignored, with
+a warning.
+
+## The summary
+
+Every metric's last value is tracked automatically. Explicit assignments win and are
+never overwritten:
+
+```python
+et.log({"acc": 0.91})
+et.summary()["best_acc"] = 0.93   # survives later log() calls
+dict(et.summary())                # {"acc": 0.91, "best_acc": 0.93}
+```
+
+The summary is saved on `finish()` and by the exit hook if the process dies first.
+
+## Printing to screen
+
+```python
+et.init(..., print_to_screen=True)                 # print every committed row
+et.init(..., print_to_screen=True, print_handle=logger.info)
+```
+
+Only committed rows are printed, so you see one line per step. A failing handle is
+logged and never interrupts training.
+
+## Cost
+
+`log()` costs about 26 µs (p50) without rules and 74 µs with five rules,
+around 33,000 calls/s. The write buffer batches lines, and the p99 stays under
+5 ms so a flush cannot stall the loop.