expr-tracker 0.2.4__tar.gz → 0.2.5__tar.gz

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/.github/workflows/docs.yaml

@@ -5,6 +5,7 @@ on:
     branches: [main]
     paths:
       - "docs/**"
+      - "examples/**"   # docs/examples.md includes examples/README.md
       - "mkdocs.yml"
       - ".github/workflows/docs.yaml"
   workflow_dispatch:

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: expr_tracker
-Version: 0.2.4
+Version: 0.2.5
 Summary: Local-first experiment tracking with queryable history and expression-based alerts on your training metrics
 Project-URL: Homepage, https://hspk.github.io/expr_tracker/
 Project-URL: Documentation, https://hspk.github.io/expr_tracker/
@@ -112,7 +112,26 @@ install command; it never crashes a run.
 | [Spans](https://hspk.github.io/expr_tracker/guide/spans/) | Time the parts of a step, and their parts. Each duration becomes a metric, so alerts and queries work on it unchanged; `et trace` exports the timeline for Perfetto. `print_fn` prints the tree live, and plugins attach CPU and GPU cost to each region. |
 | [Streams](https://hspk.github.io/expr_tracker/guide/streams/) | Independent producers — a data worker and a training loop — each with their own step cursor and file inside one run. |
 | [Distributed](https://hspk.github.io/expr_tracker/guide/distributed/) | Per-rank shards so concurrent appends cannot corrupt step order; only rank 0 alerts by default. |
-| [CLI](https://hspk.github.io/expr_tracker/guide/cli/) | `et history`, `et rules explain`, `et rules test`, `et alert`. |
+| [CLI](https://hspk.github.io/expr_tracker/guide/cli/) | `et history`, `et trace`, `et rules explain`, `et rules test`, `et alert`. |
+
+## Examples
+
+Every one runs offline, with no account and no network.
+
+| | |
+| --- | --- |
+| [`quickstart.py`](examples/quickstart.py) | The sixty-second tour: log a run, merge eval into the training step, read the history back while it is open and again afterwards. |
+| [`alert_rules.py`](examples/alert_rules.py) | Four rules against four faults — a loss spike, a non-finite loss, a stalled curve and a regression that must persist. `--fault none` fires nothing, which is the point. |
+| [`profile_step.py`](examples/profile_step.py) | Where a step goes. Nested spans become metrics, a plugin adds CPU cost, and the tree exports to Perfetto. |
+| [`early_stopping.py`](examples/early_stopping.py) | Querying your own history mid-run to decay the learning rate on a plateau and stop when it stops paying. |
+| [`checkpoints.py`](examples/checkpoints.py) | Checkpoints as versioned artifacts, deduplicated by content and fetched later by alias. |
+| [`multiprocess_pipeline.py`](examples/multiprocess_pipeline.py) | Four data producers and four trainers as eight processes in one run, with bounded staleness. Each worker gets its own stream and its own lane in the trace, and the blocking spans show which side is the bottleneck. |
+
+```bash
+uv run python examples/quickstart.py
+uv run python examples/alert_rules.py --fault spike
+uv run python examples/multiprocess_pipeline.py --produce-ms 10 --train-ms 40
+```
 
 ## wandb compatibility
 
@@ -155,6 +174,7 @@ uv run ruff format src tests
 | `test_expr_properties.py` | DSL properties: render round-trip stability, precedence, the whole `M` builder |
 | `test_trace.py` | Chrome Trace export: lane layout, stream and step selection, the CLI |
 | `test_spans.py` | nesting, aggregation, decorator and async forms, errors, thread and task isolation |
+| `test_examples.py` | the shipped examples run, and their backpressure claims hold |
 | `test_span_plugins.py` | `print_fn` output and indentation, the plugin protocol, failure isolation, CPU/GPU built-ins |
 | `test_streams.py` | stream naming and validation, isolation, resolution order, backend grouping, two-process runs |
 | `test_distributed.py` | rank shards, `alert_on_rank`, real multi-process runs |

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/README.md

@@ -62,7 +62,26 @@ install command; it never crashes a run.
 | [Spans](https://hspk.github.io/expr_tracker/guide/spans/) | Time the parts of a step, and their parts. Each duration becomes a metric, so alerts and queries work on it unchanged; `et trace` exports the timeline for Perfetto. `print_fn` prints the tree live, and plugins attach CPU and GPU cost to each region. |
 | [Streams](https://hspk.github.io/expr_tracker/guide/streams/) | Independent producers — a data worker and a training loop — each with their own step cursor and file inside one run. |
 | [Distributed](https://hspk.github.io/expr_tracker/guide/distributed/) | Per-rank shards so concurrent appends cannot corrupt step order; only rank 0 alerts by default. |
-| [CLI](https://hspk.github.io/expr_tracker/guide/cli/) | `et history`, `et rules explain`, `et rules test`, `et alert`. |
+| [CLI](https://hspk.github.io/expr_tracker/guide/cli/) | `et history`, `et trace`, `et rules explain`, `et rules test`, `et alert`. |
+
+## Examples
+
+Every one runs offline, with no account and no network.
+
+| | |
+| --- | --- |
+| [`quickstart.py`](examples/quickstart.py) | The sixty-second tour: log a run, merge eval into the training step, read the history back while it is open and again afterwards. |
+| [`alert_rules.py`](examples/alert_rules.py) | Four rules against four faults — a loss spike, a non-finite loss, a stalled curve and a regression that must persist. `--fault none` fires nothing, which is the point. |
+| [`profile_step.py`](examples/profile_step.py) | Where a step goes. Nested spans become metrics, a plugin adds CPU cost, and the tree exports to Perfetto. |
+| [`early_stopping.py`](examples/early_stopping.py) | Querying your own history mid-run to decay the learning rate on a plateau and stop when it stops paying. |
+| [`checkpoints.py`](examples/checkpoints.py) | Checkpoints as versioned artifacts, deduplicated by content and fetched later by alias. |
+| [`multiprocess_pipeline.py`](examples/multiprocess_pipeline.py) | Four data producers and four trainers as eight processes in one run, with bounded staleness. Each worker gets its own stream and its own lane in the trace, and the blocking spans show which side is the bottleneck. |
+
+```bash
+uv run python examples/quickstart.py
+uv run python examples/alert_rules.py --fault spike
+uv run python examples/multiprocess_pipeline.py --produce-ms 10 --train-ms 40
+```
 
 ## wandb compatibility
 
@@ -105,6 +124,7 @@ uv run ruff format src tests
 | `test_expr_properties.py` | DSL properties: render round-trip stability, precedence, the whole `M` builder |
 | `test_trace.py` | Chrome Trace export: lane layout, stream and step selection, the CLI |
 | `test_spans.py` | nesting, aggregation, decorator and async forms, errors, thread and task isolation |
+| `test_examples.py` | the shipped examples run, and their backpressure claims hold |
 | `test_span_plugins.py` | `print_fn` output and indentation, the plugin protocol, failure isolation, CPU/GPU built-ins |
 | `test_streams.py` | stream naming and validation, isolation, resolution order, backend grouping, two-process runs |
 | `test_distributed.py` | rank shards, `alert_on_rank`, real multi-process runs |

expr_tracker-0.2.5/docs/examples.md

@@ -0,0 +1,10 @@
+---
+title: Examples
+---
+
+<!-- Single source: examples/README.md, so the two cannot drift apart. -->
+
+--8<-- "examples/README.md"
+
+The source lives in
+[`examples/`](https://github.com/HSPK/expr_tracker/tree/main/examples).

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/docs/getting-started.md

@@ -85,12 +85,16 @@ et.get_run().step        # 1000 - the cursor picked up where it left off
 len(et.history(-1))      # 1000 - the old rows are still there
 ```
 
+This is unconditional: `resume` is passed on to wandb and trackio, but the local
+file always continues. Use a different name to start clean.
+
 A process that dies without `finish()` still leaves a complete, valid file: the open
 row is committed and the summary saved by an exit hook, and a torn trailing line is
 repaired on the next run.
 
 ## Next
 
+- [Examples](examples.md) — six runnable programs, all offline.
 - [Logging metrics](guide/logging.md) for commit semantics and out-of-order steps.
 - [Alerts](guide/alerts.md) to get notified instead of watching curves.
 - [CLI](guide/cli.md) to inspect runs without writing code.

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/docs/guide/cli.md

@@ -11,6 +11,7 @@ et history tracker/jsonl/demo/run-1
 et history tracker/jsonl/demo/run-1 -n 50 --metrics loss,lr
 et history tracker/jsonl/demo/run-1 --step-range 100:200 --format json
 et history tracker/jsonl/demo/run-1 -n -1 --format csv > run.csv
+et history runs/llm/sft-1 --stream data
 ```
 
 | Option | Default | Meaning |
@@ -18,6 +19,7 @@ et history tracker/jsonl/demo/run-1 -n -1 --format csv > run.csv
 | `-n` | 20 | number of steps, `-1` for all |
 | `--metrics` | all | comma-separated names |
 | `--step-range` | — | `start:end`, end exclusive; either side may be empty |
+| `--stream` | the unnamed producer | which [stream](streams.md) to read |
 | `--format` | `table` | `table`, `json` or `csv` |
 
 The argument is a run directory or a `metrics.jsonl` file.

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/docs/guide/distributed.md

@@ -50,6 +50,29 @@ its rules, and reports them through `info()`:
 et.info()["rank"]     # which rank this process was detected as
 ```
 
+## Remote backends
+
+wandb and trackio identify a run by id, and neither can merge two step axes into
+one run. If every rank opened the same id they would interleave their steps into
+it — exactly what the local shards exist to prevent. So only rank 0 opens a
+remote run by default:
+
+```python
+et.init(..., backends=["wandb"])                        # rank 0 reports (default)
+et.init(..., backends=["wandb"], backend_on_rank=2)     # rank 2 reports instead
+et.init(..., backends=["wandb"], backend_on_rank=None)  # every rank reports
+```
+
+A silenced rank still writes its full local history; it simply never calls the
+backend. When every rank does report, each gets its own run id and they are tied
+together with `group`, which both backends understand:
+
+| | id | group |
+| --- | --- | --- |
+| rank 0 | `sft-1` | `sft-1` |
+| rank 2 | `sft-1-rank2` | `sft-1` |
+| rank 2 of stream `data` | `sft-1-data-rank2` | `sft-1` |
+
 ## Typical setup
 
 ```python
@@ -64,5 +87,5 @@ et.init(
 )
 ```
 
-Every rank writes to `/shared/runs/llm/sft-<id>/`, each into its own shard, and only
-rank 0 pages you.
+Every rank writes to `/shared/runs/llm/sft-<id>/`, each into its own shard. Only
+rank 0 pages you, and only rank 0 opens the wandb run.

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/docs/guide/history.md

@@ -30,6 +30,14 @@ et.history(-1, metrics=["eval/acc"], dropna=True)     # drop steps without it
 `dropna` drops a row when **all** selected metrics are missing, which is what you
 want for sparse eval metrics logged every N steps.
 
+`n` selects rows first and `dropna` filters them afterwards. Asking for the last
+4 rows of a metric logged every 10 steps therefore finds nothing — take every
+eval point and slice:
+
+```python
+et.history(-1, metrics=["eval/acc"], dropna=True)[-4:]   # the last 4 evals
+```
+
 ## Output types
 
 ```python

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/docs/guide/streams.md

@@ -57,6 +57,39 @@ from expr_tracker.history import list_streams
 list_streams("runs/llm/sft-1")       # [None, "data"]
 ```
 
+```bash
+et history runs/llm/sft-1 --stream data
+et trace runs/llm/sft-1                   # every stream, one lane each
+```
+
+## A worked example
+
+`examples/multiprocess_pipeline.py` runs four data producers and four trainers as
+eight processes sharing one run, with a queue that lets a producer run at most
+`--staleness` batches ahead of the trainers. Each worker writes its own stream,
+so the exported trace gives each one a lane and the blocking spans show which
+side is the bottleneck:
+
+```bash
+# producers faster than trainers: they stall on a full queue
+uv run python examples/multiprocess_pipeline.py --produce-ms 10 --train-ms 40
+
+# trainers faster than producers: they starve waiting for batches
+uv run python examples/multiprocess_pipeline.py --produce-ms 40 --train-ms 10
+```
+
+```
+  producers  0.38s in produce
+    read             0.15s  37.9%
+    enqueue          0.13s  35.0%  <- backpressure
+    decode           0.09s  24.5%
+
+  trainers  1.05s in step
+    backward         0.59s  55.9%
+    forward          0.40s  38.1%
+    wait_for_batch   0.05s   4.9%  <- starvation
+```
+
 ## Alerts
 
 Each process alerts on what it can see, which is its own stream. That is usually

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/docs/index.md

@@ -42,6 +42,7 @@ uv add "expr_tracker[all]"          # + wandb, trackio, lark, pandas, polars
 ## Next
 
 - [Getting started](getting-started.md) — a complete run, end to end.
+- [Examples](examples.md) — six runnable programs, all offline.
 - [Logging metrics](guide/logging.md) — commit semantics and the step model.
 - [Alerts](guide/alerts.md) — rules, channels and delivery policy.
 - [Design](design.md) — the data model and the invariants behind it.

{expr_tracker-0.2.4 → expr_tracker-0.2.5}/docs/reference/api.md

@@ -26,6 +26,8 @@ et.init(
     stream: str | None = None,
     alert=None,
     alert_rules: Sequence = (),
+    alert_on_rank: int | None = 0,
+    backend_on_rank: int | None = 0,
     **history_options,
 ) -> Run
 ```
@@ -34,6 +36,14 @@ Starts a run and publishes it as the process-wide current run. `name` defaults t
 timestamp. `dir` defaults to `./tracker/jsonl`. Extra keyword arguments are
 [history options](configuration.md#history-options).
 
+`resume` is forwarded to wandb and trackio. The local history always continues an
+existing run directory, so re-running with the same project and name picks up the
+step cursor whatever `resume` says; use a new name to start clean.
+
+`alert_on_rank` and `backend_on_rank` pick which rank alerts and which opens the
+remote backend run; `None` means every rank. See
+[Distributed runs](../guide/distributed.md).
+
 Calling `init()` twice without `finish()` raises.
 
 ### `finish`

expr_tracker-0.2.5/examples/README.md

@@ -0,0 +1,65 @@
+# Examples
+
+Every example runs offline: no account, no network, no GPU required. They write
+to `runs/` by default, which is gitignored — pass `--dir` to put them elsewhere.
+
+```bash
+uv run python examples/quickstart.py
+uv run python examples/<name>.py --help      # every example takes arguments
+```
+
+## [`quickstart.py`](https://github.com/HSPK/expr_tracker/blob/main/examples/quickstart.py)
+
+The sixty-second tour. Logs a training run, merges a sparse eval metric into the
+training step with `commit=False`, queries the history while the run is still
+open, and reads it back afterwards from the directory.
+
+## [`alert_rules.py`](https://github.com/HSPK/expr_tracker/blob/main/examples/alert_rules.py)
+
+Four rules against four faults: a loss spike caught by z-score, a non-finite
+loss, a curve that goes flat, and an accuracy regression that has to persist for
+three steps before it counts.
+
+```bash
+uv run python examples/alert_rules.py --fault spike   # or nan, stall, none
+```
+
+`--fault none` sends nothing. That is the interesting case: warm-up, missing
+data and NaN all evaluate to UNKNOWN rather than False, so rules cannot cry wolf
+before they have the evidence.
+
+## [`profile_step.py`](https://github.com/HSPK/expr_tracker/blob/main/examples/profile_step.py)
+
+Where a training step actually goes. Nested spans become metrics on the step's
+row, so timings query and alert like any other metric; a plugin attaches CPU
+cost; the tree exports to a Chrome Trace for Perfetto.
+
+```bash
+uv run python examples/profile_step.py --print-spans
+```
+
+## [`early_stopping.py`](https://github.com/HSPK/expr_tracker/blob/main/examples/early_stopping.py)
+
+The loop reading its own history to decide what to do next: decay the learning
+rate when the eval metric plateaus, and stop once decaying no longer pays. This
+is what a local, queryable history buys you over shipping metrics out.
+
+## [`checkpoints.py`](https://github.com/HSPK/expr_tracker/blob/main/examples/checkpoints.py)
+
+Checkpoints as artifacts — versioned, deduplicated by content, aliased `best`,
+and fetched by a later run that knows nothing about which run wrote them.
+
+## [`multiprocess_pipeline.py`](https://github.com/HSPK/expr_tracker/blob/main/examples/multiprocess_pipeline.py)
+
+Four data producers and four trainers as eight processes sharing one run, with a
+queue that lets a producer run at most `--staleness` batches ahead. Each worker
+writes its own stream and gets its own lane in the trace, and the blocking spans
+show which side is the bottleneck.
+
+```bash
+# producers faster than trainers: they stall on a full queue
+uv run python examples/multiprocess_pipeline.py --produce-ms 10 --train-ms 40
+
+# trainers faster than producers: they starve waiting for batches
+uv run python examples/multiprocess_pipeline.py --produce-ms 40 --train-ms 10
+```

expr_tracker-0.2.5/examples/alert_rules.py

@@ -0,0 +1,115 @@
+"""Alert rules that catch the four ways a run goes wrong.
+
+Rules are expressions over a rolling window of your metrics, evaluated once per
+committed step. This sends to a local handler so it runs offline; swap the
+channel for lark/slack/email and nothing else changes.
+
+    uv run python examples/alert_rules.py
+    uv run python examples/alert_rules.py --fault stall
+
+Faults: spike, nan, stall, none.
+"""
+
+from __future__ import annotations
+
+import argparse
+import math
+import random
+import time
+
+import expr_tracker as et
+
+RULES = [
+    # Never fires during warm-up: too few points evaluates to UNKNOWN, not False
+    "zscore(train/loss[30]) > 4    => error: loss spike {train/loss:.4f} @ step {step}",
+    "isnan(train/loss)             => critical: loss went non-finite",
+    "stalled(train/loss[20])       => warning: loss flat for 20 steps",
+    # Must hold for 3 consecutive steps, so one noisy eval does not page anyone
+    {
+        "name": "accuracy_regression",
+        "condition": "eval/accuracy < 0.5",
+        "level": "warning",
+        "for_steps": 3,
+        "notify_recovery": True,
+    },
+]
+
+
+def channel(sink):
+    """A channel that appends to a list. Synchronous, so output stays ordered."""
+    return {
+        "channels": [
+            {
+                "type": "callable",
+                "name": "local",
+                "options": {"handler": sink.append},
+                "policy": {"async_send": False, "dedup_window": 0},
+            }
+        ]
+    }
+
+
+def losses(args):
+    """A loss curve with the requested fault injected into it."""
+    rng = random.Random(args.seed)
+    for step in range(args.steps):
+        loss = math.exp(-step / 60) + rng.uniform(0, 0.02)
+        if args.fault == "spike" and step == args.at:
+            loss *= 12  # a sudden jump the z-score will notice
+        elif args.fault == "nan" and step >= args.at:
+            loss = float("nan")
+        elif args.fault == "stall" and step >= args.at:
+            loss = 0.5  # exactly flat, which stalled() is looking for
+        yield step, loss
+
+
+def main(argv=None) -> list:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument(
+        "--fault", default="spike", choices=["spike", "nan", "stall", "none"]
+    )
+    parser.add_argument("--steps", type=int, default=80)
+    parser.add_argument("--at", type=int, default=50, help="step to inject the fault")
+    parser.add_argument("--seed", type=int, default=0)
+    parser.add_argument("--dir", default="runs")
+    parser.add_argument("--name", default=None)
+    args = parser.parse_args(argv)
+    # A fresh run each time: resuming would leave the previous run's losses in
+    # the rolling window, and the jump back to a high loss reads as a spike
+    args.name = args.name or f"{args.fault}-{time.strftime('%H%M%S')}"
+
+    fired: list = []
+    et.init(
+        project="alerts",
+        name=args.name,
+        dir=args.dir,
+        backends=[],
+        alert=channel(fired),
+        alert_rules=RULES,
+    )
+
+    for step, loss in losses(args):
+        accuracy = 0.4 if args.fault == "stall" and step > args.at else 0.9
+        et.log({"eval/accuracy": accuracy}, commit=False)
+        et.log({"train/loss": loss})
+
+    print(f"fault: {args.fault}, {args.steps} steps")
+    print(f"\n{len(fired)} message(s):")
+    for message in fired:
+        print(f"  [{message.level.value}] {message.title}")
+        print(f"      {message.text.splitlines()[0]}")
+
+    state = et.info()["alerts"]["rules"]
+    print("\nrule state:")
+    for rule in et.list_alert_rules():
+        # An unnamed rule is known by its condition, which is what you read
+        label = rule.condition if rule.auto_named else rule.name
+        counts = state[rule.name]
+        print(f"  {label:<34} fires={counts['fires']} firing={counts['firing']}")
+
+    et.finish()
+    return fired
+
+
+if __name__ == "__main__":
+    main()

expr_tracker-0.2.5/examples/checkpoints.py

@@ -0,0 +1,83 @@
+"""Checkpoints as artifacts: versioned, deduplicated, and found again later.
+
+Artifacts are stored per project and shared by its runs, so a later run can ask
+for ``model:best`` without knowing which run produced it.
+
+    uv run python examples/checkpoints.py
+"""
+
+from __future__ import annotations
+
+import argparse
+import time
+from pathlib import Path
+
+import expr_tracker as et
+
+
+def write_checkpoint(directory: Path, step: int, accuracy: float) -> Path:
+    """Stand-in for torch.save."""
+    directory.mkdir(parents=True, exist_ok=True)
+    path = directory / "model.pt"
+    path.write_text(f"weights@{step}:{accuracy:.3f}", encoding="utf-8")
+    return path
+
+
+def train(args, workdir: Path) -> str:
+    run = et.init(
+        project="checkpoints",
+        name=args.name,
+        dir=args.dir,
+        backends=[],
+        config={"steps": args.steps},
+    )
+    best = -1.0
+
+    for step in range(0, args.steps, args.every):
+        accuracy = 0.7 + 0.1 * (step / args.steps)
+        et.log({"eval/accuracy": accuracy}, commit=False)
+        et.log({"train/loss": 1.0 / (step + 1)})
+
+        path = write_checkpoint(workdir, step, accuracy)
+        aliases = ["best"] if accuracy > best else []
+        artifact = et.log_artifact(path, name="model", type="model", aliases=aliases)
+        if accuracy > best:
+            best = accuracy
+            et.summary()["best_accuracy"] = accuracy
+        print(f"step {step:>3}: acc {accuracy:.3f} -> model:v{artifact.version}")
+
+    # The same bytes again: deduplicated, so no new version is created
+    repeat = et.log_artifact(workdir / "model.pt", name="model", type="model")
+    print(f"logging the same file again -> v{repeat.version} (deduplicated)")
+
+    et.finish()
+    return run.name
+
+
+def restore(args, into: Path) -> None:
+    """A different run, later, asking for the best checkpoint."""
+    et.init(project="checkpoints", name=f"{args.name}-eval", dir=args.dir, backends=[])
+    artifact = et.use_artifact("model:best")
+    path = Path(artifact.download(str(into)))
+    payload = (path / "model.pt").read_text(encoding="utf-8")
+    print(f"\nrestored model:best (v{artifact.version}) -> {payload}")
+    print(f"downloaded to {path}")
+    et.finish()
+
+
+def main(argv=None) -> None:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--steps", type=int, default=40)
+    parser.add_argument("--every", type=int, default=10, help="checkpoint interval")
+    parser.add_argument("--dir", default="runs")
+    parser.add_argument("--name", default=None)
+    args = parser.parse_args(argv)
+    args.name = args.name or time.strftime("ckpt-%H%M%S")
+
+    workdir = Path(args.dir) / "scratch" / args.name
+    train(args, workdir)
+    restore(args, workdir / "restored")
+
+
+if __name__ == "__main__":
+    main()

expr_tracker-0.2.5/examples/early_stopping.py

@@ -0,0 +1,99 @@
+"""Reading your own history mid-run to decide what to do next.
+
+Most trackers only ship metrics out. Because history is local and queryable
+while the run is open, the loop can ask what has been happening and act on it:
+decay the learning rate on a plateau, stop early, skip a bad batch.
+
+    uv run python examples/early_stopping.py
+    uv run python examples/early_stopping.py --patience 3 --plateau 0.002
+"""
+
+from __future__ import annotations
+
+import argparse
+import math
+import random
+import time
+
+import expr_tracker as et
+
+
+def plateaued(window: int, threshold: float) -> bool:
+    """Has the eval metric stopped improving over the last `window` evals?
+
+    `n` counts rows and `dropna` filters them afterwards, so asking for the last
+    4 rows of a metric logged every 10 steps finds nothing. Take the eval points
+    first, then the last few.
+    """
+    points = et.history(-1, metrics=["eval/loss"], dropna=True)[-window:]
+    if len(points) < window:
+        return False  # not enough evidence yet, so do nothing
+    values = [row["eval/loss"] for row in points]
+    return max(values) - min(values) < threshold
+
+
+def main(argv=None) -> dict:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--steps", type=int, default=600)
+    parser.add_argument("--eval-every", type=int, default=10)
+    parser.add_argument("--patience", type=int, default=4, help="evals before acting")
+    parser.add_argument("--plateau", type=float, default=0.004, help="no-progress band")
+    parser.add_argument("--lr", type=float, default=1e-3)
+    parser.add_argument("--min-lr", type=float, default=1e-5)
+    parser.add_argument("--seed", type=int, default=0)
+    parser.add_argument("--dir", default="runs")
+    parser.add_argument("--name", default=None)
+    args = parser.parse_args(argv)
+    args.name = args.name or time.strftime("earlystop-%H%M%S")
+
+    et.init(
+        project="control",
+        name=args.name,
+        dir=args.dir,
+        backends=[],
+        config={"lr": args.lr, "patience": args.patience},
+    )
+    rng = random.Random(args.seed)
+    lr, decays, stopped_at, since = args.lr, 0, None, 0
+
+    for step in range(args.steps):
+        # Improvement restarts after each decay and then flattens again, so
+        # there is a real plateau to find. Noise stays under --plateau.
+        floor = 0.25 * (0.6**decays)
+        loss = floor + 0.8 * math.exp(-(step - since) / 30) + rng.uniform(0, 0.001)
+
+        if step % args.eval_every == 0 and step:
+            et.log({"eval/loss": loss * 0.98}, commit=False)
+            if plateaued(args.patience, args.plateau):
+                if lr > args.min_lr:
+                    lr, decays, since = lr / 10, decays + 1, step
+                    print(f"step {step:>3}: plateau -> lr {lr:.1e}")
+                    et.log({"event/lr_decay": decays}, commit=False)
+                else:
+                    stopped_at = step
+                    print(f"step {step:>3}: plateau at the minimum lr, stopping")
+                    et.summary()["stopped_early"] = True
+
+        et.log({"train/loss": loss, "train/lr": lr})
+        if stopped_at is not None:
+            break
+
+    et.summary()["decays"] = decays
+    et.summary()["steps_run"] = et.get_run().step
+
+    rows = et.history(-1, metrics=["eval/loss"], dropna=True)
+    print(f"\n{len(rows)} evals, {decays} decay(s)")
+    print(f"first eval {rows[0]['eval/loss']:.4f} -> last {rows[-1]['eval/loss']:.4f}")
+    if stopped_at is None:
+        print(f"ran all {args.steps} steps without stopping")
+    else:
+        saved = args.steps - stopped_at
+        print(f"stopped at step {stopped_at}, saving {saved} steps of compute")
+
+    info = dict(et.summary())
+    et.finish()
+    return info
+
+
+if __name__ == "__main__":
+    main()