trainmeter 0.0.2__tar.gz → 0.0.4__tar.gz

{trainmeter-0.0.2 → trainmeter-0.0.4}/CLAUDE.md

@@ -68,6 +68,7 @@ These are the kind that locally reasonable code breaks quietly.
    New entry points inside the training process get the same wrapper and a test that forces an internal error.
 2. **Peaks are dense, and every peak cites its datasheet.**
    NVIDIA headlines the 2:4 sparse figure, which is exactly twice the dense one; using it halves MFU.
+   A peak lives in a spec file in `hardware/specs/` next to the datasheet lines it comes from, and a figure no datasheet gives is `null` with a note, never an estimate.
    An unknown device returns `None`, and MFU is then not reported.
    Never fall back to another card's peak.
 3. **A FLOPs number always carries its convention.**
@@ -99,13 +100,13 @@ These are the kind that locally reasonable code breaks quietly.
 
 What exists today:
 
-- `peaks.py`, `flops.py`, `metrics.py` - peaks (dense BF16, HBM), FLOPs per token under named conventions, pure formulas (MFU, OFU, arithmetic intensity as `step_intensity`, ETA, EMA, and dollar helpers that only overlays may use).
+- `hardware/` (device specs, one JSON file per GPU in `hardware/specs/` with the datasheet lines they come from), `flops.py`, `metrics.py` - dense BF16 peaks and memory bandwidth, FLOPs per token under named conventions, pure formulas (MFU, OFU, arithmetic intensity as `step_intensity`, ETA, EMA, and dollar helpers that only overlays may use).
 - `records.py` (wire schema and redaction), `facts.py` (provenance, precedence), `catalog.py` (series aliases and the metric catalog), `engine.py` (records in, snapshots out), `timeline.py` (chart history), `passport.py` (the model-card passport), `replay.py` (rebuild a run from its log), `config.py` (declared flags), `report.py` (the summary line) - the pure core.
-- `cli.py`, `doctor.py` (`tm doctor`), `supervisor/` (`launcher.py`, `store.py`, `ingest.py`), `sources/host.py`, `sources/gpu.py` (NVML and GPM behind a `Backend` seam, with a fake), `supervisor/live.py` (engine and timeline under one lock), `web/` (`server.py` and the static dashboard), `viewer.py` (`tm view`, `tm attach`), `commands.py` (`tm passport`, `tm export`) and `export/` (passport files, W&B) - the `tm` process.
+- `cli.py`, `doctor.py` (`tm doctor`), `supervisor/` (`launcher.py`, `store.py`, `ingest.py`), `sources/host.py`, `sources/gpu/` (`backend.py`: NVML and GPM behind a `Backend` seam, with a fake; `platform.py`: the adapter interface and the row schema; `server.py` and `spark.py`: one adapter per kind of machine, named by the device's spec; `sampler.py`), `supervisor/live.py` (engine and timeline under one lock), `web/` (`server.py` and the static dashboard), `viewer.py` (`tm view`, `tm attach`), `commands.py` (`tm passport`, `tm export`) and `export/` (passport files, W&B) - the `tm` process.
 - `agent/` (`core.py`, `taps.py`, `wrap.py`, `sender.py`, `bootstrap/sitecustomize.py`) and `emit.py` - what runs inside the training process.
 - `tests/` - one test file per module; numbers for the reference 1.04B model are pinned in `test_flops.py`. Tests that need torch skip without it, and CI runs them in a second job; those that use wandb, TensorBoard or MLflow also skip without the tracker (`uv pip install torch tensorboard wandb mlflow-skinny`).
 
-What is planned, from `docs/architecture.md`: the remaining probes and views.
+What is planned: `docs/roadmap.md`, which follows from the user analysis in `docs/users.md`.
 `docs/hardware-checklist.md` lists what only a real GPU node can verify.
 
 The dependency rule: edges import the core, and the core imports nothing but the standard library.

{trainmeter-0.0.2 → trainmeter-0.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: trainmeter
-Version: 0.0.2
+Version: 0.0.4
 Summary: Where your training FLOPs go and how well the GPUs are used: loss, FLOPs, MFU and hardware counters for a wrapped training run.
 Project-URL: Homepage, https://github.com/Almaz-KG/trainmeter
 Project-URL: Repository, https://github.com/Almaz-KG/trainmeter
@@ -31,13 +31,20 @@ trainmeter starts your command, watches the node from outside, and serves a live
 The dashboard shows what a training run needs to be judged: training and validation loss, FLOPs spent and required, MFU with its FLOPs convention spelled out, and how busy the GPUs really are, from `GPU-Util` down to MFU, next to arithmetic intensity.
 Your training code stays as it is.
 
-Status: early scaffold, version 0.0.1.
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="docs/images/dashboard-dark.png">
+  <img alt="The trainmeter dashboard: progress, throughput, MFU and OFU, goodput and the loss curve of a run on four H100s" src="docs/images/dashboard-light.png">
+</picture>
+
+<sub>The dashboard on a synthetic run.</sub>
+
+Status: early scaffold.
 `tm` runs any command untouched, records the run, and turns what a PyTorch loop does into tokens, FLOPs spent and required, ETA, MFU (given a device) and the losses you `emit()`.
 GPU counters (NVML and GPM: the utilization ladder, OFU, observed FLOPs, arithmetic intensity) and `tm doctor` are built and tested against fakes, not yet on a real GPU.
 Series your loop already logs to wandb, TensorBoard or MLflow are picked up without a code change (`--no-tap` turns it off), and DataLoader waits, checkpoint saves and eval spans feed a "where the time goes" view.
 One real step is also counted with `FlopCounterMode` (shown beside MFU as the `counted` convention), and layers and width are read from a model config when there is one.
 Each run ends with a run passport (`passport.md` and `passport.json`) for a model card.
-The dashboard (Cockpit, Efficiency, Hardware, Where the time goes, Plan vs actual and Run passport) is served on localhost while the run goes; it has been checked in a browser on synthetic data, not on a real run.
+The dashboard, one page of progress, tiles and charts with the run details and model card folded away, is served on localhost while the run goes; it has been checked in a browser on synthetic data, not on a real run.
 Not yet used on a real run.
 To try it, `examples/` has a tiny GPT to run under `tm` on a laptop and a run planner that needs no GPU.
 

{trainmeter-0.0.2 → trainmeter-0.0.4}/README.md

@@ -7,13 +7,20 @@ trainmeter starts your command, watches the node from outside, and serves a live
 The dashboard shows what a training run needs to be judged: training and validation loss, FLOPs spent and required, MFU with its FLOPs convention spelled out, and how busy the GPUs really are, from `GPU-Util` down to MFU, next to arithmetic intensity.
 Your training code stays as it is.
 
-Status: early scaffold, version 0.0.1.
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="docs/images/dashboard-dark.png">
+  <img alt="The trainmeter dashboard: progress, throughput, MFU and OFU, goodput and the loss curve of a run on four H100s" src="docs/images/dashboard-light.png">
+</picture>
+
+<sub>The dashboard on a synthetic run.</sub>
+
+Status: early scaffold.
 `tm` runs any command untouched, records the run, and turns what a PyTorch loop does into tokens, FLOPs spent and required, ETA, MFU (given a device) and the losses you `emit()`.
 GPU counters (NVML and GPM: the utilization ladder, OFU, observed FLOPs, arithmetic intensity) and `tm doctor` are built and tested against fakes, not yet on a real GPU.
 Series your loop already logs to wandb, TensorBoard or MLflow are picked up without a code change (`--no-tap` turns it off), and DataLoader waits, checkpoint saves and eval spans feed a "where the time goes" view.
 One real step is also counted with `FlopCounterMode` (shown beside MFU as the `counted` convention), and layers and width are read from a model config when there is one.
 Each run ends with a run passport (`passport.md` and `passport.json`) for a model card.
-The dashboard (Cockpit, Efficiency, Hardware, Where the time goes, Plan vs actual and Run passport) is served on localhost while the run goes; it has been checked in a browser on synthetic data, not on a real run.
+The dashboard, one page of progress, tiles and charts with the run details and model card folded away, is served on localhost while the run goes; it has been checked in a browser on synthetic data, not on a real run.
 Not yet used on a real run.
 To try it, `examples/` has a tiny GPT to run under `tm` on a laptop and a run planner that needs no GPU.
 

{trainmeter-0.0.2 → trainmeter-0.0.4}/docs/architecture.md

@@ -65,7 +65,7 @@ How the invariants in `CLAUDE.md` map onto this:
 | Invariant | In this architecture |
 | --- | --- |
 | 1. Never raises into the training loop | Rule 6. Agent probes, taps and `emit()` are wrapped and self-disabling, and everything else runs in another process. Each of them gets a test that forces an internal error. |
-| 2. Peaks are dense and cite a datasheet | Unchanged. `peaks.py` stays in the pure core, the GPU sampler reports the device name, and an unknown device means no MFU. |
+| 2. Peaks are dense and cite a datasheet | Each device has a spec file in `hardware/specs/` with its figures and the datasheet lines they come from. The GPU sampler reports the device name, and an unknown device, or one whose datasheet gives no dense BF16 figure, means no MFU. |
 | 3. A FLOPs number carries its convention | Rule 4. Value and convention are one fact and are never separated in the log, the API or the dashboard. |
 | 4. Counter metrics are not model metrics | The engine keeps modeled keys (MFU, FLOPs spent), sampled keys (OFU, observed FLOPs, DRAM active) and arithmetic intensity, which uses both, apart. The dashboard shows them side by side and labels each by origin. |
 | 5. Only rank 0 writes | Generalized by rule 5: only the supervisor writes. Rank 0 is the step clock and the other ranks feed straggler detection. |
@@ -341,42 +341,37 @@ Overlays are computed from the primitives at view time and never stored.
 
 The dashboard is one static page fed by the read-only API of section 10.
 It renders from the catalog, so a new series appears without a page change.
-The views, the rendering rules and the x-axis switch are specified in `docs/spec.md` (Screens).
+The page, the rendering rules and the x-axis selector are specified in `docs/spec.md` (Screens).
 Plotting loss against FLOPs makes runs comparable across batch sizes and sequence lengths.
-Cockpit, Efficiency and Hardware arrive in slice 4, Where the time goes in slice 5, and the run passport and Plan vs actual in slice 7.
+The views of slices 4, 5 and 7 were later folded into this one page without tabs.
 
-Layout sketches, with placeholders instead of numbers:
+Layout sketch, with placeholders instead of numbers:
 
 ```text
 +----------------------------------------------------------------------+
-| <command> | <n> x <GPU> | FLOPs per token <value> (<convention>)     |
-+----------------------------------+-----------------------------------+
-| PROGRESS                         | THROUGHPUT                        |
-| FLOPs spent of FLOPs required    | tokens/s and step time            |
-| tokens seen, ETA                 | data wait share of the step       |
-+----------------------------------+-----------------------------------+
-| TRAIN LOSS and VAL LOSS          | MFU and OFU                       |
-| x axis: step, tokens, FLOPs      | the gap between them              |
-| or wall time                     | latest utilization ladder         |
-+----------------------------------+-----------------------------------+
-| LR, GRAD NORM and every other series the loop logs                   |
+| trainmeter  <state>  <n> x <GPU>                              (theme)|
++----------------------------------------------------------------------+
+| PROGRESS <percent>     tokens, FLOPs of required, ETA, elapsed       |
++-----------+-----------+-----------+-----------+-----------+----------+
+| tokens/s  | step time | MFU       | OFU       | goodput   | FLOPs/tok|
+| sparkline | sparkline | sparkline | sparkline | sparkline |          |
++-----------+-----------+-----------+-----------+-----------+----------+
+| TRAINING                                   [x axis: step|tokens|...] |
+| loss (full width), tokens/s, MFU and OFU, lr, grad norm,             |
+| every other logged series, plan against actual                       |
++----------------------------------------------------------------------+
+| GPU EFFICIENCY: ladder now and over time, roofline, arithmetic       |
+| intensity, FLOPs spent (modeled next to observed)                    |
++----------------------------------------------------------------------+
+| NODE: time split, host CPU and memory                                |
+| GPUS: one small chart per counter, one line per GPU; ranks           |
++----------------------------------------------------------------------+
+| > Run details (collapsed)    > Model card (collapsed)                |
 +----------------------------------------------------------------------+
 ```
 
-```text
-+----------------------------------+-----------------------------------+
-| UTILIZATION LADDER over time     | ROOFLINE                          |
-| GPU-Util, SM active,             | arithmetic intensity against      |
-| tensor active, OFU, MFU          | achieved FLOP/s, with the ridge   |
-|                                  | and the peak of the device        |
-+----------------------------------+-----------------------------------+
-| FLOPs SPENT                      | ARITHMETIC INTENSITY over time    |
-| modeled next to observed         | aggregate over the window         |
-+----------------------------------+-----------------------------------+
-```
-
-The optional price field is the one place where money exists.
-It computes dollars in the browser from `elapsed`, `gpu_hours` and `tokens_seen`, keeps the price in that browser, and never sends it to the supervisor or writes it to the run log.
+The dashboard shows no price and no dollars.
+Money stays an overlay over `elapsed`, `gpu_hours` and `tokens_seen`, and a price never reaches the supervisor or the run log.
 
 ## 8. Run lifecycle
 
@@ -536,7 +531,7 @@ flowchart TB
     subgraph sup["Inside the tm process"]
         cli["cli.py, config.py"]
         supervisor["supervisor/<br/>launcher, ingest, store"]
-        sources["sources/<br/>gpu.py, host.py"]
+        sources["sources/<br/>gpu/, host.py"]
         web["web/<br/>server, dashboard"]
     end
 
@@ -545,7 +540,7 @@ flowchart TB
         catalog["catalog.py<br/>metric names,<br/>units, aliases"]
         facts["facts.py<br/>provenance,<br/>precedence"]
         records["records.py<br/>wire schema,<br/>codec"]
-        formulas["peaks.py, flops.py,<br/>metrics.py<br/>exist today"]
+        formulas["hardware/, flops.py,<br/>metrics.py<br/>exist today"]
     end
 
     inproc -->|"import records.py only"| core
@@ -562,13 +557,13 @@ The dependency rule is one sentence: edges import the core, and the core imports
 
 | Module | Layer | Role | State |
 | --- | --- | --- | --- |
-| `peaks.py`, `flops.py`, `metrics.py` | pure core | peaks, FLOPs conventions, formulas | exists |
+| `hardware/`, `flops.py`, `metrics.py` | pure core | device specs and peaks, FLOPs conventions, formulas | exists |
 | `records.py` | pure core | wire schema, codec, schema version | new |
 | `facts.py` | pure core | provenance, precedence, conflict warnings | new |
 | `catalog.py` | pure core | canonical metric names, units, labels, aliases, served to the dashboard | new |
 | `engine.py` | pure core | records in, snapshots out; takes over the arithmetic inside `Meter._step` | new |
 | `supervisor/` | tm process | launcher, ingest, store | new |
-| `sources/` | tm process | `gpu.py` (NVML behind a seam), `host.py` (`/proc`) | new |
+| `sources/` | tm process | `gpu/` (NVML behind a seam, one adapter per kind of machine), `host.py` (`/proc`) | new |
 | `web/` | tm process | standard-library server, static dashboard | new |
 | `agent/` | training process | bootstrap, probes and taps | new |
 | `emit.py` | training process | the public `emit()` | new |
@@ -579,6 +574,22 @@ The engine is a pure function of records and an injected clock.
 Live it is fed by the ingest queue, and in `tm view` it is fed by `events.jsonl`, so both paths produce the same numbers.
 A recorded run is also the test fixture: golden logs go in, snapshots come out.
 
+### GPU platforms
+
+Machines differ in what NVML can tell about them, so the GPU sampler has three layers, and only the middle one knows which machine it is on.
+
+| Layer | Module | Job |
+| --- | --- | --- |
+| Device access | `sources/gpu/backend.py` | NVML and GPM calls; a field the device does not offer is left out |
+| Platform adapter | `sources/gpu/platform.py`, `server.py`, `spark.py` | Turns what the device offers into one row schema, and names each field the machine never gives with the reason |
+| Sampler | `sources/gpu/sampler.py` | Picks the GPUs, asks each one's adapter, emits `sample.gpu` |
+
+The row schema in `platform.py` is the contract: the engine, the dashboard and the passport read only those fields, never the machine.
+The device's spec in `hardware/specs/` names its adapter, and a device without a spec is read as a server GPU.
+`server` passes NVML through, which is right for discrete cards with their own memory.
+`spark` covers the DGX Spark, whose GB10 shares the node's memory with the CPU: it reads memory from `/proc/meminfo` and leaves out the memory clock, PCIe and NVLink, which do not describe that chip.
+A new kind of machine is a new adapter and a spec file; nothing downstream changes.
+
 ## 12. Decisions and alternatives
 
 | # | Decision | Why | Rejected |

{trainmeter-0.0.2 → trainmeter-0.0.4}/docs/hardware-checklist.md

@@ -10,16 +10,23 @@ Every item names the code it verifies and what a wrong answer looks like.
 - [ ] No root and no DCGM are needed.
   If something asks for them, that is a bug.
 
+## Device specs (`hardware/specs/`)
+
+The spec files were transcribed without access to the linked pages from the build environment.
+
+- [ ] For each device you can reach, open every `sources` link in its spec file, compare each figure, and set `checked` to the date.
+  A figure that disagrees with the page is a bug, and a sparse figure taken as dense halves MFU.
+
 ## `tm doctor`
 
 - [ ] `tm doctor` prints `ok` for NVML with the driver version.
-- [ ] Each GPU line shows the right model and its dense BF16 peak from the table.
+- [ ] Each GPU line shows the right model and its dense BF16 peak from its spec, and the `platform` line names the right adapter.
 - [ ] Each `GPU n GPM` line is `ok` and prints a live read.
 - [ ] On a provider that blocks GPM, the line is `warn` with the reason, and `tm` still runs.
 - [ ] `CUDA_VISIBLE_DEVICES=1 tm doctor` selects GPU 1, and so does `tm doctor --gpus 1`.
   With `CUDA_DEVICE_ORDER` unset on a mixed-GPU node, the ordering warning appears.
 
-## Scale of the counters (`sources/gpu.py`, `NvmlBackend.gpm`)
+## Scale of the counters (`sources/gpu/backend.py`, `NvmlBackend.gpm`)
 
 The code divides SM, tensor and DRAM activity by 100 because the NVML headers document percent.
 Nothing has confirmed it on a device.
@@ -54,12 +61,23 @@ Nothing has confirmed it on a device.
 - [ ] The counted step shows no extra device synchronization (compare step time of step 6 with steps 5 and 7) and no extra GPU memory.
 - [ ] With `torch.compile` the probe reports that nothing was counted and the counted card is absent, instead of showing a small number.
 
-## `tm attach` (`sources/gpu.py`, `processes`)
+## `tm attach` (`sources/gpu/backend.py`, `processes`)
 
 - [ ] `tm attach <pid>` on a program running on GPU 1 samples GPU 1 only and `n_gpus` says `detected`.
 - [ ] Inside a container the pids do not match, and the fallback to every GPU shows its warning; `--gpus` fixes it.
 - [ ] Attaching does not change the program's step time, and stopping `tm` with Ctrl-C leaves it running.
 
+## DGX Spark (`sources/gpu/spark.py`)
+
+The GB10 adapter is built from public reports of what NVML returns on that chip, not from a run on one.
+
+- [ ] `tm doctor` names the `spark` adapter, says the memory is unified, and lists the memory clock, PCIe and NVLink as not on this machine.
+- [ ] The GPU line says GB10 has no dense BF16 figure, and MFU is absent with the same reason on the dashboard.
+- [ ] The `GPU 0 GPM` line says whether GPM works on GB10; record the answer here, since NVIDIA does not publish it.
+  If it works, run the dense bf16 matmul loop from the counters section and check the scale the same way.
+- [ ] During training, the "Memory used, shared with the CPU" chart follows `free -b` (total minus available) within a few percent.
+- [ ] Utilization, power, temperature and SM clock move with the load, and nothing in the GPU section is an empty chart.
+
 ## Agent device probe (`agent/core.py`)
 
 - [ ] `n_gpus` in `summary.json` says `detected` under `torchrun --nproc_per_node=N`, and `assumed` with a warning under a non-Python trainer.

trainmeter-0.0.4/docs/roadmap.md

@@ -0,0 +1,136 @@
+# Roadmap
+
+This plan follows from the user analysis in [`users.md`](users.md).
+Every item is tied to a pain or a point of friction from there.
+Sizes are an estimate, not a commitment: S is up to a couple of days, M up to a week, L more than a week.
+There are no dates, because they depend on access to a real GPU node.
+
+The slide version is the deck [trainmeter: development plan](https://claude.ai/artifact/QDxC41C2o4AxuVX9onhdN4) (Russian, private until shared).
+
+## Three themes, ordered by the cost of a mistake
+
+1. **Trust in the numbers.** A wrong MFU costs more than a missing one, because budgets are planned on it.
+   If a number is wrong, everything else only delivers the mistake faster.
+2. **A short path to the first MFU.** Until the user reaches a number, the value of the product is invisible.
+3. **A quiet night.** A hang or a throughput drop costs hours of rental, and the run should not vanish with the node.
+   It is the most expensive pain, but it matters to those who already use the product.
+
+## Theme 1: trust in the numbers
+
+| What to do | What the user gets | Size |
+| --- | --- | --- |
+| Go through `hardware-checklist.md` on an H100 | MFU, OFU and GPM checked against a real node | M |
+| Peaks for popular rented GPUs | MFU on L40S, RTX 4090 and others, each with its datasheet | S |
+| "How it is computed" on every tile | The formula, its inputs and where they came from, checkable by hand | M |
+| Counted FLOPs under `torch.compile` | Counting before compilation instead of an empty card | L |
+| Node calibration | How much of the datasheet peak this particular node delivers | M |
+
+The checklist comes first: until it passes, no number is confirmed.
+New peaks are dense BF16 with a datasheet only, or MFU is not shown (invariant 2).
+Counting FLOPs under `torch.compile` is marked in the architecture as needing a spike, hence L.
+Node calibration runs CUDA and conflicts with invariant 9; see the decisions below.
+
+## Theme 2: a short path
+
+| What to do | The friction it removes | Size |
+| --- | --- | --- |
+| An observe-only banner with the reason | The agent is silent: say which Python lacks trainmeter | S |
+| `tm doctor --python` | Check the training environment, not the one `tm` runs in | S |
+| A URL that does not get lost, and `tm open` | The link and a ready `ssh -L` command in the run directory | S |
+| Ready flags when there is no MFU | The reason and a command to copy | S |
+| Choosing the convention in the dashboard | The MFU range becomes a number in one click | M |
+
+Choosing the convention works at view time and writes nothing to the run log.
+It is possible because FLOPs per token under all four conventions follow from the model shape.
+
+The first run, step by step:
+
+| Step | Today | Target |
+| --- | --- | --- |
+| 1. Install | `pip` refuses, `uv tool` goes to the wrong environment | One `uv add` into the training environment |
+| 2. Run | The agent may fail to attach without saying so | The banner says at once what is seen and what is not |
+| 3. Open | The URL is in the log, the tunnel is set up by hand | `tm open` and a ready `ssh -L` command |
+| 4. See MFU | A range, or nothing without flags | A number with its convention, or an exact hint |
+
+The measure is the spec's: the first MFU on screen within 10 minutes of installation, timed on a clean rented node.
+
+## Theme 3: a quiet night
+
+| What to do | What the user gets | Size |
+| --- | --- | --- |
+| Alerts: hung, throughput dropped | Slice 6 finished; the rules are already on the paused branch | S |
+| A heartbeat to a URL | An outside service notices that the node died | M |
+| A notification through a webhook | A message in Telegram, Slack or ntfy | S |
+| `tm report RUN` | The run's dashboard as one HTML file | M |
+| A copy of the run off the node | The run directory survives the end of the rental | M |
+
+A dead node cannot report its own death, so it needs an outside observer.
+A heartbeat every minute to a URL such as healthchecks.io raises the alarm when the signals stop.
+The heartbeat is off by default: data leaves the node only on an explicit flag.
+`tm report` replaces what the W&B export was expected to give: our own dashboard with every label, as one file that opens anywhere.
+
+## Profilers alongside `tm`
+
+`tm` shows that the GPUs are underused across the whole run; a profiler explains why, inside a window of a few steps.
+We do not build a profiler of our own, but we stay correct next to one and read what it produces.
+
+| What to do | What the user gets | Size |
+| --- | --- | --- |
+| Mark steps taken under `torch.profiler` | Profiled steps leave the EMA, ETA and MFU averages and show as a band on the charts | S |
+| Detect `ncu` and `nsys` and pause GPM | Counter metrics read "profiler attached" instead of wrong values | S |
+| `ncu` as the reference in `hardware-checklist.md` | Observed FLOPs, arithmetic intensity and peaks checked kernel by kernel on a real node | S |
+| Import a `torch.profiler` trace | The step time split into GEMM, attention, exposed NCCL, copies and GPU idle, top kernels, and a per-kernel roofline on our peak table | M |
+| Import an `ncu` report | Speed-of-light and roofline for the kernels of one step | M |
+| Trace capture on demand | A "trace the next steps" button with no change to the training code | L |
+
+`torch.profiler` matters more to us than `ncu`.
+`ncu` says how well one kernel runs, while a trace says where the step time goes, which is where low MFU usually comes from.
+Under `ncu` kernels are replayed and serialized and clocks are locked, so its timings never mix with the run's own.
+Every imported result is labeled as a window under a profiler, never shown in place of the run's metrics (invariants 3 and 4).
+Trace analysis reads JSON with the standard library and runs in `tm`, so nothing new enters the training process; traces reach hundreds of megabytes and are parsed as a stream.
+Capture on demand would have the agent start a profiler inside the run: the profiler may synchronize the device and writes its own file, against invariants 5 and 8, so it waits for a decision below.
+Kineto's on-demand tracing through dynolog is the path to study first, because it puts no code of ours in the process.
+
+## Releases
+
+| Release | Goal | Contents |
+| --- | --- | --- |
+| 0.1 | Correct numbers, quick start | The checklist on an H100, peaks for rented GPUs, the observe-only banner, `tm doctor --python`, `tm open` and the URL, profiled steps marked and GPM paused under a profiler |
+| 0.2 | A quiet night | Alerts from slice 6, heartbeat and webhook, `tm report`, a copy of the run off the node |
+| 0.3 | Deeper into the numbers | Choosing the convention, "how it is computed", counted FLOPs under compile, run comparison, `torch.profiler` trace import |
+
+Run comparison sits in 0.3 conditionally: it needs the hypothesis from `users.md` confirmed first.
+
+## How we will know it worked
+
+The four thresholds are already in the success criteria of `docs/spec.md`:
+
+| Threshold | Measure | Closed by |
+| --- | --- | --- |
+| 10 minutes | From installation to the first MFU on screen | Theme 2 |
+| 1 percentage point | MFU against a hand calculation under the same convention | Theme 1 |
+| 5 minutes | From the node's death to an alert, with the heartbeat on | Theme 3 |
+| Under 0.5% | Overhead per training step | Theme 1, on a real node |
+
+And the standing condition: the loss curve with and without `tm` is identical.
+
+## Deliberately not in this plan
+
+- Multi-node runs, Kubernetes, Slurm: the primary segment works on one node.
+- AMD, TPU, Apple Silicon: NVIDIA first, and checked on it.
+- A profiler of our own: kernels are for `torch.profiler` and Nsight, and `tm` only reads what they produce.
+- Pausing or stopping a run: `tm` only observes and changes nothing.
+- Billing and prices: money stays a view-time overlay.
+- Panels in W&B: `tm report` instead.
+
+The first five match the out-of-scope list in `docs/spec.md`.
+
+## Decisions needed from the owner before 0.1
+
+1. Where and when to get an H100 node for the checklist: [provider, hours].
+2. Node calibration runs CUDA: is it acceptable as a separate command run before training, outside invariant 9?
+3. Which GPUs to add to the peak table first: [L40S, RTX 4090, ...].
+4. The notification channel: a generic webhook, or Telegram and Slack directly?
+5. The W&B export: keep it, write into the run the loop already logs to, or remove it?
+6. Trace capture on demand: is an opt-in exception to invariants 5 and 8 acceptable, or does capture stay with the user?
+7. Trace analysis: our own breakdown on the standard library, or HTA or TraceLens as an optional extra?

{trainmeter-0.0.2 → trainmeter-0.0.4}/docs/spec.md

@@ -73,7 +73,7 @@ Rules:
 
 1. Unknown stays unknown.
    A metric that lacks an input is absent and names the input that would enable it, and a zero is never shown in its place.
-2. An assumed fact carries a visible badge.
+2. An assumed fact is marked `assumed` in the run details, and the dashboard raises a notice for it.
    A declared value that disagrees with a detected one is used, and a warning shows both.
 3. `n_gpus` is the number of GPUs the run uses, taken from the ranks.
    Falling back to the visible devices is an assumed fact with a warning.
@@ -107,7 +107,7 @@ The engine maps the raw key the loop used to a canonical series:
 | `lr` | `lr`, `learning_rate`, `train/lr`, `train/learning_rate` |
 | `grad_norm` | `grad_norm`, `train/grad_norm`, `gradient_norm` |
 
-A mapped series carries a `mapped by name` badge and its original key.
+A mapped series shows its original key, with `mapped by name` in its tooltip.
 `tm.toml` can pin or override a mapping.
 A number that matches no alias still appears as a custom series under its own key.
 
@@ -149,7 +149,7 @@ The last column is the label the dashboard shows beside the number.
 | MFU | FLOPs per token x tokens/s / (GPUs x dense BF16 peak) | model, peak table | convention and peak source |
 | OFU | tensor active x SM clock / max SM clock | GPM | `counters` |
 | MFU-OFU gap | OFU minus MFU | both | both labels |
-| Arithmetic intensity | ridge x MFU / DRAM active, in FLOP per byte of HBM traffic | MFU, GPM, peak table | `aggregate over the window, not per kernel` |
+| Arithmetic intensity | ridge x MFU / DRAM active, in FLOP per byte of memory traffic | MFU, GPM, peak table | `aggregate over the window, not per kernel` |
 | Roofline point | arithmetic intensity against achieved FLOP/s (MFU x peak), on the device's ridge and dense peak | as above | as above |
 
 OFU follows the May 2026 preprint "Instant GPU Efficiency Visibility at Fleet Scale" (arXiv 2605.20799).
@@ -186,7 +186,7 @@ Across ranks: the spread of step time, for straggler detection.
 Overlays are computed from the primitives at view time and never stored in the run log.
 Dollars are GPU-hours times a price per GPU-hour, or elapsed hours times a price per node-hour.
 GPU energy is the integral of the sampled power.
-A price is typed into the dashboard, stays in that browser, and is never sent to the supervisor.
+The dashboard shows no dollars, and a price is never sent to the supervisor.
 
 ## Run log
 
@@ -257,23 +257,35 @@ The run ends when the program does, with no exit status, because `tm` did not st
 
 ## Screens
 
-| Screen | Shows | Answers |
+The dashboard is one page with no tabs.
+Its header holds only the icon, the run state, the rig (GPU count and model) and the theme toggle.
+The x-axis selector sits in the heading of the Training section, next to the charts it changes.
+The look follows beszel: one warm light theme and one cool dark theme, cards with a title and one muted line under it, flat translucent area charts and meter bars.
+Below it, top to bottom:
+
+| Part | Shows | Answers |
 | --- | --- | --- |
-| Cockpit | progress (FLOPs spent of required, tokens, ETA), train and validation loss, tokens/s and step time, MFU and OFU with their gap, lr, grad norm and every other logged series | Is the run on plan? |
-| Efficiency | the utilization ladder over time, MFU against OFU, arithmetic intensity over time, the roofline point, FLOPs spent (modeled next to observed) | Are the GPUs used well, and where does the gap open? |
-| Hardware | per GPU: tensor, DRAM and SM active, power, clocks, memory, temperature, NVLink, stragglers; and the process tree's CPU (cores busy) and resident memory | Which card is slow? |
-| Where the time goes | node time split: boot, startup and compile, data, compute, NCCL, checkpoints, eval, idle | What did the node spend time on without training? |
-| Run passport | architecture, FLOPs per token, hardware, topology, versions, final losses | What goes into the model card? |
-| Plan vs actual | the declared budget and sizing assumptions (MFU, time) next to measured values | How wrong was my plan? |
+| Progress | percent of the token budget, tokens and FLOPs spent of required, ETA, elapsed and GPU-hours | Is the run on plan? |
+| Tiles | tokens/s, step time, MFU, OFU, goodput and FLOPs per token, each a number with a sparkline | How is it going right now? |
+| Training | train and validation loss, tokens/s, MFU against OFU, lr, grad norm, every other logged series, and plan against actual | Is the model learning, and how wrong was my plan? |
+| GPU efficiency | the utilization ladder now and over time, the roofline point, arithmetic intensity over time, FLOPs spent (modeled next to observed) | Are the GPUs used well, and where does the gap open? |
+| Node | node time split (startup and compile, data, checkpoints, eval, compute), and the process tree's CPU (cores busy) and resident memory | What did the node spend time on without training? |
+| GPUs | per GPU: tensor, DRAM and SM active, power, clocks, memory, temperature, NVLink, and step time per rank | Which card is slow? |
+| Run details (collapsed) | the command, host, directory, declared flags, model, FLOPs per token under every convention, hardware and versions | What exactly ran? |
+| Model card (collapsed) | the run passport as Markdown, with a copy button | What goes into the model card? |
+
+A chart with no data is left out, and a part with nothing to draw is one line that names what would fill it.
 
 Rendering rules:
 
-1. Every number shows its convention and provenance, and `assumed` and `mapped by name` carry a visible badge.
-2. A metric that lacks an input names the input that would enable it, such as "budget not declared: pass --budget-tokens", and never shows a zero.
+1. Every MFU and FLOPs number shows its convention next to it in plain text, and a series mapped by name shows its raw key.
+   Provenance (`declared`, `assumed`, `measured`) is listed next to each fact in the run details, and a guessed input also raises a notice at the top of the page.
+   A tile is a label, a number, one short line and a sparkline, with no badges; any longer explanation is in its tooltip.
+2. A metric that lacks an input shows "n/a", names the input that would enable it in its tooltip, such as "budget not declared: pass --budget-tokens", and never shows a zero.
 3. Arithmetic intensity is labeled as an aggregate over the sampling window.
-4. Every time series has an x-axis switch: step, tokens, FLOPs spent (modeled, with its convention shown) or wall time.
+4. Every time series follows one x-axis selector: step, tokens, FLOPs spent (modeled, with its convention shown) or wall time.
    Until a step has been seen, and for a program watched from outside, the charts use wall time, because every point would otherwise sit at zero.
-5. The header has an optional price field that adds dollars spent and dollars per billion tokens, as described under Overlays.
+5. The dashboard shows no price and no dollars.
 6. The dashboard is static files with hand-written SVG charts, no build step and no CDN.
 
 ## MVP
@@ -281,7 +293,7 @@ Rendering rules:
 The first build is slices 1 to 4 of the build order in `docs/architecture.md`.
 It answers two questions: how is the model training, and how well are the GPUs used.
 
-- [x] Peak table (dense BF16 and HBM bandwidth) with datasheet links
+- [x] Device specs (dense BF16 and memory bandwidth), one file per GPU with the datasheet lines they come from
 - [x] FLOPs per token under four named conventions
 - [x] Pure formulas in `metrics.py`: MFU, OFU, arithmetic intensity, ETA, EMA
 - [x] Launcher: `tm -- cmd` with the signal and exit-status contract, run directory, run log, host sampler and exit summary (slice 1)