inference-vitals 0.2.0__tar.gz

inference_vitals-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Urban Morph and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

inference_vitals-0.2.0/PKG-INFO

@@ -0,0 +1,121 @@
+Metadata-Version: 2.4
+Name: inference-vitals
+Version: 0.2.0
+Summary: A dashboard for running and fine-tuning AI models on an Apple Silicon Mac or Linux with NVIDIA GPUs: speed, loss, memory, heat, power and battery
+Keywords: llm,inference,fine-tuning,apple-silicon,mlx,nvidia,prometheus,dashboard,monitoring
+Author: Urban Morph
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Web Environment
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/urbanmorph/inference-vitals
+Project-URL: Documentation, https://github.com/urbanmorph/inference-vitals/tree/main/docs
+Project-URL: Issues, https://github.com/urbanmorph/inference-vitals/issues
+Project-URL: Changelog, https://github.com/urbanmorph/inference-vitals/releases
+Description-Content-Type: text/markdown
+
+<h1>
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://github.com/urbanmorph/inference-vitals/raw/v0.2.0/docs/brand/logo-dark.svg">
+    <img alt="inference-vitals" src="https://github.com/urbanmorph/inference-vitals/raw/v0.2.0/docs/brand/logo-light.svg" height="48">
+  </picture>
+</h1>
+
+[![PyPI](https://img.shields.io/pypi/v/inference-vitals)](https://pypi.org/project/inference-vitals/)
+[![Python](https://img.shields.io/pypi/pyversions/inference-vitals)](https://pypi.org/project/inference-vitals/)
+[![Licence: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/LICENSE)
+[![Tests](https://github.com/urbanmorph/inference-vitals/actions/workflows/test.yml/badge.svg)](https://github.com/urbanmorph/inference-vitals/actions/workflows/test.yml)
+[![Apple Silicon · Linux with NVIDIA](https://img.shields.io/badge/runs_on-Apple_Silicon_%C2%B7_Linux_%2B_NVIDIA-555)](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/install.md)
+
+**One screen for running and fine-tuning AI models on an Apple Silicon Mac, or on Linux with NVIDIA GPUs
+(experimental): how fast the model is (or how well it is learning), how close it is to running out of
+memory, and how hot the machine is getting, on the same timeline.**
+
+![At a glance, during a run: speed, context, answers, memory, swap, GPU, heat and power](https://github.com/urbanmorph/inference-vitals/raw/v0.2.0/docs/screenshots/at-a-glance-dark.png)
+
+Activity Monitor (or `nvtop`) makes you switch views and knows nothing about models. Benchmark apps time
+the requests they send themselves. inference-vitals sits beside whatever you are running and records both
+sides, the model and the machine, so you can see *why* a run slowed down: the GPU got hot, the system
+started swapping, or a page was simply longer.
+
+## What you get
+
+- **Model runs**, with any local engine (Ollama, LM Studio, llama.cpp, the MLX servers, Jan, LocalAI) or
+  your own code: speed live while the answer is written, time to first token, context used, answers cut
+  off or invalid.
+- **Fine-tunes**: training and validation loss, steps and time left, and whether the model is still
+  learning or has started memorising its examples.
+- **The machine**: GPU memory left, memory pressure, swap, GPU use, temperatures, thermal state, power.
+- **Energy**: what the machine used over any range, how much of it was the AI, and the cost at your rate.
+- **Battery** on a laptop: the charge, and on battery how long it will last at this load, with a warning
+  when it will not outlast the run.
+
+It fits one screen: the live numbers stay in view while tabs swap the charts below them, and you star the
+charts you want together in **Favourites**. Every tile and tab: [docs/dashboard.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/dashboard.md).
+
+## Quick start
+
+On an Apple Silicon Mac with [Homebrew](https://brew.sh):
+
+```sh
+brew install uv prometheus macmon
+uv tool install inference-vitals
+inference-vitals start
+open http://127.0.0.1:3000
+```
+
+The machine's numbers appear straight away. Linux with NVIDIA, Grafana, troubleshooting:
+[docs/install.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/install.md).
+
+## Connect what you run
+
+| You run… | Then | More |
+| --- | --- | --- |
+| An engine with an HTTP API (Ollama, LM Studio, llama.cpp, MLX servers) | `inference-vitals proxy --to http://127.0.0.1:11434` and point your app at `http://127.0.0.1:8401` | [connect.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/connect.md) |
+| Your own Python code | Three calls from `inference_vitals.metrics` | [connect.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/connect.md#your-own-python-code-the-library) |
+| A fine-tune (mlx-vlm, mlx-lm, or any loop) | `inference-vitals train -- <your training command>` | [training.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/training.md) |
+
+## Privacy
+
+Everything runs on your machine and listens on `127.0.0.1` only; the page loads nothing from the internet;
+only numbers are collected, never prompts, answers or file names. More: [docs/limits.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/limits.md).
+
+## Documentation
+
+| | |
+| --- | --- |
+| [Install](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/install.md) | Mac and Linux, commands, Grafana, troubleshooting, uninstall |
+| [Connect](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/connect.md) | The proxy and the Python library for model runs; run conditions |
+| [Training](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/training.md) | Following a fine-tune three ways; how "Learning?" is decided |
+| [Dashboard](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/dashboard.md) | Every tile, training mode, energy and cost, battery, address options |
+| [Metrics](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/metrics.md) | Every metric name, for other tools that read Prometheus |
+| [Limits](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/limits.md) | Privacy in full, overhead, what it cannot show |
+| [AGENTS.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/AGENTS.md) | For coding agents helping you: tasks, checks, symptoms, rules |
+| [CONTRIBUTING.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/CONTRIBUTING.md) | Tests, checks, adding a trainer, where things are |
+
+It is [Prometheus](https://prometheus.io) with a dashboard page of its own (or
+[Grafana](https://grafana.com)), [macmon](https://github.com/vladkens/macmon) for the chip on a Mac (on
+Linux, a small exporter that reads `nvidia-smi`), small exporters for memory and the battery, the proxy,
+and a tiny Python library, with no dependencies beyond Python itself.
+
+## Credits
+
+inference-vitals runs [macmon](https://github.com/vladkens/macmon) (MIT) and
+[Prometheus](https://github.com/prometheus/prometheus) (Apache-2.0), and optionally
+[Grafana](https://github.com/grafana/grafana) (AGPL-3.0), installed separately (Homebrew on a Mac, your
+package manager on Linux); none of them is bundled here.
+
+## Licence
+
+MIT. See [LICENSE](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/LICENSE).

inference_vitals-0.2.0/README.md

@@ -0,0 +1,94 @@
+<h1>
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://github.com/urbanmorph/inference-vitals/raw/v0.2.0/docs/brand/logo-dark.svg">
+    <img alt="inference-vitals" src="https://github.com/urbanmorph/inference-vitals/raw/v0.2.0/docs/brand/logo-light.svg" height="48">
+  </picture>
+</h1>
+
+[![PyPI](https://img.shields.io/pypi/v/inference-vitals)](https://pypi.org/project/inference-vitals/)
+[![Python](https://img.shields.io/pypi/pyversions/inference-vitals)](https://pypi.org/project/inference-vitals/)
+[![Licence: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/LICENSE)
+[![Tests](https://github.com/urbanmorph/inference-vitals/actions/workflows/test.yml/badge.svg)](https://github.com/urbanmorph/inference-vitals/actions/workflows/test.yml)
+[![Apple Silicon · Linux with NVIDIA](https://img.shields.io/badge/runs_on-Apple_Silicon_%C2%B7_Linux_%2B_NVIDIA-555)](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/install.md)
+
+**One screen for running and fine-tuning AI models on an Apple Silicon Mac, or on Linux with NVIDIA GPUs
+(experimental): how fast the model is (or how well it is learning), how close it is to running out of
+memory, and how hot the machine is getting, on the same timeline.**
+
+![At a glance, during a run: speed, context, answers, memory, swap, GPU, heat and power](https://github.com/urbanmorph/inference-vitals/raw/v0.2.0/docs/screenshots/at-a-glance-dark.png)
+
+Activity Monitor (or `nvtop`) makes you switch views and knows nothing about models. Benchmark apps time
+the requests they send themselves. inference-vitals sits beside whatever you are running and records both
+sides, the model and the machine, so you can see *why* a run slowed down: the GPU got hot, the system
+started swapping, or a page was simply longer.
+
+## What you get
+
+- **Model runs**, with any local engine (Ollama, LM Studio, llama.cpp, the MLX servers, Jan, LocalAI) or
+  your own code: speed live while the answer is written, time to first token, context used, answers cut
+  off or invalid.
+- **Fine-tunes**: training and validation loss, steps and time left, and whether the model is still
+  learning or has started memorising its examples.
+- **The machine**: GPU memory left, memory pressure, swap, GPU use, temperatures, thermal state, power.
+- **Energy**: what the machine used over any range, how much of it was the AI, and the cost at your rate.
+- **Battery** on a laptop: the charge, and on battery how long it will last at this load, with a warning
+  when it will not outlast the run.
+
+It fits one screen: the live numbers stay in view while tabs swap the charts below them, and you star the
+charts you want together in **Favourites**. Every tile and tab: [docs/dashboard.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/dashboard.md).
+
+## Quick start
+
+On an Apple Silicon Mac with [Homebrew](https://brew.sh):
+
+```sh
+brew install uv prometheus macmon
+uv tool install inference-vitals
+inference-vitals start
+open http://127.0.0.1:3000
+```
+
+The machine's numbers appear straight away. Linux with NVIDIA, Grafana, troubleshooting:
+[docs/install.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/install.md).
+
+## Connect what you run
+
+| You run… | Then | More |
+| --- | --- | --- |
+| An engine with an HTTP API (Ollama, LM Studio, llama.cpp, MLX servers) | `inference-vitals proxy --to http://127.0.0.1:11434` and point your app at `http://127.0.0.1:8401` | [connect.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/connect.md) |
+| Your own Python code | Three calls from `inference_vitals.metrics` | [connect.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/connect.md#your-own-python-code-the-library) |
+| A fine-tune (mlx-vlm, mlx-lm, or any loop) | `inference-vitals train -- <your training command>` | [training.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/training.md) |
+
+## Privacy
+
+Everything runs on your machine and listens on `127.0.0.1` only; the page loads nothing from the internet;
+only numbers are collected, never prompts, answers or file names. More: [docs/limits.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/limits.md).
+
+## Documentation
+
+| | |
+| --- | --- |
+| [Install](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/install.md) | Mac and Linux, commands, Grafana, troubleshooting, uninstall |
+| [Connect](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/connect.md) | The proxy and the Python library for model runs; run conditions |
+| [Training](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/training.md) | Following a fine-tune three ways; how "Learning?" is decided |
+| [Dashboard](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/dashboard.md) | Every tile, training mode, energy and cost, battery, address options |
+| [Metrics](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/metrics.md) | Every metric name, for other tools that read Prometheus |
+| [Limits](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/docs/limits.md) | Privacy in full, overhead, what it cannot show |
+| [AGENTS.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/AGENTS.md) | For coding agents helping you: tasks, checks, symptoms, rules |
+| [CONTRIBUTING.md](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/CONTRIBUTING.md) | Tests, checks, adding a trainer, where things are |
+
+It is [Prometheus](https://prometheus.io) with a dashboard page of its own (or
+[Grafana](https://grafana.com)), [macmon](https://github.com/vladkens/macmon) for the chip on a Mac (on
+Linux, a small exporter that reads `nvidia-smi`), small exporters for memory and the battery, the proxy,
+and a tiny Python library, with no dependencies beyond Python itself.
+
+## Credits
+
+inference-vitals runs [macmon](https://github.com/vladkens/macmon) (MIT) and
+[Prometheus](https://github.com/prometheus/prometheus) (Apache-2.0), and optionally
+[Grafana](https://github.com/grafana/grafana) (AGPL-3.0), installed separately (Homebrew on a Mac, your
+package manager on Linux); none of them is bundled here.
+
+## Licence
+
+MIT. See [LICENSE](https://github.com/urbanmorph/inference-vitals/blob/v0.2.0/LICENSE).

inference_vitals-0.2.0/pyproject.toml

@@ -0,0 +1,59 @@
+[project]
+name = "inference-vitals"
+version = "0.2.0"
+description = "A dashboard for running and fine-tuning AI models on an Apple Silicon Mac or Linux with NVIDIA GPUs: speed, loss, memory, heat, power and battery"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+dependencies = []
+keywords = [
+    "llm",
+    "inference",
+    "fine-tuning",
+    "apple-silicon",
+    "mlx",
+    "nvidia",
+    "prometheus",
+    "dashboard",
+    "monitoring",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Web Environment",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: MacOS",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: System :: Monitoring",
+]
+
+[[project.authors]]
+name = "Urban Morph"
+
+[project.urls]
+Homepage = "https://github.com/urbanmorph/inference-vitals"
+Documentation = "https://github.com/urbanmorph/inference-vitals/tree/main/docs"
+Issues = "https://github.com/urbanmorph/inference-vitals/issues"
+Changelog = "https://github.com/urbanmorph/inference-vitals/releases"
+
+[project.scripts]
+inference-vitals = "inference_vitals.monitor:main"
+
+[build-system]
+requires = ["uv_build>=0.13.7,<0.14.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.2.0",
+    "ruff>=0.17.3",
+]
+
+[tool.ruff]
+line-length = 110

inference_vitals-0.2.0/pyproject.toml.orig

@@ -0,0 +1,49 @@
+[project]
+name = "inference-vitals"
+version = "0.2.0"
+description = "A dashboard for running and fine-tuning AI models on an Apple Silicon Mac or Linux with NVIDIA GPUs: speed, loss, memory, heat, power and battery"
+authors = [
+    { name = "Urban Morph" }
+]
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+dependencies = []
+keywords = ["llm", "inference", "fine-tuning", "apple-silicon", "mlx", "nvidia", "prometheus", "dashboard", "monitoring"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Web Environment",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: MacOS",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: System :: Monitoring",
+]
+
+[project.urls]
+Homepage = "https://github.com/urbanmorph/inference-vitals"
+Documentation = "https://github.com/urbanmorph/inference-vitals/tree/main/docs"
+Issues = "https://github.com/urbanmorph/inference-vitals/issues"
+Changelog = "https://github.com/urbanmorph/inference-vitals/releases"
+
+[project.scripts]
+inference-vitals = "inference_vitals.monitor:main"
+
+[build-system]
+requires = ["uv_build>=0.13.7,<0.14.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.2.0",
+    "ruff>=0.17.3",
+]
+
+[tool.ruff]
+line-length = 110

inference_vitals-0.2.0/src/inference_vitals/conditions.py

@@ -0,0 +1,63 @@
+"""What the Mac was doing when a run happened, so a slow page can be told apart from a hot or busy Mac.
+
+Read from macOS's own commands (pmset, sysctl, sw_vers, ps), none of which need sudo. Process names are
+kept, never their arguments, since arguments can hold file names.
+"""
+
+import re
+import subprocess
+import time
+
+MODES = {"0": "normal", "1": "low power", "2": "high power"}
+
+
+def _out(*cmd) -> str:
+    """A command's output, or "" when the command does not exist (macOS tools on Linux)."""
+    try:
+        return subprocess.run(cmd, capture_output=True, text=True, check=False).stdout
+    except FileNotFoundError:
+        return ""
+
+
+def thermal(text: str) -> dict:
+    def level(pattern, default):
+        m = re.search(pattern, text, re.IGNORECASE)
+        return int(m.group(1)) if m else default
+
+    return {
+        "thermal_warning": level(r"thermal warning level set to (\d+)", 0),
+        "performance_warning": level(r"performance warning level set to (\d+)", 0),
+        "cpu_speed_limit": level(r"CPU_Speed_Limit\s*=\s*(\d+)", 100),
+    }
+
+
+def on_mains(batt: str) -> bool:
+    return "'AC Power'" in batt
+
+
+def power_mode(pm: str) -> str:
+    m = re.search(r"\bpowermode\s+(\d)", pm)
+    return MODES.get(m.group(1), "unknown") if m else "unknown"
+
+
+def busy_processes(ps: str, over: float = 20.0) -> list:
+    busy = []
+    for line in ps.splitlines():
+        m = re.match(r"\s*([\d.]+)\s+(.+)$", line)
+        if m and float(m.group(1)) >= over:
+            busy.append([m.group(2).rstrip().rsplit("/", 1)[-1], float(m.group(1))])
+    return busy
+
+
+def snapshot() -> dict:
+    limit = _out("sysctl", "-n", "iogpu.wired_limit_mb").strip()
+    return {
+        "at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+        "macos": _out("sw_vers", "-productVersion").strip(),
+        "model": _out("sysctl", "-n", "hw.model").strip(),
+        "on_mains": on_mains(_out("pmset", "-g", "batt")),
+        "power_mode": power_mode(_out("pmset", "-g")),
+        **thermal(_out("pmset", "-g", "therm")),
+        "gpu_wired_limit_mb": int(limit) if limit.isdigit() else None,
+        "busy_processes": busy_processes(_out("ps", "-Ao", "pcpu=,comm=", "-r")),
+    }

inference_vitals-0.2.0/src/inference_vitals/follow.py

@@ -0,0 +1,196 @@
+"""Follow a fine-tuning run from what its trainer prints, and publish it for the dashboard. The trainer
+needs no change.
+
+  inference-vitals train -- python -m mlx_vlm.train --model-path ... --data ...   wrap the command
+  inference-vitals watch-training train.log                                     follow a log file
+
+Read: mlx-vlm's trainer (metric blocks such as "Training (iteration 10):" then name=value pairs, its
+progress bar, and "Saved adapter weights" lines) and mlx-lm's ("Iter 10: Train loss 1.23, ..."). Lines
+it does not recognise change nothing. A trainer of your own can publish directly with
+inference_vitals.training instead.
+"""
+
+import argparse
+import codecs
+import os
+import re
+import subprocess
+import sys
+import time
+
+from inference_vitals import training
+from inference_vitals.metrics import DEFAULT_PORT
+
+ANSI = re.compile(r"\x1b\[[0-9;]*m")
+TOTAL = re.compile(r"Starting training\.\.\., (?:iterations|iters): (\d+)")
+HEADER = re.compile(r"^(Training|Validation) \(iteration (\d+)\):")
+PAIR = re.compile(r"\b([a-z_]+)=(-?[\d,]*\.?\d+(?:e[-+]?\d+)?)")
+BAR = re.compile(r"\|\s*(\d+)/(\d+) \[[^\]]*?loss=([\d.]+)(?:, lr=([\d.e+-]+))?")
+SAVED = re.compile(r"Saved (?:final )?(?:adapter|trainable) weights")  # mlx-lm; mlx-vlm 0.7
+LM_TRAIN = re.compile(
+    r"Iter (\d+): Train loss ([\d.]+), Learning Rate ([\d.e+-]+).*?Tokens/sec ([\d.]+)"
+    r"(?:, Trained Tokens (\d+))?.*?Peak mem ([\d.]+) GB"
+)
+LM_VAL = re.compile(r"Iter (\d+): Val loss ([\d.]+)")
+KEYS = {  # mlx-vlm's names -> training.Training.step's
+    "train_loss": "loss",
+    "learning_rate": "learning_rate",
+    "tokens_per_second": "tokens_per_second",
+    "peak_memory": "peak_memory_gb",
+    "trained_tokens": "trained_tokens",
+    "step_time": "step_seconds",  # its own timing: lines arrive in bursts, so timing them would mislead
+}
+
+
+class Reader:
+    """Turns a trainer's printed lines into calls on a training.Training run."""
+
+    def __init__(self, run: training.Training):
+        self.run, self.block, self.step = run, None, 0
+
+    def feed(self, text: str):
+        for part in ANSI.sub("", text).split("\r"):
+            self._part(part.strip())
+
+    def _part(self, s: str):
+        if not s:
+            return
+        run = self.run
+        if m := TOTAL.search(s):
+            run.total_steps = int(m.group(1))
+        elif m := HEADER.match(s):
+            self.block, self.step = m.group(1), int(m.group(2))
+            if self.block == "Validation":
+                run.validating()
+        elif SAVED.search(s):
+            run.saving()
+        elif m := LM_VAL.search(s):
+            run.validating()
+            run.validate(float(m.group(2)))
+        elif m := LM_TRAIN.search(s):
+            step, loss, lr, tps, tokens, mem = m.groups()
+            run.step(int(step), loss=float(loss), learning_rate=float(lr), tokens_per_second=float(tps),
+                     peak_memory_gb=float(mem), trained_tokens=int(tokens) if tokens else None)  # fmt: skip
+        elif m := BAR.search(s):
+            step, total, loss, lr = m.groups()
+            run.total_steps = run.total_steps or int(total)
+            run.step(int(step), loss=float(loss), learning_rate=float(lr) if lr else None)
+        elif self.block and (pairs := PAIR.findall(s)):
+            nums = {k: float(v.replace(",", "")) for k, v in pairs}
+            if self.block == "Validation":
+                if "val_loss" in nums:
+                    run.validate(nums["val_loss"])
+            elif mapped := {KEYS[k]: v for k, v in nums.items() if k in KEYS}:
+                run.step(self.step, **mapped)
+
+
+def _parts(chunks):
+    """The pieces of text between line ends and carriage returns, as chunks arrive."""
+    buf = ""
+    for chunk in chunks:
+        buf += chunk
+        *done, buf = re.split(r"[\r\n]", buf)
+        yield from done
+    if buf:
+        yield buf
+
+
+def train(
+    cmd: list[str], name: str, port: int = DEFAULT_PORT, epochs=None, on_finish=None, linger=20.0
+) -> int:
+    """Run the trainer, pass its output through untouched, and publish what it reports while it runs."""
+    run = training.start(name, epochs=epochs, port=port)
+    reader = Reader(run)
+    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                            env={**os.environ, "PYTHONUNBUFFERED": "1"})  # fmt: skip
+
+    decode = codecs.getincrementaldecoder("utf-8")(errors="replace").decode  # a character may span two reads
+
+    def chunks():
+        while data := os.read(proc.stdout.fileno(), 4096):
+            sys.stdout.buffer.write(data)  # the trainer's own bytes, untouched
+            sys.stdout.buffer.flush()
+            yield decode(data)
+
+    try:
+        for part in _parts(chunks()):
+            reader.feed(part)
+        return proc.wait()
+    except KeyboardInterrupt:  # Ctrl-C: stop the trainer too, and do not wait long for the last read
+        proc.terminate()
+        proc.wait(10)
+        linger = min(linger, 5)
+        return 130
+    finally:
+        run.finish(linger)
+        if on_finish:
+            on_finish(run)
+
+
+IDLE_AFTER = 300  # seconds without a new line before a watched run counts as no longer training
+
+
+def watch(
+    path, run: training.Training, stop=lambda: False, poll=1.0, clock=time.monotonic, idle_after=IDLE_AFTER
+):
+    """Read a log file as it grows, until stop() says so at its end (Ctrl-C from the command line). A log
+    that stops growing for `idle_after` seconds marks the run idle, so it is not shown training for days."""
+    reader = Reader(run)
+    with open(path, errors="replace") as f:
+
+        def chunks():
+            last = clock()
+            while True:
+                text = f.read()
+                if text:
+                    last = clock()
+                    yield text
+                elif stop():
+                    return
+                else:
+                    if clock() - last > idle_after and run.phase != training.PHASES["idle"]:
+                        run.idle()
+                    time.sleep(poll)
+
+        for part in _parts(chunks()):
+            reader.feed(part)
+
+
+def main(action: str, argv: list[str]):
+    p = argparse.ArgumentParser(prog=f"inference-vitals {action}", description=__doc__,
+                                formatter_class=argparse.RawDescriptionHelpFormatter)  # fmt: skip
+    p.add_argument("--name", help="the run's name on the dashboard (default: from the command or file)")
+    p.add_argument("--epochs", type=int)
+    p.add_argument("--port", type=int, default=DEFAULT_PORT, help="where the dashboard reads numbers")
+    if action == "train":
+        own, _, cmd = _split(argv)
+        args = p.parse_args(own)
+        if not cmd:
+            p.error("give the training command after --")
+        return train(cmd, args.name or default_name(cmd), args.port, args.epochs)
+    p.add_argument("log", help="the trainer's log file")
+    args = p.parse_args(argv)
+    run = training.start(args.name or os.path.basename(args.log), epochs=args.epochs, port=args.port)
+    try:
+        watch(args.log, run)
+    except KeyboardInterrupt:
+        pass
+    finally:
+        run.finish()
+    return 0
+
+
+def default_name(cmd: list[str]) -> str:
+    """A run's name when none is given: the module after -m, else the script, else the program."""
+    if "-m" in cmd[:-1]:
+        return cmd[cmd.index("-m") + 1]
+    script = next((a for a in cmd[1:] if a.endswith(".py")), None)
+    return os.path.basename(script or cmd[0])
+
+
+def _split(argv: list[str]):
+    """(our options, "--", the trainer's command)."""
+    if "--" in argv:
+        i = argv.index("--")
+        return argv[:i], "--", argv[i + 1 :]
+    return argv, "", []