torchwatch-tui 0.1.0__tar.gz

torchwatch_tui-0.1.0/.gitignore

@@ -0,0 +1,18 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+
+# Virtual environments
+.venv/
+venv/
+
+# Tooling caches
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+
+# OS
+.DS_Store

torchwatch_tui-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jay Patel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

torchwatch_tui-0.1.0/PKG-INFO

@@ -0,0 +1,138 @@
+Metadata-Version: 2.4
+Name: torchwatch-tui
+Version: 0.1.0
+Summary: A btop for your PyTorch GPU jobs: live terminal dashboard for training runs
+Project-URL: Homepage, https://github.com/JayPatel095/TorchWatch
+Project-URL: Repository, https://github.com/JayPatel095/TorchWatch
+Project-URL: Issues, https://github.com/JayPatel095/TorchWatch/issues
+Author-email: Jay Patel <jayashish.patel@mail.utoronto.ca>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: gpu,monitoring,pytorch,terminal,training,tui
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.10
+Requires-Dist: click
+Requires-Dist: nvidia-ml-py
+Requires-Dist: psutil
+Requires-Dist: textual
+Provides-Extra: dev
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# torchwatch
+
+A btop for your PyTorch GPU jobs with live per-GPU utilization, VRAM pressure, training
+throughput, loss curve, and ETA, all in your terminal, with zero code changes.
+
+![torchwatch demo](https://raw.githubusercontent.com/JayPatel095/TorchWatch/main/docs/demo.gif)
+
+## Install
+
+```
+pip install torchwatch-tui
+```
+
+(The command and package are `torchwatch`; only the PyPI name carries the suffix.)
+
+## Run
+
+The easiest way is to launch your training command under torchwatch:
+
+```
+torchwatch run -- python train.py --epochs 10
+```
+
+Your script runs as a child of torchwatch under a pseudo-terminal. Its output is
+parsed live, and quitting the dashboard (`q`) terminates it.
+
+Attach to a job that's already running:
+
+```
+torchwatch list           # list running PyTorch processes
+torchwatch --pid 12345    # attach to one of them
+```
+
+Attaching reads the process's stdout via `/proc` (Linux only), and only works when
+stdout is redirected to a file (`python train.py > train.log`). If it isn't,
+torchwatch tells you why and suggests alternatives instead of guessing.
+
+Smoke-test the dashboard on synthetic data
+
+```
+torchwatch --demo
+```
+
+## Current Features
+
+- **Per-GPU panels** — utilization, VRAM (used/total and a pressure-colored gauge),
+  temperature, and power with one tile per device
+- **Loss sparkline** — scrolling curve of recent loss values
+- **Throughput + ETA** — steps/sec over a rolling window, elapsed time, and a
+  time-to-completion estimate through epoch restarts
+- **Alerts area** — appears only when something needs attention, and lingers long
+  enough to read:
+  - VRAM above 95% on a specific GPU, with concrete fixes (smaller batch size,
+    mixed precision, gradient checkpointing)
+  - loss stalled (no meaningful change over the last 100 updates)
+  - loss spike (latest value far above the recent average)
+
+Keys: `q` quit · `p` pause
+
+## Supported log formats
+
+torchwatch parses training progress straight from stdout with no imports or callbacks.
+Detected automatically:
+
+| format | example line |
+|---|---|
+| tqdm | ` 42/500 [00:12<02:15, 3.4it/s, loss=0.693]` |
+| PyTorch Lightning | `Epoch 3/10: 42/500 [... loss=0.693 ...]` |
+| HF Trainer | `{'loss': 0.693, 'learning_rate': 5e-05, 'epoch': 1.2}` |
+| plain | `step 42/500 loss: 0.693` |
+
+The header shows which format was detected. Lines that match nothing are ignored.
+
+## Notes
+
+- GPU stats come from NVML (NVIDIA). Without a usable NVML (e.g. macOS, no driver),
+  torchwatch falls back to clearly-labeled mock data so the dashboard still runs.
+- `--pid` attach and process detection use `/proc` and process memory maps, so they
+  are Linux-only. `torchwatch run` works everywhere.
+
+## Development
+
+```
+pip install -e ".[dev]"
+pytest
+ruff check .
+```
+
+## Planned
+
+Roughly in order of likelihood:
+
+- **More alert rules** — NaN/inf loss (the run is dead, say so immediately),
+  GPU temperature, sustained throughput drops (often a dataloader bottleneck)
+- **Alert logging** — opt-in `--alert-log alerts.log`: a timestamped record of
+  every alert, so a spike at 3am is still explainable at 9am
+- **Configurable thresholds** — `--vram-warn` / `--vram-alert` instead of the
+  built-in 85/95
+- **Generic metric sparklines** — track any `key=value` metric your logs already
+  print (accuracy, val_loss, grad_norm, …) and chart the ones you pick
+- **Appearance** — theme selection, optional extra GPU stats
+- **Config file** — persistent defaults once the flag count justifies it
+- **Raw output pane** — the wrapped process's actual stdout, scrolling under
+  the dashboard
+- **Multi-job dashboard** — one dashboard, several training processes

torchwatch_tui-0.1.0/README.md

@@ -0,0 +1,105 @@
+# torchwatch
+
+A btop for your PyTorch GPU jobs with live per-GPU utilization, VRAM pressure, training
+throughput, loss curve, and ETA, all in your terminal, with zero code changes.
+
+![torchwatch demo](https://raw.githubusercontent.com/JayPatel095/TorchWatch/main/docs/demo.gif)
+
+## Install
+
+```
+pip install torchwatch-tui
+```
+
+(The command and package are `torchwatch`; only the PyPI name carries the suffix.)
+
+## Run
+
+The easiest way is to launch your training command under torchwatch:
+
+```
+torchwatch run -- python train.py --epochs 10
+```
+
+Your script runs as a child of torchwatch under a pseudo-terminal. Its output is
+parsed live, and quitting the dashboard (`q`) terminates it.
+
+Attach to a job that's already running:
+
+```
+torchwatch list           # list running PyTorch processes
+torchwatch --pid 12345    # attach to one of them
+```
+
+Attaching reads the process's stdout via `/proc` (Linux only), and only works when
+stdout is redirected to a file (`python train.py > train.log`). If it isn't,
+torchwatch tells you why and suggests alternatives instead of guessing.
+
+Smoke-test the dashboard on synthetic data
+
+```
+torchwatch --demo
+```
+
+## Current Features
+
+- **Per-GPU panels** — utilization, VRAM (used/total and a pressure-colored gauge),
+  temperature, and power with one tile per device
+- **Loss sparkline** — scrolling curve of recent loss values
+- **Throughput + ETA** — steps/sec over a rolling window, elapsed time, and a
+  time-to-completion estimate through epoch restarts
+- **Alerts area** — appears only when something needs attention, and lingers long
+  enough to read:
+  - VRAM above 95% on a specific GPU, with concrete fixes (smaller batch size,
+    mixed precision, gradient checkpointing)
+  - loss stalled (no meaningful change over the last 100 updates)
+  - loss spike (latest value far above the recent average)
+
+Keys: `q` quit · `p` pause
+
+## Supported log formats
+
+torchwatch parses training progress straight from stdout with no imports or callbacks.
+Detected automatically:
+
+| format | example line |
+|---|---|
+| tqdm | ` 42/500 [00:12<02:15, 3.4it/s, loss=0.693]` |
+| PyTorch Lightning | `Epoch 3/10: 42/500 [... loss=0.693 ...]` |
+| HF Trainer | `{'loss': 0.693, 'learning_rate': 5e-05, 'epoch': 1.2}` |
+| plain | `step 42/500 loss: 0.693` |
+
+The header shows which format was detected. Lines that match nothing are ignored.
+
+## Notes
+
+- GPU stats come from NVML (NVIDIA). Without a usable NVML (e.g. macOS, no driver),
+  torchwatch falls back to clearly-labeled mock data so the dashboard still runs.
+- `--pid` attach and process detection use `/proc` and process memory maps, so they
+  are Linux-only. `torchwatch run` works everywhere.
+
+## Development
+
+```
+pip install -e ".[dev]"
+pytest
+ruff check .
+```
+
+## Planned
+
+Roughly in order of likelihood:
+
+- **More alert rules** — NaN/inf loss (the run is dead, say so immediately),
+  GPU temperature, sustained throughput drops (often a dataloader bottleneck)
+- **Alert logging** — opt-in `--alert-log alerts.log`: a timestamped record of
+  every alert, so a spike at 3am is still explainable at 9am
+- **Configurable thresholds** — `--vram-warn` / `--vram-alert` instead of the
+  built-in 85/95
+- **Generic metric sparklines** — track any `key=value` metric your logs already
+  print (accuracy, val_loss, grad_norm, …) and chart the ones you pick
+- **Appearance** — theme selection, optional extra GPU stats
+- **Config file** — persistent defaults once the flag count justifies it
+- **Raw output pane** — the wrapped process's actual stdout, scrolling under
+  the dashboard
+- **Multi-job dashboard** — one dashboard, several training processes

torchwatch_tui-0.1.0/pyproject.toml

@@ -0,0 +1,66 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+# PyPI distribution name; the import package and CLI are still `torchwatch`.
+# (pypi.org/project/torchwatch is an abandoned 0.0.0 placeholder.)
+name = "torchwatch-tui"
+version = "0.1.0"
+description = "A btop for your PyTorch GPU jobs: live terminal dashboard for training runs"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+authors = [
+    { name = "Jay Patel", email = "jayashish.patel@mail.utoronto.ca" },
+]
+keywords = ["pytorch", "gpu", "monitoring", "tui", "terminal", "training"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: System :: Monitoring",
+]
+dependencies = [
+    "textual",
+    "nvidia-ml-py",
+    "psutil",
+    "click",
+]
+
+[project.urls]
+Homepage = "https://github.com/JayPatel095/TorchWatch"
+Repository = "https://github.com/JayPatel095/TorchWatch"
+Issues = "https://github.com/JayPatel095/TorchWatch/issues"
+
+[project.scripts]
+torchwatch = "torchwatch.cli:main"
+
+[project.optional-dependencies]
+dev = [
+    "ruff",
+    "mypy",
+    "pytest",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["torchwatch"]
+
+[tool.hatch.build.targets.sdist]
+# Ship the code and tests, not the demo assets or planning docs.
+exclude = ["docs/", "torchwatch_project_brief.md"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.mypy]
+python_version = "3.10"
+strict = true

torchwatch_tui-0.1.0/tests/fixtures/README.md

@@ -0,0 +1,14 @@
+# Log fixtures
+
+Real or faithful captures of training-log formats the stdout parser must handle.
+
+| file | provenance |
+|---|---|
+| `tqdm.log` | **Real capture**: `tqdm==4.x` `trange` with a `loss` postfix, stderr piped to a file (non-tty — exactly what torchwatch sees when attached). Frames are `\r`-separated. |
+| `hf_trainer.log` | Hand-written to match HuggingFace `Trainer` stdout: dict-repr log lines interleaved with bare tqdm progress. The `eval_loss` / `train_loss` lines are traps — they must NOT parse as training loss. |
+| `plain.log` | Hand-written `print()`-style lines, plus one `Loss=` capitalization variant. |
+| `lightning.log` | Hand-written PyTorch Lightning progress lines (Lightning wraps tqdm, hence the resemblance). |
+| `noise.log` | Typical non-metric stdout: warnings, wandb banner, torchvision download note, a metrics dict with no loss. Nothing here should match. |
+
+When a real-world format breaks the parser, capture the offending lines into a
+new fixture *first*, then fix the regex against it.

torchwatch_tui-0.1.0/tests/fixtures/hf_trainer.log

@@ -0,0 +1,7 @@
+  0%|          | 0/500 [00:00<?, ?it/s]
+{'loss': 2.4915, 'grad_norm': 8.301953315734863, 'learning_rate': 4.95e-05, 'epoch': 0.1}
+ 10%|█         | 50/500 [00:12<01:52,  4.01it/s]
+{'loss': 1.9847, 'grad_norm': 6.192838191986084, 'learning_rate': 4.85e-05, 'epoch': 0.2}
+{'eval_loss': 1.7523119449615479, 'eval_runtime': 12.4817, 'eval_samples_per_second': 80.117, 'epoch': 1.0}
+100%|██████████| 500/500 [02:05<00:00,  4.00it/s]
+{'train_runtime': 125.9012, 'train_samples_per_second': 79.428, 'train_steps_per_second': 3.972, 'train_loss': 1.8912, 'epoch': 2.0}

torchwatch_tui-0.1.0/tests/fixtures/lightning.log

@@ -0,0 +1,5 @@
+GPU available: True (cuda), used: True
+Epoch 0/9:   0%|          | 0/500 [00:00<?, ?it/s]
+Epoch 0/9:  45%|████▌     | 225/500 [00:05<00:06, 41.72it/s, loss=1.892, v_num=3]
+Epoch 0/9: 100%|██████████| 500/500 [00:12<00:00, 41.20it/s, loss=1.204, v_num=3]
+Epoch 1/9:  60%|██████    | 300/500 [00:07<00:04, 41.55it/s, loss=0.987, v_num=3]

torchwatch_tui-0.1.0/tests/fixtures/noise.log

@@ -0,0 +1,6 @@
+/usr/lib/python3/site-packages/torch/cuda/__init__.py:123: UserWarning: CUDA initialization: forked subprocess
+INFO: Using device cuda:0 (NVIDIA A100 80GB)
+wandb: View run at https://wandb.ai/jay/proj/runs/abc123
+Files already downloaded and verified
+{'accuracy': 0.912, 'f1': 0.884}
+Saving checkpoint to ./checkpoints/epoch-1/

torchwatch_tui-0.1.0/tests/fixtures/plain.log

@@ -0,0 +1,6 @@
+Starting training with 1000 steps, lr=0.001, batch_size=64
+Step 100/1000 | loss: 0.4230 | lr: 0.001
+Step 200/1000 | loss: 0.3891 | lr: 0.001
+Step 300/1000 | loss: 0.3514 | lr: 0.0009
+Epoch 2 | Step 350/1000 | Loss=0.3312
+Step 400/1000 | loss: 0.3120 | lr: 0.0009

torchwatch_tui-0.1.0/tests/fixtures/tqdm.log

@@ -0,0 +1 @@
+

torchwatch_tui-0.1.0/tests/test_alert_log.py

@@ -0,0 +1,65 @@
+"""Acceptance tests for AlertLog: dedup, lingering, and expiry."""
+
+from torchwatch.alerts import AlertLog
+
+
+def test_reported_alert_is_active():
+    log = AlertLog(ttl_s=10.0)
+    log.report("vram:0", "vram high", now=0.0)
+    (alert,) = log.active(now=0.0)
+    assert alert.key == "vram:0"
+    assert alert.message == "vram high"
+    assert alert.first_seen == 0.0
+    assert alert.last_seen == 0.0
+
+
+def test_empty_log_has_no_active_alerts():
+    assert AlertLog().active(now=100.0) == []
+
+
+def test_rereport_dedups_by_key():
+    log = AlertLog(ttl_s=10.0)
+    log.report("vram:0", "vram at 95.1%", now=0.0)
+    log.report("vram:0", "vram at 96.4%", now=1.0)
+    (alert,) = log.active(now=1.0)
+    assert alert.message == "vram at 96.4%"  # latest numbers win
+    assert alert.first_seen == 0.0  # but the alert is still the same event
+    assert alert.last_seen == 1.0
+
+
+def test_alert_lingers_after_condition_stops():
+    log = AlertLog(ttl_s=10.0)
+    log.report("spike", "loss spike", now=0.0)
+    assert len(log.active(now=5.0)) == 1  # condition gone, still readable
+    assert len(log.active(now=10.0)) == 1  # boundary: now - last_seen == ttl
+
+
+def test_alert_expires_after_ttl():
+    log = AlertLog(ttl_s=10.0)
+    log.report("spike", "loss spike", now=0.0)
+    assert log.active(now=10.1) == []
+
+
+def test_rereport_refreshes_lifespan():
+    log = AlertLog(ttl_s=10.0)
+    log.report("stall", "loss stalled", now=0.0)
+    log.report("stall", "loss stalled", now=8.0)
+    assert len(log.active(now=15.0)) == 1  # 15 - 8 = 7 < ttl
+    assert log.active(now=18.1) == []
+
+
+def test_active_orders_by_first_seen():
+    log = AlertLog(ttl_s=100.0)
+    log.report("b-later-refire", "b", now=0.0)
+    log.report("a-newcomer", "a", now=1.0)
+    log.report("b-later-refire", "b", now=2.0)  # re-fire must not reorder
+    assert [a.key for a in log.active(now=2.0)] == ["b-later-refire", "a-newcomer"]
+
+
+def test_expired_alert_can_fire_again_as_new():
+    log = AlertLog(ttl_s=10.0)
+    log.report("spike", "loss spike", now=0.0)
+    assert log.active(now=50.0) == []  # long gone
+    log.report("spike", "loss spike again", now=50.0)
+    (alert,) = log.active(now=50.0)
+    assert alert.first_seen == 50.0  # a fresh event, not the old one revived

torchwatch_tui-0.1.0/tests/test_alert_wiring.py

@@ -0,0 +1,145 @@
+"""Alerts must flow from the rules through the AlertLog into the alerts area."""
+
+import asyncio
+
+from textual.app import App
+
+from torchwatch.alerts import Alert
+from torchwatch.app import TorchwatchApp
+from torchwatch.collector.nvidia import GiB, GpuSample
+from torchwatch.collector.stdout import TrainingUpdate
+from torchwatch.widgets.alert_panel import AlertPanel
+from torchwatch.widgets.gpu_panel import GpuPanel
+
+
+def _sample(vram_pct: float) -> GpuSample:
+    total = 80 * GiB
+    return GpuSample(
+        index=0,
+        name="Fake A100",
+        utilization_pct=50,
+        vram_used_bytes=int(total * vram_pct / 100),
+        vram_total_bytes=total,
+        temperature_c=60,
+        power_w=250.0,
+    )
+
+
+class FakeCollector:
+    """Always reports the same sample, so alert conditions are deterministic."""
+
+    is_mock = True
+
+    def __init__(self, vram_pct: float) -> None:
+        self._vram_pct = vram_pct
+
+    def gpu_count(self) -> int:
+        return 1
+
+    def sample(self) -> list[GpuSample]:
+        return [_sample(self._vram_pct)]
+
+
+class ScriptedSource:
+    """Replays a fixed list of losses, then goes quiet."""
+
+    label = "scripted"
+    interval_s = 0.01
+
+    def __init__(self, losses: list[float]) -> None:
+        self._updates = [
+            TrainingUpdate(step=i + 1, total_steps=len(losses), loss=loss, format_name="plain")
+            for i, loss in enumerate(losses)
+        ]
+
+    def next_update(self) -> TrainingUpdate | None:
+        return self._updates.pop(0) if self._updates else None
+
+    def close(self) -> None:
+        pass
+
+
+class PanelHarness(App[None]):
+    def compose(self):
+        yield AlertPanel(id="alert-panel")
+
+
+def test_alert_panel_hidden_when_empty():
+    async def scenario() -> None:
+        app = PanelHarness()
+        async with app.run_test(size=(120, 20)) as pilot:
+            panel = app.query_one(AlertPanel)
+            assert panel.display is False  # collapsed from the start
+
+            panel.show_alerts(
+                [Alert(key="k", message="something bad", first_seen=0.0, last_seen=0.0)]
+            )
+            await pilot.pause()
+            assert panel.display is True
+            assert "something bad" in str(panel.render())
+
+            panel.show_alerts([])
+            await pilot.pause()
+            assert panel.display is False
+
+    asyncio.run(scenario())
+
+
+async def _wait_for_alert(app: TorchwatchApp, pilot, needle: str) -> str:
+    for _ in range(60):
+        await pilot.pause(0.05)
+        panel = app.query_one("#alert-panel", AlertPanel)
+        rendered = str(panel.render())
+        if panel.display and needle in rendered.lower():
+            return rendered
+    raise AssertionError(f"no active alert containing {needle!r} ever appeared")
+
+
+def test_vram_alert_reaches_alert_panel():
+    async def scenario() -> None:
+        app = TorchwatchApp(poll_ms=25, collector=FakeCollector(vram_pct=97.0))
+        async with app.run_test(size=(120, 30)) as pilot:
+            rendered = await _wait_for_alert(app, pilot, "batch")
+            assert "vram" in rendered.lower()
+            assert "gpu 0" in rendered.lower()  # says WHICH gpu is under pressure
+            # the suggestion lives in the alerts area now, not the GPU panel
+            gpu_text = str(app.query_one(GpuPanel).render())
+            assert "batch" not in gpu_text.lower()
+
+    asyncio.run(scenario())
+
+
+def test_healthy_vram_raises_no_alert():
+    async def scenario() -> None:
+        app = TorchwatchApp(poll_ms=25, collector=FakeCollector(vram_pct=60.0))
+        async with app.run_test(size=(120, 30)) as pilot:
+            await pilot.pause(0.5)
+            assert app.query_one("#alert-panel", AlertPanel).display is False
+
+    asyncio.run(scenario())
+
+
+def test_loss_spike_reaches_alert_panel():
+    async def scenario() -> None:
+        app = TorchwatchApp(
+            poll_ms=200,
+            collector=FakeCollector(vram_pct=50.0),
+            metrics_source=ScriptedSource([1.0] * 19 + [2.5]),
+        )
+        async with app.run_test(size=(120, 30)) as pilot:
+            await _wait_for_alert(app, pilot, "spike")
+
+    asyncio.run(scenario())
+
+
+def test_loss_stall_reaches_alert_panel():
+    async def scenario() -> None:
+        app = TorchwatchApp(
+            poll_ms=200,
+            collector=FakeCollector(vram_pct=50.0),
+            metrics_source=ScriptedSource([0.5] * 100),
+        )
+        async with app.run_test(size=(120, 30)) as pilot:
+            await _wait_for_alert(app, pilot, "stall")
+
+    asyncio.run(scenario())

torchwatch_tui-0.1.0/tests/test_alerts.py

@@ -0,0 +1,49 @@
+"""Acceptance tests for alert rules."""
+
+from torchwatch.alerts import is_spiking, is_stalled, vram_suggestion
+
+
+def test_no_suggestion_below_alert_threshold():
+    assert vram_suggestion(50.0) is None
+    assert vram_suggestion(94.9) is None  # warn zone colors, but no nagging
+
+
+def test_suggestion_at_alert_threshold():
+    hint = vram_suggestion(95.0)
+    assert hint is not None
+    lowered = hint.lower()
+    assert "batch" in lowered  # cheapest fix must be mentioned
+    assert "amp" in lowered or "precision" in lowered
+    assert "checkpoint" in lowered
+
+
+def test_flat_loss_stalls():
+    assert is_stalled([1.0] * 100)
+
+
+def test_stall_needs_a_full_window():
+    assert not is_stalled([1.0] * 99)
+    assert not is_stalled([])
+
+
+def test_improving_loss_is_not_stalled():
+    losses = [1.0 - 0.005 * i for i in range(100)]
+    assert not is_stalled(losses)
+
+
+def test_stall_looks_only_at_recent_window():
+    # improved long ago, flat for the last 100 → stalled
+    losses = [2.0 - 0.01 * i for i in range(100)] + [0.5] * 100
+    assert is_stalled(losses)
+
+
+def test_spike_detected():
+    assert is_spiking([1.0] * 19 + [2.5])
+
+
+def test_steady_loss_no_spike():
+    assert not is_spiking([1.0] * 20)
+
+
+def test_spike_needs_a_full_window():
+    assert not is_spiking([1.0, 5.0])