nvmon 0.3.2__tar.gz → 0.3.3__tar.gz

{nvmon-0.3.2 → nvmon-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: nvmon
-Version: 0.3.2
+Version: 0.3.3
 Summary: A fancy NVIDIA GPU monitor for the terminal. One Python file, no dependencies.
 Project-URL: Source, https://github.com/Han-DongHeun/nvmon
 Project-URL: Issues, https://github.com/Han-DongHeun/nvmon/issues
@@ -50,13 +50,13 @@ ssh -t HOST nvmon  # over ssh: -t gives it a terminal
 
 - **Top edge**: number and name, power against its limit, temperature, fan (if any), clock. Warnings
   only when something is off: `SLOWED: power cap | too hot | hw brake`, `PCIe DEGRADED: x16 -> x8`.
-- **Graph**: utilization over time, newest on the right.
+- **Graph**: utilization over time, newest on the right. Point at it for the value at that moment.
 - **GPU**: how much of the time a kernel ran. **cores / tensor** (H100 and newer): how busy that kept
   the chip, and how much was Tensor Core math. One small kernel running nonstop reads GPU 100 % with
   cores at 30 %.
 - **MEM**: memory in use. **CPU -> GPU**, **GPU -> CPU**: PCIe traffic against the link's top speed.
 - **Bottom edge**: the processes, with run time and memory, by account and conda environment (yours
-  unlabelled). `+` opens the process list.
+  unlabelled); `+3` for those that did not fit.
 
 The bottom line has the GPU numbers (click one, or press its key, to hide or show it), the keys, and a
 newer release when there is one.
@@ -85,13 +85,15 @@ Shift+wheel or ← → sideways.
 
 A **job** is a process, or a launcher (torchrun, deepspeed…) with its workers.
 
-- **Pick** one: click a process, or press Tab (Tab and the arrows move on). Its GPUs stand out, and the
-  bottom line sums it up.
-- `f`, or a double click: only its GPUs; again for all.
+- **Pick** one: click a process, or press Tab (Tab and the arrows move on). Its GPUs stand out, the
+  list opens with it picked, and the bottom line sums it up.
+- **Pick a GPU's jobs**: click anywhere else in its box. The same again lets go, and closes the list it
+  opened.
+- `f`, or a double click: only the picked jobs' GPUs; again for all.
 - `p`: every job in a list. Click a heading to sort, again to reverse (or `s`, `r`). ← → scroll the
   picked job's command. Esc, or a click on an empty spot, lets go.
-- `t` stops a job of yours (SIGTERM), `k` kills it (SIGKILL): each after a `y`, and only if it is still
-  the job you picked.
+- `t` stops the picked jobs of yours (SIGTERM), `k` kills them (SIGKILL): after a `y`, and only those
+  still the jobs you picked.
 
 The list, busiest first, one job picked; and that job with `f`:
 
@@ -119,8 +121,9 @@ The list, busiest first, one job picked; and that job with `f`:
 | `c` / `C` | next / previous theme |
 | `p` | process list, open or close |
 | click, Tab, ↑ ↓ | pick a job |
-| `f`, double click | only the picked job's GPUs, or all |
-| `t`, `k`, then `y` | stop (SIGTERM) or kill (SIGKILL) the picked job |
+| click a GPU | pick its jobs |
+| `f`, double click | only the picked jobs' GPUs, or all |
+| `t`, `k`, then `y` | stop (SIGTERM) or kill (SIGKILL) the picked jobs |
 | click a heading, `s`, `r` | sort the list, by the next heading, the other way |
 | wheel, ← → | scroll; sideways the commands |
 | PgUp, PgDn | scroll the GPUs |

{nvmon-0.3.2 → nvmon-0.3.3}/README.md

@@ -28,13 +28,13 @@ ssh -t HOST nvmon  # over ssh: -t gives it a terminal
 
 - **Top edge**: number and name, power against its limit, temperature, fan (if any), clock. Warnings
   only when something is off: `SLOWED: power cap | too hot | hw brake`, `PCIe DEGRADED: x16 -> x8`.
-- **Graph**: utilization over time, newest on the right.
+- **Graph**: utilization over time, newest on the right. Point at it for the value at that moment.
 - **GPU**: how much of the time a kernel ran. **cores / tensor** (H100 and newer): how busy that kept
   the chip, and how much was Tensor Core math. One small kernel running nonstop reads GPU 100 % with
   cores at 30 %.
 - **MEM**: memory in use. **CPU -> GPU**, **GPU -> CPU**: PCIe traffic against the link's top speed.
 - **Bottom edge**: the processes, with run time and memory, by account and conda environment (yours
-  unlabelled). `+` opens the process list.
+  unlabelled); `+3` for those that did not fit.
 
 The bottom line has the GPU numbers (click one, or press its key, to hide or show it), the keys, and a
 newer release when there is one.
@@ -63,13 +63,15 @@ Shift+wheel or ← → sideways.
 
 A **job** is a process, or a launcher (torchrun, deepspeed…) with its workers.
 
-- **Pick** one: click a process, or press Tab (Tab and the arrows move on). Its GPUs stand out, and the
-  bottom line sums it up.
-- `f`, or a double click: only its GPUs; again for all.
+- **Pick** one: click a process, or press Tab (Tab and the arrows move on). Its GPUs stand out, the
+  list opens with it picked, and the bottom line sums it up.
+- **Pick a GPU's jobs**: click anywhere else in its box. The same again lets go, and closes the list it
+  opened.
+- `f`, or a double click: only the picked jobs' GPUs; again for all.
 - `p`: every job in a list. Click a heading to sort, again to reverse (or `s`, `r`). ← → scroll the
   picked job's command. Esc, or a click on an empty spot, lets go.
-- `t` stops a job of yours (SIGTERM), `k` kills it (SIGKILL): each after a `y`, and only if it is still
-  the job you picked.
+- `t` stops the picked jobs of yours (SIGTERM), `k` kills them (SIGKILL): after a `y`, and only those
+  still the jobs you picked.
 
 The list, busiest first, one job picked; and that job with `f`:
 
@@ -97,8 +99,9 @@ The list, busiest first, one job picked; and that job with `f`:
 | `c` / `C` | next / previous theme |
 | `p` | process list, open or close |
 | click, Tab, ↑ ↓ | pick a job |
-| `f`, double click | only the picked job's GPUs, or all |
-| `t`, `k`, then `y` | stop (SIGTERM) or kill (SIGKILL) the picked job |
+| click a GPU | pick its jobs |
+| `f`, double click | only the picked jobs' GPUs, or all |
+| `t`, `k`, then `y` | stop (SIGTERM) or kill (SIGKILL) the picked jobs |
 | click a heading, `s`, `r` | sort the list, by the next heading, the other way |
 | wheel, ← → | scroll; sideways the commands |
 | PgUp, PgDn | scroll the GPUs |

{nvmon-0.3.2 → nvmon-0.3.3}/nvmon.py

@@ -29,7 +29,7 @@ from ctypes import byref, c_int, c_uint, c_ulonglong, c_void_p
 from functools import lru_cache
 from typing import NamedTuple, Optional
 
-__version__ = "0.3.2"  # "+dev": work past this release; the release commit sets the next number
+__version__ = "0.3.3"  # "+dev": work past this release; the release commit sets the next number
 REPO = "https://github.com/Han-DongHeun/nvmon"
 RELEASES = REPO + "/releases"                 # release notes, and nvmon.py for those who copy the file
 PYPI_JSON = "https://pypi.org/pypi/nvmon/json"
@@ -795,6 +795,12 @@ def ranges(numbers):
     return ",".join(str(a) if a == b else "{}-{}".format(a, b) for a, b in spans)
 
 
+def gpu_numbers(indices):
+    """"GPU 3", "GPUs 4-5,7"."""
+    indices = sorted(set(indices))
+    return "GPU{} {}".format("s" if len(indices) > 1 else "", ranges(indices))
+
+
 def account(job):
     """"kim/torch": whose it is, and the conda environment; our own go without an account."""
     return "/".join(part for part in (job.owner, job.env) if part)
@@ -804,7 +810,7 @@ def about(job):
     """"kim/torch · GPUs 4-5 · util 86% · 30.2G · 2h13m · torchrun job" as segments, util in its color;
     the launcher, often a long name, comes last, where a narrow line cuts first."""
     parts = [(account(job), DIM)] if account(job) else []
-    parts.append(("GPU{} {}".format("s" if len(job.gpus) > 1 else "", ranges(job.gpus)), DIM))
+    parts.append((gpu_numbers(job.gpus), DIM))
     if job.util is not None:
         parts.append(("util ", DIM, "{:.0f}%".format(job.util), heat(job.util / 100)))
     parts.append(("{:.1f}G".format(job.mem), DIM))
@@ -817,6 +823,14 @@ def about(job):
     return out
 
 
+def about_all(jobs):
+    """"GPUs 4-7 · 61.2G · 2 yours" for several jobs picked together, as segments."""
+    parts = [gpu_numbers(index for job in jobs for index in job.gpus), "{:.1f}G".format(sum(j.mem for j in jobs))]
+    ours = sum(job.owner is None for job in jobs)
+    parts += ["{} yours".format(ours)] if 0 < ours < len(jobs) else []
+    return [(" · ".join(parts), DIM)]
+
+
 def stop(job, name):
     """Send signal `name` ("SIGTERM" or "SIGKILL") to `job`, once /proc confirms its PIDs still are the
     processes we saw: started when we saw them, holding a GPU open (inside a container, NVML can report
@@ -986,6 +1000,31 @@ def graph(history, width, height, two_tone=True):
     return rows
 
 
+# The pointer over a graph (View.hover): a guide up the empty part of its column, and a chip with that moment.
+GUIDE = _rgb((150, 150, 150))
+CHIP = "\x1b[48;2;235;235;235m" + _rgb((24, 24, 27))
+COLOR_24 = re.compile(r"(\x1b\[[34]8;2;)(\d+);(\d+);(\d+)m")
+
+
+def lit(style):
+    """`style` lit up, for the graph column under the pointer: its colors halfway to white; on PAPER, where
+    restyle turns lightness around, halfway to black."""
+    paper = ON_PAPER if style.startswith(ON_PAPER) else ""
+    to = 0 if paper else 255
+    return paper + COLOR_24.sub(lambda m: m.group(1) + ";".join(
+        str((int(v) + to) // 2) for v in m.group(2, 3, 4)) + "m", style[len(paper):])
+
+
+def ago(seconds):
+    """How long ago, as short as a run time: now, 14.5s ago, 3m05s ago, 1h02m ago."""
+    if seconds < 0.05:
+        return "now"
+    if seconds < 60:
+        return "{:g}s ago".format(round(seconds, 1))
+    m, s = divmod(int(round(seconds)), 60)
+    return "{}m{:02d}s ago".format(m, s) if m < 60 else "{}h{:02d}m ago".format(m // 60, m % 60)
+
+
 def row(left, right=()):
     """One stats line: `left`, then the `right` segments flush right; exactly INFO_W cells."""
     room = INFO_W - width_of(right)
@@ -1034,12 +1073,12 @@ def degraded(gpu):
     return bool(s.pcie_width and gpu.pcie_max_width and s.pcie_width < gpu.pcie_max_width)
 
 
-def panel(gpu, width, height, selected=None, left="graph", scroll=(0, 0)):
+def panel(gpu, width, height, selected=frozenset(), left="graph", scroll=(0, 0)):
     """One GPU's box: beside the stats, as `left` says (see LEFTS), the utilization graph, or the GPU's
     processes one a line, moved `scroll` = (lines, cells) on (see process_lines); or the stats alone. Faded
-    when a job is `selected` and it does not run here."""
+    when jobs are `selected` (their PIDs) and none runs here."""
     s = gpu.now
-    faded = selected is not None and not any(p.job == selected for p in s.processes)
+    faded = bool(selected) and not any(p.job in selected for p in s.processes)
     # Both sides as (segments, rank); when space runs out the lowest rank goes first:
     # fan, name, power, clock, temperature, PCIe warning, slowdown warning. The GPU number stays.
     # Fixed widths keep things from shifting.
@@ -1053,20 +1092,20 @@ def panel(gpu, width, height, selected=None, left="graph", scroll=(0, 0)):
     parts += [] if s.fan is None else [([("FAN {:>3}%".format(s.fan), "")], 1)]
     parts += [] if s.clock is None else [([("{:>4} MHz".format(s.clock), s.slowdown[1] if s.slowdown else "")], 4)]
     top = edge(width, label, parts)
-    if left in CARDS:  # the numbers only: of the processes, just the way to the list
+    if left in CARDS:  # the numbers only, no processes
         split = width - 1
         body = [[("├" + "─" * (width - 2) + "┤", DIM)] if i is None else [("│ ", DIM)] + i + [(" │", DIM)]
                 for i in info(gpu, height)]
-        label = [("+", DIM, ("list", gpu.index))]
+        label = []
     else:
         graph_w = max(0, width - CHROME_W - INFO_W)
         split = 2 + graph_w + 1  # column of the graph | stats divider
         rule = [(" ├" + "─" * (INFO_W + 2) + "┤", DIM)]  # across the stats, joined to the borders
         if left == "graph":
             beside = graph(gpu.history, graph_w, height, not faded)
-            label = process_label(s.processes, split - 4, selected, gpu.index)
+            label = process_label(s.processes, split - 4, selected)
         else:
-            beside, label = process_lines(s.processes, graph_w, height, selected, gpu.index, *scroll)
+            beside, label = process_lines(s.processes, graph_w, height, selected, *scroll)
         body = [[("│ ", DIM)] + g + (rule if i is None else [(" │ ", DIM)] + i + [(" │", DIM)])
                 for g, i in zip(beside, info(gpu, height))]
     lines = [top] + body + [bottom_edge(width, split, label)]
@@ -1126,9 +1165,9 @@ def process_entries(processes):
             for process in group]
 
 
-def process_name(process, selected=None):
-    """The segment of `process`'s name: clicked, it means its job; the `selected` job's is reversed."""
-    return process.name, THEME.selected if process.job == selected else THEME.accent, process.job
+def process_name(process, selected=frozenset()):
+    """The segment of `process`'s name: clicked, it means its job; those of `selected` jobs reversed."""
+    return process.name, THEME.selected if process.job in selected else THEME.accent, process.job
 
 
 def run_time(process):
@@ -1139,10 +1178,9 @@ def memory(process):
     return "-" if process.mem is None else "{:.1f}G".format(process.mem)
 
 
-def process_label(processes, room, selected=None, index=None):
+def process_label(processes, room, selected=frozenset()):
     """The processes (process_entries) on one line: "migi: train.py 2h13m 15.1G · eval.py 5m 0.5G   kim/torch:
-    a.py 1d4h 9.0G". As many whole entries as fit in `room` cells, then "+N" for the rest, or "+" when none
-    is left out: clicked, it opens the process list at GPU `index`."""
+    a.py 1d4h 9.0G". As many whole entries as fit in `room` cells, then "+N" for the rest."""
     entries, last = [], None
     for tag, process in process_entries(processes):
         gap = [] if not entries else [(" · ", DIM)] if tag == last else [("   ", "")]
@@ -1152,29 +1190,29 @@ def process_label(processes, room, selected=None, index=None):
         last = tag
 
     def more(n):
-        return ("  +{}".format(n) if n else "  +", DIM, ("list", index))
+        return [("  +{}".format(n), DIM)] if n else []
     label = []
     for i, entry in enumerate(entries):
         rest = len(entries) - i - 1
-        if width_of(label + entry) + len(more(rest)[0]) > room:
+        if width_of(label + entry + more(rest)) > room:
             # The first entry is always shown, cut if need be; later ones collapse into the count.
-            return clip(entry, room - len(more(rest)[0])) + [more(rest)] if not label else label + [more(rest + 1)]
+            return clip(entry, room - width_of(more(rest))) + more(rest) if not label else label + more(rest + 1)
         label += entry
-    return label + [more(0)] if label else []
+    return label
 
 
-def process_lines(processes, room, height, selected=None, index=None, top=0, shift=0):
+def process_lines(processes, room, height, selected=frozenset(), top=0, shift=0):
     """The processes (process_listing) one a line, `height` lines of `room` cells, from
     line `top` on and `shift` cells in (see listing_extent), and the label for the box's bottom edge: which
-    lines show of how many, when not all do, and "+", which clicked opens the process list at GPU `index`."""
+    lines show of how many, when not all do."""
     lines = process_listing(processes, selected)
     # Sideways the command moves; account, run time and memory, the first segment, stay.
     shown = [pad(clip(line[:1] + skip(line[1:], shift), room), room) for line in lines[top:top + height]]
-    label = [("{}-{} of {}  ".format(top + 1, top + len(shown), len(lines)), DIM)] if len(lines) > height else []
-    return shown + [[(" " * room, "")]] * (height - len(shown)), label + [("+", DIM, ("list", index))]
+    label = [("{}-{} of {}".format(top + 1, top + len(shown), len(lines)), DIM)] if len(lines) > height else []
+    return shown + [[(" " * room, "")]] * (height - len(shown)), label
 
 
-def process_listing(processes, selected=None):
+def process_listing(processes, selected=frozenset()):
     """process_lines's lines, whole: account and environment, run time and memory in columns (one segment),
     then the command line, the name in the process names' color and the arguments dim: "kim/torch:  2d5h
     26.1G  train.py --lr 3". The columns are as wide as the GPU's longest."""
@@ -1240,7 +1278,7 @@ def layout(count, width, height, left="graph"):
     return cols, max(MIN_INNER_H, min(MAX_INNER_H, height // rows - 2))
 
 
-def render(gpus, width, height, selected=None, shape=None, left="graph", scroll=None):
+def render(gpus, width, height, selected=frozenset(), shape=None, left="graph", scroll=None):
     """The GPU boxes, in `shape` = (columns, inner height) or else the layout that fits: exactly `height`
     lines of exactly `width` cells. `left`: see panel; the processes moved on as `scroll` has it for each
     GPU (its index: (lines, cells))."""
@@ -1287,10 +1325,10 @@ def cells(values, width, fill=("",)):
     return line
 
 
-def process_list(jobs, width, rows, selected, top, sort, shift=0):
+def process_list(jobs, width, rows, selected, focus, top, sort, shift=0):
     """The list p opens, `rows` lines: the headings (clicking one sorts by it; `sort` = (column, descending)),
-    the jobs from index `top` on (clicking one means it; the `selected` one reversed, its command moved
-    `shift` characters on), how many more. A command too long to show ends in "…"."""
+    the jobs from index `top` on (clicking one means it; the `selected` ones reversed, the command of the one
+    in `focus` moved `shift` characters on), how many more. A command too long to show ends in "…"."""
     room = max(0, width - COMMAND_AT - 1)
     column, descending = sort
     head = []
@@ -1301,9 +1339,9 @@ def process_list(jobs, width, rows, selected, top, sort, shift=0):
     lines = [cells(head, width)]
     shown = jobs[top:top + rows - 2]
     for job in shown:
-        chosen = job.pid == selected
+        chosen = job.pid in selected
         values = [cell(job) for _, _, cell, _ in COLUMNS]
-        command = "…" + job.command[shift:] if chosen and shift else job.command
+        command = "…" + job.command[shift:] if job.pid == focus and shift else job.command
         values[-1] = (command if len(command) <= room else command[:max(0, room - 1)] + "…", DIM)
         lines.append(cells([(text, THEME.selected if chosen else color, job.pid) for text, color in values],
                            width, (THEME.selected if chosen else "", job.pid)))
@@ -1395,22 +1433,24 @@ MOVES = {"tab": 1, "down": 1, "up": -1, "backtab": -1, "pgdn": 10, "pgup": -10,
 
 
 class View:
-    """What the screen shows around the numbers, and what keys and clicks do to it: the selected job,
-    only its GPUs, the process list, a question before stopping, a note on how that went."""
+    """What the screen shows around the numbers, and what keys and clicks do to it: the picked jobs (one, or
+    a GPU's), only their GPUs, the process list, a question before stopping, a note on how that went."""
 
     def __init__(self):
         self.shows = "truecolor"          # the colors the terminal shows, see color_mode
         self.theme = THEMES[0]            # the look, see THEMES (c)
         self.left = "graph"               # what the boxes have beside the stats, see LEFTS (v)
         self.hidden, self.indices = set(), []  # the GPUs left out, and all there are
-        self.job = None                   # the selected job's PID
-        self.only = False                 # show only the GPUs it uses
+        self.picked = set()               # the picked jobs' PIDs
+        self.job = None                   # of them the one in focus: the arrows go on from it, ← → move its command
+        self.only = False                 # show only the GPUs they use
         self.listing = False              # the process list is open
-        self.top, self.follow = 0, True   # the list's first job; keep the selected one in view
+        self.opened = False               # by a pick, and so to close when the picks are let go
+        self.top, self.follow = 0, True   # the list's first job; keep the one in focus in view
         self.sort = ("GPUs", False)       # the list's order, and the order Tab and the arrows go in
         self.asking = None                # "SIGTERM" or "SIGKILL" awaiting y or n
         self.note = None                  # (segments, until): what the last stop did
-        self.sent = None                  # (PID, when) of a SIGTERM, to point at k if the job lives on
+        self.sent = None                  # (PIDs, when) of a SIGTERM, to point at k if a job lives on
         self.lines, self.jobs = [], []    # the last screen drawn, and the jobs on it
         self.gpu_top, self.gpu_page = 0, 1  # the first row of boxes shown, and how many rows fit
         self.list_rows = range(0)         # the screen rows the list takes, when it is open
@@ -1418,6 +1458,10 @@ class View:
         self.clicked = (0, None)          # (when, what): the last click, to tell a double click
         self.scroll = {}                  # GPU index: (lines, cells) its processes are moved on
         self.areas = []                   # (rows, columns, GPU) where those processes are on screen
+        self.graphs = []                  # (rows, columns, GPU) where the graphs are on screen
+        self.panels = []                  # (rows, columns, GPU index) where the boxes are on screen
+        self.pointer = None               # (row, column) of the mouse pointer, as the terminal last said
+        self.interval = DEFAULT_INTERVAL  # seconds from one graph column to the next
 
     def restore(self, saved, indices):
         """Take up what an earlier run kept (Settings.load); of the GPUs `indices`, one always stays shown."""
@@ -1431,8 +1475,19 @@ class View:
         """What is kept for the next run: see Settings."""
         return {"left": self.left, "theme": self.theme.name, "sort": self.sort, "hidden": set(self.hidden)}
 
+    def pick(self, pids):
+        """Pick the jobs `pids`, in the list's order, the first in focus; none lets all go, and closes the
+        list if a pick opened it."""
+        self.picked = set(pids)
+        self.job = next((job.pid for job in self.jobs if job.pid in self.picked), None)
+        if not self.picked:
+            self.only = False
+            self.listing = self.listing and not self.opened
+            self.opened = False
+
     def selected(self):
-        return next((job for job in self.jobs if job.pid == self.job), None)
+        """The picked jobs, in the list's order."""
+        return [job for job in self.jobs if job.pid in self.picked]
 
     def screen(self, gpus, width, height, top_line, newer):
         """The whole screen: `top_line`, the GPU boxes (and the list, when open), the bottom line."""
@@ -1440,12 +1495,17 @@ class View:
         THEME, SHOWS = self.theme, self.shows
         column, descending = self.sort
         self.jobs = sorted(jobs(gpus), key=lambda job: (SORTS[column](job), job.pid), reverse=descending)
-        job = self.selected()
-        if job is None:  # nothing selected, or its job ended
+        self.picked &= {job.pid for job in self.jobs}  # those that ended go
+        if not self.picked:
             self.job, self.only, self.asking = None, False, None
+        elif self.job not in self.picked:
+            self.job = next(job.pid for job in self.jobs if job.pid in self.picked)
+        picked = self.selected()
+        focus = next(job for job in picked if job.pid == self.job) if picked else None
         self.remind()
         self.indices = [g.index for g in gpus]
-        shown = [g for g in gpus if (g.index in job.gpus if self.only else g.index not in self.hidden)]
+        used = {index for job in picked for index in job.gpus}
+        shown = [g for g in gpus if (g.index in used if self.only else g.index not in self.hidden)]
         rows = min(len(self.jobs) + 2, max(4, (height - 2) // 2)) if self.listing else 0
         body = self.boxes(shown, width, height - 2 - rows)
         self.list_rows = range(1 + len(body), 1 + len(body) + rows)
@@ -1453,9 +1513,10 @@ class View:
             if self.shifted != self.job:
                 self.shift, self.shifted = 0, self.job
             room = max(0, width - COMMAND_AT - 1)
-            self.shift = max(0, min(self.shift, len(job.command) + 1 - room)) if job else 0
-            body += process_list(self.jobs, width, rows, self.job, self.scrolled(rows - 2), self.sort, self.shift)
-        self.lines = [top_line] + body + [self.bottom_line(width, job, newer, gpus)]
+            self.shift = max(0, min(self.shift, len(focus.command) + 1 - room)) if focus else 0
+            body += process_list(self.jobs, width, rows, self.picked, self.job, self.scrolled(rows - 2), self.sort,
+                                 self.shift)
+        self.lines = self.hover([top_line] + body + [self.bottom_line(width, picked, newer, gpus)])
         return self.lines
 
     def boxes(self, gpus, width, height):
@@ -1474,34 +1535,83 @@ class View:
             height -= 1
         else:
             self.gpu_top, self.gpu_page = 0, rows
-        box_w, room = width // cols, max(0, width // cols - CHROME_W - INFO_W)
-        self.areas = []
-        for k, g in enumerate(gpus if self.left == "processes" else []):  # their offsets in range, where they are
-            most = listing_extent(g.now.processes, room, inner)
+        box_w = CARD_W if self.left in CARDS else width // cols
+        room = max(0, box_w - CHROME_W - INFO_W)
+        self.areas, self.graphs = [], []
+        self.panels = [(range(1 + k // cols * (inner + 2), 1 + (k // cols + 1) * (inner + 2)),
+                        range(k % cols * box_w, (k % cols + 1) * box_w), g.index) for k, g in enumerate(gpus)]
+        for k, g in enumerate(gpus if self.left in ("graph", "processes") else []):
+            y, x = 2 + k // cols * (inner + 2), k % cols * box_w + 2  # 2: the top line and the box's edge
+            if self.left == "graph":
+                self.graphs.append((range(y, y + inner), range(x, x + room), g))
+                continue
+            most = listing_extent(g.now.processes, room, inner)  # the processes' offsets in range
             top, shift = self.scroll.get(g.index, (0, 0))
             self.scroll[g.index] = min(max(top, 0), most[0]), min(max(shift, 0), most[1])
-            y, x = 2 + k // cols * (inner + 2), k % cols * box_w + 2  # 2: the top line and the box's edge
             self.areas.append((range(y, y + inner), range(x, x + room), g.index))
-        return render(gpus, width, height, self.job, (cols, inner), self.left, self.scroll) + note
+        return render(gpus, width, height, self.picked, (cols, inner), self.left, self.scroll) + note
+
+    def hovered(self):
+        """(GPU, graph rows, graph columns, the column pointed at, its sample) when the pointer is over a graph
+        column that has one; else None."""
+        if self.pointer is None:
+            return None
+        row, col = self.pointer
+        for rows, cols, gpu in self.graphs:
+            if row in rows and col in cols:
+                back = cols.stop - 1 - col  # samples back from the newest, on the right
+                if back < len(gpu.history):
+                    return gpu, rows, cols, col, gpu.history[-1 - back]
+        return None
+
+    def pointed(self):
+        """(GPU index, column) of the graph column under the pointer, or None: what the chip is about."""
+        hovered = self.hovered()
+        return hovered and (hovered[0].index, hovered[3])
+
+    def hover(self, lines):
+        """`lines` with the column under the pointer lit up, and beside the pointer a chip with that moment:
+        the utilization, and how long ago."""
+        hovered = self.hovered()
+        if hovered is None:
+            return lines
+        gpu, rows, cols, col, value = hovered
+        back = (cols.stop - 1 - col) * self.interval
+        chip = " {}% · {} ".format(value, ago(back))
+        lines = list(lines)
+        for row in rows:
+            cells = [(ch,) + tuple(seg[1:]) for seg in lines[row] for ch in seg[0]]
+            ch, style = cells[col][:2]
+            paper = ON_PAPER if style.startswith(ON_PAPER) else ""
+            cells[col] = ("│", paper + GUIDE) if ch == " " else (ch, lit(style))
+            if row == self.pointer[0]:
+                start = col + 2 if col + 2 + len(chip) <= cols.stop else col - 1 - len(chip)
+                if start >= cols.start:
+                    bold = chip.index("%") + 1
+                    for k, c in enumerate(chip):
+                        cells[start + k] = (c, paper + CHIP + (BOLD if k < bold else ""))
+            lines[row] = [("".join(c[0] for c in run),) + key for key, run in
+                          itertools.groupby(cells, key=lambda c: c[1:])]
+        return lines
 
     def detail(self, gpus):
-        """The GPUs whose processes' utilization is shown: all with the list open, else the selected
-        job's. Only those are asked, as the driver takes some 2 ms a GPU to answer."""
+        """The GPUs whose processes' utilization is shown: all with the list open, else the picked jobs'.
+        Only those are asked, as the driver takes some 2 ms a GPU to answer."""
         if self.listing:
             return {g.index for g in gpus}
-        job = self.selected()
-        return set(job.gpus) if job else set()
+        return {index for job in self.selected() for index in job.gpus}
 
     def remind(self):
         """A job still there 5 s after its SIGTERM may be stuck, or slow to save: say that k ends it."""
         if not self.sent:
             return
-        pid, when = self.sent
-        job = next((job for job in self.jobs if job.pid == pid), None)
-        if job and time.monotonic() - when > 5:
-            self.note = ([("{} is still running 5 s after SIGTERM; k kills it at once".format(job.name), WARN)],
+        pids, when = self.sent
+        living = [job for job in self.jobs if job.pid in pids]
+        if living and time.monotonic() - when > 5:
+            what = living[0].name + " is" if len(living) == 1 else "{} jobs are".format(len(living))
+            self.note = ([("{} still running 5 s after SIGTERM; k kills at once".format(what), WARN)],
                          time.monotonic() + 6)
-        if not job or time.monotonic() - when > 5:
+        if not living or time.monotonic() - when > 5:
             self.sent = None
 
     def scrolled(self, room):
@@ -1521,14 +1631,18 @@ class View:
         return [("GPUs", DIM)] + [(" {}".format(g.index), FAINT if g.index in self.hidden else BOLD, ("gpu", g.index))
                                   for g in gpus]
 
-    def bottom_line(self, width, job, newer, gpus):
+    def bottom_line(self, width, picked, newer, gpus):
         if self.asking:
-            return spread(width, [(self.question(job), WARN)], keys(("y", "yes"), ("n", "no")))
+            return spread(width, [(self.question(picked), WARN)], keys(("y", "yes"), ("n", "no")))
         note = self.note[0] if self.note and time.monotonic() < self.note[1] else None
-        if job:
+        if picked:
             hints = [("f", "all GPUs" if self.only else "only these GPUs")]
-            hints += [("t", "stop"), ("k", "kill")] if job.owner is None and os.name != "nt" else []
-            left = [(job.name, THEME.accent, job.pid), ("  ", DIM)] + about(job)
+            ours = any(job.owner is None for job in picked) and os.name != "nt"
+            hints += [("t", "stop"), ("k", "kill")] if ours else []
+            if len(picked) == 1:
+                left = [(picked[0].name, THEME.accent, picked[0].pid), ("  ", DIM)] + about(picked[0])
+            else:
+                left = [("{} jobs".format(len(picked)), THEME.accent), ("  ", DIM)] + about_all(picked)
             return spread(width, note or left, keys(*hints, ("Esc", "back")))
         if self.listing:
             return spread(width, note or [("click a job or a heading", DIM)],
@@ -1537,14 +1651,24 @@ class View:
             return spread(width, note, keys(("p", "processes"), ("Esc / q", "quit")))
         return footer(width, newer, self.picker(gpus))
 
-    def question(self, job):
-        """"stop train.py: SIGTERM to its torchrun (PID 48213), which ends its 2 workers?" and the like."""
+    def question(self, picked):
+        """"stop train.py on GPUs 4-5: SIGTERM to its torchrun (PID 48213), which ends its 2 workers?" and the
+        like; for several jobs "stop your 2 of 3 jobs on GPUs 4-5,7, SIGTERM to each: train.py, eval.py?". The
+        GPUs are those the signal reaches, whichever was clicked."""
         what = "stop" if self.asking == "SIGTERM" else "kill"
+        ours = [job for job in picked if job.owner is None]
+        where = "on " + gpu_numbers(index for job in ours for index in job.gpus)
+        if len(picked) > 1:
+            which = "{} jobs".format(len(ours)) if len(ours) == len(picked) else "your {} of {} jobs".format(
+                len(ours), len(picked))
+            return "{} {} {}, {} to each: {}?".format(what, which, where, self.asking,
+                                                     ", ".join(job.name for job in ours))
+        job = ours[0]
         if not job.launcher:
-            return "{} {} (PID {}) with {}?".format(what, job.name, job.pid, self.asking)
+            return "{} {} {} (PID {}) with {}?".format(what, job.name, where, job.pid, self.asking)
         passed = ", which ends its" if self.asking == "SIGTERM" else " and to its"
-        return "{} {}: {} to its {} (PID {}){} {} workers?".format(what, job.name, self.asking, job.launcher,
-                                                                  job.pid, passed, len(job.members))
+        return "{} {} {}: {} to its {} (PID {}){} {} workers?".format(what, job.name, where, self.asking,
+                                                                     job.launcher, job.pid, passed, len(job.members))
 
     def target(self, row, col):
         """What a click at (row, col) of the last screen means; None for nothing."""
@@ -1559,6 +1683,9 @@ class View:
     def handle(self, event):
         """Act on one key or mouse event; False means quit."""
         kind, value = event[0], event[1:]
+        if kind == "move":
+            self.pointer = value
+            return True
         if kind in ("wheel", "hwheel"):  # over a GPU's processes, they scroll, either way;
             step, row, col = value     # over the list, the list, or the picked job's command sideways;
             area = next((index for rows, cols, index in self.areas  # anywhere else, the GPU boxes
@@ -1577,10 +1704,6 @@ class View:
             else:
                 self.gpu_top += step
             return True
-        if kind == "list":  # "+" under a GPU: the list, from that GPU's first job
-            self.listing, self.follow = True, False
-            self.top = next((i for i, job in enumerate(self.jobs) if value[0] in job.gpus), self.top)
-            return True
         if kind == "gpu":  # a GPU's number at the bottom: hide that GPU, or show it again; one always stays
             index = value[0]
             if index in self.hidden:
@@ -1593,26 +1716,40 @@ class View:
             self.sort, self.follow = (value[0], not descending if value[0] == column else False), True
             return True
         if kind == "click":
-            target = self.target(*value)
-            if isinstance(target, tuple):  # a key hint, a heading of the list, "+" under a GPU
+            row, col = value
+            target = self.target(row, col)
+            if isinstance(target, tuple):  # a key hint, a heading of the list, a GPU's number at the bottom
                 return self.handle(target)
-            # A job picks it; the picked job again, or a place meaning nothing, lets it go, and outside the
-            # list closes that too. A double click on a job picks it and shows only its GPUs, or all again.
+            if target is None:  # anywhere else in a GPU's box: that GPU
+                target = next((("box", index) for rows, cols, index in self.panels if row in rows and col in cols),
+                              None)
             now = time.monotonic()
-            if target is None and value[0] not in self.list_rows:
-                self.listing = False
-            if target is not None and self.clicked[1] == target and now - self.clicked[0] < 0.4:
-                self.job, self.only = target, not self.only
-            else:
-                self.job = target if target is not None and target != self.job else None
+            double = target is not None and self.clicked[1] == target and now - self.clicked[0] < 0.4
             self.asking, self.follow, self.clicked = None, True, (now, target)
+            # A job picks it, a GPU the jobs on it, and outside the list opens that; the same again lets them
+            # go, as does a place meaning nothing, which outside the list closes that too. A double click picks
+            # them and shows only their GPUs, or all again.
+            if target is None:
+                self.pick(())
+                self.listing = self.listing and row in self.list_rows
+                return True
+            pids = [job.pid for job in self.jobs if target[1] in job.gpus] if isinstance(target, tuple) else [target]
+            if not pids:
+                self.note = ([("no processes on GPU {}".format(target[1]), DIM)], now + 2)
+            elif double:
+                self.pick(pids)
+                self.only = not self.only
+            elif set(pids) == self.picked:
+                self.pick(())
+            else:
+                self.pick(pids)
+                self.opened = self.opened or not self.listing
+                self.listing = True
             return True
         key = value[0].lower() if len(value[0]) == 1 else value[0]
         if self.asking:  # y stops; anything else, n and Esc among them, lets it be
             if key == "y":
-                note, sent = stop(self.selected(), self.asking)
-                self.note = (note, time.monotonic() + 4)
-                self.sent = (self.job, time.monotonic()) if sent and self.asking == "SIGTERM" else None
+                self.stop_picked()
             self.asking = None
             return key != "q"
         if key == "q":
@@ -1622,12 +1759,12 @@ class View:
                 self.only = False
             elif self.listing:
                 self.listing = False
-            elif self.job is not None:
-                self.job = None
+            elif self.picked:
+                self.pick(())
             else:
                 return False
-        elif key == "p":
-            self.listing = not self.listing
+        elif key == "p":  # opened or closed by hand, the list stays so when the picks go
+            self.listing, self.opened = not self.listing, False
         elif key == "v":  # the graph, the processes, the stats alone, GPU and MEM alone; V the one before
             self.left = LEFTS[(LEFTS.index(self.left) + (-1 if value[0] == "V" else 1)) % len(LEFTS)]
             name = {"graph": "graphs", "processes": "processes", "numbers": "numbers only",
@@ -1656,16 +1793,32 @@ class View:
         elif key in MOVES and self.jobs:
             order = [job.pid for job in self.jobs]
             i = order.index(self.job) if self.job in order else (-1 if MOVES[key] > 0 else len(order))
-            self.job, self.follow = order[min(max(i + MOVES[key], 0), len(order) - 1)], True
-        elif self.job is not None and key in ("f", "enter"):
+            self.pick([order[min(max(i + MOVES[key], 0), len(order) - 1)]])
+            self.follow = True
+        elif self.picked and key in ("f", "enter"):
             self.only = not self.only
-        elif self.job is not None and key in ("t", "k"):
-            if self.selected().owner is None:
+        elif self.picked and key in ("t", "k"):
+            if any(job.owner is None for job in self.selected()):
                 self.asking = "SIGTERM" if key == "t" else "SIGKILL"
             else:
                 self.note = ([("only your own processes can be stopped", WARN)], time.monotonic() + 3)
         return True
 
+    def stop_picked(self):
+        """Send the signal asked for to each picked job of ours (see stop), and say how that went."""
+        ours = [job for job in self.selected() if job.owner is None]
+        results = [stop(job, self.asking) for job in ours]
+        sent = {job.pid for job, (_, ok) in zip(ours, results) if ok}
+        failed = [note for note, ok in results if not ok]
+        if len(ours) == 1:
+            note = results[0][0]
+        elif not failed:
+            note = [("sent {} to {} jobs".format(self.asking, len(sent)), "")]
+        else:
+            note = [("sent {} to {} of {} jobs; ".format(self.asking, len(sent), len(ours)), WARN)] + failed[0]
+        self.note = (note, time.monotonic() + 4)
+        self.sent = (sent, time.monotonic()) if sent and self.asking == "SIGTERM" else None
+
 
 # For terminals that draw the line, block and other characters above two cells wide, as they may for Korean,
 # Chinese and Japanese (Unicode calls their width "ambiguous"): ASCII look-alikes, one for one.
@@ -1675,26 +1828,21 @@ ASCII = str.maketrans({"─": "-", "│": "|", "├": "+", "┤": "+", "╭": "+
                        "°": " ", "·": "-", "…": "~", "↑": "^", "↓": "v", "▲": "^", "▼": "v"})
 
 
-def paint(lines, mode="truecolor", ascii=False, gray=False):
-    """Join lines into one string in color `mode` (with `gray`, grays only; see restyle), and with `ascii` in
-    ASCII, sending a style code only where the style changes."""
+def paint(line, mode="truecolor", ascii=False, gray=False):
+    """One line as the terminal gets it, from plain to plain: in color `mode` (with `gray`, grays only; see
+    restyle), with `ascii` in ASCII, a style code only where the style changes."""
     out, current = [], ""
-    for i, line in enumerate(lines):
-        if i:
-            out.append("\r\n")
-        for text, style, *_ in line:
-            if ascii:
-                text = text.translate(ASCII)
-            if mode != "truecolor" or gray:
-                style = restyle(style, mode, gray)
-            if style != current:
-                # One color replaces another directly; anything else (bold, a background, plain) needs a
-                # reset first.
-                colors = (style.startswith("\x1b[38") and current.startswith("\x1b[38")
-                          and "\x1b[48" not in current)
-                out.append(style if colors else RESET + style)
-                current = style
-            out.append(text)
+    for text, style, *_ in line:
+        if ascii:
+            text = text.translate(ASCII)
+        if mode != "truecolor" or gray:
+            style = restyle(style, mode, gray)
+        if style != current:
+            # One color replaces another directly; anything else (bold, a background, plain) needs a reset first.
+            colors = style.startswith("\x1b[38") and current.startswith("\x1b[38") and "\x1b[48" not in current
+            out.append(style if colors else RESET + style)
+            current = style
+        out.append(text)
     return "".join(out) + RESET
 
 
@@ -1782,7 +1930,7 @@ INPUT = re.compile(rb"\x1b\[<(?P<button>\d+);(?P<x>\d+);(?P<y>\d+)(?P<act>[Mm])"
 def events(data):
     """The input in `data`, the bytes a terminal sends: ("key", name) events, ("click", row, column) for
     a left-button press, ("wheel", +1 down or -1 up, row, column), ("hwheel", +1 right or -1 left, row,
-    column)."""
+    column), ("move", row, column) where the pointer went."""
     out = []
     for m in INPUT.finditer(data):
         if m.group("button") or m.group("old"):
@@ -1795,7 +1943,9 @@ def events(data):
             if button & 64:  # the wheel: 64 up, 65 down, 66 left, 67 right; with Shift (+4) sideways too
                 sideways = button & 2 or button & 4
                 out.append(("hwheel" if sideways else "wheel", 1 if button & 1 else -1, y - 1, x - 1))
-            elif press and button & 3 == 0 and not button & 32:  # left button down, not a drag
+            elif button & 32:  # a move, a button held or not
+                out.append(("move", y - 1, x - 1))
+            elif press and button & 3 == 0:  # left button down
                 out.append(("click", y - 1, x - 1))
         elif m.group("report"):
             continue
@@ -1830,15 +1980,17 @@ TRUECOLOR_ANSWERS = re.compile(
 
 
 class Screen:
-    """Alternate screen, hidden cursor, no auto-wrap, unbuffered keys, mouse clicks and wheel; all
+    """Alternate screen, hidden cursor, no auto-wrap, unbuffered keys, mouse clicks, wheel and moves; all
     restored on exit."""
 
     def __enter__(self):
         self._keys = sys.stdin.isatty()
         self._mouse = self._keys and os.name != "nt"  # Windows reads keys with msvcrt, which sees no mouse
         self._restore = [enable_vt(), raw_keys() if self._keys else (lambda: None)]
-        # 1000 + 1006: clicks and the wheel come in as SGR-encoded sequences (Shift+drag still selects text).
-        self._write("\x1b[?1049h\x1b[?25l\x1b[?7l" + ("\x1b[?1000h\x1b[?1006h" if self._mouse else ""))
+        self._painted, self._size = [], None  # the lines last sent, and the window they were sent to
+        # 1000 + 1003 + 1006: clicks, the wheel and every move of the pointer come in as SGR-encoded sequences
+        # (Shift+drag still selects text). A terminal without 1003 sends clicks and the wheel all the same.
+        self._write("\x1b[?1049h\x1b[?25l\x1b[?7l" + ("\x1b[?1000h\x1b[?1003h\x1b[?1006h" if self._mouse else ""))
         # Windows Terminal and the console of Windows 10 on show 24-bit color and draw lines one cell wide.
         self.wide, self.truecolor = self.probe() if self._mouse else (False, os.name == "nt")
         return self
@@ -1873,7 +2025,8 @@ class Screen:
         return bool(cursor) and int(cursor.group(1)) > 2, bool(TRUECOLOR_ANSWERS.search(data))
 
     def __exit__(self, *exc):
-        self._write(RESET + ("\x1b[?1006l\x1b[?1000l" if self._mouse else "") + "\x1b[?7h\x1b[?25h\x1b[?1049l")
+        mouse = "\x1b[?1006l\x1b[?1003l\x1b[?1000l" if self._mouse else ""
+        self._write(RESET + mouse + "\x1b[?7h\x1b[?25h\x1b[?1049l")
         for undo in self._restore:
             undo()
 
@@ -1907,9 +2060,16 @@ class Screen:
             self._keys = False
         return events(data)
 
-    def draw(self, lines, mode, gray):
+    def draw(self, lines, mode, gray, size):
+        """The screen `lines`, in a window of `size`: only the lines that changed since the last time, so that
+        a frame, or the pointer moving over a graph, sends a few hundred bytes rather than the whole screen."""
+        painted = [paint(line, mode, self.wide, gray) for line in lines]
+        last = self._painted if self._size == size else []  # a new size: all anew
+        changed = "".join("\x1b[{};1H{}".format(row + 1, text) for row, text in enumerate(painted)
+                          if row >= len(last) or last[row] != text)
+        self._painted, self._size = painted, size
         # 2026 = synchronized output: terminals that know it swap the frame in at once.
-        self._write("\x1b[?2026h\x1b[H" + paint(lines, mode, self.wide, gray) + "\x1b[?2026l")
+        self._write("\x1b[?2026h" + changed + "\x1b[?2026l")
 
     @staticmethod
     def _write(text):
@@ -2009,6 +2169,7 @@ def main():
         host, settings, indices = socket.gethostname(), Settings(), [g.index for g in gpus]
         poller, view = Poller(gpus), View()
         view.restore(settings.load(host), indices)
+        view.interval = args.interval
         kept = view.kept()
         with Screen() as screen:
             view.shows = color_mode(screen.truecolor)
@@ -2042,16 +2203,19 @@ def main():
                     width, height = os.get_terminal_size()
                     screen.draw(view.screen(gpus, width, height,
                                             header(width, args.interval, driver, poller.waiting()), update.newer),
-                                view.shows, view.theme.gray)
-                # Keys and clicks redraw at once; the numbers move on at the next frame.
+                                view.shows, view.theme.gray, (width, height))
+                # Keys and clicks redraw at once, the pointer when it goes to another graph column; the
+                # numbers move on at the next frame.
                 events = screen.read(0.01 if peek is not None else (poll_at or frame_at) - time.monotonic())
+                pointed = view.pointed()
                 going = all(map(view.handle, events))
                 if view.kept() != kept:  # a setting changed: keep it for next time
                     kept = view.kept()
                     settings.save(kept, indices, host)
                 if not going:
                     break
-                redraw, detail = bool(events), view.detail(gpus)
+                redraw = any(e[0] != "move" for e in events) or view.pointed() != pointed
+                detail = view.detail(gpus)
                 if detail - poller.detail:  # at the latest the next round, or the one after the round running
                     busy = not poller.done.is_set()
                     peek = (detail - poller.detail | (peek[0] if peek else set()), poller.rounds + 1 + busy)