nvmon 0.3.2__tar.gz → 0.3.4__tar.gz

{nvmon-0.3.2 → nvmon-0.3.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: nvmon
-Version: 0.3.2
+Version: 0.3.4
 Summary: A fancy NVIDIA GPU monitor for the terminal. One Python file, no dependencies.
 Project-URL: Source, https://github.com/Han-DongHeun/nvmon
 Project-URL: Issues, https://github.com/Han-DongHeun/nvmon/issues
@@ -24,7 +24,7 @@ Description-Content-Type: text/markdown
 
 A fancy NVIDIA GPU monitor for the terminal. One Python file, no dependencies.
 
-![nvmon showing four busy H100 GPUs one under the other, the other four hidden](https://raw.githubusercontent.com/Han-DongHeun/nvmon/main/docs/demo-wide.gif)
+![nvmon showing four H100 GPUs one under the other, two people's jobs on them, the other four hidden, the pointer moving along a graph with the utilization and memory then beside it](https://raw.githubusercontent.com/Han-DongHeun/nvmon/main/docs/demo-wide.gif)
 
 ## Install
 
@@ -50,13 +50,14 @@ ssh -t HOST nvmon  # over ssh: -t gives it a terminal
 
 - **Top edge**: number and name, power against its limit, temperature, fan (if any), clock. Warnings
   only when something is off: `SLOWED: power cap | too hot | hw brake`, `PCIe DEGRADED: x16 -> x8`.
-- **Graph**: utilization over time, newest on the right.
+- **Graph**: utilization over time, newest on the right. Point at it for the utilization and memory at that
+  moment.
 - **GPU**: how much of the time a kernel ran. **cores / tensor** (H100 and newer): how busy that kept
   the chip, and how much was Tensor Core math. One small kernel running nonstop reads GPU 100 % with
   cores at 30 %.
 - **MEM**: memory in use. **CPU -> GPU**, **GPU -> CPU**: PCIe traffic against the link's top speed.
 - **Bottom edge**: the processes, with run time and memory, by account and conda environment (yours
-  unlabelled). `+` opens the process list.
+  underlined); `+3` for those that did not fit.
 
 The bottom line has the GPU numbers (click one, or press its key, to hide or show it), the keys, and a
 newer release when there is one.
@@ -77,21 +78,27 @@ Shift+wheel or ← → sideways.
 
 **Numbers only** and **GPU and MEM only**: narrow, always one column, for a window beside another.
 
-| Numbers only, scrolled down | GPU and MEM only |
-|---|---|
+<div align="center">
+
+| Numbers only | GPU and MEM only |
+|:---:|:---:|
 | <img src="https://raw.githubusercontent.com/Han-DongHeun/nvmon/main/docs/view-numbers.png" alt="nvmon with the numbers alone" width="200"> | <img src="https://raw.githubusercontent.com/Han-DongHeun/nvmon/main/docs/view-brief.png" alt="nvmon with GPU and MEM alone" width="200"> |
 
+</div>
+
 ## Processes and jobs: `p`
 
 A **job** is a process, or a launcher (torchrun, deepspeed…) with its workers.
 
-- **Pick** one: click a process, or press Tab (Tab and the arrows move on). Its GPUs stand out, and the
-  bottom line sums it up.
-- `f`, or a double click: only its GPUs; again for all.
+- **Pick** one: click a process, or press Tab (Tab and the arrows move on). Its GPUs stand out, the
+  list opens with it picked, and the bottom line sums it up.
+- **Pick a GPU's jobs**: click anywhere else in its box. The same again lets go, and closes the list it
+  opened.
+- `f`, or a double click: only the picked jobs' GPUs; again for all.
 - `p`: every job in a list. Click a heading to sort, again to reverse (or `s`, `r`). ← → scroll the
   picked job's command. Esc, or a click on an empty spot, lets go.
-- `t` stops a job of yours (SIGTERM), `k` kills it (SIGKILL): each after a `y`, and only if it is still
-  the job you picked.
+- `t` stops the picked jobs of yours (SIGTERM), `k` kills them (SIGKILL): after a `y`, and only those
+  still the jobs you picked.
 
 The list, busiest first, one job picked; and that job with `f`:
 
@@ -118,9 +125,11 @@ The list, busiest first, one job picked; and that job with `f`:
 | `v` / `V` | next / previous view |
 | `c` / `C` | next / previous theme |
 | `p` | process list, open or close |
+| point at a graph | utilization and memory then |
 | click, Tab, ↑ ↓ | pick a job |
-| `f`, double click | only the picked job's GPUs, or all |
-| `t`, `k`, then `y` | stop (SIGTERM) or kill (SIGKILL) the picked job |
+| click a GPU | pick its jobs |
+| `f`, double click | only the picked jobs' GPUs, or all |
+| `t`, `k`, then `y` | stop (SIGTERM) or kill (SIGKILL) the picked jobs |
 | click a heading, `s`, `r` | sort the list, by the next heading, the other way |
 | wheel, ← → | scroll; sideways the commands |
 | PgUp, PgDn | scroll the GPUs |
@@ -146,4 +155,5 @@ The view, theme, list order and hidden GPUs, for each computer you connect from,
 
 ## License
 
-MIT
+MIT. I designed nvmon's features and usability and tested it on shared H100 servers; most of the code was
+written with an AI coding assistant.

{nvmon-0.3.2 → nvmon-0.3.4}/README.md

@@ -2,7 +2,7 @@
 
 A fancy NVIDIA GPU monitor for the terminal. One Python file, no dependencies.
 
-![nvmon showing four busy H100 GPUs one under the other, the other four hidden](https://raw.githubusercontent.com/Han-DongHeun/nvmon/main/docs/demo-wide.gif)
+![nvmon showing four H100 GPUs one under the other, two people's jobs on them, the other four hidden, the pointer moving along a graph with the utilization and memory then beside it](https://raw.githubusercontent.com/Han-DongHeun/nvmon/main/docs/demo-wide.gif)
 
 ## Install
 
@@ -28,13 +28,14 @@ ssh -t HOST nvmon  # over ssh: -t gives it a terminal
 
 - **Top edge**: number and name, power against its limit, temperature, fan (if any), clock. Warnings
   only when something is off: `SLOWED: power cap | too hot | hw brake`, `PCIe DEGRADED: x16 -> x8`.
-- **Graph**: utilization over time, newest on the right.
+- **Graph**: utilization over time, newest on the right. Point at it for the utilization and memory at that
+  moment.
 - **GPU**: how much of the time a kernel ran. **cores / tensor** (H100 and newer): how busy that kept
   the chip, and how much was Tensor Core math. One small kernel running nonstop reads GPU 100 % with
   cores at 30 %.
 - **MEM**: memory in use. **CPU -> GPU**, **GPU -> CPU**: PCIe traffic against the link's top speed.
 - **Bottom edge**: the processes, with run time and memory, by account and conda environment (yours
-  unlabelled). `+` opens the process list.
+  underlined); `+3` for those that did not fit.
 
 The bottom line has the GPU numbers (click one, or press its key, to hide or show it), the keys, and a
 newer release when there is one.
@@ -55,21 +56,27 @@ Shift+wheel or ← → sideways.
 
 **Numbers only** and **GPU and MEM only**: narrow, always one column, for a window beside another.
 
-| Numbers only, scrolled down | GPU and MEM only |
-|---|---|
+<div align="center">
+
+| Numbers only | GPU and MEM only |
+|:---:|:---:|
 | <img src="https://raw.githubusercontent.com/Han-DongHeun/nvmon/main/docs/view-numbers.png" alt="nvmon with the numbers alone" width="200"> | <img src="https://raw.githubusercontent.com/Han-DongHeun/nvmon/main/docs/view-brief.png" alt="nvmon with GPU and MEM alone" width="200"> |
 
+</div>
+
 ## Processes and jobs: `p`
 
 A **job** is a process, or a launcher (torchrun, deepspeed…) with its workers.
 
-- **Pick** one: click a process, or press Tab (Tab and the arrows move on). Its GPUs stand out, and the
-  bottom line sums it up.
-- `f`, or a double click: only its GPUs; again for all.
+- **Pick** one: click a process, or press Tab (Tab and the arrows move on). Its GPUs stand out, the
+  list opens with it picked, and the bottom line sums it up.
+- **Pick a GPU's jobs**: click anywhere else in its box. The same again lets go, and closes the list it
+  opened.
+- `f`, or a double click: only the picked jobs' GPUs; again for all.
 - `p`: every job in a list. Click a heading to sort, again to reverse (or `s`, `r`). ← → scroll the
   picked job's command. Esc, or a click on an empty spot, lets go.
-- `t` stops a job of yours (SIGTERM), `k` kills it (SIGKILL): each after a `y`, and only if it is still
-  the job you picked.
+- `t` stops the picked jobs of yours (SIGTERM), `k` kills them (SIGKILL): after a `y`, and only those
+  still the jobs you picked.
 
 The list, busiest first, one job picked; and that job with `f`:
 
@@ -96,9 +103,11 @@ The list, busiest first, one job picked; and that job with `f`:
 | `v` / `V` | next / previous view |
 | `c` / `C` | next / previous theme |
 | `p` | process list, open or close |
+| point at a graph | utilization and memory then |
 | click, Tab, ↑ ↓ | pick a job |
-| `f`, double click | only the picked job's GPUs, or all |
-| `t`, `k`, then `y` | stop (SIGTERM) or kill (SIGKILL) the picked job |
+| click a GPU | pick its jobs |
+| `f`, double click | only the picked jobs' GPUs, or all |
+| `t`, `k`, then `y` | stop (SIGTERM) or kill (SIGKILL) the picked jobs |
 | click a heading, `s`, `r` | sort the list, by the next heading, the other way |
 | wheel, ← → | scroll; sideways the commands |
 | PgUp, PgDn | scroll the GPUs |
@@ -124,4 +133,5 @@ The view, theme, list order and hidden GPUs, for each computer you connect from,
 
 ## License
 
-MIT
+MIT. I designed nvmon's features and usability and tested it on shared H100 servers; most of the code was
+written with an AI coding assistant.

{nvmon-0.3.2 → nvmon-0.3.4}/nvmon.py

@@ -29,7 +29,7 @@ from ctypes import byref, c_int, c_uint, c_ulonglong, c_void_p
 from functools import lru_cache
 from typing import NamedTuple, Optional
 
-__version__ = "0.3.2"  # "+dev": work past this release; the release commit sets the next number
+__version__ = "0.3.4"  # "+dev": work past this release; the release commit sets the next number
 REPO = "https://github.com/Han-DongHeun/nvmon"
 RELEASES = REPO + "/releases"                 # release notes, and nvmon.py for those who copy the file
 PYPI_JSON = "https://pypi.org/pypi/nvmon/json"
@@ -238,6 +238,7 @@ SHOWS = "truecolor"  # what the terminal shows (color_mode), set with it
 # Warnings on the top edge, the same in every theme: yellow = worth a look, orange = slowed, red = act.
 WARN, SLOW, ALERT = _rgb(YELLOW), _rgb(ORANGE), _rgb(RED)
 DIM, FAINT = _fg(240), _fg(238)
+UNDERLINE = ""  # first in a style, so that the next one starts with a reset (see paint)
 BOLD, RESET = "\x1b[1m", "\x1b[0m"
 
 
@@ -282,7 +283,6 @@ class ProcessInfo(ctypes.Structure):  # nvmlProcessInfo_v2_t
 
 
 NVML_VALUE_NOT_AVAILABLE = 2**64 - 1  # usedGpuMemory under Windows WDDM
-MAX_PROCESSES = 64
 
 
 class GpmSupport(ctypes.Structure):  # nvmlGpmSupport_t
@@ -349,7 +349,8 @@ class Process(NamedTuple):
     name: str                     # script name for Python, else the executable
     command: str                  # the command line from `name` on, e.g. "train.py --lr 3e-4"
     mem: Optional[float]          # GPU memory, GiB
-    owner: Optional[str]          # account; None for our own processes
+    owner: Optional[str]          # account; None where unknown (Windows)
+    mine: bool                    # ours: its account underlined, and only ours can be stopped
     env: Optional[str]            # conda environment
     started: Optional[float]      # start time, seconds since the epoch
     job: int                      # the job it is part of: its launcher's PID (torchrun...), else its own
@@ -388,7 +389,8 @@ class Gpu:
         nv.nvmlDeviceGetName(handle, name, NVML_DEVICE_NAME_BUFFER_SIZE)
         name = name.value.decode("utf-8", "replace")
         self.name = name[len("NVIDIA "):] if name.startswith("NVIDIA ") else name
-        self.history = deque(maxlen=HISTORY)
+        self.history = deque(maxlen=HISTORY)  # utilization, one a frame: the graph
+        self.memory = deque(maxlen=HISTORY)   # and the memory in use then (GiB), for the chip (View.hover)
         self.sample = None  # the newest reading, written by poll()
         self.now = None     # the reading on screen, taken from `sample` by frame()
         self.pcie_max_gen = self._uint("nvmlDeviceGetMaxPcieLinkGeneration")
@@ -398,7 +400,8 @@ class Gpu:
         self._facts = {}  # pid -> what _facts_of says: fixed for a process's life, so read once
         self._listed, self._listed_mem = [], None  # process list, and the memory in use when it was read
         # Per-process utilization: the driver's samples since `seen` (µs), read at most every second.
-        self._util, self._util_due, self._util_seen, self._util_buf = None, 0, 0, (ProcessUtil * 128)()
+        self._util, self._util_due, self._util_seen = None, 0, 0
+        self._sizes = {}  # entry type -> how many the last array of them held, see _entries
 
     def _call(self, fn, *args):
         """Run the device query `fn`; its NVML return code. A query this GPU or driver cannot answer is not
@@ -475,21 +478,35 @@ class Gpu:
             return None
         return next(((label, color) for bits, label, color in SLOWDOWNS if reasons.value & bits), None)
 
+    def _entries(self, kind, query):
+        """(return code, entries) of `query(array, count)`, a query that fills an array of `kind` and sets
+        `count`. The array is made bigger while the driver says it is too small (more may come between its
+        answer and asking again), and starts that big the next time."""
+        size = self._sizes.get(kind, 64)
+        for _ in range(3):
+            array, count = (kind * size)(), c_uint(size)
+            rc = query(array, count)
+            if rc != NVML_ERROR_INSUFFICIENT_SIZE:
+                break
+            size = 2 * max(count.value, size)
+        self._sizes[kind] = size
+        return rc, array[:count.value] if rc == NVML_SUCCESS else []
+
     def _processes(self):
-        infos, count = (ProcessInfo * MAX_PROCESSES)(), c_uint(MAX_PROCESSES)
-        if not self._ok("nvmlDeviceGetComputeRunningProcesses_v3", byref(count), infos):
+        rc, running = self._entries(ProcessInfo, lambda array, count: self._call(
+            "nvmlDeviceGetComputeRunningProcesses_v3", byref(count), array))
+        if rc != NVML_SUCCESS:
             return []
-        running = infos[:count.value]
         self._facts = {p.pid: self._facts.get(p.pid) or self._facts_of(p.pid) for p in running}
         procs = []
         for p in running:
             mem = None if p.usedGpuMemory == NVML_VALUE_NOT_AVAILABLE else p.usedGpuMemory / 2**30
-            name, command, owner, env, started, job, launcher = self._facts[p.pid]
-            procs.append(Process(p.pid, name, command, mem, owner, env, started, job, launcher))
+            name, command, owner, mine, env, started, job, launcher = self._facts[p.pid]
+            procs.append(Process(p.pid, name, command, mem, owner, mine, env, started, job, launcher))
         return procs
 
     def _facts_of(self, pid):
-        """(name, command, owner, conda environment, start time, job, launcher name) of `pid`."""
+        """(name, command, owner, ours, conda environment, start time, job, launcher name) of `pid`."""
         launcher = launcher_of(pid)
         return (self._command(pid) + process_facts(pid)
                 + ((launcher, self._command(launcher)[0]) if launcher else (pid, None)))
@@ -520,18 +537,12 @@ class Gpu:
     def _process_util(self):
         """{pid: % of the time its kernels ran} since the last call; None where the GPU cannot tell
         (with MIG, say). The driver takes some 2 ms to answer, so this is asked only while it is shown."""
-        for _ in range(3):
-            count = c_uint(len(self._util_buf))
-            rc = self._call("nvmlDeviceGetProcessUtilization", self._util_buf, byref(count),
-                            c_ulonglong(self._util_seen))
-            if rc != NVML_ERROR_INSUFFICIENT_SIZE:
-                break
-            self._util_buf = (ProcessUtil * (2 * count.value))()
+        rc, samples = self._entries(ProcessUtil, lambda array, count: self._call(
+            "nvmlDeviceGetProcessUtilization", array, byref(count), c_ulonglong(self._util_seen)))
         if rc == NVML_ERROR_NOT_FOUND:  # no new samples: nothing ran
             return {}
         if rc != NVML_SUCCESS:
             return None
-        samples = self._util_buf[:count.value]
         self._util_seen = max([self._util_seen] + [s.timeStamp for s in samples])
         runs = {}
         for s in samples:
@@ -578,6 +589,7 @@ class Gpu:
         """Put the newest reading on screen for the next frame; the graph gains one column per frame."""
         self.now = self.sample
         self.history.append(self.now.util or 0)
+        self.memory.append(self.now.mem_used)
 
 
 class Poller(threading.Thread):
@@ -633,8 +645,10 @@ def versions(nv):
 
 
 def process_facts(pid):
-    """(owner, conda environment, start time) of `pid` from /proc; Nones where unknown (e.g. Windows)."""
-    return process_owner(pid), conda_env(pid), start_time(pid)
+    """(owner, whether it is ours, conda environment, start time) of `pid` from /proc; Nones, and not ours,
+    where unknown (e.g. Windows)."""
+    uid = uid_of(pid)
+    return account_name(uid), uid is not None and uid == os.getuid(), conda_env(pid), start_time(pid)
 
 
 def conda_env(pid):
@@ -666,10 +680,9 @@ def start_time(pid):
     return boot + ticks / os.sysconf("SC_CLK_TCK")
 
 
-def process_owner(pid):
-    """Account running `pid`; None for our own processes or where there is no /proc (Windows)."""
-    uid = uid_of(pid)
-    if uid is None or uid == os.getuid():
+def account_name(uid):
+    """The name of the account `uid`; None for None."""
+    if uid is None:
         return None
     import pwd
     try:
@@ -757,6 +770,7 @@ class Job(NamedTuple):
     mem: float                    # GiB, on all of them
     started: Optional[float]
     owner: Optional[str]
+    mine: bool
     env: Optional[str]
     command: str                  # its biggest process's command line
 
@@ -780,7 +794,7 @@ def jobs(gpus):
         found.append(Job(pid, main.name, main.launcher, tuple(sorted({(p.pid, p.started) for p in procs})),
                          tuple(sorted({gpu.index for gpu, _ in entries})),
                          sum(shares.values()) / len(shares) if shares else None, sum(p.mem or 0 for p in procs),
-                         min(starts) if starts else None, main.owner, main.env, main.command))
+                         min(starts) if starts else None, main.owner, main.mine, main.env, main.command))
     return sorted(found, key=lambda job: (job.gpus, job.pid))
 
 
@@ -795,16 +809,29 @@ def ranges(numbers):
     return ",".join(str(a) if a == b else "{}-{}".format(a, b) for a, b in spans)
 
 
+def gpu_numbers(indices):
+    """"GPU 3", "GPUs 4-5,7"."""
+    indices = sorted(set(indices))
+    return "GPU{} {}".format("s" if len(indices) > 1 else "", ranges(indices))
+
+
 def account(job):
-    """"kim/torch": whose it is, and the conda environment; our own go without an account."""
+    """"kim/torch": whose it is, and the conda environment."""
     return "/".join(part for part in (job.owner, job.env) if part)
 
 
+def tagged(who):
+    """"kim/torch", whose a process or job (`who`) is and its conda environment, as segments: dim, and ours
+    underlined, so that they stand apart without standing out."""
+    text = "/".join(part for part in (who.owner, who.env) if part)
+    return [(text, UNDERLINE + DIM if who.mine else DIM)] if text else []
+
+
 def about(job):
     """"kim/torch · GPUs 4-5 · util 86% · 30.2G · 2h13m · torchrun job" as segments, util in its color;
     the launcher, often a long name, comes last, where a narrow line cuts first."""
-    parts = [(account(job), DIM)] if account(job) else []
-    parts.append(("GPU{} {}".format("s" if len(job.gpus) > 1 else "", ranges(job.gpus)), DIM))
+    parts = [sum(tagged(job), ())] if account(job) else []
+    parts.append((gpu_numbers(job.gpus), DIM))
     if job.util is not None:
         parts.append(("util ", DIM, "{:.0f}%".format(job.util), heat(job.util / 100)))
     parts.append(("{:.1f}G".format(job.mem), DIM))
@@ -817,6 +844,14 @@ def about(job):
     return out
 
 
+def about_all(jobs):
+    """"GPUs 4-7 · 61.2G · 2 yours" for several jobs picked together, as segments."""
+    parts = [gpu_numbers(index for job in jobs for index in job.gpus), "{:.1f}G".format(sum(j.mem for j in jobs))]
+    ours = sum(job.mine for job in jobs)
+    parts += ["{} yours".format(ours)] if 0 < ours < len(jobs) else []
+    return [(" · ".join(parts), DIM)]
+
+
 def stop(job, name):
     """Send signal `name` ("SIGTERM" or "SIGKILL") to `job`, once /proc confirms its PIDs still are the
     processes we saw: started when we saw them, holding a GPU open (inside a container, NVML can report
@@ -986,6 +1021,31 @@ def graph(history, width, height, two_tone=True):
     return rows
 
 
+# The pointer over a graph (View.hover): a guide up the empty part of its column, and a chip with that moment.
+GUIDE = _rgb((150, 150, 150))
+CHIP = "\x1b[48;2;235;235;235m" + _rgb((24, 24, 27))
+COLOR_24 = re.compile(r"(\x1b\[[34]8;2;)(\d+);(\d+);(\d+)m")
+
+
+def lit(style):
+    """`style` lit up, for the graph column under the pointer: its colors halfway to white; on PAPER, where
+    restyle turns lightness around, halfway to black."""
+    paper = ON_PAPER if style.startswith(ON_PAPER) else ""
+    to = 0 if paper else 255
+    return paper + COLOR_24.sub(lambda m: m.group(1) + ";".join(
+        str((int(v) + to) // 2) for v in m.group(2, 3, 4)) + "m", style[len(paper):])
+
+
+def ago(seconds):
+    """How long ago, as short as a run time: now, 14.5s ago, 3m05s ago, 1h02m ago."""
+    if seconds < 0.05:
+        return "now"
+    if seconds < 60:
+        return "{:g}s ago".format(round(seconds, 1))
+    m, s = divmod(int(round(seconds)), 60)
+    return "{}m{:02d}s ago".format(m, s) if m < 60 else "{}h{:02d}m ago".format(m // 60, m % 60)
+
+
 def row(left, right=()):
     """One stats line: `left`, then the `right` segments flush right; exactly INFO_W cells."""
     room = INFO_W - width_of(right)
@@ -1034,12 +1094,12 @@ def degraded(gpu):
     return bool(s.pcie_width and gpu.pcie_max_width and s.pcie_width < gpu.pcie_max_width)
 
 
-def panel(gpu, width, height, selected=None, left="graph", scroll=(0, 0)):
+def panel(gpu, width, height, selected=frozenset(), left="graph", scroll=(0, 0)):
     """One GPU's box: beside the stats, as `left` says (see LEFTS), the utilization graph, or the GPU's
     processes one a line, moved `scroll` = (lines, cells) on (see process_lines); or the stats alone. Faded
-    when a job is `selected` and it does not run here."""
+    when jobs are `selected` (their PIDs) and none runs here."""
     s = gpu.now
-    faded = selected is not None and not any(p.job == selected for p in s.processes)
+    faded = bool(selected) and not any(p.job in selected for p in s.processes)
     # Both sides as (segments, rank); when space runs out the lowest rank goes first:
     # fan, name, power, clock, temperature, PCIe warning, slowdown warning. The GPU number stays.
     # Fixed widths keep things from shifting.
@@ -1053,20 +1113,20 @@ def panel(gpu, width, height, selected=None, left="graph", scroll=(0, 0)):
     parts += [] if s.fan is None else [([("FAN {:>3}%".format(s.fan), "")], 1)]
     parts += [] if s.clock is None else [([("{:>4} MHz".format(s.clock), s.slowdown[1] if s.slowdown else "")], 4)]
     top = edge(width, label, parts)
-    if left in CARDS:  # the numbers only: of the processes, just the way to the list
+    if left in CARDS:  # the numbers only, no processes
         split = width - 1
         body = [[("├" + "─" * (width - 2) + "┤", DIM)] if i is None else [("│ ", DIM)] + i + [(" │", DIM)]
                 for i in info(gpu, height)]
-        label = [("+", DIM, ("list", gpu.index))]
+        label = []
     else:
         graph_w = max(0, width - CHROME_W - INFO_W)
         split = 2 + graph_w + 1  # column of the graph | stats divider
         rule = [(" ├" + "─" * (INFO_W + 2) + "┤", DIM)]  # across the stats, joined to the borders
         if left == "graph":
             beside = graph(gpu.history, graph_w, height, not faded)
-            label = process_label(s.processes, split - 4, selected, gpu.index)
+            label = process_label(s.processes, split - 4, selected)
         else:
-            beside, label = process_lines(s.processes, graph_w, height, selected, gpu.index, *scroll)
+            beside, label = process_lines(s.processes, graph_w, height, selected, *scroll)
         body = [[("│ ", DIM)] + g + (rule if i is None else [(" │ ", DIM)] + i + [(" │", DIM)])
                 for g, i in zip(beside, info(gpu, height))]
     lines = [top] + body + [bottom_edge(width, split, label)]
@@ -1115,7 +1175,7 @@ def elapsed(started):
 
 def process_entries(processes):
     """Processes grouped by owner and conda environment, groups and processes by memory, biggest first, as
-    (tag, process): the tag "kim/torch", "" for our own processes."""
+    (tag, process): the tag "kim/torch", "" where neither is known."""
     def mem(process):
         return process.mem or 0
     groups = {}
@@ -1126,9 +1186,9 @@ def process_entries(processes):
             for process in group]
 
 
-def process_name(process, selected=None):
-    """The segment of `process`'s name: clicked, it means its job; the `selected` job's is reversed."""
-    return process.name, THEME.selected if process.job == selected else THEME.accent, process.job
+def process_name(process, selected=frozenset()):
+    """The segment of `process`'s name: clicked, it means its job; those of `selected` jobs reversed."""
+    return process.name, THEME.selected if process.job in selected else THEME.accent, process.job
 
 
 def run_time(process):
@@ -1139,45 +1199,45 @@ def memory(process):
     return "-" if process.mem is None else "{:.1f}G".format(process.mem)
 
 
-def process_label(processes, room, selected=None, index=None):
+def process_label(processes, room, selected=frozenset()):
     """The processes (process_entries) on one line: "migi: train.py 2h13m 15.1G · eval.py 5m 0.5G   kim/torch:
-    a.py 1d4h 9.0G". As many whole entries as fit in `room` cells, then "+N" for the rest, or "+" when none
-    is left out: clicked, it opens the process list at GPU `index`."""
+    a.py 1d4h 9.0G". As many whole entries as fit in `room` cells, then "+N" for the rest."""
     entries, last = [], None
     for tag, process in process_entries(processes):
         gap = [] if not entries else [(" · ", DIM)] if tag == last else [("   ", "")]
         details = " ".join(part for part in (run_time(process), memory(process)) if part != "-")
-        entries.append(gap + ([(tag + ": ", DIM)] if tag and tag != last else []) + [process_name(process, selected)]
-                       + ([(" " + details, DIM, process.job)] if details else []))
+        entries.append(gap + (tagged(process) + [(": ", DIM)] if tag and tag != last else [])
+                       + [process_name(process, selected)] + ([(" " + details, DIM, process.job)] if details else []))
         last = tag
 
     def more(n):
-        return ("  +{}".format(n) if n else "  +", DIM, ("list", index))
+        return [("  +{}".format(n), DIM)] if n else []
     label = []
     for i, entry in enumerate(entries):
         rest = len(entries) - i - 1
-        if width_of(label + entry) + len(more(rest)[0]) > room:
+        if width_of(label + entry + more(rest)) > room:
             # The first entry is always shown, cut if need be; later ones collapse into the count.
-            return clip(entry, room - len(more(rest)[0])) + [more(rest)] if not label else label + [more(rest + 1)]
+            return clip(entry, room - width_of(more(rest))) + more(rest) if not label else label + more(rest + 1)
         label += entry
-    return label + [more(0)] if label else []
+    return label
 
 
-def process_lines(processes, room, height, selected=None, index=None, top=0, shift=0):
+def process_lines(processes, room, height, selected=frozenset(), top=0, shift=0):
     """The processes (process_listing) one a line, `height` lines of `room` cells, from
     line `top` on and `shift` cells in (see listing_extent), and the label for the box's bottom edge: which
-    lines show of how many, when not all do, and "+", which clicked opens the process list at GPU `index`."""
+    lines show of how many, when not all do."""
     lines = process_listing(processes, selected)
-    # Sideways the command moves; account, run time and memory, the first segment, stay.
-    shown = [pad(clip(line[:1] + skip(line[1:], shift), room), room) for line in lines[top:top + height]]
-    label = [("{}-{} of {}  ".format(top + 1, top + len(shown), len(lines)), DIM)] if len(lines) > height else []
-    return shown + [[(" " * room, "")]] * (height - len(shown)), label + [("+", DIM, ("list", index))]
+    # Sideways the command moves; account, run time and memory stay.
+    shown = [pad(clip(fixed + skip(moving, shift), room), room) for fixed, moving in lines[top:top + height]]
+    label = [("{}-{} of {}".format(top + 1, top + len(shown), len(lines)), DIM)] if len(lines) > height else []
+    return shown + [[(" " * room, "")]] * (height - len(shown)), label
 
 
-def process_listing(processes, selected=None):
-    """process_lines's lines, whole: account and environment, run time and memory in columns (one segment),
-    then the command line, the name in the process names' color and the arguments dim: "kim/torch:  2d5h
-    26.1G  train.py --lr 3". The columns are as wide as the GPU's longest."""
+def process_listing(processes, selected=frozenset()):
+    """process_lines's lines, whole, as (what stays, what moves sideways): account and environment (see
+    tagged), run time and memory in columns; then the command line, the name in the process names' color and
+    the arguments dim: "kim/torch:  2d5h  26.1G  train.py --lr 3". The columns are as wide as the GPU's
+    longest."""
     entries = process_entries(processes)
     tag_w = max([0] + [len(tag) + 1 for tag, _ in entries])  # the longest "kim/torch:", when any has one
     time_w = max([0] + [len(run_time(process)) for _, process in entries])
@@ -1186,16 +1246,18 @@ def process_listing(processes, selected=None):
     for tag, process in entries:
         command = process.command
         rest = command[len(process.name):] if command.startswith(process.name) else " " + command
-        columns = ("{:<{}}  ".format(tag + ":" if tag else "", tag_w) if tag_w else "") + "{:>{}}  {:>{}}  ".format(
+        who = tagged(process) + [(":", DIM)] if tag else []
+        numbers = " " * (tag_w + 2 - width_of(who) if tag_w else 0) + "{:>{}}  {:>{}}  ".format(
             run_time(process), time_w, memory(process), mem_w)
-        lines.append([(columns, DIM, process.job), process_name(process, selected), (rest, DIM, process.job)])
+        lines.append(([seg[:2] + (process.job,) for seg in who] + [(numbers, DIM, process.job)],
+                      [process_name(process, selected), (rest, DIM, process.job)]))
     return lines
 
 
 def listing_extent(processes, room, height):
     """How far process_lines can move on for these processes: (lines, cells)."""
     lines = process_listing(processes)
-    return max(0, len(lines) - height), max([0] + [width_of(line) - room for line in lines])
+    return max(0, len(lines) - height), max([0] + [width_of(fixed + moving) - room for fixed, moving in lines])
 
 
 def skip(line, cells):
@@ -1240,7 +1302,7 @@ def layout(count, width, height, left="graph"):
     return cols, max(MIN_INNER_H, min(MAX_INNER_H, height // rows - 2))
 
 
-def render(gpus, width, height, selected=None, shape=None, left="graph", scroll=None):
+def render(gpus, width, height, selected=frozenset(), shape=None, left="graph", scroll=None):
     """The GPU boxes, in `shape` = (columns, inner height) or else the layout that fits: exactly `height`
     lines of exactly `width` cells. `left`: see panel; the processes moved on as `scroll` has it for each
     GPU (its index: (lines, cells))."""
@@ -1256,57 +1318,60 @@ def render(gpus, width, height, selected=None, shape=None, left="graph", scroll=
     return lines + [[(" " * width, "")]] * (height - len(lines))
 
 
-# The process list's columns: heading, width (">" when flush right), the cell for a job as (text, style),
-# and the key it sorts by. "command" takes whatever width is left.
+# The process list's columns: heading, width (">" when flush right), the cell for a job as segments, and the
+# key it sorts by. "command" takes whatever width is left.
 COLUMNS = [
-    ("PID", 8, lambda j: (str(j.pid), ""), lambda j: j.pid),
-    ("account/env", 14, lambda j: (account(j), ""), account),
-    ("process", 22, lambda j: (j.name + ("  {} workers".format(len(j.members)) if len(j.members) > 1 else ""),
-                               THEME.accent), lambda j: j.name.lower()),
-    ("GPUs", 8, lambda j: (ranges(j.gpus), ""), lambda j: j.gpus),
-    (">util", 5, lambda j: share(j.util, 100)[0] if j.util is not None else ("-", ""),
+    ("PID", 8, lambda j: [(str(j.pid), "")], lambda j: j.pid),
+    ("account/env", 14, lambda j: tagged(j), account),
+    ("process", 22, lambda j: [(j.name + ("  {} workers".format(len(j.members)) if len(j.members) > 1 else ""),
+                                THEME.accent)], lambda j: j.name.lower()),
+    ("GPUs", 8, lambda j: [(ranges(j.gpus), "")], lambda j: j.gpus),
+    (">util", 5, lambda j: share(j.util, 100) if j.util is not None else [("-", "")],
      lambda j: -1 if j.util is None else j.util),
-    (">memory", 7, lambda j: ("{:.1f}G".format(j.mem), ""), lambda j: j.mem),
-    (">time", 7, lambda j: (elapsed(j.started) if j.started is not None else "-", ""),
+    (">memory", 7, lambda j: [("{:.1f}G".format(j.mem), "")], lambda j: j.mem),
+    (">time", 7, lambda j: [(elapsed(j.started) if j.started is not None else "-", "")],
      lambda j: -(j.started or 0)),
-    ("command", 0, lambda j: (j.command, DIM), lambda j: j.command),
+    ("command", 0, lambda j: [(j.command, DIM)], lambda j: j.command),
 ]
 SORTS = {heading.lstrip(">"): key for heading, _, _, key in COLUMNS}
 COMMAND_AT = 1 + sum(size + 2 for _, size, _, _ in COLUMNS[:-1])  # the column where commands start
 
 
 def cells(values, width, fill=("",)):
-    """One list line of `width` cells from (text, style, ...) values, one per column, two spaces apart;
-    `fill` = (style, ...) of the spaces between, so a selected row stays one bar and clicks anywhere."""
+    """One list line of `width` cells from the columns' segments, two spaces apart; `fill` = (style, ...) of
+    the spaces, so that a selected row stays one bar and clicks anywhere: those padding a column mean what
+    its last segment does, but take nothing else of it, an underline, say."""
     line, used = [(" ",) + fill], 1
-    for (heading, size, _, _), (text, *rest) in zip(COLUMNS, values):
+    for (heading, size, _, _), segments in zip(COLUMNS, values):
         size = size or max(0, width - used - 1)
-        text = text[:size].rjust(size) if heading.startswith(">") else text[:size].ljust(size)
-        line += [(text,) + tuple(rest), (" " if heading == "command" else "  ",) + fill]
+        segments = clip(segments, size)
+        padding = [(" " * (size - width_of(segments)),) + fill[:1] + (segments[-1][2:3] if segments else fill[1:])]
+        line += padding + segments if heading.startswith(">") else segments + padding
+        line.append((" " if heading == "command" else "  ",) + fill)
         used += size + 2
     return line
 
 
-def process_list(jobs, width, rows, selected, top, sort, shift=0):
+def process_list(jobs, width, rows, selected, focus, top, sort, shift=0):
     """The list p opens, `rows` lines: the headings (clicking one sorts by it; `sort` = (column, descending)),
-    the jobs from index `top` on (clicking one means it; the `selected` one reversed, its command moved
-    `shift` characters on), how many more. A command too long to show ends in "…"."""
+    the jobs from index `top` on (clicking one means it; the `selected` ones reversed, the command of the one
+    in `focus` moved `shift` characters on), how many more. A command too long to show ends in "…"."""
     room = max(0, width - COMMAND_AT - 1)
     column, descending = sort
     head = []
     for heading, _, _, _ in COLUMNS:
         name = heading.lstrip(">")
-        head.append((name + ("▼" if descending else "▲") if name == column else name,
-                     "" if name == column else DIM, ("sort", name)))
+        head.append([(name + ("▼" if descending else "▲") if name == column else name,
+                      "" if name == column else DIM, ("sort", name))])
     lines = [cells(head, width)]
     shown = jobs[top:top + rows - 2]
     for job in shown:
-        chosen = job.pid == selected
+        chosen = job.pid in selected
         values = [cell(job) for _, _, cell, _ in COLUMNS]
-        command = "…" + job.command[shift:] if chosen and shift else job.command
-        values[-1] = (command if len(command) <= room else command[:max(0, room - 1)] + "…", DIM)
-        lines.append(cells([(text, THEME.selected if chosen else color, job.pid) for text, color in values],
-                           width, (THEME.selected if chosen else "", job.pid)))
+        command = "…" + job.command[shift:] if job.pid == focus and shift else job.command
+        values[-1] = [(command if len(command) <= room else command[:max(0, room - 1)] + "…", DIM)]
+        lines.append(cells([[(seg[0], THEME.selected if chosen else seg[1], job.pid) for seg in column]
+                            for column in values], width, (THEME.selected if chosen else "", job.pid)))
     if not jobs:
         lines.append([(" no processes on these GPUs", DIM)])
     above, below = top, len(jobs) - top - len(shown)
@@ -1395,22 +1460,24 @@ MOVES = {"tab": 1, "down": 1, "up": -1, "backtab": -1, "pgdn": 10, "pgup": -10,
 
 
 class View:
-    """What the screen shows around the numbers, and what keys and clicks do to it: the selected job,
-    only its GPUs, the process list, a question before stopping, a note on how that went."""
+    """What the screen shows around the numbers, and what keys and clicks do to it: the picked jobs (one, or
+    a GPU's), only their GPUs, the process list, a question before stopping, a note on how that went."""
 
     def __init__(self):
         self.shows = "truecolor"          # the colors the terminal shows, see color_mode
         self.theme = THEMES[0]            # the look, see THEMES (c)
         self.left = "graph"               # what the boxes have beside the stats, see LEFTS (v)
         self.hidden, self.indices = set(), []  # the GPUs left out, and all there are
-        self.job = None                   # the selected job's PID
-        self.only = False                 # show only the GPUs it uses
+        self.picked = set()               # the picked jobs' PIDs
+        self.job = None                   # of them the one in focus: the arrows go on from it, ← → move its command
+        self.only = False                 # show only the GPUs they use
         self.listing = False              # the process list is open
-        self.top, self.follow = 0, True   # the list's first job; keep the selected one in view
+        self.opened = False               # by a pick, and so to close when the picks are let go
+        self.top, self.follow = 0, True   # the list's first job; keep the one in focus in view
         self.sort = ("GPUs", False)       # the list's order, and the order Tab and the arrows go in
         self.asking = None                # "SIGTERM" or "SIGKILL" awaiting y or n
         self.note = None                  # (segments, until): what the last stop did
-        self.sent = None                  # (PID, when) of a SIGTERM, to point at k if the job lives on
+        self.sent = None                  # (PIDs, when) of a SIGTERM, to point at k if a job lives on
         self.lines, self.jobs = [], []    # the last screen drawn, and the jobs on it
         self.gpu_top, self.gpu_page = 0, 1  # the first row of boxes shown, and how many rows fit
         self.list_rows = range(0)         # the screen rows the list takes, when it is open
@@ -1418,6 +1485,10 @@ class View:
         self.clicked = (0, None)          # (when, what): the last click, to tell a double click
         self.scroll = {}                  # GPU index: (lines, cells) its processes are moved on
         self.areas = []                   # (rows, columns, GPU) where those processes are on screen
+        self.graphs = []                  # (rows, columns, GPU) where the graphs are on screen
+        self.panels = []                  # (rows, columns, GPU index) where the boxes are on screen
+        self.pointer = None               # (row, column) of the mouse pointer, as the terminal last said
+        self.interval = DEFAULT_INTERVAL  # seconds from one graph column to the next
 
     def restore(self, saved, indices):
         """Take up what an earlier run kept (Settings.load); of the GPUs `indices`, one always stays shown."""
@@ -1431,8 +1502,19 @@ class View:
         """What is kept for the next run: see Settings."""
         return {"left": self.left, "theme": self.theme.name, "sort": self.sort, "hidden": set(self.hidden)}
 
+    def pick(self, pids):
+        """Pick the jobs `pids`, in the list's order, the first in focus; none lets all go, and closes the
+        list if a pick opened it."""
+        self.picked = set(pids)
+        self.job = next((job.pid for job in self.jobs if job.pid in self.picked), None)
+        if not self.picked:
+            self.only = False
+            self.listing = self.listing and not self.opened
+            self.opened = False
+
     def selected(self):
-        return next((job for job in self.jobs if job.pid == self.job), None)
+        """The picked jobs, in the list's order."""
+        return [job for job in self.jobs if job.pid in self.picked]
 
     def screen(self, gpus, width, height, top_line, newer):
         """The whole screen: `top_line`, the GPU boxes (and the list, when open), the bottom line."""
@@ -1440,12 +1522,17 @@ class View:
         THEME, SHOWS = self.theme, self.shows
         column, descending = self.sort
         self.jobs = sorted(jobs(gpus), key=lambda job: (SORTS[column](job), job.pid), reverse=descending)
-        job = self.selected()
-        if job is None:  # nothing selected, or its job ended
+        self.picked &= {job.pid for job in self.jobs}  # those that ended go
+        if not self.picked:
             self.job, self.only, self.asking = None, False, None
+        elif self.job not in self.picked:
+            self.job = next(job.pid for job in self.jobs if job.pid in self.picked)
+        picked = self.selected()
+        focus = next(job for job in picked if job.pid == self.job) if picked else None
         self.remind()
         self.indices = [g.index for g in gpus]
-        shown = [g for g in gpus if (g.index in job.gpus if self.only else g.index not in self.hidden)]
+        used = {index for job in picked for index in job.gpus}
+        shown = [g for g in gpus if (g.index in used if self.only else g.index not in self.hidden)]
         rows = min(len(self.jobs) + 2, max(4, (height - 2) // 2)) if self.listing else 0
         body = self.boxes(shown, width, height - 2 - rows)
         self.list_rows = range(1 + len(body), 1 + len(body) + rows)
@@ -1453,9 +1540,10 @@ class View:
             if self.shifted != self.job:
                 self.shift, self.shifted = 0, self.job
             room = max(0, width - COMMAND_AT - 1)
-            self.shift = max(0, min(self.shift, len(job.command) + 1 - room)) if job else 0
-            body += process_list(self.jobs, width, rows, self.job, self.scrolled(rows - 2), self.sort, self.shift)
-        self.lines = [top_line] + body + [self.bottom_line(width, job, newer, gpus)]
+            self.shift = max(0, min(self.shift, len(focus.command) + 1 - room)) if focus else 0
+            body += process_list(self.jobs, width, rows, self.picked, self.job, self.scrolled(rows - 2), self.sort,
+                                 self.shift)
+        self.lines = self.hover([top_line] + body + [self.bottom_line(width, picked, newer, gpus)])
         return self.lines
 
     def boxes(self, gpus, width, height):
@@ -1474,34 +1562,87 @@ class View:
             height -= 1
         else:
             self.gpu_top, self.gpu_page = 0, rows
-        box_w, room = width // cols, max(0, width // cols - CHROME_W - INFO_W)
-        self.areas = []
-        for k, g in enumerate(gpus if self.left == "processes" else []):  # their offsets in range, where they are
-            most = listing_extent(g.now.processes, room, inner)
+        box_w = CARD_W if self.left in CARDS else width // cols
+        room = max(0, box_w - CHROME_W - INFO_W)
+        self.areas, self.graphs = [], []
+        self.panels = [(range(1 + k // cols * (inner + 2), 1 + (k // cols + 1) * (inner + 2)),
+                        range(k % cols * box_w, (k % cols + 1) * box_w), g.index) for k, g in enumerate(gpus)]
+        for k, g in enumerate(gpus if self.left in ("graph", "processes") else []):
+            y, x = 2 + k // cols * (inner + 2), k % cols * box_w + 2  # 2: the top line and the box's edge
+            if self.left == "graph":
+                self.graphs.append((range(y, y + inner), range(x, x + room), g))
+                continue
+            most = listing_extent(g.now.processes, room, inner)  # the processes' offsets in range
             top, shift = self.scroll.get(g.index, (0, 0))
             self.scroll[g.index] = min(max(top, 0), most[0]), min(max(shift, 0), most[1])
-            y, x = 2 + k // cols * (inner + 2), k % cols * box_w + 2  # 2: the top line and the box's edge
             self.areas.append((range(y, y + inner), range(x, x + room), g.index))
-        return render(gpus, width, height, self.job, (cols, inner), self.left, self.scroll) + note
+        return render(gpus, width, height, self.picked, (cols, inner), self.left, self.scroll) + note
+
+    def hovered(self):
+        """(GPU, graph rows, graph columns, the column pointed at, how many samples back from the newest that
+        is) when the pointer is over a graph column that has a sample; else None."""
+        if self.pointer is None:
+            return None
+        row, col = self.pointer
+        for rows, cols, gpu in self.graphs:
+            if row in rows and col in cols:
+                back = cols.stop - 1 - col  # the newest is on the right
+                if back < len(gpu.history):
+                    return gpu, rows, cols, col, back
+        return None
+
+    def pointed(self):
+        """(GPU index, column) of the graph column under the pointer, or None: what the chip is about."""
+        hovered = self.hovered()
+        return hovered and (hovered[0].index, hovered[3])
+
+    def hover(self, lines):
+        """`lines` with the column under the pointer lit up, and beside the pointer a chip with that moment:
+        the utilization, the memory in use, and how long ago."""
+        hovered = self.hovered()
+        if hovered is None:
+            return lines
+        gpu, rows, cols, col, back = hovered
+        mem = gpu.memory[-1 - back]
+        parts = ["{}%".format(gpu.history[-1 - back]), None if mem is None else "{:.1f}G".format(mem),
+                 ago(back * self.interval)]
+        # As much as fits on the pointer's right, or else its left: all, without the memory, the utilization alone.
+        chips = [" {} ".format(" · ".join(filter(None, shown))) for shown in (parts, parts[::2], parts[:1])]
+        place = next(((chip, start) for chip in chips for start in (col + 2, col - 1 - len(chip))
+                      if cols.start <= start and start + len(chip) <= cols.stop), None)
+        lines = list(lines)
+        for row in rows:
+            cells = [(ch,) + tuple(seg[1:]) for seg in lines[row] for ch in seg[0]]
+            ch, style = cells[col][:2]
+            paper = ON_PAPER if style.startswith(ON_PAPER) else ""
+            cells[col] = ("│", paper + GUIDE) if ch == " " else (ch, lit(style))
+            if row == self.pointer[0] and place:
+                chip, start = place
+                bold = chip.index("%") + 1
+                for k, c in enumerate(chip):
+                    cells[start + k] = (c, paper + CHIP + (BOLD if k < bold else ""))
+            lines[row] = [("".join(c[0] for c in run),) + key for key, run in
+                          itertools.groupby(cells, key=lambda c: c[1:])]
+        return lines
 
     def detail(self, gpus):
-        """The GPUs whose processes' utilization is shown: all with the list open, else the selected
-        job's. Only those are asked, as the driver takes some 2 ms a GPU to answer."""
+        """The GPUs whose processes' utilization is shown: all with the list open, else the picked jobs'.
+        Only those are asked, as the driver takes some 2 ms a GPU to answer."""
         if self.listing:
             return {g.index for g in gpus}
-        job = self.selected()
-        return set(job.gpus) if job else set()
+        return {index for job in self.selected() for index in job.gpus}
 
     def remind(self):
         """A job still there 5 s after its SIGTERM may be stuck, or slow to save: say that k ends it."""
         if not self.sent:
             return
-        pid, when = self.sent
-        job = next((job for job in self.jobs if job.pid == pid), None)
-        if job and time.monotonic() - when > 5:
-            self.note = ([("{} is still running 5 s after SIGTERM; k kills it at once".format(job.name), WARN)],
+        pids, when = self.sent
+        living = [job for job in self.jobs if job.pid in pids]
+        if living and time.monotonic() - when > 5:
+            what = living[0].name + " is" if len(living) == 1 else "{} jobs are".format(len(living))
+            self.note = ([("{} still running 5 s after SIGTERM; k kills at once".format(what), WARN)],
                          time.monotonic() + 6)
-        if not job or time.monotonic() - when > 5:
+        if not living or time.monotonic() - when > 5:
             self.sent = None
 
     def scrolled(self, room):
@@ -1521,14 +1662,18 @@ class View:
         return [("GPUs", DIM)] + [(" {}".format(g.index), FAINT if g.index in self.hidden else BOLD, ("gpu", g.index))
                                   for g in gpus]
 
-    def bottom_line(self, width, job, newer, gpus):
+    def bottom_line(self, width, picked, newer, gpus):
         if self.asking:
-            return spread(width, [(self.question(job), WARN)], keys(("y", "yes"), ("n", "no")))
+            return spread(width, [(self.question(picked), WARN)], keys(("y", "yes"), ("n", "no")))
         note = self.note[0] if self.note and time.monotonic() < self.note[1] else None
-        if job:
+        if picked:
             hints = [("f", "all GPUs" if self.only else "only these GPUs")]
-            hints += [("t", "stop"), ("k", "kill")] if job.owner is None and os.name != "nt" else []
-            left = [(job.name, THEME.accent, job.pid), ("  ", DIM)] + about(job)
+            ours = any(job.mine for job in picked) and os.name != "nt"
+            hints += [("t", "stop"), ("k", "kill")] if ours else []
+            if len(picked) == 1:
+                left = [(picked[0].name, THEME.accent, picked[0].pid), ("  ", DIM)] + about(picked[0])
+            else:
+                left = [("{} jobs".format(len(picked)), THEME.accent), ("  ", DIM)] + about_all(picked)
             return spread(width, note or left, keys(*hints, ("Esc", "back")))
         if self.listing:
             return spread(width, note or [("click a job or a heading", DIM)],
@@ -1537,14 +1682,24 @@ class View:
             return spread(width, note, keys(("p", "processes"), ("Esc / q", "quit")))
         return footer(width, newer, self.picker(gpus))
 
-    def question(self, job):
-        """"stop train.py: SIGTERM to its torchrun (PID 48213), which ends its 2 workers?" and the like."""
+    def question(self, picked):
+        """"stop train.py on GPUs 4-5: SIGTERM to its torchrun (PID 48213), which ends its 2 workers?" and the
+        like; for several jobs "stop your 2 of 3 jobs on GPUs 4-5,7, SIGTERM to each: train.py, eval.py?". The
+        GPUs are those the signal reaches, whichever was clicked."""
         what = "stop" if self.asking == "SIGTERM" else "kill"
+        ours = [job for job in picked if job.mine]
+        where = "on " + gpu_numbers(index for job in ours for index in job.gpus)
+        if len(picked) > 1:
+            which = "{} jobs".format(len(ours)) if len(ours) == len(picked) else "your {} of {} jobs".format(
+                len(ours), len(picked))
+            return "{} {} {}, {} to each: {}?".format(what, which, where, self.asking,
+                                                     ", ".join(job.name for job in ours))
+        job = ours[0]
         if not job.launcher:
-            return "{} {} (PID {}) with {}?".format(what, job.name, job.pid, self.asking)
+            return "{} {} {} (PID {}) with {}?".format(what, job.name, where, job.pid, self.asking)
         passed = ", which ends its" if self.asking == "SIGTERM" else " and to its"
-        return "{} {}: {} to its {} (PID {}){} {} workers?".format(what, job.name, self.asking, job.launcher,
-                                                                  job.pid, passed, len(job.members))
+        return "{} {} {}: {} to its {} (PID {}){} {} workers?".format(what, job.name, where, self.asking,
+                                                                     job.launcher, job.pid, passed, len(job.members))
 
     def target(self, row, col):
         """What a click at (row, col) of the last screen means; None for nothing."""
@@ -1559,6 +1714,9 @@ class View:
     def handle(self, event):
         """Act on one key or mouse event; False means quit."""
         kind, value = event[0], event[1:]
+        if kind == "move":
+            self.pointer = value
+            return True
         if kind in ("wheel", "hwheel"):  # over a GPU's processes, they scroll, either way;
             step, row, col = value     # over the list, the list, or the picked job's command sideways;
             area = next((index for rows, cols, index in self.areas  # anywhere else, the GPU boxes
@@ -1577,10 +1735,6 @@ class View:
             else:
                 self.gpu_top += step
             return True
-        if kind == "list":  # "+" under a GPU: the list, from that GPU's first job
-            self.listing, self.follow = True, False
-            self.top = next((i for i, job in enumerate(self.jobs) if value[0] in job.gpus), self.top)
-            return True
         if kind == "gpu":  # a GPU's number at the bottom: hide that GPU, or show it again; one always stays
             index = value[0]
             if index in self.hidden:
@@ -1593,26 +1747,40 @@ class View:
             self.sort, self.follow = (value[0], not descending if value[0] == column else False), True
             return True
         if kind == "click":
-            target = self.target(*value)
-            if isinstance(target, tuple):  # a key hint, a heading of the list, "+" under a GPU
+            row, col = value
+            target = self.target(row, col)
+            if isinstance(target, tuple):  # a key hint, a heading of the list, a GPU's number at the bottom
                 return self.handle(target)
-            # A job picks it; the picked job again, or a place meaning nothing, lets it go, and outside the
-            # list closes that too. A double click on a job picks it and shows only its GPUs, or all again.
+            if target is None:  # anywhere else in a GPU's box: that GPU
+                target = next((("box", index) for rows, cols, index in self.panels if row in rows and col in cols),
+                              None)
             now = time.monotonic()
-            if target is None and value[0] not in self.list_rows:
-                self.listing = False
-            if target is not None and self.clicked[1] == target and now - self.clicked[0] < 0.4:
-                self.job, self.only = target, not self.only
-            else:
-                self.job = target if target is not None and target != self.job else None
+            double = target is not None and self.clicked[1] == target and now - self.clicked[0] < 0.4
             self.asking, self.follow, self.clicked = None, True, (now, target)
+            # A job picks it, a GPU the jobs on it, and outside the list opens that; the same again lets them
+            # go, as does a place meaning nothing, which outside the list closes that too. A double click picks
+            # them and shows only their GPUs, or all again.
+            if target is None:
+                self.pick(())
+                self.listing = self.listing and row in self.list_rows
+                return True
+            pids = [job.pid for job in self.jobs if target[1] in job.gpus] if isinstance(target, tuple) else [target]
+            if not pids:
+                self.note = ([("no processes on GPU {}".format(target[1]), DIM)], now + 2)
+            elif double:
+                self.pick(pids)
+                self.only = not self.only
+            elif set(pids) == self.picked:
+                self.pick(())
+            else:
+                self.pick(pids)
+                self.opened = self.opened or not self.listing
+                self.listing = True
             return True
         key = value[0].lower() if len(value[0]) == 1 else value[0]
         if self.asking:  # y stops; anything else, n and Esc among them, lets it be
             if key == "y":
-                note, sent = stop(self.selected(), self.asking)
-                self.note = (note, time.monotonic() + 4)
-                self.sent = (self.job, time.monotonic()) if sent and self.asking == "SIGTERM" else None
+                self.stop_picked()
             self.asking = None
             return key != "q"
         if key == "q":
@@ -1622,12 +1790,12 @@ class View:
                 self.only = False
             elif self.listing:
                 self.listing = False
-            elif self.job is not None:
-                self.job = None
+            elif self.picked:
+                self.pick(())
             else:
                 return False
-        elif key == "p":
-            self.listing = not self.listing
+        elif key == "p":  # opened or closed by hand, the list stays so when the picks go
+            self.listing, self.opened = not self.listing, False
         elif key == "v":  # the graph, the processes, the stats alone, GPU and MEM alone; V the one before
             self.left = LEFTS[(LEFTS.index(self.left) + (-1 if value[0] == "V" else 1)) % len(LEFTS)]
             name = {"graph": "graphs", "processes": "processes", "numbers": "numbers only",
@@ -1656,16 +1824,32 @@ class View:
         elif key in MOVES and self.jobs:
             order = [job.pid for job in self.jobs]
             i = order.index(self.job) if self.job in order else (-1 if MOVES[key] > 0 else len(order))
-            self.job, self.follow = order[min(max(i + MOVES[key], 0), len(order) - 1)], True
-        elif self.job is not None and key in ("f", "enter"):
+            self.pick([order[min(max(i + MOVES[key], 0), len(order) - 1)]])
+            self.follow = True
+        elif self.picked and key in ("f", "enter"):
             self.only = not self.only
-        elif self.job is not None and key in ("t", "k"):
-            if self.selected().owner is None:
+        elif self.picked and key in ("t", "k"):
+            if any(job.mine for job in self.selected()):
                 self.asking = "SIGTERM" if key == "t" else "SIGKILL"
             else:
                 self.note = ([("only your own processes can be stopped", WARN)], time.monotonic() + 3)
         return True
 
+    def stop_picked(self):
+        """Send the signal asked for to each picked job of ours (see stop), and say how that went."""
+        ours = [job for job in self.selected() if job.mine]
+        results = [stop(job, self.asking) for job in ours]
+        sent = {job.pid for job, (_, ok) in zip(ours, results) if ok}
+        failed = [note for note, ok in results if not ok]
+        if len(ours) == 1:
+            note = results[0][0]
+        elif not failed:
+            note = [("sent {} to {} jobs".format(self.asking, len(sent)), "")]
+        else:
+            note = [("sent {} to {} of {} jobs; ".format(self.asking, len(sent), len(ours)), WARN)] + failed[0]
+        self.note = (note, time.monotonic() + 4)
+        self.sent = (sent, time.monotonic()) if sent and self.asking == "SIGTERM" else None
+
 
 # For terminals that draw the line, block and other characters above two cells wide, as they may for Korean,
 # Chinese and Japanese (Unicode calls their width "ambiguous"): ASCII look-alikes, one for one.
@@ -1675,26 +1859,21 @@ ASCII = str.maketrans({"─": "-", "│": "|", "├": "+", "┤": "+", "╭": "+
                        "°": " ", "·": "-", "…": "~", "↑": "^", "↓": "v", "▲": "^", "▼": "v"})
 
 
-def paint(lines, mode="truecolor", ascii=False, gray=False):
-    """Join lines into one string in color `mode` (with `gray`, grays only; see restyle), and with `ascii` in
-    ASCII, sending a style code only where the style changes."""
+def paint(line, mode="truecolor", ascii=False, gray=False):
+    """One line as the terminal gets it, from plain to plain: in color `mode` (with `gray`, grays only; see
+    restyle), with `ascii` in ASCII, a style code only where the style changes."""
     out, current = [], ""
-    for i, line in enumerate(lines):
-        if i:
-            out.append("\r\n")
-        for text, style, *_ in line:
-            if ascii:
-                text = text.translate(ASCII)
-            if mode != "truecolor" or gray:
-                style = restyle(style, mode, gray)
-            if style != current:
-                # One color replaces another directly; anything else (bold, a background, plain) needs a
-                # reset first.
-                colors = (style.startswith("\x1b[38") and current.startswith("\x1b[38")
-                          and "\x1b[48" not in current)
-                out.append(style if colors else RESET + style)
-                current = style
-            out.append(text)
+    for text, style, *_ in line:
+        if ascii:
+            text = text.translate(ASCII)
+        if mode != "truecolor" or gray:
+            style = restyle(style, mode, gray)
+        if style != current:
+            # One color replaces another directly; anything else (bold, a background, plain) needs a reset first.
+            colors = style.startswith("\x1b[38") and current.startswith("\x1b[38") and "\x1b[48" not in current
+            out.append(style if colors else RESET + style)
+            current = style
+        out.append(text)
     return "".join(out) + RESET
 
 
@@ -1730,32 +1909,27 @@ class Settings:
         return data if isinstance(data, dict) else {}
 
     def load(self, host):
-        """This client's settings as View.restore takes them, the GPUs hidden those on `host`."""
-        mine = self._read().get(self.client)
-        mine = mine if isinstance(mine, dict) else {}
-        out = {"left": mine["left"]} if mine.get("left") in LEFTS else {}
-        if isinstance(mine.get("theme"), str):
-            out["theme"] = mine["theme"]
-        sort = mine.get("sort")
-        if isinstance(sort, list) and len(sort) == 2 and sort[0] in SORTS and isinstance(sort[1], bool):
-            out["sort"] = tuple(sort)
-        hidden = mine.get("hidden")
-        if isinstance(hidden, dict) and isinstance(hidden.get(host), list):
-            out["hidden"] = {i for i in hidden[host] if isinstance(i, int)}
-        return out
+        """This client's settings as View.restore takes them, the GPUs hidden those on `host`; none when they
+        are not as nvmon writes them (edited by hand, say)."""
+        try:
+            mine = self._read()[self.client]
+            kept = {"left": mine["left"], "theme": mine["theme"], "sort": (mine["sort"][0], bool(mine["sort"][1])),
+                    "hidden": set(map(int, mine["hidden"].get(host, [])))}
+            return kept if kept["left"] in LEFTS and kept["sort"][0] in SORTS else {}
+        except (KeyError, IndexError, TypeError, ValueError, AttributeError):
+            return {}
 
     def save(self, kept, indices, host):
-        """Keep `kept` (View.kept) for this client; for GPUs that are not `indices` (nvmon -g), what was
-        kept for them. Everyone else's settings stay as the file has them now."""
+        """Keep `kept` (View.kept) for this client; for GPUs that are not `indices` (nvmon -g), and on other
+        hosts, what was kept for them. Everyone else's settings stay as the file has them now."""
         data = self._read()
-        mine = data.get(self.client) if isinstance(data.get(self.client), dict) else {}
-        hidden = mine.get("hidden") if isinstance(mine.get("hidden"), dict) else {}
-        earlier = hidden.get(host) if isinstance(hidden.get(host), list) else []
-        hidden[host] = sorted({i for i in earlier if isinstance(i, int) and i not in indices} | kept["hidden"])
-        mine.pop("graphs", None)  # what "left" says now
-        mine.update(left=kept["left"], theme=kept["theme"], sort=list(kept["sort"]),
-                    hidden={name: gpus for name, gpus in hidden.items() if gpus})
-        data[self.client] = mine
+        try:
+            hidden = {name: set(map(int, gpus)) for name, gpus in data[self.client]["hidden"].items()}
+        except (KeyError, TypeError, ValueError, AttributeError):
+            hidden = {}
+        hidden[host] = hidden.get(host, set()) - set(indices) | kept["hidden"]
+        data[self.client] = {"left": kept["left"], "theme": kept["theme"], "sort": list(kept["sort"]),
+                             "hidden": {name: sorted(gpus) for name, gpus in hidden.items() if gpus}}
         temp = "{}.{}".format(self.path, os.getpid())
         try:
             os.makedirs(os.path.dirname(self.path), exist_ok=True)
@@ -1782,7 +1956,7 @@ INPUT = re.compile(rb"\x1b\[<(?P<button>\d+);(?P<x>\d+);(?P<y>\d+)(?P<act>[Mm])"
 def events(data):
     """The input in `data`, the bytes a terminal sends: ("key", name) events, ("click", row, column) for
     a left-button press, ("wheel", +1 down or -1 up, row, column), ("hwheel", +1 right or -1 left, row,
-    column)."""
+    column), ("move", row, column) where the pointer went."""
     out = []
     for m in INPUT.finditer(data):
         if m.group("button") or m.group("old"):
@@ -1795,7 +1969,9 @@ def events(data):
             if button & 64:  # the wheel: 64 up, 65 down, 66 left, 67 right; with Shift (+4) sideways too
                 sideways = button & 2 or button & 4
                 out.append(("hwheel" if sideways else "wheel", 1 if button & 1 else -1, y - 1, x - 1))
-            elif press and button & 3 == 0 and not button & 32:  # left button down, not a drag
+            elif button & 32:  # a move, a button held or not
+                out.append(("move", y - 1, x - 1))
+            elif press and button & 3 == 0:  # left button down
                 out.append(("click", y - 1, x - 1))
         elif m.group("report"):
             continue
@@ -1830,15 +2006,17 @@ TRUECOLOR_ANSWERS = re.compile(
 
 
 class Screen:
-    """Alternate screen, hidden cursor, no auto-wrap, unbuffered keys, mouse clicks and wheel; all
+    """Alternate screen, hidden cursor, no auto-wrap, unbuffered keys, mouse clicks, wheel and moves; all
     restored on exit."""
 
     def __enter__(self):
         self._keys = sys.stdin.isatty()
         self._mouse = self._keys and os.name != "nt"  # Windows reads keys with msvcrt, which sees no mouse
         self._restore = [enable_vt(), raw_keys() if self._keys else (lambda: None)]
-        # 1000 + 1006: clicks and the wheel come in as SGR-encoded sequences (Shift+drag still selects text).
-        self._write("\x1b[?1049h\x1b[?25l\x1b[?7l" + ("\x1b[?1000h\x1b[?1006h" if self._mouse else ""))
+        self._painted, self._size = [], None  # the lines last sent, and the window they were sent to
+        # 1000 + 1003 + 1006: clicks, the wheel and every move of the pointer come in as SGR-encoded sequences
+        # (Shift+drag still selects text). A terminal without 1003 sends clicks and the wheel all the same.
+        self._write("\x1b[?1049h\x1b[?25l\x1b[?7l" + ("\x1b[?1000h\x1b[?1003h\x1b[?1006h" if self._mouse else ""))
         # Windows Terminal and the console of Windows 10 on show 24-bit color and draw lines one cell wide.
         self.wide, self.truecolor = self.probe() if self._mouse else (False, os.name == "nt")
         return self
@@ -1873,7 +2051,8 @@ class Screen:
         return bool(cursor) and int(cursor.group(1)) > 2, bool(TRUECOLOR_ANSWERS.search(data))
 
     def __exit__(self, *exc):
-        self._write(RESET + ("\x1b[?1006l\x1b[?1000l" if self._mouse else "") + "\x1b[?7h\x1b[?25h\x1b[?1049l")
+        mouse = "\x1b[?1006l\x1b[?1003l\x1b[?1000l" if self._mouse else ""
+        self._write(RESET + mouse + "\x1b[?7h\x1b[?25h\x1b[?1049l")
         for undo in self._restore:
             undo()
 
@@ -1907,9 +2086,16 @@ class Screen:
             self._keys = False
         return events(data)
 
-    def draw(self, lines, mode, gray):
+    def draw(self, lines, mode, gray, size):
+        """The screen `lines`, in a window of `size`: only the lines that changed since the last time, so that
+        a frame, or the pointer moving over a graph, sends a few hundred bytes rather than the whole screen."""
+        painted = [paint(line, mode, self.wide, gray) for line in lines]
+        last = self._painted if self._size == size else []  # a new size: all anew
+        changed = "".join("\x1b[{};1H{}".format(row + 1, text) for row, text in enumerate(painted)
+                          if row >= len(last) or last[row] != text)
+        self._painted, self._size = painted, size
         # 2026 = synchronized output: terminals that know it swap the frame in at once.
-        self._write("\x1b[?2026h\x1b[H" + paint(lines, mode, self.wide, gray) + "\x1b[?2026l")
+        self._write("\x1b[?2026h" + changed + "\x1b[?2026l")
 
     @staticmethod
     def _write(text):
@@ -1964,13 +2150,14 @@ DEFAULT_INTERVAL, MIN_INTERVAL = 0.5, 0.1  # seconds
 
 def interval(text):
     """argparse type for -i: seconds between updates. The GPU's own readings change every 0.1-0.2 s, so
-    anything faster would only keep the driver busier and counts as MIN_INTERVAL; what is not a number of
-    seconds (abc, nan, inf) means the default."""
+    anything faster would only keep the driver busier and counts as MIN_INTERVAL."""
     try:
         seconds = float(text)
     except ValueError:
-        return DEFAULT_INTERVAL
-    return max(MIN_INTERVAL, seconds) if math.isfinite(seconds) else DEFAULT_INTERVAL
+        seconds = math.nan
+    if not seconds >= 0 or math.isinf(seconds):  # abc, nan, -1, inf
+        raise argparse.ArgumentTypeError("expected seconds such as 0.5 or 2, got {!r}".format(text))
+    return max(MIN_INTERVAL, seconds)
 
 
 def main():
@@ -2009,6 +2196,7 @@ def main():
         host, settings, indices = socket.gethostname(), Settings(), [g.index for g in gpus]
         poller, view = Poller(gpus), View()
         view.restore(settings.load(host), indices)
+        view.interval = args.interval
         kept = view.kept()
         with Screen() as screen:
             view.shows = color_mode(screen.truecolor)
@@ -2042,16 +2230,19 @@ def main():
                     width, height = os.get_terminal_size()
                     screen.draw(view.screen(gpus, width, height,
                                             header(width, args.interval, driver, poller.waiting()), update.newer),
-                                view.shows, view.theme.gray)
-                # Keys and clicks redraw at once; the numbers move on at the next frame.
+                                view.shows, view.theme.gray, (width, height))
+                # Keys and clicks redraw at once, the pointer when it goes to another graph column; the
+                # numbers move on at the next frame.
                 events = screen.read(0.01 if peek is not None else (poll_at or frame_at) - time.monotonic())
+                pointed = view.pointed()
                 going = all(map(view.handle, events))
                 if view.kept() != kept:  # a setting changed: keep it for next time
                     kept = view.kept()
                     settings.save(kept, indices, host)
                 if not going:
                     break
-                redraw, detail = bool(events), view.detail(gpus)
+                redraw = any(e[0] != "move" for e in events) or view.pointed() != pointed
+                detail = view.detail(gpus)
                 if detail - poller.detail:  # at the latest the next round, or the one after the round running
                     busy = not poller.done.is_set()
                     peek = (detail - poller.detail | (peek[0] if peek else set()), poller.rounds + 1 + busy)