sqtop 0.1.0__tar.gz

sqtop-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yanting Yang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sqtop-0.1.0/PKG-INFO

@@ -0,0 +1,324 @@
+Metadata-Version: 2.4
+Name: sqtop
+Version: 0.1.0
+Summary: Slurm dashboard: partitions, account/QOS limits and estimated start times
+Keywords: slurm,hpc,tui,dashboard,squeue,sbatch
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Clustering
+Classifier: Topic :: System :: Monitoring
+Requires-Dist: textual>=8,<9
+Requires-Python: ==3.13.*
+Project-URL: Homepage, https://github.com/yanting-yang/sqtop
+Project-URL: Issues, https://github.com/yanting-yang/sqtop/issues
+Description-Content-Type: text/markdown
+
+# sqtop
+
+A terminal dashboard for Slurm, in the spirit of
+[hunoutl/slurmtop](https://github.com/hunoutl/slurmtop) and
+[slmtop](https://github.com/dawnmy/slmtop), that answers "where can I run, under which
+account/QOS, and when would it start?", and can email you that report every hour while
+it runs.
+
+```
+ killarney  fir  tamia
+╸━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+ someone@klogin02  killarney  Slurm 25.05.9                                                              updated 13:49:06 (10s ago) · probed 13:49:06
+╭─ Partitions ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ Partition        Use  CPU/node  Mem/node  GPU/node  Load           Free         Idle   Pend  Max                                                   │
+│ gpubase_h100_b1  yes  48/96     2011G     8x h100   ██████▒▒  75%  4/80 h100    0/10   117   0-03:00:00                                            │
+│ gpubase_h100_b2  yes  48/96     2011G     8x h100   ███████▒  86%  1/64 h100    0/8    63    0-12:00:00                                            │
+│ gpubase_interac  yes  64        502G      4x l40s   ███████░  90%  6/100 l40s   0/25   1     0-03:00:00                                            │
+│ gpubase_l40s_b1  yes  64        502G      4x l40s   ███████▒  90%  18/672 l40s  1/168  2571  0-03:00:00                                            │
+│ gpubase_l40s_b3  yes  64        502G      4x l40s   ███████▒  88%  8/336 l40s   1/84   90    1-00:00:00                                            │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Start estimates · aip-abc123/normal ──────────────────────────────────────────────────╮╭─ Accounts & QOS ─────────────────────────────────────────╮
+│ Request   0-03:00:00    0-12:00:00    1-00:00:00    3-00:00:00    7-00:00:00           ││ Account      QOS      Jobs  GPUs  Wall        FS         │
+│ 1x h100     0-22:05:00    1-05:38:00    3-02:01:00    7-23:35:00   39-23:02:00         ││ aip-abc123*  interac  0/1   ∞     0-03:00:00  0.14       │
+│ 8x h100     1-04:16:00    1-10:21:00    3-06:44:00    8-11:56:00   40-05:13:00         ││ aip-abc123*  normal*  0/∞   ∞     ∞           0.14       │
+│ 1x l40s     0-02:12:00    0-01:27:00    0-01:48:00    0-19:17:00    7-15:47:00         │╰────────────────── * left out of commands · enter: probe ─╯
+│ 4x l40s     0-02:12:00    0-02:17:00    0-01:48:00    0-21:38:00    7-18:03:00         │╭─ My jobs ────────────────────────────────────────────────╮
+│ CPU only    0-02:12:00    0-01:27:00    0-01:48:00    0-19:17:00    7-15:47:00         ││ Job    Name   St  Start        Time  Nodes  Reason       │
+│                                                                                        ││ No jobs in the queue                                     │
+╰───────────────────────────────────────────────────────────────── sbatch -c4 --mem=32G ─╯╰──────────────────────────────────────────────────────────╯
+╭─ Details ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ gpubase_h100_b1  UP  usable  · max time 0-03:00:00 · priority tier 1                                                                               │
+│ Nodes 10 (5 mixed, 3 allocated, 2 unavailable) · pending jobs 117                                                                                  │
+│ 9 nodes: 48 CPUs / 2011 GB / 8x h100 · 1 node: 96 CPUs / 2011 GB / 8x h100                                                                         │
+│ CPUs 104 free / 328 allocated / 96 unavailable of 528 · Memory 12045 GB free of 20117 GB                                                           │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+ Refresh   Re-probe   Edit request   Email: off   Config   Quit
+```
+
+## Requirements
+
+- [uv](https://docs.astral.sh/uv/) (it fetches Python 3.13 if you do not have it)
+- A Slurm login node (`scontrol` and `squeue` on `PATH`; estimates need `sbatch`/`srun`)
+
+## Usage
+
+```bash
+uvx sqtop                  # run it from PyPI: one tab per cluster, this one first
+uvx sqtop > report.txt     # text report of this cluster when not a terminal
+uv tool install sqtop      # or install it once, then just type `sqtop`
+
+# From a checkout
+uv run sqtop
+```
+
+There are no options. The first tab is the cluster you are logged in to (from
+`ClusterName` in `scontrol show config`), and it is the only one checked at startup.
+Every other cluster with an `ssh` host in the [config file](#configuration) (Fir,
+Killarney and Tamia) gets a tab too, but is only contacted, over ssh, once you open its
+tab (click it). Every probe starts as 4 CPUs and 32 GB at each partition
+time limit with `sbatch`; change a tab's request with `Edit request` (CPUs, memory,
+walltimes, extra `sbatch` options), and click the command under Start estimates to
+switch between `sbatch` and `srun`. The visible tab refreshes every minute and re-probes
+every five. Click `Email: off` to email the report every hour until you quit
+([Hourly email](#hourly-email)). Everything is a click: see [Mouse](#mouse).
+
+Probes use `--test-only`, so nothing is submitted or cancelled until you confirm a
+Submit or Release (see below), and it is safe to run on a shared login node. Run it on
+the login node: on a compute node, such as inside an interactive job, `srun` probes fail
+with "Requested operation is presently disabled".
+
+### Remote clusters
+
+Remote tabs run Slurm on the cluster's login node through `ssh`, with your
+`~/.ssh/config` (aliases, users, keys). Nothing is sent to a remote cluster until you
+open its tab, and only the visible tab refreshes.
+
+sqtop never asks for a password or MFA. For a cluster that needs one, let ssh share
+connections and log in with `ssh` in another terminal first; sqtop reuses that
+connection:
+
+```
+# ~/.ssh/config
+Host *.alliancecan.ca
+    ControlMaster auto
+    ControlPath ~/.ssh/cm-%C
+    ControlPersist 4h
+```
+
+After `ssh fir.alliancecan.ca`, the connection stays open for 4 hours after your last
+session there closes. With keys that log in without a prompt, nothing is needed:
+sqtop opens its own shared connection, which closes 10 minutes after the last use.
+A tab that cannot connect shows ssh's error and how to connect; then click `Refresh`.
+
+The text report printed when output is not a terminal covers the cluster you are on;
+it covers the remote ones only where Slurm is not available.
+
+Each refresh of a remote tab is one ssh session, and the cache of limits is fetched
+only for your accounts and QOS, because login nodes can take seconds to start a
+session and the full cache can be tens of MB. Commands for a remote cluster are shown
+with its host as the prompt (`fir.alliancecan.ca$ sbatch ...`): run them there.
+
+### Mouse
+
+sqtop has no keyboard shortcuts: everything is a click. The bar at the bottom holds
+the dashboard's actions, and each dialog closes with its own buttons.
+
+| Click | Action |
+| --- | --- |
+| a cluster tab | Show that cluster |
+| a row or cell | Select it; the Details panel follows the selection |
+| a highlighted Accounts & QOS row | Probe with that account/QOS |
+| the Partitions title | Switch between Partitions and Nodes by hardware |
+| the account/QOS above the estimates | Probe with the next account/QOS |
+| the command under Start estimates | Switch between `sbatch` and `srun` probes |
+| `Submit` in Details | Submit the selected `sbatch` request, after you confirm |
+| `Copy` after a command in Details | Copy that command |
+| `Release` in a job's Details | Release the job with `scancel`, after you confirm |
+| `Refresh` / `Re-probe` | Refresh everything / re-probe now |
+| `Edit request` | Edit the probe request: whole node, CPUs, memory, walltimes, extra options such as `--partition` or `--constraint`, and the email address (blank for none) |
+| `Email: off` / `Email: on` | Email the report now and every hour until you quit / stop; the label says which is on |
+| `Config` | Edit the [config](#configuration) in a form; saving writes the file and applies it |
+| `Quit` | Quit |
+
+## Reading the dashboard
+
+- **Partitions**: `Use` says whether your accounts, QOS and Unix groups may submit
+  there. `CPU/node`, `Mem/node` and `GPU/node` are each node's hardware (CPUs exclude
+  reserved cores); a partition with two kinds of node shows both (`48/96`), with more
+  a range. `Load` is the share of GPUs (CPUs on CPU-only partitions) in use: `█`
+  allocated, `▒` unavailable (down, drained, reserved), `░` free. `Free` is free/total
+  per GPU type, `Idle` counts wholly idle nodes, `Pend` counts pending jobs from all
+  users, and `Max` is the time limit. Details list each kind of node with its count,
+  then memory, `AllowAccounts`/`AllowQos`, the partition QOS and priority tier.
+- **Accounts & QOS**: every account/QOS pair you can submit under; `*` marks what
+  commands leave out because Slurm picks it anyway: the account when it is your only
+  one (with several, commands always name it, since sites such as Fir require that
+  for GPU jobs) and the account's default QOS. `Jobs` is your running jobs over the tightest per-user
+  cap, `GPUs` and `Wall` are the tightest GPU and wall-time caps from any scope, and
+  `FS` is your fair-share factor. Details list every cap with its usage: QOS per user,
+  per account, per job and per node, plus association limits on you and your account.
+  `∞` means no cap, `?` unknown.
+- **Start estimates** (titled with the account/QOS probed): rows are request sizes
+  (each GPU type up to the tightest GPU cap, then CPU only), columns are walltimes. Each
+  cell is how long until the scheduler expects the job to start: `now`, a wait such as
+  `0-10:54:00`, `no` when it cannot run (Details gives the reason), or `-` above the
+  account/QOS wall-time cap. The earliest start in each row is underlined. Here a 1-day
+  L40S job starts sooner than a 3-hour one, because short jobs queue in the busier `b1`
+  partition.
+- **Details** for an estimate shows the partition it would land in, the probe that
+  produced the estimate with the scheduler's reply, and the command to submit the same
+  request; click `Copy` after either to copy it. The probe has only what the estimate
+  depends on; the submit command drops `--test-only` and adds your email options and
+  `--output=/dev/null`, so no `slurm-<jobid>.out` file is left behind.
+  `--account`/`--qos` are left out when Slurm would pick them anyway.
+- **Submit**: click the `Submit` label in Details to run an `sbatch` command, after a
+  dialog shows it and asks you to confirm; `Cancel` submits nothing. The job is submitted in the background (on the cluster's login node
+  for a remote tab), its id pops up, and My jobs refreshes; it holds the allocation with
+  `sleep infinity` until the time limit; its details in My jobs show the `scancel`
+  command that releases it. An `srun` command opens an interactive shell, so `Submit`
+  is not clickable for it: click `Copy` and run it in a terminal. A request the probe
+  was refused for is not submitted.
+- **My jobs** lists your queued and running jobs: `Partition` is where it runs or was
+  submitted to (several for some pending jobs), `Start` is when a job started, or when
+  the scheduler expects a pending one to (`-` until it has planned it), `Nodes` is where
+  it runs, or the nodes planned for a pending job (dimmed, as they can still change),
+  and `Reason` is why a pending job waits. A job's details show the command that
+  attaches a shell to it (`srun --jobid=<id> --overlap --pty <shell>`, once it runs)
+  and the one that releases it (`scancel <id>`), each with `Copy`. Clicking `Release`
+  runs `scancel` after a dialog asks you to confirm (Cancel is selected first): a
+  running job is killed and its allocation released, a pending one leaves the queue.
+
+## Configuration
+
+All settings live in one file, `~/.config/sqtop/config.toml`
+(`$XDG_CONFIG_HOME/sqtop/config.toml` if you set that): email (`[mail]`, `[smtp]`)
+and cluster profiles (`[clusters.<name>]`). Click `Config` in the dashboard to edit it
+in a form: the job email address and states, the hourly email's server, port, sender,
+password and recipient, and one row per cluster (name, GPU counts picked from `pow2`,
+`all` and `whole_node`, and ssh host) with `Add cluster` and `Remove`. `Cancel` closes
+it unchanged. `Save` checks the settings (a problem is shown under the form, which stays
+open), writes the file privately (`chmod 600`, since it holds the SMTP password) and
+applies what changed without a restart: clusters added get tabs and removed ones lose
+theirs, changed profiles and `[mail]` settings are probed again, and the hourly email
+switches to a changed `[smtp]`.
+
+Without the file, `sqtop` shows only the cluster you are logged in to, with no email
+settings. The file is plain TOML, so you can also edit it by hand; the form rewrites it
+without comments.
+
+### Other clusters
+
+Nothing is hardcoded per cluster: partitions, GPU types, time tiers, accounts and
+limits are read from Slurm. A profile records only what Slurm cannot report, keyed by
+the cluster name. To add a cluster, or a tab for one you are not logged in to, add a
+row in the config form (`Config`); in the file it is a table:
+
+```toml
+[clusters.mycluster]          # the ClusterName from `scontrol show config`
+gpu_counts = "all"            # "pow2" (1, 2, 4, ... default), "all" or "whole_node"
+                              # (GPU jobs then take whole nodes: --exclusive --mem=0)
+ssh = "login.mycluster.org"   # a tab for it from anywhere; host or ~/.ssh/config alias
+```
+
+For example, the Alliance clusters this was written on need no rules, only an `ssh`
+host for a tab, except `tamia`, which allocates GPUs by whole node:
+
+```toml
+[clusters.fir]
+ssh = "fir.alliancecan.ca"
+
+[clusters.killarney]
+ssh = "killarney.alliancecan.ca"
+
+[clusters.tamia]
+ssh = "tamia.alliancecan.ca"
+gpu_counts = "whole_node"
+```
+
+The walltimes probed are not a setting: they are the distinct time limits of the
+partitions you can use, which is where routing changes, and `Edit request` overrides
+them for a session.
+
+### Whole nodes
+
+Tick "Whole node" in the request editor (`Edit request`) to probe whole nodes: every command then
+asks for `--exclusive --mem=0` (every CPU and all the memory) instead of `-c`/`--mem`,
+and the GPU rows are each type's full node, such as `8x h100` or `4x l40s`. Untick it
+to go back to the CPUs and memory in the editor.
+
+On clusters that only allocate GPUs by node (`gpu_counts = "whole_node"`, such as
+Tamia) the GPU rows always ask for the whole node this way, so a job that holds all of
+a node's GPUs also gets its CPUs and memory; the CPU-only row keeps the editor's
+values. The estimates panel then notes "GPUs by whole node".
+
+### Email notifications
+
+The `[mail]` table in your config (`Config`) adds `--mail-type` and `--mail-user` to every
+command, so a job you submit from one emails you when it starts, ends or fails:
+
+```toml
+[mail]
+user = "you@example.com"
+type = "BEGIN,END,FAIL"       # any of sbatch's --mail-type values
+```
+
+Only the commands to submit carry them: probes leave them out, since email does not
+change when a job would start. Remove the table to leave email out, or clear the
+address for a session with `Edit request`.
+
+### Hourly email
+
+Click `Email: off` in the dashboard to email the report now and every hour until you
+quit `sqtop` (or click `Email: on` to stop). Each email covers the cluster you are on and every
+remote tab you have opened, with each tab's current request. Every email has the subject
+`sqtop` and replies to the first one, so your mail client keeps them in one
+conversation instead of separate emails. Each starts with your job counts and what
+happened since the previous email, followed by the text report:
+
+```
+killarney: 1 running, 2 pending. Since 14:00:
+  6284538 (wrap) started on kn021
+  6233018 (wrap) ended: TIMEOUT
+  6290112 (eval) submitted (Priority), expected to start 10-17 09:30
+```
+
+The server is read from `[smtp]` in your config (`Config`) when you turn it on:
+
+```toml
+[smtp]
+server = "mail.example.org"
+port = 587                       # STARTTLS; 465 for SSL
+sender = "you@example.org"       # also the login
+password = "..."
+recipient = "you@example.org"    # default: user in [mail]
+```
+
+The bottom bar says whether the hourly email is on (`Email: on`) or off (`Email: off`).
+While it is on, the top bar shows when the next email goes out (`next email 16:37`),
+and marks it `(last failed)` if one could not be sent. The thread and the jobs last reported are kept
+in `~/.local/state/sqtop/mail.json`, so the next run of the dashboard continues the
+same conversation; delete the file to start a new one. Gmail starts a new conversation
+after 100 emails, about every four days.
+
+### When `sacctmgr` fails
+
+On some login nodes `sacctmgr` fails with "Connection refused" because `slurmdbd` runs
+on the controller. Accounting is still enforced: `sqtop` reads limits from
+`scontrol show assoc_mgr` instead. Without `sacctmgr` it cannot list your QOS
+assignments, so it test-submits each cached account/QOS pair and hides the ones Slurm
+rejects.
+
+## Development
+
+```bash
+uv sync
+uv run python -m unittest discover -s tests -t tests
+uvx ruff check src tests
+```
+
+See [AGENTS.md](AGENTS.md) for the code layout and site-specific Slurm quirks.
+
+## License
+
+MIT; see [LICENSE](LICENSE).

sqtop-0.1.0/README.md

@@ -0,0 +1,304 @@
+# sqtop
+
+A terminal dashboard for Slurm, in the spirit of
+[hunoutl/slurmtop](https://github.com/hunoutl/slurmtop) and
+[slmtop](https://github.com/dawnmy/slmtop), that answers "where can I run, under which
+account/QOS, and when would it start?", and can email you that report every hour while
+it runs.
+
+```
+ killarney  fir  tamia
+╸━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+ someone@klogin02  killarney  Slurm 25.05.9                                                              updated 13:49:06 (10s ago) · probed 13:49:06
+╭─ Partitions ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ Partition        Use  CPU/node  Mem/node  GPU/node  Load           Free         Idle   Pend  Max                                                   │
+│ gpubase_h100_b1  yes  48/96     2011G     8x h100   ██████▒▒  75%  4/80 h100    0/10   117   0-03:00:00                                            │
+│ gpubase_h100_b2  yes  48/96     2011G     8x h100   ███████▒  86%  1/64 h100    0/8    63    0-12:00:00                                            │
+│ gpubase_interac  yes  64        502G      4x l40s   ███████░  90%  6/100 l40s   0/25   1     0-03:00:00                                            │
+│ gpubase_l40s_b1  yes  64        502G      4x l40s   ███████▒  90%  18/672 l40s  1/168  2571  0-03:00:00                                            │
+│ gpubase_l40s_b3  yes  64        502G      4x l40s   ███████▒  88%  8/336 l40s   1/84   90    1-00:00:00                                            │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Start estimates · aip-abc123/normal ──────────────────────────────────────────────────╮╭─ Accounts & QOS ─────────────────────────────────────────╮
+│ Request   0-03:00:00    0-12:00:00    1-00:00:00    3-00:00:00    7-00:00:00           ││ Account      QOS      Jobs  GPUs  Wall        FS         │
+│ 1x h100     0-22:05:00    1-05:38:00    3-02:01:00    7-23:35:00   39-23:02:00         ││ aip-abc123*  interac  0/1   ∞     0-03:00:00  0.14       │
+│ 8x h100     1-04:16:00    1-10:21:00    3-06:44:00    8-11:56:00   40-05:13:00         ││ aip-abc123*  normal*  0/∞   ∞     ∞           0.14       │
+│ 1x l40s     0-02:12:00    0-01:27:00    0-01:48:00    0-19:17:00    7-15:47:00         │╰────────────────── * left out of commands · enter: probe ─╯
+│ 4x l40s     0-02:12:00    0-02:17:00    0-01:48:00    0-21:38:00    7-18:03:00         │╭─ My jobs ────────────────────────────────────────────────╮
+│ CPU only    0-02:12:00    0-01:27:00    0-01:48:00    0-19:17:00    7-15:47:00         ││ Job    Name   St  Start        Time  Nodes  Reason       │
+│                                                                                        ││ No jobs in the queue                                     │
+╰───────────────────────────────────────────────────────────────── sbatch -c4 --mem=32G ─╯╰──────────────────────────────────────────────────────────╯
+╭─ Details ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ gpubase_h100_b1  UP  usable  · max time 0-03:00:00 · priority tier 1                                                                               │
+│ Nodes 10 (5 mixed, 3 allocated, 2 unavailable) · pending jobs 117                                                                                  │
+│ 9 nodes: 48 CPUs / 2011 GB / 8x h100 · 1 node: 96 CPUs / 2011 GB / 8x h100                                                                         │
+│ CPUs 104 free / 328 allocated / 96 unavailable of 528 · Memory 12045 GB free of 20117 GB                                                           │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+ Refresh   Re-probe   Edit request   Email: off   Config   Quit
+```
+
+## Requirements
+
+- [uv](https://docs.astral.sh/uv/) (it fetches Python 3.13 if you do not have it)
+- A Slurm login node (`scontrol` and `squeue` on `PATH`; estimates need `sbatch`/`srun`)
+
+## Usage
+
+```bash
+uvx sqtop                  # run it from PyPI: one tab per cluster, this one first
+uvx sqtop > report.txt     # text report of this cluster when not a terminal
+uv tool install sqtop      # or install it once, then just type `sqtop`
+
+# From a checkout
+uv run sqtop
+```
+
+There are no options. The first tab is the cluster you are logged in to (from
+`ClusterName` in `scontrol show config`), and it is the only one checked at startup.
+Every other cluster with an `ssh` host in the [config file](#configuration) (Fir,
+Killarney and Tamia) gets a tab too, but is only contacted, over ssh, once you open its
+tab (click it). Every probe starts as 4 CPUs and 32 GB at each partition
+time limit with `sbatch`; change a tab's request with `Edit request` (CPUs, memory,
+walltimes, extra `sbatch` options), and click the command under Start estimates to
+switch between `sbatch` and `srun`. The visible tab refreshes every minute and re-probes
+every five. Click `Email: off` to email the report every hour until you quit
+([Hourly email](#hourly-email)). Everything is a click: see [Mouse](#mouse).
+
+Probes use `--test-only`, so nothing is submitted or cancelled until you confirm a
+Submit or Release (see below), and it is safe to run on a shared login node. Run it on
+the login node: on a compute node, such as inside an interactive job, `srun` probes fail
+with "Requested operation is presently disabled".
+
+### Remote clusters
+
+Remote tabs run Slurm on the cluster's login node through `ssh`, with your
+`~/.ssh/config` (aliases, users, keys). Nothing is sent to a remote cluster until you
+open its tab, and only the visible tab refreshes.
+
+sqtop never asks for a password or MFA. For a cluster that needs one, let ssh share
+connections and log in with `ssh` in another terminal first; sqtop reuses that
+connection:
+
+```
+# ~/.ssh/config
+Host *.alliancecan.ca
+    ControlMaster auto
+    ControlPath ~/.ssh/cm-%C
+    ControlPersist 4h
+```
+
+After `ssh fir.alliancecan.ca`, the connection stays open for 4 hours after your last
+session there closes. With keys that log in without a prompt, nothing is needed:
+sqtop opens its own shared connection, which closes 10 minutes after the last use.
+A tab that cannot connect shows ssh's error and how to connect; then click `Refresh`.
+
+The text report printed when output is not a terminal covers the cluster you are on;
+it covers the remote ones only where Slurm is not available.
+
+Each refresh of a remote tab is one ssh session, and the cache of limits is fetched
+only for your accounts and QOS, because login nodes can take seconds to start a
+session and the full cache can be tens of MB. Commands for a remote cluster are shown
+with its host as the prompt (`fir.alliancecan.ca$ sbatch ...`): run them there.
+
+### Mouse
+
+sqtop has no keyboard shortcuts: everything is a click. The bar at the bottom holds
+the dashboard's actions, and each dialog closes with its own buttons.
+
+| Click | Action |
+| --- | --- |
+| a cluster tab | Show that cluster |
+| a row or cell | Select it; the Details panel follows the selection |
+| a highlighted Accounts & QOS row | Probe with that account/QOS |
+| the Partitions title | Switch between Partitions and Nodes by hardware |
+| the account/QOS above the estimates | Probe with the next account/QOS |
+| the command under Start estimates | Switch between `sbatch` and `srun` probes |
+| `Submit` in Details | Submit the selected `sbatch` request, after you confirm |
+| `Copy` after a command in Details | Copy that command |
+| `Release` in a job's Details | Release the job with `scancel`, after you confirm |
+| `Refresh` / `Re-probe` | Refresh everything / re-probe now |
+| `Edit request` | Edit the probe request: whole node, CPUs, memory, walltimes, extra options such as `--partition` or `--constraint`, and the email address (blank for none) |
+| `Email: off` / `Email: on` | Email the report now and every hour until you quit / stop; the label says which is on |
+| `Config` | Edit the [config](#configuration) in a form; saving writes the file and applies it |
+| `Quit` | Quit |
+
+## Reading the dashboard
+
+- **Partitions**: `Use` says whether your accounts, QOS and Unix groups may submit
+  there. `CPU/node`, `Mem/node` and `GPU/node` are each node's hardware (CPUs exclude
+  reserved cores); a partition with two kinds of node shows both (`48/96`), with more
+  a range. `Load` is the share of GPUs (CPUs on CPU-only partitions) in use: `█`
+  allocated, `▒` unavailable (down, drained, reserved), `░` free. `Free` is free/total
+  per GPU type, `Idle` counts wholly idle nodes, `Pend` counts pending jobs from all
+  users, and `Max` is the time limit. Details list each kind of node with its count,
+  then memory, `AllowAccounts`/`AllowQos`, the partition QOS and priority tier.
+- **Accounts & QOS**: every account/QOS pair you can submit under; `*` marks what
+  commands leave out because Slurm picks it anyway: the account when it is your only
+  one (with several, commands always name it, since sites such as Fir require that
+  for GPU jobs) and the account's default QOS. `Jobs` is your running jobs over the tightest per-user
+  cap, `GPUs` and `Wall` are the tightest GPU and wall-time caps from any scope, and
+  `FS` is your fair-share factor. Details list every cap with its usage: QOS per user,
+  per account, per job and per node, plus association limits on you and your account.
+  `∞` means no cap, `?` unknown.
+- **Start estimates** (titled with the account/QOS probed): rows are request sizes
+  (each GPU type up to the tightest GPU cap, then CPU only), columns are walltimes. Each
+  cell is how long until the scheduler expects the job to start: `now`, a wait such as
+  `0-10:54:00`, `no` when it cannot run (Details gives the reason), or `-` above the
+  account/QOS wall-time cap. The earliest start in each row is underlined. Here a 1-day
+  L40S job starts sooner than a 3-hour one, because short jobs queue in the busier `b1`
+  partition.
+- **Details** for an estimate shows the partition it would land in, the probe that
+  produced the estimate with the scheduler's reply, and the command to submit the same
+  request; click `Copy` after either to copy it. The probe has only what the estimate
+  depends on; the submit command drops `--test-only` and adds your email options and
+  `--output=/dev/null`, so no `slurm-<jobid>.out` file is left behind.
+  `--account`/`--qos` are left out when Slurm would pick them anyway.
+- **Submit**: click the `Submit` label in Details to run an `sbatch` command, after a
+  dialog shows it and asks you to confirm; `Cancel` submits nothing. The job is submitted in the background (on the cluster's login node
+  for a remote tab), its id pops up, and My jobs refreshes; it holds the allocation with
+  `sleep infinity` until the time limit; its details in My jobs show the `scancel`
+  command that releases it. An `srun` command opens an interactive shell, so `Submit`
+  is not clickable for it: click `Copy` and run it in a terminal. A request the probe
+  was refused for is not submitted.
+- **My jobs** lists your queued and running jobs: `Partition` is where it runs or was
+  submitted to (several for some pending jobs), `Start` is when a job started, or when
+  the scheduler expects a pending one to (`-` until it has planned it), `Nodes` is where
+  it runs, or the nodes planned for a pending job (dimmed, as they can still change),
+  and `Reason` is why a pending job waits. A job's details show the command that
+  attaches a shell to it (`srun --jobid=<id> --overlap --pty <shell>`, once it runs)
+  and the one that releases it (`scancel <id>`), each with `Copy`. Clicking `Release`
+  runs `scancel` after a dialog asks you to confirm (Cancel is selected first): a
+  running job is killed and its allocation released, a pending one leaves the queue.
+
+## Configuration
+
+All settings live in one file, `~/.config/sqtop/config.toml`
+(`$XDG_CONFIG_HOME/sqtop/config.toml` if you set that): email (`[mail]`, `[smtp]`)
+and cluster profiles (`[clusters.<name>]`). Click `Config` in the dashboard to edit it
+in a form: the job email address and states, the hourly email's server, port, sender,
+password and recipient, and one row per cluster (name, GPU counts picked from `pow2`,
+`all` and `whole_node`, and ssh host) with `Add cluster` and `Remove`. `Cancel` closes
+it unchanged. `Save` checks the settings (a problem is shown under the form, which stays
+open), writes the file privately (`chmod 600`, since it holds the SMTP password) and
+applies what changed without a restart: clusters added get tabs and removed ones lose
+theirs, changed profiles and `[mail]` settings are probed again, and the hourly email
+switches to a changed `[smtp]`.
+
+Without the file, `sqtop` shows only the cluster you are logged in to, with no email
+settings. The file is plain TOML, so you can also edit it by hand; the form rewrites it
+without comments.
+
+### Other clusters
+
+Nothing is hardcoded per cluster: partitions, GPU types, time tiers, accounts and
+limits are read from Slurm. A profile records only what Slurm cannot report, keyed by
+the cluster name. To add a cluster, or a tab for one you are not logged in to, add a
+row in the config form (`Config`); in the file it is a table:
+
+```toml
+[clusters.mycluster]          # the ClusterName from `scontrol show config`
+gpu_counts = "all"            # "pow2" (1, 2, 4, ... default), "all" or "whole_node"
+                              # (GPU jobs then take whole nodes: --exclusive --mem=0)
+ssh = "login.mycluster.org"   # a tab for it from anywhere; host or ~/.ssh/config alias
+```
+
+For example, the Alliance clusters this was written on need no rules, only an `ssh`
+host for a tab, except `tamia`, which allocates GPUs by whole node:
+
+```toml
+[clusters.fir]
+ssh = "fir.alliancecan.ca"
+
+[clusters.killarney]
+ssh = "killarney.alliancecan.ca"
+
+[clusters.tamia]
+ssh = "tamia.alliancecan.ca"
+gpu_counts = "whole_node"
+```
+
+The walltimes probed are not a setting: they are the distinct time limits of the
+partitions you can use, which is where routing changes, and `Edit request` overrides
+them for a session.
+
+### Whole nodes
+
+Tick "Whole node" in the request editor (`Edit request`) to probe whole nodes: every command then
+asks for `--exclusive --mem=0` (every CPU and all the memory) instead of `-c`/`--mem`,
+and the GPU rows are each type's full node, such as `8x h100` or `4x l40s`. Untick it
+to go back to the CPUs and memory in the editor.
+
+On clusters that only allocate GPUs by node (`gpu_counts = "whole_node"`, such as
+Tamia) the GPU rows always ask for the whole node this way, so a job that holds all of
+a node's GPUs also gets its CPUs and memory; the CPU-only row keeps the editor's
+values. The estimates panel then notes "GPUs by whole node".
+
+### Email notifications
+
+The `[mail]` table in your config (`Config`) adds `--mail-type` and `--mail-user` to every
+command, so a job you submit from one emails you when it starts, ends or fails:
+
+```toml
+[mail]
+user = "you@example.com"
+type = "BEGIN,END,FAIL"       # any of sbatch's --mail-type values
+```
+
+Only the commands to submit carry them: probes leave them out, since email does not
+change when a job would start. Remove the table to leave email out, or clear the
+address for a session with `Edit request`.
+
+### Hourly email
+
+Click `Email: off` in the dashboard to email the report now and every hour until you
+quit `sqtop` (or click `Email: on` to stop). Each email covers the cluster you are on and every
+remote tab you have opened, with each tab's current request. Every email has the subject
+`sqtop` and replies to the first one, so your mail client keeps them in one
+conversation instead of separate emails. Each starts with your job counts and what
+happened since the previous email, followed by the text report:
+
+```
+killarney: 1 running, 2 pending. Since 14:00:
+  6284538 (wrap) started on kn021
+  6233018 (wrap) ended: TIMEOUT
+  6290112 (eval) submitted (Priority), expected to start 10-17 09:30
+```
+
+The server is read from `[smtp]` in your config (`Config`) when you turn it on:
+
+```toml
+[smtp]
+server = "mail.example.org"
+port = 587                       # STARTTLS; 465 for SSL
+sender = "you@example.org"       # also the login
+password = "..."
+recipient = "you@example.org"    # default: user in [mail]
+```
+
+The bottom bar says whether the hourly email is on (`Email: on`) or off (`Email: off`).
+While it is on, the top bar shows when the next email goes out (`next email 16:37`),
+and marks it `(last failed)` if one could not be sent. The thread and the jobs last reported are kept
+in `~/.local/state/sqtop/mail.json`, so the next run of the dashboard continues the
+same conversation; delete the file to start a new one. Gmail starts a new conversation
+after 100 emails, about every four days.
+
+### When `sacctmgr` fails
+
+On some login nodes `sacctmgr` fails with "Connection refused" because `slurmdbd` runs
+on the controller. Accounting is still enforced: `sqtop` reads limits from
+`scontrol show assoc_mgr` instead. Without `sacctmgr` it cannot list your QOS
+assignments, so it test-submits each cached account/QOS pair and hides the ones Slurm
+rejects.
+
+## Development
+
+```bash
+uv sync
+uv run python -m unittest discover -s tests -t tests
+uvx ruff check src tests
+```
+
+See [AGENTS.md](AGENTS.md) for the code layout and site-specific Slurm quirks.
+
+## License
+
+MIT; see [LICENSE](LICENSE).