slurmpast 0.2.0__tar.gz

slurmpast-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 slurmpost contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

slurmpast-0.2.0/PKG-INFO

@@ -0,0 +1,256 @@
+Metadata-Version: 2.4
+Name: slurmpast
+Version: 0.2.0
+Summary: Why did your Slurm jobs fail? A post-mortem dashboard for finished jobs.
+Author-email: Youzhi Yu <yuyouzhi666@icloud.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/PursuitOfDataScience/slurmpast
+Project-URL: Repository, https://github.com/PursuitOfDataScience/slurmpast.git
+Project-URL: Documentation, https://github.com/PursuitOfDataScience/slurmpast#readme
+Project-URL: Issues, https://github.com/PursuitOfDataScience/slurmpast/issues
+Keywords: slurm,hpc,sacct,postmortem,job-failure,tui,seff
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Natural Language :: English
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Topic :: System :: Monitoring
+Classifier: Topic :: Utilities
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: textual<9,>=0.86
+Requires-Dist: rich>=13.3.3
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: pytest-cov>=4; extra == "dev"
+Requires-Dist: ruff<0.17,>=0.15; extra == "dev"
+Requires-Dist: mypy<3,>=2.3; extra == "dev"
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+<h1 align="center">slurmpast</h1>
+
+<p align="center">
+  <strong>How your finished Slurm jobs actually ran — and what the next one should ask for.</strong>
+</p>
+
+<p align="center">
+  <a href="https://github.com/PursuitOfDataScience/slurmpast/actions/workflows/ci.yml"><img src="https://github.com/PursuitOfDataScience/slurmpast/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+  <img src="https://img.shields.io/badge/python-3.10%2B-blue.svg" alt="Python 3.10+">
+  <img src="https://img.shields.io/badge/license-MIT-green.svg" alt="MIT License">
+  <img src="https://img.shields.io/badge/tests-791-brightgreen.svg" alt="791 tests">
+</p>
+
+<p align="center">
+  <img src="assets/screenshot-overview.svg" width="900" alt="slurmpast overview: finished jobs rolled into workloads, ranked by resource use.">
+</p>
+
+**Before** the run, [`slurmate`](https://github.com/PursuitOfDataScience/slurmate)
+builds the request. **During** it,
+[`slurmwatch`](https://github.com/PursuitOfDataScience/slurmwatch) watches.
+**After** it, `slurmpast` tells you what happened and what to change.
+
+```bash
+pip install slurmpast
+```
+
+```bash
+slurmpast                  # dashboard, last 7 days
+slurmpast --sizing         # what to request next time, per workload
+slurmpast 51170455         # one job, every field Slurm recorded
+sp                         # short alias
+```
+
+Keys: `enter` open · `q` back · digits jump to a row · `/` search · `f` filter ·
+`s` sort · `n` nodes · `p` patterns · `y` copy · `?` help.
+No Slurm to hand? `slurmpast --demo`.
+
+---
+
+## What to request next time
+
+The reason to read finished jobs. Over-requesting narrows which nodes can host a
+job and reserves capacity nobody else can use; under-requesting kills the run.
+Your own history settles both.
+
+```
+argonne35-pretrain   test · 101 runs
+  --time            raise to 10:00:00   (requested 8h05m, observed 7h58m03s at p95)
+      p95 of 90 completed runs is 7h58m03s (longest 7h58m23s); +25% headroom.
+  --mem             already about right
+  --cpus-per-task   already about right
+  #SBATCH --time=10:00:00
+```
+
+That workload was running on a **seven-minute margin**. Four rules keep the
+advice honest:
+
+- A **TIMEOUT never sizes walltime down** — its elapsed is truncated at the
+  limit, so it bounds the true runtime only from below.
+- A **hung run is not evidence of needing more time.** One real workload hit a
+  30-minute wall 99 times on a median of 0.56 CPU-seconds; a longer limit buys a
+  longer hang.
+- An **OOM kill outranks MaxRSS.** It is an event, not a sample.
+- Below three usable runs it says **"not enough evidence"** rather than guessing.
+
+## One job
+
+Same row idiom as `slurmwatch`, so a job you watched running looks like the same
+object afterwards. The marker carries the resource's identity, the bar and the
+number carry the magnitude, and the verdict lives in the findings below.
+
+```
+  ● TIME   ████████████████▉░        94.0%   · 01:52:49 of the 02:00:00 limit
+  ● CPU    █████████████▎░░░░        73.5%   · 2.9 of 4 cores busy
+  ● MEM    ████████████████▋░        92.3%   · 184.6 GiB of the 200.0 GiB limit
+```
+
+Three rows, because a gauge needs a ceiling to be a fraction of. Kernel share,
+disk rate and GPU count have none, so they are printed as numbers below rather
+than as bars that can never fill — an unfillable `░░░░` reads as a measured zero.
+
+Below that: every field Slurm recorded, then the findings.
+
+```
+  gpu
+    devices          3                       gpu-hours        5.6
+    utilization      87.4%
+    device memory    35.4 GiB peak
+```
+
+That `utilization` is real wherever `gres.conf` sets `AutoDetect=nvml`. Where the
+cluster does not gather it, the row says so and names the setting instead of
+printing a zero.
+
+## Why the numbers differ from `seff`
+
+Seven traps, each found on a real record, each with a regression test naming the
+job id:
+
+| Trap | Reality |
+|---|---|
+| `ReqMem` is `0n` on **2,130 of 6,574** jobs | the real ceiling is in `AllocTRES` |
+| `AllocTRES` `mem=` is the **allocation total** | `--mem` and the cgroup are per *node*; a 2-node `--mem=8G` job records `mem=16G`, so dividing MaxRSS by it understates memory by the node count |
+| `MaxRSS` reports **51.25 GiB against a 40 GiB limit** that OOM-killed | under `jobacct_gather/linux` it sums RSS across the process tree, double-counting shared pages — so the caveat is read from your `JobAcctGatherType`, not assumed |
+| `MaxRSS` differs **4000×** between steps of one job | take the max, not a step |
+| `TotalCPU` exists **only on steps** | read it from `.batch` |
+| `State=RUNNING, End=Unknown` months after death | `Elapsed` becomes *now − start*; one record was 65% of a GPU-hour total |
+| `sacct --state=X` returns **zero rows** without `-E` | always pass an end time |
+
+A value that cannot be read prints `n/a`, never `0`.
+
+## Any Slurm cluster
+
+Developed on one cluster, but nothing here is calibrated to it. Every place the
+scheduler differs between sites is negotiated rather than assumed, and each of
+these is a regression test in `tests/test_portability.py`:
+
+| What varies | What it does about it |
+|---|---|
+| **Field names change.** Slurm 23.02 renamed `Reserved` → `Planned` | asks for whichever spelling your `sacct` accepts and reads it back under one name. Treating it as merely *optional* silently dropped queue wait on every cluster from 23.02 on |
+| **`SLURM_TIME_FORMAT`** rewrites every timestamp | pinned to `standard` for the child process. A site exporting `relative` turns `2026-04-29T14:55:48` into `29 Apr 14:55`, which stops dates parsing and makes the chronological sort alphabetical on a month name |
+| **`--constraint="v100\|a100"`** puts a pipe in a column, and `--parsable2` does not escape it | asks for an ASCII unit separator via `--delimiter` (in sacct since 17.11), falling back if absent. A pipe in field 32 shifted every memory, CPU and disk column after it |
+| **Hostlists** can be `unit[0-31]rack[0-41]`, `node[0001-0010]-int`, `cn_[01-02]` | full expansion, differential-tested against `scontrol show hostnames`. The multi-range form used to yield `unit0rack[0-2]` *as a node name*, hiding every node in such an allocation |
+| **GPUs** appear as `gres/gpu=4`, `gres/gpu:a100=4`, or pre-20.11 `AllocGRES=gpu:4` | all three read; typed entries summed |
+| **`gres/gpuutil`** is gathered wherever `gres.conf` sets `AutoDetect=nvml` | real GPU utilization and peak HBM where recorded — and the idle-GPU finding becomes a measurement instead of an inference from host CPU |
+| **`JobAcctGatherType`** decides what MaxRSS *is* | `jobacct_gather/cgroup` gives a genuine high-water mark, so the "double-counts shared pages" warning is withheld there rather than repeated |
+| **`StdOut`/`StdErr`** exist from Slurm 21.08 | used, with `%j`/`%A`/`%a`/`%x`/`%N` expanded, before falling back to guessing a filename |
+| **`--me`** needs Slurm 20.02 | falls back to `-u <you>`, not to an unfiltered `squeue` over the whole cluster |
+| **Accounting may be off**, or `sacct` absent | a one-line explanation and exit 2, never a traceback; the query has a timeout so an unreachable `slurmdbd` cannot hang the dashboard |
+
+`slurmpast --demo` pins a synthetic cluster config too, so the demo looks the
+same on a login node and on a laptop.
+
+## Built for a real history
+
+6,581 jobs in seven months. A flat list of that is not an interface, so the
+landing screen is a **workload rollup ranked by resource use** — a 5-run group
+that cost 400 GPU-hours outranks 400 two-second probes. The top 20 rows carry
+93.4% of it, and what is below the fold is stated rather than dropped.
+
+Job names encode parameters, so raw names barely group: 1,624 distinct names roll
+up to 1,687 groups. Collapsing digit runs (`s1e20`, `s2e47` → `s#e#`) folds that
+to 592, and a `NAMES` column says how many real names a pattern covers.
+
+No cache: the full seven-month query is 1.54s and the index builds in 0.04s,
+which does not justify a database. Logs are read only when you open a job.
+
+## Everything it extracts
+
+85 `sacct` fields — every one of the 107 this Slurm offers that carries a
+distinct measurement, with pure redundancy (`DBIndex`, `BlockID`, duplicate
+spellings of the same TRES) left out. The wide query costs 2.26s over seven
+months against 1.38s for a minimal 27, so nobody has to re-run it because the
+number they wanted was never collected.
+
+| group | captured |
+|---|---|
+| **cpu** | total, **user vs kernel split**, utilization, frequency, per-task min/average, slowest task with its node |
+| **memory** | ceiling **per node and per allocation**, peak with the node and task that hit it, average, task imbalance, virtual size, page faults |
+| **filesystem** | **read and write separately**, rate |
+| **gpu** | devices, GPU-hours, and **utilization and peak HBM where the site gathers them** |
+| **timing** | submit/start/end, elapsed, limit, queue wait, **backfill vs main scheduler**, priority |
+| **outcome** | state, exit code and signal, worst step exit, reason |
+| **provenance** | work directory, recorded stdout/stderr paths, **the submit line itself** |
+| per step | all of the above, for `.batch`, `.extern` and each srun step |
+
+Two were entirely invisible with a narrow `--format`: **disk writes** (one run
+read 112 GB and wrote 139 GB) and the **user/kernel CPU split** (791 of 6,582
+jobs exceed 30% kernel time — syscall overhead no other tool surfaces).
+
+`--json` emits all of it — 174 values per job. Nothing is captured and then hidden; a test fails if a field is read but never surfaced.
+
+## Also finds
+
+**Repeated failure.** `seff` describes one job; `slurmwatch` describes one live
+run. Neither can say *"you submitted this 115 times and it died 99 times."*
+
+**Hand-searched memory.** One real series walked `48G → 32G → 32G → 17G → 12G →
+12G → 14G → 16G → 18G`, then succeeded at 32G — a value that had already OOM'd,
+proving `--mem` was never the variable.
+
+**Nodes that eat jobs**, controlled for workload, with Wilson intervals and a
+ready-to-paste `--exclude`. Uncontrolled, one node looked 25% bad almost entirely
+because a buggy campaign landed there — so a node is only called `worse` when its
+interval clears the baseline.
+
+## Notes
+
+Selecting text just works: mouse capture is off by default, so your terminal
+handles selection as it does anywhere else. `M` hands the mouse to the app;
+`y`/`Y` copy via OSC 52 and also write `~/.cache/slurmpast/clip.txt`.
+
+The analysis modules import no third-party package and work as a library:
+
+```python
+from slurmpast import History, Sacct
+from slurmpast.sizing import recommend
+
+history = History(Sacct().history(user="you", since="now-30days"))
+for advice in recommend(history.groups[0].jobs):
+    print(advice.flag, advice.verdict, advice.suggestion)
+```
+
+Developed against Slurm 20.11.8 and Textual 0.89–8.2; verified against the Slurm
+documentation and release notes from 17.11 to 26.05, and against `scontrol show
+hostnames` for node-name expansion. CI covers Python 3.10–3.13, both Textual
+ends, and a no-Slurm machine.
+
+What is *measured* is portable; what is *judged* is not. The thresholds — when a
+CPU share is low, when kernel time is heavy, when a node counts as worse — are
+heuristics from one cluster's history. They are conservative and each one is a
+named constant, so recalibrating means editing a number rather than the logic.
+`--nodes` is the piece most likely to need it.
+
+MIT.

slurmpast-0.2.0/README.md

@@ -0,0 +1,215 @@
+<h1 align="center">slurmpast</h1>
+
+<p align="center">
+  <strong>How your finished Slurm jobs actually ran — and what the next one should ask for.</strong>
+</p>
+
+<p align="center">
+  <a href="https://github.com/PursuitOfDataScience/slurmpast/actions/workflows/ci.yml"><img src="https://github.com/PursuitOfDataScience/slurmpast/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+  <img src="https://img.shields.io/badge/python-3.10%2B-blue.svg" alt="Python 3.10+">
+  <img src="https://img.shields.io/badge/license-MIT-green.svg" alt="MIT License">
+  <img src="https://img.shields.io/badge/tests-791-brightgreen.svg" alt="791 tests">
+</p>
+
+<p align="center">
+  <img src="assets/screenshot-overview.svg" width="900" alt="slurmpast overview: finished jobs rolled into workloads, ranked by resource use.">
+</p>
+
+**Before** the run, [`slurmate`](https://github.com/PursuitOfDataScience/slurmate)
+builds the request. **During** it,
+[`slurmwatch`](https://github.com/PursuitOfDataScience/slurmwatch) watches.
+**After** it, `slurmpast` tells you what happened and what to change.
+
+```bash
+pip install slurmpast
+```
+
+```bash
+slurmpast                  # dashboard, last 7 days
+slurmpast --sizing         # what to request next time, per workload
+slurmpast 51170455         # one job, every field Slurm recorded
+sp                         # short alias
+```
+
+Keys: `enter` open · `q` back · digits jump to a row · `/` search · `f` filter ·
+`s` sort · `n` nodes · `p` patterns · `y` copy · `?` help.
+No Slurm to hand? `slurmpast --demo`.
+
+---
+
+## What to request next time
+
+The reason to read finished jobs. Over-requesting narrows which nodes can host a
+job and reserves capacity nobody else can use; under-requesting kills the run.
+Your own history settles both.
+
+```
+argonne35-pretrain   test · 101 runs
+  --time            raise to 10:00:00   (requested 8h05m, observed 7h58m03s at p95)
+      p95 of 90 completed runs is 7h58m03s (longest 7h58m23s); +25% headroom.
+  --mem             already about right
+  --cpus-per-task   already about right
+  #SBATCH --time=10:00:00
+```
+
+That workload was running on a **seven-minute margin**. Four rules keep the
+advice honest:
+
+- A **TIMEOUT never sizes walltime down** — its elapsed is truncated at the
+  limit, so it bounds the true runtime only from below.
+- A **hung run is not evidence of needing more time.** One real workload hit a
+  30-minute wall 99 times on a median of 0.56 CPU-seconds; a longer limit buys a
+  longer hang.
+- An **OOM kill outranks MaxRSS.** It is an event, not a sample.
+- Below three usable runs it says **"not enough evidence"** rather than guessing.
+
+## One job
+
+Same row idiom as `slurmwatch`, so a job you watched running looks like the same
+object afterwards. The marker carries the resource's identity, the bar and the
+number carry the magnitude, and the verdict lives in the findings below.
+
+```
+  ● TIME   ████████████████▉░        94.0%   · 01:52:49 of the 02:00:00 limit
+  ● CPU    █████████████▎░░░░        73.5%   · 2.9 of 4 cores busy
+  ● MEM    ████████████████▋░        92.3%   · 184.6 GiB of the 200.0 GiB limit
+```
+
+Three rows, because a gauge needs a ceiling to be a fraction of. Kernel share,
+disk rate and GPU count have none, so they are printed as numbers below rather
+than as bars that can never fill — an unfillable `░░░░` reads as a measured zero.
+
+Below that: every field Slurm recorded, then the findings.
+
+```
+  gpu
+    devices          3                       gpu-hours        5.6
+    utilization      87.4%
+    device memory    35.4 GiB peak
+```
+
+That `utilization` is real wherever `gres.conf` sets `AutoDetect=nvml`. Where the
+cluster does not gather it, the row says so and names the setting instead of
+printing a zero.
+
+## Why the numbers differ from `seff`
+
+Seven traps, each found on a real record, each with a regression test naming the
+job id:
+
+| Trap | Reality |
+|---|---|
+| `ReqMem` is `0n` on **2,130 of 6,574** jobs | the real ceiling is in `AllocTRES` |
+| `AllocTRES` `mem=` is the **allocation total** | `--mem` and the cgroup are per *node*; a 2-node `--mem=8G` job records `mem=16G`, so dividing MaxRSS by it understates memory by the node count |
+| `MaxRSS` reports **51.25 GiB against a 40 GiB limit** that OOM-killed | under `jobacct_gather/linux` it sums RSS across the process tree, double-counting shared pages — so the caveat is read from your `JobAcctGatherType`, not assumed |
+| `MaxRSS` differs **4000×** between steps of one job | take the max, not a step |
+| `TotalCPU` exists **only on steps** | read it from `.batch` |
+| `State=RUNNING, End=Unknown` months after death | `Elapsed` becomes *now − start*; one record was 65% of a GPU-hour total |
+| `sacct --state=X` returns **zero rows** without `-E` | always pass an end time |
+
+A value that cannot be read prints `n/a`, never `0`.
+
+## Any Slurm cluster
+
+Developed on one cluster, but nothing here is calibrated to it. Every place the
+scheduler differs between sites is negotiated rather than assumed, and each of
+these is a regression test in `tests/test_portability.py`:
+
+| What varies | What it does about it |
+|---|---|
+| **Field names change.** Slurm 23.02 renamed `Reserved` → `Planned` | asks for whichever spelling your `sacct` accepts and reads it back under one name. Treating it as merely *optional* silently dropped queue wait on every cluster from 23.02 on |
+| **`SLURM_TIME_FORMAT`** rewrites every timestamp | pinned to `standard` for the child process. A site exporting `relative` turns `2026-04-29T14:55:48` into `29 Apr 14:55`, which stops dates parsing and makes the chronological sort alphabetical on a month name |
+| **`--constraint="v100\|a100"`** puts a pipe in a column, and `--parsable2` does not escape it | asks for an ASCII unit separator via `--delimiter` (in sacct since 17.11), falling back if absent. A pipe in field 32 shifted every memory, CPU and disk column after it |
+| **Hostlists** can be `unit[0-31]rack[0-41]`, `node[0001-0010]-int`, `cn_[01-02]` | full expansion, differential-tested against `scontrol show hostnames`. The multi-range form used to yield `unit0rack[0-2]` *as a node name*, hiding every node in such an allocation |
+| **GPUs** appear as `gres/gpu=4`, `gres/gpu:a100=4`, or pre-20.11 `AllocGRES=gpu:4` | all three read; typed entries summed |
+| **`gres/gpuutil`** is gathered wherever `gres.conf` sets `AutoDetect=nvml` | real GPU utilization and peak HBM where recorded — and the idle-GPU finding becomes a measurement instead of an inference from host CPU |
+| **`JobAcctGatherType`** decides what MaxRSS *is* | `jobacct_gather/cgroup` gives a genuine high-water mark, so the "double-counts shared pages" warning is withheld there rather than repeated |
+| **`StdOut`/`StdErr`** exist from Slurm 21.08 | used, with `%j`/`%A`/`%a`/`%x`/`%N` expanded, before falling back to guessing a filename |
+| **`--me`** needs Slurm 20.02 | falls back to `-u <you>`, not to an unfiltered `squeue` over the whole cluster |
+| **Accounting may be off**, or `sacct` absent | a one-line explanation and exit 2, never a traceback; the query has a timeout so an unreachable `slurmdbd` cannot hang the dashboard |
+
+`slurmpast --demo` pins a synthetic cluster config too, so the demo looks the
+same on a login node and on a laptop.
+
+## Built for a real history
+
+6,581 jobs in seven months. A flat list of that is not an interface, so the
+landing screen is a **workload rollup ranked by resource use** — a 5-run group
+that cost 400 GPU-hours outranks 400 two-second probes. The top 20 rows carry
+93.4% of it, and what is below the fold is stated rather than dropped.
+
+Job names encode parameters, so raw names barely group: 1,624 distinct names roll
+up to 1,687 groups. Collapsing digit runs (`s1e20`, `s2e47` → `s#e#`) folds that
+to 592, and a `NAMES` column says how many real names a pattern covers.
+
+No cache: the full seven-month query is 1.54s and the index builds in 0.04s,
+which does not justify a database. Logs are read only when you open a job.
+
+## Everything it extracts
+
+85 `sacct` fields — every one of the 107 this Slurm offers that carries a
+distinct measurement, with pure redundancy (`DBIndex`, `BlockID`, duplicate
+spellings of the same TRES) left out. The wide query costs 2.26s over seven
+months against 1.38s for a minimal 27, so nobody has to re-run it because the
+number they wanted was never collected.
+
+| group | captured |
+|---|---|
+| **cpu** | total, **user vs kernel split**, utilization, frequency, per-task min/average, slowest task with its node |
+| **memory** | ceiling **per node and per allocation**, peak with the node and task that hit it, average, task imbalance, virtual size, page faults |
+| **filesystem** | **read and write separately**, rate |
+| **gpu** | devices, GPU-hours, and **utilization and peak HBM where the site gathers them** |
+| **timing** | submit/start/end, elapsed, limit, queue wait, **backfill vs main scheduler**, priority |
+| **outcome** | state, exit code and signal, worst step exit, reason |
+| **provenance** | work directory, recorded stdout/stderr paths, **the submit line itself** |
+| per step | all of the above, for `.batch`, `.extern` and each srun step |
+
+Two were entirely invisible with a narrow `--format`: **disk writes** (one run
+read 112 GB and wrote 139 GB) and the **user/kernel CPU split** (791 of 6,582
+jobs exceed 30% kernel time — syscall overhead no other tool surfaces).
+
+`--json` emits all of it — 174 values per job. Nothing is captured and then hidden; a test fails if a field is read but never surfaced.
+
+## Also finds
+
+**Repeated failure.** `seff` describes one job; `slurmwatch` describes one live
+run. Neither can say *"you submitted this 115 times and it died 99 times."*
+
+**Hand-searched memory.** One real series walked `48G → 32G → 32G → 17G → 12G →
+12G → 14G → 16G → 18G`, then succeeded at 32G — a value that had already OOM'd,
+proving `--mem` was never the variable.
+
+**Nodes that eat jobs**, controlled for workload, with Wilson intervals and a
+ready-to-paste `--exclude`. Uncontrolled, one node looked 25% bad almost entirely
+because a buggy campaign landed there — so a node is only called `worse` when its
+interval clears the baseline.
+
+## Notes
+
+Selecting text just works: mouse capture is off by default, so your terminal
+handles selection as it does anywhere else. `M` hands the mouse to the app;
+`y`/`Y` copy via OSC 52 and also write `~/.cache/slurmpast/clip.txt`.
+
+The analysis modules import no third-party package and work as a library:
+
+```python
+from slurmpast import History, Sacct
+from slurmpast.sizing import recommend
+
+history = History(Sacct().history(user="you", since="now-30days"))
+for advice in recommend(history.groups[0].jobs):
+    print(advice.flag, advice.verdict, advice.suggestion)
+```
+
+Developed against Slurm 20.11.8 and Textual 0.89–8.2; verified against the Slurm
+documentation and release notes from 17.11 to 26.05, and against `scontrol show
+hostnames` for node-name expansion. CI covers Python 3.10–3.13, both Textual
+ends, and a no-Slurm machine.
+
+What is *measured* is portable; what is *judged* is not. The thresholds — when a
+CPU share is low, when kernel time is heavy, when a node counts as worse — are
+heuristics from one cluster's history. They are conservative and each one is a
+named constant, so recalibrating means editing a number rather than the logic.
+`--nodes` is the piece most likely to need it.
+
+MIT.

slurmpast-0.2.0/pyproject.toml

@@ -0,0 +1,123 @@
+[build-system]
+requires = ["setuptools>=77", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "slurmpast"
+dynamic = ["version"]
+description = "Why did your Slurm jobs fail? A post-mortem dashboard for finished jobs."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [{ name = "Youzhi Yu", email = "yuyouzhi666@icloud.com" }]
+keywords = ["slurm", "hpc", "sacct", "postmortem", "job-failure", "tui", "seff"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Environment :: Console",
+  "Intended Audience :: Science/Research",
+  "Natural Language :: English",
+  "Operating System :: POSIX :: Linux",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Scientific/Engineering",
+  "Topic :: System :: Distributed Computing",
+  "Topic :: System :: Monitoring",
+  "Topic :: Utilities",
+  "Typing :: Typed",
+]
+dependencies = [
+  # Same floor as slurmwatch: >=0.86 is where the theme system landed, which the
+  # dashboard uses to register its own theme, and it covers DataTable/Header/
+  # Footer/Input. Matching the floor means the two tools resolve to one Textual
+  # in a shared environment instead of fighting over it.
+  "textual>=0.86,<9",
+  # Imported directly by render.py and tui.py (`rich.text.Text`), and render.py
+  # is on the plain-text path too, so `--plain` needs it as much as the
+  # dashboard does. It resolved anyway as a transitive dependency of textual,
+  # which is exactly why it was easy to miss -- a package you import belongs in
+  # your own dependency list, not borrowed from someone else's. Floor matches
+  # what textual itself requires.
+  "rich>=13.3.3",
+]
+
+[project.urls]
+Homepage = "https://github.com/PursuitOfDataScience/slurmpast"
+Repository = "https://github.com/PursuitOfDataScience/slurmpast.git"
+Documentation = "https://github.com/PursuitOfDataScience/slurmpast#readme"
+Issues = "https://github.com/PursuitOfDataScience/slurmpast/issues"
+
+[project.scripts]
+slurmpast = "slurmpast.cli:main"
+# "sp" is a short alias for the same entry point -- same behaviour, less typing.
+# Parallels slurmwatch's "sw"; verified free on PATH.
+sp = "slurmpast.cli:main"
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=7",
+  "pytest-asyncio>=0.23",
+  "pytest-cov>=4",
+  # Bounded on purpose: an unpinned linter makes CI a moving target, and a
+  # minor ruff release changing its rule defaults is exactly what broke the
+  # first CI run here.
+  "ruff>=0.15,<0.17",
+  "mypy>=2.3,<3",  # >=2.3: 2.2 added the MRO override check CI tripped on
+  "build>=1.0",
+  "twine>=5.0",
+]
+
+[tool.setuptools.dynamic]
+version = { attr = "slurmpast._version.__version__" }
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["slurmpast*"]
+
+[tool.setuptools.package-data]
+slurmpast = ["py.typed"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+# Pinned explicitly, matching slurmwatch. Without a `select` the gate is whatever
+# the installed ruff defaults to -- which is how CI failed on a commit that was
+# clean locally: 0.15 and 0.16 disagree about the default set, and 0.16 surfaced
+# 206 findings the local run never saw. An explicit list makes the gate mean the
+# same thing on every machine.
+select = ["E", "F", "W", "I", "N", "UP", "B", "SIM", "ARG", "C4", "T20"]
+# UP031 (%-format -> f-string) fires on 206 sites. Ruff auto-converts 113 of them
+# with all tests still passing, so the migration is safe -- but it also introduces
+# 28 line-length violations and leaves 93 sites for hand conversion. That is a
+# refactor, not a lint fix, and a half-migrated file is worse than a consistent
+# one. Recorded here rather than silently dropped.
+ignore = ["UP031"]
+
+[tool.ruff.lint.per-file-ignores]
+# A CLI writes to stdout; that is the point of it.
+"src/slurmpast/cli.py" = ["T201"]
+# Tests deliberately accept arguments they do not use (stub runners that must
+# match a real signature) and build fixtures with sacct's own CamelCase names.
+"tests/*" = ["ARG", "N802", "N803", "N806", "N815"]
+
+[tool.ruff.format]
+quote-style = "double"
+
+[tool.mypy]
+python_version = "3.10"
+ignore_missing_imports = true
+warn_redundant_casts = true
+warn_unreachable = true
+# NOT strict = true, which is what slurmwatch runs. Under --strict this codebase
+# reports 495 findings, almost all missing parameter/return annotations across
+# ~4,700 lines. That is worth doing, but it is an annotation project rather than
+# part of setting up CI, so the gate is set where it currently passes clean and
+# the gap is recorded here rather than silently skipped.
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+asyncio_mode = "strict"

slurmpast-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

slurmpast-0.2.0/src/slurmpast/__init__.py

@@ -0,0 +1,47 @@
+"""slurmpast — why did your Slurm jobs fail?
+
+A post-mortem for finished jobs. ``slurmate`` builds the request, ``slurmwatch``
+watches the run, this reads the wreckage.
+
+The analysis modules (``sacct``, ``diagnose``, ``patterns``, ``nodes``,
+``index``, ``site``) import no third-party package, so they are usable as a
+library on a login node without a UI framework. Textual is only pulled in by
+``tui``.
+
+Nothing here is specific to one cluster. Field names, output delimiters and
+timestamp formats are negotiated with the local ``sacct``; what a measurement
+means where sites differ (``JobAcctGatherType``, whether GPUs reach accounting at
+all) is read from ``scontrol show config`` by ``site``. See that module and the
+traps listed in ``sacct``.
+"""
+
+from ._version import __version__
+from .diagnose import diagnose, looks_like_noop
+from .index import GroupStats, History, build_groups
+from .model import CRITICAL, INFO, WARNING, Finding, Job, Step, Verdict
+from .sacct import Sacct, SacctError
+
+# Only the type, deliberately. Re-exporting the `site()` accessor here would bind
+# the name `slurmpast.site` to a function and shadow the submodule of the same
+# name, so `slurmpast.site.reset_cache` would stop resolving. Import it as
+# `from slurmpast.site import site`.
+from .site import Site
+
+__all__ = [
+    "__version__",
+    "Sacct",
+    "SacctError",
+    "Job",
+    "Step",
+    "Finding",
+    "Verdict",
+    "History",
+    "GroupStats",
+    "Site",
+    "build_groups",
+    "diagnose",
+    "looks_like_noop",
+    "CRITICAL",
+    "WARNING",
+    "INFO",
+]