gpusched 0.3.0__tar.gz

gpusched-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 gpusched contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gpusched-0.3.0/PKG-INFO

@@ -0,0 +1,320 @@
+Metadata-Version: 2.4
+Name: gpusched
+Version: 0.3.0
+Summary: VRAM-aware single-node GPU job scheduler: queue shell commands, place them by free GPU memory, and verify declared-vs-actual VRAM per job.
+License: MIT
+Project-URL: Homepage, https://github.com/ceruleane/gpusched
+Project-URL: Issues, https://github.com/ceruleane/gpusched/issues
+Project-URL: Changelog, https://github.com/ceruleane/gpusched/blob/main/CHANGELOG.md
+Keywords: gpu,scheduler,vram,nvidia,queue,cuda
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# gpusched
+
+A VRAM-aware GPU job scheduler for a single machine. You write shell commands
+in a text file, optionally declaring how much GPU memory each needs; gpusched
+runs them, placing each job on a GPU with enough free memory the moment one
+opens up, measures how much VRAM each job actually used, and tells you when
+your declarations were wrong.
+
+```
+$ cat jobs.txt
+[vram=18G] python train.py --config a.yaml
+[vram=18G] python train.py --config b.yaml
+python preprocess.py
+
+$ gpusched jobs.txt --watch -v
+[14:02:10] job 1 started on gpu [0] (1/3 dispatched, declared 18432 MiB/gpu) — ...
+[14:02:10] job 2 started on gpu [1] (2/3 dispatched, declared 18432 MiB/gpu) — ...
+[14:31:44] job 1 finished [OK] in 1774s — peak vram gpu0:17910 MiB | declared 18432 MiB → within ±10% | avg gpu util 96%
+```
+
+## What this is, and what it is not
+
+gpusched is a **single-node, single-user research tool**: roughly a thousand
+lines you can read in an afternoon, with no daemon, no database, and zero
+Python dependencies. It exists for one workflow — you have a box with a few
+GPUs, a pile of training/eval commands, and you want them to run unattended
+with minimal GPU idle time and without OOM-ing each other.
+
+It is **not** a cluster scheduler, and several of its guarantees are honest
+best-effort rather than enforcement:
+
+- **Declarations are advisory.** gpusched places jobs based on what you
+  declare and warns loudly when reality diverges, but it cannot cap a
+  process's VRAM (nothing can, short of MIG partitions). A job that blows
+  through its declaration can still OOM a neighbor — you get a warning, not
+  protection. `--exclusive` (one scheduled job per GPU) is the zero-risk mode.
+- **It polls.** GPU state is sampled every `--poll` seconds (default 5) via
+  `nvidia-smi`. VRAM spikes shorter than the poll interval are invisible to
+  both placement and peak reporting.
+- **Co-locating compute-heavy jobs can be slower than serializing them.**
+  Packing optimizes VRAM occupancy, not throughput. Two trainers that each
+  saturate the SMs will roughly halve each other's speed; packing pays off
+  for memory-light or compute-light neighbors (eval scripts, dataloader-bound
+  jobs, inference). The per-job utilization report helps you see which case
+  you're in.
+- **Per-job VRAM attribution can fail in some container setups** where
+  `nvidia-smi` reports host-namespace PIDs that don't match the container's.
+  gpusched then reports the peak as `n/a` rather than a wrong number;
+  placement still works because it uses device-level totals.
+- **Testing honesty:** the test suite (65 tests) exercises the full scheduler
+  against simulated GPU backends and real subprocesses; the `nvidia-smi`
+  parsing layer is straightforward but thin. Do a small smoke run on your
+  hardware before trusting it with a week of compute.
+
+## If your needs are bigger than this
+
+Use the right tool instead of stretching this one:
+
+| Need | Look at |
+|---|---|
+| Multiple machines, multiple users, fairness, accounting | Slurm (or PBS/LSF) |
+| Hyperparameter optimization with early stopping / ASHA / PBT | Ray Tune, Optuna, W&B Sweeps |
+| Distributed training orchestration | Ray, torchrun + Slurm, Kubernetes + device plugins |
+| Workflow DAGs (job B consumes job A's output) | Snakemake, Makefile, Airflow |
+| Hard VRAM/compute isolation between jobs | NVIDIA MIG (partitioning), MPS (limits) |
+| Just a per-GPU FIFO queue, even simpler than this | task-spooler (`ts`), simple_gpu_scheduler |
+
+For sweeps specifically, the intended pattern is: a 10-line script (or
+Optuna in ask-and-tell mode) *generates* the jobs file; gpusched stays a dumb
+command queue. The moment you want trials stopped early based on metrics, you
+have outgrown this tool — that requires bidirectional communication with
+running jobs, which is deliberately out of scope.
+
+## Install
+
+```
+uv tool install gpusched        # isolated env, `gpusched` on PATH
+# or: pip install gpusched
+# or, from a clone: uv tool install .
+```
+
+Requires Python ≥ 3.10 and Linux with `nvidia-smi` (the scheduler itself has
+zero Python dependencies). `uvx gpusched jobs.txt` runs it without installing.
+
+## Sixty-second tour (no GPU required)
+
+```
+echo '[vram=8G] python3 -m gpusched.simjob --vram 8000 --ramp 2 --hold 5' > jobs.txt
+gpusched jobs.txt --sim 2 --poll 0.5 -v
+```
+
+`--sim N` runs against N simulated 24 GiB GPUs; `gpusched.simjob` is a fake
+GPU job that ramps and holds a declared amount of fake VRAM. Everything below
+behaves identically in sim and on real hardware.
+
+## The jobs file
+
+One shell command per line; blank lines and `#` comments are skipped. An
+optional leading `[...]` block declares per-job attributes:
+
+```
+# no declaration -> runs only on a fully idle GPU, alone
+python preprocess.py
+
+# declared max VRAM (per GPU): may share a GPU when the declared amount fits
+[vram=18000] python train.py --config a.yaml
+[vram=22G]   bash run_eval.sh
+
+# multi-GPU: 2 GPUs, EACH with >= 30 GiB free; CUDA_VISIBLE_DEVICES gets both
+[vram=30G gpus=2] torchrun --nproc_per_node=2 train_big.py
+
+# auto-retry on CUDA OOM, declaration bumped ~1.25x of observed peak per retry
+[vram=8G retries=2] python sweep.py --seed 3
+
+# opt-in walltime: SIGTERM at 2h, SIGKILL +10s. No timeout attribute = runs
+# forever; the scheduler never guesses which long-running jobs are hung.
+[timeout=2h] python flaky_eval.py
+```
+
+`vram` accepts MiB integers or `G`/`GiB` suffixes and is always per GPU.
+Declare your honest worst case; the completion report tells you how close you
+were, so declarations converge after a run or two.
+
+## How placement works
+
+A job **without** a declaration gets a GPU only when it is fully idle (below
+`--idle-threshold`, default 200 MiB, with no other scheduled job) — the safe
+default when you don't know what a job needs.
+
+A job **with** a declaration of E MiB can be placed wherever *effective
+headroom* ≥ E + `--margin` (default 512 MiB). Effective headroom accounts for
+three things. First, a just-launched job that hasn't allocated its CUDA
+context yet still reserves everything it declared — this closes the classic
+double-booking race in poll-based schedulers, where a GPU looks empty for the
+few seconds before a process materializes. Second, every process gpusched did
+not launch is tracked to its observed per-GPU peak and held to
+`peak × (1 + --spike-buffer)` until it exits, so a fluctuating external
+process's momentary trough is not treated as packable space. Third, a
+scheduled job that exceeds its own declaration stops being trusted: its
+budget escalates from the declaration to its buffered observed peak.
+
+Queue order is file order with backfill: if the next job can't fit right now,
+smaller jobs behind it run first. A job that could not fit even on a
+completely empty GPU fails immediately as `INFEASIBLE` rather than stalling
+the queue. Multi-GPU jobs take N distinct GPUs, each meeting the per-GPU
+requirement, chosen best-fit to preserve large contiguous headroom.
+
+(One physical reality worth knowing: PyTorch's caching allocator rarely
+returns VRAM to the driver, so for torch jobs `nvidia-smi` already reads near
+the high-water mark — the spike-buffer machinery matters most for processes
+that genuinely release memory between phases.)
+
+## Monitoring: declared vs actual
+
+Each scheduled job runs in its own session (`setsid`), so all its descendant
+processes share one process-group id; each poll, gpusched maps `nvidia-smi`'s
+per-process VRAM onto jobs by pgid and tracks per-GPU peaks. Two asymmetric
+notifications, tuned by `--tolerance` (default ±10%):
+
+**Under-declaration warns immediately** — the first poll where actual exceeds
+declared, you get `WARN job N EXCEEDS declared VRAM ... neighbors may OOM`,
+because at that moment the packing math other jobs were placed under is
+already violated. **Over-declaration is reported at completion** — a
+fluctuating job may legitimately peak late, so it can only be judged once it
+exits: `declared 12288 MiB → over-declared (-59%); lowering it frees packing
+headroom`.
+
+Every completion line is streamed the moment the job finishes and includes
+its per-GPU peak and average device-level GPU utilization (device-level: when
+two jobs share a GPU the number is confounded — treat it as a diagnostic for
+spotting dataloader-bound runs, not a per-process metric). `--verbose` adds
+live usage lines as a job's peak grows.
+
+## Live queue, resume, and the status board
+
+The jobs file is **user-owned and never written by the scheduler**; it is
+re-read every poll. Each line has a stable identity (hash of its command text
+plus an occurrence counter for duplicate lines), and an append-only journal
+(`<log_dir>/journal.jsonl`) records attempts and outcomes per identity.
+"Pending" is defined as: lines in the file that are neither running nor
+terminal in the journal. Everything follows from that one definition —
+append a line from any terminal and it is dispatched within a poll; delete a
+pending line and it is dequeued; reorder pending lines and you have reordered
+the queue (file order among pending IS the priority; there is no separate
+priority mechanism); edit a pending line and you have replaced it. Edits to
+running or completed lines do nothing. A malformed mid-edit save is rejected
+with a warning and the last good queue is kept; in-flight jobs are never
+affected.
+
+Re-running the same command after a crash or Ctrl-C skips everything the
+journal marks done — that is resume. `--fresh` wipes the journal to re-run
+all; to re-run one job, change its line trivially (new identity). `--watch`
+keeps the scheduler alive after the queue drains, waiting for appended lines.
+
+A live board is rendered to `<log_dir>/status.txt` every poll
+(`▶` running, `·` pending, `↻` retrying after OOM, `✓`/`✗` done):
+
+```
+watch -n2 cat gpusched_logs/status.txt
+```
+
+Known limitation: with continuous submission, a large blocked job can be
+starved by a stream of small backfilled ones. There is no aging policy — you
+are the priority mechanism. Move the big job's line up and hold small
+submissions, or run it under `--exclusive`.
+
+## OOM retry and timeouts
+
+A failed job whose log tail matches CUDA OOM signatures and that declared
+`[retries=N]` (or ran under `--oom-retries N`) is requeued instead of
+terminal-failed: its declaration is bumped to ~1.25× of max(observed peak,
+old declaration), recorded in the journal (so it survives scheduler
+restarts), and applied on the next attempt — the retry is scheduled with
+honest requirements instead of repeating the same collision. Non-OOM failures
+never consume retries, so retry loops cannot mask code bugs.
+
+Timeouts are strictly per-job opt-in (`[timeout=90s|15m|2h|1d]`).
+Distinguishing a hung process from a legitimate three-day run is your
+declaration, never a heuristic — heuristics (e.g. "0% util for 10 minutes")
+kill legitimate CPU phases like preprocessing and checkpoint serialization,
+so none are included.
+
+## Running detached (tmux)
+
+The scheduler is an ordinary foreground process — run it inside tmux and
+disconnect freely:
+
+```
+tmux new -s sched
+gpusched jobs.txt --watch -v            # pane 1
+# Ctrl-b c -> pane 2:
+watch -n2 cat gpusched_logs/status.txt
+# Ctrl-b d to detach; later: tmux attach -t sched
+```
+
+You can also append jobs over ssh without attaching at all — the file is
+re-read every poll. Failure-mode hierarchy, honestly: tmux protects
+everything from SSH drops; the journal protects queue state from scheduler
+death; nothing recovers *tracking* of jobs orphaned by a dead scheduler (they
+keep running in their own sessions, but their exit codes are lost and a
+restarted scheduler treats them as not-done).
+
+## CLI reference
+
+```
+gpusched jobs.txt
+  --gpus 0,1,3          restrict to these GPU indices (default: all visible)
+  --idle-threshold 200  MiB below which a GPU counts as idle (undeclared jobs)
+  --margin 512          MiB safety margin added to every declaration
+  --tolerance 0.10      band before flagging over/under-declaration
+  --spike-buffer 0.10   buffer over observed VRAM maxima of fluctuating processes
+  --poll 5              seconds between scheduling rounds
+  --exclusive           one scheduled job per GPU, even when declarations fit
+  --watch               keep running after drain; pick up appended lines
+  --oom-retries N       default CUDA-OOM auto-retries ([retries=N] overrides)
+  --fresh               ignore + remove the journal: re-run everything
+  --log-dir DIR         per-job logs, journal, status board (default: gpusched_logs)
+  -v / --verbose        stream live per-job VRAM as peaks grow
+  --sim N               dry-run on N simulated 24 GiB GPUs (no hardware)
+```
+
+Exit code: 0 if every job succeeded, otherwise the max failing job exit code.
+
+## Architecture and extending
+
+```
+src/gpusched/
+  jobspec.py     parsing: [vram=.. gpus=.. timeout=.. retries=..] cmd -> JobSpec
+  backend.py     GpuBackend protocol; NvidiaSmiBackend (2 queries/poll); pgid attribution
+  allocation.py  PURE placement function: headroom, reservations, best-fit
+  journal.py     append-only JSONL: attempts + terminal outcomes per job identity
+  scheduler.py   tick loop: snapshot -> attribute -> warn -> timeouts -> reap -> dispatch
+  testing.py     FakeBackend (unit tests), SimBackend (integration / --sim)
+  simjob.py      simulated GPU job for tests and dry runs
+  cli.py         argparse front end
+```
+
+The deliberate seams: `allocation.find_allocation` is pure (snapshot +
+occupants in, GPU list out), so new placement rules — GPU-type constraints,
+NVLink-aware pairing — are filters there plus an attribute in the jobspec
+parser, with nothing else touched. Alternative monitors (pynvml, DCGM)
+implement the two-method `GpuBackend` protocol. The journal is the only
+persistent state.
+
+## Development
+
+```
+git clone <repo> && cd gpusched
+uv venv && uv pip install -e ".[dev]"
+uv run pytest -q          # 65 tests, ~20s, no GPU required
+```
+
+Tests drive the scheduler against fake/simulated backends with real
+subprocesses; several are timing-based (sub-second sim jobs with fast polls),
+so a heavily loaded machine can occasionally need a re-run.
+
+## License
+
+MIT.

gpusched-0.3.0/README.md

@@ -0,0 +1,299 @@
+# gpusched
+
+A VRAM-aware GPU job scheduler for a single machine. You write shell commands
+in a text file, optionally declaring how much GPU memory each needs; gpusched
+runs them, placing each job on a GPU with enough free memory the moment one
+opens up, measures how much VRAM each job actually used, and tells you when
+your declarations were wrong.
+
+```
+$ cat jobs.txt
+[vram=18G] python train.py --config a.yaml
+[vram=18G] python train.py --config b.yaml
+python preprocess.py
+
+$ gpusched jobs.txt --watch -v
+[14:02:10] job 1 started on gpu [0] (1/3 dispatched, declared 18432 MiB/gpu) — ...
+[14:02:10] job 2 started on gpu [1] (2/3 dispatched, declared 18432 MiB/gpu) — ...
+[14:31:44] job 1 finished [OK] in 1774s — peak vram gpu0:17910 MiB | declared 18432 MiB → within ±10% | avg gpu util 96%
+```
+
+## What this is, and what it is not
+
+gpusched is a **single-node, single-user research tool**: roughly a thousand
+lines you can read in an afternoon, with no daemon, no database, and zero
+Python dependencies. It exists for one workflow — you have a box with a few
+GPUs, a pile of training/eval commands, and you want them to run unattended
+with minimal GPU idle time and without OOM-ing each other.
+
+It is **not** a cluster scheduler, and several of its guarantees are honest
+best-effort rather than enforcement:
+
+- **Declarations are advisory.** gpusched places jobs based on what you
+  declare and warns loudly when reality diverges, but it cannot cap a
+  process's VRAM (nothing can, short of MIG partitions). A job that blows
+  through its declaration can still OOM a neighbor — you get a warning, not
+  protection. `--exclusive` (one scheduled job per GPU) is the zero-risk mode.
+- **It polls.** GPU state is sampled every `--poll` seconds (default 5) via
+  `nvidia-smi`. VRAM spikes shorter than the poll interval are invisible to
+  both placement and peak reporting.
+- **Co-locating compute-heavy jobs can be slower than serializing them.**
+  Packing optimizes VRAM occupancy, not throughput. Two trainers that each
+  saturate the SMs will roughly halve each other's speed; packing pays off
+  for memory-light or compute-light neighbors (eval scripts, dataloader-bound
+  jobs, inference). The per-job utilization report helps you see which case
+  you're in.
+- **Per-job VRAM attribution can fail in some container setups** where
+  `nvidia-smi` reports host-namespace PIDs that don't match the container's.
+  gpusched then reports the peak as `n/a` rather than a wrong number;
+  placement still works because it uses device-level totals.
+- **Testing honesty:** the test suite (65 tests) exercises the full scheduler
+  against simulated GPU backends and real subprocesses; the `nvidia-smi`
+  parsing layer is straightforward but thin. Do a small smoke run on your
+  hardware before trusting it with a week of compute.
+
+## If your needs are bigger than this
+
+Use the right tool instead of stretching this one:
+
+| Need | Look at |
+|---|---|
+| Multiple machines, multiple users, fairness, accounting | Slurm (or PBS/LSF) |
+| Hyperparameter optimization with early stopping / ASHA / PBT | Ray Tune, Optuna, W&B Sweeps |
+| Distributed training orchestration | Ray, torchrun + Slurm, Kubernetes + device plugins |
+| Workflow DAGs (job B consumes job A's output) | Snakemake, Makefile, Airflow |
+| Hard VRAM/compute isolation between jobs | NVIDIA MIG (partitioning), MPS (limits) |
+| Just a per-GPU FIFO queue, even simpler than this | task-spooler (`ts`), simple_gpu_scheduler |
+
+For sweeps specifically, the intended pattern is: a 10-line script (or
+Optuna in ask-and-tell mode) *generates* the jobs file; gpusched stays a dumb
+command queue. The moment you want trials stopped early based on metrics, you
+have outgrown this tool — that requires bidirectional communication with
+running jobs, which is deliberately out of scope.
+
+## Install
+
+```
+uv tool install gpusched        # isolated env, `gpusched` on PATH
+# or: pip install gpusched
+# or, from a clone: uv tool install .
+```
+
+Requires Python ≥ 3.10 and Linux with `nvidia-smi` (the scheduler itself has
+zero Python dependencies). `uvx gpusched jobs.txt` runs it without installing.
+
+## Sixty-second tour (no GPU required)
+
+```
+echo '[vram=8G] python3 -m gpusched.simjob --vram 8000 --ramp 2 --hold 5' > jobs.txt
+gpusched jobs.txt --sim 2 --poll 0.5 -v
+```
+
+`--sim N` runs against N simulated 24 GiB GPUs; `gpusched.simjob` is a fake
+GPU job that ramps and holds a declared amount of fake VRAM. Everything below
+behaves identically in sim and on real hardware.
+
+## The jobs file
+
+One shell command per line; blank lines and `#` comments are skipped. An
+optional leading `[...]` block declares per-job attributes:
+
+```
+# no declaration -> runs only on a fully idle GPU, alone
+python preprocess.py
+
+# declared max VRAM (per GPU): may share a GPU when the declared amount fits
+[vram=18000] python train.py --config a.yaml
+[vram=22G]   bash run_eval.sh
+
+# multi-GPU: 2 GPUs, EACH with >= 30 GiB free; CUDA_VISIBLE_DEVICES gets both
+[vram=30G gpus=2] torchrun --nproc_per_node=2 train_big.py
+
+# auto-retry on CUDA OOM, declaration bumped ~1.25x of observed peak per retry
+[vram=8G retries=2] python sweep.py --seed 3
+
+# opt-in walltime: SIGTERM at 2h, SIGKILL +10s. No timeout attribute = runs
+# forever; the scheduler never guesses which long-running jobs are hung.
+[timeout=2h] python flaky_eval.py
+```
+
+`vram` accepts MiB integers or `G`/`GiB` suffixes and is always per GPU.
+Declare your honest worst case; the completion report tells you how close you
+were, so declarations converge after a run or two.
+
+## How placement works
+
+A job **without** a declaration gets a GPU only when it is fully idle (below
+`--idle-threshold`, default 200 MiB, with no other scheduled job) — the safe
+default when you don't know what a job needs.
+
+A job **with** a declaration of E MiB can be placed wherever *effective
+headroom* ≥ E + `--margin` (default 512 MiB). Effective headroom accounts for
+three things. First, a just-launched job that hasn't allocated its CUDA
+context yet still reserves everything it declared — this closes the classic
+double-booking race in poll-based schedulers, where a GPU looks empty for the
+few seconds before a process materializes. Second, every process gpusched did
+not launch is tracked to its observed per-GPU peak and held to
+`peak × (1 + --spike-buffer)` until it exits, so a fluctuating external
+process's momentary trough is not treated as packable space. Third, a
+scheduled job that exceeds its own declaration stops being trusted: its
+budget escalates from the declaration to its buffered observed peak.
+
+Queue order is file order with backfill: if the next job can't fit right now,
+smaller jobs behind it run first. A job that could not fit even on a
+completely empty GPU fails immediately as `INFEASIBLE` rather than stalling
+the queue. Multi-GPU jobs take N distinct GPUs, each meeting the per-GPU
+requirement, chosen best-fit to preserve large contiguous headroom.
+
+(One physical reality worth knowing: PyTorch's caching allocator rarely
+returns VRAM to the driver, so for torch jobs `nvidia-smi` already reads near
+the high-water mark — the spike-buffer machinery matters most for processes
+that genuinely release memory between phases.)
+
+## Monitoring: declared vs actual
+
+Each scheduled job runs in its own session (`setsid`), so all its descendant
+processes share one process-group id; each poll, gpusched maps `nvidia-smi`'s
+per-process VRAM onto jobs by pgid and tracks per-GPU peaks. Two asymmetric
+notifications, tuned by `--tolerance` (default ±10%):
+
+**Under-declaration warns immediately** — the first poll where actual exceeds
+declared, you get `WARN job N EXCEEDS declared VRAM ... neighbors may OOM`,
+because at that moment the packing math other jobs were placed under is
+already violated. **Over-declaration is reported at completion** — a
+fluctuating job may legitimately peak late, so it can only be judged once it
+exits: `declared 12288 MiB → over-declared (-59%); lowering it frees packing
+headroom`.
+
+Every completion line is streamed the moment the job finishes and includes
+its per-GPU peak and average device-level GPU utilization (device-level: when
+two jobs share a GPU the number is confounded — treat it as a diagnostic for
+spotting dataloader-bound runs, not a per-process metric). `--verbose` adds
+live usage lines as a job's peak grows.
+
+## Live queue, resume, and the status board
+
+The jobs file is **user-owned and never written by the scheduler**; it is
+re-read every poll. Each line has a stable identity (hash of its command text
+plus an occurrence counter for duplicate lines), and an append-only journal
+(`<log_dir>/journal.jsonl`) records attempts and outcomes per identity.
+"Pending" is defined as: lines in the file that are neither running nor
+terminal in the journal. Everything follows from that one definition —
+append a line from any terminal and it is dispatched within a poll; delete a
+pending line and it is dequeued; reorder pending lines and you have reordered
+the queue (file order among pending IS the priority; there is no separate
+priority mechanism); edit a pending line and you have replaced it. Edits to
+running or completed lines do nothing. A malformed mid-edit save is rejected
+with a warning and the last good queue is kept; in-flight jobs are never
+affected.
+
+Re-running the same command after a crash or Ctrl-C skips everything the
+journal marks done — that is resume. `--fresh` wipes the journal to re-run
+all; to re-run one job, change its line trivially (new identity). `--watch`
+keeps the scheduler alive after the queue drains, waiting for appended lines.
+
+A live board is rendered to `<log_dir>/status.txt` every poll
+(`▶` running, `·` pending, `↻` retrying after OOM, `✓`/`✗` done):
+
+```
+watch -n2 cat gpusched_logs/status.txt
+```
+
+Known limitation: with continuous submission, a large blocked job can be
+starved by a stream of small backfilled ones. There is no aging policy — you
+are the priority mechanism. Move the big job's line up and hold small
+submissions, or run it under `--exclusive`.
+
+## OOM retry and timeouts
+
+A failed job whose log tail matches CUDA OOM signatures and that declared
+`[retries=N]` (or ran under `--oom-retries N`) is requeued instead of
+terminal-failed: its declaration is bumped to ~1.25× of max(observed peak,
+old declaration), recorded in the journal (so it survives scheduler
+restarts), and applied on the next attempt — the retry is scheduled with
+honest requirements instead of repeating the same collision. Non-OOM failures
+never consume retries, so retry loops cannot mask code bugs.
+
+Timeouts are strictly per-job opt-in (`[timeout=90s|15m|2h|1d]`).
+Distinguishing a hung process from a legitimate three-day run is your
+declaration, never a heuristic — heuristics (e.g. "0% util for 10 minutes")
+kill legitimate CPU phases like preprocessing and checkpoint serialization,
+so none are included.
+
+## Running detached (tmux)
+
+The scheduler is an ordinary foreground process — run it inside tmux and
+disconnect freely:
+
+```
+tmux new -s sched
+gpusched jobs.txt --watch -v            # pane 1
+# Ctrl-b c -> pane 2:
+watch -n2 cat gpusched_logs/status.txt
+# Ctrl-b d to detach; later: tmux attach -t sched
+```
+
+You can also append jobs over ssh without attaching at all — the file is
+re-read every poll. Failure-mode hierarchy, honestly: tmux protects
+everything from SSH drops; the journal protects queue state from scheduler
+death; nothing recovers *tracking* of jobs orphaned by a dead scheduler (they
+keep running in their own sessions, but their exit codes are lost and a
+restarted scheduler treats them as not-done).
+
+## CLI reference
+
+```
+gpusched jobs.txt
+  --gpus 0,1,3          restrict to these GPU indices (default: all visible)
+  --idle-threshold 200  MiB below which a GPU counts as idle (undeclared jobs)
+  --margin 512          MiB safety margin added to every declaration
+  --tolerance 0.10      band before flagging over/under-declaration
+  --spike-buffer 0.10   buffer over observed VRAM maxima of fluctuating processes
+  --poll 5              seconds between scheduling rounds
+  --exclusive           one scheduled job per GPU, even when declarations fit
+  --watch               keep running after drain; pick up appended lines
+  --oom-retries N       default CUDA-OOM auto-retries ([retries=N] overrides)
+  --fresh               ignore + remove the journal: re-run everything
+  --log-dir DIR         per-job logs, journal, status board (default: gpusched_logs)
+  -v / --verbose        stream live per-job VRAM as peaks grow
+  --sim N               dry-run on N simulated 24 GiB GPUs (no hardware)
+```
+
+Exit code: 0 if every job succeeded, otherwise the max failing job exit code.
+
+## Architecture and extending
+
+```
+src/gpusched/
+  jobspec.py     parsing: [vram=.. gpus=.. timeout=.. retries=..] cmd -> JobSpec
+  backend.py     GpuBackend protocol; NvidiaSmiBackend (2 queries/poll); pgid attribution
+  allocation.py  PURE placement function: headroom, reservations, best-fit
+  journal.py     append-only JSONL: attempts + terminal outcomes per job identity
+  scheduler.py   tick loop: snapshot -> attribute -> warn -> timeouts -> reap -> dispatch
+  testing.py     FakeBackend (unit tests), SimBackend (integration / --sim)
+  simjob.py      simulated GPU job for tests and dry runs
+  cli.py         argparse front end
+```
+
+The deliberate seams: `allocation.find_allocation` is pure (snapshot +
+occupants in, GPU list out), so new placement rules — GPU-type constraints,
+NVLink-aware pairing — are filters there plus an attribute in the jobspec
+parser, with nothing else touched. Alternative monitors (pynvml, DCGM)
+implement the two-method `GpuBackend` protocol. The journal is the only
+persistent state.
+
+## Development
+
+```
+git clone <repo> && cd gpusched
+uv venv && uv pip install -e ".[dev]"
+uv run pytest -q          # 65 tests, ~20s, no GPU required
+```
+
+Tests drive the scheduler against fake/simulated backends with real
+subprocesses; several are timing-based (sub-second sim jobs with fast polls),
+so a heavily loaded machine can occasionally need a re-run.
+
+## License
+
+MIT.

gpusched-0.3.0/pyproject.toml

@@ -0,0 +1,34 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "gpusched"
+version = "0.3.0"
+description = "VRAM-aware single-node GPU job scheduler: queue shell commands, place them by free GPU memory, and verify declared-vs-actual VRAM per job."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+keywords = ["gpu", "scheduler", "vram", "nvidia", "queue", "cuda"]
+classifiers = [
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Topic :: System :: Distributed Computing",
+]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/ceruleane/gpusched"
+Issues = "https://github.com/ceruleane/gpusched/issues"
+Changelog = "https://github.com/ceruleane/gpusched/blob/main/CHANGELOG.md"
+
+[project.optional-dependencies]
+dev = ["pytest>=7"]
+
+[project.scripts]
+gpusched = "gpusched.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

gpusched-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+