gpu-dev 0.7.4__tar.gz → 0.7.6__tar.gz

{gpu_dev-0.7.4 → gpu_dev-0.7.6}/CLAUDE.md

@@ -79,7 +79,7 @@ Big push on warm pools + instant claims + prebuilt pytorch. Tracking state here
 - [ ] **Prebuilt built WITHOUT cuDNN** — `import torch` warns "compiled without cuDNN/MIOpen". CI/nightly build with cudnn9. Add libcudnn to the gpu-dev image + `USE_CUDNN=1` to the build recipe for fidelity (conv/cudnn-dependent ops + tests). Irrelevant for flex-attention int64 test; matters generally.
 - [ ] **`--ref pr/N` uses `pull/N/head`, not `/merge`** — `/head` is the PR author's raw branch tip (often based on old trunk, missing trunk-added tests); CI tests `/merge` (PR merged onto current trunk). For CI-repro fidelity, `pr/N` should fetch `pull/N/merge` (fall back to `/head` if no merge ref). `stage-pytorch` REF case in `index.py`. (This is why `pull/185479/head` lacked `test_large_kv_int64_pointer_math_cuda`.)
 - [ ] **Misleading disconnect/expiry message** — on `gpu-dev connect` connection loss OR reservation expiry, the CLI prints "❌ Authentication failed. You don't have SSH access... ask the primary user to add you" even for the PRIMARY user's own expired/cancelled reservation. Distinguish: (a) reservation expired -> "Reservation <id> expired at <time>"; (b) cancelled -> "Reservation was cancelled"; (c) connection dropped but still active -> "Connection lost, reconnect with gpu-dev connect <id>"; (d) genuine auth failure -> the current add-user message. Check reservation status before assuming auth failure.
-- [ ] **`gpu-dev cancel` from inside the pod** — show "Shutting down this reservation..." (graceful message) instead of an abrupt SSH drop, so the user knows the disconnect was intentional.
+- [x] **`gpu-dev cancel` from inside the pod** (DONE, 0.7.5) — two bugs: (1) cancel inside a **warm-claimed** pod failed with "GitHub username not configured" because the warm pod was pre-booted with `user_id="warm"` and the claim never stamped the real identity → `GPU_DEV_USER_ID/GPU_DEV_GITHUB_USER/AWS_ROLE_SESSION_NAME` stayed `"warm"`/empty. Fix: `try_claim_warm_pod` now seds the real `user_id`/`github_user` into both `.bashrc_ext`/`.zshrc_ext` + writes `GPU_DEV_RESERVATION_ID` (full id). Cold `_ext` derives `GPU_DEV_RESERVATION_ID` from the hostname (8-char prefix; cancellation resolves by prefix). (2) `gpu-dev cancel` (no id) inside a pod now fast-paths: cancels THIS reservation directly via `GPU_DEV_RESERVATION_ID`+`GPU_DEV_USER_ID` (no github_user/interactive) with the graceful "🛑 Shutting down..." message. Needs `tf apply` (lambda) + image rebuild (CLI in pods).
 - [ ] SSH CA certs to drop the ~0.33s `kubectl exec` key injection on warm claim (auth-model change).
 - [ ] AMI baker re-bakes on every base-EKS-AMI roll (5 baked AMIs in 2 days): pin the base AMI version + clean up old `gpu-dev-baked-*`.
 - [ ] **Warm pods: gate `warm-state=ready` on staging completion** (NOW MORE IMPORTANT — the built tree is ~30GB, and on GPU nodes it's a `cp` not reflink, so staging takes ~1-3min; a claim in that window hands over a half-copied tree). Two options: (a) claim-time check — exec `[ -f /home/dev/.pytorch-staging ]` in `try_claim_warm_pod`, skip pods still staging (simple, but adds ~0.5s exec to every warm claim); (b) label-flip — create with `warm-state=provisioning`, reconciler exec-checks staging + flips to `ready` (no claim latency, but 4 interacting changes: create label + reconciler flip + eviction must also target `provisioning` + claim already filters `ready`). Prefer (b). Marker: `.pytorch-staging` present during, removed when done; `.pytorch-ready` written at end.

{gpu_dev-0.7.4 → gpu_dev-0.7.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gpu-dev
-Version: 0.7.4
+Version: 0.7.6
 Summary: CLI + Python SDK for PyTorch GPU developer server reservations
 Author: PyTorch Team
 Requires-Python: >=3.10

{gpu_dev-0.7.4 → gpu_dev-0.7.6}/cli-tools/gpu-dev-cli/gpu_dev_cli/cli.py

@@ -1523,12 +1523,19 @@ def reserve(
 @click.option("--gpu-type", default="b200", show_default=True, help="GPU type for the repro box.")
 @click.option("--gpus", type=int, default=1, show_default=True)
 @click.option("--hours", type=float, default=3.0, show_default=True,
-              help="Lifetime ceiling; the box auto-cancels when the test exits unless --keep.")
+              help="Lifetime ceiling for the box.")
+@click.option("--no-connect", is_flag=True, default=False,
+              help="CI mode: run the test, auto-cancel, exit code = test result. Default (on a TTY) drops you into the box to iterate.")
 @click.option("--keep", is_flag=True, default=False,
-              help="Keep the reservation after the test exits (default: auto-cancel).")
+              help="Never cancel the box (skip the cancel prompt / auto-cancel).")
 @click.pass_context
-def repro(ctx, ref, test_args, gpu_type, gpus, hours, keep):
-    """Reserve a GPU, check out a PR/commit, run a test, then auto-cancel.
+def repro(ctx, ref, test_args, gpu_type, gpus, hours, no_connect, keep):
+    """Reserve a GPU, check out a PR/commit, run a test, then drop you into the box.
+
+    By default (in a terminal) repro runs the test and then **connects you into the
+    box** at ~/pytorch — the ref is checked out, so you can fix and re-run. The box
+    stays alive until you cancel it (you're prompted on exit). Use --no-connect for
+    CI/scripts (run the test, auto-cancel, process exit code = the test result).
 
     REF: pr/<N>, #<N>, a bare PR number, a branch, or a commit sha. PRs use
     pull/<N>/merge (what CI tests), falling back to /head.
@@ -1539,6 +1546,7 @@ def repro(ctx, ref, test_args, gpu_type, gpus, hours, keep):
     """
     import shlex
     import subprocess
+    import sys
     config = load_config()
     reservation_mgr = ReservationManager(config)
     try:
@@ -1602,21 +1610,55 @@ def repro(ctx, ref, test_args, gpu_type, gpus, hours, keep):
     if "StrictHostKeyChecking" not in ssh_cmd:
         ssh_cmd = ssh_cmd.replace("ssh ", "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR ", 1)
     rprint(f"[dim]→ {ssh_cmd}[/dim]\n")
+    rid8 = str(rid)[:8]
     rc = 1
     try:
         rc = subprocess.run(f"{ssh_cmd} {shlex.quote(remote)}", shell=True).returncode
     except KeyboardInterrupt:
-        rprint("\n[yellow]interrupted[/yellow]")
-    finally:
+        rprint("\n[yellow]interrupted[/yellow]"); rc = 130
+
+    verdict = "[green]✓ test passed[/green]" if rc == 0 else f"[red]✗ test failed (exit {rc})[/red]"
+
+    # Default (TTY): drop into the box so you can fix and re-run. --no-connect is the
+    # CI path: auto-cancel and exit with the test's code.
+    connect = (not no_connect) and sys.stdout.isatty()
+    if connect:
+        rprint(f"\n{verdict} — dropping you into the box at ~/pytorch ({ref} checked out).")
+        rprint(f"[dim]  re-run:  python {testcmd}[/dim]")
+        rprint(f"[dim]  finish:  gpu-dev cancel  (from inside)  •  or exit this shell[/dim]\n")
+        shell_cmd = f"{ssh_cmd} -t {shlex.quote('cd /home/dev/pytorch 2>/dev/null; exec ${SHELL:-bash} -l')}"
+        try:
+            subprocess.run(shell_cmd, shell=True)
+        except KeyboardInterrupt:
+            pass
         if keep:
-            rprint(f"[cyan]📌 kept {str(rid)[:8]} — gpu-dev connect {str(rid)[:8]} • gpu-dev cancel {str(rid)[:8]}[/cyan]")
-        else:
+            rprint(f"[cyan]📌 left {rid8} running — connect: gpu-dev connect {rid8} • cancel: gpu-dev cancel {rid8}[/cyan]")
+            return
+        try:
+            drop = click.confirm(f"Cancel repro box {rid8}?", default=True)
+        except (KeyboardInterrupt, EOFError, click.Abort):
+            drop = False
+        if drop:
             try:
                 reservation_mgr.cancel_reservation(rid, user_info["user_id"])
-                rprint(f"[green]🧹 cancelled repro box {str(rid)[:8]}[/green]")
+                rprint(f"[green]🧹 cancelled {rid8}[/green]")
             except Exception as e:
-                rprint(f"[yellow]auto-cancel failed for {str(rid)[:8]}: {e}[/yellow]")
-    rprint(f"\n[bold]repro exit code: {rc}[/bold]")
+                rprint(f"[yellow]cancel failed for {rid8}: {e}[/yellow]")
+        else:
+            rprint(f"[cyan]📌 left {rid8} running — connect: gpu-dev connect {rid8} • cancel: gpu-dev cancel {rid8}[/cyan]")
+        return
+
+    # --no-connect / non-TTY: auto-cancel unless --keep, exit code = test result.
+    if keep:
+        rprint(f"[cyan]📌 kept {rid8} — gpu-dev connect {rid8} • gpu-dev cancel {rid8}[/cyan]")
+    else:
+        try:
+            reservation_mgr.cancel_reservation(rid, user_info["user_id"])
+            rprint(f"[green]🧹 cancelled repro box {rid8}[/green]")
+        except Exception as e:
+            rprint(f"[yellow]auto-cancel failed for {rid8}: {e}[/yellow]")
+    rprint(f"\n[bold]repro exit code: {rc}[/bold] ({verdict})")
+    sys.exit(rc)
 
 
 _SUBMIT_GPU_TYPES = ["b300", "b200", "b200-mig-1g", "b200-mig-2g", "b200-mig-3g", "h200", "h100",
@@ -2668,6 +2710,24 @@ def cancel(
             rprint("[red]❌ Cannot specify both --all and a reservation ID[/red]")
             return
 
+        # Inside a gpu-dev pod, `gpu-dev cancel` (no id) shuts down THIS reservation
+        # directly. The pod knows its own reservation (GPU_DEV_RESERVATION_ID) and
+        # owner (GPU_DEV_USER_ID), so we skip the github_user / interactive list —
+        # which can't work in a pod that has no `gpu-dev config set github_user`.
+        pod_rid = os.environ.get("GPU_DEV_RESERVATION_ID", "").strip()
+        pod_uid = os.environ.get("GPU_DEV_USER_ID", "").strip()
+        if pod_rid and pod_uid and pod_uid != "warm" and not reservation_id and not all:
+            rprint("[yellow]🛑 Shutting down this reservation — if you're connected to this pod, your session will close shortly.[/yellow]")
+            try:
+                reservation_mgr = ReservationManager(load_config())
+                ok = reservation_mgr.cancel_reservation(pod_rid, pod_uid)
+            except Exception as e:
+                ok = False
+                rprint(f"[red]❌ Could not cancel from inside the pod: {e}[/red]")
+            if not ok:
+                rprint(f"[dim]If that didn't work, cancel from your laptop: gpu-dev cancel {pod_rid[:8]}[/dim]")
+            return
+
         # Handle --all flag (non-interactive)
         if all:
             with Live(

{gpu_dev-0.7.4 → gpu_dev-0.7.6}/docs/SDK_REPRO.md

@@ -2,7 +2,10 @@
 
 Reserve a **warm** GPU box in ~1s, run code on it, and auto-clean up — from Python
 or one CLI command. Backed by a pool of pre-booted pods with **PyTorch prebuilt**
-(viable/strict), so `import torch` works instantly with no build.
+(viable/strict), so `import torch` works instantly with no build. And when you *do*
+have to compile — your ref moved past viable/strict, or you touch C++ — a shared
+compiler cache (ccache) makes it an **incremental, not a cold, build** (see
+[Builds are cached](#builds-are-cached-shared-ccache)).
 
 > Requires `gpu-dev` ≥ 0.7.1 (CLI **and** SDK in one package): `pip install --upgrade gpu-dev`
 
@@ -32,14 +35,18 @@ with client.reserve(gpu_type="b200", gpu_count=1, hours=1) as sb:
 PyTorch is pre-staged at `~/pytorch` (importable). To reproduce a failure, point at
 the **PR or commit** and run the test.
 
-**One CLI command** (reserve → checkout → run → auto-cancel):
+**One CLI command** (reserve → checkout → run → **drop you into the box to fix**):
 ```bash
 gpu-dev repro pr/185264 test/inductor/test_flex_attention.py TestFlexAttentionCUDA.test_large_kv_int64_pointer_math_cuda
 ```
 - `REF`: `pr/<N>`, `#<N>`, a bare PR number, a branch, or a commit sha.
 - PRs use **`pull/<N>/merge`** (what CI actually tests — the PR merged onto current
   trunk), falling back to `/head`. Use this, not the raw branch.
-- `--keep` to inspect afterward instead of auto-cancelling.
+- By default (in a terminal) repro runs the test, prints the verdict, then **lands
+  you in the box** at `~/pytorch` with the ref checked out so you can fix and re-run;
+  it stays alive until you cancel (prompted on exit).
+- `--no-connect` = CI mode: run, auto-cancel, process exit code = the test result.
+- `--keep` never cancels (no prompt). `--gpu-type` / `--gpus` / `--hours` to size it.
 
 **From the SDK:**
 ```python
@@ -62,12 +69,48 @@ pip install -e . --no-build-isolation
 ```
 Python-only changes need no rebuild — `PYTHONPATH=~/pytorch` already resolves.
 
+## Builds are cached (shared ccache)
+
+Two layers of caching mean you almost never pay for a cold, from-scratch build —
+including the full C++/CUDA compile (gcc/nvcc):
+
+1. **Prebuilt tree.** Every box gets PyTorch already built at viable/strict and
+   staged at `~/pytorch`, so `import torch` works with **zero build**.
+2. **Shared compiler cache (ccache).** `CCACHE_DIR=/ccache_shared` is an EFS volume
+   mounted in **every** dev pod *and* the dedicated build node, so all the C++/CUDA
+   object compiles are cached and **shared across users and the build node**. When
+   you check out a ref past viable/strict — or edit C++ — the rebuild reuses those
+   cached objects (and the warm `build/` for ninja) instead of recompiling from
+   scratch. So even a "full" `pip install -e .` is a warm build, not a cold one.
+
+Measured (m7i build node, 128 jobs, CUDA 13.2):
+
+| scenario | time |
+|---|---|
+| `import torch` (prebuilt, no build) | ~0s |
+| incremental (1 kernel changed + relink) | ~40s |
+| ninja no-op (nothing changed) | ~20s |
+| from-scratch `build/` with warm ccache (~86% hit) | ~21 min |
+
+(A true cold build from an empty ccache is far longer.) The cache stays warm on its
+own: an hourly build-node job compiles each viable/strict bump into `/ccache_shared`,
+so the objects you need are usually already there by the time you build — and your
+own compiles populate it for the next person too.
+
 ## Gotchas
 - **`/merge` vs `/head`**: `/head` is the PR author's raw branch and often lacks
   trunk-added tests; `/merge` is what CI ran. `gpu-dev repro` / `--ref` use `/merge`.
 - **The prebuilt is viable/strict.** If your ref moved past it and a test needs new
-  C++, do the one incremental `pip install -e . --no-build-isolation`.
+  C++, do the one incremental `pip install -e . --no-build-isolation` — it's fast
+  (warm shared ccache), not a cold build. See [Builds are cached](#builds-are-cached-shared-ccache).
 - **Ephemeral by design.** Repro boxes have no persistent disk; bring code via
   `--ref`, `sb.upload`, or git.
+- **Reproducing a reverted PR / an OOM.** `pr/N` uses `/merge` = the PR re-applied
+  onto *current* trunk — so if the PR was reverted, `/merge` effectively un-reverts
+  it and you test the **fixed** tree (it'll pass). To repro the failing trunk state,
+  check out the **exact land commit** instead (`gpu-dev repro <sha> …`). And match the
+  CI runner's GPU: an **OOM** only reproduces on a GPU as small as the runner's — the
+  default `b200` has far more memory, so a memory-bound failure won't show there
+  (`--gpu-type h100`/`a100`/… to match).
 
 See also: `sdk/python/README.md` and `sdk/python/examples/`.

{gpu_dev-0.7.4 → gpu_dev-0.7.6}/gpu_dev.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gpu-dev
-Version: 0.7.4
+Version: 0.7.6
 Summary: CLI + Python SDK for PyTorch GPU developer server reservations
 Author: PyTorch Team
 Requires-Python: >=3.10

{gpu_dev-0.7.4 → gpu_dev-0.7.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gpu-dev"
-version = "0.7.4"
+version = "0.7.6"
 description = "CLI + Python SDK for PyTorch GPU developer server reservations"
 authors = [{name = "PyTorch Team"}]
 readme = "cli-tools/gpu-dev-cli/README.md"

{gpu_dev-0.7.4 → gpu_dev-0.7.6}/sdk/python/src/gpu_dev/__init__.py

@@ -63,4 +63,4 @@ try:
 
     __version__ = _pkg_version("gpu-dev")
 except Exception:
-    __version__ = "0.7.4"
+    __version__ = "0.7.6"

{gpu_dev-0.7.4 → gpu_dev-0.7.6}/terraform-gpu-devservers/lambda/reservation_processor/index.py

@@ -1633,7 +1633,18 @@ def try_claim_warm_pod(body: dict) -> bool:
             "while IFS= read -r k; do [ -n \"$k\" ] && ! grep -Fq \"$k\" /home/dev/.ssh/authorized_keys && echo \"$k\" >> /home/dev/.ssh/authorized_keys; done <<'KEOF'\n"
             f"{github_public_key}\n"
             "KEOF\n"
-            "chmod 700 /home/dev/.ssh && chmod 600 /home/dev/.ssh/authorized_keys && chown -R 1081:1081 /home/dev/.ssh"
+            "chmod 700 /home/dev/.ssh && chmod 600 /home/dev/.ssh/authorized_keys && chown -R 1081:1081 /home/dev/.ssh\n"
+            # Warm pods were pre-booted with user_id='warm'; stamp the real claimant's
+            # identity into the managed shell-ext files so `gpu-dev` inside the pod
+            # (cancel/list/...) authenticates as the user and IRSA assumes the right
+            # session. The user connects AFTER the claim, so their login shell picks
+            # these up. Also record the reservation id for `gpu-dev cancel`.
+            "for f in /home/dev/.bashrc_ext /home/dev/.zshrc_ext; do [ -f \"$f\" ] || continue\n"
+            f"  sed -i -e 's|^export GPU_DEV_USER_ID=.*|export GPU_DEV_USER_ID=\"{user_id}\"|'"
+            f" -e 's|^export GPU_DEV_GITHUB_USER=.*|export GPU_DEV_GITHUB_USER=\"{github_user}\"|'"
+            f" -e 's|^export AWS_ROLE_SESSION_NAME=.*|export AWS_ROLE_SESSION_NAME=\"{user_id}\"|' \"$f\"\n"
+            f"  grep -q '^export GPU_DEV_RESERVATION_ID=' \"$f\" && sed -i 's|^export GPU_DEV_RESERVATION_ID=.*|export GPU_DEV_RESERVATION_ID=\"{reservation_id}\"|' \"$f\" || echo 'export GPU_DEV_RESERVATION_ID=\"{reservation_id}\"' >> \"$f\"\n"
+            "done"
         )
         stream(
             v1.connect_get_namespaced_pod_exec, pod_name, "gpu-dev",
@@ -5272,6 +5283,11 @@ EOF_PROFILE
 
 # User identification
 export GPU_DEV_USER_ID="{user_id or 'dev'}"
+# Reservation id from the pod hostname; warm claims overwrite it with the full id,
+# cold pods keep the 8-char prefix. Used by gpu-dev cancel (no args) inside the pod.
+# NOTE: escape the dollar so this is evaluated when the shell sources the file, NOT
+# command-substituted while this (unquoted) heredoc is written at pod startup.
+export GPU_DEV_RESERVATION_ID="\$(hostname | sed -e 's/^gpu-dev-//')"
 
 # Multinode peer info — inlined from container env at pod startup. sshd strips
 # container env vars from login shells, so we materialize the values into rc files.
@@ -5338,6 +5354,11 @@ EOF_BASHRC_EXT
 
 # User identification
 export GPU_DEV_USER_ID="{user_id or 'dev'}"
+# Reservation id from the pod hostname; warm claims overwrite it with the full id,
+# cold pods keep the 8-char prefix. Used by gpu-dev cancel (no args) inside the pod.
+# NOTE: escape the dollar so this is evaluated when the shell sources the file, NOT
+# command-substituted while this (unquoted) heredoc is written at pod startup.
+export GPU_DEV_RESERVATION_ID="\$(hostname | sed -e 's/^gpu-dev-//')"
 
 # Multinode peer info — inlined from container env at pod startup. sshd strips
 # container env vars from login shells, so we materialize the values into rc files.