gpu-dev 0.5.21__tar.gz → 0.5.23__tar.gz

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gpu-dev
-Version: 0.5.21
+Version: 0.5.23
 Summary: CLI tool for PyTorch GPU developer server reservations
 Author: PyTorch Team
 Requires-Python: >=3.10

gpu_dev-0.5.23/README.md

@@ -0,0 +1,143 @@
+# osdc — Open Source Developer Cloud
+
+A self-hosted developer platform for GPU work. Devs ask for `1 / 2 / 4 / 8`
+GPUs of a given type, the platform parks them on a Kubernetes pod with SSH
+access, and tears it down when the reservation expires.
+
+Built for PyTorch contributors — auth is via the GitHub public keys of users
+with commit access — but the design is generic enough to plug into other
+groups.
+
+## What you get
+
+- **Python CLI** (`gpu-dev`) with `reserve`, `list`, `extend`, `cancel`, and
+  `config` commands. Real-time polling until your pod is ready.
+- **GPU types**: T4, L4, A100, H100, B200. Pick the count (1, 2, 4, 8) and the
+  duration in hours (fractional is fine, e.g. `--hours 0.25`).
+- **SSH** straight into the pod via NodePort, with **your own GitHub public
+  keys** injected — no separate credentials to manage.
+- **Persistent disk** that survives between reservations (opt-in), backed by
+  EBS snapshots. Or run with `--no-persist` for a clean `EmptyDir` workspace.
+- **20 TB shared EFS** mounted at `/shared` with per-user folders.
+- **NVIDIA profiling** ready out of the box (`ncu` / `nsys` work without
+  manual driver tweaks), with one node per GPU type reserved as
+  profiling-dedicated.
+- **Grafana** dashboard at `<node-ip>:30080` with NVIDIA DCGM exporter
+  metrics — utilization, memory, temp, power.
+- **Multi-node NCCL** working over EFA with `OFI_NCCL_PROTOCOL=SENDRECV`.
+  Tree algo gets ~21 GB/s bus bandwidth across 2× p5.48xlarge (16 H100).
+
+## How it fits together
+
+```
+   ┌────────┐  reserve     ┌────────┐  enqueue  ┌────────────┐
+   │  CLI   │ ───────────► │   API  │ ────────► │    SQS     │
+   └────────┘              └────────┘           └─────┬──────┘
+        ▲ poll                                        │
+        │                                             ▼
+        │              ┌──────────────────────────────────────┐
+        │              │  Lambda  reservation processor       │
+        │              │  - pick a node with free GPUs        │
+        │              │  - attach EBS, mount /shared (EFS)   │
+        │              │  - create K8s pod, inject GH keys    │
+        │              └────────────────┬─────────────────────┘
+        │                               │
+        │                               ▼
+        │                     ┌──────────────────┐
+        │                     │    EKS (k8s)     │
+        │  SSH (NodePort)     │  GPU node groups │
+        └─────────────────────┤   T4 / L4 / H100 │
+                              │   B200 / ...     │
+                              └──────────────────┘
+
+   DynamoDB holds reservation state & history; CloudWatch logs the lambdas.
+```
+
+## Repository layout
+
+```
+.
+├── cli-tools/             # `gpu-dev` Python CLI (pyproject.toml)
+├── terraform-gpu-devservers/
+│                          # OpenTofu modules for EKS, node groups,
+│                          # SQS, Lambda, DynamoDB, EFS, monitoring
+├── admin/                 # operator scripts
+├── docs/                  # user guide and architecture notes
+└── tests/
+```
+
+## Getting started — as a user
+
+You need: GitHub access to the configured org (PyTorch by default), and your
+public keys uploaded to GitHub.
+
+```bash
+# 1. Install the CLI
+pip install -e ./cli-tools/gpu-dev-cli
+
+# 2. Point it at your deployment
+gpu-dev config        # walks you through API URL + GitHub username
+
+# 3. Reserve a GPU
+gpu-dev reserve -g 1 -t h100 -h 2          # 1× H100 for 2 hours
+gpu-dev reserve -g 8 -t b200 -h 24         # 8× B200 for a day
+gpu-dev reserve -g 1 -t t4  -h 0.25        # 1× T4 for 15 minutes
+
+# 4. Watch it come up; SSH instructions print when ready
+gpu-dev list
+
+# 5. Extend if you need more time (max total 48 h)
+gpu-dev extend <reservation-id> --hours 12
+
+# 6. Done? Free it up.
+gpu-dev cancel <reservation-id>
+```
+
+Each reservation drops an SSH config file at
+`~/.devgpu/<reservation_id>-sshconfig`, so connecting is just:
+
+```bash
+ssh -F ~/.devgpu/<reservation_id>-sshconfig gpu-dev
+```
+
+## Getting started — as an operator
+
+You need: an AWS account with EC2 GPU capacity (reserved or on-demand), an
+OpenTofu workstation, and credentials for whatever IAM role the modules
+assume.
+
+```bash
+cd terraform-gpu-devservers
+tf init                  # `tf` is aliased to `opentofu` in this repo
+tf plan                  # read-only — agents are restricted to this
+tf apply                 # only on a real workstation, not via the agent
+```
+
+Important variables to set in your `*.tfvars`:
+
+- `aws_region` (defaults to `us-east-2`)
+- node group sizing per GPU type (T4 / L4 / H100 / B200)
+- `grafana_admin_password`
+- the GitHub org/team that's allowed to reserve
+
+Once nodes are up, label one per GPU type as profiling-dedicated so DCGM
+doesn't fight Nsight for the device:
+
+```bash
+kubectl label node <h100-node> gpu.monitoring/profiling-dedicated=true
+kubectl label node <b200-node> gpu.monitoring/profiling-dedicated=true
+```
+
+Grafana lands at `http://<node-ip>:30080` (admin / your configured password).
+Pre-loaded dashboards: NVIDIA DCGM (community ID 12239) and a custom GPU
+overview.
+
+## Status
+
+Working end-to-end on T4 / L4 / H100. B200 supported with on-demand capacity.
+Active development — see [`PROGRESS.md`](PROGRESS.md) and [`TODO.md`](TODO.md)
+for what's in flight and what's queued.
+
+## License
+
+See [`LICENSE`](LICENSE) once added. For now: ask before reusing.

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/cli-tools/gpu-dev-cli/gpu_dev.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gpu-dev
-Version: 0.5.21
+Version: 0.5.23
 Summary: CLI tool for PyTorch GPU developer server reservations
 Author: PyTorch Team
 Requires-Python: >=3.10

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/cli-tools/gpu-dev-cli/gpu_dev.egg-info/SOURCES.txt

@@ -2,6 +2,7 @@
 CLAUDE.md
 PROGRESS.md
 PR_DESCRIPTION.md
+README.md
 TODO.md
 post.md
 pyproject.toml
@@ -44,6 +45,7 @@ terraform-gpu-devservers/efs.tf
 terraform-gpu-devservers/eks.tf
 terraform-gpu-devservers/expiry.tf
 terraform-gpu-devservers/git-cache.tf
+terraform-gpu-devservers/gpu-dev-pod-irsa.tf
 terraform-gpu-devservers/kubernetes.tf
 terraform-gpu-devservers/lambda.tf
 terraform-gpu-devservers/main.tf

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/cli-tools/gpu-dev-cli/gpu_dev_cli/config.py

@@ -240,8 +240,17 @@ class Config:
         return self.user_config.get(key)
 
     def get_github_username(self) -> Optional[str]:
-        """Get GitHub username from config."""
-        return self.user_config.get("github_user")
+        """Get GitHub username, falling back to GPU_DEV_GITHUB_USER env var.
+
+        Lambda sets GPU_DEV_GITHUB_USER on every pod from the reservation's
+        github_user field, so a user running gpu-dev from inside their dev pod
+        doesn\'t have to `gpu-dev config set github_user <name>` first.
+        """
+        v = self.user_config.get("github_user")
+        if v:
+            return v
+        v = os.environ.get("GPU_DEV_GITHUB_USER")
+        return v or None
 
 
 def load_config() -> Config:

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/cli-tools/gpu-dev-cli/gpu_dev_cli/interactive.py

@@ -89,6 +89,7 @@ def select_gpu_type_interactive(
     table = Table()
     table.add_column("GPU Type", style="cyan")
     table.add_column("Avail", style="green")
+    table.add_column("Max\nReservable", style="bright_green")
     table.add_column("Total", style="blue")
     table.add_column("Queue\nLength", style="yellow")
     table.add_column("Est. Wait Time", style="magenta")
@@ -96,6 +97,7 @@ def select_gpu_type_interactive(
     choices = []
     for gpu_type, info in visible_info.items():
         available = info.get("available", 0)
+        max_reservable = info.get("max_reservable", 0)
         total = info.get("total", 0)
         queue_length = info.get("queue_length", 0)
         est_wait = info.get("estimated_wait_minutes", 0)
@@ -134,6 +136,7 @@ def select_gpu_type_interactive(
         table.add_row(
             gpu_type.upper(),
             available_display,
+            "-" if is_maintenance else str(max_reservable),
             str(total),
             str(queue_length) if not is_maintenance else "-",
             wait_display,

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gpu-dev"
-version = "0.5.21"
+version = "0.5.23"
 description = "CLI tool for PyTorch GPU developer server reservations"
 authors = [{name = "PyTorch Team"}]
 readme = "cli-tools/gpu-dev-cli/README.md"

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/terraform-gpu-devservers/docker/Dockerfile

@@ -103,6 +103,8 @@ ENV NCCL_ASYNC_ERROR_HANDLING=1
 ENV SUPPORTS_EFA=true
 
 # Install Python packages (Jupyter and common ML packages)
+# gpu-dev itself is bundled so users can run `gpu-dev submit` from inside their pod
+# (combined with IRSA on the pod's service account, no manual aws sso login needed).
 RUN pip install --no-cache-dir --break-system-packages \
         jupyterlab \
         ipywidgets \
@@ -112,7 +114,8 @@ RUN pip install --no-cache-dir --break-system-packages \
         numpy \
         scikit-learn \
         plotly \
-        tensorboard
+        tensorboard \
+        gpu-dev
 
 # Create dev user with UID 1081 to avoid conflicts with common base image users (e.g., ubuntu=1000)
 RUN useradd -u 1081 -m -s /usr/bin/zsh dev && \

gpu_dev-0.5.23/terraform-gpu-devservers/gpu-dev-pod-irsa.tf

@@ -0,0 +1,88 @@
+# IRSA wiring for user-facing gpu-dev pods.
+#
+# Goal: when a user SSHs into their CPU dev pod (or any gpu-dev pod) and runs
+# `gpu-dev submit ...`, boto3 picks up temporary AWS credentials via the
+# IAM-roles-for-service-accounts mechanism — no manual `aws sso login` needed.
+#
+# Identity preservation: Lambda sets AWS_ROLE_SESSION_NAME=<user identity>
+# on the pod env, so STS GetCallerIdentity returns
+#   arn:aws:sts::<acct>:assumed-role/<role>/<user>
+# and the existing `authenticate_user` ARN-tail parsing keeps working unchanged.
+
+# Policy mirrors cli-tools/gpu-dev-cli/minimal-iam-policy.json — same scope a
+# user gets when they `aws sso login` from their laptop.
+resource "aws_iam_role" "gpu_dev_pod_role" {
+  name = "gpu-dev-pod-role-${local.current_config.environment}"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect = "Allow"
+        Principal = {
+          Federated = aws_iam_openid_connect_provider.eks.arn
+        }
+        Action = "sts:AssumeRoleWithWebIdentity"
+        Condition = {
+          StringEquals = {
+            "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub" = "system:serviceaccount:gpu-dev:gpu-dev-pod-sa"
+            "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:aud" = "sts.amazonaws.com"
+          }
+        }
+      }
+    ]
+  })
+
+  tags = {
+    Name        = "GPU Dev Pod IRSA Role"
+    Environment = local.current_config.environment
+  }
+}
+
+resource "aws_iam_role_policy" "gpu_dev_pod_policy" {
+  name = "gpu-dev-pod-policy"
+  role = aws_iam_role.gpu_dev_pod_role.id
+
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect = "Allow"
+        Action = [
+          "sqs:SendMessage",
+          "sqs:GetQueueUrl",
+          "sqs:GetQueueAttributes"
+        ]
+        Resource = "arn:aws:sqs:*:*:pytorch-gpu-dev-reservation-queue"
+      },
+      {
+        Effect = "Allow"
+        Action = [
+          "dynamodb:GetItem",
+          "dynamodb:Query",
+          "dynamodb:Scan"
+        ]
+        Resource = [
+          "arn:aws:dynamodb:*:*:table/pytorch-gpu-dev-reservations",
+          "arn:aws:dynamodb:*:*:table/pytorch-gpu-dev-reservations/index/*",
+          "arn:aws:dynamodb:*:*:table/pytorch-gpu-dev-gpu-availability"
+        ]
+      },
+      {
+        Effect = "Allow"
+        Action = "sts:GetCallerIdentity"
+        Resource = "*"
+      }
+    ]
+  })
+}
+
+resource "kubernetes_service_account" "gpu_dev_pod" {
+  metadata {
+    name      = "gpu-dev-pod-sa"
+    namespace = kubernetes_namespace.gpu_dev.metadata[0].name
+    annotations = {
+      "eks.amazonaws.com/role-arn" = aws_iam_role.gpu_dev_pod_role.arn
+    }
+  }
+}

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/terraform-gpu-devservers/lambda/reservation_processor/index.py

@@ -4486,6 +4486,18 @@ export MULTINODE_SIZE="$MULTINODE_SIZE"
 export MASTER_ADDR="$MASTER_ADDR"
 export MASTER_PORT="$MASTER_PORT"
 
+# IRSA + region — same reason as MULTINODE: sshd strips these from login shells, so
+# we bake the current container values into the rc file. Lets gpu-dev / aws / boto3
+# inside an SSH session pick up the gpu-dev-pod-sa IAM role automatically.
+export AWS_ROLE_ARN="$AWS_ROLE_ARN"
+export AWS_WEB_IDENTITY_TOKEN_FILE="$AWS_WEB_IDENTITY_TOKEN_FILE"
+export AWS_ROLE_SESSION_NAME="$AWS_ROLE_SESSION_NAME"
+export AWS_REGION="$AWS_REGION"
+export AWS_DEFAULT_REGION="$AWS_DEFAULT_REGION"
+export AWS_STS_REGIONAL_ENDPOINTS="$AWS_STS_REGIONAL_ENDPOINTS"
+# CLI falls back to this when ~/.config/gpu-dev/config.json has no github_user
+export GPU_DEV_GITHUB_USER="$GPU_DEV_GITHUB_USER"
+
 # Function to check for GPU reservation expiry warnings and startup script status
 check_warnings() {{
     # Check for startup script still running
@@ -4539,6 +4551,15 @@ export MULTINODE_SIZE="$MULTINODE_SIZE"
 export MASTER_ADDR="$MASTER_ADDR"
 export MASTER_PORT="$MASTER_PORT"
 
+# IRSA + region (see .bashrc_ext for rationale)
+export AWS_ROLE_ARN="$AWS_ROLE_ARN"
+export AWS_WEB_IDENTITY_TOKEN_FILE="$AWS_WEB_IDENTITY_TOKEN_FILE"
+export AWS_ROLE_SESSION_NAME="$AWS_ROLE_SESSION_NAME"
+export AWS_REGION="$AWS_REGION"
+export AWS_DEFAULT_REGION="$AWS_DEFAULT_REGION"
+export AWS_STS_REGIONAL_ENDPOINTS="$AWS_STS_REGIONAL_ENDPOINTS"
+export GPU_DEV_GITHUB_USER="$GPU_DEV_GITHUB_USER"
+
 # Function to check for GPU reservation expiry warnings and startup script status
 check_warnings() {{
     # Check for startup script still running
@@ -4577,6 +4598,16 @@ EOF_ZSHRC_EXT
                         chown 1081:1081 /home/dev/.bashrc_ext /home/dev/.zshrc_ext
                         echo "[STARTUP] ✓ Shell extension files written"
 
+                        # Background-refresh gpu-dev so older images / persistent disks pick up the
+                        # latest CLI without forcing the user to pip install it themselves. The
+                        # baseline gpu-dev is already in the image; this just upgrades.
+                        (
+                            pip install --no-cache-dir --break-system-packages --upgrade gpu-dev \
+                                > /tmp/gpu-dev-upgrade.log 2>&1 \
+                                && echo "[STARTUP] gpu-dev upgraded to $(gpu-dev --version 2>&1 | tail -1)" \
+                                || echo "[STARTUP] gpu-dev upgrade failed (non-fatal); see /tmp/gpu-dev-upgrade.log"
+                        ) &
+
                         # Ensure existing rc files source the extensions (for persistent disks with old configs)
                         for rcfile in /home/dev/.bashrc /home/dev/.zshrc; do
                             if [ -f "$rcfile" ]; then
@@ -5301,6 +5332,12 @@ EOF
                         ),
                         client.V1EnvVar(
                             name="NVIDIA_DRIVER_CAPABILITIES", value="compute,utility"
+                        ),
+                        client.V1EnvVar(
+                            name="AWS_ROLE_SESSION_NAME", value=(user_id or "gpu-dev-pod")[:64]
+                        ),
+                        client.V1EnvVar(
+                            name="GPU_DEV_GITHUB_USER", value=github_user or ""
                         )
                     ] + get_nccl_env_vars(gpu_type) + get_cpu_thread_env_vars(gpu_count, gpu_type) + _get_multinode_env_vars(multinode_peer_pods, multinode_rank),
                     resources=client.V1ResourceRequirements(
@@ -5483,6 +5520,15 @@ EOF
             ] if not gpu_type.startswith("cpu-") else [],
             # Faster pod deletion (default is 30s)
             termination_grace_period_seconds=10,
+            # IRSA: bind the pod to the gpu-dev-pod-sa service account so boto3 inside
+            # the pod gets temporary creds via STS AssumeRoleWithWebIdentity. Combined
+            # with the AWS_ROLE_SESSION_NAME env var below this lets users run
+            # `gpu-dev submit` from inside their dev pod with no manual aws sso login.
+            service_account_name="gpu-dev-pod-sa",
+            # fs_group=1081 makes the IRSA-projected token (default 0600 root:root)
+            # readable by the dev user. Without it boto3-as-dev falls through to IMDS
+            # and gets the node's IAM role, which doesn't have DDB/SQS permissions.
+            security_context=client.V1PodSecurityContext(fs_group=1081),
             # EFA requires host network namespace for RDMA access to efa0 interface
             **({
                 "host_network": True,

{gpu_dev-0.5.21 → gpu_dev-0.5.23}/terraform-gpu-devservers/lambda.tf

@@ -180,7 +180,7 @@ resource "aws_lambda_function" "reservation_processor" {
       HOSTED_ZONE_ID                     = local.effective_domain_name != "" ? local.hosted_zone_id : ""
       SSH_DOMAIN_MAPPINGS_TABLE          = local.effective_domain_name != "" ? aws_dynamodb_table.ssh_domain_mappings.name : ""
       SSL_CERTIFICATE_ARN                = local.effective_domain_name != "" ? aws_acm_certificate.wildcard[0].arn : ""
-      LAMBDA_VERSION                     = "0.5.22"
+      LAMBDA_VERSION                     = "0.5.24"
       MIN_CLI_VERSION                    = "0.5.16"
       DISK_CONTENTS_BUCKET               = aws_s3_bucket.disk_contents.bucket
       OPERATIONS_TABLE                   = aws_dynamodb_table.operations.name