swm-gpu 0.2.4__tar.gz → 0.2.6__tar.gz

{swm_gpu-0.2.4 → swm_gpu-0.2.6}/.agents/skills/swm-gpu-workflow/SKILL.md

@@ -11,7 +11,7 @@ license: Apache-2.0
 compatibility: "macOS or Linux, Python 3.11+, SSH client"
 metadata:
   author: swm-gpu
-  version: "0.2.3"
+  version: "0.2.4"
   docs-url: https://swmgpu.com
   repository: https://github.com/swm-gpu/swm
 ---
@@ -38,10 +38,11 @@ One CLI to search, provision, install, sync, and tear down GPU pods across 10 cl
 ## Phase 0 — State check
 
 ```bash
-swm pod list && swm use --show
+swm pod list && swm use --show               # running pods + active-pod selection
+swm storage ls                               # existing workspaces in the default bucket
 ```
 
-If a matching pod is `RUNNING`, set it active (`swm use <name>`) and skip to Phase 5. Otherwise continue.
+If a matching pod is `RUNNING`, set it active (`swm use <name>`) and skip to Phase 5. If a workspace prefix you want already exists in `swm storage ls`, restore it with `pod create -w <name>`. Otherwise continue.
 
 ## Phase 1 — Clarify
 
@@ -144,10 +145,11 @@ swm models pull <pod> ollama:llama3                              # Ollama
 swm models pull <pod> Comfy-Org/Wan_2.2_ComfyUI_Repackaged \
   --as lora --filename split_files/loras/wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors
 
-swm models info <ref>             # metadata before pulling
+swm models search <query> [--sort downloads] [-n 20]   # find models on HuggingFace Hub
+swm models info <ref>             # metadata before pulling (HF or Civitai)
 swm models list <pod> [--all]     # tracked (+ untracked) files
 swm models link <pod> /abs/path/to/file --as vae
-swm models remove <pod> <name>
+swm models remove <pod> <name> [-y]
 ```
 
 API keys (set once; legacy `hf_token` also read):
@@ -217,41 +219,75 @@ Unknown provider slugs raise `UsageError` listing valid ones.
 swm pod create -p runpod -g h200 -n s1 -y
 swm pod create -p vastai -g h200 -n train --gpu-count 4 --volume 200 --cloud-type SECURE -y
 swm pod list                                  # auto-prunes stale entries vs live provider state
-swm pod status / stop / start / down / prune <pod>
+swm pod status  <pod>                          # GPU, cost, uptime + live sync health (autosync/watcher/last-push)
+swm pod stop    <pod>                          # halt, preserve volume (resume with pod start)
+swm pod start   <pod>                          # resume a stopped pod, wait for SSH
+swm pod prune                                  # drop config entries for pods gone from the provider
 ```
 
+**Teardown — two distinct commands:**
+
+```bash
+swm pod down <pod>                             # SAFE: push /workspace → storage, then terminate
+swm pod down <pod> --no-sync                   #   skip the push (workspace already synced)
+swm pod down <pod> --force-down                #   terminate even if push left no recent stamp (DANGEROUS)
+swm pod terminate <pod> -y                     # DESTRUCTIVE: delete pod + volume, NO sync (irreversible)
+```
+
+`pod down` is the everyday teardown — it refuses to wipe `/workspace` or terminate unless the push wrote a fresh push stamp (guards against silent data loss). `pod terminate` skips all sync and deletes the volume; only use it for throwaway / `--no-storage` pods.
+
 ## Sync
 
 ```bash
-swm sync pull  [pod]              # storage → pod
+swm sync pull  [pod]              # storage → pod (non-destructive: skips existing, never deletes)
 swm sync pull  [pod] --tar        # tarball mode (600k+ small files; fails loudly on error)
-swm sync push  [pod]              # pod → storage
-swm sync watch [pod]              # inotify auto-push
-swm sync status [pod]             # s5cmd, watcher, autosync, pending changes
+swm sync push  [pod]              # pod → storage (incremental)
+swm sync push  [pod] --force      # kill any running transfer, re-baseline from scratch
+swm sync push  [pod] --delete     # also delete remote files removed locally (watcher must be alive)
+swm sync push  [pod] --tar        # pack workspace to one tarball object (fast for many small files)
+swm sync status [pod]             # s5cmd, watcher, autosync daemon, last-push stamp, pending changes
 ```
 
-Three-tier sync: inotify watcher → incremental s5cmd → tar. Push reconciles inotify events with a `find -newer` scan so `pip install` bursts aren't missed. Pull re-runs framework link-repair steps so `/workspace/models/` symlinks stay wired after restore.
+**Watcher vs daemon — two layers:**
+
+```bash
+swm sync watch [pod]              # start the inotify file-change watcher only (records what changed)
+swm sync watch [pod] --stop       # stop the watcher
+swm sync auto  [pod]              # start the background daemon: reads watcher log every 60s, pushes
+swm sync auto  [pod] -i 30        # custom interval (seconds)
+swm sync auto  [pod] --status     # daemon state + recent log tail
+swm sync auto  [pod] --stop       # stop the daemon
+swm sync auto  [pod] --force      # start without a prior successful pull/push (DANGEROUS — see below)
+```
+
+`swm pod create -w <name>` wires up the watcher + auto daemon for you; reach for `sync watch` / `sync auto` only to inspect or recover. **`sync auto --force` is dangerous**: it bypasses the safety check that requires a known-good baseline, so local deletions can propagate to storage. Only force when you've just `sync push --force`ed and know the pod is authoritative.
+
+Three-tier sync: inotify watcher → incremental s5cmd → tar. Push reconciles inotify events with a `find -newer` scan so `pip install` bursts aren't missed (and falls back to a full `find` count when the watcher is dead). Pull re-runs framework link-repair steps so `/workspace/models/` symlinks stay wired after restore.
 
 ## Guard
 
 ```bash
-swm guard set <pod> --mode auto-down --idle-timeout 30   # set/update policy
+swm guard set <pod> --mode auto-down --idle-timeout 30   # set/update policy (auto-down/auto-stop/remind/manual)
 swm guard set <pod> --mode remind                        # notify-only
 swm guard disable <pod> [--force-manual]                 # remove override / pin manual
-swm guard defaults --mode auto-down --idle-timeout 30    # for new pods
-swm guard list                                           # live status
-swm guard stop-daemon
+swm guard defaults --mode auto-down --idle-timeout 30    # default policy applied to new pods
+swm guard list                                           # all guarded pods + live idle status
+swm guard run [pods…] [--once] [--interval N]            # run the loop in foreground (debug/cron)
+swm guard stop-daemon                                    # stop the background daemon
 ```
 
-Monitors 4 signals: SSH sessions (`who`), GPU utilization, filesystem writes, active processes. Setting `auto-stop`/`auto-down` auto-starts the local daemon. `--idle-timeout 0` and `--poll-interval 0` are honored as explicit values (not unset).
+Monitors 4 signals: SSH sessions (`who`), GPU utilization, filesystem writes, active processes. Setting `auto-stop`/`auto-down` auto-starts the local daemon; `guard run` is the explicit foreground runner (use `--once` for a single sweep). `--idle-timeout 0` and `--poll-interval 0` are honored as explicit values (not unset). `auto-down` runs the same push-stamp safety check as `pod down` — it leaves the pod alive rather than terminate on a partial push.
 
 ## Costs
 
 ```bash
-swm costs live                                # right-now burn
+swm costs live                                # right-now burn across active pods
 swm costs summary [--period today|...]        # breakdown by provider/pod (today = UTC day)
-swm costs reconcile                           # verify vs provider billing (RunPod, Vast.ai)
-swm costs budget set 100                      # $/month alert
+swm costs log [-n 50] [-p vastai]             # detailed per-session history
+swm costs reconcile [-p vastai]               # compare local tracking vs provider card charges
+swm costs budget set 100 [-s global] [-t month]   # set a spending alert
+swm costs budget show                         # active budgets + current spend
+swm costs budget remove <scope> [-t month]    # remove a budget
 ```
 
 ## Remote
@@ -266,8 +302,9 @@ swm download [pod] remote/file -d ~/Downloads
 ## Pricing reference
 
 ```bash
-swm pricing compare  --gpu h100|h200|b200|a100|rtx-4090|l40s
-swm pricing estimate -g h200 --hours 24
+swm pricing compare  --gpu h100|h200|b200|a100|rtx-4090|l40s   # per-GPU/hr across providers
+swm pricing estimate -g h200 --hours 24 [--storage GB] [--tier secure]   # projected workload cost
+swm pricing specs    --gpu h200                                 # hardware specs (VRAM, bandwidth) side-by-side
 ```
 
 ## Instance ID format
@@ -289,18 +326,70 @@ swm pricing estimate -g h200 --hours 24
 | `azure` | az CLI (`pip install swm-gpu[azure]`) | yes |
 | `coreweave` | Kubernetes | yes |
 
-## Storage
+## Storage (buckets & workspaces)
+
+Inspect and manage cloud-storage buckets directly — **no running pod required**. Top-level prefixes in your default bucket are your workspace names, so `swm storage ls` is the fastest "what's in my bucket?" probe.
+
+```bash
+swm storage list                              # configured buckets across all providers
+swm storage ls                                # default bucket root = workspace names
+swm storage ls studio-live/                   # drill into one workspace (sizes/mtimes show here)
+swm storage ls -b b2:other-bucket             # explicit provider:bucket
+swm storage create my-models -p b2            # create a bucket (-l location, -c storage-class)
+swm storage upload   ./file remote/path       # one-off upload
+swm storage download remote/path ./file       # one-off download
+swm storage rm workspace2/ --dry-run          # COUNT objects under a prefix, delete nothing
+swm storage rm workspace2/ -y                 # delete all objects under prefix (S3 batch delete)
+```
+
+`storage rm` uses the S3 batch-delete API (1000 objects/request) and prompts before deleting unless `-y`. It's a hard delete on S3-compatible backends (B2/S3) — always `--dry-run` first. Top-level `ls` shows `DIR` with no size/mtime (B2 `CommonPrefixes` carry neither); drill into a prefix for real sizes.
 
-S3-compatible via s5cmd. Config keys:
+Backend config keys (S3-compatible via s5cmd):
 
 | Backend | Keys |
 |---|---|
-| Backblaze B2 | `b2.key_id`, `b2.app_key`, `b2.bucket` |
+| Backblaze B2 | `b2.key_id`, `b2.app_key`, `b2.bucket`, `b2.s3_endpoint` |
 | Amazon S3 | `s3.access_key`, `s3.secret_key`, `s3.bucket` |
 | Google GCS | `gcs.hmac_access`, `gcs.hmac_secret`, `gcp.bucket` |
 
 Default: `swm config set storage.default b2:<bucket>`. Credentials are masked by suffix in `config get/list` and never written to the pod (passed as transient env vars per s5cmd call).
 
+## Frameworks & workspace attach (setup)
+
+```bash
+swm setup list                                # available frameworks
+swm setup install <name> [pod]                # idempotent install into its own venv
+swm setup start   <name> [pod] [-p port] [--model M] [--extra-args "…"]
+swm setup stop    <name> [pod]
+swm setup storage [pod] [-p b2|gcs|s3|all]    # install s5cmd + configure creds on an existing pod
+swm setup workspace [pod] [-n name] [-b provider:bucket] [--force]
+```
+
+**`swm setup workspace` is the recovery command** for the silent-data-loss scenario: it does the full workspace bootstrap in one shot (installs s5cmd, configures creds, pulls or initializes the workspace, starts the auto-sync daemon) and persists the pod ↔ workspace ↔ bucket association. Use it when:
+- you created a pod with `--no-storage` and now want persistence,
+- `pod create`'s SSH probe timed out so bootstrap was skipped,
+- you want to reattach a pod to a different workspace (`--force`).
+
+## Config
+
+```bash
+swm config set <key> <value>                  # e.g. swm config set runpod.api_key sk-xxx
+swm config get <key>                          # single value (secrets suffix-masked)
+swm config list                               # all values (secrets masked)
+swm config delete <key>                       # remove a key
+swm config path                               # location of the config file
+```
+
+Common keys: provider `<slug>.api_key`; storage `b2.*` / `s3.*` / `gcs.*` + `storage.default`; model tokens `hf.api_key`, `civitai.api_key`; guard `guard.*` defaults.
+
+## Images
+
+```bash
+swm images list [-p provider] [--cuda 12.8] [--refresh] [-n 30] [--all]
+```
+
+Lists Docker images discoverable for a provider (newest-first) for use with `swm pod create --image`. `--refresh` re-queries the provider instead of using cached results.
+
 ---
 
 ## Anti-patterns
@@ -314,4 +403,7 @@ Default: `swm config set storage.default b2:<bucket>`. Credentials are masked by
 - **Never tear down before `du -sh /workspace`** — 10–30 GB bloat extends the next pull.
 - **Never guess nested HF file paths** — run `swm models info <repo>` and copy the exact `rfilename`.
 - **Never skip Phase 0** — a forgotten running pod bills silently.
+- **Never confuse `pod down` with `pod terminate`** — `down` syncs `/workspace` first; `terminate` deletes the volume with no sync. Use `terminate` only on `--no-storage` / throwaway pods.
+- **Never trust a green `sync push` alone** — confirm `swm pod status` shows a recent last-push stamp and autosync running before tearing down. A push can exit 0 yet skip the stamp on a partial transfer.
+- **Never use `swm storage rm` without `--dry-run` first** — it's a hard S3 batch delete with no undo.
 - **Never hand off** before Phase 5 passes.

{swm_gpu-0.2.4 → swm_gpu-0.2.6}/CHANGELOG.md

@@ -6,6 +6,45 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.2.6] - 2026-06-16
+
+### Fixed
+- **Auto-down never fired on workspace-backed pods (cost bug).** The lifecycle
+  guard's filesystem-activity signal keyed off the mtime of the autosync watcher
+  log (`/workspace/.swm_changes.log`), which autosync rewrites every cycle — so
+  pods were reported perpetually "active," the idle timer never elapsed, and
+  `auto-down`/`auto-stop` never triggered. The guard now detects *real*
+  workspace writes via `find`, excluding swm bookkeeping (`.swm_*`,
+  `.swm_guard`), build caches (`.uv-cache`, `.cache`, `__pycache__`), logs,
+  `terminfo`, `.git`, and `.nv`.
+- `swm setup start` no longer crashes with a traceback when the post-start
+  health probe hits a connection reset. `_probe_url` caught `ConnectError`/
+  `TimeoutException` but not `httpx.ReadError` ("connection reset by peer" while
+  a freshly-opened SSH tunnel settles); it now catches all `httpx.HTTPError`.
+
+### Changed
+- Workspace sync now excludes `/.uv-cache/` (uv build/wheel cache) and
+  `/terminfo/` (managed-CPython terminal database). The former churned
+  constantly and caused delete-reconciliation failures and ~11 GB of wasted
+  storage; the latter's dedup-hardlinked files broke hardlink staging.
+
+### Docs
+- Rewrote the `swm-gpu-workflow` skill for full command coverage — added the
+  entire `swm storage` group, `swm config`, `swm images`, `swm sync auto`,
+  `swm setup workspace`/`storage`, `pod terminate` vs `pod down`, and more.
+
+## [0.2.5] - 2026-05-29
+
+### Fixed
+- Framework SSH tunnels (e.g. `swm setup start comfyui`) now use the
+  configured SSH identity. `_open_tunnel` built a bare `ssh` command without
+  `-i <key_path>`, so when a custom `ssh.key_path` (or `<provider>.ssh_key`)
+  was set it silently fell back to the default key, failed authentication, and
+  — because of `ExitOnForwardFailure=yes` with output discarded — exited
+  immediately without forwarding the port. The tunnel now resolves its
+  host/port/user/key via the same path as `swm run`/`swm ssh`, so it also
+  targets the direct IP and mapped port-22 endpoint when available.
+
 ## [0.2.4] - 2026-05-27
 
 ### Fixed

{swm_gpu-0.2.4 → swm_gpu-0.2.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: swm-gpu
-Version: 0.2.4
+Version: 0.2.6
 Summary: One CLI to search, provision, and manage cloud GPUs across 10 providers
 Project-URL: Homepage, https://swmgpu.com
 Project-URL: Repository, https://github.com/swm-gpu/swm

{swm_gpu-0.2.4 → swm_gpu-0.2.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "swm-gpu"
-version = "0.2.4"
+version = "0.2.6"
 description = "One CLI to search, provision, and manage cloud GPUs across 10 providers"
 readme = "README.md"
 requires-python = ">=3.11"

swm_gpu-0.2.6/src/swm/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.6"

{swm_gpu-0.2.4 → swm_gpu-0.2.6}/src/swm/commands/_helpers.py

@@ -180,29 +180,36 @@ def _open_tunnel(inst, ports: dict[int, str]) -> list[int] | None:
     """
     import subprocess
 
+    from swm.remote.ssh import _ssh_config_for
+
     if not inst.ssh_host:
         return None
     unmapped = [p for p in ports if p not in (inst.ports or {})]
     if not unmapped:
         return None
 
-    ssh_port = inst.ssh_port or 22
-    user = inst.ssh_user or "root"
+    # Resolve host/port/user/key the same way `swm run`/`swm ssh` do, so the
+    # tunnel uses the configured identity (e.g. ssh.key_path) and the direct
+    # IP + mapped port-22 when available. Building a bare `ssh` command here
+    # silently fell back to default keys and could miss a custom key.
+    c = _ssh_config_for(inst)
 
-    cmd = [
-        "ssh", "-N",
+    cmd = ["ssh", "-N"]
+    if c["key_path"]:
+        cmd.extend(["-i", str(c["key_path"])])
+    cmd.extend([
         "-o", "StrictHostKeyChecking=no",
         "-o", "UserKnownHostsFile=/dev/null",
         "-o", "LogLevel=ERROR",
         "-o", "ExitOnForwardFailure=yes",
         "-o", "ServerAliveInterval=30",
         "-o", "ServerAliveCountMax=3",
-    ]
+    ])
     for p in unmapped:
         cmd.extend(["-L", f"{p}:localhost:{p}"])
-    if ssh_port != 22:
-        cmd.extend(["-p", str(ssh_port)])
-    cmd.append(f"{user}@{inst.ssh_host}")
+    if c["port"] != 22:
+        cmd.extend(["-p", str(c["port"])])
+    cmd.append(f"{c['user']}@{c['host']}")
 
     subprocess.Popen(
         cmd,
@@ -226,7 +233,12 @@ def _probe_url(url: str, timeout: int = 60) -> bool:
             r = httpx.get(url, timeout=5, follow_redirects=True)
             if r.status_code < 500:
                 return True
-        except (httpx.ConnectError, httpx.TimeoutException, httpx.RemoteProtocolError, OSError):
+        except (httpx.HTTPError, OSError):
+            # httpx.HTTPError covers all transport-level failures —
+            # ConnectError, ReadError ("connection reset by peer" while a
+            # freshly-opened SSH tunnel settles), WriteError, TimeoutException,
+            # RemoteProtocolError, etc. A probe must never crash the command;
+            # an unreachable URL just means "not ready yet, retry".
             pass
         remaining = deadline - time.monotonic()
         time.sleep(min(interval, max(remaining, 0)))

{swm_gpu-0.2.4 → swm_gpu-0.2.6}/src/swm/guard.py

@@ -81,10 +81,21 @@ def avg_gpu_util() -> float:
 
 
 def recent_fs_write(now: float) -> bool:
-    try:
-        return os.path.getmtime(WATCH_LOG) >= now - max(POLL * 2, 120)
-    except OSError:
-        return False
+    # Detect REAL workspace writes (user data, generation output, installs),
+    # excluding swm's own bookkeeping (autosync change-log/stamps/guard), build
+    # caches, logs, and framework noise. Keying off WATCH_LOG mtime was a bug:
+    # the autosync watcher rewrites that file every cycle, so the pod looked
+    # perpetually active and auto-down could never fire.
+    minutes = max(2, int(max(POLL * 2, 120) / 60))
+    out = sh(
+        "find /workspace -mindepth 1 -mmin -%d -type f "
+        "-not -path '*/.swm_*' -not -path '*/.swm_guard/*' "
+        "-not -path '*/.uv-cache/*' -not -path '*/.cache/*' "
+        "-not -path '*/__pycache__/*' -not -name '*.log' "
+        "-not -path '*/terminfo/*' -not -path '*/.git/*' "
+        "-not -path '*/.nv/*' 2>/dev/null | head -1" % minutes
+    )
+    return bool(out.strip())
 
 
 def transfer_locked() -> bool:

{swm_gpu-0.2.4 → swm_gpu-0.2.6}/src/swm/sync/paths.py

@@ -43,4 +43,12 @@ WATCH_EXCLUDES: tuple[str, ...] = (
     r"\.log$",
     r"/\.cache/",
     r"/\.nv/",
+    # uv's build/wheel cache (workspace-owned uv from v0.2.x). Regenerable,
+    # and its ephemeral builds-v0/.tmp* dirs churn constantly — syncing it
+    # both wastes storage and causes delete-reconciliation failures.
+    r"/\.uv-cache/",
+    # Managed-CPython terminfo database (workspace-owned python). Its entries
+    # are dedup-hardlinked, which breaks hardlink-staging (s5cmd then can't
+    # open them → push fails). Non-essential for any venv/ML workload.
+    r"/terminfo/",
 )

swm_gpu-0.2.4/src/swm/__init__.py

@@ -1 +0,0 @@
-__version__ = "0.2.4"