paralleliq-skypilot-plugin 0.1.0__tar.gz

paralleliq_skypilot_plugin-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.venv/

paralleliq_skypilot_plugin-0.1.0/PKG-INFO

@@ -0,0 +1,63 @@
+Metadata-Version: 2.4
+Name: paralleliq-skypilot-plugin
+Version: 0.1.0
+Summary: SkyPilot AdminPolicy that surfaces piqc GPU waste scanning when a GPU workload is launched on Kubernetes
+Author-email: Paralleliq <engineering@paralleliq.ai>
+License: Apache-2.0
+Keywords: gpu,inference,kubernetes,paralleliq,piqc,skypilot
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.11
+Requires-Dist: skypilot>=0.7.0
+Description-Content-Type: text/markdown
+
+# paralleliq-skypilot-plugin
+
+A SkyPilot [AdminPolicy](https://docs.skypilot.co/en/latest/cloud-setup/policy.html) that surfaces [piqc](https://github.com/paralleliq/piqc) — Paralleliq's GPU waste scanner — when a GPU workload is launched on a Kubernetes cluster.
+
+## What it does
+
+When you run `sky launch` with GPU accelerators targeting a Kubernetes cluster, the policy prints instructions to run a free piqc scan on your cluster before the job starts.
+
+## Install
+
+```bash
+pip install paralleliq-skypilot-plugin
+```
+
+## Configure
+
+Add one line to `~/.sky/config.yaml`:
+
+```yaml
+admin_policy: paralleliq_skypilot_plugin.ParalleliqPolicy
+```
+
+## Usage
+
+Run any GPU workload as normal:
+
+```bash
+sky launch -c my-cluster task.yaml
+```
+
+If the task requests GPU accelerators on a Kubernetes backend, you will see:
+
+```
+  Paralleliq — GPU workload detected on Kubernetes.
+  Run a free GPU waste scan on this cluster before launching:
+
+    kubectl apply -f https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/rbac.yaml
+    kubectl apply -f https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/scan-job.yaml
+    kubectl logs -n kube-system job/piqc-scan
+
+  github.com/paralleliq/piqc  ·  paralleliq.ai
+```
+
+## License
+
+Apache-2.0

paralleliq_skypilot_plugin-0.1.0/README.md

@@ -0,0 +1,46 @@
+# paralleliq-skypilot-plugin
+
+A SkyPilot [AdminPolicy](https://docs.skypilot.co/en/latest/cloud-setup/policy.html) that surfaces [piqc](https://github.com/paralleliq/piqc) — Paralleliq's GPU waste scanner — when a GPU workload is launched on a Kubernetes cluster.
+
+## What it does
+
+When you run `sky launch` with GPU accelerators targeting a Kubernetes cluster, the policy prints instructions to run a free piqc scan on your cluster before the job starts.
+
+## Install
+
+```bash
+pip install paralleliq-skypilot-plugin
+```
+
+## Configure
+
+Add one line to `~/.sky/config.yaml`:
+
+```yaml
+admin_policy: paralleliq_skypilot_plugin.ParalleliqPolicy
+```
+
+## Usage
+
+Run any GPU workload as normal:
+
+```bash
+sky launch -c my-cluster task.yaml
+```
+
+If the task requests GPU accelerators on a Kubernetes backend, you will see:
+
+```
+  Paralleliq — GPU workload detected on Kubernetes.
+  Run a free GPU waste scan on this cluster before launching:
+
+    kubectl apply -f https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/rbac.yaml
+    kubectl apply -f https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/scan-job.yaml
+    kubectl logs -n kube-system job/piqc-scan
+
+  github.com/paralleliq/piqc  ·  paralleliq.ai
+```
+
+## License
+
+Apache-2.0

paralleliq_skypilot_plugin-0.1.0/pyproject.toml

@@ -0,0 +1,25 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "paralleliq-skypilot-plugin"
+version = "0.1.0"
+description = "SkyPilot AdminPolicy that surfaces piqc GPU waste scanning when a GPU workload is launched on Kubernetes"
+readme = "README.md"
+license = { text = "Apache-2.0" }
+authors = [{ name = "Paralleliq", email = "engineering@paralleliq.ai" }]
+keywords = ["skypilot", "gpu", "kubernetes", "inference", "piqc", "paralleliq"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: Apache Software License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+requires-python = ">=3.11"
+dependencies = ["skypilot>=0.7.0"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/paralleliq_skypilot_plugin"]

paralleliq_skypilot_plugin-0.1.0/skypilot-plugin-devlog.md

@@ -0,0 +1,277 @@
+# SkyPilot AdminPolicy Integration — Developer Experience Log
+
+This document captures the experience of a third-party developer building an AdminPolicy plugin for SkyPilot.
+Written from the perspective of Paralleliq integrating with SkyPilot's policy system.
+Shared with Zongheng Yang and the SkyPilot team as honest feedback.
+
+*Note: We previously built a plugin for dstack — that experience is the baseline for several comparisons below.*
+
+---
+
+## What We Are Building
+
+A `paralleliq-skypilot-plugin` Python package that hooks into SkyPilot's `AdminPolicy` system to surface our open source GPU scanner ([piqc](https://github.com/paralleliq/piqc)) when a user launches a GPU workload on a Kubernetes cluster. Goal: when a user runs `sky launch`, they immediately see instructions to run a piqc GPU waste scan on their cluster before the job starts.
+
+---
+
+## Step 1 — Installing SkyPilot
+
+**What we did:**
+Installed with `pip install "skypilot[kubernetes]"`. Used `[kubernetes]` rather than `[all]` since our test clusters are on GKE and we didn't need every cloud SDK.
+
+**One dependency conflict to note:**
+SkyPilot and dstack require different versions of pydantic. SkyPilot upgraded pydantic to 2.x, which broke dstack (which requires pydantic <2.0). In a production environment where both tools are used, separate virtual environments would be the clean fix.
+
+**No issues with the install itself** — straightforward.
+
+---
+
+## Step 2 — Finding the AdminPolicy System
+
+**What we did:**
+Searched the SkyPilot docs for "admin policy" and found a dedicated page immediately. This was a significantly better experience than dstack, where we had to navigate the source code directly to find the plugin API.
+
+**What the docs tell you:**
+- Subclass `sky.AdminPolicy`
+- Implement `validate_and_mutate(cls, user_request: UserRequest) -> MutatedUserRequest`
+- Register via one line in `~/.sky/config.yaml`
+
+**How we verified the interface:**
+Read the installed source at `sky/admin_policy.py`. Found the full class hierarchy:
+- `AdminPolicy` — the class to subclass, `validate_and_mutate` is a `@classmethod`
+- `UserRequest` — wraps `sky.Task`, `sky.Config`, `request_name`, `request_options`
+- `MutatedUserRequest` — wraps `sky.Task`, `sky.Config` (return this unchanged to pass through)
+
+**Bonus finding — `RestfulAdminPolicy`:**
+SkyPilot ships a built-in `RestfulAdminPolicy` class that calls a REST endpoint for validation. This is a distribution model we hadn't considered: instead of installing a pip package, a cluster admin could point to a Paralleliq API URL. Worth exploring in a future version.
+
+**Feedback for SkyPilot:**
+> The AdminPolicy docs are well-structured and the interface is immediately clear. Compared to building the equivalent for dstack, ramp time was significantly shorter. One addition that would help: a minimal working example in the docs (10–15 lines) that a developer can copy, install, and run as a starting point.
+
+---
+
+## Step 3 — Understanding the Task and Resources Model
+
+**What we needed to know:**
+How to detect a GPU workload — specifically, how to check if the user is requesting GPU accelerators on a Kubernetes backend.
+
+**What we found in `sky.Task`:**
+`task.resources` is a **list** of `sky.Resources` objects, not a single object. SkyPilot supports ordered resource preferences (try A100 first, fall back to V100, etc.), so the list can have multiple entries.
+
+**What we found in `sky.Resources`:**
+- `.accelerators` — returns `Optional[Dict[str, Union[int, float]]]`. Always a dict when set, e.g., `{'A100': 4}`. Returns `None` if no GPU is requested.
+- `.cloud` — the cloud target. When Kubernetes is specified, `str(r.cloud).lower()` returns `'kubernetes'`.
+
+**The check we implemented:**
+```python
+has_gpu = any(r.accelerators for r in task.resources)
+is_k8s = any(str(r.cloud).lower() == "kubernetes" for r in task.resources if r.cloud)
+```
+
+**One gotcha:**
+`str(r.cloud)` returns `'kubernetes'` (lowercase), not `'Kubernetes'` as suggested by the class's `_REPR = 'Kubernetes'`. A case-insensitive comparison is required.
+
+---
+
+## Step 4 — Building the Plugin Package
+
+**Package structure** (mirrored from `paralleliq-dstack-plugin`):
+```
+paralleliq-skypilot-plugin/
+  pyproject.toml
+  README.md
+  test_policy.py
+  src/paralleliq_skypilot_plugin/
+    __init__.py
+    policy.py
+```
+
+**The full policy implementation:**
+```python
+from sky.admin_policy import AdminPolicy, MutatedUserRequest, UserRequest
+
+_PIQC_RBAC_URL = "https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/rbac.yaml"
+_PIQC_JOB_URL = "https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/scan-job.yaml"
+
+_PIQC_MESSAGE = (
+    "\n"
+    "  Paralleliq — GPU workload detected on Kubernetes.\n"
+    "  Run a free GPU waste scan on this cluster before launching:\n"
+    "\n"
+    f"    kubectl apply -f {_PIQC_RBAC_URL}\n"
+    f"    kubectl apply -f {_PIQC_JOB_URL}\n"
+    "    kubectl logs -n kube-system job/piqc-scan\n"
+    "\n"
+    "  github.com/paralleliq/piqc  ·  paralleliq.ai\n"
+)
+
+class ParalleliqPolicy(AdminPolicy):
+    @classmethod
+    def validate_and_mutate(cls, user_request: UserRequest) -> MutatedUserRequest:
+        resources = user_request.task.resources or []
+        if _has_gpu(resources) and _is_kubernetes(resources):
+            print(_PIQC_MESSAGE)
+        return MutatedUserRequest(
+            task=user_request.task,
+            skypilot_config=user_request.skypilot_config,
+        )
+```
+
+**Registration in `~/.sky/config.yaml`:**
+```yaml
+admin_policy: paralleliq_skypilot_plugin.ParalleliqPolicy
+```
+
+**Key advantage over dstack:**
+The dstack plugin message appears only in the dstack server log — not in the user's terminal. The SkyPilot AdminPolicy fires **client-side**, so `print()` goes directly to the user's terminal at `sky launch` time. This is a materially better developer and user experience.
+
+---
+
+## Step 5 — Local Testing
+
+**What we built:**
+A `test_policy.py` with 3 test cases that exercise the policy directly without a cluster.
+
+**Test results:**
+```
+=== Test 1: GPU + Kubernetes -> should print message ===
+
+  Paralleliq — GPU workload detected on Kubernetes.
+  Run a free GPU waste scan on this cluster before launching:
+
+    kubectl apply -f https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/rbac.yaml
+    kubectl apply -f https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/scan-job.yaml
+    kubectl logs -n kube-system job/piqc-scan
+
+  github.com/paralleliq/piqc  ·  paralleliq.ai
+
+PASS
+
+=== Test 2: GPU + no cloud -> should NOT print message ===
+PASS (no message above = correct)
+
+=== Test 3: No GPU + Kubernetes -> should NOT print message ===
+PASS (no message above = correct)
+
+All tests passed.
+```
+
+**One bug caught during testing:**
+Initial check used `str(r.cloud) == "Kubernetes"` (capital K). The actual value is `'kubernetes'` (lowercase). Fixed to `.lower() == "kubernetes"`. This would have been a silent failure in production — the plugin would install and load but never fire.
+
+---
+
+## Step 6 — GKE Cluster Testing
+
+**Test environment:**
+- `piqc-cluster-1` — GKE, us-central1-a, e2-standard-2, 2 nodes
+- `piqc-cluster-2` — GKE, us-west1-a, e2-standard-2, 2 nodes
+- CPU-only nodes (no GPUs) — testing plugin fire behavior, not GPU optimization
+
+### 6a — Prerequisite: verify piqc still works on the cluster
+
+Before layering in SkyPilot, confirmed piqc scans cleanly on piqc-cluster-1:
+
+```bash
+kubectl apply -f https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/rbac.yaml
+kubectl apply -f https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/scan-job.yaml
+kubectl logs -n kube-system job/piqc-scan
+```
+
+Result: scan completed successfully. Cluster healthy, piqc image pulls correctly.
+
+### 6b — Connect SkyPilot to the cluster
+
+```bash
+sky check kubernetes
+```
+
+**First attempt failed** — SkyPilot's default Kubernetes networking mode (`portforward`) requires `socat` and GNU `netcat`, neither of which ship with macOS.
+
+Fix:
+```bash
+brew install socat netcat
+sky check kubernetes
+```
+
+Second attempt succeeded:
+```
+Kubernetes: enabled [compute]
+  Allowed contexts:
+  └── gke_piqc-483417_us-central1-a_piqc-cluster-1: enabled.
+```
+
+Expected warning about no GPU/TPU resources — clusters are CPU-only for this test.
+
+### 6c — GPU task: policy fires, scheduling fails (expected)
+
+```bash
+sky launch test-task.yaml  # requests A100:1 on kubernetes
+```
+
+**Result:**
+```
+Applying client admin policy: ParalleliqPolicy
+
+  Paralleliq — GPU workload detected on Kubernetes.
+  Run a free GPU waste scan on this cluster before launching:
+
+    kubectl apply -f .../rbac.yaml
+    kubectl apply -f .../scan-job.yaml
+    kubectl logs -n kube-system job/piqc-scan
+
+  github.com/paralleliq/piqc  ·  paralleliq.ai
+
+Applying server admin policy: ParalleliqPolicy
+ResourcesUnavailableError: Kubernetes cluster does not contain any instances satisfying the request: 1x Kubernetes({'A100': 1}).
+```
+
+Policy fired exactly once (client-side), message visible to the user. SkyPilot then correctly rejected the request — no A100s on a CPU-only cluster. This is the real-world scenario: user is about to launch a GPU job, our message tells them to scan first before consuming GPU capacity.
+
+**One bug found and fixed during this step:**
+The policy was initially printing twice — once client-side, once server-side. SkyPilot runs a local API server even for local use, and fires the AdminPolicy on both sides. Fixed by checking `user_request.at_client_side`:
+
+```python
+if user_request.at_client_side and _has_gpu(resources) and _is_kubernetes(resources):
+    print(_PIQC_MESSAGE)
+```
+
+Confirmed via debug logging: client call has `at_client_side=True`, server call has `at_client_side=False`.
+
+### 6d — CPU task: policy silent, task runs successfully
+
+```bash
+sky launch test-task-cpu.yaml  # 1 CPU, kubernetes, no GPU
+```
+
+**Result:**
+```
+Applying client admin policy: ParalleliqPolicy
+Applying server admin policy: ParalleliqPolicy
+Considered resources (1 node):
+ Kubernetes (gke_piqc-48...c-cluster-1)   1 vCPU   1GB   -   $0.00   ✔
+
+(task, pid=965) piqc cluster connectivity test - OK
+✓ Job finished (status: SUCCEEDED).
+```
+
+Policy stayed completely silent — no GPU, no message. Task ran to completion on piqc-cluster-1. End-to-end cluster connectivity confirmed.
+
+**Feedback for SkyPilot:**
+> The `at_client_side` field on `UserRequest` is well-designed and exactly what we needed to deduplicate messages across client and server. One thing worth documenting: the AdminPolicy fires twice by default (client + server), and `at_client_side` is the mechanism to control that. This isn't obvious until you hit the double-print behavior in testing.
+
+---
+
+## Step 7 — PyPI Publish
+
+*[Pending — after GKE test]*
+
+---
+
+## Summary for Zongheng
+
+*[To be written after Steps 6 and 7 are complete]*
+
+---
+
+*Last updated: 2026-06-14*

paralleliq_skypilot_plugin-0.1.0/src/paralleliq_skypilot_plugin/__init__.py

@@ -0,0 +1,4 @@
+from paralleliq_skypilot_plugin.policy import ParalleliqPolicy
+
+__all__ = ["ParalleliqPolicy"]
+__version__ = "0.1.0"

paralleliq_skypilot_plugin-0.1.0/src/paralleliq_skypilot_plugin/policy.py

@@ -0,0 +1,41 @@
+from sky.admin_policy import AdminPolicy, MutatedUserRequest, UserRequest
+
+_PIQC_RBAC_URL = "https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/rbac.yaml"
+_PIQC_JOB_URL = "https://raw.githubusercontent.com/paralleliq/piqc/main/deploy/scan-job.yaml"
+
+_PIQC_MESSAGE = (
+    "\n"
+    "  Paralleliq — GPU workload detected on Kubernetes.\n"
+    "  Run a free GPU waste scan on this cluster before launching:\n"
+    "\n"
+    f"    kubectl apply -f {_PIQC_RBAC_URL}\n"
+    f"    kubectl apply -f {_PIQC_JOB_URL}\n"
+    "    kubectl logs -n kube-system job/piqc-scan\n"
+    "\n"
+    "  github.com/paralleliq/piqc  ·  paralleliq.ai\n"
+)
+
+
+def _has_gpu(resources_list) -> bool:
+    """Return True if any resource in the list requests GPU accelerators."""
+    return any(r.accelerators for r in resources_list)
+
+
+def _is_kubernetes(resources_list) -> bool:
+    """Return True if any resource in the list explicitly targets Kubernetes."""
+    return any(
+        r.cloud is not None and str(r.cloud).lower() == "kubernetes"
+        for r in resources_list
+    )
+
+
+class ParalleliqPolicy(AdminPolicy):
+    @classmethod
+    def validate_and_mutate(cls, user_request: UserRequest) -> MutatedUserRequest:
+        resources = user_request.task.resources or []
+        if user_request.at_client_side and _has_gpu(resources) and _is_kubernetes(resources):
+            print(_PIQC_MESSAGE)
+        return MutatedUserRequest(
+            task=user_request.task,
+            skypilot_config=user_request.skypilot_config,
+        )

paralleliq_skypilot_plugin-0.1.0/test-task-cpu.yaml

@@ -0,0 +1,5 @@
+resources:
+  cloud: kubernetes
+  cpus: 1
+
+run: echo "piqc cluster connectivity test - OK"

paralleliq_skypilot_plugin-0.1.0/test-task.yaml

@@ -0,0 +1,5 @@
+resources:
+  cloud: kubernetes
+  accelerators: A100:1
+
+run: echo "hello from GPU task"

paralleliq_skypilot_plugin-0.1.0/test_policy.py

@@ -0,0 +1,41 @@
+"""Local test for ParalleliqPolicy — no cluster required."""
+import sys
+sys.path.insert(0, "src")
+
+import sky
+from sky.admin_policy import UserRequest, MutatedUserRequest
+from sky.server.requests.request_names import AdminPolicyRequestName
+from paralleliq_skypilot_plugin import ParalleliqPolicy
+
+
+def make_request(accelerators=None, cloud=None):
+    resources = sky.Resources(accelerators=accelerators, cloud=cloud)
+    task = sky.Task()
+    task.set_resources(resources)
+    config = sky.Config()
+    return UserRequest(
+        task=task,
+        skypilot_config=config,
+        request_name=AdminPolicyRequestName.CLUSTER_LAUNCH,
+    )
+
+
+print("=== Test 1: GPU + Kubernetes -> should print message ===")
+req = make_request(accelerators="A100:1", cloud="kubernetes")
+result = ParalleliqPolicy.validate_and_mutate(req)
+assert isinstance(result, MutatedUserRequest)
+print("PASS\n")
+
+print("=== Test 2: GPU + no cloud -> should NOT print message ===")
+req = make_request(accelerators="A100:1", cloud=None)
+result = ParalleliqPolicy.validate_and_mutate(req)
+assert isinstance(result, MutatedUserRequest)
+print("PASS (no message above = correct)\n")
+
+print("=== Test 3: No GPU + Kubernetes -> should NOT print message ===")
+req = make_request(accelerators=None, cloud="kubernetes")
+result = ParalleliqPolicy.validate_and_mutate(req)
+assert isinstance(result, MutatedUserRequest)
+print("PASS (no message above = correct)\n")
+
+print("All tests passed.")