dstack 0.20.20__tar.gz → 0.20.21__tar.gz

{dstack-0.20.20 → dstack-0.20.21}/.github/workflows/build-artifacts.yml

@@ -74,7 +74,7 @@ jobs:
       - name: Install dependencies
         run: uv sync --all-extras
       - name: Run pyright
-        uses: jakebailey/pyright-action@v2
+        uses: jakebailey/pyright-action@v3
         with:
           pylance-version: latest-release
       - name: Download frontend build

{dstack-0.20.20 → dstack-0.20.21}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dstack
-Version: 0.20.20
+Version: 0.20.21
 Summary: dstack is an open-source orchestration engine for running AI workloads on any cloud or on-premises.
 Project-URL: Homepage, https://dstack.ai
 Project-URL: Source, https://github.com/dstackai/dstack
@@ -22,7 +22,7 @@ Requires-Dist: cryptography
 Requires-Dist: cursor
 Requires-Dist: filelock
 Requires-Dist: gitpython
-Requires-Dist: gpuhunt==0.1.21
+Requires-Dist: gpuhunt==0.1.22
 Requires-Dist: ignore-python>=0.2.0
 Requires-Dist: jsonschema
 Requires-Dist: orjson

{dstack-0.20.20 → dstack-0.20.21}/mkdocs/docs/concepts/backends.md

@@ -918,6 +918,26 @@ projects:
 
     </div>
 
+### JarvisLabs
+
+Log into your [JarvisLabs](https://cloud.jarvislabs.ai/) account and create an API key.
+
+Then, go ahead and configure the backend:
+
+<div editor-title="~/.dstack/server/config.yml">
+
+```yaml
+projects:
+- name: main
+  backends:
+    - type: jarvislabs
+      creds:
+        type: api_key
+        api_key: ...
+```
+
+</div>
+
 ### CloudRift
 
 Log into your [CloudRift](https://console.cloudrift.ai/) console, click `API Keys` in the sidebar and click the button to create a new API key.
@@ -1051,9 +1071,9 @@ Compared to [VM-based](#vm-based) backends, they offer less fine-grained control
 
 ### Kubernetes
 
-Regardless of whether it’s on-prem Kubernetes or managed, `dstack` can orchestrate container-based runs across your clusters.
+Regardless of whether it’s on-prem Kubernetes or managed, `dstack` can orchestrate container-based runs across your clusters. A single `kubernetes` backend can manage one or many clusters — each cluster is selected via a kubeconfig [context](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#context).
 
-To use the `kubernetes` backend with `dstack`, you need to configure it with the path to the kubeconfig file, the IP address of any node in the cluster, and the port that `dstack` will use for proxying SSH traffic. 
+The recommended way is to enable clusters explicitly via the `contexts` property:
 
 <div editor-title="~/.dstack/server/config.yml">
 
@@ -1066,22 +1086,48 @@ projects:
     kubeconfig:
       filename: ~/.kube/config
 
-    proxy_jump:
-      hostname: 204.12.171.137
-      port: 32000
+    contexts:
+    - name: gpu-cluster-a
+    - name: gpu-cluster-b
 ```
 
 </div>
 
 !!! info "Proxy jump"
-    To allow the `dstack` server and CLI to access runs via SSH, `dstack` requires a node that acts as a jump host to proxy SSH traffic into containers.  
+    To allow the `dstack` server and CLI to access runs via SSH, `dstack` uses a node in each cluster as a jump host to proxy SSH traffic into containers. No additional setup is required — `dstack` configures and manages the proxy automatically.
+
+    By default, `dstack` autodetects the jump host:
+
+    - `hostname` — picks the `ExternalIP` of the jump pod's node, or a random node `ExternalIP` from the cluster if the jump pod's node has none. If no node in the cluster has an `ExternalIP`, provisioning fails and you must set `hostname` explicitly.
+    - `port` — Kubernetes allocates a port from the cluster's NodePort range.
+
+    Set `proxy_jump.hostname` and `proxy_jump.port` per context to override autodetection — useful when nodes lack `ExternalIP`s, or when you want a stable, firewall-friendly port:
+
+    ```yaml
+    contexts:
+    - name: gpu-cluster-a
+      proxy_jump:
+        hostname: 204.12.171.137
+        port: 32000
+    ```
+
+    Both fields are independent — you can set just one.
 
-    To configure this node, specify `hostname` and `port` under the `proxy_jump` property:  
+    The jump host can be a GPU node or a CPU-only node — it makes no difference. The only requirement is that both the `dstack` server and CLI can reach `hostname:port`.
 
-    - `hostname` — the IP address of any cluster node selected as the jump host. Both the `dstack` server and CLI must be able to reach it. This node can be either a GPU node or a CPU-only node — it makes no difference.  
-    - `port` — any accessible port on that node, which `dstack` uses to forward SSH traffic.  
+!!! info "Region and namespace"
+    Each enabled context becomes its own `dstack` region, named after the context. When creating a `dstack` [volume](volumes.md) or [gateway](gateways.md), the `region` field selects which cluster the resource is provisioned in.
 
-    No additional setup is required — `dstack` configures and manages the proxy automatically.
+    The namespace `dstack` uses for managed resources is taken from each kubeconfig context's `namespace` property, defaulting to `default` if not set:
+
+    ```yaml
+    contexts:
+    - name: gpu-cluster-a
+      context:
+        cluster: gpu-cluster-a
+        user: kubernetes-admin
+        namespace: dstack
+    ```
 
 ??? info "User interface"
     If you are configuring the `kubernetes` backend on the [project settings page](projects.md#backends),
@@ -1091,17 +1137,16 @@ projects:
 
     ```yaml
     type: kubernetes
-    
+
     kubeconfig:
       data: |
         apiVersion: v1
         kind: Config
-        current-context: kubernetes-admin@gpu-cluster
 
         clusters:
-        - name: gpu-cluster
+        - name: gpu-cluster-a
           cluster:
-            server: https://gpu-cluster.internal.example.com:6443
+            server: https://gpu-cluster-a.internal.example.com:6443
             certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...LS0tLQo=
 
         users:
@@ -1111,17 +1156,50 @@ projects:
             client-key-data: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0t...LS0tLQo=
 
         contexts:
-        - name: kubernetes-admin@gpu-cluster
+        - name: gpu-cluster-a
           context:
-            cluster: gpu-cluster
+            cluster: gpu-cluster-a
             user: kubernetes-admin
-    
-    proxy_jump:
-      hostname: 204.12.171.137
-      port: 32000
+            namespace: dstack
+
+    contexts:
+    - name: gpu-cluster-a
+      proxy_jump:
+        hostname: 204.12.171.137
+        port: 32000
+    ```
+
+    </div>
+
+??? warning "Legacy configuration (without `contexts`)"
+    If `contexts` is not set, `dstack` falls back to using the kubeconfig's `current-context` as the only cluster, and the top-level `proxy_jump` and `namespace` properties apply:
+
+    <div editor-title="~/.dstack/server/config.yml">
+
+    ```yaml
+    projects:
+    - name: main
+      backends:
+      - type: kubernetes
+
+        kubeconfig:
+          filename: ~/.kube/config
+
+        namespace: dstack
+
+        proxy_jump:
+          hostname: 204.12.171.137
+          port: 32000
     ```
 
     </div>
+
+    This mode is not recommended and may be deprecated and removed in the future. It also has a namespace-handling quirk: the top-level `namespace` property **overrides** the kubeconfig context's namespace (defaulting to `default` if not set in the config), unlike the `contexts` mode where the kubeconfig is authoritative. A warning is logged when the two disagree. To prepare for a possible future change, set the same value in both your kubeconfig context and the backend config.
+
+    With this configuration, the cluster's region is an empty string. When creating a `dstack` volume or gateway, set `region: ''` explicitly in the configuration.
+
+    !!! warning "Migrating from legacy to `contexts`"
+        Switching an existing backend from the legacy mode to `contexts` is not transparent for already-provisioned resources: their region changes from an empty string to the context name, so `dstack` can no longer terminate them. Terminate all jobs, gateways, and volumes managed by the backend before changing the configuration.
     
 ??? info "Required operators"
     === "NVIDIA"
@@ -1149,7 +1227,7 @@ projects:
       --8<-- "snippets/kubernetes/dstack-backend-role.yaml"
       ```
 
-    Ensure you've created a ClusterRoleBinding to grant the role to the user or the service account you're using.
+    Ensure you've created a ClusterRoleBinding and RoleBinding to grant the roles to the user or the service account you're using.
 
 ??? info "Resources and offers"
     If you use ranges with [`resources`](../concepts/tasks.md#resources) (e.g. `gpu: 1..8` or `memory: 64GB..`) in fleet or run configurations, other backends collect and try all offers that satisfy the range.

{dstack-0.20.20 → dstack-0.20.21}/mkdocs/docs/examples/inference/dynamo.md

@@ -1,6 +1,6 @@
 ---
-title: Dynamo
-description: Deploying zai-org/GLM-4.5-Air-FP8 using NVIDIA Dynamo
+title: NVIDIA Dynamo
+description: Deploying zai-org/GLM-4.5-Air-FP8 using NVIDIA Dynamo with Prefill-Decode disaggregation.
 ---
 
 # Dynamo

{dstack-0.20.20 → dstack-0.20.21}/mkdocs/docs/reference/dstack.yml/volume.md

@@ -60,3 +60,5 @@ The `volume` configuration type allows creating, registering, and updating [volu
             show_root_heading: false
             backend:
                 required: true
+            region:
+                required: true

{dstack-0.20.20 → dstack-0.20.21}/mkdocs/docs/reference/server/config.yml.md

@@ -278,6 +278,18 @@ to configure [backends](../../concepts/backends.md) and other [server-level sett
     yq -o=json ~/.kube/config | jq -c | jq -R
     ```
 
+###### `projects[n].backends[type=kubernetes].contexts[n]` { #kubernetes-contexts data-toc-label="contexts" }
+
+#SCHEMA# dstack._internal.core.backends.kubernetes.models.KubernetesContextConfig
+    overrides:
+        show_root_heading: false
+
+###### `projects[n].backends[type=kubernetes].contexts[n].proxy_jump` { #kubernetes-contexts-proxy_jump data-toc-label="proxy_jump" }
+
+#SCHEMA# dstack._internal.core.backends.kubernetes.models.KubernetesProxyJumpConfig
+    overrides:
+        show_root_heading: false
+
 ###### `projects[n].backends[type=kubernetes].proxy_jump` { #kubernetes-proxy_jump data-toc-label="proxy_jump" }
 
 #SCHEMA# dstack._internal.core.backends.kubernetes.models.KubernetesProxyJumpConfig
@@ -369,6 +381,23 @@ to configure [backends](../../concepts/backends.md) and other [server-level sett
         type:
             required: true
 
+##### `projects[n].backends[type=jarvislabs]` { #jarvislabs data-toc-label="jarvislabs" }
+
+#SCHEMA# dstack._internal.core.backends.jarvislabs.models.JarvisLabsBackendFileConfigWithCreds
+    overrides:
+        show_root_heading: false
+        type:
+            required: true
+        item_id_prefix: jarvislabs-
+
+###### `projects[n].backends[type=jarvislabs].creds` { #jarvislabs-creds data-toc-label="creds" }
+
+#SCHEMA# dstack._internal.core.backends.jarvislabs.models.JarvisLabsAPIKeyCreds
+    overrides:
+        show_root_heading: false
+        type:
+            required: true
+
 ##### `projects[n].backends[type=cloudrift]` { #cloudrift data-toc-label="cloudrift" }
 
 #SCHEMA# dstack._internal.core.backends.cloudrift.models.CloudRiftBackendConfigWithCreds

{dstack-0.20.20 → dstack-0.20.21}/pyproject.toml

@@ -32,7 +32,7 @@ dependencies = [
     "python-multipart>=0.0.16",
     "filelock",
     "psutil",
-    "gpuhunt==0.1.21",
+    "gpuhunt==0.1.22",
     "argcomplete>=3.5.0",
     "ignore-python>=0.2.0",
     "orjson",
@@ -152,19 +152,21 @@ dev = [
     "testcontainers>=4.9.2",
     "pytest-xdist>=3.6.1",
     "pyinstrument>=5.0.0",
-    "kubernetes-stubs-elephant-fork",
-    # docs
-    "dstack[server]; python_version >= '3.11'",
-    "dstack-plugin-server; python_version >= '3.11'",
-    "pillow; python_version >= '3.11'",
-    "cairosvg; python_version >= '3.11'",
-    "mkdocs-material>=9.7.0; python_version >= '3.11'",
-    "mkdocs-material[imaging]; python_version >= '3.11'",
-    "mkdocs-material-extensions; python_version >= '3.11'",
-    "mkdocs-redirects; python_version >= '3.11'",
-    "mkdocs-gen-files; python_version >= '3.11'",
-    "mkdocstrings[python]; python_version >= '3.11'",
-    "mkdocs-render-swagger-plugin; python_version >= '3.11'",
+    "kubernetes-stubs-elephant-fork>=35.0.0.post3",
+    {include-group = "docs"},
+]
+docs = [
+    "dstack[server]",
+    "dstack-plugin-server",
+    "pillow",
+    "cairosvg",
+    "mkdocs-material>=9.7.0",
+    "mkdocs-material[imaging]",
+    "mkdocs-material-extensions",
+    "mkdocs-redirects",
+    "mkdocs-gen-files",
+    "mkdocstrings[python]",
+    "mkdocs-render-swagger-plugin",
 ]
 
 [project.optional-dependencies]

{dstack-0.20.20 → dstack-0.20.21}/scripts/docs/gen_schema_reference.py

@@ -27,9 +27,10 @@ logger.info("Generating schema reference...")
 
 def _is_linkable_type(annotation: Any) -> bool:
     """Check if a type annotation contains a BaseModel subclass (excluding Range)."""
-    if inspect.isclass(annotation):
-        return issubclass(annotation, BaseModel) and not issubclass(annotation, Range)
     origin = get_origin(annotation)
+    type_ = origin if origin is not None else annotation
+    if inspect.isclass(type_):
+        return issubclass(type_, BaseModel) and not issubclass(type_, Range)
     if origin is Annotated:
         return _is_linkable_type(get_args(annotation)[0])
     if origin is Union:
@@ -239,12 +240,12 @@ def generate_schema_reference(
             default_repr = None
         elif isinstance(default, (list, tuple, dict)) and len(default) == 0:
             default_repr = None
-        elif isinstance(default, str):
-            default_repr = default
-        elif isinstance(default, BaseModel):
-            default_repr = str(default)
         elif isinstance(default, Enum):
             default_repr = str(default.value)
+        elif isinstance(default, BaseModel):
+            default_repr = str(default)
+        elif isinstance(default, str):
+            default_repr = default
         else:
             default_repr = json.dumps(default)
         friendly_type = get_friendly_type(field.annotation)

dstack-0.20.21/scripts/merge_kubeconfigs.sh

@@ -0,0 +1,12 @@
+#!/bin/sh
+set -eu
+
+if [ ${#} -lt 2 ]; then
+    echo "usage: $(basename "${0}") PATH1 PATH2 [PATH3 ...]" >&2
+    exit 1
+fi
+
+# Windows is not supported; on Windows a path separator is ';', not ':'
+KUBECONFIG=$(IFS=':'; echo "${*}")
+export KUBECONFIG
+kubectl config view --raw --flatten | grep -Ev '^current-context: '

{dstack-0.20.20 → dstack-0.20.21}/scripts/setup_kubernetes.py

@@ -201,7 +201,9 @@ def generate_kubeconfig(
     service_account_token: str,
 ) -> str:
     logging.info("generating kubeconfig")
-    kubeconfig_content = kubectl.call("config", "view", "--minify", "--raw", capture_stdout=True)
+    kubeconfig_content = kubectl.call(
+        "config", "view", "--minify", "--raw", "--flatten", capture_stdout=True
+    )
     with tempfile.NamedTemporaryFile("w+") as f:
         f.write(kubeconfig_content)
         f.flush()

{dstack-0.20.20 → dstack-0.20.21}/src/dstack/_internal/core/backends/configurators.py

@@ -87,6 +87,15 @@ try:
 except ImportError:
     pass
 
+try:
+    from dstack._internal.core.backends.jarvislabs.configurator import (
+        JarvisLabsConfigurator,
+    )
+
+    _CONFIGURATOR_CLASSES.append(JarvisLabsConfigurator)
+except ImportError:
+    pass
+
 try:
     from dstack._internal.core.backends.kubernetes.configurator import (
         KubernetesConfigurator,