dstack 0.19.12__tar.gz → 0.19.13__tar.gz

{dstack-0.19.12 → dstack-0.19.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dstack
-Version: 0.19.12
+Version: 0.19.13
 Summary: dstack is an open-source orchestration engine for running AI workloads on any cloud or on-premises.
 Project-URL: Homepage, https://dstack.ai
 Project-URL: Source, https://github.com/dstackai/dstack

{dstack-0.19.12 → dstack-0.19.13}/docker/base/Dockerfile

@@ -19,7 +19,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
     ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime && \
     apt-get install -y tzdata && \
     dpkg-reconfigure --frontend noninteractive tzdata && \
-    apt-get install -y bzip2 ca-certificates curl build-essential git libglib2.0-0 libsm6 libxext6 libxrender1 mercurial openssh-server subversion wget && \
+    apt-get install -y bzip2 ca-certificates curl build-essential git libglib2.0-0 libsm6 libxext6 libxrender1 mercurial openssh-server subversion wget \
+    libibverbs1 ibverbs-providers ibverbs-utils libibverbs-dev infiniband-diags && \
     sed -i "s/.*PasswordAuthentication.*/PasswordAuthentication no/g" /etc/ssh/sshd_config && mkdir /run/sshd && \
     mkdir ~/.ssh && chmod 700 ~/.ssh && touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys && rm /etc/ssh/ssh_host_*
 

{dstack-0.19.12 → dstack-0.19.13}/docs/blog/posts/efa.md

@@ -117,7 +117,7 @@ name: efa-task
 # The size of the cluster
 nodes: 2
 
-python: "3.12"
+python: 3.12
 
 # Commands to run on each node
 commands:

{dstack-0.19.12 → dstack-0.19.13}/docs/docs/concepts/dev-environments.md

@@ -99,66 +99,6 @@ init:
 
 </div>
 
-### Inactivity duration
-
-Set [`inactivity_duration`](../reference/dstack.yml/dev-environment.md#inactivity_duration)
-to automatically stop the dev environment after a configured period of inactivity.
-
-<div editor-title=".dstack.yml">
-
-```yaml
-type: dev-environment
-name: vscode
-ide: vscode
-
-# Stop if inactive for 2 hours
-inactivity_duration: 2h
-```
-
-</div>
-
-The dev environment becomes inactive when you close the remote VS Code window,
-close any `ssh <run name>` shells, and stop the `dstack apply` or `dstack attach` command.
-If you go offline without stopping anything manually, the dev environment will also become inactive
-within about 3 minutes.
-
-If `inactivity_duration` is configured for your dev environment, you can see how long
-it has been inactive in `dstack ps --verbose`.
-
-<div class="termy">
-
-```shell
-$ dstack ps --verbose
- NAME    BACKEND  RESOURCES       PRICE    STATUS                 SUBMITTED
- vscode  cudo     2xCPU, 8GB,     $0.0286  running                8 mins ago
-                  100.0GB (disk)           (inactive for 2m 34s)
-```
-
-</div>
-
-If you reattach to the dev environment using [`dstack attach`](../reference/cli/dstack/attach.md),
-the inactivity timer will be reset within a few seconds.
-
-??? info "In-place update"
-    As long as the configuration defines the `name` property, the value of `inactivity_duration`
-    can be changed for a running dev environment without a restart.
-    Just change the value in the configuration and run `dstack apply` again.
-
-    <div class="termy">
-
-    ```shell
-    $ dstack apply -f .dstack.yml
-
-    Detected configuration changes that can be updated in-place: ['inactivity_duration']
-    Update the run? [y/n]:
-    ```
-
-    </div>
-
-> `inactivity_duration` is not to be confused with [`idle_duration`](#idle-duration).
-> The latter determines how soon the underlying cloud instance will be terminated
-> _after_ the dev environment is stopped.
-
 ### Resources
 
 When you specify a resource value like `cpu` or `memory`,
@@ -307,19 +247,6 @@ If you don't assign a value to an environment variable (see `HF_TOKEN` above),
     | `DSTACK_REPO_ID`        | The ID of the repo                      |
     | `DSTACK_GPUS_NUM`       | The total number of GPUs in the run     |
 
-### Spot policy
-
-By default, `dstack` uses on-demand instances. However, you can change that
-via the [`spot_policy`](../reference/dstack.yml/dev-environment.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`.
-
-!!! info "Reference"
-    Dev environments support many more configuration options,
-    incl. [`backends`](../reference/dstack.yml/dev-environment.md#backends), 
-    [`regions`](../reference/dstack.yml/dev-environment.md#regions), 
-    [`max_price`](../reference/dstack.yml/dev-environment.md#max_price), and
-    [`max_duration`](../reference/dstack.yml/dev-environment.md#max_duration), 
-    among [others](../reference/dstack.yml/dev-environment.md).
-
 ### Retry policy
 
 By default, if `dstack` can't find capacity or the instance is interrupted, the run will fail.
@@ -345,8 +272,108 @@ retry:
 
 </div>
 
+### Inactivity duration
+
+Set [`inactivity_duration`](../reference/dstack.yml/dev-environment.md#inactivity_duration)
+to automatically stop the dev environment after a configured period of inactivity.
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: dev-environment
+name: vscode
+ide: vscode
+
+# Stop if inactive for 2 hours
+inactivity_duration: 2h
+```
+
+</div>
+
+The dev environment becomes inactive when you close the remote VS Code window,
+close any `ssh <run name>` shells, and stop the `dstack apply` or `dstack attach` command.
+If you go offline without stopping anything manually, the dev environment will also become inactive
+within about 3 minutes.
+
+If `inactivity_duration` is configured for your dev environment, you can see how long
+it has been inactive in `dstack ps --verbose`.
+
+<div class="termy">
+
+```shell
+$ dstack ps --verbose
+ NAME    BACKEND  RESOURCES       PRICE    STATUS                 SUBMITTED
+ vscode  cudo     2xCPU, 8GB,     $0.0286  running                8 mins ago
+                  100.0GB (disk)           (inactive for 2m 34s)
+```
+
+</div>
+
+If you reattach to the dev environment using [`dstack attach`](../reference/cli/dstack/attach.md),
+the inactivity timer will be reset within a few seconds.
+
+??? info "In-place update"
+    As long as the configuration defines the `name` property, the value of `inactivity_duration`
+    can be changed for a running dev environment without a restart.
+    Just change the value in the configuration and run `dstack apply` again.
+
+    <div class="termy">
+
+    ```shell
+    $ dstack apply -f .dstack.yml
+
+    Detected configuration changes that can be updated in-place: ['inactivity_duration']
+    Update the run? [y/n]:
+    ```
+
+    </div>
+
+> `inactivity_duration` is not to be confused with [`idle_duration`](#idle-duration).
+> The latter determines how soon the underlying cloud instance will be terminated
+> _after_ the dev environment is stopped.
+
+### Utilization policy
+
+Sometimes it’s useful to track whether a dev environment is fully utilizing all GPUs. While you can check this with
+[`dstack metrics`](../reference/cli/dstack/metrics.md), `dstack` also lets you set a policy to auto-terminate the run if any GPU is underutilized.
+
+Below is an example of a dev environment that auto-terminate if any GPU stays below 10% utilization for 1 hour.
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: dev-environment
+name: my-dev
+
+python: 3.12
+ide: cursor
+
+resources:
+  gpu: H100:8
+
+utilization_policy:
+  min_gpu_utilization: 10
+  time_window: 1h
+```
+
+</div>
+
+### Spot policy
+
+By default, `dstack` uses on-demand instances. However, you can change that
+via the [`spot_policy`](../reference/dstack.yml/dev-environment.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`.
+
 --8<-- "docs/concepts/snippets/manage-fleets.ext"
 
+!!! info "Reference"
+    Dev environments support many more configuration options,
+    incl. [`backends`](../reference/dstack.yml/dev-environment.md#backends), 
+    [`regions`](../reference/dstack.yml/dev-environment.md#regions), 
+    [`max_price`](../reference/dstack.yml/dev-environment.md#max_price), and
+    [`max_duration`](../reference/dstack.yml/dev-environment.md#max_duration), 
+    among [others](../reference/dstack.yml/dev-environment.md).
+
+
 --8<-- "docs/concepts/snippets/manage-runs.ext"
 
 !!! info "What's next?"

{dstack-0.19.12 → dstack-0.19.13}/docs/docs/concepts/repos.md

@@ -28,6 +28,8 @@ If the directory is cloned Git repo, [`dstack apply`](../reference/cli/dstack/ap
 If the directory is not a cloned Git repo, it uploads the entire directory.
 
 Uploads are limited to 2MB. Use `.gitignore` to exclude unnecessary files from being uploaded.
+You can set the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable to increase the default server limit.
+Increasing the limit is recommended only if you [configure an object storage](../guides/server-deployment.md).
 
 ### Initialize as a local directory
 

{dstack-0.19.12 → dstack-0.19.13}/docs/docs/concepts/services.md

@@ -14,13 +14,13 @@ type: service
 name: llama31
 
 # If `image` is not specified, dstack uses its default image
-python: "3.11"
+python: 3.12
 env:
   - HF_TOKEN
   - MODEL_ID=meta-llama/Meta-Llama-3.1-8B-Instruct
   - MAX_MODEL_LEN=4096
 commands:
-  - pip install vllm
+  - uv pip install vllm
   - vllm serve $MODEL_ID
     --max-model-len $MAX_MODEL_LEN
     --tensor-parallel-size $DSTACK_GPUS_NUM
@@ -128,13 +128,13 @@ type: service
 # The name is optional, if not specified, generated randomly
 name: llama31-service
 
-python: "3.10"
+python: 3.12
 
 # Required environment variables
 env:
   - HF_TOKEN
 commands:
-  - pip install vllm
+  - uv pip install vllm
   - vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --max-model-len 4096
 # Expose the port of the service
 port: 8000
@@ -184,7 +184,7 @@ name: http-server-service
 # Disable authorization
 auth: false
 
-python: "3.10"
+python: 3.12
 
 # Commands of the service
 commands:
@@ -220,7 +220,7 @@ env:
   - DASH_ROUTES_PATHNAME_PREFIX=/proxy/services/main/dash/
 
 commands:
-  - pip install dash
+  - uv pip install dash
   # Assuming the Dash app is in your repo at app.py
   - python app.py
 
@@ -303,11 +303,11 @@ type: service
 # The name is optional, if not specified, generated randomly
 name: llama31-service
 
-python: "3.10"
+python: 3.12
 
 # Commands of the service
 commands:
-  - pip install vllm
+  - uv pip install vllm
   - python -m vllm.entrypoints.openai.api_server
     --model mistralai/Mixtral-8X7B-Instruct-v0.1
     --host 0.0.0.0
@@ -384,7 +384,7 @@ type: service
 name: http-server-service    
 
 # If `image` is not specified, dstack uses its base image
-python: "3.10"
+python: 3.12
 
 # Commands of the service
 commands:
@@ -407,7 +407,7 @@ port: 8000
     name: http-server-service    
     
     # If `image` is not specified, dstack uses its base image
-    python: "3.10"
+    python: 3.12
     # Ensure nvcc is installed (req. for Flash Attention) 
     nvcc: true
 
@@ -480,7 +480,7 @@ type: service
 # The name is optional, if not specified, generated randomly
 name: llama-2-7b-service
 
-python: "3.10"
+python: 3.12
 
 # Environment variables
 env:
@@ -488,7 +488,7 @@ env:
   - MODEL=NousResearch/Llama-2-7b-chat-hf
 # Commands of the service
 commands:
-  - pip install vllm
+  - uv pip install vllm
   - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000
 # The port of the service
 port: 8000
@@ -512,18 +512,6 @@ resources:
     | `DSTACK_REPO_ID`        | The ID of the repo                      |
     | `DSTACK_GPUS_NUM`       | The total number of GPUs in the run     |
 
-### Spot policy
-
-By default, `dstack` uses on-demand instances. However, you can change that
-via the [`spot_policy`](../reference/dstack.yml/service.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`.
-
-!!! info "Reference"
-    Services support many more configuration options,
-    incl. [`backends`](../reference/dstack.yml/service.md#backends), 
-    [`regions`](../reference/dstack.yml/service.md#regions), 
-    [`max_price`](../reference/dstack.yml/service.md#max_price), and
-    among [others](../reference/dstack.yml/service.md).
-
 ### Retry policy
 
 By default, if `dstack` can't find capacity, or the service exits with an error, or the instance is interrupted, the run will fail.
@@ -550,8 +538,52 @@ retry:
 If one replica of a multi-replica service fails with retry enabled,
 `dstack` will resubmit only the failed replica while keeping active replicas running.
 
+### Spot policy
+
+By default, `dstack` uses on-demand instances. However, you can change that
+via the [`spot_policy`](../reference/dstack.yml/service.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`.
+
+### Utilization policy
+
+Sometimes it’s useful to track whether a service is fully utilizing all GPUs. While you can check this with
+[`dstack metrics`](../reference/cli/dstack/metrics.md), `dstack` also lets you set a policy to auto-terminate the run if any GPU is underutilized.
+
+Below is an example of a service that auto-terminate if any GPU stays below 10% utilization for 1 hour.
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: service
+name: llama-2-7b-service
+
+python: 3.12
+env:
+  - HF_TOKEN
+  - MODEL=NousResearch/Llama-2-7b-chat-hf
+commands:
+  - uv pip install vllm
+  - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000
+port: 8000
+
+resources:
+  gpu: 24GB
+
+utilization_policy:
+  min_gpu_utilization: 10
+  time_window: 1h
+```
+
+</div>
+
 --8<-- "docs/concepts/snippets/manage-fleets.ext"
 
+!!! info "Reference"
+    Services support many more configuration options,
+    incl. [`backends`](../reference/dstack.yml/service.md#backends), 
+    [`regions`](../reference/dstack.yml/service.md#regions), 
+    [`max_price`](../reference/dstack.yml/service.md#max_price), and
+    among [others](../reference/dstack.yml/service.md).
+
 --8<-- "docs/concepts/snippets/manage-runs.ext"
 
 !!! info "What's next?"

{dstack-0.19.12 → dstack-0.19.13}/docs/docs/concepts/tasks.md

@@ -81,11 +81,11 @@ type: task
 # The name is optional, if not specified, generated randomly
 name: streamlit-hello
 
-python: "3.10"
+python: 3.12
 
 # Commands of the task
 commands:
-  - pip3 install streamlit
+  - uv pip install streamlit
   - streamlit hello
 # Expose the port to access the web app
 ports: 
@@ -102,33 +102,33 @@ application.
 By default, a task runs on a single node.
 However, you can run it on a cluster of nodes by specifying `nodes`.
 
-<div editor-title="train.dstack.yml">
+<div editor-title="examples/distributed-training/torchrun/.dstack.yml">
 
 ```yaml
 type: task
-# The name is optional, if not specified, generated randomly
 name: train-distrib
 
 # The size of the cluster
 nodes: 2
 
-python: "3.12"
-
-# Commands to run on each node
+python: 3.12
+env:
+  - NCCL_DEBUG=INFO
 commands:
-  - git clone https://github.com/pytorch/examples.git
-  - cd examples/distributed/ddp-tutorial-series
-  - pip install -r requirements.txt
-  - torchrun
-    --nproc-per-node=$DSTACK_GPUS_PER_NODE
-    --node-rank=$DSTACK_NODE_RANK
-    --nnodes=$DSTACK_NODES_NUM
-    --master-addr=$DSTACK_MASTER_NODE_IP
-    --master-port=12345
-    multinode.py 50 10
+  - git clone https://github.com/pytorch/examples.git pytorch-examples
+  - cd pytorch-examples/distributed/ddp-tutorial-series
+  - uv pip install -r requirements.txt
+  - |
+    torchrun \
+      --nproc-per-node=$DSTACK_GPUS_PER_NODE \
+      --node-rank=$DSTACK_NODE_RANK \
+      --nnodes=$DSTACK_NODES_NUM \
+      --master-addr=$DSTACK_MASTER_NODE_IP \
+      --master-port=12345 \
+      multinode.py 50 10
 
 resources:
-  gpu: 24GB
+  gpu: 24GB:1..2
   # Uncomment if using multiple GPUs
   #shm_size: 24GB
 ```
@@ -139,6 +139,15 @@ Nodes can communicate using their private IP addresses.
 Use `DSTACK_MASTER_NODE_IP`, `DSTACK_NODES_IPS`, `DSTACK_NODE_RANK`, and other
 [System environment variables](#system-environment-variables) for inter-node communication.
 
+`dstack` is easy to use with `accelerate`, `torchrun`, Ray, Spark, and any other distributed frameworks.
+
+
+!!! info "MPI"
+    If want to use MPI, you can set `startup_order` to `workers-first` and `stop_criteria` to `master-done`, and use `DSTACK_MPI_HOSTFILE`.
+    See the [NCCL](../../examples/clusters/nccl-tests/index.md) or [RCCL](../../examples/clusters/rccl-tests/index.md) examples.
+
+> For detailed examples, see [distributed training](../../examples.md#distributed-training) examples.
+
 ??? info "Network interface"
     Distributed frameworks usually detect the correct network interface automatically,
     but sometimes you need to specify it explicitly.
@@ -170,7 +179,7 @@ Use `DSTACK_MASTER_NODE_IP`, `DSTACK_NODES_IPS`, `DSTACK_NODE_RANK`, and other
     recommended to create them via a fleet configuration
     to ensure the highest level of inter-node connectivity.
 
-`dstack` is easy to use with `accelerate`, `torchrun`, Ray, Spark, and any other distributed frameworks.
+> See the [Clusters](../guides/clusters.md) guide for more details on how to use `dstack` on clusters.
 
 ### Resources
 
@@ -187,7 +196,7 @@ name: train
 
 # Commands of the task
 commands:
-  - pip install -r fine-tuning/qlora/requirements.txt
+  - uv pip install -r fine-tuning/qlora/requirements.txt
   - python fine-tuning/qlora/train.py
   
 resources:
@@ -224,7 +233,7 @@ If the vendor is not specified, `dstack` tries to infer it from the GPU name usi
     # The name is optional, if not specified, generated randomly
     name: train    
     
-    python: "3.10"
+    python: 3.12
     
     # Commands of the task
     commands:
@@ -258,7 +267,7 @@ type: task
 name: train    
 
 # If `image` is not specified, dstack uses its base image
-python: "3.10"
+python: 3.12
 
 # Commands of the task
 commands:
@@ -279,7 +288,7 @@ commands:
     name: train    
 
     # If `image` is not specified, dstack uses its base image
-    python: "3.10"
+    python: 3.12
     # Ensure nvcc is installed (req. for Flash Attention) 
     nvcc: true
     
@@ -346,7 +355,7 @@ type: task
 # The name is optional, if not specified, generated randomly
 name: train
 
-python: "3.10"
+python: 3.12
 
 # Environment variables
 env:
@@ -380,19 +389,6 @@ If you don't assign a value to an environment variable (see `HF_TOKEN` above),
     | `DSTACK_NODES_IPS`      | The list of internal IP addresses of all nodes delimited by "\n" |
     | `DSTACK_MPI_HOSTFILE`   | The path to a pre-populated MPI hostfile                         |
 
-### Spot policy
-
-By default, `dstack` uses on-demand instances. However, you can change that
-via the [`spot_policy`](../reference/dstack.yml/task.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`.
-
-!!! info "Reference"
-    Tasks support many more configuration options,
-    incl. [`backends`](../reference/dstack.yml/task.md#backends), 
-    [`regions`](../reference/dstack.yml/task.md#regions), 
-    [`max_price`](../reference/dstack.yml/task.md#max_price), and
-    [`max_duration`](../reference/dstack.yml/task.md#max_duration), 
-    among [others](../reference/dstack.yml/task.md).
-
 ### Retry policy
 
 By default, if `dstack` can't find capacity, or the task exits with an error, or the instance is interrupted, 
@@ -405,14 +401,12 @@ If you'd like `dstack` to automatically retry, configure the
 
 ```yaml
 type: task
-# The name is optional, if not specified, generated randomly
 name: train    
 
-python: "3.10"
+python: 3.12
 
-# Commands of the task
 commands:
-  - pip install -r fine-tuning/qlora/requirements.txt
+  - uv pip install -r fine-tuning/qlora/requirements.txt
   - python fine-tuning/qlora/train.py
 
 retry:
@@ -439,11 +433,10 @@ This can be done by specifying the [`priority`](../reference/dstack.yml/task.md)
 type: task
 name: train
 
-python: "3.10"
+python: 3.12
 
-# Commands of the task
 commands:
-  - pip install -r fine-tuning/qlora/requirements.txt
+  - uv pip install -r fine-tuning/qlora/requirements.txt
   - python fine-tuning/qlora/train.py
 
 priority: 50
@@ -455,8 +448,49 @@ priority: 50
 Note that if a high priority run cannot be scheduled,
 it does not block other runs with lower priority from scheduling.
 
+### Utilization policy
+
+Sometimes it’s useful to track whether a task is fully utilizing all GPUs. While you can check this with
+[`dstack metrics`](../reference/cli/dstack/metrics.md), `dstack` also lets you set a policy to auto-terminate the run if any GPU is underutilized.
+
+Below is an example of a task that auto-terminate if any GPU stays below 10% utilization for 1 hour.
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: task
+name: train
+
+python: 3.12
+commands:
+  - uv pip install -r fine-tuning/qlora/requirements.txt
+  - python fine-tuning/qlora/train.py
+
+resources:
+  gpu: H100:8
+
+utilization_policy:
+  min_gpu_utilization: 10
+  time_window: 1h
+```
+
+</div>
+
+### Spot policy
+
+By default, `dstack` uses on-demand instances. However, you can change that
+via the [`spot_policy`](../reference/dstack.yml/task.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`.
+
 --8<-- "docs/concepts/snippets/manage-fleets.ext"
 
+!!! info "Reference"
+    Tasks support many more configuration options,
+    incl. [`backends`](../reference/dstack.yml/task.md#backends), 
+    [`regions`](../reference/dstack.yml/task.md#regions), 
+    [`max_price`](../reference/dstack.yml/task.md#max_price), and
+    [`max_duration`](../reference/dstack.yml/task.md#max_duration), 
+    among [others](../reference/dstack.yml/task.md).
+
 --8<-- "docs/concepts/snippets/manage-runs.ext"
 
 !!! info "What's next?"