dstack 0.19.11rc2__tar.gz → 0.19.12__tar.gz

{dstack-0.19.11rc2 → dstack-0.19.12}/.github/workflows/build.yml

@@ -293,7 +293,7 @@ jobs:
       - name: Build
         run: |
           uv pip install pillow cairosvg
-          sudo apt-get install -y libcairo2-dev libfreetype6-dev libffi-dev libjpeg-dev libpng-dev libz-dev
+          sudo apt-get update && sudo apt-get install -y libcairo2-dev libfreetype6-dev libffi-dev libjpeg-dev libpng-dev libz-dev
           uv pip install mkdocs-material "mkdocs-material[imaging]" mkdocs-material-extensions mkdocs-redirects mkdocs-gen-files "mkdocstrings[python]" mkdocs-render-swagger-plugin --upgrade
           uv pip install git+https://${{ secrets.GH_TOKEN }}@github.com/squidfunk/mkdocs-material-insiders.git
           uv run mkdocs build -s

{dstack-0.19.11rc2 → dstack-0.19.12}/.github/workflows/docs.yaml

@@ -25,7 +25,7 @@ jobs:
       - name: Build
         run: |
           uv pip install pillow cairosvg
-          sudo apt-get install -y libcairo2-dev libfreetype6-dev libffi-dev libjpeg-dev libpng-dev libz-dev
+          sudo apt-get update && sudo apt-get install -y libcairo2-dev libfreetype6-dev libffi-dev libjpeg-dev libpng-dev libz-dev
           uv pip install mkdocs-material "mkdocs-material[imaging]" mkdocs-material-extensions mkdocs-redirects mkdocs-gen-files "mkdocstrings[python]" mkdocs-render-swagger-plugin --upgrade
           uv pip install git+https://${{ secrets.GH_TOKEN }}@github.com/squidfunk/mkdocs-material-insiders.git
           uv run mkdocs build -s

dstack-0.19.12/.pre-commit-config.yaml

@@ -0,0 +1,38 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.11.6  # Should match pyproject.toml
+    hooks:
+      - id: ruff
+        name: ruff common
+        args: ['--fix']
+      - id: ruff-format
+  - repo: https://github.com/golangci/golangci-lint
+    rev: v1.62.0  # Should match .github/workflows/build.yml
+    hooks:
+      - id: golangci-lint-full
+        language_version: 1.23.0  # Should match runner/go.mod
+        entry: bash -c 'cd runner && golangci-lint run'
+        stages: [manual]
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: end-of-file-fixer
+
+  # TODO(oleg): fix ESLint warnings
+  # - repo: https://github.com/pre-commit/mirrors-eslint
+  #   rev: 'v8.31.0'
+  #   hooks:
+  #     - id: eslint
+  #       name: ESLint
+  #       entry: eslint --fix --config frontend/.eslintrc --ignore-path frontend/.eslintignore
+  #       language: node
+  #       pass_filenames: false
+  #       cwd: "frontend/"
+  #       additional_dependencies:
+  #         - eslint@8.31.0
+  #         - eslint-config-prettier@8.10.0
+  #         - eslint-plugin-i18n@2.4.0
+  #         - eslint-plugin-prettier@4.2.1
+  #         - eslint-plugin-simple-import-sort@10.0.0
+  #         - '@typescript-eslint/eslint-plugin@5.48.1'
+  #         - '@typescript-eslint/parser@5.48.1'

{dstack-0.19.11rc2 → dstack-0.19.12}/CONTRIBUTING.md

@@ -54,6 +54,8 @@ uv run pytest src/tests --runpostgres
 
 If you'd like to integrate a new cloud provider to `dstack`, follow [contributing/BACKENDS.md](contributing/BACKENDS.md).
 
-## Get help
+## What's next
+
+You can find more subject-focused guides in the [contributing](contributing/) directory.
 
 If you have any questions, you can always get help in our [Discord](https://discord.gg/u8SmfwPpMd) community.

{dstack-0.19.11rc2 → dstack-0.19.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dstack
-Version: 0.19.11rc2
+Version: 0.19.12
 Summary: dstack is an open-source orchestration engine for running AI workloads on any cloud or on-premises.
 Project-URL: Homepage, https://dstack.ai
 Project-URL: Source, https://github.com/dstackai/dstack
@@ -22,6 +22,7 @@ Requires-Dist: cursor
 Requires-Dist: filelock
 Requires-Dist: gitpython
 Requires-Dist: gpuhunt==0.1.6
+Requires-Dist: ignore-python>=0.2.0
 Requires-Dist: jsonschema
 Requires-Dist: packaging
 Requires-Dist: paramiko>=3.2.0

dstack-0.19.12/contributing/PROXY.md

@@ -0,0 +1,207 @@
+# `dstack-proxy`
+
+`dstack-proxy` is a set of `dstack` components responsible for exposing [services](https://dstack.ai/docs/concepts/services/).
+
+- By default, services are published at `dstack` server URL subpaths. The component that handles traffic to such services is called **in-server proxy**. It runs as part of the `dstack` server. It is implemented in `dstack._internal.server.services.proxy`.
+- Users can optionally deploy a **gateway** to handle traffic to their services. The gateway app runs on a dedicated instance. Although it requires additional configuration, it provides higher performance and supports more features than the in-server proxy. It is implemented in `dstack._internal.proxy.gateway`.
+- The in-server proxy and the gateway share some business logic, such as the OpenAI-compatible API and the connection pool implementation. The common details are implemented in `dstack._internal.proxy.lib`.
+
+## Proxy functions and modules
+
+### Reverse proxy
+
+`dstack-proxy` acts as a reverse proxy and load balancer for services.
+
+The in-server proxy uses a custom reverse proxy implementation based on FastAPI and httpx. It routes requests based on the `/proxy/services/<project>/<service>` path. It performs load balancing by selecting a random service replica for each forwarded request.
+
+The gateway uses Nginx. It automatically maintains Nginx configs for each service in `/etc/nginx/sites-enabled/*`. Each service is published at a subdomain. Nginx performs load balancing using the round-robin method. Nginx forwards requests directly to service replicas, so traffic does not go through the gateway app.
+
+### HTTPS
+
+The in-server proxy is part of the `dstack` server, so services are only available over HTTPS if the `dstack` server is deployed with HTTPS.
+
+The gateway can enforce HTTPS in the Nginx config. It uses Certbot to obtain TLS certificates from Let's Encrypt or another configured CA when the service is registered, unless the service configuration specifies `https: false`.
+
+### Auth
+
+Unless the service configuration specifies `auth: false`, `dstack-proxy` checks the authorization of incoming requests based on `dstack` user tokens from the `Authorization` header.
+
+The in-server proxy validates the tokens by querying the correct token from the database.
+
+On gateways, Nginx makes a headers-only subrequest to the gateway app to check authorization for each incoming request. The gateway app then makes a request to the `dstack` server to validate the token. Responses from the `dstack` server are cached for 60 seconds. If the `dstack` server does not respond, the incoming request is denied.
+
+### OpenAI-compatible API
+
+The OpenAI interface emulates the real OpenAI API for chat completion models. It can list running models in the project, convert between OpenAI and TGI request formats, and forward requests to the correct service.
+
+The in-server proxy forwards requests directly to service replicas.
+
+```mermaid
+sequenceDiagram
+    User->>dstack server: http(s)://<server>/proxy/models/<project>/
+    dstack server->>Service replica: http+unix:///path/to/ssh/socket
+```
+
+The gateway uses Nginx to forward requests so they are included in access logs and service stats.
+
+```mermaid
+sequenceDiagram
+    User->>Nginx: http(s)://gateway.<gateway>/
+    Nginx->>Gateway app: http://localhost:8000/api/models/<project>/
+    Gateway app->>Nginx: http://<service>.<gateway>/
+    Nginx->>Service replica: http+unix:///path/to/ssh/socket
+```
+
+### Stats collector
+
+`dstack-proxy` collects service usage stats that are then used by the `dstack` server for autoscaling. Stats collection is only supported on gateways and is implemented by reading `/var/log/nginx/dstack.access.log`
+
+### Service connection pool
+
+`dstack-proxy` connects to service replicas via SSH and forwards their service port to a local Unix socket, which is then used by the reverse proxy. All SSH connections are added to a pool and reused between requests.
+
+The in-server proxy opens an SSH connection when it needs to forward the first request to it, so there may be a delay when forwarding the first request.
+
+The gateway opens SSH connections when each replica is registered.
+
+### Communication with the `dstack` server
+
+The in-server proxy is part of the `dstack` server, so no network communication takes place between them.
+
+Gateway-to-server communication happens over an SSH connection that is established by the `dstack` server when it creates a new gateway or starts. The SSH connection includes bidirectional port forwarding, enabling the gateway and the `dstack` server to call each other's APIs:
+
+- The server calls the gateway to register services, fetch stats, etc.
+- The gateway calls the server to validate user tokens.
+
+### Storage
+
+`dstack-proxy` has a set of stored models and a common storage repo interface.
+
+The in-server proxy repo implementation fetches the models from the database. It doesn't write to the database, as all the details about services and replicas are written by the `dstack` server.
+
+The gateway maintains its own in-memory storage repo. A copy of the repo is also stored in `~/dstack/state-v2.json`. The copy is updated on every write operation and is used for data recovery after restarts. The repo is populated when new services and replicas are registered.
+
+## Dependency injection
+
+When a module has to be implemented differently for the in-server proxy and the gateway, `dstack-proxy` uses interfaces with multiple implementations. For example, there are common interfaces for the storage repo and for auth checks. The in-server proxy and the gateway provide their own implementations for both interfaces.
+
+```mermaid
+classDiagram
+    class BaseProxyRepo {
+        <<abstract>>
+    }
+    BaseProxyRepo <|-- ServerProxyRepo
+    BaseProxyRepo <|-- GatewayProxyRepo
+
+    class BaseProxyAuthProvider {
+        <<abstract>>
+    }
+    BaseProxyAuthProvider <|-- ServerProxyAuthProvider
+    BaseProxyAuthProvider <|-- GatewayProxyAuthProvider
+```
+
+`dstack-proxy` then uses dependency injection to select the relevant implementation. Both the in-server proxy and the gateway provide an "injector" class that is used to obtain concrete interface implementations.
+
+```mermaid
+classDiagram
+    class ProxyDependencyInjector {
+        <<abstract>>
+        +get_repo() BaseProxyRepo
+        +get_auth_provider() BaseProxyAuthProvider
+    }
+    class ServerProxyDependencyInjector {
+        +get_repo() ServerProxyRepo
+        +get_auth_provider() ServerProxyAuthProvider
+    }
+    class GatewayDependencyInjector {
+        +get_repo() GatewayProxyRepo
+        +get_auth_provider() GatewayProxyAuthProvider
+    }
+    ProxyDependencyInjector <|-- ServerProxyDependencyInjector
+    ProxyDependencyInjector <|-- GatewayDependencyInjector
+```
+
+An instance of the relevant injector class is stored in the FastAPI global app state and can be accessed from FastAPI path operations. There are helper functions that can be used as FastAPI path operation dependencies to obtain the injector or a module implementation: `get_injector`, `get_proxy_repo`, `get_proxy_auth_provider`, etc.
+
+There are also similar gateway-specific helper functions to obtain gateway-specific module implementations: `get_gateway_proxy_repo` (guaranteed to return the gateway repo, which has more methods than the base repo interface), `get_nginx`, `get_stats_collector`, etc.
+
+## Gateway operations
+
+Gateway instances are managed by the `dstack` server. A gateway is associated with a project and some backend in the project. In `dstack` Sky, there is also a global gateway associated with all projects.
+
+### Creation
+
+Users can create a gateway using the `dstack apply` command. The gateway YAML configuration must specify the gateway's wildcard domain - all direct subdomains should resolve to the gateway IP address. Since the IP address is unknown during provisioning, `dstack` doesn't check DNS records.
+
+Provisioning happens as follows:
+1. Launch a non-GPU instance (usually the smallest) with all ports exposed.
+2. Install Nginx, Certbot, and patch configs.
+3. Create blue-green virtual environments.
+4. Install the latest `dstack-gateway` package from the S3 bucket. `dstack-gateway` is a thin package that depends on the `dstack` package, which contains the actual gateway implementation.
+5. Run the systemd service `dstack.gateway.service`.
+
+### Update
+
+The gateway has a "blue-green deployment"-like configuration: there are two virtual environments to be swapped on update. The systemd service uses the newly installed package after a restart.
+
+The update process looks like this:
+1. Install the new package to the unused venv.
+2. Update scripts and systemd service config.
+3. Swap the active venv name in `~/dstack/version`.
+4. Restart the systemd service.
+
+## Gateway development
+
+The gateway app needs to interact with Nginx and certbot, so running it locally can be challenging. One way to test your code is to upload your development branch to an existing gateway and run the gateway app from source.
+
+1. Run `dstack server` with `DSTACK_SKIP_GATEWAY_UPDATE=1` environment variable. This will prevent `dstack` from updating and starting the standard gateway version on each server restart.
+
+1. Provision a gateway through `dstack`:
+
+   ```shell
+   dstack apply -f my-gateway.dstack.yml
+   ```
+
+1. Save the gateway key to a file:
+
+   ```shell
+   sqlite3 ~/.dstack/server/data/sqlite.db "SELECT ssh_private_key FROM gateway_computes WHERE deleted = 0 AND ip_address = '<gateway-ip-addr>'" > /tmp/gateway.key
+   chmod 600 /tmp/gateway.key
+   ```
+
+1. Deliver your code to the gateway. For example, clone it from a remote repo:
+
+   ```shell
+   ssh -i /tmp/gateway.key ubuntu@gateway.example "git clone https://github.com/dstackai/dstack.git ~/dstack-repo"
+   ```
+
+   Or push it from your machine:
+
+   ```shell
+   ssh -i /tmp/gateway.key ubuntu@gateway.example "git init ~/dstack-repo"
+   git remote add gateway ubuntu@gateway.example:~/dstack-repo
+   GIT_SSH_COMMAND='ssh -i /tmp/gateway.key' git push gateway branch_name
+   ```
+
+1. Connect to the gateway:
+
+   ```shell
+   ssh -i /tmp/gateway.key ubuntu@gateway.example
+   ```
+
+1. Prepare an environment with your development branch on the gateway:
+
+   ```shell
+   cd ~/dstack-repo
+   git checkout branch_name
+   curl -LsSf https://astral.sh/uv/install.sh | sh
+   source ~/.local/bin/env
+   uv sync --extra gateway
+   ```
+
+1. Stop the gateway service and start your development version from source:
+
+   ```shell
+   sudo systemctl stop dstack.gateway.service
+   uv run uvicorn dstack._internal.proxy.gateway.main:app
+   ```

{dstack-0.19.11rc2 → dstack-0.19.12}/contributing/RUNS-AND-JOBS.md

@@ -11,7 +11,7 @@ Runs are created from run configurations. There are three types of run configura
 
 1. `dev-environment` — runs a VS Code server.
 2. `task` — runs the user's bash script until completion.
-3. `service` — runs the user's bash script and exposes a port through the [gateway](GATEWAY.md) or the server built-in proxy.
+3. `service` — runs the user's bash script and exposes a port through [dstack-proxy](PROXY.md).
 
 A run can spawn one or multiple jobs, depending on the configuration. A task that specifies multiple `nodes` spawns a job for every node (a multi-node task). A service that specifies multiple `replicas` spawns a job for every replica. A job submission is always assigned to one particular instance. If a job fails and the configuration allows retrying, the server creates a new job submission for the job.
 

{dstack-0.19.11rc2 → dstack-0.19.12}/docs/blog/posts/intel-gaudi.md

@@ -98,7 +98,7 @@ model using [Optimum for Intel Gaudi :material-arrow-top-right-thin:{ .external
 and [DeepSpeed :material-arrow-top-right-thin:{ .external }](https://docs.habana.ai/en/latest/PyTorch/DeepSpeed/DeepSpeed_User_Guide/DeepSpeed_User_Guide.html#deepspeed-user-guide){:target="_blank"} with 
 the [`lvwerra/stack-exchange-paired` :material-arrow-top-right-thin:{ .external }](https://huggingface.co/datasets/lvwerra/stack-exchange-paired){:target="_blank"} dataset:
 
-<div editor-title="examples/fine-tuning/trl/intel/.dstack.yml">
+<div editor-title="examples/single-node-training/trl/intel/.dstack.yml">
     
 ```yaml
 type: task
@@ -152,7 +152,7 @@ Submit the task using the [`dstack apply`](../../docs/reference/cli/dstack/apply
 <div class="termy">
 
 ```shell
-$ dstack apply -f examples/fine-tuning/trl/intel/.dstack.yml -R
+$ dstack apply -f examples/single-node-training/trl/intel/.dstack.yml -R
 ```
 
 </div>

{dstack-0.19.11rc2 → dstack-0.19.12}/docs/blog/posts/tpu-on-gcp.md

@@ -158,7 +158,7 @@ Below is an example of fine-tuning Llama 3.1 8B using [Optimum TPU :material-arr
 and the [Abirate/english_quotes :material-arrow-top-right-thin:{ .external }](https://huggingface.co/datasets/Abirate/english_quotes){:target="_blank"}
 dataset.
 
-<div editor-title="examples/fine-tuning/optimum-tpu/llama31/train.dstack.yml"> 
+<div editor-title="examples/single-node-training/optimum-tpu/llama31/train.dstack.yml"> 
 
 ```yaml
 type: task
@@ -171,8 +171,8 @@ env:
 commands:
   - git clone -b add_llama_31_support https://github.com/dstackai/optimum-tpu.git
   - mkdir -p optimum-tpu/examples/custom/
-  - cp examples/fine-tuning/optimum-tpu/llama31/train.py optimum-tpu/examples/custom/train.py
-  - cp examples/fine-tuning/optimum-tpu/llama31/config.yaml optimum-tpu/examples/custom/config.yaml
+  - cp examples/single-node-training/optimum-tpu/llama31/train.py optimum-tpu/examples/custom/train.py
+  - cp examples/single-node-training/optimum-tpu/llama31/config.yaml optimum-tpu/examples/custom/config.yaml
   - cd optimum-tpu
   - pip install -e . -f https://storage.googleapis.com/libtpu-releases/index.html
   - pip install datasets evaluate

{dstack-0.19.11rc2 → dstack-0.19.12}/docs/docs/concepts/tasks.md

@@ -10,7 +10,7 @@ The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml`
 
 [//]: # (TODO: Make tabs - single machine & distributed tasks & web app)
 
-<div editor-title="examples/fine-tuning/axolotl/train.dstack.yml"> 
+<div editor-title="examples/single-node-training/axolotl/train.dstack.yml"> 
 
 ```yaml
 type: task
@@ -26,7 +26,7 @@ env:
   - WANDB_API_KEY
 # Commands of the task
 commands:
-  - accelerate launch -m axolotl.cli.train examples/fine-tuning/axolotl/config.yaml
+  - accelerate launch -m axolotl.cli.train examples/single-node-training/axolotl/config.yaml
 
 resources:
   gpu:
@@ -378,6 +378,7 @@ If you don't assign a value to an environment variable (see `HF_TOKEN` above),
     | `DSTACK_NODE_RANK`      | The rank of the node                                             |
     | `DSTACK_MASTER_NODE_IP` | The internal IP address of the master node                          |
     | `DSTACK_NODES_IPS`      | The list of internal IP addresses of all nodes delimited by "\n" |
+    | `DSTACK_MPI_HOSTFILE`   | The path to a pre-populated MPI hostfile                         |
 
 ### Spot policy
 
@@ -461,4 +462,4 @@ it does not block other runs with lower priority from scheduling.
 !!! info "What's next?"
     1. Read about [dev environments](dev-environments.md), [services](services.md), and [repos](repos.md)
     2. Learn how to manage [fleets](fleets.md)
-    3. Check the [Axolotl](/examples/fine-tuning/axolotl) example
+    3. Check the [Axolotl](/examples/single-node-training/axolotl) example

{dstack-0.19.11rc2 → dstack-0.19.12}/docs/docs/reference/environment-variables.md

@@ -77,6 +77,7 @@ tasks, and services:
      ```
 
 - `DSTACK_NODES_IPS`{ #DSTACK_NODES_IPS } – The list of internal IP addresses of all nodes delimited by `"\n"`.
+- `DSTACK_MPI_HOSTFILE`{ #DSTACK_MPI_HOSTFILE } – The path to a pre-populated MPI hostfile that can be used directly as `mpirun --hostfile $DSTACK_MPI_HOSTFILE`.
 
 ## Server
 

{dstack-0.19.11rc2 → dstack-0.19.12}/docs/examples.md

@@ -12,10 +12,21 @@ hide:
 }
 </style>
 
-## Fine-tuning
+## Single-node training
 
 <div class="tx-landing__highlights_grid">
-    <a href="/examples/fine-tuning/axolotl"
+    <a href="/examples/single-node-training/trl"
+       class="feature-cell">
+        <h3>
+            TRL
+        </h3>
+
+        <p>
+            Fine-tune Llama 3.1 8B on a custom dataset using TRL.
+        </p>
+    </a>
+
+    <a href="/examples/single-node-training/axolotl"
        class="feature-cell">
         <h3>
             Axolotl
@@ -25,19 +36,47 @@ hide:
             Fine-tune Llama 4 on a custom dataset using Axolotl.
         </p>
     </a>
+</div>
 
-    <a href="/examples/fine-tuning/trl"
-       class="feature-cell">
+## Distributed training
+
+<div class="tx-landing__highlights_grid">
+    <a href="/examples/distributed-training/trl"
+       class="feature-cell sky">
         <h3>
             TRL
         </h3>
 
         <p>
-            Fine-tune Llama 3.1 8B on a custom dataset using TRL.
+            Fine-tune LLM on multiple nodes
+            with TRL, Accelerate, and Deepspeed.
+        </p>
+    </a>
+    <a href="/examples/distributed-training/axolotl"
+       class="feature-cell sky">
+        <h3>
+            Axolotl
+        </h3>
+
+        <p>
+            Fine-tune LLM on multiple nodes
+            with Axolotl.
+        </p>
+    </a>
+    <a href="/examples/distributed-training/ray-ragen"
+       class="feature-cell sky">
+        <h3>
+            Ray+RAGEN
+        </h3>
+
+        <p>
+            Fine-tune an agent on multiple nodes
+            with RAGEN, verl, and Ray.
         </p>
     </a>
 </div>
 
+
 ## Clusters
 
 <div class="tx-landing__highlights_grid">
@@ -83,22 +122,6 @@ hide:
     </a>
 </div>
 
-## Distributed training
-
-<div class="tx-landing__highlights_grid">
-    <a href="/examples/distributed-training/ray-ragen"
-       class="feature-cell sky">
-        <h3>
-            Ray+RAGEN
-        </h3>
-
-        <p>
-            Fine-tune an agent on multiple nodes
-            with RAGEN, verl, and Ray.
-        </p>
-    </a>
-</div>
-
 ## Inference
 
 <div class="tx-landing__highlights_grid">
@@ -197,31 +220,6 @@ hide:
     </a>
 </div>
 
-## LLMs
-
-<div class="tx-landing__highlights_grid">
-    <a href="/examples/llms/deepseek"
-       class="feature-cell sky">
-        <h3>
-            Deepseek
-        </h3>
-
-        <p>
-            Deploy and train Deepseek models
-        </p>
-    </a>
-    <a href="/examples/llms/llama"
-       class="feature-cell sky">
-        <h3>
-            Llama
-        </h3>
-
-        <p>
-            Deploy Llama 4 models
-        </p>
-    </a>
-</div>
-
 ## Misc
 
 <div class="tx-landing__highlights_grid">

{dstack-0.19.11rc2 → dstack-0.19.12}/docs/overrides/main.html

@@ -117,12 +117,11 @@ Support dstack with a ⭐ on
 
                 <div class="tx-footer__section">
                     <div class="tx-footer__section-title">Examples</div>
-                    <a href="/examples#fine-tuning" class="tx-footer__section-link">Fine-tuning</a>
-                    <a href="/examples#clusters" class="tx-footer__section-link">Clusters</a>
+                    <a href="/examples#fine-tuning" class="tx-footer__section-link">Single-node training</a>
                     <a href="/examples#distributed-training" class="tx-footer__section-link">Distributed training</a>
+                    <a href="/examples#clusters" class="tx-footer__section-link">Clusters</a>
                     <a href="/examples#inference" class="tx-footer__section-link">Inference</a>
                     <a href="/examples#accelerators" class="tx-footer__section-link">Accelerators</a>
-                    <a href="/examples#llms" class="tx-footer__section-link">LLMs</a>
                     <!-- <a href="/examples#misc" class="tx-footer__section-link">Misc</a> -->
                 </div>
 

{dstack-0.19.11rc2 → dstack-0.19.12}/examples/accelerators/amd/README.md

@@ -114,7 +114,7 @@ To request multiple GPUs, specify the quantity after the GPU name, separated by
     and the [`mlabonne/guanaco-llama2-1k` :material-arrow-top-right-thin:{ .external }](https://huggingface.co/datasets/mlabonne/guanaco-llama2-1k){:target="_blank"}
     dataset.
     
-    <div editor-title="examples/fine-tuning/trl/amd/.dstack.yml">
+    <div editor-title="examples/single-node-training/trl/amd/.dstack.yml">
     
     ```yaml
     type: task
@@ -140,7 +140,7 @@ To request multiple GPUs, specify the quantity after the GPU name, separated by
       - pip install peft
       - pip install transformers datasets huggingface-hub scipy
       - cd ..
-      - python examples/fine-tuning/trl/amd/train.py
+      - python examples/single-node-training/trl/amd/train.py
     
     # Uncomment to leverage spot instances
     #spot_policy: auto
@@ -157,7 +157,7 @@ To request multiple GPUs, specify the quantity after the GPU name, separated by
     and the [tatsu-lab/alpaca :material-arrow-top-right-thin:{ .external }](https://huggingface.co/datasets/tatsu-lab/alpaca){:target="_blank"}
     dataset.
     
-    <div editor-title="examples/fine-tuning/axolotl/amd/.dstack.yml">
+    <div editor-title="examples/single-node-training/axolotl/amd/.dstack.yml">
     
     ```yaml
     type: task
@@ -213,7 +213,7 @@ To request multiple GPUs, specify the quantity after the GPU name, separated by
 
     > To speed up installation of `flash-attention` and `xformers `, we use pre-built binaries uploaded to S3. 
     > You can find the tasks that build and upload the binaries
-    > in [`examples/fine-tuning/axolotl/amd/` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/axolotl/amd/){:target="_blank"}.
+    > in [`examples/single-node-training/axolotl/amd/` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/single-node-training/axolotl/amd/){:target="_blank"}.
 
 ## Running a configuration
 
@@ -238,8 +238,8 @@ $ dstack apply -f examples/inference/vllm/amd/.dstack.yml
 The source-code of this example can be found in 
 [`examples/inference/tgi/amd` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/inference/tgi/amd){:target="_blank"},
 [`examples/inference/vllm/amd` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/inference/vllm/amd){:target="_blank"},
-[`examples/fine-tuning/axolotl/amd` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/axolotl/amd){:target="_blank"} and
-[`examples/fine-tuning/trl/amd` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/trl/amd){:target="_blank"}
+[`examples/single-node-training/axolotl/amd` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/single-node-training/axolotl/amd){:target="_blank"} and
+[`examples/single-node-training/trl/amd` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/single-node-training/trl/amd){:target="_blank"}
 
 ## What's next?
 

{dstack-0.19.11rc2 → dstack-0.19.12}/examples/accelerators/intel/README.md

@@ -102,7 +102,7 @@ using [Optimum for Intel Gaudi :material-arrow-top-right-thin:{ .external }](htt
 and [DeepSpeed :material-arrow-top-right-thin:{ .external }](https://docs.habana.ai/en/latest/PyTorch/DeepSpeed/DeepSpeed_User_Guide/DeepSpeed_User_Guide.html#deepspeed-user-guide){:target="_blank"} with 
 the [`lvwerra/stack-exchange-paired` :material-arrow-top-right-thin:{ .external }](https://huggingface.co/datasets/lvwerra/stack-exchange-paired){:target="_blank"} dataset. 
     
-<div editor-title="examples/fine-tuning/trl/intel/.dstack.yml">
+<div editor-title="examples/single-node-training/trl/intel/.dstack.yml">
     
 ```yaml
 type: task

{dstack-0.19.11rc2 → dstack-0.19.12}/examples/accelerators/tpu/README.md

@@ -127,7 +127,7 @@ Below is an example of fine-tuning Llama 3.1 8B using [Optimum TPU :material-arr
 and the [`Abirate/english_quotes` :material-arrow-top-right-thin:{ .external }](https://huggingface.co/datasets/Abirate/english_quotes){:target="_blank"}
 dataset.
 
-<div editor-title="examples/fine-tuning/optimum-tpu/llama31/.dstack.yml"> 
+<div editor-title="examples/single-node-training/optimum-tpu/llama31/.dstack.yml"> 
 
 ```yaml
 type: task
@@ -139,8 +139,8 @@ env:
 commands:
   - git clone -b add_llama_31_support https://github.com/dstackai/optimum-tpu.git
   - mkdir -p optimum-tpu/examples/custom/
-  - cp examples/fine-tuning/optimum-tpu/llama31/train.py optimum-tpu/examples/custom/train.py
-  - cp examples/fine-tuning/optimum-tpu/llama31/config.yaml optimum-tpu/examples/custom/config.yaml
+  - cp examples/single-node-training/optimum-tpu/llama31/train.py optimum-tpu/examples/custom/train.py
+  - cp examples/single-node-training/optimum-tpu/llama31/config.yaml optimum-tpu/examples/custom/config.yaml
   - cd optimum-tpu
   - pip install -e . -f https://storage.googleapis.com/libtpu-releases/index.html
   - pip install datasets evaluate
@@ -155,7 +155,7 @@ resources:
 </div>
 
 [//]: # (### Fine-Tuning with TRL)
-[//]: # (Use the example `examples/fine-tuning/optimum-tpu/gemma/train.dstack.yml` to Finetune `Gemma-2B` model using `trl` with `dstack` and `optimum-tpu`. )
+[//]: # (Use the example `examples/single-node-training/optimum-tpu/gemma/train.dstack.yml` to Finetune `Gemma-2B` model using `trl` with `dstack` and `optimum-tpu`. )
 
 ### Memory requirements
 
@@ -181,7 +181,7 @@ Note, `v5litepod` is optimized for fine-tuning transformer-based models. Each co
 The source-code of this example can be found in 
 [`examples/inference/tgi/tpu` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/inference/tgi/tpu){:target="_blank"},
 [`examples/inference/vllm/tpu` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/inference/vllm/tpu){:target="_blank"},
-and [`examples/fine-tuning/optimum-tpu` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/trl){:target="_blank"}.
+and [`examples/single-node-training/optimum-tpu` :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/single-node-training/trl){:target="_blank"}.
 
 ## What's next?
 

dstack-0.19.12/examples/clusters/nccl-tests/.dstack.yml

@@ -0,0 +1,28 @@
+type: task
+name: nccl-tests
+
+nodes: 2
+startup_order: workers-first
+stop_criteria: master-done
+
+image: dstackai/efa
+env:
+  - NCCL_DEBUG=INFO
+commands:
+  - |
+    if [ ${DSTACK_NODE_RANK} -eq 0 ]; then
+      cd /root/nccl-tests/build
+      MPIRUN="mpirun --allow-run-as-root --hostfile $DSTACK_MPI_HOSTFILE"
+      # Run NCCL Tests
+      ${MPIRUN} \
+        -n ${DSTACK_GPUS_NUM} -N ${DSTACK_GPUS_PER_NODE} \
+        --mca btl_tcp_if_exclude lo,docker0 \
+        --bind-to none \
+        ./all_reduce_perf -b 8 -e 8G -f 2 -g 1
+    else
+      sleep infinity
+    fi
+
+resources:
+  gpu: nvidia:4:16GB
+  shm_size: 16GB