dstack 0.19.10__tar.gz → 0.19.11rc2__tar.gz

{dstack-0.19.10 → dstack-0.19.11rc2}/.github/workflows/build.yml

@@ -287,7 +287,8 @@ jobs:
           python-version: 3.11
       - name: Install dstack
         run: |
-          uv sync --extra server
+          uv pip install examples/plugins/example_plugin_server
+          uv pip install -e '.[server]'
       # Move these deps into an extra and install that way
       - name: Build
         run: |

{dstack-0.19.10 → dstack-0.19.11rc2}/.github/workflows/docs.yaml

@@ -16,10 +16,11 @@ jobs:
           python-version: 3.11
       - name: Install dstack
         run: |
+          uv pip install examples/plugins/example_plugin_server
           if [ -n "${{ inputs.release_tag }}" ]; then
             uv pip install "dstack[server]==${{ inputs.release_tag }}"
           else
-            uv sync --extra server
+            uv pip install -e '.[server]'
           fi
       - name: Build
         run: |

{dstack-0.19.10 → dstack-0.19.11rc2}/.gitignore

@@ -24,3 +24,4 @@ build/
 .vscode
 .aider*
 uv.lock
+.local/

{dstack-0.19.10 → dstack-0.19.11rc2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dstack
-Version: 0.19.10
+Version: 0.19.11rc2
 Summary: dstack is an open-source orchestration engine for running AI workloads on any cloud or on-premises.
 Project-URL: Homepage, https://dstack.ai
 Project-URL: Source, https://github.com/dstackai/dstack
@@ -22,7 +22,6 @@ Requires-Dist: cursor
 Requires-Dist: filelock
 Requires-Dist: gitpython
 Requires-Dist: gpuhunt==0.1.6
-Requires-Dist: humanize>=4.12.3
 Requires-Dist: jsonschema
 Requires-Dist: packaging
 Requires-Dist: paramiko>=3.2.0

dstack-0.19.11rc2/docs/blog/posts/ea-gtc25.md

@@ -0,0 +1,87 @@
+---
+title: "Case study: how EA uses dstack to fast-track AI development"
+date: 2025-05-22
+description: "TBA"  
+slug: ea-gtc25
+image: https://dstack.ai/static-assets/static-assets/images/dstack-ea-slide-2-background-min.png
+categories:
+  - Case studies
+  - NVIDIA
+---
+
+# Case study: how EA uses dstack to fast-track AI development
+
+At NVIDIA GTC 2025, Electronic Arts [shared :material-arrow-top-right-thin:{ .external }](https://www.nvidia.com/en-us/on-demand/session/gtc25-s73667/){:target="_blank"} how they’re scaling AI development and managing infrastructure across teams. They highlighted using tools like `dstack` to provision GPUs quickly, flexibly, and cost-efficiently. This case study summarizes key insights from their talk.
+
+<img src="https://dstack.ai/static-assets/static-assets/images/dstack-ea-slide-1.png" width="630" />
+
+EA has over 100+ AI projects running, and the number keeps growing. There are many teams with AI needs—game dev, ML engineers, AI researchers, and platform teams—supported by a central tech team. Some need full MLOps support; others have in-house expertise but need flexible tooling and infrastructure.
+
+<!-- more -->
+
+The central tech team ensures all teams have what they require, including tools, infrastructure, and expertise.
+
+<!-- <img src="https://dstack.ai/static-assets/static-assets/images/dstack-ea-slide-1-1.png" width="630" style="border: 0.5px dotted black"/> -->
+
+As EA’s AI efforts grew, they faced major challenges:
+
+* **Tool fragmentation**: Teams used different tools and workflows, leading to duplicated effort and poor collaboration.  
+* **High GPU costs**: Spinning up GPUs could take days or weeks. To avoid delays, teams often left machines running idle, increasing costs.  
+* **Heavy engineering burden**: ML engineers spent time managing infrastructure—setting up clusters, configuring environments, and deploying models—instead of building AI.
+
+The typical AI workflow at EA includes:
+
+1. Development and training  
+2. Model storage and distribution  
+3. Serving and scaling
+
+Each stage comes with scaling challenges, from GPU compute provisioning efficiency to fragmented tooling and complex project setups.
+
+<img src="https://dstack.ai/static-assets/static-assets/images/dstack-ea-slide-2.png" width="630" style="border: 0.5px dotted black"/>
+
+EA's centralized approach uses these core ML tools:
+
+* `dstack` – for provisioning compute for AI workloads at scale, covering everything related to ML development and training  
+* ML Artifactory – for managing  artifacts at scale  
+* AXS (Kubernetes+) – for scalable inference and production serving
+
+EA uses `dstack` to streamline GPU provisioning and AI workflow orchestration. It's open-source, cloud-agnostic, automated, and integrates seamlessly with teams' existing dev workflows.
+
+<!-- In addition to the cloud-agnostic interface for ML teams, dstack eliminates the need for filing infra tickets and waiting days or weeks to get a GPU box or cluster, it just spins up what you need in minutes. -->
+
+> *Because our teams are fragmented, we want them to be able to run on any environment of their choosing... It has to work with all of these. That means a centralized, unified interface to talk to all of them.*
+>
+> *— Wah Loon Keng, Sr. AI Engineer, Electronic Arts*
+
+EA teams use `dstack` for three types of ML workloads:
+
+* [Dev environments](../../docs/concepts/dev-environments.md): spining up GPU boxes pre-setup with a Gitrepo, and ready to use via desktop IDE such as VS Code, Cursor, etc  
+* [Tasks](../../docs/concepts/tasks.md): seamless single-node or distributed training using open-source PyTorch libraries  
+* [Services](../../docs/concepts/services.md): running model endpoints and Streamlit-style apps for quick internal demos and prototyping
+
+Introducing `dstack` had a significant impact on EA’s ML teams. Before, getting access to GPU infrastructure could take days or even weeks. With dstack, teams can now spin up what they need in just minutes. This shift accelerated development by removing delays and freeing engineers to focus on building models.
+
+> *With dstack, what used to take weeks,  provisioning GPUs, setting up environments, now takes minutes. It changed how fast teams at EA can move.*
+>
+> *— Wah Loon Keng, Sr. AI Engineer, Electronic Arts*
+
+Costs dropped by nearly a factor of three, largely due to dstack’s ability to automatically start and stop resources using spot and on-demand instances.
+
+<img src="https://dstack.ai/static-assets/static-assets/images/dstack-ea-slide-3.png" width="630" />
+
+Workflows became standardized, reproducible, and easier to trace—thanks to the use of version-controlled YAML configurations. Teams across different departments and cloud providers now follow the same setup and processes.
+
+> `dstack` provisions compute on demand and automatically shuts it down when no longer needed. That alone saves you over three times in cost.”
+>
+> — Wah Loon Keng, Sr. AI Engineer, Electronic Arts
+
+<!-- EA’s experience highlights how critical standardized, open-source tooling is for scaling AI across teams. Instead of each group reinventing infrastructure and workflows, they’ve moved toward a common stack that supports fast iteration, reproducibility, and cost-efficient use of compute.  -->
+
+By adopting tools that are cloud-agnostic and developer-friendly, EA has reduced friction—from provisioning GPUs to deploying models—and enabled teams to spend more time on actual ML work.
+
+*Huge thanks to Kris and Keng from EA’s central tech team for sharing these insights. For more details, including the recording and slides, check out the full talk on the [NVIDIA GTC website :material-arrow-top-right-thin:{ .external }](https://www.nvidia.com/en-us/on-demand/session/gtc25-s73667/){:target="_blank"}.*
+
+!!! info "What's next?"
+    1. Check [dev environments](../../docs/concepts/dev-environments.md), [tasks](../../docs/concepts/tasks.md), [services](../../docs/concepts/services.md), and [fleets](../../docs/concepts/fleets.md)
+    2. Follow [Quickstart](../../docs/quickstart.md)
+    3. Browse [Examples](../../examples.md)

{dstack-0.19.10 → dstack-0.19.11rc2}/docs/docs/concepts/fleets.md

@@ -328,10 +328,7 @@ This is required if you'd like to use the fleet for [distributed tasks](tasks.md
 
 #### Blocks { #ssh-blocks }
 
-By default, a single job occupies the entire instance, so if the instance has 8 GPUs, the job will use all of them.
-
-To make it more efficient, you can set the blocks property to specify how many blocks you’d like the instance to be
-divided into, allowing multiple jobs to use these blocks concurrently.
+By default, a job uses the entire instance—e.g., all 8 GPUs. To allow multiple jobs on the same instance, set the `blocks` property to divide the instance. Each job can then use one or more blocks, up to the full instance.
 
 <div editor-title=".dstack.yml">
 
@@ -355,14 +352,15 @@ divided into, allowing multiple jobs to use these blocks concurrently.
 
 </div>
 
-For instance, with 8 GPUs, 128 CPUs, and 2TB of memory, setting blocks to 8 would assign 1 GPU, 16 CPUs, and 256 GB of
-memory to each block. These blocks can be used concurrently, and a single job can occupy multiple blocks if needed.
+All resources (GPU, CPU, memory) are split evenly across blocks, while disk is shared.
 
-> GPUs and CPUs must be divisible by the number of blocks. All resources (GPU, CPU, memory) are split proportionally,
-> except disk storage, which is shared.
+For example, with 8 GPUs, 128 CPUs, and 2TB RAM, setting `blocks` to `8` gives each block 1 GPU, 16 CPUs, and 256 GB RAM.
 
-You can also set `blocks` to `auto`, which automatically sets the number of blocks to match the number of GPUs.
+Set `blocks` to `auto` to match the number of blocks to the number of GPUs.
 
+!!! info "Distributed tasks"
+    Distributed tasks require exclusive access to all host resources and therefore must use all blocks on each node.
+    
 #### Environment variables
 
 If needed, you can specify environment variables that will be used by `dstack-shim` and passed to containers.

{dstack-0.19.10 → dstack-0.19.11rc2}/docs/docs/guides/clusters.md

@@ -76,5 +76,5 @@ Refer to [instance volumes](../concepts/volumes.md#instance) for an example.
 
 !!! info "What's next?"
     1. Read about [distributed tasks](../concepts/tasks.md#distributed-tasks), [fleets](../concepts/fleets.md), and [volumes](../concepts/volumes.md)
-    2. Browse the [Clusters](../../examples.md#clusters) examples
+    2. Browse the [Clusters](../../examples.md#clusters) and [Distributed training](../../examples.md#distributed-training) examples
     

{dstack-0.19.10 → dstack-0.19.11rc2}/docs/docs/guides/plugins.md

@@ -113,4 +113,16 @@ class ExamplePolicy(ApplyPolicy):
 
 </div>
 
-For more information on the plugin development, see the [plugin example](https://github.com/dstackai/dstack/tree/master/examples/plugins/example_plugin).
+## Built-in Plugins
+
+### REST Plugin
+
+`rest_plugin` is a builtin `dstack` plugin that allows writing your custom plugins as API servers, so you don't need to install plugins as Python packages.
+
+Plugins implemented as API servers have advantages over plugins implemented as Python packages in some cases:
+
+* No dependency conflicts with `dstack`.
+* You can use any programming language.
+* If you run the `dstack` server via Docker, you don't need to extend the `dstack` server image with plugins or map them via volumes.
+
+To get started, check out the [plugin server example](https://github.com/dstackai/dstack/tree/master/examples/plugins/example_plugin_server). The `rest_plugin` server API is documented [here](../reference/plugins/rest_plugin/index.md).

{dstack-0.19.10 → dstack-0.19.11rc2}/docs/docs/guides/protips.md

@@ -197,6 +197,26 @@ replace it with a corresponding command to start Docker daemon.
 
 See more Docker examples [here](https://github.com/dstackai/dstack/tree/master/examples/misc/docker-compose).
 
+## Projects
+
+If you're using multiple `dstack` projects (e.g., from different `dstack` servers),  
+you can switch between them using the [`dstack project`](../reference/cli/dstack/project.md) command.
+
+Alternatively, you can install [`direnv` :material-arrow-top-right-thin:{ .external }](https://direnv.net/){:target="_blank"}  
+to automatically apply environment variables from the `.envrc` file in your project directory.
+
+<div editor-title=".envrc"> 
+
+```shell
+export DSTACK_PROJECT=main
+```
+
+</div>
+
+Now, `dstack` will always use this project within this directory.
+
+Remember to add `.envrc` to `.gitignore` to avoid committing it to the repo. 
+
 ## Environment variables
 
 If a configuration requires an environment variable that you don't want to hardcode in the YAML, you can define it
@@ -250,9 +270,9 @@ $ dstack apply -e HF_TOKEN=... -f .dstack.yml
     </div>
     
     If you install [`direnv` :material-arrow-top-right-thin:{ .external }](https://direnv.net/){:target="_blank"},
-    it will automatically pass the environment variables from the `.env` file to the `dstack apply` command.
+    it will automatically apply the environment variables from the `.envrc` file to the `dstack apply` command.
 
-    Remember to add `.env` to `.gitignore` to avoid pushing it to the repo.    
+    Remember to add `.envrc` to `.gitignore` to avoid committing it to the repo.    
 
 [//]: # (## Profiles)
 [//]: # ()

{dstack-0.19.10 → dstack-0.19.11rc2}/docs/docs/reference/cli/dstack/project.md

@@ -6,6 +6,33 @@ Before the CLI can be used, it must be configured with a [project](../../../conc
 
 The `dstack project set-default` command can be used to switch between multiple projects.
 
+??? info "Environment variable"
+    Alternatively to `dstack project set-default`, you can set the `DSTACK_PROJECT` environment variable. It overrides the default project set in `~/.dstack/config.yml`.
+
+    <div class="termy">
+    
+    ```shell
+    $ DSTACK_PROJECT=main
+    $ dstack apply -f examples/.dstack.yml
+    ```
+    
+    </div>
+
+    Also, you can install [`direnv` :material-arrow-top-right-thin:{ .external }](https://direnv.net/){:target="_blank"}  
+    to automatically apply environment variables from the `.envrc` file in your project directory.
+
+    <div editor-title=".envrc"> 
+
+    ```shell
+    export DSTACK_PROJECT=main
+    ```
+
+    </div>
+
+    Now, `dstack` will always use this project within this directory.
+
+    Remember to add `.envrc` to `.gitignore` to avoid committing it to the repo. 
+
 ## dstack project add
 
 This command adds a new project configuration.

dstack-0.19.11rc2/docs/docs/reference/plugins/rest_plugin/index.md

@@ -0,0 +1,33 @@
+---
+title: REST PLUGIN API
+---
+
+`rest_plugin` is a builtin `dstack` plugin that allows writing custom plugins as API servers. The following is an OpenAPI documentation for a `rest_plugin` API server.
+
+<style>
+    .swagger-ui .info {
+        margin: 0 !important;
+    }
+
+    .swagger-ui .info h1, .swagger-ui .info h2, .swagger-ui .info h3, .swagger-ui .info h4, .swagger-ui .info h5 {
+        font-weight: 800 !important;
+        letter-spacing: -1px !important;
+        color: rgb(0, 0, 0) !important;
+        text-transform: none !important;
+        font-family: var(--md-text-font-family) !important;
+    }
+
+    .swagger-ui .info .title {
+        padding: 0 !important;
+    }
+
+    .swagger-ui .info li, .swagger-ui .info p, .swagger-ui .info table {
+        line-height: 1.3rem !important;
+        font-size: 0.8rem !important;
+        font-family: var(--md-text-font-family) !important;
+    }
+</style>
+
+<br></br>
+
+!!swagger rest_plugin_openapi.json!!

{dstack-0.19.10 → dstack-0.19.11rc2}/docs/docs/reference/server/config.yml.md

@@ -15,7 +15,7 @@ to configure [backends](../../concepts/backends.md) and other [sever-level setti
     overrides:
         show_root_heading: false
         backends:
-            type: 'Union[AWSBackendConfigWithCreds, AzureBackendConfigWithCreds, GCPBackendConfigWithCreds, LambdaBackendConfigWithCreds, NebiusBackendConfigWithCreds, RunpodBackendConfigWithCreds, TensorDockBackendConfigWithCreds, VastAIBackendConfigWithCreds, KubernetesConfig]'
+            type: 'Union[AWSBackendConfigWithCreds, AzureBackendConfigWithCreds, GCPBackendConfigWithCreds, LambdaBackendConfigWithCreds, NebiusBackendConfigWithCreds, RunpodBackendConfigWithCreds, VastAIBackendConfigWithCreds, KubernetesConfig]'
 
 #### `projects[n].backends` { #backends data-toc-label="backends" }
 
@@ -194,7 +194,7 @@ to configure [backends](../../concepts/backends.md) and other [sever-level setti
         type:
             required: true
 
-##### `projects[n].backends[type=tensordock]` { #tensordock data-toc-label="tensordock" }
+<!-- ##### `projects[n].backends[type=tensordock]` { #tensordock data-toc-label="tensordock" }
 
 #SCHEMA# dstack._internal.core.backends.tensordock.models.TensorDockBackendConfigWithCreds
     overrides:
@@ -209,7 +209,7 @@ to configure [backends](../../concepts/backends.md) and other [sever-level setti
     overrides:
         show_root_heading: false
         type:
-            required: true
+            required: true -->
 
 ##### `projects[n].backends[type=oci]` { #oci data-toc-label="oci" }
 

{dstack-0.19.10 → dstack-0.19.11rc2}/docs/examples.md

@@ -83,6 +83,22 @@ hide:
     </a>
 </div>
 
+## Distributed training
+
+<div class="tx-landing__highlights_grid">
+    <a href="/examples/distributed-training/ray-ragen"
+       class="feature-cell sky">
+        <h3>
+            Ray+RAGEN
+        </h3>
+
+        <p>
+            Fine-tune an agent on multiple nodes
+            with RAGEN, verl, and Ray.
+        </p>
+    </a>
+</div>
+
 ## Inference
 
 <div class="tx-landing__highlights_grid">
@@ -128,7 +144,7 @@ hide:
            TensorRT-LLM
        </h3>
        <p>
-            Deploy DeepSeek R1 and its distilled version with TensorRT-LLM
+            Deploy DeepSeek models with TensorRT-LLM
         </p>
     </a>
 </div>

{dstack-0.19.10 → dstack-0.19.11rc2}/docs/overrides/main.html

@@ -119,6 +119,7 @@ Support dstack with a ⭐ on
                     <div class="tx-footer__section-title">Examples</div>
                     <a href="/examples#fine-tuning" class="tx-footer__section-link">Fine-tuning</a>
                     <a href="/examples#clusters" class="tx-footer__section-link">Clusters</a>
+                    <a href="/examples#distributed-training" class="tx-footer__section-link">Distributed training</a>
                     <a href="/examples#inference" class="tx-footer__section-link">Inference</a>
                     <a href="/examples#accelerators" class="tx-footer__section-link">Accelerators</a>
                     <a href="/examples#llms" class="tx-footer__section-link">LLMs</a>

{dstack-0.19.10 → dstack-0.19.11rc2}/examples/.dstack.yml

@@ -2,14 +2,15 @@ type: dev-environment
 # The name is optional, if not specified, generated randomly
 name: vscode
 
-python: "3.11"
-# Uncomment to use a custom Docker image
-#image: dstackai/base:py3.13-0.7-cuda-12.1
+#python: "3.11"
+
+image: un1def/dstack-base:py3.12-dev-cuda-12.1
 
 ide: vscode
 
 # Use either spot or on-demand instances
-spot_policy: auto
+#spot_policy: auto
 
 resources:
-  gpu: 1
+  cpu: x86:8..32
+  gpu: 24GB..:1

dstack-0.19.11rc2/examples/distributed-training/ray-ragen/.dstack.yml

@@ -0,0 +1,39 @@
+type: task
+name: ray-ragen-cluster
+
+nodes: 2
+
+env:
+- WANDB_API_KEY
+image: whatcanyousee/verl:ngc-cu124-vllm0.8.5-sglang0.4.6-mcore0.12.0-te2.2
+commands:
+  - wget -O miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+  - bash miniconda.sh -b -p /workflow/miniconda
+  - eval "$(/workflow/miniconda/bin/conda shell.bash hook)"
+  - git clone https://github.com/RAGEN-AI/RAGEN.git
+  - cd RAGEN
+  - bash scripts/setup_ragen.sh
+  - conda activate ragen
+  - cd verl
+  - pip install --no-deps -e .
+  - pip install hf_transfer hf_xet
+  - pip uninstall -y ray
+  - pip install -U "ray[default]"
+  - |
+    if [ $DSTACK_NODE_RANK = 0 ]; then 
+        ray start --head --port=6379;
+    else
+        ray start --address=$DSTACK_MASTER_NODE_IP:6379
+    fi
+
+# Expose Ray dashboard port
+ports:
+  - 8265
+
+resources:
+  gpu: 80GB:8
+  shm_size: 128GB
+
+# Save checkpoints on the instance
+volumes:
+  - /checkpoints:/checkpoints

dstack-0.19.11rc2/examples/distributed-training/ray-ragen/README.md

@@ -0,0 +1,133 @@
+# Ray + RAGEN
+
+This example shows how use `dstack` and [RAGEN :material-arrow-top-right-thin:{ .external }](https://github.com/RAGEN-AI/RAGEN){:target="_blank"} 
+to fine-tune an agent on mulitiple nodes.
+
+Under the hood `RAGEN` uses [verl :material-arrow-top-right-thin:{ .external }](https://github.com/volcengine/verl){:target="_blank"} for Reinforcement Learning and [Ray :material-arrow-top-right-thin:{ .external }](https://docs.ray.io/en/latest/){:target="_blank"} for ditributed training.
+
+## Create fleet
+
+Before submitted disributed training runs, make sure to create a fleet with a `placement` set to `cluster`.
+
+> For more detials on how to use clusters with `dstack`, check the [Clusters](https://dstack.ai/docs/guides/clusters) guide.
+
+## Run a Ray cluster
+
+If you want to use Ray with `dstack`, you have to first run a Ray cluster.
+
+The task below runs a Ray cluster on an existing fleet:
+
+<div editor-title="examples/distributed-training/ray-ragen/.dstack.yml">
+
+```yaml
+type: task
+name: ray-ragen-cluster
+
+nodes: 2
+
+env:
+- WANDB_API_KEY
+image: whatcanyousee/verl:ngc-cu124-vllm0.8.5-sglang0.4.6-mcore0.12.0-te2.2
+commands:
+  - wget -O miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+  - bash miniconda.sh -b -p /workflow/miniconda
+  - eval "$(/workflow/miniconda/bin/conda shell.bash hook)"
+  - git clone https://github.com/RAGEN-AI/RAGEN.git
+  - cd RAGEN
+  - bash scripts/setup_ragen.sh
+  - conda activate ragen
+  - cd verl
+  - pip install --no-deps -e .
+  - pip install hf_transfer hf_xet
+  - pip uninstall -y ray
+  - pip install -U "ray[default]"
+  - |
+    if [ $DSTACK_NODE_RANK = 0 ]; then 
+        ray start --head --port=6379;
+    else
+        ray start --address=$DSTACK_MASTER_NODE_IP:6379
+    fi
+
+# Expose Ray dashboard port
+ports:
+  - 8265
+
+resources:
+  gpu: 80GB:8
+  shm_size: 128GB
+
+# Save checkpoints on the instance
+volumes:
+  - /checkpoints:/checkpoints
+```
+
+</div>
+
+We are using verl's docker image for vLLM with FSDP. See [Installation :material-arrow-top-right-thin:{ .external }](https://verl.readthedocs.io/en/latest/start/install.html){:target="_blank"} for more.
+
+The `RAGEN` setup script `scripts/setup_ragen.sh` isolates dependencies within Conda environment.
+
+Note that the Ray setup in the RAGEN environment is missing the dashboard, so we reinstall it using `ray[default]`.
+
+Now, if you run this task via `dstack apply`, it will automatically forward the Ray's dashboard port to `localhost:8265`.
+
+<div class="termy">
+
+```shell
+$ dstack apply -f examples/distributed-training/ray-ragen/.dstack.yml
+```
+
+</div>
+
+As long as the `dstack apply` is attached, you can use `localhost:8265` to submit Ray jobs for execution.
+If `dstack apply` is detached, you can use `dstack attach` to re-attach.
+
+## Submit Ray jobs
+
+Before you can submit Ray jobs, ensure to install `ray` locally:
+
+<div class="termy">
+
+```shell
+$ pip install ray
+```
+
+</div>
+
+Now you can submit the training job to the Ray cluster which is available at `localhost:8265`:
+
+<div class="termy">
+
+```shell
+$ RAY_ADDRESS=http://localhost:8265
+$ ray job submit \
+  -- bash -c "\
+    export PYTHONPATH=/workflow/RAGEN; \
+    cd /workflow/RAGEN; \
+    /workflow/miniconda/envs/ragen/bin/python train.py \
+      --config-name base \
+      system.CUDA_VISIBLE_DEVICES=[0,1,2,3,4,5,6,7] \
+      model_path=Qwen/Qwen2.5-7B-Instruct \
+      trainer.experiment_name=agent-fine-tuning-Qwen2.5-7B \
+      trainer.n_gpus_per_node=8 \
+      trainer.nnodes=2 \
+      micro_batch_size_per_gpu=2 \
+      trainer.default_local_dir=/checkpoints \
+      trainer.save_freq=50 \
+      actor_rollout_ref.rollout.tp_size_check=False \
+      actor_rollout_ref.rollout.tensor_model_parallel_size=4"
+```
+
+</div>
+
+!!! info "Training parameters"
+    1. `actor_rollout_ref.rollout.tensor_model_parallel_size=4`, because `Qwen/Qwen2.5-7B-Instruct` has 28 attention heads and number of attention heads should be divisible by `tensor_model_parallel_size`
+    2. `actor_rollout_ref.rollout.tp_size_check=False`, if True `tensor_model_parallel_size` should be equal to `trainer.n_gpus_per_node`
+    3. `micro_batch_size_per_gpu=2`, to keep the RAGEN-paper's `rollout_filter_ratio` and `es_manager` settings as it is for world size `16`
+
+Using Ray via `dstack` is a powerful way to get access to the rich Ray ecosystem while benefiting from `dstack`'s provisioning capabilities.
+
+!!! info "What's next"
+    1. Check the [Clusters](https://dstack.ai/docs/guides/clusters) guide
+    2. Read about [distributed tasks](https://dstack.ai/docs/concepts/tasks#distributed-tasks) and [fleets](https://dstack.ai/docs/concepts/fleets)
+    3. Browse Ray's [docs :material-arrow-top-right-thin:{ .external }](https://docs.ray.io/en/latest/train/examples.html){:target="_blank"} for other examples.

{dstack-0.19.10 → dstack-0.19.11rc2}/examples/misc/ray/README.md

@@ -33,7 +33,7 @@ name: ray-cluster
 nodes: 4
 commands:
   - pip install -U "ray[default]"
-  - >
+  - |
     if [ $DSTACK_NODE_RANK = 0 ]; then 
       ray start --head --port=6379;
     else

dstack-0.19.11rc2/examples/plugins/example_plugin_server/.python-version

@@ -0,0 +1 @@
+3.11

dstack-0.19.11rc2/examples/plugins/example_plugin_server/README.md

@@ -0,0 +1,29 @@
+## Overview
+
+If you wish to hook up your own plugin server through `dstack` builtin `rest_plugin`, here's a basic example on how to do so.
+
+## Steps
+
+1. Install the plugin server:
+
+    ```bash
+    uv pip install examples/plugins/example_plugin_server
+    ```
+
+2. Start the plugin server:
+
+    ```bash
+    python -m example_plugin_server.main
+    ```
+
+3. Enable `rest_plugin` in `server/config.yaml`:
+
+    ```yaml
+   plugins:
+    - rest_plugin
+    ```
+
+4. Point the `dstack` server to your plugin server:
+    ```bash
+    export DSTACK_PLUGIN_SERVICE_URI=http://127.0.0.1:8000
+    ```

dstack-0.19.11rc2/examples/plugins/example_plugin_server/pyproject.toml

@@ -0,0 +1,17 @@
+[project]
+name = "dstack-plugin-server"
+version = "0.1.0"
+description = "Example plugin server"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "fastapi[standard]>=0.115.12",
+    "dstack",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/example_plugin_server"]