dstack 0.19.23rc1__tar.gz → 0.19.24__tar.gz

{dstack-0.19.23rc1 → dstack-0.19.24}/.github/workflows/build.yml

@@ -126,7 +126,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [ubuntu-latest]
+        os: [ubuntu-latest, macos-latest]
     steps:
       - uses: actions/checkout@v4
       - name: Set up Go

{dstack-0.19.23rc1 → dstack-0.19.24}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dstack
-Version: 0.19.23rc1
+Version: 0.19.24
 Summary: dstack is an open-source orchestration engine for running AI workloads on any cloud or on-premises.
 Project-URL: Homepage, https://dstack.ai
 Project-URL: Source, https://github.com/dstackai/dstack

dstack-0.19.24/docs/blog/posts/gpu-health-checks.md

@@ -0,0 +1,73 @@
+---
+title: Introducing passive GPU health checks
+date: 2025-08-12
+description: "TBA"  
+slug: gpu-helth-checks
+image: https://dstack.ai/static-assets/static-assets/images/gpu-health-checks.png
+categories:
+  - Changelog
+---
+
+# Introducing passive GPU health checks
+
+In large-scale training, a single bad GPU can derail progress. Sometimes the failure is obvious — jobs crash outright. Other times it’s subtle: correctable memory errors, intermittent instability, or thermal throttling that quietly drags down throughput. In big experiments, these issues can go unnoticed for hours or days, wasting compute and delaying results.
+
+`dstack` already supports GPU telemetry monitoring through NVIDIA DCGM [metrics](../../docs/guides/metrics.md), covering utilization, memory, and temperature. This release extends that capability with passive hardware health checks powered by DCGM [background health checks](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#background-health-checks). With these, `dstack` continuously evaluates fleet GPUs for hardware reliability and displays their status before scheduling workloads.
+
+<img src="https://dstack.ai/static-assets/static-assets/images/gpu-health-checks.png" width="630"/>
+
+<!-- more -->
+
+## Why this matters
+
+Multi-GPU and multi-node workloads are only as strong as their weakest component. GPU cloud providers increasingly rely on automated health checks to prevent degraded hardware from reaching customers. Problems can stem from ECC memory errors, faulty PCIe links, overheating, or other hardware-level issues. Some are fatal, others allow the GPU to run but at reduced performance or with higher failure risk.
+
+Passive checks like these run in the background. They continuously monitor hardware telemetry and system events, evaluating them against NVIDIA’s known failure patterns — all without pausing workloads.
+
+## How it works in dstack
+
+`dstack` automatically queries DCGM for each fleet instance and appends a health status:
+
+* An `idle` status means no issues have been detected.
+* An `idle (warning)` status indicates a non-fatal issue, such as a correctable ECC error. The instance remains usable but should be monitored.
+* An `idle (failure)` status points to a fatal issue, and the instance is automatically excluded from scheduling.
+
+<div class="termy">
+
+```shell
+$ dstack fleet
+
+ FLEET     INSTANCE  BACKEND          RESOURCES  STATUS          PRICE   CREATED
+ my-fleet  0         aws (us-east-1)  T4:16GB:1  idle            $0.526  11 mins ago
+           1         aws (us-east-1)  T4:16GB:1  idle (warning)  $0.526  11 mins ago
+           2         aws (us-east-1)  T4:16GB:1  idle (failure)  $0.526  11 mins ago
+```
+
+</div>
+
+A healthy instance is ready for workloads. A warning means you should monitor it closely. A failure removes it from scheduling entirely.
+
+## Passive vs active checks
+
+This release focuses on passive checks using DCGM background health checks. These run continuously and do not interrupt workloads.
+
+For active checks today, you can run [NCCL tests](../../examples/clusters/nccl-tests/index.md) as a [distributed task](../../docs/concepts/tasks.md#distributed-tasks) to verify GPU-to-GPU communication and bandwidth across a fleet. Active tests like these can reveal network or interconnect issues that passive monitoring might miss. More built-in support for active diagnostics is planned.
+
+## Supported backends
+
+Passive GPU health checks work on AWS (except with custom `os_images`), Azure (except A10 GPUs), GCP, OCI, and [SSH fleets](../../docs/concepts/fleets.md#ssh) where DCGM is installed and configured for background checks. 
+
+> Fleets created before version 0.19.22 need to be recreated to enable this feature.
+
+## Looking ahead
+
+This update is about visibility: giving engineers real-time insight into GPU health before jobs run. Next comes automation — policies to skip GPUs with warnings, and self-healing workflows that replace unhealthy instances without manual steps.
+
+If you have experience with GPU reliability or ideas for automated recovery, join the conversation on
+[Discord :material-arrow-top-right-thin:{ .external }](https://discord.gg/u8SmfwPpMd){:target="_blank"}.
+
+!!! info "What's next?"
+    1. Check [Quickstart](../../docs/quickstart.md)
+    2. Explore the [clusters](../../docs/guides/clusters.md) guide
+    3. Learn more about [metrics](../../docs/guides/metrics.md)
+    4. Join [Discord :material-arrow-top-right-thin:{ .external }](https://discord.gg/u8SmfwPpMd){:target="_blank"}

dstack-0.19.24/docs/blog/posts/hotaisle.md

@@ -0,0 +1,114 @@
+---
+title: Supporting Hot Aisle AMD AI Developer Cloud
+date: 2025-08-11
+description: "TBA"  
+slug: hotaisle
+image: https://dstack.ai/static-assets/static-assets/images/dstack-hotaisle.png
+categories:
+  - Changelog
+---
+
+# Supporting Hot Aisle AMD AI Developer Cloud
+
+As the ecosystem around AMD GPUs matures, developers are looking for easier ways to experiment with ROCm, benchmark new architectures, and run cost-effective workloads—without manual infrastructure setup.  
+
+`dstack` is an open-source orchestrator designed for AI workloads, providing a lightweight, container-native alternative to Kubernetes and Slurm.
+
+<img src="https://dstack.ai/static-assets/static-assets/images/dstack-hotaisle.png" width="630"/>
+
+Today, we’re excited to announce native integration with [Hot Aisle :material-arrow-top-right-thin:{ .external }](https://www.hotaisle.io/){:target="_blank"}, an AMD-only GPU neocloud offering VMs and clusters at highly competitive on-demand pricing.  
+
+<!-- more -->
+
+## About Hot Aisle
+
+Hot Aisle is a next-generation GPU cloud built around AMD’s flagship AI accelerators.  
+
+Highlights:
+
+- AMD’s flagship AI-optimized accelrators
+- On-demand pricing: $1.99/hour for 1-GPU VMs
+- No commitment – start and stop when you want
+- First AMD-only GPU backend in `dstack`
+
+While it has already been possible to use HotAisle’s 8-GPU MI300X bare-metal clusters via [`SSH fleets`](../../docs/concepts/fleets.md#ssh-fleets), this integration now enables automated provisioning of VMs—made possible by HotAisle’s newly added API for MI300X instances.
+
+## Why dstack
+
+`dstack` is a new open-source container orchestrator built specifically for GPU workloads.  
+It fills the gaps left by Kubernetes and Slurm when it comes to GPU provisioning and orchestration:
+
+- Unlike Kubernetes, `dstack` offers a high-level, AI-engineer-friendly interface, and GPUs work out of the box, with no need to wrangle custom operators, device plugins, or other low-level setup.
+- Unlike Slurm, it’s use-case agnostic — equally suited for training, inference, benchmarking, or even setting up long-running dev environments.
+- It works across clouds and on-prem without vendor lock-in.
+
+With the new Hot Aisle backend, you can automatically provision MI300X VMs for any workload — from experiments to production — with a single `dstack` CLI command.
+
+## Getting started
+
+Before configuring `dstack` to use Hot Aisle’s VMs, complete these steps:
+
+1. Create a project via `ssh admin.hotaisle.app`
+2. Get credits or approve a payment method
+3. Create an API key
+
+Then, configure the backend in `~/.dstack/server/config.yml`:
+
+<div editor-title="~/.dstack/server/config.yml">
+
+```yaml
+projects:
+- name: main
+  backends:
+    - type: hotaisle
+      team_handle: hotaisle-team-handle
+      creds:
+        type: api_key
+        api_key: 9c27a4bb7a8e472fae12ab34.3f2e3c1db75b9a0187fd2196c6b3e56d2b912e1c439ba08d89e7b6fcd4ef1d3f
+```
+
+</div>
+
+Install and start the `dstack` server:
+
+<div class="termy">
+
+```shell
+$ pip install "dstack[server]"
+$ dstack server
+```
+
+</div>
+
+For more details, see [Installation](../../docs/installation/index.md).
+
+Use the `dstack` CLI to
+manage [dev environments](../../docs/concepts/dev-environments.md), [tasks](../../docs/concepts/tasks.md),
+and [services](../../docs/concepts/services.md).
+
+<div class="termy">
+
+```shell
+$ dstack apply -f .dstack.yml
+
+ #  BACKEND                   RESOURCES                                     INSTANCE TYPE                     PRICE   
+ 1  hotaisle (us-michigan-1)  cpu=13 mem=224GB disk=12288GB MI300X:192GB:1  1x MI300X 13x Xeon Platinum 8470  $1.99
+ 2  hotaisle (us-michigan-1)  cpu=8 mem=224GB disk=12288GB MI300X:192GB:1   1x MI300X 8x Xeon Platinum 8470   $1.99
+ 
+ Submit the run? [y/n]:
+```
+
+</div>
+
+Currently, `dstack` supports 1xGPU Hot Aisle VMs. Support for 8xGPU VMs will be added once Hot Aisle supports it.
+
+> If you prefer to use Hot Aisle’s bare-metal 8-GPU clusters with dstack, you can create an [SSH fleet](../../docs/concepts/fleets.md#ssh-fleets).
+> This way, you’ll be able to run [distributed tasks](../../docs/concepts/tasks.md#distributed-tasks) efficiently across the cluster.
+
+!!! info "What's next?"
+    1. Check [Quickstart](../../docs/quickstart.md)
+    2. Learn more about [Hot Aisle :material-arrow-top-right-thin:{ .external }](https://hotaisle.xyz/){:target="_blank"}
+    3. Explore [dev environments](../../docs/concepts/dev-environments.md), 
+        [tasks](../../docs/concepts/tasks.md), [services](../../docs/concepts/services.md), 
+        and [fleets](../../docs/concepts/fleets.md)
+    4. Join [Discord :material-arrow-top-right-thin:{ .external }](https://discord.gg/u8SmfwPpMd){:target="_blank"}

{dstack-0.19.23rc1 → dstack-0.19.24}/frontend/src/pages/Project/Secrets/index.tsx

@@ -3,7 +3,7 @@ import { useTranslation } from 'react-i18next';
 
 import { Button, ButtonWithConfirmation, Header, ListEmptyMessage, Modal, Pagination, SpaceBetween, Table } from 'components';
 
-import { useCollection, useNotifications, usePermissionGuard } from 'hooks';
+import { useAppSelector, useCollection, useNotifications, usePermissionGuard } from 'hooks';
 import { getServerError } from 'libs';
 import {
     useDeleteSecretsMutation,
@@ -13,6 +13,9 @@ import {
 } from 'services/secrets';
 import { GlobalUserRole, ProjectUserRole } from 'types';
 
+import { selectUserData } from 'App/slice';
+
+import { getProjectRoleByUserName } from '../utils';
 import { SecretForm } from './Form';
 
 import { IProps, TFormValues } from './types';
@@ -21,6 +24,8 @@ import styles from './styles.module.scss';
 
 export const ProjectSecrets: React.FC<IProps> = ({ project, loading }) => {
     const { t } = useTranslation();
+    const userData = useAppSelector(selectUserData);
+    const userName = userData?.username ?? '';
     const [initialFormValues, setInitialFormValues] = useState<TFormValues | undefined>();
     const projectName = project?.project_name ?? '';
     const [pushNotification] = useNotifications();
@@ -28,6 +33,7 @@ export const ProjectSecrets: React.FC<IProps> = ({ project, loading }) => {
     const [hasPermissionForSecretsManaging] = usePermissionGuard({
         allowedProjectRoles: [ProjectUserRole.ADMIN],
         allowedGlobalRoles: [GlobalUserRole.ADMIN],
+        projectRole: project ? (getProjectRoleByUserName(project, userName) ?? undefined) : undefined,
     });
 
     const { data, isLoading, isFetching } = useGetAllSecretsQuery(

{dstack-0.19.23rc1 → dstack-0.19.24}/runner/cmd/shim/main.go

@@ -204,7 +204,7 @@ func start(ctx context.Context, args shim.CLIArgs, serviceMode bool) (err error)
 	}
 
 	var dcgmExporter *dcgm.DCGMExporter
-	var dcgmWrapper *dcgm.DCGMWrapper
+	var dcgmWrapper dcgm.DCGMWrapperInterface
 
 	if common.GetGpuVendor() == common.GpuVendorNvidia {
 		dcgmExporterPath, err := dcgm.GetDCGMExporterExecPath(ctx)

{dstack-0.19.23rc1 → dstack-0.19.24}/runner/internal/shim/api/handlers.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"errors"
 	"net/http"
+	"reflect"
 
 	"github.com/dstackai/dstack/runner/internal/api"
 	"github.com/dstackai/dstack/runner/internal/log"
@@ -24,7 +25,7 @@ func (s *ShimServer) HealthcheckHandler(w http.ResponseWriter, r *http.Request)
 func (s *ShimServer) InstanceHealthHandler(w http.ResponseWriter, r *http.Request) (interface{}, error) {
 	ctx := r.Context()
 	response := InstanceHealthResponse{}
-	if s.dcgmWrapper != nil {
+	if !reflect.ValueOf(s.dcgmWrapper).IsNil() {
 		if dcgmHealth, err := s.dcgmWrapper.GetHealth(); err != nil {
 			log.Error(ctx, "failed to get health from DCGM", "err", err)
 		} else {

{dstack-0.19.23rc1 → dstack-0.19.24}/runner/internal/shim/api/server.go

@@ -29,14 +29,14 @@ type ShimServer struct {
 	runner TaskRunner
 
 	dcgmExporter *dcgm.DCGMExporter
-	dcgmWrapper  *dcgm.DCGMWrapper
+	dcgmWrapper  dcgm.DCGMWrapperInterface
 
 	version string
 }
 
 func NewShimServer(
 	ctx context.Context, address string, version string,
-	runner TaskRunner, dcgmExporter *dcgm.DCGMExporter, dcgmWrapper *dcgm.DCGMWrapper,
+	runner TaskRunner, dcgmExporter *dcgm.DCGMExporter, dcgmWrapper dcgm.DCGMWrapperInterface,
 ) *ShimServer {
 	r := api.NewRouter()
 	s := &ShimServer{

dstack-0.19.24/runner/internal/shim/dcgm/wrapper.go

@@ -0,0 +1,29 @@
+package dcgm
+
+type HealthStatus string
+
+const (
+	HealthStatusHealthy HealthStatus = "healthy"
+	HealthStatusWarning HealthStatus = "warning"
+	HealthStatusFailure HealthStatus = "failure"
+)
+
+type HealthIncident struct {
+	System        int    `json:"system"`
+	Health        int    `json:"health"`
+	ErrorMessage  string `json:"error_message"`
+	ErrorCode     int    `json:"error_code"`
+	EntityGroupID int    `json:"entity_group_id"`
+	EntityID      int    `json:"entity_id"`
+}
+
+type Health struct {
+	OverallHealth int              `json:"overall_health"`
+	Incidents     []HealthIncident `json:"incidents"`
+}
+
+type DCGMWrapperInterface interface {
+	Shutdown() error
+	EnableHealthChecks() error
+	GetHealth() (Health, error)
+}

dstack-0.19.24/runner/internal/shim/dcgm/wrapper_darwin.go

@@ -0,0 +1,9 @@
+//go:build darwin
+
+package dcgm
+
+import "errors"
+
+func NewDCGMWrapper(address string) (DCGMWrapperInterface, error) {
+	return nil, errors.New("macOS is not supported")
+}

dstack-0.19.23rc1/runner/internal/shim/dcgm/wrapper.go → dstack-0.19.24/runner/internal/shim/dcgm/wrapper_linux.go

@@ -1,3 +1,5 @@
+//go:build linux
+
 package dcgm
 
 import (
@@ -8,28 +10,6 @@ import (
 	godcgm "github.com/NVIDIA/go-dcgm/pkg/dcgm"
 )
 
-type HealthStatus string
-
-const (
-	HealthStatusHealthy HealthStatus = "healthy"
-	HealthStatusWarning HealthStatus = "warning"
-	HealthStatusFailure HealthStatus = "failure"
-)
-
-type HealthIncident struct {
-	System        int    `json:"system"`
-	Health        int    `json:"health"`
-	ErrorMessage  string `json:"error_message"`
-	ErrorCode     int    `json:"error_code"`
-	EntityGroupID int    `json:"entity_group_id"`
-	EntityID      int    `json:"entity_id"`
-}
-
-type Health struct {
-	OverallHealth int              `json:"overall_health"`
-	Incidents     []HealthIncident `json:"incidents"`
-}
-
 // DCGMWrapper is a wrapper around go-dcgm (which, in turn, is a wrapper around libdcgm.so)
 type DCGMWrapper struct {
 	group              godcgm.GroupHandle

dstack-0.19.23rc1/runner/internal/shim/dcgm/wrapper_test.go → dstack-0.19.24/runner/internal/shim/dcgm/wrapper_linux_test.go

@@ -1,3 +1,5 @@
+//go:build linux
+
 package dcgm
 
 import (
@@ -73,7 +75,7 @@ func getGpuID(t *testing.T) uint {
 	return gpuIDs[0]
 }
 
-func injectError(t *testing.T, gpuID uint, fieldID godcgm.Short, fieldType uint, value interface{}) {
+func injectError(t *testing.T, gpuID uint, fieldID godcgm.Short, fieldType uint, value any) {
 	t.Helper()
 	err := godcgm.InjectFieldValue(gpuID, fieldID, fieldType, 0, time.Now().UnixMicro(), value)
 	require.NoError(t, err)

{dstack-0.19.23rc1 → dstack-0.19.24}/src/dstack/_internal/core/models/instances.py

@@ -165,6 +165,9 @@ class InstanceAvailability(Enum):
     AVAILABLE = "available"
     NOT_AVAILABLE = "not_available"
     NO_QUOTA = "no_quota"
+    NO_BALANCE = (
+        "no_balance"  # Introduced in 0.19.24, may be used after a short compatibility period
+    )
     IDLE = "idle"
     BUSY = "busy"
 

{dstack-0.19.23rc1 → dstack-0.19.24}/src/dstack/_internal/core/models/runs.py

@@ -350,15 +350,17 @@ class JobSubmission(CoreModel):
     deployment_num: int = 0  # default for compatibility with pre-0.19.14 servers
     submitted_at: datetime
     last_processed_at: datetime
-    finished_at: Optional[datetime]
-    inactivity_secs: Optional[int]
+    finished_at: Optional[datetime] = None
+    inactivity_secs: Optional[int] = None
     status: JobStatus
     status_message: str = ""  # default for backward compatibility
-    termination_reason: Optional[JobTerminationReason]
-    termination_reason_message: Optional[str]
-    exit_status: Optional[int]
-    job_provisioning_data: Optional[JobProvisioningData]
-    job_runtime_data: Optional[JobRuntimeData]
+    # termination_reason stores JobTerminationReason.
+    # str allows adding new enum members without breaking compatibility with old clients.
+    termination_reason: Optional[str] = None
+    termination_reason_message: Optional[str] = None
+    exit_status: Optional[int] = None
+    job_provisioning_data: Optional[JobProvisioningData] = None
+    job_runtime_data: Optional[JobRuntimeData] = None
     error: Optional[str] = None
     probes: list[Probe] = []
 
@@ -508,7 +510,9 @@ class Run(CoreModel):
     last_processed_at: datetime
     status: RunStatus
     status_message: str = ""  # default for backward compatibility
-    termination_reason: Optional[RunTerminationReason] = None
+    # termination_reason stores RunTerminationReason.
+    # str allows adding new enum members without breaking compatibility with old clients.
+    termination_reason: Optional[str] = None
     run_spec: RunSpec
     jobs: List[Job]
     latest_job_submission: Optional[JobSubmission] = None

{dstack-0.19.23rc1 → dstack-0.19.24}/src/dstack/_internal/server/background/tasks/process_running_jobs.py

@@ -455,8 +455,7 @@ async def _wait_for_instance_provisioning_data(job_model: JobModel):
 
     if job_model.instance.status == InstanceStatus.TERMINATED:
         job_model.status = JobStatus.TERMINATING
-        # TODO use WAITING_INSTANCE_LIMIT_EXCEEDED after 0.19.x
-        job_model.termination_reason = JobTerminationReason.FAILED_TO_START_DUE_TO_NO_CAPACITY
+        job_model.termination_reason = JobTerminationReason.WAITING_INSTANCE_LIMIT_EXCEEDED
         return
 
     job_model.job_provisioning_data = job_model.instance.job_provisioning_data

{dstack-0.19.23rc1 → dstack-0.19.24}/src/dstack/_internal/server/background/tasks/process_runs.py

@@ -574,7 +574,7 @@ def _should_retry_job(run: Run, job: Job, job_model: JobModel) -> Optional[datet
 
     if (
         last_provisioned_submission.termination_reason is not None
-        and last_provisioned_submission.termination_reason.to_retry_event()
+        and JobTerminationReason(last_provisioned_submission.termination_reason).to_retry_event()
         in job.job_spec.retry.on_events
     ):
         return common.get_current_datetime() - last_provisioned_submission.last_processed_at