gpustack-runtime 0.2.5.post4__tar.gz → 0.2.7__tar.gz

gpustack_runtime-0.2.7/.agents/skills/runtime-release/SKILL.md

@@ -0,0 +1,100 @@
+---
+name: runtime-release
+description: Cut a gpustack-runtime package release end to end (select and push the tag, monitor the release workflow, curate the GitHub Release notes).
+---
+
+# Runtime release
+
+One maintainer duty: cut a package release. Push a version tag; `release.yml` builds and publishes
+it. Then write the GitHub Release notes.
+
+## Cut a package release
+
+The git tag is the single source of truth: hatch-vcs derives the package version from it at build
+time, and there is no version file to bump. Pushing a `v*.*.*` tag runs
+`.github/workflows/release.yml`: build the distributions, create the GitHub Release with the
+wheel/sdist assets, publish to PyPI, then verify an isolated install. Branch pushes to `main` or
+`v*-dev` run the same job against TestPyPI and create no release.
+
+Two workflow facts shape the procedure:
+
+- The GitHub Release is a pre-release only when the tag contains `rc`. A final tag goes public and
+  becomes `latest` immediately. There is no promotion gate.
+- The workflow generates no baseline notes. The release body starts empty, so curating the notes is
+  not polish; it is the entire text the reader gets.
+
+PyPI is the irreversible step: a version once published there cannot be re-uploaded or edited.
+
+### Hard rules
+
+- Tag shapes: `vX.Y.Z` final, `vX.Y.ZrcN` pre-release, `vX.Y.Z.postN` post-release. PEP 440
+  canonical spellings only; history contains the non-dotted `v0.2.5postN` — do not repeat it.
+- A tag whose PyPI publish succeeded is burned. NEVER delete or move it; fix forward with a new
+  version. Delete-and-recreate is limited to a tag that never reached PyPI and is interactive-only.
+- An `rc` tag stays a pre-release; never mark it latest.
+- Release from `origin/main` HEAD by default. A maintenance line (`v*-dev` branch) targets that
+  branch's HEAD; flag the choice in the summary.
+- Interactive mode confirms every mutating step (tag push, note replacement). Auto mode runs
+  unattended through note curation and reports the release URL.
+
+### 1. Preflight (read-only)
+
+```sh
+gh auth status
+git fetch origin --tags --prune
+git status --porcelain                            # must be empty
+git tag -l 'v*' --sort=-v:refname | head -5
+```
+
+### 2. Version and commit (confirm)
+
+- Pick the target ref: `origin/main` by default, or the `v*-dev` branch for a maintenance line. The
+  steps below call it `$TARGET`; `$SHA` is its HEAD (`git rev-parse "$TARGET"`).
+- Read the range since the previous tag: `git log --no-merges --oneline <last>..$TARGET`.
+- Propose the bump from the Conventional Commits in range: any `feat` → minor; fixes, docs, and
+  chores → patch. Use `.postN` only to re-issue the same version line (packaging metadata or a
+  docs-only rebuild of unchanged content).
+- Confirm the (version, commit) pair before tagging. A final tag publishes immediately, so this
+  confirmation is the only gate.
+
+### 3. Draft notes
+
+Group the range by type: Features / Fixes / Docs / Other; omit empty sections. Keep highlights and
+fold noisy `chore`/`ci`/`test` entries. Link PRs from `(#NNN)` subjects. Do not open the body with
+a heading that repeats the tag. End with the Full Changelog compare URL
+(`https://github.com/gpustack/runtime/compare/<last>...<VER>`). Write the draft to `notes.md`;
+step 5 publishes that file.
+
+### 4. Tag and monitor (confirm, then read-only)
+
+```sh
+git tag -a "$VER" "$SHA" -m "Release $VER"
+git push origin "$VER"
+gh run list --branch "$VER" --limit 5     # the release.yml run for the tag
+gh run watch <run-id> --exit-status       # background it; wait for the completion notification
+```
+
+The job's own verify step installs the package from PyPI in an isolated env, but it is unpinned and
+only prints the installed version. A green run proves the upload succeeded, not that PyPI already
+serves the new version; read the verify log's `version =` line and confirm it equals `$VER`. On
+failure read `gh run view <run-id> --log-failed`; retry the same run on transient errors. If the
+failure is real and PyPI already holds the version, cut the next version instead — never re-tag.
+
+### 5. Curate notes (confirm)
+
+```sh
+gh release view "$VER" --json body,isPrerelease     # the body starts empty
+gh release edit "$VER" --notes-file notes.md
+```
+
+A maintenance-line final keeps `--latest=false` when a newer stable tag exists. Verify the badge
+with `gh api repos/gpustack/runtime/releases/latest --jq .tag_name`. An `rc` release stays a
+pre-release and needs no further action.
+
+## Scope
+
+Package releases and their notes run here. No other repository document changes with a release:
+the version comes from the tag, and no README or docs table tracks it. Container image builds are
+`make package` (`pack/`) and deployment manifests live in `deploy/`; both are outside this skill.
+Docs-only changes never trigger a release: the release workflow ignores docs paths on branch
+pushes, so docs updates need no tag.

{gpustack_runtime-0.2.5.post4 → gpustack_runtime-0.2.7}/.gitignore

@@ -3,7 +3,10 @@
 .vscode/
 .DS_Store
 *.swp
-.claude/
+**/.claude/*
+!/.claude/skills
+/.claude/skills/*
+!/.claude/skills/runtime-release
 
 # Python-generated files
 _version.py

{gpustack_runtime-0.2.5.post4 → gpustack_runtime-0.2.7}/AGENTS.md

@@ -59,6 +59,8 @@ Setup is `make install` (uv sync plus pre-commit hooks). Day-to-day targets:
 `gpustack_runtime/_version.py` is generated by hatch-vcs and `_version_appendix.py` by
 `make prepare`; never hand-edit either.
 
+Cut package releases with the [runtime-release](.agents/skills/runtime-release/SKILL.md) skill.
+
 ### Python conventions
 
 - Prefer clarity over cleverness to simplify long-term maintenance.

{gpustack_runtime-0.2.5.post4 → gpustack_runtime-0.2.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gpustack-runtime
-Version: 0.2.5.post4
+Version: 0.2.7
 Summary: GPUStack Runtime is library for detecting GPU resources and launching GPU workloads.
 Project-URL: Homepage, https://github.com/gpustack/runtime
 Project-URL: Bug Tracker, https://github.com/gpustack/gpustack/issues
@@ -16,7 +16,7 @@ Requires-Python: >=3.10
 Requires-Dist: argcomplete>=3.6.3
 Requires-Dist: cachetools>=5.5.2
 Requires-Dist: docker>=7.1.0
-Requires-Dist: gpustack-runner>=0.1.29
+Requires-Dist: gpustack-runner>=0.1.32.post2
 Requires-Dist: kubernetes>=33.1.0
 Requires-Dist: mthreads-ml-py>=2.2.10
 Requires-Dist: nvidia-ml-py>=13.580.65

{gpustack_runtime-0.2.5.post4 → gpustack_runtime-0.2.7}/gpustack_runtime/_version.py

@@ -27,8 +27,8 @@ version_tuple: VERSION_TUPLE
 __commit_id__: COMMIT_ID
 commit_id: COMMIT_ID
 
-__version__ = version = '0.2.5.post4'
-__version_tuple__ = version_tuple = (0, 2, 5, 'post4')
+__version__ = version = '0.2.7'
+__version_tuple__ = version_tuple = (0, 2, 7)
 try:
     from ._version_appendix import git_commit
     __commit_id__ = commit_id = git_commit

gpustack_runtime-0.2.7/gpustack_runtime/_version_appendix.py

@@ -0,0 +1 @@
+git_commit = "489860c"

{gpustack_runtime-0.2.5.post4 → gpustack_runtime-0.2.7}/gpustack_runtime/detector/__types__.py

@@ -1,6 +1,9 @@
 from __future__ import annotations as __future_annotations__
 
+import copy
 import logging
+import threading
+import time
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
 from enum import Enum
@@ -9,10 +12,17 @@ from typing import Any
 
 from dataclasses_json import dataclass_json
 
+from .. import envs
 from ..logging import debug_log_exception, debug_log_warning
 
 logger = logging.getLogger(__name__)
 
+_monotonic = time.monotonic
+"""
+The clock the identity cache's time to live is measured on. A module level
+indirection, so tests can inject a clock of their own.
+"""
+
 
 class ManufacturerEnum(str, Enum):
     """
@@ -606,6 +616,285 @@ def _usage_fields_to_merge(usage: Device) -> tuple[str, ...]:
     return tuple(field for field in _DEVICE_USAGE_FIELDS if field != "memory_status")
 
 
+def _clear_health_verdict(devices: Devices) -> None:
+    """
+    Drop the health verdict of the given devices, in place.
+
+    A verdict read when the devices were detected says nothing about them
+    now, and the refresh that could have confirmed it failed. So the devices
+    are served as inventory, with no verdict, rather than with one that went
+    stale.
+
+    Args:
+        devices:
+            The devices to drop the verdict of.
+
+    """
+    for dev in devices:
+        dev.memory_status = DeviceMemoryStatusEnum.UNKNOWN
+        # A MIG device is an entry of the card's appendix, and carries its own
+        # verdict, which the CLI reports as well.
+        for mig_dev in (dev.appendix or {}).get("mig_devices") or []:
+            mig_dev["memory_status"] = DeviceMemoryStatusEnum.UNKNOWN
+
+
+@dataclass
+class _IdentityCacheEntry:
+    """
+    A cached device identity and the moment it expires.
+    """
+
+    devices: Devices
+    """
+    The detected identity, handed out as a copy.
+    """
+
+    expires_at: float
+    """
+    The clock reading after which the identity is stale.
+    """
+
+
+class _RefreshFlight:
+    """
+    One identity refresh of a manufacturer.
+
+    A flight carries its own completion, so a caller that joined it takes
+    that flight's outcome, whatever a later flight does.
+    """
+
+    def __init__(self, stale: Devices | None = None):
+        self.done = threading.Event()
+        self.devices: Devices | None = None
+        self.error: BaseException | None = None
+        self.stale = stale
+        """
+        The last good identity when the flight started: the fallback a failed
+        flight owes its callers.
+        """
+
+    def complete(self, devices: Devices | None, error: BaseException | None) -> None:
+        """
+        Complete the flight with the devices it detected or the error it
+        raised.
+
+        Args:
+            devices:
+                The devices the flight detected, empty when it detected none.
+            error:
+                The exception the flight raised, None when it succeeded.
+
+        """
+        self.devices = devices
+        self.error = error
+        self.done.set()
+
+    def outcome(self) -> Devices | None:
+        """
+        Return the devices the flight detected, or the last good identity
+        when the flight failed.
+
+        The last good identity is served without its health verdict: the
+        verdict was read when the identity was detected, and the refresh that
+        could have confirmed it failed.
+
+        Returns:
+            The devices to serve, as a copy.
+
+        Raises:
+            The exception the flight raised, when no identity is cached to
+            fall back on.
+
+        """
+        if self.error is not None:
+            if self.stale is None:
+                raise self.error
+            # The last good identity is better than nothing, and the next call
+            # retries the detection.
+            devices = copy.deepcopy(self.stale)
+            _clear_health_verdict(devices)
+            return devices
+        return copy.deepcopy(self.devices)
+
+
+class _VendorIdentityState:
+    """
+    The identity cache state of one manufacturer.
+
+    The condition guards the entry and the live flight. A refresh runs
+    outside the lock, so a wedged vendor delays only the callers waiting for
+    that vendor.
+    """
+
+    def __init__(self):
+        self.condition = threading.Condition()
+        self.entry: _IdentityCacheEntry | None = None
+        self.flight: _RefreshFlight | None = None
+        """
+        The refresh running now, if any. A caller that finds one joins it
+        rather than starting a second detection of the same vendor.
+        """
+
+
+class _IdentityCache:
+    """
+    The process wide cache of detected device identities.
+
+    One state per manufacturer, so a slow refresh of one vendor delays only
+    the callers waiting for that vendor.
+    """
+
+    def __init__(self):
+        self._lock = threading.Lock()
+        self._states: dict[ManufacturerEnum, _VendorIdentityState] = {}
+
+    def state(self, manufacturer: ManufacturerEnum) -> _VendorIdentityState:
+        """
+        Return the cache state of the given manufacturer.
+
+        Args:
+            manufacturer:
+                The manufacturer to take the state of.
+
+        Returns:
+            The state of the manufacturer.
+
+        """
+        with self._lock:
+            state = self._states.get(manufacturer)
+            if state is None:
+                state = _VendorIdentityState()
+                self._states[manufacturer] = state
+            return state
+
+    def reset(self) -> None:
+        """
+        Drop every cached identity.
+        """
+        with self._lock:
+            self._states.clear()
+
+    def identity(self, detector: Detector, ttl: float) -> Devices | None:
+        """
+        Return the identity of the given detector, from the cache when it is
+        still fresh.
+
+        The identity is handed out as a copy: the usage merge writes into the
+        devices it is given, so a shared object would let one caller
+        contaminate the cache.
+
+        A fresh identity is served with the health verdict it was detected
+        with. A stale one is served without any verdict: the refresh that
+        could have confirmed it failed, so the last good identity stands for
+        the inventory only.
+
+        Args:
+            detector:
+                The detector to take the identity of.
+            ttl:
+                The time to live of a cached identity, in seconds.
+                Zero disables the cache.
+
+        Returns:
+            The devices carrying the identity, or None if there is none.
+
+        Raises:
+            The exception the detection raised, when the cache holds no
+            identity to fall back on. An interruption propagates to the caller
+            that asked for the detection, and the waiters of the same refresh
+            get its outcome.
+
+        """
+        if ttl <= 0:
+            # The cache is disabled: the identity is detected on every call.
+            return detector.detect_info()
+
+        state = self.state(detector.manufacturer)
+        with state.condition:
+            entry = state.entry
+            if entry is not None and _monotonic() < entry.expires_at:
+                return copy.deepcopy(entry.devices)
+            flight = state.flight
+            if flight is not None:
+                # A refresh is running: join it, so the callers of one vendor
+                # share one detection. The flight carries its own completion,
+                # so a later flight cannot postpone this caller's outcome.
+                state.condition.wait_for(flight.done.is_set)
+                return flight.outcome()
+            # No refresh is running: start one. The last good identity is the
+            # fallback the flight owes its callers when it fails.
+            flight = _RefreshFlight(
+                stale=entry.devices if entry is not None else None,
+            )
+            state.flight = flight
+
+        # The detection runs outside the lock: a wedged vendor must not hold
+        # the cache for the other vendors.
+        devices: Devices | None = None
+        error: BaseException | None = None
+        try:
+            devices = detector.detect_info()
+        except Exception as e:
+            debug_log_exception(
+                logger,
+                "Failed to detect %s devices identity, keeping the last known one",
+                detector.manufacturer,
+            )
+            error = e
+        except BaseException as e:
+            # An interruption settles the flight as well: a flight left
+            # pending blocks every later caller of the vendor forever. The
+            # interruption keeps propagating to the thread that asked for the
+            # detection, while the waiters get the flight's outcome.
+            debug_log_exception(
+                logger,
+                "Interrupted while detecting %s devices identity",
+                detector.manufacturer,
+            )
+            error = e
+            raise
+        finally:
+            with state.condition:
+                if error is None and devices:
+                    # Only a non empty identity is cached: one enumeration glitch
+                    # must not pin an empty inventory for the time to live.
+                    state.entry = _IdentityCacheEntry(
+                        devices=devices,
+                        expires_at=_monotonic() + ttl,
+                    )
+                flight.complete(devices, error)
+                state.flight = None
+                state.condition.notify_all()
+        return flight.outcome()
+
+
+_IDENTITY_CACHE = _IdentityCache()
+"""
+The process wide cache of detected device identities.
+"""
+
+
+def _identity_cache_ttl() -> float:
+    """
+    Return the time to live of a cached device identity, in seconds.
+
+    Zero disables the cache. The environment variable is read once per
+    process, so it must be set before the first detection.
+
+    Returns:
+        The time to live, in seconds.
+
+    """
+    return float(envs.GPUSTACK_RUNTIME_DETECT_INFO_CACHE_TTL)
+
+
+def _reset_identity_cache() -> None:
+    """
+    Drop every cached device identity.
+    """
+    _IDENTITY_CACHE.reset()
+
+
 class Detector(ABC):
     """
     Base class for all detectors.
@@ -677,6 +966,10 @@ class Detector(ABC):
         """
         Detect devices and return a list of Device objects.
 
+        The identity of the devices is cached process wide, per manufacturer,
+        so a repeated collection pays only the usage query. The usage is read
+        on every call, and so is the health it carries.
+
         Args:
             usage:
                 Whether to fetch the devices' usage as well.
@@ -685,7 +978,7 @@ class Detector(ABC):
             A list of detected Device objects, or None if detection fails.
 
         """
-        devices = self.detect_info()
+        devices = _IDENTITY_CACHE.identity(self, _identity_cache_ttl())
         if usage and devices:
             try:
                 self.detect_usage(devices)

{gpustack_runtime-0.2.5.post4 → gpustack_runtime-0.2.7}/gpustack_runtime/detector/amd.py

@@ -502,6 +502,9 @@ def _get_memory_status(dev, dev_idx: int) -> DeviceMemoryStatusEnum:
     Both queries produce it, mirroring the operator, which reports `Unhealthy`
     from `DetectAccelerator` and `MonitorAccelerator` alike.
 
+    A failed read yields no verdict, as the operator's counter read does: only
+    a readable value may condemn a card.
+
     Args:
         dev:
             The AMD SMI device handle.
@@ -521,14 +524,13 @@ def _get_memory_status(dev, dev_idx: int) -> DeviceMemoryStatusEnum:
             pyamdsmi.AmdSmiGpuBlock.UMC,
         )
     except pyamdsmi.AmdSmiException as e:
-        # Fail closed: a genuine driver error marks the device unhealthy, while
-        # a card without the counter cannot be judged. The stub binding raises
-        # a codeless error when amdsmi is absent, in which case ROCm SMI is
-        # the fallback, as it is for the memory read.
+        # Fail open: a genuine driver error yields no verdict, as the
+        # operator's counter read does. The stub binding raises a codeless
+        # error when amdsmi is absent, in which case ROCm SMI is the
+        # fallback, as it is for the memory read.
         err_code = getattr(e, "err_code", None)
         if err_code is not None:
-            if err_code != getattr(pyamdsmi, "AMDSMI_STATUS_NOT_SUPPORTED", None):
-                return DeviceMemoryStatusEnum.UNHEALTHY
+            debug_log_warning(logger, "Failed to read the UMC ECC error count")
             return DeviceMemoryStatusEnum.HEALTHY
     else:
         if dev_ecc_count.get("uncorrectable_count", 0) > 0:
@@ -539,19 +541,11 @@ def _get_memory_status(dev, dev_idx: int) -> DeviceMemoryStatusEnum:
         dev_ecc_count = pyrocmsmi.rsmi_dev_ecc_count_get(dev_idx)
         if dev_ecc_count.uncorrectable_err > 0:
             return DeviceMemoryStatusEnum.UNHEALTHY
-    except pyrocmsmi.ROCMSMIError as e:
-        # rsmi_status_t comes from the ROCm-installed rsmiBindings, so it is
-        # looked up late: without ROCm the module has no such attribute.
-        rsmi_not_supported = getattr(
-            getattr(pyrocmsmi, "rsmi_status_t", None),
-            "RSMI_STATUS_NOT_SUPPORTED",
-            None,
-        )
-        if e.value not in (
-            pyrocmsmi.ROCMSMI_ERROR_FUNCTION_NOT_FOUND,
-            rsmi_not_supported,
-        ):
-            return DeviceMemoryStatusEnum.UNHEALTHY
+    except pyrocmsmi.ROCMSMIError:
+        # Fail open: a terminal error inside the fallback yields no verdict
+        # either, so the card stays unjudged.
+        debug_log_warning(logger, "Failed to read the ROCm SMI ECC error count")
+        return DeviceMemoryStatusEnum.HEALTHY
 
     return DeviceMemoryStatusEnum.HEALTHY