cortexgrid 0.3.11__tar.gz → 0.3.12__tar.gz

{cortexgrid-0.3.11 → cortexgrid-0.3.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cortexgrid
-Version: 0.3.11
+Version: 0.3.12
 Summary: Connect your ML code to the RoboLab compute cluster — Ray, MLflow, and S3
 Project-URL: Homepage, https://github.com/robodatalab/cortexgrid
 Project-URL: Repository, https://github.com/robodatalab/cortexgrid
@@ -215,8 +215,10 @@ app = FastAPI()
 
 @serve.ingress(app)
 class MyServeApp:
-    def __init__(self, family: str, suffix: str, run_name: str) -> None:
-        self._weights_dir = cortexgrid.load_model(family, suffix, run_name)
+    def __init__(self, deployment: cortexgrid.DeploymentKey) -> None:
+        self._weights_dir = cortexgrid.load_model(
+            deployment.family, deployment.suffix, deployment.run_name
+        )
 
     @app.post("/complete")
     async def complete(self, body: dict): ...
@@ -232,11 +234,11 @@ print(deployed.url)
 
 The requirements are part of the model, not of the serve-app class: GPUs, RAM and VRAM (GiB, 0 meaning no requirement) are stored with it and matched against what the cluster's hosts have free. Among the hosts that fit, the model goes to the **smallest GPU** that does, so a 4 GiB model does not occupy a 128 GiB card a bigger one needs; it moves up only once the smaller cards are full. `num_gpus` may be a fraction (`0.25`) to share one card between models, in which case `vram_gb` is what keeps them from overcommitting it. Correct them later with `cortexgrid.set_model_requirements(family, suffix, run_name, requirements)` or on the model card in the dashboard; `cortexgrid.deploy_model(..., num_replicas=2)` chooses how many copies to run.
 
-Anything else the serve-app has to know about the model - which model a provider should be asked for, an endpoint, the name of a secret to read - goes in a free-form string mapping on the same entry: `save_model(..., config={"model": "claude-opus-5"})`. The serve-app reads it in `__init__` with `cortexgrid.model_config(family, suffix, run_name)`; `cortexgrid.set_model_config(family, suffix, run_name, config)` or the model card replaces it. cortexgrid stores the mapping without interpreting it, and a tag is readable by anyone with registry access, so a credential belongs in `set_secret` with only its name in the config.
+Anything else the serve-app has to know about the model - which model a provider should be asked for, an endpoint, the name of a secret to read - goes in a free-form string mapping on the same entry: `save_model(..., config={"model": "claude-opus-5"})`. The serve-app reads it in `__init__` with `cortexgrid.model_config(deployment)`; `cortexgrid.set_model_config(family, suffix, run_name, config)` or the model card replaces it. How one deployment serves the model - thinking on or off, say - is that deployment's own: `deploy_model(..., config={"thinking": "false"})` deploys the model once per distinct config, each deployment named by the `DeploymentKey` on the returned `Deployment`, and `model_config` lays the deployment's config over the model's. cortexgrid stores the mapping without interpreting it, and a tag is readable by anyone with registry access, so a credential belongs in `set_secret` with only its name in the config.
 
 `save_model` saves a new copy under every run - meant for weights the run produced (e.g. a fine-tune). For a model produced elsewhere (e.g. a pretrained base model), `cortexgrid.import_model(source, MyServeApp, family, suffix)` uploads it once under `run_name=cortexgrid.IMPORTED` and on later runs only re-bundles `MyServeApp` if its code changed; deploy it with `deploy_model(family, suffix, cortexgrid.IMPORTED)`. A model with no weights to stage - one behind a provider's API, e.g. Gemini or OpenAI - is registered the same way by `cortexgrid.register_model(MyServeApp, family, suffix, config=...)`: same key, same reuse, only the bundle stored. [Serving a hosted-API model](https://github.com/robodatalab/cortexgrid/blob/main/docs/cortexgrid/model-serving.md#serving-a-hosted-api-model) walks through one end to end - serve-app, API key, registration, deploy.
 
-`save_model` is synchronous (registry lifecycle: `uploading` -> `ready`); `deploy_model` schedules the serving lifecycle (`deploying` -> `running`). With `wait=True` a failed deploy raises `cortexgrid.ModelDeployFailed`; `cortexgrid.wait_for_model_serving(family, suffix, run_name, timeout=...)` waits on a deploy started elsewhere, and re-deploying a failed model retries it from scratch. See [model-serving.md](https://github.com/robodatalab/cortexgrid/blob/main/docs/cortexgrid/model-serving.md) for both lifecycles end to end - upload/deploy/undeploy/delete, status queries (`model_registry_status`, `model_serving_status`), and error handling.
+`save_model` is synchronous (registry lifecycle: `uploading` -> `ready`); `deploy_model` schedules the serving lifecycle (`deploying` -> `running`). With `wait=True` a failed deploy raises `cortexgrid.ModelDeployFailed`; `cortexgrid.wait_for_model_serving(deployment.key, timeout=...)` waits on a deploy started elsewhere, and re-deploying a failed model retries it from scratch. See [model-serving.md](https://github.com/robodatalab/cortexgrid/blob/main/docs/cortexgrid/model-serving.md) for both lifecycles end to end - upload/deploy/undeploy/delete, status queries (`model_registry_status`, `model_serving_status`), and error handling.
 
 ### API reference
 

{cortexgrid-0.3.11 → cortexgrid-0.3.12}/cortexgrid/__init__.py

@@ -101,12 +101,16 @@ from cortexgrid.model_storage import register_model as _register_model_storage
 from cortexgrid.model_storage import save_model as _save_model_storage
 from cortexgrid.model_serving import (
     Deployment,
+    DeploymentConfig,
+    DeploymentKey,
     ModelDeployFailed,
+    ModelNotDeployed,
     ModelRequirements,
     ServingStatus,
     deploy_model,
     list_deployed_models,
     model_serving_status,
+    redeploy_model,
     undeploy_model,
     wait_for_model_serving,
 )
@@ -308,9 +312,13 @@ __all__ = [
     "delete_model",
     # Model serving
     "Deployment",
+    "DeploymentConfig",
+    "DeploymentKey",
     "ModelDeployFailed",
+    "ModelNotDeployed",
     "ServingStatus",
     "deploy_model",
+    "redeploy_model",
     "wait_for_model_serving",
     "model_serving_status",
     "undeploy_model",

{cortexgrid-0.3.11 → cortexgrid-0.3.12}/cortexgrid/_serve_entry.py

@@ -6,13 +6,14 @@ On the cluster replica, `build` imports the serve-app class bundled at
 ingress with the app it was marked with by `cortexgrid.serve.ingress` (again on
 each replica, see `_IngressOnReplica`), wraps it as a Ray Serve deployment
 with the replica count and Ray resource requests `deploy_model` derived from
-the model's requirements, and binds it with the (family, suffix, run_name)
-identifiers.
+the model's requirements, and binds it with its deployment's
+`cortexgrid.DeploymentKey`.
 
 The serve-app owns everything about traffic: its own routes, request schemas,
 streaming, and timeouts. cortexgrid does not interpose a request/response
-contract - it only schedules the app and hands it the identifiers it needs to
-fetch its own weights via `cortexgrid.load_model`.
+contract - it only schedules the app and hands it the key it needs to fetch
+its own weights via `cortexgrid.load_model` and its settings via
+`cortexgrid.model_config`.
 
 The serve-app declares no resources: the hardware a replica needs belongs to
 the model and is stored in the registry (`cortexgrid.ModelRequirements`), and
@@ -28,6 +29,7 @@ from ray import serve
 from ray.serve.deployment import Application
 
 from cortexgrid._model_scheduler import model_autoscaling_config
+from cortexgrid.model_serving.deployment_key import DeploymentKey
 from cortexgrid.serve import ingress_app
 
 
@@ -80,4 +82,11 @@ def build(args: dict[str, Any]) -> Application:
         # had on Ray 2.9.
         max_ongoing_requests=_MAX_ONGOING_REQUESTS,
         ray_actor_options=args.get("ray_actor_options", {}),
-    ).bind(args["family"], args["suffix"], args["run_name"])
+    ).bind(
+        DeploymentKey(
+            args["family"],
+            args["suffix"],
+            args["run_name"],
+            args.get("config_fingerprint", ""),
+        )
+    )

cortexgrid-0.3.12/cortexgrid/model_serving/__init__.py

@@ -0,0 +1,73 @@
+"""Cortexgrid wrappers around Ray Serve.
+
+Caller stays HTTP-only: deploy/undeploy/list talk to the Ray dashboard's
+declarative `/api/serve/applications/` endpoint via [cortexgrid.ray_util],
+never `ray.init`. The deployment class is bundled at `save_model` time, zipped,
+uploaded to MinIO under
+`serve-bundles/<run_name>/<family>__<suffix>/<fingerprint>.zip`, and
+referenced via `runtime_env.working_dir` so Ray workers fetch it from there.
+The bundle URL, class import path, pip list, and fingerprint are persisted as
+tags on the model's registry entry so `deploy_model` can find them later
+without the caller holding the class object. So are the model's
+`ModelRequirements`, which `deploy_model` turns into the replica's Ray resource
+requests.
+
+Every deployment `deploy_model` puts on Ray Serve gets a record with the jobs
+control plane, keyed by its `DeploymentKey` - the model's (family, suffix,
+run_name) plus a fingerprint of the config it was deployed with: that config,
+the spec it PUT, plus the phase, message and replica
+placements the control plane last observed (`observe_deployments`, run on
+every poll cycle). Listings and status reads come from those records; waits
+ask the Serve controller directly.
+
+Naming: the Ray Serve application is named "<family>__<suffix>__<run_name>",
+followed by "__<config_fingerprint>" for a deployment given a config.
+This relies on family/suffix/run_name not containing the literal "__".
+
+See [docs/cortexgrid/model-serving.md](../../docs/cortexgrid/model-serving.md)
+for the end-to-end design.
+"""
+
+from cortexgrid.model_serving.application_spec import app_name
+from cortexgrid.model_serving.deployment_key import (
+    DeploymentConfig,
+    DeploymentKey,
+    deployment_key,
+)
+from cortexgrid.model_serving.lifecycle import (
+    ModelDeployFailed,
+    ModelNotDeployed,
+    deploy_model,
+    redeploy_model,
+    undeploy_model,
+    wait_for_model_serving,
+)
+from cortexgrid.model_serving.placement import ModelRequirements, vram_tiers
+from cortexgrid.model_serving.registry_tags import (
+    bundle_fingerprint_from_tags,
+    has_requirement_tags,
+    metadata_from_tags,
+    metadata_to_tags,
+    requirements_from_tags,
+    requirements_to_tags,
+)
+from cortexgrid.model_serving.serve_bundle import (
+    BundleMetadata,
+    ServeBundle,
+    build_bundle,
+    bundle_class,
+    bundle_fingerprint_from_url,
+    upload_bundle,
+)
+from cortexgrid.model_serving.status import (
+    Deployment,
+    ReplicaPlacement,
+    ServingMessage,
+    ServingStatus,
+    deployment_config,
+    list_deployed_models,
+    model_replica_placements,
+    model_serving_messages,
+    model_serving_status,
+    observe_deployments,
+)

cortexgrid-0.3.12/cortexgrid/model_serving/application_spec.py

@@ -0,0 +1,70 @@
+from __future__ import annotations
+
+from typing import Any
+
+from cortexgrid.model_serving.deployment_key import DeploymentKey
+from cortexgrid.model_serving.placement import ModelRequirements, ray_actor_options
+from cortexgrid.model_serving.serve_bundle import (
+    BundleMetadata,
+    bundle_fingerprint_from_url,
+)
+
+
+def app_name(key: DeploymentKey) -> str:
+    return "__".join(_name_segments(key))
+
+
+def route_prefix(key: DeploymentKey) -> str:
+    return "/r/" + "/".join(_name_segments(key))
+
+
+def _name_segments(key: DeploymentKey) -> list[str]:
+    model_segments = [key.family, key.suffix, key.run_name]
+    if not key.config_fingerprint:
+        return model_segments
+    return [*model_segments, key.config_fingerprint]
+
+
+def build_application_spec(
+    key: DeploymentKey,
+    meta: BundleMetadata,
+    requirements: ModelRequirements,
+    num_replicas: int,
+    tiers: list[int],
+) -> dict[str, Any]:
+    """Assemble a Ray Serve application schema from pre-bundled metadata, the
+    model's requirements, and the cluster's GPU size classes."""
+    # working_dir carries the serve-app's own source; Ray pip-installs the
+    # third-party distributions the image lacks into a per-node cached
+    # virtualenv layered on the image. No pip key when there are none, so Ray
+    # builds no virtualenv.
+    runtime_env: dict[str, Any] = {"working_dir": meta.bundle_url}
+    if meta.pip_requirements:
+        runtime_env["pip"] = meta.pip_requirements
+    return {
+        "name": app_name(key),
+        "route_prefix": route_prefix(key),
+        # Ray Serve REST requires import_path to point at an Application builder
+        # (callable returning a bound node) or an already-bound node. A bare
+        # Deployment class is rejected, so cortexgrid.deploy_model goes through
+        # a generic builder that re-imports the user's class and binds it.
+        "import_path": "cortexgrid._serve_entry:build",
+        "args": {
+            "class_import_path": meta.class_import_path,
+            "family": key.family,
+            "suffix": key.suffix,
+            "run_name": key.run_name,
+            "config_fingerprint": key.config_fingerprint,
+            "num_replicas": num_replicas,
+            "ray_actor_options": ray_actor_options(requirements, tiers),
+        },
+        "runtime_env": runtime_env,
+    }
+
+
+def bundle_fingerprint_in_spec(spec: dict[str, Any]) -> str:
+    return bundle_fingerprint_from_url(spec["runtime_env"]["working_dir"])
+
+
+def replica_count_in_spec(spec: dict[str, Any]) -> int:
+    return spec["args"]["num_replicas"]

cortexgrid-0.3.12/cortexgrid/model_serving/deployment_key.py

@@ -0,0 +1,38 @@
+from __future__ import annotations
+
+import hashlib
+import json
+from dataclasses import dataclass
+
+
+DeploymentConfig = dict[str, str]
+
+_CONFIG_FINGERPRINT_LENGTH = 12
+
+
+@dataclass(frozen=True)
+class DeploymentKey:
+    family: str
+    suffix: str
+    run_name: str
+    config_fingerprint: str = ""
+
+
+def deployment_key(
+    family: str, suffix: str, run_name: str, config: DeploymentConfig
+) -> DeploymentKey:
+    return DeploymentKey(family, suffix, run_name, _config_fingerprint(config))
+
+
+def _config_fingerprint(config: DeploymentConfig) -> str:
+    for name, value in config.items():
+        if not isinstance(name, str) or not isinstance(value, str):
+            raise ValueError(
+                f"Deployment config must map strings to strings: {name!r}: {value!r}"
+            )
+    if not config:
+        return ""
+    canonical_config = json.dumps(config, sort_keys=True)
+    return hashlib.sha256(canonical_config.encode()).hexdigest()[
+        :_CONFIG_FINGERPRINT_LENGTH
+    ]

cortexgrid-0.3.12/cortexgrid/model_serving/deployment_records.py

@@ -0,0 +1,46 @@
+from __future__ import annotations
+
+from typing import Any
+
+from cortexgrid import state
+from cortexgrid.model_serving.deployment_key import DeploymentKey
+
+
+DeploymentRecord = dict[str, Any]
+
+
+def get_deployment_record(key: DeploymentKey) -> DeploymentRecord | None:
+    return state.get(*_path(key), params=_params(key))
+
+
+def put_deployment_record(key: DeploymentKey, body: DeploymentRecord) -> None:
+    state.put(*_path(key), body=body, params=_params(key))
+
+
+def patch_deployment_record(key: DeploymentKey, body: DeploymentRecord) -> bool:
+    return state.patch(*_path(key), body=body, params=_params(key))
+
+
+def delete_deployment_record(key: DeploymentKey) -> None:
+    state.delete(*_path(key), params=_params(key))
+
+
+def list_deployment_records() -> list[DeploymentRecord]:
+    return state.get("deployments")
+
+
+def key_of_record(record: DeploymentRecord) -> DeploymentKey:
+    return DeploymentKey(
+        record["family"],
+        record["suffix"],
+        record["run_name"],
+        record["config_fingerprint"],
+    )
+
+
+def _path(key: DeploymentKey) -> tuple[str, ...]:
+    return ("deployments", key.family, key.suffix, key.run_name)
+
+
+def _params(key: DeploymentKey) -> dict[str, str]:
+    return {"config_fingerprint": key.config_fingerprint}

cortexgrid-0.3.12/cortexgrid/model_serving/lifecycle.py

@@ -0,0 +1,338 @@
+from __future__ import annotations
+
+import time
+from typing import Any
+
+from ray.serve.schema import ApplicationStatus
+
+from cortexgrid.infra import get_ray_serve_uri
+from cortexgrid.model_serving.application_spec import (
+    app_name,
+    build_application_spec,
+    bundle_fingerprint_in_spec,
+    replica_count_in_spec,
+    route_prefix,
+)
+from cortexgrid.model_serving.deployment_key import (
+    DeploymentConfig,
+    DeploymentKey,
+    deployment_key,
+)
+from cortexgrid.model_serving.deployment_records import (
+    DeploymentRecord,
+    delete_deployment_record,
+    get_deployment_record,
+    patch_deployment_record,
+    put_deployment_record,
+)
+from cortexgrid.model_serving.placement import ModelRequirements, vram_tiers
+from cortexgrid.model_serving.registry_tags import load_deploy_metadata
+from cortexgrid.model_serving.serve_bundle import BundleMetadata
+from cortexgrid.model_serving.status import (
+    PHASE_PAUSED,
+    Deployment,
+    deployment_of_record,
+    observed,
+    replaced_bundle_fingerprint_until_rolled_out,
+)
+from cortexgrid.ray_util import get_serve_details, put_serve_applications
+
+
+def _current_application_specs() -> list[dict[str, Any]]:
+    """Reconstruct the most recently PUT applications list from GET output.
+
+    Ray stores the originally-deployed `ServeApplicationSchema` for each app
+    under `applications[<name>].deployed_app_config`, which is what we need to
+    PUT-round-trip. Apps without a `deployed_app_config` (e.g. created via
+    `serve.run` in-cluster) are skipped: we can't faithfully reproduce them
+    from the read-only view.
+    """
+    details = get_serve_details()
+    specs: list[dict[str, Any]] = []
+    for app in details.get("applications", {}).values():
+        cfg = app.get("deployed_app_config")
+        if cfg is not None:
+            specs.append(cfg)
+    return specs
+
+
+class ModelDeployFailed(RuntimeError):
+    """A model's Serve app cannot reach RUNNING: the controller reported
+    DEPLOY_FAILED, or no app exists for the model. Subclasses RuntimeError, which
+    `deploy_model(wait=True)` raised before this type existed."""
+
+
+class ModelNotDeployed(LookupError):
+    pass
+
+
+_SERVING_POLL_INTERVAL_S = 2.0
+
+
+def _deadline(timeout: float | None) -> float | None:
+    return None if timeout is None else time.monotonic() + timeout
+
+
+def _past(deadline: float | None) -> bool:
+    return deadline is not None and time.monotonic() >= deadline
+
+
+def wait_for_model_serving(key: DeploymentKey, timeout: float | None = None) -> None:
+    """Block until the model's Serve app is RUNNING.
+
+    Raises ModelDeployFailed on DEPLOY_FAILED, carrying the controller's message,
+    and as soon as no app exists for the model: never deployed, undeployed, or
+    dropped by a concurrent `deploy_model` (each one GETs the applications list,
+    splices its own app in and PUTs the whole list back, so a later PUT can drop
+    an app an earlier one added). NOT_STARTED, DEPLOYING, UNHEALTHY and DELETING
+    are transient; a DELETING app ends up missing. Exceeding a finite `timeout`
+    raises TimeoutError; with `timeout=None` there is no deadline.
+    """
+    _wait_for_application_running(app_name(key), timeout, _deadline(timeout))
+
+
+def _wait_for_application_running(
+    name: str, timeout: float | None, deadline: float | None
+) -> None:
+    """`wait_for_model_serving` against a deadline already running. `timeout`
+    only labels the TimeoutError."""
+    while True:
+        app = get_serve_details().get("applications", {}).get(name)
+        if app is None:
+            raise ModelDeployFailed(f"Serve app {name!r} does not exist")
+        status = str(app.get("status", "(missing)"))
+        message = str(app.get("message", ""))
+        if status == ApplicationStatus.RUNNING.value:
+            return
+        if status == ApplicationStatus.DEPLOY_FAILED.value:
+            raise ModelDeployFailed(f"Serve app {name!r} DEPLOY_FAILED: {message}")
+        if _past(deadline):
+            raise TimeoutError(
+                f"Serve app {name!r} did not reach RUNNING within {timeout}s "
+                f"(last status={status!r}, message={message!r})"
+            )
+        time.sleep(_SERVING_POLL_INTERVAL_S)
+
+
+def _clear_failed_application(
+    key: DeploymentKey, timeout: float | None, deadline: float | None
+) -> None:
+    """Remove the model's DEPLOY_FAILED Serve app, and wait until it, or an app
+    already DELETING, is gone.
+
+    Ray resets a failed deployment only when a deploy arrives after the
+    deployment is marked for deletion or its version changes. Re-PUTting an
+    identical spec over a failed app, or PUTting it back before the controller's
+    next tick has processed an undeploy, leaves the failed deployment in place,
+    and the app reports DEPLOY_FAILED again without retrying."""
+    name = app_name(key)
+    app = get_serve_details().get("applications", {}).get(name)
+    if app is None:
+        return
+    status = app.get("status")
+    if status == ApplicationStatus.DEPLOY_FAILED.value:
+        undeploy_model(key)
+    elif status != ApplicationStatus.DELETING.value:
+        return
+    while name in get_serve_details().get("applications", {}):
+        if _past(deadline):
+            raise TimeoutError(
+                f"Serve app {name!r} was not removed within {timeout}s"
+            )
+        time.sleep(_SERVING_POLL_INTERVAL_S)
+
+
+def _spec_already_deployed(spec: dict[str, Any]) -> bool:
+    """True when this exact spec is already the app's target, so PUTting it
+    again would only disturb the controller.
+
+    Re-PUTting is not a no-op. While an app's build task is in flight its target
+    code version is unset, so Ray cancels that build and starts a new one no
+    matter what the config says - and since every PUT re-sends the whole
+    applications list, it restarts the in-flight builds of the other apps too.
+    Our build task downloads the model bundle and may create a pip virtualenv,
+    so a caller redeploying faster than that could keep it from ever finishing.
+
+    A DEPLOY_FAILED or DELETING app never reaches here: `_clear_failed_application`
+    has already removed it, and an identical PUT over a failed app is exactly the
+    no-op that leaves it failed.
+    """
+    app = get_serve_details().get("applications", {}).get(spec["name"])
+    return app is not None and app.get("deployed_app_config") == spec
+
+
+# Phases in which a deployment record stands for a live app: one a redeploy
+# with the same spec may leave alone. A failed app has to be cleared and
+# re-PUT, a deleting one waited out, and a missing one PUT again.
+_LIVE_PHASES = ("running", "deploying", "not_started", "unhealthy", PHASE_PAUSED)
+
+
+def _record_is_current(
+    record: DeploymentRecord,
+    key: DeploymentKey,
+    meta: BundleMetadata,
+    requirements: ModelRequirements,
+    num_replicas: int,
+) -> bool:
+    """True when the deployment record shows the model live with exactly the
+    spec this deploy would PUT.
+
+    The spec is rebuilt against the GPU tiers the record was deployed with,
+    so the check asks neither Ray's state API nor the Serve controller: a
+    redeploy that changes nothing costs one lookup. A tier added to or gone
+    from the cluster since reaches the spec on the next deploy that changes
+    anything else; until then the placement fallback keeps larger new GPUs
+    usable (see `_placement_preferences`)."""
+    if record["phase"] not in _LIVE_PHASES:
+        return False
+    spec = build_application_spec(key, meta, requirements, num_replicas, record["tiers"])
+    return spec == record["spec"]
+
+
+def _bundle_fingerprint_replaced_by(
+    record: DeploymentRecord | None, meta: BundleMetadata
+) -> str:
+    if record is None or record["phase"] not in _LIVE_PHASES:
+        return ""
+    rollout_origin = record["replaced_bundle_fingerprint"] or bundle_fingerprint_in_spec(
+        record["spec"]
+    )
+    return "" if rollout_origin == meta.fingerprint else rollout_origin
+
+
+def _observation_of_application(
+    name: str, replaced_bundle_fingerprint: str
+) -> DeploymentRecord:
+    observation = observed(get_serve_details().get("applications", {}).get(name, {}))
+    observation["replaced_bundle_fingerprint"] = (
+        replaced_bundle_fingerprint_until_rolled_out(
+            replaced_bundle_fingerprint, observation["phase"]
+        )
+    )
+    return observation
+
+
+def deploy_model(
+    family: str,
+    suffix: str,
+    run_name: str,
+    num_replicas: int = 1,
+    wait: bool = False,
+    timeout: float | None = 300.0,
+    config: DeploymentConfig | None = None,
+) -> Deployment:
+    """Schedule a Ray Serve app for a previously-saved model and return a
+    handle carrying its base URL. The caller (e.g. model-gateway) builds
+    whatever client the app's routes need - streaming, long timeouts, custom
+    request schemas - against that URL; cortexgrid imposes no traffic contract.
+
+    The serve-app class is pulled from the registry entry's tags `save_model`
+    wrote at save time; the caller does not need to hold the class object.
+    Each of the `num_replicas` replicas requests the model's `ModelRequirements`
+    from Ray, so it is placed only on a node that has them free.
+
+    A DEPLOY_FAILED app left by an earlier attempt is undeployed first, and it,
+    or an app still DELETING, is waited out before the new spec is PUT, so the
+    deploy starts afresh instead of Ray reusing the failed deployment.
+
+    Re-deploying a model that is already live with exactly this spec skips the
+    PUT rather than restating it: see `_spec_already_deployed` for what a
+    redundant PUT costs. When the model's deployment record already shows it
+    live with this spec, the call returns from the record without asking Ray
+    at all (see `_record_is_current`). The call still reports the app's phase,
+    and with `wait=True` still blocks until it is RUNNING.
+
+    `config` holds this deployment's own settings. A model gets one deployment
+    per distinct config, told apart by the `DeploymentKey` on the returned
+    `Deployment`; its replicas read the config, laid over the model's own, with
+    `model_config`.
+
+    Records the deployment - its config, the spec it PUT, where the app is
+    served, and its phase - with the jobs control plane before waiting.
+
+    With `wait=True`, blocks as `wait_for_model_serving` does until the Serve
+    controller reports the app RUNNING. `timeout` (default 300) caps the whole
+    call, clearing a failed app included; exceeding it raises TimeoutError, and
+    DEPLOY_FAILED raises ModelDeployFailed. With `timeout=None` there is no cap.
+    Tradeoff: an app that never reaches a terminal state (e.g. GPU-starved,
+    stuck in DEPLOYING) will hang forever.
+    """
+    deadline = _deadline(timeout)
+    deployment_config = config or {}
+    key = deployment_key(family, suffix, run_name, deployment_config)
+    meta, requirements = load_deploy_metadata(family, suffix, run_name)
+    record = get_deployment_record(key)
+    if record is not None and _record_is_current(
+        record, key, meta, requirements, num_replicas
+    ):
+        if wait:
+            _wait_for_application_running(record["spec"]["name"], timeout, deadline)
+            record = {
+                **record,
+                "phase": observed(
+                    get_serve_details().get("applications", {}).get(record["spec"]["name"])
+                )["phase"],
+            }
+        return deployment_of_record(record)
+    # Read afresh on every deploy that reaches Ray: the tiers are what the
+    # model is placed against, so a GPU joining or leaving the cluster has to
+    # change the spec (and therefore re-PUT it).
+    replaced_bundle_fingerprint = _bundle_fingerprint_replaced_by(record, meta)
+    tiers = vram_tiers()
+    spec = build_application_spec(key, meta, requirements, num_replicas, tiers)
+    _clear_failed_application(key, timeout, deadline)
+    if not _spec_already_deployed(spec):
+        existing = [
+            a for a in _current_application_specs() if a["name"] != spec["name"]
+        ]
+        # The controller registers the app, sets it DEPLOYING and stamps
+        # last_deployed_time_s before the PUT returns, so the wait below neither
+        # misses the app nor reads a status left by an earlier deploy.
+        put_serve_applications([*existing, spec])
+    url = f"{get_ray_serve_uri()}{route_prefix(key)}"
+    observation = _observation_of_application(spec["name"], replaced_bundle_fingerprint)
+    put_deployment_record(
+        key,
+        {
+            "config": deployment_config,
+            "spec": spec,
+            "tiers": tiers,
+            "url": url,
+            **observation,
+        },
+    )
+    if wait:
+        _wait_for_application_running(spec["name"], timeout, deadline)
+        observation = _observation_of_application(
+            spec["name"], replaced_bundle_fingerprint
+        )
+        patch_deployment_record(key, observation)
+    return Deployment(
+        key=key,
+        config=deployment_config,
+        url=url,
+        phase=observation["phase"],
+        bundle_fingerprint=meta.fingerprint,
+        replaced_bundle_fingerprint=observation["replaced_bundle_fingerprint"],
+    )
+
+
+def redeploy_model(key: DeploymentKey) -> Deployment:
+    record = get_deployment_record(key)
+    if record is None:
+        raise ModelNotDeployed(f"{key} is not deployed")
+    return deploy_model(
+        key.family,
+        key.suffix,
+        key.run_name,
+        num_replicas=replica_count_in_spec(record["spec"]),
+        config=record["config"],
+    )
+
+
+def undeploy_model(key: DeploymentKey) -> None:
+    """Tear down the Ray Serve app of this deployment and drop its record."""
+    name = app_name(key)
+    remaining = [a for a in _current_application_specs() if a["name"] != name]
+    put_serve_applications(remaining)
+    delete_deployment_record(key)