cortexgrid 0.2.96__tar.gz → 0.3.0__tar.gz

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cortexgrid
-Version: 0.2.96
+Version: 0.3.0
 Summary: Connect your ML code to the RoboLab compute cluster — Ray, MLflow, and S3
 Project-URL: Homepage, https://github.com/robodatalab/cortexgrid
 Project-URL: Repository, https://github.com/robodatalab/cortexgrid
@@ -107,11 +107,61 @@ def train_step(batch):
     # MLflow and S3 env vars are injected automatically
     return loss
 
-job_id = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
-print(f"Submitted: {job_id}")
+job = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
+print(f"Submitted: {job.job_id}")
 ```
 
-`cortexgrid.remote` submits a job *request* (a pickled payload plus a `JobLifecycle` record) to MLflow and returns a job ID string immediately. It does not wait for the job to run or finish — use the UI at `http://<DGX_IP>:8000`, or poll `cortexgrid.list_experiment_run_jobs(run_id)`, to observe status.
+`cortexgrid.remote` submits a job *request* (a pickled payload plus a `JobLifecycle` record) to MLflow and returns a `JobFuture` immediately. It does not wait for the job to run or finish — use the UI at `http://<DGX_IP>:8000`, `job.status()`, or poll `cortexgrid.list_experiment_run_jobs(run_id)`, to observe status.
+
+##### Blocking on the result
+
+`JobFuture.result(timeout=None)` blocks until Ray reports the job terminal, then behaves like a local call: it returns whatever the function returned, or raises whatever the function raised.
+
+```python
+def score(batch) -> dict[str, float]:
+    # runs on the DGX
+    return {"loss": 0.12}
+
+# Fire-and-forget — the form training uses. Nothing blocks, and a retry=True
+# job cannot be waited on at all (see below).
+training = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
+print(training.job_id, training.status().value)   # "blue-42 running"
+
+# Blocking — the value comes back as if score() had run locally.
+job = cortexgrid.remote(score, batch, num_gpus=1)
+metrics = job.result()                            # {"loss": 0.12}
+
+# ...or with a deadline. On TimeoutError the job keeps running; wait again later.
+metrics = job.result(timeout=600)
+
+# From another process — the handle is gone, the job id is enough.
+metrics = cortexgrid.get_job_result(job_id, timeout=600)
+```
+
+Failures arrive as exceptions, not as a status to inspect:
+
+```python
+try:
+    metrics = job.result()
+except BadBatch as exc:                # exactly what score() raised on the DGX
+    print(exc.__cause__)               # JobFailed, carrying the remote traceback
+except cortexgrid.JobFailed:           # the job never got as far as returning
+    ...                                # submission failed, driver died, or stopped
+except cortexgrid.JobResultUnavailable:
+    ...                                # it returned, but the value is not transportable
+```
+
+The driver cloudpickles the outcome to `job/{job_id}/result.pkl`, beside the payload manifest and the lifecycle, and the waiting side reads it back:
+
+| Outcome | What `result()` does |
+|---|---|
+| The function returned | Returns its value |
+| The function raised | Raises that exception, with the remote traceback attached as a chained `JobFailed` cause |
+| The job never got as far as returning (submission failed, driver died, job stopped) | Raises `JobFailed` |
+| The value or the exception did not survive cloudpickle | Raises `JobResultUnavailable` (value) or `JobFailed` (exception). A value that cannot be pickled never fails the job itself |
+| `timeout` elapsed | Raises `TimeoutError`; the job keeps running |
+
+Waiting on a `retry=True` job raises `ValueError`: retries are unbounded by design (see below), so the wait would have no end. Fire-and-forget submission is the form training uses — submit, then watch the UI.
 
 A separate service — the **jobs control plane** — polls MLflow for pending job requests, matches them against the set of Ray submissions the cluster already has, and submits anything missing. It is also responsible for retrying failed jobs and honouring user-requested stops.
 
@@ -165,19 +215,23 @@ app = FastAPI()
 
 @serve.ingress(app)
 class MyServeApp:
-    num_gpus = 1
-
     def __init__(self, family: str, suffix: str, run_name: str) -> None:
         self._weights_dir = cortexgrid.load_model(family, suffix, run_name)
 
     @app.post("/complete")
     async def complete(self, body: dict): ...
 
-saved = cortexgrid.save_model(weights_dir, MyServeApp, family="qwen", suffix="instruct")
+saved = cortexgrid.save_model(
+    weights_dir, MyServeApp, family="qwen", suffix="instruct",
+    # What one replica needs; the model is deployed only on a host that has it.
+    requirements=cortexgrid.ModelRequirements(num_gpus=1, ram_gb=8, vram_gb=16),
+)
 deployed = cortexgrid.deploy_model("qwen", "instruct", saved.run_name, wait=True)
 print(deployed.url)
 ```
 
+The requirements are part of the model, not of the serve-app class: GPUs, RAM and VRAM (GiB, 0 meaning no requirement) are stored with it and matched against what the cluster's hosts have free. Correct them later with `cortexgrid.set_model_requirements(family, suffix, run_name, requirements)` or on the model card in the dashboard; `cortexgrid.deploy_model(..., num_replicas=2)` chooses how many copies to run.
+
 `save_model` saves a new copy under every run - meant for weights the run produced (e.g. a fine-tune). For a model produced elsewhere (e.g. a pretrained base model), `cortexgrid.import_model(source, MyServeApp, family, suffix)` uploads it once under `run_name=cortexgrid.IMPORTED` and on later runs only re-bundles `MyServeApp` if its code changed; deploy it with `deploy_model(family, suffix, cortexgrid.IMPORTED)`.
 
 `save_model` is synchronous (registry lifecycle: `uploading` -> `ready`); `deploy_model` schedules the serving lifecycle (`deploying` -> `running`). With `wait=True` a failed deploy raises `cortexgrid.ModelDeployFailed`; `cortexgrid.wait_for_model_serving(family, suffix, run_name, timeout=...)` waits on a deploy started elsewhere, and re-deploying a failed model retries it from scratch. See [model-serving.md](https://github.com/robodatalab/cortexgrid/blob/main/docs/cortexgrid/model-serving.md) for both lifecycles end to end - upload/deploy/undeploy/delete, status queries (`model_registry_status`, `model_serving_status`), and error handling.
@@ -193,7 +247,9 @@ print(deployed.url)
 | `cortexgrid.log_artifact(path, artifact_path)` | Log a file as an artifact |
 | `cortexgrid.checkpoint()` | Context manager returning an attribute-based checkpoint saved to MLflow on exit |
 | `cortexgrid.resume()` | Load the latest checkpoint for the current job, or `None` |
-| `cortexgrid.remote(fn, *args, num_gpus=0, num_cpus=1, retry=False, **kwargs)` | Submit a function to the jobs control plane; returns a job ID |
+| `cortexgrid.remote(fn, *args, num_gpus=0, num_cpus=1, retry=False, **kwargs)` | Submit a function to the jobs control plane; returns a `JobFuture` |
+| `JobFuture.status()` / `.done()` / `.result(timeout=None)` | Live status of a submitted job, and its function's return value (blocking) |
+| `cortexgrid.get_job_result(job_id, timeout=None)` | Block on a job of the current run by id; returns its value or raises its exception |
 | `cortexgrid.list_experiment_run_jobs(run_id)` | List `JobLifecycle` records for every cortexgrid job in a run |
 | `cortexgrid.stop_experiment_run_jobs(run_id)` | Request every job in a run to stop (flips the `stop_requested` latch) |
 | `cortexgrid.get_ray_job_status(ray_job_id)` | Live Ray status for a submission id |

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/cortexgrid/__init__.py

@@ -2,14 +2,18 @@
 
 import cortexgrid
 
-# Fire-and-forget training on the DGX; returns a job id immediately.
+# Fire-and-forget training on the DGX; returns a handle immediately.
 # The jobs control plane picks up the submission and dispatches it to Ray.
-job_id = cortexgrid.remote(my_train, config, num_gpus=1, retry=True)
-print(f"Submitted: {job_id}")
+training = cortexgrid.remote(my_train, config, num_gpus=1, retry=True)
+print(f"Submitted: {training.job_id} ({training.status().value})")
 
-# Check on it later
-for job in cortexgrid.list_experiment_run_jobs(run_id):
-    status = cortexgrid.get_ray_job_status(job.get_ray_job_id())
+# Or block on the function's return value, as if it had run locally.
+# The job's own exception is what a failed job raises here.
+job = cortexgrid.remote(score, batch, num_gpus=1)
+loss = job.result(timeout=600)
+
+# In another process, the job id is enough
+loss = cortexgrid.get_job_result(job_id)
 
 # Inside the training function — checkpoint after each epoch
 with cortexgrid.checkpoint() as ckpt:
@@ -39,7 +43,12 @@ from cortexgrid.jobs import (
     schedule_remote_job,
     list_experiment_run_jobs,
     stop_experiment_run_jobs,
+    wait_for_job_result,
+    JobFailed,
+    JobFuture,
     JobLifecycle,
+    JobResult,
+    JobResultUnavailable,
     LifecycleEvent,
     Payload,
 )
@@ -80,12 +89,14 @@ from cortexgrid.model_storage import (
     list_models,
     load_model,
     model_registry_status,
+    set_model_requirements,
 )
 from cortexgrid.model_storage import import_model as _import_model_storage
 from cortexgrid.model_storage import save_model as _save_model_storage
 from cortexgrid.model_serving import (
     Deployment,
     ModelDeployFailed,
+    ModelRequirements,
     ServingStatus,
     deploy_model,
     list_deployed_models,
@@ -102,8 +113,11 @@ def remote(
     num_cpus: int = 1,
     retry: bool = False,
     **kwargs: Any,
-) -> str:
-    """Submit a function to the control plane. Returns a job ID."""
+) -> JobFuture:
+    """Submit a function to the control plane. Returns a handle on the job.
+
+    The call does not block: the returned `JobFuture` carries the job id and
+    offers `status()`, `done()` and `result()`. Only `result()` waits."""
     experiment = Experiment.get_instance()
     return schedule_remote_job(
         experiment.experiment_name,
@@ -117,11 +131,27 @@ def remote(
     )
 
 
+def get_job_result(job_id: str, timeout: float | None = None) -> Any:
+    """Block until the job finishes, then return what its function returned.
+
+    For a job of the current Experiment's run, known only by its id — the
+    handle `remote` returned may live in another process. Raises whatever the
+    job's function raised, so the call reads like a local one; see
+    `wait_for_job_result` for the other errors and for `timeout`."""
+    experiment = Experiment.get_instance()
+    return wait_for_job_result(experiment.run_id, job_id, timeout)
+
+
 def save_model(
-    weights_dir: str | Path, serve_app: type, family: str, suffix: str
+    weights_dir: str | Path,
+    serve_app: type,
+    family: str,
+    suffix: str,
+    requirements: ModelRequirements | None = None,
 ) -> SavedModel:
     """Persist a weights directory under the current Experiment's run, paired
-    with the serve-app class that will front it at deploy time.
+    with the serve-app class that will front it at deploy time and the
+    hardware one replica of it needs.
 
     Every run saves a new copy under its own run_name - meant for weights the
     run produced (e.g. a fine-tune). For a model produced elsewhere that should
@@ -134,6 +164,7 @@ def save_model(
         family,
         run_id=experiment.run_id,
         run_name=experiment.run_name(),
+        requirements=requirements,
     )
 
 
@@ -142,6 +173,7 @@ def import_model(
     serve_app: type,
     family: str,
     suffix: str,
+    requirements: ModelRequirements | None = None,
 ) -> SavedModel:
     """Register a model produced elsewhere once, reuse it on every later call,
     and record on the current Experiment's run which imported model it used.
@@ -151,7 +183,7 @@ def import_model(
     `imported_model/<family>/<suffix>` holds the version's `created_at`, set
     whether this call uploaded the model or reused it."""
     experiment = Experiment.get_instance()
-    model = _import_model_storage(source, serve_app, family, suffix)
+    model = _import_model_storage(source, serve_app, family, suffix, requirements)
     get_mlflow_client().set_tag(
         experiment.run_id, f"imported_model/{family}/{suffix}", model.created_at
     )
@@ -164,11 +196,17 @@ __all__ = [
     "delete_run",
     # Ray / jobs
     "remote",
+    "get_job_result",
+    "wait_for_job_result",
     "get_ray_job_status",
     "list_experiment_run_jobs",
     "stop_experiment_run_jobs",
     "JobStatus",
+    "JobFailed",
+    "JobFuture",
     "JobLifecycle",
+    "JobResult",
+    "JobResultUnavailable",
     "LifecycleEvent",
     "Payload",
     "get_ray_status",
@@ -213,6 +251,8 @@ __all__ = [
     "load_model",
     "list_models",
     "model_registry_status",
+    "ModelRequirements",
+    "set_model_requirements",
     "delete_model",
     # Model serving
     "Deployment",

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/cortexgrid/_ray_job_driver.py

@@ -9,11 +9,23 @@ from pathlib import Path
 
 from cortexgrid.checkpoint import set_cortexgrid_job_id
 from cortexgrid.experiment import Experiment
-from cortexgrid.jobs import Payload
+from cortexgrid.jobs import JobResult, Payload
 
 log = logging.getLogger("ray-job-driver")
 
 
+def record(result: JobResult) -> None:
+    """Persist the job's outcome beside its payload.
+
+    Best effort: a job that ran must not be reported as failed because its
+    result could not be uploaded. A caller waiting on the result gets
+    JobResultUnavailable instead, and the reason is in this log."""
+    try:
+        result.save_to_mlflow()
+    except Exception:
+        log.exception("Failed to record result for job %s", result.job_id)
+
+
 def main(payload_path: str) -> None:
     logging.basicConfig(
         level=logging.INFO,
@@ -41,7 +53,12 @@ def main(payload_path: str) -> None:
         payload.args,
         payload.kwargs,
     )
-    payload.fn(*payload.args, **payload.kwargs)
+    try:
+        value = payload.fn(*payload.args, **payload.kwargs)
+    except BaseException as exc:
+        record(JobResult.from_exception(payload, exc))
+        raise
+    record(JobResult.from_value(payload, value))
 
 
 if __name__ == "__main__":

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/cortexgrid/_serve_entry.py

@@ -4,9 +4,9 @@ Ray Serve's REST `import_path` resolves to `cortexgrid._serve_entry:build`.
 On the cluster replica, `build` imports the serve-app class bundled at
 `save_model` time (its import path was stored as an MLflow tag), applies Ray's
 ingress with the app it was marked with by `cortexgrid.serve.ingress` (again on
-each replica, see `_IngressOnReplica`), reads its
-`num_gpus`/`num_replicas` class attributes for actor placement, wraps it as a
-Ray Serve deployment, and binds it with the (family, suffix, run_name)
+each replica, see `_IngressOnReplica`), wraps it as a Ray Serve deployment
+with the replica count and Ray resource requests `deploy_model` derived from
+the model's requirements, and binds it with the (family, suffix, run_name)
 identifiers.
 
 The serve-app owns everything about traffic: its own routes, request schemas,
@@ -14,11 +14,9 @@ streaming, and timeouts. cortexgrid does not interpose a request/response
 contract - it only schedules the app and hands it the identifiers it needs to
 fetch its own weights via `cortexgrid.load_model`.
 
-Design note: resource needs (`num_gpus`/`num_replicas`) are read from plain
-class attributes rather than a cortexgrid decorator or base class. This is a
-deliberate, provisional choice - kept minimal until we see how serve-apps
-declare resources in practice; revisit if plain class attributes prove too
-limited.
+The serve-app declares no resources: the hardware a replica needs belongs to
+the model and is stored in the registry (`cortexgrid.ModelRequirements`), and
+the replica count is chosen per `deploy_model`.
 """
 
 from __future__ import annotations
@@ -72,12 +70,13 @@ def build(args: dict[str, Any]) -> Application:
             {"_serve_app": serve_app},
         )
         serve_app = serve.ingress(app)(on_replica)
-    num_gpus = getattr(serve_app, "num_gpus", 0)
-    num_replicas = getattr(serve_app, "num_replicas", 1)
     return serve.deployment(serve_app).options(
-        num_replicas=num_replicas,
+        # This builder ships in the bundle, frozen at save time, while `args`
+        # come from the cortexgrid that deploys it; one older than the bundle
+        # sends neither key.
+        num_replicas=args.get("num_replicas", 1),
         # Ray 2.32 lowered the default from 100 to 5; keep what serve-apps
         # had on Ray 2.9.
         max_ongoing_requests=_MAX_ONGOING_REQUESTS,
-        ray_actor_options={"num_gpus": num_gpus},
+        ray_actor_options=args.get("ray_actor_options", {}),
     ).bind(args["family"], args["suffix"], args["run_name"])

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/cortexgrid/jobs.py

@@ -12,18 +12,39 @@ from pathlib import Path
 import sys
 import tarfile
 import tempfile
+import time
+import traceback
 from typing import Any, Callable
 
 from cortexgrid import s3_util
 from cortexgrid._bundle import bundle, stage, worker_provides
 from cortexgrid.infra import get_mlflow_tracking_uri
-from cortexgrid.ray_util import get_ray_job_id_for_cortexgrid_job
+from cortexgrid.ray_util import (
+    JobStatus,
+    get_ray_job_id_for_cortexgrid_job,
+    get_ray_job_status,
+)
 from haikunator import Haikunator  # type: ignore
 from mlflow.tracking import MlflowClient
 from pydantic import BaseModel, ConfigDict
 
 log = logging.getLogger(__name__)
 
+_JOB_POLL_INTERVAL_S = 5.0
+_TERMINAL_JOB_STATES = (JobStatus.FINISHED, JobStatus.FAILED, JobStatus.STOPPED)
+
+
+class JobFailed(RuntimeError):
+    """A job did not produce a return value: its function raised, or the job
+    ended in a terminal state before the function ever returned. Carries the
+    remote traceback when there is one."""
+
+
+class JobResultUnavailable(RuntimeError):
+    """The job's function returned, but its value cannot be handed back here:
+    it did not survive cloudpickle on the cluster, cannot be unpickled by this
+    process, or was never recorded."""
+
 
 @dataclass
 class LifecycleEvent:
@@ -218,6 +239,215 @@ class Payload(BaseModel):
         return payload
 
 
+def _try_dumps(obj: Any) -> tuple[bytes | None, str | None]:
+    """cloudpickle ``obj``, or report why it could not be pickled.
+
+    Returns ``(blob, None)`` on success and ``(None, reason)`` on failure.
+    Nothing a job produces is allowed to turn into a job failure, so every
+    pickling error is captured rather than raised."""
+    try:
+        return cloudpickle.dumps(obj), None
+    except Exception as exc:
+        return None, f"{type(obj).__name__} did not survive cloudpickle: {exc!r}"
+
+
+@dataclass
+class JobResult:
+    """What a job's function returned or raised, recorded by the driver.
+
+    Written to ``job/{job_id}/result.pkl``, beside the payload manifest and
+    the lifecycle. The value and the exception are pickled separately from
+    the envelope so an object that cannot be pickled costs only its own
+    field: a job whose return value does not survive cloudpickle still
+    finishes, and the reason is still readable here.
+    """
+
+    experiment_name: str
+    run_id: str
+    job_id: str
+    ok: bool
+    value_pickle: bytes | None = None  # cloudpickled return value
+    value_error: str | None = None  # why value_pickle is None
+    exception_pickle: bytes | None = None  # cloudpickled exception, when picklable
+    traceback: str | None = None  # formatted remote traceback
+
+    @classmethod
+    def from_value(cls, payload: "Payload", value: Any) -> "JobResult":
+        blob, error = _try_dumps(value)
+        return cls(
+            experiment_name=payload.experiment_name,
+            run_id=payload.run_id,
+            job_id=payload.job_id,
+            ok=True,
+            value_pickle=blob,
+            value_error=error,
+        )
+
+    @classmethod
+    def from_exception(cls, payload: "Payload", exc: BaseException) -> "JobResult":
+        blob, _ = _try_dumps(exc)
+        return cls(
+            experiment_name=payload.experiment_name,
+            run_id=payload.run_id,
+            job_id=payload.job_id,
+            ok=False,
+            exception_pickle=blob,
+            traceback="".join(traceback.format_exception(exc)),
+        )
+
+    def save_to_mlflow(self) -> None:
+        artifact_path = f"job/{self.job_id}"
+        client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())
+        log.info("Recording result for job %s (ok=%s)", self.job_id, self.ok)
+        with tempfile.TemporaryDirectory() as tmp_dir:
+            local_path = Path(tmp_dir, "result.pkl")
+            local_path.write_bytes(cloudpickle.dumps(self))
+            client.log_artifact(
+                self.run_id, str(local_path), artifact_path=artifact_path
+            )
+
+    @classmethod
+    def load_from_mlflow(cls, run_id: str, job_id: str) -> "JobResult":
+        client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())
+        result_rel = f"job/{job_id}/result.pkl"
+        if not any(
+            a.path == result_rel for a in client.list_artifacts(run_id, f"job/{job_id}")
+        ):
+            raise FileNotFoundError(f"artifact {result_rel} not found in run {run_id}")
+        local_path = client.download_artifacts(run_id, result_rel)
+        result = cloudpickle.loads(Path(local_path).read_bytes())
+        if not isinstance(result, cls):
+            raise TypeError(f"{result_rel} in run {run_id} is not a JobResult")
+        return result
+
+    def unwrap(self) -> Any:
+        """Return the value the job's function returned, or raise what it raised.
+
+        A failed job raises the original exception, rebuilt from its pickle, so
+        a blocking caller sees what a local call would have raised; the remote
+        traceback rides along as the chained ``JobFailed`` cause. When the
+        exception itself did not survive pickling, ``JobFailed`` is raised
+        instead.
+        """
+        if not self.ok:
+            failure = JobFailed(f"job {self.job_id} failed:\n{self.traceback}")
+            remote_exc = self._unpickled_exception()
+            if remote_exc is None:
+                raise failure
+            raise remote_exc from failure
+        if self.value_pickle is None:
+            raise JobResultUnavailable(
+                f"job {self.job_id} finished, but its return value was not "
+                f"recorded: {self.value_error}"
+            )
+        try:
+            return cloudpickle.loads(self.value_pickle)
+        except Exception as exc:
+            raise JobResultUnavailable(
+                f"job {self.job_id} finished, but its return value could not be "
+                f"unpickled here: {exc!r}"
+            ) from exc
+
+    def _unpickled_exception(self) -> BaseException | None:
+        """The remote exception object, or None when it cannot be rebuilt here."""
+        if self.exception_pickle is None:
+            return None
+        try:
+            exc = cloudpickle.loads(self.exception_pickle)
+        except Exception:
+            log.warning("Job %s: remote exception could not be unpickled", self.job_id)
+            return None
+        return exc if isinstance(exc, BaseException) else None
+
+
+def _missing_result(
+    lifecycle: JobLifecycle, ray_job_id: str | None, status: JobStatus
+) -> Exception:
+    """The error for a job that reached ``status`` leaving no result behind.
+
+    Two ways in: the job's function never ran (the driver died, or the control
+    plane could not hand the job to Ray), or it ran and the upload of its
+    result failed. Either way the detail lives in the Ray logs, and a
+    submission error lives on the lifecycle."""
+    error = next((e.error for e in reversed(lifecycle.history) if e.error), None)
+    detail = f": {error}" if error else f" (ray job {ray_job_id}; check its logs)"
+    if status == JobStatus.FINISHED:
+        return JobResultUnavailable(
+            f"job {lifecycle.job_id} finished but recorded no result{detail}"
+        )
+    return JobFailed(
+        f"job {lifecycle.job_id} ended in {status.value} before its function "
+        f"returned{detail}"
+    )
+
+
+def wait_for_job_result(run_id: str, job_id: str, timeout: float | None = None) -> Any:
+    """Block until the job reaches a terminal Ray state, then return what its
+    function returned — or raise what it raised, as a local call would.
+
+    Raises ``ValueError`` for a job submitted with ``retry=True``: retries are
+    unbounded by design, so a blocking wait would sit through however many
+    resubmissions the control plane makes. Raises ``TimeoutError`` once a
+    finite ``timeout`` elapses (``timeout=0`` polls once and gives up);
+    ``timeout=None`` has no deadline. Raises ``JobFailed`` if the job never
+    got as far as returning, and ``JobResultUnavailable`` if it returned a
+    value that cannot be handed back here.
+    """
+    deadline = None if timeout is None else time.monotonic() + timeout
+    while True:
+        lifecycle = JobLifecycle.load_from_mlflow(run_id, job_id)
+        if lifecycle.retry:
+            raise ValueError(
+                f"job {job_id} was submitted with retry=True, which resubmits "
+                "without bound; waiting on its result is not supported. Poll "
+                "list_experiment_run_jobs instead."
+            )
+        ray_job_id = lifecycle.get_ray_job_id()
+        status = get_ray_job_status(ray_job_id)
+        if status in _TERMINAL_JOB_STATES:
+            try:
+                result = JobResult.load_from_mlflow(run_id, job_id)
+            except FileNotFoundError:
+                raise _missing_result(lifecycle, ray_job_id, status) from None
+            return result.unwrap()
+        if deadline is not None and time.monotonic() >= deadline:
+            raise TimeoutError(
+                f"job {job_id} did not finish within {timeout}s "
+                f"(last status={status.value})"
+            )
+        time.sleep(_JOB_POLL_INTERVAL_S)
+
+
+@dataclass(frozen=True)
+class JobFuture:
+    """A handle on a submitted job: its identity, its live status, and its result.
+
+    ``schedule_remote_job`` returns one as soon as the job request is recorded.
+    Nothing here blocks until :meth:`result` is called.
+    """
+
+    experiment_name: str
+    run_id: str
+    job_id: str
+
+    def status(self) -> JobStatus:
+        """The job's live status, derived from Ray. ``PENDING`` covers both
+        'the control plane has not picked it up yet' and 'queued in Ray'."""
+        lifecycle = JobLifecycle.load_from_mlflow(self.run_id, self.job_id)
+        return get_ray_job_status(lifecycle.get_ray_job_id())
+
+    def done(self) -> bool:
+        """True once Ray reports the job finished, failed or stopped."""
+        return self.status() in _TERMINAL_JOB_STATES
+
+    def result(self, timeout: float | None = None) -> Any:
+        """Block until the job finishes and return its function's value.
+
+        Raises exactly what :func:`wait_for_job_result` raises — including the
+        job's own exception, so a blocking call reads like a local one."""
+        return wait_for_job_result(self.run_id, self.job_id, timeout)
+
+
 def schedule_remote_job(
     experiment_name: str,
     run_id: str,
@@ -227,8 +457,8 @@ def schedule_remote_job(
     num_cpus: int = 1,
     retry: bool = False,
     **kwargs: Any,
-) -> str:
-    """Submit a function to the control plane. Returns a job ID."""
+) -> JobFuture:
+    """Submit a function to the control plane. Returns a handle on the job."""
     job_id = Haikunator().haikunate(token_length=2, token_chars="0123456789")
     entry_file = Path(inspect.getfile(fn)).resolve()
     driver_file = Path(__file__).with_name("_ray_job_driver.py")
@@ -263,7 +493,7 @@ def schedule_remote_job(
             num_cpus=num_cpus,
             pip_requirements=pip_requirements,
         ).save_to_mlflow()
-    return job_id
+    return JobFuture(experiment_name=experiment_name, run_id=run_id, job_id=job_id)
 
 
 def list_experiment_run_jobs(run_id: str) -> list[JobLifecycle]:

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/cortexgrid/model_serving.py

@@ -8,7 +8,8 @@ uploaded to MinIO under
 referenced via `runtime_env.working_dir` so Ray workers fetch it from there.
 The bundle URL, class import path, pip list, and fingerprint are persisted as
 MLflow tags on the ModelVersion so `deploy_model` can find them later without
-the caller holding the class object.
+the caller holding the class object. So are the model's `ModelRequirements`,
+which `deploy_model` turns into the replica's Ray resource requests.
 
 Naming: the Ray Serve application is named "<family>__<suffix>__<run_name>".
 This relies on family/suffix/run_name not containing the literal "__".
@@ -206,10 +207,41 @@ def upload_bundle(
     )
 
 
+# Custom Ray resource each GPU worker advertises: the MiB of memory its GPUs
+# have (see the ray-worker DaemonSet). A replica requests its vram_gb of it in
+# MiB, so Ray places it only on a node with that much VRAM left. MiB because
+# nvidia-smi reports MiB and a GPU's memory is not a whole number of GiB.
+_VRAM_RESOURCE = "vram_mib"
+
+_MIB_PER_GIB = 1024
+
+_GIB = 1024**3
+
+
+def _ray_actor_options(requirements: ModelRequirements) -> dict[str, Any]:
+    """Translate ModelRequirements into a replica's Ray actor resource requests.
+    Ray places the replica only on a node with that much free and reserves it
+    there; a zero requirement requests nothing."""
+    options: dict[str, Any] = {"num_gpus": requirements.num_gpus}
+    if requirements.ram_gb > 0:
+        options["memory"] = int(requirements.ram_gb * _GIB)
+    if requirements.vram_gb > 0:
+        options["resources"] = {
+            _VRAM_RESOURCE: round(requirements.vram_gb * _MIB_PER_GIB)
+        }
+    return options
+
+
 def _build_application_spec(
-    family: str, suffix: str, run_name: str, meta: BundleMetadata
+    family: str,
+    suffix: str,
+    run_name: str,
+    meta: BundleMetadata,
+    requirements: ModelRequirements,
+    num_replicas: int,
 ) -> dict[str, Any]:
-    """Assemble a Ray Serve application schema from pre-bundled metadata."""
+    """Assemble a Ray Serve application schema from pre-bundled metadata and
+    the model's requirements."""
     # working_dir carries the serve-app's own source; Ray pip-installs the
     # third-party distributions the image lacks into a per-node cached
     # virtualenv layered on the image. No pip key when there are none, so Ray
@@ -230,6 +262,8 @@ def _build_application_spec(
             "family": family,
             "suffix": suffix,
             "run_name": run_name,
+            "num_replicas": num_replicas,
+            "ray_actor_options": _ray_actor_options(requirements),
         },
         "runtime_env": runtime_env,
     }
@@ -268,10 +302,66 @@ def metadata_from_tags(tags: dict[str, str]) -> BundleMetadata:
     )
 
 
-def _load_bundle_metadata(
+@dataclass
+class ModelRequirements:
+    """Hardware one replica of a model needs to be served, in GiB. Persisted as
+    tags on the ModelVersion next to the bundle metadata, so it is read without
+    touching the weights or importing the serve-app class.
+
+    Zero means no requirement: a model with no requirements is served on any
+    node, CPU-only included. Models saved before requirements existed carry
+    no tags and read as the defaults."""
+
+    num_gpus: int = 0
+    ram_gb: float = 0.0
+    # GPU memory across the replica's num_gpus GPUs, so it needs num_gpus >= 1.
+    vram_gb: float = 0.0
+
+    def __post_init__(self) -> None:
+        if self.num_gpus < 0 or self.ram_gb < 0 or self.vram_gb < 0:
+            raise ValueError(f"Model requirements cannot be negative: {self}")
+        if self.vram_gb > 0 and self.num_gpus == 0:
+            raise ValueError(
+                f"vram_gb={self.vram_gb} needs a GPU; set num_gpus >= 1"
+            )
+
+
+# MLflow tag keys for the ModelRequirements.
+_NUM_GPUS_TAG = "num_gpus"
+_RAM_GB_TAG = "ram_gb"
+_VRAM_GB_TAG = "vram_gb"
+
+
+def has_requirement_tags(tags: dict[str, str]) -> bool:
+    """Whether requirements were ever stored on the ModelVersion."""
+    return any(key in tags for key in (_NUM_GPUS_TAG, _RAM_GB_TAG, _VRAM_GB_TAG))
+
+
+def requirements_to_tags(requirements: ModelRequirements) -> dict[str, str]:
+    """Serialise ModelRequirements to MLflow tags. The inverse of
+    `requirements_from_tags`."""
+    return {
+        _NUM_GPUS_TAG: str(requirements.num_gpus),
+        _RAM_GB_TAG: str(requirements.ram_gb),
+        _VRAM_GB_TAG: str(requirements.vram_gb),
+    }
+
+
+def requirements_from_tags(tags: dict[str, str]) -> ModelRequirements:
+    """Deserialise ModelRequirements from a ModelVersion's MLflow tags; a
+    missing tag reads as no requirement."""
+    return ModelRequirements(
+        num_gpus=int(tags.get(_NUM_GPUS_TAG, "0")),
+        ram_gb=float(tags.get(_RAM_GB_TAG, "0")),
+        vram_gb=float(tags.get(_VRAM_GB_TAG, "0")),
+    )
+
+
+def _load_deploy_metadata(
     family: str, suffix: str, run_name: str
-) -> BundleMetadata:
-    """Read the bundle metadata `save_model` persisted on the ModelVersion."""
+) -> tuple[BundleMetadata, ModelRequirements]:
+    """Read the bundle metadata and requirements `save_model` persisted on the
+    ModelVersion."""
     client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())
     name = f"{family}__{suffix}"
     versions = client.search_model_versions(
@@ -281,8 +371,9 @@ def _load_bundle_metadata(
         raise ValueError(
             f"No saved model for {family}/{suffix}/{run_name}; cannot deploy."
         )
+    tags = versions[0].tags or {}
     try:
-        return metadata_from_tags(versions[0].tags or {})
+        return metadata_from_tags(tags), requirements_from_tags(tags)
     except KeyError as exc:
         raise ValueError(
             f"Saved model {family}/{suffix}/{run_name} is missing the deployment "
@@ -398,6 +489,7 @@ def deploy_model(
     family: str,
     suffix: str,
     run_name: str,
+    num_replicas: int = 1,
     wait: bool = False,
     timeout: float | None = 300.0,
 ) -> Deployment:
@@ -408,6 +500,8 @@ def deploy_model(
 
     The serve-app class is pulled from the MLflow ModelVersion tags `save_model`
     wrote at save time; the caller does not need to hold the class object.
+    Each of the `num_replicas` replicas requests the model's `ModelRequirements`
+    from Ray, so it is placed only on a node that has them free.
 
     A DEPLOY_FAILED app left by an earlier attempt is undeployed first, and it,
     or an app still DELETING, is waited out before the new spec is PUT, so the
@@ -421,8 +515,10 @@ def deploy_model(
     stuck in DEPLOYING) will hang forever.
     """
     deadline = _deadline(timeout)
-    meta = _load_bundle_metadata(family, suffix, run_name)
-    spec = _build_application_spec(family, suffix, run_name, meta)
+    meta, requirements = _load_deploy_metadata(family, suffix, run_name)
+    spec = _build_application_spec(
+        family, suffix, run_name, meta, requirements, num_replicas
+    )
     _clear_failed_application(family, suffix, run_name, timeout, deadline)
     existing = [a for a in _current_application_specs() if a["name"] != spec["name"]]
     # The controller registers the app, sets it DEPLOYING and stamps

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/cortexgrid/model_storage.py

@@ -8,6 +8,7 @@ Mapping cortexgrid taxonomy <-> MLflow Registry:
                            "s3://<bucket>/models/<run_name>/<family>/<suffix>/weights/"
     run linkage         -> ModelVersion.run_id  (built-in MLflow field; unset
                            for imported models)
+    requirements        -> ModelVersion.tags["num_gpus"], ["ram_gb"], ["vram_gb"]
 
 Two ways in: `save_model` registers a fresh copy under the calling run's
 run_name every time it runs (fine-tuned output); `import_model` registers a
@@ -36,10 +37,14 @@ from mlflow.tracking import MlflowClient
 from cortexgrid import s3_util
 from cortexgrid.infra import get_mlflow_tracking_uri, get_s3_bucket
 from cortexgrid.model_serving import (
+    ModelRequirements,
     build_bundle,
     bundle_class,
+    has_requirement_tags,
     metadata_from_tags,
     metadata_to_tags,
+    requirements_from_tags,
+    requirements_to_tags,
     upload_bundle,
 )
 
@@ -81,6 +86,8 @@ class SavedModel:
     # _UPLOAD_DEADLINE (writer presumed dead). Versions written before this tag
     # existed report "ready".
     phase: str
+    # Hardware one replica needs; defaults for versions stored without it.
+    requirements: ModelRequirements
 
 
 def _phase_for(version: Any) -> str:
@@ -111,6 +118,7 @@ def _to_saved_model(version: Any) -> SavedModel:
         data_blob_path=version.source,
         size_bytes=int(version.tags.get("size_bytes", "0")),
         phase=_phase_for(version),
+        requirements=requirements_from_tags(version.tags),
     )
 
 
@@ -150,6 +158,7 @@ def save_model(
     family: str,
     run_id: str,
     run_name: str,
+    requirements: ModelRequirements | None = None,
 ) -> SavedModel:
     """Upload a weights directory to S3 and register a new MLflow ModelVersion
     paired with the serve-app that fronts it.
@@ -168,8 +177,13 @@ def save_model(
     weights.
     Its code is bundled and its import path, bundle URL, and pip list are
     stored as tags on the ModelVersion so `deploy_model` can bind it later
-    without the caller holding the class object."""
-    return _upload_model(weights_dir, serve_app, suffix, family, run_id, run_name)
+    without the caller holding the class object.
+
+    `requirements` is the hardware one replica needs; None stores none, which
+    reads as no requirement. Change it later with `set_model_requirements`."""
+    return _upload_model(
+        weights_dir, serve_app, suffix, family, run_id, run_name, requirements
+    )
 
 
 def import_model(
@@ -177,6 +191,7 @@ def import_model(
     serve_app: type,
     family: str,
     suffix: str,
+    requirements: ModelRequirements | None = None,
 ) -> SavedModel:
     """Register a model produced elsewhere (e.g. a pretrained base model) under
     the fixed key (family, suffix, IMPORTED), once.
@@ -190,7 +205,9 @@ def import_model(
     If a version is already registered under the key:
       - "ready": returns it without calling `source`. If `serve_app`'s code no
         longer matches the stored bundle, it is re-bundled first and the
-        weights are kept (see `_refresh_bundle`). To replace the weights,
+        weights are kept (see `_refresh_bundle`). `requirements` are stored
+        only if the version has none yet, so values changed since with
+        `set_model_requirements` are kept. To replace the weights,
         `delete_model` it first.
       - "uploading": raises RuntimeError - another process is importing it.
       - "upload_failed" / "broken": deleted and imported again.
@@ -202,6 +219,10 @@ def import_model(
     if existing is not None:
         if existing.phase == _PHASE_READY:
             _refresh_bundle(serve_app, family, suffix)
+            if requirements is not None:
+                existing.requirements = _set_missing_requirements(
+                    family, suffix, requirements
+                )
             return existing
         if existing.phase == _PHASE_UPLOADING:
             raise RuntimeError(
@@ -209,7 +230,9 @@ def import_model(
                 "another process"
             )
         delete_model(family, suffix, IMPORTED)
-    return _upload_model(source, serve_app, suffix, family, None, IMPORTED)
+    return _upload_model(
+        source, serve_app, suffix, family, None, IMPORTED, requirements
+    )
 
 
 def _refresh_bundle(serve_app: type, family: str, suffix: str) -> None:
@@ -233,6 +256,23 @@ def _refresh_bundle(serve_app: type, family: str, suffix: str) -> None:
         client.set_model_version_tag(name, version.version, key, value)
 
 
+def _set_missing_requirements(
+    family: str, suffix: str, requirements: ModelRequirements
+) -> ModelRequirements:
+    """Store `requirements` on an imported model whose version has none yet.
+    Returns the requirements the version holds afterwards."""
+    name = f"{family}__{suffix}"
+    client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())
+    version = client.search_model_versions(
+        f"name='{name}' and tags.run_name='{IMPORTED}'"
+    )[0]
+    if has_requirement_tags(version.tags):
+        return requirements_from_tags(version.tags)
+    for key, value in requirements_to_tags(requirements).items():
+        client.set_model_version_tag(name, version.version, key, value)
+    return requirements
+
+
 def _upload_model(
     weights: str | Path | Callable[[], str | Path],
     serve_app: type,
@@ -240,6 +280,7 @@ def _upload_model(
     family: str,
     run_id: str | None,
     run_name: str,
+    requirements: ModelRequirements | None,
 ) -> SavedModel:
     """Register a ModelVersion in "uploading", resolve `weights` to a directory
     (calling it when it is a callable), upload the weights and the serve-app
@@ -255,7 +296,8 @@ def _upload_model(
     # fetched and uploaded, and a concurrent `import_model` sees the import in
     # flight for the whole download instead of starting one of its own. The
     # size is stamped once the directory exists; the bundle tags and the flip
-    # to "ready" happen only after the upload lands.
+    # to "ready" happen only after the upload lands. No requirements leaves
+    # their tags unset, so a later `import_model` can still store them.
     version = client.create_model_version(
         name=name,
         source=source,
@@ -265,6 +307,7 @@ def _upload_model(
             "suffix": suffix,
             "run_name": run_name,
             _LIFECYCLE_TAG: _PHASE_UPLOADING,
+            **(requirements_to_tags(requirements) if requirements is not None else {}),
         },
     )
     try:
@@ -333,6 +376,23 @@ def model_registry_status(
     return _to_saved_model(versions[0]) if versions else None
 
 
+def set_model_requirements(
+    family: str, suffix: str, run_name: str, requirements: ModelRequirements
+) -> None:
+    """Replace the hardware requirements stored on a model. Takes effect on
+    its next `deploy_model`; a replica already running keeps its placement.
+    Raises ValueError if the model was never registered."""
+    client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())
+    name = f"{family}__{suffix}"
+    versions = client.search_model_versions(
+        f"name='{name}' and tags.run_name='{run_name}'"
+    )
+    if not versions:
+        raise ValueError(f"No model {family}/{suffix}/{run_name}")
+    for key, value in requirements_to_tags(requirements).items():
+        client.set_model_version_tag(name, versions[0].version, key, value)
+
+
 def delete_model(family: str, suffix: str, run_name: str) -> None:
     """Delete the ModelVersion in MLflow, its weights blob, and its serve bundle."""
     client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/docs/cortexgrid/README.md

@@ -79,11 +79,61 @@ def train_step(batch):
     # MLflow and S3 env vars are injected automatically
     return loss
 
-job_id = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
-print(f"Submitted: {job_id}")
+job = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
+print(f"Submitted: {job.job_id}")
 ```
 
-`cortexgrid.remote` submits a job *request* (a pickled payload plus a `JobLifecycle` record) to MLflow and returns a job ID string immediately. It does not wait for the job to run or finish — use the UI at `http://<DGX_IP>:8000`, or poll `cortexgrid.list_experiment_run_jobs(run_id)`, to observe status.
+`cortexgrid.remote` submits a job *request* (a pickled payload plus a `JobLifecycle` record) to MLflow and returns a `JobFuture` immediately. It does not wait for the job to run or finish — use the UI at `http://<DGX_IP>:8000`, `job.status()`, or poll `cortexgrid.list_experiment_run_jobs(run_id)`, to observe status.
+
+##### Blocking on the result
+
+`JobFuture.result(timeout=None)` blocks until Ray reports the job terminal, then behaves like a local call: it returns whatever the function returned, or raises whatever the function raised.
+
+```python
+def score(batch) -> dict[str, float]:
+    # runs on the DGX
+    return {"loss": 0.12}
+
+# Fire-and-forget — the form training uses. Nothing blocks, and a retry=True
+# job cannot be waited on at all (see below).
+training = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
+print(training.job_id, training.status().value)   # "blue-42 running"
+
+# Blocking — the value comes back as if score() had run locally.
+job = cortexgrid.remote(score, batch, num_gpus=1)
+metrics = job.result()                            # {"loss": 0.12}
+
+# ...or with a deadline. On TimeoutError the job keeps running; wait again later.
+metrics = job.result(timeout=600)
+
+# From another process — the handle is gone, the job id is enough.
+metrics = cortexgrid.get_job_result(job_id, timeout=600)
+```
+
+Failures arrive as exceptions, not as a status to inspect:
+
+```python
+try:
+    metrics = job.result()
+except BadBatch as exc:                # exactly what score() raised on the DGX
+    print(exc.__cause__)               # JobFailed, carrying the remote traceback
+except cortexgrid.JobFailed:           # the job never got as far as returning
+    ...                                # submission failed, driver died, or stopped
+except cortexgrid.JobResultUnavailable:
+    ...                                # it returned, but the value is not transportable
+```
+
+The driver cloudpickles the outcome to `job/{job_id}/result.pkl`, beside the payload manifest and the lifecycle, and the waiting side reads it back:
+
+| Outcome | What `result()` does |
+|---|---|
+| The function returned | Returns its value |
+| The function raised | Raises that exception, with the remote traceback attached as a chained `JobFailed` cause |
+| The job never got as far as returning (submission failed, driver died, job stopped) | Raises `JobFailed` |
+| The value or the exception did not survive cloudpickle | Raises `JobResultUnavailable` (value) or `JobFailed` (exception). A value that cannot be pickled never fails the job itself |
+| `timeout` elapsed | Raises `TimeoutError`; the job keeps running |
+
+Waiting on a `retry=True` job raises `ValueError`: retries are unbounded by design (see below), so the wait would have no end. Fire-and-forget submission is the form training uses — submit, then watch the UI.
 
 A separate service — the **jobs control plane** — polls MLflow for pending job requests, matches them against the set of Ray submissions the cluster already has, and submits anything missing. It is also responsible for retrying failed jobs and honouring user-requested stops.
 
@@ -137,19 +187,23 @@ app = FastAPI()
 
 @serve.ingress(app)
 class MyServeApp:
-    num_gpus = 1
-
     def __init__(self, family: str, suffix: str, run_name: str) -> None:
         self._weights_dir = cortexgrid.load_model(family, suffix, run_name)
 
     @app.post("/complete")
     async def complete(self, body: dict): ...
 
-saved = cortexgrid.save_model(weights_dir, MyServeApp, family="qwen", suffix="instruct")
+saved = cortexgrid.save_model(
+    weights_dir, MyServeApp, family="qwen", suffix="instruct",
+    # What one replica needs; the model is deployed only on a host that has it.
+    requirements=cortexgrid.ModelRequirements(num_gpus=1, ram_gb=8, vram_gb=16),
+)
 deployed = cortexgrid.deploy_model("qwen", "instruct", saved.run_name, wait=True)
 print(deployed.url)
 ```
 
+The requirements are part of the model, not of the serve-app class: GPUs, RAM and VRAM (GiB, 0 meaning no requirement) are stored with it and matched against what the cluster's hosts have free. Correct them later with `cortexgrid.set_model_requirements(family, suffix, run_name, requirements)` or on the model card in the dashboard; `cortexgrid.deploy_model(..., num_replicas=2)` chooses how many copies to run.
+
 `save_model` saves a new copy under every run - meant for weights the run produced (e.g. a fine-tune). For a model produced elsewhere (e.g. a pretrained base model), `cortexgrid.import_model(source, MyServeApp, family, suffix)` uploads it once under `run_name=cortexgrid.IMPORTED` and on later runs only re-bundles `MyServeApp` if its code changed; deploy it with `deploy_model(family, suffix, cortexgrid.IMPORTED)`.
 
 `save_model` is synchronous (registry lifecycle: `uploading` -> `ready`); `deploy_model` schedules the serving lifecycle (`deploying` -> `running`). With `wait=True` a failed deploy raises `cortexgrid.ModelDeployFailed`; `cortexgrid.wait_for_model_serving(family, suffix, run_name, timeout=...)` waits on a deploy started elsewhere, and re-deploying a failed model retries it from scratch. See [model-serving.md](https://github.com/robodatalab/cortexgrid/blob/main/docs/cortexgrid/model-serving.md) for both lifecycles end to end - upload/deploy/undeploy/delete, status queries (`model_registry_status`, `model_serving_status`), and error handling.
@@ -165,7 +219,9 @@ print(deployed.url)
 | `cortexgrid.log_artifact(path, artifact_path)` | Log a file as an artifact |
 | `cortexgrid.checkpoint()` | Context manager returning an attribute-based checkpoint saved to MLflow on exit |
 | `cortexgrid.resume()` | Load the latest checkpoint for the current job, or `None` |
-| `cortexgrid.remote(fn, *args, num_gpus=0, num_cpus=1, retry=False, **kwargs)` | Submit a function to the jobs control plane; returns a job ID |
+| `cortexgrid.remote(fn, *args, num_gpus=0, num_cpus=1, retry=False, **kwargs)` | Submit a function to the jobs control plane; returns a `JobFuture` |
+| `JobFuture.status()` / `.done()` / `.result(timeout=None)` | Live status of a submitted job, and its function's return value (blocking) |
+| `cortexgrid.get_job_result(job_id, timeout=None)` | Block on a job of the current run by id; returns its value or raises its exception |
 | `cortexgrid.list_experiment_run_jobs(run_id)` | List `JobLifecycle` records for every cortexgrid job in a run |
 | `cortexgrid.stop_experiment_run_jobs(run_id)` | Request every job in a run to stop (flips the `stop_requested` latch) |
 | `cortexgrid.get_ray_job_status(ray_job_id)` | Live Ray status for a submission id |

{cortexgrid-0.2.96 → cortexgrid-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "cortexgrid"
-version = "0.2.96"
+version = "0.3.0"
 description = "Connect your ML code to the RoboLab compute cluster — Ray, MLflow, and S3"
 readme = "docs/cortexgrid/README.md"
 license = "Apache-2.0"