cortexgrid 0.2.97__tar.gz → 0.3.0__tar.gz

{cortexgrid-0.2.97 → cortexgrid-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cortexgrid
-Version: 0.2.97
+Version: 0.3.0
 Summary: Connect your ML code to the RoboLab compute cluster — Ray, MLflow, and S3
 Project-URL: Homepage, https://github.com/robodatalab/cortexgrid
 Project-URL: Repository, https://github.com/robodatalab/cortexgrid
@@ -107,11 +107,61 @@ def train_step(batch):
     # MLflow and S3 env vars are injected automatically
     return loss
 
-job_id = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
-print(f"Submitted: {job_id}")
+job = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
+print(f"Submitted: {job.job_id}")
 ```
 
-`cortexgrid.remote` submits a job *request* (a pickled payload plus a `JobLifecycle` record) to MLflow and returns a job ID string immediately. It does not wait for the job to run or finish — use the UI at `http://<DGX_IP>:8000`, or poll `cortexgrid.list_experiment_run_jobs(run_id)`, to observe status.
+`cortexgrid.remote` submits a job *request* (a pickled payload plus a `JobLifecycle` record) to MLflow and returns a `JobFuture` immediately. It does not wait for the job to run or finish — use the UI at `http://<DGX_IP>:8000`, `job.status()`, or poll `cortexgrid.list_experiment_run_jobs(run_id)`, to observe status.
+
+##### Blocking on the result
+
+`JobFuture.result(timeout=None)` blocks until Ray reports the job terminal, then behaves like a local call: it returns whatever the function returned, or raises whatever the function raised.
+
+```python
+def score(batch) -> dict[str, float]:
+    # runs on the DGX
+    return {"loss": 0.12}
+
+# Fire-and-forget — the form training uses. Nothing blocks, and a retry=True
+# job cannot be waited on at all (see below).
+training = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
+print(training.job_id, training.status().value)   # "blue-42 running"
+
+# Blocking — the value comes back as if score() had run locally.
+job = cortexgrid.remote(score, batch, num_gpus=1)
+metrics = job.result()                            # {"loss": 0.12}
+
+# ...or with a deadline. On TimeoutError the job keeps running; wait again later.
+metrics = job.result(timeout=600)
+
+# From another process — the handle is gone, the job id is enough.
+metrics = cortexgrid.get_job_result(job_id, timeout=600)
+```
+
+Failures arrive as exceptions, not as a status to inspect:
+
+```python
+try:
+    metrics = job.result()
+except BadBatch as exc:                # exactly what score() raised on the DGX
+    print(exc.__cause__)               # JobFailed, carrying the remote traceback
+except cortexgrid.JobFailed:           # the job never got as far as returning
+    ...                                # submission failed, driver died, or stopped
+except cortexgrid.JobResultUnavailable:
+    ...                                # it returned, but the value is not transportable
+```
+
+The driver cloudpickles the outcome to `job/{job_id}/result.pkl`, beside the payload manifest and the lifecycle, and the waiting side reads it back:
+
+| Outcome | What `result()` does |
+|---|---|
+| The function returned | Returns its value |
+| The function raised | Raises that exception, with the remote traceback attached as a chained `JobFailed` cause |
+| The job never got as far as returning (submission failed, driver died, job stopped) | Raises `JobFailed` |
+| The value or the exception did not survive cloudpickle | Raises `JobResultUnavailable` (value) or `JobFailed` (exception). A value that cannot be pickled never fails the job itself |
+| `timeout` elapsed | Raises `TimeoutError`; the job keeps running |
+
+Waiting on a `retry=True` job raises `ValueError`: retries are unbounded by design (see below), so the wait would have no end. Fire-and-forget submission is the form training uses — submit, then watch the UI.
 
 A separate service — the **jobs control plane** — polls MLflow for pending job requests, matches them against the set of Ray submissions the cluster already has, and submits anything missing. It is also responsible for retrying failed jobs and honouring user-requested stops.
 
@@ -197,7 +247,9 @@ The requirements are part of the model, not of the serve-app class: GPUs, RAM an
 | `cortexgrid.log_artifact(path, artifact_path)` | Log a file as an artifact |
 | `cortexgrid.checkpoint()` | Context manager returning an attribute-based checkpoint saved to MLflow on exit |
 | `cortexgrid.resume()` | Load the latest checkpoint for the current job, or `None` |
-| `cortexgrid.remote(fn, *args, num_gpus=0, num_cpus=1, retry=False, **kwargs)` | Submit a function to the jobs control plane; returns a job ID |
+| `cortexgrid.remote(fn, *args, num_gpus=0, num_cpus=1, retry=False, **kwargs)` | Submit a function to the jobs control plane; returns a `JobFuture` |
+| `JobFuture.status()` / `.done()` / `.result(timeout=None)` | Live status of a submitted job, and its function's return value (blocking) |
+| `cortexgrid.get_job_result(job_id, timeout=None)` | Block on a job of the current run by id; returns its value or raises its exception |
 | `cortexgrid.list_experiment_run_jobs(run_id)` | List `JobLifecycle` records for every cortexgrid job in a run |
 | `cortexgrid.stop_experiment_run_jobs(run_id)` | Request every job in a run to stop (flips the `stop_requested` latch) |
 | `cortexgrid.get_ray_job_status(ray_job_id)` | Live Ray status for a submission id |

{cortexgrid-0.2.97 → cortexgrid-0.3.0}/cortexgrid/__init__.py

@@ -2,14 +2,18 @@
 
 import cortexgrid
 
-# Fire-and-forget training on the DGX; returns a job id immediately.
+# Fire-and-forget training on the DGX; returns a handle immediately.
 # The jobs control plane picks up the submission and dispatches it to Ray.
-job_id = cortexgrid.remote(my_train, config, num_gpus=1, retry=True)
-print(f"Submitted: {job_id}")
+training = cortexgrid.remote(my_train, config, num_gpus=1, retry=True)
+print(f"Submitted: {training.job_id} ({training.status().value})")
 
-# Check on it later
-for job in cortexgrid.list_experiment_run_jobs(run_id):
-    status = cortexgrid.get_ray_job_status(job.get_ray_job_id())
+# Or block on the function's return value, as if it had run locally.
+# The job's own exception is what a failed job raises here.
+job = cortexgrid.remote(score, batch, num_gpus=1)
+loss = job.result(timeout=600)
+
+# In another process, the job id is enough
+loss = cortexgrid.get_job_result(job_id)
 
 # Inside the training function — checkpoint after each epoch
 with cortexgrid.checkpoint() as ckpt:
@@ -39,7 +43,12 @@ from cortexgrid.jobs import (
     schedule_remote_job,
     list_experiment_run_jobs,
     stop_experiment_run_jobs,
+    wait_for_job_result,
+    JobFailed,
+    JobFuture,
     JobLifecycle,
+    JobResult,
+    JobResultUnavailable,
     LifecycleEvent,
     Payload,
 )
@@ -104,8 +113,11 @@ def remote(
     num_cpus: int = 1,
     retry: bool = False,
     **kwargs: Any,
-) -> str:
-    """Submit a function to the control plane. Returns a job ID."""
+) -> JobFuture:
+    """Submit a function to the control plane. Returns a handle on the job.
+
+    The call does not block: the returned `JobFuture` carries the job id and
+    offers `status()`, `done()` and `result()`. Only `result()` waits."""
     experiment = Experiment.get_instance()
     return schedule_remote_job(
         experiment.experiment_name,
@@ -119,6 +131,17 @@ def remote(
     )
 
 
+def get_job_result(job_id: str, timeout: float | None = None) -> Any:
+    """Block until the job finishes, then return what its function returned.
+
+    For a job of the current Experiment's run, known only by its id — the
+    handle `remote` returned may live in another process. Raises whatever the
+    job's function raised, so the call reads like a local one; see
+    `wait_for_job_result` for the other errors and for `timeout`."""
+    experiment = Experiment.get_instance()
+    return wait_for_job_result(experiment.run_id, job_id, timeout)
+
+
 def save_model(
     weights_dir: str | Path,
     serve_app: type,
@@ -173,11 +196,17 @@ __all__ = [
     "delete_run",
     # Ray / jobs
     "remote",
+    "get_job_result",
+    "wait_for_job_result",
     "get_ray_job_status",
     "list_experiment_run_jobs",
     "stop_experiment_run_jobs",
     "JobStatus",
+    "JobFailed",
+    "JobFuture",
     "JobLifecycle",
+    "JobResult",
+    "JobResultUnavailable",
     "LifecycleEvent",
     "Payload",
     "get_ray_status",

{cortexgrid-0.2.97 → cortexgrid-0.3.0}/cortexgrid/_ray_job_driver.py

@@ -9,11 +9,23 @@ from pathlib import Path
 
 from cortexgrid.checkpoint import set_cortexgrid_job_id
 from cortexgrid.experiment import Experiment
-from cortexgrid.jobs import Payload
+from cortexgrid.jobs import JobResult, Payload
 
 log = logging.getLogger("ray-job-driver")
 
 
+def record(result: JobResult) -> None:
+    """Persist the job's outcome beside its payload.
+
+    Best effort: a job that ran must not be reported as failed because its
+    result could not be uploaded. A caller waiting on the result gets
+    JobResultUnavailable instead, and the reason is in this log."""
+    try:
+        result.save_to_mlflow()
+    except Exception:
+        log.exception("Failed to record result for job %s", result.job_id)
+
+
 def main(payload_path: str) -> None:
     logging.basicConfig(
         level=logging.INFO,
@@ -41,7 +53,12 @@ def main(payload_path: str) -> None:
         payload.args,
         payload.kwargs,
     )
-    payload.fn(*payload.args, **payload.kwargs)
+    try:
+        value = payload.fn(*payload.args, **payload.kwargs)
+    except BaseException as exc:
+        record(JobResult.from_exception(payload, exc))
+        raise
+    record(JobResult.from_value(payload, value))
 
 
 if __name__ == "__main__":

{cortexgrid-0.2.97 → cortexgrid-0.3.0}/cortexgrid/jobs.py

@@ -12,18 +12,39 @@ from pathlib import Path
 import sys
 import tarfile
 import tempfile
+import time
+import traceback
 from typing import Any, Callable
 
 from cortexgrid import s3_util
 from cortexgrid._bundle import bundle, stage, worker_provides
 from cortexgrid.infra import get_mlflow_tracking_uri
-from cortexgrid.ray_util import get_ray_job_id_for_cortexgrid_job
+from cortexgrid.ray_util import (
+    JobStatus,
+    get_ray_job_id_for_cortexgrid_job,
+    get_ray_job_status,
+)
 from haikunator import Haikunator  # type: ignore
 from mlflow.tracking import MlflowClient
 from pydantic import BaseModel, ConfigDict
 
 log = logging.getLogger(__name__)
 
+_JOB_POLL_INTERVAL_S = 5.0
+_TERMINAL_JOB_STATES = (JobStatus.FINISHED, JobStatus.FAILED, JobStatus.STOPPED)
+
+
+class JobFailed(RuntimeError):
+    """A job did not produce a return value: its function raised, or the job
+    ended in a terminal state before the function ever returned. Carries the
+    remote traceback when there is one."""
+
+
+class JobResultUnavailable(RuntimeError):
+    """The job's function returned, but its value cannot be handed back here:
+    it did not survive cloudpickle on the cluster, cannot be unpickled by this
+    process, or was never recorded."""
+
 
 @dataclass
 class LifecycleEvent:
@@ -218,6 +239,215 @@ class Payload(BaseModel):
         return payload
 
 
+def _try_dumps(obj: Any) -> tuple[bytes | None, str | None]:
+    """cloudpickle ``obj``, or report why it could not be pickled.
+
+    Returns ``(blob, None)`` on success and ``(None, reason)`` on failure.
+    Nothing a job produces is allowed to turn into a job failure, so every
+    pickling error is captured rather than raised."""
+    try:
+        return cloudpickle.dumps(obj), None
+    except Exception as exc:
+        return None, f"{type(obj).__name__} did not survive cloudpickle: {exc!r}"
+
+
+@dataclass
+class JobResult:
+    """What a job's function returned or raised, recorded by the driver.
+
+    Written to ``job/{job_id}/result.pkl``, beside the payload manifest and
+    the lifecycle. The value and the exception are pickled separately from
+    the envelope so an object that cannot be pickled costs only its own
+    field: a job whose return value does not survive cloudpickle still
+    finishes, and the reason is still readable here.
+    """
+
+    experiment_name: str
+    run_id: str
+    job_id: str
+    ok: bool
+    value_pickle: bytes | None = None  # cloudpickled return value
+    value_error: str | None = None  # why value_pickle is None
+    exception_pickle: bytes | None = None  # cloudpickled exception, when picklable
+    traceback: str | None = None  # formatted remote traceback
+
+    @classmethod
+    def from_value(cls, payload: "Payload", value: Any) -> "JobResult":
+        blob, error = _try_dumps(value)
+        return cls(
+            experiment_name=payload.experiment_name,
+            run_id=payload.run_id,
+            job_id=payload.job_id,
+            ok=True,
+            value_pickle=blob,
+            value_error=error,
+        )
+
+    @classmethod
+    def from_exception(cls, payload: "Payload", exc: BaseException) -> "JobResult":
+        blob, _ = _try_dumps(exc)
+        return cls(
+            experiment_name=payload.experiment_name,
+            run_id=payload.run_id,
+            job_id=payload.job_id,
+            ok=False,
+            exception_pickle=blob,
+            traceback="".join(traceback.format_exception(exc)),
+        )
+
+    def save_to_mlflow(self) -> None:
+        artifact_path = f"job/{self.job_id}"
+        client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())
+        log.info("Recording result for job %s (ok=%s)", self.job_id, self.ok)
+        with tempfile.TemporaryDirectory() as tmp_dir:
+            local_path = Path(tmp_dir, "result.pkl")
+            local_path.write_bytes(cloudpickle.dumps(self))
+            client.log_artifact(
+                self.run_id, str(local_path), artifact_path=artifact_path
+            )
+
+    @classmethod
+    def load_from_mlflow(cls, run_id: str, job_id: str) -> "JobResult":
+        client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())
+        result_rel = f"job/{job_id}/result.pkl"
+        if not any(
+            a.path == result_rel for a in client.list_artifacts(run_id, f"job/{job_id}")
+        ):
+            raise FileNotFoundError(f"artifact {result_rel} not found in run {run_id}")
+        local_path = client.download_artifacts(run_id, result_rel)
+        result = cloudpickle.loads(Path(local_path).read_bytes())
+        if not isinstance(result, cls):
+            raise TypeError(f"{result_rel} in run {run_id} is not a JobResult")
+        return result
+
+    def unwrap(self) -> Any:
+        """Return the value the job's function returned, or raise what it raised.
+
+        A failed job raises the original exception, rebuilt from its pickle, so
+        a blocking caller sees what a local call would have raised; the remote
+        traceback rides along as the chained ``JobFailed`` cause. When the
+        exception itself did not survive pickling, ``JobFailed`` is raised
+        instead.
+        """
+        if not self.ok:
+            failure = JobFailed(f"job {self.job_id} failed:\n{self.traceback}")
+            remote_exc = self._unpickled_exception()
+            if remote_exc is None:
+                raise failure
+            raise remote_exc from failure
+        if self.value_pickle is None:
+            raise JobResultUnavailable(
+                f"job {self.job_id} finished, but its return value was not "
+                f"recorded: {self.value_error}"
+            )
+        try:
+            return cloudpickle.loads(self.value_pickle)
+        except Exception as exc:
+            raise JobResultUnavailable(
+                f"job {self.job_id} finished, but its return value could not be "
+                f"unpickled here: {exc!r}"
+            ) from exc
+
+    def _unpickled_exception(self) -> BaseException | None:
+        """The remote exception object, or None when it cannot be rebuilt here."""
+        if self.exception_pickle is None:
+            return None
+        try:
+            exc = cloudpickle.loads(self.exception_pickle)
+        except Exception:
+            log.warning("Job %s: remote exception could not be unpickled", self.job_id)
+            return None
+        return exc if isinstance(exc, BaseException) else None
+
+
+def _missing_result(
+    lifecycle: JobLifecycle, ray_job_id: str | None, status: JobStatus
+) -> Exception:
+    """The error for a job that reached ``status`` leaving no result behind.
+
+    Two ways in: the job's function never ran (the driver died, or the control
+    plane could not hand the job to Ray), or it ran and the upload of its
+    result failed. Either way the detail lives in the Ray logs, and a
+    submission error lives on the lifecycle."""
+    error = next((e.error for e in reversed(lifecycle.history) if e.error), None)
+    detail = f": {error}" if error else f" (ray job {ray_job_id}; check its logs)"
+    if status == JobStatus.FINISHED:
+        return JobResultUnavailable(
+            f"job {lifecycle.job_id} finished but recorded no result{detail}"
+        )
+    return JobFailed(
+        f"job {lifecycle.job_id} ended in {status.value} before its function "
+        f"returned{detail}"
+    )
+
+
+def wait_for_job_result(run_id: str, job_id: str, timeout: float | None = None) -> Any:
+    """Block until the job reaches a terminal Ray state, then return what its
+    function returned — or raise what it raised, as a local call would.
+
+    Raises ``ValueError`` for a job submitted with ``retry=True``: retries are
+    unbounded by design, so a blocking wait would sit through however many
+    resubmissions the control plane makes. Raises ``TimeoutError`` once a
+    finite ``timeout`` elapses (``timeout=0`` polls once and gives up);
+    ``timeout=None`` has no deadline. Raises ``JobFailed`` if the job never
+    got as far as returning, and ``JobResultUnavailable`` if it returned a
+    value that cannot be handed back here.
+    """
+    deadline = None if timeout is None else time.monotonic() + timeout
+    while True:
+        lifecycle = JobLifecycle.load_from_mlflow(run_id, job_id)
+        if lifecycle.retry:
+            raise ValueError(
+                f"job {job_id} was submitted with retry=True, which resubmits "
+                "without bound; waiting on its result is not supported. Poll "
+                "list_experiment_run_jobs instead."
+            )
+        ray_job_id = lifecycle.get_ray_job_id()
+        status = get_ray_job_status(ray_job_id)
+        if status in _TERMINAL_JOB_STATES:
+            try:
+                result = JobResult.load_from_mlflow(run_id, job_id)
+            except FileNotFoundError:
+                raise _missing_result(lifecycle, ray_job_id, status) from None
+            return result.unwrap()
+        if deadline is not None and time.monotonic() >= deadline:
+            raise TimeoutError(
+                f"job {job_id} did not finish within {timeout}s "
+                f"(last status={status.value})"
+            )
+        time.sleep(_JOB_POLL_INTERVAL_S)
+
+
+@dataclass(frozen=True)
+class JobFuture:
+    """A handle on a submitted job: its identity, its live status, and its result.
+
+    ``schedule_remote_job`` returns one as soon as the job request is recorded.
+    Nothing here blocks until :meth:`result` is called.
+    """
+
+    experiment_name: str
+    run_id: str
+    job_id: str
+
+    def status(self) -> JobStatus:
+        """The job's live status, derived from Ray. ``PENDING`` covers both
+        'the control plane has not picked it up yet' and 'queued in Ray'."""
+        lifecycle = JobLifecycle.load_from_mlflow(self.run_id, self.job_id)
+        return get_ray_job_status(lifecycle.get_ray_job_id())
+
+    def done(self) -> bool:
+        """True once Ray reports the job finished, failed or stopped."""
+        return self.status() in _TERMINAL_JOB_STATES
+
+    def result(self, timeout: float | None = None) -> Any:
+        """Block until the job finishes and return its function's value.
+
+        Raises exactly what :func:`wait_for_job_result` raises — including the
+        job's own exception, so a blocking call reads like a local one."""
+        return wait_for_job_result(self.run_id, self.job_id, timeout)
+
+
 def schedule_remote_job(
     experiment_name: str,
     run_id: str,
@@ -227,8 +457,8 @@ def schedule_remote_job(
     num_cpus: int = 1,
     retry: bool = False,
     **kwargs: Any,
-) -> str:
-    """Submit a function to the control plane. Returns a job ID."""
+) -> JobFuture:
+    """Submit a function to the control plane. Returns a handle on the job."""
     job_id = Haikunator().haikunate(token_length=2, token_chars="0123456789")
     entry_file = Path(inspect.getfile(fn)).resolve()
     driver_file = Path(__file__).with_name("_ray_job_driver.py")
@@ -263,7 +493,7 @@ def schedule_remote_job(
             num_cpus=num_cpus,
             pip_requirements=pip_requirements,
         ).save_to_mlflow()
-    return job_id
+    return JobFuture(experiment_name=experiment_name, run_id=run_id, job_id=job_id)
 
 
 def list_experiment_run_jobs(run_id: str) -> list[JobLifecycle]:

{cortexgrid-0.2.97 → cortexgrid-0.3.0}/docs/cortexgrid/README.md

@@ -79,11 +79,61 @@ def train_step(batch):
     # MLflow and S3 env vars are injected automatically
     return loss
 
-job_id = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
-print(f"Submitted: {job_id}")
+job = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
+print(f"Submitted: {job.job_id}")
 ```
 
-`cortexgrid.remote` submits a job *request* (a pickled payload plus a `JobLifecycle` record) to MLflow and returns a job ID string immediately. It does not wait for the job to run or finish — use the UI at `http://<DGX_IP>:8000`, or poll `cortexgrid.list_experiment_run_jobs(run_id)`, to observe status.
+`cortexgrid.remote` submits a job *request* (a pickled payload plus a `JobLifecycle` record) to MLflow and returns a `JobFuture` immediately. It does not wait for the job to run or finish — use the UI at `http://<DGX_IP>:8000`, `job.status()`, or poll `cortexgrid.list_experiment_run_jobs(run_id)`, to observe status.
+
+##### Blocking on the result
+
+`JobFuture.result(timeout=None)` blocks until Ray reports the job terminal, then behaves like a local call: it returns whatever the function returned, or raises whatever the function raised.
+
+```python
+def score(batch) -> dict[str, float]:
+    # runs on the DGX
+    return {"loss": 0.12}
+
+# Fire-and-forget — the form training uses. Nothing blocks, and a retry=True
+# job cannot be waited on at all (see below).
+training = cortexgrid.remote(train_step, batch, num_gpus=1, retry=True)
+print(training.job_id, training.status().value)   # "blue-42 running"
+
+# Blocking — the value comes back as if score() had run locally.
+job = cortexgrid.remote(score, batch, num_gpus=1)
+metrics = job.result()                            # {"loss": 0.12}
+
+# ...or with a deadline. On TimeoutError the job keeps running; wait again later.
+metrics = job.result(timeout=600)
+
+# From another process — the handle is gone, the job id is enough.
+metrics = cortexgrid.get_job_result(job_id, timeout=600)
+```
+
+Failures arrive as exceptions, not as a status to inspect:
+
+```python
+try:
+    metrics = job.result()
+except BadBatch as exc:                # exactly what score() raised on the DGX
+    print(exc.__cause__)               # JobFailed, carrying the remote traceback
+except cortexgrid.JobFailed:           # the job never got as far as returning
+    ...                                # submission failed, driver died, or stopped
+except cortexgrid.JobResultUnavailable:
+    ...                                # it returned, but the value is not transportable
+```
+
+The driver cloudpickles the outcome to `job/{job_id}/result.pkl`, beside the payload manifest and the lifecycle, and the waiting side reads it back:
+
+| Outcome | What `result()` does |
+|---|---|
+| The function returned | Returns its value |
+| The function raised | Raises that exception, with the remote traceback attached as a chained `JobFailed` cause |
+| The job never got as far as returning (submission failed, driver died, job stopped) | Raises `JobFailed` |
+| The value or the exception did not survive cloudpickle | Raises `JobResultUnavailable` (value) or `JobFailed` (exception). A value that cannot be pickled never fails the job itself |
+| `timeout` elapsed | Raises `TimeoutError`; the job keeps running |
+
+Waiting on a `retry=True` job raises `ValueError`: retries are unbounded by design (see below), so the wait would have no end. Fire-and-forget submission is the form training uses — submit, then watch the UI.
 
 A separate service — the **jobs control plane** — polls MLflow for pending job requests, matches them against the set of Ray submissions the cluster already has, and submits anything missing. It is also responsible for retrying failed jobs and honouring user-requested stops.
 
@@ -169,7 +219,9 @@ The requirements are part of the model, not of the serve-app class: GPUs, RAM an
 | `cortexgrid.log_artifact(path, artifact_path)` | Log a file as an artifact |
 | `cortexgrid.checkpoint()` | Context manager returning an attribute-based checkpoint saved to MLflow on exit |
 | `cortexgrid.resume()` | Load the latest checkpoint for the current job, or `None` |
-| `cortexgrid.remote(fn, *args, num_gpus=0, num_cpus=1, retry=False, **kwargs)` | Submit a function to the jobs control plane; returns a job ID |
+| `cortexgrid.remote(fn, *args, num_gpus=0, num_cpus=1, retry=False, **kwargs)` | Submit a function to the jobs control plane; returns a `JobFuture` |
+| `JobFuture.status()` / `.done()` / `.result(timeout=None)` | Live status of a submitted job, and its function's return value (blocking) |
+| `cortexgrid.get_job_result(job_id, timeout=None)` | Block on a job of the current run by id; returns its value or raises its exception |
 | `cortexgrid.list_experiment_run_jobs(run_id)` | List `JobLifecycle` records for every cortexgrid job in a run |
 | `cortexgrid.stop_experiment_run_jobs(run_id)` | Request every job in a run to stop (flips the `stop_requested` latch) |
 | `cortexgrid.get_ray_job_status(ray_job_id)` | Live Ray status for a submission id |

{cortexgrid-0.2.97 → cortexgrid-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "cortexgrid"
-version = "0.2.97"
+version = "0.3.0"
 description = "Connect your ML code to the RoboLab compute cluster — Ray, MLflow, and S3"
 readme = "docs/cortexgrid/README.md"
 license = "Apache-2.0"