cortexgrid 0.2.88__tar.gz → 0.2.89__tar.gz

{cortexgrid-0.2.88 → cortexgrid-0.2.89}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cortexgrid
-Version: 0.2.88
+Version: 0.2.89
 Summary: Connect your ML code to the RoboLab compute cluster — Ray, MLflow, and S3
 Project-URL: Homepage, https://github.com/robodatalab/cortexgrid
 Project-URL: Repository, https://github.com/robodatalab/cortexgrid
@@ -55,7 +55,7 @@ import cortexgrid
 cortexgrid.init(experiment="weather-forecast")
 ```
 
-That single call reads the service URLs from the head's secrets server at `$CORTEXGRID_HEAD_URL` and connects to all services through them. It also creates (or finds) the named MLflow experiment and starts a new run inside it. Omit `experiment=` to auto-generate a unique name like `funky-koval-12`.
+That single call reads the service URLs from the head's secrets server at `$CORTEXGRID_HEAD_URL` and connects to all services through them. It also creates (or finds) the named MLflow experiment and starts a new run inside it. If an experiment of that name was deleted (e.g. from the UI), a new experiment is created under the name: MLflow keeps a deleted experiment's name reserved, so the deleted one is renamed to `<name>__deleted__<id>` first (`delete_experiment` does that rename at deletion). Omit `experiment=` to auto-generate a unique name like `funky-koval-12`.
 
 **One experiment per binary run.** `cortexgrid.init()` may only be called once per process. Every subsequent `cortexgrid.log_metric`, `cortexgrid.log_artifact`, checkpoint, and `cortexgrid.remote()` submission is scoped to that experiment+run. Remote jobs dispatched by the control plane inherit the experiment+run via the pickled payload, so their logging flows into the same MLflow run as the parent binary.
 

{cortexgrid-0.2.88 → cortexgrid-0.2.89}/cortexgrid/experiment.py

@@ -10,6 +10,7 @@ from cortexgrid.jobs import stop_experiment_run_jobs
 from cortexgrid.ray_util import list_ray_jobs_with_submission_id, stop_ray_job
 from cortexgrid.model_storage import delete_models_for_run
 from haikunator import Haikunator  # type: ignore
+from mlflow.entities import Experiment as MlflowExperiment
 from mlflow.tracking import MlflowClient
 
 
@@ -122,7 +123,12 @@ def _try_create_experiment_and_run(
         experiment = name_gen.haikunate(token_length=2, token_chars="0123456789")
 
     client = MlflowClient(tracking_uri=mlflow_tracking_uri)
+    # get_experiment_by_name also returns deleted experiments. A deleted one
+    # cannot take new runs, so a fresh experiment is created under its name.
     experiment_obj = client.get_experiment_by_name(name=experiment)
+    if experiment_obj is not None and experiment_obj.lifecycle_stage != "active":
+        _release_deleted_experiment_name(client, experiment_obj)
+        experiment_obj = None
     if experiment_obj:
         experiment_id = experiment_obj.experiment_id
     else:
@@ -134,6 +140,33 @@ def _try_create_experiment_and_run(
     return (experiment, run.info.run_id)
 
 
+def _deleted_experiment_name(name: str, experiment_id: str) -> str:
+    """The name a deleted experiment is moved to, freeing `name` for reuse.
+    Unique because experiment ids are."""
+    return f"{name}__deleted__{experiment_id}"
+
+
+def _release_deleted_experiment_name(
+    client: MlflowClient, experiment: MlflowExperiment
+) -> None:
+    """Move a deleted experiment off its name, so a new experiment can take it.
+
+    MLflow keeps a deleted experiment's name reserved (experiment names are
+    unique across every lifecycle stage) and refuses to rename a deleted
+    experiment, so it is restored only for the rename and deleted again."""
+    log.info(
+        "Experiment %r (id %s) is deleted; renaming it to release the name",
+        experiment.name,
+        experiment.experiment_id,
+    )
+    client.restore_experiment(experiment.experiment_id)
+    client.rename_experiment(
+        experiment.experiment_id,
+        _deleted_experiment_name(experiment.name, experiment.experiment_id),
+    )
+    client.delete_experiment(experiment.experiment_id)
+
+
 def delete_run(run_id: str) -> None:
     """Soft-delete a run in MLflow, cancel its Ray attempts, and wipe its
     S3 job packages so it cannot be relaunched or re-read.
@@ -174,6 +207,10 @@ def list_run_ids_in_experiment(name: str) -> list[str]:
 def delete_experiment(name: str) -> None:
     """Soft-delete every run in the experiment, then the experiment itself.
 
+    The experiment is renamed (`<name>__deleted__<id>`) before it is deleted:
+    MLflow keeps a deleted experiment's name reserved, and the rename frees it
+    so `Experiment.init(name)` can create a new experiment under it.
+
     Idempotent: already-deleted experiments are treated as success."""
     log.info("delete_experiment(%r): start", name)
     client = MlflowClient(tracking_uri=get_mlflow_tracking_uri())
@@ -190,6 +227,9 @@ def delete_experiment(name: str) -> None:
     log.info("delete_experiment(%r): %d active run(s) to delete", name, len(runs))
     for run in runs:
         delete_run(run.info.run_id)
+    client.rename_experiment(
+        exp.experiment_id, _deleted_experiment_name(name, exp.experiment_id)
+    )
     client.delete_experiment(exp.experiment_id)
     log.info("delete_experiment(%r): done", name)
 

{cortexgrid-0.2.88 → cortexgrid-0.2.89}/docs/cortexgrid/README.md

@@ -27,7 +27,7 @@ import cortexgrid
 cortexgrid.init(experiment="weather-forecast")
 ```
 
-That single call reads the service URLs from the head's secrets server at `$CORTEXGRID_HEAD_URL` and connects to all services through them. It also creates (or finds) the named MLflow experiment and starts a new run inside it. Omit `experiment=` to auto-generate a unique name like `funky-koval-12`.
+That single call reads the service URLs from the head's secrets server at `$CORTEXGRID_HEAD_URL` and connects to all services through them. It also creates (or finds) the named MLflow experiment and starts a new run inside it. If an experiment of that name was deleted (e.g. from the UI), a new experiment is created under the name: MLflow keeps a deleted experiment's name reserved, so the deleted one is renamed to `<name>__deleted__<id>` first (`delete_experiment` does that rename at deletion). Omit `experiment=` to auto-generate a unique name like `funky-koval-12`.
 
 **One experiment per binary run.** `cortexgrid.init()` may only be called once per process. Every subsequent `cortexgrid.log_metric`, `cortexgrid.log_artifact`, checkpoint, and `cortexgrid.remote()` submission is scoped to that experiment+run. Remote jobs dispatched by the control plane inherit the experiment+run via the pickled payload, so their logging flows into the same MLflow run as the parent binary.
 

{cortexgrid-0.2.88 → cortexgrid-0.2.89}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "cortexgrid"
-version = "0.2.88"
+version = "0.2.89"
 description = "Connect your ML code to the RoboLab compute cluster — Ray, MLflow, and S3"
 readme = "docs/cortexgrid/README.md"
 license = "Apache-2.0"