zenml-nightly 0.82.1.dev20250528__tar.gz → 0.83.0.dev20250530__tar.gz

{zenml_nightly-0.82.1.dev20250528 → zenml_nightly-0.83.0.dev20250530}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: zenml-nightly
-Version: 0.82.1.dev20250528
+Version: 0.83.0.dev20250530
 Summary: ZenML: Write production-ready ML code.
 License: Apache-2.0
 Keywords: machine learning,production,pipeline,mlops,devops
@@ -532,7 +532,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.82.1 is out. Check out the release notes
+    🎉 Version 0.83.0 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

{zenml_nightly-0.82.1.dev20250528 → zenml_nightly-0.83.0.dev20250530}/README.md

@@ -386,7 +386,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.82.1 is out. Check out the release notes
+    🎉 Version 0.83.0 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

{zenml_nightly-0.82.1.dev20250528 → zenml_nightly-0.83.0.dev20250530}/RELEASE_NOTES.md

@@ -1,4 +1,73 @@
 <!-- markdown-link-check-disable -->
+# 0.83.0
+
+🚀 **Major Performance Release** - ZenML 0.83.0 introduces significant performance improvements and response optimizations that dramatically reduce API response times and database query overhead. This release includes breaking changes and is **not compatible with earlier client/server versions**.
+
+## ⚡ Performance Improvements
+
+This release delivers substantial performance enhancements across the ZenML server:
+
+- **Optimized API Responses**: Pipeline run responses no longer include unpaginated step lists, dramatically reducing response sizes for large pipelines
+- **Database Query Optimization**: Improved query patterns with strategic joined loads to minimize database roundtrips
+- **Reduced Response Payloads**: Many attributes moved from `body` to `resources` to avoid unnecessary data transfer when objects are embedded in other responses
+
+## 🔄 Breaking Changes
+
+⚠️ **Client/Server Compatibility**: This version is **not compatible** with earlier ZenML client/server versions. Please ensure both client and server are upgraded to 0.83.0.
+
+### API Response Changes
+- Pipeline run responses no longer include `metadata.steps` or `metadata.step_substitutions`
+- Many model attributes moved from `body` to `resources` for performance
+- Project metadata structure simplified across all responses
+- Model version responses no longer include comprehensive artifact and run ID lists
+
+### Method Deprecations
+- `Model.get_pipeline_run(...)` and `ModelVersionResponse.get_pipeline_run(...)` have been removed
+
+For a comprehensive list of all response changes and migration details, see [PR #3675](https://github.com/zenml-io/zenml/pull/3675).
+
+## 🚀 Orchestrator Enhancements
+
+### Kubernetes Orchestrator
+- **Enhanced Caching**: Added caching capabilities in Kubernetes orchestrator entrypoint to improve performance and reduce unnecessary pod creations
+
+### Skypilot Orchestrator  
+- **Updated Version and Settings**: Updated to new Skypilot version and added new setting options
+
+## What's Changed
+* Adding 0.82.0 to the legacy docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3671
+* New deployment scenarios by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3666
+* Add instruction to `WANDB_DISABLED` to `True` for the quickstart by @bcdurak in https://github.com/zenml-io/zenml/pull/3673
+* Fix service connector docs example by @strickvl in https://github.com/zenml-io/zenml/pull/3679
+* Fix YAML extension check by @strickvl in https://github.com/zenml-io/zenml/pull/3677
+* Fix typos, bugs, and improve test precision by @strickvl in https://github.com/zenml-io/zenml/pull/3678
+* Performance boost fix: don't fetch entire pipeline run to verify pipeline API token validity by @stefannica in https://github.com/zenml-io/zenml/pull/3684
+* Async wrapper for FastAPI endpoints to run serialization in event loop by @schustmi in https://github.com/zenml-io/zenml/pull/3685
+* Don't log parent image digest warning if build is skipped by @schustmi in https://github.com/zenml-io/zenml/pull/3676
+* Convert string to raw to avoid warnings in python 3.12+ by @jlopezpena in https://github.com/zenml-io/zenml/pull/3687
+* Use correct artifact store for nested materializers by @schustmi in https://github.com/zenml-io/zenml/pull/3670
+* Tiny Discord docs fix by @strickvl in https://github.com/zenml-io/zenml/pull/3691
+* Store validated config with converted types in DB by @schustmi in https://github.com/zenml-io/zenml/pull/3668
+* Added zenml codespace env detection by @htahir1 in https://github.com/zenml-io/zenml/pull/3686
+* Fix setuptools vulnerabilities and deprecate pip as DockerSettings default by @stefannica in https://github.com/zenml-io/zenml/pull/3694
+* Improve alerter documentation with comprehensive ask step coverage by @strickvl in https://github.com/zenml-io/zenml/pull/3693
+* Add comprehensive agent guidelines with `AGENTS.md` and `CLAUDE.md` by @strickvl in https://github.com/zenml-io/zenml/pull/3680
+* Performance test utilities, stats and stress test pipeline updates by @stefannica in https://github.com/zenml-io/zenml/pull/3690
+* Fix GCP service connector expiry by @stefannica in https://github.com/zenml-io/zenml/pull/3697
+* Fix trivy image scanning GitHub actions by @stefannica in https://github.com/zenml-io/zenml/pull/3699
+* Add ability to strip the timestamps from the logs, on request by @avishniakov in https://github.com/zenml-io/zenml/pull/3683
+* Send POST request for RBAC permission checks to avoid URL length limits by @schustmi in https://github.com/zenml-io/zenml/pull/3702
+* Response and database improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3675
+* Add Caching in Kubernetes orchestrator entrypoint by @schustmi in https://github.com/zenml-io/zenml/pull/3703
+* New Pro onboarding by @schustmi in https://github.com/zenml-io/zenml/pull/3704
+* Suppress repeated DockerSettings warnings by @stefannica in https://github.com/zenml-io/zenml/pull/3705
+* New OSS dashboard fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3706
+* Remove unused import from metadata docs by @strickvl in https://github.com/zenml-io/zenml/pull/3707
+* Project consistency fixes for `Model` and `ModelVersionResponse` by @schustmi in https://github.com/zenml-io/zenml/pull/3708
+* Update Skypilot orchestrator settings and features by @htahir1 in https://github.com/zenml-io/zenml/pull/3612
+
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.82.1...0.83.0
 
 # 0.82.1
 

{zenml_nightly-0.82.1.dev20250528 → zenml_nightly-0.83.0.dev20250530}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "zenml-nightly"
-version = "0.82.1.dev20250528"
+version = "0.83.0.dev20250530"
 packages = [{ include = "zenml", from = "src" }]
 description = "ZenML: Write production-ready ML code."
 authors = ["ZenML GmbH <info@zenml.io>"]

zenml_nightly-0.83.0.dev20250530/src/zenml/VERSION

@@ -0,0 +1 @@
+0.83.0.dev20250530

{zenml_nightly-0.82.1.dev20250528 → zenml_nightly-0.83.0.dev20250530}/src/zenml/integrations/skypilot/flavors/skypilot_orchestrator_base_vm_config.py

@@ -13,7 +13,7 @@
 #  permissions and limitations under the License.
 """Skypilot orchestrator base config and settings."""
 
-from typing import Dict, List, Literal, Optional, Union
+from typing import Any, Dict, List, Literal, Optional, Union
 
 from pydantic import Field
 
@@ -67,6 +67,14 @@ class SkypilotBaseOrchestratorSettings(BaseSettings):
         disk_size: the size of the OS disk in GiB.
         disk_tier: the disk performance tier to use. If None, defaults to
             ``'medium'``.
+        ports: Ports to expose. Could be an integer, a range, or a list of
+            integers and ranges. All ports will be exposed to the public internet.
+        labels: Labels to apply to instances as key-value pairs. These are
+            mapped to cloud-specific implementations (instance tags in AWS,
+            instance labels in GCP, etc.)
+        any_of: List of candidate resources to try in order of preference based on
+            cost (determined by the optimizer).
+        ordered: List of candidate resources to try in the specified order.
 
         cluster_name: name of the cluster to create/reuse.  If None,
             auto-generate a name.
@@ -88,6 +96,19 @@ class SkypilotBaseOrchestratorSettings(BaseSettings):
         stream_logs: if True, show the logs in the terminal.
         docker_run_args: Optional arguments to pass to the `docker run` command
             running inside the VM.
+        workdir: Working directory to sync to the VM. Synced to ~/sky_workdir.
+        task_name: Task name used for display purposes.
+        file_mounts: File and storage mounts configuration for remote cluster.
+        envs: Environment variables for the task.
+        task_settings: Dictionary of arbitrary settings to pass to sky.Task().
+            This allows passing future parameters added by SkyPilot without
+            requiring updates to ZenML.
+        resources_settings: Dictionary of arbitrary settings to pass to
+            sky.Resources(). This allows passing future parameters added
+            by SkyPilot without requiring updates to ZenML.
+        launch_settings: Dictionary of arbitrary settings to pass to
+            sky.launch(). This allows passing future parameters added
+            by SkyPilot without requiring updates to ZenML.
     """
 
     # Resources
@@ -103,14 +124,18 @@ class SkypilotBaseOrchestratorSettings(BaseSettings):
     )
     accelerator_args: Optional[Dict[str, str]] = None
     use_spot: Optional[bool] = None
-    job_recovery: Optional[str] = None
+    job_recovery: Union[None, str, Dict[str, Any]] = Field(
+        default=None, union_mode="left_to_right"
+    )
     region: Optional[str] = None
     zone: Optional[str] = None
     image_id: Union[Dict[str, str], str, None] = Field(
         default=None, union_mode="left_to_right"
     )
     disk_size: Optional[int] = None
-    disk_tier: Optional[Literal["high", "medium", "low"]] = None
+    disk_tier: Optional[Literal["high", "medium", "low", "ultra", "best"]] = (
+        None
+    )
 
     # Run settings
     cluster_name: Optional[str] = None
@@ -118,9 +143,25 @@ class SkypilotBaseOrchestratorSettings(BaseSettings):
     idle_minutes_to_autostop: Optional[int] = 30
     down: bool = True
     stream_logs: bool = True
-
     docker_run_args: List[str] = []
 
+    # Additional SkyPilot features
+    ports: Union[None, int, str, List[Union[int, str]]] = Field(
+        default=None, union_mode="left_to_right"
+    )
+    labels: Optional[Dict[str, str]] = None
+    any_of: Optional[List[Dict[str, Any]]] = None
+    ordered: Optional[List[Dict[str, Any]]] = None
+    workdir: Optional[str] = None
+    task_name: Optional[str] = None
+    file_mounts: Optional[Dict[str, Any]] = None
+    envs: Optional[Dict[str, str]] = None
+
+    # Future-proofing settings dictionaries
+    task_settings: Dict[str, Any] = {}
+    resources_settings: Dict[str, Any] = {}
+    launch_settings: Dict[str, Any] = {}
+
 
 class SkypilotBaseOrchestratorConfig(
     BaseOrchestratorConfig, SkypilotBaseOrchestratorSettings

{zenml_nightly-0.82.1.dev20250528 → zenml_nightly-0.83.0.dev20250530}/src/zenml/integrations/skypilot/orchestrators/skypilot_base_vm_orchestrator.py

@@ -14,12 +14,12 @@
 """Implementation of the Skypilot base VM orchestrator."""
 
 import os
-import re
 from abc import abstractmethod
 from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, cast
 from uuid import uuid4
 
 import sky
+from sky import StatusRefreshMode
 
 from zenml.entrypoints import PipelineEntrypointConfiguration
 from zenml.enums import StackComponentType
@@ -31,6 +31,15 @@ from zenml.integrations.skypilot.flavors.skypilot_orchestrator_base_vm_config im
 from zenml.integrations.skypilot.orchestrators.skypilot_orchestrator_entrypoint_configuration import (
     SkypilotOrchestratorEntrypointConfiguration,
 )
+from zenml.integrations.skypilot.utils import (
+    create_docker_run_command,
+    prepare_docker_setup,
+    prepare_launch_kwargs,
+    prepare_resources_kwargs,
+    prepare_task_kwargs,
+    sanitize_cluster_name,
+    sky_job_get,
+)
 from zenml.logger import get_logger
 from zenml.orchestrators import (
     ContainerizedOrchestrator,
@@ -252,32 +261,21 @@ class SkypilotBaseOrchestrator(ContainerizedOrchestrator):
         entrypoint_str = " ".join(command)
         arguments_str = " ".join(args)
 
-        task_envs = environment
-        docker_environment_str = " ".join(
-            f"-e {k}={v}" for k, v in environment.items()
-        )
-        custom_run_args = " ".join(settings.docker_run_args)
-        if custom_run_args:
-            custom_run_args += " "
-
-        instance_type = settings.instance_type or self.DEFAULT_INSTANCE_TYPE
+        task_envs = environment.copy()
 
         # Set up credentials
         self.setup_credentials()
 
-        # Guaranteed by stack validation
-        assert stack is not None and stack.container_registry is not None
+        # Prepare Docker setup
+        setup, docker_creds_envs = prepare_docker_setup(
+            container_registry_uri=stack.container_registry.config.uri,
+            credentials=stack.container_registry.credentials,
+            use_sudo=True,  # Base orchestrator uses sudo
+        )
 
-        if docker_creds := stack.container_registry.credentials:
-            docker_username, docker_password = docker_creds
-            setup = (
-                f"sudo docker login --username $DOCKER_USERNAME --password "
-                f"$DOCKER_PASSWORD {stack.container_registry.config.uri}"
-            )
-            task_envs["DOCKER_USERNAME"] = docker_username
-            task_envs["DOCKER_PASSWORD"] = docker_password
-        else:
-            setup = None
+        # Update task_envs with Docker credentials
+        if docker_creds_envs:
+            task_envs.update(docker_creds_envs)
 
         # Run the entire pipeline
 
@@ -291,45 +289,49 @@ class SkypilotBaseOrchestrator(ContainerizedOrchestrator):
                 down = False
                 idle_minutes_to_autostop = None
             else:
-                run_command = f"sudo docker run --rm {custom_run_args}{docker_environment_str} {image} {entrypoint_str} {arguments_str}"
+                run_command = create_docker_run_command(
+                    image=image,
+                    entrypoint_str=entrypoint_str,
+                    arguments_str=arguments_str,
+                    environment=task_envs,
+                    docker_run_args=settings.docker_run_args,
+                    use_sudo=True,  # Base orchestrator uses sudo
+                )
                 down = settings.down
                 idle_minutes_to_autostop = settings.idle_minutes_to_autostop
-            task = sky.Task(
-                run=run_command,
+
+            # Create the Task with all parameters and task settings
+            task_kwargs = prepare_task_kwargs(
+                settings=settings,
+                run_command=run_command,
                 setup=setup,
-                envs=task_envs,
+                task_envs=task_envs,
+                task_name=f"{orchestrator_run_name}",
             )
+
+            task = sky.Task(**task_kwargs)
             logger.debug(f"Running run: {run_command}")
 
-            task = task.set_resources(
-                sky.Resources(
-                    cloud=self.cloud,
-                    instance_type=instance_type,
-                    cpus=settings.cpus,
-                    memory=settings.memory,
-                    accelerators=settings.accelerators,
-                    accelerator_args=settings.accelerator_args,
-                    use_spot=settings.use_spot,
-                    job_recovery=settings.job_recovery,
-                    region=settings.region,
-                    zone=settings.zone,
-                    image_id=image
-                    if isinstance(self.cloud, sky.clouds.Kubernetes)
-                    else settings.image_id,
-                    disk_size=settings.disk_size,
-                    disk_tier=settings.disk_tier,
-                )
+            # Set resources with all parameters and resource settings
+            resources_kwargs = prepare_resources_kwargs(
+                cloud=self.cloud,
+                settings=settings,
+                default_instance_type=self.DEFAULT_INSTANCE_TYPE,
+                kubernetes_image=image
+                if isinstance(self.cloud, sky.clouds.Kubernetes)
+                else None,
             )
-            # Do not detach run if logs are being streamed
-            # Otherwise, the logs will not be streamed after the task is submitted
-            # Could also be a parameter in the settings to control this behavior
-            detach_run = not settings.stream_logs
+
+            task = task.set_resources(sky.Resources(**resources_kwargs))
 
             launch_new_cluster = True
             if settings.cluster_name:
-                cluster_info = sky.status(
-                    refresh=True, cluster_names=settings.cluster_name
+                status_request_id = sky.status(
+                    refresh=StatusRefreshMode.AUTO,
+                    cluster_names=[settings.cluster_name],
                 )
+                cluster_info = sky.stream_and_get(status_request_id)
+
                 if cluster_info:
                     logger.info(
                         f"Found existing cluster {settings.cluster_name}. Reusing..."
@@ -342,7 +344,7 @@ class SkypilotBaseOrchestrator(ContainerizedOrchestrator):
                     )
                     cluster_name = settings.cluster_name
             else:
-                cluster_name = self.sanitize_cluster_name(
+                cluster_name = sanitize_cluster_name(
                     f"{orchestrator_run_name}"
                 )
                 logger.info(
@@ -350,33 +352,55 @@ class SkypilotBaseOrchestrator(ContainerizedOrchestrator):
                 )
 
             if launch_new_cluster:
-                sky.launch(
+                # Prepare launch parameters with additional launch settings
+                launch_kwargs = prepare_launch_kwargs(
+                    settings=settings,
+                    down=down,
+                    idle_minutes_to_autostop=idle_minutes_to_autostop,
+                )
+                logger.info(
+                    f"Launching the task on a new cluster: {cluster_name}"
+                )
+                launch_job_id = sky.launch(
                     task,
                     cluster_name,
-                    retry_until_up=settings.retry_until_up,
-                    idle_minutes_to_autostop=idle_minutes_to_autostop,
-                    down=down,
-                    stream_logs=settings.stream_logs,
-                    backend=None,
-                    detach_setup=True,
-                    detach_run=detach_run,
+                    **launch_kwargs,
                 )
+                sky_job_get(launch_job_id, settings.stream_logs, cluster_name)
+
             else:
-                # Make sure the cluster is up -
-                # If the cluster is already up, this will not do anything
-                sky.start(
+                # Prepare exec parameters with additional launch settings
+                exec_kwargs = {
+                    "down": down,
+                    "backend": None,
+                    **settings.launch_settings,  # Can reuse same settings for exec
+                }
+
+                # Remove None values to avoid overriding SkyPilot defaults
+                exec_kwargs = {
+                    k: v for k, v in exec_kwargs.items() if v is not None
+                }
+
+                # Make sure the cluster is up
+                start_request_id = sky.start(
                     settings.cluster_name,
                     down=down,
                     idle_minutes_to_autostop=idle_minutes_to_autostop,
                     retry_until_up=settings.retry_until_up,
                 )
-                sky.exec(
+                sky.stream_and_get(start_request_id)
+
+                logger.info(
+                    f"Executing the task on the cluster: {settings.cluster_name}"
+                )
+                exec_job_id = sky.exec(
                     task,
-                    settings.cluster_name,
-                    down=down,
-                    stream_logs=settings.stream_logs,
-                    backend=None,
-                    detach_run=detach_run,
+                    cluster_name=settings.cluster_name,
+                    **exec_kwargs,
+                )
+                assert settings.cluster_name is not None
+                sky_job_get(
+                    exec_job_id, settings.stream_logs, settings.cluster_name
                 )
 
         except Exception as e:
@@ -386,19 +410,3 @@ class SkypilotBaseOrchestrator(ContainerizedOrchestrator):
         finally:
             # Unset the service connector AWS profile ENV variable
             self.prepare_environment_variable(set=False)
-
-    def sanitize_cluster_name(self, name: str) -> str:
-        """Sanitize the value to be used in a cluster name.
-
-        Args:
-            name: Arbitrary input cluster name.
-
-        Returns:
-            Sanitized cluster name.
-        """
-        name = re.sub(
-            r"[^a-z0-9-]", "-", name.lower()
-        )  # replaces any character that is not a lowercase letter, digit, or hyphen with a hyphen
-        name = re.sub(r"^[-]+", "", name)  # trim leading hyphens
-        name = re.sub(r"[-]+$", "", name)  # trim trailing hyphens
-        return name