zenml-nightly 0.80.1.dev20250409__tar.gz → 0.80.2.dev20250411__tar.gz

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: zenml-nightly
-Version: 0.80.1.dev20250409
+Version: 0.80.2.dev20250411
 Summary: ZenML: Write production-ready ML code.
 License: Apache-2.0
 Keywords: machine learning,production,pipeline,mlops,devops
@@ -531,7 +531,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.80.1 is out. Check out the release notes
+    🎉 Version 0.80.2 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/README.md

@@ -386,7 +386,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.80.1 is out. Check out the release notes
+    🎉 Version 0.80.2 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/RELEASE_NOTES.md

@@ -1,5 +1,78 @@
 <!-- markdown-link-check-disable -->
 
+# 0.80.2
+
+ The `0.80.2` release focuses on several bug fixes, usability enhancements and documentation improvements. Key highlights include fixing the yanked fsspec pip resolution issue, adding sample visualization for Pandas materializer, and upgrading pydantic to 2.11 for improved performance. This release also adds new tutorials and guides, fixes several critical issues, and introduces new functionality for pipeline development.
+
+ ## Features
+
+ - Added poetry add support for CLI
+ - Added retry option on step.with_options
+ - Added service annotations to helm chart
+ - Added step name to console logs for better tracing
+
+ ## Improvements
+
+ - Added sample visualization for Pandas materializer
+ - Improved run templates with small enhancements and hidden option support
+ - Fixed service connector list-resources endpoint and CLI command
+
+ ## Fixes
+
+ - Fixed timestamps in logs
+ - Fixed yanked fsspec pip resolution issue
+ - Fixed Kubernetes Orchestrator step pod failure status updates
+ - Fixed CI failures and test issues
+ - Updated error message for setting pipeline parameters via API
+
+ ## Documentation
+
+ - Added new ZenML Tutorials section to user guide
+ - Added mini-tutorials for [pipeline scheduling](https://docs.zenml.io/user-guides/tutorial/managing-scheduled-pipelines) and [organization with tags/projects](https://docs.zenml.io/user-guides/tutorial/organizing-pipelines-and-models)
+ - Updated documentation for [MCP Server](https://docs.zenml.io/how-to/manage-zenml-server/mcp-chat-with-server) and [VSCode Extension](https://docs.zenml.io/how-to/manage-zenml-server/vscode-extension)
+ - Fixed various documentation issues
+
+## What's Changed
+
+* Updating the missing legacy docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3473
+* add new version to the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3471
+* Add docs for MCP Server and VSCode Extension by @strickvl in https://github.com/zenml-io/zenml/pull/3475
+* Log job completion for Vertex orchestrator by @strickvl in https://github.com/zenml-io/zenml/pull/3478
+* Cleaning up the CI failures by @bcdurak in https://github.com/zenml-io/zenml/pull/3480
+* Remove remaining use of tj-actions/changed-files by @stefannica in https://github.com/zenml-io/zenml/pull/3481
+* Fixing the timestamps in logs by @bcdurak in https://github.com/zenml-io/zenml/pull/3479
+* Fix incorrect step access method in documentation by @marwan37 in https://github.com/zenml-io/zenml/pull/3483
+* Relaxing the `mlflow` dependency by @bcdurak in https://github.com/zenml-io/zenml/pull/3446
+* Add two mini-tutorials (pipeline scheduling and organization with tags/projects) by @strickvl in https://github.com/zenml-io/zenml/pull/3485
+* Add ZenML Tutorials section to user guide by @htahir1 in https://github.com/zenml-io/zenml/pull/3488
+* Add service annotations to helm chart by @jsybel in https://github.com/zenml-io/zenml/pull/3482
+* Adding step name to console logs by @bcdurak in https://github.com/zenml-io/zenml/pull/3486
+* Fix yanked fsspec pip resolution issue by @stefannica in https://github.com/zenml-io/zenml/pull/3493
+* Fixed some docs here and there by @htahir1 in https://github.com/zenml-io/zenml/pull/3484
+* Update error message for setting pipeline parameters via API by @htahir1 in https://github.com/zenml-io/zenml/pull/3491
+* Fix the service connector list-resources endpoint and CLI command by @stefannica in https://github.com/zenml-io/zenml/pull/3492
+* Restructure organization viewer role in ZenML Pro by @htahir1 in https://github.com/zenml-io/zenml/pull/3501
+* Add a sample visualization for Pandas materializer by @htahir1 in https://github.com/zenml-io/zenml/pull/3494
+* Add retry option on step.with_options by @schustmi in https://github.com/zenml-io/zenml/pull/3499
+* Fix tutorials + small improvements to docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3489
+* CI fixes with `huggingface` and crud tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3502
+* Upgrading pydantic to 2.11 by @bcdurak in https://github.com/zenml-io/zenml/pull/3472
+* Update step status if Kubernetes Orchestrator step pods fail by @stefannica in https://github.com/zenml-io/zenml/pull/3497
+* Add `poetry add` support for CLI by @avishniakov in https://github.com/zenml-io/zenml/pull/3470
+* Small run template improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3503
+* Move uv env var to constants by @schustmi in https://github.com/zenml-io/zenml/pull/3504
+* Add hidden option for run templates by @schustmi in https://github.com/zenml-io/zenml/pull/3500
+* Fix security check for poetry by @bcdurak in https://github.com/zenml-io/zenml/pull/3509
+* Minor auth improvement by @schustmi in https://github.com/zenml-io/zenml/pull/3512
+
+## New Contributors
+
+* @marwan37 made their first contribution in https://github.com/zenml-io/zenml/pull/3483
+* @jsybel made their first contribution in https://github.com/zenml-io/zenml/pull/3482
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.80.1...0.80.2
+
+
 # 0.80.1
 
 The `0.80.1` release focuses on bug fixes and performance improvements following the major `0.80.0` update. This release addresses several critical issues, particularly improving the CLI functionality when used with the REST API through a deployed ZenML instance. Additionally, this version introduces [a restructured documentation architecture](https://docs.zenml.io) for improved user experience.

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "zenml-nightly"
-version = "0.80.1.dev20250409"
+version = "0.80.2.dev20250411"
 packages = [{ include = "zenml", from = "src" }]
 description = "ZenML: Write production-ready ML code."
 authors = ["ZenML GmbH <info@zenml.io>"]

zenml_nightly-0.80.2.dev20250411/src/zenml/VERSION

@@ -0,0 +1 @@
+0.80.2.dev20250411

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/src/zenml/cli/__init__.py

@@ -2414,6 +2414,7 @@ If you want to create a run template based on your pipeline that can later be us
 ```bash
 zenml pipeline create-run-template <PIPELINE_SOURCE_PATH> \
    --name=<TEMPLATE_NAME>
+```
 
 To specify a config file, use the `--config/-c` option. If you would like to use a different stack than the active one, use the `--stack` option.
 

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/src/zenml/constants.py

@@ -184,6 +184,7 @@ ENV_ZENML_CODE_REPOSITORY_IGNORE_UNTRACKED_FILES = (
 ENV_ZENML_MATERIALIZER_ALLOW_NON_ASCII_JSON_DUMPS = (
     "ZENML_MATERIALIZER_ALLOW_NON_ASCII_JSON_DUMPS"
 )
+ENV_ZENML_DISABLE_PATH_MATERIALIZER = "ZENML_DISABLE_PATH_MATERIALIZER"
 
 # ZenML Server environment variables
 ENV_ZENML_SERVER_PREFIX = "ZENML_SERVER_"

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/src/zenml/enums.py

@@ -411,9 +411,15 @@ class OnboardingStep(StrEnum):
     STACK_WITH_REMOTE_ORCHESTRATOR_CREATED = (
         "stack_with_remote_orchestrator_created"
     )
+    STACK_WITH_REMOTE_ARTIFACT_STORE_CREATED = (
+        "stack_with_remote_artifact_store_created"
+    )
     PIPELINE_RUN_WITH_REMOTE_ORCHESTRATOR = (
         "pipeline_run_with_remote_orchestrator"
     )
+    PIPELINE_RUN_WITH_REMOTE_ARTIFACT_STORE = (
+        "pipeline_run_with_remote_artifact_store"
+    )
     PRODUCTION_SETUP_COMPLETED = "production_setup_completed"
 
 

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/src/zenml/logging/step_logging.py

@@ -263,7 +263,7 @@ class StepLogsStorage:
             formatted_message = (
                 f"[{timestamp} UTC] {remove_ansi_escape_codes(text)}"
             )
-            self.buffer.append(formatted_message)
+            self.buffer.append(formatted_message.rstrip())
             self.save_to_file()
 
     @property

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/src/zenml/materializers/__init__.py

@@ -26,14 +26,14 @@ from zenml.materializers.built_in_materializer import (
 from zenml.materializers.cloudpickle_materializer import (
     CloudpickleMaterializer,
 )
+from zenml.materializers.path_materializer import PathMaterializer
+from zenml.materializers.pydantic_materializer import PydanticMaterializer
+from zenml.materializers.service_materializer import ServiceMaterializer
 from zenml.materializers.structured_string_materializer import (
     StructuredStringMaterializer,
 )
-from zenml.materializers.pydantic_materializer import PydanticMaterializer
-from zenml.materializers.service_materializer import ServiceMaterializer
 from zenml.materializers.uuid_materializer import UUIDMaterializer
 
-
 __all__ = [
     "BuiltInContainerMaterializer",
     "BuiltInMaterializer",
@@ -43,4 +43,5 @@ __all__ = [
     "PydanticMaterializer",
     "ServiceMaterializer",
     "UUIDMaterializer",
+    "PathMaterializer",
 ]

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/src/zenml/materializers/built_in_materializer.py

@@ -107,6 +107,19 @@ class BuiltInMaterializer(BaseMaterializer):
             ensure_ascii=not ZENML_MATERIALIZER_ALLOW_NON_ASCII_JSON_DUMPS,
         )
 
+    def save_visualizations(
+        self, data: Union[bool, float, int, str]
+    ) -> Dict[str, VisualizationType]:
+        """Save visualizations for the given basic type.
+
+        Args:
+            data: The data to save visualizations for.
+
+        Returns:
+            A dictionary of visualization URIs and their types.
+        """
+        return {self.data_path.replace("\\", "/"): VisualizationType.JSON}
+
     def extract_metadata(
         self, data: Union[bool, float, int, str]
     ) -> Dict[str, "MetadataType"]:
@@ -165,6 +178,17 @@ class BytesMaterializer(BaseMaterializer):
         with self.artifact_store.open(self.data_path, "wb") as file_:
             file_.write(data)
 
+    def save_visualizations(self, data: bytes) -> Dict[str, VisualizationType]:
+        """Save visualizations for the given bytes data.
+
+        Args:
+            data: The bytes data to save visualizations for.
+
+        Returns:
+            A dictionary of visualization URIs and their types.
+        """
+        return {self.data_path.replace("\\", "/"): VisualizationType.MARKDOWN}
+
 
 def _all_serializable(iterable: Iterable[Any]) -> bool:
     """For an iterable, check whether all of its elements are JSON-serializable.

zenml_nightly-0.80.2.dev20250411/src/zenml/materializers/path_materializer.py

@@ -0,0 +1,119 @@
+#  Copyright (c) ZenML GmbH 2025. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at:
+#
+#       https://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+#  or implied. See the License for the specific language governing
+#  permissions and limitations under the License.
+"""Implementation of the materializer for Path objects."""
+
+import os
+import shutil
+import tarfile
+from pathlib import Path
+from typing import Any, ClassVar, Tuple, Type
+
+from zenml.constants import (
+    ENV_ZENML_DISABLE_PATH_MATERIALIZER,
+    handle_bool_env_var,
+)
+from zenml.enums import ArtifactType
+from zenml.io import fileio
+from zenml.materializers.base_materializer import BaseMaterializer
+
+
+class PathMaterializer(BaseMaterializer):
+    """Materializer for Path objects.
+
+    This materializer handles `pathlib.Path` objects by storing their contents
+    in a compressed tar archive within the artifact store if it's a directory,
+    or directly copying the file if it's a single file.
+    """
+
+    ASSOCIATED_TYPES: ClassVar[Tuple[Type[Any], ...]] = (Path,)
+    ASSOCIATED_ARTIFACT_TYPE: ClassVar[ArtifactType] = ArtifactType.DATA
+    ARCHIVE_NAME: ClassVar[str] = "data.tar.gz"
+    FILE_NAME: ClassVar[str] = "file_data"
+
+    # Skip registration if the environment variable is set
+    SKIP_REGISTRATION: ClassVar[bool] = handle_bool_env_var(
+        ENV_ZENML_DISABLE_PATH_MATERIALIZER, default=False
+    )
+
+    def load(self, data_type: Type[Any]) -> Any:
+        """Copy the artifact files to a local temp directory or file.
+
+        Args:
+            data_type: Unused.
+
+        Returns:
+            Path to the local directory or file that contains the artifact.
+
+        Raises:
+            FileNotFoundError: If the artifact is not found in the artifact store.
+        """
+        # Create a temporary directory that will persist until step execution ends
+        with self.get_temporary_directory(delete_at_exit=False) as directory:
+            # Check if we're loading a file or directory by looking for the archive
+            archive_path_remote = os.path.join(self.uri, self.ARCHIVE_NAME)
+            file_path_remote = os.path.join(self.uri, self.FILE_NAME)
+
+            if fileio.exists(archive_path_remote):
+                # This is a directory artifact
+                archive_path_local = os.path.join(directory, self.ARCHIVE_NAME)
+                fileio.copy(archive_path_remote, archive_path_local)
+
+                # Extract the archive to the temporary directory
+                with tarfile.open(archive_path_local, "r:gz") as tar:
+                    tar.extractall(path=directory)
+
+                # Clean up the archive file
+                os.remove(archive_path_local)
+                return Path(directory)
+            elif fileio.exists(file_path_remote):
+                # This is a single file artifact
+                file_path_local = os.path.join(
+                    directory, os.path.basename(file_path_remote)
+                )
+                fileio.copy(file_path_remote, file_path_local)
+                return Path(file_path_local)
+            else:
+                raise FileNotFoundError(
+                    f"Could not find artifact at {archive_path_remote} or {file_path_remote}"
+                )
+
+    def save(self, data: Any) -> None:
+        """Store the directory or file in the artifact store.
+
+        Args:
+            data: Path to a local directory or file to store. Must be a Path object.
+        """
+        assert isinstance(data, Path)
+
+        if data.is_dir():
+            # Handle directory artifact
+            with self.get_temporary_directory(
+                delete_at_exit=True
+            ) as directory:
+                archive_base = os.path.join(directory, "data")
+
+                # Create tar.gz archive - automatically uses relative paths
+                shutil.make_archive(
+                    base_name=archive_base, format="gztar", root_dir=str(data)
+                )
+
+                # Copy the archive to the artifact store
+                fileio.copy(
+                    f"{archive_base}.tar.gz",
+                    os.path.join(self.uri, self.ARCHIVE_NAME),
+                )
+        else:
+            # Handle single file artifact
+            file_path_remote = os.path.join(self.uri, self.FILE_NAME)
+            fileio.copy(str(data), file_path_remote)

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/src/zenml/orchestrators/step_launcher.py

@@ -154,8 +154,8 @@ class StepLauncher:
         if step_logging_enabled:
             # Configure the logs
             logs_uri = step_logging.prepare_logs_uri(
-                self._stack.artifact_store,
-                self._step.config.name,
+                artifact_store=self._stack.artifact_store,
+                step_name=self._step_name,
             )
 
             logs_context = step_logging.StepLogsStorageContext(

zenml_nightly-0.80.2.dev20250411/src/zenml/zen_stores/migrations/versions/0.80.2_release.py

@@ -0,0 +1,23 @@
+"""Release [0.80.2].
+
+Revision ID: 0.80.2
+Revises: 6611d4bcc95b
+Create Date: 2025-04-08 16:18:18.112107
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "0.80.2"
+down_revision = "6611d4bcc95b"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    """Upgrade database schema and/or data, creating a new revision."""
+    pass
+
+
+def downgrade() -> None:
+    """Downgrade database schema and/or data back to the previous revision."""
+    pass

{zenml_nightly-0.80.1.dev20250409 → zenml_nightly-0.80.2.dev20250411}/src/zenml/zen_stores/sql_zen_store.py

@@ -7635,6 +7635,17 @@ class SqlZenStore(BaseZenStore):
                                 },
                                 session=session,
                             )
+                    if (
+                        defined_component.type
+                        == StackComponentType.ARTIFACT_STORE
+                    ):
+                        if defined_component.flavor != "local":
+                            self._update_onboarding_state(
+                                completed_steps={
+                                    OnboardingStep.STACK_WITH_REMOTE_ARTIFACT_STORE_CREATED
+                                },
+                                session=session,
+                            )
 
                 return new_stack_schema.to_model(
                     include_metadata=True, include_resources=True
@@ -8466,6 +8477,12 @@ class SqlZenStore(BaseZenStore):
                     completed_onboarding_steps.update(
                         {
                             OnboardingStep.PIPELINE_RUN_WITH_REMOTE_ORCHESTRATOR,
+                        }
+                    )
+                if stack_metadata["artifact_store"] != "local":
+                    completed_onboarding_steps.update(
+                        {
+                            OnboardingStep.PIPELINE_RUN_WITH_REMOTE_ARTIFACT_STORE,
                             OnboardingStep.PRODUCTION_SETUP_COMPLETED,
                         }
                     )

zenml_nightly-0.80.1.dev20250409/src/zenml/VERSION

@@ -1 +0,0 @@
-0.80.1.dev20250409