zenml-nightly 0.63.0.dev20240801__py3-none-any.whl → 0.64.0.dev20240811__py3-none-any.whl

README.md

@@ -1,6 +1,6 @@
 <div align="center">
   <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=0fcbab94-8fbe-4a38-93e8-c2348450a42e" />
-  <h1 align="center">Create an internal MLOps platform for your entire machine learning team.
+  <h1 align="center">Connecting data science teams seamlessly to cloud infrastructure.
 </h1>
 </div>
 
@@ -327,7 +327,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.63.0 is out. Check out the release notes
+    🎉 Version 0.64.0 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

RELEASE_NOTES.md

@@ -1,5 +1,84 @@
 <!-- markdown-link-check-disable -->
 
+# 0.64.0
+
+## New Features and Improvements
+
+### Notebook Integration
+
+ZenML now supports running steps defined in notebook cells with remote 
+orchestrators and step operators. This feature enhances the development 
+workflow by allowing seamless transition from experimentation to production.
+
+- **Details**: [Running remote pipelines from notebooks](https://docs.zenml.io/v/docs/how-to/run-remote-pipelines-from-notebooks)
+
+### Reduced Docker Builds with Code Uploads
+
+We've introduced an option to upload code to the artifact store, enabling 
+Docker build reuse. This feature can significantly speed up iteration, 
+especially when working with remote stacks.
+
+- **Default**: Enabled
+- **Configuration**: To disable, set `DockerSettings.allow_download_from_artifact_store=False` 
+for steps or pipelines
+- **Benefits**: 
+  - Faster development cycles
+  - No need to register a code repository to reuse builds
+  - Builds only occur when requirements or DockerSettings change
+- **Documentation**: [Which files are built into the image](https://docs.zenml.io/how-to/customize-docker-builds/which-files-are-built-into-the-image)
+
+### AzureML Orchestrator Support
+
+ZenML now supports [AzureML](https://azure.microsoft.com/en-gb/free/machine-learning) 
+as an orchestrator, expanding our list of supported cloud platforms.
+
+- **Full Azure Guide**: [Setting up an Azure stack](https://docs.zenml.io/how-to/popular-integrations/azure-guide)
+- **Documentation**: [AzureML orchestrator](https://docs.zenml.io/stack-components/orchestrators/azureml)
+
+### Terraform Modules
+
+We've released new Terraform modules on the Hashicorp registry for 
+provisioning complete MLOps stacks across major cloud providers.
+
+- **Features**: 
+  - Automate infrastructure setup for ZenML stack deployment
+  - Handle registration of configurations to ZenML server
+- **More Information**: [MLOps Terraform ZenML blog post](https://www.zenml.io/blog/mlops-terraform-zenml)
+
+These updates aim to streamline the MLOps workflow, making it easier to 
+develop, deploy, and manage machine learning pipelines with ZenML.
+
+## What's Changed
+
+* Add 0.63.0 to migration testing by @bcdurak in https://github.com/zenml-io/zenml/pull/2893
+* Document terraform stack deployment modules by @stefannica in https://github.com/zenml-io/zenml/pull/2898
+* README update by @htahir1 in https://github.com/zenml-io/zenml/pull/2901
+* Enable `Databricks` Unity Catalog for MLflow by @safoinme in https://github.com/zenml-io/zenml/pull/2900
+* Make urls pop out from the sea of purple/cyan in the logs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2894
+* Add terraform as a supported stack deployment provider by @stefannica in https://github.com/zenml-io/zenml/pull/2902
+* Fix `Model` imports in docs by @strickvl in https://github.com/zenml-io/zenml/pull/2907
+* Remove hub references by @schustmi in https://github.com/zenml-io/zenml/pull/2905
+* Bump NLP template by @avishniakov in https://github.com/zenml-io/zenml/pull/2912
+* Updated step operator docs by @htahir1 in https://github.com/zenml-io/zenml/pull/2908
+* Added lightning studio check by @htahir1 in https://github.com/zenml-io/zenml/pull/2910
+* Upload code to artifact store by @schustmi in https://github.com/zenml-io/zenml/pull/2895
+* AzureML orchestrator by @bcdurak in https://github.com/zenml-io/zenml/pull/2873
+* Run steps defined in notebooks with remote orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/2899
+* Fix broken / unparsable md docs file by @strickvl in https://github.com/zenml-io/zenml/pull/2916
+* Bump mlflow to 2.15.0 by @christianversloot in https://github.com/zenml-io/zenml/pull/2896
+* Remove extra button by @schustmi in https://github.com/zenml-io/zenml/pull/2918
+* Added last timestamp to zenserver by @htahir1 in https://github.com/zenml-io/zenml/pull/2913
+* A pipeline can't finish successfully in this case by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2903
+* Fix the GCP Workload Identity Federation support in the GCP Service Connector by @stefannica in https://github.com/zenml-io/zenml/pull/2914
+* Embeddings finetuning guide for LLMOps guide by @strickvl in https://github.com/zenml-io/zenml/pull/2917
+
+## 🥳 Community Contributions 🥳
+
+We'd like to give a special thanks to @christianversloot who contributed to 
+this release by bumping the `mlflow` version to 2.15.0
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.63.0...0.64.0
+
 # 0.63.0
 
 Moving forward from the last two releases, we have further improved the 

zenml/VERSION

@@ -1 +1 @@
-0.63.0.dev20240801
+0.64.0.dev20240811

zenml/__init__.py

@@ -27,10 +27,6 @@ from zenml.logger import init_logging  # noqa
 
 init_logging()
 
-# The following code is needed for `zenml.hub` subpackages to be found
-from pkgutil import extend_path
-
-__path__ = extend_path(__path__, __name__)
 
 # Need to import zenml.models before zenml.config to avoid circular imports
 from zenml.models import *  # noqa: F401

zenml/analytics/enums.py

@@ -99,11 +99,5 @@ class AnalyticsEvent(str, Enum):
     ZENML_SERVER_DEPLOYED = "ZenML server deployed"
     ZENML_SERVER_DESTROYED = "ZenML server destroyed"
 
-    # ZenML Hub events
-    ZENML_HUB_PLUGIN_INSTALL = "ZenML Hub plugin installed"
-    ZENML_HUB_PLUGIN_UNINSTALL = "ZenML Hub plugin uninstalled"
-    ZENML_HUB_PLUGIN_CLONE = "ZenML Hub plugin pulled"
-    ZENML_HUB_PLUGIN_SUBMIT = "ZenML Hub plugin pushed"
-
     # Server Settings
     SERVER_SETTINGS_UPDATED = "Server Settings Updated"

zenml/cli/__init__.py

@@ -2541,66 +2541,6 @@ zenml artifact-store deploy -f gcp -p gcp -r us-east1 -x project_id=zenml-core b
 
 For full documentation on this functionality, please refer to [the dedicated
 documentation on stack component deploy](https://docs.zenml.io/how-to/stack-deployment/deploy-a-stack-component).
-
-Interacting with the ZenML Hub
-------------------------------
-
-The ZenML Hub is a central location for discovering and sharing third-party 
-ZenML code, such as custom integrations, components, steps, pipelines, 
-materializers, and more. 
-You can browse the ZenML Hub at [https://hub.zenml.io](https://hub.zenml.io).
-
-The ZenML CLI provides various commands to interact with the ZenML Hub:
-
-- Listing all plugins available on the Hub:
-```bash
-zenml hub list
-```
-
-- Installing a Hub plugin:
-```bash
-zenml hub install
-```
-Installed plugins can be imported via `from zenml.hub.<plugin_name> import ...`. 
-
-
-- Uninstalling a Hub plugin:
-```bash
-zenml hub uninstall
-```
-
-- Cloning the source code of a Hub plugin (without installing it):
-```bash
-zenml hub clone
-```
-This is useful, e.g., for extending an existing plugin or for getting the 
-examples of a plugin.
-
-- Submitting/contributing a plugin to the Hub (requires login, see below):
-```bash
-zenml hub submit
-```
-If you are unsure about which arguments you need to set, you can run the
-command in interactive mode:
-```bash
-zenml hub submit --interactive
-```
-This will ask for and validate inputs one at a time.
-
-- Logging in to the Hub:
-```bash
-zenml hub login
-```
-
-- Logging out of the Hub:
-```bash
-zenml hub logout
-```
-
-- Viewing the build logs of a plugin you submitted to the Hub:
-```bash
-zenml hub logs
-```
 """
 
 from zenml.cli.version import *  # noqa
@@ -2612,7 +2552,6 @@ from zenml.cli.code_repository import *  # noqa
 from zenml.cli.config import *  # noqa
 from zenml.cli.downgrade import *  # noqa
 from zenml.cli.feature import *  # noqa
-from zenml.cli.hub import *  # noqa
 from zenml.cli.integration import *  # noqa
 from zenml.cli.model import *  # noqa
 from zenml.cli.model_registry import *  # noqa

zenml/cli/base.py

@@ -87,7 +87,7 @@ ZENML_PROJECT_TEMPLATES = dict(
     ),
     nlp=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-nlp",
-        github_tag="2024.06.14",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
+        github_tag="2024.08.07",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
     ),
     llm_finetuning=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-llm-finetuning",

zenml/cli/web_login.py

@@ -28,6 +28,7 @@ from zenml.constants import (
     DEVICE_AUTHORIZATION,
     LOGIN,
     VERSION_1,
+    ZENML_PRO_CONNECTION_ISSUES_SUSPENDED_PAUSED_TENANT_HINT,
 )
 from zenml.exceptions import AuthorizationException, OAuthError
 from zenml.logger import get_logger
@@ -93,6 +94,11 @@ def web_login(url: str, verify_ssl: Union[str, bool]) -> str:
     # Get rid of any trailing slashes to prevent issues when having double
     # slashes in the URL
     url = url.rstrip("/")
+    zenml_pro_extra = ""
+    if ".zenml.io" in url:
+        zenml_pro_extra = (
+            ZENML_PRO_CONNECTION_ISSUES_SUSPENDED_PAUSED_TENANT_HINT
+        )
     try:
         auth_url = url + API + VERSION_1 + DEVICE_AUTHORIZATION
         response = requests.post(
@@ -111,6 +117,7 @@ def web_login(url: str, verify_ssl: Union[str, bool]) -> str:
             logger.info(f"Error: {response.status_code} {response.text}")
             raise AuthorizationException(
                 "Could not connect to API server. Please check the URL."
+                + zenml_pro_extra
             )
     except (requests.exceptions.JSONDecodeError, ValueError, TypeError):
         logger.exception("Bad response received from API server.")
@@ -121,6 +128,7 @@ def web_login(url: str, verify_ssl: Union[str, bool]) -> str:
         logger.exception("Could not connect to API server.")
         raise AuthorizationException(
             "Could not connect to API server. Please check the URL."
+            + zenml_pro_extra
         )
 
     # Open the verification URL in the user's browser

zenml/client.py

@@ -868,7 +868,6 @@ class Client(metaclass=ClientMetaClass):
         updated_full_name: Optional[str] = None,
         updated_email: Optional[str] = None,
         updated_email_opt_in: Optional[bool] = None,
-        updated_hub_token: Optional[str] = None,
         updated_password: Optional[str] = None,
         old_password: Optional[str] = None,
         updated_is_admin: Optional[bool] = None,
@@ -883,7 +882,6 @@ class Client(metaclass=ClientMetaClass):
             updated_full_name: The new full name of the user.
             updated_email: The new email of the user.
             updated_email_opt_in: The new email opt-in status of the user.
-            updated_hub_token: Update the hub token
             updated_password: The new password of the user.
             old_password: The old password of the user. Required for password
                 update.
@@ -911,8 +909,6 @@ class Client(metaclass=ClientMetaClass):
             )
         if updated_email_opt_in is not None:
             user_update.email_opted_in = updated_email_opt_in
-        if updated_hub_token is not None:
-            user_update.hub_token = updated_hub_token
         if updated_password is not None:
             user_update.password = updated_password
             if old_password is None:

zenml/config/build_configuration.py

@@ -14,11 +14,13 @@
 """Build configuration class."""
 
 import hashlib
+import json
 from typing import TYPE_CHECKING, Dict, Optional
 
 from pydantic import BaseModel
 
-from zenml.config.docker_settings import DockerSettings, SourceFileMode
+from zenml.config.docker_settings import DockerSettings
+from zenml.utils import json_utils
 
 if TYPE_CHECKING:
     from zenml.code_repositories import BaseCodeRepository
@@ -60,7 +62,14 @@ class BuildConfiguration(BaseModel):
             The checksum.
         """
         hash_ = hashlib.md5()  # nosec
-        hash_.update(self.settings.model_dump_json().encode())
+        settings_json = json.dumps(
+            self.settings.model_dump(
+                mode="json", exclude={"prevent_build_reuse"}
+            ),
+            sort_keys=True,
+            default=json_utils.pydantic_encoder,
+        )
+        hash_.update(settings_json.encode())
         if self.entrypoint:
             hash_.update(self.entrypoint.encode())
 
@@ -72,7 +81,7 @@ class BuildConfiguration(BaseModel):
             PipelineDockerImageBuilder,
         )
 
-        pass_code_repo = self.should_download_files(
+        pass_code_repo = self.should_download_files_from_code_repository(
             code_repository=code_repository
         )
         requirements_files = (
@@ -101,34 +110,51 @@ class BuildConfiguration(BaseModel):
         Returns:
             Whether files should be included in the image.
         """
-        if self.settings.source_files == SourceFileMode.INCLUDE:
-            return True
+        if self.should_download_files(code_repository=code_repository):
+            return False
 
-        if (
-            self.settings.source_files == SourceFileMode.DOWNLOAD_OR_INCLUDE
-            and not code_repository
+        return self.settings.allow_including_files_in_images
+
+    def should_download_files(
+        self,
+        code_repository: Optional["BaseCodeRepository"],
+    ) -> bool:
+        """Whether files should be downloaded in the image.
+
+        Args:
+            code_repository: Code repository that can be used to download files
+                inside the image.
+
+        Returns:
+            Whether files should be downloaded in the image.
+        """
+        if self.should_download_files_from_code_repository(
+            code_repository=code_repository
         ):
             return True
 
+        if self.settings.allow_download_from_artifact_store:
+            return True
+
         return False
 
-    def should_download_files(
+    def should_download_files_from_code_repository(
         self,
         code_repository: Optional["BaseCodeRepository"],
     ) -> bool:
-        """Whether files should be downloaded in the image.
+        """Whether files should be downloaded from the code repository.
 
         Args:
             code_repository: Code repository that can be used to download files
                 inside the image.
 
         Returns:
-            Whether files should be downloaded in the image.
+            Whether files should be downloaded from the code repository.
         """
-        if not code_repository:
-            return False
+        if (
+            code_repository
+            and self.settings.allow_download_from_code_repository
+        ):
+            return True
 
-        return self.settings.source_files in {
-            SourceFileMode.DOWNLOAD,
-            SourceFileMode.DOWNLOAD_OR_INCLUDE,
-        }
+        return False

zenml/config/docker_settings.py

@@ -16,8 +16,7 @@
 from enum import Enum
 from typing import Any, Dict, List, Optional, Union
 
-from pydantic import BaseModel, Field, model_validator
-from pydantic_settings import SettingsConfigDict
+from pydantic import BaseModel, ConfigDict, Field, model_validator
 
 from zenml.config.base_settings import BaseSettings
 from zenml.logger import get_logger
@@ -49,15 +48,6 @@ class PythonEnvironmentExportMethod(Enum):
         }[self]
 
 
-class SourceFileMode(Enum):
-    """Different methods to handle source files in Docker images."""
-
-    INCLUDE = "include"
-    DOWNLOAD_OR_INCLUDE = "download_or_include"
-    DOWNLOAD = "download"
-    IGNORE = "ignore"
-
-
 class PythonPackageInstaller(Enum):
     """Different installers for python packages."""
 
@@ -102,7 +92,6 @@ class DockerSettings(BaseSettings):
     Depending on the configuration of this object, requirements will be
     installed in the following order (each step optional):
     - The packages installed in your local python environment
-    - The packages specified via the `required_hub_plugins` attribute
     - The packages required by the stack unless this is disabled by setting
       `install_stack_requirements=False`.
     - The packages specified via the `required_integrations`
@@ -135,10 +124,9 @@ class DockerSettings(BaseSettings):
             when the `dockerfile` attribute is set. If this is left empty, the
             build context will only contain the Dockerfile.
         parent_image_build_config: Configuration for the parent image build.
-        build_options: DEPRECATED, use parent_image_build_config.build_options
-            instead.
         skip_build: If set to `True`, the parent image will be used directly to
             run the steps of your pipeline.
+        prevent_build_reuse: Prevent the reuse of an existing build.
         target_repository: Name of the Docker repository to which the
             image should be pushed. This repository will be appended to the
             registry URI of the container registry of your stack and should
@@ -161,11 +149,7 @@ class DockerSettings(BaseSettings):
         required_integrations: List of ZenML integrations that should be
             installed. All requirements for the specified integrations will
             be installed inside the Docker image.
-        required_hub_plugins: List of ZenML Hub plugins to install.
-            Expected format: '(<author_username>/)<plugin_name>==<version>'.
-            If no version is specified, the latest version is taken. The
-            packages of required plugins and all their dependencies will be
-            installed inside the Docker image.
+        required_hub_plugins: DEPRECATED/UNUSED.
         install_stack_requirements: If `True`, ZenML will automatically detect
             if components of your active stack are part of a ZenML integration
             and install the corresponding requirements and apt packages.
@@ -176,33 +160,32 @@ class DockerSettings(BaseSettings):
         environment: Dictionary of environment variables to set inside the
             Docker image.
         build_config: Configuration for the main image build.
-        dockerignore: DEPRECATED, use build_config.dockerignore instead.
-        copy_files: DEPRECATED, use the `source_files` attribute instead.
-        copy_global_config: DEPRECATED/UNUSED.
         user: If not `None`, will set the user, make it owner of the `/app`
             directory which contains all the user code and run the container
             entrypoint as this user.
-        source_files: Defines how the user source files will be handled when
-            building the Docker image.
-            * INCLUDE: The files will be included in the Docker image.
-            * DOWNLOAD: The files will be downloaded when running the image. If
-              this is specified, the files must be inside a registered code
-              repository and the repository must have no local changes,
-              otherwise the build will fail.
-            * DOWNLOAD_OR_INCLUDE: The files will be downloaded if they're
-              inside a registered code repository and the repository has no
-              local changes, otherwise they will be included in the image.
-            * IGNORE: The files will not be included or downloaded in the image.
-              If you use this option, you're responsible that all the files
-              to run your steps exist in the right place.
+        allow_including_files_in_images: If `True`, code can be included in the
+            Docker images if code download from a code repository or artifact
+            store is disabled or not possible.
+        allow_download_from_code_repository: If `True`, code can be downloaded
+            from a code repository if possible.
+        allow_download_from_artifact_store: If `True`, code can be downloaded
+            from the artifact store.
+        build_options: DEPRECATED, use parent_image_build_config.build_options
+            instead.
+        dockerignore: DEPRECATED, use build_config.dockerignore instead.
+        copy_files: DEPRECATED/UNUSED.
+        copy_global_config: DEPRECATED/UNUSED.
+        source_files: DEPRECATED. Use allow_including_files_in_images,
+            allow_download_from_code_repository and
+            allow_download_from_artifact_store instead.
     """
 
     parent_image: Optional[str] = None
     dockerfile: Optional[str] = None
     build_context_root: Optional[str] = None
-    build_options: Dict[str, Any] = {}
     parent_image_build_config: Optional[DockerBuildConfig] = None
     skip_build: bool = False
+    prevent_build_reuse: bool = False
     target_repository: Optional[str] = None
     python_package_installer: PythonPackageInstaller = (
         PythonPackageInstaller.PIP
@@ -215,49 +198,89 @@ class DockerSettings(BaseSettings):
         default=None, union_mode="left_to_right"
     )
     required_integrations: List[str] = []
-    required_hub_plugins: List[str] = []
     install_stack_requirements: bool = True
     apt_packages: List[str] = []
     environment: Dict[str, Any] = {}
-    dockerignore: Optional[str] = None
-    copy_files: bool = True
-    copy_global_config: bool = True
     user: Optional[str] = None
     build_config: Optional[DockerBuildConfig] = None
 
-    source_files: SourceFileMode = SourceFileMode.DOWNLOAD_OR_INCLUDE
+    allow_including_files_in_images: bool = True
+    allow_download_from_code_repository: bool = True
+    allow_download_from_artifact_store: bool = True
+
+    # Deprecated attributes
+    build_options: Dict[str, Any] = {}
+    dockerignore: Optional[str] = None
+    copy_files: bool = True
+    copy_global_config: bool = True
+    source_files: Optional[str] = None
+    required_hub_plugins: List[str] = []
 
     _deprecation_validator = deprecation_utils.deprecate_pydantic_attributes(
-        "copy_files", "copy_global_config"
+        "copy_files",
+        "copy_global_config",
+        "source_files",
+        "required_hub_plugins",
     )
 
     @model_validator(mode="before")
     @classmethod
     @before_validator_handler
-    def _migrate_copy_files(cls, data: Dict[str, Any]) -> Dict[str, Any]:
-        """Migrates the value from the old copy_files attribute.
+    def _migrate_source_files(cls, data: Dict[str, Any]) -> Dict[str, Any]:
+        """Migrate old source_files values.
 
         Args:
-            data: The settings values.
+            data: The model data.
+
+        Raises:
+            ValueError: If an invalid source file mode is specified.
 
         Returns:
-            The migrated settings values.
+            The migrated data.
         """
-        copy_files = data.get("copy_files", None)
+        source_files = data.get("source_files", None)
 
-        if copy_files is None:
+        if source_files is None:
             return data
 
-        if data.get("source_files", None):
-            # Ignore the copy files value in favor of the new source files
+        replacement_attributes = [
+            "allow_including_files_in_images",
+            "allow_download_from_code_repository",
+            "allow_download_from_artifact_store",
+        ]
+        if any(v in data for v in replacement_attributes):
             logger.warning(
-                "Both `copy_files` and `source_files` specified for the "
-                "DockerSettings, ignoring the `copy_files` value."
+                "Both `source_files` and one of %s specified for the "
+                "DockerSettings, ignoring the `source_files` value.",
+                replacement_attributes,
             )
-        elif copy_files is True:
-            data["source_files"] = SourceFileMode.INCLUDE
-        elif copy_files is False:
-            data["source_files"] = SourceFileMode.IGNORE
+            return data
+
+        allow_including_files_in_images = False
+        allow_download_from_code_repository = False
+        allow_download_from_artifact_store = False
+
+        if source_files == "download":
+            allow_download_from_code_repository = True
+        elif source_files == "include":
+            allow_including_files_in_images = True
+        elif source_files == "download_or_include":
+            allow_including_files_in_images = True
+            allow_download_from_code_repository = True
+        elif source_files == "ignore":
+            pass
+        else:
+            raise ValueError(f"Invalid source file mode `{source_files}`.")
+
+        data["allow_including_files_in_images"] = (
+            allow_including_files_in_images
+        )
+        data["allow_download_from_code_repository"] = (
+            allow_download_from_code_repository
+        )
+        data["allow_download_from_artifact_store"] = (
+            allow_download_from_artifact_store
+        )
 
         return data
 
@@ -282,7 +305,7 @@ class DockerSettings(BaseSettings):
 
         return self
 
-    model_config = SettingsConfigDict(
+    model_config = ConfigDict(
         # public attributes are immutable
         frozen=True,
         # prevent extra attributes during model initialization