zenml-nightly 0.62.0.dev20240729__py3-none-any.whl → 0.64.0.dev20240809__py3-none-any.whl

README.md

@@ -1,6 +1,6 @@
 <div align="center">
   <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=0fcbab94-8fbe-4a38-93e8-c2348450a42e" />
-  <h1 align="center">Create an internal MLOps platform for your entire machine learning team.
+  <h1 align="center">Connecting data science teams seamlessly to cloud infrastructure.
 </h1>
 </div>
 
@@ -327,7 +327,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.62.0 is out. Check out the release notes
+    🎉 Version 0.64.0 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

RELEASE_NOTES.md

@@ -1,4 +1,124 @@
 <!-- markdown-link-check-disable -->
+
+# 0.64.0
+
+## New Features and Improvements
+
+### Notebook Integration
+
+ZenML now supports running steps defined in notebook cells with remote 
+orchestrators and step operators. This feature enhances the development 
+workflow by allowing seamless transition from experimentation to production.
+
+- **Details**: [Running remote pipelines from notebooks](https://docs.zenml.io/v/docs/how-to/run-remote-pipelines-from-notebooks)
+
+### Reduced Docker Builds with Code Uploads
+
+We've introduced an option to upload code to the artifact store, enabling 
+Docker build reuse. This feature can significantly speed up iteration, 
+especially when working with remote stacks.
+
+- **Default**: Enabled
+- **Configuration**: To disable, set `DockerSettings.allow_download_from_artifact_store=False` 
+for steps or pipelines
+- **Benefits**: 
+  - Faster development cycles
+  - No need to register a code repository to reuse builds
+  - Builds only occur when requirements or DockerSettings change
+- **Documentation**: [Which files are built into the image](https://docs.zenml.io/how-to/customize-docker-builds/which-files-are-built-into-the-image)
+
+### AzureML Orchestrator Support
+
+ZenML now supports [AzureML](https://azure.microsoft.com/en-gb/free/machine-learning) 
+as an orchestrator, expanding our list of supported cloud platforms.
+
+- **Full Azure Guide**: [Setting up an Azure stack](https://docs.zenml.io/how-to/popular-integrations/azure-guide)
+- **Documentation**: [AzureML orchestrator](https://docs.zenml.io/stack-components/orchestrators/azureml)
+
+### Terraform Modules
+
+We've released new Terraform modules on the Hashicorp registry for 
+provisioning complete MLOps stacks across major cloud providers.
+
+- **Features**: 
+  - Automate infrastructure setup for ZenML stack deployment
+  - Handle registration of configurations to ZenML server
+- **More Information**: [MLOps Terraform ZenML blog post](https://www.zenml.io/blog/mlops-terraform-zenml)
+
+These updates aim to streamline the MLOps workflow, making it easier to 
+develop, deploy, and manage machine learning pipelines with ZenML.
+
+## What's Changed
+
+* Add 0.63.0 to migration testing by @bcdurak in https://github.com/zenml-io/zenml/pull/2893
+* Document terraform stack deployment modules by @stefannica in https://github.com/zenml-io/zenml/pull/2898
+* README update by @htahir1 in https://github.com/zenml-io/zenml/pull/2901
+* Enable `Databricks` Unity Catalog for MLflow by @safoinme in https://github.com/zenml-io/zenml/pull/2900
+* Make urls pop out from the sea of purple/cyan in the logs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2894
+* Add terraform as a supported stack deployment provider by @stefannica in https://github.com/zenml-io/zenml/pull/2902
+* Fix `Model` imports in docs by @strickvl in https://github.com/zenml-io/zenml/pull/2907
+* Remove hub references by @schustmi in https://github.com/zenml-io/zenml/pull/2905
+* Bump NLP template by @avishniakov in https://github.com/zenml-io/zenml/pull/2912
+* Updated step operator docs by @htahir1 in https://github.com/zenml-io/zenml/pull/2908
+* Added lightning studio check by @htahir1 in https://github.com/zenml-io/zenml/pull/2910
+* Upload code to artifact store by @schustmi in https://github.com/zenml-io/zenml/pull/2895
+* AzureML orchestrator by @bcdurak in https://github.com/zenml-io/zenml/pull/2873
+* Run steps defined in notebooks with remote orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/2899
+* Fix broken / unparsable md docs file by @strickvl in https://github.com/zenml-io/zenml/pull/2916
+* Bump mlflow to 2.15.0 by @christianversloot in https://github.com/zenml-io/zenml/pull/2896
+* Remove extra button by @schustmi in https://github.com/zenml-io/zenml/pull/2918
+* Added last timestamp to zenserver by @htahir1 in https://github.com/zenml-io/zenml/pull/2913
+* A pipeline can't finish successfully in this case by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2903
+* Fix the GCP Workload Identity Federation support in the GCP Service Connector by @stefannica in https://github.com/zenml-io/zenml/pull/2914
+* Embeddings finetuning guide for LLMOps guide by @strickvl in https://github.com/zenml-io/zenml/pull/2917
+
+## 🥳 Community Contributions 🥳
+
+We'd like to give a special thanks to @christianversloot who contributed to 
+this release by bumping the `mlflow` version to 2.15.0
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.63.0...0.64.0
+
+# 0.63.0
+
+Moving forward from the last two releases, we have further improved the 
+1-click deployment tool and the stack wizard by adding support for Azure.
+
+Moreover, we implemented a new step operator that allows you to run individual 
+steps of your pipeline in Kubernetes pods.
+
+Lastly, we have simplified our pipeline models by removing their versions.
+
+## What's Changed
+* Enable cloud build service in GCP stack deployment by @stefannica in https://github.com/zenml-io/zenml/pull/2864
+* Adding a `logo_url` and the of the `integration` to component responses by @bcdurak in https://github.com/zenml-io/zenml/pull/2866
+* Use REST in Model tests by @avishniakov in https://github.com/zenml-io/zenml/pull/2834
+* Add Azure stack wizard by @avishniakov in https://github.com/zenml-io/zenml/pull/2841
+* Migration testing for 0.62.0 by @schustmi in https://github.com/zenml-io/zenml/pull/2860
+* Fix RBAC in combination with lazy loaders by @schustmi in https://github.com/zenml-io/zenml/pull/2869
+* Misc cleanup after release by @schustmi in https://github.com/zenml-io/zenml/pull/2861
+* Disable notebook error for Kubernetes orchestrator by @strickvl in https://github.com/zenml-io/zenml/pull/2870
+* Added ability to add labels to k8s pod by @htahir1 in https://github.com/zenml-io/zenml/pull/2872
+* Fix zenml pro links by @schustmi in https://github.com/zenml-io/zenml/pull/2875
+* Fix mlstacks docs typo by @begoechavarren in https://github.com/zenml-io/zenml/pull/2878
+* Fix requests vulnerability by @stefannica in https://github.com/zenml-io/zenml/pull/2843
+* Fixed some minor docs things i noticed by @htahir1 in https://github.com/zenml-io/zenml/pull/2881
+* Serialize source as Any to keep subclass attributes by @schustmi in https://github.com/zenml-io/zenml/pull/2880
+* Fix node selectors for Vertex orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/2876
+* Kubernetes step operator by @schustmi in https://github.com/zenml-io/zenml/pull/2883
+* Automatically populate GCP/azure path when using wizard from the frontend by @schustmi in https://github.com/zenml-io/zenml/pull/2886
+* Remove pipeline versioning and add run templates by @schustmi in https://github.com/zenml-io/zenml/pull/2830
+* Implement the Azure 1-click stack deployment by @stefannica in https://github.com/zenml-io/zenml/pull/2887
+* Better error message sagemaker, better documentation server env vars by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2885
+* Azure Stack Wizard docs by @bcdurak in https://github.com/zenml-io/zenml/pull/2890
+* Docs update mlflow deploy function call by @safoinme in https://github.com/zenml-io/zenml/pull/2863
+* Fix databricks resource setting by @safoinme in https://github.com/zenml-io/zenml/pull/2889
+
+## New Contributors
+* @begoechavarren made their first contribution in https://github.com/zenml-io/zenml/pull/2878
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.62.0...0.63.0
+
 # 0.62.0
 
 Building on top of the last release, this release adds a new and easy way to deploy a GCP ZenML stack from the dashboard and the CLI. Give it a try by going to the `Stacks` section in the dashboard or running the `zenml stack deploy` command! For more information on this new feature, please do check out [the video and blog](https://www.zenml.io/blog/easy-mlops-pipelines) from our previous release.

zenml/VERSION

@@ -1 +1 @@
-0.62.0.dev20240729
+0.64.0.dev20240809

zenml/__init__.py

@@ -27,10 +27,6 @@ from zenml.logger import init_logging  # noqa
 
 init_logging()
 
-# The following code is needed for `zenml.hub` subpackages to be found
-from pkgutil import extend_path
-
-__path__ = extend_path(__path__, __name__)
 
 # Need to import zenml.models before zenml.config to avoid circular imports
 from zenml.models import *  # noqa: F401

zenml/actions/pipeline_run/pipeline_run_action.py

@@ -33,8 +33,8 @@ from zenml.models import (
 )
 from zenml.models.v2.base.base import BaseResponse
 from zenml.zen_server.auth import AuthContext
-from zenml.zen_server.pipeline_deployment.utils import run_pipeline
 from zenml.zen_server.rbac.models import ResourceType
+from zenml.zen_server.template_execution.utils import run_template
 from zenml.zen_server.utils import server_config
 
 logger = get_logger(__name__)
@@ -45,7 +45,7 @@ logger = get_logger(__name__)
 class PipelineRunActionConfiguration(ActionConfig):
     """Configuration class to configure a pipeline run action."""
 
-    pipeline_deployment_id: UUID
+    template_id: UUID
     run_config: Optional[PipelineRunConfiguration] = None
 
 
@@ -95,10 +95,10 @@ class PipelineRunActionHandler(BaseActionHandler):
 
         assert isinstance(config, PipelineRunActionConfiguration)
 
-        deployment = zen_store().get_deployment(config.pipeline_deployment_id)
-        logger.debug("Running deployment:", deployment)
-        run_pipeline(
-            deployment=deployment,
+        template = zen_store().get_run_template(config.template_id)
+        logger.debug("Running template:", template)
+        run_template(
+            template=template,
             run_config=config.run_config,
             auth_context=auth_context,
         )
@@ -114,13 +114,14 @@ class PipelineRunActionHandler(BaseActionHandler):
         Raises:
             ValueError: In case no deployment can be found with the deployment_id
         """
-        deployment_id = config.pipeline_deployment_id
         zen_store = GlobalConfiguration().zen_store
 
         try:
-            zen_store.get_deployment(deployment_id=deployment_id)
+            zen_store.get_run_template(template_id=config.template_id)
         except KeyError:
-            raise ValueError(f"No deployment found with id {deployment_id}.")
+            raise ValueError(
+                f"No template found with id {config.template_id}."
+            )
 
     def _validate_action_request(
         self, action: ActionRequest, config: ActionConfig
@@ -176,27 +177,28 @@ class PipelineRunActionHandler(BaseActionHandler):
             List of resources related to the action.
 
         Raises:
-            ValueError: In case the deployment_id does not exist.
+            ValueError: In case the specified template does not exist.
         """
         assert isinstance(action_config, PipelineRunActionConfiguration)
 
-        deployment_id = action_config.pipeline_deployment_id
         zen_store = GlobalConfiguration().zen_store
 
         try:
-            deployment = zen_store.get_deployment(
-                deployment_id=deployment_id, hydrate=hydrate
+            template = zen_store.get_run_template(
+                template_id=action_config.template_id, hydrate=hydrate
             )
         except KeyError:
-            raise ValueError(f"No deployment found with id {deployment_id}.")
+            raise ValueError(
+                f"No template found with id {action_config.template_id}."
+            )
 
         resources: Dict[ResourceType, BaseResponse[Any, Any, Any]] = {
-            ResourceType.PIPELINE_DEPLOYMENT: deployment
+            ResourceType.RUN_TEMPLATE: template
         }
 
-        if deployment.pipeline is not None:
+        if template.pipeline is not None:
             pipeline = zen_store.get_pipeline(
-                pipeline_id=deployment.pipeline.id, hydrate=hydrate
+                pipeline_id=template.pipeline.id, hydrate=hydrate
             )
             resources[ResourceType.PIPELINE] = pipeline
 

zenml/analytics/enums.py

@@ -48,6 +48,10 @@ class AnalyticsEvent(str, Enum):
     CREATED_TRIGGER = "Trigger created"
     UPDATED_TRIGGER = "Trigger updated"
 
+    # Templates
+    CREATED_RUN_TEMPLATE = "Run template created"
+    EXECUTED_RUN_TEMPLATE = "Run templated executed"
+
     # Model Control Plane
     MODEL_DEPLOYED = "Model deployed"
     CREATED_MODEL = "Model created"
@@ -95,11 +99,5 @@ class AnalyticsEvent(str, Enum):
     ZENML_SERVER_DEPLOYED = "ZenML server deployed"
     ZENML_SERVER_DESTROYED = "ZenML server destroyed"
 
-    # ZenML Hub events
-    ZENML_HUB_PLUGIN_INSTALL = "ZenML Hub plugin installed"
-    ZENML_HUB_PLUGIN_UNINSTALL = "ZenML Hub plugin uninstalled"
-    ZENML_HUB_PLUGIN_CLONE = "ZenML Hub plugin pulled"
-    ZENML_HUB_PLUGIN_SUBMIT = "ZenML Hub plugin pushed"
-
     # Server Settings
     SERVER_SETTINGS_UPDATED = "Server Settings Updated"

zenml/cli/__init__.py

@@ -1684,7 +1684,7 @@ def my_pipeline(...):
 
 You can register your pipeline like this:
 ```bash
-zenml pipeline register my_pipeline
+zenml pipeline register run.my_pipeline
 ```
 
 To list all registered pipelines, use:
@@ -1693,21 +1693,15 @@ To list all registered pipelines, use:
 zenml pipeline list
 ```
 
-Since every pipeline run creates a new pipeline by default, you might
-occasionally want to delete a pipeline, which you can do via:
+To delete a pipeline, run:
 
 ```bash
 zenml pipeline delete <PIPELINE_NAME>
 ```
 
-This will delete the latest pipeline version and change all corresponding
+This will delete the pipeline and change all corresponding
 pipeline runs to become unlisted (not linked to any pipeline).
 
-If you want to delete all versions of a pipeline, you can do so as follows:
-
-```bash
-zenml pipeline delete <PIPELINE_NAME> --all-versions
-```
 
 To list all pipeline runs that you have executed, use:
 
@@ -2378,11 +2372,30 @@ zenml code-repository delete <REPOSITORY_NAME_OR_ID>
 Building an image without Runs
 ------------------------------
 
+To build or run a pipeline from the CLI, you need to know the source path of
+your pipeline. Let's imagine you have defined your pipeline in a python file
+called `run.py` like this:
+
+```python
+from zenml import pipeline
+
+@pipeline
+def my_pipeline(...):
+   # Connect your pipeline steps here
+   pass
+```
+
+The source path of your pipeline will be `run.my_pipeline`. In a generalized
+way, this will be `<MODULE_PATH>.<PIPELINE_FUNCTION_NAME>`. If the python file
+defining the pipeline is not in your current directory, the module path consists
+of the full path to the file, separated by dots, e.g.
+`some_directory.some_file.my_pipeline`.
+
 To [build Docker images for your pipeline](https://docs.zenml.io/how-to/customize-docker-builds)
 without actually running the pipeline, use:
 
 ```bash
-zenml pipeline build <PIPELINE_ID_OR_NAME>
+zenml pipeline build <PIPELINE_SOURCE_PATH>
 ```
 
 To specify settings for the Docker builds, use the `--config/-c` option of the
@@ -2390,20 +2403,20 @@ command. For more information about the structure of this configuration file,
 check out the `zenml.pipelines.base_pipeline.BasePipeline.build(...)` method.
 
 ```bash
-zenml pipeline build <PIPELINE_ID_OR_NAME> --config=<PATH_TO_CONFIG_YAML>
+zenml pipeline build <PIPELINE_SOURCE_PATH> --config=<PATH_TO_CONFIG_YAML>
 ```
 
 If you want to build the pipeline for a stack other than your current active
 stack, use the `--stack` option.
 
 ```bash
-zenml pipeline build <PIPELINE_ID_OR_NAME> --stack=<STACK_ID_OR_NAME>
+zenml pipeline build <PIPELINE_SOURCE_PATH> --stack=<STACK_ID_OR_NAME>
 ```
 
 To run a pipeline that was previously registered, use:
 
 ```bash
-zenml pipeline run  <PIPELINE_ID_OR_NAME>
+zenml pipeline run <PIPELINE_SOURCE_PATH>
 ```
 
 To specify settings for the pipeline, use the `--config/-c` option of the
@@ -2411,14 +2424,14 @@ command. For more information about the structure of this configuration file,
 check out the `zenml.pipelines.base_pipeline.BasePipeline.run(...)` method.
 
 ```bash
-zenml pipeline run <PIPELINE_ID_OR_NAME> --config=<PATH_TO_CONFIG_YAML>
+zenml pipeline run <PIPELINE_SOURCE_PATH> --config=<PATH_TO_CONFIG_YAML>
 ```
 
 If you want to run the pipeline on a stack different than your current active
 stack, use the `--stack` option.
 
 ```bash
-zenml pipeline run <PIPELINE_ID_OR_NAME> --stack=<STACK_ID_OR_NAME>
+zenml pipeline run <PIPELINE_SOURCE_PATH> --stack=<STACK_ID_OR_NAME>
 ```
 
 Tagging your resources with ZenML
@@ -2528,66 +2541,6 @@ zenml artifact-store deploy -f gcp -p gcp -r us-east1 -x project_id=zenml-core b
 
 For full documentation on this functionality, please refer to [the dedicated
 documentation on stack component deploy](https://docs.zenml.io/how-to/stack-deployment/deploy-a-stack-component).
-
-Interacting with the ZenML Hub
-------------------------------
-
-The ZenML Hub is a central location for discovering and sharing third-party 
-ZenML code, such as custom integrations, components, steps, pipelines, 
-materializers, and more. 
-You can browse the ZenML Hub at [https://hub.zenml.io](https://hub.zenml.io).
-
-The ZenML CLI provides various commands to interact with the ZenML Hub:
-
-- Listing all plugins available on the Hub:
-```bash
-zenml hub list
-```
-
-- Installing a Hub plugin:
-```bash
-zenml hub install
-```
-Installed plugins can be imported via `from zenml.hub.<plugin_name> import ...`. 
-
-
-- Uninstalling a Hub plugin:
-```bash
-zenml hub uninstall
-```
-
-- Cloning the source code of a Hub plugin (without installing it):
-```bash
-zenml hub clone
-```
-This is useful, e.g., for extending an existing plugin or for getting the 
-examples of a plugin.
-
-- Submitting/contributing a plugin to the Hub (requires login, see below):
-```bash
-zenml hub submit
-```
-If you are unsure about which arguments you need to set, you can run the
-command in interactive mode:
-```bash
-zenml hub submit --interactive
-```
-This will ask for and validate inputs one at a time.
-
-- Logging in to the Hub:
-```bash
-zenml hub login
-```
-
-- Logging out of the Hub:
-```bash
-zenml hub logout
-```
-
-- Viewing the build logs of a plugin you submitted to the Hub:
-```bash
-zenml hub logs
-```
 """
 
 from zenml.cli.version import *  # noqa
@@ -2599,7 +2552,6 @@ from zenml.cli.code_repository import *  # noqa
 from zenml.cli.config import *  # noqa
 from zenml.cli.downgrade import *  # noqa
 from zenml.cli.feature import *  # noqa
-from zenml.cli.hub import *  # noqa
 from zenml.cli.integration import *  # noqa
 from zenml.cli.model import *  # noqa
 from zenml.cli.model_registry import *  # noqa

zenml/cli/base.py

@@ -83,11 +83,11 @@ ZENML_PROJECT_TEMPLATES = dict(
     ),
     starter=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-starter",
-        github_tag="2024.06.06",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
+        github_tag="2024.07.26",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
     ),
     nlp=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-nlp",
-        github_tag="2024.06.14",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
+        github_tag="2024.08.07",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
     ),
     llm_finetuning=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-llm-finetuning",