zenml-nightly 0.70.0.dev20241125__py3-none-any.whl → 0.71.0.dev20241220__py3-none-any.whl

README.md

@@ -147,7 +147,7 @@ Or, through our CLI command:
 zenml stack deploy --provider aws
 ```
 
-Alternatively, if the necessary pieces of infrastructure is already deployed, you can register a cloud stack seamlessly through the stack wizard:
+Alternatively, if the necessary pieces of infrastructure are already deployed, you can register a cloud stack seamlessly through the stack wizard:
 
 ```bash
 zenml stack register <STACK_NAME> --provider aws
@@ -195,9 +195,9 @@ def trainer(training_df: pd.DataFrame) -> Annotated["model", torch.nn.Module]:
 
 ![Exploring ZenML Models](/docs/book/.gitbook/assets/readme_mcp.gif)
 
-### Purpose built for machine learning with integration to you favorite tools
+### Purpose built for machine learning with integrations to your favorite tools
 
-While ZenML brings a lot of value of the box, it also integrates into your existing tooling and infrastructure without you having to be locked in.
+While ZenML brings a lot of value out of the box, it also integrates into your existing tooling and infrastructure without you having to be locked in.
 
 ```python
 from bentoml._internal.bento import bento
@@ -333,7 +333,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.70.0 is out. Check out the release notes
+    🎉 Version 0.71.0 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

RELEASE_NOTES.md

@@ -1,5 +1,117 @@
 <!-- markdown-link-check-disable -->
 
+# 0.71.0
+
+ZenML version 0.71.0 delivers a new Modal step operator integration as its 
+core feature, enabling efficient cloud execution for ML pipelines with granular 
+hardware configuration options. The release strengthens enterprise capabilities 
+through improved token management and dashboard features, while expanding 
+artifact handling with dynamic naming and enhanced visualization support. 
+Additionally, it includes various infrastructure improvements and bug fixes 
+that enhance the platform's stability and usability, particularly around 
+Docker connectivity, Kubernetes management, and service connector operations.
+
+## New Feature: Modal Step Operator Integration
+
+ZenML now [integrates with Modal](https://modal.com/), bringing lightning-fast 
+cloud execution capabilities to your ML pipelines. This new step operator[ 
+https://docs.zenml.io/stack-components/step-operators/modal] allows you to 
+execute individual pipeline steps on Modal's specialized compute instances, 
+offering notable speed particularly for Docker image building and hardware 
+provisioning. With simple configuration options, you can precisely specify 
+hardware requirements like GPU type, CPU count, and memory for each step, 
+making it ideal for resource-intensive ML workloads.
+
+## Other Highlights
+
+- **Workload API Token Management:** Refactored token management for improved 
+security with a generic API token dispenser.
+- **Dashboard Enhancements:** 
+  - Introduced service account management capabilities.
+  - Added API key creation and integration features.
+- **Dynamic Artifact Naming:** Introduced capability to dynamically name 
+artifacts.
+- **Visualization Enhancements:** Made dictionaries and lists visualizable, 
+added JSON visualization type.
+
+## Additional Features and Improvements
+
+- Improved error messages for Docker daemon connectivity
+- Enhanced SageMaker URL handling
+- Simplified model version artifact linkage
+- Added testing for pipeline templates
+- Improved Kubernetes pod and label length management
+- Allowed skipping type annotations for step inputs
+- Enabled using feature service instances instead of just names
+
+## Bug Fixes
+
+- Fixed issues with getting out of an inaccessible active stack
+- Fixed race conditions in the service connector type registry
+- Resolved migration test complications
+- Corrected documentation links
+- Fixed artifact store and artifact URI handling
+- Addressed various scalability and compatibility issues
+
+## Documentation Updates
+
+- Added documentation redirects
+- Updated PyTorch documentation links
+- Improved service connector documentation
+
+## What's Changed
+* Refactored workload API token management for better security and implemented generic API token dispenser by @stefannica in https://github.com/zenml-io/zenml/pull/3154
+* Add 0.70.0 to the migration tests by @avishniakov in https://github.com/zenml-io/zenml/pull/3190
+* Adjustments to the PR template by @bcdurak in https://github.com/zenml-io/zenml/pull/3194
+* [docs] Fix links in the how-to section of docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3196
+* Fixing sagemaker urls to take the settings into consideration by @bcdurak in https://github.com/zenml-io/zenml/pull/3195
+* Add cached run into testing of migrations by @avishniakov in https://github.com/zenml-io/zenml/pull/3199
+* Fix service connector type registry race conditions by @stefannica in https://github.com/zenml-io/zenml/pull/3202
+* Refactor container resource configuration in Vertex Orchestrator test by @avishniakov in https://github.com/zenml-io/zenml/pull/3203
+* [docs] Add missing redirects by @wjayesh in https://github.com/zenml-io/zenml/pull/3200
+* Add links to `uv` new PyTorch documentation by @strickvl in https://github.com/zenml-io/zenml/pull/3204
+* Fix broken docs link by @strickvl in https://github.com/zenml-io/zenml/pull/3208
+* Bugfix for getting out of an inaccessible active stack with no permissions by @bcdurak in https://github.com/zenml-io/zenml/pull/3198
+* Simplify model version artifact linkage by @schustmi in https://github.com/zenml-io/zenml/pull/3175
+* Reenable macos testing by @avishniakov in https://github.com/zenml-io/zenml/pull/3205
+* Various fixes and improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3211
+* Pass config path during zenml pipeline build by @schustmi in https://github.com/zenml-io/zenml/pull/3212
+* Add test for running templates by @schustmi in https://github.com/zenml-io/zenml/pull/3192
+* Fix service connector docstring by @schustmi in https://github.com/zenml-io/zenml/pull/3216
+* Improve error message when docker daemon is not reachable by @schustmi in https://github.com/zenml-io/zenml/pull/3214
+* Don't run migration for empty updates by @schustmi in https://github.com/zenml-io/zenml/pull/3210
+* Remove `--check` from format script and fix naming by @safoinme in https://github.com/zenml-io/zenml/pull/3218
+* More scalability improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3206
+* Use correct keyword for artifact store open by @schustmi in https://github.com/zenml-io/zenml/pull/3220
+* Fix passing of some sagemaker settings by @schustmi in https://github.com/zenml-io/zenml/pull/3221
+* Add hint when trying to connect with api key by @schustmi in https://github.com/zenml-io/zenml/pull/3222
+* Allow passing None values as parameter for optional complex types by @schustmi in https://github.com/zenml-io/zenml/pull/3215
+* Limit kubernetes pod and label length by @schustmi in https://github.com/zenml-io/zenml/pull/3217
+* Updating the quickstart example to use the new `log_metadata` by @bcdurak in https://github.com/zenml-io/zenml/pull/3188
+* Allow skipping type annotations for step inputs by @schustmi in https://github.com/zenml-io/zenml/pull/3223
+* Modal Step Operator by @strickvl in https://github.com/zenml-io/zenml/pull/2948
+* Add dynamic artifacts naming, documentation and tests by @avishniakov in https://github.com/zenml-io/zenml/pull/3201
+* Run template CLI command and bugfix by @schustmi in https://github.com/zenml-io/zenml/pull/3225
+* Make dicts/lists visualizable and add JSON as viz type by @wjayesh in https://github.com/zenml-io/zenml/pull/2882
+* Instances of the `FeatureService`s are now used instead of only the names of the FeatureServices.  by @aiakide in https://github.com/zenml-io/zenml/pull/3209
+* Quickstart fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3227
+* Add missing docs by @schustmi in https://github.com/zenml-io/zenml/pull/3226
+* Misc cleanup by @schustmi in https://github.com/zenml-io/zenml/pull/3229
+* Fix input resolution for steps with dynamic artifact names by @schustmi in https://github.com/zenml-io/zenml/pull/3228
+* Follow-up on the `run_metadata` changes by @bcdurak in https://github.com/zenml-io/zenml/pull/3193
+* Fixed broken links by @htahir1 in https://github.com/zenml-io/zenml/pull/3232
+* Fixed wandb login problem in Quickstart by @htahir1 in https://github.com/zenml-io/zenml/pull/3233
+* Misc bugfixes by @schustmi in https://github.com/zenml-io/zenml/pull/3234
+* Add additional way to fetch docker repo digest by @schustmi in https://github.com/zenml-io/zenml/pull/3231
+* AWS Image Builder implementation by @stefannica in https://github.com/zenml-io/zenml/pull/2904
+* Disable client-side caching for some orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/3235
+* Fix artifact uris for artifacts with name placeholders by @schustmi in https://github.com/zenml-io/zenml/pull/3237
+* Materializer test fix on Windows by @bcdurak in https://github.com/zenml-io/zenml/pull/3238
+* Fix GET step run endpoint to return unhydrated response if requested by @schustmi in https://github.com/zenml-io/zenml/pull/3240
+* Pipeline run API token fixes and improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3242
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.70.0...0.71.0
+
 # 0.70.0
 
 The **ZenML 0.70.0** release includes a significant number of database schema changes and migrations, which means upgrading to this version will require extra caution. As always, please make sure to make a copy of your production database before upgrading.

zenml/VERSION

@@ -1 +1 @@
-0.70.0.dev20241125
+0.71.0.dev20241220

zenml/artifact_stores/base_artifact_store.py

@@ -267,11 +267,11 @@ class BaseArtifactStore(StackComponent):
 
     # --- User interface ---
     @abstractmethod
-    def open(self, name: PathType, mode: str = "r") -> Any:
+    def open(self, path: PathType, mode: str = "r") -> Any:
         """Open a file at the given path.
 
         Args:
-            name: The path of the file to open.
+            path: The path of the file to open.
             mode: The mode to open the file.
 
         Returns:

zenml/artifacts/artifact_config.py

@@ -45,7 +45,13 @@ class ArtifactConfig(BaseModel):
     ```
 
     Attributes:
-        name: The name of the artifact.
+        name: The name of the artifact:
+            - static string e.g. "name"
+            - dynamic string e.g. "name_{date}_{time}_{custom_placeholder}"
+            If you use any placeholders besides `date` and `time`,
+            you need to provide the values for them in the `substitutions`
+            argument of the step decorator or the `substitutions` argument
+            of `with_options` of the step.
         version: The version of the artifact.
         tags: The tags of the artifact.
         run_metadata: Metadata to add to the artifact.
@@ -98,15 +104,18 @@ class ArtifactConfig(BaseModel):
             )
         elif is_model_artifact:
             logger.warning(
-                "`ArtifactConfig.is_model_artifact` is deprecated and will be "
-                "removed soon. Use `ArtifactConfig.artifact_type` instead."
+                "`ArtifactConfig(..., is_model_artifact=True)` is deprecated "
+                "and will be removed soon. Use `ArtifactConfig(..., "
+                "artifact_type=ArtifactType.MODEL)` instead. For more info: "
+                "https://docs.zenml.io/user-guide/starter-guide/manage-artifacts"
             )
             data.setdefault("artifact_type", ArtifactType.MODEL)
         elif is_deployment_artifact:
             logger.warning(
-                "`ArtifactConfig.is_deployment_artifact` is deprecated and "
-                "will be removed soon. Use `ArtifactConfig.artifact_type` "
-                "instead."
+                "`ArtifactConfig(..., is_deployment_artifact=True)` is "
+                "deprecated and will be removed soon. Use `ArtifactConfig(..., "
+                "artifact_type=ArtifactType.SERVICE)` instead. For more info: "
+                "https://docs.zenml.io/user-guide/starter-guide/manage-artifacts"
             )
             data.setdefault("artifact_type", ArtifactType.SERVICE)
 

zenml/artifacts/utils.py

@@ -14,6 +14,7 @@
 """Utility functions for handling artifacts."""
 
 import base64
+import contextlib
 import os
 import tempfile
 import zipfile
@@ -41,7 +42,6 @@ from zenml.enums import (
     ArtifactSaveType,
     ArtifactType,
     ExecutionStatus,
-    MetadataResourceTypes,
     StackComponentType,
     VisualizationType,
 )
@@ -404,50 +404,73 @@ def log_artifact_metadata(
         artifact_version: The version of the artifact to log metadata for. If
             not provided, when being called inside a step that produces an
             artifact named `artifact_name`, the metadata will be associated to
-            the corresponding newly created artifact. Or, if not provided when
-            being called outside a step, or in a step that does not produce
-            any artifact named `artifact_name`, the metadata will be associated
-            to the latest version of that artifact.
+            the corresponding newly created artifact.
 
     Raises:
         ValueError: If no artifact name is provided and the function is not
             called inside a step with a single output, or, if neither an
             artifact nor an output with the given name exists.
+
     """
     logger.warning(
         "The `log_artifact_metadata` function is deprecated and will soon be "
-        "removed. Please use `log_metadata` instead."
+        "removed. Instead, you can consider using: "
+        "`log_metadata(metadata={...}, infer_artifact=True, ...)` instead. For more "
+        "info: https://docs.zenml.io/how-to/model-management-metrics/track-metrics-metadata/attach-metadata-to-an-artifact"
     )
-    try:
+
+    from zenml import log_metadata
+
+    if artifact_name and artifact_version:
+        assert artifact_name is not None
+
+        log_metadata(
+            metadata=metadata,
+            artifact_name=artifact_name,
+            artifact_version=artifact_version,
+        )
+
+    step_context = None
+    with contextlib.suppress(RuntimeError):
         step_context = get_step_context()
-        in_step_outputs = (artifact_name in step_context._outputs) or (
-            not artifact_name and len(step_context._outputs) == 1
+
+    if step_context and artifact_name in step_context._outputs.keys():
+        log_metadata(
+            metadata=metadata,
+            artifact_name=artifact_name,
+            infer_artifact=True,
         )
-    except RuntimeError:
-        step_context = None
-        in_step_outputs = False
-
-    if not step_context or not in_step_outputs or artifact_version:
-        if not artifact_name:
-            raise ValueError(
-                "Artifact name must be provided unless the function is called "
-                "inside a step with a single output."
-            )
+    elif step_context and len(step_context._outputs) == 1:
+        single_output_name = list(step_context._outputs.keys())[0]
+
+        log_metadata(
+            metadata=metadata,
+            artifact_name=single_output_name,
+            infer_artifact=True,
+        )
+    elif artifact_name:
         client = Client()
-        response = client.get_artifact_version(artifact_name, artifact_version)
-        client.create_run_metadata(
+        logger.warning(
+            "Deprecation warning! Currently, you are calling "
+            "`log_artifact_metadata` from a context, where we use the "
+            "`artifact_name` to fetch it and link the metadata to its "
+            "latest version. This behavior is deprecated and will be "
+            "removed in the future. To circumvent this, please check"
+            "the `log_metadata` function."
+        )
+        artifact_version_model = client.get_artifact_version(
+            name_id_or_prefix=artifact_name
+        )
+        log_metadata(
             metadata=metadata,
-            resource_id=response.id,
-            resource_type=MetadataResourceTypes.ARTIFACT_VERSION,
+            artifact_version_id=artifact_version_model.id,
         )
-
     else:
-        try:
-            step_context.add_output_metadata(
-                metadata=metadata, output_name=artifact_name
-            )
-        except StepContextError as e:
-            raise ValueError(e)
+        raise ValueError(
+            "You need to call `log_artifact_metadata` either within a step "
+            "(potentially with an artifact name) or outside of a step with an "
+            "artifact name (and/or version)."
+        )
 
 
 # -----------------
@@ -575,7 +598,7 @@ def download_artifact_files_from_response(
                     )
                     file_path = str(Path(artifact.uri) / file_str)
                     with artifact_store.open(
-                        name=file_path, mode="rb"
+                        file_path, mode="rb"
                     ) as store_file:
                         # Use a loop to read and write chunks of the file
                         # instead of reading the entire file into memory
@@ -689,7 +712,11 @@ def _link_artifact_version_to_the_step_and_model(
         client.zen_store.update_run_step(
             step_run_id=step_run.id,
             step_run_update=StepRunUpdate(
-                outputs={artifact_version.artifact.name: artifact_version.id}
+                outputs={
+                    artifact_version.artifact.name: [
+                        artifact_version.id,
+                    ]
+                }
             ),
         )
         error_message = "model"

zenml/cli/__init__.py

@@ -1987,7 +1987,7 @@ Secrets management
 ------------------
 
 ZenML offers a way to [securely store secrets associated with your other
-stack components and infrastructure](https://docs.zenml.io/getting-started/deploying-zenml/manage-the-deployed-services/secret-management).
+stack components and infrastructure](https://docs.zenml.io/getting-started/deploying-zenml/secret-management).
 A ZenML Secret is a collection or grouping of key-value pairs stored by the
 ZenML secrets store. ZenML Secrets are identified by a unique name which
 allows you to fetch or reference them in your pipelines and stacks.
@@ -2083,7 +2083,7 @@ challenge in configuring uninterrupted, secure access to infrastructure
 resources. In ZenML, Service Connectors streamline this process by abstracting
 away the complexity of authentication and help you connect your stack to your
 resources. You can find the full docs on the ZenML service connectors
-[here](https://docs.zenml.io/how-to/auth-management).
+[here](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management).
 
 The ZenML CLI features a variety of commands to help you manage your service
 connectors. First of all, to explore all the types of service connectors
@@ -2113,7 +2113,7 @@ zenml service-connector register SERVICE_CONNECTOR_NAME \
 ```
 
 For more details on how to create a service connector, please refer to our
-[docs](https://docs.zenml.io/how-to/auth-management).
+[docs](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management).
 
 To check if your service connector is registered properly, you can `verify` it.
 By doing this, you can both check if it is configured correctly and also, you
@@ -2242,6 +2242,9 @@ export ZENML_STORE_URL=https://...
 export ZENML_STORE_API_KEY=<API_KEY>
 ```
 
+You don't need to run `zenml login` after setting these two environment
+variables and can start interacting with your server right away.
+
 To see all the service accounts you've created and their API keys, use the
 following commands:
 
@@ -2364,7 +2367,7 @@ defining the pipeline is not in your current directory, the module path consists
 of the full path to the file, separated by dots, e.g.
 `some_directory.some_file.my_pipeline`.
 
-To [build Docker images for your pipeline](https://docs.zenml.io/how-to/customize-docker-builds)
+To [build Docker images for your pipeline](https://docs.zenml.io/how-to/infrastructure-deployment/customize-docker-builds)
 without actually running the pipeline, use:
 
 ```bash
@@ -2407,6 +2410,21 @@ stack, use the `--stack` option.
 zenml pipeline run <PIPELINE_SOURCE_PATH> --stack=<STACK_ID_OR_NAME>
 ```
 
+If you want to create a run template based on your pipeline that can later be used to trigger a run either from the dashboard or through an HTTP request:
+
+```bash
+zenml pipeline create-run-template <PIPELINE_SOURCE_PATH> \
+   --name=<TEMPLATE_NAME>
+
+To specify a config file, use the `--config/-c` option. If you would like to use a different stack than the active one, use the `--stack` option.
+
+```bash
+zenml pipeline create-run-template <PIPELINE_SOURCE_PATH> \
+   --name=<TEMPLATE_NAME> \
+   --config=<PATH_TO_CONFIG_YAML> \
+   --stack=<STACK_ID_OR_NAME>
+```
+
 Tagging your resources with ZenML
 ---------------------------------
 

zenml/cli/base.py

@@ -79,19 +79,19 @@ class ZenMLProjectTemplateLocation(BaseModel):
 ZENML_PROJECT_TEMPLATES = dict(
     e2e_batch=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-e2e-batch",
-        github_tag="2024.11.20",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
+        github_tag="2024.11.28",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
     ),
     starter=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-starter",
-        github_tag="2024.10.30",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
+        github_tag="2024.11.28",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
     ),
     nlp=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-nlp",
-        github_tag="2024.10.30",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
+        github_tag="2024.11.28",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
     ),
     llm_finetuning=ZenMLProjectTemplateLocation(
         github_url="zenml-io/template-llm-finetuning",
-        github_tag="2024.11.08",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
+        github_tag="2024.11.28",  # Make sure it is aligned with .github/workflows/update-templates-to-examples.yml
     ),
 )
 
@@ -264,7 +264,7 @@ def init(
         f"will only take effect when you're running ZenML from the initialized "
         f"repository root, or from a subdirectory. For more information on "
         f"repositories and configurations, please visit "
-        f"https://docs.zenml.io/user-guide/starter-guide/understand-stacks."
+        f"https://docs.zenml.io/user-guide/production-guide/understand-stacks."
     )
 
 

zenml/cli/login.py

@@ -14,6 +14,7 @@
 """CLI for managing ZenML server deployments."""
 
 import ipaddress
+import os
 import re
 import sys
 import time
@@ -413,6 +414,27 @@ def connect_to_pro_server(
     cli_utils.declare(f"Connected to ZenML Pro server: {server.name}.")
 
 
+def _fail_if_authentication_environment_variables_set() -> None:
+    """Fail if any of the authentication environment variables are set."""
+    environment_variables = [
+        "ZENML_STORE_URL",
+        "ZENML_STORE_API_KEY",
+        "ZENML_STORE_USERNAME",
+        "ZENML_STORE_PASSWORD",
+    ]
+
+    if any(env_var in os.environ for env_var in environment_variables):
+        cli_utils.error(
+            "You're running to login/logout while having one of the "
+            f"{environment_variables} environment variables set. "
+            "If you want to use those environment variables to authenticate "
+            "to your ZenML server, there is no need to login/logout, you can "
+            "start interacting with your server right away. If you want to use "
+            "the `zenml login` command for authentication, please unset these "
+            "environment variables first."
+        )
+
+
 @cli.command(
     "login",
     help=(
@@ -670,6 +692,8 @@ def login(
             dashboard on a public domain. Primarily used for accessing the
             dashboard in Colab.
     """
+    _fail_if_authentication_environment_variables_set()
+
     if local:
         if api_key:
             cli_utils.error(
@@ -849,6 +873,8 @@ def logout(
     """
     from zenml.login.credentials_store import get_credentials_store
 
+    _fail_if_authentication_environment_variables_set()
+
     credentials_store = get_credentials_store()
     gc = GlobalConfiguration()
     store_cfg = gc.store_configuration