zenml-nightly 0.70.0.dev20241201__py3-none-any.whl → 0.71.0.dev20241220__py3-none-any.whl

README.md

@@ -147,7 +147,7 @@ Or, through our CLI command:
 zenml stack deploy --provider aws
 ```
 
-Alternatively, if the necessary pieces of infrastructure is already deployed, you can register a cloud stack seamlessly through the stack wizard:
+Alternatively, if the necessary pieces of infrastructure are already deployed, you can register a cloud stack seamlessly through the stack wizard:
 
 ```bash
 zenml stack register <STACK_NAME> --provider aws
@@ -195,9 +195,9 @@ def trainer(training_df: pd.DataFrame) -> Annotated["model", torch.nn.Module]:
 
 ![Exploring ZenML Models](/docs/book/.gitbook/assets/readme_mcp.gif)
 
-### Purpose built for machine learning with integration to you favorite tools
+### Purpose built for machine learning with integrations to your favorite tools
 
-While ZenML brings a lot of value of the box, it also integrates into your existing tooling and infrastructure without you having to be locked in.
+While ZenML brings a lot of value out of the box, it also integrates into your existing tooling and infrastructure without you having to be locked in.
 
 ```python
 from bentoml._internal.bento import bento
@@ -333,7 +333,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.70.0 is out. Check out the release notes
+    🎉 Version 0.71.0 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

RELEASE_NOTES.md

@@ -1,5 +1,117 @@
 <!-- markdown-link-check-disable -->
 
+# 0.71.0
+
+ZenML version 0.71.0 delivers a new Modal step operator integration as its 
+core feature, enabling efficient cloud execution for ML pipelines with granular 
+hardware configuration options. The release strengthens enterprise capabilities 
+through improved token management and dashboard features, while expanding 
+artifact handling with dynamic naming and enhanced visualization support. 
+Additionally, it includes various infrastructure improvements and bug fixes 
+that enhance the platform's stability and usability, particularly around 
+Docker connectivity, Kubernetes management, and service connector operations.
+
+## New Feature: Modal Step Operator Integration
+
+ZenML now [integrates with Modal](https://modal.com/), bringing lightning-fast 
+cloud execution capabilities to your ML pipelines. This new step operator[ 
+https://docs.zenml.io/stack-components/step-operators/modal] allows you to 
+execute individual pipeline steps on Modal's specialized compute instances, 
+offering notable speed particularly for Docker image building and hardware 
+provisioning. With simple configuration options, you can precisely specify 
+hardware requirements like GPU type, CPU count, and memory for each step, 
+making it ideal for resource-intensive ML workloads.
+
+## Other Highlights
+
+- **Workload API Token Management:** Refactored token management for improved 
+security with a generic API token dispenser.
+- **Dashboard Enhancements:** 
+  - Introduced service account management capabilities.
+  - Added API key creation and integration features.
+- **Dynamic Artifact Naming:** Introduced capability to dynamically name 
+artifacts.
+- **Visualization Enhancements:** Made dictionaries and lists visualizable, 
+added JSON visualization type.
+
+## Additional Features and Improvements
+
+- Improved error messages for Docker daemon connectivity
+- Enhanced SageMaker URL handling
+- Simplified model version artifact linkage
+- Added testing for pipeline templates
+- Improved Kubernetes pod and label length management
+- Allowed skipping type annotations for step inputs
+- Enabled using feature service instances instead of just names
+
+## Bug Fixes
+
+- Fixed issues with getting out of an inaccessible active stack
+- Fixed race conditions in the service connector type registry
+- Resolved migration test complications
+- Corrected documentation links
+- Fixed artifact store and artifact URI handling
+- Addressed various scalability and compatibility issues
+
+## Documentation Updates
+
+- Added documentation redirects
+- Updated PyTorch documentation links
+- Improved service connector documentation
+
+## What's Changed
+* Refactored workload API token management for better security and implemented generic API token dispenser by @stefannica in https://github.com/zenml-io/zenml/pull/3154
+* Add 0.70.0 to the migration tests by @avishniakov in https://github.com/zenml-io/zenml/pull/3190
+* Adjustments to the PR template by @bcdurak in https://github.com/zenml-io/zenml/pull/3194
+* [docs] Fix links in the how-to section of docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3196
+* Fixing sagemaker urls to take the settings into consideration by @bcdurak in https://github.com/zenml-io/zenml/pull/3195
+* Add cached run into testing of migrations by @avishniakov in https://github.com/zenml-io/zenml/pull/3199
+* Fix service connector type registry race conditions by @stefannica in https://github.com/zenml-io/zenml/pull/3202
+* Refactor container resource configuration in Vertex Orchestrator test by @avishniakov in https://github.com/zenml-io/zenml/pull/3203
+* [docs] Add missing redirects by @wjayesh in https://github.com/zenml-io/zenml/pull/3200
+* Add links to `uv` new PyTorch documentation by @strickvl in https://github.com/zenml-io/zenml/pull/3204
+* Fix broken docs link by @strickvl in https://github.com/zenml-io/zenml/pull/3208
+* Bugfix for getting out of an inaccessible active stack with no permissions by @bcdurak in https://github.com/zenml-io/zenml/pull/3198
+* Simplify model version artifact linkage by @schustmi in https://github.com/zenml-io/zenml/pull/3175
+* Reenable macos testing by @avishniakov in https://github.com/zenml-io/zenml/pull/3205
+* Various fixes and improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3211
+* Pass config path during zenml pipeline build by @schustmi in https://github.com/zenml-io/zenml/pull/3212
+* Add test for running templates by @schustmi in https://github.com/zenml-io/zenml/pull/3192
+* Fix service connector docstring by @schustmi in https://github.com/zenml-io/zenml/pull/3216
+* Improve error message when docker daemon is not reachable by @schustmi in https://github.com/zenml-io/zenml/pull/3214
+* Don't run migration for empty updates by @schustmi in https://github.com/zenml-io/zenml/pull/3210
+* Remove `--check` from format script and fix naming by @safoinme in https://github.com/zenml-io/zenml/pull/3218
+* More scalability improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3206
+* Use correct keyword for artifact store open by @schustmi in https://github.com/zenml-io/zenml/pull/3220
+* Fix passing of some sagemaker settings by @schustmi in https://github.com/zenml-io/zenml/pull/3221
+* Add hint when trying to connect with api key by @schustmi in https://github.com/zenml-io/zenml/pull/3222
+* Allow passing None values as parameter for optional complex types by @schustmi in https://github.com/zenml-io/zenml/pull/3215
+* Limit kubernetes pod and label length by @schustmi in https://github.com/zenml-io/zenml/pull/3217
+* Updating the quickstart example to use the new `log_metadata` by @bcdurak in https://github.com/zenml-io/zenml/pull/3188
+* Allow skipping type annotations for step inputs by @schustmi in https://github.com/zenml-io/zenml/pull/3223
+* Modal Step Operator by @strickvl in https://github.com/zenml-io/zenml/pull/2948
+* Add dynamic artifacts naming, documentation and tests by @avishniakov in https://github.com/zenml-io/zenml/pull/3201
+* Run template CLI command and bugfix by @schustmi in https://github.com/zenml-io/zenml/pull/3225
+* Make dicts/lists visualizable and add JSON as viz type by @wjayesh in https://github.com/zenml-io/zenml/pull/2882
+* Instances of the `FeatureService`s are now used instead of only the names of the FeatureServices.  by @aiakide in https://github.com/zenml-io/zenml/pull/3209
+* Quickstart fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3227
+* Add missing docs by @schustmi in https://github.com/zenml-io/zenml/pull/3226
+* Misc cleanup by @schustmi in https://github.com/zenml-io/zenml/pull/3229
+* Fix input resolution for steps with dynamic artifact names by @schustmi in https://github.com/zenml-io/zenml/pull/3228
+* Follow-up on the `run_metadata` changes by @bcdurak in https://github.com/zenml-io/zenml/pull/3193
+* Fixed broken links by @htahir1 in https://github.com/zenml-io/zenml/pull/3232
+* Fixed wandb login problem in Quickstart by @htahir1 in https://github.com/zenml-io/zenml/pull/3233
+* Misc bugfixes by @schustmi in https://github.com/zenml-io/zenml/pull/3234
+* Add additional way to fetch docker repo digest by @schustmi in https://github.com/zenml-io/zenml/pull/3231
+* AWS Image Builder implementation by @stefannica in https://github.com/zenml-io/zenml/pull/2904
+* Disable client-side caching for some orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/3235
+* Fix artifact uris for artifacts with name placeholders by @schustmi in https://github.com/zenml-io/zenml/pull/3237
+* Materializer test fix on Windows by @bcdurak in https://github.com/zenml-io/zenml/pull/3238
+* Fix GET step run endpoint to return unhydrated response if requested by @schustmi in https://github.com/zenml-io/zenml/pull/3240
+* Pipeline run API token fixes and improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3242
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.70.0...0.71.0
+
 # 0.70.0
 
 The **ZenML 0.70.0** release includes a significant number of database schema changes and migrations, which means upgrading to this version will require extra caution. As always, please make sure to make a copy of your production database before upgrading.

zenml/VERSION

@@ -1 +1 @@
-0.70.0.dev20241201
+0.71.0.dev20241220

zenml/artifacts/artifact_config.py

@@ -104,15 +104,18 @@ class ArtifactConfig(BaseModel):
             )
         elif is_model_artifact:
             logger.warning(
-                "`ArtifactConfig.is_model_artifact` is deprecated and will be "
-                "removed soon. Use `ArtifactConfig.artifact_type` instead."
+                "`ArtifactConfig(..., is_model_artifact=True)` is deprecated "
+                "and will be removed soon. Use `ArtifactConfig(..., "
+                "artifact_type=ArtifactType.MODEL)` instead. For more info: "
+                "https://docs.zenml.io/user-guide/starter-guide/manage-artifacts"
             )
             data.setdefault("artifact_type", ArtifactType.MODEL)
         elif is_deployment_artifact:
             logger.warning(
-                "`ArtifactConfig.is_deployment_artifact` is deprecated and "
-                "will be removed soon. Use `ArtifactConfig.artifact_type` "
-                "instead."
+                "`ArtifactConfig(..., is_deployment_artifact=True)` is "
+                "deprecated and will be removed soon. Use `ArtifactConfig(..., "
+                "artifact_type=ArtifactType.SERVICE)` instead. For more info: "
+                "https://docs.zenml.io/user-guide/starter-guide/manage-artifacts"
             )
             data.setdefault("artifact_type", ArtifactType.SERVICE)
 

zenml/artifacts/utils.py

@@ -414,7 +414,9 @@ def log_artifact_metadata(
     """
     logger.warning(
         "The `log_artifact_metadata` function is deprecated and will soon be "
-        "removed. Please use `log_metadata` instead."
+        "removed. Instead, you can consider using: "
+        "`log_metadata(metadata={...}, infer_artifact=True, ...)` instead. For more "
+        "info: https://docs.zenml.io/how-to/model-management-metrics/track-metrics-metadata/attach-metadata-to-an-artifact"
     )
 
     from zenml import log_metadata

zenml/cli/__init__.py

@@ -1987,7 +1987,7 @@ Secrets management
 ------------------
 
 ZenML offers a way to [securely store secrets associated with your other
-stack components and infrastructure](https://docs.zenml.io/getting-started/deploying-zenml/manage-the-deployed-services/secret-management).
+stack components and infrastructure](https://docs.zenml.io/getting-started/deploying-zenml/secret-management).
 A ZenML Secret is a collection or grouping of key-value pairs stored by the
 ZenML secrets store. ZenML Secrets are identified by a unique name which
 allows you to fetch or reference them in your pipelines and stacks.
@@ -2083,7 +2083,7 @@ challenge in configuring uninterrupted, secure access to infrastructure
 resources. In ZenML, Service Connectors streamline this process by abstracting
 away the complexity of authentication and help you connect your stack to your
 resources. You can find the full docs on the ZenML service connectors
-[here](https://docs.zenml.io/how-to/auth-management).
+[here](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management).
 
 The ZenML CLI features a variety of commands to help you manage your service
 connectors. First of all, to explore all the types of service connectors
@@ -2113,7 +2113,7 @@ zenml service-connector register SERVICE_CONNECTOR_NAME \
 ```
 
 For more details on how to create a service connector, please refer to our
-[docs](https://docs.zenml.io/how-to/auth-management).
+[docs](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management).
 
 To check if your service connector is registered properly, you can `verify` it.
 By doing this, you can both check if it is configured correctly and also, you
@@ -2367,7 +2367,7 @@ defining the pipeline is not in your current directory, the module path consists
 of the full path to the file, separated by dots, e.g.
 `some_directory.some_file.my_pipeline`.
 
-To [build Docker images for your pipeline](https://docs.zenml.io/how-to/customize-docker-builds)
+To [build Docker images for your pipeline](https://docs.zenml.io/how-to/infrastructure-deployment/customize-docker-builds)
 without actually running the pipeline, use:
 
 ```bash

zenml/cli/base.py

@@ -264,7 +264,7 @@ def init(
         f"will only take effect when you're running ZenML from the initialized "
         f"repository root, or from a subdirectory. For more information on "
         f"repositories and configurations, please visit "
-        f"https://docs.zenml.io/user-guide/starter-guide/understand-stacks."
+        f"https://docs.zenml.io/user-guide/production-guide/understand-stacks."
     )
 
 

zenml/cli/pipeline.py

@@ -40,6 +40,35 @@ from zenml.utils.yaml_utils import write_yaml
 logger = get_logger(__name__)
 
 
+def _import_pipeline(source: str) -> Pipeline:
+    """Import a pipeline.
+
+    Args:
+        source: The pipeline source.
+
+    Returns:
+        The pipeline.
+    """
+    try:
+        pipeline_instance = source_utils.load(source)
+    except ModuleNotFoundError as e:
+        source_root = source_utils.get_source_root()
+        cli_utils.error(
+            f"Unable to import module `{e.name}`. Make sure the source path is "
+            f"relative to your source root `{source_root}`."
+        )
+    except AttributeError as e:
+        cli_utils.error("Unable to load attribute from module: " + str(e))
+
+    if not isinstance(pipeline_instance, Pipeline):
+        cli_utils.error(
+            f"The given source path `{source}` does not resolve to a pipeline "
+            "object."
+        )
+
+    return pipeline_instance
+
+
 @cli.group(cls=TagGroup, tag=CliCategories.MANAGEMENT_TOOLS)
 def pipeline() -> None:
     """Interact with pipelines, runs and schedules."""
@@ -85,22 +114,7 @@ def register_pipeline(
             "source code root."
         )
 
-    try:
-        pipeline_instance = source_utils.load(source)
-    except ModuleNotFoundError as e:
-        source_root = source_utils.get_source_root()
-        cli_utils.error(
-            f"Unable to import module `{e.name}`. Make sure the source path is "
-            f"relative to your source root `{source_root}`."
-        )
-    except AttributeError as e:
-        cli_utils.error("Unable to load attribute from module: " + str(e))
-
-    if not isinstance(pipeline_instance, Pipeline):
-        cli_utils.error(
-            f"The given source path `{source}` does not resolve to a pipeline "
-            "object."
-        )
+    pipeline_instance = _import_pipeline(source=source)
 
     parameters: Dict[str, Any] = {}
     if parameters_path:
@@ -176,24 +190,9 @@ def build_pipeline(
             "your source code root."
         )
 
-    try:
-        pipeline_instance = source_utils.load(source)
-    except ModuleNotFoundError as e:
-        source_root = source_utils.get_source_root()
-        cli_utils.error(
-            f"Unable to import module `{e.name}`. Make sure the source path is "
-            f"relative to your source root `{source_root}`."
-        )
-    except AttributeError as e:
-        cli_utils.error("Unable to load attribute from module: " + str(e))
-
-    if not isinstance(pipeline_instance, Pipeline):
-        cli_utils.error(
-            f"The given source path `{source}` does not resolve to a pipeline "
-            "object."
-        )
-
     with cli_utils.temporary_active_stack(stack_name_or_id=stack_name_or_id):
+        pipeline_instance = _import_pipeline(source=source)
+
         pipeline_instance = pipeline_instance.with_options(
             config_path=config_path
         )
@@ -277,36 +276,21 @@ def run_pipeline(
             "your source code root."
         )
 
-    try:
-        pipeline_instance = source_utils.load(source)
-    except ModuleNotFoundError as e:
-        source_root = source_utils.get_source_root()
-        cli_utils.error(
-            f"Unable to import module `{e.name}`. Make sure the source path is "
-            f"relative to your source root `{source_root}`."
-        )
-    except AttributeError as e:
-        cli_utils.error("Unable to load attribute from module: " + str(e))
-
-    if not isinstance(pipeline_instance, Pipeline):
-        cli_utils.error(
-            f"The given source path `{source}` does not resolve to a pipeline "
-            "object."
-        )
-
-    build: Union[str, PipelineBuildBase, None] = None
-    if build_path_or_id:
-        if uuid_utils.is_valid_uuid(build_path_or_id):
-            build = build_path_or_id
-        elif os.path.exists(build_path_or_id):
-            build = PipelineBuildBase.from_yaml(build_path_or_id)
-        else:
-            cli_utils.error(
-                f"The specified build {build_path_or_id} is not a valid UUID "
-                "or file path."
-            )
-
     with cli_utils.temporary_active_stack(stack_name_or_id=stack_name_or_id):
+        pipeline_instance = _import_pipeline(source=source)
+
+        build: Union[str, PipelineBuildBase, None] = None
+        if build_path_or_id:
+            if uuid_utils.is_valid_uuid(build_path_or_id):
+                build = build_path_or_id
+            elif os.path.exists(build_path_or_id):
+                build = PipelineBuildBase.from_yaml(build_path_or_id)
+            else:
+                cli_utils.error(
+                    f"The specified build {build_path_or_id} is not a valid UUID "
+                    "or file path."
+                )
+
         pipeline_instance = pipeline_instance.with_options(
             config_path=config_path,
             build=build,
@@ -369,24 +353,9 @@ def create_run_template(
             "init` at your source code root."
         )
 
-    try:
-        pipeline_instance = source_utils.load(source)
-    except ModuleNotFoundError as e:
-        source_root = source_utils.get_source_root()
-        cli_utils.error(
-            f"Unable to import module `{e.name}`. Make sure the source path is "
-            f"relative to your source root `{source_root}`."
-        )
-    except AttributeError as e:
-        cli_utils.error("Unable to load attribute from module: " + str(e))
-
-    if not isinstance(pipeline_instance, Pipeline):
-        cli_utils.error(
-            f"The given source path `{source}` does not resolve to a pipeline "
-            "object."
-        )
-
     with cli_utils.temporary_active_stack(stack_name_or_id=stack_name_or_id):
+        pipeline_instance = _import_pipeline(source=source)
+
         pipeline_instance = pipeline_instance.with_options(
             config_path=config_path
         )

zenml/cli/server.py

@@ -587,25 +587,6 @@ def server_list(verbose: bool = False, all: bool = False) -> None:
             accessible_pro_servers = client.tenant.list(member_only=not all)
         except AuthorizationException as e:
             cli_utils.warning(f"ZenML Pro authorization error: {e}")
-        else:
-            if not all:
-                accessible_pro_servers = [
-                    s
-                    for s in accessible_pro_servers
-                    if s.status == TenantStatus.AVAILABLE
-                ]
-
-            if not accessible_pro_servers:
-                cli_utils.declare(
-                    "No ZenML Pro servers that are accessible to the current "
-                    "user could be found."
-                )
-                if not all:
-                    cli_utils.declare(
-                        "Hint: use the `--all` flag to show all ZenML servers, "
-                        "including those that the client is not currently "
-                        "authorized to access or are not running."
-                    )
 
         # We update the list of stored ZenML Pro servers with the ones that the
         # client is a member of
@@ -633,6 +614,25 @@ def server_list(verbose: bool = False, all: bool = False) -> None:
                 stored_server.update_server_info(accessible_server)
                 pro_servers.append(stored_server)
 
+        if not all:
+            accessible_pro_servers = [
+                s
+                for s in accessible_pro_servers
+                if s.status == TenantStatus.AVAILABLE
+            ]
+
+        if not accessible_pro_servers:
+            cli_utils.declare(
+                "No ZenML Pro servers that are accessible to the current "
+                "user could be found."
+            )
+            if not all:
+                cli_utils.declare(
+                    "Hint: use the `--all` flag to show all ZenML servers, "
+                    "including those that the client is not currently "
+                    "authorized to access or are not running."
+                )
+
     elif pro_servers:
         cli_utils.warning(
             "The ZenML Pro authentication has expired. Please re-login "