zenml-nightly 0.70.0.dev20241204__py3-none-any.whl → 0.71.0.dev20241207__py3-none-any.whl

README.md

@@ -147,7 +147,7 @@ Or, through our CLI command:
 zenml stack deploy --provider aws
 ```
 
-Alternatively, if the necessary pieces of infrastructure is already deployed, you can register a cloud stack seamlessly through the stack wizard:
+Alternatively, if the necessary pieces of infrastructure are already deployed, you can register a cloud stack seamlessly through the stack wizard:
 
 ```bash
 zenml stack register <STACK_NAME> --provider aws
@@ -195,9 +195,9 @@ def trainer(training_df: pd.DataFrame) -> Annotated["model", torch.nn.Module]:
 
 ![Exploring ZenML Models](/docs/book/.gitbook/assets/readme_mcp.gif)
 
-### Purpose built for machine learning with integration to you favorite tools
+### Purpose built for machine learning with integrations to your favorite tools
 
-While ZenML brings a lot of value of the box, it also integrates into your existing tooling and infrastructure without you having to be locked in.
+While ZenML brings a lot of value out of the box, it also integrates into your existing tooling and infrastructure without you having to be locked in.
 
 ```python
 from bentoml._internal.bento import bento
@@ -333,7 +333,7 @@ the Apache License Version 2.0.
     <a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
     <br />
     <br />
-    🎉 Version 0.70.0 is out. Check out the release notes
+    🎉 Version 0.71.0 is out. Check out the release notes
     <a href="https://github.com/zenml-io/zenml/releases">here</a>.
     <br />
     🖥️ Download our VS Code Extension <a href="https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode">here</a>.

RELEASE_NOTES.md

@@ -1,5 +1,117 @@
 <!-- markdown-link-check-disable -->
 
+# 0.71.0
+
+ZenML version 0.71.0 delivers a new Modal step operator integration as its 
+core feature, enabling efficient cloud execution for ML pipelines with granular 
+hardware configuration options. The release strengthens enterprise capabilities 
+through improved token management and dashboard features, while expanding 
+artifact handling with dynamic naming and enhanced visualization support. 
+Additionally, it includes various infrastructure improvements and bug fixes 
+that enhance the platform's stability and usability, particularly around 
+Docker connectivity, Kubernetes management, and service connector operations.
+
+## New Feature: Modal Step Operator Integration
+
+ZenML now [integrates with Modal](https://modal.com/), bringing lightning-fast 
+cloud execution capabilities to your ML pipelines. This new step operator[ 
+https://docs.zenml.io/stack-components/step-operators/modal] allows you to 
+execute individual pipeline steps on Modal's specialized compute instances, 
+offering notable speed particularly for Docker image building and hardware 
+provisioning. With simple configuration options, you can precisely specify 
+hardware requirements like GPU type, CPU count, and memory for each step, 
+making it ideal for resource-intensive ML workloads.
+
+## Other Highlights
+
+- **Workload API Token Management:** Refactored token management for improved 
+security with a generic API token dispenser.
+- **Dashboard Enhancements:** 
+  - Introduced service account management capabilities.
+  - Added API key creation and integration features.
+- **Dynamic Artifact Naming:** Introduced capability to dynamically name 
+artifacts.
+- **Visualization Enhancements:** Made dictionaries and lists visualizable, 
+added JSON visualization type.
+
+## Additional Features and Improvements
+
+- Improved error messages for Docker daemon connectivity
+- Enhanced SageMaker URL handling
+- Simplified model version artifact linkage
+- Added testing for pipeline templates
+- Improved Kubernetes pod and label length management
+- Allowed skipping type annotations for step inputs
+- Enabled using feature service instances instead of just names
+
+## Bug Fixes
+
+- Fixed issues with getting out of an inaccessible active stack
+- Fixed race conditions in the service connector type registry
+- Resolved migration test complications
+- Corrected documentation links
+- Fixed artifact store and artifact URI handling
+- Addressed various scalability and compatibility issues
+
+## Documentation Updates
+
+- Added documentation redirects
+- Updated PyTorch documentation links
+- Improved service connector documentation
+
+## What's Changed
+* Refactored workload API token management for better security and implemented generic API token dispenser by @stefannica in https://github.com/zenml-io/zenml/pull/3154
+* Add 0.70.0 to the migration tests by @avishniakov in https://github.com/zenml-io/zenml/pull/3190
+* Adjustments to the PR template by @bcdurak in https://github.com/zenml-io/zenml/pull/3194
+* [docs] Fix links in the how-to section of docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3196
+* Fixing sagemaker urls to take the settings into consideration by @bcdurak in https://github.com/zenml-io/zenml/pull/3195
+* Add cached run into testing of migrations by @avishniakov in https://github.com/zenml-io/zenml/pull/3199
+* Fix service connector type registry race conditions by @stefannica in https://github.com/zenml-io/zenml/pull/3202
+* Refactor container resource configuration in Vertex Orchestrator test by @avishniakov in https://github.com/zenml-io/zenml/pull/3203
+* [docs] Add missing redirects by @wjayesh in https://github.com/zenml-io/zenml/pull/3200
+* Add links to `uv` new PyTorch documentation by @strickvl in https://github.com/zenml-io/zenml/pull/3204
+* Fix broken docs link by @strickvl in https://github.com/zenml-io/zenml/pull/3208
+* Bugfix for getting out of an inaccessible active stack with no permissions by @bcdurak in https://github.com/zenml-io/zenml/pull/3198
+* Simplify model version artifact linkage by @schustmi in https://github.com/zenml-io/zenml/pull/3175
+* Reenable macos testing by @avishniakov in https://github.com/zenml-io/zenml/pull/3205
+* Various fixes and improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3211
+* Pass config path during zenml pipeline build by @schustmi in https://github.com/zenml-io/zenml/pull/3212
+* Add test for running templates by @schustmi in https://github.com/zenml-io/zenml/pull/3192
+* Fix service connector docstring by @schustmi in https://github.com/zenml-io/zenml/pull/3216
+* Improve error message when docker daemon is not reachable by @schustmi in https://github.com/zenml-io/zenml/pull/3214
+* Don't run migration for empty updates by @schustmi in https://github.com/zenml-io/zenml/pull/3210
+* Remove `--check` from format script and fix naming by @safoinme in https://github.com/zenml-io/zenml/pull/3218
+* More scalability improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3206
+* Use correct keyword for artifact store open by @schustmi in https://github.com/zenml-io/zenml/pull/3220
+* Fix passing of some sagemaker settings by @schustmi in https://github.com/zenml-io/zenml/pull/3221
+* Add hint when trying to connect with api key by @schustmi in https://github.com/zenml-io/zenml/pull/3222
+* Allow passing None values as parameter for optional complex types by @schustmi in https://github.com/zenml-io/zenml/pull/3215
+* Limit kubernetes pod and label length by @schustmi in https://github.com/zenml-io/zenml/pull/3217
+* Updating the quickstart example to use the new `log_metadata` by @bcdurak in https://github.com/zenml-io/zenml/pull/3188
+* Allow skipping type annotations for step inputs by @schustmi in https://github.com/zenml-io/zenml/pull/3223
+* Modal Step Operator by @strickvl in https://github.com/zenml-io/zenml/pull/2948
+* Add dynamic artifacts naming, documentation and tests by @avishniakov in https://github.com/zenml-io/zenml/pull/3201
+* Run template CLI command and bugfix by @schustmi in https://github.com/zenml-io/zenml/pull/3225
+* Make dicts/lists visualizable and add JSON as viz type by @wjayesh in https://github.com/zenml-io/zenml/pull/2882
+* Instances of the `FeatureService`s are now used instead of only the names of the FeatureServices.  by @aiakide in https://github.com/zenml-io/zenml/pull/3209
+* Quickstart fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3227
+* Add missing docs by @schustmi in https://github.com/zenml-io/zenml/pull/3226
+* Misc cleanup by @schustmi in https://github.com/zenml-io/zenml/pull/3229
+* Fix input resolution for steps with dynamic artifact names by @schustmi in https://github.com/zenml-io/zenml/pull/3228
+* Follow-up on the `run_metadata` changes by @bcdurak in https://github.com/zenml-io/zenml/pull/3193
+* Fixed broken links by @htahir1 in https://github.com/zenml-io/zenml/pull/3232
+* Fixed wandb login problem in Quickstart by @htahir1 in https://github.com/zenml-io/zenml/pull/3233
+* Misc bugfixes by @schustmi in https://github.com/zenml-io/zenml/pull/3234
+* Add additional way to fetch docker repo digest by @schustmi in https://github.com/zenml-io/zenml/pull/3231
+* AWS Image Builder implementation by @stefannica in https://github.com/zenml-io/zenml/pull/2904
+* Disable client-side caching for some orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/3235
+* Fix artifact uris for artifacts with name placeholders by @schustmi in https://github.com/zenml-io/zenml/pull/3237
+* Materializer test fix on Windows by @bcdurak in https://github.com/zenml-io/zenml/pull/3238
+* Fix GET step run endpoint to return unhydrated response if requested by @schustmi in https://github.com/zenml-io/zenml/pull/3240
+* Pipeline run API token fixes and improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3242
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.70.0...0.71.0
+
 # 0.70.0
 
 The **ZenML 0.70.0** release includes a significant number of database schema changes and migrations, which means upgrading to this version will require extra caution. As always, please make sure to make a copy of your production database before upgrading.

zenml/VERSION

@@ -1 +1 @@
-0.70.0.dev20241204
+0.71.0.dev20241207

zenml/config/server_config.py

@@ -27,6 +27,7 @@ from zenml.constants import (
     DEFAULT_ZENML_SERVER_DEVICE_AUTH_POLLING,
     DEFAULT_ZENML_SERVER_DEVICE_AUTH_TIMEOUT,
     DEFAULT_ZENML_SERVER_GENERIC_API_TOKEN_LIFETIME,
+    DEFAULT_ZENML_SERVER_GENERIC_API_TOKEN_MAX_LIFETIME,
     DEFAULT_ZENML_SERVER_LOGIN_RATE_LIMIT_DAY,
     DEFAULT_ZENML_SERVER_LOGIN_RATE_LIMIT_MINUTE,
     DEFAULT_ZENML_SERVER_MAX_DEVICE_AUTH_ATTEMPTS,
@@ -269,6 +270,9 @@ class ServerConfiguration(BaseModel):
     generic_api_token_lifetime: PositiveInt = (
         DEFAULT_ZENML_SERVER_GENERIC_API_TOKEN_LIFETIME
     )
+    generic_api_token_max_lifetime: PositiveInt = (
+        DEFAULT_ZENML_SERVER_GENERIC_API_TOKEN_MAX_LIFETIME
+    )
 
     external_login_url: Optional[str] = None
     external_user_info_url: Optional[str] = None

zenml/constants.py

@@ -170,6 +170,9 @@ ENV_ZENML_DISABLE_STEP_LOGS_STORAGE = "ZENML_DISABLE_STEP_LOGS_STORAGE"
 ENV_ZENML_IGNORE_FAILURE_HOOK = "ZENML_IGNORE_FAILURE_HOOK"
 ENV_ZENML_CUSTOM_SOURCE_ROOT = "ZENML_CUSTOM_SOURCE_ROOT"
 ENV_ZENML_WHEEL_PACKAGE_NAME = "ZENML_WHEEL_PACKAGE_NAME"
+ENV_ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION = (
+    "ZENML_PIPELINE_API_TOKEN_EXPIRATION"
+)
 
 # ZenML Server environment variables
 ENV_ZENML_SERVER_PREFIX = "ZENML_SERVER_"
@@ -268,6 +271,9 @@ DEFAULT_ZENML_SERVER_PIPELINE_RUN_AUTH_WINDOW = 60 * 48  # 48 hours
 DEFAULT_ZENML_SERVER_LOGIN_RATE_LIMIT_MINUTE = 5
 DEFAULT_ZENML_SERVER_LOGIN_RATE_LIMIT_DAY = 1000
 DEFAULT_ZENML_SERVER_GENERIC_API_TOKEN_LIFETIME = 60 * 60  # 1 hour
+DEFAULT_ZENML_SERVER_GENERIC_API_TOKEN_MAX_LIFETIME = (
+    60 * 60 * 24 * 7
+)  # 7 days
 
 DEFAULT_ZENML_SERVER_SECURE_HEADERS_HSTS = (
     "max-age=63072000; includeSubdomains"
@@ -466,3 +472,7 @@ BANNED_NAME_CHARACTERS = "\t\n\r\v\f"
 
 
 STACK_DEPLOYMENT_API_TOKEN_EXPIRATION = 60 * 6  # 6 hours
+
+ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION = handle_int_env_var(
+    ENV_ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION, default=0
+)

zenml/integrations/wandb/flavors/wandb_experiment_tracker_flavor.py

@@ -23,7 +23,7 @@ from typing import (
     cast,
 )
 
-from pydantic import field_validator
+from pydantic import field_validator, BaseModel
 
 from zenml.config.base_settings import BaseSettings
 from zenml.experiment_trackers.base_experiment_tracker import (
@@ -60,18 +60,26 @@ class WandbExperimentTrackerSettings(BaseSettings):
         Args:
             value: The settings.
 
+        Raises:
+            ValueError: If converting the settings failed.
+
         Returns:
             Dict representation of the settings.
         """
         import wandb
 
         if isinstance(value, wandb.Settings):
-            # Depending on the wandb version, either `make_static` or `to_dict`
-            # is available to convert the settings to a dictionary
-            if hasattr(value, "make_static"):
+            # Depending on the wandb version, either `model_dump`, 
+            # `make_static` or `to_dict` is available to convert the settings 
+            # to a dictionary
+            if isinstance(value, BaseModel):
+                return value.model_dump()
+            elif hasattr(value, "make_static"):
                 return cast(Dict[str, Any], value.make_static())
-            else:
+            elif hasattr(value, "to_dict"):
                 return value.to_dict()
+            else:
+                raise ValueError("Unable to convert wandb settings to dict.")
         else:
             return value
 

zenml/orchestrators/utils.py

@@ -26,10 +26,12 @@ from zenml.constants import (
     ENV_ZENML_ACTIVE_STACK_ID,
     ENV_ZENML_ACTIVE_WORKSPACE_ID,
     ENV_ZENML_DISABLE_CREDENTIALS_DISK_CACHING,
+    ENV_ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION,
     ENV_ZENML_SERVER,
     ENV_ZENML_STORE_PREFIX,
+    ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION,
 )
-from zenml.enums import AuthScheme, StackComponentType, StoreType
+from zenml.enums import APITokenType, AuthScheme, StackComponentType, StoreType
 from zenml.logger import get_logger
 from zenml.stack import StackComponent
 
@@ -137,37 +139,63 @@ def get_config_environment_vars(
         url = global_config.store_configuration.url
         api_token = credentials_store.get_token(url, allow_expired=False)
         if schedule_id or pipeline_run_id or step_run_id:
-            # When connected to an authenticated ZenML server, if a schedule ID,
-            # pipeline run ID or step run ID is supplied, we need to fetch a new
-            # workload API token scoped to the schedule, pipeline run or step
-            # run.
             assert isinstance(global_config.zen_store, RestZenStore)
 
-            # If only a schedule is given, the pipeline run credentials will
-            # be valid for the entire duration of the schedule.
-            api_key = credentials_store.get_api_key(url)
-            if not api_key and not pipeline_run_id and not step_run_id:
+            # The user has the option to manually set an expiration for the API
+            # token generated for a pipeline run. In this case, we generate a new
+            # generic API token that will be valid for the indicated duration.
+            if (
+                pipeline_run_id
+                and ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION != 0
+            ):
                 logger.warning(
-                    "An API token without an expiration time will be generated "
-                    "and used to run this pipeline on a schedule. This is very "
-                    "insecure because the API token will be valid for the "
-                    "entire lifetime of the schedule and can be used to access "
-                    "your user account if accidentally leaked. When deploying "
-                    "a pipeline on a schedule, it is strongly advised to use a "
-                    "service account API key to authenticate to the ZenML "
-                    "server instead of your regular user account. For more "
-                    "information, see "
-                    "https://docs.zenml.io/how-to/connecting-to-zenml/connect-with-a-service-account"
+                    f"An unscoped API token will be generated for this pipeline "
+                    f"run that will expire after "
+                    f"{ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION} "
+                    f"seconds instead of being scoped to the pipeline run "
+                    f"and not having an expiration time. This is more insecure "
+                    f"because the API token will remain valid even after the "
+                    f"pipeline run completes its execution. This option has "
+                    "been explicitly enabled by setting the "
+                    f"{ENV_ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION} environment "
+                    f"variable"
+                )
+                new_api_token = global_config.zen_store.get_api_token(
+                    token_type=APITokenType.GENERIC,
+                    expires_in=ZENML_PIPELINE_RUN_API_TOKEN_EXPIRATION,
                 )
 
-            # The schedule, pipeline run or step run credentials are scoped to
-            # the schedule, pipeline run or step run and will only be valid for
-            # the duration of the schedule/pipeline run/step run.
-            new_api_token = global_config.zen_store.get_api_token(
-                schedule_id=schedule_id,
-                pipeline_run_id=pipeline_run_id,
-                step_run_id=step_run_id,
-            )
+            else:
+                # If a schedule ID, pipeline run ID or step run ID is supplied,
+                # we need to fetch a new workload API token scoped to the
+                # schedule, pipeline run or step run.
+
+                # If only a schedule is given, the pipeline run credentials will
+                # be valid for the entire duration of the schedule.
+                api_key = credentials_store.get_api_key(url)
+                if not api_key and not pipeline_run_id and not step_run_id:
+                    logger.warning(
+                        "An API token without an expiration time will be generated "
+                        "and used to run this pipeline on a schedule. This is very "
+                        "insecure because the API token will be valid for the "
+                        "entire lifetime of the schedule and can be used to access "
+                        "your user account if accidentally leaked. When deploying "
+                        "a pipeline on a schedule, it is strongly advised to use a "
+                        "service account API key to authenticate to the ZenML "
+                        "server instead of your regular user account. For more "
+                        "information, see "
+                        "https://docs.zenml.io/how-to/connecting-to-zenml/connect-with-a-service-account"
+                    )
+
+                # The schedule, pipeline run or step run credentials are scoped to
+                # the schedule, pipeline run or step run and will only be valid for
+                # the duration of the schedule/pipeline run/step run.
+                new_api_token = global_config.zen_store.get_api_token(
+                    token_type=APITokenType.WORKLOAD,
+                    schedule_id=schedule_id,
+                    pipeline_run_id=pipeline_run_id,
+                    step_run_id=step_run_id,
+                )
 
             environment_vars[ENV_ZENML_STORE_PREFIX + "API_TOKEN"] = (
                 new_api_token

zenml/zen_server/auth.py

@@ -413,10 +413,7 @@ def authenticate_credentials(
                 logger.error(error)
                 raise CredentialsNotValid(error)
 
-            if pipeline_run_status in [
-                ExecutionStatus.FAILED,
-                ExecutionStatus.COMPLETED,
-            ]:
+            if pipeline_run_status.is_finished:
                 error = (
                     f"The execution of pipeline run "
                     f"{decoded_token.pipeline_run_id} has already concluded and "
@@ -461,10 +458,7 @@ def authenticate_credentials(
                 logger.error(error)
                 raise CredentialsNotValid(error)
 
-            if step_run_status in [
-                ExecutionStatus.FAILED,
-                ExecutionStatus.COMPLETED,
-            ]:
+            if step_run_status.is_finished:
                 error = (
                     f"The execution of step run "
                     f"{decoded_token.step_run_id} has already concluded and "
@@ -821,7 +815,10 @@ def generate_access_token(
         response: The FastAPI response object.
         device: The device used for authentication.
         api_key: The service account API key used for authentication.
-        expires_in: The number of seconds until the token expires.
+        expires_in: The number of seconds until the token expires. If not set,
+            the default value is determined automatically based on the server
+            configuration and type of token. If set to 0, the token will not
+            expire.
         schedule_id: The ID of the schedule to scope the token to.
         pipeline_run_id: The ID of the pipeline run to scope the token to.
         step_run_id: The ID of the step run to scope the token to.
@@ -835,7 +832,9 @@ def generate_access_token(
     # according to the values configured in the server config. Device tokens are
     # handled separately from regular user tokens.
     expires: Optional[datetime] = None
-    if expires_in:
+    if expires_in == 0:
+        expires_in = None
+    elif expires_in is not None:
         expires = datetime.utcnow() + timedelta(seconds=expires_in)
     elif device:
         # If a device was used for authentication, the token will expire

zenml/zen_server/dashboard/assets/{404-NVXKFp-x.js → 404-Cqu3EDCm.js}

@@ -1 +1 @@
-import{j as e}from"./@radix-DeK6qiuw.js";import{f as s,r as t}from"./index-CCOPpudF.js";import{E as r}from"./EmptyState-BzdlCwp3.js";import{S as a}from"./help-Cc9bBIJH.js";import{L as o}from"./@react-router-B3Z5rLr2.js";import"./@tanstack-DT5WLu9C.js";import"./@reactflow-CK0KJUen.js";function d(){return e.jsx("div",{className:"flex min-h-screen w-full flex-col",children:e.jsx(r,{icon:e.jsx(a,{className:"h-[120px] w-[120px] fill-neutral-300"}),children:e.jsxs("div",{className:"text-center",children:[e.jsx("h1",{className:"mb-2 text-display-xs font-semibold",children:"We can't find the page you are looking for"}),e.jsx("p",{className:"text-lg text-theme-text-secondary",children:"You can try typing a different URL or we can bring you back to your Homepage."}),e.jsx("div",{className:"mt-5 flex justify-center",children:e.jsx(s,{size:"md",asChild:!0,children:e.jsx(o,{className:"w-min self-center whitespace-nowrap",to:t.home,children:e.jsx("span",{className:"px-0.5",children:"Go to Home"})})})})]})})})}export{d as default};
+import{j as e}from"./@radix-DeK6qiuw.js";import{f as s,r as t}from"./index-FO-p0GU7.js";import{E as r}from"./EmptyState-BzdlCwp3.js";import{S as a}from"./help-Cc9bBIJH.js";import{L as o}from"./@react-router-B3Z5rLr2.js";import"./@tanstack-DT5WLu9C.js";import"./@reactflow-D2Y7BWwz.js";function d(){return e.jsx("div",{className:"flex min-h-screen w-full flex-col",children:e.jsx(r,{icon:e.jsx(a,{className:"h-[120px] w-[120px] fill-neutral-300"}),children:e.jsxs("div",{className:"text-center",children:[e.jsx("h1",{className:"mb-2 text-display-xs font-semibold",children:"We can't find the page you are looking for"}),e.jsx("p",{className:"text-lg text-theme-text-secondary",children:"You can try typing a different URL or we can bring you back to your Homepage."}),e.jsx("div",{className:"mt-5 flex justify-center",children:e.jsx(s,{size:"md",asChild:!0,children:e.jsx(o,{className:"w-min self-center whitespace-nowrap",to:t.home,children:e.jsx("span",{className:"px-0.5",children:"Go to Home"})})})})]})})})}export{d as default};