sagemaker-mlops 1.0__tar.gz

sagemaker_mlops-1.0/PKG-INFO

@@ -0,0 +1,143 @@
+Metadata-Version: 2.4
+Name: sagemaker-mlops
+Version: 1.0
+Summary: SageMaker MLOps package for workflow orchestration and model building
+Author: Amazon Web Services
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: sagemaker-core>=2.0.0
+Requires-Dist: sagemaker-train>=0.1.0
+Requires-Dist: sagemaker-serve>=0.1.0
+Requires-Dist: boto3<2.0,>=1.35.75
+Requires-Dist: botocore<2.0,>=1.35.75
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Requires-Dist: mock; extra == "test"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: mock; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+
+# SageMaker MLOps Package
+
+The `sagemaker-mlops` package provides high-level orchestration capabilities for Amazon SageMaker workflows, including pipeline definitions, step implementations, and model building utilities.
+
+## Purpose
+
+This package sits at the top of the SageMaker SDK dependency hierarchy and orchestrates components from the Core, Train, and Serve packages. It resolves architectural violations by providing a dedicated home for workflow orchestration logic that needs to import from multiple lower-level packages.
+
+## Architecture
+
+The SageMaker SDK follows a clean 4-package architecture:
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                   Package Architecture                       │
+└─────────────────────────────────────────────────────────────┘
+
+                    MLOps (orchestration)
+                   ↙      ↓      ↘
+              Train     Core     Serve
+                   ↘      ↓      ↙
+                       Core
+
+Dependency Rules:
+✓ Core → nothing (foundation layer)
+✓ Train → Core only
+✓ Serve → Core only
+✓ MLOps → Train, Serve, Core (orchestration layer)
+```
+
+### Package Responsibilities
+
+- **sagemaker-core**: Foundation primitives (entities, parameters, functions, conditions, properties)
+- **sagemaker-train**: Training functionality (estimators, processors, tuners)
+- **sagemaker-serve**: Serving functionality (models, predictors, endpoints)
+- **sagemaker-mlops**: Workflow orchestration (pipelines, steps, model building)
+
+## What's in This Package
+
+### Workflow Orchestration (from Core)
+
+The following files were moved from `sagemaker-core/src/sagemaker/core/workflow/` to establish clean architectural boundaries:
+
+**Core Orchestration (2 files):**
+- `pipeline.py` - Pipeline class for workflow definition
+- `steps.py` - Base Step class and common step logic
+
+**Step Implementations (13 files):**
+- `automl_step.py` - AutoML training steps
+- `model_step.py` - Model creation and registration steps
+- `callback_step.py` - Callback steps for custom logic
+- `clarify_check_step.py` - Model bias and explainability checks
+- `condition_step.py` - Conditional execution steps
+- `emr_step.py` - EMR cluster steps
+- `fail_step.py` - Explicit failure steps
+- `function_step.py` - Lambda function steps
+- `lambda_step.py` - AWS Lambda invocation steps
+- `monitor_batch_transform_step.py` - Batch transform monitoring
+- `notebook_job_step.py` - Notebook execution steps
+- `quality_check_step.py` - Model quality checks
+- `step_collections.py` - Step collection utilities
+- `step_outputs.py` - Step output handling
+
+**Utilities (6 files):**
+- `_utils.py` - Internal utilities
+- `_steps_compiler.py` - Step compilation logic
+- `_repack_model.py` - Model repackaging utilities
+- `_event_bridge_client_helper.py` - EventBridge integration
+- `triggers.py` - Pipeline triggers
+- `utilities.py` - Public utility functions
+
+**Configuration (6 files):**
+- `check_job_config.py` - Quality check configuration
+- `parallelism_config.py` - Parallel execution configuration
+- `pipeline_definition_config.py` - Pipeline definition settings
+- `pipeline_experiment_config.py` - Experiment tracking configuration
+- `retry.py` - Retry policies
+- `selective_execution_config.py` - Selective execution settings
+
+### Model Building
+
+ModelBuilder is now located in the `sagemaker-serve` package but is re-exported from MLOps for convenience.
+
+### What Stayed in Core
+
+The following primitive files remain in `sagemaker-core/src/sagemaker/core/workflow/`:
+
+- `entities.py` - Base Entity and PipelineVariable classes
+- `parameters.py` - Parameter type definitions
+- `functions.py` - Pipeline functions (Join, JsonGet)
+- `execution_variables.py` - ExecutionVariable
+- `conditions.py` - Condition primitives
+- `properties.py` - Property definitions
+- `pipeline_context.py` - PipelineSession (refactored to remove Train/Serve imports)
+- `__init__.py` - Package initialization
+
+## Installation
+
+Install the package in editable mode for development:
+
+```bash
+pip install -e sagemaker-mlops
+```
+
+Or install all SageMaker packages together:
+
+```bash
+pip install -e sagemaker-core
+pip install -e sagemaker-train
+pip install -e sagemaker-serve
+pip install -e sagemaker-mlops
+```

sagemaker_mlops-1.0/README.md

@@ -0,0 +1,112 @@
+# SageMaker MLOps Package
+
+The `sagemaker-mlops` package provides high-level orchestration capabilities for Amazon SageMaker workflows, including pipeline definitions, step implementations, and model building utilities.
+
+## Purpose
+
+This package sits at the top of the SageMaker SDK dependency hierarchy and orchestrates components from the Core, Train, and Serve packages. It resolves architectural violations by providing a dedicated home for workflow orchestration logic that needs to import from multiple lower-level packages.
+
+## Architecture
+
+The SageMaker SDK follows a clean 4-package architecture:
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                   Package Architecture                       │
+└─────────────────────────────────────────────────────────────┘
+
+                    MLOps (orchestration)
+                   ↙      ↓      ↘
+              Train     Core     Serve
+                   ↘      ↓      ↙
+                       Core
+
+Dependency Rules:
+✓ Core → nothing (foundation layer)
+✓ Train → Core only
+✓ Serve → Core only
+✓ MLOps → Train, Serve, Core (orchestration layer)
+```
+
+### Package Responsibilities
+
+- **sagemaker-core**: Foundation primitives (entities, parameters, functions, conditions, properties)
+- **sagemaker-train**: Training functionality (estimators, processors, tuners)
+- **sagemaker-serve**: Serving functionality (models, predictors, endpoints)
+- **sagemaker-mlops**: Workflow orchestration (pipelines, steps, model building)
+
+## What's in This Package
+
+### Workflow Orchestration (from Core)
+
+The following files were moved from `sagemaker-core/src/sagemaker/core/workflow/` to establish clean architectural boundaries:
+
+**Core Orchestration (2 files):**
+- `pipeline.py` - Pipeline class for workflow definition
+- `steps.py` - Base Step class and common step logic
+
+**Step Implementations (13 files):**
+- `automl_step.py` - AutoML training steps
+- `model_step.py` - Model creation and registration steps
+- `callback_step.py` - Callback steps for custom logic
+- `clarify_check_step.py` - Model bias and explainability checks
+- `condition_step.py` - Conditional execution steps
+- `emr_step.py` - EMR cluster steps
+- `fail_step.py` - Explicit failure steps
+- `function_step.py` - Lambda function steps
+- `lambda_step.py` - AWS Lambda invocation steps
+- `monitor_batch_transform_step.py` - Batch transform monitoring
+- `notebook_job_step.py` - Notebook execution steps
+- `quality_check_step.py` - Model quality checks
+- `step_collections.py` - Step collection utilities
+- `step_outputs.py` - Step output handling
+
+**Utilities (6 files):**
+- `_utils.py` - Internal utilities
+- `_steps_compiler.py` - Step compilation logic
+- `_repack_model.py` - Model repackaging utilities
+- `_event_bridge_client_helper.py` - EventBridge integration
+- `triggers.py` - Pipeline triggers
+- `utilities.py` - Public utility functions
+
+**Configuration (6 files):**
+- `check_job_config.py` - Quality check configuration
+- `parallelism_config.py` - Parallel execution configuration
+- `pipeline_definition_config.py` - Pipeline definition settings
+- `pipeline_experiment_config.py` - Experiment tracking configuration
+- `retry.py` - Retry policies
+- `selective_execution_config.py` - Selective execution settings
+
+### Model Building
+
+ModelBuilder is now located in the `sagemaker-serve` package but is re-exported from MLOps for convenience.
+
+### What Stayed in Core
+
+The following primitive files remain in `sagemaker-core/src/sagemaker/core/workflow/`:
+
+- `entities.py` - Base Entity and PipelineVariable classes
+- `parameters.py` - Parameter type definitions
+- `functions.py` - Pipeline functions (Join, JsonGet)
+- `execution_variables.py` - ExecutionVariable
+- `conditions.py` - Condition primitives
+- `properties.py` - Property definitions
+- `pipeline_context.py` - PipelineSession (refactored to remove Train/Serve imports)
+- `__init__.py` - Package initialization
+
+## Installation
+
+Install the package in editable mode for development:
+
+```bash
+pip install -e sagemaker-mlops
+```
+
+Or install all SageMaker packages together:
+
+```bash
+pip install -e sagemaker-core
+pip install -e sagemaker-train
+pip install -e sagemaker-serve
+pip install -e sagemaker-mlops
+```

sagemaker_mlops-1.0/VERSION

@@ -0,0 +1 @@
+1.0

sagemaker_mlops-1.0/pyproject.toml

@@ -0,0 +1,81 @@
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sagemaker-mlops"
+dynamic = ["version"]
+description = "SageMaker MLOps package for workflow orchestration and model building"
+readme = "README.md"
+requires-python = ">=3.9"
+authors = [
+    {name = "Amazon Web Services"},
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: Apache Software License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+    "sagemaker-core>=2.0.0",
+    "sagemaker-train>=0.1.0",
+    "sagemaker-serve>=0.1.0",
+    "boto3>=1.35.75,<2.0",
+    "botocore>=1.35.75,<2.0",
+]
+
+[project.optional-dependencies]
+test = [
+    "pytest",
+    "pytest-cov",
+    "mock",
+]
+dev = [
+    "pytest",
+    "pytest-cov", 
+    "mock",
+    "black",
+    "flake8",
+]
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+include-package-data = true
+
+[tool.setuptools.dynamic]
+version = { file = "VERSION"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["sagemaker*"]
+namespaces = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+python_classes = ["Test*"]
+python_functions = ["test_*"]
+addopts = "-v --tb=short"
+
+[tool.black]
+line-length = 100
+target-version = ['py38']
+include = '\.pyi?$'
+extend-exclude = '''
+/(
+  # directories
+  \.eggs
+  | \.git
+  | \.hg
+  | \.mypy_cache
+  | \.tox
+  | \.venv
+  | build
+  | dist
+)/
+'''

sagemaker_mlops-1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sagemaker_mlops-1.0/src/sagemaker/__init__.py

@@ -0,0 +1,2 @@
+"""Namespace package for SageMaker."""
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)

sagemaker_mlops-1.0/src/sagemaker/mlops/__init__.py

@@ -0,0 +1,33 @@
+"""SageMaker MLOps package for workflow orchestration and model building.
+
+This package provides high-level orchestration capabilities for SageMaker workflows,
+including pipeline definitions, step implementations, and model building utilities.
+
+The MLOps package sits at the top of the dependency hierarchy and can import from:
+- sagemaker.core (foundation primitives)
+- sagemaker.train (training functionality)
+- sagemaker.serve (serving functionality)
+
+Key components:
+- workflow: Pipeline and step orchestration
+- model_builder: Model building and orchestration
+
+Example usage:
+    from sagemaker.mlops import ModelBuilder
+    from sagemaker.mlops.workflow import Pipeline, TrainingStep
+"""
+from __future__ import absolute_import
+
+__version__ = "0.1.0"
+
+# Model building
+from sagemaker.serve.model_builder import ModelBuilder
+
+# Workflow submodule is available via:
+#   from sagemaker.mlops import workflow
+#   from sagemaker.mlops.workflow import Pipeline, TrainingStep, etc.
+
+__all__ = [
+    "ModelBuilder",
+    "workflow",  # Submodule
+]

sagemaker_mlops-1.0/src/sagemaker/mlops/local/__init__.py

@@ -0,0 +1,24 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+"""Local pipeline execution for SageMaker MLOps."""
+from __future__ import absolute_import
+
+from sagemaker.mlops.local.local_pipeline_session import LocalPipelineSession  # noqa: F401
+
+# Pipeline execution is now in MLOps
+# For backward compatibility, users should update imports:
+# OLD: from sagemaker.core.local import LocalSession
+# NEW: from sagemaker.core.local import LocalSession (for jobs)
+#      from sagemaker.mlops.local import LocalPipelineSession (for pipelines)
+
+__all__ = ["LocalPipelineSession"]

sagemaker_mlops-1.0/src/sagemaker/mlops/local/exceptions.py

@@ -0,0 +1,29 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+"""Exceptions for local pipeline execution."""
+from __future__ import absolute_import
+
+
+class StepExecutionException(Exception):
+    """Exception raised when a pipeline step execution fails in local mode."""
+
+    def __init__(self, step_name, message):
+        """Initialize StepExecutionException.
+
+        Args:
+            step_name (str): Name of the step that failed
+            message (str): Failure message
+        """
+        self.step_name = step_name
+        self.message = message
+        super().__init__(f"Step '{step_name}' failed: {message}")

sagemaker_mlops-1.0/src/sagemaker/mlops/local/local_pipeline_session.py

@@ -0,0 +1,154 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+"""Local Pipeline Session - extends LocalSession with pipeline execution capabilities."""
+from __future__ import absolute_import
+
+import logging
+from datetime import datetime
+from botocore.exceptions import ClientError
+
+from sagemaker.core.local import LocalSession
+from sagemaker.core.telemetry.telemetry_logging import _telemetry_emitter
+from sagemaker.core.telemetry.constants import Feature
+from sagemaker.mlops.local.pipeline_entities import _LocalPipeline
+
+logger = logging.getLogger(__name__)
+
+
+class LocalPipelineSession(LocalSession):
+    """Extends LocalSession with pipeline execution capabilities.
+    
+    This class provides local pipeline execution functionality that was previously
+    in LocalSession. It's now in the MLOps package since pipeline orchestration
+    is an MLOps concern.
+    
+    Usage:
+        from sagemaker.mlops.local import LocalPipelineSession
+        from sagemaker.mlops.workflow import Pipeline
+        
+        session = LocalPipelineSession()
+        session.create_pipeline(pipeline, "My pipeline")
+    """
+    
+    def __init__(self, *args, **kwargs):
+        """Initialize LocalPipelineSession.
+        
+        Accepts the same arguments as LocalSession.
+        """
+        super().__init__(*args, **kwargs)
+        # Add pipeline storage to the sagemaker_client
+        if not hasattr(self.sagemaker_client, '_pipelines'):
+            self.sagemaker_client._pipelines = {}
+    
+    @_telemetry_emitter(Feature.LOCAL_MODE, "local_pipeline_session.create_pipeline")
+    def create_pipeline(
+        self, pipeline, pipeline_description, **kwargs  # pylint: disable=unused-argument
+    ):
+        """Create a local pipeline.
+
+        Args:
+            pipeline (Pipeline): Pipeline object
+            pipeline_description (str): Description of the pipeline
+
+        Returns:
+            Pipeline metadata (PipelineArn)
+        """
+        local_pipeline = _LocalPipeline(
+            pipeline=pipeline,
+            pipeline_description=pipeline_description,
+            local_session=self,
+        )
+        self.sagemaker_client._pipelines[pipeline.name] = local_pipeline
+        return {"PipelineArn": pipeline.name}
+
+    def update_pipeline(
+        self, pipeline, pipeline_description, **kwargs  # pylint: disable=unused-argument
+    ):
+        """Update a local pipeline.
+
+        Args:
+            pipeline (Pipeline): Pipeline object
+            pipeline_description (str): Description of the pipeline
+
+        Returns:
+            Pipeline metadata (PipelineArn)
+        """
+        if pipeline.name not in self.sagemaker_client._pipelines:
+            error_response = {
+                "Error": {
+                    "Code": "ResourceNotFound",
+                    "Message": "Pipeline {} does not exist".format(pipeline.name),
+                }
+            }
+            raise ClientError(error_response, "update_pipeline")
+        self.sagemaker_client._pipelines[pipeline.name].pipeline_description = pipeline_description
+        self.sagemaker_client._pipelines[pipeline.name].pipeline = pipeline
+        self.sagemaker_client._pipelines[pipeline.name].last_modified_time = (
+            datetime.now().timestamp()
+        )
+        return {"PipelineArn": pipeline.name}
+
+    def describe_pipeline(self, PipelineName):
+        """Describe the pipeline.
+
+        Args:
+          PipelineName (str): Name of the pipeline
+
+        Returns:
+            Pipeline metadata (PipelineArn, PipelineDefinition, LastModifiedTime, etc)
+        """
+        if PipelineName not in self.sagemaker_client._pipelines:
+            error_response = {
+                "Error": {
+                    "Code": "ResourceNotFound",
+                    "Message": "Pipeline {} does not exist".format(PipelineName),
+                }
+            }
+            raise ClientError(error_response, "describe_pipeline")
+        return self.sagemaker_client._pipelines[PipelineName].describe()
+
+    def delete_pipeline(self, PipelineName):
+        """Delete the local pipeline.
+
+        Args:
+          PipelineName (str): Name of the pipeline
+
+        Returns:
+            Pipeline metadata (PipelineArn)
+        """
+        if PipelineName in self.sagemaker_client._pipelines:
+            del self.sagemaker_client._pipelines[PipelineName]
+        return {"PipelineArn": PipelineName}
+
+    def start_pipeline_execution(self, PipelineName, **kwargs):
+        """Start the pipeline.
+
+        Args:
+          PipelineName (str): Name of the pipeline
+
+        Returns: 
+            _LocalPipelineExecution object
+        """
+        if "ParallelismConfiguration" in kwargs:
+            logger.warning("Parallelism configuration is not supported in local mode.")
+        if "SelectiveExecutionConfig" in kwargs:
+            raise ValueError("SelectiveExecutionConfig is not supported in local mode.")
+        if PipelineName not in self.sagemaker_client._pipelines:
+            error_response = {
+                "Error": {
+                    "Code": "ResourceNotFound",
+                    "Message": "Pipeline {} does not exist".format(PipelineName),
+                }
+            }
+            raise ClientError(error_response, "start_pipeline_execution")
+        return self.sagemaker_client._pipelines[PipelineName].start(**kwargs)