synapse-sdk 1.0.0a11__py3-none-any.whl → 2026.1.1b2__py3-none-any.whl

synapse_sdk/plugins/docs/STEP.md

@@ -0,0 +1,656 @@
+# Step Implementation Guide
+
+Step-based workflows enable complex multi-phase operations with progress tracking, conditional execution, and automatic rollback on failure.
+
+## Overview
+
+Steps are composable building blocks for workflows. Each step:
+- Has a unique name and progress weight
+- Executes independently with shared context
+- Can be skipped conditionally
+- Supports rollback on workflow failure
+
+```mermaid
+flowchart LR
+    subgraph Workflow["Step-Based Workflow"]
+        S1["Step 1<br/>weight: 0.2"] --> S2["Step 2<br/>weight: 0.5"]
+        S2 --> S3["Step 3<br/>weight: 0.3"]
+    end
+
+    Context["Shared Context"] -.-> S1
+    Context -.-> S2
+    Context -.-> S3
+
+    style Workflow fill:#e1f5fe
+    style Context fill:#fff3e0
+```
+
+---
+
+## Core Components
+
+### BaseStep[C]
+
+Location: `synapse_sdk/plugins/steps/base.py`
+
+Abstract base class for all workflow steps. Generic type `C` must extend `BaseStepContext`.
+
+```python
+from abc import ABC, abstractmethod
+from synapse_sdk.plugins.steps import BaseStep, StepResult
+
+class BaseStep[C: BaseStepContext](ABC):
+    @property
+    @abstractmethod
+    def name(self) -> str:
+        """Unique step identifier."""
+        ...
+
+    @property
+    @abstractmethod
+    def progress_weight(self) -> float:
+        """Weight for progress calculation (0.0-1.0)."""
+        ...
+
+    @abstractmethod
+    def execute(self, context: C) -> StepResult:
+        """Execute the step logic."""
+        ...
+
+    def can_skip(self, context: C) -> bool:
+        """Optional: Return True to skip this step."""
+        return False
+
+    def rollback(self, context: C, result: StepResult) -> None:
+        """Optional: Cleanup when a later step fails."""
+        pass
+```
+
+### StepResult
+
+Location: `synapse_sdk/plugins/steps/base.py`
+
+Dataclass representing step execution outcome.
+
+```python
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any
+
+@dataclass
+class StepResult:
+    success: bool = True                              # Did step complete successfully?
+    data: dict[str, Any] = field(default_factory=dict)  # Output data
+    error: str | None = None                          # Error message if failed
+    rollback_data: dict[str, Any] = field(default_factory=dict)  # Data for rollback
+    skipped: bool = False                             # Was step skipped?
+    timestamp: datetime = field(default_factory=datetime.now)
+```
+
+### BaseStepContext
+
+Location: `synapse_sdk/plugins/steps/context.py`
+
+Base class for shared state between steps.
+
+```python
+from dataclasses import dataclass, field
+from synapse_sdk.plugins.context import RuntimeContext
+
+@dataclass
+class BaseStepContext:
+    runtime_ctx: RuntimeContext                    # Logger, env, clients
+    step_results: list[StepResult] = field(default_factory=list)
+    errors: list[str] = field(default_factory=list)
+    current_step: str | None = None               # Set by Orchestrator
+
+    def log(self, event: str, data: dict, file: str | None = None) -> None:
+        """Log an event via runtime context."""
+        self.runtime_ctx.log(event, data, file)
+
+    def set_progress(self, current: int, total: int, category: str | None = None) -> None:
+        """Set progress. Uses current_step as category if not provided."""
+        effective_category = category or self.current_step
+        self.runtime_ctx.set_progress(current, total, effective_category)
+
+    def set_metrics(self, value: dict, category: str | None = None) -> None:
+        """Set metrics. Uses current_step as category if not provided."""
+        effective_category = category or self.current_step
+        self.runtime_ctx.set_metrics(value, effective_category)
+```
+
+### StepRegistry[C]
+
+Location: `synapse_sdk/plugins/steps/registry.py`
+
+Manages ordered collection of steps.
+
+```python
+from synapse_sdk.plugins.steps import StepRegistry, BaseStep
+
+registry = StepRegistry[MyContext]()
+
+# Register steps in order
+registry.register(Step1())
+registry.register(Step2())
+registry.register(Step3())
+
+# Dynamic insertion
+registry.insert_after('step1', NewStep())   # Insert after step1
+registry.insert_before('step3', NewStep())  # Insert before step3
+
+# Remove step
+registry.unregister('step_name')
+
+# Access
+steps = registry.get_steps()      # Returns copy of step list
+total = registry.total_weight     # Sum of all progress weights
+```
+
+### Orchestrator[C]
+
+Location: `synapse_sdk/plugins/steps/orchestrator.py`
+
+Executes steps with progress tracking and automatic rollback.
+
+```python
+from synapse_sdk.plugins.steps import Orchestrator, StepRegistry
+
+orchestrator = Orchestrator(
+    registry=registry,
+    context=context,
+    progress_callback=lambda curr, total: print(f"{curr}%"),
+)
+
+result = orchestrator.execute()
+# Returns: {'success': True, 'steps_executed': 3, 'steps_total': 3}
+```
+
+---
+
+## Creating Custom Steps
+
+### Step 1: Define Context
+
+Create a context class with fields for shared state:
+
+```python
+from dataclasses import dataclass, field
+from synapse_sdk.plugins.steps import BaseStepContext
+
+@dataclass
+class ImageProcessingContext(BaseStepContext):
+    """Context for image processing workflow."""
+
+    # Input parameters
+    input_dir: str = ''
+    output_dir: str = ''
+    resize_width: int = 800
+
+    # Shared state populated by steps
+    image_paths: list[str] = field(default_factory=list)
+    processed_images: list[str] = field(default_factory=list)
+    failed_images: list[str] = field(default_factory=list)
+    total_bytes_saved: int = 0
+```
+
+### Step 2: Implement Steps
+
+```python
+import os
+from synapse_sdk.plugins.steps import BaseStep, StepResult
+
+class DiscoverImagesStep(BaseStep[ImageProcessingContext]):
+    """Find all images in input directory."""
+
+    @property
+    def name(self) -> str:
+        return 'discover_images'
+
+    @property
+    def progress_weight(self) -> float:
+        return 0.1  # 10% of workflow
+
+    def execute(self, context: ImageProcessingContext) -> StepResult:
+        # Find images
+        images = []
+        for file in os.listdir(context.input_dir):
+            if file.lower().endswith(('.jpg', '.jpeg', '.png')):
+                images.append(os.path.join(context.input_dir, file))
+
+        if not images:
+            return StepResult(
+                success=False,
+                error=f"No images found in {context.input_dir}",
+            )
+
+        # Store in context for next steps
+        context.image_paths = images
+
+        return StepResult(
+            success=True,
+            data={'image_count': len(images)},
+        )
+
+
+class ResizeImagesStep(BaseStep[ImageProcessingContext]):
+    """Resize all discovered images."""
+
+    @property
+    def name(self) -> str:
+        return 'resize_images'
+
+    @property
+    def progress_weight(self) -> float:
+        return 0.7  # 70% of workflow
+
+    def execute(self, context: ImageProcessingContext) -> StepResult:
+        processed = []
+        failed = []
+        bytes_saved = 0
+
+        for i, path in enumerate(context.image_paths):
+            # Report progress within step
+            context.set_progress(i + 1, len(context.image_paths))
+
+            try:
+                output_path, saved = self._resize_image(
+                    path,
+                    context.output_dir,
+                    context.resize_width,
+                )
+                processed.append(output_path)
+                bytes_saved += saved
+            except Exception as e:
+                failed.append(path)
+                context.log('resize_failed', {'path': path, 'error': str(e)})
+
+        # Update context
+        context.processed_images = processed
+        context.failed_images = failed
+        context.total_bytes_saved = bytes_saved
+
+        return StepResult(
+            success=True,
+            data={
+                'processed': len(processed),
+                'failed': len(failed),
+                'bytes_saved': bytes_saved,
+            },
+            rollback_data={'output_files': processed},  # For cleanup
+        )
+
+    def rollback(self, context: ImageProcessingContext, result: StepResult) -> None:
+        """Delete processed files on failure."""
+        for path in result.rollback_data.get('output_files', []):
+            try:
+                os.remove(path)
+            except OSError:
+                context.errors.append(f"Failed to delete: {path}")
+
+    def _resize_image(self, path: str, output_dir: str, width: int) -> tuple[str, int]:
+        # Image resizing implementation
+        ...
+
+
+class GenerateReportStep(BaseStep[ImageProcessingContext]):
+    """Generate summary report."""
+
+    @property
+    def name(self) -> str:
+        return 'generate_report'
+
+    @property
+    def progress_weight(self) -> float:
+        return 0.2  # 20% of workflow
+
+    def can_skip(self, context: ImageProcessingContext) -> bool:
+        # Skip if nothing was processed
+        return len(context.processed_images) == 0
+
+    def execute(self, context: ImageProcessingContext) -> StepResult:
+        report = {
+            'total_images': len(context.image_paths),
+            'processed': len(context.processed_images),
+            'failed': len(context.failed_images),
+            'bytes_saved': context.total_bytes_saved,
+        }
+
+        # Log report
+        context.log('processing_complete', report)
+
+        return StepResult(success=True, data=report)
+```
+
+### Step 3: Run Workflow
+
+```python
+from synapse_sdk.plugins.steps import StepRegistry, Orchestrator
+
+def process_images(runtime_ctx, input_dir: str, output_dir: str) -> dict:
+    # Create registry and register steps
+    registry = StepRegistry[ImageProcessingContext]()
+    registry.register(DiscoverImagesStep())
+    registry.register(ResizeImagesStep())
+    registry.register(GenerateReportStep())
+
+    # Create context
+    context = ImageProcessingContext(
+        runtime_ctx=runtime_ctx,
+        input_dir=input_dir,
+        output_dir=output_dir,
+    )
+
+    # Execute with progress callback
+    def on_progress(current: int, total: int):
+        runtime_ctx.set_progress(current, total, category='overall')
+
+    orchestrator = Orchestrator(
+        registry=registry,
+        context=context,
+        progress_callback=on_progress,
+    )
+
+    try:
+        result = orchestrator.execute()
+        return {
+            'success': True,
+            'processed': len(context.processed_images),
+            'failed': len(context.failed_images),
+        }
+    except RuntimeError as e:
+        return {
+            'success': False,
+            'error': str(e),
+            'errors': context.errors,
+        }
+```
+
+---
+
+## Orchestrator Execution Flow
+
+```mermaid
+flowchart TD
+    A["orchestrator.execute()"] --> B["For each step in registry"]
+    B --> C["Set context.current_step = step.name"]
+    C --> D{"step.can_skip(context)?"}
+
+    D -->|Yes| E["Create skipped StepResult"]
+    D -->|No| F["Try: result = step.execute(context)"]
+
+    F -->|Exception| G["result = StepResult(success=False, error)"]
+    F -->|Success| H["result from execute()"]
+
+    E --> I["Append result to context.step_results"]
+    G --> I
+    H --> I
+
+    I --> J{"result.success?"}
+
+    J -->|No| K["_rollback() in reverse order"]
+    K --> L["Raise RuntimeError"]
+
+    J -->|Yes| M["Update progress via callback"]
+    M --> N{"More steps?"}
+
+    N -->|Yes| B
+    N -->|No| O["Clear context.current_step"]
+    O --> P["Return success summary"]
+
+    style A fill:#e8f5e9
+    style P fill:#e8f5e9
+    style L fill:#ffcdd2
+    style D fill:#fff9c4
+    style J fill:#fff9c4
+    style N fill:#fff9c4
+```
+
+### Key Behaviors
+
+1. **Sequential Execution**: Steps run in registration order
+2. **Auto-Category**: `context.current_step` is set automatically for logging/metrics
+3. **Progress Calculation**: Based on cumulative step weights
+4. **Automatic Rollback**: On failure, `rollback()` called in reverse order
+5. **Best-Effort Rollback**: Rollback errors logged but don't stop other rollbacks
+
+---
+
+## Utility Steps
+
+### LoggingStep
+
+Wraps any step with start/end logging:
+
+```python
+from synapse_sdk.plugins.steps.utils import LoggingStep
+
+# Wrap a step
+logged_step = LoggingStep(ResizeImagesStep())
+
+# Logs:
+# - step_start: {step: 'logged_resize_images'}
+# - step_end: {step: 'logged_resize_images', elapsed: 1.23, success: True}
+```
+
+### TimingStep
+
+Wraps any step to measure execution duration:
+
+```python
+from synapse_sdk.plugins.steps.utils import TimingStep
+
+timed_step = TimingStep(ResizeImagesStep())
+
+# Result includes:
+# result.data['duration_seconds'] = 1.234567
+```
+
+### ValidationStep
+
+Validates context state before proceeding:
+
+```python
+from synapse_sdk.plugins.steps.utils import ValidationStep
+
+def validate_input(context: ImageProcessingContext) -> tuple[bool, str | None]:
+    if not context.input_dir:
+        return False, "input_dir is required"
+    if not os.path.isdir(context.input_dir):
+        return False, f"Directory not found: {context.input_dir}"
+    return True, None
+
+validate_step = ValidationStep(
+    validator=validate_input,
+    name='validate_input',
+    progress_weight=0.05,
+)
+
+registry.register(validate_step)
+registry.register(DiscoverImagesStep())
+# ...
+```
+
+### Composing Utility Steps
+
+```python
+from synapse_sdk.plugins.steps.utils import LoggingStep, TimingStep
+
+# Combine wrappers
+step = LoggingStep(TimingStep(ResizeImagesStep()))
+
+# Result: logged + timed step
+# Logs start/end AND measures duration
+```
+
+---
+
+## Integration with Actions
+
+### BaseTrainAction Example
+
+```python
+from synapse_sdk.plugins.actions.train import BaseTrainAction, TrainContext
+from synapse_sdk.plugins.steps import StepRegistry
+
+class MyTrainAction(BaseTrainAction[TrainParams]):
+    def setup_steps(self, registry: StepRegistry[TrainContext]) -> None:
+        """Override to register training steps."""
+        registry.register(PrepareDatasetStep())
+        registry.register(InitializeModelStep())
+        registry.register(TrainModelStep())
+        registry.register(SaveCheckpointStep())
+        registry.register(UploadModelStep())
+
+    def execute(self) -> dict:
+        # BaseTrainAction handles orchestration
+        return super().execute()
+```
+
+### BaseUploadAction Example
+
+```python
+from synapse_sdk.plugins.actions.upload import BaseUploadAction, UploadContext
+from synapse_sdk.plugins.steps import StepRegistry
+
+class MyUploadAction(BaseUploadAction[UploadParams]):
+    def setup_steps(self, registry: StepRegistry[UploadContext]) -> None:
+        registry.register(ValidateFilesStep())
+        registry.register(OrganizeFilesStep())
+        registry.register(UploadFilesStep())
+        registry.register(VerifyUploadStep())
+```
+
+---
+
+## Progress Weight Guidelines
+
+Progress weights should reflect **perceived duration** for better UX:
+
+| Step Type | Typical Weight | Notes |
+|-----------|---------------|-------|
+| Validation | 0.05 - 0.10 | Quick checks |
+| Data loading | 0.10 - 0.20 | I/O bound |
+| Processing | 0.50 - 0.80 | Main work |
+| Cleanup/Report | 0.05 - 0.10 | Finalization |
+
+**Example Distribution:**
+
+```python
+class ValidateStep(BaseStep):
+    progress_weight = 0.05   # 5%
+
+class LoadDataStep(BaseStep):
+    progress_weight = 0.15   # 15%
+
+class ProcessStep(BaseStep):
+    progress_weight = 0.70   # 70%
+
+class SaveResultsStep(BaseStep):
+    progress_weight = 0.10   # 10%
+# Total: 1.0 (100%)
+```
+
+---
+
+## Best Practices
+
+### Context Design
+
+```python
+# Good: Clear field purposes
+@dataclass
+class GoodContext(BaseStepContext):
+    # Input (set at creation)
+    config_path: str
+
+    # Intermediate state (populated by steps)
+    loaded_data: dict = field(default_factory=dict)
+
+    # Output (final results)
+    results: list = field(default_factory=list)
+```
+
+### Error Handling
+
+```python
+def execute(self, context: MyContext) -> StepResult:
+    try:
+        result = self._do_work(context)
+        return StepResult(success=True, data=result)
+    except ValidationError as e:
+        # Expected error - return failure
+        return StepResult(success=False, error=str(e))
+    except Exception as e:
+        # Unexpected error - log and return failure
+        context.log('unexpected_error', {'error': str(e)})
+        return StepResult(success=False, error=f"Unexpected: {e}")
+```
+
+### Rollback Design
+
+```python
+def execute(self, context: MyContext) -> StepResult:
+    created_files = []
+
+    for item in context.items:
+        path = self._create_file(item)
+        created_files.append(path)
+
+    return StepResult(
+        success=True,
+        rollback_data={'files': created_files},  # Save for cleanup
+    )
+
+def rollback(self, context: MyContext, result: StepResult) -> None:
+    for path in result.rollback_data.get('files', []):
+        try:
+            os.remove(path)
+        except OSError as e:
+            # Log but continue rollback
+            context.errors.append(f"Rollback failed for {path}: {e}")
+```
+
+### Step Independence
+
+```python
+# Good: Step reads from context, writes to context
+class ProcessStep(BaseStep[MyContext]):
+    def execute(self, context: MyContext) -> StepResult:
+        # Read from context (set by previous step)
+        data = context.loaded_data
+
+        # Process
+        results = self._process(data)
+
+        # Write to context (for next step)
+        context.results = results
+
+        return StepResult(success=True)
+```
+
+---
+
+## File Structure
+
+```
+synapse_sdk/plugins/steps/
+├── __init__.py          # Public exports
+├── base.py              # BaseStep, StepResult
+├── context.py           # BaseStepContext
+├── registry.py          # StepRegistry
+├── orchestrator.py      # Orchestrator
+└── utils/
+    ├── __init__.py
+    ├── logging.py       # LoggingStep
+    ├── timing.py        # TimingStep
+    └── validation.py    # ValidationStep
+```
+
+---
+
+## Related Documentation
+
+- **[ARCHITECTURE.md](ARCHITECTURE.md)** - Overall plugin system architecture
+- **[OVERVIEW.md](OVERVIEW.md)** - Getting started with plugins
+- **[README.md](README.md)** - Quick reference and extension guide