synapse-sdk 1.0.0a11__py3-none-any.whl → 2026.1.1b2__py3-none-any.whl

synapse_sdk/plugins/docs/ARCHITECTURE.md

@@ -0,0 +1,1225 @@
+# Plugin System Architecture
+
+This document provides a detailed technical overview of the Synapse SDK plugin system architecture, covering all major components, their interactions, and extension points.
+
+---
+
+## System Architecture
+
+### High-Level Overview
+
+```mermaid
+flowchart TB
+    subgraph UserCode["User Code"]
+        decorator["@action decorator"]
+        baseaction["BaseAction subclass"]
+    end
+
+    subgraph PluginFramework["Plugin Framework"]
+        discovery["PluginDiscovery<br/>- from_path()<br/>- from_module()<br/>- get_action_class()"]
+        actionclass["Action Class<br/>- params_model<br/>- result_model<br/>- execute()"]
+        context["RuntimeContext<br/>- logger<br/>- env<br/>- job_id"]
+        executor["Executor<br/>- LocalExecutor<br/>- RayActorExecutor<br/>- RayJobExecutor"]
+    end
+
+    subgraph Config["Configuration"]
+        configyaml["config.yaml<br/>- actions<br/>- method"]
+        pluginenv["PluginEnvironment<br/>- from_environ()<br/>- from_file()"]
+    end
+
+    decorator -->|discover| discovery
+    baseaction -->|discover| discovery
+    discovery -->|load| actionclass
+    actionclass -->|instantiate| context
+    pluginenv -->|inject| context
+    context -->|execute via| executor
+
+    style UserCode fill:#e1f5fe
+    style PluginFramework fill:#f3e5f5
+    style Config fill:#fff3e0
+```
+
+### Component Interaction Flow
+
+```mermaid
+flowchart TD
+    A["run_plugin(plugin_code, action, params, mode)"] --> B["_discover_action"]
+    B --> C["PluginDiscovery.get_action_class()"]
+    C --> D{"Select Executor"}
+
+    D -->|"mode='local'"| E1["LocalExecutor"]
+    D -->|"mode='task'"| E2["RayActorExecutor"]
+    D -->|"mode='job'"| E3["RayJobExecutor"]
+
+    E1 --> F["executor.execute()"]
+    E2 --> F
+    E3 --> F
+
+    F --> G["1. Validate params<br/>(Pydantic)"]
+    G --> H["2. Create RuntimeContext<br/>(logger, env)"]
+    H --> I["3. Instantiate action<br/>action_cls(params, ctx)"]
+    I --> J["4. Call execute()<br/>(user's method)"]
+    J --> K["5. Validate result<br/>(optional schema)"]
+    K --> L["Result"]
+
+    style A fill:#e8f5e9
+    style L fill:#e8f5e9
+    style D fill:#fff9c4
+```
+
+---
+
+## Core Components
+
+### BaseAction[P] - Action Base Class
+
+Location: `synapse_sdk/plugins/action.py`
+
+The `BaseAction` class is a generic abstract base class parameterized by a Pydantic model type `P` representing the action's parameters.
+
+#### Class Definition
+
+```python
+from abc import ABC, abstractmethod
+from typing import Generic, TypeVar
+from pydantic import BaseModel
+
+P = TypeVar('P', bound=BaseModel)
+
+class BaseAction(ABC, Generic[P]):
+    # Class attributes (optional, can be injected from config)
+    action_name: str | None = None
+    category: PluginCategory | None = None
+
+    # Semantic type declarations for pipeline validation
+    input_type: type[DataType] | None = None
+    output_type: type[DataType] | None = None
+
+    # Auto-extracted from generic parameter
+    params_model: type[P]
+
+    # Optional result schema for validation
+    result_model: type[BaseModel] | type[NoResult] = NoResult
+
+    # Instance attributes (set in __init__)
+    params: P
+    ctx: RuntimeContext
+
+    @abstractmethod
+    def execute(self) -> Any:
+        """Execute the action logic."""
+        ...
+```
+
+#### Generic Parameter Extraction
+
+The `params_model` is automatically extracted from the generic type parameter via `__init_subclass__`:
+
+```python
+class TrainParams(BaseModel):
+    epochs: int = 10
+
+class TrainAction(BaseAction[TrainParams]):  # P = TrainParams
+    pass
+
+# After class creation:
+TrainAction.params_model  # == TrainParams (auto-extracted)
+```
+
+#### Complete Implementation Example
+
+```python
+from synapse_sdk.plugins import BaseAction
+from synapse_sdk.plugins.enums import PluginCategory
+from synapse_sdk.plugins.types import YOLODataset, ModelWeights
+from pydantic import BaseModel, Field
+
+class TrainParams(BaseModel):
+    """Training parameters."""
+    epochs: int = Field(default=100, ge=1, le=1000)
+    batch_size: int = Field(default=16, ge=1, le=256)
+    learning_rate: float = Field(default=0.001, gt=0)
+    model_size: str = Field(default='n', pattern='^[nsmlx]$')
+
+class TrainResult(BaseModel):
+    """Training result."""
+    weights_path: str
+    final_loss: float
+    best_map: float
+
+class YOLOTrainAction(BaseAction[TrainParams]):
+    """Train a YOLO model on a dataset."""
+
+    # Metadata (optional, can come from config.yaml)
+    action_name = 'train'
+    category = PluginCategory.NEURAL_NET
+
+    # Semantic types for pipeline compatibility
+    input_type = YOLODataset
+    output_type = ModelWeights
+
+    # Enable result validation
+    result_model = TrainResult
+
+    def execute(self) -> TrainResult:
+        # Enable automatic framework logging
+        self.autolog('ultralytics')
+
+        # Access parameters
+        epochs = self.params.epochs
+        batch_size = self.params.batch_size
+
+        # Log start event
+        self.log('train_start', {
+            'epochs': epochs,
+            'batch_size': batch_size,
+        })
+
+        # Training loop with progress tracking
+        model = self._create_model()
+
+        for epoch in range(epochs):
+            loss = self._train_epoch(model, epoch)
+            self.set_progress(epoch + 1, epochs, category='train')
+            self.set_metrics({'loss': loss}, category='epoch')
+
+        # Save model
+        weights_path = self._save_model(model)
+
+        # Log completion
+        self.log('train_complete', {'weights_path': weights_path})
+
+        return TrainResult(
+            weights_path=weights_path,
+            final_loss=model.final_loss,
+            best_map=model.best_map,
+        )
+
+    def _create_model(self):
+        # Model creation logic
+        ...
+
+    def _train_epoch(self, model, epoch):
+        # Training epoch logic
+        ...
+
+    def _save_model(self, model):
+        # Model saving logic
+        ...
+```
+
+#### Method Reference
+
+| Method | Description |
+|--------|-------------|
+| `execute()` | Abstract. Implement your action logic here. |
+| `log(event, data, file=None)` | Log a structured event. |
+| `set_progress(current, total, category=None)` | Update progress tracking. |
+| `set_metrics(value, category)` | Record metrics for a category. |
+| `autolog(framework)` | Enable automatic logging for ML frameworks. |
+
+| Property | Description |
+|----------|-------------|
+| `logger` | Access the logger from context. |
+| `params` | Validated parameters (Pydantic model instance). |
+| `ctx` | RuntimeContext with services. |
+
+---
+
+### RuntimeContext - Dependency Injection Container
+
+Location: `synapse_sdk/plugins/context/__init__.py`
+
+The `RuntimeContext` provides runtime services to actions.
+
+#### Structure
+
+```python
+from dataclasses import dataclass
+from typing import Any
+
+@dataclass
+class RuntimeContext:
+    """Runtime context for action execution."""
+
+    logger: BaseLogger          # Logging interface
+    env: dict[str, Any]        # Environment variables
+    job_id: str | None = None  # Job tracking ID
+    client: BackendClient | None = None  # API client
+    agent_client: AgentClient | None = None  # Ray operations
+    checkpoint: dict | None = None  # Pretrained model info
+```
+
+#### Usage Patterns
+
+```python
+def execute(self) -> dict:
+    # Logging
+    self.ctx.logger.info('Starting execution')
+    self.ctx.logger.warning('Rate limit approaching')
+    self.ctx.logger.error('Failed to connect')
+
+    # Environment access
+    api_key = self.ctx.env.get('API_KEY', '')
+    debug_mode = self.ctx.env.get('DEBUG', 'false') == 'true'
+
+    # Structured logging
+    self.ctx.log('process_start', {'input_size': 1000})
+
+    # Progress tracking (with optional category)
+    self.ctx.set_progress(50, 100)  # 50% complete
+    self.ctx.set_progress(5, 10, category='download')  # Category-specific
+
+    # Metrics recording
+    self.ctx.set_metrics({'loss': 0.05}, category='train')
+    self.ctx.set_metrics({'accuracy': 0.95}, category='eval')
+
+    # User-visible messages
+    self.ctx.log_message('Model training complete!')
+
+    # Developer debug events
+    self.ctx.log_dev_event('checkpoint_saved', {'epoch': 10})
+
+    # Finalize logging
+    self.ctx.end_log()
+
+    return {'status': 'done'}
+```
+
+---
+
+### PluginEnvironment - Configuration Management
+
+Location: `synapse_sdk/plugins/context/env.py`
+
+Type-safe configuration loading from environment or files.
+
+#### Loading Methods
+
+```python
+from synapse_sdk.plugins.context.env import PluginEnvironment
+
+# From environment variables with prefix
+# PLUGIN_API_KEY=xxx, PLUGIN_BATCH_SIZE=32, etc.
+env = PluginEnvironment.from_environ(prefix='PLUGIN_')
+
+# From TOML file
+env = PluginEnvironment.from_file('config.toml')
+
+# From explicit dictionary
+env = PluginEnvironment({'API_KEY': 'xxx', 'BATCH_SIZE': 32})
+
+# Merge environments (later values override)
+merged = env1.merge(env2)
+```
+
+#### Type-Safe Getters
+
+```python
+# String (required - raises if missing)
+api_key = env.get_str('API_KEY')
+
+# String with default
+mode = env.get_str('MODE', default='production')
+
+# Integer
+batch_size = env.get_int('BATCH_SIZE', default=32)
+
+# Float
+learning_rate = env.get_float('LEARNING_RATE', default=0.001)
+
+# Boolean (true/false, 1/0, yes/no)
+debug = env.get_bool('DEBUG', default=False)
+
+# List (comma-separated string or actual list)
+tags = env.get_list('TAGS', default=[])
+```
+
+---
+
+### PluginConfig & ActionConfig
+
+Location: `synapse_sdk/plugins/config.py`
+
+Configuration models for plugin metadata.
+
+#### ActionConfig
+
+```python
+from pydantic import BaseModel
+from synapse_sdk.plugins.enums import RunMethod
+
+class ActionConfig(BaseModel):
+    name: str                           # Display name
+    description: str = ''               # Human-readable description
+    entrypoint: str                     # Module path (module:Class or module.Class)
+    method: RunMethod = RunMethod.TASK  # Execution method
+    params_schema: type[BaseModel] | None = None  # Parameter schema
+    input_type: str | None = None       # Semantic input type name
+    output_type: str | None = None      # Semantic output type name
+```
+
+#### PluginConfig
+
+```python
+class PluginConfig(BaseModel):
+    name: str                           # Plugin display name
+    code: str                           # Unique identifier
+    version: str = '0.1.0'              # Semantic version
+    category: PluginCategory            # Plugin category
+
+    # Package management
+    package_manager: PackageManager = PackageManager.PIP
+    package_manager_options: list[str] = []
+    wheels_dir: str = 'wheels'
+
+    # Runtime configuration
+    env: dict[str, Any] = {}            # Default environment
+    runtime_env: dict[str, Any] = {}    # Ray runtime_env
+
+    # Data type configuration
+    data_type: DataType | None = None   # IMAGE, TEXT, VIDEO, etc.
+    tasks: list[str] = []               # Supported annotation tasks
+
+    # Smart tool configuration
+    annotation_category: AnnotationCategory | None = None
+    annotation_type: AnnotationType | None = None
+    smart_tool: SmartToolType | None = None
+
+    # Actions
+    actions: dict[str, ActionConfig] = {}
+```
+
+#### config.yaml Example
+
+```yaml
+name: YOLO Training Plugin
+code: yolo_train
+version: 1.0.0
+category: neural_net
+data_type: image
+package_manager: uv
+
+env:
+  DEFAULT_MODEL: yolov8n.pt
+  MAX_EPOCHS: 300
+
+runtime_env:
+  pip:
+    - ultralytics>=8.0.0
+    - torch>=2.0.0
+
+actions:
+  train:
+    name: Train YOLO Model
+    description: Train a YOLO model on a labeled dataset
+    entrypoint: yolo_plugin.actions:TrainAction
+    method: job
+    input_type: YOLODataset
+    output_type: ModelWeights
+
+  export:
+    name: Export Model
+    description: Export trained model to ONNX format
+    entrypoint: yolo_plugin.actions:ExportAction
+    method: task
+    input_type: ModelWeights
+    output_type: ONNXModel
+```
+
+---
+
+## Step-Based Workflow System
+
+For complex multi-phase operations, use the step-based workflow system.
+
+### BaseStep[C] - Step Base Class
+
+Location: `synapse_sdk/plugins/steps/base.py`
+
+```python
+from abc import ABC, abstractmethod
+
+class BaseStep[C: BaseStepContext](ABC):
+    """Abstract base for workflow steps."""
+
+    @property
+    @abstractmethod
+    def name(self) -> str:
+        """Unique step identifier."""
+        ...
+
+    @property
+    @abstractmethod
+    def progress_weight(self) -> float:
+        """Weight for progress calculation (0.0-1.0)."""
+        ...
+
+    @abstractmethod
+    def execute(self, context: C) -> StepResult:
+        """Execute the step."""
+        ...
+
+    def can_skip(self, context: C) -> bool:
+        """Optional skip condition."""
+        return False
+
+    def rollback(self, context: C, result: StepResult) -> None:
+        """Optional cleanup on workflow failure."""
+        pass
+```
+
+### StepResult - Step Outcome
+
+```python
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any
+
+@dataclass
+class StepResult:
+    success: bool = True
+    data: dict[str, Any] = field(default_factory=dict)
+    error: str | None = None
+    rollback_data: dict[str, Any] = field(default_factory=dict)
+    skipped: bool = False
+    timestamp: datetime = field(default_factory=datetime.now)
+```
+
+### Creating Custom Steps
+
+```python
+from synapse_sdk.plugins.steps import BaseStep, StepResult, BaseStepContext
+from dataclasses import dataclass, field
+
+# 1. Define context with shared state
+@dataclass
+class UploadContext(BaseStepContext):
+    """Context for upload workflow."""
+    files: list[str] = field(default_factory=list)
+    uploaded_urls: list[str] = field(default_factory=list)
+    total_bytes: int = 0
+
+# 2. Implement steps
+class ValidateFilesStep(BaseStep[UploadContext]):
+    @property
+    def name(self) -> str:
+        return 'validate_files'
+
+    @property
+    def progress_weight(self) -> float:
+        return 0.1  # 10% of workflow
+
+    def execute(self, context: UploadContext) -> StepResult:
+        # Validate each file exists
+        missing = []
+        for path in context.files:
+            if not os.path.exists(path):
+                missing.append(path)
+
+        if missing:
+            return StepResult(
+                success=False,
+                error=f"Missing files: {missing}",
+            )
+
+        # Calculate total size
+        context.total_bytes = sum(
+            os.path.getsize(f) for f in context.files
+        )
+
+        return StepResult(
+            success=True,
+            data={'total_bytes': context.total_bytes},
+        )
+
+
+class UploadFilesStep(BaseStep[UploadContext]):
+    @property
+    def name(self) -> str:
+        return 'upload_files'
+
+    @property
+    def progress_weight(self) -> float:
+        return 0.8  # 80% of workflow
+
+    def execute(self, context: UploadContext) -> StepResult:
+        uploaded = []
+
+        for i, path in enumerate(context.files):
+            # Report per-file progress
+            context.set_progress(i, len(context.files))
+
+            url = upload_to_storage(path)
+            uploaded.append(url)
+            context.uploaded_urls.append(url)
+
+        return StepResult(
+            success=True,
+            data={'urls': uploaded},
+            rollback_data={'urls': uploaded},  # For cleanup
+        )
+
+    def rollback(self, context: UploadContext, result: StepResult) -> None:
+        # Clean up uploaded files on failure
+        for url in result.rollback_data.get('urls', []):
+            try:
+                delete_from_storage(url)
+            except Exception:
+                context.errors.append(f"Failed to delete: {url}")
+
+
+class FinalizeStep(BaseStep[UploadContext]):
+    @property
+    def name(self) -> str:
+        return 'finalize'
+
+    @property
+    def progress_weight(self) -> float:
+        return 0.1  # 10% of workflow
+
+    def can_skip(self, context: UploadContext) -> bool:
+        # Skip if nothing was uploaded
+        return len(context.uploaded_urls) == 0
+
+    def execute(self, context: UploadContext) -> StepResult:
+        # Register uploads in database
+        register_uploads(context.uploaded_urls)
+
+        return StepResult(
+            success=True,
+            data={'finalized': True},
+        )
+```
+
+### StepRegistry - Step Management
+
+Location: `synapse_sdk/plugins/steps/registry.py`
+
+```python
+from synapse_sdk.plugins.steps import StepRegistry
+
+# Create registry
+registry = StepRegistry[UploadContext]()
+
+# Register steps in order
+registry.register(ValidateFilesStep())
+registry.register(UploadFilesStep())
+registry.register(FinalizeStep())
+
+# Dynamic insertion
+registry.insert_after('validate_files', CompressionStep())
+registry.insert_before('finalize', VerifyStep())
+
+# Remove step
+registry.unregister('compression')
+
+# Access steps
+steps = registry.get_steps()  # Returns copy of step list
+total = registry.total_weight  # Sum of all progress weights
+```
+
+### Orchestrator - Step Execution
+
+Location: `synapse_sdk/plugins/steps/orchestrator.py`
+
+```python
+from synapse_sdk.plugins.steps import Orchestrator, StepRegistry
+
+# Setup
+registry = StepRegistry[UploadContext]()
+registry.register(ValidateFilesStep())
+registry.register(UploadFilesStep())
+registry.register(FinalizeStep())
+
+# Create context
+context = UploadContext(
+    runtime_ctx=runtime_ctx,
+    files=['/data/file1.jpg', '/data/file2.jpg'],
+)
+
+# Optional progress callback
+def on_progress(current: int, total: int):
+    print(f"Progress: {current}/{total}%")
+
+# Execute workflow
+orchestrator = Orchestrator(
+    registry=registry,
+    context=context,
+    progress_callback=on_progress,
+)
+
+try:
+    result = orchestrator.execute()
+    # {'success': True, 'steps_executed': 3, 'steps_total': 3}
+    print(f"Uploaded: {context.uploaded_urls}")
+except RuntimeError as e:
+    print(f"Workflow failed: {e}")
+    # Rollback already performed automatically
+```
+
+### Orchestrator Execution Flow
+
+```mermaid
+flowchart TD
+    A["orchestrator.execute()"] --> B["For each step in registry"]
+    B --> C["Set context.current_step = step.name"]
+    C --> D{"step.can_skip(context)?"}
+
+    D -->|Yes| E["Create skipped StepResult"]
+    D -->|No| F["Try: result = step.execute(context)"]
+
+    F -->|Exception| G["result = StepResult(success=False, error)"]
+    F -->|Success| H["result from execute()"]
+
+    E --> I["Append result to context.step_results"]
+    G --> I
+    H --> I
+
+    I --> J{"result.success?"}
+
+    J -->|No| K["_rollback() in reverse order"]
+    K --> L["Raise RuntimeError"]
+
+    J -->|Yes| M["Update progress via callback"]
+    M --> N{"More steps?"}
+
+    N -->|Yes| B
+    N -->|No| O["Clear context.current_step"]
+    O --> P["Return success summary"]
+
+    style A fill:#e8f5e9
+    style P fill:#e8f5e9
+    style L fill:#ffcdd2
+    style D fill:#fff9c4
+    style J fill:#fff9c4
+    style N fill:#fff9c4
+```
+
+### Utility Steps
+
+#### LoggingStep - Execution Logging
+
+```python
+from synapse_sdk.plugins.steps.utils import LoggingStep
+
+# Wrap any step with logging
+logged_upload = LoggingStep(UploadFilesStep())
+
+# Will log:
+# - step_start: {step: 'logged_upload_files'}
+# - step_end: {step: 'logged_upload_files', elapsed: 1.23, success: True}
+```
+
+#### TimingStep - Execution Timing
+
+```python
+from synapse_sdk.plugins.steps.utils import TimingStep
+
+# Wrap any step with timing
+timed_upload = TimingStep(UploadFilesStep())
+
+# Result will include:
+# result.data['duration_seconds'] = 1.234567
+```
+
+#### ValidationStep - Context Validation
+
+```python
+from synapse_sdk.plugins.steps.utils import ValidationStep
+
+def validate_files_exist(context: UploadContext) -> tuple[bool, str | None]:
+    if not context.files:
+        return False, "No files provided"
+    return True, None
+
+# Create validation step
+validate_step = ValidationStep(
+    validator=validate_files_exist,
+    name='validate_input',
+    progress_weight=0.05,
+)
+```
+
+---
+
+## Action Specializations
+
+### BaseTrainAction
+
+Location: `synapse_sdk/plugins/actions/train/`
+
+Base class for ML training actions with step-based workflow support.
+
+```python
+from synapse_sdk.plugins.actions.train import BaseTrainAction, TrainContext
+from synapse_sdk.plugins.steps import StepRegistry
+
+class MyTrainAction(BaseTrainAction[TrainParams]):
+    def setup_steps(self, registry: StepRegistry[TrainContext]) -> None:
+        """Override to register training steps."""
+        registry.register(PrepareDatasetStep())
+        registry.register(InitModelStep())
+        registry.register(TrainStep())
+        registry.register(SaveModelStep())
+
+    def execute(self) -> dict:
+        # Base class handles step orchestration
+        return super().execute()
+```
+
+TrainContext provides:
+- `dataset`: Loaded dataset reference
+- `model_path`: Path to model weights
+- `model`: Model instance
+- Progress categories: `DATASET`, `TRAIN`, `MODEL_UPLOAD`
+
+### BaseExportAction
+
+Location: `synapse_sdk/plugins/actions/export/`
+
+```python
+from synapse_sdk.plugins.actions.export import BaseExportAction, ExportContext
+
+class ONNXExportAction(BaseExportAction[ExportParams]):
+    def execute(self) -> dict:
+        # Load model
+        model = load_model(self.params.model_path)
+
+        # Export to ONNX
+        self.set_progress(0, 100, category='export')
+        output_path = export_to_onnx(model, self.params.output_path)
+        self.set_progress(100, 100, category='export')
+
+        return {'output_path': output_path}
+```
+
+### BaseUploadAction
+
+Location: `synapse_sdk/plugins/actions/upload/`
+
+```python
+from synapse_sdk.plugins.actions.upload import BaseUploadAction, UploadContext
+
+class S3UploadAction(BaseUploadAction[UploadParams]):
+    def setup_steps(self, registry: StepRegistry[UploadContext]) -> None:
+        registry.register(ValidateFilesStep())
+        registry.register(UploadToS3Step())
+        registry.register(VerifyUploadStep())
+```
+
+UploadContext provides:
+- `organized_files`: Files prepared for upload
+- `uploaded_files`: Successfully uploaded file references
+- `data_units`: Data unit tracking
+- Progress categories: `VALIDATION`, `UPLOAD`, `FINALIZATION`
+
+### BaseInferenceAction & BaseDeploymentAction
+
+Location: `synapse_sdk/plugins/actions/inference/`
+
+```python
+from synapse_sdk.plugins.actions.inference import (
+    BaseInferenceAction,
+    BaseDeploymentAction,
+    InferenceContext,
+)
+
+class YOLOInferenceAction(BaseInferenceAction[InferenceParams]):
+    def execute(self) -> dict:
+        # Load model
+        self.set_progress(0, 100, category='model_download')
+        model = load_model(self.params.model_path)
+        self.set_progress(100, 100, category='model_download')
+
+        # Run inference
+        results = []
+        images = self.params.images
+        for i, image in enumerate(images):
+            self.set_progress(i, len(images), category='inference')
+            result = model.predict(image)
+            results.append(result)
+
+        return {'predictions': results}
+```
+
+---
+
+## Executor System
+
+### LocalExecutor
+
+Location: `synapse_sdk/plugins/executors/local.py`
+
+Synchronous, in-process execution for development.
+
+```python
+from synapse_sdk.plugins.executors import LocalExecutor
+
+executor = LocalExecutor(
+    env={'DEBUG': 'true', 'API_KEY': 'xxx'},
+    job_id='local-test-001',
+)
+
+result = executor.execute(
+    action_cls=TrainAction,
+    params={'epochs': 10, 'batch_size': 32},
+)
+
+# Result is returned directly
+print(result)
+```
+
+### RayActorExecutor
+
+Location: `synapse_sdk/plugins/executors/ray/task.py`
+
+Persistent Ray Actor for serial task execution.
+
+```python
+from synapse_sdk.plugins.executors.ray import RayActorExecutor
+
+executor = RayActorExecutor(
+    working_dir='/path/to/plugin',  # Plugin source code
+    num_gpus=1,                      # GPU allocation per task
+    num_cpus=4,                      # CPU allocation per task
+    runtime_env={                    # Additional Ray runtime_env
+        'pip': ['torch>=2.0'],
+    },
+)
+
+# Execute action on Ray cluster
+result = executor.execute(
+    action_cls=TrainAction,
+    params={'epochs': 100},
+)
+
+# Cleanup actor
+executor.shutdown()
+```
+
+### RayJobExecutor
+
+Location: `synapse_sdk/plugins/executors/ray/job.py`
+
+Full job isolation via Ray Job API.
+
+```python
+from synapse_sdk.plugins.executors.ray import RayJobExecutor
+
+executor = RayJobExecutor(
+    dashboard_url='http://ray-head:8265',
+    working_dir='/path/to/plugin',
+    runtime_env={
+        'pip': ['ultralytics>=8.0'],
+        'env_vars': {'CUDA_VISIBLE_DEVICES': '0'},
+    },
+)
+
+# Submit job (returns immediately)
+job_id = executor.submit(
+    action_name='train',
+    params={'epochs': 100},
+)
+
+# Monitor status
+status = executor.get_status(job_id)
+# Returns: 'PENDING', 'RUNNING', 'SUCCEEDED', 'FAILED', 'STOPPED'
+
+# Stream logs
+for line in executor.stream_logs(job_id):
+    print(line)
+
+# Wait for completion with timeout
+result = executor.wait(job_id, timeout_seconds=3600)
+
+# Or get logs after completion
+logs = executor.get_logs(job_id)
+```
+
+### RayPipelineExecutor
+
+Location: `synapse_sdk/plugins/executors/ray/pipeline.py`
+
+Multi-action workflow execution. See [PIPELINE_GUIDE.md](PIPELINE_GUIDE.md) for details.
+
+```python
+from synapse_sdk.plugins.pipelines import ActionPipeline
+from synapse_sdk.plugins.executors.ray import RayPipelineExecutor
+
+# Define pipeline
+pipeline = ActionPipeline([
+    DownloadAction,
+    PreprocessAction,
+    TrainAction,
+    ExportAction,
+])
+
+# Create executor
+executor = RayPipelineExecutor(
+    ray_address='auto',
+    working_dir='/path/to/plugin',
+    pipeline_service_url='http://localhost:8100',
+)
+
+# Submit pipeline
+run_id = pipeline.submit({'dataset_id': 123}, executor)
+
+# Monitor progress
+progress = executor.get_progress(run_id)
+
+# Wait for result
+result = pipeline.wait(run_id, executor)
+```
+
+---
+
+## Plugin Discovery System
+
+Location: `synapse_sdk/plugins/discovery.py`
+
+### PluginDiscovery Class
+
+```python
+from synapse_sdk.plugins.discovery import PluginDiscovery
+
+# Load from config file
+discovery = PluginDiscovery.from_path('/path/to/plugin')
+
+# Load from Python module
+discovery = PluginDiscovery.from_module('my_plugin')
+
+# List available actions
+actions = discovery.list_actions()
+# ['train', 'export', 'inference']
+
+# Get action class with metadata injection
+action_cls = discovery.get_action_class('train')
+# action_cls.action_name == 'train' (injected from config)
+# action_cls.category == PluginCategory.NEURAL_NET
+
+# Get params model for validation
+params_model = discovery.get_action_params_model('train')
+
+# Get result model
+result_model = discovery.get_action_result_model('train')
+
+# Get semantic types
+input_type = discovery.get_action_input_type('train')  # YOLODataset
+output_type = discovery.get_action_output_type('train')  # ModelWeights
+```
+
+### UI Schema Generation
+
+Generate FormKit UI schemas from Pydantic models:
+
+```python
+# Get UI schema for action params
+ui_schema = discovery.get_action_ui_schema('train')
+
+# Returns FormKit-compatible schema:
+# {
+#     '$formkit': 'group',
+#     'children': [
+#         {
+#             '$formkit': 'number',
+#             'name': 'epochs',
+#             'label': 'Epochs',
+#             'value': 10,
+#             'validation': 'required|min:1|max:1000',
+#         },
+#         ...
+#     ]
+# }
+```
+
+### Config Synchronization
+
+Update config.yaml from discovered code:
+
+```python
+# Discover actions and update config file
+PluginDiscovery.sync_config_file('/path/to/plugin/config.yaml')
+
+# This will:
+# 1. Scan Python files for BaseAction subclasses
+# 2. Extract entrypoints, input_type, output_type
+# 3. Add new actions to config.yaml
+# 4. Update existing action entrypoints
+```
+
+### Static Discovery (AST-Based)
+
+Discover actions without importing (safer for broken modules):
+
+```python
+# Returns list of discovered action info
+actions = PluginDiscovery.discover_actions('/path/to/plugin')
+
+# Each action contains:
+# {
+#     'name': 'train',
+#     'entrypoint': 'my_plugin.actions:TrainAction',
+#     'input_type': 'YOLODataset',
+#     'output_type': 'ModelWeights',
+# }
+```
+
+---
+
+## Type System
+
+Location: `synapse_sdk/plugins/types.py`
+
+### DataType Hierarchy
+
+```python
+class DataType:
+    """Base class for semantic data types."""
+    name: ClassVar[str]
+    format: ClassVar[str | None] = None
+    description: ClassVar[str] = ''
+
+    @classmethod
+    def is_compatible_with(cls, other: type[DataType]) -> bool:
+        """Check if this type is compatible with another."""
+        return issubclass(cls, other) or issubclass(other, cls)
+
+# Built-in types
+class Dataset(DataType):
+    name = 'dataset'
+
+class YOLODataset(Dataset):
+    name = 'yolo_dataset'
+    format = 'yolo'
+
+class COCODataset(Dataset):
+    name = 'coco_dataset'
+    format = 'coco'
+
+class Model(DataType):
+    name = 'model'
+
+class ModelWeights(Model):
+    name = 'model_weights'
+
+class ONNXModel(Model):
+    name = 'onnx_model'
+    format = 'onnx'
+```
+
+### Creating Custom Types
+
+```python
+from synapse_sdk.plugins.types import DataType
+
+class CustomDataset(DataType):
+    name = 'custom_dataset'
+    format = 'custom'
+    description = 'My custom dataset format'
+
+class CustomModel(DataType):
+    name = 'custom_model'
+    format = 'custom'
+```
+
+### Pipeline Compatibility
+
+Types enable automatic pipeline validation:
+
+```python
+from synapse_sdk.plugins.pipelines import ActionPipeline
+
+# This validates that:
+# - Action1.output_type is compatible with Action2.input_type
+# - etc.
+pipeline = ActionPipeline([Action1, Action2, Action3])
+```
+
+---
+
+## Directory Structure
+
+```
+synapse_sdk/plugins/
+├── __init__.py                 # Public API exports (136 items)
+├── action.py                   # BaseAction[P], NoResult, validate_result
+├── decorators.py               # @action decorator
+├── config.py                   # PluginConfig, ActionConfig
+├── discovery.py                # PluginDiscovery class
+├── runner.py                   # run_plugin() entry point
+├── types.py                    # DataType hierarchy
+├── enums.py                    # PluginCategory, RunMethod, etc.
+├── errors.py                   # Exception hierarchy
+├── utils.py                    # Utility functions
+│
+├── context/
+│   ├── __init__.py             # RuntimeContext
+│   └── env.py                  # PluginEnvironment
+│
+├── steps/
+│   ├── __init__.py             # Step exports
+│   ├── base.py                 # BaseStep[C], StepResult
+│   ├── context.py              # BaseStepContext
+│   ├── registry.py             # StepRegistry[C]
+│   ├── orchestrator.py         # Orchestrator[C]
+│   └── utils/
+│       ├── __init__.py
+│       ├── logging.py          # LoggingStep
+│       ├── timing.py           # TimingStep
+│       └── validation.py       # ValidationStep
+│
+├── actions/
+│   ├── __init__.py             # Action exports
+│   ├── train/                  # BaseTrainAction, TrainContext
+│   ├── export/                 # BaseExportAction, ExportContext
+│   ├── upload/                 # BaseUploadAction, UploadContext
+│   ├── inference/              # BaseInferenceAction, BaseDeploymentAction
+│   └── dataset/                # DatasetAction
+│
+├── executors/
+│   ├── __init__.py             # ExecutorProtocol, LocalExecutor
+│   ├── local.py                # LocalExecutor
+│   └── ray/
+│       ├── __init__.py         # Ray executor exports
+│       ├── base.py             # BaseRayExecutor
+│       ├── task.py             # RayTaskExecutor, RayActorExecutor
+│       ├── job.py              # RayJobExecutor
+│       ├── pipeline.py         # RayPipelineExecutor
+│       └── packaging.py        # Dependency packaging
+│
+├── pipelines/
+│   ├── __init__.py             # Pipeline exports
+│   ├── action_pipeline.py      # ActionPipeline
+│   ├── context.py              # PipelineContext
+│   ├── display.py              # Progress display utilities
+│   └── (see docs/PIPELINE_GUIDE.md)
+│
+├── models/
+│   ├── logger.py               # LogLevel, ActionProgress, PipelineProgress
+│   └── pipeline.py             # RunStatus, ActionStatus
+│
+├── schemas/
+│   └── results.py              # WeightsResult, TrainResult, etc.
+│
+├── datasets/
+│   ├── formats/                # Format definitions (DM, YOLO)
+│   └── converters/             # Format converters
+│
+├── templates/                  # Plugin generation templates
+│   └── base/                   # Base plugin templates
+│
+└── testing/
+    └── sample_actions.py       # Test fixtures
+```
+
+---
+
+## Next Steps
+
+- **[OVERVIEW.md](OVERVIEW.md)**: Quick start and tutorials
+- **[ACTION_DEV_GUIDE.md](ACTION_DEV_GUIDE.md)**: Action development guide
+- **[STEP.md](STEP.md)**: Step implementations guide
+- **[LOGGING_SYSTEM.md](LOGGING_SYSTEM.md)**: Logging and progress tracking
+- **[README.md](README.md)**: Extension guide and best practices
+- **[PIPELINE_GUIDE.md](PIPELINE_GUIDE.md)**: Multi-action pipelines