synapse-sdk 1.0.0a11__py3-none-any.whl → 2026.1.1b2__py3-none-any.whl

synapse_sdk/plugins/docs/OVERVIEW.md

@@ -0,0 +1,531 @@
+# Plugin System Overview
+
+The Synapse SDK Plugin System is a comprehensive framework for building, discovering, and executing plugin actions. It provides a flexible architecture for creating reusable components that can run locally or distributed across Ray clusters.
+
+## Target Audience
+
+This documentation is for **plugin system developers** who need to:
+- Create custom plugin actions for ML training, export, upload, inference, etc.
+- Understand the plugin architecture for extension and customization
+- Build step-based workflows with progress tracking and rollback support
+
+## Prerequisites
+
+- Python 3.12+
+- Familiarity with Pydantic v2 for data validation
+- Basic understanding of async/await patterns (for Ray execution)
+
+---
+
+## Key Concepts
+
+### Actions
+
+Actions are the fundamental units of work in the plugin system. Each action:
+- Receives typed parameters (Pydantic models)
+- Has access to a runtime context (logger, environment, clients)
+- Produces a result (optionally typed)
+
+```python
+from synapse_sdk.plugins import BaseAction
+from pydantic import BaseModel
+
+class TrainParams(BaseModel):
+    epochs: int = 10
+    learning_rate: float = 0.001
+
+class TrainAction(BaseAction[TrainParams]):
+    def execute(self) -> dict:
+        for epoch in range(self.params.epochs):
+            self.set_progress(epoch + 1, self.params.epochs)
+            # Training logic here
+        return {'status': 'completed'}
+```
+
+### Contexts
+
+Contexts provide runtime services and shared state:
+
+- **RuntimeContext**: Injected into all actions, provides logger, environment, job tracking
+- **BaseStepContext**: Extended by step-based workflows to share state between steps
+- **Specialized Contexts**: TrainContext, UploadContext, etc. for domain-specific state
+
+### Steps
+
+Steps are composable workflow building blocks for multi-phase operations:
+
+```python
+from synapse_sdk.plugins.steps import BaseStep, StepResult
+
+class LoadDataStep(BaseStep[MyContext]):
+    @property
+    def name(self) -> str:
+        return 'load_data'
+
+    @property
+    def progress_weight(self) -> float:
+        return 0.3  # 30% of total progress
+
+    def execute(self, context: MyContext) -> StepResult:
+        context.data = load_data()
+        return StepResult(success=True)
+```
+
+### Executors
+
+Executors determine where and how actions run:
+
+| Executor | Use Case | Startup Time |
+|----------|----------|--------------|
+| `LocalExecutor` | Development, testing | Instant |
+| `RayActorExecutor` | Light parallel tasks | <1 second |
+| `RayJobExecutor` | Heavy workloads, isolation | ~30 seconds |
+| `RayPipelineExecutor` | Multi-action workflows | Variable |
+
+### Discovery
+
+Plugin discovery loads actions from:
+- **Config files**: `config.yaml` with action metadata
+- **Python modules**: Scans for `@action` decorators and `BaseAction` subclasses
+- **AST scanning**: Static analysis without import failures
+
+---
+
+## Tutorial: Creating Your First Plugin
+
+This tutorial walks through creating a complete plugin from scratch.
+
+### Step 1: Define Parameters Model
+
+Create a Pydantic model for your action's input parameters:
+
+```python
+# my_plugin/params.py
+from pydantic import BaseModel, Field
+
+class ProcessParams(BaseModel):
+    """Parameters for the process action."""
+
+    input_path: str = Field(..., description='Path to input file')
+    output_path: str = Field(..., description='Path to output file')
+    batch_size: int = Field(default=32, ge=1, le=1024)
+    verbose: bool = False
+```
+
+### Step 2: Create Action Class
+
+Implement your action by subclassing `BaseAction`:
+
+```python
+# my_plugin/actions.py
+from synapse_sdk.plugins import BaseAction
+from synapse_sdk.plugins.enums import PluginCategory
+from my_plugin.params import ProcessParams
+
+class ProcessAction(BaseAction[ProcessParams]):
+    """Process files with progress tracking."""
+
+    action_name = 'process'
+    category = PluginCategory.CUSTOM
+
+    def execute(self) -> dict:
+        # Access validated parameters
+        input_path = self.params.input_path
+        output_path = self.params.output_path
+
+        # Log events
+        self.log('process_start', {'input': input_path})
+
+        # Simulate processing with progress
+        total_items = 100
+        for i in range(total_items):
+            # Update progress (current, total)
+            self.set_progress(i + 1, total_items)
+
+            if self.params.verbose:
+                self.log('item_processed', {'index': i})
+
+        # Log metrics
+        self.set_metrics({'items_processed': total_items}, category='process')
+
+        # Return result
+        return {
+            'status': 'completed',
+            'output_path': output_path,
+            'items_processed': total_items,
+        }
+```
+
+### Step 3: Write config.yaml
+
+Create a configuration file for your plugin:
+
+```yaml
+# my_plugin/config.yaml
+name: My Plugin
+code: my_plugin
+version: 0.1.0
+category: custom
+
+actions:
+  process:
+    name: Process Files
+    description: Process input files with batch processing
+    entrypoint: my_plugin.actions:ProcessAction
+    method: task
+```
+
+### Step 4: Run Locally
+
+Execute your action using `run_plugin`:
+
+```python
+# run.py
+from synapse_sdk.plugins import run_plugin
+
+result = run_plugin(
+    plugin_code='my_plugin',  # Module path
+    action='process',
+    params={
+        'input_path': '/data/input.csv',
+        'output_path': '/data/output.csv',
+        'batch_size': 64,
+        'verbose': True,
+    },
+    mode='local',  # Run in current process
+)
+
+print(f"Result: {result}")
+# Output: Result: {'status': 'completed', 'output_path': '/data/output.csv', 'items_processed': 100}
+```
+
+### Step 5: Add Result Schema (Optional)
+
+For typed results, create a result model:
+
+```python
+# my_plugin/results.py
+from pydantic import BaseModel
+
+class ProcessResult(BaseModel):
+    """Typed result for process action."""
+    status: str
+    output_path: str
+    items_processed: int
+```
+
+Then update your action:
+
+```python
+# my_plugin/actions.py
+from my_plugin.results import ProcessResult
+
+class ProcessAction(BaseAction[ProcessParams]):
+    result_model = ProcessResult  # Enable result validation
+
+    def execute(self) -> ProcessResult:
+        # ... processing logic ...
+        return ProcessResult(
+            status='completed',
+            output_path=self.params.output_path,
+            items_processed=total_items,
+        )
+```
+
+---
+
+## Two Ways to Define Actions
+
+### Class-Based Actions (Recommended for Complex Actions)
+
+Use `BaseAction` subclass for:
+- Complex actions with multiple methods
+- Actions that need class attributes (input_type, output_type)
+- Step-based workflows
+- Actions requiring autolog integration
+
+```python
+from synapse_sdk.plugins import BaseAction
+from synapse_sdk.plugins.enums import PluginCategory
+from synapse_sdk.plugins.types import YOLODataset, ModelWeights
+
+class TrainAction(BaseAction[TrainParams]):
+    """Train a YOLO model with comprehensive features."""
+
+    # Optional class attributes
+    action_name = 'train'
+    category = PluginCategory.NEURAL_NET
+    input_type = YOLODataset      # Semantic input type
+    output_type = ModelWeights    # Semantic output type
+    result_model = TrainResult    # Typed result schema
+
+    def execute(self) -> TrainResult:
+        # Enable automatic logging for Ultralytics
+        self.autolog('ultralytics')
+
+        # Access environment variables
+        api_key = self.ctx.env.get('API_KEY', '')
+
+        # Use the logger
+        self.logger.info('Starting training')
+
+        # Training implementation
+        model = train_model(self.params)
+
+        return TrainResult(
+            weights_path=model.path,
+            final_loss=model.loss,
+        )
+```
+
+### Function-Based Actions (Simple Actions)
+
+Use `@action` decorator for:
+- Simple, stateless operations
+- Quick prototyping
+- Actions without complex dependencies
+
+```python
+from synapse_sdk.plugins import action
+from synapse_sdk.plugins.context import RuntimeContext
+from pydantic import BaseModel
+
+class ConvertParams(BaseModel):
+    input_format: str
+    output_format: str
+    file_path: str
+
+class ConvertResult(BaseModel):
+    output_path: str
+    converted: bool
+
+@action(
+    name='convert',
+    description='Convert file between formats',
+    params=ConvertParams,
+    result=ConvertResult,
+)
+def convert(params: ConvertParams, context: RuntimeContext) -> ConvertResult:
+    """Convert a file from one format to another."""
+
+    # Log progress
+    context.set_progress(0, 100)
+
+    # Conversion logic
+    output_path = convert_file(
+        params.file_path,
+        params.input_format,
+        params.output_format,
+    )
+
+    context.set_progress(100, 100)
+
+    return ConvertResult(
+        output_path=output_path,
+        converted=True,
+    )
+```
+
+### Comparison: When to Use Each
+
+| Aspect | Class-Based (`BaseAction`) | Function-Based (`@action`) |
+|--------|---------------------------|---------------------------|
+| Complexity | Complex, multi-method | Simple, single function |
+| State | Can have instance state | Stateless |
+| Type declarations | `input_type`, `output_type` | Not supported |
+| Autolog | `self.autolog()` | Not available |
+| Step workflows | Supported via mixins | Not supported |
+| Testing | Easier mocking | Direct function calls |
+| Config discovery | Full introspection | Limited metadata |
+
+---
+
+## Plugin Categories
+
+Plugins are organized into categories for management and UI presentation:
+
+| Category | Value | Description | Example Use Cases |
+|----------|-------|-------------|-------------------|
+| Neural Net | `neural_net` | ML model training | YOLO training, classification |
+| Export | `export` | Model conversion/export | ONNX export, TensorRT |
+| Upload | `upload` | Data/model uploads | S3 upload, model registry |
+| Smart Tool | `smart_tool` | Interactive annotation | Auto-labeling, SAM |
+| Pre-Annotation | `pre_annotation` | Before annotation | Data preprocessing |
+| Post-Annotation | `post_annotation` | After annotation | QA checks, format conversion |
+| Data Validation | `data_validation` | Data quality checks | Schema validation |
+| Custom | `custom` | Custom functionality | Anything else |
+
+Usage in actions:
+
+```python
+from synapse_sdk.plugins.enums import PluginCategory
+
+class MyAction(BaseAction[MyParams]):
+    category = PluginCategory.NEURAL_NET
+```
+
+---
+
+## Execution Modes
+
+### Local Execution
+
+Best for development and testing. Runs in the current process:
+
+```python
+from synapse_sdk.plugins import run_plugin
+
+result = run_plugin(
+    plugin_code='my_plugin',
+    action='train',
+    params={'epochs': 10},
+    mode='local',
+)
+```
+
+Or using the executor directly:
+
+```python
+from synapse_sdk.plugins.executors import LocalExecutor
+
+executor = LocalExecutor(
+    env={'DEBUG': 'true'},  # Environment variables
+    job_id='test-123',       # Optional job tracking ID
+)
+
+result = executor.execute(
+    action_cls=TrainAction,
+    params={'epochs': 10},
+)
+```
+
+### Ray Task Execution
+
+For parallel tasks with fast startup. Uses Ray Actors:
+
+```python
+from synapse_sdk.plugins.executors.ray import RayActorExecutor
+
+executor = RayActorExecutor(
+    working_dir='/path/to/plugin',  # Plugin code directory
+    num_gpus=1,                      # GPU allocation
+    num_cpus=4,                      # CPU allocation
+)
+
+result = executor.execute(
+    action_cls=TrainAction,
+    params={'epochs': 10},
+)
+
+# Clean up when done
+executor.shutdown()
+```
+
+### Ray Job Execution
+
+For heavy workloads requiring full isolation:
+
+```python
+from synapse_sdk.plugins.executors.ray import RayJobExecutor
+
+executor = RayJobExecutor(
+    dashboard_url='http://localhost:8265',
+    working_dir='/path/to/plugin',
+)
+
+# Submit job (async)
+job_id = executor.submit('train', {'epochs': 100})
+
+# Check status
+status = executor.get_status(job_id)
+print(f"Status: {status}")  # PENDING, RUNNING, SUCCEEDED, FAILED
+
+# Get logs
+logs = executor.get_logs(job_id)
+
+# Wait for completion
+result = executor.wait(job_id, timeout_seconds=3600)
+```
+
+### Execution Mode Comparison
+
+| Mode | Startup | Isolation | Best For |
+|------|---------|-----------|----------|
+| `local` | Instant | None | Development, testing |
+| `task` | <1s | Process | Light parallel tasks |
+| `job` | ~30s | Full | Production, heavy workloads |
+
+---
+
+## RuntimeContext Reference
+
+All actions receive a `RuntimeContext` with these capabilities:
+
+```python
+def execute(self) -> dict:
+    # Access the logger
+    self.ctx.logger.info('Processing started')
+
+    # Access environment variables
+    api_key = self.ctx.env.get('API_KEY')
+    debug = self.ctx.env.get('DEBUG', 'false') == 'true'
+
+    # Log structured events
+    self.ctx.log('event_name', {'key': 'value'})
+
+    # Track progress
+    self.ctx.set_progress(current=50, total=100, category='training')
+
+    # Record metrics
+    self.ctx.set_metrics({'loss': 0.05, 'accuracy': 0.95}, category='train')
+
+    # Log user-visible messages
+    self.ctx.log_message('Training complete!')
+
+    # Log developer debug events
+    self.ctx.log_dev_event('checkpoint_saved', {'path': '/model.pt'})
+
+    # Access job ID for tracking
+    job_id = self.ctx.job_id
+
+    # Access backend client (if available)
+    if self.ctx.client:
+        self.ctx.client.upload_file(...)
+```
+
+---
+
+## PluginEnvironment
+
+Load configuration from environment variables or files:
+
+```python
+from synapse_sdk.plugins.context.env import PluginEnvironment
+
+# From environment variables with prefix
+env = PluginEnvironment.from_environ(prefix='PLUGIN_')
+
+# From TOML file
+env = PluginEnvironment.from_file('config.toml')
+
+# Type-safe getters
+api_key = env.get_str('API_KEY')
+batch_size = env.get_int('BATCH_SIZE', default=32)
+learning_rate = env.get_float('LEARNING_RATE', default=0.001)
+debug = env.get_bool('DEBUG', default=False)
+tags = env.get_list('TAGS', default=[])
+
+# Merge multiple environments
+merged = env.merge(another_env)
+```
+
+---
+
+## Next Steps
+
+- **[ACTION_DEV_GUIDE.md](ACTION_DEV_GUIDE.md)**: Complete action development guide
+- **[ARCHITECTURE.md](ARCHITECTURE.md)**: Deep dive into system architecture
+- **[STEP.md](STEP.md)**: Step implementations guide
+- **[LOGGING_SYSTEM.md](LOGGING_SYSTEM.md)**: Logging and progress tracking
+- **[README.md](README.md)**: Quick reference and extension guide
+- **[PIPELINE_GUIDE.md](PIPELINE_GUIDE.md)**: Multi-action pipelines

synapse_sdk/plugins/docs/PIPELINE_GUIDE.md

@@ -0,0 +1,145 @@
+# Remote Pipeline Execution
+
+Multi-action pipeline execution on Ray clusters with progress tracking and checkpointing.
+
+## Architecture
+
+```mermaid
+flowchart LR
+    subgraph Local["SDK (local)"]
+        pipeline["ActionPipeline<br/>.submit()<br/>.wait()"]
+        executor["RayPipelineExecutor<br/>- get_progress()<br/>- get_result()"]
+    end
+
+    subgraph Ray["Ray Cluster"]
+        actor["PipelineActor<br/>- run_pipeline<br/>- checkpoints"]
+    end
+
+    subgraph Backend["Backend Service"]
+        devapi["Backend Service<br/>/api/v1/runs/<br/>/api/v1/progress/"]
+    end
+
+    pipeline -->|"submit()"| actor
+    executor -->|"poll"| devapi
+    actor -->|"report"| devapi
+    devapi <-->|"HTTP"| executor
+
+    style Local fill:#e1f5fe
+    style Ray fill:#f3e5f5
+    style Backend fill:#fff3e0
+```
+
+**Key Design Decision:** Ray Workflows is deprecated. We use a single persistent Ray actor + custom orchestration via a backend service.
+
+## Quick Start
+
+```python
+from synapse_sdk.plugins.pipelines import ActionPipeline
+from synapse_sdk.plugins.executors.ray import RayPipelineExecutor
+
+pipeline = ActionPipeline([DownloadAction, ConvertAction, TrainAction])
+
+executor = RayPipelineExecutor(
+    ray_address='auto',
+    working_dir='/path/to/plugin',
+    pipeline_service_url='http://localhost:8100',
+)
+
+# Remote execution
+run_id = pipeline.submit({'dataset_id': 123}, executor)
+progress = executor.get_progress(run_id)
+result = pipeline.wait(run_id, executor)
+
+# Stream progress with Rich display (sync)
+from synapse_sdk.plugins.pipelines import display_progress
+final = display_progress(executor.stream_progress(run_id))
+
+# Stream progress with Rich display (async)
+from synapse_sdk.plugins.pipelines import display_progress_async
+final = await display_progress_async(executor.stream_progress_async(run_id))
+
+# Resume from a failed run
+new_run_id = pipeline.submit({'dataset_id': 123}, executor, resume_from=run_id)
+
+# Local execution (unchanged)
+result = pipeline.execute(params, ctx)
+```
+
+## File Structure
+
+```
+synapse_sdk/
+├── plugins/pipelines/
+│   ├── action_pipeline.py   # ActionPipeline with submit()/wait()
+│   ├── context.py           # PipelineContext (shared working dir)
+│   ├── display.py           # Rich console progress display
+│   └── models.py            # PipelineProgress, ActionProgress, RunStatus
+├── clients/pipeline/
+│   └── client.py            # PipelineServiceClient (backend client, SSE streaming)
+└── plugins/executors/ray/
+    └── pipeline.py          # RayPipelineExecutor, _PipelineExecutorActor
+```
+
+## API Reference
+
+### SDK Classes
+
+| Class/Function | Location | Purpose |
+|----------------|----------|---------|
+| `ActionPipeline` | `plugins/pipelines/action_pipeline.py` | Chain actions, local or remote |
+| `RayPipelineExecutor` | `plugins/executors/ray/pipeline.py` | Submit to Ray, track progress |
+| `PipelineServiceClient` | `clients/pipeline/client.py` | Communicate with backend service |
+| `PipelineContext` | `plugins/pipelines/context.py` | Shared working directory |
+| `PipelineProgress` | `plugins/pipelines/models.py` | Progress state model |
+| `display_progress()` | `plugins/pipelines/display.py` | Rich console progress (sync) |
+| `display_progress_async()` | `plugins/pipelines/display.py` | Rich console progress (async) |
+| `stream_progress()` | `RayPipelineExecutor` method | SSE progress generator (sync) |
+| `stream_progress_async()` | `RayPipelineExecutor` method | SSE progress generator (async) |
+
+## Design Decisions
+
+| Decision | Choice | Rationale |
+|----------|--------|-----------|
+| Orchestration | Custom (not Ray Workflows) | Ray Workflows deprecated |
+| Execution | Single persistent Ray actor | Shared filesystem, simpler state |
+| Working dir | `/tmp/synapse_pipelines/{id}` | Ephemeral, auto-cleanup on reboot |
+| Progress store | External backend service | Decoupled from Ray, queryable |
+| Checkpointing | Yes | Enable resume on failure |
+
+## Remaining Work
+
+### Phase 2: Progress Streaming (Complete)
+- [x] SSE endpoint: `GET /runs/{id}/progress/stream`
+- [x] `executor.stream_progress()` generator
+- [x] Rich console progress display (`display_progress()`)
+
+### Phase 3: Checkpointing & Resume (Complete)
+- [x] `submit(resume_from=run_id)` - Resume from latest checkpoint of a previous run
+- [x] Skip completed actions on resume (marks them as SKIPPED)
+- [x] Restore accumulated params from checkpoint's `params_snapshot`
+
+### Phase 4: DAG Pipelines (Future)
+- [ ] Parallel action branches
+- [ ] Per-action resource requirements
+- [ ] Dependency graph execution
+
+## Troubleshooting
+
+**Ray actor fails to import actions:**
+- Ensure `working_dir` contains the plugin code
+- Check `include_sdk=True` if using local SDK changes
+- Actions must be in a proper module (not `__main__`), importable on the remote worker
+
+**Remote cluster connectivity:**
+- Use `actor_pipeline_service_url` parameter for the URL the actor should use
+- The local SDK uses `pipeline_service_url` (e.g., `localhost:8100`)
+- The remote actor uses `actor_pipeline_service_url` (e.g., internal network address)
+
+```python
+executor = RayPipelineExecutor(
+    ray_address="ray://remote-cluster:10001",
+    pipeline_service_url="http://localhost:8100",  # Local SDK access
+    actor_pipeline_service_url="http://internal-service:8100",  # Actor access
+    include_sdk=True,
+)
+```