synapse-sdk 1.0.0a35__py3-none-any.whl → 2025.11.7__py3-none-any.whl

synapse_sdk/devtools/docs/docs/plugins/export-plugins.md

@@ -0,0 +1,1092 @@
+---
+id: export-plugins
+title: Export Plugins
+sidebar_position: 2
+---
+
+# Export Plugins
+
+Export plugins provide data export and transformation operations for exporting annotated data, ground truth datasets, assignments, and tasks from the Synapse platform.
+
+## Overview
+
+**Available Actions:**
+
+- `export` - Export data from various sources (assignments, ground truth, tasks) with customizable processing
+
+**Use Cases:**
+
+- Exporting annotated datasets for training
+- Converting ground truth data to custom formats
+- Creating data packages for distribution
+- Batch processing of assignment results
+- Transforming annotation data for external tools
+
+**Supported Export Targets:**
+
+- `assignment` - Export assignment data with annotations
+- `ground_truth` - Export ground truth dataset versions
+- `task` - Export task data with associated annotations
+
+## BaseExporter and Exporter Class Architecture
+
+The following diagrams illustrate the relationship between BaseExporter and Exporter classes and their method implementations:
+
+```mermaid
+classDiagram
+    %% Light/Dark mode compatible colors using CSS variables
+    classDef baseClass fill:#f0f8ff,stroke:#4169e1,stroke-width:2px,color:#000
+    classDef childClass fill:#f0fff0,stroke:#228b22,stroke-width:2px,color:#000
+    classDef method fill:#fff8dc,stroke:#daa520,stroke-width:1px,color:#000
+    classDef abstractMethod fill:#ffe4e1,stroke:#dc143c,stroke-width:1px,color:#000
+    classDef helperMethod fill:#f5f5f5,stroke:#696969,stroke-width:1px,color:#000
+
+    class BaseExporter {
+        %% Core attributes
+        +run: object
+        +export_items: Generator
+        +path_root: Path
+        +params: dict
+
+        %% Main workflow methods
+        +export(export_items, results, **kwargs) dict
+        +process_data_conversion(export_item) object
+        +process_file_saving(...) bool
+        +setup_output_directories(path, flag) dict
+
+        %% Abstract methods (to be implemented)
+        +convert_data(data)* object
+        +before_convert(data)* object
+        +after_convert(data)* object
+
+        %% File operations
+        +save_original_file(result, path, errors) ExportStatus
+        +save_as_json(result, path, errors) ExportStatus
+        +get_original_file_name(files) str
+
+        %% Helper methods
+        -_create_unique_export_path(name) Path
+        -_save_error_list(path, json_errors, file_errors)
+        -_process_original_file_saving(...) bool
+        -_process_json_file_saving(...) bool
+    }
+
+    class Exporter {
+        %% Inherited from BaseExporter
+        +export(export_items, results, **kwargs) dict
+
+        %% Implemented abstract methods
+        +convert_data(data) object
+        +before_convert(data) object
+        +after_convert(data) object
+    }
+
+    %% Inheritance relationship
+    BaseExporter <|-- Exporter
+
+    %% Apply styles
+    class BaseExporter baseClass
+    class Exporter childClass
+```
+
+### Method Execution Flow
+
+This flowchart shows the complete execution flow of export operations:
+
+```mermaid
+flowchart TD
+    %% Start
+    A[export method called] --> B[Initialize paths and metrics]
+    B --> C[Setup output directories]
+    C --> D[Loop through export_items]
+
+    %% Data processing pipeline
+    D --> E[process_data_conversion]
+    E --> F[before_convert]
+    F --> G[convert_data]
+    G --> H[after_convert]
+
+    %% File saving pipeline
+    H --> I[process_file_saving]
+    I --> J{save_original_file?}
+    J -->|Yes| K[_process_original_file_saving]
+    J -->|No| L[_process_json_file_saving]
+    K --> L
+
+    %% Continue or finish
+    L --> M{More items?}
+    M -->|Yes| D
+    M -->|No| N[Save error lists]
+    N --> O[Return export path]
+
+    %% Error handling
+    K --> P{Original file failed?}
+    P -->|Yes| Q[Skip to next item]
+    P -->|No| L
+
+    L --> R{JSON file failed?}
+    R -->|Yes| Q
+    R -->|No| S[Update metrics]
+    S --> M
+    Q --> M
+
+    %% Styling for light/dark compatibility
+    classDef startEnd fill:#e1f5fe,stroke:#01579b,color:#000
+    classDef process fill:#f3e5f5,stroke:#4a148c,color:#000
+    classDef decision fill:#fff3e0,stroke:#e65100,color:#000
+    classDef data fill:#e8f5e8,stroke:#2e7d32,color:#000
+    classDef error fill:#ffebee,stroke:#c62828,color:#000
+
+    class A,O startEnd
+    class B,C,E,F,G,H,I,K,L,N,S process
+    class J,M,P,R decision
+    class D data
+    class Q error
+```
+
+### Key Relationships and Responsibilities
+
+**BaseExporter (Abstract Base Class)**
+
+- **Core Functionality**: Provides complete export workflow infrastructure
+- **Template Methods**: `export()` method orchestrates the entire process
+- **Hook Methods**: `convert_data()`, `before_convert()`, `after_convert()` for customization
+- **Utilities**: File operations, directory setup, error handling, progress tracking
+
+**Exporter (Concrete Implementation)**
+
+- **Inheritance**: Extends `BaseExporter`
+- **Minimal Implementation**: Provides basic implementations of abstract methods
+- **Pass-through Behavior**: Most methods delegate to parent class
+- **Customization Points**: Override conversion methods for specific logic
+
+### Method Categories
+
+- **🔵 Core Workflow**: Main export orchestration methods
+- **🟢 Template/Hook**: Methods designed to be overridden by subclasses
+- **🟡 File Operations**: Concrete file saving and handling methods
+- **🔸 Helper/Utility**: Private methods for internal operations
+
+The design follows the **Template Method Pattern** where `BaseExporter.export()` defines the algorithm skeleton, and subclasses customize specific steps through the hook methods.
+
+## Plugin Configuration
+
+Export plugin templates include configuration fields for filtering and plugin discovery:
+
+```yaml
+actions:
+  export:
+    entrypoint: plugin.export.Exporter
+    annotation_types:
+      - image
+      - video
+      - audio
+      - text
+      - pcd
+      - prompt
+
+data_types:
+  - image
+  - video
+  - audio
+  - text
+  - pcd
+```
+
+### Configuration Fields
+
+- **data_types**: List of supported data types for filtering export plugins (plugin-level filter)
+
+  - Supported values: `image`, `video`, `audio`, `text`, `pcd`
+  - Used by the platform to filter and display relevant export plugins to users based on their data type
+
+- **annotation_types**: List of supported annotation types for filtering export plugins (action-level filter)
+  - Supported values: `image`, `video`, `audio`, `text`, `pcd`, `prompt`
+  - Defined within each action's configuration (e.g., `actions.export.annotation_types`)
+  - Used by the platform to filter and display relevant export plugins to users based on their annotation type
+  - Each action can have different annotation type requirements
+
+**Best Practice**: Customize these fields to accurately reflect your plugin's capabilities. The template includes all common types as examples, but you should modify the list to match what your plugin actually supports.
+
+## BaseExporter Class Structure
+
+The new BaseExporter class provides an object-oriented approach for export plugins:
+
+```python
+from synapse_sdk.plugins.categories.export.templates.plugin import BaseExporter
+
+class Exporter(BaseExporter):
+    """Plugin export action interface for organizing files."""
+
+    def __init__(self, run, export_items, path_root, **params):
+        """Initialize the plugin export action class."""
+        super().__init__(run, export_items, path_root, **params)
+
+    def convert_data(self, data):
+        """Converts the data."""
+        return data
+
+    def before_convert(self, data):
+        """Preprocesses the data before conversion."""
+        return data
+
+    def after_convert(self, data):
+        """Post-processes the data after conversion."""
+        return data
+```
+
+## BaseExporter Core Functionality
+
+### Auto-provided Utilities
+
+- **Complete Export Workflow**: `export()` method handles the complete export process
+- **Data Conversion Pipeline**: `process_data_conversion()` handles before_convert → convert_data → after_convert processing
+- **File Saving Management**: `process_file_saving()` handles original file and JSON file saving operations (can be overridden)
+- **Output Directory Setup**: `setup_output_directories()` creates output directory structure (can be overridden)
+
+### Required Methods (should be implemented by subclasses)
+
+- **convert_data()**: Transform data during export
+
+### Optional Methods (can be overridden by subclasses)
+
+- **save_original_file()**: Save original files from export items
+- **save_as_json()**: Save data as JSON files
+- **before_convert()**: Pre-process data before conversion
+- **after_convert()**: Post-process data after conversion
+- **process_file_saving()**: Custom file saving logic
+- **additional_file_saving()**: Save additional files after processing all export items
+
+### Helper Methods
+
+- **\_process_original_file_saving()**: Handle original file saving with metrics
+- **\_process_json_file_saving()**: Handle JSON file saving with metrics
+
+### Auto-provided Utilities
+
+- Progress tracking via `self.run.set_progress()`
+- Logging via `self.run.log_message()` and other run methods
+- Error handling and metrics collection via self.run methods
+
+## Additional File Saving
+
+The `additional_file_saving()` method is called after all export items have been processed and is designed for saving files that depend on the collective data from all processed items. This is useful for creating:
+
+- Metadata files (e.g., dataset statistics, class mappings)
+- Configuration files (e.g., dataset.yaml for YOLO, classes.txt)
+- Summary files (e.g., export reports, processing logs)
+- Index files (e.g., file lists, directory structures)
+
+### Method Signature
+
+```python
+def additional_file_saving(self, unique_export_path):
+    """Save additional files after processing all export items.
+
+    This method is called after the main export loop completes and is intended
+    for saving files that need to be created based on the collective data from
+    all processed export items (e.g., metadata files, configuration files,
+    summary files, etc.).
+
+    Args:
+        unique_export_path (str): The unique export directory path where
+            additional files should be saved.
+    """
+    pass
+```
+
+### Example Usage
+
+```python
+class YOLOExporter(BaseExporter):
+    def __init__(self, run, export_items, path_root, **params):
+        super().__init__(run, export_items, path_root, **params)
+        self.class_names = set()
+        self.dataset_stats = {
+            'total_images': 0,
+            'total_annotations': 0,
+            'class_distribution': {}
+        }
+
+    def convert_data(self, data):
+        # Track classes and stats during conversion
+        for annotation in data.get('annotations', []):
+            class_name = annotation['class_name']
+            self.class_names.add(class_name)
+            self.dataset_stats['class_distribution'][class_name] = \
+                self.dataset_stats['class_distribution'].get(class_name, 0) + 1
+
+        self.dataset_stats['total_images'] += 1
+        self.dataset_stats['total_annotations'] += len(data.get('annotations', []))
+
+        return data  # ... rest of conversion logic
+
+    def additional_file_saving(self, unique_export_path):
+        """Save YOLO configuration and metadata files."""
+        data_dir = Path(unique_export_path) / 'data'
+        data_dir.mkdir(exist_ok=True)
+
+        # 1. Save classes.txt file
+        classes_file = data_dir / 'classes.txt'
+        with classes_file.open('w') as f:
+            for class_name in sorted(self.class_names):
+                f.write(f"{class_name}\n")
+        self.run.log_message(f"Saved classes file: {classes_file}")
+
+        # 2. Save dataset.yaml file
+        dataset_config = {
+            'path': str(unique_export_path),
+            'train': 'images',
+            'val': 'images',
+            'names': {i: name for i, name in enumerate(sorted(self.class_names))}
+        }
+
+        dataset_file = data_dir / 'dataset.yaml'
+        with dataset_file.open('w') as f:
+            yaml.dump(dataset_config, f, default_flow_style=False)
+        self.run.log_message(f"Saved dataset config: {dataset_file}")
+
+        # 3. Save export statistics
+        stats_file = data_dir / 'export_stats.json'
+        with stats_file.open('w') as f:
+            json.dump(self.dataset_stats, f, indent=2)
+        self.run.log_message(f"Saved export statistics: {stats_file}")
+```
+
+### Common Use Cases
+
+#### 1. Dataset Configuration Files
+
+```python
+def additional_file_saving(self, unique_export_path):
+    # Create dataset configuration for training frameworks
+    config = {
+        'dataset_name': self.params.get('name'),
+        'created_at': datetime.now().isoformat(),
+        'total_samples': len(self.processed_items),
+        'classes': list(self.class_mapping.keys())
+    }
+
+    config_file = Path(unique_export_path) / 'dataset_config.json'
+    with config_file.open('w') as f:
+        json.dump(config, f, indent=2)
+```
+
+#### 2. Export Summary Reports
+
+```python
+def additional_file_saving(self, unique_export_path):
+    # Generate export summary
+    summary = {
+        'export_info': {
+            'plugin_name': self.__class__.__name__,
+            'export_time': datetime.now().isoformat(),
+            'export_path': str(unique_export_path)
+        },
+        'statistics': self.get_export_statistics(),
+        'errors': self.get_error_summary()
+    }
+
+    summary_file = Path(unique_export_path) / 'export_summary.json'
+    with summary_file.open('w') as f:
+        json.dump(summary, f, indent=2)
+```
+
+#### 3. Index and Manifest Files
+
+```python
+def additional_file_saving(self, unique_export_path):
+    # Create file index for processed items
+    file_index = []
+    for item in self.processed_items:
+        file_index.append({
+            'original_file': item['original_filename'],
+            'json_file': f"{item['stem']}.json",
+            'processed_at': item['timestamp']
+        })
+
+    index_file = Path(unique_export_path) / 'file_index.json'
+    with index_file.open('w') as f:
+        json.dump(file_index, f, indent=2)
+```
+
+## Key Features
+
+- **Progress Tracking**: Built-in progress monitoring with `run.set_progress()`
+- **Error Handling**: Automatic error collection and reporting
+- **Metrics Logging**: Track success/failure rates with `run.log_metrics()`
+- **File Management**: Handles both original files and processed JSON data
+- **Logging**: Comprehensive logging with `run.log_message()` and custom events
+
+## Practical Examples
+
+### YOLO Format Exporter with Custom Directory Structure
+
+Here's a complete example that exports data in YOLO format while leveraging `setup_output_directories` and `process_file_saving`:
+
+```python
+from synapse_sdk.plugins.categories.export.templates.plugin import BaseExporter
+import os
+import json
+
+class YOLOExporter(BaseExporter):
+    """Plugin to export data in YOLO format."""
+
+    def __init__(self, run, export_items, path_root, **params):
+        super().__init__(run, export_items, path_root, **params)
+        self.class_mapping = {}
+
+    def setup_output_directories(self, unique_export_path, save_original_file_flag):
+        """Create directories matching YOLO project structure."""
+        directories = ['images', 'labels', 'data']
+
+        for directory in directories:
+            dir_path = os.path.join(unique_export_path, directory)
+            os.makedirs(dir_path, exist_ok=True)
+            self.run.log_message(f"Created YOLO directory: {dir_path}")
+
+        return unique_export_path
+
+    def convert_data(self, data):
+        """Convert annotation data to YOLO format."""
+        converted_annotations = []
+
+        for annotation in data.get('annotations', []):
+            # Convert bounding box to YOLO format
+            bbox = annotation['geometry']['bbox']
+            image_width = data['image']['width']
+            image_height = data['image']['height']
+
+            # YOLO format: center_x, center_y, width, height (normalized)
+            center_x = (bbox['x'] + bbox['width'] / 2) / image_width
+            center_y = (bbox['y'] + bbox['height'] / 2) / image_height
+            width = bbox['width'] / image_width
+            height = bbox['height'] / image_height
+
+            # Class ID mapping
+            class_name = annotation['class_name']
+            if class_name not in self.class_mapping:
+                self.class_mapping[class_name] = len(self.class_mapping)
+
+            class_id = self.class_mapping[class_name]
+
+            converted_annotations.append({
+                'class_id': class_id,
+                'center_x': center_x,
+                'center_y': center_y,
+                'width': width,
+                'height': height
+            })
+
+        return {
+            'yolo_annotations': converted_annotations,
+            'class_mapping': self.class_mapping,
+            'image_info': data['image']
+        }
+
+    def process_file_saving(
+        self,
+        final_data,
+        unique_export_path,
+        save_original_file_flag,
+        errors_json_file_list,
+        errors_original_file_list,
+        original_file_metrics_record,
+        data_file_metrics_record,
+        current_index,
+    ):
+        """Handle file saving in YOLO format."""
+        try:
+            export_item = self.export_items[current_index - 1]
+            base_name = os.path.splitext(export_item.original_file.name)[0]
+
+            # 1. Save image file to images folder
+            if save_original_file_flag:
+                images_dir = os.path.join(unique_export_path, 'images')
+                image_path = os.path.join(images_dir, export_item.original_file.name)
+                import shutil
+                shutil.copy2(export_item.original_file.path, image_path)
+                self.run.log_message(f"Saved image: {image_path}")
+
+            # 2. Save YOLO label file to labels folder
+            labels_dir = os.path.join(unique_export_path, 'labels')
+            label_path = os.path.join(labels_dir, f"{base_name}.txt")
+
+            with open(label_path, 'w') as f:
+                for ann in final_data.get('yolo_annotations', []):
+                    line = f"{ann['class_id']} {ann['center_x']} {ann['center_y']} {ann['width']} {ann['height']}\n"
+                    f.write(line)
+
+            self.run.log_message(f"Saved YOLO label: {label_path}")
+
+            # 3. Save class mapping file (only once)
+            if current_index == 1:  # Only when processing first file
+                classes_path = os.path.join(unique_export_path, 'data', 'classes.txt')
+                with open(classes_path, 'w') as f:
+                    for class_name, class_id in sorted(final_data['class_mapping'].items(), key=lambda x: x[1]):
+                        f.write(f"{class_name}\n")
+                self.run.log_message(f"Saved classes file: {classes_path}")
+
+            return True
+
+        except Exception as e:
+            self.run.log_message(f"Error during file saving: {str(e)}", level="error")
+            errors_json_file_list.append(f"Export item {current_index}: {str(e)}")
+            return True  # Return True to continue processing other files
+```
+
+This example demonstrates how to leverage BaseExporter's key extension points `setup_output_directories` and `process_file_saving` to:
+
+- Create YOLO project structure (`images/`, `labels/`, `data/`)
+- Save image files and YOLO label files in appropriate locations
+- Manage class mapping files
+- Track progress and handle errors
+
+## Quick Start Guide
+
+Step-by-step guide to create a simple plugin using BaseExporter:
+
+### Step 1: Inherit Base Class
+
+```python
+from synapse_sdk.plugins.categories.export.templates.plugin import BaseExporter
+
+class MyExporter(BaseExporter):
+    def convert_data(self, data):
+        # Required: Implement data transformation logic
+        return data  # or return transformed data
+```
+
+### Step 2: Override Additional Methods as Needed
+
+```python
+def before_convert(self, data):
+    # Optional: Pre-processing before conversion
+    return data
+
+def after_convert(self, converted_data):
+    # Optional: Post-processing after conversion
+    return converted_data
+
+def save_as_json(self, converted_data, output_path):
+    # Optional: Custom save format
+    # By default saves as JSON format
+    pass
+```
+
+### Step 3: Register Plugin
+
+Plugin directory structure:
+
+```
+my_plugin/
+├── __init__.py
+├── plugin.py          # MyExporter class definition
+└── manifest.yaml       # Plugin metadata
+```
+
+## Export Action Architecture
+
+The export system has been refactored into a modular architecture with specialized components for different aspects of data export processing:
+
+```mermaid
+classDiagram
+    %% Light/Dark mode compatible colors with semi-transparency
+    classDef baseClass fill:#e1f5fe80,stroke:#0288d1,stroke-width:2px
+    classDef childClass fill:#c8e6c980,stroke:#388e3c,stroke-width:2px
+    classDef modelClass fill:#fff9c480,stroke:#f57c00,stroke-width:2px
+    classDef utilClass fill:#f5f5f580,stroke:#616161,stroke-width:2px
+    classDef enumClass fill:#ffccbc80,stroke:#d32f2f,stroke-width:2px
+
+    class ExportAction {
+        +name: str = "export"
+        +category: PluginCategory.EXPORT
+        +method: RunMethod.JOB
+        +run_class: ExportRun
+        +params_model: ExportParams
+        +progress_categories: dict
+        +metrics_categories: dict
+
+        +start() dict
+        +get_exporter(...) object
+        +_get_export_items(target, filter) Generator
+        +_create_target_handler(target) object
+    }
+
+    class ExportRun {
+        +log_message_with_code(code, args, level) None
+        +log_export_event(code, args, level) None
+        +export_log_json_file(id, file_info, status) None
+        +export_log_original_file(id, file_info, status) None
+        +ExportEventLog: BaseModel
+        +DataFileLog: BaseModel
+        +MetricsRecord: BaseModel
+    }
+
+    class ExportParams {
+        +name: str
+        +storage: int
+        +target: Literal["assignment", "ground_truth", "task"]
+        +filter: dict
+        +path: str
+        +save_original_file: bool = True
+        +extra_params: dict = {}
+
+        +check_storage_exists(value) str
+        +validate_target_filter(cls, values) dict
+    }
+
+    class LogCode {
+        +EXPORT_STARTED: str
+        +ITEMS_DISCOVERED: str
+        +CONVERSION_STARTED: str
+        +CONVERSION_COMPLETED: str
+        +FILE_SAVED: str
+        +EXPORT_COMPLETED: str
+        +EXPORT_FAILED: str
+    }
+
+    class ExportStatus {
+        +SUCCESS: str = "success"
+        +FAILED: str = "failed"
+        +SKIPPED: str = "skipped"
+    }
+
+    class ExportError {
+        +message: str
+        +code: str
+    }
+
+    class ExportValidationError {
+        +message: str
+        +field: str
+    }
+
+    class ExportTargetError {
+        +message: str
+        +target: str
+    }
+
+    %% Relationships
+    ExportAction --> ExportRun : uses
+    ExportAction --> ExportParams : validates with
+    ExportRun --> LogCode : logs with
+    ExportRun --> ExportStatus : tracks status
+    ExportAction --> ExportError : may raise
+    ExportAction --> ExportValidationError : may raise
+    ExportAction --> ExportTargetError : may raise
+
+    %% Apply styles
+    class ExportAction baseClass
+    class ExportRun childClass
+    class ExportParams modelClass
+    class LogCode,ExportStatus enumClass
+    class ExportError,ExportValidationError,ExportTargetError utilClass
+```
+
+### Modular Structure
+
+The export action follows a clean modular organization:
+
+```
+synapse_sdk/plugins/categories/export/actions/export/
+├── __init__.py        # Clean module interfaces
+├── action.py          # ExportAction class
+├── enums.py          # ExportStatus, LogCode + LOG_MESSAGES
+├── exceptions.py     # Export-specific exceptions
+├── models.py         # ExportParams model
+├── run.py           # ExportRun class
+└── utils.py         # Target handlers and utilities
+```
+
+**Key Benefits:**
+
+- **Enhanced Maintainability**: Smaller, focused files are easier to understand and modify
+- **Code Consistency**: Export action now follows the same pattern as upload action
+- **Better Organization**: Related functionality is grouped logically
+- **Improved Readability**: Clear separation of concerns across modules
+
+## Creating Export Plugins
+
+Export plugins now use the BaseExporter class-based approach for better organization and reusability. Here's how to create a custom export plugin:
+
+### Step 1: Generate Export Plugin Template
+
+```bash
+synapse plugin create
+# Select 'export' as category
+# Plugin will be created with export template
+```
+
+### Step 2: Customize Export Parameters
+
+The `ExportParams` model defines the required parameters:
+
+```python
+from synapse_sdk.plugins.categories.export.actions.export.models import ExportParams
+from pydantic import BaseModel
+from typing import Literal
+
+class CustomExportParams(ExportParams):
+    # Add custom parameters
+    output_format: Literal['json', 'csv', 'xml'] = 'json'
+    include_metadata: bool = True
+    compression: bool = False
+```
+
+### Step 3: Implement Data Transformation
+
+Implement the required methods in your `Exporter` class in `plugin/export.py`:
+
+```python
+from datetime import datetime
+from synapse_sdk.plugins.categories.export.templates.plugin import BaseExporter
+
+class Exporter(BaseExporter):
+    """Custom export plugin with COCO format conversion."""
+
+    def convert_data(self, data):
+        """Convert annotation data to your desired format."""
+        # Example: Convert to COCO format
+        if data.get('data_type') == 'image_detection':
+            return self.convert_to_coco_format(data)
+        elif data.get('data_type') == 'image_classification':
+            return self.convert_to_classification_format(data)
+        return data
+
+    def before_convert(self, export_item):
+        """Preprocess data before conversion."""
+        # Add validation, filtering, or preprocessing
+        if not export_item.get('data'):
+            return None  # Skip empty items
+
+        # Add custom metadata
+        export_item['processed_at'] = datetime.now().isoformat()
+        return export_item
+
+    def after_convert(self, converted_data):
+        """Post-process converted data."""
+        # Add final touches, validation, or formatting
+        if 'annotations' in converted_data:
+            converted_data['annotation_count'] = len(converted_data['annotations'])
+        return converted_data
+
+    def convert_to_coco_format(self, data):
+        """Example: Convert to COCO detection format."""
+        coco_data = {
+            "images": [],
+            "annotations": [],
+            "categories": []
+        }
+
+        # Transform annotation data to COCO format
+        for annotation in data.get('annotations', []):
+            coco_annotation = {
+                "id": annotation['id'],
+                "image_id": annotation['image_id'],
+                "category_id": annotation['category_id'],
+                "bbox": annotation['bbox'],
+                "area": annotation.get('area', 0),
+                "iscrowd": 0
+            }
+            coco_data["annotations"].append(coco_annotation)
+
+        return coco_data
+```
+
+### Step 4: Configure Export Targets
+
+The export action supports different data sources:
+
+```python
+# Filter examples for different targets
+filters = {
+    # For ground truth export
+    "ground_truth": {
+        "ground_truth_dataset_version": 123,
+        "expand": ["data"]
+    },
+
+    # For assignment export
+    "assignment": {
+        "project": 456,
+        "status": "completed",
+        "expand": ["data"]
+    },
+
+    # For task export
+    "task": {
+        "project": 456,
+        "assignment": 789,
+        "expand": ["data_unit", "assignment"]
+    }
+}
+```
+
+### Step 5: Handle File Operations
+
+Customize file saving and organization by overriding BaseExporter methods:
+
+```python
+import json
+from pathlib import Path
+from synapse_sdk.plugins.categories.export.actions.export.enums import ExportStatus
+
+class Exporter(BaseExporter):
+    """Custom export plugin with multiple format support."""
+
+    def save_as_json(self, result, base_path, error_file_list):
+        """Custom JSON saving with different formats."""
+        file_name = Path(self.get_original_file_name(result['files'])).stem
+
+        # Choose output format based on params
+        if self.params.get('output_format') == 'csv':
+            return self.save_as_csv(result, base_path, error_file_list)
+        elif self.params.get('output_format') == 'xml':
+            return self.save_as_xml(result, base_path, error_file_list)
+
+        # Default JSON handling
+        json_data = result['data']
+        file_info = {'file_name': f'{file_name}.json'}
+
+        try:
+            with (base_path / f'{file_name}.json').open('w', encoding='utf-8') as f:
+                json.dump(json_data, f, indent=4, ensure_ascii=False)
+            status = ExportStatus.SUCCESS
+        except Exception as e:
+            error_file_list.append([f'{file_name}.json', str(e)])
+            status = ExportStatus.FAILED
+
+        self.run.export_log_json_file(result['id'], file_info, status)
+        return status
+
+    def setup_output_directories(self, unique_export_path, save_original_file_flag):
+        """Custom directory structure."""
+        # Create format-specific directories
+        output_paths = super().setup_output_directories(unique_export_path, save_original_file_flag)
+
+        # Add custom directories based on output format
+        format_dir = unique_export_path / self.params.get('output_format', 'json')
+        format_dir.mkdir(parents=True, exist_ok=True)
+        output_paths['format_output_path'] = format_dir
+
+        return output_paths
+```
+
+### Step 6: Usage Examples
+
+Running export plugins with different configurations:
+
+```bash
+# Basic export of ground truth data
+synapse plugin run export '{
+  "name": "my_export",
+  "storage": 1,
+  "target": "ground_truth",
+  "filter": {"ground_truth_dataset_version": 123},
+  "path": "exports/ground_truth",
+  "save_original_file": true
+}' --plugin my-export-plugin
+
+# Export assignments with custom parameters
+synapse plugin run export '{
+  "name": "assignment_export",
+  "storage": 1,
+  "target": "assignment",
+  "filter": {"project": 456, "status": "completed"},
+  "path": "exports/assignments",
+  "save_original_file": false,
+  "extra_params": {
+    "output_format": "coco",
+    "include_metadata": true
+  }
+}' --plugin custom-coco-export
+```
+
+## Common Export Patterns
+
+```python
+# Pattern 1: Format-specific conversion
+class Exporter(BaseExporter):
+    def convert_data(self, data):
+        """Convert to YOLO format."""
+        if data.get('task_type') == 'object_detection':
+            return self.convert_to_yolo_format(data)
+        return data
+
+# Pattern 2: Conditional file organization
+class Exporter(BaseExporter):
+    def setup_output_directories(self, unique_export_path, save_original_file_flag):
+        # Call parent method
+        output_paths = super().setup_output_directories(unique_export_path, save_original_file_flag)
+
+        # Create separate folders by category
+        for category in ['train', 'val', 'test']:
+            category_path = unique_export_path / category
+            category_path.mkdir(parents=True, exist_ok=True)
+            output_paths[f'{category}_path'] = category_path
+
+        return output_paths
+
+# Pattern 3: Batch processing with validation
+class Exporter(BaseExporter):
+    def before_convert(self, export_item):
+        # Validate required fields
+        required_fields = ['data', 'files', 'id']
+        for field in required_fields:
+            if field not in export_item:
+                raise ValueError(f"Missing required field: {field}")
+        return export_item
+```
+
+## Development Tips and Best Practices
+
+### 1. Error Handling
+
+```python
+def convert_data(self, data):
+    try:
+        # Conversion logic
+        result = self.process_annotations(data)
+        return result
+    except Exception as e:
+        self.run.log_message(f"Error during conversion: {str(e)}", level="error")
+        raise  # BaseExporter handles errors automatically
+```
+
+### 2. Progress Tracking
+
+```python
+def convert_data(self, data):
+    annotations = data.get('annotations', [])
+    total = len(annotations)
+
+    for i, annotation in enumerate(annotations):
+        # Update progress (value between 0-100)
+        progress = int((i / total) * 100)
+        self.run.set_progress(progress)
+
+        # Conversion logic...
+
+    return converted_data
+```
+
+### 3. Metrics Collection
+
+```python
+def after_convert(self, converted_data):
+    # Collect useful metrics
+    metrics = {
+        'total_exported': len(converted_data.get('annotations', [])),
+        'processing_time': time.time() - self.start_time,
+        'success_rate': self.calculate_success_rate(),
+    }
+
+    self.run.log_metrics(metrics)
+    return converted_data
+```
+
+### 4. Logging Usage
+
+```python
+def convert_data(self, data):
+    self.run.log_message("Starting data conversion", level="info")
+
+    if not data.get('annotations'):
+        self.run.log_message("No annotation data found", level="warning")
+        return data
+
+    # Conversion logic...
+
+    self.run.log_message(f"Conversion complete: processed {len(result)} items", level="success")
+    return result
+```
+
+### 5. Parameter Handling
+
+```python
+def __init__(self, run, export_items, path_root, **params):
+    super().__init__(run, export_items, path_root, **params)
+
+    # Handle custom parameters
+    self.output_format = params.get('output_format', 'json')
+    self.include_metadata = params.get('include_metadata', True)
+    self.compression = params.get('compression', False)
+```
+
+## Best Practices
+
+### Data Processing
+
+- **Memory Efficiency**: Use generators for processing large datasets
+- **Error Recovery**: Implement graceful error handling for individual items
+- **Progress Reporting**: Update progress regularly for long-running exports
+- **Data Validation**: Validate data structure before conversion
+
+```python
+class Exporter(BaseExporter):
+    def export(self, export_items=None, results=None, **kwargs):
+        """Override the main export method for custom processing."""
+        # Use tee to count items without consuming generator
+        items_to_process = export_items if export_items is not None else self.export_items
+        export_items_count, export_items_process = tee(items_to_process)
+        total = sum(1 for _ in export_items_count)
+
+        # Custom processing with error handling
+        for no, export_item in enumerate(export_items_process, start=1):
+            try:
+                # Use the built-in data conversion pipeline
+                processed_item = self.process_data_conversion(export_item)
+                self.run.set_progress(no, total, category='dataset_conversion')
+            except Exception as e:
+                self.run.log_message(f"Error processing item {no}: {str(e)}", "ERROR")
+                continue
+
+        # Call parent's export method for standard processing
+        # or implement your own complete workflow
+        return super().export(export_items, results, **kwargs)
+```
+
+### File Management
+
+- **Unique Paths**: Prevent file collisions with timestamp or counter suffixes
+- **Directory Structure**: Organize output files logically
+- **Error Logging**: Track failed files for debugging
+- **Cleanup**: Remove temporary files on completion
+
+```python
+class Exporter(BaseExporter):
+    def setup_output_directories(self, unique_export_path, save_original_file_flag):
+        """Create unique export directory structure."""
+        # BaseExporter already handles unique path creation via _create_unique_export_path
+        # This method sets up the internal directory structure
+        output_paths = super().setup_output_directories(unique_export_path, save_original_file_flag)
+
+        # Add custom subdirectories as needed
+        custom_dir = unique_export_path / 'custom_output'
+        custom_dir.mkdir(parents=True, exist_ok=True)
+        output_paths['custom_output_path'] = custom_dir
+
+        return output_paths
+```
+
+### Format Conversion
+
+- **Flexible Templates**: Design templates that work with multiple data types
+- **Schema Validation**: Validate output against expected schemas
+- **Metadata Preservation**: Maintain important metadata during conversion
+- **Version Compatibility**: Handle different data schema versions
+
+## Frequently Asked Questions
+
+**Q: Can I implement plugins directly without using BaseExporter?**  
+A: Yes, but it's not recommended. BaseExporter provides essential features like progress tracking, error handling, and metrics collection automatically.
+
+**Q: Can I export to multiple file formats simultaneously?**  
+A: Yes, you can override the `process_file_saving()` method to save in multiple formats.
+
+**Q: How can I optimize memory usage when processing large datasets?**  
+A: Consider using streaming processing in `convert_data()` rather than loading all data at once.
+
+**Q: What should I do if progress is not displaying correctly?**  
+A: Make sure you're calling `self.run.set_progress()` at appropriate intervals and using integer values between 0-100.