synapse-sdk 1.0.0b24__py3-none-any.whl → 2025.9.3__py3-none-any.whl

synapse_sdk/devtools/docs/docs/plugins/export-plugins.md

@@ -47,43 +47,43 @@ classDiagram
         +export_items: Generator
         +path_root: Path
         +params: dict
-        
+
         %% Main workflow methods
         +export(export_items, results, **kwargs) dict
         +process_data_conversion(export_item) object
         +process_file_saving(...) bool
         +setup_output_directories(path, flag) dict
-        
+
         %% Abstract methods (to be implemented)
         +convert_data(data)* object
-        +before_convert(data)* object  
+        +before_convert(data)* object
         +after_convert(data)* object
-        
+
         %% File operations
         +save_original_file(result, path, errors) ExportStatus
         +save_as_json(result, path, errors) ExportStatus
         +get_original_file_name(files) str
-        
+
         %% Helper methods
         -_create_unique_export_path(name) Path
         -_save_error_list(path, json_errors, file_errors)
         -_process_original_file_saving(...) bool
         -_process_json_file_saving(...) bool
     }
-    
+
     class Exporter {
         %% Inherited from BaseExporter
         +export(export_items, results, **kwargs) dict
-        
+
         %% Implemented abstract methods
         +convert_data(data) object
         +before_convert(data) object
         +after_convert(data) object
     }
-    
+
     %% Inheritance relationship
     BaseExporter <|-- Exporter
-    
+
     %% Apply styles
     class BaseExporter baseClass
     class Exporter childClass
@@ -99,44 +99,44 @@ flowchart TD
     A[export method called] --> B[Initialize paths and metrics]
     B --> C[Setup output directories]
     C --> D[Loop through export_items]
-    
+
     %% Data processing pipeline
     D --> E[process_data_conversion]
     E --> F[before_convert]
     F --> G[convert_data]
     G --> H[after_convert]
-    
+
     %% File saving pipeline
     H --> I[process_file_saving]
     I --> J{save_original_file?}
     J -->|Yes| K[_process_original_file_saving]
     J -->|No| L[_process_json_file_saving]
     K --> L
-    
+
     %% Continue or finish
     L --> M{More items?}
     M -->|Yes| D
     M -->|No| N[Save error lists]
     N --> O[Return export path]
-    
+
     %% Error handling
     K --> P{Original file failed?}
     P -->|Yes| Q[Skip to next item]
     P -->|No| L
-    
+
     L --> R{JSON file failed?}
     R -->|Yes| Q
     R -->|No| S[Update metrics]
     S --> M
     Q --> M
-    
+
     %% Styling for light/dark compatibility
     classDef startEnd fill:#e1f5fe,stroke:#01579b,color:#000
     classDef process fill:#f3e5f5,stroke:#4a148c,color:#000
     classDef decision fill:#fff3e0,stroke:#e65100,color:#000
     classDef data fill:#e8f5e8,stroke:#2e7d32,color:#000
     classDef error fill:#ffebee,stroke:#c62828,color:#000
-    
+
     class A,O startEnd
     class B,C,E,F,G,H,I,K,L,N,S process
     class J,M,P,R decision
@@ -147,20 +147,23 @@ flowchart TD
 ### Key Relationships and Responsibilities
 
 **BaseExporter (Abstract Base Class)**
+
 - **Core Functionality**: Provides complete export workflow infrastructure
 - **Template Methods**: `export()` method orchestrates the entire process
 - **Hook Methods**: `convert_data()`, `before_convert()`, `after_convert()` for customization
 - **Utilities**: File operations, directory setup, error handling, progress tracking
 
-**Exporter (Concrete Implementation)**  
+**Exporter (Concrete Implementation)**
+
 - **Inheritance**: Extends `BaseExporter`
 - **Minimal Implementation**: Provides basic implementations of abstract methods
 - **Pass-through Behavior**: Most methods delegate to parent class
 - **Customization Points**: Override conversion methods for specific logic
 
 ### Method Categories
+
 - **🔵 Core Workflow**: Main export orchestration methods
-- **🟢 Template/Hook**: Methods designed to be overridden by subclasses  
+- **🟢 Template/Hook**: Methods designed to be overridden by subclasses
 - **🟡 File Operations**: Concrete file saving and handling methods
 - **🔸 Helper/Utility**: Private methods for internal operations
 
@@ -240,12 +243,12 @@ The `additional_file_saving()` method is called after all export items have been
 ```python
 def additional_file_saving(self, unique_export_path):
     """Save additional files after processing all export items.
-    
+
     This method is called after the main export loop completes and is intended
     for saving files that need to be created based on the collective data from
     all processed export items (e.g., metadata files, configuration files,
     summary files, etc.).
-    
+
     Args:
         unique_export_path (str): The unique export directory path where
             additional files should be saved.
@@ -265,7 +268,7 @@ class YOLOExporter(BaseExporter):
             'total_annotations': 0,
             'class_distribution': {}
         }
-    
+
     def convert_data(self, data):
         # Track classes and stats during conversion
         for annotation in data.get('annotations', []):
@@ -273,24 +276,24 @@ class YOLOExporter(BaseExporter):
             self.class_names.add(class_name)
             self.dataset_stats['class_distribution'][class_name] = \
                 self.dataset_stats['class_distribution'].get(class_name, 0) + 1
-        
+
         self.dataset_stats['total_images'] += 1
         self.dataset_stats['total_annotations'] += len(data.get('annotations', []))
-        
+
         return data  # ... rest of conversion logic
-    
+
     def additional_file_saving(self, unique_export_path):
         """Save YOLO configuration and metadata files."""
         data_dir = Path(unique_export_path) / 'data'
         data_dir.mkdir(exist_ok=True)
-        
+
         # 1. Save classes.txt file
         classes_file = data_dir / 'classes.txt'
         with classes_file.open('w') as f:
             for class_name in sorted(self.class_names):
                 f.write(f"{class_name}\n")
         self.run.log_message(f"Saved classes file: {classes_file}")
-        
+
         # 2. Save dataset.yaml file
         dataset_config = {
             'path': str(unique_export_path),
@@ -298,12 +301,12 @@ class YOLOExporter(BaseExporter):
             'val': 'images',
             'names': {i: name for i, name in enumerate(sorted(self.class_names))}
         }
-        
+
         dataset_file = data_dir / 'dataset.yaml'
         with dataset_file.open('w') as f:
             yaml.dump(dataset_config, f, default_flow_style=False)
         self.run.log_message(f"Saved dataset config: {dataset_file}")
-        
+
         # 3. Save export statistics
         stats_file = data_dir / 'export_stats.json'
         with stats_file.open('w') as f:
@@ -314,6 +317,7 @@ class YOLOExporter(BaseExporter):
 ### Common Use Cases
 
 #### 1. Dataset Configuration Files
+
 ```python
 def additional_file_saving(self, unique_export_path):
     # Create dataset configuration for training frameworks
@@ -323,13 +327,14 @@ def additional_file_saving(self, unique_export_path):
         'total_samples': len(self.processed_items),
         'classes': list(self.class_mapping.keys())
     }
-    
+
     config_file = Path(unique_export_path) / 'dataset_config.json'
     with config_file.open('w') as f:
         json.dump(config, f, indent=2)
 ```
 
 #### 2. Export Summary Reports
+
 ```python
 def additional_file_saving(self, unique_export_path):
     # Generate export summary
@@ -342,13 +347,14 @@ def additional_file_saving(self, unique_export_path):
         'statistics': self.get_export_statistics(),
         'errors': self.get_error_summary()
     }
-    
+
     summary_file = Path(unique_export_path) / 'export_summary.json'
     with summary_file.open('w') as f:
         json.dump(summary, f, indent=2)
 ```
 
 #### 3. Index and Manifest Files
+
 ```python
 def additional_file_saving(self, unique_export_path):
     # Create file index for processed items
@@ -359,7 +365,7 @@ def additional_file_saving(self, unique_export_path):
             'json_file': f"{item['stem']}.json",
             'processed_at': item['timestamp']
         })
-    
+
     index_file = Path(unique_export_path) / 'file_index.json'
     with index_file.open('w') as f:
         json.dump(file_index, f, indent=2)
@@ -540,6 +546,127 @@ my_plugin/
 └── manifest.yaml       # Plugin metadata
 ```
 
+## Export Action Architecture
+
+The export system has been refactored into a modular architecture with specialized components for different aspects of data export processing:
+
+```mermaid
+classDiagram
+    %% Light/Dark mode compatible colors with semi-transparency
+    classDef baseClass fill:#e1f5fe80,stroke:#0288d1,stroke-width:2px
+    classDef childClass fill:#c8e6c980,stroke:#388e3c,stroke-width:2px
+    classDef modelClass fill:#fff9c480,stroke:#f57c00,stroke-width:2px
+    classDef utilClass fill:#f5f5f580,stroke:#616161,stroke-width:2px
+    classDef enumClass fill:#ffccbc80,stroke:#d32f2f,stroke-width:2px
+
+    class ExportAction {
+        +name: str = "export"
+        +category: PluginCategory.EXPORT
+        +method: RunMethod.JOB
+        +run_class: ExportRun
+        +params_model: ExportParams
+        +progress_categories: dict
+        +metrics_categories: dict
+
+        +start() dict
+        +get_exporter(...) object
+        +_get_export_items(target, filter) Generator
+        +_create_target_handler(target) object
+    }
+
+    class ExportRun {
+        +log_message_with_code(code, args, level) None
+        +log_export_event(code, args, level) None
+        +export_log_json_file(id, file_info, status) None
+        +export_log_original_file(id, file_info, status) None
+        +ExportEventLog: BaseModel
+        +DataFileLog: BaseModel
+        +MetricsRecord: BaseModel
+    }
+
+    class ExportParams {
+        +name: str
+        +storage: int
+        +target: Literal["assignment", "ground_truth", "task"]
+        +filter: dict
+        +path: str
+        +save_original_file: bool = True
+        +extra_params: dict = {}
+
+        +check_storage_exists(value) str
+        +validate_target_filter(cls, values) dict
+    }
+
+    class LogCode {
+        +EXPORT_STARTED: str
+        +ITEMS_DISCOVERED: str
+        +CONVERSION_STARTED: str
+        +CONVERSION_COMPLETED: str
+        +FILE_SAVED: str
+        +EXPORT_COMPLETED: str
+        +EXPORT_FAILED: str
+    }
+
+    class ExportStatus {
+        +SUCCESS: str = "success"
+        +FAILED: str = "failed"
+        +SKIPPED: str = "skipped"
+    }
+
+    class ExportError {
+        +message: str
+        +code: str
+    }
+
+    class ExportValidationError {
+        +message: str
+        +field: str
+    }
+
+    class ExportTargetError {
+        +message: str
+        +target: str
+    }
+
+    %% Relationships
+    ExportAction --> ExportRun : uses
+    ExportAction --> ExportParams : validates with
+    ExportRun --> LogCode : logs with
+    ExportRun --> ExportStatus : tracks status
+    ExportAction --> ExportError : may raise
+    ExportAction --> ExportValidationError : may raise
+    ExportAction --> ExportTargetError : may raise
+
+    %% Apply styles
+    class ExportAction baseClass
+    class ExportRun childClass
+    class ExportParams modelClass
+    class LogCode,ExportStatus enumClass
+    class ExportError,ExportValidationError,ExportTargetError utilClass
+```
+
+### Modular Structure
+
+The export action follows a clean modular organization:
+
+```
+synapse_sdk/plugins/categories/export/actions/export/
+├── __init__.py        # Clean module interfaces
+├── action.py          # ExportAction class
+├── enums.py          # ExportStatus, LogCode + LOG_MESSAGES
+├── exceptions.py     # Export-specific exceptions
+├── models.py         # ExportParams model
+├── run.py           # ExportRun class
+└── utils.py         # Target handlers and utilities
+```
+
+**Key Benefits:**
+
+- **Enhanced Maintainability**: Smaller, focused files are easier to understand and modify
+- **Code Consistency**: Export action now follows the same pattern as upload action
+- **Better Organization**: Related functionality is grouped logically
+- **Improved Readability**: Clear separation of concerns across modules
+
 ## Creating Export Plugins
 
 Export plugins now use the BaseExporter class-based approach for better organization and reusability. Here's how to create a custom export plugin:
@@ -557,7 +684,7 @@ synapse plugin create
 The `ExportParams` model defines the required parameters:
 
 ```python
-from synapse_sdk.plugins.categories.export.actions.export import ExportParams
+from synapse_sdk.plugins.categories.export.actions.export.models import ExportParams
 from pydantic import BaseModel
 from typing import Literal
 
@@ -664,7 +791,7 @@ Customize file saving and organization by overriding BaseExporter methods:
 ```python
 import json
 from pathlib import Path
-from synapse_sdk.plugins.categories.export.enums import ExportStatus
+from synapse_sdk.plugins.categories.export.actions.export.enums import ExportStatus
 
 class Exporter(BaseExporter):
     """Custom export plugin with multiple format support."""
@@ -923,4 +1050,4 @@ A: Yes, you can override the `process_file_saving()` method to save in multiple
 A: Consider using streaming processing in `convert_data()` rather than loading all data at once.
 
 **Q: What should I do if progress is not displaying correctly?**  
-A: Make sure you're calling `self.run.set_progress()` at appropriate intervals and using integer values between 0-100.
+A: Make sure you're calling `self.run.set_progress()` at appropriate intervals and using integer values between 0-100.