synapse-sdk 2025.9.5__py3-none-any.whl → 2025.10.6__py3-none-any.whl

synapse_sdk/devtools/docs/docs/plugins/categories/upload-plugins/upload-plugin-overview.md

@@ -0,0 +1,585 @@
+---
+id: upload-plugin-overview
+title: Upload Plugin Overview
+sidebar_position: 1
+---
+
+# Upload Plugin Overview
+
+Upload plugins provide comprehensive file upload and data ingestion operations for processing files into the Synapse platform with metadata support, security validation, and organized data unit generation.
+
+## Quick Overview
+
+**Category:** Upload
+**Available Actions:** `upload`
+**Execution Method:** Job-based execution
+
+## Key Features
+
+- **Multi-Path Mode Support**: Upload files from different locations with individual path settings for each asset
+- **Excel Metadata Integration**: Automatic metadata annotation from Excel files
+- **Flexible File Organization**: Single-path or multi-path modes for different use cases
+- **Batch Processing**: Optimized batch processing for large-scale uploads
+- **Progress Tracking**: Real-time progress updates across workflow stages
+- **Security Validation**: Comprehensive file and Excel security checks
+
+## Use Cases
+
+- Bulk file uploads with metadata annotation
+- Excel-based metadata mapping and validation
+- Recursive directory processing
+- Type-based file organization
+- Batch data unit creation
+- Multi-source dataset uploads (sensors, cameras, annotations from different locations)
+- Secure file processing with size and content validation
+
+## Supported Upload Sources
+
+- Local file system paths (files and directories)
+- Recursive directory scanning
+- Excel metadata files for enhanced file annotation
+- Mixed file types with automatic organization
+- Distributed data sources with per-asset path configuration
+
+## Configuration Modes
+
+### Mode 1: Single Path Mode (Default - `use_single_path: true`)
+
+All assets share one base directory. The system expects subdirectories for each file specification.
+
+```json
+{
+  "name": "Standard Upload",
+  "use_single_path": true,
+  "path": "/data/experiment",
+  "is_recursive": true,
+  "storage": 1,
+  "data_collection": 5
+}
+```
+
+**Expected Directory Structure:**
+
+```
+/data/experiment/
+├── pcd_1/           # Point clouds
+│   └── *.pcd
+├── image_1/         # Images
+│   └── *.jpg
+└── json_meta_1/     # Metadata
+    └── *.json
+```
+
+### Mode 2: Multi-Path Mode (`use_single_path: false`)
+
+Each asset has its own path and recursive setting. Perfect for distributed data sources.
+
+```json
+{
+  "name": "Multi-Source Upload",
+  "use_single_path": false,
+  "assets": {
+    "pcd_1": {
+      "path": "/sensors/lidar/scan_001",
+      "is_recursive": false
+    },
+    "image_1": {
+      "path": "/sensors/camera/front",
+      "is_recursive": true
+    },
+    "json_meta_1": {
+      "path": "/metadata/annotations",
+      "is_recursive": false
+    }
+  },
+  "storage": 1,
+  "data_collection": 5
+}
+```
+
+**Optional File Specs:**
+
+In multi-path mode, file specifications can be marked as optional in the data collection's file specification template:
+
+- **Required specs** (`is_required: true`): Must have an asset path in the `assets` parameter
+- **Optional specs** (`is_required: false`): Can be omitted from `assets` - the system will skip them
+
+Example with optional spec omitted:
+
+```json
+{
+  "name": "Multi-Source Upload",
+  "use_single_path": false,
+  "assets": {
+    "pcd_1": {"path": "/sensors/lidar", "is_recursive": false},
+    "image_1": {"path": "/cameras/front", "is_recursive": true}
+    // "json_meta_1" is optional and omitted
+  },
+  "storage": 1,
+  "data_collection": 5
+}
+```
+
+The system logs: `"Skipping optional spec json_meta_1: no asset path configured"`
+
+## Basic Usage
+
+### CLI Usage
+
+```bash
+# Single path mode (traditional)
+synapse plugin run upload '{
+  "name": "Dataset Upload",
+  "use_single_path": true,
+  "path": "/data/training",
+  "is_recursive": true,
+  "storage": 1,
+  "data_collection": 5
+}'
+
+# Multi-path mode (advanced)
+synapse plugin run upload '{
+  "name": "Multi-Sensor Upload",
+  "use_single_path": false,
+  "assets": {
+    "lidar": {"path": "/sensors/lidar", "is_recursive": true},
+    "camera": {"path": "/sensors/camera", "is_recursive": false}
+  },
+  "storage": 1,
+  "data_collection": 5
+}'
+```
+
+### Python API Usage
+
+```python
+from synapse_sdk.plugins.categories.upload.actions.upload.action import UploadAction
+
+# Configure upload parameters
+params = {
+    "name": "Dataset Upload",
+    "use_single_path": true,
+    "path": "/data/training_images",
+    "is_recursive": True,
+    "storage": 1,
+    "data_collection": 5,
+    "max_file_size_mb": 100
+}
+
+action = UploadAction(params=params, plugin_config=plugin_config)
+result = action.start()
+
+print(f"Uploaded {result['uploaded_files_count']} files")
+print(f"Generated {result['generated_data_units_count']} data units")
+```
+
+## Configuration Parameters
+
+### Required Parameters
+
+| Parameter         | Type  | Description        | Example       |
+| ----------------- | ----- | ------------------ | ------------- |
+| `name`            | `str` | Upload name        | `"My Upload"` |
+| `storage`         | `int` | Storage ID         | `1`           |
+| `data_collection` | `int` | Data collection ID | `5`           |
+
+### Mode-Specific Required Parameters
+
+**Single Path Mode** (`use_single_path: true`):
+
+- `path` (str): Base directory path
+
+**Multi-Path Mode** (`use_single_path: false`):
+
+- `assets` (dict): Asset-specific configurations with `path` and `is_recursive` per asset
+
+### Optional Parameters
+
+| Parameter                       | Type          | Default | Description                       |
+| ------------------------------- | ------------- | ------- | --------------------------------- |
+| `description`                   | `str \| None` | `None`  | Upload description                |
+| `project`                       | `int \| None` | `None`  | Project ID                        |
+| `use_single_path`               | `bool`        | `true`  | Mode toggle                       |
+| `is_recursive`                  | `bool`        | `true`  | Recursive scan (single path mode) |
+| `excel_metadata_path`           | `str \| None` | `None`  | Excel metadata file path (will be deprecated in future, use `excel_metadata` instead) |
+| `excel_metadata`                | `dict \| None`| `None`  | Excel metadata (base64 encoded, recommended) |
+| `max_file_size_mb`              | `int`         | `50`    | Maximum file size in MB           |
+| `creating_data_unit_batch_size` | `int`         | `100`   | Batch size for data units         |
+| `use_async_upload`              | `bool`        | `True`  | Use asynchronous processing       |
+
+## Excel Metadata Support
+
+The upload plugin provides advanced Excel metadata processing with flexible header support, comprehensive filename matching, and two distinct input methods for different use cases.
+
+### Input Methods
+
+There are two separate parameters for providing Excel metadata:
+
+#### 1. File Path Method (`excel_metadata_path`)
+
+:::info Future Deprecation Notice
+This parameter will be deprecated in a future version.
+We recommend using the `excel_metadata` parameter with base64 encoding for new implementations.
+:::
+
+**Use case:** Traditional file-based uploads where the Excel file exists on the server's file system.
+
+Simple string path to an Excel file:
+
+```json
+{
+  "excel_metadata_path": "/data/metadata.xlsx"
+}
+```
+
+**Advantages:**
+- Backward compatible with existing implementations
+- Simple and straightforward
+- Direct file system access
+
+**Note:** Consider migrating to base64 encoded method for future compatibility (see Method 2 below)
+
+#### 2. Base64 Encoded Method (`excel_metadata`) - **RECOMMENDED**
+
+**Use case:** Web frontends, APIs, and cloud integrations where files are transmitted as encoded data.
+
+Send Excel file as base64-encoded data with original filename:
+
+```json
+{
+  "excel_metadata": {
+    "data": "UEsDBBQABgAIAAAAIQDd4Z...",  // base64 encoded Excel file
+    "filename": "metadata.xlsx"
+  }
+}
+```
+
+**Advantages:**
+- No intermediate file storage required
+- Perfect for web upload forms
+- API-friendly JSON payload
+- Automatic temporary file cleanup
+- **This is the recommended method going forward**
+
+**Python Example:**
+```python
+import base64
+
+# Read Excel file and encode to base64
+with open("metadata.xlsx", "rb") as f:
+    excel_bytes = f.read()
+    encoded_excel = base64.b64encode(excel_bytes).decode("utf-8")
+
+# Use in upload parameters
+upload_params = {
+    "name": "Web Upload",
+    "path": "/data/files",
+    "storage": 1,
+    "data_collection": 5,
+    "excel_metadata": {
+        "data": encoded_excel,
+        "filename": "metadata.xlsx"
+    }
+}
+```
+
+:::warning Important
+You **cannot** use both `excel_metadata_path` and `excel_metadata` at the same time. Choose the method that best fits your use case:
+- Use `excel_metadata_path` for server-side files (will be deprecated in future versions)
+- Use `excel_metadata` for all use cases (recommended for new implementations)
+:::
+
+### Future-Proof Migration Example
+
+For new implementations, we recommend using `excel_metadata`. Here's how to migrate:
+
+```python
+import base64
+
+# Current approach (will be deprecated in future):
+upload_params = {
+    "name": "My Upload",
+    "path": "/data/files",
+    "storage": 1,
+    "data_collection": 5,
+    "excel_metadata_path": "/data/metadata.xlsx"  # Will be deprecated
+}
+
+# Recommended approach for new implementations:
+with open("/data/metadata.xlsx", "rb") as f:
+    excel_bytes = f.read()
+    encoded_excel = base64.b64encode(excel_bytes).decode("utf-8")
+
+upload_params = {
+    "name": "My Upload",
+    "path": "/data/files",
+    "storage": 1,
+    "data_collection": 5,
+    "excel_metadata": {
+        "data": encoded_excel,
+        "filename": "metadata.xlsx"
+    }
+}
+```
+
+### Excel Format
+
+Both header formats are supported (case-insensitive):
+
+**Option 1: "filename" header**
+| filename | category | description | custom_field |
+|----------|----------|-------------|--------------|
+| image1.jpg | nature | Mountain landscape | high_res |
+| image2.png | urban | City skyline | processed |
+
+**Option 2: "filename" header**
+| file_name | category | description | custom_field |
+|-----------|----------|-------------|--------------|
+| image1.jpg | nature | Mountain landscape | high_res |
+| image2.png | urban | City skyline | processed |
+
+### Filename Matching
+
+The system uses a 5-tier priority matching algorithm:
+
+1. **Exact stem match** (highest priority): `image1` matches `image1.jpg`
+2. **Exact filename match**: `image1.jpg` matches `image1.jpg`
+3. **Metadata key stem match**: `path/image1.ext` stem matches `image1`
+4. **Partial path matching**: `/uploads/image1.jpg` contains `image1`
+5. **Full path matching**: Complete path matching for complex structures
+
+### Security Validation
+
+Excel files undergo security validation:
+
+```python
+# Default security limits
+max_file_size_mb: 10      # File size limit
+max_rows: 100000          # Row count limit
+max_columns: 50           # Column count limit
+```
+
+### Configuration
+
+Configure Excel security in `config.yaml`:
+
+```yaml
+actions:
+  upload:
+    excel_config:
+      max_file_size_mb: 10
+      max_rows: 100000
+      max_columns: 50
+```
+
+## Progress Tracking
+
+The upload action tracks progress across three main phases:
+
+| Category              | Proportion | Description                         |
+| --------------------- | ---------- | ----------------------------------- |
+| `analyze_collection`  | 2%         | Parameter validation and setup      |
+| `upload_data_files`   | 38%        | File upload processing              |
+| `generate_data_units` | 60%        | Data unit creation and finalization |
+
+## Common Use Cases
+
+### 1. Simple Dataset Upload
+
+```json
+{
+  "name": "Training Dataset",
+  "use_single_path": true,
+  "path": "/datasets/training",
+  "is_recursive": true,
+  "storage": 1,
+  "data_collection": 2
+}
+```
+
+### 2. Multi-Source Sensor Data
+
+```json
+{
+  "name": "Multi-Camera Dataset",
+  "use_single_path": false,
+  "assets": {
+    "front_camera": { "path": "/cameras/front", "is_recursive": true },
+    "rear_camera": { "path": "/cameras/rear", "is_recursive": true },
+    "lidar": { "path": "/sensors/lidar", "is_recursive": false }
+  },
+  "storage": 1,
+  "data_collection": 2
+}
+```
+
+### 3. Dataset with Metadata (File Path)
+
+```json
+{
+  "name": "Annotated Dataset",
+  "use_single_path": true,
+  "path": "/data/annotated",
+  "is_recursive": true,
+  "excel_metadata_path": "/data/metadata.xlsx",
+  "storage": 1,
+  "data_collection": 5
+}
+```
+
+### 4. Dataset with Base64 Metadata (API/Web Use Case)
+
+```json
+{
+  "name": "Web Upload with Metadata",
+  "use_single_path": true,
+  "path": "/data/uploads",
+  "is_recursive": true,
+  "excel_metadata": {
+    "data": "UEsDBBQABgAIAAAAIQDd4Zg...",
+    "filename": "metadata.xlsx"
+  },
+  "storage": 1,
+  "data_collection": 5
+}
+```
+
+**Python Example - Convert Excel to Base64:**
+
+```python
+import base64
+
+# Read Excel file and convert to base64
+with open('metadata.xlsx', 'rb') as f:
+    excel_data = f.read()
+    encoded = base64.b64encode(excel_data).decode('utf-8')
+
+# Use in upload params
+params = {
+    "name": "Web Upload",
+    "path": "/data/uploads",
+    "excel_metadata": {
+        "data": encoded,
+        "filename": "metadata.xlsx"
+    },
+    "storage": 1,
+    "data_collection": 5
+}
+```
+
+## Benefits
+
+### For Users
+
+- **Flexibility**: Upload files from multiple different locations in a single operation
+- **Granular Control**: Set recursive search per asset, not globally
+- **Organization**: Map complex file structures to data collection specifications
+- **Use Case Support**: Multi-sensor data collection, distributed datasets, heterogeneous sources
+
+### For Developers
+
+- **Backward Compatible**: Existing code continues to work without changes
+- **Type Safe**: Full Pydantic validation with clear error messages
+- **Maintainable**: Clean separation between single-path and multi-path logic
+- **Extensible**: Easy to add more per-asset configuration options in the future
+
+## Next Steps
+
+- **For Plugin Developers**: See [BaseUploader Template Guide](./upload-plugin-template.md) for creating custom upload plugins with file processing logic
+- **For SDK/Action Developers**: See [Upload Action Development](./upload-plugin-action.md) for architecture details, strategy patterns, and action internals
+
+## Migration Guide
+
+### From Legacy to Current Version
+
+The upload action maintains 100% backward compatibility. The default behavior (`use_single_path=true`) works identically to the previous version.
+
+#### No Migration Required
+
+Existing configurations continue to work without changes:
+
+```python
+# This legacy usage still works
+params = {
+    "name": "My Upload",
+    "path": "/data/files",
+    "storage": 1,
+    "data_collection": 5
+}
+```
+
+#### Adopting Multi-Path Mode
+
+To use the new multi-path functionality:
+
+1. Set `use_single_path: false`
+2. Remove the `path` field (it will be ignored)
+3. Add `assets` dictionary with per-asset configurations
+
+```python
+# New multi-path mode
+params = {
+    "name": "Multi-Source Upload",
+    "use_single_path": false,
+    "assets": {
+        "pcd_1": {"path": "/sensors/lidar", "is_recursive": false},
+        "image_1": {"path": "/cameras/front", "is_recursive": true}
+    },
+    "storage": 1,
+    "data_collection": 5
+}
+```
+
+## Troubleshooting
+
+### Common Issues
+
+#### "No Files Found" Error
+
+```bash
+# Check path exists and is readable
+ls -la /path/to/data
+test -r /path/to/data && echo "Readable" || echo "Not readable"
+
+# Verify files exist
+find /path/to/data -name "*.jpg" | head -10
+```
+
+#### Excel Processing Errors
+
+```bash
+# Check file format and size
+file /path/to/metadata.xlsx
+ls -lh /path/to/metadata.xlsx
+```
+
+#### Mode Validation Errors
+
+- **Single path mode**: Ensure `path` is provided
+- **Multi-path mode**: Ensure `assets` is provided with at least one asset configuration
+
+## Best Practices
+
+### Directory Organization
+
+- Use clear, descriptive directory names
+- Keep reasonable directory sizes (< 10,000 files per directory)
+- Use absolute paths for reliability
+
+### Performance Optimization
+
+- Enable recursive only when needed
+- Keep Excel files under 5MB for best performance
+- Organize files in balanced directory structures
+
+### Security Considerations
+
+- Validate all paths before processing
+- Use read-only permissions for source data
+- Set appropriate Excel size limits
+
+## Support and Resources
+
+- **Action Development Guide**: [Upload Plugin Action Development](./upload-plugin-action.md)
+- **Template Development Guide**: [Upload Plugin Template Development](./upload-plugin-template.md)
+- **API Reference**: See action development documentation for detailed API reference