synapse-sdk 1.0.0a35__py3-none-any.whl → 2025.11.7__py3-none-any.whl

synapse_sdk/devtools/docs/docs/api/clients/backend.md

@@ -0,0 +1,420 @@
+---
+id: backend
+title: BackendClient
+sidebar_position: 1
+---
+
+# BackendClient
+
+Main client for interacting with the Synapse backend API.
+
+## Overview
+
+The `BackendClient` provides comprehensive access to all backend operations including data management, plugin execution, annotations, and machine learning workflows. It aggregates functionality from multiple specialized mixins:
+
+- **AnnotationClientMixin**: Task and annotation management
+- **CoreClientMixin**: File upload and core operations
+- **DataCollectionClientMixin**: Data collection and file management
+- **HITLClientMixin**: Human-in-the-loop assignment operations
+- **IntegrationClientMixin**: Plugin and job management
+- **MLClientMixin**: Machine learning models and ground truth operations
+
+## Constructor
+
+```python
+BackendClient(
+    base_url: str,
+    api_token: str = None,
+    agent_token: str = None,
+    timeout: dict = None
+)
+```
+
+### Parameters
+
+- **base_url** (`str`): The base URL of the Synapse backend API
+- **api_token** (`str`, optional): API authentication token. Can also be set via `SYNAPSE_API_TOKEN` environment variable
+- **agent_token** (`str`, optional): Agent authentication token. Can also be set via `SYNAPSE_AGENT_TOKEN` environment variable
+- **timeout** (`dict`, optional): Custom timeout settings. Defaults to `{'connect': 5, 'read': 30}`
+
+### Example
+
+```python
+from synapse_sdk.clients.backend import BackendClient
+
+# Create client with explicit token
+client = BackendClient(
+    base_url="https://api.synapse.sh",
+    api_token="your-api-token"
+)
+
+# Or use environment variables
+import os
+os.environ['SYNAPSE_API_TOKEN'] = "your-api-token"
+client = BackendClient(base_url="https://api.synapse.sh")
+```
+
+## API Methods
+
+### Annotation Operations
+
+#### `get_project(pk)`
+
+Get project details by ID.
+
+```python
+project = client.get_project(123)
+```
+
+#### `get_task(pk, params)`
+
+Get task details with optional parameters.
+
+```python
+task = client.get_task(456, params={'expand': 'data_unit'})
+```
+
+#### `annotate_task_data(pk, data)`
+
+Submit annotation data for a task.
+
+```python
+result = client.annotate_task_data(456, {
+    'annotations': [
+        {'type': 'bbox', 'coordinates': [10, 10, 100, 100]}
+    ]
+})
+```
+
+#### `list_tasks(params=None, url_conversion=None, list_all=False)`
+
+List tasks with filtering and pagination.
+
+```python
+# Get tasks for a project
+tasks = client.list_tasks(params={'project': 123})
+
+# Get all tasks (handles pagination automatically)
+all_tasks = client.list_tasks(list_all=True)
+```
+
+#### `create_tasks(data)`
+
+Create new tasks.
+
+```python
+new_tasks = client.create_tasks([
+    {'project': 123, 'data_unit': 789},
+    {'project': 123, 'data_unit': 790}
+])
+```
+
+#### `set_tags_tasks(data, params=None)`
+
+Set tags for multiple tasks.
+
+```python
+client.set_tags_tasks({
+    'task_ids': [456, 457],
+    'tag_ids': [1, 2, 3]
+})
+```
+
+### Core Operations
+
+#### `create_chunked_upload(file_path)`
+
+Upload large files using chunked upload for optimal performance.
+
+```python
+from pathlib import Path
+
+result = client.create_chunked_upload(Path('/path/to/large_file.zip'))
+print(f"Upload completed: {result}")
+```
+
+**Features:**
+
+- Uses 50MB chunks for optimal performance
+- Automatic retry and resume capability
+- MD5 integrity verification
+- Progress tracking support
+
+### Data Collection Operations
+
+#### `list_data_collection()`
+
+List all available data collections.
+
+```python
+collections = client.list_data_collection()
+```
+
+#### `get_data_collection(data_collection_id)`
+
+Get detailed information about a specific data collection.
+
+```python
+collection = client.get_data_collection(123)
+file_specs = collection['file_specifications']
+```
+
+#### `create_data_file(file_path, use_chunked_upload=False)`
+
+Create and upload a data file to the backend.
+
+```python
+from pathlib import Path
+
+# Regular upload
+data_file = client.create_data_file(Path('/path/to/file.jpg'))
+
+# Chunked upload for large files
+large_file = client.create_data_file(
+    Path('/path/to/large_file.zip'),
+    use_chunked_upload=True
+)
+```
+
+#### `upload_data_file(organized_file, collection_id, use_chunked_upload=False)`
+
+Upload organized file data to a collection.
+
+```python
+result = client.upload_data_file(
+    organized_file={'files': {...}, 'meta': {...}},
+    collection_id=123,
+    use_chunked_upload=False
+)
+```
+
+#### `create_data_units(uploaded_files)`
+
+Create data units from uploaded files.
+
+```python
+data_units = client.create_data_units([
+    {'id': 1, 'file': {...}},
+    {'id': 2, 'file': {...}}
+])
+```
+
+### HITL (Human-in-the-Loop) Operations
+
+#### `get_assignment(pk)`
+
+Get assignment details by ID.
+
+```python
+assignment = client.get_assignment(789)
+```
+
+#### `list_assignments(params=None, url_conversion=None, list_all=False)`
+
+List assignments with filtering options.
+
+```python
+# Get assignments for a project
+assignments = client.list_assignments(params={'project': 123})
+
+# Get all assignments
+all_assignments = client.list_assignments(list_all=True)
+```
+
+#### `set_tags_assignments(data, params=None)`
+
+Set tags for multiple assignments.
+
+```python
+client.set_tags_assignments({
+    'assignment_ids': [789, 790],
+    'tag_ids': [1, 2]
+})
+```
+
+### Integration Operations
+
+#### `health_check_agent(token)`
+
+Check agent health status.
+
+```python
+status = client.health_check_agent('agent-token-123')
+```
+
+#### `get_plugin(pk)` / `create_plugin(data)` / `update_plugin(pk, data)`
+
+Manage plugins.
+
+```python
+# Get plugin
+plugin = client.get_plugin(123)
+
+# Create plugin
+new_plugin = client.create_plugin({
+    'name': 'My Plugin',
+    'description': 'Plugin description'
+})
+
+# Update plugin
+updated = client.update_plugin(123, {'description': 'Updated description'})
+```
+
+#### `run_plugin(pk, data)`
+
+Execute a plugin with provided data.
+
+```python
+result = client.run_plugin(123, {
+    'parameters': {'input': 'value'},
+    'context': {...}
+})
+```
+
+#### Plugin Release Management
+
+```python
+# Create plugin release
+release = client.create_plugin_release({
+    'plugin': 123,
+    'version': '1.0.0',
+    'file': open('/path/to/plugin.zip', 'rb')
+})
+
+# Get release details
+release_info = client.get_plugin_release(456)
+```
+
+#### Job Management
+
+```python
+# List jobs
+jobs = client.list_jobs(params={'status': 'running'})
+
+# Get job details
+job = client.get_job(789, params={'expand': 'logs'})
+
+# Update job status
+client.update_job(789, {'status': 'completed'})
+
+# Get job console logs
+logs = client.list_job_console_logs(789)
+```
+
+#### Storage Operations
+
+```python
+# List storages
+storages = client.list_storages()
+
+# Get storage details
+storage = client.get_storage(123)
+
+# Create storage
+new_storage = client.create_storage({
+    'name': 'My Storage',
+    'provider': 'amazon_s3',
+    'configuration': {...}
+})
+```
+
+### Machine Learning Operations
+
+#### `list_models(params=None)` / `get_model(pk, params=None, url_conversion=None)`
+
+Manage ML models.
+
+```python
+# List models
+models = client.list_models(params={'project': 123})
+
+# Get model details
+model = client.get_model(456, params={'expand': 'metrics'})
+```
+
+#### `create_model(data)`
+
+Create a new ML model with file upload.
+
+```python
+new_model = client.create_model({
+    'name': 'My Model',
+    'project': 123,
+    'file': '/path/to/model.pkl'
+})
+```
+
+#### Ground Truth Operations
+
+```python
+# List ground truth events
+events = client.list_ground_truth_events(
+    params={'ground_truth_dataset_versions': [123]},
+    list_all=True
+)
+
+# Get ground truth version
+version = client.get_ground_truth_version(123)
+```
+
+## Storage Models
+
+The backend client includes predefined models for storage operations:
+
+### StorageCategory
+
+- `INTERNAL`: Internal storage systems
+- `EXTERNAL`: External storage providers
+
+### StorageProvider
+
+- `AMAZON_S3`: Amazon S3
+- `AZURE`: Microsoft Azure Blob Storage
+- `DIGITAL_OCEAN`: DigitalOcean Spaces
+- `FILE_SYSTEM`: Local file system
+- `FTP` / `SFTP`: FTP protocols
+- `MINIO`: MinIO storage
+- `GCP`: Google Cloud Storage
+
+## Error Handling
+
+All API methods may raise `ClientError` exceptions for various error conditions:
+
+```python
+from synapse_sdk.clients.exceptions import ClientError
+
+try:
+    project = client.get_project(999)
+except ClientError as e:
+    print(f"API Error: {e}")
+    print(f"Status Code: {e.status_code}")
+    print(f"Response: {e.response}")
+```
+
+## Pagination
+
+Methods supporting `list_all=True` will automatically handle pagination:
+
+```python
+# Manual pagination
+tasks_page1 = client.list_tasks(params={'page': 1, 'page_size': 100})
+
+# Automatic pagination (recommended)
+all_tasks = client.list_tasks(list_all=True)
+```
+
+## URL Conversion
+
+Some methods support URL conversion for file fields:
+
+```python
+# Custom URL conversion
+tasks = client.list_tasks(
+    url_conversion={'files': lambda url: f"https://cdn.example.com{url}"}
+)
+```
+
+## See Also
+
+- [AgentClient](./agent.md) - For agent-specific operations
+- [BaseClient](./base.md) - Base client implementation

synapse_sdk/devtools/docs/docs/api/clients/base.md

@@ -0,0 +1,257 @@
+---
+id: base
+title: BaseClient
+sidebar_position: 3
+---
+
+# BaseClient
+
+Base class for all Synapse SDK clients providing core HTTP operations and pagination.
+
+## Overview
+
+The `BaseClient` provides common functionality for HTTP operations, error handling, request management, and pagination used by all other clients. It implements efficient pagination handling with automatic file URL conversion capabilities.
+
+## Features
+
+- HTTP request handling with retry logic
+- Automatic timeout management
+- Efficient pagination with generators
+- File URL to local path conversion
+- Pydantic model validation
+- Connection pooling
+
+## Core HTTP Methods
+
+The BaseClient provides low-level HTTP methods that are used internally by all client mixins:
+
+- `_get()` - GET requests with optional response model validation
+- `_post()` - POST requests with request/response validation
+- `_put()` - PUT requests with model validation
+- `_patch()` - PATCH requests with model validation
+- `_delete()` - DELETE requests with model validation
+
+These methods are typically not called directly. Instead, use the higher-level methods provided by client mixins.
+
+## Pagination Methods
+
+### `_list(path, url_conversion=None, list_all=False, params=None, **kwargs)`
+
+List resources from a paginated API endpoint with optional automatic pagination and file URL conversion.
+
+**Parameters:**
+
+- `path` (str): URL path to request
+- `url_conversion` (dict, optional): Configuration for converting file URLs to local paths
+  - Structure: `{'files_fields': ['field1', 'field2'], 'is_list': True}`
+  - Automatically downloads files and replaces URLs with local paths
+- `list_all` (bool): If True, returns all results across all pages using a generator
+- `params` (dict, optional): Query parameters (filters, sorting, etc.)
+- `**kwargs`: Additional request arguments
+
+**Returns:**
+
+- If `list_all=False`: Dict with `results`, `count`, `next`, `previous`
+- If `list_all=True`: Tuple of `(generator, total_count)`
+
+**Examples:**
+
+```python
+# Get first page only
+response = client._list('api/tasks/')
+tasks = response['results']  # First page of tasks
+total = response['count']     # Total number of tasks
+
+# Get all results using generator (memory efficient)
+generator, total_count = client._list('api/tasks/', list_all=True)
+all_tasks = list(generator)  # Fetches all pages automatically
+
+# With filters
+params = {'status': 'pending', 'priority': 'high'}
+response = client._list('api/tasks/', params=params)
+
+# With url_conversion for file fields
+url_conversion = {'files_fields': ['files'], 'is_list': True}
+generator, count = client._list(
+    'api/data_units/',
+    url_conversion=url_conversion,
+    list_all=True,
+    params={'status': 'active'}
+)
+# File URLs in 'files' field are automatically downloaded and converted to local paths
+for unit in generator:
+    print(unit['files'])  # Local file paths, not URLs
+```
+
+### `_list_all(path, url_conversion=None, params=None, **kwargs)`
+
+Generator that yields all results from a paginated API endpoint.
+
+This method is called internally by `_list()` when `list_all=True`. It handles pagination automatically by following `next` URLs and uses an iterative approach (while loop) instead of recursion to avoid stack overflow with deep pagination.
+
+**Key Improvements (SYN-5757):**
+
+1. **No duplicate page_size**: The `page_size` parameter is only added to the first request. Subsequent requests use the `next` URL directly, which already contains all necessary parameters.
+
+2. **Proper params handling**: User-specified query parameters are correctly passed to the first request and preserved through pagination via the `next` URL.
+
+3. **url_conversion on all pages**: URL conversion is applied to every page, not just the first one.
+
+4. **Iterative instead of recursive**: Uses a while loop instead of recursion for better memory efficiency and to prevent stack overflow on large datasets.
+
+**Parameters:**
+
+- `path` (str): Initial URL path
+- `url_conversion` (dict, optional): Applied to all pages
+- `params` (dict, optional): Query parameters for first request only
+- `**kwargs`: Additional request arguments
+
+**Yields:**
+
+Individual result items from all pages, fetched lazily.
+
+**Examples:**
+
+```python
+# Basic: iterate through all tasks
+for task in client._list_all('api/tasks/'):
+    process_task(task)
+
+# With filters
+params = {'status': 'pending'}
+for task in client._list_all('api/tasks/', params=params):
+    print(task['id'])
+
+# With url_conversion for nested file fields
+url_conversion = {'files_fields': ['data.files', 'metadata.attachments'], 'is_list': True}
+for item in client._list_all('api/items/', url_conversion=url_conversion):
+    print(item['data']['files'])  # Local paths
+
+# Collect all results (memory intensive for large datasets)
+all_results = list(client._list_all('api/tasks/'))
+```
+
+## URL Conversion for File Downloads
+
+The `url_conversion` parameter enables automatic downloading of files referenced by URLs in API responses. This is particularly useful when working with data units, tasks, or any resources that include file references.
+
+### URL Conversion Structure
+
+```python
+url_conversion = {
+    'files_fields': ['files', 'images', 'data.attachments'],  # Field paths
+    'is_list': True  # Whether processing a list of items
+}
+```
+
+- `files_fields`: List of field paths (supports dot notation for nested fields)
+- `is_list`: Set to `True` for paginated list responses
+
+### How It Works
+
+1. API returns responses with file URLs
+2. `url_conversion` identifies fields containing URLs
+3. Files are downloaded automatically to a temporary directory
+4. URLs are replaced with local file paths
+5. Your code receives responses with local paths instead of URLs
+
+### Examples
+
+```python
+# Simple file field
+url_conversion = {'files_fields': ['image_url'], 'is_list': True}
+generator, count = client._list(
+    'api/photos/',
+    url_conversion=url_conversion,
+    list_all=True
+)
+for photo in generator:
+    # photo['image_url'] is now a local Path object, not a URL
+    with open(photo['image_url'], 'rb') as f:
+        process_image(f)
+
+# Multiple file fields
+url_conversion = {
+    'files_fields': ['thumbnail', 'full_image', 'raw_data'],
+    'is_list': True
+}
+
+# Nested fields using dot notation
+url_conversion = {
+    'files_fields': ['data.files', 'metadata.preview', 'annotations.image'],
+    'is_list': True
+}
+
+# With async download for better performance
+from synapse_sdk.utils.file import files_url_to_path_from_objs
+
+results = client._list('api/data_units/')['results']
+files_url_to_path_from_objs(
+    results,
+    files_fields=['files'],
+    is_list=True,
+    is_async=True  # Download all files concurrently
+)
+```
+
+## Performance Considerations
+
+### Memory Efficiency
+
+When working with large datasets, use generators instead of loading all results into memory:
+
+```python
+# ❌ Memory intensive - loads all results
+all_tasks = list(client._list('api/tasks/', list_all=True)[0])
+
+# ✅ Memory efficient - processes one at a time
+generator, _ = client._list('api/tasks/', list_all=True)
+for task in generator:
+    process_task(task)
+    # Task is processed and can be garbage collected
+```
+
+### Pagination Best Practices
+
+1. **Use list_all=True** for datasets larger than one page
+2. **Set appropriate page_size** in params if default (100) isn't optimal
+3. **Use url_conversion** only when you need to process files
+4. **Consider async downloads** for multiple files per item
+
+```python
+# Optimal pagination for large dataset
+params = {'page_size': 50}  # Smaller pages for faster first response
+generator, total = client._list(
+    'api/large_dataset/',
+    list_all=True,
+    params=params
+)
+
+# Process with progress tracking
+from tqdm import tqdm
+for item in tqdm(generator, total=total):
+    process_item(item)
+```
+
+## Usage in Client Mixins
+
+The BaseClient pagination methods are used internally by all client mixins:
+
+```python
+# DataCollectionClientMixin
+def list_data_units(self, params=None, url_conversion=None, list_all=False):
+    return self._list('data_units/', params=params,
+                     url_conversion=url_conversion, list_all=list_all)
+
+# AnnotationClientMixin
+def list_tasks(self, params=None, url_conversion=None, list_all=False):
+    return self._list('sdk/tasks/', params=params,
+                     url_conversion=url_conversion, list_all=list_all)
+```
+
+## See Also
+
+- [BackendClient](./backend.md) - Main client implementation
+- [AgentClient](./agent.md) - Agent-specific operations
+- [DataCollectionClientMixin](./data-collection-mixin.md) - Data and file operations
+- [AnnotationClientMixin](./annotation-mixin.md) - Task and annotation management