synapse-sdk 1.0.0a13__py3-none-any.whl → 2025.11.7__py3-none-any.whl

synapse_sdk/devtools/docs/docs/features/utils/network.md

@@ -0,0 +1,378 @@
+---
+id: network
+title: Network Utilities
+sidebar_position: 4
+---
+
+# Network Utilities
+
+Comprehensive networking utilities for streaming, validation, and connection management.
+
+## Overview
+
+The `synapse_sdk.utils.network` module provides essential networking components used by Ray and other clients for secure, robust streaming operations and input validation.
+
+## StreamLimits
+
+Configuration class for streaming operation limits to prevent resource exhaustion.
+
+### Constructor
+
+```python
+from synapse_sdk.utils.network import StreamLimits
+
+limits = StreamLimits(
+    max_messages=10000,
+    max_lines=50000,
+    max_bytes=50 * 1024 * 1024,  # 50MB
+    max_message_size=10240,      # 10KB
+    queue_size=1000,
+    exception_queue_size=10
+)
+```
+
+### Parameters
+
+- `max_messages` (int): Maximum number of messages to process (default: 10,000)
+- `max_lines` (int): Maximum number of lines to stream (default: 50,000)
+- `max_bytes` (int): Maximum total bytes to process (default: 50MB)
+- `max_message_size` (int): Maximum size per message (default: 10KB)
+- `queue_size` (int): Internal queue size for message buffering (default: 1,000)
+- `exception_queue_size` (int): Queue size for exception handling (default: 10)
+
+### Usage
+
+```python
+# Custom limits for high-volume streaming
+custom_limits = StreamLimits(
+    max_messages=50000,
+    max_lines=100000,
+    max_bytes=100 * 1024 * 1024  # 100MB
+)
+
+# Use with stream managers
+websocket_manager = WebSocketStreamManager(thread_pool, custom_limits)
+```
+
+## WebSocketStreamManager
+
+Manages WebSocket connections for real-time log streaming with automatic error handling and resource cleanup.
+
+### Constructor
+
+```python
+from synapse_sdk.utils.network import WebSocketStreamManager
+from concurrent.futures import ThreadPoolExecutor
+
+thread_pool = ThreadPoolExecutor(max_workers=5)
+limits = StreamLimits()
+manager = WebSocketStreamManager(thread_pool, limits)
+```
+
+### Methods
+
+#### `stream_logs(ws_url, headers, timeout, context)`
+
+Stream logs from a WebSocket endpoint.
+
+```python
+# Stream logs with error handling
+try:
+    for log_line in manager.stream_logs(
+        ws_url="wss://ray-cluster:8265/logs/ws/",
+        headers={"Authorization": "Bearer token"},
+        timeout=30.0,
+        context="job job-12345"
+    ):
+        print(log_line.strip())
+except ClientError as e:
+    print(f"Streaming failed: {e}")
+```
+
+### Error Handling
+
+- **500**: WebSocket library not available
+- **503**: Connection failed
+- **408**: Connection timeout
+- **429**: Stream limits exceeded
+
+## HTTPStreamManager
+
+Manages HTTP streaming connections using chunked transfer encoding for reliable log streaming.
+
+### Constructor
+
+```python
+from synapse_sdk.utils.network import HTTPStreamManager
+import requests
+
+session = requests.Session()
+limits = StreamLimits()
+manager = HTTPStreamManager(session, limits)
+```
+
+### Methods
+
+#### `stream_logs(url, headers, timeout, context)`
+
+Stream logs from an HTTP endpoint with chunked transfer encoding.
+
+```python
+# Stream logs with custom timeout
+for log_line in manager.stream_logs(
+    url="http://ray-cluster:8265/logs/stream",
+    headers={"Authorization": "Bearer token"},
+    timeout=(5, 60),  # (connect, read) timeout
+    context="job job-12345"
+):
+    if 'ERROR' in log_line:
+        print(f"Error detected: {log_line}")
+```
+
+### Features
+
+- **Automatic Resource Cleanup**: HTTP responses properly closed
+- **Line Size Filtering**: Oversized lines (>10KB) automatically filtered
+- **Stream Limits**: Prevents memory exhaustion
+- **Error Recovery**: Robust error handling with proper cleanup
+
+### Error Handling
+
+- **503**: Connection refused or network error
+- **408**: Connection or read timeout
+- **404**: Endpoint not found
+- **429**: Stream limits exceeded
+- **500**: Unexpected streaming error
+
+## Validation Functions
+
+### `validate_resource_id(resource_id, resource_name='resource')`
+
+Validates resource identifiers to prevent injection attacks.
+
+```python
+from synapse_sdk.utils.network import validate_resource_id
+
+# Valid usage
+job_id = validate_resource_id('job-12345', 'job')
+node_id = validate_resource_id('node_abc_123', 'node')
+
+# Invalid usage raises ClientError
+try:
+    validate_resource_id('job/../malicious', 'job')
+except ClientError as e:
+    print(f"Invalid ID: {e}")  # Status 400
+```
+
+#### Validation Rules
+
+- Must not be empty
+- Only alphanumeric characters, hyphens, and underscores allowed
+- Maximum length: 100 characters
+- Pattern: `^[a-zA-Z0-9\-_]+$`
+
+#### Parameters
+
+- `resource_id` (Any): The ID to validate (converted to string)
+- `resource_name` (str): Name for error messages (default: 'resource')
+
+#### Returns
+
+- `str`: Validated resource ID
+
+#### Raises
+
+- `ClientError` (400): If validation fails
+
+### `validate_timeout(timeout, max_timeout=300)`
+
+Validates timeout values with bounds checking.
+
+```python
+from synapse_sdk.utils.network import validate_timeout
+
+# Valid timeouts
+timeout = validate_timeout(30)      # 30 seconds
+timeout = validate_timeout(10.5)    # 10.5 seconds
+
+# Invalid timeouts raise ClientError
+try:
+    validate_timeout(-1)     # Negative timeout
+    validate_timeout(500)    # Exceeds maximum
+except ClientError as e:
+    print(f"Invalid timeout: {e}")  # Status 400
+```
+
+#### Parameters
+
+- `timeout` (int|float): Timeout value in seconds
+- `max_timeout` (int): Maximum allowed timeout (default: 300)
+
+#### Returns
+
+- `float`: Validated timeout value
+
+#### Raises
+
+- `ClientError` (400): If timeout is invalid
+
+## URL Utilities
+
+### `http_to_websocket_url(http_url)`
+
+Converts HTTP URLs to WebSocket URLs.
+
+```python
+from synapse_sdk.utils.network import http_to_websocket_url
+
+# Convert URLs
+ws_url = http_to_websocket_url("http://ray-cluster:8265/logs/")
+# Result: "ws://ray-cluster:8265/logs/"
+
+wss_url = http_to_websocket_url("https://ray-cluster:8265/logs/")
+# Result: "wss://ray-cluster:8265/logs/"
+```
+
+#### Parameters
+
+- `http_url` (str): HTTP or HTTPS URL
+
+#### Returns
+
+- `str`: WebSocket URL (ws:// or wss://)
+
+## Error Utilities
+
+### `sanitize_error_message(message, context)`
+
+Sanitizes error messages to prevent information leakage while maintaining debugging context.
+
+```python
+from synapse_sdk.utils.network import sanitize_error_message
+
+# Sanitize errors for logging
+clean_message = sanitize_error_message(
+    "Connection failed: Invalid token abc123",
+    "job job-12345"
+)
+# Result: Sanitized error message with context
+```
+
+#### Parameters
+
+- `message` (str): Original error message
+- `context` (str): Context for debugging (e.g., "job job-12345")
+
+#### Returns
+
+- `str`: Sanitized error message
+
+## Complete Example
+
+```python
+import asyncio
+from concurrent.futures import ThreadPoolExecutor
+from synapse_sdk.utils.network import (
+    StreamLimits,
+    WebSocketStreamManager,
+    HTTPStreamManager,
+    validate_resource_id,
+    validate_timeout,
+    http_to_websocket_url
+)
+
+class StreamingExample:
+    def __init__(self):
+        # Setup components
+        self.thread_pool = ThreadPoolExecutor(max_workers=3)
+        self.limits = StreamLimits(max_lines=10000)
+        self.ws_manager = WebSocketStreamManager(self.thread_pool, self.limits)
+
+        # HTTP session for HTTP streaming
+        import requests
+        self.session = requests.Session()
+        self.http_manager = HTTPStreamManager(self.session, self.limits)
+
+    def stream_job_logs(self, job_id, protocol='websocket'):
+        # Validate inputs
+        validated_id = validate_resource_id(job_id, 'job')
+        timeout = validate_timeout(30)
+
+        if protocol == 'websocket':
+            return self._websocket_stream(validated_id, timeout)
+        else:
+            return self._http_stream(validated_id, timeout)
+
+    def _websocket_stream(self, job_id, timeout):
+        url = f"http://ray-cluster:8265/jobs/{job_id}/logs/"
+        ws_url = http_to_websocket_url(url)
+        headers = {"Authorization": "Bearer token"}
+        context = f"job {job_id}"
+
+        return self.ws_manager.stream_logs(ws_url, headers, timeout, context)
+
+    def _http_stream(self, job_id, timeout):
+        url = f"http://ray-cluster:8265/jobs/{job_id}/logs/stream"
+        headers = {"Authorization": "Bearer token"}
+        timeout_tuple = (5, timeout)
+        context = f"job {job_id}"
+
+        return self.http_manager.stream_logs(url, headers, timeout_tuple, context)
+
+# Usage
+example = StreamingExample()
+
+# Stream with WebSocket
+for log_line in example.stream_job_logs('job-12345', 'websocket'):
+    print(log_line.strip())
+
+# Stream with HTTP fallback
+for log_line in example.stream_job_logs('job-12345', 'http'):
+    print(log_line.strip())
+```
+
+## Best Practices
+
+### 1. Resource Management
+
+```python
+# Always use proper cleanup
+thread_pool = ThreadPoolExecutor(max_workers=5)
+try:
+    manager = WebSocketStreamManager(thread_pool, StreamLimits())
+    # Use manager...
+finally:
+    thread_pool.shutdown(wait=True)
+```
+
+### 2. Error Handling
+
+```python
+from synapse_sdk.clients.exceptions import ClientError
+
+try:
+    job_id = validate_resource_id(user_input, 'job')
+    timeout = validate_timeout(user_timeout)
+    # Proceed with validated inputs...
+except ClientError as e:
+    logger.error(f"Validation failed: {e}")
+    return error_response(e.status, str(e))
+```
+
+### 3. Stream Limits Configuration
+
+```python
+# Configure limits based on use case
+production_limits = StreamLimits(
+    max_messages=20000,      # Higher for production
+    max_lines=100000,        # More lines allowed
+    max_bytes=200 * 1024 * 1024,  # 200MB for large logs
+    queue_size=2000          # Larger buffer
+)
+```
+
+## See Also
+
+- [RayClient](../../api/clients/ray.md) - Primary consumer of network utilities
+- [File Utils](./file.md) - File operations and handling
+- [Storage](./storage.md) - Storage providers

synapse_sdk/devtools/docs/docs/features/utils/storage.md

@@ -0,0 +1,57 @@
+---
+id: storage
+title: Storage Providers
+sidebar_position: 2
+---
+
+# Storage Providers
+
+Storage abstraction layer supporting multiple cloud providers.
+
+## Supported Providers
+
+### Amazon S3
+Amazon Simple Storage Service integration.
+
+```python
+from synapse_sdk.utils.storage.providers.s3 import S3Provider
+
+provider = S3Provider(
+    bucket="my-bucket",
+    region="us-west-2"
+)
+```
+
+### Google Cloud Storage
+Google Cloud Storage integration.
+
+```python
+from synapse_sdk.utils.storage.providers.gcp import GCPProvider
+
+provider = GCPProvider(
+    bucket="my-bucket",
+    project="my-project"
+)
+```
+
+### SFTP
+Secure File Transfer Protocol support.
+
+```python
+from synapse_sdk.utils.storage.providers.sftp import SFTPProvider
+
+provider = SFTPProvider(
+    host="sftp.example.com",
+    username="user"
+)
+```
+
+## Usage
+
+```python
+from synapse_sdk.utils.storage import get_storage
+
+# Automatic provider detection from URL
+storage = get_storage("s3://my-bucket/file.csv")
+local_path = storage.download()
+```

synapse_sdk/devtools/docs/docs/features/utils/types.md

@@ -0,0 +1,51 @@
+---
+id: types
+title: Custom Types
+sidebar_position: 3
+---
+
+# Custom Types
+
+Custom types and Pydantic fields used throughout the SDK.
+
+## FileField
+
+Custom Pydantic field for handling file URLs with automatic download.
+
+```python
+from synapse_sdk.types import FileField
+from pydantic import BaseModel
+
+class MyParams(BaseModel):
+    input_file: FileField  # Automatically downloads files
+
+def process(params: MyParams):
+    file_path = params.input_file  # Local file path
+    # Process the file...
+```
+
+### Features
+
+- Automatic file download from URLs
+- Temporary file management
+- Support for various file formats
+- Validation of file existence
+
+## Usage Examples
+
+```python
+# In plugin parameters
+class ProcessParams(BaseModel):
+    data_file: FileField
+    config_file: FileField = None  # Optional file
+
+# The FileField automatically:
+# 1. Downloads the file from URL
+# 2. Validates file existence
+# 3. Returns local file path
+# 4. Cleans up temporary files
+```
+
+## Type Validation
+
+Custom validators for ensuring type safety across the SDK.

synapse_sdk/devtools/docs/docs/installation.md

@@ -0,0 +1,94 @@
+---
+id: installation
+title: Installation & Setup
+sidebar_position: 2
+---
+
+# Installation & Setup
+
+Get started with Synapse SDK in minutes.
+
+## Prerequisites
+
+Before installing Synapse SDK, ensure you have:
+
+- **Python 3.10 or higher** installed
+
+## Installation Methods
+
+### Install from PyPI
+
+The easiest way to install Synapse SDK is via pip:
+
+```bash
+pip install synapse-sdk
+```
+
+### Install with Optional Dependencies
+
+For additional features, install with extras:
+
+```bash
+# Install with all dependencies (distributed computing, optimization libraries)
+pip install synapse-sdk[all]
+
+# Install with dashboard dependencies (FastAPI, Uvicorn)
+pip install synapse-sdk[devtools]
+
+# Install both
+pip install "synapse-sdk[all,devtools]"
+```
+
+### Install from Source
+
+To get the latest development version:
+
+```bash
+git clone https://github.com/datamaker/synapse-sdk.git
+cd synapse-sdk
+pip install -e .
+
+# With optional dependencies
+pip install -e ".[all,devtools]"
+```
+
+## Verify Installation
+
+After installation, verify everything is working:
+
+```bash
+# Check version
+synapse --version
+
+# Run interactive CLI
+synapse
+
+# Run with devtools
+synapse --dev-tools
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **ImportError: No module named 'synapse_sdk'**
+   - Ensure you've activated your virtual environment
+   - Check Python path: `python -c "import sys; print(sys.path)"`
+
+2. **Connection timeout to backend**
+   - Verify your API token is correct
+   - Check network connectivity
+   - Ensure backend URL is accessible
+
+### Getting Help
+
+If you encounter issues:
+
+1. Check the [Troubleshooting Guide](./troubleshooting.md)
+2. Search [GitHub Issues](https://github.com/datamaker/synapse-sdk/issues)
+3. Join our [Discord Community](https://discord.gg/synapse-sdk)
+
+## Next Steps
+
+- Follow the [Quickstart Guide](./quickstart.md)
+- Learn about [Core Concepts](./concepts/index.md)

synapse_sdk/devtools/docs/docs/introduction.md

@@ -0,0 +1,47 @@
+---
+id: introduction
+title: Introduction
+sidebar_position: 1
+---
+
+# Synapse SDK
+
+Build ML and data processing plugins.
+
+## Overview
+
+Synapse SDK is..
+
+### Key Features
+
+- **🚀 Plugin Development and Testing**: Develop modular, reusable components organized by categories for Synapse
+- **⚡ Distributed Computing**: Built for scalable, distributed execution
+- **🔧 Multiple Execution Modes**: Support for Jobs, Tasks, and REST APIs
+- **📦 Isolated Runtime Environments**: Each plugin runs in its own environment with dependency management
+- **📊 Progress Tracking**: Built-in progress monitoring and metrics reporting
+
+### Plugin Categories
+
+The SDK organizes plugins into specific categories:
+
+1. **Neural Network** (`NEURAL_NET`): ML model training, inference, and deployment
+2. **Export** (`EXPORT`): Data export and transformation operations
+3. **Upload** (`UPLOAD`): File and data upload functionality
+4. **Smart Tools** (`SMART_TOOL`): Intelligent automation tools
+5. **Post-Annotation** (`POST_ANNOTATION`): Post-processing after data annotation
+6. **Pre-Annotation** (`PRE_ANNOTATION`): Pre-processing before data annotation
+7. **Data Validation** (`DATA_VALIDATION`): Data quality and validation checks
+
+## Getting Started
+
+To get started with the Synapse SDK:
+
+1. [Install the SDK](./installation.md)
+2. [Follow the Quickstart Guide](./quickstart.md)
+3. [Explore the API Reference](./api/index.md)
+4. [Check out Examples](./examples/index.md)
+
+## Next Steps
+
+- Learn about [Core Concepts](./concepts/index.md)
+- Configure [Backend Connection](./configuration.md)