pyarallel 0.1.1__tar.gz → 0.1.3__tar.gz

{pyarallel-0.1.1 → pyarallel-0.1.3}/.gitignore

@@ -42,3 +42,7 @@ htmlcov/
 # Misc
 .DS_Store
 *.log
+
+
+# docs 
+site

pyarallel-0.1.3/Makefile

@@ -0,0 +1,36 @@
+.PHONY: help test docs-serve docs-deploy format lint clean
+
+help:
+	@echo "Available commands:"
+	@echo "  make test         Run pytest suite"
+	@echo "  make docs-serve   Start mkdocs development server"
+	@echo "  make docs-deploy  Deploy documentation to GitHub Pages"
+	@echo "  make format      Format code with black and isort"
+	@echo "  make lint        Run mypy for type checking"
+	@echo "  make clean       Remove build artifacts"
+
+test:
+	pytest tests/ -v
+
+docs-serve:
+	mkdocs serve
+
+docs-deploy:
+	python -c "import pyarallel; from pathlib import Path; yml = Path('mkdocs.yml').read_text(); Path('mkdocs.yml').write_text(yml.replace('version: .*', f'version: {pyarallel.__version__}'))"
+	mkdocs gh-deploy
+
+format:
+	black .
+	isort .
+
+lint:
+	mypy pyarallel/
+
+clean:
+	rm -rf build/
+	rm -rf dist/
+	rm -rf *.egg-info/
+	find . -type d -name __pycache__ -exec rm -rf {} +
+	find . -type f -name '*.pyc' -delete
+	find . -type f -name '*.pyo' -delete
+	find . -type f -name '*.pyd' -delete

{pyarallel-0.1.1 → pyarallel-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyarallel
-Version: 0.1.1
+Version: 0.1.3
 Summary: A powerful parallel execution library for Python
 Project-URL: Homepage, https://github.com/oneryalcin/pyarallel
 Project-URL: Repository, https://github.com/oneryalcin/pyarallel.git
@@ -39,6 +39,9 @@ Description-Content-Type: text/markdown
 
 # Pyarallel
 
+[![Docs](https://img.shields.io/badge/docs-live-brightgreen)](https://oneryalcin.github.io/pyarallel/) [![PyPI version](https://img.shields.io/pypi/v/pyarallel)](https://pypi.org/project/pyarallel/) [![PyPI Downloads](https://static.pepy.tech/badge/pyarallel/month)](https://pepy.tech/project/pyarallel)
+
+
 A powerful,feature-rich parallel execution library for Python that makes concurrent programming easy and efficient.
 
 ## Features
@@ -56,6 +59,10 @@ A powerful,feature-rich parallel execution library for Python that makes concurr
   - Memory-efficient with automatic cleanup
   - Comprehensive error handling
 
+## Documentation
+
+Check out the [documentation](https://oneryalcin.github.io/pyarallel/) for detailed usage instructions and examples.
+
 ## Installation
 
 ```bash
@@ -91,6 +98,56 @@ def analyze_text(text: str) -> dict:
     return text_analysis(text)
 ```
 
+## Usage Examples
+
+### Basic Function
+```python
+from pyarallel import parallel
+
+@parallel
+def process_item(x):
+    return x * 2
+
+results = process_item([1, 2, 3])  # [2, 4, 6]
+```
+
+### Instance Methods
+```python
+class DataProcessor:
+    def __init__(self, multiplier):
+        self.multiplier = multiplier
+    
+    @parallel
+    def process(self, x):
+        return x * self.multiplier
+
+processor = DataProcessor(3)
+results = processor.process([1, 2, 3])  # [3, 6, 9]
+```
+
+### Class Methods
+```python
+class StringFormatter:
+    @classmethod
+    @parallel
+    def format_all(cls, items):
+        return [f"Formatted-{item}" for item in items]
+
+results = StringFormatter.format_all(['a', 'b', 'c'])
+# ['Formatted-a', 'Formatted-b', 'Formatted-c']
+```
+
+### Static Methods
+```python
+class MathUtils:
+    @staticmethod
+    @parallel
+    def square_all(numbers):
+        return [n**2 for n in numbers]
+
+results = MathUtils.square_all([1, 2, 3])  # [1, 4, 9]
+```
+
 ## Advanced Usage
 
 ### Rate Limiting
@@ -180,6 +237,157 @@ results = process_large_dataset(items)
            return {"error": str(e), "item": item}
    ```
 
+## Configuration
+
+Pyarallel features a robust configuration system built on Pydantic, offering type validation, environment variable support, and thread-safe configuration management.
+
+### Basic Configuration
+
+```python
+from pyarallel import ConfigManager
+
+# Get the thread-safe singleton configuration manager
+config = ConfigManager.get_instance()
+
+# Update configuration with type validation
+config.update_config({
+    "execution": {
+        "default_max_workers": 8,
+        "default_executor_type": "thread",
+        "default_batch_size": 100,
+        "prewarm_pools": True
+    },
+    "rate_limiting": {
+        "default_rate": 1000,
+        "default_interval": "minute",
+        "burst_tolerance": 1.5
+    }
+})
+
+# Access configuration using dot notation
+workers = config.execution.default_max_workers
+rate = config.rate_limiting.default_rate
+
+# Category-specific updates
+config.update_execution(max_workers=16)
+config.update_rate_limiting(rate=2000)
+```
+
+### Environment Variables
+
+Configure Pyarallel using environment variables with the `PYARALLEL_` prefix. The system automatically handles type coercion and validation:
+
+```bash
+# Execution settings
+export PYARALLEL_MAX_WORKERS=4
+export PYARALLEL_EXECUTOR_TYPE=thread
+export PYARALLEL_BATCH_SIZE=100
+
+# Rate limiting
+export PYARALLEL_RATE_LIMIT=100/minute
+export PYARALLEL_FAIL_FAST=true
+
+# Complex values (using JSON)
+export PYARALLEL_RETRY_CONFIG='{"max_attempts": 3, "backoff": 1.5}'
+```
+
+### Configuration Schema
+
+The configuration system uses a structured schema with the following categories:
+
+```python
+{
+    "execution": {
+        "default_max_workers": int,        # Default worker count
+        "default_executor_type": str,      # "thread" or "process"
+        "default_batch_size": Optional[int], # Default batch size
+        "prewarm_pools": bool             # Enable worker prewarming
+    },
+    "rate_limiting": {
+        "default_rate": Optional[float],   # Default operations per interval
+        "default_interval": str,          # "second", "minute", "hour"
+        "burst_tolerance": float          # Burst allowance factor
+    },
+    "error_handling": {
+        "max_retries": int,               # Maximum retry attempts
+        "retry_backoff": float,           # Backoff multiplier
+        "fail_fast": bool                 # Stop on first error
+    },
+    "monitoring": {
+        "enable_logging": bool,           # Enable detailed logging
+        "log_level": str,                # Logging level
+        "sentry_dsn": Optional[str],     # Sentry integration
+        "metrics_enabled": bool          # Enable metrics collection
+    }
+}
+```
+
+### Best Practices
+
+1. **Use Environment Variables for Deployment**:
+   - Keep configuration in environment variables for different environments
+   - Use the `PYARALLEL_` prefix to avoid conflicts
+   - Complex values can be passed as JSON strings
+
+2. **Validate Configuration Early**:
+   - Set up configuration at application startup
+   - Use type validation to catch issues early
+   - Test configuration with sample data
+
+3. **Thread-Safe Updates**:
+   - Always use `ConfigManager.get_instance()` for thread-safe access
+   - Make configuration changes before starting parallel operations
+   - Use category-specific update methods for better type safety
+
+4. **Configuration Inheritance**:
+   - Global settings serve as defaults
+   - Decorator arguments override global configuration
+   - Environment variables take precedence over code-based configuration
+
+### Runtime Configuration Warnings
+
+Pyarallel includes built-in warnings to help identify potential performance issues:
+
+```python
+# Warning for high worker count
+@parallel(max_workers=150)  # Triggers warning about system impact
+def high_worker_task(): ...
+
+# Warning for inefficient process pool configuration
+@parallel(
+    executor_type="process",
+    batch_size=1  # Triggers warning about inefficient batch size
+)
+def inefficient_task(): ...
+```
+
+### Configuration Inheritance
+
+Pyarallel uses a hierarchical configuration system:
+
+1. **Default Values**: Built-in defaults (4 workers, thread executor, batch size 10)
+2. **Global Configuration**: Set via ConfigManager
+3. **Environment Variables**: Override global config
+4. **Decorator Arguments**: Highest precedence, override all other settings
+
+```python
+# Global configuration (lowest precedence)
+config = ConfigManager.get_instance()
+config.update_config({
+    "execution": {
+        "default_max_workers": 8,
+        "default_executor_type": "thread"
+    }
+})
+
+# Environment variables (middle precedence)
+# export PYARALLEL_MAX_WORKERS=16
+
+# Decorator arguments (highest precedence)
+@parallel(max_workers=4)  # This value wins
+def my_func(): ...
+```
+
 ## Roadmap
 
 ### Observability & Debugging
@@ -235,7 +443,6 @@ results = process_large_dataset(items)
   - Log analysis utilities
   - Telemetry visualization
 
-
 ### Enterprise Features
 - **Integration**
   - Distributed tracing (OpenTelemetry)
@@ -280,61 +487,4 @@ Contributions are welcome! Please check out our [Contributing Guide](CONTRIBUTIN
 
 ## License
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-## Configuration
-
-Pyarallel provides a flexible configuration system that allows you to customize its behavior globally or per-function:
-
-### Basic Configuration
-
-```python
-from pyarallel import ConfigManager
-
-# Get the global configuration manager
-config = ConfigManager.get_instance()
-
-# Update configuration
-config.update_config({
-    "max_workers": 8,
-    "timeout": 60.0,
-    "debug": True
-})
-```
-
-### Configuration Options
-
-- **Execution Settings**
-  - `max_workers`: Maximum number of worker processes/threads (default: 4)
-  - `timeout`: Default timeout for parallel operations in seconds (default: 30.0)
-
-- **Resource Management**
-  - `memory_limit`: Memory limit per worker in bytes (default: None)
-  - `cpu_affinity`: Enable CPU affinity for workers (default: False)
-
-- **Logging and Debugging**
-  - `debug`: Enable debug mode (default: False)
-  - `log_level`: Logging level (default: "INFO")
-
-### Environment Variables
-
-You can configure Pyarallel using environment variables with the `PYARALLEL_` prefix:
-
-```bash
-PYARALLEL_MAX_WORKERS=4
-PYARALLEL_TIMEOUT=60.0
-PYARALLEL_DEBUG=true
-```
-
-### Configuration Files
-
-Load configuration from JSON, YAML, or TOML files:
-
-```python
-from pyarallel import PyarallelConfig
-
-# Load from file
-config = PyarallelConfig.from_file("pyarallel.yaml")
-
-# Convert to dictionary
-config_dict = config.to_dict()
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

{pyarallel-0.1.1 → pyarallel-0.1.3}/README.md

@@ -1,5 +1,8 @@
 # Pyarallel
 
+[![Docs](https://img.shields.io/badge/docs-live-brightgreen)](https://oneryalcin.github.io/pyarallel/) [![PyPI version](https://img.shields.io/pypi/v/pyarallel)](https://pypi.org/project/pyarallel/) [![PyPI Downloads](https://static.pepy.tech/badge/pyarallel/month)](https://pepy.tech/project/pyarallel)
+
+
 A powerful,feature-rich parallel execution library for Python that makes concurrent programming easy and efficient.
 
 ## Features
@@ -17,6 +20,10 @@ A powerful,feature-rich parallel execution library for Python that makes concurr
   - Memory-efficient with automatic cleanup
   - Comprehensive error handling
 
+## Documentation
+
+Check out the [documentation](https://oneryalcin.github.io/pyarallel/) for detailed usage instructions and examples.
+
 ## Installation
 
 ```bash
@@ -52,6 +59,56 @@ def analyze_text(text: str) -> dict:
     return text_analysis(text)
 ```
 
+## Usage Examples
+
+### Basic Function
+```python
+from pyarallel import parallel
+
+@parallel
+def process_item(x):
+    return x * 2
+
+results = process_item([1, 2, 3])  # [2, 4, 6]
+```
+
+### Instance Methods
+```python
+class DataProcessor:
+    def __init__(self, multiplier):
+        self.multiplier = multiplier
+    
+    @parallel
+    def process(self, x):
+        return x * self.multiplier
+
+processor = DataProcessor(3)
+results = processor.process([1, 2, 3])  # [3, 6, 9]
+```
+
+### Class Methods
+```python
+class StringFormatter:
+    @classmethod
+    @parallel
+    def format_all(cls, items):
+        return [f"Formatted-{item}" for item in items]
+
+results = StringFormatter.format_all(['a', 'b', 'c'])
+# ['Formatted-a', 'Formatted-b', 'Formatted-c']
+```
+
+### Static Methods
+```python
+class MathUtils:
+    @staticmethod
+    @parallel
+    def square_all(numbers):
+        return [n**2 for n in numbers]
+
+results = MathUtils.square_all([1, 2, 3])  # [1, 4, 9]
+```
+
 ## Advanced Usage
 
 ### Rate Limiting
@@ -141,6 +198,157 @@ results = process_large_dataset(items)
            return {"error": str(e), "item": item}
    ```
 
+## Configuration
+
+Pyarallel features a robust configuration system built on Pydantic, offering type validation, environment variable support, and thread-safe configuration management.
+
+### Basic Configuration
+
+```python
+from pyarallel import ConfigManager
+
+# Get the thread-safe singleton configuration manager
+config = ConfigManager.get_instance()
+
+# Update configuration with type validation
+config.update_config({
+    "execution": {
+        "default_max_workers": 8,
+        "default_executor_type": "thread",
+        "default_batch_size": 100,
+        "prewarm_pools": True
+    },
+    "rate_limiting": {
+        "default_rate": 1000,
+        "default_interval": "minute",
+        "burst_tolerance": 1.5
+    }
+})
+
+# Access configuration using dot notation
+workers = config.execution.default_max_workers
+rate = config.rate_limiting.default_rate
+
+# Category-specific updates
+config.update_execution(max_workers=16)
+config.update_rate_limiting(rate=2000)
+```
+
+### Environment Variables
+
+Configure Pyarallel using environment variables with the `PYARALLEL_` prefix. The system automatically handles type coercion and validation:
+
+```bash
+# Execution settings
+export PYARALLEL_MAX_WORKERS=4
+export PYARALLEL_EXECUTOR_TYPE=thread
+export PYARALLEL_BATCH_SIZE=100
+
+# Rate limiting
+export PYARALLEL_RATE_LIMIT=100/minute
+export PYARALLEL_FAIL_FAST=true
+
+# Complex values (using JSON)
+export PYARALLEL_RETRY_CONFIG='{"max_attempts": 3, "backoff": 1.5}'
+```
+
+### Configuration Schema
+
+The configuration system uses a structured schema with the following categories:
+
+```python
+{
+    "execution": {
+        "default_max_workers": int,        # Default worker count
+        "default_executor_type": str,      # "thread" or "process"
+        "default_batch_size": Optional[int], # Default batch size
+        "prewarm_pools": bool             # Enable worker prewarming
+    },
+    "rate_limiting": {
+        "default_rate": Optional[float],   # Default operations per interval
+        "default_interval": str,          # "second", "minute", "hour"
+        "burst_tolerance": float          # Burst allowance factor
+    },
+    "error_handling": {
+        "max_retries": int,               # Maximum retry attempts
+        "retry_backoff": float,           # Backoff multiplier
+        "fail_fast": bool                 # Stop on first error
+    },
+    "monitoring": {
+        "enable_logging": bool,           # Enable detailed logging
+        "log_level": str,                # Logging level
+        "sentry_dsn": Optional[str],     # Sentry integration
+        "metrics_enabled": bool          # Enable metrics collection
+    }
+}
+```
+
+### Best Practices
+
+1. **Use Environment Variables for Deployment**:
+   - Keep configuration in environment variables for different environments
+   - Use the `PYARALLEL_` prefix to avoid conflicts
+   - Complex values can be passed as JSON strings
+
+2. **Validate Configuration Early**:
+   - Set up configuration at application startup
+   - Use type validation to catch issues early
+   - Test configuration with sample data
+
+3. **Thread-Safe Updates**:
+   - Always use `ConfigManager.get_instance()` for thread-safe access
+   - Make configuration changes before starting parallel operations
+   - Use category-specific update methods for better type safety
+
+4. **Configuration Inheritance**:
+   - Global settings serve as defaults
+   - Decorator arguments override global configuration
+   - Environment variables take precedence over code-based configuration
+
+### Runtime Configuration Warnings
+
+Pyarallel includes built-in warnings to help identify potential performance issues:
+
+```python
+# Warning for high worker count
+@parallel(max_workers=150)  # Triggers warning about system impact
+def high_worker_task(): ...
+
+# Warning for inefficient process pool configuration
+@parallel(
+    executor_type="process",
+    batch_size=1  # Triggers warning about inefficient batch size
+)
+def inefficient_task(): ...
+```
+
+### Configuration Inheritance
+
+Pyarallel uses a hierarchical configuration system:
+
+1. **Default Values**: Built-in defaults (4 workers, thread executor, batch size 10)
+2. **Global Configuration**: Set via ConfigManager
+3. **Environment Variables**: Override global config
+4. **Decorator Arguments**: Highest precedence, override all other settings
+
+```python
+# Global configuration (lowest precedence)
+config = ConfigManager.get_instance()
+config.update_config({
+    "execution": {
+        "default_max_workers": 8,
+        "default_executor_type": "thread"
+    }
+})
+
+# Environment variables (middle precedence)
+# export PYARALLEL_MAX_WORKERS=16
+
+# Decorator arguments (highest precedence)
+@parallel(max_workers=4)  # This value wins
+def my_func(): ...
+```
+
 ## Roadmap
 
 ### Observability & Debugging
@@ -196,7 +404,6 @@ results = process_large_dataset(items)
   - Log analysis utilities
   - Telemetry visualization
 
-
 ### Enterprise Features
 - **Integration**
   - Distributed tracing (OpenTelemetry)
@@ -241,61 +448,4 @@ Contributions are welcome! Please check out our [Contributing Guide](CONTRIBUTIN
 
 ## License
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-## Configuration
-
-Pyarallel provides a flexible configuration system that allows you to customize its behavior globally or per-function:
-
-### Basic Configuration
-
-```python
-from pyarallel import ConfigManager
-
-# Get the global configuration manager
-config = ConfigManager.get_instance()
-
-# Update configuration
-config.update_config({
-    "max_workers": 8,
-    "timeout": 60.0,
-    "debug": True
-})
-```
-
-### Configuration Options
-
-- **Execution Settings**
-  - `max_workers`: Maximum number of worker processes/threads (default: 4)
-  - `timeout`: Default timeout for parallel operations in seconds (default: 30.0)
-
-- **Resource Management**
-  - `memory_limit`: Memory limit per worker in bytes (default: None)
-  - `cpu_affinity`: Enable CPU affinity for workers (default: False)
-
-- **Logging and Debugging**
-  - `debug`: Enable debug mode (default: False)
-  - `log_level`: Logging level (default: "INFO")
-
-### Environment Variables
-
-You can configure Pyarallel using environment variables with the `PYARALLEL_` prefix:
-
-```bash
-PYARALLEL_MAX_WORKERS=4
-PYARALLEL_TIMEOUT=60.0
-PYARALLEL_DEBUG=true
-```
-
-### Configuration Files
-
-Load configuration from JSON, YAML, or TOML files:
-
-```python
-from pyarallel import PyarallelConfig
-
-# Load from file
-config = PyarallelConfig.from_file("pyarallel.yaml")
-
-# Convert to dictionary
-config_dict = config.to_dict()
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.