pyarallel 0.1.0__tar.gz

pyarallel-0.1.0/.gitignore

@@ -0,0 +1,43 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual Environment
+venv/
+env/
+ENV/
+.venv
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Test
+.coverage
+htmlcov/
+.pytest_cache/
+.tox/
+
+# Misc
+.DS_Store
+*.log

pyarallel-0.1.0/.python-version

@@ -0,0 +1 @@
+3.12

pyarallel-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,56 @@
+# Contributing to Pyarallel
+
+We love your input! We want to make contributing to Pyarallel as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+- Becoming a maintainer
+
+## Development Process
+
+We use GitHub to host code, to track issues and feature requests, as well as accept pull requests.
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. Issue that pull request!
+
+## Any Contributions You Make Will Be Under the MIT License
+In short, when you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern.
+
+## Report Bugs Using GitHub's [Issue Tracker](https://github.com/oneryalcin/pyarallel/issues)  
+
+We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/oneryalcin/pyarallel/issues/new); it's that easy!
+
+## Write Bug Reports With Detail, Background, and Sample Code
+
+**Great Bug Reports** tend to have:
+
+- A quick summary and/or background
+- Steps to reproduce
+  - Be specific!
+  - Give sample code if you can.
+- What you expected would happen
+- What actually happens
+- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
+
+## Use a Consistent Coding Style
+
+* Use [Black](https://github.com/psf/black) for Python code formatting
+* Keep line length to 88 characters (Black default)
+* Use type hints for function arguments and return values
+* Write docstrings for all public functions and classes
+
+## Running Tests
+
+```bash
+pytest tests/
+```
+
+## License
+
+By contributing, you agree that your contributions will be licensed under its MIT License.

pyarallel-0.1.0/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Pyarallel Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pyarallel-0.1.0/PKG-INFO

@@ -0,0 +1,283 @@
+Metadata-Version: 2.4
+Name: pyarallel
+Version: 0.1.0
+Summary: A powerful parallel execution library for Python
+Project-URL: Homepage, https://github.com/oneryalcin/pyarallel
+Project-URL: Repository, https://github.com/oneryalcin/pyarallel.git
+Project-URL: Documentation, https://github.com/oneryalcin/pyarallel
+Author-email: Mehmet Oner Yalcin <oneryalcin@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 Pyarallel Contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE.md
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+
+# Pyarallel
+
+A powerful,feature-rich parallel execution library for Python that makes concurrent programming easy and efficient.
+
+## Features
+
+- **Simple Decorator-Based API**: Just add `@parallel` to your functions
+- **Flexible Parallelism**: Choose between threads (I/O-bound) and processes (CPU-bound)
+- **Smart Rate Limiting**: Control execution rates with per-second, per-minute, or per-hour limits
+- **Batch Processing**: Handle large datasets efficiently with automatic batching
+- **Performance Optimized**: 
+  - Automatic worker pool reuse
+  - Optional worker prewarming for latency-critical applications
+  - Smart defaults based on your system
+- **Production Ready**:
+  - Thread-safe implementation
+  - Memory-efficient with automatic cleanup
+  - Comprehensive error handling
+
+## Installation
+
+```bash
+pip install pyarallel
+```
+
+## Quick Start
+
+```python
+from pyarallel import parallel
+
+# Basic parallel processing
+@parallel(max_workers=4)
+def fetch_url(url: str) -> dict:
+    return requests.get(url).json()
+
+# Process multiple URLs in parallel
+urls = ["http://api1.com", "http://api2.com"]
+results = fetch_url(urls)
+
+# Rate-limited CPU-intensive task
+@parallel(
+    max_workers=4,
+    executor_type="process",
+    rate_limit=(100, "minute")  # 100 ops/minute
+)
+def process_image(image: bytes) -> bytes:
+    return heavy_processing(image)
+
+# Memory-efficient batch processing
+@parallel(max_workers=4, batch_size=10)
+def analyze_text(text: str) -> dict:
+    return text_analysis(text)
+```
+
+## Advanced Usage
+
+### Rate Limiting
+
+Control execution rates using various formats:
+
+```python
+# Operations per second
+@parallel(rate_limit=2.0)
+def func1(): ...
+
+# Operations per minute
+@parallel(rate_limit=(100, "minute"))
+def func2(): ...
+
+# Custom rate limit object
+from pyarallel import RateLimit
+rate = RateLimit(1000, "hour")
+@parallel(rate_limit=rate)
+def func3(): ...
+```
+
+### CPU-Bound Tasks
+
+Use process-based parallelism for CPU-intensive operations:
+
+```python
+@parallel(
+    max_workers=4,
+    executor_type="process",  # Use processes instead of threads
+    batch_size=10            # Process in batches of 10
+)
+def cpu_intensive(data: bytes) -> bytes:
+    return heavy_computation(data)
+```
+
+### Latency-Critical Applications
+
+Prewarm workers to minimize cold start latency:
+
+```python
+@parallel(
+    max_workers=4,
+    prewarm=True  # Start workers immediately
+)
+def latency_critical(item): ...
+```
+
+### Memory-Efficient Processing
+
+Handle large datasets with batch processing:
+
+```python
+@parallel(
+    max_workers=4,
+    batch_size=100  # Process items in batches of 100
+)
+def process_large_dataset(item): ...
+
+# Process millions of items without memory issues
+items = range(1_000_000)
+results = process_large_dataset(items)
+```
+
+## Best Practices
+
+1. **Choose the Right Executor**:
+   - Use `executor_type="thread"` (default) for I/O-bound tasks (network, disk)
+   - Use `executor_type="process"` for CPU-bound tasks (computation)
+
+2. **Optimize Worker Count**:
+   - For I/O-bound: `max_workers = cpu_count * 5` (default)
+   - For CPU-bound: `max_workers = cpu_count` (default)
+
+3. **Control Resource Usage**:
+   - Use `batch_size` for large datasets
+   - Use `rate_limit` to prevent overwhelming resources
+   - Only use `prewarm=True` when cold start latency is critical
+
+4. **Handle Errors Properly**:
+   ```python
+   @parallel()
+   def my_func(item):
+       try:
+           return process(item)
+       except Exception as e:
+           return {"error": str(e), "item": item}
+   ```
+
+## Roadmap
+
+### Observability & Debugging
+- **Advanced Telemetry System**
+  - Task execution metrics (duration, wait times, queue times)
+  - Worker utilization tracking
+  - Error frequency analysis
+  - SQLite persistence for historical data
+  - Interactive visualizations with Plotly
+  - Performance bottleneck identification
+
+- **Rich Logging System**
+  - Configurable log levels per component
+  - Structured logging for machine parsing
+  - Contextual information for debugging
+  - Log rotation and management
+  - Integration with popular logging frameworks
+
+### Advanced Features
+- **Callback System**
+  - Pre/post execution hooks
+  - Error handling callbacks
+  - Progress tracking
+  - Custom metrics collection
+  - State management hooks
+
+- **Smart Scheduling**
+  - Priority queues for tasks
+  - Deadline-aware scheduling
+  - Resource-aware task distribution
+  - Adaptive batch sizing
+  - Dynamic worker scaling
+
+- **Fault Tolerance**
+  - Automatic retries with backoff
+  - Circuit breaker pattern
+  - Fallback strategies
+  - Dead letter queues
+  - Task timeout handling
+
+- **Resource Management**
+  - Memory usage monitoring
+  - CPU utilization tracking
+  - Network bandwidth control
+  - Disk I/O rate limiting
+  - Resource quotas per task
+
+### Developer Experience
+- **CLI Tools**
+  - Task monitoring dashboard
+  - Performance profiling
+  - Configuration management
+  - Log analysis utilities
+  - Telemetry visualization
+
+
+### Enterprise Features
+- **Integration**
+  - Distributed tracing (OpenTelemetry)
+  - Metrics export (Prometheus)
+  - Log aggregation (ELK Stack)
+
+Want to contribute? Check out our [CONTRIBUTING.md](CONTRIBUTING.md) guide!
+
+## API Reference
+
+### @parallel Decorator
+
+```python
+@parallel(
+    max_workers: int = None,          # Maximum workers (default: based on CPU)
+    batch_size: int = None,           # Items per batch (default: all at once)
+    rate_limit: Union[                # Rate limiting configuration
+        float,                        # - Operations per second
+        Tuple[float, str],           # - (count, interval)
+        RateLimit                     # - RateLimit object
+    ] = None,
+    executor_type: str = "thread",    # "thread" or "process"
+    prewarm: bool = False            # Prewarm workers
+)
+```
+
+### RateLimit Class
+
+```python
+class RateLimit:
+    def __init__(self, count: float, interval: str = "second"):
+        """
+        Args:
+            count: Operations allowed per interval
+            interval: "second", "minute", or "hour"
+        """
+```
+
+## Contributing
+
+Contributions are welcome! Please check out our [Contributing Guide](CONTRIBUTING.md).
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

pyarallel-0.1.0/README.md

@@ -0,0 +1,244 @@
+# Pyarallel
+
+A powerful,feature-rich parallel execution library for Python that makes concurrent programming easy and efficient.
+
+## Features
+
+- **Simple Decorator-Based API**: Just add `@parallel` to your functions
+- **Flexible Parallelism**: Choose between threads (I/O-bound) and processes (CPU-bound)
+- **Smart Rate Limiting**: Control execution rates with per-second, per-minute, or per-hour limits
+- **Batch Processing**: Handle large datasets efficiently with automatic batching
+- **Performance Optimized**: 
+  - Automatic worker pool reuse
+  - Optional worker prewarming for latency-critical applications
+  - Smart defaults based on your system
+- **Production Ready**:
+  - Thread-safe implementation
+  - Memory-efficient with automatic cleanup
+  - Comprehensive error handling
+
+## Installation
+
+```bash
+pip install pyarallel
+```
+
+## Quick Start
+
+```python
+from pyarallel import parallel
+
+# Basic parallel processing
+@parallel(max_workers=4)
+def fetch_url(url: str) -> dict:
+    return requests.get(url).json()
+
+# Process multiple URLs in parallel
+urls = ["http://api1.com", "http://api2.com"]
+results = fetch_url(urls)
+
+# Rate-limited CPU-intensive task
+@parallel(
+    max_workers=4,
+    executor_type="process",
+    rate_limit=(100, "minute")  # 100 ops/minute
+)
+def process_image(image: bytes) -> bytes:
+    return heavy_processing(image)
+
+# Memory-efficient batch processing
+@parallel(max_workers=4, batch_size=10)
+def analyze_text(text: str) -> dict:
+    return text_analysis(text)
+```
+
+## Advanced Usage
+
+### Rate Limiting
+
+Control execution rates using various formats:
+
+```python
+# Operations per second
+@parallel(rate_limit=2.0)
+def func1(): ...
+
+# Operations per minute
+@parallel(rate_limit=(100, "minute"))
+def func2(): ...
+
+# Custom rate limit object
+from pyarallel import RateLimit
+rate = RateLimit(1000, "hour")
+@parallel(rate_limit=rate)
+def func3(): ...
+```
+
+### CPU-Bound Tasks
+
+Use process-based parallelism for CPU-intensive operations:
+
+```python
+@parallel(
+    max_workers=4,
+    executor_type="process",  # Use processes instead of threads
+    batch_size=10            # Process in batches of 10
+)
+def cpu_intensive(data: bytes) -> bytes:
+    return heavy_computation(data)
+```
+
+### Latency-Critical Applications
+
+Prewarm workers to minimize cold start latency:
+
+```python
+@parallel(
+    max_workers=4,
+    prewarm=True  # Start workers immediately
+)
+def latency_critical(item): ...
+```
+
+### Memory-Efficient Processing
+
+Handle large datasets with batch processing:
+
+```python
+@parallel(
+    max_workers=4,
+    batch_size=100  # Process items in batches of 100
+)
+def process_large_dataset(item): ...
+
+# Process millions of items without memory issues
+items = range(1_000_000)
+results = process_large_dataset(items)
+```
+
+## Best Practices
+
+1. **Choose the Right Executor**:
+   - Use `executor_type="thread"` (default) for I/O-bound tasks (network, disk)
+   - Use `executor_type="process"` for CPU-bound tasks (computation)
+
+2. **Optimize Worker Count**:
+   - For I/O-bound: `max_workers = cpu_count * 5` (default)
+   - For CPU-bound: `max_workers = cpu_count` (default)
+
+3. **Control Resource Usage**:
+   - Use `batch_size` for large datasets
+   - Use `rate_limit` to prevent overwhelming resources
+   - Only use `prewarm=True` when cold start latency is critical
+
+4. **Handle Errors Properly**:
+   ```python
+   @parallel()
+   def my_func(item):
+       try:
+           return process(item)
+       except Exception as e:
+           return {"error": str(e), "item": item}
+   ```
+
+## Roadmap
+
+### Observability & Debugging
+- **Advanced Telemetry System**
+  - Task execution metrics (duration, wait times, queue times)
+  - Worker utilization tracking
+  - Error frequency analysis
+  - SQLite persistence for historical data
+  - Interactive visualizations with Plotly
+  - Performance bottleneck identification
+
+- **Rich Logging System**
+  - Configurable log levels per component
+  - Structured logging for machine parsing
+  - Contextual information for debugging
+  - Log rotation and management
+  - Integration with popular logging frameworks
+
+### Advanced Features
+- **Callback System**
+  - Pre/post execution hooks
+  - Error handling callbacks
+  - Progress tracking
+  - Custom metrics collection
+  - State management hooks
+
+- **Smart Scheduling**
+  - Priority queues for tasks
+  - Deadline-aware scheduling
+  - Resource-aware task distribution
+  - Adaptive batch sizing
+  - Dynamic worker scaling
+
+- **Fault Tolerance**
+  - Automatic retries with backoff
+  - Circuit breaker pattern
+  - Fallback strategies
+  - Dead letter queues
+  - Task timeout handling
+
+- **Resource Management**
+  - Memory usage monitoring
+  - CPU utilization tracking
+  - Network bandwidth control
+  - Disk I/O rate limiting
+  - Resource quotas per task
+
+### Developer Experience
+- **CLI Tools**
+  - Task monitoring dashboard
+  - Performance profiling
+  - Configuration management
+  - Log analysis utilities
+  - Telemetry visualization
+
+
+### Enterprise Features
+- **Integration**
+  - Distributed tracing (OpenTelemetry)
+  - Metrics export (Prometheus)
+  - Log aggregation (ELK Stack)
+
+Want to contribute? Check out our [CONTRIBUTING.md](CONTRIBUTING.md) guide!
+
+## API Reference
+
+### @parallel Decorator
+
+```python
+@parallel(
+    max_workers: int = None,          # Maximum workers (default: based on CPU)
+    batch_size: int = None,           # Items per batch (default: all at once)
+    rate_limit: Union[                # Rate limiting configuration
+        float,                        # - Operations per second
+        Tuple[float, str],           # - (count, interval)
+        RateLimit                     # - RateLimit object
+    ] = None,
+    executor_type: str = "thread",    # "thread" or "process"
+    prewarm: bool = False            # Prewarm workers
+)
+```
+
+### RateLimit Class
+
+```python
+class RateLimit:
+    def __init__(self, count: float, interval: str = "second"):
+        """
+        Args:
+            count: Operations allowed per interval
+            interval: "second", "minute", or "hour"
+        """
+```
+
+## Contributing
+
+Contributions are welcome! Please check out our [Contributing Guide](CONTRIBUTING.md).
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

pyarallel-0.1.0/hello.py

@@ -0,0 +1,6 @@
+def main():
+    print("Hello from pyarallel!")
+
+
+if __name__ == "__main__":
+    main()

pyarallel-0.1.0/pyarallel/__init__.py

@@ -0,0 +1,10 @@
+from .core import parallel, RateLimit
+
+__version__ = "0.1.0"
+__all__ = ["parallel", "RateLimit"]
+
+__doc__ = f"""
+pyarallel v{__version__}
+
+Simple parallel processing framework for Python.
+"""