adaptive-executor 0.1.0__tar.gz

adaptive_executor-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Teut2711
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

adaptive_executor-0.1.0/MANIFEST.in

@@ -0,0 +1,6 @@
+include README.md
+include LICENSE
+include pyproject.toml
+recursive-include src/adaptive_executor *.py
+recursive-exclude * __pycache__
+recursive-exclude * .pyc

adaptive_executor-0.1.0/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.4
+Name: adaptive-executor
+Version: 0.1.0
+Summary: Adaptive thread pool executor with dynamic scaling policies
+Author-email: Teut2711 <manglavishesh64@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Teut2711/adaptive-executor
+Project-URL: Documentation, https://Teut2711.github.io/adaptive-executor
+Project-URL: Repository, https://github.com/Teut2711/adaptive-executor
+Project-URL: Bug Tracker, https://github.com/Teut2711/adaptive-executor/issues
+Keywords: threading,concurrency,executor,adaptive,scaling,thread,pool
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: time
+Requires-Dist: pytz>=2023.3; extra == "time"
+Provides-Extra: cpu
+Requires-Dist: psutil>=5.9.0; extra == "cpu"
+Provides-Extra: standard
+Requires-Dist: pytz>=2023.3; extra == "standard"
+Requires-Dist: psutil>=5.9.0; extra == "standard"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: pytest-mock>=3.10.0; extra == "dev"
+Requires-Dist: freezegun>=1.5.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: pre-commit>=3.0.0; extra == "dev"
+Requires-Dist: pytz>=2023.3; extra == "dev"
+Requires-Dist: psutil>=5.9.0; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=6.0.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme>=1.2.0; extra == "docs"
+Requires-Dist: myst-parser>=1.0.0; extra == "docs"
+Dynamic: license-file
+
+
+# Adaptive Executor
+
+Adaptive thread pool executor that scales concurrency based on policies.
+
+## Features
+- Time-based scaling
+- CPU & memory criteria
+- Multi-criterion aggregation
+- Graceful shutdown
+
+## Installation
+
+**Minimal Installation (Core Only)**
+```bash
+pip install adaptive-executor
+```
+
+**Feature-Specific Installations**
+```bash
+# Time-based scaling only
+pip install adaptive-executor[time]
+
+# CPU/Memory monitoring only
+pip install adaptive-executor[cpu]
+
+# Complete feature set (recommended)
+pip install adaptive-executor[standard]
+
+# Development with all tools
+pip install adaptive-executor[dev]
+```
+
+## Quick Start
+
+```python
+from adaptive_executor import AdaptiveExecutor, MultiCriterionPolicy
+from adaptive_executor.criteria import TimeCriterion, CpuCriterion
+from datetime import time
+
+# Create scaling criteria
+time_policy = TimeCriterion(
+    worker_count=8,
+    active_start=time(22, 0),
+    active_end=time(6, 0),
+    timezone="UTC",
+)
+cpu_policy = CpuCriterion(threshold=75, workers=4)
+
+# Combine criteria
+policy = MultiCriterionPolicy([time_policy, cpu_policy], hard_cap=10)
+
+# Create and use executor
+executor = AdaptiveExecutor(max_workers=15, policy=policy)
+
+# Submit tasks
+def my_task(task_id):
+    print(f"Processing task {task_id}")
+
+for i in range(5):
+    executor.submit(my_task, i)
+
+executor.join()
+executor.shutdown()
+```
+
+## Development Setup
+
+This project uses `setuptools` with a `src/` package layout (`src/adaptive_executor`).
+
+```bash
+git clone https://github.com/Teut2711/adaptive-executor.git
+cd adaptive-executor
+uv venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+## Logging Configuration
+
+The adaptive-executor provides comprehensive logging that can be configured to suit your needs. By default, it logs to the console with INFO level.
+
+### Basic Logging Control
+
+```python
+from adaptive_executor import setup_logger
+
+# Disable all logging
+setup_logger(level=logging.CRITICAL)
+
+# Or set a specific log level
+import logging
+setup_logger(level=logging.WARNING)  # Only show warnings and above
+```
+
+### Advanced Logging Configuration
+
+You can also configure file logging with rotation:
+
+```python
+# Configure file logging with rotation (10MB per file, keep 5 backups)
+setup_logger(
+    level=logging.DEBUG,
+    log_file="/var/log/adaptive_executor.log",
+    max_bytes=10*1024*1024,  # 10MB
+    backup_count=5
+)
+```
+
+### Environment Variables
+
+You can control logging via environment variables:
+
+```bash
+# Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
+ADAPTIVE_EXECUTOR_LOG_LEVEL=WARNING
+
+# Enable file logging
+ADAPTIVE_EXECUTOR_LOG_FILE=/path/to/logfile.log
+
+# Set max log file size (in bytes)
+ADAPTIVE_EXECUTOR_LOG_MAX_BYTES=10485760  # 10MB
+
+# Set number of backup files to keep
+ADAPTIVE_EXECUTOR_LOG_BACKUP_COUNT=5
+```
+
+### Disabling Logging Completely
+
+To completely disable all logging:
+
+```python
+import logging
+logging.getLogger('adaptive_executor').addHandler(logging.NullHandler())
+logging.getLogger('adaptive_executor').propagate = False
+```
+
+## Documentation
+
+Full documentation is available at [https://Teut2711.github.io/adaptive-executor](https://Teut2711.github.io/adaptive-executor)
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.

adaptive_executor-0.1.0/README.md

@@ -0,0 +1,144 @@
+
+# Adaptive Executor
+
+Adaptive thread pool executor that scales concurrency based on policies.
+
+## Features
+- Time-based scaling
+- CPU & memory criteria
+- Multi-criterion aggregation
+- Graceful shutdown
+
+## Installation
+
+**Minimal Installation (Core Only)**
+```bash
+pip install adaptive-executor
+```
+
+**Feature-Specific Installations**
+```bash
+# Time-based scaling only
+pip install adaptive-executor[time]
+
+# CPU/Memory monitoring only
+pip install adaptive-executor[cpu]
+
+# Complete feature set (recommended)
+pip install adaptive-executor[standard]
+
+# Development with all tools
+pip install adaptive-executor[dev]
+```
+
+## Quick Start
+
+```python
+from adaptive_executor import AdaptiveExecutor, MultiCriterionPolicy
+from adaptive_executor.criteria import TimeCriterion, CpuCriterion
+from datetime import time
+
+# Create scaling criteria
+time_policy = TimeCriterion(
+    worker_count=8,
+    active_start=time(22, 0),
+    active_end=time(6, 0),
+    timezone="UTC",
+)
+cpu_policy = CpuCriterion(threshold=75, workers=4)
+
+# Combine criteria
+policy = MultiCriterionPolicy([time_policy, cpu_policy], hard_cap=10)
+
+# Create and use executor
+executor = AdaptiveExecutor(max_workers=15, policy=policy)
+
+# Submit tasks
+def my_task(task_id):
+    print(f"Processing task {task_id}")
+
+for i in range(5):
+    executor.submit(my_task, i)
+
+executor.join()
+executor.shutdown()
+```
+
+## Development Setup
+
+This project uses `setuptools` with a `src/` package layout (`src/adaptive_executor`).
+
+```bash
+git clone https://github.com/Teut2711/adaptive-executor.git
+cd adaptive-executor
+uv venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+## Logging Configuration
+
+The adaptive-executor provides comprehensive logging that can be configured to suit your needs. By default, it logs to the console with INFO level.
+
+### Basic Logging Control
+
+```python
+from adaptive_executor import setup_logger
+
+# Disable all logging
+setup_logger(level=logging.CRITICAL)
+
+# Or set a specific log level
+import logging
+setup_logger(level=logging.WARNING)  # Only show warnings and above
+```
+
+### Advanced Logging Configuration
+
+You can also configure file logging with rotation:
+
+```python
+# Configure file logging with rotation (10MB per file, keep 5 backups)
+setup_logger(
+    level=logging.DEBUG,
+    log_file="/var/log/adaptive_executor.log",
+    max_bytes=10*1024*1024,  # 10MB
+    backup_count=5
+)
+```
+
+### Environment Variables
+
+You can control logging via environment variables:
+
+```bash
+# Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
+ADAPTIVE_EXECUTOR_LOG_LEVEL=WARNING
+
+# Enable file logging
+ADAPTIVE_EXECUTOR_LOG_FILE=/path/to/logfile.log
+
+# Set max log file size (in bytes)
+ADAPTIVE_EXECUTOR_LOG_MAX_BYTES=10485760  # 10MB
+
+# Set number of backup files to keep
+ADAPTIVE_EXECUTOR_LOG_BACKUP_COUNT=5
+```
+
+### Disabling Logging Completely
+
+To completely disable all logging:
+
+```python
+import logging
+logging.getLogger('adaptive_executor').addHandler(logging.NullHandler())
+logging.getLogger('adaptive_executor').propagate = False
+```
+
+## Documentation
+
+Full documentation is available at [https://Teut2711.github.io/adaptive-executor](https://Teut2711.github.io/adaptive-executor)
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.

adaptive_executor-0.1.0/pyproject.toml

@@ -0,0 +1,104 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "adaptive-executor"
+version = "0.1.0"
+description = "Adaptive thread pool executor with dynamic scaling policies"
+readme = "README.md"
+license = {text = "MIT"}
+authors = [
+    {name = "Teut2711", email = "manglavishesh64@gmail.com"}
+]
+keywords = ["threading", "concurrency", "executor", "adaptive", "scaling", "thread", "pool"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Topic :: System :: Distributed Computing",
+]
+requires-python = ">=3.9"
+dependencies = []
+
+[project.optional-dependencies]
+time = [
+    "pytz>=2023.3",
+]
+cpu = [
+    "psutil>=5.9.0",
+]
+standard = [
+    "pytz>=2023.3",
+    "psutil>=5.9.0",
+]
+dev = [
+    "pytest>=7.0.0",
+    "pytest-cov>=4.0.0",
+    "pytest-asyncio>=0.21.0",
+    "pytest-mock>=3.10.0",
+    "freezegun>=1.5.0",
+    "ruff>=0.4.0",        # replaces black + isort + flake8
+    "mypy>=1.0.0",
+    "pre-commit>=3.0.0",
+    "pytz>=2023.3",
+    "psutil>=5.9.0",
+]
+docs = [
+    "sphinx>=6.0.0",
+    "sphinx-rtd-theme>=1.2.0",
+    "myst-parser>=1.0.0",
+]
+[project.urls]
+Homepage = "https://github.com/Teut2711/adaptive-executor"
+Documentation = "https://Teut2711.github.io/adaptive-executor"
+Repository = "https://github.com/Teut2711/adaptive-executor"
+"Bug Tracker" = "https://github.com/Teut2711/adaptive-executor/issues"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["adaptive_executor*"]
+
+[tool.ruff]
+line-length = 88
+target-version = "py39"
+
+[tool.ruff.lint]
+select = ["E", "F", "I"]   # pycodestyle + pyflakes + isort
+
+[tool.mypy]
+python_version = "3.9"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+python_classes = ["Test*"]
+python_functions = ["test_*"]
+addopts = "--cov=adaptive_executor --cov-report=term-missing"
+markers = [
+    "integration: slow integration tests",
+    "unit: fast isolated tests",
+]
+
+[tool.coverage.run]
+source = ["adaptive_executor"]
+
+[tool.coverage.report]
+exclude_lines = [
+    "pragma: no cover",
+    "def __repr__",
+    "raise AssertionError",
+    "raise NotImplementedError",
+]

adaptive_executor-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

adaptive_executor-0.1.0/src/adaptive_executor/__init__.py

@@ -0,0 +1,29 @@
+from .criteria import (
+    ConditionalCriterion,
+    CpuCriterion,
+    DateTimeCriterion,
+    MemoryCriterion,
+    MultiCriterion,
+    ScalingCriterion,
+    TimeCriterion,
+    from_dict,
+)
+from .executor import AdaptiveExecutor
+from .policies import MultiCriterionPolicy
+from .utils import get_logger, logger, setup_logger
+
+__all__ = [
+    "AdaptiveExecutor",
+    "MultiCriterionPolicy",
+    "ScalingCriterion",
+    "TimeCriterion",
+    "DateTimeCriterion",
+    "CpuCriterion",
+    "MemoryCriterion",
+    "MultiCriterion",
+    "ConditionalCriterion",
+    "from_dict",
+    "get_logger",
+    "setup_logger",
+    "logger",
+]

adaptive_executor-0.1.0/src/adaptive_executor/criteria/__init__.py

@@ -0,0 +1,59 @@
+"""Scaling criteria for adaptive executor."""
+
+from typing import Any, Dict
+
+# Also expose the modules for patching
+from . import cpu, datetime, memory, multi, time
+from .base import ScalingCriterion
+from .cpu import CpuCriterion
+from .datetime import DateTimeCriterion
+from .memory import MemoryCriterion
+from .multi import ConditionalCriterion, MultiCriterion
+from .time import TimeCriterion
+
+
+def from_dict(data: Dict[str, Any]) -> ScalingCriterion:
+    """Create a criterion from a dictionary.
+
+    Args:
+        data: Dictionary containing serialized criterion data
+
+    Returns:
+        ScalingCriterion: An instance of the appropriate criterion class
+
+    Raises:
+        ValueError: If the criterion type is unknown
+    """
+    criterion_type = data.get("type")
+
+    if criterion_type == "TimeCriterion":
+        return TimeCriterion.from_dict(data)
+    elif criterion_type == "DateTimeCriterion":
+        return DateTimeCriterion.from_dict(data)
+    elif criterion_type == "CpuCriterion":
+        return CpuCriterion.from_dict(data)
+    elif criterion_type == "MemoryCriterion":
+        return MemoryCriterion.from_dict(data)
+    elif criterion_type == "MultiCriterion":
+        return MultiCriterion.from_dict(data)
+    elif criterion_type == "ConditionalCriterion":
+        return ConditionalCriterion.from_dict(data)
+    else:
+        raise ValueError(f"Unknown criterion type: {criterion_type}")
+
+
+__all__ = [
+    "ScalingCriterion",
+    "TimeCriterion",
+    "DateTimeCriterion",
+    "CpuCriterion",
+    "MemoryCriterion",
+    "MultiCriterion",
+    "ConditionalCriterion",
+    "from_dict",
+    "time",
+    "datetime",
+    "cpu",
+    "memory",
+    "multi",
+]

adaptive_executor-0.1.0/src/adaptive_executor/criteria/base.py

@@ -0,0 +1,111 @@
+"""Base class for all scaling criteria."""
+
+import datetime
+import json
+from typing import Any, Dict, Type, TypeVar
+
+from ..utils import get_logger
+
+logger = get_logger(__name__)
+
+# Type variable for criterion classes
+C = TypeVar("C", bound="ScalingCriterion")
+
+
+class ScalingCriterion:
+    """Base class for all scaling criteria.
+
+    Subclasses must implement the max_workers() method to define their scaling logic.
+    """
+
+    def max_workers(self) -> int:
+        """Calculate the maximum number of workers based on this criterion.
+
+        Returns:
+            int: The maximum number of workers (must be at least 1)
+
+        Raises:
+            NotImplementedError: If the subclass doesn't implement this method
+        """
+        raise NotImplementedError("Subclasses must implement max_workers()")
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Serialize criterion to a dictionary.
+
+        Returns:
+            Dict[str, Any]: A dictionary representation of the criterion
+
+        Raises:
+            NotImplementedError: If the subclass doesn't implement this method
+        """
+        raise NotImplementedError("Subclasses must implement to_dict()")
+
+    @classmethod
+    def from_dict(cls: Type[C], data: Dict[str, Any]) -> C:
+        """Deserialize criterion from a dictionary.
+
+        Args:
+            data: Dictionary containing serialized criterion data
+
+        Returns:
+            An instance of the criterion
+
+        Raises:
+            NotImplementedError: If the subclass doesn't implement this method
+            ValueError: If the data is invalid
+        """
+        raise NotImplementedError("Subclasses must implement from_dict()")
+
+    def to_json(self) -> str:
+        """Serialize criterion to a JSON string.
+
+        Returns:
+            str: JSON string representation of the criterion
+
+        Raises:
+            json.JSONEncodeError: If the criterion cannot be serialized to JSON
+        """
+        try:
+            return json.dumps(self.to_dict())
+        except Exception as e:
+            logger.error(
+                "Failed to serialize criterion to JSON: %s", str(e), exc_info=True
+            )
+            raise
+
+    @classmethod
+    def from_json(cls: Type[C], json_str: str) -> C:
+        """Deserialize criterion from a JSON string.
+
+        Args:
+            json_str: JSON string containing serialized criterion data
+
+        Returns:
+            An instance of the criterion
+
+        Raises:
+            json.JSONDecodeError: If the JSON string is invalid
+            ValueError: If the data is invalid
+            NotImplementedError: If the criterion type is not supported
+        """
+        try:
+            data = json.loads(json_str)
+            return cls.from_dict(data)
+        except json.JSONDecodeError as e:
+            logger.error("Invalid JSON string: %s", str(e))
+            raise
+        except Exception as e:
+            logger.error(
+                "Failed to deserialize criterion from JSON: %s", str(e), exc_info=True
+            )
+            raise
+
+    @staticmethod
+    def _parse_datetime(dt_val: str) -> datetime.datetime:
+        """Parse a datetime from string or timestamp."""
+        return datetime.datetime.fromisoformat(dt_val)
+
+    @staticmethod
+    def _format_with_tz(dt: datetime.datetime) -> str:
+        """Format a datetime with timezone if available."""
+        return dt.isoformat()