rl-template 0.1.0__tar.gz

rl_template-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+.opencode

rl_template-0.1.0/.python-version

@@ -0,0 +1 @@
+3.11

rl_template-0.1.0/AGENTS.md

@@ -0,0 +1,82 @@
+# AGENTS.md
+
+## Project
+
+RL training template built on PyTorch + Gymnasium. Provides abstract base classes (`BaseAgent`, `BaseEnv`, `BaseTrain`) and a concrete PPO (Proximal Policy Optimization) implementation. Includes a full test suite (107 tests, all passing).
+
+## Package management
+
+- Managed with **uv** (lockfile: `uv.lock`, Python version: `.python-version` → 3.11)
+- Dependencies are pinned in `requirements.txt`, not declared in `pyproject.toml`
+
+## Lint / typecheck
+
+```
+ruff check rl_template/       # linter (Pyflakes only, ruff.toml)
+mypy rl_template/             # strict: disallow_untyped_defs, warn_unreachable (mypy.ini)
+```
+
+No pre-commit hooks, no CI, no formatter config beyond ruff defaults.
+
+## Import style gotcha
+
+- `train.py` uses **relative imports** (`from .env import BaseEnv`) — requires package-level import
+- `ppo.py` uses **relative imports** (`from ...common import Buffer`) — assumes package-level import
+- `algorithms/__init__.py` and `algorithms/ppo/__init__.py` exist (empty) — required for relative imports in `ppo.py` to resolve
+
+## Structure
+
+```
+rl_template/
+  __init__.py            # empty package marker
+  agent.py               # BaseAgent (ABC, nn.Module) — abstract agent interface
+  env.py                 # BaseEnv (ABC) — Gymnasium v1 API wrapper interface
+  train.py               # BaseTrain (ABC) — rollout/update/save training loop
+  common.py              # Buffer — pre-allocated numpy rollout buffer with size property
+  config.py              # PPOConfig (frozen), TrainConfig, WandbConfig dataclasses
+  errors.py              # EmptyBufferError with detailed __str__
+  algorithms/
+    __init__.py          # package marker (empty)
+    ppo/
+      __init__.py        # package marker (empty)
+      ppo.py             # PPOTrainer — GAE, clipped surrogate loss, linear LR decay
+tests/
+  __init__.py
+  test_common.py         # Buffer unit tests (21 tests)
+  test_config.py         # Config dataclass tests (13 tests)
+  test_errors.py         # EmptyBufferError tests (7 tests)
+  test_ppo.py            # PPOTrainer tests (12 tests)
+  test_buffer_integration.py  # Buffer integration tests (10 tests)
+  test_agent.py         # BaseAgent abstract interface tests (12 tests)
+  test_env.py           # BaseEnv abstract interface tests (11 tests)
+  test_train.py         # BaseTrain tests: init, save_model, update_weights (6 tests)
+```
+
+## Conventions
+
+- Abstract base classes use `@abstractmethod` with concrete bodies (template method pattern) — subclasses override and optionally call `super()`
+- `PPOConfig` is `frozen=True` (immutable); `TrainConfig` uses `__post_init__` for computed fields
+- `Buffer` pre-allocates numpy arrays and uses a slice pointer with a `size` property and bounds checking in `insert()`
+- GPU detection is automatic in `TrainConfig.device`
+- Tests use pytest, import rl_template as a package (`from rl_template.common import Buffer`)
+
+## Testing
+
+```
+python -m pytest tests/ -v    # run all 107 tests
+```
+
+## Fixes applied
+
+- Created `algorithms/__init__.py` and `algorithms/ppo/__init__.py` for package resolution
+- Added `size` property to `Buffer` class (returns current element count)
+- Added bounds checking in `Buffer.insert()` — raises `ValueError` when buffer is full
+- Fixed `save_model()` to use `os.makedirs(os.path.dirname(...), exist_ok=True)` instead of creating a `.pt` directory
+- Fixed PPO `update()` loss tracking — `index_loss` now increments correctly, final mean excludes trailing zeros
+- Fixed typo `BaseAgnt` → `BaseAgent` in `agent.py`
+- Fixed `ruff.toml` `target-version` from `"py313"` to `"py311"` to match `.python-version`
+- Removed `print()` side effects from `TrainConfig.__post_init__`
+- Fixed `Buffer.get_all()` actions dtype from `torch.long` to `torch.float32` (continuous actions were silently truncated to integers)
+- Fixed `Distributions` → `Distribution` import typo in `agent.py` (pre-existing bug, class is singular in PyTorch)
+- Changed `train.py` from bare imports to relative imports for package-level compatibility
+- Rewrote all docstrings and comments across source and test files for consistency

rl_template-0.1.0/PKG-INFO

@@ -0,0 +1,145 @@
+Metadata-Version: 2.4
+Name: rl-template
+Version: 0.1.0
+Summary: A type-safe, algorithm-agnostic Reinforcement Learning framework
+Author-email: Mohamed Tine <mohamedtine17@gmail.com>
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: torch>=2.0.0
+Description-Content-Type: text/markdown
+
+# RL Template
+
+A modular reinforcement learning training framework built on PyTorch and Gymnasium, providing abstract base classes and a concrete PPO implementation.
+
+## Features
+
+- **Abstract base classes** — `BaseAgent`, `BaseEnv`, `BaseTrain` with template method pattern for easy extension
+- **PPO implementation** — Clipped surrogate loss, GAE, linear LR decay, entropy bonus, gradient clipping
+- **Pre-allocated buffer** — Zero-allocation rollout storage with bounds checking, supports discrete and continuous actions
+- **Frozen configs** — Immutable `PPOConfig` and computed `TrainConfig` dataclasses
+- **Auto GPU detection** — Automatic CUDA/CPU device selection
+- **107 tests** — Full test suite covering all components
+
+## Project Structure
+
+```
+rl_template/
+  __init__.py
+  agent.py               # BaseAgent — abstract policy/value interface
+  env.py                 # BaseEnv — Gymnasium v1 API wrapper
+  train.py               # BaseTrain — rollout/update training loop
+  common.py              # Buffer — pre-allocated numpy rollout buffer
+  config.py              # PPOConfig, TrainConfig, WandbConfig
+  errors.py              # EmptyBufferError
+  algorithms/
+    ppo/
+      ppo.py             # PPOTrainer — GAE + clipped surrogate loss
+tests/
+  test_agent.py          # BaseAgent tests (12)
+  test_env.py            # BaseEnv tests (11)
+  test_train.py          # BaseTrain tests (6)
+  test_common.py         # Buffer tests (21)
+  test_config.py         # Config tests (13)
+  test_errors.py         # Error tests (7)
+  test_ppo.py            # PPOTrainer tests (15)
+  test_buffer_integration.py  # Buffer integration tests (6)
+  test_continuous_actions.py  # Continuous action tests (16)
+```
+
+## Installation
+
+Requires Python 3.11+.
+
+```bash
+# With uv (recommended)
+uv sync
+
+# With pip
+pip install -r requirements.txt
+```
+
+## Usage
+
+```python
+import numpy as np
+from rl_template.agent import BaseAgent
+from rl_template.env import BaseEnv
+from rl_template.train import BaseTrain
+from rl_template.common import Buffer
+from rl_template.config import TrainConfig, PPOConfig
+from rl_template.algorithms.ppo.ppo import PPOTrainer
+
+
+# 1. Implement your agent
+class MyAgent(BaseAgent):
+    # Implement forward(), get_distribution()
+    ...
+
+# 2. Implement your environment
+class MyEnv(BaseEnv):
+    # Implement reset(), step(), close()
+    ...
+
+# 3. Configure training
+config = TrainConfig(model_name="my_agent", model_saved_path="./checkpoints")
+ppo_config = PPOConfig(lr=3e-4, gamma=0.99, clip_eps=0.2)
+
+# 4. Wire everything together
+agent = MyAgent()
+env = MyEnv()
+buffer = Buffer(step=config.rollout_steps, state_shape=(4,))
+ppo_trainer = PPOTrainer(agent, lr=ppo_config.lr, gamma=ppo_config.gamma)
+
+# 5. Create trainer and run
+class MyTrain(BaseTrain):
+    def rollout_phase(self, state): super().rollout_phase(state)
+    def update_weights(self, step): super().update_weights(step)
+    def save_model(self): super().save_model()
+
+trainer = MyTrain(agent, env, buffer, config, ppo_trainer)
+state, _ = env.reset()
+
+for step in range(config.num_update):
+    trainer.rollout_phase(state)
+    trainer.update_weights(step)
+    trainer.save_model()
+```
+
+## Key Classes
+
+| Class | Description |
+|-------|-------------|
+| `BaseAgent` | Abstract agent combining ABC and nn.Module. Subclasses implement `forward()` and `get_distribution()`. The `get_action()` template method handles sampling and log-probability computation. |
+| `BaseEnv` | Abstract Gymnasium v1 environment wrapper. Subclasses implement `reset()`, `step()`, and `close()`. |
+| `BaseTrain` | Abstract training loop orchestrating rollout collection, GAE computation, and PPO updates. |
+| `Buffer` | Pre-allocated numpy buffer for rollout data. Supports discrete and continuous actions with automatic dtype handling. |
+| `PPOTrainer` | PPO algorithm implementation with GAE, clipped surrogate loss, value loss, entropy bonus, and linear LR decay. |
+| `PPOConfig` | Frozen (immutable) PPO hyperparameters. |
+| `TrainConfig` | Training configuration with auto-computed `model_path` and `num_update`. |
+
+## PPO Algorithm
+
+The PPO trainer implements:
+
+- **Generalized Advantage Estimation (GAE)** — Blended n-step returns for low-variance advantage estimates
+- **Clipped surrogate loss** — Prevents destructive policy updates
+- **Value function loss** — MSE loss for the critic network
+- **Entropy bonus** — Encourages exploration
+- **Linear LR decay** — Learning rate anneals to zero over training
+- **Gradient clipping** — Max norm of 1.0 for stability
+
+## Testing
+
+```bash
+python -m pytest tests/ -v
+```
+
+All 107 tests cover unit tests for each component, integration tests for buffer workflows, and continuous action space validation.
+
+## License
+
+MIT

rl_template-0.1.0/README.md

@@ -0,0 +1,132 @@
+# RL Template
+
+A modular reinforcement learning training framework built on PyTorch and Gymnasium, providing abstract base classes and a concrete PPO implementation.
+
+## Features
+
+- **Abstract base classes** — `BaseAgent`, `BaseEnv`, `BaseTrain` with template method pattern for easy extension
+- **PPO implementation** — Clipped surrogate loss, GAE, linear LR decay, entropy bonus, gradient clipping
+- **Pre-allocated buffer** — Zero-allocation rollout storage with bounds checking, supports discrete and continuous actions
+- **Frozen configs** — Immutable `PPOConfig` and computed `TrainConfig` dataclasses
+- **Auto GPU detection** — Automatic CUDA/CPU device selection
+- **107 tests** — Full test suite covering all components
+
+## Project Structure
+
+```
+rl_template/
+  __init__.py
+  agent.py               # BaseAgent — abstract policy/value interface
+  env.py                 # BaseEnv — Gymnasium v1 API wrapper
+  train.py               # BaseTrain — rollout/update training loop
+  common.py              # Buffer — pre-allocated numpy rollout buffer
+  config.py              # PPOConfig, TrainConfig, WandbConfig
+  errors.py              # EmptyBufferError
+  algorithms/
+    ppo/
+      ppo.py             # PPOTrainer — GAE + clipped surrogate loss
+tests/
+  test_agent.py          # BaseAgent tests (12)
+  test_env.py            # BaseEnv tests (11)
+  test_train.py          # BaseTrain tests (6)
+  test_common.py         # Buffer tests (21)
+  test_config.py         # Config tests (13)
+  test_errors.py         # Error tests (7)
+  test_ppo.py            # PPOTrainer tests (15)
+  test_buffer_integration.py  # Buffer integration tests (6)
+  test_continuous_actions.py  # Continuous action tests (16)
+```
+
+## Installation
+
+Requires Python 3.11+.
+
+```bash
+# With uv (recommended)
+uv sync
+
+# With pip
+pip install -r requirements.txt
+```
+
+## Usage
+
+```python
+import numpy as np
+from rl_template.agent import BaseAgent
+from rl_template.env import BaseEnv
+from rl_template.train import BaseTrain
+from rl_template.common import Buffer
+from rl_template.config import TrainConfig, PPOConfig
+from rl_template.algorithms.ppo.ppo import PPOTrainer
+
+
+# 1. Implement your agent
+class MyAgent(BaseAgent):
+    # Implement forward(), get_distribution()
+    ...
+
+# 2. Implement your environment
+class MyEnv(BaseEnv):
+    # Implement reset(), step(), close()
+    ...
+
+# 3. Configure training
+config = TrainConfig(model_name="my_agent", model_saved_path="./checkpoints")
+ppo_config = PPOConfig(lr=3e-4, gamma=0.99, clip_eps=0.2)
+
+# 4. Wire everything together
+agent = MyAgent()
+env = MyEnv()
+buffer = Buffer(step=config.rollout_steps, state_shape=(4,))
+ppo_trainer = PPOTrainer(agent, lr=ppo_config.lr, gamma=ppo_config.gamma)
+
+# 5. Create trainer and run
+class MyTrain(BaseTrain):
+    def rollout_phase(self, state): super().rollout_phase(state)
+    def update_weights(self, step): super().update_weights(step)
+    def save_model(self): super().save_model()
+
+trainer = MyTrain(agent, env, buffer, config, ppo_trainer)
+state, _ = env.reset()
+
+for step in range(config.num_update):
+    trainer.rollout_phase(state)
+    trainer.update_weights(step)
+    trainer.save_model()
+```
+
+## Key Classes
+
+| Class | Description |
+|-------|-------------|
+| `BaseAgent` | Abstract agent combining ABC and nn.Module. Subclasses implement `forward()` and `get_distribution()`. The `get_action()` template method handles sampling and log-probability computation. |
+| `BaseEnv` | Abstract Gymnasium v1 environment wrapper. Subclasses implement `reset()`, `step()`, and `close()`. |
+| `BaseTrain` | Abstract training loop orchestrating rollout collection, GAE computation, and PPO updates. |
+| `Buffer` | Pre-allocated numpy buffer for rollout data. Supports discrete and continuous actions with automatic dtype handling. |
+| `PPOTrainer` | PPO algorithm implementation with GAE, clipped surrogate loss, value loss, entropy bonus, and linear LR decay. |
+| `PPOConfig` | Frozen (immutable) PPO hyperparameters. |
+| `TrainConfig` | Training configuration with auto-computed `model_path` and `num_update`. |
+
+## PPO Algorithm
+
+The PPO trainer implements:
+
+- **Generalized Advantage Estimation (GAE)** — Blended n-step returns for low-variance advantage estimates
+- **Clipped surrogate loss** — Prevents destructive policy updates
+- **Value function loss** — MSE loss for the critic network
+- **Entropy bonus** — Encourages exploration
+- **Linear LR decay** — Learning rate anneals to zero over training
+- **Gradient clipping** — Max norm of 1.0 for stability
+
+## Testing
+
+```bash
+python -m pytest tests/ -v
+```
+
+All 107 tests cover unit tests for each component, integration tests for buffer workflows, and continuous action space validation.
+
+## License
+
+MIT

rl_template-0.1.0/mypy.ini

@@ -0,0 +1,4 @@
+[mypy]
+files = rl_template
+disallow_untyped_defs = True
+warn_unreachable = True

rl_template-0.1.0/pyproject.toml

@@ -0,0 +1,22 @@
+[project]
+name = "rl-template"
+version = "0.1.0"
+description = "A type-safe, algorithm-agnostic Reinforcement Learning framework"
+readme = "README.md"
+requires-python = ">=3.11"
+authors = [
+    { name = "Mohamed Tine", email = "mohamedtine17@gmail.com" }
+]
+classifiers = [
+    "Programming Language :: Python :: 3.11",
+    "License :: OSI Approved :: MIT License",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence"
+]
+dependencies = [
+    "torch>=2.0.0",
+    "numpy>=1.24.0"
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"

rl_template-0.1.0/requirements.txt

@@ -0,0 +1,79 @@
+alembic==1.18.5
+annotated-types==0.7.0
+bottle==0.13.4
+certifi==2026.6.17
+charset-normalizer==3.4.9
+click==8.4.2
+cloudpickle==3.1.2
+colorlog==6.10.1
+contourpy==1.3.3
+cuda-bindings==13.3.1
+cuda-pathfinder==1.5.6
+cuda-toolkit==13.0.3.0
+cycler==0.12.1
+farama-notifications==0.0.6
+filelock==3.29.7
+fonttools==4.63.0
+fsspec==2026.6.0
+gitdb==4.0.12
+gitpython==3.1.50
+greenlet==3.5.3
+gymnasium==1.3.0
+idna==3.18
+jinja2==3.1.6
+joblib==1.5.3
+kiwisolver==1.5.0
+mako==1.3.12
+markupsafe==3.0.3
+matplotlib==3.11.0
+mpmath==1.3.0
+narwhals==2.23.0
+networkx==3.6.1
+numpy==2.4.6
+nvidia-cublas==13.1.1.3
+nvidia-cuda-cupti==13.0.85
+nvidia-cuda-nvrtc==13.0.88
+nvidia-cuda-runtime==13.0.96
+nvidia-cudnn-cu13==9.20.0.48
+nvidia-cufft==12.0.0.61
+nvidia-cufile==1.15.1.6
+nvidia-curand==10.4.0.35
+nvidia-cusolver==12.0.4.66
+nvidia-cusparse==12.6.3.3
+nvidia-cusparselt-cu13==0.8.1
+nvidia-nccl-cu13==2.29.7
+nvidia-nvjitlink==13.3.33
+nvidia-nvshmem-cu13==3.4.5
+nvidia-nvtx==13.0.85
+optuna==4.9.0
+optuna-dashboard==0.20.0
+packaging==26.2
+pandas==3.0.3
+pillow==12.3.0
+platformdirs==4.10.0
+plotly==6.8.0
+protobuf==7.35.1
+pydantic==2.13.4
+pydantic-core==2.46.4
+pyparsing==3.3.2
+python-dateutil==2.9.0.post0
+pyyaml==6.0.3
+requests==2.34.2
+ruff==0.15.20
+scikit-learn==1.9.0
+scipy==1.17.1
+seaborn==0.13.2
+sentry-sdk==2.64.0
+setuptools==83.0.0
+six==1.17.0
+smmap==5.0.3
+sqlalchemy==2.0.51
+sympy==1.14.0
+threadpoolctl==3.6.0
+torch==2.13.0
+tqdm==4.68.4
+triton==3.7.1
+typing-extensions==4.16.0
+typing-inspection==0.4.2
+urllib3==2.7.0
+wandb==0.28.0

rl_template-0.1.0/rl_template/agent.py

@@ -0,0 +1,69 @@
+"""Abstract base agent interface for reinforcement learning.
+
+Provides BaseAgent, an ABC + nn.Module hybrid that enforces a consistent
+policy/value interface for all RL agent implementations.
+"""
+
+import numpy as np
+import torch.nn as nn
+from torch.distributions import Distribution
+from torch import Tensor
+from abc import ABC, abstractmethod
+
+
+class BaseAgent(ABC, nn.Module):
+    """Abstract base class for all RL agents.
+
+    Subclasses must implement forward() and get_distribution(). The
+    get_action() template method composes those two to sample actions,
+    compute log probabilities, and return critic values.
+    """
+
+    def __init__(self):
+        super().__init__()
+
+    @abstractmethod
+    def forward(self, state: np.ndarray) -> tuple[Tensor, Tensor]:
+        """Compute raw policy logits and value estimate.
+
+        Args:
+            state: Environment observation.
+
+        Returns:
+            Tuple of (policy_logits, value) tensors.
+        """
+        pass
+
+    @abstractmethod
+    def get_distribution(self, state: np.ndarray) -> tuple[Distribution, Tensor]:
+        """Build a distribution over actions for the given state.
+
+        Args:
+            state: Environment observation.
+
+        Returns:
+            Tuple of (distribution, value) where distribution is a
+            torch.distributions.Distribution instance.
+        """
+        pass
+
+    def get_action(self, state: np.ndarray, action: int | None = None) -> tuple[Tensor, Tensor, Tensor, Tensor]:
+        """Sample or evaluate an action under the current policy.
+
+        Template method: calls get_distribution(), then samples if no
+        action is provided.
+
+        Args:
+            state: Environment observation.
+            action: Optional pre-selected action. When None, samples from
+                    the policy distribution.
+
+        Returns:
+            Tuple of (action, log_prob, entropy, value).
+        """
+        dist, value = self.get_distribution(state)
+        if action is None:
+            action = dist.sample()
+        log_prob = dist.log_prob(action)
+        dist_entropy = dist.entropy()
+        return (action, log_prob, dist_entropy, value)