saferl-lite 0.1.0__tar.gz

saferl_lite-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Satyam Mishra
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

saferl_lite-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include README.md
+include requirements.txt
+include LICENSE
+recursive-include saferl_lite *.py
+recursive-include docs *

saferl_lite-0.1.0/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: saferl-lite
+Version: 0.1.0
+Summary: A lightweight, explainable, and constrained reinforcement learning toolkit.
+Home-page: https://github.com/satyamcser/saferl-lite
+Author: Satyam Mishra
+Author-email: satyam@example.com
+Project-URL: Documentation, https://satyamcser.github.io/saferl-lite/
+Project-URL: Source, https://github.com/satyamcser/saferl-lite
+Project-URL: Bug Tracker, https://github.com/satyamcser/saferl-lite/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: gym
+Requires-Dist: gymnasium
+Requires-Dist: numpy
+Requires-Dist: torch
+Requires-Dist: matplotlib
+Requires-Dist: seaborn
+Requires-Dist: pre-commit
+Requires-Dist: flake8
+Requires-Dist: pyyaml
+Requires-Dist: shap
+Requires-Dist: captum
+Requires-Dist: typer
+Requires-Dist: scikit-learn
+Requires-Dist: pandas
+Requires-Dist: pytest
+Requires-Dist: pytest-cov
+Requires-Dist: coverage
+Requires-Dist: mkdocs
+Requires-Dist: wandb
+Requires-Dist: mkdocs>=1.5
+Requires-Dist: mkdocs-material>=9.5
+Requires-Dist: mkdocstrings[python]
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# 🔐 SafeRL-Lite
+
+A **lightweight, explainable, and modular** Python library for **Constrained Reinforcement Learning (Safe RL)** with real-time **SHAP & saliency-based explainability**, custom metrics, and Gym-compatible wrappers.
+
+<p align="center">
+  <img src="https://img.shields.io/github/license/satyamcser/saferl-lite?style=flat-square">
+  <img src="https://img.shields.io/github/stars/satyamcser/saferl-lite?style=flat-square">
+  <img src="https://img.shields.io/pypi/v/saferl-lite?style=flat-square">
+  <img src="https://img.shields.io/github/actions/workflow/status/satyamcser/saferl-lite/ci.yml?branch=main&style=flat-square">
+</p>
+
+---
+
+## 🌟 Overview
+
+**SafeRL-Lite** empowers reinforcement learning agents to act under **safety constraints**, while remaining **interpretable** and **modular** for fast experimentation. It wraps standard Gym environments and DQN-based agents with:
+
+- ✅ Safety constraint logic
+- 🔍 Visual explainability (SHAP, saliency maps)
+- 📊 Violation and reward tracking
+- 🧪 Built-in testing and evaluations
+
+---
+
+## 🔧 Installation
+
+> 📦 PyPI (coming soon)
+```bash
+pip install saferl-lite
+```
+
+## 🛠️ From source:
+
+```bash
+git clone https://github.com/satyamcser/saferl-lite.git
+cd saferl-lite
+pip install -e .
+```
+
+## 🚀 Quickstart
+Train a constrained DQN agent with saliency-based explainability:
+
+```bash
+python train.py --env CartPole-v1 --constraint pole_angle --explain shap
+```
+
+🔹 This:
+
+- Adds a pole-angle constraint wrapper to the Gym env
+
+- Logs violations
+
+- Displays SHAP or saliency explanations for agent decisions
+
+## 🧠 Features
+#### ✅ Constrained RL
+- Add custom constraints via wrapper or logic class
+
+- Violation logging and reward shaping
+
+- Safe vs unsafe episode tracking
+
+#### 🔍 Explainability
+- SaliencyExplainer — gradient-based visual heatmaps
+
+- SHAPExplainer — feature contribution values per decision
+
+- Compatible with any PyTorch-based agent
+
+#### 📊 Metrics
+- Constraint violation rate
+
+- Episode reward
+
+- Cumulative safe reward
+
+- Action entropy & temporal behavior stats
+
+#### 📚 Modularity
+- Swap out agents, constraints, evaluators, or explainers
+
+- Supports Gym environments
+
+- Configurable training pipeline
+
+## 📜 Citation
+Coming soon after arXiv/preprint release.

saferl_lite-0.1.0/README.md

@@ -0,0 +1,87 @@
+# 🔐 SafeRL-Lite
+
+A **lightweight, explainable, and modular** Python library for **Constrained Reinforcement Learning (Safe RL)** with real-time **SHAP & saliency-based explainability**, custom metrics, and Gym-compatible wrappers.
+
+<p align="center">
+  <img src="https://img.shields.io/github/license/satyamcser/saferl-lite?style=flat-square">
+  <img src="https://img.shields.io/github/stars/satyamcser/saferl-lite?style=flat-square">
+  <img src="https://img.shields.io/pypi/v/saferl-lite?style=flat-square">
+  <img src="https://img.shields.io/github/actions/workflow/status/satyamcser/saferl-lite/ci.yml?branch=main&style=flat-square">
+</p>
+
+---
+
+## 🌟 Overview
+
+**SafeRL-Lite** empowers reinforcement learning agents to act under **safety constraints**, while remaining **interpretable** and **modular** for fast experimentation. It wraps standard Gym environments and DQN-based agents with:
+
+- ✅ Safety constraint logic
+- 🔍 Visual explainability (SHAP, saliency maps)
+- 📊 Violation and reward tracking
+- 🧪 Built-in testing and evaluations
+
+---
+
+## 🔧 Installation
+
+> 📦 PyPI (coming soon)
+```bash
+pip install saferl-lite
+```
+
+## 🛠️ From source:
+
+```bash
+git clone https://github.com/satyamcser/saferl-lite.git
+cd saferl-lite
+pip install -e .
+```
+
+## 🚀 Quickstart
+Train a constrained DQN agent with saliency-based explainability:
+
+```bash
+python train.py --env CartPole-v1 --constraint pole_angle --explain shap
+```
+
+🔹 This:
+
+- Adds a pole-angle constraint wrapper to the Gym env
+
+- Logs violations
+
+- Displays SHAP or saliency explanations for agent decisions
+
+## 🧠 Features
+#### ✅ Constrained RL
+- Add custom constraints via wrapper or logic class
+
+- Violation logging and reward shaping
+
+- Safe vs unsafe episode tracking
+
+#### 🔍 Explainability
+- SaliencyExplainer — gradient-based visual heatmaps
+
+- SHAPExplainer — feature contribution values per decision
+
+- Compatible with any PyTorch-based agent
+
+#### 📊 Metrics
+- Constraint violation rate
+
+- Episode reward
+
+- Cumulative safe reward
+
+- Action entropy & temporal behavior stats
+
+#### 📚 Modularity
+- Swap out agents, constraints, evaluators, or explainers
+
+- Supports Gym environments
+
+- Configurable training pipeline
+
+## 📜 Citation
+Coming soon after arXiv/preprint release.

saferl_lite-0.1.0/agents/constrained_dqn.py

@@ -0,0 +1,82 @@
+# agents/constrained_dqn.py
+
+import numpy as np
+import random
+import torch
+import torch.nn as nn
+import torch.optim as optim
+from collections import deque
+from agents.constraints import Constraint
+
+
+class DQNetwork(nn.Module):
+    def __init__(self, input_dim, output_dim):
+        super().__init__()
+        self.net = nn.Sequential(
+            nn.Linear(input_dim, 128),
+            nn.ReLU(),
+            nn.Linear(128, 128),
+            nn.ReLU(),
+            nn.Linear(128, output_dim),
+        )
+
+    def forward(self, x):
+        return self.net(x)
+
+
+class ConstrainedDQNAgent:
+    def __init__(self, state_dim, action_dim, constraint: Constraint = None):
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        self.q_net = DQNetwork(state_dim, action_dim).to(self.device)
+        self.target_net = DQNetwork(state_dim, action_dim).to(self.device)
+        self.target_net.load_state_dict(self.q_net.state_dict())
+
+        self.optimizer = optim.Adam(self.q_net.parameters(), lr=1e-3)
+        self.memory = deque(maxlen=10000)
+        self.gamma = 0.99
+        self.batch_size = 64
+
+        self.constraint = constraint
+        self.action_dim = action_dim
+
+    def select_action(self, state, epsilon):
+        if random.random() < epsilon:
+            return random.randint(0, self.action_dim - 1)
+        state = torch.tensor(state, dtype=torch.float32).to(self.device)
+        with torch.no_grad():
+            q_values = self.q_net(state)
+        return q_values.argmax().item()
+
+    def store_transition(self, s, a, r, s_next, done):
+        self.memory.append((s, a, r, s_next, done))
+
+    def update(self):
+        if len(self.memory) < self.batch_size:
+            return
+        batch = random.sample(self.memory, self.batch_size)
+        s, a, r, s_next, done = zip(*batch)
+
+        s = torch.tensor(s, dtype=torch.float32).to(self.device)
+        a = torch.tensor(a).to(self.device)
+        r = torch.tensor(r, dtype=torch.float32).to(self.device)
+        s_next = torch.tensor(s_next, dtype=torch.float32).to(self.device)
+        done = torch.tensor(done, dtype=torch.float32).to(self.device)
+
+        q_values = self.q_net(s).gather(1, a.unsqueeze(1)).squeeze()
+        with torch.no_grad():
+            target_q = r + self.gamma * self.target_net(s_next).max(1)[0] * (1 - done)
+
+        loss = nn.functional.mse_loss(q_values, target_q)
+        self.optimizer.zero_grad()
+        loss.backward()
+        self.optimizer.step()
+
+    def apply_constraint(self, state, action, reward):
+        if self.constraint:
+            penalty = self.constraint.compute_penalty(state, action, reward)
+            return reward - penalty, penalty
+        return reward, 0.0
+
+    def reset_constraints(self):
+        if self.constraint:
+            self.constraint.reset()

saferl_lite-0.1.0/agents/constraints.py

@@ -0,0 +1,39 @@
+# agents/constraints.py
+
+from abc import ABC, abstractmethod
+
+
+class Constraint(ABC):
+    """Abstract base class for constraints in SafeRL agents."""
+
+    @abstractmethod
+    def compute_penalty(self, state, action, reward) -> float:
+        """Return penalty value for a given step."""
+        pass
+
+    def reset(self):
+        """Optional: reset internal counters for new episode."""
+        pass
+
+
+class ActionBudgetConstraint(Constraint):
+    def __init__(self, max_actions: int):
+        self.max_actions = max_actions
+        self.counter = 0
+
+    def compute_penalty(self, state, action, reward) -> float:
+        self.counter += 1
+        return 1.0 if self.counter > self.max_actions else 0.0
+
+    def reset(self):
+        self.counter = 0
+
+
+class EnergyPenaltyConstraint(Constraint):
+    def __init__(self, energy_fn, max_energy):
+        self.energy_fn = energy_fn
+        self.max_energy = max_energy
+
+    def compute_penalty(self, state, action, reward) -> float:
+        energy = self.energy_fn(state, action)
+        return max(0.0, energy - self.max_energy)

saferl_lite-0.1.0/docs/about.md

@@ -0,0 +1,65 @@
+# 🙌 About SafeRL-Lite
+
+**SafeRL-Lite** is a lightweight, explainable, and modular Python library for constrained reinforcement learning (Safe RL). It is designed for researchers, educators, and engineers who want to build safe, interpretable agents in Gym-compatible environments with minimal setup.
+
+---
+
+## 🎯 Mission
+
+SafeRL-Lite aims to:
+- Enable **constrained RL** with built-in safety logic.
+- Support **real-time explainability** using SHAP and saliency maps.
+- Provide **modular components** for easy experimentation and extension.
+- Be **minimal**, **readable**, and **production-friendly**.
+
+---
+
+## 🧱 Key Components
+
+- **Constrained DQN agent** with runtime violation tracking.
+- **Environment wrappers** for safety constraints and logging.
+- **Evaluation module** to track rewards, constraint violations, and success metrics.
+- **Explainability tools**:
+  - Gradient-based **saliency maps**
+  - **SHAP** values for input importance
+
+---
+
+## 👩‍🔬 Who Should Use This?
+
+SafeRL-Lite is ideal for:
+- 📚 **Educators** teaching RL and safety concepts.
+- 🔬 **Researchers** in safe RL, explainable AI (XAI), or constrained optimization.
+- 🧪 **Experimenters** building proof-of-concepts for safety-critical environments (e.g., healthcare, robotics).
+
+---
+
+## 🛠️ Design Philosophy
+
+- ✅ Simplicity over complexity  
+- 🧩 Component reusability  
+- 🔍 Built-in explainability  
+- 🧪 Test-driven development  
+- 📦 PyPI-ready structure  
+
+---
+
+## 🤝 Contributing
+
+We welcome contributions from the community! Check out the [CONTRIBUTING.md](https://github.com/satyamcser/saferl-lite/blob/main/CONTRIBUTING.md) for guidelines.
+
+---
+
+## 📜 License
+
+This project is licensed under the **MIT License**. See the [LICENSE](https://github.com/satyamcser/saferl-lite/blob/main/LICENSE) file for details.
+
+---
+
+## 🌐 Links
+
+- GitHub: [https://github.com/satyamcser/saferl-lite](https://github.com/satyamcser/saferl-lite)
+- PyPI: *Coming soon*
+- Docs: *This site*
+
+---

saferl_lite-0.1.0/docs/docs/index.md

@@ -0,0 +1,34 @@
+# SafeRL-Lite
+
+Welcome to **SafeRL-Lite**, a lightweight, explainable, and constrained reinforcement learning library built in Python.
+
+---
+
+## ✨ Features
+
+- ♻️ **Constrained Agents**: Supports safe training using budgeted penalties and constraint wrappers.
+- 🔍 **Explainability Tools**: Includes SHAP and saliency-based attribution methods.
+- 🎮 **Custom Environments**: Easy wrappers for constraint-aware OpenAI Gym environments.
+- 📊 **Evaluation Metrics**: Regret, safe episode rate, violation count.
+- 🧪 **Tested and Documented**: Full unit test suite with CI + MkDocs support.
+
+---
+
+## 🔧 Repository Info
+
+- **Author**: [Satyam Mishra](https://github.com/satyamcser)
+- **GitHub**: [github.com/satyamcser/saferl-lite](https://github.com/satyamcser/saferl-lite)
+
+---
+
+## 📦 Installation
+
+See [Usage → Installation](usage/installation.md) for details.
+
+## 🚀 Quickstart
+
+See [Usage → Quickstart](usage/quickstart.md) to get up and running in 60 seconds.
+
+## 🧪 Example
+
+- [CartPole with Constraints](examples/cartpole.md)

saferl_lite-0.1.0/docs/evaluation/metrics.md

@@ -0,0 +1,113 @@
+# 📊 SafeRL-Lite Evaluation Metrics
+
+SafeRL-Lite provides built-in evaluation metrics to assess both **performance** and **safety** of RL agents in constrained environments.
+
+---
+
+## 🎯 1. `evaluate_safety_violations`
+
+```python
+evaluate_safety_violations(violations: List[int]) -> float
+```
+
+#### Description:
+Calculates the percentage of time steps where the agent violated a safety constraint.
+
+#### Arguments:
+
+- violations (List[int]): A binary list where 1 indicates a violation at that timestep.
+
+Returns:
+
+- float: Fraction of violated steps (e.g. 0.12 means 12% violation rate).
+
+### Usage Example:
+
+```python
+violations = [0, 1, 0, 1, 1]
+violation_rate = evaluate_safety_violations(violations)
+# Output: 0.6
+```
+---
+
+## 📈 2. evaluate_cumulative_reward
+
+```python
+evaluate_cumulative_reward(rewards: List[float]) -> float
+
+```
+
+#### Description:
+Computes the total accumulated reward across an episode.
+
+#### Arguments:
+
+- rewards (List[float]): A list of numerical rewards at each timestep.
+
+Returns:
+
+- float: Total reward over the episode.
+
+### Usage Example:
+
+```python
+rewards = [1.0, -0.2, 2.0, 0.5]
+total_reward = evaluate_cumulative_reward(rewards)
+# Output: 3.3
+```
+
+
+---
+
+## ⚖️ 3. evaluate_constraint_satisfaction_rate
+
+```python
+evaluate_constraint_satisfaction_rate(satisfied: List[bool]) -> float
+```
+
+#### Description:
+Measures the fraction of time steps where all constraints were satisfied.
+
+#### Arguments:
+
+- satisfied (List[bool]): A list where True means constraints were satisfied at that step.
+
+Returns:
+
+- float: Constraint satisfaction rate (e.g. 0.9 means 90% of steps were constraint-safe).
+
+### Usage Example:
+
+```python
+satisfied = [True, False, True, True]
+rate = evaluate_constraint_satisfaction_rate(satisfied)
+# Output: 0.75
+
+```
+
+
+
+---
+
+## 📊 4. Visualization Hooks (WIP)
+
+Future versions will include utilities for:
+
+- Violation trend plots
+
+- Reward-violation scatter plots
+
+- Safety-performance Pareto frontiers
+
+#### 🧠 Pro Tip
+These metrics are modular, you can log them live during training or compute them post hoc from rollout logs using:
+
+```python
+from evaluations.metrics import (
+    evaluate_safety_violations,
+    evaluate_cumulative_reward,
+    evaluate_constraint_satisfaction_rate,
+)
+
+```
+

saferl_lite-0.1.0/docs/examples/cartpole.md

@@ -0,0 +1,84 @@
+# 🧪 Example: Safe CartPole
+
+This example demonstrates how to train a **constrained DQN agent** on the classic `CartPole-v1` task with constraint-aware wrappers, SHAP-based explainability, and safe policy evaluation.
+
+---
+
+## 🧠 Objective
+
+Balance the pole while **minimizing unsafe actions**, defined by custom constraints (e.g., excessive cart velocity).
+
+---
+
+## 📦 Setup
+
+Make sure you’ve followed the [Installation Guide](../usage/installation.md).
+
+---
+
+## 🔧 1. Import Required Modules
+
+```python
+import gymnasium as gym
+from agents.constrained_dqn import ConstrainedDQNAgent
+from envs.wrappers import ConstraintWrapper
+from evaluations.metrics import evaluate_agent
+from explainability.shap_explainer import SHAPExplainer
+```
+
+---
+
+## 🏗️ 2. Create a Safe Environment
+
+```python
+env = gym.make("CartPole-v1", render_mode=None)
+safe_env = ConstraintWrapper(env, constraint_threshold=0.5)
+
+```
+
+--- 
+
+## 🤖 3. Train a Constrained Agent
+
+```python
+agent = ConstrainedDQNAgent(env=safe_env, budget=1.0)
+agent.train(episodes=10)
+
+
+```
+
+--- 
+
+## 📊 4. Evaluate the Agent
+
+```python
+metrics = evaluate_agent(agent, safe_env, episodes=5)
+
+print("=== Evaluation Metrics ===")
+print("Safe episode rate:", metrics["safe_rate"])
+print("Average violations:", metrics["avg_violations"])
+print("Total regret:", metrics["regret"])
+
+```
+
+--- 
+
+## 🔍 5. Explain a Sample Episode with SHAP
+
+```python
+explainer = SHAPExplainer(agent, safe_env)
+explainer.explain_episode(max_steps=50)
+```
+
+--- 
+
+## ✅ Output Snapshot
+
+```bash
+Episode 3: reward=1.0, penalty=0.0, violations=0.0
+SHAP: ['0.04', '0.01']
+Saliency: ['0.19', '0.05']
+
+```
+
+---