rlshiftlab 0.2.0__tar.gz

rlshiftlab-0.2.0/.github/workflows/ci.yml

@@ -0,0 +1,22 @@
+name: CI
+
+on: [push, pull_request]
+
+jobs:
+  checks:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install development dependencies
+        run: python -m pip install -e ".[dev]"
+      - name: Run tests
+        run: python -m pytest -q
+      - name: Run Ruff
+        run: python -m ruff check .

rlshiftlab-0.2.0/.gitignore

@@ -0,0 +1,8 @@
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+*.egg-info/
+build/
+dist/

rlshiftlab-0.2.0/CHANGELOG.md

@@ -0,0 +1,41 @@
+# Changelog
+
+## 0.2.0 (unreleased)
+
+- Added evaluation of caller-supplied groups of frozen policies, with equally
+  weighted member means, descriptive observed member spread, and pointwise
+  crossed member/episode bootstrap confidence intervals.
+- Added direct left-minus-right comparisons for single policies and groups
+  on complete, paired evaluation episode keys. Group comparisons resample
+  training members independently on each side.
+- Added step-contribution and episode-summary callbacks for custom scalar
+  per-episode metrics, with raw episode records for audit.
+- Added separate group, member, custom-metric, episode, and comparison CSVs;
+  custom metric and group plots; and self-contained HTML reports that can
+  include explicitly selected comparisons.
+- Added a multi-seed CliffWalking example and a small custom-metrics API
+  example. The package continues to evaluate already frozen policies; the
+  examples keep any training code outside the package.
+- Preserved the v0.1 single-policy API and its 14-column CSV, with a v0.1
+  baseline archive and reproducibility manifest in `artifacts/v0.1/`.
+
+Intervals are pointwise and approximate, and observed member spread also
+contains finite evaluation noise. Local packaging and install validation is
+recorded in `WORK_LOG.md`; publication remains a separate human action.
+
+## 0.1.0 (unreleased)
+
+- Added an installable, typed Python package with three controlled shifts:
+  observation noise, action slip, and sticky actions.
+- Added paired, seeded evaluation of frozen callable policies and confidence
+  intervals for mean return, nominal-minus-shifted drop, and optional success
+  rate.
+- Added deterministic tidy CSV export, a text summary, headless degradation
+  plots, and a self-contained HTML report.
+- Added runnable CliffWalking Q-learning/SARSA and CartPole heuristic examples,
+  including generated CSV, HTML, and PNG artifacts.
+- Added a pytest suite and GitHub Actions pytest and Ruff checks on Python
+  3.9 and 3.12.
+- Added the project specification, usage documentation, and MIT license.
+
+Its packaging check was deferred to the v0.2 final validation gate.

rlshiftlab-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 RLShiftLab contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rlshiftlab-0.2.0/PKG-INFO

@@ -0,0 +1,341 @@
+Metadata-Version: 2.5
+Name: rlshiftlab
+Version: 0.2.0
+Summary: Stress-test frozen RL policies under controlled shifts and get degradation curves with confidence intervals.
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.9
+Requires-Dist: gymnasium<2,>=0.29
+Requires-Dist: matplotlib>=3.6
+Requires-Dist: numpy>=1.23
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# RLShiftLab
+
+RLShiftLab 0.2.0 evaluates frozen policies under controlled observation and
+action shifts. A single-policy run produces a degradation curve: shift
+strength on the x axis, mean episode return with a pointwise 95% confidence
+interval on the y axis. Version 0.2 also evaluates groups of already trained
+policies, reports the observed spread across their members, compares policies
+directly on paired evaluation episodes, and records custom per-episode
+metrics. Each shift is varied separately. Reports describe measured
+differences and do not establish their causes.
+
+## Install locally
+
+Requires Python 3.9 or newer. From a checkout of this repository:
+
+```sh
+python -m pip install -e .
+```
+
+For tests and lint checks, use `python -m pip install -e ".[dev]"`, then
+`python -m pytest -q` and `python -m ruff check .`.
+
+## Quickstart
+
+Save this as `quickstart.py` and run `python quickstart.py` from a writable
+directory after installing the package:
+
+```python
+import gymnasium as gym
+from rlshiftlab import ActionSlip, ObservationNoise, Suite
+
+def policy(obs):
+    return int(obs[2] + 0.5 * obs[3] > 0)
+
+suite = Suite(
+    env_fn=lambda: gym.make("CartPole-v1"),
+    shifts=[ObservationNoise(levels=[0.01, 0.05]),
+            ActionSlip(levels=[0.05, 0.1])],
+    n_episodes=20,
+    seed=0,
+    success_fn=lambda ep: ep.truncated,
+)
+results = suite.evaluate({"angle+velocity": policy})
+
+print(results.summary())
+results.to_csv("results.csv")
+results.plot("degradation.png")
+results.to_html("report.html")
+```
+
+`policy` is a callable from observation to action. RLShiftLab does not train
+or alter it. Use a deterministic policy for paired comparisons; if a policy
+has a `reset()` method, RLShiftLab calls it at the start of each episode.
+The suite gives every policy and condition the same sequence of environment
+reset seeds. Shift randomness is also seeded per episode, and reused across
+levels and policies for the same shift. Repeated runs with the same inputs and
+seed produce identical CSV rows.
+
+The nominal row has no shift. For shifted rows, **drop** is the nominal episode
+return minus the shifted return, paired by episode seed; a positive value means
+lower return under the shift. Mean-return and drop intervals use percentile
+bootstrap over episodes. Optional success-rate intervals use the Wilson
+method. The CSV includes mean episode length and, when `success_fn` is given,
+success rate. A policy's behavior can still be stateful or stochastic if its
+callable is written that way; the suite cannot guarantee reproducibility for
+randomness that the policy manages independently. This original
+`Suite.evaluate()` interface and its 14-column `to_csv()` export remain
+compatible with v0.1.
+
+## Shifts and outputs
+
+- `ObservationNoise`: Gaussian noise for Box observations; random replacement
+  for Discrete observations.
+- `ActionSlip`: random replacement of an action before it reaches the environment.
+- `StickyActions`: repeat the previously executed action with a given probability.
+
+Discrete random replacement may select the original value. At replacement
+probability `p` with `n` equally likely values, the observed mismatch rate is
+`p * (1 - 1/n)`. The level is the replacement probability, not a guaranteed
+mismatch rate.
+
+`Results.to_csv()` writes tidy rows; `Results.plot()` returns a Matplotlib
+figure and optionally writes a PNG; `Results.to_html()` writes a self-contained
+report with the plot, setup, summary, and full results table. Plotting uses the
+headless Agg canvas.
+
+## v0.2: groups of frozen policies
+
+Supply a mapping from group name to member ID to an **already trained**
+callable policy. Each member ID should identify a separately trained frozen
+policy, such as a run with its own training seed. Multiple checkpoints from
+one training run are not independent training members. RLShiftLab evaluates
+the supplied members; it does not train them or verify their independence.
+The runnable multi-seed CliffWalking example in `examples/` shows how to
+train the members outside the package and pass their frozen policies to
+`evaluate_groups()`.
+
+For example, after creating four frozen CliffWalking callables in your own
+training code, configure `suite` with that CliffWalking environment and its
+evaluation shifts. The earlier CartPole quickstart suite is for CartPole
+policies and cannot evaluate these CliffWalking tables:
+
+```python
+groups = {
+    "Q-learning": {"train-0": q_policy_0, "train-1": q_policy_1},
+    "SARSA": {"train-0": sarsa_policy_0, "train-1": sarsa_policy_1},
+}
+group_results = suite.evaluate_groups(groups)
+group_results.to_csv("groups.csv")
+group_results.to_member_csv("members.csv")
+group_results.to_episode_csv("episodes.csv")
+group_results.plot("group_degradation.png", band="ci")
+group_results.plot("member_spread.png", band="std")
+
+contrast = group_results.compare("Q-learning", "SARSA", metric="return")
+contrast.to_csv("comparison.csv")
+contrast.plot("comparison.png")
+group_results.to_html("group_report.html", comparisons=[contrast])
+```
+
+The same `Suite` configuration works for both single policies and groups.
+At least two members are required per group, but groups may have different
+member counts. Each member receives the same ordered evaluation reset seeds
+at every condition. The group mean first averages episodes within a member,
+then gives each member equal weight. The reported `member_std`, minimum, and
+maximum describe the observed spread of member means. That spread includes
+finite evaluation noise and is neither a confidence interval nor a pure
+estimate of training variance.
+
+Group confidence intervals use a crossed bootstrap that resamples members
+and evaluation episode keys. They are approximate **pointwise** 95%
+intervals, not simultaneous bands or multiple-comparison-adjusted claims.
+Intervals from only a few members may be unstable. Single-policy success
+rates retain Wilson intervals; group success rates use the crossed bootstrap.
+The [pigeonhole bootstrap paper](https://arxiv.org/abs/0712.1111) informs the
+two-axis resampling design, and [Agarwal et al.'s RL evaluation
+paper](https://arxiv.org/abs/2108.13264) motivates careful reporting across
+trained runs. Neither source guarantees exact coverage for a particular
+RLShiftLab experiment.
+
+## Direct comparisons and custom metrics
+
+Call `Results.compare(left, right, metric="return")` for two individual
+policies, or `GroupResults.compare(...)` for two groups. Every nominal and
+shifted comparison uses the same evaluation episode keys on both sides;
+missing or duplicate keys are errors. The estimate is **left minus right**.
+A positive value means the left side has a larger measured metric, which is
+not always better (for example, more falls or longer episodes). Single-policy
+comparisons bootstrap the paired episode differences. Group comparisons
+resample members independently on each side while sharing evaluation episode
+keys; equal member ID strings do not assert matched training runs. Each
+contrast has its own pointwise 95% interval and can be saved as CSV, a
+zero-reference plot, or standalone HTML. Pass a list to `to_html(...,
+comparisons=[contrast])` to include chosen comparisons in an evaluation
+report.
+
+Pass `step_metrics` and `episode_metrics` to `Suite` to collect scalar metrics
+in both single and group runs. Each step callback contributes a finite
+number; RLShiftLab sums it over the episode. For the configured CliffWalking
+environment, a reward of -100 marks a fall, so this metric counts falls:
+
+```python
+suite = Suite(
+    env_fn=make_cliffwalking_env,
+    shifts=[ActionSlip([0.1, 0.2])],
+    n_episodes=100,
+    seed=1000,
+    step_metrics={"cliff_falls": lambda t: float(t.reward == -100.0)},
+    episode_metrics={"return_per_step": lambda ep: ep.episode_return / ep.length},
+)
+```
+
+Here `make_cliffwalking_env` is your environment factory. The -100 rule is
+specific to that environment, not a generic fall detector. Step callbacks
+receive a `Transition` with policy-visible observations, requested action,
+next observation, reward, termination/truncation flags, step index, episode
+seed, and step `info`. Its requested action may differ from the action
+actually executed after action slip or sticky actions. Episode callbacks
+receive a copy of the final `EpisodeResult`, including step metric totals,
+length, return, flags, and final step `info`. Each callback must return one
+finite numeric scalar; no full trajectory is retained. The resulting
+`metric_rows` and `to_metrics_csv()` summarize each metric, while
+`to_episode_csv()` includes scalar values under `metric:<name>` columns.
+
+The raw episode CSV records policy/group, member, shift, level, episode index,
+reset seed, return, length, flags, optional success, and custom scalars. It
+does not serialize arbitrary observations or `info` objects. These records
+make pairing and reported aggregates auditable.
+
+Repeated runs with the same inputs, software versions, suite seed, and pure
+deterministic or consistently resettable policy and metric callbacks produce
+the same CSV bytes. Suite seeds control Gymnasium resets and shift randomness;
+they do not control randomness inside arbitrary user policies or callbacks.
+
+| Result object | Main export | Additional exports |
+| --- | --- | --- |
+| `Results` | `to_csv(path)` keeps the v0.1 14 columns | `to_metrics_csv(path)`, `to_episode_csv(path)`, `plot(metric=...)`, `to_html(path, comparisons=...)` |
+| `GroupResults` | `to_csv(path)` writes aggregate rows | `to_member_csv(path)`, `to_metrics_csv(path)`, `to_episode_csv(path)`, `plot(metric=..., band="ci" or "std")`, `to_html(path, comparisons=...)` |
+| `ComparisonResults` | `to_csv(path)` writes left-minus-right rows | `plot(path)`, `to_html(path)`, `summary()` |
+
+## v0.2 examples
+
+Run either example from the repository root:
+
+```sh
+python examples/cliffwalking_multiseed.py
+python examples/custom_metrics.py
+```
+
+The [multi-seed CliffWalking example](examples/cliffwalking_multiseed.py)
+trains its Q-learning and SARSA tables in the example code, then evaluates
+them as frozen members. It writes separate group, member, metric, episode,
+and paired-comparison files, plus plots and HTML, under
+[`artifacts/v0.2/cliffwalking/`](artifacts/v0.2/cliffwalking/).
+The [small custom-metrics example](examples/custom_metrics.py) uses fixed toy
+policies to demonstrate step and episode metric callbacks and comparison
+exports under [`artifacts/v0.2/custom_metrics/`](artifacts/v0.2/custom_metrics/).
+Its outputs illustrate the API, not a research benchmark. The package does
+not perform the training in either case.
+
+### Measured multi-seed CliffWalking run
+
+The generated [v0.2 group results](artifacts/v0.2/cliffwalking/group.csv)
+used deterministic CliffWalking-v1, ten Q-learning training seeds (0–9) and
+ten SARSA training seeds (100–109), with 500 training episodes per member.
+Each frozen member was evaluated on the same 100 reset seeds per condition
+for one nominal condition and eight shift levels (18,000 raw episodes).
+[run.json](artifacts/v0.2/cliffwalking/run.json) records the configuration,
+software versions, and source/output hashes.
+
+Nominal mean return across members was -13.000 for Q-learning (observed
+member SD 0.000) and -35.300 for SARSA (SD 57.870). In the
+[member results](artifacts/v0.2/cliffwalking/member.csv), SARSA training seed
+100 had nominal return -200, length 200, and no success; each of the other
+nine SARSA seeds had return -17, length 17, and success. These member scores
+underlie the reported nominal spread; they do not identify why that one
+training outcome occurred.
+
+At action slip 0.3, group mean returns were -149.928 for Q-learning and
+-41.324 for SARSA. The direct Q-learning-minus-SARSA return difference was
+-108.604, with a pointwise 95% CI of [-138.084, -79.740]. The corresponding
+[cliff-fall difference](artifacts/v0.2/cliffwalking/falls_comparison.csv) was
++1.066 falls per episode [0.808, 1.329], so Q-learning had more recorded
+falls under that shift in this configured run. These are measured differences
+for these trained tables, environment settings, and evaluation seeds; they
+are not causal explanations or a general ranking of the algorithms.
+
+![v0.2 multi-seed CliffWalking group return curves](artifacts/v0.2/cliffwalking/mean_return_ci.png)
+
+The [return comparison CSV](artifacts/v0.2/cliffwalking/return_comparison.csv),
+[episode audit CSV](artifacts/v0.2/cliffwalking/episodes.csv), and
+[standalone report](artifacts/v0.2/cliffwalking/report.html) provide the
+condition-level and per-episode evidence. A repeated execution with the
+same configuration produced byte-identical hashes for all 13 generated
+outputs. That repeatability applies to this deterministic example and
+software setup, not to arbitrary user-supplied stochastic callbacks.
+
+The [toy custom-metrics outputs](artifacts/v0.2/custom_metrics/) use two
+fixed policies and 20 paired evaluation episodes per condition. At action
+slip 0.5, the measured always-zero-minus-always-one return difference was
+0.85 [0.5, 1.2], while its unsafe-event difference was -0.5 [-0.7, -0.3].
+This demonstrates the metric and comparison APIs; it is not a research
+benchmark.
+
+## Historical v0.1 examples and measured results
+
+The original v0.1 repository contains a CliffWalking Q-learning versus SARSA example and a
+CartPole comparison of two hand-coded policies in `examples/`. Training code
+for CliffWalking stays in the example; the RLShiftLab package evaluates only
+the resulting frozen policies. The CliffWalking example uses the registered
+`CliffWalking-v0` when available, or `CliffWalking-v1` with
+`is_slippery=False` on newer Gymnasium versions. See the [Gymnasium
+CliffWalking documentation](https://gymnasium.farama.org/v1.3.0/environments/toy_text/cliff_walking/)
+for the v1 environment and its optional slippery mode.
+
+Run the examples from the repository root:
+
+```sh
+python examples/cliffwalking_q_vs_sarsa.py
+python examples/cartpole_heuristic.py
+```
+
+The CliffWalking run writes [results.csv](results.csv),
+[report.html](report.html), and the figure below. The CartPole run writes
+[results.csv](artifacts/cartpole/results.csv),
+[report.html](artifacts/cartpole/report.html), and
+[degradation.png](artifacts/cartpole/degradation.png) under
+`artifacts/cartpole/`.
+
+![Measured CliffWalking degradation curves](docs/cliffwalking_degradation.png)
+
+In the generated CliffWalking results (200 evaluation episodes per condition),
+nominal mean return was -13.000 for Q-learning and -19.000 for SARSA. At action
+slip 0.3, mean return was -198.250 for Q-learning and -48.815 for SARSA. The
+paired drops from nominal were 185.250 (95% CI [154.945, 220.342]) and 29.815
+(95% CI [23.120, 37.440]), respectively. This run measured a larger drop for
+Q-learning at that shift level. The example trained one table for each
+algorithm with 500 episodes before evaluation; it does not measure variation
+across training seeds.
+
+In the generated CartPole results (100 evaluation episodes per condition),
+nominal mean return was 500.000 for the angle+velocity heuristic and 41.040
+for the angle-only heuristic. At observation noise 0.2, their mean returns were
+267.400 and 38.310, respectively. These values describe the two particular
+frozen policies and the configured evaluation seeds.
+
+## What v0.2 does not do
+
+Version 0.2 does not train or adapt policies inside the package, measure
+recovery, apply reward shifts or delays, change environment dynamics, support
+vectorized environments or a CLI, or accept Dict, MultiDiscrete, or Tuple
+observation spaces. A group member ID is caller-provided provenance, not
+proof that training runs were independent. Reports do not provide causal
+explanations or multiplicity-adjusted confidence intervals.
+
+## Roadmap after v0.2
+
+Future work may add learners and adaptation with recovery metrics, reward
+shifts and delays, environment-specific dynamics hooks, and Dict and
+MultiDiscrete spaces.
+
+## License
+
+MIT; see [LICENSE](LICENSE).