multireward-grpo 0.1.0__tar.gz

multireward_grpo-0.1.0/.gitignore

@@ -0,0 +1,33 @@
+# Secrets — never commit
+.env
+.env.local
+*.pem
+*.key
+
+# Python / uv
+__pycache__/
+*.py[co]
+.venv/
+*.egg-info/
+.pytest_cache/
+
+# Build artifacts
+/dist/
+/build/
+
+# Generated data — NOT published (datasets/models live on Hugging Face; see README)
+data/
+
+# Editor / OS
+.DS_Store
+*~
+.vscode/
+.idea/
+*.swp
+
+# WSL metadata
+*Zone.Identifier*
+
+# Runtime artifacts
+*.lock.tmp
+/tmp/id_ed25519_runpod

multireward_grpo-0.1.0/CLAUDE.md

@@ -0,0 +1,117 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this repository is
+
+This is a **research package**, not an application. It develops a finite-sample,
+correlation-aware *theory* of decoupled and conditioned multi-reward GRPO
+advantage estimators, and ships a verification harness that checks every
+theoretical claim against simulation and real-LLM data. The deliverable is the
+paper (the `*.md` files) backed by reproducible code (`scripts/`) and released
+HuggingFace artifacts.
+
+The prose and the code are tightly coupled: each script verifies a specific
+named claim, writes a specific figure, and the `README.md` "Script → figure →
+claim map" table is the source of truth for that mapping. **When you change a
+script's numerical output, update the claim's status in `README.md` and the
+relevant prose file** (`empirical-section.md`, `proposed-solutions.md`,
+`proofs.md`). Several first-draft claims were *falsified* by the harness and the
+prose was corrected — that falsification record is a feature, not a bug; preserve
+it.
+
+### Document map (read in this order)
+- `literature-review.md` — related work, the gap, citation positioning
+- `problem-statement.md` — **canonical setup/notation referenced by all other files**; defines AN/NA, conditioning, the 4 questions Q1–Q4
+- `proposed-solutions.md` — the results: Prop 1 (background, = MO-GRPO), Prop 2, Thm 3, Prop 4/4′
+- `proofs.md` — full proofs (appendix)
+- `empirical-section.md` — the 3-tier verification story
+- `huggingface_assets.md` — released datasets (3) + fine-tuned models (3) under HF user `eagle0504`
+
+## Core domain concepts (needed to read any script)
+
+- **AN — Aggregate-then-Normalize**: scalarize rewards `s = wᵀr`, then group-normalize. The GRPO baseline. Suffers Prop 1 (high-variance channel dominates) and Prop 2 (resolution collapse).
+- **NA — Normalize-then-Aggregate**: per-channel standardize, then weighted-sum. The "decoupled" estimator from MO-GRPO/GDPO — **the object analyzed**. Restores weight-proportional influence.
+- These two orderings are implemented canonically in `gdpo_adapter.py::compute_multireward_advantage(rewards, w, mode)` and again, per-pipeline, as `advantage_AN` / `advantage_NA` in `scripts/llm_analysis.py` and `compute_advantages` in `scripts/grpo_train.py`. Keep all three consistent.
+- **Conditioning** ("reward b counts only if a passes"): three impls compared — `none`, `zero_fill` (recommended, preserves U-statistic structure), `subgroup` (degenerates when <2 rollouts pass the gate). See `gdpo_adapter.py::apply_conditioning`.
+- **Headline result (Thm 3)**: MSE `= (τ²/m)·wᵀCw + O(m⁻²)` — reward correlation `C` sets the achievable MSE floor.
+- **"Money plot"**: predicted-vs-realized gradient-noise MSE scatter, computed *without* an oracle gradient by using the seed-mean as a proxy oracle (`measure_gradient_mse` / `realized_mse`).
+
+## Environment & common commands
+
+Uses **uv** (not pip/conda). Python pinned to `>=3.12,<3.13`.
+
+```bash
+uv sync                      # base deps (CPU: numpy/scipy/matplotlib/datasets/hf)
+uv sync --extra llm          # adds torch (cu124 wheel index)/transformers/peft — GPU host only
+uv run scripts/<name>.py     # always run scripts through uv
+```
+
+There is **no test suite, linter, or build step**. Verification = running the
+harness scripts and checking their printed "sim-vs-theory" lines and PNG output.
+The closest thing to a unit test is the mock-mode gate (see below).
+
+### Synthetic harness (CPU, seconds–1 min each; writes PNGs to CWD)
+```bash
+uv run scripts/grpo_harness.py    # Prop 1, 2; Thm 3 core + U-statistic; budget m*  (fig1*, fig2*)
+uv run scripts/grpo_thm4.py       # Prop 4 / 4' conditioning  (figT*)
+uv run scripts/grpo_selfnorm.py   # Thm 3 self-normalized lift  (figS*)
+uv run scripts/grpo_mstar.py      # group size vs correlation  (figM1 — a FALSIFIED claim)
+uv run scripts/grpo_prop2.py      # Prop 2 resolution lattice  (figP2)
+```
+
+### LLM validation pipeline
+```bash
+# Mock mode = the GATING experiment. CPU, ~20s. If it doesn't reproduce Thm 3 to
+# ~1% on synthetic Bernoulli rewards, there's a bug in the analysis pipeline —
+# do NOT spend GPU money until mock passes.
+uv run scripts/llm_validate.py --mode mock
+
+# Real GSM8K (needs GPU + --extra llm). --subsample-from-max is ~4x cheaper.
+uv run scripts/llm_validate.py --mode gsm8k --model Qwen/Qwen2.5-1.5B-Instruct \
+    --n-prompts 100 --K 8 --m-grid 4 8 16 32 --subsample-from-max
+```
+
+### RunPod orchestration (spawns H100 pod, runs, retrieves, terminates)
+Requires `RUNPOD_API_KEY` in `.env` (copy from `.env.example`) and an SSH key at
+`~/.ssh/id_ed25519`. See `runpod_user_guide.md` and `validation_plan.md` for the
+full GPU budget and debugging guide.
+```bash
+uv run scripts/runpod_launch.py --n-prompts 50 --K 8 --m-grid 4 8 16 32   # Thm 3 GSM8K run
+uv run scripts/runpod_train.py --mode both --n-steps 200                  # Tier-3 GRPO fine-tune
+uv run scripts/runpod_train_multiseed.py                                  # multi-seed training curves
+```
+
+## Architecture of the script layer
+
+Scripts fall into four families (prefix = family):
+
+- **`grpo_*` — synthetic harness.** Self-contained, NumPy-only. Each draws rewards with a known correlation matrix, computes AN/NA advantages, and overlays the closed-form theory on a PNG. `grpo_harness.py` is the main one (Prop 1, Prop 2, Thm 3, U-statistic bias/variance, budget-optimal `m*`). `grpo_train.py`/`grpo_eval.py` are the GPU fine-tuning + eval (Tier 3, fintech domain).
+
+- **`llm_*` — real-LLM validation pipeline.** Three-stage, importable modules:
+  1. `llm_generation.py` — `Backend` protocol with `MockBackend` (synthetic, CPU) and `QwenBackend` (real, GPU). `run_corpus` → `pack_for_analysis` produces a `(P, K, m, R)` tensor (prompts × seeds × rollouts × reward channels).
+  2. `llm_rewards.py` — the R reward channels for GSM8K (correctness/length/format).
+  3. `llm_analysis.py` — `analyze()` returns a `Thm3Result`; produces money-plot scatters and influence-law numbers.
+  `llm_validate.py` is the entry point orchestrating all three; `llm_prop4_real.py` does the Prop 4 γ-sweep on saved rewards.
+
+- **`runpod_*` — cloud GPU orchestration.** `runpod_lib.py` is the shared library (pod create/wait/ssh/rsync/delete via the RunPod REST API). `runpod_launch.py` (Thm 3), `runpod_train.py` / `runpod_train_multiseed.py` (Tier 3 training), `runpod_fintech.py` are thin task-specific drivers. These rsync the repo to a pod, run a `uv` command remotely, rsync results back, then terminate the pod. **Note `runpod_launch.py` duplicates much of `runpod_lib.py` rather than importing it** — keep them in sync if editing the pod lifecycle logic.
+
+- **`hf_*` / `fintech_*` — asset publishing + the fintech domain.** `fintech_scenarios.py`/`fintech_rewards.py`/`fintech_generate.py` define the synthetic fintech-customer-comms domain (Tier 3); `hf_push_*.py` / `hf_add_parquet.py` publish datasets and LoRA adapters to the `eagle0504` HuggingFace account.
+
+`plot_grpo_training.py` / `plot_grpo_training_multiseed.py` render the training
+curves produced by the Tier-3 fine-tune (`grpo_train.py` / `runpod_train*.py`);
+`hf_enrich_gsm8k_parquet.py` post-processes a released GSM8K parquet before
+re-pushing it. These are post-hoc plotting/publishing utilities, not part of the
+verification harness.
+
+`gdpo_adapter.py` (repo root) is a **non-runnable stub** documenting the drop-in
+contract for wiring AN/NA + conditioning into an external verl/TRL/GDPO training
+loop — it is reference, not part of any pipeline.
+
+## Conventions specific to this repo
+
+- **`.txt` and `:Zone.Identifier` files**: `data/pdfs/*.txt` are extracted text of the source papers (read these instead of the PDFs). `*:Zone.Identifier` files are Windows/WSL download-provenance cruft — ignore them.
+- **Figures**: synthetic-harness scripts write PNGs to the **current working directory** (`FIG = "."`), so run them from `scripts/` or `figures/` as intended; the LLM pipeline writes to `figures/` via `--out-dir`. `*_mock*` figures are committed CPU-reproducible outputs; bare-name figures come from GPU runs.
+- **Reproducibility**: synthetic scripts use fixed seeds (`np.random.default_rng(7)` etc.). Changing a seed can shift the last printed digit — don't "fix" a claim by reseeding.
+- **Falsified claims are documented on purpose** in `README.md` (§"Verification status") and the prose. If your edit changes a result, update both the table row and the narrative; never silently flip a "FALSIFIED" to "verified".
+- **Not a git repository**: this working copy is not under version control, so there is no commit/branch/PR workflow to follow — edit files in place. Secrets live in `.env` (RunPod + HF tokens); `.env.example` is the template.

multireward_grpo-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yiqiao Yin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

multireward_grpo-0.1.0/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.4
+Name: multireward-grpo
+Version: 0.1.0
+Summary: Decoupled & conditioned multi-reward GRPO advantage estimators, a generalized trainer, and the Theorem-3 verification harness from the paper 'When and Why Decoupling and Conditioning Beat Reweighting in Multi-Reward GRPO'.
+Project-URL: Homepage, https://github.com/yiqiao-yin/multireward-grpo
+Project-URL: Repository, https://github.com/yiqiao-yin/multireward-grpo
+Project-URL: Hugging Face, https://huggingface.co/eagle0504
+Author-email: Yiqiao Yin <eagle0504@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: grpo,llm,multi-reward,reinforcement-learning,rlhf,rlvr,u-statistic
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Requires-Dist: scipy>=1.10
+Provides-Extra: data
+Requires-Dist: datasets>=3.0; extra == 'data'
+Requires-Dist: huggingface-hub>=0.24; extra == 'data'
+Requires-Dist: pandas>=2.0; extra == 'data'
+Requires-Dist: pyarrow>=14.0; extra == 'data'
+Provides-Extra: llm
+Requires-Dist: accelerate>=1.1; extra == 'llm'
+Requires-Dist: peft>=0.13; extra == 'llm'
+Requires-Dist: protobuf>=5.28; extra == 'llm'
+Requires-Dist: sentencepiece>=0.2; extra == 'llm'
+Requires-Dist: torch<2.10,>=2.5; extra == 'llm'
+Requires-Dist: transformers>=4.46; extra == 'llm'
+Provides-Extra: research
+Requires-Dist: datasets>=3.0; extra == 'research'
+Requires-Dist: huggingface-hub>=0.24; extra == 'research'
+Requires-Dist: matplotlib>=3.7; extra == 'research'
+Requires-Dist: pandas>=2.0; extra == 'research'
+Requires-Dist: pyarrow>=14.0; extra == 'research'
+Requires-Dist: requests>=2.28; extra == 'research'
+Provides-Extra: runpod
+Requires-Dist: requests>=2.28; extra == 'runpod'
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.7; extra == 'viz'
+Description-Content-Type: text/markdown
+
+# multireward-grpo
+
+**Decoupled & conditioned multi-reward GRPO** — advantage estimators, a
+generalized trainer, and the Theorem-3 verification harness from the paper
+*"When and Why Decoupling and Conditioning Beat Reweighting in Multi-Reward
+GRPO: A U-Statistic Treatment."*
+
+This package modularizes the experiment code so you can train your own
+multi-reward GRPO models, verify the correlation-aware MSE law on your own
+rollouts, and (optionally) run the whole thing on a cloud GPU.
+
+```bash
+pip install multireward-grpo            # core (numpy/scipy): advantage + analysis
+pip install "multireward-grpo[llm]"     # + torch/transformers/peft: training & real generation
+pip install "multireward-grpo[viz]"     # + matplotlib: the money-plot figure
+pip install "multireward-grpo[data]"    # + datasets/hf-hub: dataset loaders & model push
+pip install "multireward-grpo[runpod]"  # + requests: cloud GPU orchestration
+```
+
+## The two orderings
+
+Given a group of `m` rollouts each scored on `R` reward channels with weights `w`:
+
+- **AN — Aggregate-then-Normalize** (classic GRPO baseline): scalarize `s = wᵀr`,
+  then group-normalize. The high-variance channel dominates (Prop 1) and the
+  advantage resolution collapses under heterogeneous scales (Prop 2).
+- **NA — Normalize-then-Aggregate** (the decoupled estimator, = MO-GRPO/GDPO):
+  group-normalize each channel, then take the weighted sum. Restores
+  weight-proportional influence and gives the correlation-aware gradient-MSE
+  floor `(τ²/m)·wᵀCw` (Theorem 3).
+
+```python
+import numpy as np
+from multireward_grpo import compute_advantage
+
+rewards = np.array([[1.0, 0.3, 1.0],   # (m=4 rollouts, R=3 channels)
+                    [0.0, 0.9, 1.0],
+                    [1.0, 0.1, 0.0],
+                    [0.0, 0.5, 1.0]])
+w = np.array([1.0, 1.0, 0.5])
+A_na = compute_advantage(rewards, w, mode="na")   # recommended
+A_an = compute_advantage(rewards, w, mode="an")   # GRPO baseline
+```
+
+## Train your own model
+
+Bring **your own prompts** and **your own reward function**; the trainer runs
+group-relative policy optimization with a KL anchor and saves a LoRA adapter.
+
+```python
+from multireward_grpo import GRPOConfig, GRPOTrainer
+
+# prompts: list of strings, chat-message lists, or dicts with metadata
+prompts = ["Write a polite refusal to a refund demand.", ...]
+
+# reward_fn(completion, prompt) -> R channel scores (len == len(weights))
+def reward_fn(completion, prompt):
+    return (compliance(completion), politeness(completion), action(completion))
+
+cfg = GRPOConfig(model="Qwen/Qwen2.5-1.5B-Instruct",
+                 mode="na", weights=(1.0, 1.0, 0.5), n_steps=200, m=8)
+history = GRPOTrainer(cfg, reward_fn, prompts).train()
+```
+
+### Data format
+
+| Input | Shape / type | Notes |
+|---|---|---|
+| `prompts` | `list[str \| list[dict] \| dict]` | a string (user msg), chat messages `[{"role","content"}]`, or `{"prompt": ..., "gold": ...}` with metadata passed through to the reward fn |
+| `reward_fn(completion, prompt)` | returns `Sequence[float]` of length `R` | one score per reward channel; **channel 0 is the gate** for conditioning |
+| `weights` | `tuple[float, ...]` length `R` | objective weights `w` |
+| `mode` | `"na" \| "an" \| "single"` | `na` is the paper's recommendation |
+
+Reward tensors for the analysis tools use shape **`(P, K, m, R)`** = prompts ×
+seeds × rollouts × reward channels.
+
+### Ready-made examples
+
+```python
+from multireward_grpo.examples import FintechRewardFunction, make_fintech_prompts
+from multireward_grpo import GRPOConfig, GRPOTrainer
+
+prompts = make_fintech_prompts(400, seed=0)
+cfg = GRPOConfig(mode="na", weights=(1.0, 1.0, 0.5))
+GRPOTrainer(cfg, FintechRewardFunction(), prompts).train()
+```
+
+`multireward_grpo.examples.gsm8k` provides GSM8K loaders paired with
+`multireward_grpo.rewards.MathRewardFunction` (correctness / length / format).
+
+## Verify Theorem 3 on your rollouts
+
+```python
+from multireward_grpo import analyze, summary_print
+from multireward_grpo.generation import MockBackend, run_corpus, pack_for_analysis
+import numpy as np
+
+C = np.array([[1, 0.5, 0], [0.5, 1, 0], [0, 0, 1]])   # reward correlation
+corpus = run_corpus(MockBackend(C=C), [(f"p{i}", "0") for i in range(40)],
+                    m_grid=[8], K_seeds=200)
+rewards = pack_for_analysis(corpus, m=8)               # (P, K, m, R)
+result = analyze(rewards, w=np.array([1.0, 1.0, 0.5]))
+summary_print(result)
+```
+
+Or from the shell:
+
+```bash
+multireward-grpo thm3-check --rho 0.5      # CPU, no GPU
+multireward-grpo train --mode na --n-steps 50   # needs [llm] + GPU
+```
+
+## Run on a cloud GPU (RunPod)
+
+```python
+from multireward_grpo.runpod import RunPodClient
+client = RunPodClient()  # reads RUNPOD_API_KEY from env or .env
+client.run_command('pip install "multireward-grpo[llm]" && multireward-grpo train --mode na',
+                   wall_clock_cap=1800)
+```
+
+## Released artifacts (Hugging Face)
+
+Datasets and fine-tuned models from the paper live under the
+[`eagle0504`](https://huggingface.co/eagle0504) namespace:
+
+**Datasets**
+- [multireward-grpo-gsm8k-rewards](https://huggingface.co/datasets/eagle0504/multireward-grpo-gsm8k-rewards) — 76,800 Qwen2.5-1.5B GSM8K rollouts (rewards + chains-of-thought)
+- [multireward-grpo-gsm8k-rewards-qwen2.5-7b](https://huggingface.co/datasets/eagle0504/multireward-grpo-gsm8k-rewards-qwen2.5-7b) — 25,600 Qwen2.5-7B rollouts
+- [multireward-grpo-fintech-customer-comms](https://huggingface.co/datasets/eagle0504/multireward-grpo-fintech-customer-comms) — 2,400 fintech conversations
+
+**Models** (LoRA adapters for Qwen2.5-1.5B-Instruct)
+- [multireward-grpo-fintech-na-qwen2.5-1.5b](https://huggingface.co/eagle0504/multireward-grpo-fintech-na-qwen2.5-1.5b) — NA (paper's recommendation)
+- [multireward-grpo-fintech-an-qwen2.5-1.5b](https://huggingface.co/eagle0504/multireward-grpo-fintech-an-qwen2.5-1.5b) — AN baseline
+- [multireward-grpo-fintech-single-qwen2.5-1.5b](https://huggingface.co/eagle0504/multireward-grpo-fintech-single-qwen2.5-1.5b) — single-reward ablation
+
+## Citation
+
+If you use this package, please cite the paper (see the
+[GitHub repository](https://github.com/yiqiao-yin/multireward-grpo) for the
+current BibTeX entry).
+
+## License
+
+MIT

multireward_grpo-0.1.0/README.md

@@ -0,0 +1,107 @@
+# Conditioned Multi-Reward Advantage Estimation — research package
+
+A literature gap, a problem statement with candidate theorems, fully written
+proofs, and a simulation+LLM harness that verifies every claim.
+
+## Paper skeleton (read in this order)
+
+```
+literature-review.md         Three strands of related work, the gap, positioning, citation checklist.
+problem-statement.md         The gap (summary), canonical formal setup/notation, the 4 questions.
+proposed-solutions.md        The 4 results (Prop 1, 2; Thm 3; Prop 4/4') with intuition + what each solves.
+proofs.md                    [APPENDIX] Full proofs, cross-referenced to figures.
+empirical-section.md         3-tier verification: synthetic harness, real GSM8K (2 scales), GRPO fine-tuning.
+huggingface_assets.md        Released datasets (3) + fine-tuned models (3), with schemas + URLs.
+```
+
+## Supporting files
+
+```
+runpod_user_guide.md         How to reproduce the RunPod runs end-to-end.
+validation_plan.md           LLM validation pipeline details + GPU budget + debugging guide.
+gdpo_reproduction_plan.md    Anchor reproduction scaffold (DAPO fallback included).
+gdpo_adapter.py              Framework-agnostic AN/NA + conditioning switch stub.
+.env.example                 Template for RUNPOD_API_KEY + HF_TOKEN.
+scripts/                     Synthetic harness (grpo_*.py) + LLM pipeline (llm_*.py) + RunPod orchestration (runpod_*.py).
+figures/                     Synthetic-harness PNGs + LLM money plots + GSM8K/fintech/training outputs.
+```
+
+## The gap (one line)
+
+Multi-reward RLVR defaults to GRPO; GDPO (arXiv:2601.05242) showed scalarize-then-normalize
+collapses reward combinations and that conditioning helps — but offers no theory. The
+single-reward U-statistic theory (arXiv:2603.01162) and shrinkage-baseline work
+(arXiv:2511.03710) exist but have not been extended to the multi-reward / conditioned setting.
+This package develops and tests that extension.
+
+## How to run
+
+```bash
+# environment (uv-based; pyproject.toml already in repo)
+uv sync
+
+# synthetic harness — each prints sim-vs-theory checks and writes PNGs
+uv run scripts/grpo_harness.py    # Prop 1, 2; Thm 3 core + U-statistic; budget m*
+uv run scripts/grpo_thm4.py       # Prop 4 / 4' conditioning
+uv run scripts/grpo_selfnorm.py   # Thm 3 self-normalized lift
+uv run scripts/grpo_mstar.py      # Thm 3 group-size vs correlation
+uv run scripts/grpo_prop2.py      # Prop 2 resolution (enumeration)
+
+# LLM validation pipeline
+uv run scripts/llm_validate.py --mode mock   # CPU, ~20 sec — gating experiment
+uv run scripts/llm_validate.py --mode gsm8k --model Qwen/Qwen2.5-1.5B-Instruct \
+    --n-prompts 100 --K 8 --m-grid 4 8 16 32 --subsample-from-max   # requires GPU
+
+# RunPod orchestration (spawn H100 pod, run, retrieve results, terminate)
+# requires RUNPOD_API_KEY in .env and an SSH key at ~/.ssh/id_ed25519
+uv run scripts/runpod_launch.py --n-prompts 50 --K 8 --m-grid 4 8 16 32
+```
+Runtime: synthetic harness is seconds to ~1 min each on a laptop; mock LLM
+validation is ~20 sec on CPU; real GSM8K experiment is ~1.5 hr on a single
+RTX 4090. See `validation_plan.md` for the full GPU budget and debug guide.
+
+## Script -> figure -> claim map
+
+| Script | Figure | Claim tested | Result |
+|---|---|---|---|
+| grpo_harness.py | fig1a_wCw_scaling | Thm 3: MSE $= (\tau^2/m)\, w^\top Cw$ at $\tau^2=1$ | verified, 3 digits |
+| grpo_harness.py | fig1b_influence_law | Prop 1: AN influence $\propto w_\ell\sigma_\ell$, NA $\propto w_\ell$ | verified (het. scales) |
+| grpo_harness.py | fig1c_coeff_vs_rho | Thm 3: $m\cdot$MSE traces $w^\top Cw$ | verified |
+| grpo_harness.py | fig2a_bias | U-statistic self-centering bias $(m{-}1)/m$ | verified |
+| grpo_harness.py | fig2b_variance | Hoeffding two-term variance $a/m+b/m^2$ | verified (inverse-variance-weighted fit; pure $a/m$ is rejected) |
+| grpo_harness.py | fig2c_groupsize_law | budget-optimal $m^\star$ grows as $N^{1/3}$ (single reward) | verified by log-log fit over 6 budgets (exponent printed by harness) |
+| grpo_thm4.py | figT1_crossover_pa | Prop 4': non-monotone subgroup MSE (degenerates structurally at low $p_a$ but wins on raw MSE there because the format channel is small); subgroup loses in mid-$p_a$; zero-fill best on average | verified |
+| grpo_thm4.py | figT2_bias_law | Prop 4: bias $=p_b(1{-}p_a)[\gamma(1{-}p_b)-\alpha_c p_a]$, sign-changing | verified, $2\times10^{-3}$ |
+| grpo_thm4.py | figT3_phase_diagram | Prop 4': condition vs not over $(p_a,\gamma)$; boundary = bias-zero curve | verified |
+| grpo_selfnorm.py | figS1_selfnorm_mse | self-norm Thm 3: $m\cdot$MSE $=\mathbb{E}[w^\top\hat Cw]\to w^\top Cw$ | verified |
+| grpo_selfnorm.py | figS2_selfnorm_vs_rho | self-norm structure preserved over $\rho$ | verified |
+| grpo_selfnorm.py | figS3_selfnorm_bias | self-norm bias is $O(1/m)$ with distribution-dependent coefficient | Gaussian $\to -3\theta/4$ (exact $\Gamma$-formula); Bernoulli($\tfrac12$) $\to -\theta/2$; original $+\theta/4$ approx FALSIFIED |
+| grpo_mstar.py | figM1_mstar_vs_rho | Thm 3 group size vs reward correlation | **FALSIFIED** $\sqrt{w^\top Cw}$; $m^\star$ flat in $\rho$ |
+| grpo_prop2.py | figP2_resolution | Prop 2: NA product lattice $L^R$ vs AN sum lattice | verified (het. scales only) |
+| llm_validate.py --mode mock | llm_money_scatter_mock_m{m}, llm_mse_vs_m_mock | Thm 3 + Prop 1 on synthetic LLM-shaped data | NA realized / pred = 0.94–1.02 across m=4–32; AN drifts to 0.62–0.75 (Prop 1) |
+| llm_validate.py --mode gsm8k | llm_money_scatter_gsm8k_m{m}, llm_mse_vs_m_gsm8k | Same on real LLM generations | pending GPU run |
+
+## Verification status (summary)
+
+All claims are simulation-backed. Four first-draft claims were caught wrong and corrected:
+1. **Prop 4 bias** — original dropped a cross-term; corrected form is sign-changing.
+2. **Prop 4' variance penalty** — the claimed $1/p_a$ inflation does not appear. The real
+   structural pathology is subgroup-baseline degeneracy at low pass-rate (figT1 gray curve), but
+   the subgroup MSE profile is **non-monotone** in $p_a$: subgroup loses to unconditioned only in
+   the mid-range $p_a\in[0.29, 0.74]$, not below a single threshold. Zero-fill conditioning is the
+   recommended estimator (wins ~75% of the $(p_a,\gamma)$ plane).
+3. **Thm 3 group-size scaling** — $m^\star \propto \sqrt{w^\top Cw}$ is false; $m^\star$ is set by
+   prompt heterogeneity and bias, independent of reward correlation. Correlation sets the MSE *floor*.
+4. **Self-norm bias coefficient** — original $+\theta/4$ independence approximation is false. The
+   leading $1/m$ coefficient is distribution-dependent: Gaussian gives $-3\theta/4$ (exact via the
+   $\Gamma$-formula); Bernoulli($\tfrac12$) gives $-\theta/2$ empirically.
+
+Headline that survives: reward correlation $w^\top Cw$ governs the achievable MSE floor of
+multi-reward GRPO (positively correlated objectives are fundamentally harder); decoupled
+normalization restores weight-proportional influence and advantage resolution under heterogeneous
+scales; the self-normalized estimator's sample-std bias is $O(1/m)$ with a
+distribution-dependent leading coefficient ($-3\theta/4$ for Gaussian via the $\Gamma$-formula;
+$-\theta/2$ empirically for Bernoulli($\tfrac12$)); conditioning removes a sign-changing
+contamination bias, best implemented by zero-fill.
+
+See `problem-statement.md` Section 3 for the per-claim figure/number references.

multireward_grpo-0.1.0/README_PYPI.md

@@ -0,0 +1,145 @@
+# multireward-grpo
+
+**Decoupled & conditioned multi-reward GRPO** — advantage estimators, a
+generalized trainer, and the Theorem-3 verification harness from the paper
+*"When and Why Decoupling and Conditioning Beat Reweighting in Multi-Reward
+GRPO: A U-Statistic Treatment."*
+
+This package modularizes the experiment code so you can train your own
+multi-reward GRPO models, verify the correlation-aware MSE law on your own
+rollouts, and (optionally) run the whole thing on a cloud GPU.
+
+```bash
+pip install multireward-grpo            # core (numpy/scipy): advantage + analysis
+pip install "multireward-grpo[llm]"     # + torch/transformers/peft: training & real generation
+pip install "multireward-grpo[viz]"     # + matplotlib: the money-plot figure
+pip install "multireward-grpo[data]"    # + datasets/hf-hub: dataset loaders & model push
+pip install "multireward-grpo[runpod]"  # + requests: cloud GPU orchestration
+```
+
+## The two orderings
+
+Given a group of `m` rollouts each scored on `R` reward channels with weights `w`:
+
+- **AN — Aggregate-then-Normalize** (classic GRPO baseline): scalarize `s = wᵀr`,
+  then group-normalize. The high-variance channel dominates (Prop 1) and the
+  advantage resolution collapses under heterogeneous scales (Prop 2).
+- **NA — Normalize-then-Aggregate** (the decoupled estimator, = MO-GRPO/GDPO):
+  group-normalize each channel, then take the weighted sum. Restores
+  weight-proportional influence and gives the correlation-aware gradient-MSE
+  floor `(τ²/m)·wᵀCw` (Theorem 3).
+
+```python
+import numpy as np
+from multireward_grpo import compute_advantage
+
+rewards = np.array([[1.0, 0.3, 1.0],   # (m=4 rollouts, R=3 channels)
+                    [0.0, 0.9, 1.0],
+                    [1.0, 0.1, 0.0],
+                    [0.0, 0.5, 1.0]])
+w = np.array([1.0, 1.0, 0.5])
+A_na = compute_advantage(rewards, w, mode="na")   # recommended
+A_an = compute_advantage(rewards, w, mode="an")   # GRPO baseline
+```
+
+## Train your own model
+
+Bring **your own prompts** and **your own reward function**; the trainer runs
+group-relative policy optimization with a KL anchor and saves a LoRA adapter.
+
+```python
+from multireward_grpo import GRPOConfig, GRPOTrainer
+
+# prompts: list of strings, chat-message lists, or dicts with metadata
+prompts = ["Write a polite refusal to a refund demand.", ...]
+
+# reward_fn(completion, prompt) -> R channel scores (len == len(weights))
+def reward_fn(completion, prompt):
+    return (compliance(completion), politeness(completion), action(completion))
+
+cfg = GRPOConfig(model="Qwen/Qwen2.5-1.5B-Instruct",
+                 mode="na", weights=(1.0, 1.0, 0.5), n_steps=200, m=8)
+history = GRPOTrainer(cfg, reward_fn, prompts).train()
+```
+
+### Data format
+
+| Input | Shape / type | Notes |
+|---|---|---|
+| `prompts` | `list[str \| list[dict] \| dict]` | a string (user msg), chat messages `[{"role","content"}]`, or `{"prompt": ..., "gold": ...}` with metadata passed through to the reward fn |
+| `reward_fn(completion, prompt)` | returns `Sequence[float]` of length `R` | one score per reward channel; **channel 0 is the gate** for conditioning |
+| `weights` | `tuple[float, ...]` length `R` | objective weights `w` |
+| `mode` | `"na" \| "an" \| "single"` | `na` is the paper's recommendation |
+
+Reward tensors for the analysis tools use shape **`(P, K, m, R)`** = prompts ×
+seeds × rollouts × reward channels.
+
+### Ready-made examples
+
+```python
+from multireward_grpo.examples import FintechRewardFunction, make_fintech_prompts
+from multireward_grpo import GRPOConfig, GRPOTrainer
+
+prompts = make_fintech_prompts(400, seed=0)
+cfg = GRPOConfig(mode="na", weights=(1.0, 1.0, 0.5))
+GRPOTrainer(cfg, FintechRewardFunction(), prompts).train()
+```
+
+`multireward_grpo.examples.gsm8k` provides GSM8K loaders paired with
+`multireward_grpo.rewards.MathRewardFunction` (correctness / length / format).
+
+## Verify Theorem 3 on your rollouts
+
+```python
+from multireward_grpo import analyze, summary_print
+from multireward_grpo.generation import MockBackend, run_corpus, pack_for_analysis
+import numpy as np
+
+C = np.array([[1, 0.5, 0], [0.5, 1, 0], [0, 0, 1]])   # reward correlation
+corpus = run_corpus(MockBackend(C=C), [(f"p{i}", "0") for i in range(40)],
+                    m_grid=[8], K_seeds=200)
+rewards = pack_for_analysis(corpus, m=8)               # (P, K, m, R)
+result = analyze(rewards, w=np.array([1.0, 1.0, 0.5]))
+summary_print(result)
+```
+
+Or from the shell:
+
+```bash
+multireward-grpo thm3-check --rho 0.5      # CPU, no GPU
+multireward-grpo train --mode na --n-steps 50   # needs [llm] + GPU
+```
+
+## Run on a cloud GPU (RunPod)
+
+```python
+from multireward_grpo.runpod import RunPodClient
+client = RunPodClient()  # reads RUNPOD_API_KEY from env or .env
+client.run_command('pip install "multireward-grpo[llm]" && multireward-grpo train --mode na',
+                   wall_clock_cap=1800)
+```
+
+## Released artifacts (Hugging Face)
+
+Datasets and fine-tuned models from the paper live under the
+[`eagle0504`](https://huggingface.co/eagle0504) namespace:
+
+**Datasets**
+- [multireward-grpo-gsm8k-rewards](https://huggingface.co/datasets/eagle0504/multireward-grpo-gsm8k-rewards) — 76,800 Qwen2.5-1.5B GSM8K rollouts (rewards + chains-of-thought)
+- [multireward-grpo-gsm8k-rewards-qwen2.5-7b](https://huggingface.co/datasets/eagle0504/multireward-grpo-gsm8k-rewards-qwen2.5-7b) — 25,600 Qwen2.5-7B rollouts
+- [multireward-grpo-fintech-customer-comms](https://huggingface.co/datasets/eagle0504/multireward-grpo-fintech-customer-comms) — 2,400 fintech conversations
+
+**Models** (LoRA adapters for Qwen2.5-1.5B-Instruct)
+- [multireward-grpo-fintech-na-qwen2.5-1.5b](https://huggingface.co/eagle0504/multireward-grpo-fintech-na-qwen2.5-1.5b) — NA (paper's recommendation)
+- [multireward-grpo-fintech-an-qwen2.5-1.5b](https://huggingface.co/eagle0504/multireward-grpo-fintech-an-qwen2.5-1.5b) — AN baseline
+- [multireward-grpo-fintech-single-qwen2.5-1.5b](https://huggingface.co/eagle0504/multireward-grpo-fintech-single-qwen2.5-1.5b) — single-reward ablation
+
+## Citation
+
+If you use this package, please cite the paper (see the
+[GitHub repository](https://github.com/yiqiao-yin/multireward-grpo) for the
+current BibTeX entry).
+
+## License
+
+MIT