rewardprobe 0.1.0__tar.gz

rewardprobe-0.1.0/.gitignore

@@ -0,0 +1,22 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+.venv/
+venv/
+.env
+.pytest_cache/
+.ruff_cache/
+htmlcov/
+.coverage
+*.log
+
+# Vitals provenance data
+.vitals/
+hunt/
+.claude/
+excalidraw.log

rewardprobe-0.1.0/CLAUDE.md

@@ -0,0 +1,121 @@
+# rewardprobe — Architecture Guide
+
+## What This Is
+
+rewardprobe answers: **"What will my model learn to do against this reward function?"**
+
+Three modes:
+1. **Quick Check** — 30 deterministic probes, instant, free
+2. **Deep Analysis** — Claude reads your code + generates adversarial inputs
+3. **Simulate** — generates diverse completions, maps the reward landscape
+
+## How It Works (Plain English)
+
+### Quick Check
+We generate adversarial inputs — empty strings, format tricks, parser exploits, wrong-but-formatted answers — and feed them to your reward function. If it scores them high, that's a bug.
+
+### Deep Analysis
+We send your reward function's Python source code to Claude. Claude reads it and says: "This function uses rfind() to find the last \boxed{} tag — a model can output multiple tags and only the last one is checked." Then Claude generates realistic wrong completions and we actually run them against your function.
+
+### Simulate
+We use Claude to generate 10 completions per task, each representing a different strategy a model might adopt during training:
+
+- **perfect** — correct reasoning + correct answer
+- **correct_lazy** — just the answer, no work shown
+- **shortcut** — skips computation, guesses based on patterns
+- **format_only** — perfect format, wrong content
+- **near_miss** — plausible but slightly wrong
+- **hedge** — multiple answers hoping one matches
+- etc.
+
+Then we score every completion against your reward function and build the **strategy scoreboard** — a bar chart showing which behaviors get the most reward. If "shortcut" scores as high as "perfect," the model will learn to take shortcuts.
+
+## Architecture
+
+```
+src/rewardprobe/
+├── probe.py                    # Probe class — main entry point
+├── types.py                    # Core data models (Pydantic)
+├── report.py                   # Terminal output (Rich)
+├── simulator.py                # Reward landscape simulation
+├── adapters/
+│   ├── auto.py                 # Universal signature detection
+│   ├── verifiers.py            # verifiers framework adapter
+│   ├── trl.py                  # TRL GRPOTrainer adapter
+│   └── raw.py                  # Raw Python functions
+├── families/
+│   ├── base.py                 # Attack base classes + format detection
+│   ├── basic_sanity.py         # Empty, whitespace, range, determinism (7)
+│   ├── trivial_bypass.py       # Echo, constant, format-only (4)
+│   ├── parser_bugs.py          # Reasoning leaks, regex, malformed (6)
+│   ├── composition.py          # Weight dominance, zero-weight leaks (3)
+│   ├── schema.py               # Missing columns, type mismatches (5)
+│   └── stateful.py             # State leak, stale state, tool skip (5)
+└── tier2/
+    ├── client.py               # LLM client (Anthropic, OpenAI, Ollama)
+    ├── code_analyzer.py        # Claude reads source code
+    ├── adversarial.py          # Claude generates adversarial completions
+    ├── validator.py             # Filters false positives from Tier 1
+    └── prompts.py              # All LLM prompt templates
+```
+
+## Key Concepts
+
+### NormalizedEnvironment
+Every adapter converts framework-specific reward functions into one universal interface:
+```python
+NormalizedEnvironment(
+    reward_fns=[NormalizedRewardFn(name="...", fn=wrapped_fn, weight=1.0)],
+    tasks=[{"prompt": "...", "answer": "..."}, ...],
+)
+```
+The wrapped `fn` always has signature `(completion_text: str, task: dict) -> float`.
+
+### Auto-Detection
+`auto.py` classifies any callable into one of 5 patterns:
+- `SINGLE_PAIR` — `fn(completion, answer) -> float`
+- `SINGLE_FLEXIBLE` — `fn(completion) -> float`
+- `BATCH_SIMPLE` — `fn(completions, **kwargs) -> list[float]`
+- `BATCH_WITH_PROMPTS` — `fn(prompts, completions, **kwargs) -> list[float]`
+- `VERIFIERS_ASYNC` — `async fn(completion, answer, ...) -> float`
+
+### Format Detection
+Reward functions expect answers in different formats. Before running attacks, we probe the function with common formats (`\boxed{}`, `####`, `<answer>`, raw) to discover what it accepts. This prevents false positives on format-enforcing functions.
+
+### Answer Key Normalization
+Datasets use different column names for ground truth (`answer`, `solution`, `label`, `target`, `expected_output`). We detect and normalize to `answer` before running attacks.
+
+## Adding a New Attack
+
+```python
+class MyAttack(BaseAttack):
+    name = "my_attack"
+    family = "basic_sanity"
+    severity = Severity.WARNING
+
+    def run(self, env: NormalizedEnvironment) -> list[Finding]:
+        findings = []
+        for fn in env.reward_fns:
+            for idx, task in enumerate(env.tasks):
+                score, err = _safe_call(fn, "adversarial input", task)
+                if err is None and score is not None and score > 0.5:
+                    findings.append(Finding(
+                        attack=self.name, family=self.family,
+                        severity=self.severity,
+                        input="adversarial input",
+                        actual_reward=score,
+                        affected_tasks=[idx],
+                        recommendation="Plain English explanation of the problem",
+                    ))
+        return findings
+```
+
+Add to the family's `__init__` attack list. Add a test in `tests/test_families/`.
+
+## Design Principles
+
+1. **No false positives.** A finding the developer ignores erodes all trust. Every finding must be real and actionable.
+2. **Plain English.** No jargon in output. "Your function scores wrong answers at 0.8" not "parser_bugs/regex_greediness FAILED."
+3. **Domain agnostic.** Works for math, code, text, classification — any RL reward function. No hardcoded formats.
+4. **Auto-detect everything.** The developer should never configure their framework, format, or column names.
+5. **Seconds, not minutes.** Tier 1 is instant. Tier 2 is under a minute. Simulate is 2-3 minutes.

rewardprobe-0.1.0/PKG-INFO

@@ -0,0 +1,249 @@
+Metadata-Version: 2.4
+Name: rewardprobe
+Version: 0.1.0
+Summary: Pre-training stress-testing for reward functions. Find bugs in minutes on CPU instead of days into a $10K training run.
+Project-URL: Homepage, https://github.com/rewardprobe/rewardprobe
+Project-URL: Documentation, https://github.com/rewardprobe/rewardprobe
+Project-URL: Repository, https://github.com/rewardprobe/rewardprobe
+Author: rewardprobe contributors
+License-Expression: Apache-2.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: click>=8.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: rich>=13.0
+Provides-Extra: all
+Requires-Dist: trl>=0.15.0; extra == 'all'
+Requires-Dist: verifiers>=0.1.0; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.8.0; extra == 'dev'
+Provides-Extra: trl
+Requires-Dist: trl>=0.15.0; extra == 'trl'
+Provides-Extra: verifiers
+Requires-Dist: verifiers>=0.1.0; extra == 'verifiers'
+Description-Content-Type: text/markdown
+
+# rewardprobe
+
+**Know what your model will learn — before you train.**
+
+[![PyPI](https://img.shields.io/pypi/v/rewardprobe)](https://pypi.org/project/rewardprobe/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://python.org)
+
+---
+
+You write a reward function. You're about to spend $10K on a GRPO training run. rewardprobe tells you what the model will actually learn to do:
+
+```
+rewardprobe simulate — production_math_rlvr
+  50 completions across 5 tasks
+
+  2 critical found
+
+  1.  critical
+     'Shortcut' strategy scores 0.71
+     A model using the shortcut strategy earns 103% of what a correct
+     answer earns. It will learn to skip computation and take shortcuts
+     because that's easier AND scores higher.
+
+  2.  critical
+     'Lazy correct' strategy scores only 0.07
+     A correct answer without formatting scores near zero. Your reward
+     function punishes correct-but-unformatted answers more than it
+     punishes wrong-but-formatted ones.
+
+  Strategy scoreboard:
+    perfect              ████████████████████ 1.00
+    correct_verbose      ████████████████████ 1.00
+    shortcut             ██████████████░░░░░░ 0.71  ← problem
+    near_miss            █████░░░░░░░░░░░░░░░ 0.29
+    format_only          █████░░░░░░░░░░░░░░░ 0.29
+    garbage              ███░░░░░░░░░░░░░░░░░ 0.18
+    correct_lazy         █░░░░░░░░░░░░░░░░░░░ 0.07  ← problem
+```
+
+The **strategy scoreboard** shows exactly how your reward function scores different model behaviors. If a lazy or wrong strategy scores close to a correct one, the model will learn the lazy path. You see this in 30 seconds instead of discovering it 3 days into training.
+
+---
+
+## The Problem
+
+You write a reward function for RL training. It looks correct. You start training. Days later, the model is gaming the reward — outputting shortcuts, copying format without thinking, or guessing. OpenAI [documented](https://openai.com/index/chain-of-thought-monitoring/) this happening with `exit(0)` and `raise SkipTest`. METR [found](https://metr.org/blog/2025-06-05-recent-reward-hacking/) frontier models monkey-patching their own graders.
+
+The fix is to test reward functions **before** training, the same way you test code before deploying.
+
+---
+
+## Install
+
+```bash
+pip install rewardprobe
+```
+
+---
+
+## Three Modes
+
+### 1. Quick Check (free, instant, no API key)
+
+30 deterministic probes. Catches parser bugs, edge cases, format tricks. Runs in under a second on CPU.
+
+```bash
+rewardprobe test my_reward.py::my_fn --dataset tasks.jsonl
+```
+
+```
+rewardprobe — my_reward
+
+  1 critical, 2 warning found
+
+  1.  critical
+     Correct answer in reasoning section scores 1.0 even when the
+     answer field contains a wrong answer.
+
+  2.  warning
+     Different scores depending on answer tag order.
+
+  28/30 checks passed.
+```
+
+### 2. Deep Analysis (needs API key)
+
+Claude reads your source code, understands what each function does, and generates realistic adversarial completions. Finds bugs that static probes can't.
+
+```bash
+export ANTHROPIC_API_KEY=sk-...
+rewardprobe test my_reward.py::my_fn --dataset tasks.jsonl --deep
+```
+
+This adds:
+- **Code analysis** — Claude identifies logic bugs by reading your Python code
+- **Adversarial completions** — wrong-but-plausible model outputs tested against your function
+- **False positive filtering** — classifies each function (correctness/format/auxiliary) so findings are precise
+
+### 3. Simulate (needs API key)
+
+The flagship feature. Generates diverse completions spanning the full range of what a model might produce during training — from perfect solutions to garbage — and maps the reward landscape.
+
+```bash
+rewardprobe simulate my_reward.py::my_fn --dataset tasks.jsonl
+```
+
+The strategy scoreboard shows you at a glance:
+- **Green strategies** (perfect, correct_lazy, correct_verbose) — what you WANT the model to learn
+- **Red strategies** (shortcut, format_only, hedge, garbage) — what you DON'T want
+
+If a red strategy scores close to or higher than a green one, your reward function has a problem.
+
+---
+
+## What We Found
+
+We ran rewardprobe against reward functions from 4 major RL codebases plus 3 non-math domains. Results:
+
+| Codebase | Domain | Key Finding |
+|----------|--------|-------------|
+| **verifiers/gsm8k** (Prime Intellect) | Math | Model can skip reasoning — `correct_lazy` scores 1.0 |
+| **Open-R1** (HuggingFace) | Math | `first_match` mode lets models hedge with multiple answers |
+| **verl** (ByteDance) | Math | `format_score` parameter can reward wrong answers |
+| **willccbb GRPO gist** | Math | Returns 2.0 (outside [0,1]); rejects "42.0" for "42" |
+| Custom code reward | Code | Off-by-one bugs score 0.83 — substring matching misses logic errors |
+| Sentiment classifier | Text | Reasoned answers score 0.0, bare labels score 1.0 |
+
+---
+
+## Works With Any Framework
+
+Auto-detects your reward function's signature. No configuration.
+
+```python
+# Any of these just work:
+def my_reward(completion, answer): ...                     # Raw Python
+def accuracy_reward(completions, solution, **kwargs): ...  # TRL / GRPO
+def correctness(prompts, completions, answer, **kwargs): ... # TRL with prompts
+async def correct_answer(completion, answer): ...          # verifiers
+def compute_score(solution_str, ground_truth): ...         # ByteDance verl
+```
+
+```bash
+rewardprobe test file.py::fn --dataset tasks.jsonl    # Just works
+rewardprobe test environments/gsm8k.py                # verifiers environments too
+```
+
+---
+
+## GitHub Action
+
+```yaml
+- run: pip install rewardprobe
+- run: rewardprobe test my_reward.py::my_fn --dataset tasks.jsonl --ci
+```
+
+Exit code 1 on critical findings. Add `--deep` with `ANTHROPIC_API_KEY` secret for AI analysis in CI.
+
+---
+
+## Python API
+
+```python
+from rewardprobe import Probe
+
+# Quick check
+report = Probe().test_fn(my_reward, tasks)
+print(report.passed)  # True / False
+
+# Deep analysis
+report = Probe(deep=True).test_fn(my_reward, tasks)
+
+# Simulate
+from rewardprobe.simulator import simulate, print_simulation
+from rewardprobe.tier2.client import get_client
+from rewardprobe.adapters.auto import auto_adapt
+
+env = auto_adapt(my_reward, tasks)
+result = simulate(env, get_client("sonnet"), n_tasks=5)
+print_simulation(result)
+```
+
+---
+
+## How It Works
+
+**Quick Check** generates adversarial inputs (empty strings, format tricks, parser exploits, wrong-but-formatted answers) and tests your reward function against them. 30 probes across 6 families, all deterministic, all on CPU.
+
+**Deep Analysis** uses Claude to read your reward function's Python source code. It understands what the function checks, identifies logic bugs, and generates realistic wrong completions that a model might produce during training. Each completion is actually run against your function — only real exploits are reported.
+
+**Simulate** uses Claude to generate 10 diverse completions per task, each representing a different strategy a model might learn (perfect, lazy, shortcut, hedging, garbage, etc). Scores them all against your reward function. The strategy scoreboard shows which behaviors your reward function actually incentivizes.
+
+---
+
+## What rewardprobe Is NOT
+
+- **Not a training monitor.** We run *before* training starts.
+- **Not a formal prover.** We find bugs empirically with concrete inputs.
+- **Not a guarantee.** A clean report means "we tested these patterns and found nothing." The nastiest reward hacks are novel and environment-specific.
+
+---
+
+## Contributing
+
+See [CLAUDE.md](CLAUDE.md) for architecture, how to add attacks, and how the simulator works.
+
+```bash
+git clone https://github.com/rewardprobe/rewardprobe && cd rewardprobe
+uv sync --extra dev && pytest tests/
+```
+
+Apache 2.0

rewardprobe-0.1.0/README.md

@@ -0,0 +1,213 @@
+# rewardprobe
+
+**Know what your model will learn — before you train.**
+
+[![PyPI](https://img.shields.io/pypi/v/rewardprobe)](https://pypi.org/project/rewardprobe/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://python.org)
+
+---
+
+You write a reward function. You're about to spend $10K on a GRPO training run. rewardprobe tells you what the model will actually learn to do:
+
+```
+rewardprobe simulate — production_math_rlvr
+  50 completions across 5 tasks
+
+  2 critical found
+
+  1.  critical
+     'Shortcut' strategy scores 0.71
+     A model using the shortcut strategy earns 103% of what a correct
+     answer earns. It will learn to skip computation and take shortcuts
+     because that's easier AND scores higher.
+
+  2.  critical
+     'Lazy correct' strategy scores only 0.07
+     A correct answer without formatting scores near zero. Your reward
+     function punishes correct-but-unformatted answers more than it
+     punishes wrong-but-formatted ones.
+
+  Strategy scoreboard:
+    perfect              ████████████████████ 1.00
+    correct_verbose      ████████████████████ 1.00
+    shortcut             ██████████████░░░░░░ 0.71  ← problem
+    near_miss            █████░░░░░░░░░░░░░░░ 0.29
+    format_only          █████░░░░░░░░░░░░░░░ 0.29
+    garbage              ███░░░░░░░░░░░░░░░░░ 0.18
+    correct_lazy         █░░░░░░░░░░░░░░░░░░░ 0.07  ← problem
+```
+
+The **strategy scoreboard** shows exactly how your reward function scores different model behaviors. If a lazy or wrong strategy scores close to a correct one, the model will learn the lazy path. You see this in 30 seconds instead of discovering it 3 days into training.
+
+---
+
+## The Problem
+
+You write a reward function for RL training. It looks correct. You start training. Days later, the model is gaming the reward — outputting shortcuts, copying format without thinking, or guessing. OpenAI [documented](https://openai.com/index/chain-of-thought-monitoring/) this happening with `exit(0)` and `raise SkipTest`. METR [found](https://metr.org/blog/2025-06-05-recent-reward-hacking/) frontier models monkey-patching their own graders.
+
+The fix is to test reward functions **before** training, the same way you test code before deploying.
+
+---
+
+## Install
+
+```bash
+pip install rewardprobe
+```
+
+---
+
+## Three Modes
+
+### 1. Quick Check (free, instant, no API key)
+
+30 deterministic probes. Catches parser bugs, edge cases, format tricks. Runs in under a second on CPU.
+
+```bash
+rewardprobe test my_reward.py::my_fn --dataset tasks.jsonl
+```
+
+```
+rewardprobe — my_reward
+
+  1 critical, 2 warning found
+
+  1.  critical
+     Correct answer in reasoning section scores 1.0 even when the
+     answer field contains a wrong answer.
+
+  2.  warning
+     Different scores depending on answer tag order.
+
+  28/30 checks passed.
+```
+
+### 2. Deep Analysis (needs API key)
+
+Claude reads your source code, understands what each function does, and generates realistic adversarial completions. Finds bugs that static probes can't.
+
+```bash
+export ANTHROPIC_API_KEY=sk-...
+rewardprobe test my_reward.py::my_fn --dataset tasks.jsonl --deep
+```
+
+This adds:
+- **Code analysis** — Claude identifies logic bugs by reading your Python code
+- **Adversarial completions** — wrong-but-plausible model outputs tested against your function
+- **False positive filtering** — classifies each function (correctness/format/auxiliary) so findings are precise
+
+### 3. Simulate (needs API key)
+
+The flagship feature. Generates diverse completions spanning the full range of what a model might produce during training — from perfect solutions to garbage — and maps the reward landscape.
+
+```bash
+rewardprobe simulate my_reward.py::my_fn --dataset tasks.jsonl
+```
+
+The strategy scoreboard shows you at a glance:
+- **Green strategies** (perfect, correct_lazy, correct_verbose) — what you WANT the model to learn
+- **Red strategies** (shortcut, format_only, hedge, garbage) — what you DON'T want
+
+If a red strategy scores close to or higher than a green one, your reward function has a problem.
+
+---
+
+## What We Found
+
+We ran rewardprobe against reward functions from 4 major RL codebases plus 3 non-math domains. Results:
+
+| Codebase | Domain | Key Finding |
+|----------|--------|-------------|
+| **verifiers/gsm8k** (Prime Intellect) | Math | Model can skip reasoning — `correct_lazy` scores 1.0 |
+| **Open-R1** (HuggingFace) | Math | `first_match` mode lets models hedge with multiple answers |
+| **verl** (ByteDance) | Math | `format_score` parameter can reward wrong answers |
+| **willccbb GRPO gist** | Math | Returns 2.0 (outside [0,1]); rejects "42.0" for "42" |
+| Custom code reward | Code | Off-by-one bugs score 0.83 — substring matching misses logic errors |
+| Sentiment classifier | Text | Reasoned answers score 0.0, bare labels score 1.0 |
+
+---
+
+## Works With Any Framework
+
+Auto-detects your reward function's signature. No configuration.
+
+```python
+# Any of these just work:
+def my_reward(completion, answer): ...                     # Raw Python
+def accuracy_reward(completions, solution, **kwargs): ...  # TRL / GRPO
+def correctness(prompts, completions, answer, **kwargs): ... # TRL with prompts
+async def correct_answer(completion, answer): ...          # verifiers
+def compute_score(solution_str, ground_truth): ...         # ByteDance verl
+```
+
+```bash
+rewardprobe test file.py::fn --dataset tasks.jsonl    # Just works
+rewardprobe test environments/gsm8k.py                # verifiers environments too
+```
+
+---
+
+## GitHub Action
+
+```yaml
+- run: pip install rewardprobe
+- run: rewardprobe test my_reward.py::my_fn --dataset tasks.jsonl --ci
+```
+
+Exit code 1 on critical findings. Add `--deep` with `ANTHROPIC_API_KEY` secret for AI analysis in CI.
+
+---
+
+## Python API
+
+```python
+from rewardprobe import Probe
+
+# Quick check
+report = Probe().test_fn(my_reward, tasks)
+print(report.passed)  # True / False
+
+# Deep analysis
+report = Probe(deep=True).test_fn(my_reward, tasks)
+
+# Simulate
+from rewardprobe.simulator import simulate, print_simulation
+from rewardprobe.tier2.client import get_client
+from rewardprobe.adapters.auto import auto_adapt
+
+env = auto_adapt(my_reward, tasks)
+result = simulate(env, get_client("sonnet"), n_tasks=5)
+print_simulation(result)
+```
+
+---
+
+## How It Works
+
+**Quick Check** generates adversarial inputs (empty strings, format tricks, parser exploits, wrong-but-formatted answers) and tests your reward function against them. 30 probes across 6 families, all deterministic, all on CPU.
+
+**Deep Analysis** uses Claude to read your reward function's Python source code. It understands what the function checks, identifies logic bugs, and generates realistic wrong completions that a model might produce during training. Each completion is actually run against your function — only real exploits are reported.
+
+**Simulate** uses Claude to generate 10 diverse completions per task, each representing a different strategy a model might learn (perfect, lazy, shortcut, hedging, garbage, etc). Scores them all against your reward function. The strategy scoreboard shows which behaviors your reward function actually incentivizes.
+
+---
+
+## What rewardprobe Is NOT
+
+- **Not a training monitor.** We run *before* training starts.
+- **Not a formal prover.** We find bugs empirically with concrete inputs.
+- **Not a guarantee.** A clean report means "we tested these patterns and found nothing." The nastiest reward hacks are novel and environment-specific.
+
+---
+
+## Contributing
+
+See [CLAUDE.md](CLAUDE.md) for architecture, how to add attacks, and how the simulator works.
+
+```bash
+git clone https://github.com/rewardprobe/rewardprobe && cd rewardprobe
+uv sync --extra dev && pytest tests/
+```
+
+Apache 2.0

rewardprobe-0.1.0/action.yml

@@ -0,0 +1,46 @@
+name: "rewardprobe"
+description: "Stress-test RL reward functions before training. Catch reward hacking in CI."
+branding:
+  icon: "shield"
+  color: "red"
+
+inputs:
+  target:
+    description: "Path to reward function (file.py::fn) or verifiers environment (env.py)"
+    required: true
+  dataset:
+    description: "Path to JSONL tasks file (required for non-verifiers targets)"
+    required: false
+  python-version:
+    description: "Python version to use"
+    required: false
+    default: "3.12"
+  strict:
+    description: "Fail on warnings too (not just critical findings)"
+    required: false
+    default: "false"
+
+runs:
+  using: "composite"
+  steps:
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ inputs.python-version }}
+
+    - name: Install rewardprobe
+      shell: bash
+      run: pip install rewardprobe
+
+    - name: Run rewardprobe
+      shell: bash
+      run: |
+        ARGS="rewardprobe test ${{ inputs.target }}"
+        if [ -n "${{ inputs.dataset }}" ]; then
+          ARGS="$ARGS --dataset ${{ inputs.dataset }}"
+        fi
+        ARGS="$ARGS --ci"
+        if [ "${{ inputs.strict }}" = "true" ]; then
+          ARGS="$ARGS --strict"
+        fi
+        $ARGS

rewardprobe-0.1.0/examples/sample_tasks.jsonl

@@ -0,0 +1,10 @@
+{"prompt": "What is 2+2?", "answer": "4"}
+{"prompt": "What is 3+3?", "answer": "6"}
+{"prompt": "What is 5*5?", "answer": "25"}
+{"prompt": "What is 7+3?", "answer": "10"}
+{"prompt": "What is 6*7?", "answer": "42"}
+{"prompt": "What is 9-4?", "answer": "5"}
+{"prompt": "What is 100/4?", "answer": "25"}
+{"prompt": "What is 8*2?", "answer": "16"}
+{"prompt": "What is 15-7?", "answer": "8"}
+{"prompt": "What is 12/3?", "answer": "4"}