gameability 0.2.0__tar.gz → 0.3.1__tar.gz

{gameability-0.2.0/src/gameability.egg-info → gameability-0.3.1}/PKG-INFO

@@ -1,162 +1,202 @@
-Metadata-Version: 2.4
-Name: gameability
-Version: 0.2.0
-Summary: Find the problems in a code benchmark that accept broken solutions.
-Author: Empyrean Intelligence
-License-Expression: MIT
-Project-URL: Homepage, https://github.com/Empyrean-Intelligence/gameability
-Project-URL: Source, https://github.com/Empyrean-Intelligence/gameability
-Project-URL: Issues, https://github.com/Empyrean-Intelligence/gameability/issues
-Classifier: Programming Language :: Python :: 3
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Classifier: Topic :: Software Development :: Testing
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Provides-Extra: hf
-Requires-Dist: datasets; extra == "hf"
-Dynamic: license-file
-
-# gameability
-
-**Find the problems in a code benchmark that accept broken solutions.**
-
-When a model learns from tests, the tests are its reward. A self-improving system trained on tests that
-accept broken code learns to pass weak tests, not to write correct code. `gameability` measures how often
-that can happen. It is part of the verification layer we use at Empyrean Intelligence for self-improvement
-work, released for anyone who builds benchmarks or RL training sets.
-
-A benchmark score means something only if the benchmark rejects solutions that lack the
-skill it measures. `gameability` checks that directly. It takes each problem's reference
-solution, breaks it in one small place, confirms the result really is wrong, and then
-asks whether the problem's own tests notice.
-
-```
-pip install "gameability[hf]"
-gameability check mbpp
-```
-
-On MBPP, **17.9% of problems (169 of 942 judged) accept a broken solution**; on HumanEval,
-**10.3% (16 of 155)**. Each run takes minutes on a laptop, with no model and no API key.
-
-**Confirmed on valid inputs.** On the problems EvalPlus also covers, every accepted broken solution
-was re-run against EvalPlus's independent tests, whose inputs respect each problem's rules:
-**95.2% of flagged MBPP problems and 84.6% of flagged HumanEval problems are confirmed** (the broken
-solution passes the original tests and fails EvalPlus). The two unconfirmed HumanEval problems
-(116 and 129) are ones whose own visible tests contradict their written specification.
-
-**What changed in v0.2.** v0.1 built fresh inputs by changing values freely, which sometimes asked
-a problem about inputs it never promised to handle (an unsorted list for binary search, random
-letters for a parenthesis parser). v0.2 learns each argument's domain from the problem's own
-visible tests and stays inside it: number signs and minimums, the characters or whole tokens a
-string uses, and whether lists are sorted, unique, rectangular or a permutation. On the same
-problems this raised the confirmed share from 85.7% to 95.2% (MBPP) and 78.6% to 84.6% (HumanEval)
-while keeping 93.1% of v0.1's confirmed detections. `--inputs free` reproduces v0.1.
-
-## Scoreboard
-
-Problems that accept at least one broken solution (wrong on at least one fresh input), with 95% intervals. Defaults (24 broken
-versions per problem, 8 fresh inputs, seed 0), laptop CPU, October 2026.
-
-| Benchmark | Weak problems | Command |
-|---|---|---|
-| MBPP | 17.9% (169 of 942) [15.6, 20.5] | `gameability check mbpp` |
-| MBPP sanitized | 13.8% (57 of 413) [10.8, 17.5] | `gameability check mbpp-sanitized` |
-| HumanEval | 10.3% (16 of 155) [6.5, 16.1] | `gameability check humaneval` |
-| KodCode (RL training set, 1,000-row random sample) | 9.7% (45 of 466) [7.3, 12.7] | `gameability check kodcode:rows.jsonl` |
-| HumanEval+ (EvalPlus) | 3.4% (5 of 147) [1.5, 7.7] | `gameability check humaneval-plus` |
-| MBPP+ (EvalPlus) | 2.0% (7 of 353) [1.0, 4.0] | `gameability check mbpp-plus` |
-
-EvalPlus's stronger tests look like a large improvement, but EvalPlus also rewrote most reference
-solutions. Holding the original reference fixed (`mbpp-plus-origref`, `humaneval-plus-origref`), the weak
-share roughly halves (MBPP 8.7% to 3.1% on 254 problems, HumanEval 10.6% to 4.9% on 123), and at these
-sample sizes the intervals overlap. KodCode's judged set is the problems whose tests call the function with
-literal arguments (466 of 801 loadable). `rows.jsonl` is KodCode rows saved as JSON lines; the sample above
-is reproducible with `python tests/sample_kodcode.py rows.jsonl` (positions from `random.Random(0)`).
-
-## What it prints
-
-```
-  415 -- 6 of 11 broken solutions pass every visible test
-      -    if arr_len < 2:
-      +    if arr_len <= 2:
-      a test that catches it: assert max_Product([4, 9]) == (4, 9)
-```
-
-For every weak problem you get the one line that was changed, how many broken versions got
-through, and an assert that would have caught one.
-
-**`--fixes fixes.py` writes those asserts to a file; review them before adding them.** Measured on
-34 MBPP/HumanEval problems against a 1.5B model's real solutions: v0.2's suggested asserts wrongly
-reject **2.6% of its correct solutions** (8 of 307; v0.1: 15.3%). They target the broken versions
-gameability builds, not the mistakes a particular model makes: they caught 4 of that model's 24
-real wrong solutions. Treat them as leads, not as a test suite.
-
-## How it works
-
-For each problem the benchmark supplies (reference solution, visible tests):
-
-1. The reference must pass its own visible tests. If it doesn't, the problem is excluded and counted.
-2. Fresh inputs are built by varying the arguments in the visible tests. The reference's
-   outputs on them are the hidden answer key.
-3. The reference is mutated at a single site: a comparison flipped, an operator swapped, a
-   constant nudged, a `return` replaced with `return None`.
-4. A mutant counts as **broken** only if it gives a different answer from the reference on
-   at least one fresh input. Mutants that behave identically are thrown away, so they can't
-   inflate the number.
-5. A broken mutant that still passes every visible test is a **survivor**. A problem with at
-   least one survivor accepts a wrong solution.
-
-Benchmark code runs in a child process with a step budget, so a runaway mutant is stopped at
-the same point on any machine; a wall-clock limit is only a backstop. Mutants that never
-finish are reported as truncated, never counted as rejected.
-
-## Your own benchmark
-
-One problem per line in a `.jsonl` file:
-
-```json
-{"id": "clamp", "reference": "def clamp(x, lo, hi): ...", "tests": ["assert clamp(5, 0, 10) == 5"]}
-```
-
-Optional fields: `entry` (the function name, if it can't be inferred), `setup` (code run
-before the tests), `hidden` (held-out asserts you already have; they are used as extra
-answer-key inputs), `call_alias` (the name the tests call the solution by, e.g. `candidate`).
-
-```
-gameability check examples/tiny.jsonl --fixes fixes.py --json report.json
-```
-
-## What the number means, and what it doesn't
-
-- **The attacker is weak.** One small edit is the simplest possible way to be wrong; a model trained
-  against these tests can search far more. In that sense the rates are a floor.
-- **Fresh inputs can still break a problem's rules in ways the tests don't reveal.** v0.2 keeps each
-  argument inside its observed domain, but it does not model relationships BETWEEN arguments (a
-  length argument that must equal a list's length) or rules that only the docstring states. On the
-  problems EvalPlus covers, 4.8% of flagged MBPP and 15.4% of flagged HumanEval problems are still
-  unconfirmed (v0.1: 14.3% and 21.4%).
-- **It does not predict where a model will cheat.** On a 1.5B coder, false passes were no more
-  common on flagged problems than on others (8.4% vs 10.0%). It audits the tests' strength, not a
-  model's behaviour.
-- **It audits the tests, not any model.** Nothing here says a particular model cheated. It says
-  which problems would not have caught it.
-- **It runs the benchmark's code.** Reference solutions, mutants and tests are executed, each job in
-  its own throwaway directory (some benchmark tasks write files; v0.1 let them land in your working
-  folder). That is isolation from your files, not a security sandbox: only run it on benchmarks
-  you'd be willing to execute.
-
-## Reproducing the headline numbers
-
-```
-gameability check mbpp --json mbpp.json
-gameability check humaneval --json humaneval.json
-```
-
-Defaults: 24 mutants per problem, 8 fresh inputs, seed 0, contract inputs. Add `--inputs free` to
-reproduce v0.1's numbers (MBPP 18.5%, HumanEval 11.8%). The known-answer tests run with
-`python tests/test_gameability.py` and `python tests/test_contract_inputs.py`.
-
-## License
-
-MIT
+Metadata-Version: 2.4
+Name: gameability
+Version: 0.3.1
+Summary: Find the problems in a code benchmark that accept broken solutions, and RL environments whose reward pays for no work.
+Author: Empyrean Intelligence
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Empyrean-Intelligence/gameability
+Project-URL: Source, https://github.com/Empyrean-Intelligence/gameability
+Project-URL: Issues, https://github.com/Empyrean-Intelligence/gameability/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: hf
+Requires-Dist: datasets; extra == "hf"
+Dynamic: license-file
+
+# gameability
+
+[![PyPI](https://img.shields.io/pypi/v/gameability)](https://pypi.org/project/gameability/) [![Python](https://img.shields.io/pypi/pyversions/gameability)](https://pypi.org/project/gameability/) [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE) [![Empyrean Intelligence](https://img.shields.io/badge/by-Empyrean%20Intelligence-111)](https://empyreanintelligence.com)
+
+**Find the problems in a code benchmark that accept broken solutions.**
+
+When a model learns from tests, the tests are its reward. A self-improving system trained on tests that
+accept broken code learns to pass weak tests, not to write correct code. `gameability` measures how often
+that can happen. It is part of the verification layer we use at Empyrean Intelligence for self-improvement
+work, released for anyone who builds benchmarks or RL training sets.
+
+A benchmark score means something only if the benchmark rejects solutions that lack the
+skill it measures. `gameability` checks that directly. It takes each problem's reference
+solution, breaks it in one small place, confirms the result really is wrong, and then
+asks whether the problem's own tests notice.
+
+```
+pip install "gameability[hf]"
+gameability check mbpp
+```
+
+On MBPP, **17.9% of problems (169 of 942 judged) accept a broken solution**; on HumanEval,
+**10.3% (16 of 155)**. Each run takes minutes on a laptop, with no model and no API key.
+
+**Confirmed on valid inputs.** On the problems EvalPlus also covers, every accepted broken solution
+was re-run against EvalPlus's independent tests, whose inputs respect each problem's rules:
+**95.2% of flagged MBPP problems and 84.6% of flagged HumanEval problems are confirmed** (the broken
+solution passes the original tests and fails EvalPlus). The two unconfirmed HumanEval problems
+(116 and 129) are ones whose own visible tests contradict their written specification.
+
+**New in v0.3: `gameability probe`.** It asks the same question of an RL environment's reward: does
+it pay a policy that does no work? See [Probe an RL environment's reward](#probe-an-rl-environments-reward).
+By default it makes no model calls.
+
+**What changed in v0.2.** v0.1 built fresh inputs by changing values freely, which sometimes asked
+a problem about inputs it never promised to handle (an unsorted list for binary search, random
+letters for a parenthesis parser). v0.2 learns each argument's domain from the problem's own
+visible tests and stays inside it: number signs and minimums, the characters or whole tokens a
+string uses, and whether lists are sorted, unique, rectangular or a permutation. On the same
+problems this raised the confirmed share from 85.7% to 95.2% (MBPP) and 78.6% to 84.6% (HumanEval)
+while keeping 93.1% of v0.1's confirmed detections. `--inputs free` reproduces v0.1.
+
+## Scoreboard
+
+Problems that accept at least one broken solution (wrong on at least one fresh input), with 95% intervals. Defaults (24 broken
+versions per problem, 8 fresh inputs, seed 0), laptop CPU, October 2026.
+
+| Benchmark | Weak problems | Command |
+|---|---|---|
+| MBPP | 17.9% (169 of 942) [15.6, 20.5] | `gameability check mbpp` |
+| MBPP sanitized | 13.8% (57 of 413) [10.8, 17.5] | `gameability check mbpp-sanitized` |
+| HumanEval | 10.3% (16 of 155) [6.5, 16.1] | `gameability check humaneval` |
+| KodCode (RL training set, 1,000-row random sample) | 9.7% (45 of 466) [7.3, 12.7] | `gameability check kodcode:rows.jsonl` |
+| HumanEval+ (EvalPlus) | 3.4% (5 of 147) [1.5, 7.7] | `gameability check humaneval-plus` |
+| MBPP+ (EvalPlus) | 2.0% (7 of 353) [1.0, 4.0] | `gameability check mbpp-plus` |
+
+EvalPlus's stronger tests look like a large improvement, but EvalPlus also rewrote most reference
+solutions. Holding the original reference fixed (`mbpp-plus-origref`, `humaneval-plus-origref`), the weak
+share roughly halves (MBPP 8.7% to 3.1% on 254 problems, HumanEval 10.6% to 4.9% on 123), and at these
+sample sizes the intervals overlap. KodCode's judged set is the problems whose tests call the function with
+literal arguments (466 of 801 loadable). `rows.jsonl` is KodCode rows saved as JSON lines; the sample above
+is reproducible with `python tests/sample_kodcode.py rows.jsonl` (positions from `random.Random(0)`).
+
+## What it prints
+
+```
+  415 -- 6 of 11 broken solutions pass every visible test
+      -    if arr_len < 2:
+      +    if arr_len <= 2:
+      a test that catches it: assert max_Product([4, 9]) == (4, 9)
+```
+
+For every weak problem you get the one line that was changed, how many broken versions got
+through, and an assert that would have caught one.
+
+**`--fixes fixes.py` writes those asserts to a file; review them before adding them.** Measured on
+34 MBPP/HumanEval problems against a 1.5B model's real solutions: v0.2's suggested asserts wrongly
+reject **2.6% of its correct solutions** (8 of 307; v0.1: 15.3%). They target the broken versions
+gameability builds, not the mistakes a particular model makes: they caught 4 of that model's 24
+real wrong solutions. Treat them as leads, not as a test suite.
+
+## How it works
+
+For each problem the benchmark supplies (reference solution, visible tests):
+
+1. The reference must pass its own visible tests. If it doesn't, the problem is excluded and counted.
+2. Fresh inputs are built by varying the arguments in the visible tests. The reference's
+   outputs on them are the hidden answer key.
+3. The reference is mutated at a single site: a comparison flipped, an operator swapped, a
+   constant nudged, a `return` replaced with `return None`.
+4. A mutant counts as **broken** only if it gives a different answer from the reference on
+   at least one fresh input. Mutants that behave identically are thrown away, so they can't
+   inflate the number.
+5. A broken mutant that still passes every visible test is a **survivor**. A problem with at
+   least one survivor accepts a wrong solution.
+
+Benchmark code runs in a child process with a step budget, so a runaway mutant is stopped at
+the same point on any machine; a wall-clock limit is only a backstop. Mutants that never
+finish are reported as truncated, never counted as rejected.
+
+## Probe an RL environment's reward
+
+`gameability probe` asks a different question of an RL environment: does its reward pay a policy that
+does no work? It answers every item with an empty string, a refusal, the prompt echoed back, a wrong
+answer in the expected format, and filler text, and scores each through the environment's own rubric.
+
+```
+pip install gameability verifiers
+pip install ./my_env          # your verifiers environment
+gameability probe my_env
+```
+
+On a multiple-choice test environment with an exact-match reward (verifiers 0.1.11):
+
+```
+mmlu: 20 items, eval split, scored by the environment's own rubric
+  correct answers             1.000   (answer format: plain)
+  empty answer                0.000
+  refusal                     0.000
+  prompt echoed back          0.000
+  wrong answer, right format  0.000
+  filler text                 0.000
+  (most common answer every time earns 25% of the correct-answer reward; reported, not counted)
+VERDICT: PASS -- no null probe earns 50% of the correct-answer reward (finds only gaming that needs no capability)
+```
+
+An environment that pays for an empty answer instead prints `<-- pays for no work` on that line and
+`VERDICT: GAMEABLE`.
+
+The exit code is 1 when the environment is gameable, so it can gate CI. No model is called by default:
+OpenAI clients are pointed at a dead local address, so a reward that needs an LLM judge is reported
+UNREACHABLE instead of spending your credits (`--allow-model-calls` to permit it). A PASS means no
+*no-capability* gaming was found; it is not a proof that the reward is sound.
+
+## Your own benchmark
+
+One problem per line in a `.jsonl` file:
+
+```json
+{"id": "clamp", "reference": "def clamp(x, lo, hi): ...", "tests": ["assert clamp(5, 0, 10) == 5"]}
+```
+
+Optional fields: `entry` (the function name, if it can't be inferred), `setup` (code run
+before the tests), `hidden` (held-out asserts you already have; they are used as extra
+answer-key inputs), `call_alias` (the name the tests call the solution by, e.g. `candidate`).
+
+```
+gameability check examples/tiny.jsonl --fixes fixes.py --json report.json
+```
+
+## What the number means, and what it doesn't
+
+- **The attacker is weak.** One small edit is the simplest possible way to be wrong; a model trained
+  against these tests can search far more. In that sense the rates are a floor.
+- **Fresh inputs can still break a problem's rules in ways the tests don't reveal.** v0.2 keeps each
+  argument inside its observed domain, but it does not model relationships BETWEEN arguments (a
+  length argument that must equal a list's length) or rules that only the docstring states. On the
+  problems EvalPlus covers, 4.8% of flagged MBPP and 15.4% of flagged HumanEval problems are still
+  unconfirmed (v0.1: 14.3% and 21.4%).
+- **It does not predict where a model will cheat.** On a 1.5B coder, false passes were no more
+  common on flagged problems than on others (8.4% vs 10.0%). It audits the tests' strength, not a
+  model's behaviour.
+- **It audits the tests, not any model.** Nothing here says a particular model cheated. It says
+  which problems would not have caught it.
+- **It runs the benchmark's code.** Reference solutions, mutants and tests are executed, each job in
+  its own throwaway directory (some benchmark tasks write files; v0.1 let them land in your working
+  folder). That is isolation from your files, not a security sandbox: only run it on benchmarks
+  you'd be willing to execute.
+
+## Reproducing the headline numbers
+
+```
+gameability check mbpp --json mbpp.json
+gameability check humaneval --json humaneval.json
+```
+
+Defaults: 24 mutants per problem, 8 fresh inputs, seed 0, contract inputs. Add `--inputs free` to
+reproduce v0.1's numbers (MBPP 18.5%, HumanEval 11.8%). The known-answer tests run with
+`python tests/test_gameability.py` and `python tests/test_contract_inputs.py`.
+
+## License
+
+MIT

{gameability-0.2.0 → gameability-0.3.1}/README.md

@@ -1,5 +1,7 @@
 # gameability
 
+[![PyPI](https://img.shields.io/pypi/v/gameability)](https://pypi.org/project/gameability/) [![Python](https://img.shields.io/pypi/pyversions/gameability)](https://pypi.org/project/gameability/) [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE) [![Empyrean Intelligence](https://img.shields.io/badge/by-Empyrean%20Intelligence-111)](https://empyreanintelligence.com)
+
 **Find the problems in a code benchmark that accept broken solutions.**
 
 When a model learns from tests, the tests are its reward. A self-improving system trained on tests that
@@ -26,6 +28,10 @@ was re-run against EvalPlus's independent tests, whose inputs respect each probl
 solution passes the original tests and fails EvalPlus). The two unconfirmed HumanEval problems
 (116 and 129) are ones whose own visible tests contradict their written specification.
 
+**New in v0.3: `gameability probe`.** It asks the same question of an RL environment's reward: does
+it pay a policy that does no work? See [Probe an RL environment's reward](#probe-an-rl-environments-reward).
+By default it makes no model calls.
+
 **What changed in v0.2.** v0.1 built fresh inputs by changing values freely, which sometimes asked
 a problem about inputs it never promised to handle (an unsorted list for binary search, random
 letters for a parenthesis parser). v0.2 learns each argument's domain from the problem's own
@@ -92,6 +98,40 @@ Benchmark code runs in a child process with a step budget, so a runaway mutant i
 the same point on any machine; a wall-clock limit is only a backstop. Mutants that never
 finish are reported as truncated, never counted as rejected.
 
+## Probe an RL environment's reward
+
+`gameability probe` asks a different question of an RL environment: does its reward pay a policy that
+does no work? It answers every item with an empty string, a refusal, the prompt echoed back, a wrong
+answer in the expected format, and filler text, and scores each through the environment's own rubric.
+
+```
+pip install gameability verifiers
+pip install ./my_env          # your verifiers environment
+gameability probe my_env
+```
+
+On a multiple-choice test environment with an exact-match reward (verifiers 0.1.11):
+
+```
+mmlu: 20 items, eval split, scored by the environment's own rubric
+  correct answers             1.000   (answer format: plain)
+  empty answer                0.000
+  refusal                     0.000
+  prompt echoed back          0.000
+  wrong answer, right format  0.000
+  filler text                 0.000
+  (most common answer every time earns 25% of the correct-answer reward; reported, not counted)
+VERDICT: PASS -- no null probe earns 50% of the correct-answer reward (finds only gaming that needs no capability)
+```
+
+An environment that pays for an empty answer instead prints `<-- pays for no work` on that line and
+`VERDICT: GAMEABLE`.
+
+The exit code is 1 when the environment is gameable, so it can gate CI. No model is called by default:
+OpenAI clients are pointed at a dead local address, so a reward that needs an LLM judge is reported
+UNREACHABLE instead of spending your credits (`--allow-model-calls` to permit it). A PASS means no
+*no-capability* gaming was found; it is not a proof that the reward is sound.
+
 ## Your own benchmark
 
 One problem per line in a `.jsonl` file:

{gameability-0.2.0 → gameability-0.3.1}/pyproject.toml

@@ -1,33 +1,33 @@
-[build-system]
-requires = ["setuptools>=77"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "gameability"
-version = "0.2.0"
-description = "Find the problems in a code benchmark that accept broken solutions."
-readme = "README.md"
-license = "MIT"
-license-files = ["LICENSE"]
-requires-python = ">=3.9"
-authors = [{ name = "Empyrean Intelligence" }]
-dependencies = []
-classifiers = [
-  "Programming Language :: Python :: 3",
-  "Topic :: Scientific/Engineering :: Artificial Intelligence",
-  "Topic :: Software Development :: Testing",
-]
-
-[project.optional-dependencies]
-hf = ["datasets"]
-
-[project.urls]
-Homepage = "https://github.com/Empyrean-Intelligence/gameability"
-Source = "https://github.com/Empyrean-Intelligence/gameability"
-Issues = "https://github.com/Empyrean-Intelligence/gameability/issues"
-
-[project.scripts]
-gameability = "gameability.cli:main"
-
-[tool.setuptools.packages.find]
-where = ["src"]
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "gameability"
+version = "0.3.1"
+description = "Find the problems in a code benchmark that accept broken solutions, and RL environments whose reward pays for no work."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.9"
+authors = [{ name = "Empyrean Intelligence" }]
+dependencies = []
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Software Development :: Testing",
+]
+
+[project.optional-dependencies]
+hf = ["datasets"]
+
+[project.urls]
+Homepage = "https://github.com/Empyrean-Intelligence/gameability"
+Source = "https://github.com/Empyrean-Intelligence/gameability"
+Issues = "https://github.com/Empyrean-Intelligence/gameability/issues"
+
+[project.scripts]
+gameability = "gameability.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

{gameability-0.2.0 → gameability-0.3.1}/setup.cfg

@@ -1,4 +1,4 @@
-[egg_info]
-tag_build = 
-tag_date = 0
-
+[egg_info]
+tag_build = 
+tag_date = 0
+

{gameability-0.2.0 → gameability-0.3.1}/src/gameability/__init__.py

@@ -16,4 +16,4 @@ from .engine import Problem, audit, census, report, summarise
 from .harden import witnesses
 
 __all__ = ["Problem", "audit", "census", "summarise", "report", "witnesses"]
-__version__ = "0.2.0"
+__version__ = "0.3.1"

{gameability-0.2.0 → gameability-0.3.1}/src/gameability/cli.py

@@ -1,8 +1,9 @@
-"""Command line: gameability check <benchmark>
+"""Command line: gameability check <benchmark> | gameability probe <rl-environment>
 
     gameability check mbpp
     gameability check humaneval --limit 50
     gameability check my_benchmark.jsonl --fixes fixes.py --json report.json
+    gameability probe my_env --n 20
 """
 import argparse
 import ast
@@ -109,7 +110,8 @@ def check(a):
 
 def main(argv=None):
     ap = argparse.ArgumentParser(prog="gameability",
-                                 description="Find the problems in a code benchmark that accept broken solutions.")
+                                 description="Find the problems in a code benchmark that accept broken solutions, "
+                                             "and the RL environments whose reward pays for no work.")
     ap.add_argument("--version", action="version", version="gameability " + __version__)
     sub = ap.add_subparsers(dest="cmd", required=True)
     c = sub.add_parser("check", help="audit a benchmark")
@@ -126,9 +128,29 @@ def main(argv=None):
     c.add_argument("--show", type=int, default=5, help="print this many weak problems with the changed line")
     c.add_argument("--fixes", help="write suggested asserts to this file")
     c.add_argument("--json", help="write the full report to this file")
+    p = sub.add_parser("probe", help="check whether an RL environment's reward pays for no work")
+    p.add_argument("env", help="id of a verifiers environment installed in this Python")
+    p.add_argument("--n", type=int, default=20, help="number of items to score (default 20)")
+    p.add_argument("--args", default="{}", help="JSON keyword arguments for load_environment")
+    p.add_argument("--allow-model-calls", action="store_true",
+                   help="let rewards that use an LLM judge call your OpenAI endpoint (costs money)")
+    p.add_argument("--json", help="write the full result to this file")
     a = ap.parse_args(argv)
     if a.cmd == "check":
         return check(a)
+    if a.cmd == "probe":
+        return probe_cmd(a)
+
+
+def probe_cmd(a):
+    from .probe import probe, report as probe_report
+    res = probe(a.env, n_items=a.n, load_args=json.loads(a.args), allow_model_calls=a.allow_model_calls)
+    probe_report(res)
+    if a.json:
+        with open(a.json, "w", encoding="utf-8") as f:
+            json.dump(dict(res, version=__version__), f, indent=1, default=str)
+        print("wrote %s" % a.json)
+    return 1 if res.get("gameable") else 0
 
 
 if __name__ == "__main__":