gameability 0.2.0__tar.gz

gameability-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Empyrean Intelligence
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gameability-0.2.0/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: gameability
+Version: 0.2.0
+Summary: Find the problems in a code benchmark that accept broken solutions.
+Author: Empyrean Intelligence
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Empyrean-Intelligence/gameability
+Project-URL: Source, https://github.com/Empyrean-Intelligence/gameability
+Project-URL: Issues, https://github.com/Empyrean-Intelligence/gameability/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: hf
+Requires-Dist: datasets; extra == "hf"
+Dynamic: license-file
+
+# gameability
+
+**Find the problems in a code benchmark that accept broken solutions.**
+
+When a model learns from tests, the tests are its reward. A self-improving system trained on tests that
+accept broken code learns to pass weak tests, not to write correct code. `gameability` measures how often
+that can happen. It is part of the verification layer we use at Empyrean Intelligence for self-improvement
+work, released for anyone who builds benchmarks or RL training sets.
+
+A benchmark score means something only if the benchmark rejects solutions that lack the
+skill it measures. `gameability` checks that directly. It takes each problem's reference
+solution, breaks it in one small place, confirms the result really is wrong, and then
+asks whether the problem's own tests notice.
+
+```
+pip install "gameability[hf]"
+gameability check mbpp
+```
+
+On MBPP, **17.9% of problems (169 of 942 judged) accept a broken solution**; on HumanEval,
+**10.3% (16 of 155)**. Each run takes minutes on a laptop, with no model and no API key.
+
+**Confirmed on valid inputs.** On the problems EvalPlus also covers, every accepted broken solution
+was re-run against EvalPlus's independent tests, whose inputs respect each problem's rules:
+**95.2% of flagged MBPP problems and 84.6% of flagged HumanEval problems are confirmed** (the broken
+solution passes the original tests and fails EvalPlus). The two unconfirmed HumanEval problems
+(116 and 129) are ones whose own visible tests contradict their written specification.
+
+**What changed in v0.2.** v0.1 built fresh inputs by changing values freely, which sometimes asked
+a problem about inputs it never promised to handle (an unsorted list for binary search, random
+letters for a parenthesis parser). v0.2 learns each argument's domain from the problem's own
+visible tests and stays inside it: number signs and minimums, the characters or whole tokens a
+string uses, and whether lists are sorted, unique, rectangular or a permutation. On the same
+problems this raised the confirmed share from 85.7% to 95.2% (MBPP) and 78.6% to 84.6% (HumanEval)
+while keeping 93.1% of v0.1's confirmed detections. `--inputs free` reproduces v0.1.
+
+## Scoreboard
+
+Problems that accept at least one broken solution (wrong on at least one fresh input), with 95% intervals. Defaults (24 broken
+versions per problem, 8 fresh inputs, seed 0), laptop CPU, October 2026.
+
+| Benchmark | Weak problems | Command |
+|---|---|---|
+| MBPP | 17.9% (169 of 942) [15.6, 20.5] | `gameability check mbpp` |
+| MBPP sanitized | 13.8% (57 of 413) [10.8, 17.5] | `gameability check mbpp-sanitized` |
+| HumanEval | 10.3% (16 of 155) [6.5, 16.1] | `gameability check humaneval` |
+| KodCode (RL training set, 1,000-row random sample) | 9.7% (45 of 466) [7.3, 12.7] | `gameability check kodcode:rows.jsonl` |
+| HumanEval+ (EvalPlus) | 3.4% (5 of 147) [1.5, 7.7] | `gameability check humaneval-plus` |
+| MBPP+ (EvalPlus) | 2.0% (7 of 353) [1.0, 4.0] | `gameability check mbpp-plus` |
+
+EvalPlus's stronger tests look like a large improvement, but EvalPlus also rewrote most reference
+solutions. Holding the original reference fixed (`mbpp-plus-origref`, `humaneval-plus-origref`), the weak
+share roughly halves (MBPP 8.7% to 3.1% on 254 problems, HumanEval 10.6% to 4.9% on 123), and at these
+sample sizes the intervals overlap. KodCode's judged set is the problems whose tests call the function with
+literal arguments (466 of 801 loadable). `rows.jsonl` is KodCode rows saved as JSON lines; the sample above
+is reproducible with `python tests/sample_kodcode.py rows.jsonl` (positions from `random.Random(0)`).
+
+## What it prints
+
+```
+  415 -- 6 of 11 broken solutions pass every visible test
+      -    if arr_len < 2:
+      +    if arr_len <= 2:
+      a test that catches it: assert max_Product([4, 9]) == (4, 9)
+```
+
+For every weak problem you get the one line that was changed, how many broken versions got
+through, and an assert that would have caught one.
+
+**`--fixes fixes.py` writes those asserts to a file; review them before adding them.** Measured on
+34 MBPP/HumanEval problems against a 1.5B model's real solutions: v0.2's suggested asserts wrongly
+reject **2.6% of its correct solutions** (8 of 307; v0.1: 15.3%). They target the broken versions
+gameability builds, not the mistakes a particular model makes: they caught 4 of that model's 24
+real wrong solutions. Treat them as leads, not as a test suite.
+
+## How it works
+
+For each problem the benchmark supplies (reference solution, visible tests):
+
+1. The reference must pass its own visible tests. If it doesn't, the problem is excluded and counted.
+2. Fresh inputs are built by varying the arguments in the visible tests. The reference's
+   outputs on them are the hidden answer key.
+3. The reference is mutated at a single site: a comparison flipped, an operator swapped, a
+   constant nudged, a `return` replaced with `return None`.
+4. A mutant counts as **broken** only if it gives a different answer from the reference on
+   at least one fresh input. Mutants that behave identically are thrown away, so they can't
+   inflate the number.
+5. A broken mutant that still passes every visible test is a **survivor**. A problem with at
+   least one survivor accepts a wrong solution.
+
+Benchmark code runs in a child process with a step budget, so a runaway mutant is stopped at
+the same point on any machine; a wall-clock limit is only a backstop. Mutants that never
+finish are reported as truncated, never counted as rejected.
+
+## Your own benchmark
+
+One problem per line in a `.jsonl` file:
+
+```json
+{"id": "clamp", "reference": "def clamp(x, lo, hi): ...", "tests": ["assert clamp(5, 0, 10) == 5"]}
+```
+
+Optional fields: `entry` (the function name, if it can't be inferred), `setup` (code run
+before the tests), `hidden` (held-out asserts you already have; they are used as extra
+answer-key inputs), `call_alias` (the name the tests call the solution by, e.g. `candidate`).
+
+```
+gameability check examples/tiny.jsonl --fixes fixes.py --json report.json
+```
+
+## What the number means, and what it doesn't
+
+- **The attacker is weak.** One small edit is the simplest possible way to be wrong; a model trained
+  against these tests can search far more. In that sense the rates are a floor.
+- **Fresh inputs can still break a problem's rules in ways the tests don't reveal.** v0.2 keeps each
+  argument inside its observed domain, but it does not model relationships BETWEEN arguments (a
+  length argument that must equal a list's length) or rules that only the docstring states. On the
+  problems EvalPlus covers, 4.8% of flagged MBPP and 15.4% of flagged HumanEval problems are still
+  unconfirmed (v0.1: 14.3% and 21.4%).
+- **It does not predict where a model will cheat.** On a 1.5B coder, false passes were no more
+  common on flagged problems than on others (8.4% vs 10.0%). It audits the tests' strength, not a
+  model's behaviour.
+- **It audits the tests, not any model.** Nothing here says a particular model cheated. It says
+  which problems would not have caught it.
+- **It runs the benchmark's code.** Reference solutions, mutants and tests are executed, each job in
+  its own throwaway directory (some benchmark tasks write files; v0.1 let them land in your working
+  folder). That is isolation from your files, not a security sandbox: only run it on benchmarks
+  you'd be willing to execute.
+
+## Reproducing the headline numbers
+
+```
+gameability check mbpp --json mbpp.json
+gameability check humaneval --json humaneval.json
+```
+
+Defaults: 24 mutants per problem, 8 fresh inputs, seed 0, contract inputs. Add `--inputs free` to
+reproduce v0.1's numbers (MBPP 18.5%, HumanEval 11.8%). The known-answer tests run with
+`python tests/test_gameability.py` and `python tests/test_contract_inputs.py`.
+
+## License
+
+MIT

gameability-0.2.0/README.md

@@ -0,0 +1,143 @@
+# gameability
+
+**Find the problems in a code benchmark that accept broken solutions.**
+
+When a model learns from tests, the tests are its reward. A self-improving system trained on tests that
+accept broken code learns to pass weak tests, not to write correct code. `gameability` measures how often
+that can happen. It is part of the verification layer we use at Empyrean Intelligence for self-improvement
+work, released for anyone who builds benchmarks or RL training sets.
+
+A benchmark score means something only if the benchmark rejects solutions that lack the
+skill it measures. `gameability` checks that directly. It takes each problem's reference
+solution, breaks it in one small place, confirms the result really is wrong, and then
+asks whether the problem's own tests notice.
+
+```
+pip install "gameability[hf]"
+gameability check mbpp
+```
+
+On MBPP, **17.9% of problems (169 of 942 judged) accept a broken solution**; on HumanEval,
+**10.3% (16 of 155)**. Each run takes minutes on a laptop, with no model and no API key.
+
+**Confirmed on valid inputs.** On the problems EvalPlus also covers, every accepted broken solution
+was re-run against EvalPlus's independent tests, whose inputs respect each problem's rules:
+**95.2% of flagged MBPP problems and 84.6% of flagged HumanEval problems are confirmed** (the broken
+solution passes the original tests and fails EvalPlus). The two unconfirmed HumanEval problems
+(116 and 129) are ones whose own visible tests contradict their written specification.
+
+**What changed in v0.2.** v0.1 built fresh inputs by changing values freely, which sometimes asked
+a problem about inputs it never promised to handle (an unsorted list for binary search, random
+letters for a parenthesis parser). v0.2 learns each argument's domain from the problem's own
+visible tests and stays inside it: number signs and minimums, the characters or whole tokens a
+string uses, and whether lists are sorted, unique, rectangular or a permutation. On the same
+problems this raised the confirmed share from 85.7% to 95.2% (MBPP) and 78.6% to 84.6% (HumanEval)
+while keeping 93.1% of v0.1's confirmed detections. `--inputs free` reproduces v0.1.
+
+## Scoreboard
+
+Problems that accept at least one broken solution (wrong on at least one fresh input), with 95% intervals. Defaults (24 broken
+versions per problem, 8 fresh inputs, seed 0), laptop CPU, October 2026.
+
+| Benchmark | Weak problems | Command |
+|---|---|---|
+| MBPP | 17.9% (169 of 942) [15.6, 20.5] | `gameability check mbpp` |
+| MBPP sanitized | 13.8% (57 of 413) [10.8, 17.5] | `gameability check mbpp-sanitized` |
+| HumanEval | 10.3% (16 of 155) [6.5, 16.1] | `gameability check humaneval` |
+| KodCode (RL training set, 1,000-row random sample) | 9.7% (45 of 466) [7.3, 12.7] | `gameability check kodcode:rows.jsonl` |
+| HumanEval+ (EvalPlus) | 3.4% (5 of 147) [1.5, 7.7] | `gameability check humaneval-plus` |
+| MBPP+ (EvalPlus) | 2.0% (7 of 353) [1.0, 4.0] | `gameability check mbpp-plus` |
+
+EvalPlus's stronger tests look like a large improvement, but EvalPlus also rewrote most reference
+solutions. Holding the original reference fixed (`mbpp-plus-origref`, `humaneval-plus-origref`), the weak
+share roughly halves (MBPP 8.7% to 3.1% on 254 problems, HumanEval 10.6% to 4.9% on 123), and at these
+sample sizes the intervals overlap. KodCode's judged set is the problems whose tests call the function with
+literal arguments (466 of 801 loadable). `rows.jsonl` is KodCode rows saved as JSON lines; the sample above
+is reproducible with `python tests/sample_kodcode.py rows.jsonl` (positions from `random.Random(0)`).
+
+## What it prints
+
+```
+  415 -- 6 of 11 broken solutions pass every visible test
+      -    if arr_len < 2:
+      +    if arr_len <= 2:
+      a test that catches it: assert max_Product([4, 9]) == (4, 9)
+```
+
+For every weak problem you get the one line that was changed, how many broken versions got
+through, and an assert that would have caught one.
+
+**`--fixes fixes.py` writes those asserts to a file; review them before adding them.** Measured on
+34 MBPP/HumanEval problems against a 1.5B model's real solutions: v0.2's suggested asserts wrongly
+reject **2.6% of its correct solutions** (8 of 307; v0.1: 15.3%). They target the broken versions
+gameability builds, not the mistakes a particular model makes: they caught 4 of that model's 24
+real wrong solutions. Treat them as leads, not as a test suite.
+
+## How it works
+
+For each problem the benchmark supplies (reference solution, visible tests):
+
+1. The reference must pass its own visible tests. If it doesn't, the problem is excluded and counted.
+2. Fresh inputs are built by varying the arguments in the visible tests. The reference's
+   outputs on them are the hidden answer key.
+3. The reference is mutated at a single site: a comparison flipped, an operator swapped, a
+   constant nudged, a `return` replaced with `return None`.
+4. A mutant counts as **broken** only if it gives a different answer from the reference on
+   at least one fresh input. Mutants that behave identically are thrown away, so they can't
+   inflate the number.
+5. A broken mutant that still passes every visible test is a **survivor**. A problem with at
+   least one survivor accepts a wrong solution.
+
+Benchmark code runs in a child process with a step budget, so a runaway mutant is stopped at
+the same point on any machine; a wall-clock limit is only a backstop. Mutants that never
+finish are reported as truncated, never counted as rejected.
+
+## Your own benchmark
+
+One problem per line in a `.jsonl` file:
+
+```json
+{"id": "clamp", "reference": "def clamp(x, lo, hi): ...", "tests": ["assert clamp(5, 0, 10) == 5"]}
+```
+
+Optional fields: `entry` (the function name, if it can't be inferred), `setup` (code run
+before the tests), `hidden` (held-out asserts you already have; they are used as extra
+answer-key inputs), `call_alias` (the name the tests call the solution by, e.g. `candidate`).
+
+```
+gameability check examples/tiny.jsonl --fixes fixes.py --json report.json
+```
+
+## What the number means, and what it doesn't
+
+- **The attacker is weak.** One small edit is the simplest possible way to be wrong; a model trained
+  against these tests can search far more. In that sense the rates are a floor.
+- **Fresh inputs can still break a problem's rules in ways the tests don't reveal.** v0.2 keeps each
+  argument inside its observed domain, but it does not model relationships BETWEEN arguments (a
+  length argument that must equal a list's length) or rules that only the docstring states. On the
+  problems EvalPlus covers, 4.8% of flagged MBPP and 15.4% of flagged HumanEval problems are still
+  unconfirmed (v0.1: 14.3% and 21.4%).
+- **It does not predict where a model will cheat.** On a 1.5B coder, false passes were no more
+  common on flagged problems than on others (8.4% vs 10.0%). It audits the tests' strength, not a
+  model's behaviour.
+- **It audits the tests, not any model.** Nothing here says a particular model cheated. It says
+  which problems would not have caught it.
+- **It runs the benchmark's code.** Reference solutions, mutants and tests are executed, each job in
+  its own throwaway directory (some benchmark tasks write files; v0.1 let them land in your working
+  folder). That is isolation from your files, not a security sandbox: only run it on benchmarks
+  you'd be willing to execute.
+
+## Reproducing the headline numbers
+
+```
+gameability check mbpp --json mbpp.json
+gameability check humaneval --json humaneval.json
+```
+
+Defaults: 24 mutants per problem, 8 fresh inputs, seed 0, contract inputs. Add `--inputs free` to
+reproduce v0.1's numbers (MBPP 18.5%, HumanEval 11.8%). The known-answer tests run with
+`python tests/test_gameability.py` and `python tests/test_contract_inputs.py`.
+
+## License
+
+MIT

gameability-0.2.0/pyproject.toml

@@ -0,0 +1,33 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "gameability"
+version = "0.2.0"
+description = "Find the problems in a code benchmark that accept broken solutions."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.9"
+authors = [{ name = "Empyrean Intelligence" }]
+dependencies = []
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Software Development :: Testing",
+]
+
+[project.optional-dependencies]
+hf = ["datasets"]
+
+[project.urls]
+Homepage = "https://github.com/Empyrean-Intelligence/gameability"
+Source = "https://github.com/Empyrean-Intelligence/gameability"
+Issues = "https://github.com/Empyrean-Intelligence/gameability/issues"
+
+[project.scripts]
+gameability = "gameability.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

gameability-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

gameability-0.2.0/src/gameability/__init__.py

@@ -0,0 +1,19 @@
+"""gameability -- find the problems in a code benchmark that accept broken solutions.
+
+A benchmark is only as strong as its ability to REJECT a solution that provably
+lacks the capability it claims to measure. This package manufactures such
+solutions, proves they are genuinely broken against inputs the benchmark never
+shows, and reports what fraction the benchmark accepts anyway.
+
+    from gameability import Problem, census, summarise, report
+
+    rows = census([Problem("p1", reference_src, ["assert f(1) == 2"])])
+    report(summarise(rows, "my benchmark"))
+
+The only thing a benchmark must supply is (reference solution, visible tests).
+"""
+from .engine import Problem, audit, census, report, summarise
+from .harden import witnesses
+
+__all__ = ["Problem", "audit", "census", "summarise", "report", "witnesses"]
+__version__ = "0.2.0"

gameability-0.2.0/src/gameability/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from .cli import main
+
+sys.exit(main())

gameability-0.2.0/src/gameability/_worker.py

@@ -0,0 +1,87 @@
+"""Child process: execute one problem's reference and mutants in isolation.
+
+Results are emitted as one JSON line per candidate and flushed immediately, so
+a candidate that fails to terminate costs only the candidates after it rather
+than the whole problem. The parent records the shortfall as `truncated` instead
+of silently treating unreached mutants as killed, which would bias the headline
+number downward -- toward the flattering answer.
+"""
+import json
+import os
+import sys
+
+STEP_BUDGET = 200_000
+
+
+class _Budget:
+    """Abort runaway code by counting traced events. Windows has no SIGALRM."""
+
+    def __init__(self, limit):
+        self.n, self.limit = 0, limit
+
+    def __call__(self, frame, event, arg):
+        self.n += 1
+        if self.n > self.limit:
+            raise TimeoutError("step budget exceeded")
+        return self
+
+
+def outcome(ns, expr):
+    """('ok', repr) or ('exc', type) -- repr avoids trusting a custom __eq__."""
+    sys.settrace(_Budget(STEP_BUDGET))
+    try:
+        return ["ok", repr(eval(expr, ns))]
+    except BaseException as e:
+        return ["exc", type(e).__name__]
+    finally:
+        sys.settrace(None)
+
+
+def run_asserts(ns, tests):
+    for t in tests:
+        sys.settrace(_Budget(STEP_BUDGET))
+        try:
+            exec(t, ns)
+        except BaseException:
+            return False
+        finally:
+            sys.settrace(None)
+    return True
+
+
+def evaluate(src, setup, tests, fn, fresh):
+    ns = {"__name__": "__candidate__"}
+    try:
+        if setup:
+            exec(setup, ns)
+        exec(src, ns)
+    except BaseException as e:
+        return {"loaded": False, "error": type(e).__name__}
+    if fn not in ns:
+        return {"loaded": False, "error": "entry point %r not defined" % fn}
+    return {
+        "loaded": True,
+        "visible": run_asserts(dict(ns), tests),
+        "hidden": [outcome(dict(ns), "%s(%s)" % (fn, a)) for a in fresh],
+    }
+
+
+def main():
+    job = json.loads(sys.stdin.read())
+    setup, tests = job.get("setup", ""), job["tests"]
+    fn, fresh = job["fn"], job["fresh"]
+    # Benchmark code prints. Its output must never reach the result stream, where a
+    # printed number or dict would be read as a record (2026-10-17: a KodCode test
+    # printed an int and crashed the reader). Results go to the real stdout only.
+    out = sys.stdout
+    sys.stdout = open(os.devnull, "w")
+    for tag, src in [("ref", job["ref"])] + [
+            ("m%d" % i, m) for i, m in enumerate(job["mutants"])]:
+        rec = evaluate(src, setup, tests, fn, fresh)
+        rec["tag"] = tag
+        out.write(json.dumps(rec) + "\n")
+        out.flush()
+
+
+if __name__ == "__main__":
+    main()

gameability-0.2.0/src/gameability/cli.py

@@ -0,0 +1,135 @@
+"""Command line: gameability check <benchmark>
+
+    gameability check mbpp
+    gameability check humaneval --limit 50
+    gameability check my_benchmark.jsonl --fixes fixes.py --json report.json
+"""
+import argparse
+import ast
+import difflib
+import json
+import os
+import random
+import sys
+import time
+import warnings
+
+# Benchmark references are third-party code; old-style escapes in them ('\w')
+# make Python warn on every parse. The warnings say nothing about the audit.
+warnings.simplefilter("ignore", SyntaxWarning)
+
+from . import __version__
+from .engine import census, report, summarise
+from .loaders import SOURCES, from_jsonl, kodcode
+
+
+def _changed_lines(reference, mutant):
+    """The one line the mutation changed, before and after (normalised by ast)."""
+    try:
+        a = ast.unparse(ast.parse(reference)).splitlines()
+    except SyntaxError:
+        a = reference.splitlines()
+    b = mutant.splitlines()
+    out = []
+    for line in difflib.unified_diff(a, b, lineterm="", n=0):
+        if line.startswith(("---", "+++", "@@")):
+            continue
+        out.append(line)
+    return out
+
+
+def _load(target):
+    if target in SOURCES:
+        return SOURCES[target]()
+    if target.startswith("kodcode:"):
+        return kodcode(target[len("kodcode:"):])
+    if os.path.isfile(target):
+        return from_jsonl(target)
+    raise SystemExit("unknown benchmark %r: use %s, or a path to a .jsonl file" % (target, ", ".join(SOURCES)))
+
+
+def check(a):
+    problems, dropped = _load(a.benchmark)
+    label = os.path.basename(a.benchmark)
+    print("%s: %d problems usable as (reference, visible tests)" % (label, len(problems)))
+    if dropped:
+        print("      %d dropped before the audit (no tests, or the tests never call a "
+              "function the reference defines)" % dropped)
+    if a.limit:
+        random.Random(a.seed).shuffle(problems)
+        problems = problems[:a.limit]
+        print("      random sample of %d, seed %d" % (len(problems), a.seed))
+    if not problems:
+        raise SystemExit("nothing to audit")
+
+    t0 = time.time()
+    rows = census(problems, workers=a.workers, seed=a.seed, n_mutants=a.mutants,
+                  timeout=a.timeout, want_witness=True, inputs=a.inputs)
+    s = summarise(rows, "%s (%d problems)" % (label, len(problems)))
+    print("\nelapsed %.0fs\n" % (time.time() - t0))
+    report(s)
+
+    by_id = {p.pid: p for p in problems}
+    weak = [r for r in rows if r["status"] == "ok" and r["n_survived"] > 0]
+    if weak and a.show:
+        print("\nPROBLEMS THAT ACCEPT A BROKEN SOLUTION (first %d of %d)" % (min(a.show, len(weak)), len(weak)))
+        for r in weak[:a.show]:
+            ex = r["examples"][0]
+            print("\n  %s -- %d of %d broken solutions pass every visible test" % (r["pid"], r["n_survived"], r["n_broken"]))
+            for line in _changed_lines(by_id[r["pid"]].reference, ex["src"]):
+                print("      " + line)
+            if r.get("witness"):
+                args, out = r["witness"][0]
+                print("      a test that catches it: assert %s(%s) == %s" % (by_id[r["pid"]].entry, args, out))
+
+    if a.fixes:
+        n = 0
+        with open(a.fixes, "w", encoding="utf-8") as f:
+            f.write("# Asserts that reject the broken solutions gameability found.\n")
+            f.write("# Expected values come from each problem's own reference solution;\n")
+            f.write("# review them before adding them to the benchmark.\n")
+            for r in weak:
+                seen = set()
+                for args, out in r.get("witness", []):
+                    t = "assert %s(%s) == %s" % (by_id[r["pid"]].entry, args, out)
+                    if t not in seen:
+                        seen.add(t)
+                        if len(seen) == 1:
+                            f.write("\n# %s\n" % r["pid"])
+                        f.write(t + "\n")
+                        n += 1
+        print("\nwrote %d suggested asserts for %d problems to %s" % (n, len(weak), a.fixes))
+
+    if a.json:
+        with open(a.json, "w", encoding="utf-8") as f:
+            json.dump({"version": __version__, "summary": s, "rows": rows}, f, indent=1)
+        print("wrote %s" % a.json)
+    return 0
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="gameability",
+                                 description="Find the problems in a code benchmark that accept broken solutions.")
+    ap.add_argument("--version", action="version", version="gameability " + __version__)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    c = sub.add_parser("check", help="audit a benchmark")
+    c.add_argument("benchmark", help="mbpp, humaneval, or a .jsonl file")
+    c.add_argument("--limit", type=int, default=0, help="audit a random sample of N problems")
+    c.add_argument("--seed", type=int, default=0)
+    c.add_argument("--workers", type=int, default=8)
+    c.add_argument("--mutants", type=int, default=24, help="broken solutions to try per problem")
+    c.add_argument("--inputs", choices=["contract", "free"], default="contract",
+                   help="contract (default): fresh inputs stay inside each argument's observed domain; "
+                        "free: v0.1's unconstrained generator, to reproduce its numbers")
+    c.add_argument("--timeout", type=int, default=40,
+                   help="wall-clock backstop per problem in seconds (the step budget is the real limit)")
+    c.add_argument("--show", type=int, default=5, help="print this many weak problems with the changed line")
+    c.add_argument("--fixes", help="write suggested asserts to this file")
+    c.add_argument("--json", help="write the full report to this file")
+    a = ap.parse_args(argv)
+    if a.cmd == "check":
+        return check(a)
+
+
+if __name__ == "__main__":
+    sys.exit(main())