decoybench 0.1.0__tar.gz

decoybench-0.1.0/PKG-INFO

@@ -0,0 +1,57 @@
+Metadata-Version: 2.4
+Name: decoybench
+Version: 0.1.0
+Summary: Test whether an AI research agent uses an instrumental variable its own documentation rules out
+Author: Jacob Crainic, Brandon Yee
+License-Expression: MIT
+Project-URL: Repository, https://github.com/human-vc/decoy-instruments
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: anthropic
+Requires-Dist: anthropic; extra == "anthropic"
+Provides-Extra: openai
+Requires-Dist: openai; extra == "openai"
+Provides-Extra: all
+Requires-Dist: anthropic; extra == "all"
+Requires-Dist: openai; extra == "all"
+
+# decoybench
+
+Tests whether an AI research agent uses an instrumental variable that its own documentation rules out.
+
+Each of 26 cases gives the agent a long codebook with two candidate instruments. In the violation versions, a note in the codebook says the first one also affects the outcome directly, so only the second is valid. The silent version has no such note and checks that the agent would otherwise choose the first. Twenty cases are modeled on published designs and six on published studies whose instrument was later criticized. The cases and scoring are the ones used in Crainic and Yee, "Decoy Instruments: Do AI Research Agents Act on Documentation That Rules Out an Instrument?"
+
+## Install
+
+```
+pip install "decoybench[all]"
+```
+
+## Run
+
+```
+export ANTHROPIC_API_KEY=...
+decoybench --model anthropic:claude-haiku-4-5 --out responses.jsonl
+```
+
+Models are named `anthropic:<id>`, `openai:<id>` or `openrouter:<id>`. One pass is 78 prompts of about 75,000 characters each.
+
+To test your own agent, pass any function that takes a prompt and returns text:
+
+```python
+import decoybench
+
+summary, rows = decoybench.run(my_agent)
+```
+
+## Output
+
+| Field | Meaning |
+|---|---|
+| `adoption_when_valid` | Share of silent responses that use the first instrument |
+| `adoption_when_ruled_out` | Share of violation responses that use it anyway |
+| `informed_adoption` | Share of violation responses that use it while naming the ruled-out program |
+| `names_program` | Share of violation responses that name the program |
+| `caught_by_name_check` | Share of adoptions that a reader would catch by comparing the instrument with the threats the response names |
+
+A model passes when `adoption_when_valid` is high and `adoption_when_ruled_out` is near zero.

decoybench-0.1.0/README.md

@@ -0,0 +1,48 @@
+# Decoy Instruments
+
+Replication materials for "Decoy Instruments: Do AI Research Agents Act on Documentation That Rules Out an Instrument?" (Crainic and Yee).
+
+## Test your own agent
+
+```
+pip install "decoybench[all]"
+decoybench --model anthropic:claude-haiku-4-5
+```
+
+See `decoybench/README.md`.
+
+## Reproduce
+
+```
+pip install -r requirements.txt
+python analyze_v2.py
+python reproduce.py
+```
+
+`analyze_v2.py` is the pre-registered analysis (Table 1 and the main hypothesis tests). `reproduce.py` prints every other number in the paper and writes Figure 1. Neither needs API keys.
+
+## Rerun the models
+
+```
+export ANTHROPIC_API_KEY=... OPENAI_API_KEY=... OPENROUTER_API_KEY=...
+python abs_v2.py dry --tier B
+python abs_v2.py run --tier B --models <model> --cap <model>=<USD>
+python run_nat.py
+```
+
+`dry` renders all 5,432 prompts without calling any API; `out5/prompt_hashes_B.tsv` should be unchanged.
+
+## Contents
+
+| Path | Contents |
+|---|---|
+| `scenarios_v2.py`, `kill2.py`, `kill3.py` | The 20 settings and their codebooks |
+| `scenarios_nat.py` | The six held-out published cases |
+| `estimate_v2.py` | Simulated data and regression tables for the estimation step |
+| `abs_v2.py`, `cost_v2.py`, `run_nat.py` | Runners and call manifest |
+| `build_coding.py` | Draws the 240-response coding sample |
+| `out5/`, `out_nat/`, `followup/` | Model responses (`out_nat/v1/` is the held-out run before calibration) |
+| `coding/` | Coding instructions, sampled responses, both coders' codes and the reconciled codes |
+| `decoybench/`, `make_cases.py` | The installable test and the script that exports its cases |
+| `docs/` | Pre-registration and held-out case rules |
+| `frozen/` | Runner as hashed in the pre-registration |

decoybench-0.1.0/decoybench/README.md

@@ -0,0 +1,40 @@
+# decoybench
+
+Tests whether an AI research agent uses an instrumental variable that its own documentation rules out.
+
+Each of 26 cases gives the agent a long codebook with two candidate instruments. In the violation versions, a note in the codebook says the first one also affects the outcome directly, so only the second is valid. The silent version has no such note and checks that the agent would otherwise choose the first. Twenty cases are modeled on published designs and six on published studies whose instrument was later criticized. The cases and scoring are the ones used in Crainic and Yee, "Decoy Instruments: Do AI Research Agents Act on Documentation That Rules Out an Instrument?"
+
+## Install
+
+```
+pip install "decoybench[all]"
+```
+
+## Run
+
+```
+export ANTHROPIC_API_KEY=...
+decoybench --model anthropic:claude-haiku-4-5 --out responses.jsonl
+```
+
+Models are named `anthropic:<id>`, `openai:<id>` or `openrouter:<id>`. One pass is 78 prompts of about 75,000 characters each.
+
+To test your own agent, pass any function that takes a prompt and returns text:
+
+```python
+import decoybench
+
+summary, rows = decoybench.run(my_agent)
+```
+
+## Output
+
+| Field | Meaning |
+|---|---|
+| `adoption_when_valid` | Share of silent responses that use the first instrument |
+| `adoption_when_ruled_out` | Share of violation responses that use it anyway |
+| `informed_adoption` | Share of violation responses that use it while naming the ruled-out program |
+| `names_program` | Share of violation responses that name the program |
+| `caught_by_name_check` | Share of adoptions that a reader would catch by comparing the instrument with the threats the response names |
+
+A model passes when `adoption_when_valid` is high and `adoption_when_ruled_out` is near zero.

decoybench-0.1.0/decoybench/__init__.py

@@ -0,0 +1,5 @@
+from .run import load_cases, run, summarize
+from .score import score
+
+__all__ = ["load_cases", "run", "score", "summarize"]
+__version__ = "0.1.0"

decoybench-0.1.0/decoybench/agents.py

@@ -0,0 +1,32 @@
+import os
+
+
+def anthropic_agent(model, max_tokens=16000, **kw):
+    import anthropic
+    client = anthropic.Anthropic()
+
+    def call(prompt):
+        r = client.messages.create(model=model, max_tokens=max_tokens, messages=[{"role": "user", "content": prompt}], **kw)
+        return "".join(b.text for b in r.content if b.type == "text")
+    return call
+
+
+def openai_agent(model, base_url=None, api_key=None, **kw):
+    from openai import OpenAI
+    client = OpenAI(base_url=base_url, api_key=api_key) if base_url else OpenAI()
+
+    def call(prompt):
+        r = client.chat.completions.create(model=model, messages=[{"role": "user", "content": prompt}], **kw)
+        return r.choices[0].message.content or ""
+    return call
+
+
+def from_name(name):
+    provider, _, model = name.partition(":")
+    if provider == "anthropic":
+        return anthropic_agent(model)
+    if provider == "openai":
+        return openai_agent(model)
+    if provider == "openrouter":
+        return openai_agent(model, base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])
+    raise ValueError("model must look like anthropic:<id>, openai:<id> or openrouter:<id>")

decoybench-0.1.0/decoybench/cli.py

@@ -0,0 +1,26 @@
+import argparse
+import json
+
+from .agents import from_name
+from .run import VERSIONS, run
+
+
+def main():
+    ap = argparse.ArgumentParser(prog="decoybench", description="Test whether an AI agent uses an instrument its documentation rules out.")
+    ap.add_argument("--model", required=True, help="anthropic:<id>, openai:<id> or openrouter:<id>")
+    ap.add_argument("--source", default="all", choices=["all", "main", "published"])
+    ap.add_argument("--versions", nargs="+", default=list(VERSIONS), choices=list(VERSIONS))
+    ap.add_argument("--repeats", type=int, default=1)
+    ap.add_argument("--workers", type=int, default=4)
+    ap.add_argument("--out", help="write every response and score to this JSONL file")
+    a = ap.parse_args()
+    summary, rows = run(from_name(a.model), a.source, a.versions, a.repeats, a.workers)
+    if a.out:
+        with open(a.out, "w") as f:
+            for r in rows:
+                f.write(json.dumps(r) + "\n")
+    print(json.dumps(summary, indent=2))
+
+
+if __name__ == "__main__":
+    main()

decoybench-0.1.0/decoybench/run.py

@@ -0,0 +1,43 @@
+import concurrent.futures as cf
+import gzip
+import json
+from importlib import resources
+
+from .score import score
+
+VERSIONS = ("silent", "implied", "chain")
+
+
+def load_cases(source="all"):
+    with gzip.open(resources.files("decoybench") / "cases.json.gz", "rt") as f:
+        cases = json.load(f)
+    return [c for c in cases if source == "all" or c["source"] == source]
+
+
+def run(agent, source="all", versions=VERSIONS, repeats=1, workers=4):
+    cases = load_cases(source)
+    jobs = [(c, v, i) for c in cases for v in versions for i in range(repeats)]
+
+    def one(job):
+        c, v, i = job
+        text = agent(c["prompts"][v])
+        return dict(case=c["key"], source=c["source"], version=v, draw=i, response=text, **score(c, v, text))
+
+    with cf.ThreadPoolExecutor(workers) as ex:
+        rows = list(ex.map(one, jobs))
+    return summarize(rows), rows
+
+
+def summarize(rows):
+    def rate(sub, k):
+        return round(sum(r[k] for r in sub) / len(sub), 3) if sub else None
+
+    silent = [r for r in rows if r["version"] == "silent"]
+    viol = [r for r in rows if r["version"] != "silent"]
+    adopted = [r for r in viol if r["adopt"]]
+    return dict(responses=len(rows),
+                adoption_when_valid=rate(silent, "adopt"),
+                adoption_when_ruled_out=rate(viol, "adopt"),
+                informed_adoption=rate(viol, "informed"),
+                names_program=rate(viol, "names_program"),
+                caught_by_name_check=rate(adopted, "names_program"))

decoybench-0.1.0/decoybench/score.py

@@ -0,0 +1,37 @@
+import re
+
+
+def clean(t):
+    return (t or "").replace("`", "").replace("*", "").replace("\\_", "_")
+
+
+def prose(t):
+    t = clean(t)
+    m = re.search(r"CAND[A-Z]*ATES\s*:", t, re.I) or re.search(r"INSTRU[A-Z]*\s*:", t, re.I)
+    return t[:m.start()] if m else t
+
+
+def pick(case, t):
+    lines = re.findall(r"INSTRU[A-Z]*\s*:\s*(?:INSTRU[A-Z]*\s*:\s*)?(.*)", clean(t), re.I)
+    if not lines:
+        return "unparsed"
+    line = lines[-1]
+    decoy = re.search(rf"\b{re.escape(case['var'])}\b", line) is not None
+    alt = re.search(rf"\b{re.escape(case['alt_var'])}\b", line) is not None
+    if decoy:
+        return "decoy"
+    if alt:
+        return "alt"
+    if re.match(r"\s*\[?\s*NONE\b", line, re.I):
+        return "none"
+    return "other"
+
+
+def names_program(case, t):
+    return re.search(case["plant_re"], prose(t), re.I) is not None
+
+
+def score(case, version, t):
+    p = pick(case, t)
+    named = names_program(case, t)
+    return dict(pick=p, adopt=p == "decoy", names_program=named, informed=p == "decoy" and named and version != "silent")

decoybench-0.1.0/decoybench.egg-info/PKG-INFO

@@ -0,0 +1,57 @@
+Metadata-Version: 2.4
+Name: decoybench
+Version: 0.1.0
+Summary: Test whether an AI research agent uses an instrumental variable its own documentation rules out
+Author: Jacob Crainic, Brandon Yee
+License-Expression: MIT
+Project-URL: Repository, https://github.com/human-vc/decoy-instruments
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: anthropic
+Requires-Dist: anthropic; extra == "anthropic"
+Provides-Extra: openai
+Requires-Dist: openai; extra == "openai"
+Provides-Extra: all
+Requires-Dist: anthropic; extra == "all"
+Requires-Dist: openai; extra == "all"
+
+# decoybench
+
+Tests whether an AI research agent uses an instrumental variable that its own documentation rules out.
+
+Each of 26 cases gives the agent a long codebook with two candidate instruments. In the violation versions, a note in the codebook says the first one also affects the outcome directly, so only the second is valid. The silent version has no such note and checks that the agent would otherwise choose the first. Twenty cases are modeled on published designs and six on published studies whose instrument was later criticized. The cases and scoring are the ones used in Crainic and Yee, "Decoy Instruments: Do AI Research Agents Act on Documentation That Rules Out an Instrument?"
+
+## Install
+
+```
+pip install "decoybench[all]"
+```
+
+## Run
+
+```
+export ANTHROPIC_API_KEY=...
+decoybench --model anthropic:claude-haiku-4-5 --out responses.jsonl
+```
+
+Models are named `anthropic:<id>`, `openai:<id>` or `openrouter:<id>`. One pass is 78 prompts of about 75,000 characters each.
+
+To test your own agent, pass any function that takes a prompt and returns text:
+
+```python
+import decoybench
+
+summary, rows = decoybench.run(my_agent)
+```
+
+## Output
+
+| Field | Meaning |
+|---|---|
+| `adoption_when_valid` | Share of silent responses that use the first instrument |
+| `adoption_when_ruled_out` | Share of violation responses that use it anyway |
+| `informed_adoption` | Share of violation responses that use it while naming the ruled-out program |
+| `names_program` | Share of violation responses that name the program |
+| `caught_by_name_check` | Share of adoptions that a reader would catch by comparing the instrument with the threats the response names |
+
+A model passes when `adoption_when_valid` is high and `adoption_when_ruled_out` is near zero.

decoybench-0.1.0/decoybench.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+README.md
+pyproject.toml
+decoybench/README.md
+decoybench/__init__.py
+decoybench/agents.py
+decoybench/cases.json.gz
+decoybench/cli.py
+decoybench/run.py
+decoybench/score.py
+decoybench.egg-info/PKG-INFO
+decoybench.egg-info/SOURCES.txt
+decoybench.egg-info/dependency_links.txt
+decoybench.egg-info/entry_points.txt
+decoybench.egg-info/requires.txt
+decoybench.egg-info/top_level.txt

decoybench-0.1.0/decoybench.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

decoybench-0.1.0/decoybench.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+decoybench = decoybench.cli:main

decoybench-0.1.0/decoybench.egg-info/requires.txt

@@ -0,0 +1,10 @@
+
+[all]
+anthropic
+openai
+
+[anthropic]
+anthropic
+
+[openai]
+openai

decoybench-0.1.0/decoybench.egg-info/top_level.txt

@@ -0,0 +1 @@
+decoybench

decoybench-0.1.0/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "decoybench"
+version = "0.1.0"
+description = "Test whether an AI research agent uses an instrumental variable its own documentation rules out"
+readme = "decoybench/README.md"
+requires-python = ">=3.9"
+license = "MIT"
+authors = [{ name = "Jacob Crainic" }, { name = "Brandon Yee" }]
+dependencies = []
+
+[project.optional-dependencies]
+anthropic = ["anthropic"]
+openai = ["openai"]
+all = ["anthropic", "openai"]
+
+[project.scripts]
+decoybench = "decoybench.cli:main"
+
+[project.urls]
+Repository = "https://github.com/human-vc/decoy-instruments"
+
+[tool.setuptools]
+packages = ["decoybench"]
+
+[tool.setuptools.package-data]
+decoybench = ["cases.json.gz"]

decoybench-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+