aimpg 0.6.2__tar.gz → 0.7.0__tar.gz

{aimpg-0.6.2 → aimpg-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: aimpg
-Version: 0.6.2
+Version: 0.7.0
 Summary: Real-world energy per solved task for AI coding agents
 Project-URL: Homepage, https://github.com/kumarganduri/aimpg
 Project-URL: Issues, https://github.com/kumarganduri/aimpg/issues
@@ -28,6 +28,8 @@ uvx aimpg report
 
 **New here? [aimpg in 5 minutes](https://kumarganduri.github.io/aimpg-scoreboard/start/)** walks you from install to your first numbers, the live status line, and your team's GitHub Action, with copy buttons for every command.
 
+**Using it as a GitHub Action?** Add `uses: kumarganduri/aimpg@v0.6.3` to a workflow and every pull request shows its AI cost, and every week your team's cost per working change. Copy the workflow from [Team numbers, without a server](#6-team-numbers-without-a-server).
+
 ```
 YOUR AI CODING
   Energy   17.31 kWh – 156.95 kWh
@@ -153,7 +155,7 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with: {fetch-depth: 0}
-      - uses: kumarganduri/aimpg@v0.6.2   # or pin a commit SHA: it runs with your repo token
+      - uses: kumarganduri/aimpg@v0.6.3   # or pin a commit SHA: it runs with your repo token
         with:
           mode: ${{ github.event_name == 'pull_request' && 'pr' || 'rollup' }}
           comment: "true"
@@ -193,6 +195,7 @@ Questions people ask (is it sponsored, are the energy numbers guesses, is it saf
 
 - Claude Code and Codex CLI logs are read automatically. Cursor keeps token usage on its servers, so it needs its usage export (below), and that export doesn't say which folder the work was in: give `--cursor-repo` and requests are matched to your next own commit there by time alone (shown as "by time only"). Replays and `verify` need macOS or Linux and an API key (about $0.10–0.30 per run). On Linux install bubblewrap (`sudo apt install bubblewrap`); Ubuntu 24.04+ also needs a one-time AppArmor rule for bwrap, which aimpg prints if it's missing.
 - Windows: `report`, `live` and the team commands are untested there; replays and `verify` need macOS or Linux.
+- Replays let code in the sandbox use localhost, because tests start local servers. On macOS that includes services already listening on your machine's localhost (a local database, for example). On Linux each run gets its own private localhost. The internet stays blocked except the model's API.
 - Energy is an estimate with a wide range (model sizes are guesses within stated bounds); dollars are close (within about 7% of Claude Code's own session totals on the author's logs).
 - Tips are upper bounds and overlap; they can't be added together.
 - Replays from commit messages alone are hard (12% solved on the author's repo); `--task-mode tests` shows the agent the tests, which makes tasks easier than real work but keeps comparisons fair.

{aimpg-0.6.2 → aimpg-0.7.0}/README.md

@@ -8,6 +8,8 @@ uvx aimpg report
 
 **New here? [aimpg in 5 minutes](https://kumarganduri.github.io/aimpg-scoreboard/start/)** walks you from install to your first numbers, the live status line, and your team's GitHub Action, with copy buttons for every command.
 
+**Using it as a GitHub Action?** Add `uses: kumarganduri/aimpg@v0.6.3` to a workflow and every pull request shows its AI cost, and every week your team's cost per working change. Copy the workflow from [Team numbers, without a server](#6-team-numbers-without-a-server).
+
 ```
 YOUR AI CODING
   Energy   17.31 kWh – 156.95 kWh
@@ -133,7 +135,7 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with: {fetch-depth: 0}
-      - uses: kumarganduri/aimpg@v0.6.2   # or pin a commit SHA: it runs with your repo token
+      - uses: kumarganduri/aimpg@v0.6.3   # or pin a commit SHA: it runs with your repo token
         with:
           mode: ${{ github.event_name == 'pull_request' && 'pr' || 'rollup' }}
           comment: "true"
@@ -173,6 +175,7 @@ Questions people ask (is it sponsored, are the energy numbers guesses, is it saf
 
 - Claude Code and Codex CLI logs are read automatically. Cursor keeps token usage on its servers, so it needs its usage export (below), and that export doesn't say which folder the work was in: give `--cursor-repo` and requests are matched to your next own commit there by time alone (shown as "by time only"). Replays and `verify` need macOS or Linux and an API key (about $0.10–0.30 per run). On Linux install bubblewrap (`sudo apt install bubblewrap`); Ubuntu 24.04+ also needs a one-time AppArmor rule for bwrap, which aimpg prints if it's missing.
 - Windows: `report`, `live` and the team commands are untested there; replays and `verify` need macOS or Linux.
+- Replays let code in the sandbox use localhost, because tests start local servers. On macOS that includes services already listening on your machine's localhost (a local database, for example). On Linux each run gets its own private localhost. The internet stays blocked except the model's API.
 - Energy is an estimate with a wide range (model sizes are guesses within stated bounds); dollars are close (within about 7% of Claude Code's own session totals on the author's logs).
 - Tips are upper bounds and overlap; they can't be added together.
 - Replays from commit messages alone are hard (12% solved on the author's repo); `--task-mode tests` shows the agent the tests, which makes tasks easier than real work but keeps comparisons fair.

{aimpg-0.6.2 → aimpg-0.7.0}/action.yml

@@ -1,4 +1,4 @@
-name: aimpg
+name: AI Coding MPG
 description: AI cost and energy of pull requests and of your team (cost per working change), from aimpg git notes
 author: Kumar Ganduri
 branding:

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/chain.py

@@ -56,6 +56,8 @@ def run_chain(c1: Commit, c2: Commit, repeat: int, layout: Layout, cfg: Config,
     if not session:
         first.outcome, first.passed, first.note = "harness_error", False, "task 1 left no session id to resume"
         return [first]
+    # keep task 1's totals: the resumed session's own totals include them
+    (run_dir / "prior-result.json").write_text((run_dir / "result.json").read_text())
     cont = run_one(c2, CONTINUE, repeat, layout, cfg, attempt=attempt, solution=solutions.get(c2.sha),
                    run_id=f"{tag}-chain-continue-r{repeat}-a{attempt}", extra_args=("--resume", session),
                    resume_dir=run_dir, price_from_tokens=True)

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/cli.py

@@ -45,6 +45,7 @@ def secure_root(root: Path | None = None) -> Path:
     return root
 STATE = Path.home() / ".aimpg" / "replay"  # selections + results, unreadable to agents
 DEFAULT_MODEL = "claude-sonnet-5-5"
+SETUP_GIVE_UP = 5  # the first N candidates all fail to install: the repo can't be set up offline
 DEFAULT_SETUPS = "claude-code,claude-code+terse,claude-code+rtk"
 
 
@@ -106,7 +107,7 @@ def _select(repo: Path, args) -> int:
     commits, skipped = select.candidates(str(repo), time.time() - args.days * DAY, cutoff, any_author=any_author)
     print(f"{len(commits)} candidates after cheap filters; skipped: {skipped or 'none'}")
     layout = Layout(secure_root())
-    good = []
+    good, setup_failures = [], 0
     with AllowlistProxy() as proxy:
         for c in commits:
             if len(good) >= args.max:
@@ -116,6 +117,11 @@ def _select(repo: Path, args) -> int:
             print(f"  {mark} {c.sha[:8]} {c.subject[:60]:<60} {'' if check.ok else check.reason}")
             if check.ok:
                 good.append(c)
+            elif check.reason.startswith("setup failed") and not good:
+                setup_failures += 1
+                if setup_failures >= SETUP_GIVE_UP:
+                    print(f"  The first {SETUP_GIVE_UP} commits all fail to install here: giving up on this repo.")
+                    break
     out = _state(repo) / "selected.jsonl"
     select.save(good, out)
     checked = commits[: commits.index(good[-1]) + 1] if good else commits

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/fake_agent.py

@@ -109,6 +109,14 @@ def other_agent(mode: str, args: list[str]) -> int:
         if mode == "nocredit":
             print('{"type":"error","message":"You exceeded your current quota (insufficient_quota)"}')
             return 1
+        if mode == "quota-in-log":  # what the real Codex did in the main run: the error only in its session log
+            folder = cfg / "sessions" / "x"
+            folder.mkdir(parents=True, exist_ok=True)
+            (folder / "rollout-q.jsonl").write_text(json.dumps({"type": "event_msg", "payload": {
+                "type": "task_complete", "error": {"message": "Quota exceeded. Check your plan and billing details.",
+                                                   "codex_error_info": "usage_limit_exceeded"}}}) + "\n")
+            print("Reading additional input from stdin...", file=sys.stderr)
+            return 1
         model = args[args.index("-m") + 1] if "-m" in args else "gpt-fake"
         codex_rollout(cfg, cwd, model, 3)
         # its output can quote error text from the code it writes: never an account error

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/hints.py

@@ -157,6 +157,11 @@ def write_stubs(hint: Hint, work, kind: str) -> None:
     """Create every hinted name with an empty value: tests must still FAIL on this."""
     for path, names in hint.names.items():
         target = work / path
+        if target.is_dir():  # `from "./src"` imports the folder's index file
+            index = [target / f"index{ext}" for ext in (".ts", ".tsx", ".mts", ".js", ".mjs", ".py")]
+            target = next((p for p in index if p.is_file()), None) or (target / "__init__.py" if kind == "python" else None)
+            if target is None:
+                continue
         target.parent.mkdir(parents=True, exist_ok=True)
         existing = target.read_text() if target.exists() else ""
         if kind == "js":

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/proxy.py

@@ -7,7 +7,8 @@ host the agent wants to reach.
 
 The allowlist changes per phase:
 
-    Phase A (prepare)   pypi.org, files.pythonhosted.org / registry.npmjs.org
+    Phase A (prepare)   pypi.org, files.pythonhosted.org / registry.npmjs.org, plus github.com
+                        and codeload.github.com for dependencies pinned to a commit
     Phase B (agent)     the agent's API only (api.anthropic.com, api.openai.com, ...)
     Phase C (judge)     nothing
 """
@@ -22,8 +23,11 @@ import threading
 from pathlib import Path
 from dataclasses import dataclass, field
 
-PYPI = frozenset({"pypi.org", "files.pythonhosted.org"})
-NPM = frozenset({"registry.npmjs.org"})
+# Lockfiles may pin dependencies to exact commits on GitHub (litestar's do), so installs
+# may also reach github.com and its tarball host. Install phase only; the agent never can.
+GIT_HOSTS = frozenset({"github.com", "codeload.github.com"})
+PYPI = frozenset({"pypi.org", "files.pythonhosted.org"}) | GIT_HOSTS
+NPM = frozenset({"registry.npmjs.org"}) | GIT_HOSTS
 ANTHROPIC = frozenset({"api.anthropic.com"})
 OPENAI = frozenset({"api.openai.com"})
 NOTHING: frozenset[str] = frozenset()

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/run.py

@@ -43,7 +43,8 @@ NOT_PASSED = ("tests_failed", "timeout", "budget_hit", "agent_error")
 _ACCOUNT_ERROR = re.compile(r"credit balance|invalid x-api-key|invalid api key|authentication_error|permission_error|billing", re.I)
 _ACCOUNT_STATUS = {401, 402, 403}  # Claude Code's result carries api_error_status
 # The same for OpenAI (Codex): no quota, bad key.
-_OPENAI_ACCOUNT_ERROR = re.compile(r"insufficient_quota|invalid_api_key|incorrect api key|401 unauthorized|exceeded your current quota", re.I)
+_OPENAI_ACCOUNT_ERROR = re.compile(r"insufficient_quota|invalid_api_key|incorrect api key|401 unauthorized|exceeded your current quota|"
+                                   r"quota exceeded|usage_limit_exceeded", re.I)
 
 
 @dataclass
@@ -55,6 +56,9 @@ class Config:
     parallel: int = 3
     timeout: float = AGENT_TIMEOUT
     task_mode: str = "hint"  # "hint": message + interface hint, tests hidden | "tests": tests shown
+    # Claude Code's own --max-budget-usd uses Claude Code's price list, which the pilot found ~27x too
+    # high for Haiku 5.5; a cap at the agent's price would stop Haiku runs early. 0 = per_run_budget_usd.
+    agent_budget_flag_usd: float = 0.0
     keys: dict[str, str] = field(default_factory=dict)  # other agents' keys by env var (from the user's shell)
 
     def key_for(self, env_name: str) -> str:
@@ -141,7 +145,15 @@ def _agent_env(work: Path, cfg: Path, proxy: AllowlistProxy, api_key: str, key_e
     return env
 
 
-def _check_tokens(requests, result: dict) -> tuple[str, bool]:
+def _prior_usage(cfgdir: Path) -> dict | None:
+    """modelUsage of the step before a resumed one (its result.json is kept as prior-result.json)."""
+    try:
+        return json.loads((cfgdir.parent / "prior-result.json").read_text()).get("modelUsage")
+    except (OSError, ValueError):
+        return None
+
+
+def _check_tokens(requests, result: dict, offset: dict | None = None) -> tuple[str, bool]:
     """(tokens_check, is_harness_error). Is the transcript this run's, and complete?
 
     Learned from the first paid calibration runs: Claude Code's `usage` field
@@ -157,6 +169,8 @@ def _check_tokens(requests, result: dict) -> tuple[str, bool]:
     if isinstance(totals, dict) and totals:
         keys = ("inputTokens", "cacheCreationInputTokens", "cacheReadInputTokens", "outputTokens", "thinkingTokens")
         theirs = sum(int(m.get(k) or 0) for m in totals.values() if isinstance(m, dict) for k in keys)
+        if offset:  # a resumed session's totals include the earlier step (found in the pilot)
+            theirs -= sum(int(m.get(k) or 0) for m in offset.values() if isinstance(m, dict) for k in keys)
         if theirs == 0:
             return "unverified (modelUsage is zero)", False
         if abs(ours - theirs) / theirs > TOKEN_TOLERANCE:
@@ -239,7 +253,8 @@ def _run_agent(commit: Commit, setup: Setup, layout: Layout, cfg: Config, work:
             proxy_socket=proxy.socket_path,
         )
         res = sandbox.run(
-            [*setup.argv(task or task_text(commit, cfg.task_mode), cfg.model, cfg.per_run_budget_usd, cfgdir), *extra_args],
+            [*setup.argv(task or task_text(commit, cfg.task_mode), cfg.model, cfg.agent_budget_flag_usd or cfg.per_run_budget_usd,
+                         cfgdir), *extra_args],
             profile=profile,
             profile_path=layout.profile_path(run_id),
             env=_agent_env(work, cfgdir, proxy, key, setup.key_env),
@@ -263,9 +278,13 @@ def _run_agent(commit: Commit, setup: Setup, layout: Layout, cfg: Config, work:
     if since is not None:  # only this step's requests (a resumed session's transcript holds the earlier step too)
         parsed.requests = [r for r in parsed.requests if r.ts >= since]
     usages = [(r.usage.fresh_in, r.usage.cache_write, r.usage.cache_read, r.usage.output) for r in parsed.requests]
-    cost = float(result.get("total_cost_usd") or 0.0)
-    if since is not None:
-        cost = sum(usage_cost(r.usage, r.model) or 0.0 for r in parsed.requests)
+    # Every run is priced from its own tokens at the published price table, the same way for every
+    # setup. Claude Code's own total is kept for the record: the pilot found it ~27x too high for
+    # Haiku 5.5 while matching the table exactly for Sonnet.
+    reported = float(result.get("total_cost_usd") or 0.0)
+    priced = [usage_cost(r.usage, r.model) for r in parsed.requests]
+    cost = sum(p for p in priced if p is not None) if parsed.requests and all(p is not None for p in priced) else reported
+    measured["agent_reported_cost_usd"] = round(reported, 6)
     note = f"blocked: {blocked[:5]}" if blocked else ""
     if result.get("escapes"):
         note += f" escapes: {result['escapes']}"
@@ -274,7 +293,7 @@ def _run_agent(commit: Commit, setup: Setup, layout: Layout, cfg: Config, work:
 
     if res.timed_out:
         return record("timeout", usages=usages, cost=cost, check="n/a (timed out)", note=note)
-    check, harness_error = _check_tokens(parsed.requests, result)
+    check, harness_error = _check_tokens(parsed.requests, result, offset=_prior_usage(cfgdir) if since is not None else None)
     subtype = str(result.get("subtype", ""))
     if "budget" in subtype:
         return record("budget_hit", usages=usages, cost=cost, check=check, note=note)
@@ -309,7 +328,7 @@ def _finish_other(commit: Commit, setup: Setup, layout: Layout, work: Path, cfgd
         cost = sum(p for p in prices if p is not None) if cost_known else 0.0
         check = "unverified (codex rollout log)" if requests else "empty transcript"
     note = f"blocked: {blocked[:5]}" if blocked else ""
-    out = _codex_errors(res.stdout or "", res.stderr or "") if setup.agent == "codex" else ""
+    out = _codex_errors(res.stdout or "", res.stderr or "", cfgdir / "sessions") if setup.agent == "codex" else ""
     if res.timed_out:
         return record("timeout", usages=usages, cost=cost, check=check, note=note, model=model, cost_known=cost_known)
     if setup.agent == "codex" and _OPENAI_ACCOUNT_ERROR.search(out):
@@ -329,7 +348,7 @@ def _finish_other(commit: Commit, setup: Setup, layout: Layout, work: Path, cfgd
     return rec
 
 
-def _codex_errors(stdout: str, stderr: str) -> str:
+def _codex_errors(stdout: str, stderr: str, sessions: Path | None = None) -> str:
     """Only Codex's own error events, never the code or command output it prints.
 
     Found in the first real Codex run: the task was about handling OpenAI's
@@ -345,6 +364,17 @@ def _codex_errors(stdout: str, stderr: str) -> str:
         if isinstance(event, dict) and event.get("type") in ("error", "turn.failed"):
             lines.append(json.dumps(event.get("message") or event.get("error")))
     lines += [l for l in stderr.splitlines() if " ERROR " in l or l.startswith("Error")]
+    # Codex can also end a turn with an error that only its session log records (found in the
+    # main run: "Quota exceeded ... usage_limit_exceeded" looked like a harness error)
+    for path in sorted(sessions.rglob("*.jsonl")) if sessions is not None and sessions.is_dir() else []:
+        for line in path.read_text(errors="replace").splitlines():
+            try:
+                event = json.loads(line)
+            except ValueError:
+                continue
+            payload = event.get("payload") if isinstance(event, dict) else None
+            if isinstance(payload, dict) and payload.get("type") in ("task_complete", "error", "stream_error") and payload.get("error"):
+                lines.append(json.dumps(payload.get("error")))
     return "\n".join(lines)
 
 

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/sandbox.py

@@ -161,6 +161,7 @@ class Profile:
     deny_roots: list[Path] = field(default_factory=list)
     proxy_port: int | None = None  # None: no network at all
     proxy_socket: Path | None = None  # Linux: the proxy's Unix socket, relayed to 127.0.0.1:proxy_port inside
+    loopback: bool = True  # localhost networking for tests' own servers (never the internet)
 
     def render(self) -> str:
         home = str(Path.home().resolve())
@@ -187,6 +188,9 @@ class Profile:
         # other users' homes. This run's own folders are re-allowed below.
         for root in [*shared, *deny_roots]:
             lines.append(f'(deny file-read* file-write* (subpath "{root}"))')
+        # macOS's /usr/bin/git, clang etc. are xcrun launchers that keep a cache file in the user's
+        # temp folder; allow exactly those files, the rest of the folder stays hidden
+        lines.append('(allow file-read* file-write* (regex #"^/private/var/folders/[^/]+/[^/]+/T/xcrun_db"))')
         for d in readable:
             lines.append(f'(allow file-read* (subpath "{d}"))')
         for d in writable:
@@ -194,7 +198,15 @@ class Profile:
         for ancestor in sorted(_ancestors([*writable, *readable], [home, *shared, *deny_roots])):
             lines.append(f'(allow file-read-metadata (literal "{ancestor}"))')
         lines.append('(allow file-write-data (literal "/dev/null") (literal "/dev/zero"))')
-        if self.proxy_port is not None:
+        if self.loopback:
+            # Tests and test plugins start servers on localhost (pytest-rerunfailures binds a socket at
+            # startup; axios's tests run local HTTP servers), and agents run those tests while working.
+            # The internet stays unreachable; the proxy port is on localhost too. macOS can't give a
+            # sandbox its own loopback, so services already listening on this machine's localhost are
+            # reachable (Linux runs get a private loopback).
+            lines += ["(allow system-socket)", '(allow network-bind (local ip "localhost:*"))',
+                      '(allow network-inbound (local ip "localhost:*"))', '(allow network-outbound (remote ip "localhost:*"))']
+        elif self.proxy_port is not None:
             lines += ["(allow system-socket)", f'(allow network-outbound (remote ip "localhost:{self.proxy_port}"))']
         return "\n".join(lines) + "\n"
 

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/study.py

@@ -187,7 +187,7 @@ def holm(comparisons: list[Comparison], *, alpha: float = 0.05) -> None:
         running = max(running, min(1.0, (m - i) * c.p_per_solved))
         c.p_holm = running
     for c in comparisons:
-        need = min(5, c.repos) if c.repos >= 6 else c.repos
+        need = 5 if c.repos >= 6 else c.repos + 1  # pre-registered: >= 6 repos, direction holding in >= 5 (pilot caught 1 repo passing)
         ratio = {"per_solved": c.ratio_per_solved, "tokens": c.ratio_tokens, "per_attempt": c.ratio_per_attempt}[c.primary]
         direction = c.cheaper_in_repos if (ratio or 1) < 1 else c.repos - c.cheaper_in_repos
         c.headline = bool(c.p_holm is not None and c.p_holm < alpha and c.repos and direction >= need)

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/study_cli.py

@@ -32,6 +32,7 @@ import os
 import time
 from pathlib import Path
 
+from aimpg.gitkept import GitError
 from aimpg.replay import chain, select, study
 from aimpg.replay import run as runner
 from aimpg.replay.cli import _select, _state, secure_root
@@ -72,16 +73,60 @@ def _out(plan: dict) -> Path:
 def main(args) -> int:
     plan = load_plan(args.plan)
     if args.study_command == "select":
-        for r in plan["repos"]:
-            print(f"\n== {r['path']}")
-            _select(Path(r["path"]), argparse.Namespace(days=365, cutoff=plan["cutoff"], max=plan["commits_per_repo"] * 2,
-                                                        any_author=True))
-        return 0
+        return _select_all(plan)
     if args.study_command == "report":
         return _report(plan)
     return _run(plan, args)
 
 
+MIN_QUALIFYING = 5  # pre-registered: fewer qualifying commits than this and the repo is replaced by the next reserve
+
+
+def _qualified(repo: str) -> int:
+    path = _state(Path(repo)) / "selected.jsonl"
+    return len(select.load(path)) if path.exists() and path.read_text().strip() else 0
+
+
+def _select_all(plan: dict) -> int:
+    """Select each repo; apply the pre-registered replacement rule; write selection.json for `run`."""
+    reserves = [dict(r, path=str(Path(r["path"]).expanduser().resolve())) for r in plan.get("reserve_repos", [])]
+    summary = {"repos": [], "replaced": []}
+    queue = list(plan["repos"])
+    while queue:
+        r = queue.pop(0)
+        print(f"\n== {r['path']}")
+        if not Path(r["path"]).is_dir():
+            print(f"  not cloned: git clone --filter=blob:none --single-branch {r.get('url')} {r['path']}")
+            n = 0
+        elif (_state(Path(r["path"])) / "selected.manifest.json").exists():
+            n = _qualified(r["path"])  # finished in an earlier run (the manifest is written last)
+            print(f"  already selected: {n} qualifying (delete {_state(Path(r['path']))} to redo)")
+        else:
+            for attempt in range(3):  # blobless clones fetch file contents on demand; GitHub sometimes drops one
+                try:
+                    _select(Path(r["path"]), argparse.Namespace(days=365, cutoff=plan["cutoff"],
+                                                                max=plan["commits_per_repo"] * 2, any_author=True))
+                    break
+                except GitError as exc:
+                    print(f"  git failed ({str(exc)[:120]}); retrying in 30 s" if attempt < 2 else f"  git failed 3 times: {exc}")
+                    time.sleep(30)
+            n = _qualified(r["path"])
+        if n >= MIN_QUALIFYING:
+            summary["repos"].append({"path": r["path"], "url": r.get("url"), "qualifying": n})
+        elif reserves:
+            nxt = reserves.pop(0)
+            summary["replaced"].append({"repo": r.get("url") or r["path"], "qualifying": n, "replaced_by": nxt.get("url")})
+            print(f"  {n} qualifying (< {MIN_QUALIFYING}): replaced by the next reserve, {nxt.get('url')}")
+            queue.insert(0, nxt)
+        else:
+            summary["replaced"].append({"repo": r.get("url") or r["path"], "qualifying": n, "replaced_by": None})
+            print(f"  {n} qualifying (< {MIN_QUALIFYING}) and no reserves left: dropped")
+    (_out(plan) / "selection.json").write_text(json.dumps(summary, indent=1))
+    print(f"\nIn the study: {len(summary['repos'])} repos. Replaced: {len(summary['replaced'])}. "
+          f"Summary: {_out(plan) / 'selection.json'}")
+    return 0
+
+
 def _selected(plan: dict, repo: str, n: int):
     path = _state(Path(repo)) / "selected.jsonl"
     if not path.exists():
@@ -96,7 +141,11 @@ def _setups(plan: dict):
 
 def _run(plan: dict, args) -> int:
     setups = _setups(plan)
-    repos = plan["repos"][:1] if args.pilot else plan["repos"]
+    chosen = _out(plan) / "selection.json"
+    if not chosen.exists():
+        raise SystemExit("run `aimpg study select` first")
+    active = json.loads(chosen.read_text())["repos"]
+    repos = active[:1] if args.pilot else active
     n, repeats = (2, 1) if args.pilot else (plan["commits_per_repo"], plan["repeats"])
     chains = 1 if args.pilot else plan.get("chains_per_repo", 0)
     keys = {}
@@ -111,14 +160,20 @@ def _run(plan: dict, args) -> int:
     cap = (min(plan["cap_usd"], 15.0) if args.pilot else plan["cap_usd"])
     print(f"Study {plan['name']}{' (PILOT)' if args.pilot else ''}: {len(repos)} repos, {n_runs} agent runs, "
           f"setups {', '.join(s.name for s in setups)}, {chains} fresh-session pairs per repo, task mode {plan.get('task_mode', 'hint')}")
-    print(f"Total cap ${cap:.2f}; Claude Code runs stop at ${plan.get('per_run_budget_usd', 2.0):.2f} each.")
+    out_dir = _out(plan) / ("pilot" if args.pilot else "runs")
+    done_runs = sum(len(runner.completed(out_dir / f"{Path(r['path']).name}.jsonl")) for r in repos) if out_dir.exists() else 0
+    if done_runs:
+        print(f"{done_runs} of those runs are already done (kept); only the rest will run.")
+    print(f"Total cap ${cap:.2f}, counted at the published API prices (each run reserves "
+          f"${plan.get('per_run_budget_usd', 2.0):.2f} until its real cost is known).")
     if not args.yes and input("Spend up to that on your API keys? [y/N] ").strip().lower() != "y":
         print("Stopped. Nothing was spent.")
         return 1
     out = _out(plan) / ("pilot" if args.pilot else "runs")
     out.mkdir(exist_ok=True)
     cfg = runner.Config(model="claude-sonnet-5-5", per_run_budget_usd=plan.get("per_run_budget_usd", 2.0), total_cap_usd=cap,
-                        api_key=keys.get("ANTHROPIC_API_KEY", ""), task_mode=plan.get("task_mode", "hint"), keys=keys)
+                        api_key=keys.get("ANTHROPIC_API_KEY", ""), task_mode=plan.get("task_mode", "hint"), keys=keys,
+                        agent_budget_flag_usd=plan.get("agent_budget_flag_usd", 50.0))
     layout, spent = Layout(secure_root()), 0.0
     for repo, commits in selected.items():
         results = out / f"{Path(repo).name}.jsonl"

{aimpg-0.6.2 → aimpg-0.7.0}/aimpg/replay/workspace.py

@@ -134,6 +134,10 @@ def _tool_env(work: Path) -> dict[str, str]:
         npm_config_update_notifier="false",
         npm_config_fund="false",
         npm_config_audit="false",
+        # The workspace is a history-free export (no .git), so build backends that read the version
+        # from git (setuptools-scm, hatch-vcs: pyinfra) fail; give them a placeholder, the same for every setup
+        SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0",
+        UV_DYNAMIC_VERSIONING_BYPASS="0.0.0",  # uv-dynamic-versioning (pyinfra)
     )
     return env
 
@@ -161,6 +165,31 @@ def _check(result: sandbox.Result, what: str) -> None:
         raise WorkspaceError(f"{what} failed: {(result.stderr or result.stdout)[-800:]}")
 
 
+_GROUP = re.compile(r"^\s*([A-Za-z0-9][A-Za-z0-9_.-]*)\s*=\s*\[", re.M)
+_TEST_GROUP = re.compile(r"(^|[-_])(dev|tests?|testing)($|[-_])")
+
+
+def test_groups(project: Path) -> list[str]:
+    """uv flags for the dependency groups tests need: those named like dev/test/testing.
+
+    Installing every group pulled in docs tooling fetched straight from GitHub
+    (pydantic), which the install sandbox rightly can't reach. Falls back to all
+    groups when a project has none of those names.
+    """
+    try:
+        text = (project / "pyproject.toml").read_text()
+    except OSError:
+        return ["--all-groups"]
+    section = re.search(r"^\[dependency-groups\]\s*$(.*?)(?=^\[|\Z)", text, re.M | re.S)
+    names = _GROUP.findall(section.group(1)) if section else []
+    wanted = [n for n in names if _TEST_GROUP.search(n)]
+    if not names:
+        return []
+    if not wanted:
+        return ["--all-groups"]
+    return [f for n in wanted for f in ("--group", n)]
+
+
 def prepare(commit: Commit, layout: Layout, proxy: AllowlistProxy) -> Path:
     """Phase A, once per commit: parent tree + dependencies from the commit's lockfile (R17)."""
     dest = layout.prepared(commit.sha)
@@ -172,7 +201,7 @@ def prepare(commit: Commit, layout: Layout, proxy: AllowlistProxy) -> Path:
     if commit.kind == "python":
         proxy.set_allow(PYPI)
         reqs = _commit_requirements(commit, layout)
-        _check(_sandboxed(layout, "prepare", dest, ["uv", "sync", "--all-groups", "--quiet"], proxy=proxy, timeout=PREPARE_TIMEOUT), "uv sync")
+        _check(_sandboxed(layout, "prepare", dest, ["uv", "sync", *test_groups(dest), "--quiet"], proxy=proxy, timeout=PREPARE_TIMEOUT), "uv sync")
         if reqs:
             (dest / ".tmp").mkdir(exist_ok=True)
             (dest / ".tmp" / "commit-requirements.txt").write_text(reqs)
@@ -203,7 +232,7 @@ def _commit_requirements(commit: Commit, layout: Layout) -> str:
     (tmp / "uv.lock").write_bytes(lock)
     (tmp / "pyproject.toml").write_bytes(pyproject)
     proc = subprocess.run(
-        ["uv", "export", "--frozen", "--no-hashes", "--no-emit-project", "--all-groups", "--quiet"],
+        ["uv", "export", "--frozen", "--no-hashes", "--no-emit-project", *test_groups(tmp), "--quiet"],
         cwd=tmp,
         capture_output=True,
         text=True,
@@ -232,7 +261,7 @@ def clone_for_run(commit: Commit, layout: Layout, prepared: Path, run_id: str, *
     subprocess.run([*clone, str(prepared), str(work)], check=True)
     (work / ".prepared").unlink(missing_ok=True)
     if commit.kind == "python":
-        _check(_sandboxed(layout, run_id, work, ["uv", "sync", "--offline", "--all-groups", "--quiet"], proxy=None, timeout=300), "offline re-sync")
+        _check(_sandboxed(layout, run_id, work, ["uv", "sync", "--offline", *test_groups(work), "--quiet"], proxy=None, timeout=300), "offline re-sync")
         reqs = work / ".tmp" / "commit-requirements.txt"
         if reqs.exists():
             _check(_sandboxed(layout, run_id, work, ["uv", "pip", "install", "--offline", "--quiet", "-r", ".tmp/commit-requirements.txt"], proxy=None, timeout=300), "offline commit deps")
@@ -313,18 +342,21 @@ def overlay_commit_tree(commit: Commit, work: Path) -> None:
                         os.symlink(m.linkname, link)
 
 
-def test_argv(commit: Commit) -> list[str]:
+def test_argv(commit: Commit, work: Path | None = None) -> list[str]:
+    """Run only the commit's test files, with coverage thresholds off: a repo that demands 100%
+    coverage (lint-staged) fails any subset run even when every test passes."""
     files = [f for f in commit.test_files if f.endswith((".py", ".js", ".ts", ".jsx", ".tsx", ".mjs", ".cjs", ".mts", ".cts"))]
     if commit.runner == "pytest":
-        return [".venv/bin/python", "-m", "pytest", "-q", "-p", "no:cacheprovider", *files]
+        has_cov = work is not None and any((work / ".venv").glob("lib/python*/site-packages/pytest_cov"))
+        return [".venv/bin/python", "-m", "pytest", "-q", "-p", "no:cacheprovider", *(["--no-cov"] if has_cov else []), *files]
     if commit.runner == "vitest":
-        return ["node_modules/.bin/vitest", "run", *files]
-    return ["node_modules/.bin/jest", "--ci", *files]
+        return ["node_modules/.bin/vitest", "run", "--coverage.enabled=false", *files]
+    return ["node_modules/.bin/jest", "--ci", "--coverage=false", *files]
 
 
 def judge(commit: Commit, layout: Layout, work: Path, name: str) -> tuple[bool, str]:
     """Phase C: run only the commit's test files, no network. (passed, tail of output)."""
-    result = _sandboxed(layout, name, work, test_argv(commit), proxy=None, timeout=JUDGE_TIMEOUT, extra_env={"CI": "1"})
+    result = _sandboxed(layout, name, work, test_argv(commit, work), proxy=None, timeout=JUDGE_TIMEOUT, extra_env={"CI": "1"})
     tail = (result.stdout + result.stderr)[-600:]
     return (not result.timed_out and result.returncode == 0), tail
 

{aimpg-0.6.2 → aimpg-0.7.0}/docs/FAQ.md

@@ -25,7 +25,7 @@ A bill tells you the total. aimpg tells you what each change that shipped and la
 Maybe not. aimpg tells you how much of your agents' tokens were conversation carried over from before your last commit (57% for the author). It doesn't claim clearing saves all of that: a fresh session has to rebuild some context. Report No. 1 includes an experiment that measures the real difference on real commits.
 
 **Is it safe to run?**
-`report`, `live`, `sci` and `defaults` only read local logs and git. Replays run agents in a sandbox: Seatbelt on macOS, bubblewrap on Linux. The sandbox hides your home folder, temp folders and other users' files, and only reaches the model's API through an allowlisting proxy. A pre-launch security review found and fixed three high-severity issues (in 0.6.1), each with a regression test. `--rerun` of someone else's record shows every command, hook and prompt it would run, and asks first.
+`report`, `live`, `sci` and `defaults` only read local logs and git. Replays run agents in a sandbox: Seatbelt on macOS, bubblewrap on Linux. The sandbox hides your home folder, temp folders and other users' files, and only reaches the model's API through an allowlisting proxy. A pre-launch security review found and fixed three high-severity issues (in 0.6.1), each with a regression test. `--rerun` of someone else's record shows every command, hook and prompt it would run, and asks first. One limit: tests start local servers, so the sandbox allows localhost. On macOS that means services already listening on your machine's localhost are reachable from inside; on Linux each run gets its own private localhost.
 
 **What leaves my machine?**
 Nothing, unless you choose:

{aimpg-0.6.2 → aimpg-0.7.0}/docs/designs/launch-brief.md

@@ -61,9 +61,9 @@ One maintainer; no budget beyond a few hundred dollars of API credit; must stay
   - the stopping rule and Holm correction;
   - the RTK config approved by its maintainers (asked 2026-10-11: https://github.com/rtk-ai/rtk/discussions/4511; proceed with the stated config if no reply by 2026-10-18 and say so);
   - headline only if the direction holds in at least 5 of 6 repos.
-- [ ] D. A pilot (about $10) to size n, then the full study: about 55 commits × 3 repeats × (Haiku, Sonnet, Opus, Codex, RTK), plus the fresh-session chains. $80–250, approved by the user in the terminal.
+- [ ] D. Pilot done 2026-10-13 ($1.95 total; found and fixed Claude Code's ~27x Haiku 5.5 price, the resumed-session check and the headline rule). Then the full study: about 55 commits × 3 repeats × (Haiku, Sonnet, Opus, Codex, RTK), plus the fresh-session chains. $80–250, approved by the user in the terminal.
 - [ ] E. At least 2–3 reruns by unaffiliated people, so "Reproduced" exists; friends' reruns are labeled as reruns. Get one team quote if possible.
-- [ ] F. Assets:
+- [ ] F. Assets (done 2026-10-11: share card, PR footer, Marketplace listing https://github.com/marketplace/actions/ai-coding-mpg, FAQ, Bring Your Claim rules; to do once results exist: report page, clip, post drafts):
   - the Report No. 1 page on the scoreboard site, with charts and a forest plot;
   - a 20-second clip;
   - `aimpg report --share` card;