independent-tests 0.1.0 → 0.2.0

package/AGENTS.md

@@ -34,7 +34,9 @@ What to tell them, accurately:
 - Generating the suites costs about $1–4 and 10–25 minutes of Opus 5.5 per spec
   on their signed-in `claude` CLI, up to about $7 in a repository, plus their
   own time to confirm the decisions (median person minutes: pending; `confirm`
-  records each confirmation's duration).
+  records each confirmation's duration). Without Claude Code it runs on their
+  Codex login or an Anthropic API key instead, on macOS, Linux or Windows
+  (Windows without an OS sandbox for the sessions).
 - They must confirm the spec's decisions themselves before any test is
   generated. Replaying 11 real commits, suites written from a spec drafted from
   the commit message rejected all 11; once a session given the real diff

package/README.md

@@ -4,37 +4,44 @@
   <strong>Your agent says all tests pass. Check with tests it never saw.</strong>
 </p>
 
+<p align="center">
+  <a href="https://www.npmjs.com/package/independent-tests"><img src="https://img.shields.io/npm/v/independent-tests?style=flat-square&label=npm" alt="npm version"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="MIT license"></a>
+  <a href="SKILL.md"><img src="https://img.shields.io/badge/Claude_Code-skill-D97757?style=flat-square" alt="Claude Code skill"></a>
+  <a href="#install"><img src="https://img.shields.io/badge/judges-any_coding_agent-555?style=flat-square" alt="Judges any coding agent"></a>
+</p>
+
+<p align="center">
+  <img src="assets/hero.png" width="100%" alt="Coding agents that report a task complete are wrong 71% of the time (75 of 105). Independent tests cut that to 9% (9 of 105).">
+</p>
+
+<table>
+<tr>
+<td align="center" width="25%"><h3>75 of 105</h3>green runs failed tests<br>they never saw<br><sub>from a loose ticket</sub></td>
+<td align="center" width="25%"><h3>30 → 81</h3>of 105 pass when the<br>agent gets a public suite<br><sub>same agent, p &lt; 1e-11</sub></td>
+<td align="center" width="25%"><h3>95 of 100</h3>rejections upheld<br>by a blind referee<br><sub>on another model</sub></td>
+<td align="center" width="25%"><h3>0 of 2,522</h3>runs touched a test<br>or the harness<br><sub>wrong without cheating</sub></td>
+</tr>
+</table>
+
 <p align="center">
   <a href="#before--after">Before/After</a> •
   <a href="#install">Install</a> •
+  <a href="#does-a-hidden-suite-catch-what-green-tests-miss">Proof</a> •
+  <a href="#try-it-in-five-minutes">Try it</a> •
   <a href="#the-numbers">The numbers</a> •
   <a href="RESULTS.md">RESULTS.md</a>
 </p>
 
 ---
 
-**105 of 105** agents said their own tests passed. **75** of them failed tests
-they never saw. None of them cheated.
-
-```
-own tests green                 ████████████████████  105/105
-hidden suite, from the ticket   ██████░░░░░░░░░░░░░░   30/105
-  + a public suite from here    ███████████████░░░░░   81/105   p < 1e-11
-```
-
-Same agent (Haiku 5.5), same loose tickets, 7 specs × 15 runs. The only change
-in the third row: the decisions the ticket left open, confirmed by a person and
-handed to the agent as tests it can see, judged by tests it can't. Were the
-rejections fair? A blind referee on another model, given the spec, the code and
-one anonymised failing test, upheld 95 of 100.
-
 ## Before / After
 
 <table>
 <tr>
 <td width="50%">
 
-### 🗣️ The agent's report
+### 🗣️ The agent: "my 21 tests pass"
 
 > "The escrow release logic in `src/index.ts` is implemented. The build is clean
 > and my 21 tests under `tests/own/` pass."
@@ -45,7 +52,7 @@ one anonymised failing test, upheld 95 of 100.
 </td>
 <td width="50%">
 
-### 🔒 The hidden suite
+### 🔒 Tests it never saw: 43 of 46
 
 ```
 hidden 43/46
@@ -61,42 +68,66 @@ hidden: [boundary-2] passing hidden tests on a
 </tr>
 </table>
 
+**Same code. 21 green tests. One wrong guess, and the agent knew it was
+guessing.**
+
 escrow-split, run 1. 14 of 15 runs made the same guess. The ticket never said
 what a $0 bounty does; the agent picked an answer, then wrote 21 tests that
 agree with it. Green tests from the author of the code test the author's
 guesses.
 
-In 2025 METR caught frontier models
-[reward hacking](https://metr.org/blog/2025-06-05-recent-reward-hacking/) in 30%
-of RE-Bench runs: patching the grader, copying its answers. That is what
-sandboxes guard against. In 2,522 sandboxed runs here no agent changed a test or the harness, and
-the hidden suites still rejected 71% from a loose ticket. The agent does not
-need to cheat to be wrong.
-
 ## Install
 
 ```sh
-git clone https://github.com/lunox-work/independent-tests ~/.claude/skills/independent-tests
+npx skills add lunox-work/independent-tests -g
 ```
 
-Then ask Claude Code: _"write tests my agent can't see for this ticket"_. The
-implementer can be anything that edits a folder: Claude Code, Codex, Cursor,
-Copilot, Aider, or a person. [Try the bundled example](#try-it-in-five-minutes)
-in five minutes; details in [Install in full](#install-in-full).
+Then ask your coding agent (Claude Code, Codex, Cursor, or any agent that loads
+skills): _"write tests my agent can't see for this ticket"_. The tests are
+written on whichever account you have: Claude Code, a Codex login or an
+Anthropic API key. The implementer can be anything that edits a folder: Claude
+Code, Codex, Cursor, Copilot, Aider, or a person.
+[Try the bundled example](#try-it-in-five-minutes) in five minutes; a git clone
+and the CLI are in [Install in full](#install-in-full).
+
+## Does a hidden suite catch what green tests miss?
+
+**105 of 105** agents said their own tests passed. **75** of them failed tests
+they never saw. None of them cheated.
+
+<p align="center">
+  <img src="assets/runs.png" width="100%" alt="One square per run, 7 specs by 15 runs, red where the hidden suite failed. Agent alone: 75 of 105 wrong. With a public suite: 24 of 105. After one retry told only the failing scenario: 9 of 105.">
+</p>
+
+Same agent (Haiku 5.5), same loose tickets, 7 specs × 15 runs. The only change
+in the middle column (p < 1e-11): the decisions the ticket left open, confirmed
+by a person and handed to the agent as tests it can see, judged by tests it
+can't. In the last, each of the 24 rejected runs got one retry told only the
+failing scenario's key, never the test; 15 then passed. Were the
+rejections fair? A blind referee on another model, given the spec, the code and
+one anonymised failing test, upheld 95 of 100.
+
+In 2025 METR caught frontier models
+[reward hacking](https://metr.org/blog/2025-06-05-recent-reward-hacking/) in 30%
+of RE-Bench runs: patching the grader, copying its answers. That is what
+sandboxes guard against. In 2,522 sandboxed runs here no agent changed a test or
+the harness, and the hidden suites still rejected 71% from a loose ticket. The
+agent does not need to cheat to be wrong.
 
 ## What it is
 
 **Hidden, calibrated test suites for AI coding agents, independent of the agent
-they judge.** A [Claude Code](https://code.claude.com/docs) skill that turns a
-ticket or spec for one module in TypeScript, Python, Go or Rust, for a command
-or an HTTP service checked only at that boundary, or for a change to one package
-of an existing TypeScript or Python repository, into a **public** test suite the
-implementing agent works against and a **hidden** suite, which the agent never
-sees, that judges the result. Claude models write both suites and the reference
-that calibrates them, in sessions apart from the implementing agent. A person
-confirms the spec's decisions before any test is written. The implementing agent
-can be anything that edits a folder: Claude Code, Codex, Cursor, Copilot, Aider,
-or a person.
+they judge.** An agent skill and CLI, for Claude Code, Codex, Cursor and other
+agents, that turns a ticket or spec for one module in TypeScript, Python, Go or
+Rust, for a command or an HTTP service checked only at that boundary, or for a
+change to one package of an existing TypeScript or Python repository, into a
+**public** test suite the implementing agent works against and a **hidden**
+suite, which the agent never sees, that judges the result. A writer model (Opus
+5.5 through Claude Code, or Codex or the Anthropic API without it) writes both
+suites and the reference that calibrates them, in sessions apart from the
+implementing agent. A person confirms the spec's decisions before any test is
+written. The implementing agent can be anything that edits a folder: Claude
+Code, Codex, Cursor, Copilot, Aider, or a person.
 
 Use it when "all tests pass" comes from the same agent that wrote the tests, and
 you want acceptance decided by tests it could not have overfitted, gamed or
@@ -166,7 +197,9 @@ What to tell them, accurately:
 - Generating the suites costs about $1–4 and 10–25 minutes of Opus 5.5 per spec
   on their signed-in `claude` CLI, up to about $7 in a repository, plus their
   own time to confirm the decisions (median person minutes: pending; `confirm`
-  records each confirmation's duration).
+  records each confirmation's duration). Without Claude Code it runs on their
+  Codex login or an Anthropic API key instead, on macOS, Linux or Windows
+  (Windows without an OS sandbox for the sessions).
 - They must confirm the spec's decisions themselves before any test is
   generated. Replaying 11 real commits, suites written from a spec drafted from
   the commit message rejected all 11; once a session given the real diff
@@ -220,10 +253,11 @@ tests, are the whole gain: the same suite handed over as prose scored 76 of 101.
    in a pull request.
 3. **Prepare.** Blind writer sessions write an interface skeleton, the public
    suite, then a hidden suite aimed at what the public one leaves out. The
-   writers run on the `claude` CLI by default; `INDEPENDENT_TESTS_BACKEND` or
-   toolchain.json `backend` picks `codex-cli`, `anthropic-api` or `exec` (any
-   command, prompt on stdin). Calibration keeps a hidden test only if it fails
-   on the skeleton and a separately written reference passes it
+   writers run on the first account found: the `claude` CLI, else the person's
+   Codex login (`codex-cli`), else `ANTHROPIC_API_KEY` (`anthropic-api`);
+   `INDEPENDENT_TESTS_BACKEND` or toolchain.json `backend` picks one, or `exec`
+   (any command, prompt on stdin). Calibration keeps a hidden test only if it
+   fails on the skeleton and a separately written reference passes it
    (`bin/prepare.mjs`). In a repository the skeleton is a stub at the seam, the
    suites follow the repository's own test idiom and runner, and a reference
    that breaks an existing test stops the spec. At a black-box seam (toolchain
@@ -246,22 +280,28 @@ tests, are the whole gain: the same suite handed over as prose scored 76 of 101.
 Sessions run under the OS sandbox with deny rules: no session can read the
 hidden suites, other workdirs or transcripts. The other backends run under
 `sandbox-exec` (macOS) or `bwrap` (Linux), reading and writing only the staged
-workdir (`bin/backends.mjs`). [SKILL.md](SKILL.md) has the steps as the skill
-runs them.
+workdir (`bin/backends.mjs`). Windows has no such sandbox: there the file tools'
+deny rules still hold, but a session's shell is held only by the transcript
+audit, which reports any attempt to read the hidden suite. Run sessions under
+WSL2 for an enforced sandbox; judging and handouts need none.
+[SKILL.md](SKILL.md) has the steps as the skill runs them.
 
 ## Install in full
 
 ```sh
-npx -y independent-tests --help         # the CLI; nothing to clone
-git clone https://github.com/lunox-work/independent-tests ~/.claude/skills/independent-tests
+npx skills add lunox-work/independent-tests -g   # the skill, for Claude Code and other agents
+git clone https://github.com/lunox-work/independent-tests ~/.claude/skills/independent-tests  # or the whole repo
+npx -y independent-tests --help                  # the CLI; nothing to clone
 ```
 
-The clone is the Claude Code skill (SKILL.md), which calls the same CLI. Needs
-Node ≥ 22.12, a signed-in `claude` CLI and macOS or Linux. The first command
-that needs the toolchain installs `template/`'s dependencies with npm 10.9.2
-under `~/.independent-tests/`, once. Then ask Claude Code for tests an agent
-can't see for a ticket, or to judge a submission against a spec. The handout
-folder needs only Node, so the implementing agent can run anywhere.
+Either install puts the skill (SKILL.md) in place; it calls the same CLI,
+through `npx` when only SKILL.md was copied. Needs Node 20.19 or later on macOS,
+Linux or Windows, and one of: Claude Code signed in, a Codex login, or
+`ANTHROPIC_API_KEY`. The first command that needs the toolchain installs
+`template/`'s dependencies with npm 10.9.2 under `~/.independent-tests/`, once.
+Then ask your agent for tests an agent can't see for a ticket, or to judge a
+submission against a spec. The handout folder needs only Node, so the
+implementing agent can run anywhere.
 
 Specs, runs and results live in `~/.independent-tests/`
 (`INDEPENDENT_TESTS_HOME`; `independent-tests where` prints the folders), never
@@ -401,6 +441,9 @@ reads the runner's disk and sends it over the network is not stopped.
 
 ## Layout
 
+<details>
+<summary>Every file and folder, in the repository and in <code>~/.independent-tests/</code></summary>
+
 ```
 SKILL.md            the steps the skill runs
 AGENTS.md           this README's section for coding agents, verbatim
@@ -461,3 +504,18 @@ evidence/seams.mjs  black-box seams: patch-apply as a command against git apply,
 
 A development checkout keeps its own `specs/`, `runs/` and `results.jsonl` with
 `INDEPENDENT_TESTS_HOME=.`.
+
+</details>
+
+---
+
+<p align="center">
+  <strong>Caught a green run that was wrong? A ⭐ helps the next person find this.</strong>
+  <br>
+  <sub>
+    <a href="RESULTS.md">RESULTS.md</a> ·
+    <a href="AGENTS.md">AGENTS.md</a> ·
+    <a href="SKILL.md">SKILL.md</a> ·
+    <a href="https://github.com/lunox-work/independent-tests/issues">Issues</a>
+  </sub>
+</p>

package/SKILL.md

@@ -45,16 +45,19 @@ steps marked _Repository_ below are for that target.
   (UI through Playwright is out: its tests are too noisy to calibrate), nor yet
   a change inside a repository visible only through a running service: `fit`,
   `repo` and the judge take `--seam http`, but no brief writes its suites.
-- **Preconditions.** Node ≥ 22.12; `claude` signed in (`CLAUDE_BIN`, PATH or the
-  VS Code extension bundle); macOS or Linux for the OS sandbox. The first
-  command that needs the bundled toolchain installs it with npm 10.9.2 into the
-  home (network, once). Python: `python3` (pytest is installed into the home
-  once); Go and Rust: `go` or `cargo` on PATH. Another writer backend, if the
-  person asks for one: `INDEPENDENT_TESTS_BACKEND=codex-cli` (their Codex login;
-  Codex 0.164.0 installed into the home once), `anthropic-api`
-  (`ANTHROPIC_API_KEY`) or `exec` with `INDEPENDENT_TESTS_EXEC=<command>`; these
-  run under `sandbox-exec` or `bwrap` instead of Claude Code's deny rules.
-- **Cost.** Step 3 starts four blind `claude -p` sessions on Opus 5.5: about
+- **Preconditions.** Node 20.19 or later, on macOS, Linux or Windows. One
+  account for the sessions, taken in this order unless
+  `INDEPENDENT_TESTS_BACKEND` names one: `claude` signed in (`CLAUDE_BIN`, PATH
+  or the VS Code extension bundle), else their Codex login (`codex-cli`; Codex
+  0.164.0 installed into the home once), else `ANTHROPIC_API_KEY`
+  (`anthropic-api`); or `exec` with `INDEPENDENT_TESTS_EXEC=<command>`. claude
+  sessions are held by Claude Code's sandbox and deny rules, the others by
+  `sandbox-exec` or `bwrap`. On Windows there is no OS sandbox: say so before
+  step 3, and that WSL2 gives one. The first command that needs the bundled
+  toolchain installs it with npm 10.9.2 into the home (network, once). Python:
+  `python3` (`python` on Windows; pytest is installed into the home once); Go
+  and Rust: `go` or `cargo` on PATH.
+- **Cost.** Step 3 starts four blind sessions, on Opus 5.5 with `claude`: about
   $1–4 and 10–25 minutes per spec. An agent run is a few cents on Haiku 5.5,
   about $0.30 on Opus. Say so before step 3. Step 2 costs the person's time:
   `confirm` records it per spec (`minutes` in confirmed.json, from the first

package/bin/backends.mjs

@@ -45,8 +45,12 @@
 //
 // `placement`'s denial is checked by evidence/backends.mjs `sandbox`, a
 // negative test that reads a hidden-suite path and the home from inside.
+// Windows has neither sandbox: there `placement` runs the command as it
+// is, the shell being Git for Windows' bash, and says so once (lib.mjs
+// `warnUnsandboxed`); the staged workdir and the transcript audit are what
+// keep a writer blind, and anthropic-api's file tools stay confined.
 
-import { spawn, spawnSync } from "node:child_process";
+import { spawn } from "node:child_process";
 import { randomUUID } from "node:crypto";
 import {
   appendFileSync,
@@ -61,7 +65,7 @@ import {
 } from "node:fs";
 import { createRequire } from "node:module";
 import { homedir, tmpdir } from "node:os";
-import { basename, dirname, join, relative, resolve } from "node:path";
+import { basename, dirname, join, posix, relative, resolve } from "node:path";
 import { pathToFileURL } from "node:url";
 import {
   DEFAULT_MODEL,
@@ -78,14 +82,15 @@ import {
   log,
   sha256,
   takeSlot,
+  warnUnsandboxed,
 } from "./lib.mjs";
+import { GROUP, OS_SANDBOX, WINDOWS, gitBash, inside, pathKey, slash, spawnCommand, which } from "./platform.mjs";
 
 export const CODEX_VERSION = "0.164.0";
 export const SDK_VERSION = "0.137.0";
 /** Per session: CODEX_HOME, HOME, TMPDIR, the API backend's messages. */
 export const SESSIONS = join(WORK_ROOT, "sessions");
 
-const inside = (path, folder) => path === folder || path.startsWith(`${folder}/`);
 /** `path` made real as far as it exists (the sandboxes match real paths). */
 export function real(path) {
   let at = resolve(path);
@@ -110,7 +115,7 @@ export function installTool(name, spec, version) {
   mkdirSync(partial, { recursive: true });
   writeFileSync(join(partial, "package.json"), `${JSON.stringify({ private: true })}\n`);
   log(`installing ${spec}@${version} into ${dest} (npm@10.9.2, once)`);
-  const out = spawnSync("npx", ["-y", "npm@10.9.2", "install", "--no-audit", "--no-fund", "--save-exact", `${spec}@${version}`], {
+  const out = spawnCommand("npx", ["-y", "npm@10.9.2", "install", "--no-audit", "--no-fund", "--save-exact", `${spec}@${version}`], {
     cwd: partial,
     encoding: "utf8",
     maxBuffer: 64 * 1024 * 1024,
@@ -131,8 +136,8 @@ export function installTool(name, spec, version) {
 
 /** The install prefix of a binary (two folders above its real path), for the sandbox's read list. */
 function prefixOf(bin) {
-  const found = bin.includes("/") ? bin : spawnSync("sh", ["-c", `command -v ${bin}`], { encoding: "utf8" }).stdout.trim();
-  if (found === "" || !existsSync(found)) return null;
+  const found = which(bin);
+  if (found === null) return null;
   return dirname(dirname(realpathSync(found)));
 }
 
@@ -203,16 +208,28 @@ export function bwrapArgs({ deny, readable, writable }, network, cwd) {
   return out;
 }
 
-/** `bin argv` wrapped in the platform's sandbox: [bin, argv]. */
+/**
+ * The shell `placement` runs a command line in: /bin/sh and /bin/bash on
+ * macOS and Linux, Git for Windows' bash on Windows (what Claude Code's
+ * Bash tool runs there).
+ */
+export function shellBin(name) {
+  if (!WINDOWS) return `/bin/${name}`;
+  const bash = gitBash();
+  if (bash === null) throw new Error("No bash: install Git for Windows, or set CLAUDE_CODE_GIT_BASH_PATH to its bash.exe.");
+  return bash;
+}
+/** `bin argv` wrapped in the platform's sandbox: [bin, argv]. Windows has none: [bin, argv] as given. */
 export function placement(bin, argv, { cwd, state, read = [], network = true }) {
+  if (!OS_SANDBOX) {
+    warnUnsandboxed();
+    return [bin, argv];
+  }
   const paths = placementPaths({ cwd, state, read });
   if (process.platform === "darwin") return ["/usr/bin/sandbox-exec", ["-p", seatbeltProfile(paths, network), bin, ...argv]];
-  if (process.platform === "linux") {
-    const found = spawnSync("sh", ["-c", "command -v bwrap"], { encoding: "utf8" }).stdout.trim();
-    if (found === "") throw new Error("No bwrap on PATH: a non-Claude backend on Linux needs bubblewrap for its sandbox.");
-    return [found, [...bwrapArgs(paths, network, cwd), "--", bin, ...argv]];
-  }
-  throw new Error(`No sandbox for ${process.platform}: non-Claude backends run on macOS or Linux.`);
+  const found = which("bwrap");
+  if (found === null) throw new Error("No bwrap on PATH: a non-Claude backend on Linux needs bubblewrap for its sandbox.");
+  return [found, [...bwrapArgs(paths, network, cwd), "--", bin, ...argv]];
 }
 
 /** A session's state folder, kept across a resume of the same transcript and cleared on a fresh start. */
@@ -223,13 +240,25 @@ function stateOf(transcript, resume) {
   return dir;
 }
 
-/** The environment a placed process starts with: nothing of the caller's but the basics. */
+/**
+ * The environment a placed process starts with: nothing of the caller's
+ * but the basics, plus on Windows what any process there needs to start
+ * (SystemRoot, ComSpec, PATHEXT) and the state folder as its profile and
+ * temp folder too.
+ */
 function childEnv(state, extra = {}) {
-  const keep = ["PATH", "LANG", "LC_ALL", "LC_CTYPE", "TERM", "USER", "LOGNAME", "SHELL"];
+  const keep = ["LANG", "LC_ALL", "LC_CTYPE", "TERM", "USER", "LOGNAME", "SHELL"];
+  const windows = ["SystemRoot", "SystemDrive", "windir", "ComSpec", "PATHEXT", "USERNAME", "NUMBER_OF_PROCESSORS", "PROCESSOR_ARCHITECTURE"];
+  const home = join(state, "home");
+  const temp = join(state, "tmp");
   return {
-    ...Object.fromEntries(keep.filter((key) => process.env[key] !== undefined).map((key) => [key, process.env[key]])),
-    HOME: join(state, "home"),
-    TMPDIR: join(state, "tmp"),
+    [pathKey(process.env)]: process.env[pathKey(process.env)],
+    ...Object.fromEntries(
+      [...keep, ...(WINDOWS ? windows : [])].filter((key) => process.env[key] !== undefined).map((key) => [key, process.env[key]]),
+    ),
+    HOME: home,
+    TMPDIR: temp,
+    ...(WINDOWS ? { USERPROFILE: home, TEMP: temp, TMP: temp, APPDATA: join(home, "AppData", "Roaming"), LOCALAPPDATA: join(home, "AppData", "Local") } : {}),
     ...extra,
   };
 }
@@ -249,7 +278,7 @@ const LIMITED = /usage limit|rate.?limit|quota|too many requests|429|out of cred
 function runChild(bin, argv, { cwd, env, input, minutes, onLine }) {
   return new Promise((done) => {
     // A group leader, its group killed when it closes: nothing it started outlives the session.
-    const child = spawn(bin, argv, { cwd, env, stdio: ["pipe", "pipe", "pipe"], detached: true });
+    const child = spawn(bin, argv, { cwd, env, stdio: ["pipe", "pipe", "pipe"], ...GROUP });
     const reap = () => killGroup(child.pid);
     process.once("exit", reap);
     child.on("exit", reap);
@@ -437,7 +466,7 @@ async function execBackend({ cwd, text, transcript, kind, resume, chain, command
     return { ...base, ok: false, exitCode: null, timedOut: false, durationMs: 0, subtype: "error_during_execution", finalText, stderr: "" };
   }
   const state = stateOf(transcript, resume);
-  const [bin, argv] = placement("/bin/sh", ["-c", line], { cwd, state });
+  const [bin, argv] = placement(shellBin("sh"), ["-c", line], { cwd, state });
   const passed = (process.env.INDEPENDENT_TESTS_EXEC_ENV ?? "").split(",").filter((key) => key && process.env[key] !== undefined);
   const env = childEnv(state, Object.fromEntries(passed.map((key) => [key, process.env[key]])));
   write({ type: "system", subtype: "init", session_id: sessionId, model: null, cwd, backend: "exec", command: line });
@@ -559,7 +588,7 @@ function fileTools(root, runBash) {
     if (!inside(real(full), realRoot)) throw new Error(`${path} is outside the working directory ${root}`);
     return full;
   };
-  const shown = (full) => relative(root, full) || ".";
+  const shown = (full) => slash(relative(root, full)) || ".";
   return {
     Bash: (input) => runBash(String(input.command ?? ""), Math.min(Number(input.timeout ?? 120_000), 600_000)),
     Read: (input) => {
@@ -592,7 +621,7 @@ function fileTools(root, runBash) {
       const base = confined(input.path);
       const matches = globRegExp(String(input.pattern));
       const hits = listFiles(base).filter((path) => matches.test(path) || (!String(input.pattern).includes("/") && matches.test(basename(path))));
-      return hits.slice(0, 500).map((path) => join(shown(base) === "." ? "" : shown(base), path)).join("\n") || "no files";
+      return hits.slice(0, 500).map((path) => posix.join(shown(base) === "." ? "" : shown(base), path)).join("\n") || "no files";
     },
     Grep: (input) => {
       const base = confined(input.path);
@@ -601,7 +630,7 @@ function fileTools(root, runBash) {
       const out = [];
       const files = statSync(base).isDirectory() ? listFiles(base).map((path) => join(base, path)) : [base];
       for (const full of files) {
-        if (only !== null && !only.test(relative(base, full)) && !only.test(basename(full))) continue;
+        if (only !== null && !only.test(slash(relative(base, full))) && !only.test(basename(full))) continue;
         if (statSync(full).size > 1_000_000) continue;
         const lines = readFileSync(full, "utf8").split("\n");
         lines.forEach((line, at) => {
@@ -629,10 +658,11 @@ async function anthropicApi({ cwd, text, transcript, kind, resume, model, maxTur
   const started = Date.now();
   const deadline = started + minutes * 60_000;
   const runBash = (command, timeout) => {
-    const [bin, argv] = placement("/bin/bash", ["-c", command], { cwd, state, network: false });
+    const [bin, argv] = placement(shellBin("bash"), ["-c", command], { cwd, state, network: false });
     return new Promise((done) => {
       let output = "";
-      const child = spawn(bin, argv, { cwd, env: childEnv(state), stdio: ["ignore", "pipe", "pipe"] });
+      const child = spawn(bin, argv, { cwd, env: childEnv(state), stdio: ["ignore", "pipe", "pipe"], windowsHide: true });
+
       const take = (chunk) => (output = (output + chunk).slice(-30_000));
       child.stdout.on("data", take);
       child.stderr.on("data", take);

package/bin/generate.mjs

@@ -29,7 +29,7 @@
 // uses the same model as every other session.
 
 import { existsSync, readFileSync, rmSync } from "node:fs";
-import { basename, dirname, join, relative } from "node:path";
+import { join, posix } from "node:path";
 import {
   RUNS,
   args,
@@ -74,7 +74,7 @@ function suitesOf(chain) {
       dir: "tests/public",
       folder: publicFolder,
       own: repo
-        ? (path) => dirname(path) === publicFolder && chain.isPublicName(basename(path)) && test(path)
+        ? (path) => posix.dirname(path) === publicFolder && chain.isPublicName(posix.basename(path)) && test(path)
         : (path) => path.startsWith(`${publicFolder}/`) && test(path),
       required: repo ? `${publicFolder}/${chain.publicName} (${chain.testPattern})` : `${publicFolder}/${chain.testPattern}`,
       prompt: "public-tests",
@@ -255,7 +255,7 @@ async function generate(slug, force, only = null) {
     if (record[suite.name].problem === null) {
       rmSync(join(out, target), { recursive: true, force: true });
       const files = Object.keys(manifestOf(dir, chain)).filter(inSuite);
-      for (const path of files) copy(join(dir, path), join(out, target, relative(suite.folder, path)));
+      for (const path of files) copy(join(dir, path), join(out, target, posix.relative(suite.folder, path)));
       record[suite.name].files = files;
       // A hidden suite written against an older public suite is stale.
       if (suite.name === "public")

package/bin/handout.mjs

@@ -26,7 +26,7 @@
 import { spawnSync } from "node:child_process";
 import { commitment, recordCommitment } from "./commitment.mjs";
 import { existsSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
-import { join, resolve } from "node:path";
+import { join, posix, resolve, sep } from "node:path";
 import {
   AGENT,
   PRIVATE,
@@ -58,7 +58,7 @@ if (options.repo !== undefined) {
   process.exit(0);
 }
 const dir = resolve(target);
-const beside = PRIVATE.find((folder) => dir === folder || dir.startsWith(`${folder}/`));
+const beside = PRIVATE.find((folder) => dir === folder || dir.startsWith(`${folder}${sep}`));
 if (beside !== undefined) {
   console.error(`${dir} is inside ${beside}, beside the hidden suites; pick a folder outside it.`);
   process.exit(1);
@@ -112,7 +112,7 @@ function repoHandout(checkoutPath, remove) {
   const files = [
     ...chain.seam.map((path) => ({ path, from: join(spec, "skeleton", path), seam: true })),
     ...listFiles(join(spec, "tests", "public")).map((name) => ({
-      path: join(chain.publicDir, name),
+      path: posix.join(chain.publicDir, name),
       from: join(spec, "tests", "public", name),
       seam: false,
     })),