autor3search-typescript 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Gal Be
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,210 @@
+# autor3search-typescript
+
+Let an AI coding agent loose on your repository's performance, without letting it grade its own homework. `autor3search-typescript` freezes your tests and benchmarks, measures every change the agent proposes against a baseline it cannot touch, and only keeps a commit that is a real, statistically significant, material win.
+
+Licensed under the [MIT License](./LICENSE). Copyright (c) 2026 Gal Be.
+
+## Start here
+
+Open your coding agent inside the TypeScript repository you want to make faster, and
+paste this:
+
+```text
+Install and run autor3search-typescript on this repository, then optimize it.
+
+Setup:
+1. npm install --save-dev autor3search-typescript
+2. npx autor3search-typescript init
+   Show me the benchmarks it discovered. If it reports none, STOP and tell me:
+   this tool can only optimize what it can measure.
+3. git add -A && git commit -m "autor3search-typescript init"
+4. npx autor3search-typescript doctor
+   Show me any warnings. If the machine looks unfit to measure, stop and ask me
+   before continuing.
+5. npx autor3search-typescript baseline -tag <today, e.g. sep7>
+
+Then:
+6. Read program.md in this repository, in full. It is your instruction set for
+   the rest of this run. Follow it exactly.
+
+Rules for the whole run:
+- One hypothesis per commit. Commit before each experiment, then run
+  `npx autor3search-typescript eval --json` and apply its verdict before touching
+  anything else: KEEP means the commit stays; anything else (DISCARD, FAIL, CRASH)
+  means `git reset --hard HEAD~1`.
+- Never edit program.md, .autor3search/config.yaml, results.tsv, any
+  *.test.*/*.spec.*/*.bench.* file, package.json, a lockfile, or tsconfig.json.
+  They are not yours.
+- Never pass -force to any autor3search-typescript command. (I may run
+  `autor3search-typescript stop -force` myself; that one is mine, not yours.)
+- Print one context line before each experiment, so I can see where you are:
+  [exp <n> | <branch> | vs <measure_commit> | stop: npx autor3search-typescript stop]
+
+Run the loop until I stop you. I stop you by running
+`npx autor3search-typescript stop` in my own terminal — you will see it as
+"stop_requested": true in a verdict. When you do: apply that verdict, do not
+start another experiment, run `npx autor3search-typescript report`, summarize
+what you tried, and exit the loop.
+```
+
+That's the whole handoff. The agent installs the tool, discovers your benchmarks,
+freezes a baseline, and then follows `program.md` — generated for your repository by
+`init` — which tells it how to run the keep-or-discard loop. `program.md` names the
+benchmarks in scope, spells out the KEEP/DISCARD/FAIL/CRASH contract, lists everything
+the agent must never touch, and ends with a bank of generic V8/TypeScript performance
+ideas for when the agent is out of hypotheses.
+
+Two things worth knowing before you start it:
+
+- **It needs benchmarks.** `init` refuses to run without a discovered `bench*`
+  function in a `*.bench.ts` file — the tool optimizes what it can measure, and
+  refuses to guess.
+- **Numbers are only as good as the machine.** Run `doctor` and read it. See
+  Limitations below for how much run-to-run noise a JS runtime can add even on an
+  idle machine.
+
+## What you get back
+
+An unattended loop that, commit by commit:
+
+- measures the agent's candidate against a frozen baseline with real, repeated, interleaved timing rounds and a rank-sum significance test — not a single before/after number;
+- restores any test, spec or benchmark file the agent edited before running it, so a weakened test can never manufacture a KEEP;
+- refuses (FAILs) a change that touches `package.json`, a lockfile, `tsconfig.json`, or a file outside the declared `scope`, before anything is even measured;
+- refuses a change that adds a new test/bench file the baseline never saw, closing "add an easier benchmark";
+- appends one row per experiment to `results.tsv`, and reports a cumulative, compounding speedup across the whole run.
+
+## Who owns what
+
+| | you edit | the agent edits |
+|---|---|---|
+| `program.md` | yes, before handing it over — never after | never (enforced: editing it is a scope violation) |
+| `.autor3search/config.yaml` | yes | never (enforced: a config hash mismatch FAILs the experiment) |
+| source files under `scope` | rarely | yes — this is the whole point |
+| `*.test.ts`, `*.spec.ts`, `*.bench.ts` | yes | never (enforced: frozen content is restored before every measurement) |
+| `package.json`, lockfiles, `tsconfig.json` | yes | never (enforced: immutable regardless of `scope`) |
+| `results.tsv` | never, ordinarily | not prevented, but pointless — it is gitignored and untracked, so no gate ever sees it as "changed," and it is never read back to decide a verdict; editing it corrupts a human-readable log, nothing more |
+| baselines, locks, stop requests | never — this is what the agent cannot reach | never |
+
+The measurement state — baselines, locks, stop requests — lives outside the repository entirely (see "Where run state lives" below). If the agent could write any of it, it could grade its own work.
+
+## Quick start
+
+```bash
+npm install --save-dev autor3search-typescript
+npx autor3search-typescript init
+# review .autor3search/config.yaml and program.md, then:
+git add .autor3search/config.yaml program.md .gitignore && git commit -m "chore: add autor3search-typescript"
+npx autor3search-typescript baseline -tag <tag>
+# hand the repo and program.md to your agent, using the prompt above
+```
+
+`init` refuses to run if it finds no exported `bench*` function in a `*.bench.ts` file, or no `test` script in `package.json` — this tool has nothing to gate or measure without both.
+
+To install from `main` ahead of a release instead of the last published version, use
+`npm install --save-dev g4lb/autor3search-typescript` — the GitHub form works too,
+just builds the tool on the way in instead of using a prebuilt tarball.
+
+## Watching and stopping a run
+
+```bash
+npx autor3search-typescript status -tag <tag>   # branch, commits, worktree, experiment counts, in-flight eval — read-only
+npx autor3search-typescript stop -tag <tag>     # ask the agent to stop after its current experiment
+npx autor3search-typescript stop -tag <tag> -force   # also signal the running eval to abandon it now
+npx autor3search-typescript stop -tag <tag> -clear   # cancel a pending stop
+```
+
+`stop` never drops a commit for you — it only asks, or signals, and then prints the `git reset --hard HEAD~1` that would drop the abandoned experiment, for you to run yourself.
+
+## Commands
+
+| command | what it does |
+|---|---|
+| `init` | Discover benchmarks and write `.autor3search/config.yaml` and `program.md` |
+| `doctor` | Report whether this machine can measure reliably (informational, always exits 0) |
+| `baseline` | Freeze tests/benchmarks, pin a worktree at HEAD, install and prove it can measure |
+| `eval` | Run one experiment through the gate chain and report a verdict (0 KEEP, 1 DISCARD, 2 FAIL, 3 CRASH) |
+| `status` | Report where a run is: branch, commits, worktree, experiment counts, in-flight eval, pending stop |
+| `stop` | Ask the agent to stop after its current experiment; `-clear` cancels, `-force` also signals the running eval |
+| `report` | Summarize `results.tsv`: counts by status, cumulative speedup, largest individual wins |
+| `profile` | Run the declared benchmarks under Node's CPU profiler and print the hottest functions |
+
+Every command accepts a leading `-C <dir>` to run as if invoked from `<dir>` (its git repository root is resolved from there).
+
+## Where run state lives
+
+Everything the verdict depends on — the frozen manifest, the baseline record, the eval lock, stop requests — lives under the OS cache directory (`~/Library/Caches` on macOS, `$XDG_CACHE_HOME` or `~/.cache` on Linux, `%LOCALAPPDATA%` on Windows), keyed by the repository's own canonical path and the `-tag` you chose, never inside the repository itself. `results.tsv` and `run.log` also live in the repository but are gitignored and untracked — plain, human-readable output, not gated artifacts. `.autor3search/config.yaml` is the one exception: `init` writes `.autor3search/*` to `.gitignore` with a `!.autor3search/config.yaml` negation, so the run configuration itself is committed to version history (a KEEP has to stay reproducible and auditable later), while only its hash — not its content — is what the gate chain actually trusts; a hand-edit to it fails the next `eval` rather than silently loosening it.
+
+## The worked example
+
+This is one real, unmodified run of this tool against the demo fixture shipped in `testdata/demo/`: a `countWords` function whose hot loop rebuilds a whole new array on every character —
+
+```ts
+// before
+chars = chars.concat([c])   // allocates and copies a new array per character: O(n^2) per word
+
+// after
+chars.push(c)                // O(1) amortized per character: O(n) per word
+```
+
+This is deliberately **not** a string-concatenation fix. Plain `+=` on a JS string is not the bug here — V8 represents concatenated strings as ropes, so it never re-copies on every append the way Go's naked string concatenation does. The bug is expressed with an array instead, because that is what actually costs O(n²) in this runtime.
+
+Running `autor3search-typescript eval --json` against that one commit, on the machine this project was built on:
+
+| | value |
+|---|---|
+| baseline | 3.59 ms/op |
+| candidate | 268 µs/op |
+| improvement | 92.55% |
+| verdict | **KEEP** (exit 0) |
+
+The same live session then correctly **DISCARDed** a genuine no-op commit (a comment-only change) measured against that *just-advanced* baseline — proving `measureCommit` really moved and the no-op was not coasting on the earlier win — and **FAILed** a scope violation. All three exit codes (0, 1, 2) were observed in that one live run, and the automated end-to-end test in `src/e2e/full-run.test.ts` reproduces the KEEP-then-DISCARD sequence on every run of this project's own test suite.
+
+**How do we know the measurements themselves mean anything, not just that the harness runs?** The measurement layer was checked against a known quantity, not just against itself: a hand-timed 5 ms busy-spin measured back at **5.0002 ms/op**, and a hand-timed 1 ms spin at **1.0001 ms/op**. That is the strongest evidence in this project that the numbers are real — it is checkable, not merely self-consistent.
+
+## What the harness enforces
+
+| if the agent tries to... | what happens |
+|---|---|
+| edit `package.json`, a lockfile, or `tsconfig.json` | FAIL, unconditionally — immutable regardless of `scope` |
+| edit a file outside `scope` | FAIL — scope violation |
+| edit a frozen test/spec/bench file to make it pass trivially | the real, original content is restored before the test ever runs; a `WARNING:` line reports what was restored |
+| edit `.autor3search/config.yaml` (e.g. loosen `min_effect_pct`) | FAIL — the config hash no longer matches the baseline's |
+| add a new, easier benchmark not present at baseline | FAIL — unmanifested file |
+| commit source AND a slowed-down benchmark body together, hoping the slow body inflates its own future baseline | after a KEEP, the base worktree is repointed to the candidate commit and then the frozen benchmark bytes are immediately restored over it — so the tampered bench body never survives into the new baseline; every future comparison still measures the same original frozen bytes on both sides |
+| let its own change fail the repo's typecheck, build, or test command | FAIL, before anything is measured |
+| edit `program.md` | FAIL — scope violation, never in `scope` |
+| edit `results.tsv` | nothing stops this, and nothing needs to: it is a gitignored, untracked, human-readable log the harness only ever appends to, never a gated or tamper-evident artifact, and no gate reads it back to make a decision |
+| kill the running `eval` process (e.g. `stop -force`) while a benchmark is measuring | the detached measurement child (its own process group) is signalled and the eval lock is released before the process exits, so neither is left orphaned |
+| kill and restart the measurement process to hide a crash | CRASH is reported (exit 3), distinct from FAIL — the harness does not let a crashed child silently become "no verdict" |
+
+Anti-cheat evidence, not assertion: each row above was verified by mutation testing during development — the guard was removed, the attack was demonstrated to succeed, the guard was restored, and the attack failed again. Recorded examples: removing the measurement-commit advance made a comment-only no-op commit report KEEP; removing the post-KEEP worktree restore left a doctored benchmark in the base worktree, inflating every subsequent comparison; removing the symlink guard let a content-matching symlink survive `restore` and be mutated afterward; removing the process-group kill left a live orphaned benchmark process running after the harness exited.
+
+## Scoring
+
+Each declared benchmark is measured `count` times per side (baseline and candidate), interleaved and alternating which side goes first each round to cancel ordering bias, using a Mann-Whitney rank-sum test per benchmark and a Bonferroni correction across however many benchmarks are compared. A commit KEEPs only if:
+
+1. at least one benchmark shows a significant (`p < ALPHA / k`) improvement, and
+2. no benchmark shows a significant regression past `max_regress_pct`, and
+3. the geometric mean of all candidate/baseline ratios beats `min_effect_pct`.
+
+`count` below 4 is refused outright at config-load time: with fewer than 4 rounds per side, the exact rank-sum test cannot report `p < 0.05` no matter how large the true effect is, so every experiment would DISCARD regardless of what changed. The shipped default is 10.
+
+## Limitations — read this before trusting an overnight run
+
+**A KEEP is evidence, not proof.** Any fixed statistical threshold admits false positives; a KEEP means "this cleared the bar the config set," not "this change is definitely faster in production."
+
+**The candidate side's `node_modules` is not integrity-verified.** A Go project gets this for free from `go.sum`, which is checked on every build. npm verifies package integrity only at install time (against `package-lock.json`'s hashes) — once installed, nothing here re-checks that `node_modules` on disk still matches the lockfile before measuring against it.
+
+**JavaScript measurement is noisier than Go's.** JIT warm-up state, garbage-collection timing, and (on Apple Silicon) P/E-core scheduling all add variance a Go binary's more uniform runtime does not have to the same degree. `doctor` reports what it can about the current machine's load, power source, and core mix, but cannot eliminate any of this.
+
+**Concretely, on this project's own dev machine:** three independent measurements of the exact same fixture, on the exact same code, landed at 3.1, 3.198 and 3.59 ms/op — a spread of roughly 15% from run to run, with nothing else on the machine changed. That is the noise floor this tool exists to reason about statistically rather than paper over; it is also why the numbers above are presented as one coherent run, not averaged together into a false precision no single run actually produced.
+
+**No benchmarks, no value.** This tool cannot invent something to measure. It only ever gates and reports what your own `*.bench.ts` files already exercise.
+
+**Discovery is textual, not semantic.** It finds an exported `bench*` function in a `*.bench.ts` file. It does **not** find `export const benchX = someFactory()`, and it does not follow a cross-module re-export of a benchmark function — if your benchmark isn't a literal `export function bench...() { ... }` in the file that declares it, it will not be discovered.
+
+**Microbenchmarks are not your application.** A benchmark that KEEPs proves the benchmark got faster under the exact conditions it constructs. Whether that improvement is visible in your actual application depends on how hot that code path really is there — something this tool has no way to know.
+
+**`count` below 4 can never reach significance,** for the reason given under Scoring above, which is why it is refused at config-load time rather than silently producing a run that can only ever DISCARD.
+
+A performance tool that oversells its own certainty is worse than useless: it launders noise into a verdict a human then trusts. Every claim above is stated as plainly as it can be for exactly that reason.

package/dist/benchproto/types.d.ts

@@ -0,0 +1,32 @@
+/** One benchmark measured once, as written by the child process. */
+export type BenchResult = {
+    ok: true;
+    id: string;
+    /** Mean nanoseconds per operation across the measured window. */
+    nsPerOp: number;
+    iterations: number;
+    batches: number;
+    elapsedMs: number;
+    /** typeof the sink value, proving the return value was consumed. */
+    sinkType: string;
+} | {
+    ok: false;
+    id: string;
+    error: string;
+};
+/**
+ * Parses one child result, validating just enough that a caller never
+ * silently treats corrupt data as a real measurement.
+ *
+ * Only `ok` and `id` were checked before this fix -- documented as an
+ * accepted limitation for a self-produced protocol, until the final
+ * whole-branch review reclassified it as a defect: a SIGKILL mid-`writeFile`
+ * (the timeout path, or an OS-level kill) can land after the JSON document
+ * is syntactically complete but before every field inside it was written --
+ * e.g. an atomic-rename race that leaves `"nsPerOp":` followed by a
+ * half-written number, or a field simply absent from an otherwise
+ * well-formed object. `JSON.parse` succeeds either way, so only an explicit
+ * check here stops `nsPerOp: undefined` (or `NaN`, or `-1`) from reaching
+ * the statistics layer looking like a legitimate zero-cost measurement.
+ */
+export declare function parseBenchResult(text: string): BenchResult;

package/dist/benchproto/types.js

@@ -0,0 +1,28 @@
+/**
+ * Parses one child result, validating just enough that a caller never
+ * silently treats corrupt data as a real measurement.
+ *
+ * Only `ok` and `id` were checked before this fix -- documented as an
+ * accepted limitation for a self-produced protocol, until the final
+ * whole-branch review reclassified it as a defect: a SIGKILL mid-`writeFile`
+ * (the timeout path, or an OS-level kill) can land after the JSON document
+ * is syntactically complete but before every field inside it was written --
+ * e.g. an atomic-rename race that leaves `"nsPerOp":` followed by a
+ * half-written number, or a field simply absent from an otherwise
+ * well-formed object. `JSON.parse` succeeds either way, so only an explicit
+ * check here stops `nsPerOp: undefined` (or `NaN`, or `-1`) from reaching
+ * the statistics layer looking like a legitimate zero-cost measurement.
+ */
+export function parseBenchResult(text) {
+    const v = JSON.parse(text);
+    if (typeof v !== 'object' || v === null || !('ok' in v) || !('id' in v)) {
+        throw new Error(`malformed benchmark result: ${text.slice(0, 200)}`);
+    }
+    const r = v;
+    if (r.ok === true && !(typeof r.nsPerOp === 'number' && Number.isFinite(r.nsPerOp) && r.nsPerOp > 0)) {
+        throw new Error(`malformed benchmark result: ok:true but nsPerOp is not a finite positive number ` +
+            `(got ${JSON.stringify(v.nsPerOp)}): ${text.slice(0, 200)}`);
+    }
+    return r;
+}
+//# sourceMappingURL=types.js.map

package/dist/benchproto/types.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.js","sourceRoot":"","sources":["../../src/benchproto/types.ts"],"names":[],"mappings":"AAeA;;;;;;;;;;;;;;GAcG;AACH,MAAM,UAAU,gBAAgB,CAAC,IAAY;IAC3C,MAAM,CAAC,GAAY,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAA;IACnC,IAAI,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC,EAAE,CAAC;QACxE,MAAM,IAAI,KAAK,CAAC,+BAA+B,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAA;IACtE,CAAC;IACD,MAAM,CAAC,GAAG,CAAgB,CAAA;IAC1B,IAAI,CAAC,CAAC,EAAE,KAAK,IAAI,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC,OAAO,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,OAAO,GAAG,CAAC,CAAC,EAAE,CAAC;QACrG,MAAM,IAAI,KAAK,CACb,kFAAkF;YAChF,QAAQ,IAAI,CAAC,SAAS,CAAE,CAA2B,CAAC,OAAO,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CACzF,CAAA;IACH,CAAC;IACD,OAAO,CAAC,CAAA;AACV,CAAC"}

package/dist/benchrun/child.d.ts

@@ -0,0 +1 @@
+export {};

package/dist/benchrun/child.js

@@ -0,0 +1,222 @@
+/**
+ * The measurement child.
+ *
+ * Runs exactly one benchmark function in a fresh process and writes one JSON
+ * result to --out. Never writes its result to stdout: the module under
+ * measurement is arbitrary repository code and is free to print (a stray
+ * console.log during import, module init, or the benchmark body itself), so
+ * a protocol that parsed stdout would be broken by one such line. --out is
+ * a file this process alone controls the contents of.
+ */
+import { writeFile } from 'node:fs/promises';
+import { parseArgs } from 'node:util';
+import { pathToFileURL } from 'node:url';
+/** Batches shorter than this are dominated by timer resolution. */
+const MIN_BATCH_NS = 10000000n; // 10ms
+/**
+ * Written on every call, read once at the end via `sinkType`. Consuming the
+ * return value this way is cheap insurance against an optimizer deciding an
+ * unread result is dead and eliding the work that produced it. Measured
+ * directly (see task-12-report.md): the specific V8 build and benchmark
+ * shapes tested so far did NOT eliminate the loop even with the sink
+ * removed, so this is not known to be load-bearing today -- it is kept
+ * because that is a fact about one V8 version and a handful of shapes, not
+ * a guarantee, and the cost of keeping it is one assignment and one read.
+ */
+let sink;
+/** True for a promise or any other thenable a benchmark might return. */
+function isThenable(v) {
+    return v !== null && typeof v === 'object' && typeof v.then === 'function';
+}
+/**
+ * Arbitrary repository code can `throw` anything, not just an `Error` --
+ * `throw null` or `throw 'boom'` are both legal JS. `(e as Error).message`
+ * on a non-Error throws its own TypeError, which would crash this process
+ * with no --out file written at all: exactly the crash this module exists
+ * to avoid.
+ */
+function messageOf(e) {
+    return e instanceof Error ? e.message : String(e);
+}
+function runBatchSync(fn, n) {
+    const start = process.hrtime.bigint();
+    for (let i = 0; i < n; i++)
+        sink = fn();
+    return process.hrtime.bigint() - start;
+}
+async function runBatchAsync(fn, n) {
+    const start = process.hrtime.bigint();
+    for (let i = 0; i < n; i++)
+        sink = await fn();
+    return process.hrtime.bigint() - start;
+}
+function isNonNegativeInteger(n) {
+    return Number.isFinite(n) && Number.isInteger(n) && n >= 0;
+}
+/**
+ * Parses and validates argv. --out and --id are checked first because they
+ * are what let any OTHER failure be reported at all -- without an output
+ * path or an id to put in it, an invalid-invocation result cannot be
+ * attributed to any benchmark, so their absence is fatal (caller exits
+ * non-zero) rather than reported as a benchmark result the way every other
+ * validation failure is.
+ */
+function parseArgv() {
+    const { values } = parseArgs({
+        options: {
+            file: { type: 'string' },
+            fn: { type: 'string' },
+            id: { type: 'string' },
+            'benchtime-ms': { type: 'string' },
+            'warmup-ms': { type: 'string' },
+            out: { type: 'string' },
+        },
+    });
+    const out = values.out;
+    const id = values.id;
+    if (typeof out !== 'string' || out === '' || typeof id !== 'string' || id === '') {
+        return { ok: false, attributable: false, message: 'missing --out or --id' };
+    }
+    const file = values.file;
+    if (typeof file !== 'string' || file === '') {
+        return { ok: false, attributable: true, id, out, message: 'missing --file' };
+    }
+    const fnName = values.fn;
+    if (typeof fnName !== 'string' || fnName === '') {
+        return { ok: false, attributable: true, id, out, message: 'missing --fn' };
+    }
+    const benchtimeMs = Number(values['benchtime-ms']);
+    if (!isNonNegativeInteger(benchtimeMs)) {
+        return {
+            ok: false,
+            attributable: true,
+            id,
+            out,
+            message: `--benchtime-ms must be a non-negative integer, got ${JSON.stringify(values['benchtime-ms'])}`,
+        };
+    }
+    const warmupMs = Number(values['warmup-ms']);
+    if (!isNonNegativeInteger(warmupMs)) {
+        return {
+            ok: false,
+            attributable: true,
+            id,
+            out,
+            message: `--warmup-ms must be a non-negative integer, got ${JSON.stringify(values['warmup-ms'])}`,
+        };
+    }
+    return { ok: true, args: { file, fnName, id, benchtimeMs, warmupMs, out } };
+}
+async function measure(args) {
+    let fn;
+    try {
+        const mod = (await import(pathToFileURL(args.file).href));
+        const candidate = mod[args.fnName];
+        if (typeof candidate !== 'function') {
+            return { ok: false, id: args.id, error: `${args.file} does not export a function named ${args.fnName}` };
+        }
+        fn = candidate;
+    }
+    catch (e) {
+        return { ok: false, id: args.id, error: `import ${args.file}: ${messageOf(e)}` };
+    }
+    try {
+        // Probe once to learn whether the benchmark is async. Awaiting a
+        // synchronous function would fold microtask-queue overhead into every
+        // measurement, so the two cases get separate loops: runBatchSync never
+        // awaits, runBatchAsync always does.
+        const probe = fn();
+        const isAsync = isThenable(probe);
+        if (isAsync)
+            await probe;
+        sink = probe;
+        const batch = isAsync
+            ? (n) => runBatchAsync(fn, n)
+            : (n) => Promise.resolve(runBatchSync(fn, n));
+        // Warm up first: JIT tiering changes the speed we are about to
+        // calibrate to, so calibrating against a cold function would pick a
+        // batch size for code that no longer exists by the time we measure.
+        const warmDeadline = process.hrtime.bigint() + BigInt(args.warmupMs) * 1000000n;
+        while (process.hrtime.bigint() < warmDeadline)
+            await batch(16);
+        // Calibrate a batch size whose duration is well above timer resolution.
+        let n = 1;
+        for (;;) {
+            const took = await batch(n);
+            if (took >= MIN_BATCH_NS || n >= 1_000_000_000)
+                break;
+            n *= 2;
+        }
+        let totalNs = 0n;
+        let iterations = 0;
+        let batches = 0;
+        const deadline = process.hrtime.bigint() + BigInt(args.benchtimeMs) * 1000000n;
+        while (process.hrtime.bigint() < deadline) {
+            totalNs += await batch(n);
+            iterations += n;
+            batches++;
+        }
+        if (iterations === 0) {
+            // benchtime shorter than a single batch: still report one honest batch.
+            totalNs = await batch(n);
+            iterations = n;
+            batches = 1;
+        }
+        // The initial probe is a single, un-timed call: it decides which loop
+        // measures every batch for the rest of the run. A benchmark that took
+        // a synchronous fast path on that one call (a cache hit, an early
+        // return) but later starts returning promises (a cache miss, a lazily
+        // opened connection) would stay on runBatchSync, which never awaits --
+        // silently timing promise *construction* instead of the awaited work,
+        // and reporting a real-looking nsPerOp that is orders of magnitude too
+        // small. One check here, after the loop, is enough to catch it without
+        // adding a per-iteration cost to every measurement.
+        if (!isAsync && isThenable(sink)) {
+            return {
+                ok: false,
+                id: args.id,
+                error: `${args.fnName}: benchmark is conditionally async -- the initial probe call returned a plain value so it was measured synchronously, but a later call returned a promise; measured timings would reflect promise construction, not the awaited work`,
+            };
+        }
+        return {
+            ok: true,
+            id: args.id,
+            nsPerOp: Number(totalNs) / iterations,
+            iterations,
+            batches,
+            elapsedMs: Number(totalNs) / 1_000_000,
+            // typeof the value consumed above -- see the `sink` declaration for
+            // why it is read at all.
+            sinkType: typeof sink,
+        };
+    }
+    catch (e) {
+        return { ok: false, id: args.id, error: `${args.fnName}: ${messageOf(e)}` };
+    }
+}
+async function main() {
+    const parsed = parseArgv();
+    if (!parsed.ok && !parsed.attributable) {
+        // No --out and/or --id: there is nowhere to write a result and nothing
+        // to attribute it to, so this is the one failure mode that is not
+        // reported as a benchmark result.
+        process.stderr.write(`benchrun child: invalid invocation: ${parsed.message}\n`);
+        process.exit(1);
+    }
+    if (parsed.ok) {
+        const result = await measure(parsed.args);
+        await writeFile(parsed.args.out, JSON.stringify(result));
+        return;
+    }
+    await writeFile(parsed.out, JSON.stringify({ ok: false, id: parsed.id, error: `invalid invocation: ${parsed.message}` }));
+}
+await main();
+// A benchmark module can leave the event loop non-empty on its way out (an
+// open timer, a socket, a lingering connection pool) even though the result
+// is already written. Without an explicit exit, node would wait for that
+// handle and the parent would eventually kill this process on timeout,
+// discarding a result that was complete on disk the whole time -- the same
+// "arbitrary repository code" reasoning that justified --out in the first
+// place.
+process.exit(0);
+//# sourceMappingURL=child.js.map

package/dist/benchrun/child.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"child.js","sourceRoot":"","sources":["../../src/benchrun/child.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AACH,OAAO,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAA;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAA;AACrC,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAA;AAGxC,mEAAmE;AACnE,MAAM,YAAY,GAAG,SAAW,CAAA,CAAC,OAAO;AAExC;;;;;;;;;GASG;AACH,IAAI,IAAa,CAAA;AAIjB,yEAAyE;AACzE,SAAS,UAAU,CAAC,CAAU;IAC5B,OAAO,CAAC,KAAK,IAAI,IAAI,OAAO,CAAC,KAAK,QAAQ,IAAI,OAAQ,CAA0B,CAAC,IAAI,KAAK,UAAU,CAAA;AACtG,CAAC;AAED;;;;;;GAMG;AACH,SAAS,SAAS,CAAC,CAAU;IAC3B,OAAO,CAAC,YAAY,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAA;AACnD,CAAC;AAED,SAAS,YAAY,CAAC,EAAS,EAAE,CAAS;IACxC,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,CAAA;IACrC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE;QAAE,IAAI,GAAG,EAAE,EAAE,CAAA;IACvC,OAAO,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,KAAK,CAAA;AACxC,CAAC;AAED,KAAK,UAAU,aAAa,CAAC,EAAS,EAAE,CAAS;IAC/C,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,CAAA;IACrC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE;QAAE,IAAI,GAAG,MAAM,EAAE,EAAE,CAAA;IAC7C,OAAO,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,KAAK,CAAA;AACxC,CAAC;AAWD,SAAS,oBAAoB,CAAC,CAAS;IACrC,OAAO,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;AAC5D,CAAC;AAOD;;;;;;;GAOG;AACH,SAAS,SAAS;IAChB,MAAM,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;QAC3B,OAAO,EAAE;YACP,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YACxB,EAAE,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YACtB,EAAE,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YACtB,cAAc,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAClC,WAAW,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAC/B,GAAG,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;SACxB;KACF,CAAC,CAAA;IAEF,MAAM,GAAG,GAAG,MAAM,CAAC,GAAG,CAAA;IACtB,MAAM,EAAE,GAAG,MAAM,CAAC,EAAE,CAAA;IACpB,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAI,GAAG,KAAK,EAAE,IAAI,OAAO,EAAE,KAAK,QAAQ,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC;QACjF,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,YAAY,EAAE,KAAK,EAAE,OAAO,EAAE,uBAAuB,EAAE,CAAA;IAC7E,CAAC;IAED,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAA;IACxB,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,KAAK,EAAE,EAAE,CAAC;QAC5C,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,YAAY,EAAE,IAAI,EAAE,EAAE,EAAE,GAAG,EAAE,OAAO,EAAE,gBAAgB,EAAE,CAAA;IAC9E,CAAC;IACD,MAAM,MAAM,GAAG,MAAM,CAAC,EAAE,CAAA;IACxB,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,MAAM,KAAK,EAAE,EAAE,CAAC;QAChD,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,YAAY,EAAE,IAAI,EAAE,EAAE,EAAE,GAAG,EAAE,OAAO,EAAE,cAAc,EAAE,CAAA;IAC5E,CAAC;IACD,MAAM,WAAW,GAAG,MAAM,CAAC,MAAM,CAAC,cAAc,CAAC,CAAC,CAAA;IAClD,IAAI,CAAC,oBAAoB,CAAC,WAAW,CAAC,EAAE,CAAC;QACvC,OAAO;YACL,EAAE,EAAE,KAAK;YACT,YAAY,EAAE,IAAI;YAClB,EAAE;YACF,GAAG;YACH,OAAO,EAAE,sDAAsD,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,cAAc,CAAC,CAAC,EAAE;SACxG,CAAA;IACH,CAAC;IACD,MAAM,QAAQ,GAAG,MAAM,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC,CAAA;IAC5C,IAAI,CAAC,oBAAoB,CAAC,QAAQ,CAAC,EAAE,CAAC;QACpC,OAAO;YACL,EAAE,EAAE,KAAK;YACT,YAAY,EAAE,IAAI;YAClB,EAAE;YACF,GAAG;YACH,OAAO,EAAE,mDAAmD,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC,EAAE;SAClG,CAAA;IACH,CAAC;IACD,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,WAAW,EAAE,QAAQ,EAAE,GAAG,EAAE,EAAE,CAAA;AAC7E,CAAC;AAED,KAAK,UAAU,OAAO,CAAC,IAAU;IAC/B,IAAI,EAAS,CAAA;IACb,IAAI,CAAC;QACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,CAA4B,CAAA;QACpF,MAAM,SAAS,GAAG,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAA;QAClC,IAAI,OAAO,SAAS,KAAK,UAAU,EAAE,CAAC;YACpC,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,IAAI,CAAC,EAAE,EAAE,KAAK,EAAE,GAAG,IAAI,CAAC,IAAI,qCAAqC,IAAI,CAAC,MAAM,EAAE,EAAE,CAAA;QAC1G,CAAC;QACD,EAAE,GAAG,SAAkB,CAAA;IACzB,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,IAAI,CAAC,EAAE,EAAE,KAAK,EAAE,UAAU,IAAI,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAA;IAClF,CAAC;IAED,IAAI,CAAC;QACH,iEAAiE;QACjE,sEAAsE;QACtE,uEAAuE;QACvE,qCAAqC;QACrC,MAAM,KAAK,GAAG,EAAE,EAAE,CAAA;QAClB,MAAM,OAAO,GAAG,UAAU,CAAC,KAAK,CAAC,CAAA;QACjC,IAAI,OAAO;YAAE,MAAM,KAAK,CAAA;QACxB,IAAI,GAAG,KAAK,CAAA;QAEZ,MAAM,KAAK,GAAG,OAAO;YACnB,CAAC,CAAC,CAAC,CAAS,EAAmB,EAAE,CAAC,aAAa,CAAC,EAAE,EAAE,CAAC,CAAC;YACtD,CAAC,CAAC,CAAC,CAAS,EAAmB,EAAE,CAAC,OAAO,CAAC,OAAO,CAAC,YAAY,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAA;QAExE,+DAA+D;QAC/D,oEAAoE;QACpE,oEAAoE;QACpE,MAAM,YAAY,GAAG,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,QAAU,CAAA;QACjF,OAAO,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,YAAY;YAAE,MAAM,KAAK,CAAC,EAAE,CAAC,CAAA;QAE9D,wEAAwE;QACxE,IAAI,CAAC,GAAG,CAAC,CAAA;QACT,SAAS,CAAC;YACR,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,CAAA;YAC3B,IAAI,IAAI,IAAI,YAAY,IAAI,CAAC,IAAI,aAAa;gBAAE,MAAK;YACrD,CAAC,IAAI,CAAC,CAAA;QACR,CAAC;QAED,IAAI,OAAO,GAAG,EAAE,CAAA;QAChB,IAAI,UAAU,GAAG,CAAC,CAAA;QAClB,IAAI,OAAO,GAAG,CAAC,CAAA;QACf,MAAM,QAAQ,GAAG,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,MAAM,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,QAAU,CAAA;QAChF,OAAO,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,QAAQ,EAAE,CAAC;YAC1C,OAAO,IAAI,MAAM,KAAK,CAAC,CAAC,CAAC,CAAA;YACzB,UAAU,IAAI,CAAC,CAAA;YACf,OAAO,EAAE,CAAA;QACX,CAAC;QACD,IAAI,UAAU,KAAK,CAAC,EAAE,CAAC;YACrB,wEAAwE;YACxE,OAAO,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,CAAA;YACxB,UAAU,GAAG,CAAC,CAAA;YACd,OAAO,GAAG,CAAC,CAAA;QACb,CAAC;QAED,sEAAsE;QACtE,sEAAsE;QACtE,kEAAkE;QAClE,sEAAsE;QACtE,uEAAuE;QACvE,sEAAsE;QACtE,uEAAuE;QACvE,uEAAuE;QACvE,oDAAoD;QACpD,IAAI,CAAC,OAAO,IAAI,UAAU,CAAC,IAAI,CAAC,EAAE,CAAC;YACjC,OAAO;gBACL,EAAE,EAAE,KAAK;gBACT,EAAE,EAAE,IAAI,CAAC,EAAE;gBACX,KAAK,EAAE,GAAG,IAAI,CAAC,MAAM,sOAAsO;aAC5P,CAAA;QACH,CAAC;QAED,OAAO;YACL,EAAE,EAAE,IAAI;YACR,EAAE,EAAE,IAAI,CAAC,EAAE;YACX,OAAO,EAAE,MAAM,CAAC,OAAO,CAAC,GAAG,UAAU;YACrC,UAAU;YACV,OAAO;YACP,SAAS,EAAE,MAAM,CAAC,OAAO,CAAC,GAAG,SAAS;YACtC,oEAAoE;YACpE,yBAAyB;YACzB,QAAQ,EAAE,OAAO,IAAI;SACtB,CAAA;IACH,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,IAAI,CAAC,EAAE,EAAE,KAAK,EAAE,GAAG,IAAI,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAA;IAC7E,CAAC;AACH,CAAC;AAED,KAAK,UAAU,IAAI;IACjB,MAAM,MAAM,GAAG,SAAS,EAAE,CAAA;IAC1B,IAAI,CAAC,MAAM,CAAC,EAAE,IAAI,CAAC,MAAM,CAAC,YAAY,EAAE,CAAC;QACvC,uEAAuE;QACvE,kEAAkE;QAClE,kCAAkC;QAClC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,uCAAuC,MAAM,CAAC,OAAO,IAAI,CAAC,CAAA;QAC/E,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA;IACjB,CAAC;IACD,IAAI,MAAM,CAAC,EAAE,EAAE,CAAC;QACd,MAAM,MAAM,GAAG,MAAM,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,CAAA;QACzC,MAAM,SAAS,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC,CAAA;QACxD,OAAM;IACR,CAAC;IACD,MAAM,SAAS,CACb,MAAM,CAAC,GAAG,EACV,IAAI,CAAC,SAAS,CAAC,EAAE,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,MAAM,CAAC,EAAE,EAAE,KAAK,EAAE,uBAAuB,MAAM,CAAC,OAAO,EAAE,EAAE,CAAC,CAC7F,CAAA;AACH,CAAC;AAED,MAAM,IAAI,EAAE,CAAA;AACZ,2EAA2E;AAC3E,4EAA4E;AAC5E,yEAAyE;AACzE,uEAAuE;AACvE,2EAA2E;AAC3E,0EAA0E;AAC1E,SAAS;AACT,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA"}

package/dist/benchrun/invoke.d.ts

@@ -0,0 +1,61 @@
+import { type BenchResult } from '../benchproto/types.js';
+import { type ExecResult } from '../runner/exec.js';
+/**
+ * Picks the measurement child from a directory, preferring the built
+ * `child.js` and falling back to the source `child.ts`.
+ *
+ * Under a real install (`dist/`), only `child.js` exists next to this
+ * module -- that is the branch that actually runs for an installed user.
+ * Under Vitest, `import.meta.url` resolves to `src/benchrun/invoke.ts`,
+ * where only `child.ts` exists -- there is no build step before the test
+ * run. Either way the spawn below runs through the tsx loader, so a `.ts`
+ * path works identically to a `.js` one; this just picks whichever file is
+ * actually on disk.
+ *
+ * Pulled apart from `resolveChild` (a thin wrapper below) so both branches
+ * can be exercised directly with an injected existence check, since only
+ * one of the two ever exists in any single checkout.
+ */
+export declare function pickChildPath(dir: string, exists: (p: string) => boolean): string;
+export declare const CHILD: string;
+/**
+ * Reads and parses the child's `--out` file, turning every failure mode
+ * into `{ ok: false }` instead of a rejection: a missing file (the child
+ * died before writing at all) and a present-but-malformed one (the file
+ * exists but doesn't parse, or doesn't have the right shape).
+ *
+ * The malformed case is not exotic: our own timeout path kills the child's
+ * process group with SIGKILL, which can land mid-`writeFile`, leaving a
+ * truncated-but-readable JSON document on disk. `readFile` succeeds on
+ * that; only `JSON.parse`/`parseBenchResult` would have caught it, and
+ * uncaught, that throw would have propagated out of `runChild` as a
+ * rejected promise -- exactly the "harness bug" failure mode `runChild`
+ * exists to avoid for a benchmark-level failure.
+ *
+ * Exported for testing: it lets a test write deliberately truncated JSON
+ * to a real, controlled `--out` path and assert `ok: false` directly,
+ * without needing to race a real subprocess's SIGKILL against its own
+ * `writeFile` to reproduce the truncation.
+ */
+export declare function readChildResult(outFile: string, id: string, r: Pick<ExecResult, 'timedOut' | 'exitCode' | 'stderr'>, timeoutMs: number): Promise<BenchResult>;
+export interface RunChildOptions {
+    /** The worktree to measure in. Sets module resolution and tsconfig context. */
+    cwd: string;
+    benchFileAbs: string;
+    fn: string;
+    id: string;
+    benchtimeMs: number;
+    warmupMs: number;
+    timeoutMs: number;
+    nodeArgs: string[];
+    log?: ((s: string) => void) | undefined;
+}
+/**
+ * Runs one benchmark, once, in a fresh child process.
+ *
+ * Never rejects for a benchmark-level failure: a timeout, a crash, or a
+ * child that exits without writing --out all come back as `{ ok: false }`
+ * with a diagnosis built from the child's own stderr, so a caller can
+ * record a CRASH verdict instead of aborting the whole run.
+ */
+export declare function runChild(o: RunChildOptions): Promise<BenchResult>;