forgedock 1.31.0 → 1.32.1

package/docs/CONFIG.md

@@ -563,11 +563,11 @@ engine:
 | `lane_limits` | map | No | Cap each core lane's whole process tree (#4115), under `pipeline:` or `engine:`: `memory_max` (for example `3G`) and `cpu_quota` (for example `150%`). When set and `systemd-run --user --scope` works (probed once), the builder CLI is spawned as `systemd-run --user --scope -q -p MemoryMax=... -p CPUQuota=... -- claude ...`; elsewhere the caps are ignored. Write it nested or flat (`lane_limits_memory_max`, `lane_limits_cpu_quota`). While a builder runs the engine polls host load and memory every 15 s against the `admission` thresholds: over them it SIGSTOPs the lane's process group, under them SIGCONT, and never pauses longer than 10 minutes. |
 | `admission` | map | No | Host-resource admission for the core work-on (#4105), checked before the lease, worktree or any GitHub call. When a check fails the run ends `deferred` (code `host.busy`, exit 75) with a last line such as `issue #N → deferred (host busy: mem 1.2G < 2G; retry in 60s)`; no lease is taken and nothing is posted. Keys: `enabled` (default `true`; `false` disables admission), `min_free_mem_mb` (default 2048; `MemAvailable` from `/proc/meminfo`, else `os.freemem()`), `min_free_tmp_mb` (default 1024; free space of the run TMPDIR filesystem, `~/.forge/tmp`, and of `/tmp`, via `statfs`), `max_load_per_core` (default 4; 1-minute load average divided by cores). Write it nested (`engine:` then `admission:` with the keys indented) or flat as `admission_enabled`, `admission_min_free_mem_mb`, `admission_min_free_tmp_mb`, `admission_max_load_per_core`; the parser reads one nested level into `engine`. `/orchestrate` requeues a host-busy lane and lowers K by 1 for 5 minutes. |
 | `merge_wait` | string | No | Core work-on merge gate. `required` (default): wait only for the status checks the base branch requires: the union of classic branch protection (`gh api repos/{o}/{r}/branches/{base}/protection/required_status_checks`, contexts and checks) and repository rulesets (`gh api repos/{o}/{r}/rules/branches/{base}`, `required_status_checks` rules); with neither (404 or empty) merge right after review PASS. A merge rejected by base branch policy (PR `MERGEABLE` but `BLOCKED`) is waited on and retried within the CI budget, then left `ci-pending`; it is never base-synced and never `base.conflict`. A failing required check is `needs-human`; a pending one is polled with backoff and never `needs-human`. `all`: wait for every check (`scripts/wait-ci-green.sh`). `none`: merge right after review. Always `--match-head-commit` at the reviewed head. The policy used is recorded in the `FORGE:WORK-ON` comment. |
-| `context_pack` | string | No | Context pack (#4275). Before the plan call the core compiles the issue's GitHub/git neighbourhood into a small, budgeted pack: linked issues and PRs (parent, `Depends on`, body links, same-repo cross-references, siblings), prior attempts, merged PRs on the issue's files that were reverted or whose decisions/findings name a symbol from the issue, structured `FORGE:FACTS` records of earlier lanes on the issue's files (rejected approaches with why, hot spots, decisions, conventions, baselines; #4401), and repo-fact file pointers (found in the lane worktree or, for untracked caches, the main checkout; #4400). `shadow` (default): the pack is compiled, posted on the issue as a collapsed `FORGE:CONTEXT` comment and logged as a `CONTEXT_PACK` event; the builder prompt is unchanged. `on`: it is also appended to the plan prompt as hints. `off`: nothing is gathered, posted or logged. The pack never enters the text contract quotes are checked against. Values read from linked issues' `FORGE:STATE` are kept only when they are outcome codes. Compare with `forgedock stats --context-ab`. Unless `off`, a resumed or recovery lane also gets a bounded resume digest of its last run in the builder prompt (end reason, validate results per command, deliver requirements, files changed; logged as `RESUME_DIGEST`; #4400), whatever the mode. |
+| `context_pack` | string | No | Context pack (#4275). Before the plan call the core compiles the issue's GitHub/git neighbourhood into a small, budgeted pack: linked issues and PRs (parent, `Depends on`, body links, same-repo cross-references, siblings), prior attempts, merged PRs on the issue's files that were reverted or whose decisions/findings name a symbol from the issue, structured `FORGE:FACTS` records of earlier lanes on the issue's files (rejected approaches with why, hot spots, decisions, conventions, baselines; #4401), and repo-fact file pointers (found in the lane worktree or, for untracked caches, the main checkout; #4400). `shadow` (default): the pack is compiled, posted on the issue as a collapsed `FORGE:CONTEXT` comment and logged as a `CONTEXT_PACK` event; the builder prompt is unchanged. `on`: it is also appended to the plan prompt as hints. `off`: nothing is gathered, posted or logged. `ab` (#4444): the A/B experiment. Each issue gets a stable arm, `on` or `shadow`, from a hash of `repo#issue`, logged as that mode with `context_ab: true`; the default stays `shadow`. The pack never enters the text contract quotes are checked against. Values read from linked issues' `FORGE:STATE` are kept only when they are outcome codes. Compare with `forgedock stats --context-ab` (per lane type and mode: first-pass yield, turns and cache-read per merge, rework rate, and recovery lanes' implement attempt against the first run on the same issue). Unless `off`, a resumed or recovery lane also gets a bounded resume digest of its last run in the builder prompt (end reason, validate results per command, deliver requirements, files changed; logged as `RESUME_DIGEST`; #4400), whatever the mode. |
 | `context_pack_sources` | string | No | Comma list of the pack sources to use: `graph`, `attempts`, `pitfalls`, `ledger`, `facts` (earlier lanes' `FORGE:FACTS` on the issue's files plus repo-fact file pointers). Default: all five. |
 | `context_pack_bytes` | integer | No | Hard byte cap of the pack; items that do not fit are dropped whole and counted. Default: 6144. |
 | `context_pack_days` | integer | No | Window of the merged-PR ledger, in days. Default: 30. |
-| `lane_instructions` | string | No | Lane profile (#4395). Core lane sessions always run with Claude Code auto-memory off (`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`). Set this to a repo-relative file, for example `devdocs/lane-instructions.md`, that holds only the hard rules plus pointers into docs. It is read from the lane worktree, so it is versioned with the repo. Lanes then also skip CLAUDE.md discovery (`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`), and the file text is appended to the system prompt of every fresh builder, reviewer and verifier session. An absolute path, a `..` path, or a missing, empty or over-16 KB file is a trajectory anomaly; lanes then keep CLAUDE.md. Settings and hooks always load (no `--setting-sources`). `FORGEDOCK_FULL_SESSION=1` keeps a full session. Each `phase_attempt` telemetry record carries `lane_profile` (`full`, `memory-off` or `lane`) and `baseline_tokens`, the first turn's context of a fresh session, so baseline growth is visible. |
+| `lane_instructions` | string | No | Lane profile (#4395). Core lane sessions always run with Claude Code auto-memory off (`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`). Set this to a repo-relative file, for example `devdocs/lane-instructions.md`, that holds only the hard rules plus pointers into docs. It is read from the lane worktree, so it is versioned with the repo. Lanes then also skip CLAUDE.md discovery (`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`), and the file text is appended to the system prompt of every fresh builder, reviewer and verifier session. An absolute path, a `..` path, or a missing, empty or over-16 KB file is a trajectory anomaly; lanes then keep CLAUDE.md. Settings and hooks always load (no `--setting-sources`). `FORGEDOCK_FULL_SESSION=1` keeps a full session. Each `phase_attempt` telemetry record carries `lane_profile` (`full`, `memory-off` or `lane`) and `baseline_tokens`, the first turn's context of a fresh session, so baseline growth is visible. `forgedock stats --lane-profile [--issues <list>]` compares `baseline_tokens` per phase and the conformance rate per profile from local run logs, for the before/after check on the same issue set. |
 | `ci_pending_max_minutes` | integer | No | Minutes the engine waits for CI when the review phase ends BLOCKED with "checks still pending", or the remediate phase holds at its M7 re-gate on "CI pending" (the merge gate's wait window ended while CI was still running). Default: 90. Invalid values are ignored. This key is flat like the others but is not part of the `<key>_model` / `<key>_max_turns` pattern. |
 
 **CI still pending**: the engine polls `scripts/wait-ci-green.sh` (no model calls) one read per poll on a 60 s, 2 min, 4 min, 5 min (cap) backoff with +/-20% jitter, while the lease keeps renewing, then re-dispatches the phase that was waiting, review or remediate (idempotent at an unchanged head: review re-runs the merge gate and merges; remediate resumes at its M7 re-gate). Before each poll it reads `gh api rate_limit` (not itself rate-limited) and, below 10% of the REST budget, sleeps until the window resets instead of polling; a rate-limited or transient (5xx, network) poll is retried and never re-dispatches the phase. Red CI takes the normal ci-gate path. When the budget is exhausted the run ends `needs-human` with outcome code `review.ci_pending_timeout`; `--retry` resumes. Each wait emits a `ci_pending_wait` telemetry event (`docs/TELEMETRY.md`) with the minutes waited, polls made and rate-limit sleeps.
@@ -613,7 +613,7 @@ lean:
 | `fix_model` | string | No | Model of fresh fix-round sessions (#4363; `engine.lean_fix_model` is also read). Default: the reviewer model (`engine.lean_reviewer_model`, else Sonnet). Fix rounds for precheck, CI, docs, structure, enumerating-test and sequence-number failures, and conformance or review FAILs whose failures all cite a `file:line`, run in a fresh session (no resume of the builder's transcript) whose prompt carries only the round's failure output, the capped `git diff <base>...HEAD`, the frozen contract and the issue title and summary (under 60K characters). A fresh session that makes no commit gets one resume of the original builder session in the same round. Other rounds (block or contract corrections, CI unavailable, base-sync conflicts, failures without a file:line) resume the builder as before; round limits (`verify.max_rounds`, the one review retry) are unchanged. The run log's `PHASE_ATTEMPT_USAGE` records `mode: "fresh"\|"resume"` on each fix-round attempt. Plan and build sessions keep the builder model. |
 | `builder_routing` | string | No | `on` (default) or `off` (#4394). After the contract is frozen, the lane is classed from its deliverable paths only: `risk` (any path with a built-in risk word such as auth, billing, payments, session, secret or migration, or matching a `review.risk_paths` glob), `docs-config-test` (every path is a test, doc or config file; lockfiles count as code), `small` (at most 2 non-test deliverables) or `large` (more, or none). `docs-config-test` and `small` build on the mid tier (`builder_mid_model`); `risk` and `large` build on the top model (`--model`, else the CLI default). The plan and contract-correction calls always run on the top model. A failed validate round on the mid tier (`precheck`, `ci_red`, `merge_ci`) escalates the lane once: that fix round and every later builder call, fresh fix sessions included, run on the top model. Escalation adds no round. Conformance or review fix rounds do not escalate and keep `fix_model`. `off` keeps every builder call on the top model, as before, and still records the class. The run log gets `BUILDER_ROUTE {mode, class, tier, model, risk}` and `BUILDER_ESCALATED {from, to, trigger}`, and `RUN_TERMINAL` gets `builder_tier`, `builder_class`, `builder_escalated` and `builder_model`. `forgedock stats --tiers` reports merge, conformance and post-merge-rework rates, first-pass yield and cost per tier (`mid`, `mid->top`, `top`) and class, from local run logs. Target-repo risk areas (for example detection code) belong in `review.risk_paths`. |
 | `builder_mid_model` | string | No | Mid-tier builder model for `builder_routing` (#4394; `engine.lean_builder_mid_model` is also read). Default `claude-sonnet-5-5`. An invalid id is ignored. |
-| `light.mode` | string | No | `off`, `shadow` (default) or `on` (#4448). `bin/lean/light.mjs` decides whether an issue qualifies for the light lane: a repro, steps to reproduce, expected or actual behaviour or a code block in the body; 1 to `max_files` predicted non-test files; no `size:M`/`size:L` label; no predicted path with a built-in risk word or matching `review.risk_paths`; not an `investigate(` issue, a post-deploy issue (`post-deploy`/`awaiting-promotion` label, or only post-deploy checklist items) or a decision issue (`decision`, `needs-decision`, `product-decision`, `rfc` or `adr` label, or a `decision(`/`rfc(`/`adr(` title). A missing repro or file prediction is `ambiguous`, left to a cheap classifier. `off` is never eligible; `shadow` reports the verdict without changing the lane; `on` lets an eligible issue run light. The lane wiring is #4450: until it lands, no lane changes shape in any mode. The flat form `lean.light_mode` also works and wins. |
+| `light.mode` | string | No | `off`, `shadow` (default) or `on` (#4448). `bin/lean/light.mjs` decides whether an issue qualifies for the light lane: a repro, steps to reproduce, expected or actual behaviour or a code block in the body; 1 to `max_files` predicted non-test files; no `size:M`/`size:L` label; no predicted path with a built-in risk word or matching `review.risk_paths`; not an `investigate(` issue, a post-deploy issue (`post-deploy`/`awaiting-promotion` label, or only post-deploy checklist items) or a decision issue (`decision`, `needs-decision`, `product-decision`, `rfc` or `adr` label, or a `decision(`/`rfc(`/`adr(` title). A missing repro or file prediction is `ambiguous`, left to a cheap classifier. `off` is never eligible; `shadow` reports the verdict without changing the lane; `on` lets an eligible issue run light. The lane wiring is #4450. A light lane posts a `FORGE:LITE` receipt on the issue (outcome light or upgraded, diff stat, red/green repro, gates, true cost from the #4407 per-session deltas, eligibility reason, upgrade reasons), and the batch efficiency report ends with a `lanes:` line: the light/upgraded/full split with cost and the upgrade rate, upgraded ÷ (light + upgraded) (#4452). The flat form `lean.light_mode` also works and wins. |
 | `light.max_lines` | number | No | Upgrade trigger (#4448): a light lane whose non-test diff (added + deleted lines) exceeds this moves to the full lane. Default 40. The other triggers: more than `max_files` non-test changed files, a repro that is not red on the base, two failed test runs, a touched risk path, or a public-API-shape change (#4449). Flat `lean.light_max_lines` wins. |
 | `light.max_files` | number | No | Max predicted non-test files for eligibility and max changed non-test files before an upgrade (#4448). Default 2. Flat `lean.light_max_files` wins. |
 | `sequence.regenerate` | string | No | Command run in the lane worktree (`sh -c`) after the engine renumbers a schema-changing sequence-numbered migration (#4386), to regenerate its snapshot, e.g. `pnpm --filter @x/db db:generate`. The flat form `lean.sequence_regenerate` also works and wins. When a lane-added numbered file (`NNNN_*.sql`, a drizzle `meta/_journal.json` + `meta/NNNN_snapshot.json`, a numbered ADR) shares its number with the base, the engine renumbers it without a builder session: `git mv` to max(base)+1…, the document's own number in its heading, references to the old name repo-wide, the base journal plus the lane's entries, and a data-only migration's snapshot cloned from the base tip with a new `id`/`prevId`. A schema-changing migration without this command, or a migration in another language (`.py`, `.ts`), falls back to the builder renumber, and the trajectory records why. A base-sync whose conflicts are only in such files is resolved the same way, with no re-review. At the merge gate a lane with numbered files holds a host sequence lease per (repo, lane branch, directory) from its last renumber check through the merge, so only one numbered-file PR is in the renumber→CI→merge window at a time. |

package/docs/STATS.md

@@ -78,6 +78,19 @@ Limits: logs written before #4362 carry no trigger, so implement attempts beyond
 `unknown`. Logs written before #4407 sum running session totals. Lanes run outside a batch are in no window. `gh` (for
 merged issues only) is the one network call; when it fails, quality and acceptance print `n/a` with the reason.
 
+## Explorer A/B (`--explore-ab`, forge#4418)
+
+Compares lanes run with forge.yaml `lean.explore: on` against `off` (#4391), from the local run logs only (no GitHub
+crawl). Every RUN_START slice of each log counts, so re-running an issue with the other mode adds a second lane.
+
+- Rows per size bucket (`S`, `M`, `L` from the lane's turn record, else `unsized`) and mode: n, mean builder (plan +
+  build) turns, mean builder `cache_read_input_tokens`, merge rate (merged / finished) and conformance rate (lanes that
+  reached review with no conformance/review fix round and no shadow would-block, over reviewed; same as `--tiers`).
+- A paired table restricted to issues with a finished lane in both modes.
+- Finished lanes without an `explore` tag (before #4391) are counted and left out.
+
+`--json` prints the same data as `query.explore_ab`.
+
 ## Sections (per version and repo)
 
 - **Cost**: cache-read tokens per issue (mean, median), per phase (mean over all runs, so phases add up to the per-issue

package/docs/eval/corpus/lightlane-v1.json

@@ -0,0 +1,55 @@
+{
+  "version": "lightlane-v1",
+  "description": "Three-way light-lane benchmark (#4453, parent #4442): three more-itertools bugs reported after the model cutoff, graded by the upstream fix tests. Instructions: docs/eval/lightlane-bench.md.",
+  "upstream": "more-itertools/more-itertools",
+  "license": "MIT",
+  "clone_url": "https://github.com/more-itertools/more-itertools.git",
+  "base": "670b715b307adfe5d2fca08a87ca562cb8bfea2c",
+  "suite_command": "python3 -m unittest discover -q -s tests -t .",
+  "control_prompt": "Fix this issue in the repository in the current directory. Write a failing regression test first, then the fix, then run the targeted tests. Do not use web tools.\n\nIssue: {title}\n\n{body}",
+  "issues": [
+    {
+      "id": "lightlane-1293",
+      "upstream_issue": 1293,
+      "fix_pr": 1294,
+      "fix_sha": "1dd480a08bc2a4c399c84bf21cb1558f8b5fc838",
+      "title": "Repeated zero-cache seekable.peek() retains nested chain iterators",
+      "body": "On current master (`b5e3886`), the fix for #1268 preserves values but each `seekable(..., maxlen=0).peek()` prepends a new `chain` around the previous source. Repeated peeking without advancing retains an iterator chain proportional to the number of calls, despite having no history cache.\n\n```python\nimport tracemalloc\nfrom more_itertools import seekable\n\ns = seekable(range(3), maxlen=0)\ntracemalloc.start()\ns.peek()\nbaseline = tracemalloc.get_traced_memory()[0]\nfor _ in range(10000):\n    assert s.peek() == 0\nprint(tracemalloc.get_traced_memory()[0] - baseline)\nprint(list(s.elements()), list(s))\n```\n\nOn CPython 3.12.13/macOS this retains 2,427,672 additional bytes; output remains `[] [0, 1, 2]`. A single lookahead slot should be sufficient regardless of the number of repeated `peek()`/`bool()` calls. I am checking a fix that reuses the existing `peekable` helper only in the zero-history case, with unchanged bounded/unbounded history behavior.\n\nInvestigation performed with AI assistance.\n",
+      "oracle": {
+        "files": [
+          "tests/test_more.py"
+        ],
+        "command": "python3 -m unittest -q tests.test_more.SeekableTest.test_repeated_zero_cache_peek_has_bounded_memory",
+        "note": "670b715 is the merge parent of the fix, but the fix branch also carries the #1268/#1269 seekable maxlen=0 value fix; on the base the oracle errors with StopIteration, so an arm passes only by bounding memory and keeping the peeked values."
+      }
+    },
+    {
+      "id": "lightlane-1304",
+      "upstream_issue": 1304,
+      "fix_pr": 1305,
+      "fix_sha": "1ea82a711c69f590054987b5cb194157f8ce8ac4",
+      "title": "ichunked(..., 0) raises islice's ValueError on a nonempty iterable",
+      "body": "`ichunked` is documented as being like `chunked`, but with lazy chunks. `chunked` treats `n=0` as \"yield nothing\". `ichunked` only does that when the source is already empty; with any remaining items it calls `islice(iterator, n - 1)` and blows up.\n\nI hit this switching a streaming pipeline from `chunked` to `ichunked` so batches would not be materialized as lists. Chunk size comes from the same config in both paths; `0` means \"this stage is disabled, emit nothing.\"\n\n```python\nfrom more_itertools import chunked, ichunked\n\nrows = [\"a\", \"b\", \"c\"]  # e.g. leftover rows from a DB cursor\nchunk_size = 0          # config: skip this sink\n\nprint(list(chunked(rows, chunk_size)))\n# []\n\nprint([list(c) for c in ichunked(rows, chunk_size)])\n# ValueError: Stop argument for islice() must be None or an integer: 0 <= x <= sys.maxsize.\n```\n\n`IchunkedTests.test_zero` only uses `iterable = []`, which never enters the `for first in iterator` loop, so the `islice(..., n - 1)` path is untested.\n\nNegative `n` has the same leak: `chunked` raises `ValueError: n must be at least 0`, `ichunked` raises islice's \"Indices for islice() must be None or an integer...\".\n\nHappy to send a PR.",
+      "oracle": {
+        "files": [
+          "tests/test_more.py"
+        ],
+        "command": "python3 -m unittest -q tests.test_more.IchunkedTests.test_zero_nonempty tests.test_more.IchunkedTests.test_negative"
+      }
+    },
+    {
+      "id": "lightlane-1308",
+      "upstream_issue": 1308,
+      "fix_pr": 1309,
+      "fix_sha": "67fcf640477c78605580f5ee703a54c73c1e7a8a",
+      "title": "bucket ignores a supplied validator when the callable is falsey",
+      "body": "`bucket` replaces a supplied validator with its default when the callable has a false truth value. An empty callable set is one example:\n\n```python\nfrom more_itertools import bucket\n\nclass AllowedKeys(set):\n    def __call__(self, key):\n        return key in self\n\nallowed = AllowedKeys()\nsource = iter([1, 2, 3])\nbuckets = bucket(source, key=lambda x: x, validator=allowed)\nprint(list(buckets[1]))  # [1]; expected []\n```\n\nThe validator rejects every key, so selecting a bucket should return an empty iterator without consuming `source`. Instead, `validator or (lambda x: True)` discards it. Membership checks and iteration over bucket keys also accept keys the validator rejects. On an infinite input, a lookup for an absent, rejected key can keep scanning rather than return immediately.\n\nI reproduced this on current main (11.1.0). Changing the default selection to `(lambda x: True) if validator is None else validator` preserves the supplied callable. My regression checks rejected selection, membership, source consumption, and key iteration; the full unittest suite passes with that change.\n\nI can send the small fix and regression test if this behavior is unintended.\n",
+      "oracle": {
+        "files": [
+          "tests/test_more.py"
+        ],
+        "command": "python3 -m unittest -q tests.test_more.BucketTests.test_falsey_validator_is_honoured"
+      }
+    }
+  ]
+}

package/docs/eval/lightlane-bench.md

@@ -0,0 +1,91 @@
+# Light-lane three-way benchmark (#4453)
+
+This benchmark compares the light lane (#4450) with the full lean lane and with a plain single-session `claude -p` control. All three arms run on the same issue set, and each is graded by the **upstream oracle tests**: the test files of the real upstream fix, which fail on the base and pass on the fix. It reproduces the #4442 benchmark of 2026-10-22 and checks its acceptance targets.
+
+- Harness: `scripts/bench-lightlane.mjs`
+- Corpus: `docs/eval/corpus/lightlane-v1.json`. This is more-itertools @ `670b715` with upstream #1293, #1304 and #1308. Their fixes are PRs #1294, #1305 and #1309, and they were reported after the model's training cutoff.
+
+## Prerequisites
+
+- `python3`, `git`, `gh` (authenticated, with access to the sandbox repos), `claude` and `forgedock` on PATH. Use the ForgeDock release under test.
+- Two **private sandbox repos**, one for the light arm and one for the full arm. Each needs the upstream history up to the base, so the base SHA exists in it:
+  ```sh
+  git clone https://github.com/more-itertools/more-itertools.git mi && cd mi
+  git push git@github.com:<you>/mi-light.git 670b715b307adfe5d2fca08a87ca562cb8bfea2c:refs/heads/main
+  git push git@github.com:<you>/mi-full.git  670b715b307adfe5d2fca08a87ca562cb8bfea2c:refs/heads/main
+  ```
+  If a sandbox is a squashed snapshot instead, set `arms.<arm>.base` in the run config to its base commit.
+- A `forge.yaml` in each sandbox. The light arm sets `lean:` → `light:` → `mode: on`, and the full arm sets `mode: off`. Commit it on `main`; the harness diffs from the base, so this commit doesn't count as an arm change.
+- The corpus issues filed in each sandbox, with the title and body taken verbatim from the corpus. Labels follow the #4442 setup: `bug`.
+  ```sh
+  jq -c '.issues[]' docs/eval/corpus/lightlane-v1.json | while read -r i; do
+    gh issue create -R <you>/mi-light --title "$(jq -r .title <<<"$i")" --body "$(jq -r .body <<<"$i")" --label bug
+  done
+  ```
+
+## Run config (`runcfg.json`, not committed)
+
+```json
+{
+  "arms": {
+    "light":   { "repo": "<you>/mi-light", "issues": { "lightlane-1293": 1, "lightlane-1304": 2, "lightlane-1308": 3 } },
+    "full":    { "repo": "<you>/mi-full",  "issues": { "lightlane-1293": 1, "lightlane-1304": 2, "lightlane-1308": 3 } },
+    "control": { "model": "claude-opus-5-5" }
+  }
+}
+```
+
+Optional per arm:
+- `command`: argv prefix. The default is `["forgedock","run-issue"]`, or `["claude"]` for the control.
+- `args`: extra arguments, for example `["--model","claude-opus-5-5"]` for a lane, or the permission flags your `claude` needs for the control.
+- `clone_url` and `base` for the lane arms.
+
+Use the same model in every arm, and no web tools.
+
+## Steps
+
+1. **Calibrate** the oracles. Every issue must report `red_on_base` and `green_on_fix`, otherwise the command exits 2:
+   ```sh
+   node scripts/bench-lightlane.mjs calibrate --corpus docs/eval/corpus/lightlane-v1.json
+   ```
+2. **Run** the arms. This takes about 10 minutes per arm per issue, runs sequentially, and appends one JSON row per arm and issue:
+   ```sh
+   node scripts/bench-lightlane.mjs run --corpus docs/eval/corpus/lightlane-v1.json --config runcfg.json --out rows.jsonl
+   ```
+   Use `--arms light,control` or `--issues lightlane-1304` to rerun part of the set. Use `--keep` to keep the scratch clones.
+3. **Aggregate** the rows. This prints the metrics and targets, and exits 2 when a target fails:
+   ```sh
+   node scripts/bench-lightlane.mjs aggregate rows.jsonl
+   ```
+4. Post the aggregate JSON and the target table, plus `rows.jsonl`, on the measurement tracking issue. Note the ForgeDock version, the model and the date.
+
+## What each metric means
+
+| Field | Meaning |
+|---|---|
+| `cost_usd` | **True** cost. For the lanes, it is the run result's `cost_usd`: the sum of the #4407 per-session deltas, not the naive sum of running totals. For the control, it is `total_cost_usd`. |
+| `wall_ms` | Lane wall time inside the engine (`wall_ms`, #4451). For the control, it is `duration_ms`. `harness_wall_ms` adds CLI start-up. |
+| `model_ms` | Summed session time (`model_ms`, #4451). For the control, it is `duration_ms`, since the control is one session. |
+| `oracle_pass` | The oracle command passes after the fix's test files are overlaid on the arm's head (`git checkout <fix_sha> -- tests/test_more.py`). |
+| `suite_pass` | The repo's own suite (`suite_command`) passes on the arm's head, before the overlay. |
+| `claimed_fix` | The arm reported success and its diff is non-empty. For the lanes, success is a `merged` or `ci-pending` terminal reason; for the control, it is exit 0 with `is_error: false`. |
+| false fix | `claimed_fix` and not `oracle_pass`. |
+| `api_shape` | `bin/lean/apishape.mjs` from the base to the arm's head (signature, sync↔async, generator↔function, removed or renamed public symbol). |
+| undetected API shape | A light-arm row that finished light (`light: true`, no upgrade) although its diff changed the public API shape. |
+
+## Targets (#4442)
+
+| Target | Check |
+|---|---|
+| Light true cost ≤ 1.2× control | The cost sum of eligible light rows, divided by the control's cost on the same issues. |
+| Light wall ≤ 1.3× model time | The `wall_ms` sum of eligible light rows, divided by their `model_ms` sum. |
+| Light oracle pass rate ≥ control | Over all light rows. |
+| Light false-fix rate ≤ control | Over all light rows. |
+| 0 undetected public-API-shape changes | Over all light rows. |
+
+"Eligible" means the row's `light_eligibility` (the run result's light-lane verdict) is `eligible`. A target is `null` when an arm it needs has no rows.
+
+## Corpus notes
+
+- Oracle commands are scoped to the test ids each fix PR added or changed. Class-wide commands would include tests that fail on the base for reasons unrelated to the issue.
+- `lightlane-1293`: `670b715` is the merge parent of the fix, but the fix branch also carries the #1268/#1269 seekable `maxlen=0` value fix. On the base the oracle errors with `StopIteration`, so an arm passes only if it bounds memory and also keeps the peeked values.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "forgedock",
-  "version": "1.31.0",
+  "version": "1.32.1",
   "description": "Autonomous development pipeline for coding agents. GitHub as institutional memory, with native Claude Code and OpenCode entrypoints.",
   "bin": {
     "forgedock": "bin/forgedock.mjs"

package/scripts/bench-lightlane.mjs

@@ -0,0 +1,286 @@
+#!/usr/bin/env node
+// SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios
+// SPDX-License-Identifier: AGPL-3.0-or-later
+/**
+ * scripts/bench-lightlane.mjs — three-way benchmark: light lane vs full lane vs a plain single-session control (#4453)
+ *
+ * Runs the same issue set through three arms and grades every arm with the upstream oracle tests (the test files of
+ * the real upstream fix, overlaid on the arm's head). Zero dependencies; every process call goes through an
+ * injectable `run`, so tests use fakes. Instructions: docs/eval/lightlane-bench.md. Corpus: docs/eval/corpus/.
+ *
+ * Usage:
+ *   node scripts/bench-lightlane.mjs calibrate --corpus <corpus.json>
+ *   node scripts/bench-lightlane.mjs run --corpus <corpus.json> --config <runcfg.json> --out <rows.jsonl> [--arms light,full,control] [--issues id,...] [--keep]
+ *   node scripts/bench-lightlane.mjs aggregate <rows.jsonl|->
+ *
+ * Arms:
+ *   light, full: `forgedock run-issue <n> --repo <sandbox> --result-file <f>` on a sandbox repo whose forge.yaml sets
+ *     lean.light.mode on (light) or off (full). Cost is the run result's cost_usd (the sum of the #4407 per-session
+ *     deltas); model time is model_ms (summed session time, #4451).
+ *   control: `claude -p <prompt> --output-format json --model <model>` in a fresh clone at the base; cost is
+ *     total_cost_usd, wall and model time are duration_ms (one session).
+ *
+ * The harness never writes to GitHub itself; the only GitHub writes are those of the light/full arms' own run-issue.
+ *
+ * Exit codes: 0 ok | 1 invalid input or a failed run step | 2 a calibration or a target failed (output still emitted)
+ */
+
+import { spawn } from "node:child_process";
+import { appendFileSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { pathToFileURL } from "node:url";
+import { detectApiShape } from "../bin/lean/apishape.mjs";
+
+export const ARMS = ["light", "full", "control"];
+export const TARGETS = Object.freeze({ cost_vs_control: 1.2, wall_vs_model: 1.3 });
+const SHA_RE = /^[0-9a-f]{40}$/;
+const CLAIMED_REASONS = new Set(["merged", "ci-pending"]);
+const OUT_CAP = 1 << 20;
+
+/** Spawn `cmd args` and collect its output (the last 1 MiB of each stream); never throws. */
+export function benchDefaultRun(cmd, args, { cwd, env } = {}) {
+  return new Promise((resolve) => {
+    let stdout = "", stderr = "";
+    const cap = (s) => (s.length > OUT_CAP ? s.slice(-OUT_CAP) : s);
+    let p;
+    try { p = spawn(cmd, args, { cwd, env: env ?? process.env, stdio: ["ignore", "pipe", "pipe"] }); } catch (e) { resolve({ code: 127, stdout: "", stderr: String(e?.message ?? e) }); return; }
+    p.stdout.on("data", (d) => { stdout = cap(stdout + d); });
+    p.stderr.on("data", (d) => { stderr = cap(stderr + d); });
+    p.on("error", (e) => resolve({ code: e?.code === "ENOENT" ? 127 : 1, stdout, stderr: stderr + String(e?.message ?? e) }));
+    p.on("close", (code) => resolve({ code: code ?? 1, stdout, stderr }));
+  });
+}
+
+/** A parsed corpus, or a thrown Error naming the first bad field. */
+export function validateCorpus(c) {
+  if (!c || typeof c !== "object") throw new Error("corpus: not an object");
+  for (const k of ["clone_url", "suite_command", "control_prompt"]) if (typeof c[k] !== "string" || !c[k].trim()) throw new Error(`corpus: missing ${k}`);
+  if (!SHA_RE.test(String(c.base))) throw new Error("corpus: base must be a full 40-hex sha");
+  if (!Array.isArray(c.issues) || !c.issues.length) throw new Error("corpus: issues must be a non-empty array");
+  const ids = new Set();
+  for (const i of c.issues) {
+    if (!i?.id || ids.has(i.id)) throw new Error(`corpus: missing or duplicate issue id ${i?.id}`);
+    ids.add(i.id);
+    if (!SHA_RE.test(String(i.fix_sha))) throw new Error(`corpus: ${i.id}: fix_sha must be a full 40-hex sha`);
+    if (typeof i.title !== "string" || typeof i.body !== "string") throw new Error(`corpus: ${i.id}: title and body are required`);
+    if (!Array.isArray(i.oracle?.files) || !i.oracle.files.length || typeof i.oracle.command !== "string") throw new Error(`corpus: ${i.id}: oracle {files, command} is required`);
+  }
+  return c;
+}
+
+/** The run config for `arms`, or a thrown Error: light/full need {repo, issues: {<corpus id>: n}}, control needs {model}. */
+export function validateRunConfig(cfg, corpus, arms) {
+  const a = cfg?.arms;
+  if (!a || typeof a !== "object") throw new Error("config: missing arms");
+  for (const arm of arms) {
+    const x = a[arm];
+    if (!x) throw new Error(`config: missing arms.${arm}`);
+    if (arm === "control") { if (typeof x.model !== "string" || !x.model) throw new Error("config: arms.control.model is required"); continue; }
+    if (typeof x.repo !== "string" || !/^[\w.-]+\/[\w.-]+$/.test(x.repo)) throw new Error(`config: arms.${arm}.repo must be owner/name`);
+    for (const i of corpus.issues) if (!Number.isSafeInteger(x.issues?.[i.id]) || x.issues[i.id] <= 0) throw new Error(`config: arms.${arm}.issues.${i.id} must be the sandbox issue number`);
+  }
+  return cfg;
+}
+
+/** The control prompt for an issue: `{title}` and `{body}` replaced. */
+export const fillPrompt = (tpl, issue) => String(tpl).replaceAll("{title}", issue.title).replaceAll("{body}", issue.body);
+
+const sh = (run, command, cwd) => run("sh", ["-c", command], { cwd });
+async function git(run, args, cwd) {
+  const r = await run("git", args, { cwd });
+  if (r.code !== 0) throw new Error(`git ${args.join(" ")} failed (${r.code}): ${String(r.stderr).trim().slice(0, 300)}`);
+  return String(r.stdout).trim();
+}
+
+/** Overlay the oracle test files from the fix commit onto the checkout in `dir` and run the oracle; true when it passes. */
+async function runOracle(run, dir, issue) {
+  await git(run, ["checkout", "-q", issue.fix_sha, "--", ...issue.oracle.files], dir);
+  return (await sh(run, issue.oracle.command, dir)).code === 0;
+}
+
+/** For each corpus issue: the oracle must fail on the base and pass on the fix. -> [{id, red_on_base, green_on_fix}] */
+export async function calibrate({ corpus, run = benchDefaultRun, tmp }) {
+  const dir = join(tmp, "upstream");
+  await git(run, ["clone", "-q", corpus.clone_url, dir], tmp);
+  const out = [];
+  for (const issue of corpus.issues) {
+    await git(run, ["checkout", "-q", "-f", corpus.base], dir);
+    const red = !(await runOracle(run, dir, issue));
+    await git(run, ["checkout", "-q", "-f", issue.fix_sha], dir);
+    const green = (await sh(run, issue.oracle.command, dir)).code === 0;
+    out.push({ id: issue.id, red_on_base: red, green_on_fix: green });
+  }
+  return out;
+}
+
+const num = (x) => (Number.isFinite(Number(x)) && x !== null && x !== "" ? Number(x) : null);
+const readJson = (p) => { try { return JSON.parse(readFileSync(p, "utf8")); } catch { return null; } };
+
+/** Grade `head` from the repo in `src`: own suite, then the oracle overlay, then the API shape from `base`. */
+async function grade({ run, src, head, base, corpus, issue, tmp, tag, apiShape }) {
+  await git(run, ["branch", "-f", "bench-head", head], src);
+  const dir = join(tmp, `${tag}-grade`);
+  await git(run, ["clone", "-q", "-b", "bench-head", src, dir], tmp);
+  await git(run, ["fetch", "-q", corpus.clone_url, issue.fix_sha], dir);
+  const changed = (await run("git", ["diff", "--quiet", base, "HEAD"], { cwd: dir })).code === 1;
+  let shape;
+  try { const r = await apiShape({ cwd: dir, base, head: "HEAD" }); shape = { status: r?.status ?? "unknown", changes: Array.isArray(r?.changes) ? r.changes : [] }; }
+  catch (e) { shape = { status: "error", changes: [], error: String(e?.message ?? e).slice(0, 200) }; }
+  const suite = (await sh(run, corpus.suite_command, dir)).code === 0;
+  const oracle = await runOracle(run, dir, issue);
+  return { changed, suite_pass: suite, oracle_pass: oracle, api_shape: shape };
+}
+
+/** Run one arm on one corpus issue and grade it -> one row (see aggregate). `now` and `apiShape` are injectable. */
+export async function runArm({ arm, issue, corpus, cfg, run = benchDefaultRun, tmp, now = () => Date.now(), apiShape = detectApiShape }) {
+  const tag = `${arm}-${issue.id}`, a = cfg.arms[arm];
+  const row = { arm, id: issue.id, cost_usd: null, wall_ms: null, model_ms: null, harness_wall_ms: null, claimed_fix: false, oracle_pass: false, suite_pass: null,
+    api_shape: null, light: null, light_eligibility: null, light_upgrade: null, terminal: null };
+  const t0 = now();
+  let src = null, head = null, base = corpus.base, claimed = false;
+  if (arm === "control") {
+    src = join(tmp, `${tag}-work`);
+    await git(run, ["clone", "-q", corpus.clone_url, src], tmp);
+    await git(run, ["checkout", "-q", "-b", "bench-control", corpus.base], src);
+    const [bin, ...pre] = Array.isArray(a.command) && a.command.length ? a.command : ["claude"];
+    const r = await run(bin, [...pre, "-p", fillPrompt(corpus.control_prompt, issue), "--output-format", "json", "--model", a.model, ...(a.args ?? [])], { cwd: src });
+    row.harness_wall_ms = now() - t0;
+    let j = null;
+    try { j = JSON.parse(String(r.stdout).trim().split("\n").pop()); } catch { /* reported as terminal below */ }
+    row.cost_usd = num(j?.total_cost_usd); row.wall_ms = num(j?.duration_ms); row.model_ms = num(j?.duration_ms);
+    row.terminal = r.code !== 0 ? `exit ${r.code}` : !j ? "no-json" : j.is_error ? "error" : "ok";
+    claimed = row.terminal === "ok";
+    await git(run, ["add", "-A"], src);
+    await git(run, ["-c", "user.name=bench", "-c", "user.email=bench@localhost", "commit", "-q", "--allow-empty", "--no-verify", "-m", `control ${issue.id}`], src);
+    head = await git(run, ["rev-parse", "HEAD"], src);
+  } else {
+    const n = a.issues[issue.id], rf = join(tmp, `${tag}-result.json`);
+    const [bin, ...pre] = Array.isArray(a.command) && a.command.length ? a.command : ["forgedock", "run-issue"];
+    const r = await run(bin, [...pre, String(n), "--repo", a.repo, "--result-file", rf, ...(a.args ?? [])], {});
+    row.harness_wall_ms = now() - t0;
+    const res = readJson(rf);
+    row.terminal = res?.terminalReason ?? `exit ${r.code}`;
+    row.cost_usd = num(res?.cost_usd); row.wall_ms = num(res?.wall_ms); row.model_ms = num(res?.model_ms);
+    row.light = typeof res?.light === "boolean" ? res.light : null;
+    row.light_eligibility = res?.light_eligibility ?? null;
+    row.light_upgrade = Array.isArray(res?.light_upgrade) ? res.light_upgrade : null;
+    claimed = CLAIMED_REASONS.has(row.terminal);
+    base = a.base ?? corpus.base;
+    src = join(tmp, `${tag}-src`);
+    await git(run, ["clone", "-q", a.clone_url ?? `https://github.com/${a.repo}.git`, src], tmp);
+    if (res?.pr) {
+      const pr = String(res.pr).match(/(\d+)\s*$/)?.[1];
+      const v = await run("gh", ["pr", "view", pr, "-R", a.repo, "--json", "headRefOid", "--jq", ".headRefOid"], {});
+      await git(run, ["fetch", "-q", "origin", `refs/pull/${pr}/head`], src);
+      head = SHA_RE.test(String(v.stdout).trim()) ? String(v.stdout).trim() : await git(run, ["rev-parse", "FETCH_HEAD"], src);
+    } else if (res?.branch) {
+      await git(run, ["fetch", "-q", "origin", res.branch], src);
+      head = await git(run, ["rev-parse", "FETCH_HEAD"], src);
+    }
+  }
+  if (!head) return row; // no PR and no branch: nothing to grade; oracle_pass stays false
+  const g = await grade({ run, src, head, base, corpus, issue, tmp, tag, apiShape });
+  return { ...row, claimed_fix: claimed && g.changed, oracle_pass: g.oracle_pass, suite_pass: g.suite_pass, api_shape: g.api_shape };
+}
+
+const sum = (xs) => xs.reduce((s, x) => s + x, 0);
+const nums = (rows, k) => rows.map((r) => r[k]).filter((x) => Number.isFinite(x));
+const rate = (rows, f) => (rows.length ? rows.filter(f).length / rows.length : null);
+const r4 = (x) => (x == null ? null : Math.round(x * 10000) / 10000);
+
+/** Per-arm metrics over rows (pure). */
+export function armMetrics(rows) {
+  const cost = nums(rows, "cost_usd"), wall = nums(rows, "wall_ms"), model = nums(rows, "model_ms");
+  return {
+    n: rows.length,
+    cost_usd: { sum: r4(sum(cost)), mean: cost.length ? r4(sum(cost) / cost.length) : null },
+    wall_ms: sum(wall), model_ms: sum(model), wall_over_model: sum(model) > 0 ? r4(sum(wall) / sum(model)) : null,
+    oracle_pass_rate: r4(rate(rows, (r) => r.oracle_pass === true)),
+    false_fix_rate: r4(rate(rows, (r) => r.claimed_fix === true && r.oracle_pass !== true)),
+    suite_pass_rate: r4(rate(rows, (r) => r.suite_pass === true)),
+    api_shape_changed: rows.filter((r) => r.api_shape?.status === "changed").length,
+    undetected_api_shape: rows.filter((r) => r.arm === "light" && r.light === true && r.api_shape?.status === "changed").length,
+  };
+}
+
+/**
+ * Rows -> {arms, light_eligible, targets} (pure). Cost and wall are judged on the light rows whose eligibility verdict is
+ * `eligible` (cost against the control rows of the same issues); oracle, false fix and undetected API shape on all light
+ * rows. A target is {pass: true|false|null, value, limit}; null when an arm it needs has no rows.
+ */
+export function aggregate(rows) {
+  const by = Object.fromEntries(ARMS.map((a) => [a, rows.filter((r) => r.arm === a)]));
+  const arms = Object.fromEntries(ARMS.map((a) => [a, armMetrics(by[a])]));
+  const elig = by.light.filter((r) => r.light_eligibility === "eligible");
+  const eligIds = new Set(elig.map((r) => r.id));
+  const ctlElig = by.control.filter((r) => eligIds.has(r.id));
+  const le = armMetrics(elig), ce = armMetrics(ctlElig), L = arms.light, C = arms.control;
+  const t = (ok, value, limit) => ({ pass: ok == null ? null : ok, value, limit });
+  const costRatio = elig.length && ctlElig.length && ce.cost_usd.sum > 0 ? r4(le.cost_usd.sum / ce.cost_usd.sum) : null;
+  const targets = {
+    cost_le_1_2x_control: t(costRatio == null ? null : costRatio <= TARGETS.cost_vs_control, costRatio, TARGETS.cost_vs_control),
+    wall_le_1_3x_model: t(le.wall_over_model == null ? null : le.wall_over_model <= TARGETS.wall_vs_model, le.wall_over_model, TARGETS.wall_vs_model),
+    oracle_ge_control: t(L.n && C.n ? L.oracle_pass_rate >= C.oracle_pass_rate : null, L.oracle_pass_rate, C.oracle_pass_rate),
+    false_fix_le_control: t(L.n && C.n ? L.false_fix_rate <= C.false_fix_rate : null, L.false_fix_rate, C.false_fix_rate),
+    zero_undetected_api_shape: t(L.n ? L.undetected_api_shape === 0 : null, L.undetected_api_shape, 0),
+  };
+  return { arms, light_eligible: { ...le, control_cost_usd: ce.cost_usd.sum, ids: [...eligIds].sort() }, targets };
+}
+
+/** JSONL (or a JSON array) -> rows. */
+export function parseRows(text) {
+  const s = String(text).trim();
+  if (!s) return [];
+  if (s.startsWith("[")) return JSON.parse(s);
+  return s.split("\n").filter((l) => l.trim()).map((l) => JSON.parse(l));
+}
+
+const flag = (argv, name) => { const i = argv.indexOf(name); return i >= 0 && argv[i + 1] && !argv[i + 1].startsWith("--") ? argv[i + 1] : null; };
+const list = (v, all) => (v ? v.split(",").map((s) => s.trim()).filter(Boolean) : all);
+
+export async function main(argv = process.argv.slice(2), { run = benchDefaultRun, out = process.stdout, err = process.stderr } = {}) {
+  const cmd = argv[0];
+  try {
+    if (cmd === "aggregate") {
+      const src = argv[1];
+      const text = !src || src === "-" ? readFileSync(0, "utf8") : readFileSync(src, "utf8");
+      const agg = aggregate(parseRows(text));
+      out.write(JSON.stringify(agg, null, 2) + "\n");
+      return Object.values(agg.targets).some((x) => x.pass === false) ? 2 : 0;
+    }
+    if (cmd !== "calibrate" && cmd !== "run") { err.write("usage: bench-lightlane.mjs calibrate|run|aggregate (see the header)\n"); return 1; }
+    const cp = flag(argv, "--corpus");
+    if (!cp) throw new Error("--corpus is required");
+    const corpus = validateCorpus(JSON.parse(readFileSync(cp, "utf8")));
+    const tmp = mkdtempSync(join(tmpdir(), "bench-lightlane-"));
+    try {
+      if (cmd === "calibrate") {
+        const res = await calibrate({ corpus, run, tmp });
+        out.write(JSON.stringify(res, null, 2) + "\n");
+        return res.every((r) => r.red_on_base && r.green_on_fix) ? 0 : 2;
+      }
+      const arms = list(flag(argv, "--arms"), ARMS), rowsOut = flag(argv, "--out"), cfgPath = flag(argv, "--config");
+      if (!rowsOut || !cfgPath) throw new Error("--config and --out are required");
+      const bad = arms.filter((a) => !ARMS.includes(a));
+      if (bad.length) throw new Error(`unknown arm(s): ${bad.join(", ")}`);
+      const cfg = validateRunConfig(JSON.parse(readFileSync(cfgPath, "utf8")), corpus, arms);
+      const ids = list(flag(argv, "--issues"), corpus.issues.map((i) => i.id));
+      for (const issue of corpus.issues.filter((i) => ids.includes(i.id))) {
+        for (const arm of arms) {
+          const row = await runArm({ arm, issue, corpus, cfg, run, tmp });
+          appendFileSync(rowsOut, JSON.stringify(row) + "\n");
+          err.write(`${arm} ${issue.id}: oracle ${row.oracle_pass ? "pass" : "fail"}, $${row.cost_usd ?? "?"}, ${row.terminal}\n`);
+        }
+      }
+      return 0;
+    } finally {
+      if (!argv.includes("--keep")) rmSync(tmp, { recursive: true, force: true });
+    }
+  } catch (e) {
+    err.write(`ERROR: ${e?.message ?? e}\n`);
+    return 1;
+  }
+}
+
+if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) process.exitCode = await main();

package/scripts/refresh-bot-token.test.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios
+# SPDX-License-Identifier: AGPL-3.0-or-later
+#
+# Fixtures for scripts/refresh-bot-token.sh (#4507): hosts.yml is replaced by rename, never rewritten in place.
+# Run: bash scripts/refresh-bot-token.test.sh (bash 3.2 compatible). Stubs `node` and `gh`; no network.
+
+set -u
+DIR=$(cd "$(dirname "$0")" && pwd)
+S="$DIR/refresh-bot-token.sh"
+T=$(mktemp -d "${TMPDIR:-/tmp}/refresh-bot-token-test.XXXXXX")
+trap 'rm -rf "$T"' EXIT
+pass=0; fail=0
+ok() { pass=$((pass+1)); }
+no() { fail=$((fail+1)); echo "FAIL: $1"; }
+
+mkdir -p "$T/bin" "$T/cfg"
+echo key > "$T/key.pem"
+cat > "$T/bin/node" <<'STUB'
+#!/usr/bin/env bash
+printf 'tok-new'
+STUB
+cat > "$T/bin/gh" <<'STUB'
+#!/usr/bin/env bash
+echo "$GH_CONFIG_DIR $*" >> "$GH_LOG"
+case "$1 $2" in
+  "auth login") [ -n "${GH_FAIL:-}" ] && exit 1; cat > /dev/null; printf 'github.com:\n    oauth_token: tok-new\n' > "$GH_CONFIG_DIR/hosts.yml" ;;
+  "auth status") echo "Logged in to github.com account rapiercraft-forgedock[bot]" ;;
+esac
+exit 0
+STUB
+chmod +x "$T/bin/node" "$T/bin/gh"
+inode() { ls -i "$1" | awk '{print $1}'; }
+
+# (a) success: gh writes in a staging dir; hosts.yml is renamed into place
+printf 'github.com:\n    oauth_token: tok-old\n' > "$T/cfg/hosts.yml"
+before=$(inode "$T/cfg/hosts.yml")
+OUT=$(PATH="$T/bin:$PATH" GH_CONFIG_DIR="$T/cfg" GH_LOG="$T/log" bash "$S" --pem "$T/key.pem" 2>&1); RC=$?
+[ "$RC" = 0 ] && ok || no "(a) exit $RC: $OUT"
+grep -q 'tok-new' "$T/cfg/hosts.yml" && ok || no "(a) hosts.yml not updated"
+[ "$(inode "$T/cfg/hosts.yml")" != "$before" ] && ok || no "(a) hosts.yml rewritten in place (same inode)"
+if awk '{print $1}' "$T/log" | grep -qx "$T/cfg"; then no "(a) gh ran against the live config dir"; else ok; fi
+grep -q "auth login" "$T/log" && ok || no "(a) gh auth login not run"
+ls -a "$T/cfg" | grep -q '^\.hosts-stage\.' && no "(a) staging dir left behind" || ok
+case "$OUT" in *"token refreshed"*) ok ;; *) no "(a) success message missing: $OUT" ;; esac
+
+# (b) failed login: the live hosts.yml stays byte-identical, exit != 0
+printf 'github.com:\n    oauth_token: tok-keep\n' > "$T/cfg/hosts.yml"
+cp "$T/cfg/hosts.yml" "$T/expect"
+PATH="$T/bin:$PATH" GH_CONFIG_DIR="$T/cfg" GH_LOG="$T/log" GH_FAIL=1 bash "$S" --pem "$T/key.pem" >/dev/null 2>&1; RC=$?
+[ "$RC" != 0 ] && ok || no "(b) failed login exited 0"
+cmp -s "$T/expect" "$T/cfg/hosts.yml" && ok || no "(b) hosts.yml changed after a failed login"
+ls -a "$T/cfg" | grep -q '^\.hosts-stage\.' && no "(b) staging dir left behind" || ok
+
+echo "refresh-bot-token: $pass passed, $fail failed"
+[ "$fail" = 0 ]