@hominis/fireforge 0.47.0 → 0.47.1

package/docs/lifecycle-invariants.md

@@ -284,7 +284,10 @@ Wait ergonomics are opt-in: `pollMaxMs` (the poll interval doubles from
 `pollMs` up to the cap) and `waitProgressMs` with `onWaitProgress` (a periodic
 progress callback carrying the holder's PID, liveness, and owner-metadata
 lines from the pid file). All three are inert when unset, so existing callers
-keep byte-identical behavior.
+keep byte-identical behavior. Within one poll iteration a due progress probe
+runs before the deadline check, so a wait that times out has always reported
+the holder at least once when its budget exceeds `waitProgressMs`, however
+slow the host (`file-lock.test.ts`, "budget shorter than one iteration").
 
 ### `src/core/file-lock-owner.ts`
 
@@ -430,10 +433,23 @@ On SIGINT or SIGTERM, `bin/fireforge.ts`:
    - `waitForActiveCriticalSections(5 s)`, which holds the exit for in-flight
      "apply plus persist" pairs (invariant 4). It is bounded so that a stuck
      write cannot postpone the exit the user asked for.
+   - `waitForActiveChildShutdown(12 s)`, which holds the exit until every
+     tracked child has closed. For a process-group dispatch (`mach` under
+     `test`, `build`, `run`) "closed" includes the exec layer's post-close
+     work: the group sweep and the caller's `postCloseSweep` (the harness
+     helper reap in `fireforge test`). Both run before the child's closure
+     settles, so a forwarded SIGTERM cannot race `process.exit` past them.
 3. Force-releases the furnace lock directories of active operations, since
    `withFileLock`'s `finally` will never run past `process.exit`.
 4. Calls `process.exit(130 | 143)`.
 
+A parent that dies without signalling FireForge (SIGKILL on the `npm` above
+it) enters no handler here. `fireforge test` covers that case itself with
+the parent-exit watchdog (`src/core/parent-exit-watchdog.ts`): it writes the
+killed verdict, reaps the mach group and the tracked helpers, and lets the
+dispatch loop end the run; the process then exits through the ordinary
+error path, not through this pipeline.
+
 ## Choosing a primitive for a new mutation
 
 | Your operation                                                                                                       | Use                                                                                            |

package/docs/machine-output.md

@@ -86,15 +86,15 @@ moved.
 `FIREFORGE-VERDICT: FAIL reason=<reason>` uses a closed set
 (`FireforgeVerdictReason`, `src/commands/test-verdict.ts`):
 
-| Reason          | Meaning                                                                                                                                                                                                      |
-| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `crash`         | The harness or the browser died, so there is no trustworthy suite result.                                                                                                                                    |
-| `no-tests`      | The run dispatched but nothing ran.                                                                                                                                                                          |
-| `test-failures` | The suite ran and reported unexpected results.                                                                                                                                                               |
-| `preflight`     | The run was refused before the harness was reached.                                                                                                                                                          |
-| `inconclusive`  | A result exists, but `engine/` moved under it, so the result was discarded.                                                                                                                                  |
-| `lock-timeout`  | The run never started: the engine session lock stayed contended.                                                                                                                                             |
-| `killed`        | A signal terminated the run. Written from the signal handler so that a log tail always describes itself. A killed run must never be silent, or "killed", "still running" and "never started" all look alike. |
+| Reason          | Meaning                                                                                                                                                                                                                                                                                                                                                           |
+| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `crash`         | The harness or the browser died, so there is no trustworthy suite result.                                                                                                                                                                                                                                                                                         |
+| `no-tests`      | The run dispatched but nothing ran.                                                                                                                                                                                                                                                                                                                               |
+| `test-failures` | The suite ran and reported unexpected results.                                                                                                                                                                                                                                                                                                                    |
+| `preflight`     | The run was refused before the harness was reached.                                                                                                                                                                                                                                                                                                               |
+| `inconclusive`  | A result exists, but `engine/` moved under it, so the result was discarded.                                                                                                                                                                                                                                                                                       |
+| `lock-timeout`  | The run never started: the engine session lock stayed contended.                                                                                                                                                                                                                                                                                                  |
+| `killed`        | A signal terminated the run, or FireForge ended it because its parent process vanished. Written from the signal handler (or the parent-exit watchdog) so that a log tail always describes itself. A killed run must never be silent, or "killed", "still running" and "never started" all look alike. `signal=` names the cause: a signal name, or `parent-exit`. |
 
 ### The additive `note=` key
 
@@ -124,6 +124,19 @@ tasks _inside_ a file is harness code that reads the exported seed; FireForge
 owns the seed, the plumbing and the record. Absent when the run did not
 shuffle.
 
+### The additive `orphans-reaped=` key
+
+How many harness processes FireForge terminated itself during the run: at
+preflight, survivors of an earlier run in this objdir under `--reap-orphans`
+or `test.reapOrphans: "reap"`; at teardown, helpers this run's harness
+launched (httpd, websocket server, ssltunnel, process bridge, the browser,
+`moz-http2`) that were still alive after mach exited. Summed across the census, every
+retry attempt and every shard. Absent when nothing was reaped, so a green on
+a quiet machine prints the line it always did, and a green after a reap is
+distinguishable from it. A run ended by SIGTERM or by the parent-exit
+watchdog writes its verdict before the teardown reap runs, so that line never
+carries the key; the reap is reported on stderr instead.
+
 A preflight refusal's own text is written to stdout before the verdict line,
 and into the run log, so the verdict stays the last stdout write while the
 reason survives both a redirect and a `tail`. See

package/docs/testing.md

@@ -87,21 +87,85 @@ a slow build for a silently wrong test.
   earlier run in this project's objdir. It runs at preflight, before this run
   spawns anything, so every hit is necessarily a survivor. A match has to be
   objdir-anchored, since `xpcshell` and `server.js` are far too generic to
-  report on their own. Survivors slow every later run without appearing in
+  report on their own, and structural: the process's executable is a helper
+  binary, or an interpreter whose script argument is a helper script. A
+  process that merely mentions such a path (an editor, a `grep`, the shell
+  FireForge runs under) is not a helper, and FireForge's own ancestors are
+  never candidates. Survivors slow every later run without appearing in
   its output. The usual symptom is a three-second suite taking minutes of
-  wall clock. The census is report-only by default, and `--reap-orphans`
-  terminates what it finds. It never refuses a run. A different shape,
+  wall clock. The census is report-only by default; `--reap-orphans`, or
+  `test.reapOrphans: "reap"` in `fireforge.json`, terminates what it finds
+  (SIGTERM, a 500 ms grace, SIGKILL). It never refuses a run. Every reap is
+  stamped on the verdict line as `orphans-reaped=<n>`. A different shape,
   reparented Python `multiprocessing` workers, is covered by the
   `Orphaned harness workers` doctor check.
+- Every dispatch owns its own kill path. The harness announces what it
+  launches (`runtests.py | Server pid: <n>`, the websocket server, the SSL
+  tunnel, the websocket/process bridge, and the browser as `Application
+pid`); FireForge tracks those pids and, after mach has exited for any
+  reason (a clean finish, a harness crash before a retry, a no-output
+  timeout, a forwarded SIGTERM, the parent-exit watchdog below), terminates
+  any of them still alive. A tracked pid is re-read from `ps` first and must
+  still be anchored to this objdir and still look like the thing that was
+  launched, so a recycled pid is never signalled. One helper is never
+  announced: mozserve starts `moz-http2` in a session of its own and logs no
+  pid, so it is in neither the group nor the announcements. For it, FireForge
+  snapshots the same-objdir helpers before the dispatch spawns and, after
+  mach exits, reaps any helper-shaped process under this checkout's absolute
+  objdir that is new since that snapshot and reparented to launchd. An
+  earlier run's survivor is in the snapshot and stays the census's call. This
+  reap runs inside the exec layer's close path, so the signal handler's
+  bounded child-shutdown wait covers it. The count joins `orphans-reaped=`.
+- FireForge watches its own parent while a dispatch runs. A supervisor that
+  SIGKILLs the process above `fireforge test` (an `npm run` step killed at
+  its bound) signals nothing below it; FireForge notices the reparenting
+  within about two seconds, writes `FIREFORGE-VERDICT: FAIL reason=killed
+signal=parent-exit`, reaps the harness process group and the tracked
+  helpers, and ends the run without classifying or retrying.
 - `--perf-samples <path>` publishes a perf-sample artifact path to the
   harness (exported as `<BINARYNAME>_PERF_SAMPLE_JSON`).
 
+## Process groups and supervisors
+
+`fireforge test` spawns mach as the leader of its **own** process group
+(`detached: true`), so that FireForge can signal the whole harness tree with
+one negative-pid kill and sweep the group after mach exits. Two consequences
+for anything that supervises `fireforge test`:
+
+- A supervisor that kills _its own_ process group (`kill -- -<npm pid>`) does
+  not reach mach: mach is not in that group. Killing the `npm`/`node` chain
+  above FireForge leaves the harness tree running.
+- SIGTERM to FireForge is the right signal. FireForge forwards it to the
+  mach group, escalates to SIGKILL after a grace period, sweeps the group,
+  reaps the tracked helpers, and writes `FAIL reason=killed signal=SIGTERM`.
+  (That line, like the watchdog's `signal=parent-exit` line, is written
+  before the teardown reap runs, so it cannot carry `orphans-reaped=`; the
+  reap reports to stderr.)
+- A supervisor that has to SIGKILL FireForge should pass
+  `--pgid-file <path>`. FireForge rewrites the file with the harness group id
+  on every mach spawn (each retry attempt and shard) and removes it when the
+  run ends under its own control. After a SIGKILL the file is still there:
+  `kill -TERM -- -$(cat <path>)`, then `-KILL`, takes mach and every helper
+  that shares its group. Two processes are outside it: the browser
+  (mozprocess puts it in a group of its own) and `moz-http2` (mozserve
+  starts it in its own session). A SIGTERM to the group gives `runtests.py`
+  the chance to run its own cleanup, which normally takes the browser with
+  it; nothing takes `moz-http2`, so after a SIGKILL of FireForge it stays
+  until the next run's census, which is why that run should carry the reap
+  posture. A browser that does survive is the Marionette preflight's and
+  `--kill-stale-marionette`'s to remove.
+
+The mochitest helpers (httpd, websocket server, ssltunnel, process bridge)
+are plain children of `runtests.py` and share mach's group. The browser and
+`moz-http2` do not, which is why the teardown reap above exists alongside the
+group sweep.
+
 ## The verdict line
 
 Every test run ends with one machine-readable line:
 
 ```
-FIREFORGE-VERDICT: PASS|FAIL reason=… [note=<class>] [shuffle=<seed>] [log=<path>]
+FIREFORGE-VERDICT: PASS|FAIL reason=… [note=<class>] [shuffle=<seed>] [orphans-reaped=<n>] [log=<path>]
 ```
 
 Automation should branch on this line rather than on the raw process code,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hominis/fireforge",
-  "version": "0.47.0",
+  "version": "0.47.1",
   "description": "FireForge — a build tool for customizing Firefox",
   "type": "module",
   "main": "./dist/src/index.js",