@hominis/fireforge 0.34.0 → 0.34.2

package/CHANGELOG.md

@@ -5,6 +5,8 @@
 - Reworked the macOS resource-monitor (psutil) crash mitigation to an **in-process guard FireForge owns on its mach dispatches**: a `fireforge_mach_guard.pth` + module pair is installed directly into every discovered mach virtualenv site-packages (objdir `_virtualenvs` and the `~/.mozbuild` state dir) before each dispatch, because mach's venv re-exec drops `PYTHONPATH` and the 0.33.0 `sitecustomize.py` route never loaded (the env-var shim is retained only for the pre-venv bootstrap phase). The guard now covers the whole crash family: `psutil.virtual_memory`/`swap_memory`/`cpu_percent`/`cpu_times`/`disk_io_counters` are wrapped module-wide (which also covers the direct `psutil.virtual_memory()` call in `mozbuild.base._run_make`), and `SystemResourceMonitor` **construction** is guarded via an import hook that pre-populates `poll_interval` and degrades a partially-constructed monitor to a no-op — so the `AttributeError: ... poll_interval` variant can no longer escape a polling-only patch.
 - Routed every mach build dispatch — plain `fireforge build`, `build --ui` (`mach build faster`), and the pre-test build step of `fireforge test --build` — through one protected path (`runProtectedMachBuild`) with a **uniform recognized-crash retry budget** (default 2 retries, matching `--harness-retries`; the pre-test step forwards the operator's `--harness-retries` value). Each retry spawns a fresh mach process AND re-installs the guard (re-discovering venvs), so a venv materialized by a crashed first attempt is guarded on the next one instead of every retry dying on the same wedged state. Non-crash build failures are never retried; exhausted budgets surface the crash shape and evidence line above the regular diagnostics. The suite-specific test dispatches (`mach test` / `xpcshell-test` / `mochitest`) install the same guard.
 - Investigated the field incident where a failed pre-test build was followed by a ~64-minute full rebuild: FireForge's protected path performs **no configure/clobber beyond what `mach build faster` itself does** (retries never re-run `prepareBuildEnvironment`/`mach configure`; there is no fallback-to-full-build in FireForge). The pre-test build now also passes the previous build baseline into `prepareBuildEnvironment` exactly like `fireforge build` does, so auto-configure runs under identical conditions on both paths instead of diverging.
+- Fixed the degraded-psutil fallback crashing mozsystemmonitor on the fallback itself (downstream report: `TypeError: '_DegradedReading' object is not subscriptable` from `_build_meta`'s `psutil.virtual_memory()[0]`, aborting `mach mochitest` at startup with zero `TEST-START` lines, and `_collect failed: '_DegradedReading' object is not iterable`). The guard's degraded reading is now a **real duck type of psutil's namedtuple results**: each wrapped function degrades to a zeroed instance of psutil's own result namedtuple (`svmem`/`sswap`/`scputimes`/`sdiskio`, resolved via `getattr` without touching the failing syscall), falling back to an extended `_DegradedReading` that supports subscripting, iteration, `len()`, `_fields` (svmem field order), and `_asdict()`. `cpu_percent` degrades to a plain `0.0` float (callers do arithmetic on it), and `disk_io_counters` deliberately degrades to a zeroed struct rather than `None`. Runtime-verified by new python3-executed regression tests covering a mozsystemmonitor-style `_build_meta`/`_collect` sequence against a degraded host.
+- Taught the harness-crash classifier the two `_DegradedReading` crash signatures (`object is not subscriptable` / `object is not iterable`), so a startup abort with zero `TEST-START` lines from this family classifies as crash-not-failure and retries; the `_collect failed: '_DegradedReading' ...` evidence line is exempted from the noise-stripping that otherwise drops `_collect failed` degradation chatter, while a completed green summary still vetoes the classification.
 - Fixed the sharded/retry crash classifier marking fully green runs as `CRASH (N attempts)`: a completed green embedded summary (a `TEST_START`/`TEST-START` execution signal, `Unexpected results: 0`, and `SUITE_END`) now **vetoes** signature-based crash classification for that attempt, and resource-monitor degradation lines (`UserWarning: psutil failed to run: ...`, `_collect failed: ...`, FireForge's own degradation notice) plus mach's caught telemetry tracebacks are stripped from crash evidence entirely. Exit codes follow the corrected verdict: a run whose every shard is green exits 0, and a non-sharded (`--no-shard`) run whose embedded summary completed green exits 0 even when mach's own exit code went non-zero on harness noise (a note explains the override) — parsing embedded summaries by hand is no longer necessary. Runs with a non-zero unexpected count or real `TEST-UNEXPECTED-*` lines still fail, and the post-green shutdown re-entry shape still classifies as a crash.
 - Fixed `fireforge test <directory>` dispatching xpcshell-only directories to the mochitest runner ("could not find any mochitests under the following test path(s)"): the manifest walk now starts at the directory itself, so a directory whose own manifest is an `xpcshell.toml` dispatches to `mach xpcshell-test` just like an explicit `test_*.js` path.
 - Added `fireforge register --create-manifest`: registering a module under a directory with no `moz.build` now scaffolds the directory manifest (MPL-header + `EXTRA_JS_MODULES.<namespace>` list) and wires the parent `DIRS` chain up to the nearest existing moz.build (creating intermediates); without the flag the "Manifest not found" error names it. Also added an **xpcshell test-file pattern** (`**/test_*.js` outside `browser/base/content/test/`) that inserts the `["test_*.js"]` section into the directory's `xpcshell.toml` in mozbuild sort order — or creates the manifest and wires `XPCSHELL_TESTS_MANIFESTS` with `--create-manifest` — and extended the browser-chrome manifest rule to fork-owned `browser.toml` manifests at **arbitrary depth** under `browser/base/content/test/` (previously one level only).
@@ -13,6 +15,10 @@
 - Added `furnace scan --track`, which persists every discovered untracked component into the `stock` section of furnace.json non-interactively (same locked, rollback-journaled write path as the interactive confirm flow). Scan's help and its non-interactive output now state that scan is report-only by default and where the inventory is consumed, ending the report-persist-nothing-explain-nothing behavior.
 - Added `furnace chrome-doc create --browser-window`, a browser.xhtml-like scaffold for the document that ships as the fork's main browser window: `<html id="main-window">` root with the `windowtype="navigator:browser"`/`chromehidden`/geometry-`persist` attributes platform C++ reads before scripts run, while keeping the generic scaffold's bootstrap wiring, sentinel, and (already-correct) jar.mn registrations. When the target document matches a configured `tokenHostDocuments` entry and the flag was not passed, create warns that the browser-window variant is probably intended.
 - Added `furnace create --test-dir <dir>` to redirect the `--with-tests` scaffold (browser-chrome and xpcshell styles) to any engine-relative directory under `browser/base/content/test/` — nested manifests register correctly — and made all test scaffolds collision-safe: existing `browser.toml`/`xpcshell.toml`/`chrome.toml` manifests are appended to (with a shared-manifest notice) instead of scaffolded over, and existing `head.js`/test implementation files are never overwritten.
+- Fixed the degraded-psutil guard wedging `mach build` at exit on flapping hosts (0.34.1 post-upgrade field report: psutil's vm/swap syscalls flap between working and degraded, the fallback swap reading arrived svmem-shaped (8 fields) where mozsystemmonitor's parent reconstructs a 6-field `sswap` via `self._swap_type(*swap_mem)`, so every collector sample was rejected — `resourcemonitor.py:766: UserWarning: failed to read the received data` — upstream's handler breaks its drain loop, the pipe fills, the collector child blocks in `send()` and never reads the terminate sentinel, and mach hangs forever in its atexit `waitpid`). The guard's fallback readings are now **per-function, arity-correct, module-level namedtuples** (svmem 8 / sswap 6 / scputimes 4 / sdiskio 6; psutil's own result classes are still preferred when resolvable) that pickle by reference across the collector pipe and survive `type(reading)(*values)` reconstruction in **either** flapping direction (real type + degraded reading, or degraded type + real reading); `cpu_percent`/`cpu_times` fallbacks honour `percpu=True` (return `[]`) for the collector child's per-CPU sampling, and the last-resort `_DegradedReading` now tolerates positional-field construction. The report's "factory resolution bypassed in the collector child" observation is consistent with the two-copy load (`sitecustomize` via PYTHONPATH plus the in-venv `.pth`) or degraded-import-time `psutil._psplatform` resolution — the fix no longer depends on psutil internals resolving, so it is robust to both.
+- Added **collector suppression on degradation** to the guard: any observed psutil degradation sets a process-wide flag, after which monitors are kept inert — `start` never spawns the mozsystemmonitor collector child — and a degraded transition (any wrapped monitor method raising) or a raising `stop` best-effort terminates a live collector child (`_process.terminate()` + bounded join) and drains the parent end of the pipe, so a malformed sample stream can never fail the command or wedge shutdown even if a future shape mismatch slips through. (The upstream enabler — mozsystemmonitor warning then `break`ing out of its drain loop without draining, leaving the pipe to fill — is worth reporting to Mozilla but is not FireForge-fixable.)
+- Fixed the same degradation **failing `mach build` after a fully successful compile** ("Error running mach" with complete, valid `dist/` artifacts): degraded aggregate methods now return zeroed shapes instead of `None` where callers subscript the result (`aggregate_io` → zeroed io namedtuple, so mozbuild's `log_resource_usage` no longer dies on `usage["io"].read_bytes` at `mozbuild/controller/building.py:526`; `aggregate_cpu_percent`/`aggregate_cpu_times` return `[]`/`0.0`/zeroed shapes per their `per_cpu` mode), and the guard additionally wraps `mozbuild.controller.building.BuildMonitor.log_resource_usage` to warn-and-continue on any exception. Runtime-verified by new python3-executed regression tests covering per-function fallback arity, pickle/`type(*values)` round-trips in both flapping directions, collector suppression (stub monitor with upstream-named `_process`/`_pipe`), and the BuildMonitor wrap.
+- Taught the harness-crash classifier the post-success `AttributeError: 'NoneType' object has no attribute 'read_bytes'` traceback (protected builds retry it as crash-not-failure — with the fixed guard the incremental retry is cheap and green; real test failures still take precedence), and exempted the upstream drain-loop `failed to read the received data` warning from crash evidence so runs that complete despite the rejected-sample stream never classify as crashes.
 
 ## 0.33.0
 

package/dist/src/core/mach-resource-shim.d.ts

@@ -27,14 +27,66 @@
  *    `cpu_times` / `disk_io_counters` module-wide, so every call site —
  *    including the direct `psutil.virtual_memory()` in
  *    `mozbuild.base._run_make` — degrades to a `UserWarning` and a zeroed
- *    reading instead of aborting;
+ *    reading instead of aborting. Downstream report (0.34.0 cycle):
+ *    mozsystemmonitor subscripts (`_build_meta` does
+ *    `psutil.virtual_memory()[0]`) and iterates/unpacks the reading, so the
+ *    zeroed fallback is a full namedtuple duck type; `cpu_percent` degrades
+ *    to `0.0` because callers do arithmetic on it. Field report (0.34.1,
+ *    flapping host): the fallback must ALSO be per-function arity-correct
+ *    and reconstructible — mozsystemmonitor's parent captures reading types
+ *    at `__init__` (`self._swap_type = type(psutil.swap_memory())`) and
+ *    rebuilds each collector sample via `self._swap_type(*swap_mem)`, and an
+ *    svmem-shaped (8-field) fallback in the swap (6-field `sswap`) position
+ *    made the parent reject every sample ("failed to read the received
+ *    data") and BREAK its drain loop, so the pipe filled, the collector
+ *    child blocked in `send()` and never read the terminate sentinel, and
+ *    mach hung forever in its atexit `waitpid`. Each wrapped function now
+ *    degrades to a zeroed instance of psutil's own result namedtuple where
+ *    resolvable, else a module-level guard-owned namedtuple with the exact
+ *    macOS field order/arity for THAT function (module-level so readings
+ *    pickle by reference across the collector pipe); `_DegradedReading`
+ *    remains only as a documented last resort and now tolerates
+ *    `type(reading)(*values)` reconstruction;
  *  - guards `SystemResourceMonitor` CONSTRUCTION via an import hook:
  *    `poll_interval` is pre-populated before the real `__init__` runs, a
  *    failing `__init__` marks the instance degraded instead of raising,
  *    and every monitor method no-ops on a degraded instance — so a
  *    partially-constructed monitor can never surface the
- *    `AttributeError: poll_interval` variant.
+ *    `AttributeError: poll_interval` variant. Degraded aggregate methods
+ *    return zeroed shapes (not `None`) where callers subscript the result
+ *    (`aggregate_io` → zeroed io reading, so mozbuild's
+ *    `log_resource_usage` no longer dies on `usage["io"].read_bytes` after
+ *    a fully successful compile);
+ *  - suppresses the mozsystemmonitor collector child on degradation: once
+ *    any psutil degradation is observed in the process, monitors are kept
+ *    inert (`start` never spawns the collector), and a degraded transition
+ *    or raising `stop` best-effort terminates a live collector child and
+ *    drains the pipe — so a malformed sample stream can never wedge mach's
+ *    shutdown even if a future shape mismatch slips through;
+ *  - additionally wraps `mozbuild.controller.building.BuildMonitor
+ *    .log_resource_usage` to warn-and-continue on any exception, so
+ *    end-of-build resource reporting can never fail a build whose
+ *    artifacts are complete.
+ *
+ * Why the 0.34.1 report saw `_DegradedReading` in the collector child even
+ * though the psutil result classes resolve when probed directly: two guard
+ * copies can load (PYTHONPATH `sitecustomize` in the bootstrap phase plus
+ * the in-venv `.pth`), each with its own factory/classes, and flapping at
+ * guard-import time can perturb `psutil._psplatform` attribute resolution
+ * in the spawn child specifically. The fix is robust to either mechanism:
+ * the guard-owned fallback classes never depend on psutil internals
+ * resolving and are arity-correct in every position.
+ */
+/**
+ * Python guard body, importable both as the in-venv `fireforge_mach_guard`
+ * module (loaded by the sibling `.pth` at interpreter startup) and as the
+ * PYTHONPATH `sitecustomize` fallback. Defensive throughout: a missing or
+ * broken psutil/mozsystemmonitor leaves mach untouched, and every wrapper
+ * only intercepts the failure path. Exported for the Python-executing
+ * regression test (mach-resource-shim.python.test.ts), which asserts the
+ * degraded readings' runtime behavior rather than string-matching the source.
  */
+export declare const GUARD_PYTHON_SOURCE = "# Generated by FireForge - do not edit.\n# Degrades the broken host resource monitor family (psutil vs Darwin) from\n# fatal startup errors into non-fatal warnings, so mach builds and tests\n# proceed. Installed into the mach virtualenv site-packages via a .pth file\n# (survives mach's venv re-exec, which drops PYTHONPATH).\nimport builtins\nimport collections\nimport sys\nimport warnings\n\n# Any psutil degradation observed in this process. Once set, monitors are\n# kept inert (start never spawns the collector child) \u2014 on a flapping host a\n# healthy-looking moment must not be trusted to start a sample stream that a\n# later degraded reading could malform.\n_fireforge_host_degraded = [False]\n\n\ndef _fireforge_degraded_notice(exc):\n    _fireforge_host_degraded[0] = True\n    warnings.warn(\n        \"FireForge: host resource monitor degraded (%s); continuing without resource monitoring.\" % exc\n    )\n\n\nclass _DegradedReading(object):\n    # Last-resort duck type of psutil's namedtuple results (svmem field\n    # order): mozsystemmonitor subscripts, iterates, and unpacks readings,\n    # so the degraded fallback must survive r[0], list(r), len(r), r._fields\n    # and r._asdict(), not just attribute access. Field report (0.34.1): the\n    # parent also reconstructs readings via type(reading)(*values), so the\n    # constructor must tolerate the full positional field list. Normal\n    # degraded paths use the per-function namedtuple fallbacks below; this\n    # class only backstops a fallback that itself raised.\n    _fields = (\"total\", \"available\", \"percent\", \"used\", \"free\", \"active\", \"inactive\", \"wired\")\n    total = available = used = free = active = inactive = wired = 0\n    percent = 0.0\n\n    def __init__(self, *_args, **_kwargs):\n        pass\n\n    def __getattr__(self, _name):\n        return 0\n\n    def _values(self):\n        return tuple(getattr(self, _field) for _field in self._fields)\n\n    def __getitem__(self, index):\n        return self._values()[index]\n\n    def __iter__(self):\n        return iter(self._values())\n\n    def __len__(self):\n        return len(self._fields)\n\n    def _asdict(self):\n        return dict(zip(self._fields, self._values()))\n\n\n# Per-function fallback result classes with the exact macOS field\n# orders/arities. Module-level on purpose: readings cross the\n# mozsystemmonitor collector pipe via pickle (by reference), and the parent\n# rebuilds each sample with type(reading)(*values) \u2014 so a swap fallback must\n# be 6-field sswap-shaped in every position (field report 0.34.1: an\n# svmem-shaped fallback in the swap position made the parent reject every\n# sample, fill the pipe, block the collector child in send(), and wedge\n# mach's atexit join forever).\n_FallbackVmem = collections.namedtuple(\n    \"_FallbackVmem\",\n    (\"total\", \"available\", \"percent\", \"used\", \"free\", \"active\", \"inactive\", \"wired\"),\n)\n_FallbackSwap = collections.namedtuple(\n    \"_FallbackSwap\", (\"total\", \"used\", \"free\", \"percent\", \"sin\", \"sout\")\n)\n_FallbackCpuTimes = collections.namedtuple(\"_FallbackCpuTimes\", (\"user\", \"nice\", \"system\", \"idle\"))\n_FallbackDiskIO = collections.namedtuple(\n    \"_FallbackDiskIO\",\n    (\"read_count\", \"write_count\", \"read_bytes\", \"write_bytes\", \"read_time\", \"write_time\"),\n)\n\n_FALLBACK_CLASSES = {\n    \"virtual_memory\": _FallbackVmem,\n    \"swap_memory\": _FallbackSwap,\n    \"cpu_times\": _FallbackCpuTimes,\n    \"disk_io_counters\": _FallbackDiskIO,\n}\n\n# psutil result namedtuple per wrapped function, resolvable via getattr from\n# the platform module / _common without invoking the failing syscall.\n_PSUTIL_RESULT_CLASSES = {\n    \"virtual_memory\": \"svmem\",\n    \"swap_memory\": \"sswap\",\n    \"cpu_times\": \"scputimes\",\n    \"disk_io_counters\": \"sdiskio\",\n}\n\n\ndef _zeroed(cls):\n    return cls(*([0] * len(cls._fields)))\n\n\ndef _percpu_requested(args, kwargs, positional_index):\n    if \"percpu\" in kwargs:\n        return bool(kwargs[\"percpu\"])\n    return len(args) > positional_index and bool(args[positional_index])\n\n\ndef _degraded_result_factory(psutil, name):\n    # cpu_percent returns a plain float (or a per-CPU list) callers do\n    # arithmetic on.\n    if name == \"cpu_percent\":\n        def _cpu_percent_fallback(*args, **kwargs):\n            # cpu_percent(interval=None, percpu=False)\n            if _percpu_requested(args, kwargs, 1):\n                return []\n            return 0.0\n\n        return _cpu_percent_fallback\n    # The guard-owned class is always shape-correct for this function;\n    # psutil's own result namedtuple is still preferred when resolvable so\n    # type() captures stay identical to real readings on a flapping host.\n    _cls = _FALLBACK_CLASSES.get(name, _DegradedReading)\n    _cls_name = _PSUTIL_RESULT_CLASSES.get(name)\n    if _cls_name is not None:\n        for _mod in (getattr(psutil, \"_psplatform\", None), getattr(psutil, \"_common\", None)):\n            _real = getattr(_mod, _cls_name, None) if _mod is not None else None\n            if _real is not None and getattr(_real, \"_fields\", None):\n                _cls = _real\n                break\n    if name == \"cpu_times\":\n        def _cpu_times_fallback(*args, **kwargs):\n            # cpu_times(percpu=False); the collector child samples per-CPU.\n            if _percpu_requested(args, kwargs, 0):\n                return []\n            return _zeroed(_cls)\n\n        return _cpu_times_fallback\n\n    def _reading_fallback(*_args, **_kwargs):\n        return _zeroed(_cls)\n\n    return _reading_fallback\n\n\ndef _guard_psutil():\n    try:\n        import psutil\n    except Exception:\n        return\n\n    def _wrap(orig, fallback):\n        def wrapper(*args, **kwargs):\n            try:\n                return orig(*args, **kwargs)\n            except Exception as exc:  # noqa: BLE001\n                _fireforge_degraded_notice(exc)\n                try:\n                    return fallback(*args, **kwargs)\n                except Exception:  # noqa: BLE001\n                    return _DegradedReading()\n\n        return wrapper\n\n    for _name in (\"virtual_memory\", \"swap_memory\", \"cpu_percent\", \"cpu_times\", \"disk_io_counters\"):\n        _orig = getattr(psutil, _name, None)\n        if _orig is not None and not getattr(_orig, \"_fireforge_guarded\", False):\n            try:\n                _fallback = _degraded_result_factory(psutil, _name)\n            except Exception:  # noqa: BLE001\n                _fallback = _DegradedReading\n            _wrapped = _wrap(_orig, _fallback)\n            _wrapped._fireforge_guarded = True\n            setattr(psutil, _name, _wrapped)\n\n\n_MONITOR_METHOD_NAMES = (\n    \"start\",\n    \"stop\",\n    \"record_event\",\n    \"record_marker\",\n    \"begin_phase\",\n    \"finish_phase\",\n    \"aggregate_cpu_percent\",\n    \"aggregate_cpu_times\",\n    \"aggregate_io\",\n    \"min_memory_available\",\n    \"as_dict\",\n)\n\n\ndef _fireforge_stop_collector(monitor):\n    # Best-effort: terminate a live collector child and drain the parent end\n    # of the pipe, so a child blocked in send() on a full pipe can never\n    # keep mach's atexit join (os.waitpid) waiting forever. Attribute names\n    # follow upstream resourcemonitor.py (self._process / self._pipe); a\n    # miss on a refactored upstream is harmless \u2014 the shape-correct\n    # fallbacks above still keep the sample stream well-formed.\n    try:\n        proc = getattr(monitor, \"_process\", None)\n        if proc is not None and getattr(proc, \"is_alive\", lambda: False)():\n            proc.terminate()\n            proc.join(1)\n    except Exception:  # noqa: BLE001\n        pass\n    try:\n        pipe = getattr(monitor, \"_pipe\", None)\n        if pipe is not None:\n            while pipe.poll(0):\n                pipe.recv()\n    except Exception:  # noqa: BLE001\n        pass\n\n\ndef _monitor_per_cpu_requested(args, kwargs):\n    # aggregate_cpu_percent/aggregate_cpu_times(start=None, end=None,\n    # phase=None, per_cpu=True) \u2014 self already stripped from args.\n    if \"per_cpu\" in kwargs:\n        return bool(kwargs[\"per_cpu\"])\n    if len(args) > 3:\n        return bool(args[3])\n    return True\n\n\ndef _monitor_degraded_result(name, args, kwargs):\n    # Degraded aggregate methods return zeroed shapes, not None, where\n    # callers subscript the result: mozbuild's log_resource_usage does\n    # usage[\"io\"].read_bytes after a successful compile (field report\n    # 0.34.1: 'NoneType' object has no attribute 'read_bytes' failed the\n    # build with complete artifacts).\n    if name == \"aggregate_io\":\n        return _zeroed(_FallbackDiskIO)\n    if name == \"aggregate_cpu_percent\":\n        return [] if _monitor_per_cpu_requested(args, kwargs) else 0.0\n    if name == \"aggregate_cpu_times\":\n        return [] if _monitor_per_cpu_requested(args, kwargs) else _zeroed(_FallbackCpuTimes)\n    return None\n\n\ndef _patch_monitor_class(cls):\n    if cls is None or getattr(cls, \"_fireforge_guarded\", False):\n        return\n    orig_init = cls.__init__\n\n    def guarded_init(self, *args, **kwargs):\n        # Pre-populate the attributes a partially-constructed instance is\n        # later polled for, so a failing __init__ can never surface\n        # AttributeError: poll_interval.\n        self.poll_interval = kwargs.get(\"poll_interval\") or 1.0\n        self._fireforge_degraded = False\n        try:\n            orig_init(self, *args, **kwargs)\n        except Exception as exc:  # noqa: BLE001\n            self._fireforge_degraded = True\n            _fireforge_degraded_notice(exc)\n        if _fireforge_host_degraded[0] and not self._fireforge_degraded:\n            # Host already degraded once: keep the monitor inert so the\n            # collector child never spawns on a flapping host.\n            self._fireforge_degraded = True\n        if self._fireforge_degraded:\n            _fireforge_stop_collector(self)\n\n    cls.__init__ = guarded_init\n\n    def _wrap_method(name, orig):\n        def wrapper(self, *args, **kwargs):\n            if (\n                name == \"start\"\n                and _fireforge_host_degraded[0]\n                and not getattr(self, \"_fireforge_degraded\", False)\n            ):\n                self._fireforge_degraded = True\n                _fireforge_stop_collector(self)\n            if getattr(self, \"_fireforge_degraded\", False):\n                if name == \"stop\":\n                    _fireforge_stop_collector(self)\n                return _monitor_degraded_result(name, args, kwargs)\n            try:\n                return orig(self, *args, **kwargs)\n            except Exception as exc:  # noqa: BLE001\n                self._fireforge_degraded = True\n                _fireforge_degraded_notice(exc)\n                # A raising stop (or any degraded transition) must still\n                # take the collector child down, or the parent's atexit\n                # join hangs on a child blocked writing to a full pipe.\n                _fireforge_stop_collector(self)\n                return _monitor_degraded_result(name, args, kwargs)\n\n        return wrapper\n\n    for _name in _MONITOR_METHOD_NAMES:\n        _orig = getattr(cls, _name, None)\n        if _orig is not None:\n            setattr(cls, _name, _wrap_method(_name, _orig))\n    cls._fireforge_guarded = True\n\n\ndef _patch_build_monitor_class(cls):\n    # mozbuild's BuildMonitor is not a SystemResourceMonitor; only\n    # log_resource_usage needs guarding \u2014 end-of-build resource reporting\n    # must never fail a build whose artifacts are complete. No __init__\n    # replacement, no degraded-instance gating.\n    if cls is None or getattr(cls, \"_fireforge_guarded\", False):\n        return\n    _orig = getattr(cls, \"log_resource_usage\", None)\n    if _orig is not None:\n        def _guarded_log_resource_usage(self, *args, **kwargs):\n            try:\n                return _orig(self, *args, **kwargs)\n            except Exception as exc:  # noqa: BLE001\n                _fireforge_degraded_notice(exc)\n                return None\n\n        cls.log_resource_usage = _guarded_log_resource_usage\n    cls._fireforge_guarded = True\n\n\n_MONITOR_CLASS_PATCHES = (\n    (\"mozsystemmonitor.resourcemonitor\", \"SystemResourceMonitor\", _patch_monitor_class),\n    (\"mozbuild.resources\", \"SystemResourceMonitor\", _patch_monitor_class),\n    (\"mozbuild.controller.building\", \"BuildMonitor\", _patch_build_monitor_class),\n)\n\n\ndef _patch_loaded_monitor_modules():\n    for _mod_name, _cls_name, _patcher in _MONITOR_CLASS_PATCHES:\n        _mod = sys.modules.get(_mod_name)\n        if _mod is not None:\n            try:\n                _patcher(getattr(_mod, _cls_name, None))\n            except Exception:  # noqa: BLE001\n                pass\n\n\ndef _install_import_hook():\n    _orig_import = builtins.__import__\n    if getattr(_orig_import, \"_fireforge_guarded\", False):\n        return\n\n    def _guarding_import(name, *args, **kwargs):\n        _module = _orig_import(name, *args, **kwargs)\n        try:\n            _patch_loaded_monitor_modules()\n        except Exception:  # noqa: BLE001\n            pass\n        return _module\n\n    _guarding_import._fireforge_guarded = True\n    builtins.__import__ = _guarding_import\n\n\ntry:\n    _guard_psutil()\n    _patch_loaded_monitor_modules()\n    _install_import_hook()\nexcept Exception:  # noqa: BLE001\n    pass\n";
 /** Result of installing the mach resource guard before a dispatch. */
 export interface MachResourceGuardInstallation {
     /**

package/dist/src/core/mach-resource-shim.js

@@ -28,13 +28,55 @@
  *    `cpu_times` / `disk_io_counters` module-wide, so every call site —
  *    including the direct `psutil.virtual_memory()` in
  *    `mozbuild.base._run_make` — degrades to a `UserWarning` and a zeroed
- *    reading instead of aborting;
+ *    reading instead of aborting. Downstream report (0.34.0 cycle):
+ *    mozsystemmonitor subscripts (`_build_meta` does
+ *    `psutil.virtual_memory()[0]`) and iterates/unpacks the reading, so the
+ *    zeroed fallback is a full namedtuple duck type; `cpu_percent` degrades
+ *    to `0.0` because callers do arithmetic on it. Field report (0.34.1,
+ *    flapping host): the fallback must ALSO be per-function arity-correct
+ *    and reconstructible — mozsystemmonitor's parent captures reading types
+ *    at `__init__` (`self._swap_type = type(psutil.swap_memory())`) and
+ *    rebuilds each collector sample via `self._swap_type(*swap_mem)`, and an
+ *    svmem-shaped (8-field) fallback in the swap (6-field `sswap`) position
+ *    made the parent reject every sample ("failed to read the received
+ *    data") and BREAK its drain loop, so the pipe filled, the collector
+ *    child blocked in `send()` and never read the terminate sentinel, and
+ *    mach hung forever in its atexit `waitpid`. Each wrapped function now
+ *    degrades to a zeroed instance of psutil's own result namedtuple where
+ *    resolvable, else a module-level guard-owned namedtuple with the exact
+ *    macOS field order/arity for THAT function (module-level so readings
+ *    pickle by reference across the collector pipe); `_DegradedReading`
+ *    remains only as a documented last resort and now tolerates
+ *    `type(reading)(*values)` reconstruction;
  *  - guards `SystemResourceMonitor` CONSTRUCTION via an import hook:
  *    `poll_interval` is pre-populated before the real `__init__` runs, a
  *    failing `__init__` marks the instance degraded instead of raising,
  *    and every monitor method no-ops on a degraded instance — so a
  *    partially-constructed monitor can never surface the
- *    `AttributeError: poll_interval` variant.
+ *    `AttributeError: poll_interval` variant. Degraded aggregate methods
+ *    return zeroed shapes (not `None`) where callers subscript the result
+ *    (`aggregate_io` → zeroed io reading, so mozbuild's
+ *    `log_resource_usage` no longer dies on `usage["io"].read_bytes` after
+ *    a fully successful compile);
+ *  - suppresses the mozsystemmonitor collector child on degradation: once
+ *    any psutil degradation is observed in the process, monitors are kept
+ *    inert (`start` never spawns the collector), and a degraded transition
+ *    or raising `stop` best-effort terminates a live collector child and
+ *    drains the pipe — so a malformed sample stream can never wedge mach's
+ *    shutdown even if a future shape mismatch slips through;
+ *  - additionally wraps `mozbuild.controller.building.BuildMonitor
+ *    .log_resource_usage` to warn-and-continue on any exception, so
+ *    end-of-build resource reporting can never fail a build whose
+ *    artifacts are complete.
+ *
+ * Why the 0.34.1 report saw `_DegradedReading` in the collector child even
+ * though the psutil result classes resolve when probed directly: two guard
+ * copies can load (PYTHONPATH `sitecustomize` in the bootstrap phase plus
+ * the in-venv `.pth`), each with its own factory/classes, and flapping at
+ * guard-import time can perturb `psutil._psplatform` attribute resolution
+ * in the spawn child specifically. The fix is robust to either mechanism:
+ * the guard-owned fallback classes never depend on psutil internals
+ * resolving and are arity-correct in every position.
  */
 import { readdir } from 'node:fs/promises';
 import { homedir, tmpdir } from 'node:os';
@@ -51,31 +93,153 @@ const GUARD_MODULE_NAME = 'fireforge_mach_guard';
  * module (loaded by the sibling `.pth` at interpreter startup) and as the
  * PYTHONPATH `sitecustomize` fallback. Defensive throughout: a missing or
  * broken psutil/mozsystemmonitor leaves mach untouched, and every wrapper
- * only intercepts the failure path.
+ * only intercepts the failure path. Exported for the Python-executing
+ * regression test (mach-resource-shim.python.test.ts), which asserts the
+ * degraded readings' runtime behavior rather than string-matching the source.
  */
-const GUARD_PYTHON_SOURCE = `# Generated by FireForge - do not edit.
+export const GUARD_PYTHON_SOURCE = `# Generated by FireForge - do not edit.
 # Degrades the broken host resource monitor family (psutil vs Darwin) from
 # fatal startup errors into non-fatal warnings, so mach builds and tests
 # proceed. Installed into the mach virtualenv site-packages via a .pth file
 # (survives mach's venv re-exec, which drops PYTHONPATH).
 import builtins
+import collections
 import sys
 import warnings
 
+# Any psutil degradation observed in this process. Once set, monitors are
+# kept inert (start never spawns the collector child) — on a flapping host a
+# healthy-looking moment must not be trusted to start a sample stream that a
+# later degraded reading could malform.
+_fireforge_host_degraded = [False]
+
 
 def _fireforge_degraded_notice(exc):
+    _fireforge_host_degraded[0] = True
     warnings.warn(
         "FireForge: host resource monitor degraded (%s); continuing without resource monitoring." % exc
     )
 
 
 class _DegradedReading(object):
+    # Last-resort duck type of psutil's namedtuple results (svmem field
+    # order): mozsystemmonitor subscripts, iterates, and unpacks readings,
+    # so the degraded fallback must survive r[0], list(r), len(r), r._fields
+    # and r._asdict(), not just attribute access. Field report (0.34.1): the
+    # parent also reconstructs readings via type(reading)(*values), so the
+    # constructor must tolerate the full positional field list. Normal
+    # degraded paths use the per-function namedtuple fallbacks below; this
+    # class only backstops a fallback that itself raised.
+    _fields = ("total", "available", "percent", "used", "free", "active", "inactive", "wired")
     total = available = used = free = active = inactive = wired = 0
     percent = 0.0
 
+    def __init__(self, *_args, **_kwargs):
+        pass
+
     def __getattr__(self, _name):
         return 0
 
+    def _values(self):
+        return tuple(getattr(self, _field) for _field in self._fields)
+
+    def __getitem__(self, index):
+        return self._values()[index]
+
+    def __iter__(self):
+        return iter(self._values())
+
+    def __len__(self):
+        return len(self._fields)
+
+    def _asdict(self):
+        return dict(zip(self._fields, self._values()))
+
+
+# Per-function fallback result classes with the exact macOS field
+# orders/arities. Module-level on purpose: readings cross the
+# mozsystemmonitor collector pipe via pickle (by reference), and the parent
+# rebuilds each sample with type(reading)(*values) — so a swap fallback must
+# be 6-field sswap-shaped in every position (field report 0.34.1: an
+# svmem-shaped fallback in the swap position made the parent reject every
+# sample, fill the pipe, block the collector child in send(), and wedge
+# mach's atexit join forever).
+_FallbackVmem = collections.namedtuple(
+    "_FallbackVmem",
+    ("total", "available", "percent", "used", "free", "active", "inactive", "wired"),
+)
+_FallbackSwap = collections.namedtuple(
+    "_FallbackSwap", ("total", "used", "free", "percent", "sin", "sout")
+)
+_FallbackCpuTimes = collections.namedtuple("_FallbackCpuTimes", ("user", "nice", "system", "idle"))
+_FallbackDiskIO = collections.namedtuple(
+    "_FallbackDiskIO",
+    ("read_count", "write_count", "read_bytes", "write_bytes", "read_time", "write_time"),
+)
+
+_FALLBACK_CLASSES = {
+    "virtual_memory": _FallbackVmem,
+    "swap_memory": _FallbackSwap,
+    "cpu_times": _FallbackCpuTimes,
+    "disk_io_counters": _FallbackDiskIO,
+}
+
+# psutil result namedtuple per wrapped function, resolvable via getattr from
+# the platform module / _common without invoking the failing syscall.
+_PSUTIL_RESULT_CLASSES = {
+    "virtual_memory": "svmem",
+    "swap_memory": "sswap",
+    "cpu_times": "scputimes",
+    "disk_io_counters": "sdiskio",
+}
+
+
+def _zeroed(cls):
+    return cls(*([0] * len(cls._fields)))
+
+
+def _percpu_requested(args, kwargs, positional_index):
+    if "percpu" in kwargs:
+        return bool(kwargs["percpu"])
+    return len(args) > positional_index and bool(args[positional_index])
+
+
+def _degraded_result_factory(psutil, name):
+    # cpu_percent returns a plain float (or a per-CPU list) callers do
+    # arithmetic on.
+    if name == "cpu_percent":
+        def _cpu_percent_fallback(*args, **kwargs):
+            # cpu_percent(interval=None, percpu=False)
+            if _percpu_requested(args, kwargs, 1):
+                return []
+            return 0.0
+
+        return _cpu_percent_fallback
+    # The guard-owned class is always shape-correct for this function;
+    # psutil's own result namedtuple is still preferred when resolvable so
+    # type() captures stay identical to real readings on a flapping host.
+    _cls = _FALLBACK_CLASSES.get(name, _DegradedReading)
+    _cls_name = _PSUTIL_RESULT_CLASSES.get(name)
+    if _cls_name is not None:
+        for _mod in (getattr(psutil, "_psplatform", None), getattr(psutil, "_common", None)):
+            _real = getattr(_mod, _cls_name, None) if _mod is not None else None
+            if _real is not None and getattr(_real, "_fields", None):
+                _cls = _real
+                break
+    if name == "cpu_times":
+        def _cpu_times_fallback(*args, **kwargs):
+            # cpu_times(percpu=False); the collector child samples per-CPU.
+            if _percpu_requested(args, kwargs, 0):
+                return []
+            return _zeroed(_cls)
+
+        return _cpu_times_fallback
+
+    def _reading_fallback(*_args, **_kwargs):
+        return _zeroed(_cls)
+
+    return _reading_fallback
+
 
 def _guard_psutil():
     try:
@@ -83,20 +247,27 @@ def _guard_psutil():
     except Exception:
         return
 
-    def _wrap(orig):
+    def _wrap(orig, fallback):
         def wrapper(*args, **kwargs):
             try:
                 return orig(*args, **kwargs)
             except Exception as exc:  # noqa: BLE001
                 _fireforge_degraded_notice(exc)
-                return _DegradedReading()
+                try:
+                    return fallback(*args, **kwargs)
+                except Exception:  # noqa: BLE001
+                    return _DegradedReading()
 
         return wrapper
 
     for _name in ("virtual_memory", "swap_memory", "cpu_percent", "cpu_times", "disk_io_counters"):
         _orig = getattr(psutil, _name, None)
         if _orig is not None and not getattr(_orig, "_fireforge_guarded", False):
-            _wrapped = _wrap(_orig)
+            try:
+                _fallback = _degraded_result_factory(psutil, _name)
+            except Exception:  # noqa: BLE001
+                _fallback = _DegradedReading
+            _wrapped = _wrap(_orig, _fallback)
             _wrapped._fireforge_guarded = True
             setattr(psutil, _name, _wrapped)
 
@@ -116,6 +287,54 @@ _MONITOR_METHOD_NAMES = (
 )
 
 
+def _fireforge_stop_collector(monitor):
+    # Best-effort: terminate a live collector child and drain the parent end
+    # of the pipe, so a child blocked in send() on a full pipe can never
+    # keep mach's atexit join (os.waitpid) waiting forever. Attribute names
+    # follow upstream resourcemonitor.py (self._process / self._pipe); a
+    # miss on a refactored upstream is harmless — the shape-correct
+    # fallbacks above still keep the sample stream well-formed.
+    try:
+        proc = getattr(monitor, "_process", None)
+        if proc is not None and getattr(proc, "is_alive", lambda: False)():
+            proc.terminate()
+            proc.join(1)
+    except Exception:  # noqa: BLE001
+        pass
+    try:
+        pipe = getattr(monitor, "_pipe", None)
+        if pipe is not None:
+            while pipe.poll(0):
+                pipe.recv()
+    except Exception:  # noqa: BLE001
+        pass
+
+
+def _monitor_per_cpu_requested(args, kwargs):
+    # aggregate_cpu_percent/aggregate_cpu_times(start=None, end=None,
+    # phase=None, per_cpu=True) — self already stripped from args.
+    if "per_cpu" in kwargs:
+        return bool(kwargs["per_cpu"])
+    if len(args) > 3:
+        return bool(args[3])
+    return True
+
+
+def _monitor_degraded_result(name, args, kwargs):
+    # Degraded aggregate methods return zeroed shapes, not None, where
+    # callers subscript the result: mozbuild's log_resource_usage does
+    # usage["io"].read_bytes after a successful compile (field report
+    # 0.34.1: 'NoneType' object has no attribute 'read_bytes' failed the
+    # build with complete artifacts).
+    if name == "aggregate_io":
+        return _zeroed(_FallbackDiskIO)
+    if name == "aggregate_cpu_percent":
+        return [] if _monitor_per_cpu_requested(args, kwargs) else 0.0
+    if name == "aggregate_cpu_times":
+        return [] if _monitor_per_cpu_requested(args, kwargs) else _zeroed(_FallbackCpuTimes)
+    return None
+
+
 def _patch_monitor_class(cls):
     if cls is None or getattr(cls, "_fireforge_guarded", False):
         return
@@ -132,38 +351,81 @@ def _patch_monitor_class(cls):
         except Exception as exc:  # noqa: BLE001
             self._fireforge_degraded = True
             _fireforge_degraded_notice(exc)
+        if _fireforge_host_degraded[0] and not self._fireforge_degraded:
+            # Host already degraded once: keep the monitor inert so the
+            # collector child never spawns on a flapping host.
+            self._fireforge_degraded = True
+        if self._fireforge_degraded:
+            _fireforge_stop_collector(self)
 
     cls.__init__ = guarded_init
 
-    def _wrap_method(orig):
+    def _wrap_method(name, orig):
         def wrapper(self, *args, **kwargs):
+            if (
+                name == "start"
+                and _fireforge_host_degraded[0]
+                and not getattr(self, "_fireforge_degraded", False)
+            ):
+                self._fireforge_degraded = True
+                _fireforge_stop_collector(self)
             if getattr(self, "_fireforge_degraded", False):
-                return None
+                if name == "stop":
+                    _fireforge_stop_collector(self)
+                return _monitor_degraded_result(name, args, kwargs)
             try:
                 return orig(self, *args, **kwargs)
             except Exception as exc:  # noqa: BLE001
                 self._fireforge_degraded = True
                 _fireforge_degraded_notice(exc)
-                return None
+                # A raising stop (or any degraded transition) must still
+                # take the collector child down, or the parent's atexit
+                # join hangs on a child blocked writing to a full pipe.
+                _fireforge_stop_collector(self)
+                return _monitor_degraded_result(name, args, kwargs)
 
         return wrapper
 
     for _name in _MONITOR_METHOD_NAMES:
         _orig = getattr(cls, _name, None)
         if _orig is not None:
-            setattr(cls, _name, _wrap_method(_orig))
+            setattr(cls, _name, _wrap_method(_name, _orig))
     cls._fireforge_guarded = True
 
 
-_MONITOR_MODULES = ("mozsystemmonitor.resourcemonitor", "mozbuild.resources")
+def _patch_build_monitor_class(cls):
+    # mozbuild's BuildMonitor is not a SystemResourceMonitor; only
+    # log_resource_usage needs guarding — end-of-build resource reporting
+    # must never fail a build whose artifacts are complete. No __init__
+    # replacement, no degraded-instance gating.
+    if cls is None or getattr(cls, "_fireforge_guarded", False):
+        return
+    _orig = getattr(cls, "log_resource_usage", None)
+    if _orig is not None:
+        def _guarded_log_resource_usage(self, *args, **kwargs):
+            try:
+                return _orig(self, *args, **kwargs)
+            except Exception as exc:  # noqa: BLE001
+                _fireforge_degraded_notice(exc)
+                return None
+
+        cls.log_resource_usage = _guarded_log_resource_usage
+    cls._fireforge_guarded = True
+
+
+_MONITOR_CLASS_PATCHES = (
+    ("mozsystemmonitor.resourcemonitor", "SystemResourceMonitor", _patch_monitor_class),
+    ("mozbuild.resources", "SystemResourceMonitor", _patch_monitor_class),
+    ("mozbuild.controller.building", "BuildMonitor", _patch_build_monitor_class),
+)
 
 
 def _patch_loaded_monitor_modules():
-    for _mod_name in _MONITOR_MODULES:
+    for _mod_name, _cls_name, _patcher in _MONITOR_CLASS_PATCHES:
         _mod = sys.modules.get(_mod_name)
         if _mod is not None:
             try:
-                _patch_monitor_class(getattr(_mod, "SystemResourceMonitor", None))
+                _patcher(getattr(_mod, _cls_name, None))
             except Exception:  # noqa: BLE001
                 pass
 

package/dist/src/core/test-harness-crash.js

@@ -56,6 +56,17 @@ const NO_OUTPUT_TIMEOUT_PATTERN = /timed out after \d+ seconds with no output/i;
  * on macOS. Each is matched per-line so the evidence line in the report is
  * the concrete failure, not the whole traceback.
  */
+// Downstream report (0.34.0 cycle): a pre-fix _DegradedReading fallback that
+// wasn't a real namedtuple duck type crashed mozsystemmonitor on the fallback
+// itself (`_build_meta` subscripts the reading; `_collect` unpacks it); a
+// startup abort with zero TEST-START lines from this family is a crash, not
+// a test failure. The `_collect failed` variant is caught-and-logged, so it
+// carries no Traceback header — it gets its own zero-TEST-START check in
+// `detectHarnessCrashSignature` besides joining the traceback cluster.
+const DEGRADED_READING_CRASH_SIGNALS = [
+    /'_DegradedReading' object is not subscriptable/,
+    /'_DegradedReading' object is not iterable/,
+];
 const STARTUP_TRACEBACK_SIGNALS = [
     /AttributeError:.*SystemResourceMonitor/,
     /'SystemResourceMonitor' object has no attribute/,
@@ -64,6 +75,14 @@ const STARTUP_TRACEBACK_SIGNALS = [
     /HOST_VM_INFO64/,
     /\(ipc\/mig\) array not large enough/,
     /psutil\.[A-Za-z]*Error/,
+    // Field report (0.34.1): on a degraded host every collector sample is
+    // rejected, aggregation has nothing, and mozbuild's log_resource_usage
+    // dies on usage["io"].read_bytes AFTER a fully successful compile
+    // ("Error running mach" with complete artifacts). Environmental, not a
+    // build regression — the protected build retries it (incremental retry
+    // is cheap and the guard keeps the retry green).
+    /AttributeError: 'NoneType' object has no attribute 'read_bytes'/,
+    ...DEGRADED_READING_CRASH_SIGNALS,
 ];
 function findLine(output, patterns) {
     for (const line of output.split(/\r?\n/)) {
@@ -83,9 +102,18 @@ function findLine(output, patterns) {
 const NOISE_LINE_PATTERNS = [
     /\bUserWarning\b/,
     /psutil failed to run/i,
-    /_collect failed/i,
+    // `_collect failed` chatter is noise on green runs, EXCEPT the
+    // `_DegradedReading` variant: that is the crash evidence for the
+    // "object is not iterable" startup-abort signature above and must
+    // survive stripNonSignalNoise.
+    /_collect failed(?!.*_DegradedReading)/i,
     /FireForge: host resource monitor degraded/i,
     /warnings\.warn\(/,
+    // mozsystemmonitor's parent-side drain loop rejecting a malformed sample
+    // (field report 0.34.1); mach sometimes reprints the warning text without
+    // the `UserWarning:` token, and a run that completes despite the stream
+    // must never classify as crash on it.
+    /failed to read the received data/i,
 ];
 /** Matches the caught/telemetry context that marks a traceback as benign. */
 const BENIGN_TRACEBACK_CONTEXT = /telemetry|glean/i;
@@ -197,6 +225,18 @@ export function detectHarnessCrashSignature(output) {
     // the no-output timeout. A trailing "Passed: 0" summary is part of this
     // shape and must not be read as a result.
     if (!hasTestStart) {
+        // Startup abort on the degraded-reading fallback family: the `_collect
+        // failed: '_DegradedReading' ...` variant is caught-and-logged (no
+        // Traceback header), so the traceback cluster above never sees it.
+        if (realFailures.length === 0) {
+            const degradedLine = findLine(evidence, DEGRADED_READING_CRASH_SIGNALS);
+            if (degradedLine) {
+                return {
+                    reason: 'degraded resource reading crashed the harness fallback (_DegradedReading)',
+                    line: degradedLine,
+                };
+            }
+        }
         const timeoutLine = findLine(evidence, [NO_OUTPUT_TIMEOUT_PATTERN]);
         if (timeoutLine) {
             return { reason: 'no-output timeout before any test started', line: timeoutLine };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hominis/fireforge",
-  "version": "0.34.0",
+  "version": "0.34.2",
   "description": "FireForge — a build tool for customizing Firefox",
   "type": "module",
   "main": "./dist/src/index.js",