@hominis/fireforge 0.34.0 → 0.34.1

package/CHANGELOG.md

@@ -5,6 +5,8 @@
 - Reworked the macOS resource-monitor (psutil) crash mitigation to an **in-process guard FireForge owns on its mach dispatches**: a `fireforge_mach_guard.pth` + module pair is installed directly into every discovered mach virtualenv site-packages (objdir `_virtualenvs` and the `~/.mozbuild` state dir) before each dispatch, because mach's venv re-exec drops `PYTHONPATH` and the 0.33.0 `sitecustomize.py` route never loaded (the env-var shim is retained only for the pre-venv bootstrap phase). The guard now covers the whole crash family: `psutil.virtual_memory`/`swap_memory`/`cpu_percent`/`cpu_times`/`disk_io_counters` are wrapped module-wide (which also covers the direct `psutil.virtual_memory()` call in `mozbuild.base._run_make`), and `SystemResourceMonitor` **construction** is guarded via an import hook that pre-populates `poll_interval` and degrades a partially-constructed monitor to a no-op — so the `AttributeError: ... poll_interval` variant can no longer escape a polling-only patch.
 - Routed every mach build dispatch — plain `fireforge build`, `build --ui` (`mach build faster`), and the pre-test build step of `fireforge test --build` — through one protected path (`runProtectedMachBuild`) with a **uniform recognized-crash retry budget** (default 2 retries, matching `--harness-retries`; the pre-test step forwards the operator's `--harness-retries` value). Each retry spawns a fresh mach process AND re-installs the guard (re-discovering venvs), so a venv materialized by a crashed first attempt is guarded on the next one instead of every retry dying on the same wedged state. Non-crash build failures are never retried; exhausted budgets surface the crash shape and evidence line above the regular diagnostics. The suite-specific test dispatches (`mach test` / `xpcshell-test` / `mochitest`) install the same guard.
 - Investigated the field incident where a failed pre-test build was followed by a ~64-minute full rebuild: FireForge's protected path performs **no configure/clobber beyond what `mach build faster` itself does** (retries never re-run `prepareBuildEnvironment`/`mach configure`; there is no fallback-to-full-build in FireForge). The pre-test build now also passes the previous build baseline into `prepareBuildEnvironment` exactly like `fireforge build` does, so auto-configure runs under identical conditions on both paths instead of diverging.
+- Fixed the degraded-psutil fallback crashing mozsystemmonitor on the fallback itself (downstream report: `TypeError: '_DegradedReading' object is not subscriptable` from `_build_meta`'s `psutil.virtual_memory()[0]`, aborting `mach mochitest` at startup with zero `TEST-START` lines, and `_collect failed: '_DegradedReading' object is not iterable`). The guard's degraded reading is now a **real duck type of psutil's namedtuple results**: each wrapped function degrades to a zeroed instance of psutil's own result namedtuple (`svmem`/`sswap`/`scputimes`/`sdiskio`, resolved via `getattr` without touching the failing syscall), falling back to an extended `_DegradedReading` that supports subscripting, iteration, `len()`, `_fields` (svmem field order), and `_asdict()`. `cpu_percent` degrades to a plain `0.0` float (callers do arithmetic on it), and `disk_io_counters` deliberately degrades to a zeroed struct rather than `None`. Runtime-verified by new python3-executed regression tests covering a mozsystemmonitor-style `_build_meta`/`_collect` sequence against a degraded host.
+- Taught the harness-crash classifier the two `_DegradedReading` crash signatures (`object is not subscriptable` / `object is not iterable`), so a startup abort with zero `TEST-START` lines from this family classifies as crash-not-failure and retries; the `_collect failed: '_DegradedReading' ...` evidence line is exempted from the noise-stripping that otherwise drops `_collect failed` degradation chatter, while a completed green summary still vetoes the classification.
 - Fixed the sharded/retry crash classifier marking fully green runs as `CRASH (N attempts)`: a completed green embedded summary (a `TEST_START`/`TEST-START` execution signal, `Unexpected results: 0`, and `SUITE_END`) now **vetoes** signature-based crash classification for that attempt, and resource-monitor degradation lines (`UserWarning: psutil failed to run: ...`, `_collect failed: ...`, FireForge's own degradation notice) plus mach's caught telemetry tracebacks are stripped from crash evidence entirely. Exit codes follow the corrected verdict: a run whose every shard is green exits 0, and a non-sharded (`--no-shard`) run whose embedded summary completed green exits 0 even when mach's own exit code went non-zero on harness noise (a note explains the override) — parsing embedded summaries by hand is no longer necessary. Runs with a non-zero unexpected count or real `TEST-UNEXPECTED-*` lines still fail, and the post-green shutdown re-entry shape still classifies as a crash.
 - Fixed `fireforge test <directory>` dispatching xpcshell-only directories to the mochitest runner ("could not find any mochitests under the following test path(s)"): the manifest walk now starts at the directory itself, so a directory whose own manifest is an `xpcshell.toml` dispatches to `mach xpcshell-test` just like an explicit `test_*.js` path.
 - Added `fireforge register --create-manifest`: registering a module under a directory with no `moz.build` now scaffolds the directory manifest (MPL-header + `EXTRA_JS_MODULES.<namespace>` list) and wires the parent `DIRS` chain up to the nearest existing moz.build (creating intermediates); without the flag the "Manifest not found" error names it. Also added an **xpcshell test-file pattern** (`**/test_*.js` outside `browser/base/content/test/`) that inserts the `["test_*.js"]` section into the directory's `xpcshell.toml` in mozbuild sort order — or creates the manifest and wires `XPCSHELL_TESTS_MANIFESTS` with `--create-manifest` — and extended the browser-chrome manifest rule to fork-owned `browser.toml` manifests at **arbitrary depth** under `browser/base/content/test/` (previously one level only).

package/dist/src/core/mach-resource-shim.d.ts

@@ -27,7 +27,14 @@
  *    `cpu_times` / `disk_io_counters` module-wide, so every call site —
  *    including the direct `psutil.virtual_memory()` in
  *    `mozbuild.base._run_make` — degrades to a `UserWarning` and a zeroed
- *    reading instead of aborting;
+ *    reading instead of aborting. Downstream report (0.34.0 cycle):
+ *    mozsystemmonitor subscripts (`_build_meta` does
+ *    `psutil.virtual_memory()[0]`) and iterates/unpacks the reading, so the
+ *    zeroed fallback is a full namedtuple duck type — a zeroed instance of
+ *    psutil's own result namedtuple where resolvable without touching the
+ *    failing syscall, else `_DegradedReading` (subscriptable, iterable, with
+ *    `len()`, `_fields`, `_asdict()`); `cpu_percent` degrades to `0.0`
+ *    because callers do arithmetic on it;
  *  - guards `SystemResourceMonitor` CONSTRUCTION via an import hook:
  *    `poll_interval` is pre-populated before the real `__init__` runs, a
  *    failing `__init__` marks the instance degraded instead of raising,
@@ -35,6 +42,16 @@
  *    partially-constructed monitor can never surface the
  *    `AttributeError: poll_interval` variant.
  */
+/**
+ * Python guard body, importable both as the in-venv `fireforge_mach_guard`
+ * module (loaded by the sibling `.pth` at interpreter startup) and as the
+ * PYTHONPATH `sitecustomize` fallback. Defensive throughout: a missing or
+ * broken psutil/mozsystemmonitor leaves mach untouched, and every wrapper
+ * only intercepts the failure path. Exported for the Python-executing
+ * regression test (mach-resource-shim.python.test.ts), which asserts the
+ * degraded readings' runtime behavior rather than string-matching the source.
+ */
+export declare const GUARD_PYTHON_SOURCE = "# Generated by FireForge - do not edit.\n# Degrades the broken host resource monitor family (psutil vs Darwin) from\n# fatal startup errors into non-fatal warnings, so mach builds and tests\n# proceed. Installed into the mach virtualenv site-packages via a .pth file\n# (survives mach's venv re-exec, which drops PYTHONPATH).\nimport builtins\nimport sys\nimport warnings\n\n\ndef _fireforge_degraded_notice(exc):\n    warnings.warn(\n        \"FireForge: host resource monitor degraded (%s); continuing without resource monitoring.\" % exc\n    )\n\n\nclass _DegradedReading(object):\n    # Duck type of psutil's namedtuple results (svmem field order):\n    # mozsystemmonitor subscripts, iterates, and unpacks readings, so the\n    # degraded fallback must survive r[0], list(r), len(r), r._fields and\n    # r._asdict(), not just attribute access.\n    _fields = (\"total\", \"available\", \"percent\", \"used\", \"free\", \"active\", \"inactive\", \"wired\")\n    total = available = used = free = active = inactive = wired = 0\n    percent = 0.0\n\n    def __getattr__(self, _name):\n        return 0\n\n    def _values(self):\n        return tuple(getattr(self, _field) for _field in self._fields)\n\n    def __getitem__(self, index):\n        return self._values()[index]\n\n    def __iter__(self):\n        return iter(self._values())\n\n    def __len__(self):\n        return len(self._fields)\n\n    def _asdict(self):\n        return dict(zip(self._fields, self._values()))\n\n\n# psutil result namedtuple per wrapped function, resolvable via getattr from\n# the platform module / _common without invoking the failing syscall.\n_PSUTIL_RESULT_CLASSES = {\n    \"virtual_memory\": \"svmem\",\n    \"swap_memory\": \"sswap\",\n    \"cpu_times\": \"scputimes\",\n    \"disk_io_counters\": \"sdiskio\",\n}\n\n\ndef _degraded_result_factory(psutil, name):\n    # cpu_percent returns a plain float callers do arithmetic on.\n    if name == \"cpu_percent\":\n        return lambda: 0.0\n    _cls_name = _PSUTIL_RESULT_CLASSES.get(name)\n    if _cls_name is not None:\n        for _mod in (getattr(psutil, \"_psplatform\", None), getattr(psutil, \"_common\", None)):\n            _cls = getattr(_mod, _cls_name, None) if _mod is not None else None\n            _fields = getattr(_cls, \"_fields\", None)\n            if _cls is not None and _fields:\n                return lambda _cls=_cls, _n=len(_fields): _cls(*([0] * _n))\n    return _DegradedReading\n\n\ndef _guard_psutil():\n    try:\n        import psutil\n    except Exception:\n        return\n\n    def _wrap(orig, fallback):\n        def wrapper(*args, **kwargs):\n            try:\n                return orig(*args, **kwargs)\n            except Exception as exc:  # noqa: BLE001\n                _fireforge_degraded_notice(exc)\n                try:\n                    return fallback()\n                except Exception:  # noqa: BLE001\n                    return _DegradedReading()\n\n        return wrapper\n\n    for _name in (\"virtual_memory\", \"swap_memory\", \"cpu_percent\", \"cpu_times\", \"disk_io_counters\"):\n        _orig = getattr(psutil, _name, None)\n        if _orig is not None and not getattr(_orig, \"_fireforge_guarded\", False):\n            try:\n                _fallback = _degraded_result_factory(psutil, _name)\n            except Exception:  # noqa: BLE001\n                _fallback = _DegradedReading\n            _wrapped = _wrap(_orig, _fallback)\n            _wrapped._fireforge_guarded = True\n            setattr(psutil, _name, _wrapped)\n\n\n_MONITOR_METHOD_NAMES = (\n    \"start\",\n    \"stop\",\n    \"record_event\",\n    \"record_marker\",\n    \"begin_phase\",\n    \"finish_phase\",\n    \"aggregate_cpu_percent\",\n    \"aggregate_cpu_times\",\n    \"aggregate_io\",\n    \"min_memory_available\",\n    \"as_dict\",\n)\n\n\ndef _patch_monitor_class(cls):\n    if cls is None or getattr(cls, \"_fireforge_guarded\", False):\n        return\n    orig_init = cls.__init__\n\n    def guarded_init(self, *args, **kwargs):\n        # Pre-populate the attributes a partially-constructed instance is\n        # later polled for, so a failing __init__ can never surface\n        # AttributeError: poll_interval.\n        self.poll_interval = kwargs.get(\"poll_interval\") or 1.0\n        self._fireforge_degraded = False\n        try:\n            orig_init(self, *args, **kwargs)\n        except Exception as exc:  # noqa: BLE001\n            self._fireforge_degraded = True\n            _fireforge_degraded_notice(exc)\n\n    cls.__init__ = guarded_init\n\n    def _wrap_method(orig):\n        def wrapper(self, *args, **kwargs):\n            if getattr(self, \"_fireforge_degraded\", False):\n                return None\n            try:\n                return orig(self, *args, **kwargs)\n            except Exception as exc:  # noqa: BLE001\n                self._fireforge_degraded = True\n                _fireforge_degraded_notice(exc)\n                return None\n\n        return wrapper\n\n    for _name in _MONITOR_METHOD_NAMES:\n        _orig = getattr(cls, _name, None)\n        if _orig is not None:\n            setattr(cls, _name, _wrap_method(_orig))\n    cls._fireforge_guarded = True\n\n\n_MONITOR_MODULES = (\"mozsystemmonitor.resourcemonitor\", \"mozbuild.resources\")\n\n\ndef _patch_loaded_monitor_modules():\n    for _mod_name in _MONITOR_MODULES:\n        _mod = sys.modules.get(_mod_name)\n        if _mod is not None:\n            try:\n                _patch_monitor_class(getattr(_mod, \"SystemResourceMonitor\", None))\n            except Exception:  # noqa: BLE001\n                pass\n\n\ndef _install_import_hook():\n    _orig_import = builtins.__import__\n    if getattr(_orig_import, \"_fireforge_guarded\", False):\n        return\n\n    def _guarding_import(name, *args, **kwargs):\n        _module = _orig_import(name, *args, **kwargs)\n        try:\n            _patch_loaded_monitor_modules()\n        except Exception:  # noqa: BLE001\n            pass\n        return _module\n\n    _guarding_import._fireforge_guarded = True\n    builtins.__import__ = _guarding_import\n\n\ntry:\n    _guard_psutil()\n    _patch_loaded_monitor_modules()\n    _install_import_hook()\nexcept Exception:  # noqa: BLE001\n    pass\n";
 /** Result of installing the mach resource guard before a dispatch. */
 export interface MachResourceGuardInstallation {
     /**

package/dist/src/core/mach-resource-shim.js

@@ -28,7 +28,14 @@
  *    `cpu_times` / `disk_io_counters` module-wide, so every call site —
  *    including the direct `psutil.virtual_memory()` in
  *    `mozbuild.base._run_make` — degrades to a `UserWarning` and a zeroed
- *    reading instead of aborting;
+ *    reading instead of aborting. Downstream report (0.34.0 cycle):
+ *    mozsystemmonitor subscripts (`_build_meta` does
+ *    `psutil.virtual_memory()[0]`) and iterates/unpacks the reading, so the
+ *    zeroed fallback is a full namedtuple duck type — a zeroed instance of
+ *    psutil's own result namedtuple where resolvable without touching the
+ *    failing syscall, else `_DegradedReading` (subscriptable, iterable, with
+ *    `len()`, `_fields`, `_asdict()`); `cpu_percent` degrades to `0.0`
+ *    because callers do arithmetic on it;
  *  - guards `SystemResourceMonitor` CONSTRUCTION via an import hook:
  *    `poll_interval` is pre-populated before the real `__init__` runs, a
  *    failing `__init__` marks the instance degraded instead of raising,
@@ -51,9 +58,11 @@ const GUARD_MODULE_NAME = 'fireforge_mach_guard';
  * module (loaded by the sibling `.pth` at interpreter startup) and as the
  * PYTHONPATH `sitecustomize` fallback. Defensive throughout: a missing or
  * broken psutil/mozsystemmonitor leaves mach untouched, and every wrapper
- * only intercepts the failure path.
+ * only intercepts the failure path. Exported for the Python-executing
+ * regression test (mach-resource-shim.python.test.ts), which asserts the
+ * degraded readings' runtime behavior rather than string-matching the source.
  */
-const GUARD_PYTHON_SOURCE = `# Generated by FireForge - do not edit.
+export const GUARD_PYTHON_SOURCE = `# Generated by FireForge - do not edit.
 # Degrades the broken host resource monitor family (psutil vs Darwin) from
 # fatal startup errors into non-fatal warnings, so mach builds and tests
 # proceed. Installed into the mach virtualenv site-packages via a .pth file
@@ -70,12 +79,56 @@ def _fireforge_degraded_notice(exc):
 
 
 class _DegradedReading(object):
+    # Duck type of psutil's namedtuple results (svmem field order):
+    # mozsystemmonitor subscripts, iterates, and unpacks readings, so the
+    # degraded fallback must survive r[0], list(r), len(r), r._fields and
+    # r._asdict(), not just attribute access.
+    _fields = ("total", "available", "percent", "used", "free", "active", "inactive", "wired")
     total = available = used = free = active = inactive = wired = 0
     percent = 0.0
 
     def __getattr__(self, _name):
         return 0
 
+    def _values(self):
+        return tuple(getattr(self, _field) for _field in self._fields)
+
+    def __getitem__(self, index):
+        return self._values()[index]
+
+    def __iter__(self):
+        return iter(self._values())
+
+    def __len__(self):
+        return len(self._fields)
+
+    def _asdict(self):
+        return dict(zip(self._fields, self._values()))
+
+
+# psutil result namedtuple per wrapped function, resolvable via getattr from
+# the platform module / _common without invoking the failing syscall.
+_PSUTIL_RESULT_CLASSES = {
+    "virtual_memory": "svmem",
+    "swap_memory": "sswap",
+    "cpu_times": "scputimes",
+    "disk_io_counters": "sdiskio",
+}
+
+
+def _degraded_result_factory(psutil, name):
+    # cpu_percent returns a plain float callers do arithmetic on.
+    if name == "cpu_percent":
+        return lambda: 0.0
+    _cls_name = _PSUTIL_RESULT_CLASSES.get(name)
+    if _cls_name is not None:
+        for _mod in (getattr(psutil, "_psplatform", None), getattr(psutil, "_common", None)):
+            _cls = getattr(_mod, _cls_name, None) if _mod is not None else None
+            _fields = getattr(_cls, "_fields", None)
+            if _cls is not None and _fields:
+                return lambda _cls=_cls, _n=len(_fields): _cls(*([0] * _n))
+    return _DegradedReading
+
 
 def _guard_psutil():
     try:
@@ -83,20 +136,27 @@ def _guard_psutil():
     except Exception:
         return
 
-    def _wrap(orig):
+    def _wrap(orig, fallback):
         def wrapper(*args, **kwargs):
             try:
                 return orig(*args, **kwargs)
             except Exception as exc:  # noqa: BLE001
                 _fireforge_degraded_notice(exc)
-                return _DegradedReading()
+                try:
+                    return fallback()
+                except Exception:  # noqa: BLE001
+                    return _DegradedReading()
 
         return wrapper
 
     for _name in ("virtual_memory", "swap_memory", "cpu_percent", "cpu_times", "disk_io_counters"):
         _orig = getattr(psutil, _name, None)
         if _orig is not None and not getattr(_orig, "_fireforge_guarded", False):
-            _wrapped = _wrap(_orig)
+            try:
+                _fallback = _degraded_result_factory(psutil, _name)
+            except Exception:  # noqa: BLE001
+                _fallback = _DegradedReading
+            _wrapped = _wrap(_orig, _fallback)
             _wrapped._fireforge_guarded = True
             setattr(psutil, _name, _wrapped)
 

package/dist/src/core/test-harness-crash.js

@@ -56,6 +56,17 @@ const NO_OUTPUT_TIMEOUT_PATTERN = /timed out after \d+ seconds with no output/i;
  * on macOS. Each is matched per-line so the evidence line in the report is
  * the concrete failure, not the whole traceback.
  */
+// Downstream report (0.34.0 cycle): a pre-fix _DegradedReading fallback that
+// wasn't a real namedtuple duck type crashed mozsystemmonitor on the fallback
+// itself (`_build_meta` subscripts the reading; `_collect` unpacks it); a
+// startup abort with zero TEST-START lines from this family is a crash, not
+// a test failure. The `_collect failed` variant is caught-and-logged, so it
+// carries no Traceback header — it gets its own zero-TEST-START check in
+// `detectHarnessCrashSignature` besides joining the traceback cluster.
+const DEGRADED_READING_CRASH_SIGNALS = [
+    /'_DegradedReading' object is not subscriptable/,
+    /'_DegradedReading' object is not iterable/,
+];
 const STARTUP_TRACEBACK_SIGNALS = [
     /AttributeError:.*SystemResourceMonitor/,
     /'SystemResourceMonitor' object has no attribute/,
@@ -64,6 +75,7 @@ const STARTUP_TRACEBACK_SIGNALS = [
     /HOST_VM_INFO64/,
     /\(ipc\/mig\) array not large enough/,
     /psutil\.[A-Za-z]*Error/,
+    ...DEGRADED_READING_CRASH_SIGNALS,
 ];
 function findLine(output, patterns) {
     for (const line of output.split(/\r?\n/)) {
@@ -83,7 +95,11 @@ function findLine(output, patterns) {
 const NOISE_LINE_PATTERNS = [
     /\bUserWarning\b/,
     /psutil failed to run/i,
-    /_collect failed/i,
+    // `_collect failed` chatter is noise on green runs, EXCEPT the
+    // `_DegradedReading` variant: that is the crash evidence for the
+    // "object is not iterable" startup-abort signature above and must
+    // survive stripNonSignalNoise.
+    /_collect failed(?!.*_DegradedReading)/i,
     /FireForge: host resource monitor degraded/i,
     /warnings\.warn\(/,
 ];
@@ -197,6 +213,18 @@ export function detectHarnessCrashSignature(output) {
     // the no-output timeout. A trailing "Passed: 0" summary is part of this
     // shape and must not be read as a result.
     if (!hasTestStart) {
+        // Startup abort on the degraded-reading fallback family: the `_collect
+        // failed: '_DegradedReading' ...` variant is caught-and-logged (no
+        // Traceback header), so the traceback cluster above never sees it.
+        if (realFailures.length === 0) {
+            const degradedLine = findLine(evidence, DEGRADED_READING_CRASH_SIGNALS);
+            if (degradedLine) {
+                return {
+                    reason: 'degraded resource reading crashed the harness fallback (_DegradedReading)',
+                    line: degradedLine,
+                };
+            }
+        }
         const timeoutLine = findLine(evidence, [NO_OUTPUT_TIMEOUT_PATTERN]);
         if (timeoutLine) {
             return { reason: 'no-output timeout before any test started', line: timeoutLine };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hominis/fireforge",
-  "version": "0.34.0",
+  "version": "0.34.1",
   "description": "FireForge — a build tool for customizing Firefox",
   "type": "module",
   "main": "./dist/src/index.js",