@ai-ecoverse/slicc-shared-web 1.9.1 → 1.10.0

package/README.md

@@ -5,12 +5,12 @@ Configuration shared by SLICC's web repos ([slicc-bios](https://github.com/ai-ec
 | File | What it is | How a repo uses it |
 |---|---|---|
 | `default.json` | Root Renovate preset: automerge on, one dependency per PR, exact pins, 7-day release age (14 for majors, 0 for vulnerability fixes and for our own `@ai-ecoverse` packages, which also jump the PR queue), and a manager for jsDelivr pins inside `src/` | `renovate.json`: `{ "extends": ["github>ai-ecoverse/slicc-shared-web"] }` |
-| `.github/workflows/node-ci.yml` | Reusable CI: `npm ci`, lint, Chromium's system libraries, integration tests, hotspots in the job summary, coverage/profiles/screenshots as artifacts. The browser itself comes from each repo's `pretest` | `jobs: { ci: { uses: ai-ecoverse/slicc-shared-web/.github/workflows/node-ci.yml@vX.Y.Z } }` |
+| `.github/workflows/node-ci.yml` | Reusable CI: `npm ci`, lint, Chromium's system libraries, integration tests, hotspots and the stack of every crash dump in the job summary, coverage/profiles/screenshots/crash dumps as artifacts. The browser itself comes from each repo's `pretest` | `jobs: { ci: { uses: ai-ecoverse/slicc-shared-web/.github/workflows/node-ci.yml@vX.Y.Z } }` |
 | `biome.json` | Formatter settings and slicc's strict linter: cognitive complexity ≤ 25, functions ≤ 150 lines, no unused variables or imports, no floating or misused promises, naming conventions, `import type`/`export type`. Test files under `test/` get slicc's test exemptions | `biome.json`: `{ "extends": ["@ai-ecoverse/slicc-shared-web/biome"] }` |
 | `tsconfig.json` | TypeScript base for Node's type stripping | `tsconfig.json`: `{ "extends": "@ai-ecoverse/slicc-shared-web/tsconfig.json", "include": ["src/**/*.ts"] }` |
 | `lefthook.yml` | Pre-commit hooks: Biome, typecheck, no comments, no unit tests in git, AGENTS.md, diff coverage | `lefthook.yml`: `extends: [node_modules/@ai-ecoverse/slicc-shared-web/lefthook.yml]` |
 | `mcr.config.mjs` | Unit-test coverage for JS and TS sources in `src/` (TS types are stripped first), written to `coverage/unit/lcov.info` | `test:unit`: `mcr -c node_modules/@ai-ecoverse/slicc-shared-web/mcr.config.mjs node --test 'test/unit/**/*.test.mjs'` |
-| `bin/` | `slicc-lint` runs every check below plus Biome and, if the repo has one, `typecheck`. `slicc-lint-comments` runs the [slicc no-comment checker](https://github.com/ai-ecoverse/slicc/tree/main/packages/dev-tools/no-comment) at a pinned commit, allowing AGENTS.md. `slicc-agents-md`, `slicc-no-unit-tests` and `slicc-diff-cover` complete the set | `lint`: `slicc-lint`, and the hooks above |
+| `bin/` | `slicc-lint` runs every check below plus Biome and, if the repo has one, `typecheck`. `slicc-lint-comments` runs the [slicc no-comment checker](https://github.com/ai-ecoverse/slicc/tree/main/packages/dev-tools/no-comment) at a pinned commit, allowing AGENTS.md. `slicc-agents-md`, `slicc-no-unit-tests` and `slicc-diff-cover` complete the set. `slicc-minidump <file.dmp>...` prints a crash dump's stack, described [below](#crash-dumps) | `lint`: `slicc-lint`, and the hooks above |
 | `pnpmfile.mjs` | pnpm's counterpart to the Renovate preset: an `updateConfig` hook that adds `@ai-ecoverse/*` to `minimumReleaseAgeExclude`, so pnpm 12 doesn't hold back our own fresh releases (`ERR_PNPM_MINIMUM_RELEASE_AGE_VIOLATION`) while third-party packages keep its age check | in a repo with a pnpm lockfile, as a [config dependency](https://pnpm.io/config-dependencies) in `pnpm-workspace.yaml`: `configDependencies: { '@ai-ecoverse/slicc-shared-web': X.Y.Z }` plus `pnpmfile: node_modules/.pnpm-config/@ai-ecoverse/slicc-shared-web/pnpmfile.mjs` |
 | `harness/` | The CDP integration-test harness, described [below](#integration-test-harness) | `import { launch } from '@ai-ecoverse/slicc-shared-web/harness'`, and `--test-global-setup=node_modules/@ai-ecoverse/slicc-shared-web/harness/global.mjs` |
 
@@ -70,6 +70,7 @@ test('boots', async (t) => {
 - when a renderer crashes, every pending and later call to that target, and to the dedicated workers it started, rejects with `<method>: page renderer crashed (<status>, code <n>) at <url>` (or `worker`, `shared worker`, `service worker`), and `console.log` gets the same line;
 - `until`, `within`, `goto` and `reload` stop retrying and rethrow it;
 - both errors end with the last five lines of the browser's stderr, and each test keeps up to 200 lines it wrote during the test in `chrome-stderr.log`.
+- a test that saw a renderer crash, or a browser that died of anything but `SIGKILL`, keeps Chrome's minidump of it as `<id>.dmp` in its artifact directory (see [Crash dumps](#crash-dumps)).
 
  When one evaluation runs past `stallAfter`, the harness probes once, without interrupting the wait:
 - it adds a `pending` line to `console.log` for every navigation and request of the page that has no answer yet, with its age. A worker's script loads in the worker's own session, so its request is left out. Chrome holds CDP messages to a page while a navigation is waiting for its response, so an evaluation that hangs while the page is idle usually means one of these;
@@ -94,3 +95,18 @@ Why it works the way it does:
 - The attach event can arrive before `Target.createTarget` returns, so new tabs wait on a slot keyed by target id.
 - A worker blocked in `Atomics.wait`, or one that is already gone, may never answer; snapshots give up after 3 s, and closing a context or the browser after 10 s. A call to a target that detaches or crashes fails instead of waiting forever, and a new tab that never attaches fails after 10 s.
 - A WebSocket that has closed takes new messages without an error, and a crashed target never answers, so after a crash every call used to wait out its own deadline: a browser that died 4 s in showed up as `no answer` 15 minutes later. The harness now remembers the crash or the exit and fails each call with it.
+
+## Crash dumps
+
+Chrome for Testing runs crashpad by default and ignores `--crash-dumps-dir` and `--enable-crash-reporter`: on Linux it writes every dump to `~/.config/google-chrome-for-testing/Crash Reports/pending/`. `launch` points it at a fresh directory per browser through `BREAKPAD_DUMP_LOCATION`, waits up to 10 s after a crash for crashpad to finish writing, copies the new dumps into the test's artifact directory and deletes the directory on `close()`. Nothing is uploaded.
+
+Google publishes no symbols for Chrome for Testing: no Breakpad files next to the `chrome-linux64.zip` downloads, and the binary is stripped (only `.eh_frame` and a `.gnu_debuglink` to a file that isn't public). Chrome keeps frame pointers, though, so the stack walks cleanly as `module + offset`, and the dump carries crashpad's annotations: `ver`, `ptype` (`renderer`, `gpu-process`, `browser` and so on), the GPU (`gpu-gl-renderer` shows SwiftShader in CI), the page's origin, and V8 crash keys when V8 sets them.
+
+`slicc-minidump <file.dmp>...` downloads a pinned, checksummed [`minidump-stackwalk`](https://github.com/rust-minidump/rust-minidump) 0.27.0 into `~/.cache/slicc-minidump/` (Linux x64 and macOS) and prints, per dump, the signal and address, the faulting module and offset, the annotations without the command-line switches, the crashing thread's first 40 frames and the module list with build ids. The reusable workflow runs it on every `artifacts/**/*.dmp`, writes the full report next to the dump as `<id>.txt` and puts everything above the module list into the job summary. To read a dump from a CI run locally:
+
+```sh
+gh run download <run-id> -n coverage-profiles-screenshots -D ci
+npx slicc-minidump ci/artifacts/<suite>/<test>/*.dmp
+```
+
+A Linux dump reads fine on a Mac. Offsets are only comparable across dumps from the same `ver`. A frame in `chrome` with the `ud2` instruction and `SIGTRAP` is a deliberate `CHECK` or `IMMEDIATE_CRASH`; `SIGSEGV` with a wild address points at memory corruption, and a top frame in `libvk_swiftshader.so`, `libGLESv2.so` or `libEGL.so` at the software GPU.

package/bin/slicc-minidump.mjs

@@ -0,0 +1,125 @@
+#!/usr/bin/env node
+import { execFile } from 'node:child_process';
+import { createHash } from 'node:crypto';
+import { access, mkdir, writeFile } from 'node:fs/promises';
+import { homedir } from 'node:os';
+import { join } from 'node:path';
+import process, { arch, argv, env, platform, stderr, stdout } from 'node:process';
+import { promisify } from 'node:util';
+
+const run = promisify(execFile);
+const VERSION = '0.27.0';
+const FRAMES = 40;
+const builds = {
+  'linux-x64': [
+    'x86_64-unknown-linux-gnu',
+    '0020324c54cc359596e927ee907204f4c8d4da6718765536edcabaa1622122ad',
+  ],
+  'linux-x64-musl': [
+    'x86_64-unknown-linux-musl',
+    'bfaf97804965c87155b91765f4042440a38bb002f20a62fc4da9b6b6bcb20faf',
+  ],
+  'darwin-arm64': [
+    'aarch64-apple-darwin',
+    '5fcd4d25a5b0bbdc8faac6f4f81629bc67483c3eb458b86a8304733a7c6a2ca9',
+  ],
+  'darwin-x64': [
+    'x86_64-apple-darwin',
+    '801fa500d159fe474260ef1aad80706bed822434aced7fdb21c1b2c7a06afe6f',
+  ],
+};
+
+function host() {
+  const musl = platform === 'linux' && !process.report.getReport().header.glibcVersionRuntime;
+  return `${platform}-${arch}${musl ? '-musl' : ''}`;
+}
+
+async function stackwalk() {
+  const build = builds[host()];
+  if (!build) throw new Error(`no minidump-stackwalk ${VERSION} build for ${host()}`);
+  const [target, sha256] = build;
+  const dir = join(env.XDG_CACHE_HOME ?? join(homedir(), '.cache'), 'slicc-minidump', VERSION);
+  const binary = join(dir, `minidump-stackwalk-${target}`, 'minidump-stackwalk');
+  const cached = await access(binary).then(
+    () => true,
+    () => false
+  );
+  if (cached) return binary;
+  const name = `minidump-stackwalk-${target}.tar.xz`;
+  const url = `https://github.com/rust-minidump/rust-minidump/releases/download/v${VERSION}/${name}`;
+  const response = await fetch(url);
+  if (!response.ok) throw new Error(`${url}: HTTP ${response.status}`);
+  const archive = Buffer.from(await response.arrayBuffer());
+  const digest = createHash('sha256').update(archive).digest('hex');
+  if (digest !== sha256) throw new Error(`${url}: sha256 ${digest}, expected ${sha256}`);
+  await mkdir(dir, { recursive: true });
+  await writeFile(join(dir, name), archive);
+  await run('tar', ['-xJf', join(dir, name), '-C', dir]);
+  return binary;
+}
+
+function annotations(dump) {
+  const pattern =
+    /^\s*(?:simple_annotations|module_list\[\d+\]\.annotation_objects)\["([^"]+)"\] = (.*)$/;
+  return dump
+    .split('\n')
+    .map((line) => line.match(pattern))
+    .filter((match) => match && !/^(switch-|variations$)/.test(match[1]))
+    .map(([, key, value]) => `  ${key} = ${value}`);
+}
+
+function frame({ frame: n, module, module_offset, offset, function: fn, trust }) {
+  const where = module ? `${module} + ${module_offset}` : offset;
+  const named = fn ? ` ${fn}` : '';
+  return `${String(n).padStart(4)}  ${where}${named}  (${trust})`;
+}
+
+function report(walked, dump) {
+  const { crash_info: crash, crashing_thread: thread, modules, system_info: system } = walked;
+  const top = thread?.frames?.[0];
+  return [
+    `crash: ${crash?.type ?? 'none'} at ${crash?.address ?? '?'}`,
+    `faulting: ${top ? frame(top).trim() : 'no crashing thread'}`,
+    `instruction: ${crash?.instruction ?? '?'}`,
+    `system: ${system?.os} ${system?.os_ver} ${system?.cpu_arch}, ${walked.process_uptime ?? '?'} s up`,
+    'annotations:',
+    ...annotations(dump),
+    `thread ${crash?.crashing_thread} ${thread?.thread_name ?? ''} (${thread?.frame_count ?? 0} frames):`,
+    ...(thread?.frames ?? []).slice(0, FRAMES).map(frame),
+    'modules:',
+    ...(modules ?? []).map(
+      ({ base_addr, end_addr, filename, code_id }) =>
+        `  ${base_addr}-${end_addr}  ${filename}  ${code_id ?? ''}`
+    ),
+    '',
+  ].join('\n');
+}
+
+async function describe(binary, file) {
+  const options = { maxBuffer: 256 * 1024 * 1024 };
+  const [json, dump] = await Promise.all([
+    run(binary, ['--json', file], options),
+    run(binary, ['--dump', file], options),
+  ]);
+  return report(JSON.parse(json.stdout), dump.stdout);
+}
+
+async function main(files) {
+  if (files.length === 0) {
+    stderr.write('usage: slicc-minidump <file.dmp>...\n');
+    return 2;
+  }
+  const binary = await stackwalk();
+  for (const file of files) stdout.write(`== ${file}\n${await describe(binary, file)}\n`);
+  return 0;
+}
+
+main(argv.slice(2)).then(
+  (code) => {
+    process.exitCode = code;
+  },
+  (error) => {
+    stderr.write(`slicc-minidump: ${error.message}\n`);
+    process.exitCode = 1;
+  }
+);

package/harness/chrome.mjs

@@ -2,9 +2,11 @@ import { spawn } from 'node:child_process';
 import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
 import { tmpdir } from 'node:os';
 import { basename, join } from 'node:path';
+import { env as inherited } from 'node:process';
 import { chromium } from 'playwright-core';
 import { cdn } from './cdn.mjs';
-import { connect } from './cdp.mjs';
+import { CRASHED, connect } from './cdp.mjs';
+import { crashpad } from './dumps.mjs';
 import { extensionSource, unpacked } from './extensions.mjs';
 import { page } from './page.mjs';
 import { breakpoints, recorder } from './recorder.mjs';
@@ -66,9 +68,10 @@ export function exitOf(code, signal) {
 export async function start(
   profile,
   args = commandLine(profile),
-  executable = chromium.executablePath()
+  executable = chromium.executablePath(),
+  env = inherited
 ) {
-  const child = spawn(executable, args, { stdio: ['ignore', 'ignore', 'pipe'] });
+  const child = spawn(executable, args, { env, stdio: ['ignore', 'ignore', 'pipe'] });
   const stderr = lines(child.stderr);
   const drained = new Promise((resolve) => child.stderr.once('close', resolve));
   const exited = new Promise((resolve) => {
@@ -91,6 +94,11 @@ export async function start(
   return { child, url, exited, stderr };
 }
 
+export function died(child) {
+  if (child.signalCode) return child.signalCode !== 'SIGKILL';
+  return (child.exitCode ?? 0) !== 0;
+}
+
 export async function stop(cdp, child) {
   const exited = new Promise((resolve) => child.once('exit', resolve));
   await bounded(cdp.send('Browser.close').catch(ignore), 'Browser.close');
@@ -121,11 +129,23 @@ export async function launch({
   const shared = extensions.length > 0;
   const profile = await mkdtemp(join(tmpdir(), 'slicc-harness-'));
   const loaded = [...found.values()].map((root) => root.slice(0, -1));
-  const { child, url, exited, stderr } = await start(profile, commandLine(profile, loaded, args));
+  const dumps = await crashpad();
+  const { child, url, exited, stderr } = await start(
+    profile,
+    commandLine(profile, loaded, args),
+    chromium.executablePath(),
+    dumps.env
+  );
   const cdp = await connect(url, {
     exited,
     tail: () => stderr.tail().map((line) => `stderr: ${line}`),
   });
+  let crashes = 0;
+  let open = 0;
+  let closed = false;
+  cdp.on(({ method }) => {
+    if (method === CRASHED || method === 'Target.targetCrashed') crashes += 1;
+  });
   const record = recorder(cdp, files, { coverage, exits: await breakpoints(server, exits) });
   const remote = await cdn(cdp, intercept);
   await cdp.send('Target.setDiscoverTargets', { discover: true });
@@ -146,9 +166,12 @@ export async function launch({
     await Promise.all(closing.map((sent) => sent.catch(ignore)));
   }
 
-  async function finish(pages, browserContextId, dir, before, mark) {
+  async function finish(pages, browserContextId, dir, before, mark, seen) {
     const log = stderr.since(mark).map((line) => `${line}\n`);
     await writeFile(new URL('chrome-stderr.log', dir), log.join(''));
+    if (crashes > seen || died(child)) await dumps.keep(dir).catch(ignore);
+    open -= 1;
+    if (closed && open === 0) await dumps.remove();
     await bounded(Promise.all(pages.map((opened) => opened.probed())), 'probes', 30000);
     const shots = pages.map((opened, i) =>
       opened.screenshot(new URL(`tab-${i + 1}.png`, dir)).catch(ignore)
@@ -181,7 +204,9 @@ export async function launch({
       server.overridden.clear();
       const pages = [];
       const mark = stderr.total;
-      t.after(() => finish(pages, browserContextId, dir, before, mark));
+      const seen = crashes;
+      open += 1;
+      t.after(() => finish(pages, browserContextId, dir, before, mark, seen));
       const tab = async () => {
         const opened = page(cdp, await record.open(browserContextId), server, timeout, {
           stallAfter,
@@ -198,6 +223,8 @@ export async function launch({
     async close() {
       await stop(cdp, child);
       await rm(profile, { recursive: true, force: true });
+      closed = true;
+      if (open === 0) await dumps.remove();
       await server.close();
     },
   };

package/harness/dumps.mjs

@@ -0,0 +1,46 @@
+import { copyFile, mkdtemp, readdir, rm } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { basename, join } from 'node:path';
+import { env } from 'node:process';
+import { sleep } from './util.mjs';
+
+export const DUMP_WAIT = 10000;
+
+async function list(dir, sub) {
+  return (await readdir(join(dir, sub)).catch(() => [])).map((file) => join(sub, file));
+}
+
+export async function crashpad() {
+  const dir = await mkdtemp(join(tmpdir(), 'slicc-crashes-'));
+  const kept = new Set();
+  const scan = async () => {
+    const [pending, completed, writing] = await Promise.all(
+      ['pending', 'completed', 'new'].map((sub) => list(dir, sub))
+    );
+    const fresh = [...pending, ...completed].filter(
+      (file) => file.endsWith('.dmp') && !kept.has(file)
+    );
+    return { fresh, writing: writing.some((file) => file.endsWith('.dmp')) };
+  };
+  return {
+    dir,
+    env: { ...env, BREAKPAD_DUMP_LOCATION: dir },
+    async keep(target, ms = DUMP_WAIT) {
+      const deadline = Date.now() + ms;
+      let found = await scan();
+      while ((found.fresh.length === 0 || found.writing) && Date.now() < deadline) {
+        await sleep(250);
+        found = await scan();
+      }
+      const names = [];
+      for (const file of found.fresh) {
+        kept.add(file);
+        const name = basename(file);
+        await copyFile(join(dir, file), new URL(name, target));
+        names.push(name);
+      }
+      return names;
+    },
+    remove: () => rm(dir, { recursive: true, force: true }),
+  };
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ai-ecoverse/slicc-shared-web",
-  "version": "1.9.1",
+  "version": "1.10.0",
   "description": "Shared Renovate, Biome, TypeScript, lefthook and GitHub Actions configuration, plus the CDP integration-test harness, for SLICC's web repos",
   "type": "module",
   "license": "Apache-2.0",
@@ -36,6 +36,7 @@
     "slicc-diff-cover": "bin/slicc-diff-cover.sh",
     "slicc-lint": "bin/slicc-lint.sh",
     "slicc-lint-comments": "bin/slicc-lint-comments.sh",
+    "slicc-minidump": "bin/slicc-minidump.mjs",
     "slicc-no-unit-tests": "bin/slicc-no-unit-tests.sh"
   },
   "files": [