testaro 75.2.2 → 76.0.0

package/.dockerignore

@@ -0,0 +1,12 @@
+.git
+node_modules
+temp
+tmp
+watch
+.env
+.npmrc
+.DS_Store
+.vscode
+validation/tests/old
+Dockerfile
+docker-compose.yml

package/AGENTS.md

@@ -25,6 +25,11 @@
 - **Error handling**: Try-catch blocks, report failures via `prevented: true` in results
 - **Comments**: Explain complex logic, but keep concise
 
+## Assistant Mode Behavior
+
+- The user deliberately selects Code, Ask, or Plan mode for each session; do not suggest or invite switching modes (e.g., “switch to Code mode so I can…”).
+- In Ask mode, propose exact changes/commands for the user to apply without prompting a mode change.
+
 ## License
 
 © 2025–2026 Jonathan Robert Pool.

package/CONTAINERS.md

@@ -0,0 +1,88 @@
+# Containerized deployment
+
+## Introduction
+
+This document describes how to deploy Testaro in a container, using the reference `Dockerfile` and `docker-compose.yml` at the project root. It accompanies the evaluation requested in [issue 32](https://github.com/jrpool/testaro/issues/32).
+
+Containerization is practical. The reference image runs every Testaro tool:
+
+- The image is based on the official Playwright image (`mcr.microsoft.com/playwright`), which contains the Chromium, Firefox, and WebKit browsers and their system libraries, plus standard fonts, which keep page rendering consistent across hosts.
+- QualWeb’s own (Puppeteer) Chromium is downloaded at image build time, pinned by `package-lock.json`.
+- A headless Java runtime is installed for the `nuVnu` test, so `vnu-jar` does not download one over the network on first use.
+- The `nuVal` test is pure HTTP; the reference Compose stack pairs Testaro with a self-hosted Nu Html Checker sidecar (`ghcr.io/validator/validator`) via the `TESTARO_NU_URL` environment variable, so it does not depend on the public W3C service (which is rate-limited and rejects large documents).
+- The `wave` test calls an external API and needs only a `WAVE_KEY`.
+
+## Version coupling
+
+The tag of the base image in the `Dockerfile` (`mcr.microsoft.com/playwright:v1.60.0-noble`) must match the version of the `playwright` package in `package-lock.json`, so that the browsers baked into the image are the ones the installed Playwright expects. When upgrading Playwright, change both together.
+
+## The Chromium sandbox
+
+Chromium’s sandbox requires unprivileged user-namespace cloning, which the default Docker seccomp policy blocks (and which some hardened hosts prohibit even outside containers). There are two ways to run:
+
+1. **Without the sandbox** (the image default): the `TESTARO_CHROMIUM_NO_SANDBOX=true` environment variable makes Testaro launch Playwright’s Chromium with `chromiumSandbox: false` and QualWeb’s Chromium with `--no-sandbox`. This is the usual choice when the container itself is the isolation boundary and only untrusted web content is being visited.
+2. **With the sandbox**: set `TESTARO_CHROMIUM_NO_SANDBOX=false` and run the container with a seccomp profile that permits user-namespace cloning, such as [the profile published by Playwright](https://github.com/microsoft/playwright/blob/main/utils/docker/seccomp_profile.json):
+
+    ```bash
+    docker run --security-opt seccomp=seccomp_profile.json …
+    ```
+
+WebKit and Firefox are unaffected either way.
+
+## Quick start with Compose
+
+```bash
+mkdir -p watch/jobs/todo watch/jobs/done watch/reports/raw
+chmod -R a+w watch
+docker compose up --build
+```
+
+This builds the image, starts the Nu Html Checker sidecar, and starts Testaro watching `watch/jobs/todo` (checking every 300 seconds, per the default command `node call dirWatch true 300`). Drop a job file into `watch/jobs/todo` on the host; the report appears in `watch/reports/raw`, and the job file moves to `watch/jobs/done`.
+
+The container runs as the unprivileged `pwuser` from the Playwright base image, so the mounted host directories must be writable by that user (the `chmod` above, or `chown` to the matching UID).
+
+## One-shot job
+
+To perform a single job and exit:
+
+```bash
+docker compose run --rm testaro node call run
+```
+
+Or without Compose (no Nu sidecar; `nuVal` then uses the public W3C service):
+
+```bash
+docker build -t testaro .
+docker run --rm --init \
+  -v "$PWD/watch/jobs:/home/pwuser/testaro/watch/jobs" \
+  -v "$PWD/watch/reports:/home/pwuser/testaro/watch/reports" \
+  testaro node call run
+```
+
+## Server polling
+
+To poll a server for jobs instead of watching a directory, override the command and provide the `netWatch` variables:
+
+```bash
+docker run --rm --init \
+  -e AGENT=agent1 \
+  -e NETWATCH_URL_JOB=https://example.com/api/testaro-agent/job \
+  -e NETWATCH_URL_REPORT=https://example.com/api/testaro-agent/report \
+  -e NETWATCH_URL_AUTH=password \
+  testaro node call netWatch true 300 true
+```
+
+## Operational notes
+
+- **Init process**: Testaro forks a child process per test act, and some tools relaunch browsers repeatedly, so zombie reaping matters more than for typical browser-automation applications. Always run with an init process (`docker run --init`; in Compose, `init: true`).
+- **Memory**: allow headroom for browser relaunch spikes; multiple tools’ browsers can briefly coexist with the catalog pass. The reference Compose stack allows 4 GB.
+- **`/dev/shm`**: Testaro launches Chromium with `--disable-dev-shm-usage`, so the small default `/dev/shm` of containers is not a problem and no `--shm-size` option is needed.
+- **Environment variables**: all variables in `env.example` work as usual; pass them with `-e`/`--env-file` or the Compose `environment` key rather than baking an `.env` file into the image.
+
+## License
+
+© 2026 Jeff Witt.
+
+Licensed under the [MIT License](https://opensource.org/license/mit/). See [LICENSE](LICENSE) file at the project root for details.
+
+SPDX-License-Identifier: MIT

package/Dockerfile

@@ -0,0 +1,53 @@
+# © 2026 Jeff Witt.
+#
+# Licensed under the MIT License. See LICENSE file at the project root or
+# https://opensource.org/license/mit/ for details.
+#
+# SPDX-License-Identifier: MIT
+
+# Reference container image for Testaro. See CONTAINERS.md for usage.
+#
+# The base-image tag must match the version of the playwright package in
+# package-lock.json, so that the browsers baked into the image are the ones
+# the installed Playwright expects.
+FROM mcr.microsoft.com/playwright:v1.60.0-noble
+
+# Install a headless Java runtime for the nuVnu test (vnu-jar); baking it in
+# prevents vnu-jar from downloading a Temurin runtime over the network on
+# first use. Also install unzip, which Puppeteer's browser downloader needs
+# during `npm ci` and which the base image lacks.
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends openjdk-17-jre-headless unzip \
+  && rm -rf /var/lib/apt/lists/*
+
+# Run as the unprivileged user provided by the Playwright base image.
+USER pwuser
+WORKDIR /home/pwuser/testaro
+
+# Install the dependencies first, for build-layer caching. QualWeb's
+# Puppeteer downloads its own Chromium during this step (into
+# ~/.cache/puppeteer), so that browser is also baked into the image, pinned
+# by package-lock.json. Playwright's browsers are already in the base image.
+COPY --chown=pwuser:pwuser package.json package-lock.json ./
+RUN npm ci
+
+# Copy the application.
+COPY --chown=pwuser:pwuser . .
+
+# Directories for jobs and reports when watching a directory; mount volumes
+# here (see CONTAINERS.md and docker-compose.yml).
+ENV JOBDIR=/home/pwuser/testaro/watch/jobs \
+    REPORTDIR=/home/pwuser/testaro/watch/reports
+RUN mkdir -p watch/jobs/todo watch/jobs/done watch/reports/raw
+
+# Chromium's sandbox requires unprivileged user-namespace cloning, which the
+# default Docker seccomp policy blocks. This default disables the sandbox
+# (for Playwright's Chromium and QualWeb's). To keep the sandbox instead,
+# run the container with a seccomp profile that permits user-namespace
+# cloning and set this variable to false. See CONTAINERS.md.
+ENV TESTARO_CHROMIUM_NO_SANDBOX=true
+
+# Testaro forks a child process per test act, and some tools relaunch
+# browsers repeatedly, so zombie reaping matters: run the container with an
+# init process (`docker run --init`; in Compose, `init: true`).
+CMD ["node", "call", "dirWatch", "true", "300"]

package/README.md

@@ -575,7 +575,7 @@ The behavior of Testaro as a dependency of an application deployed on a virtual
 
 ### Containerized deployment
 
-Experimental deployments of Testaro as a dependency in a containerized application has sometimes resulted in thrown errors that are not thrown when the same application is deployed without containerization.
+A reference container image for stand-alone deployment, in which all tools run, is defined by the `Dockerfile` and `docker-compose.yml` files at the project root and documented in [CONTAINERS.md](CONTAINERS.md).
 
 ### Headless browser fidelity
 

package/call.js

@@ -53,13 +53,14 @@ const callRun = async jobIDStart => {
     // Get it.
     const jobJSON = await fs.readFile(`${todoDir}/${jobFileName}`, 'utf8');
     let report = JSON.parse(jobJSON);
+    const {id} = report;
     // Run the job.
     report = await doJob(report);
     // Archive it.
-    await fs.rename(`${todoDir}/${jobFileName}`, `${jobDir}/done/${jobFileName}`);
+    await fs.rename(`${todoDir}/${jobFileName}`, `${jobDir}/done/${id}.json`);
     // Save the report.
-    await fs.writeFile(`${rawDir}/${jobFileName}`, JSON.stringify(report, null, 2));
-    console.log(`Job completed and report ${report.id}.json saved in ${rawDir}`);
+    await fs.writeFile(`${rawDir}/${id}.json`, `${JSON.stringify(report, null, 2)}\n`);
+    console.log(`Job completed and report ${id} saved in ${rawDir}`);
   }
   // Otherwise, i.e. if the job does not exist.
   else {

package/dirWatch.js

@@ -52,7 +52,7 @@ const writeDirReport = async report => {
   }
 };
 // Archives a job.
-const archiveJob = async (job, isFile) => {
+const archiveJob = async (job, todoFileName) => {
   // Save the job in the done subdirectory.
   const {id} = job;
   const jobJSON = JSON.stringify(job, null, 2);
@@ -60,11 +60,11 @@ const archiveJob = async (job, isFile) => {
   await fs.mkdir(doneDir, {recursive: true});
   await fs.writeFile(`${doneDir}/${id}.json`, `${jobJSON}\n`);
   // If the job had been saved as a file in the todo subdirectory:
-  if (isFile) {
+  if (todoFileName) {
     // Delete the file.
-    await fs.rm(`${jobDir}/todo/${id}.json`);
+    await fs.rm(`${jobDir}/todo/${todoFileName}`);
   }
-  console.log(`Job ${id} archived in ${doneDir} (${nowString()})`);
+  console.log(`Job ${id} from ${todoFileName} archived in ${doneDir} (${nowString()})`);
 };
 // Waits.
 const wait = ms => {
@@ -92,11 +92,12 @@ exports.dirWatch = async (isForever, intervalInSeconds) => {
       const toDoFileNames = await fs.readdir(`${jobDir}/todo`);
       const jobFileNames = toDoFileNames.filter(fileName => fileName.endsWith('.json'));
       if (jobFileNames.length) {
+        const jobFileName = jobFileNames[0];
         // If the first one is ready to do:
-        const firstJobTimeStamp = jobFileNames[0].replace(/-.+$/, '');
+        const firstJobTimeStamp = jobFileName.replace(/-.+$/, '');
         if (Date.now() > dateOf(firstJobTimeStamp)) {
           // Get it.
-          const jobJSON = await fs.readFile(`${jobDir}/todo/${jobFileNames[0]}`, 'utf8');
+          const jobJSON = await fs.readFile(`${jobDir}/todo/${jobFileName}`, 'utf8');
           try {
             const job = JSON.parse(jobJSON);
             let report = JSON.parse(jobJSON);
@@ -108,7 +109,7 @@ exports.dirWatch = async (isForever, intervalInSeconds) => {
             // Save the report.
             await writeDirReport(report);
             // Archive the job.
-            await archiveJob(job, true);
+            await archiveJob(job, jobFileName);
           }
           catch(error) {
             console.log(`ERROR processing directory job (${error.message})`);

package/dist/nameComputation.js

@@ -1,7 +1,11 @@
 var computeAccessibleName = (() => {
   var __getOwnPropNames = Object.getOwnPropertyNames;
   var __commonJS = (cb, mod) => function __require() {
-    return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
+    try {
+      return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
+    } catch (e) {
+      throw mod = 0, e;
+    }
   };
 
   // node_modules/dom-accessibility-api/dist/polyfills/array.from.js

package/docker-compose.yml

@@ -0,0 +1,49 @@
+# © 2026 Jeff Witt.
+#
+# Licensed under the MIT License. See LICENSE file at the project root or
+# https://opensource.org/license/mit/ for details.
+#
+# SPDX-License-Identifier: MIT
+
+# Reference Compose stack for Testaro: the Testaro image watching a job
+# directory, plus a self-hosted Nu Html Checker as a sidecar so the nuVal
+# test does not depend on the public W3C service (which is rate-limited and
+# rejects large documents). See CONTAINERS.md for usage.
+services:
+  testaro:
+    build: .
+    # Reap zombie processes from forked test acts and browser relaunches.
+    init: true
+    environment:
+      # Disable the Chromium sandbox (default seccomp blocks the user
+      # namespaces it needs). Alternative: set this to "false" and run with
+      # a seccomp profile that permits user-namespace cloning, e.g.
+      #   security_opt:
+      #     - seccomp:seccomp_profile.json
+      # using the profile published by Playwright:
+      # https://github.com/microsoft/playwright/blob/main/utils/docker/seccomp_profile.json
+      TESTARO_CHROMIUM_NO_SANDBOX: "true"
+      # Send nuVal requests to the sidecar instead of the W3C service.
+      TESTARO_NU_URL: "http://nu:8888/?parser=html&out=json"
+      # JODBIR and REPORTDIR from Dockerfile override those from .env.
+      # Import other specific environment variables from .env file.
+      AGENT: ${AGENT}
+      WAVE_KEY: ${WAVE_KEY}
+      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
+    volumes:
+      # Host directories for jobs and reports. The container runs as the
+      # unprivileged pwuser; the host directories must be writable by it
+      # (see CONTAINERS.md).
+      - ./watch/jobs:/home/pwuser/testaro/watch/jobs
+      - ./watch/reports:/home/pwuser/testaro/watch/reports
+    depends_on:
+      - nu
+    # Headroom for browser relaunch spikes: multiple tools' browsers can
+    # briefly coexist with the catalog pass.
+    mem_limit: 4g
+
+  nu:
+    image: ghcr.io/validator/validator:latest
+    # Uncomment to also reach the checker from the host:
+    # ports:
+    #   - "8888:8888"

package/env.example

@@ -31,6 +31,11 @@ TIMEOUT_MULTIPLIER=1
 ABORT_ASSERTIVELY=false
 # URL of any non-default (e.g., self-hosted) Nu Html Checker API (see ghcr.io/validator/validator).
 TESTARO_NU_URL=__placeholder__
+# Whether to launch Chromium (Playwright's and QualWeb's) without its sandbox (normally false).
+# The sandbox requires unprivileged user-namespace cloning, which default container
+# seccomp policies and some hardened hosts prohibit. Set this to true in containers,
+# or run the container with a seccomp profile that permits user-namespace cloning.
+TESTARO_CHROMIUM_NO_SANDBOX=false
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "75.2.2",
+  "version": "76.0.0",
   "description": "Run 1300 web accessibility tests from 10 tools and get a standardized report",
   "main": "index.js",
   "scripts": {
@@ -55,5 +55,16 @@
     "dom-accessibility-api": "*",
     "esbuild": "*",
     "eslint": "*"
+  },
+  "overrides": {
+    "adm-zip": "^0.6.0"
+  },
+  "allowScripts": {
+    "accessibility-checker": true,
+    "puppeteer": true,
+    "chromedriver": true,
+    "esbuild": true,
+    "fsevents": true,
+    "vnu-jar": true
   }
 }

package/procs/launch.js

@@ -18,6 +18,14 @@
 
 const {addError} = require('./error');
 const headedBrowser = process.env.HEADED_BROWSER === 'true';
+// Whether to launch Chromium without its sandbox. The sandbox requires
+// unprivileged user-namespace cloning, which the default container seccomp
+// policies and some hardened hosts prohibit. Setting
+// TESTARO_CHROMIUM_NO_SANDBOX=true permits Chromium to run in such
+// environments; the alternative is to run the container with a seccomp
+// profile that permits user-namespace cloning. Applies only to Chromium;
+// WebKit and Firefox have no equivalent option.
+const chromiumNoSandbox = process.env.TESTARO_CHROMIUM_NO_SANDBOX === 'true';
 // Two flavors of Playwright:
 // - `playwrightCore`: the upstream Playwright SDK with no plugins attached.
 // - `playwrightExtra`: the playwright-extra wrapper. `run.js` registers
@@ -320,6 +328,11 @@ const launchOnce = async opts => {
       slowMo: waits || 0,
       args: browserOptionArgs
     };
+    // If launching Chromium without its sandbox was specified:
+    if (browserID === 'chromium' && chromiumNoSandbox) {
+      // Disable the sandbox.
+      browserOptions.chromiumSandbox = false;
+    }
     let browser, browserContext;
     try {
       // Create a browser of the specified type.

package/tests/qualWeb.js

@@ -72,11 +72,19 @@ exports.reporter = async (page, report, actIndex, timeLimit) => {
       instances: []
     };
   }
+  // Specify the options for the Puppeteer browser launched by QualWeb.
+  const puppeteerOptions = {
+    headless: true
+  };
+  // If launching Chromium without its sandbox was specified (e.g., in a
+  // container or on a host that restricts unprivileged user namespaces):
+  if (process.env.TESTARO_CHROMIUM_NO_SANDBOX === 'true') {
+    // Disable the sandbox of the QualWeb browser, too.
+    puppeteerOptions.args = ['--no-sandbox'];
+  }
   try {
     // Start the QualWeb core engine.
-    await qualWeb.start(clusterOptions, {
-      headless: true
-    });
+    await qualWeb.start(clusterOptions, puppeteerOptions);
   }
   // If the start fails:
   catch(error) {

package/validation/watch/done/240101T1200-simple-example.json

@@ -1,29 +0,0 @@
-{
-  "id": "240101T1200-simple-example",
-  "what": "Test example.edu with bulk",
-  "strict": true,
-  "timeLimit": 10,
-  "acts": [
-    {
-      "type": "launch",
-      "which": "chromium",
-      "url": "https://example.edu/",
-      "what": "Chromium browser"
-    },
-    {
-      "type": "test",
-      "which": "testaro",
-      "withItems": false,
-      "stopOnFail": true,
-      "rules": [
-        "y",
-        "bulk"
-      ]
-    }
-  ],
-  "sources": {},
-  "standard": "only",
-  "timeStamp": "240101T1500",
-  "creationTimeStamp": "240101T1200",
-  "sendReportTo": ""
-}

package/validation/watch/done/README.md

@@ -1,12 +0,0 @@
-# done
-
-Directory watched by executor `watchDir`.
-
-## License
-
-© 2023 CVS Health and/or one of its affiliates. All rights reserved.
-
-Licensed under the [MIT License](https://opensource.org/license/mit/). See [LICENSE](../../LICENSE) file
-at the project root for details.
-
-SPDX-License-Identifier: MIT

package/validation/watch/todo/README.md

@@ -1,12 +0,0 @@
-# todo
-
-Directory watched by executor `watchDir`.
-
-## License
-
-© 2023 CVS Health and/or one of its affiliates. All rights reserved.
-
-Licensed under the [MIT License](https://opensource.org/license/mit/). See [LICENSE](../../LICENSE) file
-at the project root for details.
-
-SPDX-License-Identifier: MIT