testaro 78.0.8 → 78.4.0

package/docs/standard-result-outcome.md

@@ -0,0 +1,165 @@
+# Certainty in the standard result: the `outcome` field
+
+**Status:** Phase A shipped in Testaro 78.1.0 (2026-09-01). Phases B–D are recommendations for
+the Testilo fork and yra-monitor, and for a later Testaro major version.
+**Companion:** yra-monitor `docs/plans/engine-candidacy-pipeline.md` (Stage-1 instance policy,
+Stage-3b taxonomy mapping).
+
+## Why
+
+Rule engines report two different things about a finding: how bad it is (impact) and whether
+they are sure it is a defect (certainty). Every Testaro adapter used to fold certainty into
+`ordinalSeverity`, and each did it differently:
+
+| tool | uncertain native value | ordinal | certain native value | ordinal |
+| --- | --- | --- | --- | --- |
+| axe, pour, surea11y | incomplete / cantTell | 0–1 | violation / fail | 2–3 |
+| htmlcs | Warning | 0 | Error | 2 |
+| ibm | recommendation | 0 | violation and potential violation | 2 |
+| alfa | cantTell (rule ID rewritten) | 0 | failed | 2 |
+| aslint | warning | 1 | error | 2 |
+| qualWeb act-rules | warning | 1 | failed | 3 |
+| qualWeb best-practices | warning | 0 | failed | 1 |
+| wave | alert | 0 | error, contrast | 3 |
+| ed11y | dismissable warning | 0 | error | 2 |
+| nuVal, nuVnu | info | 0 | error | 3 |
+
+So severity 0 meant "definitely minor" for some tools and "possibly critical" for others, and
+nothing downstream could recover certainty except by the heuristic "ordinal 2 or above means
+asserted". The ACT validation harness had to read native results to score sensitivity and
+specificity by certainty band. The Testilo issue classification (tic) grew 57 `*Risk` issues to
+hold uncertain findings at weight 1, and its scorer counted an axe `incomplete` contrast finding
+exactly like an asserted violation because it maps rule IDs only.
+
+The two new candidate engines make the loss visible: pour separates `violations`, `incomplete`,
+and `manualReview`; surea11y grades per occurrence (`fail` / `cantTell`) and publishes six
+machine-readable `uncertainty` codes with a `needed` sentence saying what a reviewer must
+determine.
+
+## The model
+
+Four dimensions, kept separate:
+
+| dimension | meaning | where it lives |
+| --- | --- | --- |
+| outcome (per instance) | did the engine assert a failure, or could it not tell | standard instance, set by the adapter |
+| rule certainty (per rule) | is the check deterministic, or a heuristic that can only say "suspicious" | tic tool-rule entry, as an override |
+| impact (per instance) | how bad for a user | `ordinalSeverity`, once it means impact only |
+| issue weight (per issue) | severity of the defect class | tic `weight` |
+
+Certainty exists at two levels because engines both under-assert (axe `incomplete`) and
+over-assert (a heuristic reported as an error). The instance level is Testaro's job; the rule
+level is the taxonomy's.
+
+## Phase A: Testaro 78.1.0 (shipped)
+
+Every standard instance now has:
+
+```js
+{
+  ruleID, what, ordinalSeverity, count, catalogIndex,   // unchanged
+  outcome: 'failed' | 'cantTell',                        // required; ACT Rules Format vocabulary
+  uncertainty: 'not-computable' | 'judgement-required' | 'runtime-dependent'
+             | 'spec-only' | 'equivalence-unknown' | 'out-of-scope',   // optional, cantTell only
+  needed: '<what a reviewer must determine>'             // optional, cantTell only, ≤ 300 chars
+}
+```
+
+and `standardResult.outcomeTotals = {failed, cantTell}` (count-weighted).
+
+- `procs/standard.ts` is the one place the shape is built (`getStandardResult`, `getInstance`,
+  `addInstance`, `pushInstance`). Adapters that compute `totals` from native counts (ibm, htmlcs,
+  ed11y, wave, testaro) use `pushInstance`; the rest use `addInstance`.
+- `ordinalSeverity` and `totals` are unchanged for every tool. The golden diff on the pour and
+  surea11y validation jobs is identical after stripping the new fields.
+- Per-tool mapping: axe/pour/surea11y violations → `failed`, incomplete → `cantTell`; htmlcs
+  Warning → `cantTell`; ibm potential violation, potential recommendation, and manual →
+  `cantTell`; alfa `cantTell` → `cantTell` (the adapter no longer rewrites the rule ID to
+  `cantTell` / `cantTellTextContrast` or prefixes `what`; r66 and r69 carry
+  `judgement-required`); aslint warning, qualWeb warning, wave alert, nuVal/nuVnu info →
+  `cantTell`; ed11y dismissable warning → `cantTell` with `judgement-required`.
+- surea11y forwards `uncertainty.code` and `uncertainty.needed` (occurrence first, then check).
+  Its `confidence` value is dropped as non-portable.
+- Testaro's own rules default to `failed`. An `allRules` entry may set `outcome` and
+  `uncertainty` (`allCaps` is `cantTell` / `judgement-required`, consistent with the earlier
+  ruling that AI confidence must not be encoded in `ordinalSeverity`). A violation description
+  may carry a prefix: `2:` severity only, `2?:` severity plus `cantTell`, `?:` `cantTell` at the
+  rule's default severity. `getBasicResult` accepts `outcome`, `uncertainty`, and `needed` on
+  each violation.
+- `validation/act/capture.js` records `outcomeTotals` and per-instance `outcome` and
+  `uncertainty`, and marks a row as an error if any instance lacks a valid outcome.
+
+Why `outcome` and not `certainty`: the ACT Rules Format and EARL already use `failed` /
+`cantTell`, the ACT testcase feed the harness scores against uses `failed`, and a closed
+two-value enum keeps consumer logic total. A graded `certainty` would invite scales no two
+engines share and collide with surea11y's separate `confidence`.
+
+Why `needed` is its own field: `what` is an identity key downstream (unique-issue rows, tsp
+description sampling). Appending guidance to it would fracture identity across pages.
+
+## Phase B: Testilo fork (recommended)
+
+Tool-rule entries in the tic gain optional properties; issues gain two:
+
+```js
+tools: { htmlcs: { 'W-AAA.1_4_3.G18.BgImage': {
+  variable: false, quality: 1, what: '…',
+  outcome: 'cantTell',            // override; 'cantTell' is the common case
+  uncertainty: 'not-computable',  // rule-level default; the instance value wins
+  review: '…',                    // fallback "what to confirm" when the engine gives no `needed`
+  ignore: 'invalid'               // replaces prose [invalid]/[irrelevant]/[duplicative]/[unreliable]
+}}}
+issue: { how: '<one imperative fix sentence>', group: 'landmarkNotTop' }
+```
+
+- Precedence: `rule.outcome` > `instance.outcome` > legacy inference > `failed`. Legacy
+  inference (reports without `outcome`) is per tool: axe, pour, surea11y, htmlcs, ibm, wave,
+  qualWeb → `ordinalSeverity >= 2 ? failed : cantTell`; alfa → rule ID starts with `cantTell`;
+  testaro, ed11y, aslint, nuVal, nuVnu → `failed`.
+- `[speculative]` becomes `outcome: 'cantTell'`; the other bracket tags become `ignore`. The
+  `ignorable` pseudo-issue dissolves into per-rule `ignore`. A loader assertion rejects any
+  `what` still carrying a bracket tag.
+- Consolidate the 57 `*Risk` issues: merge into the base issue when one exists with the same fix
+  (7 name twins plus about 11 semantic twins such as contrastRisk → contrastAA/AAA by level);
+  rename (drop `Risk`) when the issue is the sole home of a defect class with a concrete fix
+  (pseudoHeadingRisk → pseudoHeading, which already holds axe `p-as-heading` and takes pour's
+  at `failed`; eventKeyboardRisk → eventKeyboard; labelConfusionRisk → labelPlacement with every
+  rule `cantTell` / `judgement-required`); keep three page-environment advisories
+  (applicationRisk, browserSupportRisk, noScriptRisk) with every rule `cantTell`. Export a
+  `migrations` map (`{contrastRisk: 'contrastAA', …}`) for downstream issue-ID rewrites.
+- Scoring: `cantTellWeight = 0.25` per instance inside the quality-weighted count (a weight-1
+  Risk issue merged into a weight-4 base is score-neutral). Element-aware dedup: an element's
+  outcome is `failed` if any tool says so, corroborated when more than one does;
+  `effectiveCount = max(old per-tool max with the factor applied, element-union count)`.
+  Output stays additive (`instanceCounts` numeric; add `outcomeCounts`, `elementCounts`,
+  `details.elements`; `details.element` byte-identical).
+- Granularity principle: one tic issue = one remediation class = one WCAG success criterion ×
+  one fix. `ruleID` and `what` carry the diagnosis; certainty is never a reason to split an
+  issue. Families get `group`; element-conditioned splits (pour `link-name` → image link vs text
+  link) are done in the adapter with a `.`-suffixed faceted rule ID, two tic entries each.
+
+## Phase C: yra-monitor (recommended)
+
+Types, `tic-manager` `resolveOutcome`, scoring-service passthrough, standardized rows and DB
+columns (`outcome`, `uncertainty`, `needed`) with a promote-only conflict rule (`failed` wins),
+issue-ID rewrite from `migrations`, a "Needs review" badge and filter, and for `cantTell` rows
+the text `instance.needed ?? rule.review ?? 'Manual review required'` under a "To confirm"
+heading.
+
+## Phase D: Testaro 79 (recommended)
+
+Redefine `ordinalSeverity` as impact only: 0 minor, 1 moderate, 2 serious, 3 critical, which is
+what yra-monitor already labels it as. `totals` keeps its shape but its composition changes
+(an axe incomplete + critical finding moves from `totals[1]` to `totals[3]`). Safe only after
+Phases B and C, when every consumer reads `outcome`.
+
+## Open decisions
+
+1. `cantTellWeight = 0.25`: confirm against Stage-4 precision on surea11y's `cantTell` band.
+2. The semantic Risk merges need a "same fix?" pass; videoCaptionRisk → videoNoText and
+   fieldSetRisk → fieldSetMissing are the least certain.
+3. Weights for renamed heuristic issues once they can carry a `failed` band.
+4. The alfa rule-ID change is visible to the current Testilo now: r66/r69 `cantTell` instances
+   map to contrastAAA/contrastAA at full weight until the scorer reads `outcome`. Pin Testaro
+   78.1 in the worker only together with the Phase-B Testilo.
+5. Whether `unique_issues.severity` should roll up (`GREATEST`) on conflict.

package/env.example

@@ -2,8 +2,6 @@
 WAVE_KEY=__placeholder__
 # You can get an Anthropic API key at https://console.anthropic.com/.
 ANTHROPIC_API_KEY=__placeholder__
-# Name to identify this Testaro instance to a server.
-AGENT=__placeholder__
 # Whether to make the browser visible (normally false).
 HEADED_BROWSER=false
 # Whether to output even forked-job logging to the primary log (normally false).
@@ -12,23 +10,44 @@ DEBUG=false
 WAITS=0
 # See https://nodejs.org/api/cli.html#options for all permitted options.
 NODE_OPTIONS='--trace-uncaught --trace-warnings'
-# Suppresses a routine warning from QualWeb.
+# Whether to suppress a routine warning from QualWeb.
 PUPPETEER_DISABLE_HEADLESS_WARNING=true
-# Replace the next 2 placeholders with http://localhost:3000 if Kilotest and Testaro are on the same host or the URL of the server (e.g., https://kilotest.com) if they are on different hosts.
-# URL to poll for available jobs when watching the network.
-NETWATCH_URL_JOB=__placeholder__/api/testaro-agent/job
-# URL to report results to when watching the network.
-NETWATCH_URL_REPORT=__placeholder__/api/testaro-agent/report
-# Password of this Testaro agent
-NETWATCH_URL_AUTH=__placeholder__
 # Directory (relative to project root) to watch for jobs.
 JOBDIR=__placeholder__
 # Directory (relative to project root) to write reports to when watching a directory for jobs.
 REPORTDIR=__placeholder__
 # Multiplier for time limits (normally 1).
 TIMEOUT_MULTIPLIER=1
+# Default isolation of test acts when a job does not specify one: process (a child process and
+# browser per test act), browser (one browser per job, a fresh context per test act), or page
+# (test acts share the live page of the current checkpoint). See docs/checkpoint-scanning.md.
+ISOLATION=process
 # Whether to abort the job when launch retries are exhausted or a test act crashes (default false).
 ABORT_ASSERTIVELY=false
+
+# Deployment options. A job property of the same meaning overrides each of these.
+
+# Load state a navigation waits for before tools run: networkidle (default), load, or
+# domcontentloaded. Retries relax it one step. networkidle never arrives on pages whose
+# network never quiets (beacons, polling), so a deployment scanning such pages may prefer load.
+NAV_WAIT_UNTIL=networkidle
+# Navigation time limit in milliseconds (default 10000).
+NAV_TIMEOUT=10000
+# Whether a 4xx response (other than 408 and 429) ends launch retries at once (default false).
+NAV_FAIL_FAST_4XX=false
+# Scanner identity sent as the X-YRA-Scanner request header, so firewalls can recognize the
+# scanner by a stable value (default none; jobs may set scannerId).
+SCANNER_ID=__placeholder__
+# Whether to scroll the full page after navigation so lazily loaded content is present before
+# cataloguing, imaging, and each tool (default false; jobs may set scroll).
+PRESCAN_SCROLL=false
+# Launch retries per rule of the testaro tool, which relaunches per contaminating rule
+# (default 2; raise for non-Chromium browsers, which navigate flakily on relaunch).
+TESTARO_RULE_RETRIES=2
+# Whether the qualWeb tool's browser runs stealth evasions and blocks ads and trackers
+# (default true; a qualWeb test act may set stealth and adBlock).
+QUALWEB_STEALTH=true
+QUALWEB_ADBLOCK=true
 # URL of any non-default (e.g., self-hosted) Nu Html Checker API (see ghcr.io/validator/validator).
 TESTARO_NU_URL=__placeholder__
 # Whether to launch the Playwright chromium browser without its sandbox (normally false).
@@ -37,6 +56,21 @@ TESTARO_NU_URL=__placeholder__
 # or run containers with a seccomp profile that permits user-namespace cloning.
 TESTARO_CHROMIUM_NO_SANDBOX=false
 
+# netWatch variables
+
+# Replace the next 2 placeholders with the URL of the server (e.g., http://localhost:3000 or  https://kilotest.com), including the path required by the server.
+# URL to poll for available jobs.
+NETWATCH_URL_JOB=__placeholder__
+# URL to send a job report.
+NETWATCH_URL_REPORT=__placeholder__
+# Type of authentication (none, pathBody, or header) required by the server.
+NETWATCH_AUTH_TYPE=__placeholder__
+# ID of this instance. Recorded in reports as sources.agent and used by header
+# authentication (where it must not contain a colon).
+NETWATCH_WORKER_ID=__placeholder__
+# Password of this instance. Required when NETWATCH_AUTH_TYPE is pathBody or header.
+NETWATCH_WORKER_SECRET=__placeholder__
+
 ## License
 
 # © 2021–2025 CVS Health and/or one of its affiliates. All rights reserved.

package/eslint.config.mjs

@@ -0,0 +1,101 @@
+/*
+  © 2026 Jeff Witt.
+
+  Licensed under the MIT License. See LICENSE file at the project root or
+  https://opensource.org/license/mit/ for details.
+
+  SPDX-License-Identifier: MIT
+*/
+
+/*
+  eslint.config.mjs
+  Flat ESLint configuration, replacing .eslintrc.json (unsupported since ESLint 9).
+*/
+
+// IMPORTS
+
+import js from '@eslint/js';
+import globals from 'globals';
+import {defineConfig, globalIgnores} from 'eslint/config';
+
+// CONFIGURATION
+
+export default defineConfig([
+  globalIgnores([
+    // Build output.
+    'dist/',
+    // Vendored HTML CodeSniffer bundle; not to be reformatted.
+    'htmlcs/HTMLCS.js',
+    // Vendored Editoria11y bundles.
+    'ed11y/',
+    // Vendored SureA11y standalone browser bundle (MPL-2.0; not to be modified).
+    'surea11y/surea11y.browser.js',
+    // Vendored Pour Engine bundle, built with esbuild (see pour/README.md).
+    'pour/pour.min.js',
+    // Files emitted by tsc from TypeScript sources (issue #73); lint the .ts sources instead.
+    'types.js',
+    'procs/catalog.js',
+    'procs/standard.js',
+    'procs/testaro.js',
+    'procs/xPath.js',
+    'testaro/*.js',
+    'tests/*.js',
+    '**/*.d.ts',
+    // Validation fixtures, some of which are intentionally defective.
+    'validation/tests/targets/'
+  ]),
+  {
+    files: ['**/*.js'],
+    extends: [js.configs.recommended],
+    languageOptions: {
+      ecmaVersion: 2021,
+      sourceType: 'commonjs',
+      globals: {
+        ...globals.browser,
+        ...globals.node
+      }
+    },
+    rules: {
+      indent: [
+        'error',
+        2,
+        {
+          MemberExpression: 0,
+          ObjectExpression: 'first'
+        }
+      ],
+      'linebreak-style': [
+        'error',
+        'unix'
+      ],
+      quotes: [
+        'error',
+        'single'
+      ],
+      semi: [
+        'error',
+        'always'
+      ],
+      'no-use-before-define': [
+        'error'
+      ],
+      'brace-style': [
+        'error',
+        'stroustrup'
+      ],
+      /*
+        Transitional demotions to warnings, so linting can gate changes now.
+        These rules have pre-existing violations, some of them latent defects.
+        Each is to be restored to an error once its violations are repaired.
+      */
+      'no-async-promise-executor': 'warn',
+      'no-const-assign': 'warn',
+      'no-constant-binary-expression': 'warn',
+      'no-empty': 'warn',
+      'no-undef': 'warn',
+      'no-unused-vars': 'warn',
+      'no-use-before-define': 'warn',
+      'no-useless-assignment': 'warn'
+    }
+  }
+]);