automatica11y 0.3.0 → 0.3.2

package/AGENTS.md

@@ -29,4 +29,4 @@ Both print files that live in `skills/automatica11y-runner/` in the repository a
 - Heavy dependencies (Playwright, esbuild, the rule engines) load only when a command needs them. A test checks that `--version`, `doctor`, `guide`, and `--plan` never import them.
 - Keep findings from axe-core and IBM Equal Access separate. Never add their counts together or convert one engine's scale into the other's.
 - A gap, an error, a failed target, or a result that isn't testable is a finding. It never counts as a pass. A report says "no automated violations found" only where an engine found none, and never says "accessible."
-- There are two skills. `skills/automatica11y/` is a tiny bootstrap that people copy. It sends an agent to `guide`. `skills/automatica11y-runner/` holds the full steps and ships with the tool. The runner names the version series it works with. The series is the major and minor version while the major version is 0 (`0.2`), and the major version alone from 1.0 on. Update the series named in `skills/automatica11y-runner/SKILL.md` (written like `0.2.x`) whenever the series of the version in `package.json` changes, for example from `0.2.x` to `0.3.0`. A change from `0.2.5` to `0.2.6` needs no update. A test fails if they disagree.
+- There are two skills. `skills/automatica11y/` is a tiny bootstrap that people copy. It sends an agent to `guide skill`. `skills/automatica11y-runner/` holds the full steps and ships with the tool. The runner names the version series it works with. The series is the major and minor version while the major version is 0 (`0.2`), and the major version alone from 1.0 on. Update the series named in `skills/automatica11y-runner/SKILL.md` (written like `0.2.x`) whenever the series of the version in `package.json` changes, for example from `0.2.x` to `0.3.0`. A change from `0.2.5` to `0.2.6` needs no update. A test fails if they disagree.

package/README.md

@@ -111,7 +111,7 @@ npx automatica11y@latest guide fixtures  # how to write the fixtures an npm pack
 
 The guidance ships with the tool, so it always matches the version you run. Tell your agent to run `npx automatica11y@latest guide` and follow it, then ask for things like "How accessible is Radix Dialog?" or "Compare the accessibility of React Aria and Headless UI." The agent needs to run shell commands and read and write files. Nothing here is tied to one agent.
 
-**Skills.** If your agent loads skills from a folder, copy [`skills/automatica11y`](skills/automatica11y) into it. That's one small file, `SKILL.md`. It advertises the tool to the agent, and sends it to `guide`. It names no version, so it doesn't go stale. The full steps are the [`automatica11y-runner`](skills/automatica11y-runner) skill, which ships in the package and is what `guide skill` prints. Copy it too if you want the steps available without the network.
+**Skills.** If your agent loads skills from a folder, copy [`skills/automatica11y`](skills/automatica11y) into it. That's one small file, `SKILL.md`. It advertises the tool to the agent, and sends it to `guide skill`. It names no version, so it doesn't go stale. The full steps are the [`automatica11y-runner`](skills/automatica11y-runner) skill, which ships in the package and is what `guide skill` prints. Copy it too if you want the steps available without the network.
 
 **AGENTS.md.** [`AGENTS.md`](AGENTS.md) is for agents that read it but don't load skills. It points to the same steps, and tells contributors how to run and change the code.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "automatica11y",
-  "version": "0.3.0",
+  "version": "0.3.2",
   "description": "Test and compare the accessibility of web pages, Storybook builds, and npm component libraries.",
   "license": "MIT",
   "type": "module",

package/skills/automatica11y/SKILL.md

@@ -13,11 +13,13 @@ automatica11y tests and compares web accessibility. This skill only gets you sta
 3. Run this, read all of what it prints, and follow it:
 
    ```bash
-   npx --yes automatica11y@latest guide
+   npx --yes automatica11y@latest guide skill
    ```
 
-   If the command exits with an error or prints no usable output, tell the user it failed, include the error text, and stop. Don't continue with partial instructions.
+   If the error mentions `ETARGET` (npm says a version doesn't exist, usually because its local list is out of date), run the command once more with `--prefer-online`: `npx --yes --prefer-online automatica11y@latest guide skill`.
 
-   After reading the guide and the steps it sends you to, pass only settings they list as supported. If the user named a setting they don't list as supported, tell the user which setting is unsupported, list the supported values from the guide, and ask which to use. Don't ask again for anything they've already said.
+   If the command exits with an error or prints no usable output, including after that retry, tell the user it failed, include the error text, and stop. Don't continue with partial instructions.
+
+   After reading the steps, pass only settings they list as supported. If the user named a setting they don't list as supported, tell the user which setting is unsupported, list the supported values from the steps, and ask which to use. Don't ask again for anything they've already said.
 
 4. If you can't run shell commands, or `npx` can't reach the npm registry, tell the user and stop. Don't guess at results.

package/skills/automatica11y-runner/SKILL.md

@@ -14,13 +14,15 @@ The tool is **not** an attestation or certification tool. Automated checks cover
 
 ## 1. Check the version
 
-This skill works with automatica11y **0.2.x**. Run:
+Do sections 1 and 2 before you run an audit. A question to the user about a missing target or an unsupported setting (section 3) can come before or after them.
+
+This skill works with automatica11y **0.3.x**. Run:
 
 ```bash
 npx --yes automatica11y@latest --version
 ```
 
-If the command exits with an error or prints nothing, tell the user it failed, include the error text, and stop. If the output doesn't start with `0.2.`, stop. Tell the user the version you got and the series this copy expects (`0.2.x`). If this copy came from a file, offer to read the matching steps with `npx --yes automatica11y@latest guide skill`. Don't run an audit until the user confirms how to proceed.
+If the command exits with an error or prints nothing, tell the user it failed, include the error text, and stop. If the output doesn't start with `0.3.`, stop. Tell the user the version you got and the series this copy expects (`0.3.x`). If this copy came from a file, offer to read the matching steps with `npx --yes automatica11y@latest guide skill`. Don't run an audit until the user confirms how to proceed.
 
 ## 2. Check the setup
 
@@ -39,7 +41,7 @@ npx --yes automatica11y@latest audit <target> [options]
 npx --yes automatica11y@latest compare <target> <target> [<target>...] [options]
 ```
 
-Use `audit` for one target and `compare` for two or more. A target is `[label=]<spec>`. The label is optional and names the target in the report.
+Use `audit` for one target and `compare` for two or more, even when they're different kinds, such as a live page and an npm package. The report then opens with a warning that the evidence isn't equivalent. A target is `[label=]<spec>`. The label is optional and names the target in the report.
 
 | The user means | The spec is |
 |---|---|
@@ -72,7 +74,7 @@ Use the target and the settings the user already gave, and ask only for what's m
 
 Pass only the options in the table above, with the values it lists. If the user names a setting the tool doesn't have, or a value outside those lists (for example, WCAG 3.0), tell them which setting is unsupported, list the supported values, and ask which to use. Don't substitute a default or invent a value.
 
-Show the user the exact command before you run it. If a target is ambiguous, ask one question, then go on.
+Show the user the exact command before you run it. If you have questions (a target that could mean two things, a missing target, an unsupported setting), ask them together in one message, then go on.
 
 Don't set the fail flags unless the user asks for gating. They change the exit code. They don't change the results.
 
@@ -82,9 +84,9 @@ A package's components can't be guessed from its name. The first run installs th
 
 1. Run the audit once. Read `<out>/mapping.json` and the report's **Archetypes** table.
 2. Treat the mapping as a guess. Check each `export` or `tag` against what the user asked about.
-3. For each archetype marked `needs-fixture` that matters to the request, write `fixtures/<target id>/<archetype>.jsx` (`.js` for web components) in the working directory. Follow the contract and the examples below. `references/fixtures.md` has more: the mapping file, states, and library accessibility options. It should sit next to this file. If you can't open it, run `npx --yes automatica11y@latest guide fixtures` to print it. The contract here is enough for the first fixture.
+3. Each archetype in `mapping.json` has a status. `needs-fixture` means the tool found the component but can't build it from a template. `no-match` means no export or custom element looks like that archetype. For each archetype marked `needs-fixture` that matters to the request, write `fixtures/<target id>/<archetype>.jsx` (`.js` for web components) in the working directory. Follow the contract and the examples below. `references/fixtures.md` has more: the mapping file, states, and library accessibility options. It should sit next to this file. If you can't open it, run `npx --yes automatica11y@latest guide fixtures` to print it. The contract here is enough for the first fixture.
 4. Run the same command again. The tool finds fixtures in that folder without `--mapping`.
-5. Don't invent fixtures for archetypes the user didn't ask about. A gap is an honest result.
+5. Don't invent fixtures for archetypes the user didn't ask about. For a `no-match` archetype, write a fixture only if the library's documentation names a component, or a documented way, to make it. Otherwise leave it as a gap. A gap is an honest result.
 
 **The fixture contract, in brief.**
 
@@ -138,7 +140,7 @@ Run the command. Note the exit code:
 
 | Code | Meaning |
 |---|---|
-| 0 | The run completed. Findings don't change this unless a fail flag was set. |
+| 0 | The run completed. Findings don't change this unless a fail flag was set. A target that failed while others ran still exits 0, and the failure is in the results. |
 | 1 | The run completed and a fail flag tripped. |
 | 2 | The command was wrong. Read the message, fix it, and run again. |
 | 3 | An environment problem. Relay the fix. |
@@ -154,7 +156,7 @@ Write the narrative from `results.json`. Never write from memory, and never repe
 2. For a comparison, say that every target used the same archetypes, WCAG version, level, and rules.
 3. Open with the coverage matrix (target by archetype by tier). Then give the findings.
 4. Keep violations, needs-review items, and passes in separate lists. Never merge them.
-5. Break findings out by archetype and by impact.
+5. Break findings out by archetype. Within an archetype, order axe-core findings by impact and IBM findings by Toolkit level.
 6. Use "no automated violations found" only for an engine that reported none for that target. For an engine that reported violations, list them as the tool reports them. Never say "accessible," "compliant," or "passes WCAG."
 7. Don't print a single score. If someone insists, pair any number with the coverage matrix and the automated-coverage caveat.
 8. Label virtual screen reader output **simulated**. Label library accessibility options **on** or **off** on every result that has one.
@@ -174,7 +176,7 @@ Say these things plainly. Don't soften them, and don't fill in a result.
 - **Not testable.** The content is a canvas with no alternative, or sits in a closed shadow root. The rule engines can't see it, so the result is untested, not clean. The virtual screen reader also can't read open shadow roots.
 - **Gap.** The archetype has no usable fixture or no matching export. Say what the archetype needs.
 - **Error.** An interaction check couldn't finish. It's untested, not failed.
-- **Failed target.** The target can't be reached, isn't a web page, or couldn't be built. The tool records it as failed with a reason. Tell the user which target failed, using the reason from `results.json`. Don't retry with guesses. If every target failed (exit code 4), stop. If others ran, report them, and list the failed target as a gap in coverage.
+- **Failed target.** The target can't be reached, isn't a web page, or couldn't be built. The tool records it as failed with a reason. Tell the user which target failed, using the reason from `results.json`. Don't retry with guesses. If an npm target failed with a network or install error (for example `ETARGET`), you may run the same command once more. If it fails again, report it. If every target failed (exit code 4), stop. If others ran, report them, and list the failed target as a gap in coverage.
 
 ## 8. Stay out of setup
 

package/src/harness/npm-install.js

@@ -19,6 +19,9 @@ export function runNpm(args, cwd, { timeoutMs = 300_000 } = {}) {
   });
 }
 
+/** npm's local copy of a package list can lag behind the registry, so a version that exists looks missing. */
+const STALE_CACHE = /ETARGET|notarget|No matching version/i;
+
 /** The version of an installed package, or null. */
 export function installedVersion(dir, name) {
   try {
@@ -39,11 +42,23 @@ export async function installPackage({ dir, name, version, flavor, run = runNpm
   if (!existsSync(join(dir, "package.json"))) writeFileSync(join(dir, "package.json"), JSON.stringify({ name: "automatica11y-target", private: true }));
   /** @type {string[]} */
   const warnings = [];
+  let refreshed = false;
+  /** Run npm. If it says a version doesn't exist, ask once more with fresh package data, since the version came from the registry. */
+  const npm = async (args) => {
+    try {
+      return await run(args, dir);
+    } catch (error) {
+      if (!STALE_CACHE.test(String(error.message)) || !args.includes("--prefer-offline")) throw error;
+      if (!refreshed) warnings.push(`npm's local list of ${name} versions was out of date, so the install refreshed it and tried again.`);
+      refreshed = true;
+      return run(args.map((arg) => (arg === "--prefer-offline" ? "--prefer-online" : arg)), dir);
+    }
+  };
   try {
-    await run(["install", `${name}@${version}`, ...NPM_FLAGS], dir);
+    await npm(["install", `${name}@${version}`, ...NPM_FLAGS]);
   } catch (error) {
     if (!/ERESOLVE|peer dep/i.test(String(error.message))) throw new Error(`npm couldn't install ${name}@${version}: ${firstLine(error.message)}`);
-    await run(["install", `${name}@${version}`, "--legacy-peer-deps", ...NPM_FLAGS], dir).catch((retry) => {
+    await npm(["install", `${name}@${version}`, "--legacy-peer-deps", ...NPM_FLAGS]).catch((retry) => {
       throw new Error(`npm couldn't install ${name}@${version}: ${firstLine(retry.message)}`);
     });
     warnings.push(`npm couldn't satisfy ${name}'s peer dependencies, so it installed them loosely (--legacy-peer-deps). Results may not match a supported setup.`);
@@ -51,10 +66,10 @@ export async function installPackage({ dir, name, version, flavor, run = runNpm
   if (flavor === "react") {
     const react = installedVersion(dir, "react");
     if (!react) {
-      await run(["install", "react", "react-dom", ...NPM_FLAGS, "--legacy-peer-deps"], dir);
+      await npm(["install", "react", "react-dom", ...NPM_FLAGS, "--legacy-peer-deps"]);
       warnings.push(`${name} didn't bring in react, so the latest react and react-dom were added.`);
     } else if (!installedVersion(dir, "react-dom")) {
-      await run(["install", `react-dom@${react}`, ...NPM_FLAGS, "--legacy-peer-deps"], dir);
+      await npm(["install", `react-dom@${react}`, ...NPM_FLAGS, "--legacy-peer-deps"]);
     }
   }
   return { dir, warnings, react: installedVersion(dir, "react"), reactDom: installedVersion(dir, "react-dom"), version: installedVersion(dir, name) };

package/src/plan/resolve-npm.js

@@ -53,7 +53,8 @@ export function detectFlavor(meta) {
   const peers = meta.peerDependencies ?? {};
   const deps = meta.dependencies ?? {};
   if ("react" in peers || "react-dom" in peers || "react" in deps) {
-    return { kind: "npm-react", framework: "React", reason: "The package lists react as a dependency." };
+    const how = "react" in peers || "react-dom" in peers ? "peer dependency" : "dependency";
+    return { kind: "npm-react", framework: "React", reason: `The package lists react as a ${how}.` };
   }
   if (meta.customElements) return { kind: "npm-wc", framework: "Web components", reason: "The package has a customElements manifest." };
   for (const [name, label] of Object.entries(OTHER_FRAMEWORKS)) {