pa11y-ci-reporter-runner 0.6.0 → 0.7.0

package/README.md

@@ -1,6 +1,6 @@
 # Pa11y CI Reporter Runner
 
-Pa11y CI Reporter Runner is designed to facilitate testing of [Pa11y CI reporters](https://github.com/pa11y/pa11y-ci#write-a-custom-reporter). Given a Pa11y CI JSON results file and optional configuration it simulates the Pa11y CI calls to the reporter, including proper tranformation of results and configuration data. Functionally, it's a mock of the Pa11y CI side of the reporter interface.
+Pa11y CI Reporter Runner is designed to facilitate testing of [Pa11y CI reporters](https://github.com/pa11y/pa11y-ci#write-a-custom-reporter). Given a Pa11y CI JSON results file and optional configuration it simulates the Pa11y CI calls to the reporter, including proper transformation of results and configuration data. Functionally, it's an emulation of the Pa11y CI side of the reporter interface with finer control over execution.
 
 ## Installation
 
@@ -15,19 +15,50 @@ npm install pa11y-ci-reporter-runner
 Pa11y CI Reporter Runner exports a factory function that creates a reporter runner. This function has four arguments:
 
 - `resultsFileName`: Path to a Pa11y CI JSON results file.
-- `reporterName`: Name of the reporter to execute. Can be a dependency (e.g. `pa11y-ci-reporter-html`) or a path to a reporter file.
+- `reporterName`: Name of the reporter to execute. Can be an npm module (e.g. `pa11y-ci-reporter-html`) or a path to a reporter file.
 - `options`: Optional [reporter options](https://github.com/pa11y/pa11y-ci#reporter-options).
 - `config`: Optional Pa11y CI configuration file that produces the Pa11y CI JSON results.
 
-The reporter runner has two functions:
+### Runner States
 
-- `runAll`: Simulates Pa11y CI running the complete analysis from the provided JSON results file, calling all associated reporter functions (`beforeAll`, `begin` and `results`/`error` for each URL, `afterAll`). Once a run has been completed the runner must be reset before running again.
-- `reset`: Resets the runner to the initial state.
+Pa11y CI Reporter Runner emulates calls from Pa11y CI to the given reporter for the given results file. There are five distinct states for the runner, as shown below:
+
+```mermaid
+flowchart LR;
+    init-->beforeAll;
+    beforeAll-->beginUrl;
+    beginUrl-->urlResults;
+    urlResults-->beginUrl;
+    urlResults-->afterAll;
+```
+
+- `init`: The initial state of the runner.
+- `beforeAll`: In this state the runner calls the reporter `beforeAll` function.
+- `beginUrl`: In this state the runner calls the reporter `begin` function for a given URL.
+- `urlResults`: In this state the runner calls the reporter `results` or `error` function depending on the result for that URL. If there are subsequent URLs, the runner next moves to `beginUrl` for the next URL. If this is the last URL, the runner moves to `afterAll`.
+- `afterAll`: In this state the runner calls the reporter `afterAll` function.
+
+When calling reporter functions, the runner transforms the Pa11y CI results and configuration data to provide the appropriate arguments. For example, the `results` function is called with the results as returned from Pa11y (slightly different than those returned from Pa11y CI) and the consolidated configuration for the analysis of that URL.
+
+### Runner Execution
+
+The reporter runner has four control functions:
+
+- `runAll()`: Simulates Pa11y CI running the analysis from the provided JSON results file from the current state through the end, calling all associated reporter functions.
+- `runNext()`: Simulates Pa11y CI running through the next state from the provided JSON results file, calling the associated reporter function as noted above.
+- `runUntil(targetState, targetUrl)`: Simulates Pa11y CI running the analysis from the provided JSON results file from the current state through the specified state/URL, calling the associated reporter functions as noted above. An error will be thrown if the end of the results are reached and the target was not found. This function takes the following arguments:
+  - `targetState`: The target state of the runner (any state above except `init`).
+  - `targetUrl`: An optional target URL. If no URL is specified, the runner will stop at the first instance of the target state.
+- `reset()`: Resets the runner to the `init` state. This can be sent from any state.
+
+These command are all asynchronous and must be completed before another is sent, otherwise an error will be thrown. In addition, once a run has been completed and the runner is in the `afterAll` state it must be `reset` before accepting any run command.
+
+### Example
 
 A complete example is provided below:
 
 ```js
-const createRunner = require("pa11y-ci-reporter-runner");
+const { createRunner, RunnerStates } = require("pa11y-ci-reporter-runner");
 
 const resultsFileName = "pa11yci-results.json";
 const reporterName = "../test-reporter.js";
@@ -46,15 +77,26 @@ const config = {
   ],
 };
 
-test('test reporter', async () => {
+test('test all reporter functions', async () => {
   const runner = createRunner(resultsFileName, reporterName, reporterOptions, config);
 
   await runner.runAll();
 
   // Test reporter results
 });
+
+test('test reporter at urlResults state', async () => {
+  const runner = createRunner(resultsFileName, reporterName, reporterOptions, config);
+
+  await runner.runUntil(RunnerStates.beginUrl, 'http://localhost:8080/page1-no-errors.html');
+  // The runner is now in the beginUrl state for http://localhost:8080/page1-no-errors.html
+  await runner.runNext();
+  // The runner is now in the urlResults state for http://localhost:8080/page1-no-errors.html
+
+  // Test reporter results
+});
 ```
 
 ## Limitations
 
-When passing config to `results`, `error`, and `afterAll`, Pa11y CI Reporter Runner includes the same properties as Pa11y CI except the `browser` property (with the `puppeteer` `browser` object used by Pa11y CI). If the `browser` object is needed, testing should be done with Pa11y CI to ensure proper `browser` configuration.
+When passing config to `results`, `error`, and `afterAll`, Pa11y CI Reporter Runner includes the same properties as Pa11y CI except the `browser` property (with the `puppeteer` `browser` object used by Pa11y CI). If the `browser` object is needed, testing should be done with Pa11y CI to ensure proper `browser` capabilities are available.

package/index.js

@@ -12,6 +12,7 @@ const formatter = require('./lib/formatter');
 const reporterBuilder = require('./lib/reporter-builder');
 const createConfig = require('./lib/config');
 const { serviceFactory } = require('./lib/pa11yci-service');
+const RunnerStates = require('./lib/runner-states');
 
 /**
  * Loads the Pa11y CI results from JSON file.
@@ -185,13 +186,41 @@ const createRunner = (resultsFileName, reporterName, options = {}, config = {})
      * @instance
      */
     const runAll = async () => {
-        await service.runAll();
+        await service.runUntil(RunnerStates.afterAll);
+    };
+
+    /**
+     * Executes the next event in the Pa11y CI sequence, calling the
+     * appropriate reporter function.
+     *
+     * @public
+     * @instance
+     */
+    const runNext = async () => {
+        await service.runNext();
+    };
+
+    /**
+     * Executes the entire Pa11y CI sequence, calling all reporter functions,
+     * until the specified state and optional URL are reached. If no URL is provided
+     * specified, the run completes on the first occurrence of the target state.
+     *
+     * @public
+     * @interface
+     * @param {string} targetState The target state to run to.
+     * @param {string} [targetUrl] The target URL to run to.
+     */
+    const runUntil = async (targetState, targetUrl) => {
+        await service.runUntil(targetState, targetUrl);
     };
 
     return {
         reset,
-        runAll
+        runAll,
+        runNext,
+        runUntil
     };
 };
 
-module.exports = createRunner;
+module.exports.createRunner = createRunner;
+module.exports.RunnerStates = RunnerStates;

package/lib/pa11yci-service.js

@@ -3,12 +3,7 @@
 const { interpret } = require('xstate');
 const machine = require('./pa11yci-machine');
 
-const RunnerStates = Object.freeze({
-    beforeAll: 'beforeAll',
-    beginUrl: 'beginUrl',
-    urlResults: 'urlResults',
-    afterAll: 'afterAll'
-});
+const RunnerStates = require('./runner-states');
 const finalState = RunnerStates.afterAll;
 
 /**
@@ -32,6 +27,32 @@ const getInitialContext = urls => ({
  */
 const hasUrl = state => state === RunnerStates.beginUrl || state === RunnerStates.urlResults;
 
+/**
+ * Validates that the given state is a valid RunnerStates value. Throws if not.
+ *
+ * @private
+ * @param {string} state The state to validate.
+ */
+const validateRunnerState = (state) => {
+    if (!Object.keys(RunnerStates).includes(state)) {
+        throw new TypeError(`targetState "${state}" invalid`);
+    }
+};
+
+/**
+ * Checks if the context matches the targetState and either no
+ * targetUrl was specified or the context matches the targetUrl.
+ *
+ * @private
+ * @param   {object}  context     The current state machine context.
+ * @param   {string}  targetState The target runner state.
+ * @param   {string}  [targetUrl] The target URL.
+ * @returns {boolean}             True if the context matches the target, otherwise false.
+ */
+const isAtTarget = (context, targetState, targetUrl) => {
+    return context.state === targetState && (!targetUrl || context.url === targetUrl);
+};
+
 /**
  * Factory function that return a pa11yci-runner service.
  *
@@ -72,30 +93,31 @@ const serviceFactory = (urls, actions) => {
     service.start();
 
     /**
-     * Sends the NEXT event to the pa11yci-runner service. Resolves when
-     * the next state has been reached and the reporter event raised.
+     * Validate that a command is allowed in the given state. Throws if invalid.
      *
      * @private
-     * @returns {Promise<void>} Promise that indicates a pending state change.
      */
-    const next = () => {
-        return new Promise((resolve, reject) => {
-            pendingCommand = { resolve, reject };
-            service.send({ type: 'NEXT' });
-        });
+    const validateCommandAllowed = () => {
+        if (pendingCommand) {
+            throw new Error('runner cannot accept a command while another command is pending, await previous command');
+        }
+        if (currentContext.state === finalState) {
+            throw new Error(`runner must be reset before executing any other functions from the ${finalState} state`);
+        }
     };
 
     /**
-     * Sends the RESET event to the pa11yci-runner service. Resolves when
+     * Sends the specified event to the pa11yci-runner service. Resolves when
      * the next state has been reached and the reporter event raised.
      *
      * @private
-     * @returns {Promise<void>} Promise that indicates a pending state change.
+     * @param   {string}        event The event name to be sent.
+     * @returns {Promise<void>}       Promise that indicates a pending state change.
      */
-    const resetInternal = () => {
+    const sendEvent = (event) => {
         return new Promise((resolve, reject) => {
             pendingCommand = { resolve, reject };
-            service.send({ type: 'RESET' });
+            service.send({ type: event });
         });
     };
 
@@ -106,31 +128,53 @@ const serviceFactory = (urls, actions) => {
      * @instance
      */
     const reset = async () => {
-        await resetInternal();
+        await sendEvent('RESET');
     };
 
     /**
-     * Executes the entire Pa11y CI sequence, calling all reporter functions.
+     * Executes the next event in the Pa11y CI sequence, calling the
+     * appropriate reporter function.
      *
      * @public
      * @instance
      */
-    const runAll = async () => {
-        if (currentContext.state === finalState) {
-            throw new Error(`runner must be reset before executing any other functions from the ${finalState} state`);
+    const runNext = async () => {
+        validateCommandAllowed();
+
+        await sendEvent('NEXT');
+    };
+
+    /**
+     * Executes the entire Pa11y CI sequence, calling all reporter functions,
+     * until the specified state and optional URL are reached. If no URL is provided
+     * specified, the run completes on the first occurrence of the target state.
+     *
+     * @public
+     * @interface
+     * @param {string} targetState The target state to run to.
+     * @param {string} [targetUrl] The target URL to run to.
+     */
+    const runUntil = async (targetState, targetUrl) => {
+        validateCommandAllowed();
+        validateRunnerState(targetState);
+
+        while (!isAtTarget(currentContext, targetState, targetUrl)
+            && currentContext.state !== finalState) {
+            await sendEvent('NEXT');
         }
 
-        // Since this runs the entire sequence, send the NEXT event
-        // until the final state is reached.
-        while (currentContext.state !== finalState) {
-            // eslint-disable-next-line node/callback-return -- not a callback
-            await next();
+        // If the finalState is reached and not at target then it is
+        // not in the results and throw to indicate the command failed.
+        if (!isAtTarget(currentContext, targetState, targetUrl)) {
+            const urlString = targetUrl ? ` for targetUrl "${targetUrl}"` : '';
+            throw new Error(`targetState "${targetState}"${urlString} was not found`);
         }
     };
 
     return {
         reset,
-        runAll
+        runNext,
+        runUntil
     };
 };
 

package/lib/runner-states.js

@@ -0,0 +1,21 @@
+'use strict';
+
+/**
+ * @module runner-states
+ */
+
+/**
+ * Enum for runner states.
+ *
+ * @enum {string}
+ * @readonly
+ * @static
+ */
+const RunnerStates = Object.freeze({
+    beforeAll: 'beforeAll',
+    beginUrl: 'beginUrl',
+    urlResults: 'urlResults',
+    afterAll: 'afterAll'
+});
+
+module.exports = RunnerStates;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pa11y-ci-reporter-runner",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "Pa11y CI Reporter Runner is designed to facilitate testing of Pa11y CI reporters. Given a Pa11y CI JSON results file and optional configuration it simulates the Pa11y CI calls to the reporter.",
   "main": "index.js",
   "scripts": {
@@ -36,7 +36,7 @@
   "homepage": "https://gitlab.com/gitlab-ci-utils/pa11y-ci-reporter-runner#readme",
   "devDependencies": {
     "@aarongoldenthal/eslint-config-standard": "^12.0.2",
-    "eslint": "^8.9.0",
+    "eslint": "^8.10.0",
     "jest": "^27.5.1",
     "jest-junit": "^13.0.0",
     "markdownlint-cli": "^0.31.1",
@@ -44,6 +44,6 @@
   },
   "dependencies": {
     "lodash": "^4.17.21",
-    "xstate": "^4.30.2"
+    "xstate": "^4.30.3"
   }
 }