pa11y-ci-reporter-runner 0.5.0 → 0.7.1

package/README.md

@@ -1,6 +1,6 @@
 # Pa11y CI Reporter Runner
 
-Pa11y CI Reporter Runner is designed to facilitate testing of [Pa11y CI reporters](https://github.com/pa11y/pa11y-ci#write-a-custom-reporter). Given a Pa11y CI JSON results file and optional configuration it simulates the Pa11y CI calls to the reporter, including proper tranformation of results and configuration data. Functionally, it's a mock of the Pa11y CI side of the reporter interface.
+Pa11y CI Reporter Runner is designed to facilitate testing of [Pa11y CI reporters](https://github.com/pa11y/pa11y-ci#write-a-custom-reporter). Given a Pa11y CI JSON results file and optional configuration it simulates the Pa11y CI calls to the reporter, including proper transformation of results and configuration data. Functionally, it's an emulation of the Pa11y CI side of the reporter interface with finer control over execution.
 
 ## Installation
 
@@ -15,18 +15,52 @@ npm install pa11y-ci-reporter-runner
 Pa11y CI Reporter Runner exports a factory function that creates a reporter runner. This function has four arguments:
 
 - `resultsFileName`: Path to a Pa11y CI JSON results file.
-- `reporterName`: Name of the reporter to execute. Can be a dependency (e.g. `pa11y-ci-reporter-html`) or a path to a reporter file.
+- `reporterName`: Name of the reporter to execute. Can be an npm module (e.g. `pa11y-ci-reporter-html`) or a path to a reporter file.
 - `options`: Optional [reporter options](https://github.com/pa11y/pa11y-ci#reporter-options).
 - `config`: Optional Pa11y CI configuration file that produces the Pa11y CI JSON results.
 
-The reporter runner currently has one function:
+### Runner States
 
-- `runAll`: Simulates Pa11y CI running the complete analysis from the provided JSON results file, calling all associated reporter functions (`beforeAll`, `begin` and `results`/`error` for each URL, `afterAll`).
+Pa11y CI Reporter Runner emulates calls from Pa11y CI to the given reporter for the given results file. There are five distinct states for the runner, as shown below:
+
+```mermaid
+%% It's unfortunate that mermaid charts don't render on npmjs.com,
+%% see the README in the repository to view the flowchart
+flowchart LR;
+    init-->beforeAll;
+    beforeAll-->beginUrl;
+    beginUrl-->urlResults;
+    urlResults-->beginUrl;
+    urlResults-->afterAll;
+```
+
+- `init`: The initial state of the runner.
+- `beforeAll`: In this state the runner calls the reporter `beforeAll` function.
+- `beginUrl`: In this state the runner calls the reporter `begin` function for a given URL.
+- `urlResults`: In this state the runner calls the reporter `results` or `error` function depending on the result for that URL. If there are subsequent URLs, the runner next moves to `beginUrl` for the next URL. If this is the last URL, the runner moves to `afterAll`.
+- `afterAll`: In this state the runner calls the reporter `afterAll` function.
+
+When calling reporter functions, the runner transforms the Pa11y CI results and configuration data to provide the appropriate arguments. For example, the `results` function is called with the results as returned from Pa11y (slightly different than those returned from Pa11y CI) and the consolidated configuration for the analysis of that URL.
+
+### Runner Execution
+
+The reporter runner has four control functions:
+
+- `runAll()`: Simulates Pa11y CI running the analysis from the provided JSON results file from the current state through the end, calling all associated reporter functions.
+- `runNext()`: Simulates Pa11y CI running through the next state from the provided JSON results file, calling the associated reporter function as noted above.
+- `runUntil(targetState, targetUrl)`: Simulates Pa11y CI running the analysis from the provided JSON results file from the current state through the specified state/URL, calling the associated reporter functions as noted above. An error will be thrown if the end of the results are reached and the target was not found. This function takes the following arguments:
+  - `targetState`: The target state of the runner (any state above except `init`).
+  - `targetUrl`: An optional target URL. If no URL is specified, the runner will stop at the first instance of the target state.
+- `reset()`: Resets the runner to the `init` state. This can be sent from any state.
+
+These command are all asynchronous and must be completed before another is sent, otherwise an error will be thrown. In addition, once a run has been completed and the runner is in the `afterAll` state it must be `reset` before accepting any run command.
+
+### Example
 
 A complete example is provided below:
 
 ```js
-const createRunner = require("pa11y-ci-reporter-runner");
+const { createRunner, RunnerStates } = require("pa11y-ci-reporter-runner");
 
 const resultsFileName = "pa11yci-results.json";
 const reporterName = "../test-reporter.js";
@@ -45,15 +79,26 @@ const config = {
   ],
 };
 
-test('test reporter', async () => {
+test('test all reporter functions', async () => {
   const runner = createRunner(resultsFileName, reporterName, reporterOptions, config);
 
   await runner.runAll();
 
   // Test reporter results
 });
+
+test('test reporter at urlResults state', async () => {
+  const runner = createRunner(resultsFileName, reporterName, reporterOptions, config);
+
+  await runner.runUntil(RunnerStates.beginUrl, 'http://localhost:8080/page1-no-errors.html');
+  // The runner is now in the beginUrl state for http://localhost:8080/page1-no-errors.html
+  await runner.runNext();
+  // The runner is now in the urlResults state for http://localhost:8080/page1-no-errors.html
+
+  // Test reporter results
+});
 ```
 
 ## Limitations
 
-When passing config to `results`, `error`, and `afterAll`, Pa11y CI Reporter Runner includes the same properties as Pa11y CI except the `browser` property (with the `puppeteer` `browser` object used by Pa11y CI). If the `browser` object is needed, testing should be done with Pa11y CI to ensure proper `browser` configuration.
+When passing config to `results`, `error`, and `afterAll`, Pa11y CI Reporter Runner includes the same properties as Pa11y CI except the `browser` property (with the `puppeteer` `browser` object used by Pa11y CI). If the `browser` object is needed, testing should be done with Pa11y CI to ensure proper `browser` capabilities are available.

package/index.js

@@ -1,7 +1,7 @@
 'use strict';
 
 /**
- * Pa11y CI Reporter Runner allows a pa11y-ci reporter to be run
+ * Pa11y CI Reporter Runner allows a Pa11y CI reporter to be run
  * from a JSON results file without using pa11y-ci.
  *
  * @module pa11y-ci-reporter-runner
@@ -9,9 +9,18 @@
 
 const fs = require('fs');
 const formatter = require('./lib/formatter');
-const reporterBuilder = require('./lib/reporterBuilder');
+const reporterBuilder = require('./lib/reporter-builder');
 const createConfig = require('./lib/config');
+const { serviceFactory } = require('./lib/pa11yci-service');
+const RunnerStates = require('./lib/runner-states');
 
+/**
+ * Loads the Pa11y CI results from JSON file.
+ *
+ * @private
+ * @param   {string} fileName Pa11y CI JSON results file name.
+ * @returns {object}          Pa11y CI results.
+ */
 const loadPa11yciResults = fileName => {
     if (typeof fileName !== 'string') {
         throw new TypeError('fileName must be a string');
@@ -21,20 +30,17 @@ const loadPa11yciResults = fileName => {
         const results = JSON.parse(fs.readFileSync(fileName, 'utf8'));
         return formatter.convertJsonToResultsObject(results);
     }
-    catch (err) {
-        throw new Error(`Error loading results file - ${err.message}`);
+    catch (error) {
+        throw new Error(`Error loading results file - ${error.message}`);
     }
 };
 
-const isError = results => results.length === 1 && results[0] instanceof Error;
-
 /**
  * Pa11y CI configuration allows config.urls entries to be either url strings
  * or object with url and other configuration, so this return a list of just
  * the url strings.
  *
  * @private
- * @static
  * @param   {object[]} values The urls array from configuration.
  * @returns {string[]}        Array of URL strings.
  */
@@ -55,13 +61,12 @@ const getUrlList = values => {
 };
 
 /**
- * Compares URLs in Pa1y CI results and configuration files to ensure
+ * Compares URLs in Pa11y CI results and configuration files to ensure
  * consistency (i.e. The same URLs are in both lists, although they
  * may not be in the same order). URLs in config are only checked if
  * provided. If specified and inconsistent, throws error.
  *
  * @private
- * @static
  * @param   {object} results Pa11y CI JSON results file.
  * @param   {object} config  Pa11y CI configuration.
  */
@@ -78,6 +83,15 @@ const validateUrls = (results, config) => {
     }
 };
 
+/**
+ * Check if the given Pa11y results are an execution error.
+ *
+ * @private
+ * @param   {object}  results Pa11y results for a single URL.
+ * @returns {boolean}         True if the results are an execution error.
+ */
+const isError = results => results.length === 1 && results[0] instanceof Error;
+
 /**
  * Factory to create a pa11y-ci reporter runner that can execute
  * a reporter with the specified pa11y-ci JSON results file.
@@ -90,6 +104,7 @@ const validateUrls = (results, config) => {
  * @param   {object} config          The Pa11y CI configuration.
  * @returns {object}                 A Pa11y CI reporter runner.
  */
+// eslint-disable-next-line max-lines-per-function
 const createRunner = (resultsFileName, reporterName, options = {}, config = {}) => {
     const pa11yciResults = loadPa11yciResults(resultsFileName);
 
@@ -97,29 +112,115 @@ const createRunner = (resultsFileName, reporterName, options = {}, config = {})
 
     const pa11yciConfig = createConfig(config);
     const reporter = reporterBuilder.buildReporter(reporterName, options, pa11yciConfig.defaults);
+    const urls = config.urls || Object.keys(pa11yciResults.results);
+
+    /**
+     * Implements the runner beforeAll event, calling reporter.beforeAll.
+     *
+     * @private
+     */
+    const beforeAll = async () => {
+        await reporter.beforeAll(urls);
+    };
 
-    const runAll = async () => {
-        await reporter.beforeAll(config.urls || Object.keys(pa11yciResults.results));
-
-        for (const url of Object.keys(pa11yciResults.results)) {
-            await reporter.begin(url);
-
-            const urlResults = pa11yciResults.results[url];
-            const urlConfig = pa11yciConfig.getConfigForUrl(url);
-            if (isError(urlResults)) {
-                await reporter.error(urlResults[0], url, urlConfig);
-            }
-            else {
-                await reporter.results(formatter.getPa11yResultsFromPa11yCiResults(url, pa11yciResults), urlConfig);
-            }
-        }
+    /**
+     * Implements the runner beginUrl event, calling reporter.begin.
+     *
+     * @private
+     * @param {string} url The url being analyzed.
+     */
+    const beginUrl = async (url) => {
+        await reporter.begin(url);
+    };
+
+    /**
+     * Implements the runner urlResults event, calling reporter.results or
+     * reporter.error as appropriate based on the results.
+     *
+     * @private
+     * @param {string} url The url being analyzed.
+     */
+    const urlResults = async (url) => {
+        const results = pa11yciResults.results[url];
+        const urlConfig = pa11yciConfig.getConfigForUrl(url);
+        await (isError(results)
+            ? reporter.error(results[0], url, urlConfig)
+            : reporter.results(formatter.getPa11yResultsFromPa11yCiResults(url, pa11yciResults), urlConfig));
+    };
 
+    /**
+     * Implements the runner afterAll event, calling reporter.afterAll.
+     *
+     * @private
+     */
+    const afterAll = async () => {
         await reporter.afterAll(pa11yciResults, pa11yciConfig.defaults);
     };
 
+    // Collection of runner actions to be passed to the state service.
+    const actions = {
+        beforeAll,
+        beginUrl,
+        urlResults,
+        afterAll
+    };
+
+    // URLs for the service is always the array of result URLs since they
+    // are used to retrieve the results.
+    const service = serviceFactory(Object.keys(pa11yciResults.results), actions);
+
+    /**
+     * Resets the runner to the initial state.
+     *
+     * @public
+     * @instance
+     */
+    const reset = async () => {
+        await service.reset();
+    };
+
+    /**
+     * Executes the entire Pa11y CI sequence, calling all reporter functions.
+     *
+     * @public
+     * @instance
+     */
+    const runAll = async () => {
+        await service.runUntil(RunnerStates.afterAll);
+    };
+
+    /**
+     * Executes the next event in the Pa11y CI sequence, calling the
+     * appropriate reporter function.
+     *
+     * @public
+     * @instance
+     */
+    const runNext = async () => {
+        await service.runNext();
+    };
+
+    /**
+     * Executes the entire Pa11y CI sequence, calling all reporter functions,
+     * until the specified state and optional URL are reached. If no URL is provided
+     * specified, the run completes on the first occurrence of the target state.
+     *
+     * @public
+     * @interface
+     * @param {string} targetState The target state to run to.
+     * @param {string} [targetUrl] The target URL to run to.
+     */
+    const runUntil = async (targetState, targetUrl) => {
+        await service.runUntil(targetState, targetUrl);
+    };
+
     return {
-        runAll
+        reset,
+        runAll,
+        runNext,
+        runUntil
     };
 };
 
-module.exports = createRunner;
+module.exports.createRunner = createRunner;
+module.exports.RunnerStates = RunnerStates;

package/lib/config.js

@@ -20,7 +20,7 @@ const normalizeConfig = (config) => {
 };
 
 /**
- * Factory function that returns a config object for managing PA11y CI
+ * Factory function that returns a config object for managing Pa11y CI
  * configuration including defaults and determining the consolidated
  * configuration for each URL.
  *

package/lib/formatter.js

@@ -6,6 +6,9 @@
  * @module formatter
  */
 
+const isError = issues => issues.length === 1 && Object.keys(issues[0]).length === 1
+    && issues[0].message;
+
 /**
  * Converts Pa11y CI JSON output to an equivalent Pa11y CI object,
  * allowing JSON result files to be used for reporter interface
@@ -25,13 +28,7 @@ const convertJsonToResultsObject = jsonResults => {
 
     for (const url of Object.keys(jsonResults.results)) {
         const issues = jsonResults.results[url];
-        let formattedIssues;
-        if (issues.length === 1 && Object.keys(issues[0]).length === 1 && issues[0].message) {
-            formattedIssues = [new Error(issues[0].message)];
-        }
-        else {
-            formattedIssues = issues;
-        }
+        const formattedIssues = isError(issues) ? [new Error(issues[0].message)] : issues;
         results.results[url] = formattedIssues;
     }
 
@@ -63,4 +60,3 @@ const getPa11yResultsFromPa11yCiResults = (url, results) => {
 
 module.exports.convertJsonToResultsObject = convertJsonToResultsObject;
 module.exports.getPa11yResultsFromPa11yCiResults = getPa11yResultsFromPa11yCiResults;
-

package/lib/pa11yci-machine.js

@@ -0,0 +1,68 @@
+'use strict';
+
+const { createMachine, assign } = require('xstate');
+
+const pa11yciMachine = createMachine(
+    {
+        id: 'pa11yci-runner',
+        initial: 'init',
+        context: {
+            urlIndex: 0,
+            urls: []
+        },
+        states: {
+            init: {
+                entry: 'setInitialUrlIndex',
+                on: {
+                    NEXT: { target: 'beforeAll' }
+                }
+            },
+            beforeAll: {
+                on: {
+                    NEXT: [
+                        { target: 'beginUrl', cond: 'hasUrls' },
+                        { target: 'afterAll' }
+                    ],
+                    RESET: { target: 'init' }
+                }
+            },
+            beginUrl: {
+                on: {
+                    NEXT: { target: 'urlResults' },
+                    RESET: { target: 'init' }
+                }
+            },
+            urlResults: {
+                on: {
+                    NEXT: [
+                        { target: 'afterAll', cond: 'isLastUrl' },
+                        { target: 'beginUrl', actions: 'incrementUrl' }
+                    ],
+                    RESET: { target: 'init' }
+                }
+            },
+            afterAll: {
+                on: {
+                    RESET: { target: 'init' }
+                }
+            }
+        }
+    },
+    {
+        actions: {
+            incrementUrl: assign({
+                urlIndex: (context) => context.urlIndex + 1
+            }),
+            setInitialUrlIndex: assign({
+                urlIndex: () => 0
+            })
+        },
+        guards: {
+            hasUrls: (context) => context.urls.length > 0,
+            isLastUrl: (context) =>
+                context.urlIndex === context.urls.length - 1
+        }
+    }
+);
+
+module.exports = pa11yciMachine;

package/lib/pa11yci-service.js

@@ -0,0 +1,183 @@
+'use strict';
+
+const { interpret } = require('xstate');
+const machine = require('./pa11yci-machine');
+
+const RunnerStates = require('./runner-states');
+const finalState = RunnerStates.afterAll;
+
+/**
+ * Gets the initial pa11yci-runner state machine context given an array of URLs.
+ *
+ * @private
+ * @param   {Array}  urls Array of URLs.
+ * @returns {object}      The initial state machine context.
+ */
+const getInitialContext = urls => ({
+    urlIndex: 0,
+    urls
+});
+
+/**
+ * Checks the state to determine whether it has an associated URL.
+ *
+ * @private
+ * @param   {RunnerStates} state The state to check.
+ * @returns {boolean}            True if the state has an associated url.
+ */
+const hasUrl = state => state === RunnerStates.beginUrl || state === RunnerStates.urlResults;
+
+/**
+ * Validates that the given state is a valid RunnerStates value. Throws if not.
+ *
+ * @private
+ * @param {string} state The state to validate.
+ */
+const validateRunnerState = (state) => {
+    if (!Object.keys(RunnerStates).includes(state)) {
+        throw new TypeError(`targetState "${state}" invalid`);
+    }
+};
+
+/**
+ * Checks if the context matches the targetState and either no
+ * targetUrl was specified or the context matches the targetUrl.
+ *
+ * @private
+ * @param   {object}  context     The current state machine context.
+ * @param   {string}  targetState The target runner state.
+ * @param   {string}  [targetUrl] The target URL.
+ * @returns {boolean}             True if the context matches the target, otherwise false.
+ */
+const isAtTarget = (context, targetState, targetUrl) => {
+    return context.state === targetState && (!targetUrl || context.url === targetUrl);
+};
+
+/**
+ * Factory function that return a pa11yci-runner service.
+ *
+ * @public
+ * @static
+ * @param   {Array}  urls    Array of URLs.
+ * @param   {object} actions Actions object with functions to execute for each event.
+ * @returns {object}         A pa11yci-runner service.
+ */
+// eslint-disable-next-line max-lines-per-function
+const serviceFactory = (urls, actions) => {
+    let pendingCommand;
+    const currentContext = {
+        state: undefined,
+        url: undefined
+    };
+
+    // Create service from pa11yci-machine with context, setup
+    // transition event handler, and start service
+    const service = interpret(machine.withContext(getInitialContext(urls)));
+    service.onTransition(async (state) => {
+        // Save the current state and url for use in manual transitions.
+        // The state machine only increments the url index, so use that
+        // to get the url from the original urls array.
+        currentContext.state = state.value;
+        currentContext.url = hasUrl(currentContext.state)
+            ? urls[state.context.urlIndex] : undefined;
+
+        if (pendingCommand) {
+            // Do not take any action in init state
+            if (currentContext.state !== 'init') {
+                await actions[currentContext.state](currentContext.url);
+            }
+            pendingCommand.resolve();
+            pendingCommand = undefined;
+        }
+    });
+    service.start();
+
+    /**
+     * Validate that a command is allowed in the given state. Throws if invalid.
+     *
+     * @private
+     */
+    const validateCommandAllowed = () => {
+        if (pendingCommand) {
+            throw new Error('runner cannot accept a command while another command is pending, await previous command');
+        }
+        if (currentContext.state === finalState) {
+            throw new Error(`runner must be reset before executing any other functions from the ${finalState} state`);
+        }
+    };
+
+    /**
+     * Sends the specified event to the pa11yci-runner service. Resolves when
+     * the next state has been reached and the reporter event raised.
+     *
+     * @private
+     * @param   {string}        event The event name to be sent.
+     * @returns {Promise<void>}       Promise that indicates a pending state change.
+     */
+    const sendEvent = (event) => {
+        return new Promise((resolve, reject) => {
+            pendingCommand = { resolve, reject };
+            service.send({ type: event });
+        });
+    };
+
+    /**
+     * Resets the service to the init state.
+     *
+     * @public
+     * @instance
+     */
+    const reset = async () => {
+        await sendEvent('RESET');
+    };
+
+    /**
+     * Executes the next event in the Pa11y CI sequence, calling the
+     * appropriate reporter function.
+     *
+     * @public
+     * @instance
+     */
+    const runNext = async () => {
+        validateCommandAllowed();
+
+        await sendEvent('NEXT');
+    };
+
+    /**
+     * Executes the entire Pa11y CI sequence, calling all reporter functions,
+     * until the specified state and optional URL are reached. If no URL is provided
+     * specified, the run completes on the first occurrence of the target state.
+     *
+     * @public
+     * @interface
+     * @param {string} targetState The target state to run to.
+     * @param {string} [targetUrl] The target URL to run to.
+     */
+    const runUntil = async (targetState, targetUrl) => {
+        validateCommandAllowed();
+        validateRunnerState(targetState);
+
+        while (!isAtTarget(currentContext, targetState, targetUrl)
+            && currentContext.state !== finalState) {
+            await sendEvent('NEXT');
+        }
+
+        // If the finalState is reached and not at target then it is
+        // not in the results and throw to indicate the command failed.
+        if (!isAtTarget(currentContext, targetState, targetUrl)) {
+            const urlString = targetUrl ? ` for targetUrl "${targetUrl}"` : '';
+            throw new Error(`targetState "${targetState}"${urlString} was not found`);
+        }
+    };
+
+    return {
+        reset,
+        runNext,
+        runUntil
+    };
+};
+
+module.exports = {
+    serviceFactory
+};

package/lib/{reporterBuilder.js → reporter-builder.js}

@@ -27,6 +27,7 @@ const loadReporter = reporterName => {
  * @param   {object} config          The Pa11y CI configuration.
  * @returns {object}                 The reporter object.
  */
+// eslint-disable-next-line sonarjs/cognitive-complexity -- allow < 10
 const buildReporter = (reporterName, options, config) => {
     if (typeof reporterName !== 'string') {
         throw new TypeError('reporterName must be a string');
@@ -37,15 +38,15 @@ const buildReporter = (reporterName, options, config) => {
         if (typeof reporter === 'function') {
             reporter = reporter(options, config);
         }
-        reporterMethods.forEach(method => {
+        for (const method of reporterMethods) {
             if (typeof reporter[method] !== 'function') {
                 reporter[method] = noop;
             }
-        });
+        }
         return reporter;
     }
-    catch (err) {
-        throw new Error(`Error loading reporter: ${err.message}`);
+    catch (error) {
+        throw new Error(`Error loading reporter: ${error.message}`);
     }
 };
 

package/lib/runner-states.js

@@ -0,0 +1,21 @@
+'use strict';
+
+/**
+ * @module runner-states
+ */
+
+/**
+ * Enum for runner states.
+ *
+ * @enum {string}
+ * @readonly
+ * @static
+ */
+const RunnerStates = Object.freeze({
+    beforeAll: 'beforeAll',
+    beginUrl: 'beginUrl',
+    urlResults: 'urlResults',
+    afterAll: 'afterAll'
+});
+
+module.exports = RunnerStates;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pa11y-ci-reporter-runner",
-  "version": "0.5.0",
+  "version": "0.7.1",
   "description": "Pa11y CI Reporter Runner is designed to facilitate testing of Pa11y CI reporters. Given a Pa11y CI JSON results file and optional configuration it simulates the Pa11y CI calls to the reporter.",
   "main": "index.js",
   "scripts": {
@@ -35,14 +35,15 @@
   },
   "homepage": "https://gitlab.com/gitlab-ci-utils/pa11y-ci-reporter-runner#readme",
   "devDependencies": {
-    "@aarongoldenthal/eslint-config-standard": "^11.0.0",
-    "eslint": "^8.8.0",
-    "jest": "^27.4.7",
+    "@aarongoldenthal/eslint-config-standard": "^12.0.2",
+    "eslint": "^8.10.0",
+    "jest": "^27.5.1",
     "jest-junit": "^13.0.0",
-    "markdownlint-cli": "^0.30.0",
+    "markdownlint-cli": "^0.31.1",
     "pa11y-ci-reporter-cli-summary": "^1.0.1"
   },
   "dependencies": {
-    "lodash": "^4.17.21"
+    "lodash": "^4.17.21",
+    "xstate": "^4.30.5"
   }
 }