pagean 10.2.0 → 11.0.0

package/README.md

@@ -45,79 +45,96 @@ a headless Chrome browser. The URLs defined in the configuration file are each
 loaded once, and after page load the applicable tests are executed. Test
 results are `passed` or `failed`, but can be configured to report `warning`
 instead of failure. Only a `failed` test causes the test process to fail and
-exit with an error code (a `warning` doesn't).
+exit with an error code (a `warning` does not). If a page URL fails to load,
+it is retried up to two additional times and if unsuccessful the URL is
+logged as a `page error` with the error message.
 
-### Horizontal scrollbar test
+### Broken link test
 
-The horizontal scrollbar test fails if the rendered page has a horizontal
-scrollbar. If a specific browser viewport size is desired for this test, that
-can be configured in the `puppeteerLaunchOptions`.
+The broken link test checks for broken links on the page. It checks any `<a>`
+tag on the page with `href` pointing to another location on the current page or
+another page (that is, only `http(s)` or `file` protocols).
 
-### Console output test
+- For links within the page, this test checks for existence of the element on
+  the page, passing if the element exists and failing otherwise (and passing
+  for cases that are always valid, for example `#` or `#top` for the current
+  page). It doesn't check the visibility of the element. Failing tests return a
+  response of "#element Not Found" (where `#element` identifies the specific
+  element).
+- For links to other pages, the test tries to most efficiently confirm whether
+  the target link is valid. It first makes a `HEAD` request for that URL and
+  checks the response. If an erroneous response is returned (>= 400 with no
+  execution error) and not code 429 (Too Many Requests), the request is retried
+  with a `GET` request. The test passes for HTTP responses < 400 and fails
+  otherwise (if HTTP response is >= 400 or another error occurs).
+  - This can result in false failure indications, specifically for `file:`
+    links (`404` or `ECONNREFUSED`) or where the browser passes a domain
+    identity with the request (page loads when tested, but `401` response for
+    links to that page). For these cases, or other false failures, the test
+    configuration allows a Boolean `checkWithBrowser` option that instead
+    checks links by loading the target in the browser (via `puppeteer`). Note
+    this can increase test execution time, in some cases substantially, due to
+    the time to open a new browser tab and plus load the page and all assets.
+  - Note that `file:` links can only be tested with the `checkWithBrowser`
+    option.
+  - If the link to another page includes a hash it's removed prior to checking.
+    The test in this case is confirming a valid link, not that the element
+    exists, which is only done for the current page.
+  - The test configuration allows an `ignoredLinks` array listing link URLs to
+    ignore for this test. Note this only applies to links to other pages, not
+    links within the page, which are always checked.
+- To optimize performance, link test results are cached and those links aren't
+  re-tested for the entire test run (across all tested URLs). The test
+  configuration allows a Boolean `ignoreDuplicates` option that can be set to
+  `false` to bypass this behavior and re-test all links. The results for any
+  failed links are included in the reports in any case.
 
-The console output test fails if any output is written to the browser console.
-An array is included in the report with all entries, as shown below:
+For any failing test, the `data` array in the test report includes the original
+URL and the response code or error as shown below.
 
 ```json
 [
   {
-    "type": "error",
-    "text": "Failed to load resource: net::ERR_NAME_NOT_RESOLVED",
-    "location": {
-      "url": "https://this.url.does.not.exist/file.js"
-    }
+    "href": "https://about.gitlab.com/not-found",
+    "status": 404
+  },
+  {
+    "href": "http://localhost:3000/brokenLinks.html#notlinked",
+    "status": "#notlinked Not Found"
+  },
+  {
+    "href": "https://this.url.does.not.exist/",
+    "status": "ENOTFOUND"
   }
 ]
 ```
 
+Note: this test checks all links on the page, and doesn't respect mechanisms
+intended to limit web crawlers such as `robots.txt` or `noindex` tags.
+
 ### Console error test
 
 The console error test fails if any error is written to the browser console,
-but is otherwise the same as the console output test. This separation allows
-for testing for console errors, but allowing any other console output.
+but is otherwise simply a subset of the console output test. This separation
+allows for testing for console errors, but allowing any other console output.
 
-### Rendered HTML test
+### Console output test
 
-The rendered HTML test is intended for cases where content is dynamically
-created prior to page load (that is, the `load` event firing). The rendered
-HTML is returned and checked with
-[HTML Hint](https://www.npmjs.com/package/htmlhint) and the test fails if any
-issues are found. An array is included in the report with all HTML Hint issues,
-as shown below:
+The console output test fails if any output is written to the browser console.
+An array is included in the report with all entries, as shown below:
 
 ```json
 [
   {
-    "col": 9,
-    "evidence": "    <div id=\"div1\"></div>",
-    "line": 6,
-    "message": "The id value [ div1 ] must be unique.",
-    "raw": " id=\"div1\"",
-    "rule": {
-      "description": "The value of id attributes must be unique.",
-      "id": "id-unique",
-      "link": "https://github.com/thedaviddias/HTMLHint/wiki/id-unique"
-    },
-    "type": "error"
+    "type": "error",
+    "text": "Failed to load resource: net::ERR_NAME_NOT_RESOLVED",
+    "location": {
+      "url": "https://this.url.does.not.exist/file.js"
+    }
   }
 ]
 ```
 
-An htmlhintrc file can be specified in the configuration file, otherwise the
-default "./.htmlhintrc" file is used (if it exists). See the Configuration
-section below.
-
-Note: this test may not find some errors in the original HTML that are
-removed/resolved as the page is parsed (for example closing tags with no
-opening tags).
-
-### Page load time test
-
-The page load time test fails if the page load time (from start through the
-`load` event) exceeds the defined threshold in the configuration file (or the
-default of 2 seconds). The actual load time is included in the report. Tests
-time out at twice the page load time threshold.
-
 ### External script test
 
 The external script test is intended to identify any externally loaded
@@ -157,68 +174,53 @@ saved filename or applicable error, as shown below.
 Each external script is saved only once, but is reported on any page where it's
 referenced.
 
-### Broken link test
+### Horizontal scrollbar test
 
-The broken link test checks for broken links on the page. It checks any `<a>`
-tag on the page with `href` pointing to another location on the current page or
-another page (that is, only `http(s)` or `file` protocols).
+The horizontal scrollbar test fails if the rendered page has a horizontal
+scrollbar. If a specific browser viewport size is desired for this test, that
+can be configured in the `puppeteerLaunchOptions`.
 
-- For links within the page, this test checks for existence of the element on
-  the page, passing if the element exists and failing otherwise (and passing
-  for cases that are always valid, for example `#` or `#top` for the current
-  page). It doesn't check the visibility of the element. Failing tests return a
-  response of "#element Not Found" (where `#element` identifies the specific
-  element).
-- For links to other pages, the test tries to most efficiently confirm whether
-  the target link is valid. It first makes a `HEAD` request for that URL and
-  checks the response. If an erroneous response is returned (>= 400 with no
-  execution error) and not code 429 (Too Many Requests), the request is retried
-  with a `GET` request. The test passes for HTTP responses < 400 and fails
-  otherwise (if HTTP response is >= 400 or another error occurs).
-  - This can result in false failure indications, specifically for `file:`
-    links (`404` or `ECONNREFUSED`) or where the browser passes a domain
-    identity with the request (page loads when tested, but `401` response for
-    links to that page). For these cases, or other false failures, the test
-    configuration allows a Boolean `checkWithBrowser` option that instead
-    checks links by loading the target in the browser (via `puppeteer`). Note
-    this can increase test execution time, in some cases substantially, due to
-    the time to open a new browser tab and plus load the page and all assets.
-  - Note that `file:` links can only be tested with the `checkWithBrowser`
-    option.
-  - If the link to another page includes a hash it's removed prior to checking.
-    The test in this case is confirming a valid link, not that the element
-    exists, which is only done for the current page.
-  - The test configuration allows an `ignoredLinks` array listing link URLs to
-    ignore for this test. Note this only applies to links to other pages, not
-    links within the page, which are always checked.
-- To optimize performance, link test results are cached and those links aren't
-  re-tested for the entire test run (across all tested URLs). The test
-  configuration allows a Boolean `ignoreDuplicates` option that can be set to
-  `false` to bypass this behavior and re-test all links. The results for any
-  failed links are included in the reports in any case.
+### Page load time test
 
-For any failing test, the `data` array in the test report includes the original
-URL and the response code or error as shown below.
+The page load time test fails if the page load time (from start through the
+`load` event) exceeds the defined threshold in the configuration file (or the
+default of 2 seconds). The actual load time is included in the report. Tests
+time out at twice the page load time threshold.
+
+### Rendered HTML test
+
+The rendered HTML test is intended for cases where content is dynamically
+created prior to page load (that is, the `load` event firing). The rendered
+HTML is returned and checked with
+[HTML Hint](https://www.npmjs.com/package/htmlhint) and the test fails if any
+issues are found. An array is included in the report with all HTML Hint issues,
+as shown below:
 
 ```json
 [
   {
-    "href": "https://about.gitlab.com/not-found",
-    "status": 404
-  },
-  {
-    "href": "http://localhost:3000/brokenLinks.html#notlinked",
-    "status": "#notlinked Not Found"
-  },
-  {
-    "href": "https://this.url.does.not.exist/",
-    "status": "ENOTFOUND"
+    "col": 9,
+    "evidence": "    <div id=\"div1\"></div>",
+    "line": 6,
+    "message": "The id value [ div1 ] must be unique.",
+    "raw": " id=\"div1\"",
+    "rule": {
+      "description": "The value of id attributes must be unique.",
+      "id": "id-unique",
+      "link": "https://github.com/thedaviddias/HTMLHint/wiki/id-unique"
+    },
+    "type": "error"
   }
 ]
 ```
 
-Note: this test checks all links on the page, and doesn't respect mechanisms
-intended to limit web crawlers such as `robots.txt` or `noindex` tags.
+An htmlhintrc file can be specified in the configuration file, otherwise the
+default "./.htmlhintrc" file is used (if it exists). See the Configuration
+section below.
+
+Note: this test may not find some errors in the original HTML that are
+removed/resolved as the page is parsed (for example closing tags with no
+opening tags).
 
 ## Reports
 
@@ -227,14 +229,15 @@ console and saved in two reports in the project root directory (any or all of
 the three):
 
 - A JSON report named
-  [`pagean-results.json`](https://gitlab-ci-utils.gitlab.io/pagean/pagean-results.json)
+  [`pagean-results.json`](https://gitlab-ci-utils.gitlab.io/pagean/pagean-results.json).
 - An HTML report named
-  [`pagean-results.html`](https://gitlab-ci-utils.gitlab.io/pagean/pagean-results.html)
+  [`pagean-results.html`](https://gitlab-ci-utils.gitlab.io/pagean/pagean-results.html).
 
 Both reports contain:
 
-- The time of test execution
-- A summary of the total tests and results (passed, warning, failed)
+- The time of test execution.
+- A summary of the total tests and results (passed, warning, failed, and page
+  errors).
 - The detailed test results, including the URL tested, list of tests performed
   on that URL with results, and, if applicable, any relevant data associated
   with the test failure (for example the console errors if the console error
@@ -256,12 +259,11 @@ Below is an example `.pageanrc.json` file, which is broken into seven major
 properties:
 
 - `htmlhintrc`: An optional path to an htmlhintrc file to be used in the
-  rendered HTML test
+  rendered HTML test.
 - `project`: An optional name of the project, which is included in HTML and
   JSON reports.
 - `puppeteerLaunchOptions`: An optional set of options to pass to Puppeteer on
-  launch. There are no default options. The complete list of available options
-  can be found at
+  launch. The complete list of available options can be found at
   https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions.
 - `reporters`: An optional array of reporters indicating the test reports that
   should be provided. There are three possible options - `cli`, `html`, and
@@ -302,8 +304,6 @@ is equivalent to the longhand:
 }
 ```
 
-All available settings with the default values are shown below.
-
 - `sitemap`: Specify a sitemap with URLs to test. If a sitemap is specified,
   the URLs from the sitemap are added to the `urls` array. If a URL is in the
   `urls` array with `settings`, those settings are retained. Note that
@@ -327,14 +327,23 @@ All available settings with the default values are shown below.
   testing that URL. The `url` string can be either an actual URL or a local
   file, as shown in the example below.
 
+The following shows all available settings, except `sitemap`, with the default
+values.
+
 ```json
 {
   "puppeteerLaunchOptions": {
-    "args": ["--no-sandbox"]
+    "headless": "new"
   },
   "reporters": ["cli", "html", "json"],
   "settings": {
-    "horizontalScrollbarTest": {
+    "brokenLinkTest": {
+      "enabled": true,
+      "failWarn": false,
+      "checkWithBrowser": false,
+      "ignoreDuplicates": true
+    },
+    "consoleErrorTest": {
       "enabled": true,
       "failWarn": false
     },
@@ -342,11 +351,11 @@ All available settings with the default values are shown below.
       "enabled": true,
       "failWarn": false
     },
-    "consoleErrorTest": {
+    "externalScriptTest": {
       "enabled": true,
-      "failWarn": false
+      "failWarn": true
     },
-    "renderedHtmlTest": {
+    "horizontalScrollbarTest": {
       "enabled": true,
       "failWarn": false
     },
@@ -355,35 +364,23 @@ All available settings with the default values are shown below.
       "failWarn": false,
       "pageLoadTimeThreshold": 2
     },
-    "externalScriptTest": {
-      "enabled": true,
-      "failWarn": true
-    },
-    "brokenLinkTest": {
+    "renderedHtmlTest": {
       "enabled": true,
-      "failWarn": false,
-      "checkWithBrowser": false,
-      "ignoreDuplicates": true,
-      "ignoredLinks": []
-    }
-  },
-  "urls": [
-    "https://gitlab.com/gitlab-ci-utils/pagean/",
-    {
-      "url": "./tests/fixtures/site/consoleLog.html",
-      "settings": {
-        "consoleOutputTest": false
-      }
+      "failWarn": false
     }
-  ]
+  }
 }
 ```
 
+Numerous example config files used in the tests can be found
+[here](tests/fixtures/configs/).
+
 ## Container images
 
 Provided with the Pagean project are container images configured to run the
-tests. All available image tags can be found in the `gitlab-ci-utils/pagean`
-repository at https://gitlab.com/gitlab-ci-utils/pagean/container_registry.
+tests. All available image tags can be found in the
+`registry.gitlab.com/gitlab-ci-utils/pagean` repository
+[here](https://gitlab.com/gitlab-ci-utils/pagean/container_registry).
 Details on each release can be found on the
 [Releases](https://gitlab.com/gitlab-ci-utils/pagean/releases) page.
 
@@ -419,7 +416,7 @@ pagean:
       - pagean-external-scripts/
 ```
 
-### Testing with static HTTP server
+### Testing with a static HTTP server
 
 The container image shown previously includes
 [`serve`](https://www.npmjs.com/package/serve) and

package/bin/pagean.js

@@ -1,11 +1,11 @@
 #!/usr/bin/env node
-'use strict';
 
-const { log, Levels } = require('ci-logger');
-const { program } = require('commander');
-const pagean = require('../index');
-const getConfig = require('../lib/config');
-const pkg = require('../package.json');
+import { Levels, log } from 'ci-logger';
+import { program } from 'commander';
+
+import { executeAllTests } from '../index.js';
+import getConfig from '../lib/config.js';
+import { pkg } from '../lib/utils.js';
 
 const defaultConfigFileName = './.pageanrc.json';
 
@@ -19,19 +19,14 @@ program
     .parse(process.argv);
 const options = program.opts();
 
-getConfig(options.config)
-    /* eslint-disable-next-line promise/always-return -- required instead
-       of top level await */
-    .then((config) => {
-        pagean.executeAllTests(config);
-    })
-    /* eslint-disable-next-line promise/prefer-await-to-callbacks --
-       required instead of top level await */
-    .catch((error) => {
-        log({
-            errorCode: 1,
-            exitOnError: true,
-            level: Levels.Error,
-            message: `Error executing pagean tests\n${error.message}`
-        });
+try {
+    const config = await getConfig(options.config);
+    executeAllTests(config);
+} catch (error) {
+    log({
+        errorCode: 1,
+        exitOnError: true,
+        level: Levels.Error,
+        message: `Error executing pagean tests\n${error.message}`
     });
+}

package/bin/pageanrc-lint.js

@@ -1,12 +1,12 @@
 #!/usr/bin/env node
-'use strict';
 
-const { log, Levels } = require('ci-logger');
-const { program } = require('commander');
-const { green, underline } = require('kleur');
-const { lintConfigFile } = require('../lib/config');
-const { formatErrors } = require('../lib/schema-errors');
-const pkg = require('../package.json');
+import { Levels, log } from 'ci-logger';
+import kleur from 'kleur';
+import { program } from 'commander';
+
+import { formatErrors } from '../lib/schema-errors.js';
+import { lintConfigFile } from '../lib/config.js';
+import { pkg } from '../lib/utils.js';
 
 const defaultConfigFileName = './.pageanrc.json';
 
@@ -34,9 +34,9 @@ const outputJsonResults = (configFileName, lintResults) => {
 
 const outputConsoleResults = (configFileName, lintResults) => {
     if (lintResults.isValid) {
-        log({ message: `\n${configFileName}: ${green('valid')}\n` });
+        log({ message: `\n${configFileName}: ${kleur.green('valid')}\n` });
     } else {
-        logError(`\n${underline(configFileName)}`, false);
+        logError(`\n${kleur.underline(configFileName)}`, false);
         for (const error of formatErrors(lintResults.errors)) {
             logError(error, false);
         }

package/index.js

@@ -1,38 +1,43 @@
-'use strict';
-
 /**
  * Pagean test framework.
  *
  * @module pagean
  */
-const puppeteer = require('puppeteer');
 
-const testLogger = require('./lib/logger');
-const testReporter = require('./lib/reporter');
-const { ...testFunctions } = require('./lib/tests');
-const { createLinkChecker } = require('./lib/link-utils');
+import puppeteer from 'puppeteer';
+
+import * as testFunctions from './lib/tests.js';
+import { createLinkChecker } from './lib/link-utils.js';
+import { saveReports } from './lib/reporter.js';
+import testLogger from './lib/logger.js';
 
 /**
  * @typedef  {object} PageOptions
  * @property {number} retryCount  - The number of times to retry navigation.
  * @property {number} timeout     - The timeout for navigation.
  */
+
+/** @type {PageOptions} */
 const defaultPageOptions = {
     retryCount: 2,
     timeout: 60_000
 };
 
+/** @typedef {import('puppeteer').Page} Page */
+
 /**
  * Opens the specified URL in the given page, with retry logic.
  *
- * @param {string}      page    The puppeteer page object.
- * @param {string}      url     The URL to navigate to.
- * @param {PageOptions} options The options for navigating to the URL.
+ * @param  {Page}        page    The puppeteer page object.
+ * @param  {string}      url     The URL to navigate to.
+ * @param  {PageOptions} options The options for navigating to the URL.
+ * @throws {Error}               If the navigation fails.
  * @static
  * @private
  */
 const gotoPage = async (page, url, options) => {
     let retryCount = 0;
+    let lastError;
     while (retryCount < options.retryCount) {
         try {
             /* eslint-disable no-await-in-loop -- loop for retry only */
@@ -44,24 +49,33 @@ const gotoPage = async (page, url, options) => {
             });
             /* eslint-enable no-await-in-loop -- loop for retry only */
             break;
-        } catch {
+        } catch (error) {
+            lastError = error;
             retryCount++;
-            console.error(
-                `Failed to navigate to ${url}. Retrying (${retryCount}/${options.retryCount})...`
-            );
         }
     }
+
+    // Normalize URLs for comparison, prevents cases like trailing slashes
+    // from causing false negatives.
+    const desiredUrl = new URL(url);
+    const currentUrl = new URL(page.url());
+    if (currentUrl.href !== desiredUrl.href) {
+        throw lastError;
+    }
 };
 
+/** @typedef {import('./lib/config.js').PageanConfig} PageanConfig */
+
 /* eslint-disable no-await-in-loop, max-lines-per-function --
    no-await-in-loop - tests are deliberately performed synchronously,
    max-lines-per-function - allowed to test executor */
 /**
  * Executes Pagean tests as specified in config and reports results.
  *
- * @param {object} config The Pagean test configuration.
+ * @param {PageanConfig} config The Pagean test configuration.
  * @static
  */
+// eslint-disable-next-line sonarjs/cognitive-complexity -- Allow < 10
 const executeAllTests = async (config) => {
     const logger = testLogger(config);
     const linkChecker = createLinkChecker();
@@ -79,7 +93,15 @@ const executeAllTests = async (config) => {
                 type: message.type()
             })
         );
-        await gotoPage(page, testUrl.url, defaultPageOptions);
+
+        try {
+            await gotoPage(page, testUrl.url, defaultPageOptions);
+        } catch (error) {
+            logger.logPageError(error.message);
+            /* eslint-disable-next-line no-continue -- tests should not be
+               executed if navigation fails */
+            continue;
+        }
 
         const testContext = {
             consoleLog,
@@ -99,9 +121,9 @@ const executeAllTests = async (config) => {
     }
     await browser.close();
     const testResults = logger.getTestResults();
-    testReporter.saveReports(testResults, config.reporters);
+    saveReports(testResults, config.reporters);
 
-    if (testResults.summary.failed > 0) {
+    if (testResults.summary.failed > 0 || testResults.summary.pageError > 0) {
         // For test harness want process to exit with error code
         // eslint-disable-next-line unicorn/no-process-exit, no-magic-numbers -- exit code
         process.exit(2);
@@ -109,4 +131,4 @@ const executeAllTests = async (config) => {
 };
 /* eslint-enable no-await-in-loop, max-lines-per-function -- enable */
 
-module.exports.executeAllTests = executeAllTests;
+export { executeAllTests };