pagean 8.0.1 → 8.0.3

package/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019 - 2022 Aaron Goldenthal
+Copyright (c) 2019 - 2023 Aaron Goldenthal
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/README.md

@@ -118,7 +118,8 @@ The broken link test checks for broken links on the page. It checks any `<a>` ta
 
 - For links within the page, this test checks for existence of the element on the page, passing if the element exists and failing otherwise (and passing for cases that are always valid, e.g. `#` or `#top` for the current page). It does not check the visibility of the element. Failing tests return a response of "#element Not Found" (where `#element` identifies the specific element).
 - For links to other pages, the test tries to most efficiently confirm whether the target link is valid. It first makes a `HEAD` request for that URL and checks the response. If an erroneous response is returned (>= 400 with no execution error) and not code 429 (Too Many Requests), the request is retried with a `GET` request. The test passes for HTTP responses < 400 and fails otherwise (if HTTP response is >= 400 or another error occurs).
-  - This can result in false failure indications, specifically for `file://` links (`404` or `ECONNREFUSED`) or where the browser passes a domain identity with the request (page loads when tested, but `401` response for links to that page). For these cases, or other false failures, the test configuration allows a boolean `checkWithBrowser` option that will instead check links by loading the target in the browser (via `puppeteer`). Note this can increase test execution time, in some cases substantially, due to the time to open a new browser tab and plus load the page and all assets.
+  - This can result in false failure indications, specifically for `file:` links (`404` or `ECONNREFUSED`) or where the browser passes a domain identity with the request (page loads when tested, but `401` response for links to that page). For these cases, or other false failures, the test configuration allows a boolean `checkWithBrowser` option that will instead check links by loading the target in the browser (via `puppeteer`). Note this can increase test execution time, in some cases substantially, due to the time to open a new browser tab and plus load the page and all assets.
+  - Note that `file:` links can only be tested with the `checkWithBrowser` option.
   - If the link to another page includes a hash it is removed prior to checking. The test in this case is confirming a valid link, not that the element exists, which is only done for the current page.
   - The test configuration allows an `ignoredLinks` array listing link URLs to ignore for this test. Note this only applies to links to other pages, not links within the page, which are always checked.
 - To optimize performance, link test results are cached and those links are not re-tested for the entire test run (across all tested URLs). The test configuration allows a boolean `ignoreDuplicates` option that can be set to `false` to bypass this behavior and re-test all links. The results for any failed links are included in the reports in any case.
@@ -142,6 +143,8 @@ For any failing test, the `data` array in the test report includes the original
 ]
 ```
 
+Note: This test will check all links on the page, and does not respect mechanisms inteded to limit web crawlers such as `robots.txt` or `noindex` tags.
+
 ## Reports
 
 Based on the `reporters` configuration, Pagean results may be displayed in the console and saved in two reports in the project root directory (any or all of the three):
@@ -254,6 +257,10 @@ Provided with the Pagean project are Docker images configured to run the tests.
 
 **Note:** Any images in the `gitlab-ci-utils/pagean/tmp` repository are temporary images used during the build process and may be deleted at any point.
 
+### Puppeteer Cache Location
+
+In [Puppeteer v19](https://github.com/puppeteer/puppeteer/releases/tag/v19.0.0) the default cache location for installing the Chrome binary was changed from within the project's `node_modules` folder to `~/.cache/puppeteer`. To simplify execution in a container, the `PUPPETEER_CACHE_DIR` environment variable is set to install the Chrome binaries in `/home/pptruser/.cache/puppeteer` during container build, so setting to another value before execution can cause errors where Puppeteer cannot find the Chrome binary.
+
 ## GitLab CI Configuration
 
 The following is an example job from a .gitlab-ci.yml file to use this image to run Pagean against another project in GitLab CI:

package/index.js

@@ -60,7 +60,7 @@ const executeAllTests = async (config) => {
 
     if (testResults.summary.failed > 0) {
         // For test harness want process to exit with error code
-        // eslint-disable-next-line unicorn/no-process-exit, no-process-exit, no-magic-numbers -- exit code
+        // eslint-disable-next-line unicorn/no-process-exit, no-magic-numbers -- exit code
         process.exit(2);
     }
 };

package/lib/external-file-utils.js

@@ -21,9 +21,13 @@ const externalFilePath = 'pagean-external-scripts';
  * @static
  */
 const shouldSaveFile = (script, page) => {
+    const validProtocols = ['http:', 'https:'];
     const scriptUrl = new URL(script);
     const pageUrl = new URL(page);
-    return scriptUrl.host !== pageUrl.host;
+    return (
+        scriptUrl.host !== pageUrl.host &&
+        validProtocols.includes(scriptUrl.protocol)
+    );
 };
 
 /**

package/lib/link-utils.js

@@ -135,6 +135,10 @@ const checkExternalPageLinkBrowser = async (page, link) => {
  */
 // eslint-disable-next-line sonarjs/cognitive-complexity -- Allow less than 10
 const checkExternalPageLink = async (page, link, useGet = false) => {
+    if (link.startsWith('file:')) {
+        return 'Cannot check "file:" URLs';
+    }
+
     // Get user-agent from page so axios uses the same value for requesting links
     const userAgent = await page.evaluate('navigator.userAgent');
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pagean",
-  "version": "8.0.1",
+  "version": "8.0.3",
   "description": "Pagean is a web page analysis tool designed to automate tests requiring web pages to be loaded in a browser window (e.g. horizontal scrollbar, console errors)",
   "bin": {
     "pagean": "./bin/pagean.js",
@@ -51,28 +51,28 @@
   },
   "homepage": "https://gitlab.com/gitlab-ci-utils/pagean",
   "devDependencies": {
-    "@aarongoldenthal/eslint-config-standard": "^17.0.1",
-    "@aarongoldenthal/stylelint-config-standard": "^11.2.1",
+    "@aarongoldenthal/eslint-config-standard": "^20.0.0",
+    "@aarongoldenthal/stylelint-config-standard": "^12.0.1",
     "bin-tester": "^3.0.0",
-    "eslint": "^8.26.0",
-    "jest": "^29.2.2",
-    "jest-junit": "^14.0.1",
-    "markdownlint-cli": "^0.32.2",
-    "prettier": "^2.7.1",
+    "eslint": "^8.33.0",
+    "jest": "^29.4.1",
+    "jest-junit": "^15.0.0",
+    "markdownlint-cli": "^0.33.0",
+    "prettier": "^2.8.3",
     "strip-ansi": "^6.0.1",
-    "stylelint": "^14.14.1"
+    "stylelint": "^14.16.1"
   },
   "dependencies": {
-    "ajv": "^8.11.0",
+    "ajv": "^8.12.0",
     "ajv-errors": "^3.0.0",
-    "axios": "^1.1.3",
+    "axios": "^1.3.1",
     "ci-logger": "^5.1.0",
-    "commander": "^9.4.1",
+    "commander": "^10.0.0",
     "handlebars": "^4.7.7",
     "htmlhint": "^1.1.4",
     "kleur": "^4.1.5",
     "normalize-url": "^6.1.0",
     "protocolify": "^3.0.0",
-    "puppeteer": "^19.2.2"
+    "puppeteer": "^19.6.3"
   }
 }