html-snapshots 1.3.2 → 1.4.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "html-snapshots",
-  "version": "1.3.2",
+  "version": "1.4.0",
   "author": {
     "name": "Alex Grant",
     "email": "alex@localnerve.com",
@@ -55,7 +55,7 @@
     "async": "3.2.4",
     "async-lock": "1.4.0",
     "combine-errors": "3.0.3",
-    "got": "12.5.2",
+    "got": "12.5.3",
     "lodash": "4.17.21",
     "mkdirp": "1.0.4",
     "phantomjs-prebuilt": "2.1.16",
@@ -64,7 +64,7 @@
     "xml2js": "0.4.23"
   },
   "devDependencies": {
-    "eslint": "8.26.0",
+    "eslint": "8.27.0",
     "express": "4.18.2",
     "mocha": "10.1.0",
     "c8": "7.12.0",

package/.c8rc.json

@@ -1,5 +0,0 @@
-{
-  "reports-dir": "coverage",
-  "reporter": "lcov",
-  "exclude": ["tmp", "docs", "node_modules", "test", ".github", "coverage", "examples"]
-}

package/.eslintignore

@@ -1,5 +0,0 @@
-examples/*
-docs/*
-tmp/*
-coverage/*
-test/mocha/html-snapshots/server

package/.eslintrc.json

@@ -1,16 +0,0 @@
-{
-  "parserOptions": {
-    "ecmaVersion": 2020,
-    "sourceType": "module"
-  },
-  "env": {
-    "node": true,
-    "es2020": true
-  },
-  "extends": [
-    "eslint:recommended"
-  ],
-  "rules": {
-    "no-console": 0
-  }
-}

package/.github/issue_template.md

@@ -1,16 +0,0 @@
-### Expected Behavior
-
-
-
-### Actual Behavior
-
-
-
-### Options
-```javascript
-  {
-    // Paste the html-snapshots options you used to create the problem here.
-  }
-```
-
-### Node Version

package/.github/workflows/verify.yml

@@ -1,32 +0,0 @@
-name: Verify
-on:
-  push:
-    branches: [ master ]
-  pull_request:
-    branches: [ master ]
-
-jobs:
-  verify:
-
-    runs-on: ubuntu-latest
-
-    strategy:
-      matrix:
-        node-version: [14.x, 16.x, 18.x]
-        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/
-
-    steps:
-    - uses: actions/checkout@v3
-    - name: Use Node.js ${{ matrix.node-version }}
-      uses: actions/setup-node@v3
-      with:
-        node-version: ${{ matrix.node-version }}
-    - run: npm ci
-    - name: Lint, Test, And Coverage
-      run: npm run lint && xvfb-run -a npm run test:cover
-    - name: Coverage Upload
-      if: ${{ success() }}
-      uses: coverallsapp/github-action@master
-      with:
-        github-token: ${{ secrets.GITHUB_TOKEN }}
-        path-to-lcov: ./coverage/lcov.info

package/docs/example-heroku-redis.md

@@ -1,43 +0,0 @@
-# A Real html-snapshots Usage Example
-
-> This article explains an implementation using [html-snapshots](https://github.com/localnerve/html-snapshots) in the wild \(with real, working code\). Please note the code referenced in this article was written by LocalNerve and is subject to the terms of this [license](https://github.com/localnerve/wpspa/blob/master/LICENSE.txt).  
-**NOTE:** This is an older article and usage style is html-snapshots version 0.13.2.
-
-## Summary
-The [WPSPA](http://github.com/localnerve/wpspa) project uses [html-snapshots](https://github.com/localnerve/html-snapshots) on [Heroku](https://www.heroku.com/) to populate a Redis store from which the snapshotted html is served on \_escaped\_fragment\_ requests. The [html-snapshots](https://github.com/localnerve/html-snapshots) library is run in a Heroku worker process that periodically keeps the snapshots up-to-date with a CMS.
-
-The application content and routes in [WPSPA](http://github.com/localnerve/wpspa) are dynamic, so this required me to run [html-snapshots](https://github.com/localnerve/html-snapshots) in a worker on the server to get the dynamic content updates over time.
-
-> Langauge: Javascript
-
-> Tools: NodeJS, ExpressJS, PhantomJS, Redis
-
-## Serving Html Snapshots in a Scalable Cloud Environment
-Since [WPSPA](http://github.com/localnerve/wpspa) runs on Heroku, it is serving the application from [Dynos](https://devcenter.heroku.com/articles/dynos). In essence, Dynos are billable CPU/Memory hours with independent, ephemeral file systems. Dynos can be created and destroyed at will to scale the application. Which one you get when you visit the site is up to the [Heroku routing mesh](https://devcenter.heroku.com/articles/http-routing).
-
-When a Dyno is started, it gets a copy of the files that it was deployed with in its ephemeral file system. So if your app serves dynamic content from a CMS \(as is the case with [WPSPA](http://github.com/localnerve/wpspa)\), anytime your Dyno starts you get an out-of-date version of the snapshots in the ephemeral file system. In general, managing and serving continually updated content with static html snapshot files will not work on Heroku \(or probably any scalable environment\).
-
-To see this problem further, the snapshots are being taken from a worker process running on a schedule in a short-lived worker Dyno. The worker does not have access or control of the other Dynos, nor should it. So, to make the snapshots available to web Dynos, they have to be stored in a commonly accessible area \(accessible to worker and web Dynos\), and persist when any or all of them are destroyed. For [WPSPA](http://github.com/localnerve/wpspa), I chose to store them in Redis.
-
-### ExpressJS
-The snapshots produced by [html-snapshots](https://github.com/localnerve/html-snapshots) are served by custom [ExpressJS](http://expressjs.com/) middleware. The middleware checks to see if it has received an \_escaped\_fragment\_ request. If it has, it retreives the snapshot from Redis and sends it to the client.
-
-> Middleware code: [wpspa/server/middleware/snapshots.js](https://github.com/localnerve/wpspa/blob/master/server/middleware/snapshots.js)
-
-The snapshots middleware is loaded higher in the stack, and an example of that placement \(in relation to other things\) can be found in the [WPSPA](http://github.com/localnerve/wpspa) main application file.
-
-> Application code: [wpspa/app.js](https://github.com/localnerve/wpspa/blob/master/app.js)
-
-### NodeJS Worker
-A NodeJS worker process runs periodically to update the html snapshots with content updates from the CMS. I call it the [snapshots](https://github.com/localnerve/wpspa/blob/master/server/workers/snapshots/lib/index.js) worker process. To do this, it gets the actual routes used in the application, produces the snapshots for them using [html-snapshots](https://github.com/localnerve/html-snapshots), then stores them in Redis to be served later on request by the ExpressJS middleware. Actually, the snapshots worker doesn't get the application routes itself, because that task alone is a handy, reusable thing. As such, that task was factored out into its own worker process called [routes](https://github.com/localnerve/wpspa/blob/master/server/workers/routes/lib/index.js).
-The [routes](https://github.com/localnerve/wpspa/blob/master/server/workers/routes/lib/index.js) worker process runs the [WPSPA](http://github.com/localnerve/wpspa) app on the server, waits for it to load up in PhantomJS, then asks it what it thinks its application routes are. It then stores those routes in Redis for consumption by the [snapshots](https://github.com/localnerve/wpspa/blob/master/server/workers/snapshots/lib/index.js) worker process as described above. \(The dynamic routes are also consumed by other typical web server things, like a dynamic robots.txt and sitemap.xml\).
-
-> Snapshots worker code: [snapshots](https://github.com/localnerve/wpspa/blob/master/server/workers/snapshots/lib/index.js)
-
-> Routes worker code: [routes](https://github.com/localnerve/wpspa/blob/master/server/workers/routes/lib/index.js)
-
-## Process Limit
-I eat my own dogfood. That's how I find out about things sometimes, and this project was no exception. Using [html-snapshots](https://github.com/localnerve/html-snapshots) on Heroku is how I found out that unconditionally spawning processes for each application route is a not-so-good feature to have. So a while back, I added a limiting [process model](https://github.com/localnerve/html-snapshots/tree/rc-7.1#process-model) and an option to control the size of the process pool, [processLimit](https://github.com/localnerve/html-snapshots/blob/master/README.md#process-control-options). For the [WPSPA](http://github.com/localnerve/wpspa) example on Heroku, the [processLimit is set to one](https://github.com/localnerve/wpspa/blob/16c8b5ef2dbea6593572e914f5f3693c0f9f73fc/server/config/config.json#L43) to force only one PhantomJS process to be running at once for the [snapshots](https://github.com/localnerve/wpspa/blob/master/server/workers/snapshots/lib/index.js) worker process, as I run it in a [one-off Dyno](https://devcenter.heroku.com/articles/one-off-dynos), or in the [scheduler](https://addons.heroku.com/scheduler), with a single billable Dyno available to the Heroku account.
-
-## Finally
-I hope this brief tour of a Heroku/Redis html-snapshots implementation in [WPSPA](http://github.com/localnerve/wpspa) helped you think about your html snapshot implementation needs.

package/docs/notes.md

@@ -1,42 +0,0 @@
-# Background and Other Notes
-
-## Why Does This Exist?
-The whole reason I created this library was so that I could be certain I snapshotted the page when expected content arrived (not after some arbitrary time expiration). This way, I could be certain the snapshot actually contained the content I expect.
-
-## What if I Don't Know About The Rendered Page Content?
-If you aren't interested in exact snapshots, or just want to explore because you don't know about the content detail yet, try this:
-  1. Use `"body"` as the `selector` option value (every page will probably have a body).
-  2. Set the `checkInterval` option high to give the page some time to render before it is written out. `checkInterval` is the time the phantomjs script waits to check for `selector`, so consider this "render time".
-  3. Set the `pollInterval` option higher than `checkInterval`.
-  4. Set the `timeout` option much higher to give the whole thing time to transpire. Consider `timeout` as a multiple of `pollInterval`.
-
-## What Happens If Your Content Does Not Appear
-If your content does not show up visibly in a page according to the page's selector you configured in options, you get a timeout error - not a snapshot.
-
-> That means the element you select **cannot be hidden** by any method.
-
-### Why Visible?
-The intention of this library is to take a snapshot when dynamic content has been rendered in the browser, and not before. One way to do that is to check for evidence that the element you selected has been rendered, and that's what the default scripts in this library do.
-
-## Caveats
-
-### Each Snapshot is Created in a Separate Process
-html-snapshots creates a new process for each page to snapshot, and they run in parallel.
-To control the parallel processing load, you can specify the `processLimit` option to limit the size of the process pool. In other words, you can limit the number of pages being processed in parallel at any one time. The default pool size is set to 4 PhantomJS processes at a time.
-
-### The Default Snapshot Scripts
-The default snapshot scripts are:
-  - ["default"](https://github.com/localnerve/html-snapshots/blob/master/lib/phantom/default.js)
-  - ["removeScripts"](https://github.com/localnerve/html-snapshots/blob/master/lib/phantom/removeScripts.js)
-  - ["customFilter"](https://github.com/localnerve/html-snapshots/blob/master/lib/phantom/customFilter.js)
-
-As of v0.6.x, these do not rely on jQuery unless you specify a true `useJQuery` option. If you set `useJQuery` to true \(globally or per-page\), the page being snapshotted **must load jQuery itself**.
-
-#### When Using jQuery
-When using jQuery, the default snapshot scripts use jQuery is(:visible) on your supplied selector to determine document readiness \(and then triggers the snapshot\). Last I checked, this is only available in Zepto by building in a special [selector module](https://github.com/madrobby/zepto/issues/323).
-
-#### When Not Using jQuery
-When not using jQuery \(the new default behavior, from v0.6.x\), [document.querySelector](https://developer.mozilla.org/en-US/docs/Web/API/document.querySelector) is used to select the element using your supplied selector. The element selected must be an HTMLElement, and is considered visible \(triggering a snapshot\) if both offsetWidth and offsetHeight exist.
-
-### Override the Default Snapshot Scripts
-To override the default snapshot scripts, supply a path to your own phantomjs script using the `snapshotScript` option.

package/examples/README.md

@@ -1,149 +0,0 @@
-# Examples
-
-+ [Per-page Selectors](#example---per-page-selectors-and-timeouts)
-+ [Per-page Output Paths](#example---per-page-special-output-paths)
-+ [Per-page jQuery](#example---per-page-selectors-and-jquery)
-+ [Array Input](#example---array)
-+ [More Array Input](./simple-promise/)
-+ [Array Input DRY](./html5rocks/)
-+ [Sitemap Index](./sitemap-index/)
-+ [Process Limit](./process-limit/)
-+ [Script Removal](#example---remote-robotstxt-remove-script-tags-from-html-snapshots)
-+ [Custom Filters](./custom/)
-+ [Debug PhantomJS w/Verbose Output](./verbose/)
-+ [Debug PhantomJS w/Attach](./debug-phantomjs/)
-+ [Debug Puppeteer in devtools](./debug-puppeteer/)
-
-### Example - Per page selectors and timeouts
-```javascript
-const htmlSnapshots = require('html-snapshots');
-htmlSnapshots.run({
-  input: 'sitemap',
-  source: 'https://host.domain/sitemap.xml',
-  outputDir: './snapshots',
-  outputDirClean: true,
-  selector: {
-    'https://host.domain/': '#home-content',
-    '__default': '#dynamic-content'
-  },
-  timeout: {
-    'https://host.domain/superslowpage/': 20000,
-    '__default': 10000
-  }
-})
-.then(completed => {
-  // completed is an array of full file paths to the completed snapshots.
-})
-.catch(error => {
-  // error is an Error instance.
-  // error.completed is an array of snapshot file paths that were completed.
-  // error.notCompleted is an array of file paths that did NOT complete.
-});
-```
-This reads the urls from your sitemap.xml and produces snapshots in the ./snapshots directory. In this example, a selector named "#dynamic-content" appears in all pages across the site except the home page, where "#home-content" appears \(the appearance of a selector in the output triggers the snapshot\). Finally, a default timeout of 10000 ms is set on all pages except http://https://host.domain/superslowpage/, where it waits 20000 ms.
-
-### Example - Per page special output paths
-```javascript
-const htmlSnapshots = require('html-snapshots');
-htmlSnapshots.run({
-  input: 'sitemap',
-  source: 'https://host.domain/sitemap.xml',
-  outputDir: './snapshots',
-  outputDirClean: true,
-  outputPath: {
-    'https://host.domain/services/?page=1': 'services/page/1',
-    'https://host.domain/services/?page=2': 'services/page/2'
-  },
-  selector: '#dynamic-content'
-})
-.then(completed => {
-  // completed is an array of full file paths to the completed snapshots.
-})
-.catch(error => {
-  // error is an Error instance.
-  // error.completed is an array of snapshot file paths that were completed.
-  // error.notCompleted is an array of file paths that did NOT complete.
-});
-```
-This example implies there are a couple of pages with query strings in sitemap.xml, and we don't want html-snapshots to create directories with query string characters in the names. We would also have a rewrite rule that reflects this same mapping when `_escaped_fragment_` shows up in the querystring of a request so we serve the snapshot from the appropriate directory.
-
-### Example - Per page selectors and jQuery
-```javascript
-const htmlSnapshots = require('html-snapshots');
-htmlSnapshots.run({
-  source: '/path/to/robots.txt',
-  hostname: 'mysite.com',
-  outputDir: './snapshots',
-  outputDirClean: true,
-  browser: 'phantomjs',
-  selector: {
-    '__default': '#dynamic-content',
-    '/jqpage': 'A-Selector-Not-Supported-By-querySelector'
-  },
-  useJQuery: {
-    '/jqpage': true,
-    '__default': false
-  }
-})
-.then(completed => {
-  // completed is an array of full file paths to the completed snapshots.
-})
-.catch(error => {
-  // error is an Error instance.
-  // error.completed is an array of snapshot file paths that were completed.
-  // error.notCompleted is an array of file paths that did NOT complete.
-});
-```
-This reads the urls from your robots.txt and produces snapshots in the ./snapshots directory. In this example, a selector named "#dynamic-content" appears in all pages across the site except in "/jqpage", where a selector not supported by [querySelector](https://developer.mozilla.org/en-US/docs/Web/API/document.querySelector) is used. Further, "/jqpage" loads jQuery itself \(required\). All the other pages don't need to use special selectors, so the default is set to `false`. Notice that since a robots.txt input is used, full URLs are **not** used to match selectors. Instead, paths \(and QueryStrings and any Hashes\) are used, just as specified in the robots.txt file itself.
-
-### Example - Array
-```javascript
-const htmlSnapshots = require('html-snapshots');
-htmlSnapshots.run({
-  input: 'array',
-  source: ['http://mysite.com', 'http://mysite.com/contact', 'http://mysite.com:82/special'],
-  outputDir: './snapshots',
-  outputDirClean: true,  
-  selector: '#dynamic-content'
-})
-.then(completed => {
-  // completed is an array of full file paths to the completed snapshots.
-})
-.catch(error => {
-  // error is an Error instance.
-  // error.completed is an array of snapshot file paths that were completed.
-  // error.notCompleted is an array of file paths that did NOT complete.
-});
-```
-Generates snapshots for "/", "/contact", and "/special" from mysite.com. "/special" uses port 82. All use http protocol. Array input can be powerful, check out a [simple example](/examples/simple-promise), or a more [complex example](/examples/html5rocks).
-
-### Example - Remote robots.txt, remove script tags from html snapshots
-```javascript
-const assert = require('assert');
-const fs = require('fs');
-const htmlSnapshots = require('html-snapshots');
-
-htmlSnapshots.run({
-  source: 'http://localhost/robots.txt',
-  outputDir: './snapshots',
-  outputDirClean: true,  
-  selector: '#dynamic-content',
-  snapshotScript: {
-    script: 'removeScripts'
-  }
-})
-.then(completed => {
-  completed.forEach(snapshotFile => {
-    const content = fs.readFileSync(snapshotFile, { encoding: 'utf8'});
-    assert.equal(false, /<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi.test(content));
-  });
-  // It didn't throw b/c there are no script tags in the html snapshots
-  console.log('stripped all script tags as expected');
-})
-.catch(error => {
-  // error is an Error instance.
-  // error.completed is an array of snapshot file paths that were completed.
-  // error.notCompleted is an array of file paths that did NOT complete.
-});
-```
-Removes all script tags from the output of the html snapshot. Custom filters are also supported, see the customFilter Example in the explanation of the `snapshotScript` option. Also, check out the concrete [example](/examples/custom).

package/examples/custom/README.md

@@ -1,22 +0,0 @@
-# Example
-
-> Snapshot a sample website using a sitemap and custom-filter
-
-> All examples assume you already have NodeJS installed on your local machine
-
-## How To Run
-1. Clone this repo
-2. Open a command prompt
-3. Change to the examples/custom directory
-4. Run `npm install`
-5. Run `npm start`
-
-## What It Does
-This example snapshots the entire website of the WPSPA example application using its sitemap to drive.
-
-It makes use of a [custom filter](https://github.com/localnerve/html-snapshots/blob/master/examples/custom/myFilter.js) to strip various tags \(and even replace some text\) from the output before the snapshots are stored.
-
-## Notes on Interpreting Results
-This example creates a local `tmp` directory where the snapshot output is stored.
-Inspect the output directory structure and contents.
-Compare snapshots with the original website to see what happened.

package/examples/custom/myFilter.js

@@ -1,22 +0,0 @@
-/*
- * Example custom filter for html-snapshots.
- * Copyright (c) 2013 - 2022, Alex Grant, LocalNerve, contributors
- *
- * Processes the snapshot output before it is finalized.
- */
-module.exports = function(content) {
-  var filterVersion = "1.0-20220921";
-
-  return content
-    // remove all script tags
-    .replace(/<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi, "")
-    // remove all meta tags
-    .replace(/<meta\s.*?(\/)?>/gi, "")
-    // remove all link tags
-    .replace(/<link\s.*?(\/)?>/gi, "")
-    // replace select words
-    .replace(/\bSitemaps\b/ig, "Replaced-Sitemaps")
-    // add an attribute to the body for tracking and identification
-    .replace(/<body\b/i, "<body data-snapshot-filter=\""+filterVersion+"\" ")
-    ;
-};