ui5-test-runner 1.1.5 → 2.0.1

package/README.md

@@ -4,210 +4,54 @@
 [![Package Quality](https://npm.packagequality.com/shield/ui5-test-runner.svg)](https://packagequality.com/#?package=ui5-test-runner)
 [![Known Vulnerabilities](https://snyk.io/test/github/ArnaudBuchholz/ui5-test-runner/badge.svg?targetFile=package.json)](https://snyk.io/test/github/ArnaudBuchholz/ui5-test-runner?targetFile=package.json)
 [![ui5-test-runner](https://badge.fury.io/js/ui5-test-runner.svg)](https://www.npmjs.org/package/ui5-test-runner)
+[![PackagePhobia](https://img.shields.io/badge/%F0%9F%93%A6package-phobia-lightgrey)](https://packagephobia.com/result?p=ui5-test-runner)
 [![MIT License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner?ref=badge_shield)
+[![Documentation](https://img.shields.io/badge/-%F0%9F%93%9Adocumentation-blueviolet)](https://arnaudbuchholz.github.io/ui5-test-runner/)
 
 A self-sufficient test runner for UI5 applications enabling parallel execution of tests.
 
-> To put it in a nutshell, some applications have so many tests that when you run them in a browser, it ends up **crashing**. The main reason is **memory consumption** : the browser process goes up to 2 GB and it blows up. JavaScript is using garbage collecting but it needs time to operate and the stress caused by executing the tests does not let enough bandwidth for the browser to free up the memory.
+> To put it in a nutshell, some UI5 applications have so many tests that when you run them in a browser, it ends up **crashing**. The main reason is **memory consumption** : the browser process goes up to 2 GB and it blows up. JavaScript is based on garbage collecting but it needs time to operate and the stress caused by executing the tests as well as the use of iframes do not let enough bandwidth for the browser to free up the memory.
 
 > This tool is designed and built as a **substitute** of the [UI5 karma runner](https://github.com/SAP/karma-ui5). It executes all the tests in **parallel** thanks to several browser instances *(which also **reduces the total execution time**)*.
 
-## Documentation
+## 📚 [Documentation](https://arnaudbuchholz.github.io/ui5-test-runner/)
 
-* Initial concept is detailed in the  article [REserve - Testing UI5](https://arnaud-buchholz.medium.com/reserve-testing-ui5-85187d5eb7f1)
-* Tool was presented during UI5Con'21 : [A different approach to UI5 tests execution](https://youtu.be/EBp0bdIqu4s)
 
-## How to install
+## 💿 How to install
 
-* Install the [LTS version of Node.js](https://nodejs.org/en/download/)
-* `npm install -g ui5-test-runner` *(takes a while because [puppeteer](https://github.com/puppeteer/puppeteer) is big)*
+* Works with [Node.js](https://nodejs.org/en/download/) >= 14
+* Local installation
+  * `npm install --save-dev ui5-test-runner`
+  * Trigger either with `npx ui5-test-runner` or through an npm script invoking `ui5-test-runner`
+* Global installation
+  * `npm install --global ui5-test-runner`
+  * Trigger with `ui5-test-runner`
 
-## How to demo
+**NOTE** : additional packages might be needed during the execution (`puppeteer`, `selenium-webdriver`, `nyc`...) . If they are found installed **locally** in the tested project, they are used. Otherwise, they are installed **globally**.
 
-* Clone the project [training-ui5con18-opa](https://github.com/ArnaudBuchholz/training-ui5con18-opa) and run `npm install`
-* Inside the project, use `npm run karma` to test with the karma runner
-* Inside the project, use `ui5-test-runner -port:8080 -ui5:https://ui5.sap.com/1.87.0/ -cache:.ui5 -keepAlive`
-* Follow the progress of the test executions using http://localhost:8080/_/progress.html
-* When the tests are completed, inspect the results by opening :
-  - http://localhost:8080/_/report.html
-  - http://localhost:8080/_/coverage/lcov-report/index.html
-
-## How to use
-
-* Clone the project you want to test
-* If the project owns library dependencies *(other than UI5)*, you must also clone them.<br/>
-  To check for project dependencies, you may look into :
-  - `POM.xml` *(for maven based builds)* :
-  ```xml
-	<dependencies>
-		<dependency>
-			<groupId>com.sap.fiori</groupId>
-			<artifactId>my.namespace.feature.project.lib</artifactId>
-			<version>...</version>
-		</dependency>
-
-  ```
-  - `manifest.json` file :
-  ```json
-  {
-    "sap.ui5": {
-		"dependencies": {
-			"libs": {
-				"my.namespace.feature.lib": {
-					"lazy": true
-				}
-	```
-
-> The following assumes that the project and its dependencies are cloned in **the same** folder. You **must** handle the **differences** between the library **project name** / **structure** and the **namespace** it implements.
-
-* In the project root folder, run the following command :
-
-`ui5-test-runner -port:8080 -cache:.ui5 -libs:my/namespace/feature/lib/=../my.namespace.feature.project.lib/src/my/namespace/feature/lib/`
-
-The list of options is detailed below but to explain the command :
-* `-port:8080` : uses the fixed http port `8080`
-
-* `-cache:.ui5` : caches UI5 resources to boost loading of pages. It stores resources in a project folder named `.ui5` *(you may use an absolute path if preferred)*.
-
-* `-libs:my/namespace/feature/lib/=../my.namespace.feature.project.lib/src/my/namespace/feature/lib/` : maps the library path (access to URL `/resources/my/namespace/feature/lib/library.js` will be mapped to the file path `../my.namespace.feature.project.lib/src/my/namespace/feature/lib/library.js`)
-
-You may also use :
-* `-ui5:https://ui5.sap.com/1.92.1/` : uses a specific version of UI5
-
-* `-coverage:false` : **ignores**  code coverage measurement *(if you don’t need it, it speeds up a bit the startup)*
-
-* `"-args:__URL__ __REPORT__ --visible"` : changes the browser spawning command line to make the browser windows **visible**
-
-* `-parallel:3` : increases *(changes)* the number of parallel execution *(by default it uses 2)*. You may even use `0` to only serve the application *(the tests are not executed)*.
-
-* `-keepAlive` : the server remains active after executing the tests
-
-  - It is a nice way to run the tests in your own browser.<br/>For instance, open http://localhost:8080/test/unit/unitTests.qunit.html
-
-  - It might be interesting to keep it running to access the detailed report *(see below)*
-
-
-**During** the test executions *(which can take some time)* you can monitor the progress by opening : http://localhost:8080/_/progress.html
-
-  ![progress](https://raw.githubusercontent.com/ArnaudBuchholz/ui5-test-runner/main/progress.png)
-
-**After** the tests are executed :
-
-* The command line output will provide a summary of executed pages and the corresponding failures :
-
-  ![cmd_report](https://raw.githubusercontent.com/ArnaudBuchholz/ui5-test-runner/main/cmd_report.png)
+## 🖥️ How to demo
 
-* The detailed test report is available from http://localhost:8080/_/report.html *(since it uses requests to load the details, the report **must** be open through a web server, don't try to open the .html from the file system... it won't work)*
-
-  ![report](https://raw.githubusercontent.com/ArnaudBuchholz/ui5-test-runner/main/report.png)
-
-* The coverage report is available from http://localhost:8080/_/coverage/lcov-report/index.html
-
-  ![coverage](https://raw.githubusercontent.com/ArnaudBuchholz/ui5-test-runner/main/coverage.png)
-
-
-* Some folders are created to support execution, you may add them to your project `.gitignore` to exclude them from git :
-
-  - `.nyc_output/` : contains coverage information
-
-  - `report/` : contains test report *(as well as screenshots and console log outputs)*
-
-  - `.ui5/` : contains cached UI5 resources
-
-  - These folder names can be changed through parameters *(see the list below)*
-
-## Parameters
-
-| name | default | description |
-|---|---|---|
-| cwd | `process.cwd()` | Current working directory |
-| port | `0` | port to use (`0` to let REserve allocate one) |
-| ui5 | `'https://ui5.sap.com'` | UI5 url |
-| libs | | Folder(s) containing dependent libraries *(relative to `cwd`)*.<br/>Might be used multiple times, two syntaxes are supported :<ul><li>`-libs:path` adds `path` to the list of libraries, mapped directly under `/resources/`</li><li>`-libs:rel/=path` adds the `path` to the list of libraries, mapped under `/resources/rel/`</li></ul> |
-| cache | `''` | Cache UI5 resources locally in the given folder *(empty to disable)* |
-| webapp | `'webapp'` | base folder of the web application *(relative to `cwd`)* |
-| testsuite | `'test/testsuite.qunit.html'` | path / URL to the testsuite file *(relative to `webapp`)* |
-| pageFilter | `''` | [regexp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp) to select which pages to execute |
-| pageParams | `''` | Parameters added to each page URL.<br/>For instance : `'sap-ui-theme=sap_belize&sap-ui-debug=true'` |
-| pageTimeout | `0` | Limit the page execution time (ms), fails the page if it takes longer than the timeout (`0` to disable the timeout) |
-| globalTimeout | `0` | Limit the pages execution time (ms), fails the execution if it takes longer than the timeout (`0` to disable the timeout) |
-| failFast | `false` | Stops the execution after the first failing page |
-| keepAlive | `false` | Keeps the server alive *(enables debugging)* |
-| watch | `false` | Monitors the webapp folder and re-execute tests on change |
-| logServer | `false` | Logs REserve traces |
-| browser | *String, see description* | Browser instantiation command, it should point to a node.js script *(absolute or relative to `cwd`)*.<br/>By default, a script will instantiate chromium through puppetteer |
-| browserRetry | `1` | Browser instantiation retries : if the command **fails** unexpectedly, it is re-executed *(`0` means no retry)*.<br/>The page **fails** if **all attempts** fail |
-| args | `'__URL__ __REPORT__'` | Browser instantiation arguments :<ul><li>`'__URL__'` is replaced with the URL to open</li><li>`'__REPORT__'` is replaced with a folder path that is associated with the current URL *(can be used to store additional traces such as console logs or screenshots)*</li><li>`'__RETRY__'` is replaced with the retry count *(0 for the first execution, can be used to put additional traces or change strategy)*</i>*</li></ul> |
-| noScreenshot | `false` | No screenshot is taken during the tests execution (faster if the browser command supports screenshot) |
-| -- | | Parameters given right after `--` are directly added to the browser instantiation arguments *(see below)* |
-| parallel | `2` | Number of parallel tests executions (`0` to ignore tests and keep alive) |
-| tstReportDir | `'report'` | Directory to output test reports *(relative to `cwd`)* |
-| coverage | `true` | Enables code coverage |
-| covSettings | *String, see description* | Path to a custom `nyc.json` file providing settings for instrumentation *(relative to `cwd`)* |
-| covTempDir | `'.nyc_output'` | Directory to output raw coverage information to *(relative to `cwd`)* |
-| covReportDir | `'coverage'` | Where to put the coverage report files *(relative to `cwd`)* |
-| covReporters | `'lcov,cobertura'` | Comma separated list of reporters to use |
-
-These two commands are equivalent :
-
-```text
-ui5-test-runner "-args:__URL__ __REPORT__ --visible"
-ui5-test-runner -- --visible
-```
-
-### Configuration file
-
-It is also possible to set these parameters by creating a JSON file named `ui5-test-runner.json` where the **runner is executed** *(i.e. `process.cwd()`)*.
-
-The file is applied **before** parsing the command line parameters, hence some parameters might be **overridden**.
-
-If you want the parameters to be **forced** *(and not be overridden by the command line)*, prefix the parameter name with `!`.
-
-For example :
-```json
-{
-  "!pageTimeout": 900000,
-  "globalTimeout": 3600000,
-  "failFast": true
-}
-```
-
-> The `pageTimeout` setting cannot be overridden by the command line parameters
-
-**NOTE** : the `libs` parameters must be converted to an array of pairs associating `relative` URL and `source` path.
-
-For instance :
-
-```json
-{
-  "libs": [{
-    "relative": "my/namespace/feature/lib/",
-    "source": "../my.namespace.feature.project.lib/src/my/namespace/feature/lib/"
-  }]
-}
-```
+* Clone the project [training-ui5con18-opa](https://github.com/ArnaudBuchholz/training-ui5con18-opa) and run `npm install`
+* Use `npm run karma` to test with the karma runner
+* *Serving the application (a.k.a. legacy mode)*
+  * `npx ui5-test-runner --port 8081 --ui5 https://ui5.sap.com/1.109.0/ --cache .ui5 --keep-alive`
+  * Follow the progress of the test executions using http://localhost:8081/_/progress.html
+  * When the tests are completed, check the code coverage with http://localhost:8081/_/coverage/lcov-report/index.html
+* *Serving the application with `@ui5/cli`*
+  * Use `npm start` to serve the application with `@ui5/cli`
+  * `npx ui5-test-runner --port 8081 --url http://localhost:8080/test/testsuite.qunit.html --keep-alive`
+  * Follow the progress of the test executions using http://localhost:8081/_/progress.html
 
-> Structure of the `libs` parameter
 
-## Tips & tricks
+## ⚖️ License
+[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner?ref=badge_large)
 
-* The runner takes a screenshot for **every** OPA assertion (`Opa5.assert.ok`)
-* To benefit from **parallelization**, split the OPA test pages per journey.<br> An example pattern :
-  - **Declare** the list of journeys in a json file : [`AllJourneys.json`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/integration/AllJourneys.json)
-  - **Enumerate** `AllJourneys.json` in [`testsuite.qunit.html`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/testsuite.qunit.html#L17) to declare as many pages as journeys
-  - **Modify** [`AllJourneys.js`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/integration/AllJourneys.js#L26) to support a `journey=` parameter and list all declared journeys
+## ⚠️ Breaking change
 
-## Building a custom browser instantiation command
+### v2
 
-* You may follow the pattern being used by [`chromium.js`](https://github.com/ArnaudBuchholz/ui5-test-runner/blob/main/defaults/chromium.js)
-* It is **mandatory** to ensure that the child process explicitly exits at some point *(see this [thread](https://github.com/nodejs/node-v0.x-archive/issues/2605) explaining the fork behavior with Node.js)*
-* The child process will receive messages that must be handled appropriately :
-  - `message.command === 'stop'` : the browser must be closed and the command line must exit
-  - `message.command === 'capabilities'` : a message must be sent back to indicate if the following features are supported *(boolean)*
-    - `screenshot` : the browser can take screenshots (in the `__REPORT__` folder, name is provided when needed)
-    - `consoleLog` : the browser serializes the console traces (in the `__REPORT__` folder with the name `console.txt`)
-  - `message.command === 'screenshot'` : should generate a screenshot (the message contains a `filename` member). To indicate that the screenshot is done, the command line must send back the same message (`process.send(message)`).
-
-## License
-[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner?ref=badge_large)
+* Command line **parameters** as well as configuration file **syntax** have changed
+* Dependencies are installed **on demand**
+* Browser instantiation command evolved in an **incompatible way** (see [documentation](https://arnaudbuchholz.github.io/ui5-test-runner/browser.html)).
+* Output is different (report, traces)

package/index.js

@@ -2,12 +2,14 @@
 
 'use strict'
 
-const output = require('./src/output')
-const { log, serve } = require('reserve')
-const jobFactory = require('./src/job')
+const { serve } = require('reserve')
+const { fromCmdLine } = require('./src/job')
+const { getOutput } = require('./src/output')
 const reserveConfigurationFactory = require('./src/reserve')
-const executeTests = require('./src/tests')
+const { execute } = require('./src/tests')
+const { capabilities } = require('./src/capabilities')
 const { watch } = require('fs')
+const { recreateDir } = require('./src/tools')
 
 function send (message) {
   if (process.send) {
@@ -18,32 +20,48 @@ function send (message) {
   }
 }
 
-async function notifyAndExecuteTests (job) {
+async function notifyAndExecuteTests (job, output) {
   send({ msg: 'begin' })
   try {
-    await executeTests(job)
-    send({ msg: 'end', status: job.failed || 0 })
+    await execute(job)
+    let status
+    if (job.failed) {
+      status = -1
+    } else {
+      status = 0
+    }
+    send({ msg: 'end', status })
   } catch (error) {
-    output.genericError(error)
+    getOutput(job).genericError(error)
     send({ msg: 'error', error })
   }
 }
 
+let job
+let output
+
 async function main () {
-  const job = jobFactory.fromCmdLine(process.cwd(), process.argv)
+  job = fromCmdLine(process.cwd(), process.argv.slice(2))
+  output = getOutput(job)
+  await recreateDir(job.reportDir)
+  if (job.mode === 'capabilities') {
+    output.version()
+    return capabilities(job)
+  }
   const configuration = await reserveConfigurationFactory(job)
   const server = serve(configuration)
   if (job.logServer) {
-    log(server)
+    server.on('redirected', output.redirected)
   }
   server
     .on('ready', async ({ url, port }) => {
       job.port = port
       send({ msg: 'ready', port: job.port })
-      if (!job.logServer) {
-        output.serving(url)
+      output.serving(url)
+      if (job.serveOnly) {
+        return
       }
-      output.report(job)
+      output.reportOnJobProgress()
       await notifyAndExecuteTests(job)
       if (job.watch) {
         delete job.start
@@ -59,14 +77,27 @@ async function main () {
         }
       } else if (job.keepAlive) {
         job.status = 'Serving'
+      } else if (job.failed) {
+        output.stop()
+        process.exit(-1)
       } else {
-        process.exit(job.failed || 0)
+        output.stop()
+        process.exit(0)
       }
     })
     .on('error', error => {
-      output.genericError(error)
+      output.serverError(error)
       send({ msg: 'error', error })
     })
 }
 
-main().catch(reason => output.genericError(reason))
+main()
+  .catch(reason => {
+    if (output) {
+      output.genericError(reason)
+      output.stop()
+    } else if (reason.name !== 'CommanderError') {
+      console.error(reason)
+    }
+    process.exit(-1)
+  })

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ui5-test-runner",
-  "version": "1.1.5",
+  "version": "2.0.1",
   "description": "Standalone test runner for UI5",
   "main": "index.js",
   "files": [
@@ -16,8 +16,14 @@
   },
   "scripts": {
     "lint": "standard --fix",
-    "test": "jest",
-    "test:debug": "jest --runInBand"
+    "test": "npm run test:unit && npm run test:integration:jsdom && npm run test:integration:puppeteer && npm run test:integration:selenium-webdriver-chrome",
+    "test:unit": "jest",
+    "test:unit:debug": "jest --runInBand",
+    "test:integration:puppeteer": "node . --capabilities --browser $/puppeteer.js",
+    "test:integration:selenium-webdriver-chrome": "node . --capabilities --browser $/selenium-webdriver.js -- --browser chrome",
+    "test:integration:jsdom": "node . --capabilities --browser $/jsdom.js",
+    "test:report": "reserve --config ./test/report/reserve.json",
+    "build:doc": "node build/doc"
   },
   "repository": {
     "type": "git",
@@ -39,16 +45,21 @@
   },
   "homepage": "https://github.com/ArnaudBuchholz/ui5-test-runner#readme",
   "dependencies": {
+    "commander": "^10.0.0",
     "mime": "^3.0.0",
-    "nyc": "^15.1.0",
-    "puppeteer": "^19.2.0",
+    "punybind": "^1.2.1",
+    "punyexpr": "^1.0.4",
     "reserve": "^1.15.2"
   },
   "devDependencies": {
-    "jest": "^29.2.2",
-    "nock": "^13.2.9",
+    "jest": "^29.5.0",
+    "nock": "^13.3.0",
+    "nyc": "^15.1.0",
     "standard": "^17.0.0"
   },
+  "optionalDependencies": {
+    "fsevents": "^2.3.2"
+  },
   "standard": {
     "env": [
       "browser",
@@ -58,15 +69,22 @@
     ]
   },
   "jest": {
+    "testTimeout": 15000,
     "setupFilesAfterEnv": [
-      "./__mocks__/setup.js"
+      "./test/setup.js"
+    ],
+    "testPathIgnorePatterns": [
+      "/node_modules/",
+      "/capabilities/"
     ],
     "collectCoverage": true,
     "collectCoverageFrom": [
       "src/*.js"
     ],
     "coveragePathIgnorePatterns": [
-      "output\\.js"
+      "\\.spec\\.js",
+      "output\\.js",
+      "b\\capabilities\\b"
     ],
     "coverageThreshold": {
       "global": {
@@ -77,4 +95,4 @@
       }
     }
   }
-}
+}

package/src/add-test-pages.js

@@ -0,0 +1,35 @@
+'use strict'
+
+const { stop } = require('./browsers')
+const { URL } = require('url')
+
+module.exports = {
+  async addTestPages (job, url, pages) {
+    let testPageUrls
+    pages = pages.map(relativeUrl => {
+      const absoluteUrl = new URL(relativeUrl, url)
+      return absoluteUrl.toString()
+    })
+    if (job.pageFilter) {
+      const filter = new RegExp(job.pageFilter)
+      testPageUrls = pages.filter(name => name.match(filter))
+    } else {
+      testPageUrls = pages
+    }
+    if (job.pageParams) {
+      testPageUrls = testPageUrls.map(url => {
+        if (url.includes('?')) {
+          return url + '&' + job.pageParams
+        }
+        return url + '?' + job.pageParams
+      })
+    }
+    job.testPageUrls = testPageUrls.reduce((uniques, url) => {
+      if (!uniques.includes(url)) {
+        uniques.push(url)
+      }
+      return uniques
+    }, job.testPageUrls || [])
+    stop(job, url)
+  }
+}

package/src/add-test-pages.spec.js

@@ -0,0 +1,95 @@
+jest.mock('./browsers.js', () => ({
+  stop: jest.fn()
+}))
+const { stop } = require('./browsers')
+const { join } = require('path')
+const { createDir } = require('./tools')
+
+const {
+  addTestPages
+} = require('./add-test-pages')
+
+describe('src/add-test-pages', () => {
+  const url = 'http://localhost:8045/index.html'
+  const reportDir = join(__dirname, '../tmp/add-test-pages')
+  let job
+
+  beforeAll(() => createDir(reportDir))
+
+  beforeEach(() => {
+    stop.mockClear()
+    job = {
+      reportDir
+    }
+  })
+
+  describe('addTestPages', () => {
+    it('keeps track of added test pages and stops browser', async () => {
+      await addTestPages(job, url, [
+        '/page1.html',
+        'page2.html'
+      ])
+      expect(job.testPageUrls).toEqual([
+        'http://localhost:8045/page1.html',
+        'http://localhost:8045/page2.html'
+      ])
+      expect(stop).toHaveBeenCalledWith(job, url)
+    })
+
+    it('supports regexp filtering', async () => {
+      job.pageFilter = 'page(1|3)'
+      await addTestPages(job, url, [
+        '/page1.html',
+        'page2.html',
+        'page3.html'
+      ])
+      expect(job.testPageUrls).toEqual([
+        'http://localhost:8045/page1.html',
+        'http://localhost:8045/page3.html'
+      ])
+      expect(stop).toHaveBeenCalledWith(job, url)
+    })
+
+    it('supports parameters injection', async () => {
+      job.pageParams = 'a&b'
+      await addTestPages(job, url, [
+        '/page1.html',
+        'page2.html?test'
+      ])
+      expect(job.testPageUrls).toEqual([
+        'http://localhost:8045/page1.html?a&b',
+        'http://localhost:8045/page2.html?test&a&b'
+      ])
+      expect(stop).toHaveBeenCalledWith(job, url)
+    })
+
+    it('filters out duplicates', async () => {
+      await addTestPages(job, url, [
+        '/page1.html',
+        'page1.html'
+      ])
+      expect(job.testPageUrls).toEqual([
+        'http://localhost:8045/page1.html'
+      ])
+      expect(stop).toHaveBeenCalledWith(job, url)
+    })
+
+    it('supports multiple calls', async () => {
+      await addTestPages(job, url, ['/page1.html'])
+      await addTestPages(job, url, ['page1.html', '/page2.html'])
+      expect(job.testPageUrls).toEqual([
+        'http://localhost:8045/page1.html',
+        'http://localhost:8045/page2.html'
+      ])
+    })
+
+    it('supports absolute urls', async () => {
+      await addTestPages(job, url, ['/page1.html'])
+      await addTestPages(job, url, ['http://localhost:8045/page2.html'])
+      expect(job.testPageUrls).toEqual([
+        'http://localhost:8045/page1.html',
+        'http://localhost:8045/page2.html'
+      ])
+    })
+  })
+})