ui5-test-runner 1.0.7 → 1.1.3

package/README.md

@@ -9,15 +9,15 @@
 [![MIT License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner?ref=badge_shield)
 
-A test runner for UI5 applications enabling parallel execution of tests.
+A self-sufficient test runner for UI5 applications enabling parallel execution of tests.
 
-> To put it in a nutshell, some applications have so many tests that when you run them in a browser, it ends up **crashing**. The main reason is **memory consumption** : the browser process goes up to 2 GB and it crashes. JavaScript is using garbage collecting but it needs time to operate and the stress caused by executing the tests does not let enough bandwidth for the browser to free up the memory.
+> To put it in a nutshell, some applications have so many tests that when you run them in a browser, it ends up **crashing**. The main reason is **memory consumption** : the browser process goes up to 2 GB and it blows up. JavaScript is using garbage collecting but it needs time to operate and the stress caused by executing the tests does not let enough bandwidth for the browser to free up the memory.
 
 > This tool is designed and built as a **substitute** of the [UI5 karma runner](https://github.com/SAP/karma-ui5). It executes all the tests in **parallel** thanks to several browser instances *(which also **reduces the total execution time**)*.
 
 ## Documentation
 
-* Concept is detailed in the  article [REserve - Testing UI5](https://arnaud-buchholz.medium.com/reserve-testing-ui5-85187d5eb7f1)
+* Initial concept is detailed in the  article [REserve - Testing UI5](https://arnaud-buchholz.medium.com/reserve-testing-ui5-85187d5eb7f1)
 * Tool was presented during UI5Con'21 : [A different approach to UI5 tests execution](https://youtu.be/EBp0bdIqu4s)
 
 ## How to install
@@ -29,8 +29,11 @@ A test runner for UI5 applications enabling parallel execution of tests.
 
 * Clone the project [training-ui5con18-opa](https://github.com/ArnaudBuchholz/training-ui5con18-opa) and run `npm install`
 * Inside the project, use `npm run karma` to test with the karma runner
-* Inside the project, use `ui5-test-runner -port:8080 -ui5:https://ui5.sap.com/1.87.0/ -cache:.ui5`
-* You may follow the progress of the test executions using http://localhost:8080/_/progress.html
+* Inside the project, use `ui5-test-runner -port:8080 -ui5:https://ui5.sap.com/1.87.0/ -cache:.ui5 -keepAlive`
+* Follow the progress of the test executions using http://localhost:8080/_/progress.html
+* When the tests are completed, inspect the results by opening :
+  - http://localhost:8080/_/report.html
+  - http://localhost:8080/_/coverage/lcov-report/index.html
 
 ## How to use
 
@@ -69,7 +72,7 @@ The list of options is detailed below but to explain the command :
 
 * `-cache:.ui5` : caches UI5 resources to boost loading of pages. It stores resources in a project folder named `.ui5` *(you may use an absolute path if preferred)*.
 
-* `-libs:my/namespace/feature/lib/=../my.namespace.feature.project.lib/src/my/namespace/feature/lib/` : maps the library path (access to URL `/resources/my/namespace/feature/lib/library.js` will be mapped to the file path `./my.namespace.feature.project.lib/src/my/namespace/feature/lib/library.js`)
+* `-libs:my/namespace/feature/lib/=../my.namespace.feature.project.lib/src/my/namespace/feature/lib/` : maps the library path (access to URL `/resources/my/namespace/feature/lib/library.js` will be mapped to the file path `../my.namespace.feature.project.lib/src/my/namespace/feature/lib/library.js`)
 
 You may also use :
 * `-ui5:https://ui5.sap.com/1.92.1/` : uses a specific version of UI5
@@ -78,7 +81,7 @@ You may also use :
 
 * `"-args:__URL__ __REPORT__ --visible"` : changes the browser spawning command line to make the browser windows **visible**
 
-* `-parallel:3` : increases *(changes)* the number of parallel execution *(by default it uses 2)*. You may even use `0` to only serve the application *(but the tests are not executed)*.
+* `-parallel:3` : increases *(changes)* the number of parallel execution *(by default it uses 2)*. You may even use `0` to only serve the application *(the tests are not executed)*.
 
 * `-keepAlive` : the server remains active after executing the tests
 
@@ -126,6 +129,7 @@ You may also use :
 | libs | | Folder(s) containing dependent libraries *(relative to `cwd`)*.<br/>Might be used multiple times, two syntaxes are supported :<ul><li>`-libs:path` adds `path` to the list of libraries, mapped directly under `/resources/`</li><li>`-libs:rel/=path` adds the `path` to the list of libraries, mapped under `/resources/rel/`</li></ul> |
 | cache | `''` | Cache UI5 resources locally in the given folder *(empty to disable)* |
 | webapp | `'webapp'` | base folder of the web application *(relative to `cwd`)* |
+| testsuite | `'test/testsuite.qunit.html'` | path / URL to the testsuite file *(relative to `webapp`)* |
 | pageFilter | `''` | [regexp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp) to select which pages to execute |
 | pageParams | `''` | Parameters added to each page URL.<br/>For instance : `'sap-ui-theme=sap_belize&sap-ui-debug=true'` |
 | pageTimeout | `0` | Limit the page execution time (ms), fails the page if it takes longer than the timeout (`0` to disable the timeout) |
@@ -137,6 +141,7 @@ You may also use :
 | browser | *String, see description* | Browser instantiation command, it should point to a node.js script *(absolute or relative to `cwd`)*.<br/>By default, a script will instantiate chromium through puppetteer |
 | browserRetry | `1` | Browser instantiation retries : if the command **fails** unexpectedly, it is re-executed *(`0` means no retry)*.<br/>The page **fails** if **all attempts** fail |
 | args | `'__URL__ __REPORT__'` | Browser instantiation arguments :<ul><li>`'__URL__'` is replaced with the URL to open</li><li>`'__REPORT__'` is replaced with a folder path that is associated with the current URL *(can be used to store additional traces such as console logs or screenshots)*</li><li>`'__RETRY__'` is replaced with the retry count *(0 for the first execution, can be used to put additional traces or change strategy)*</i>*</li></ul> |
+| noScreenshot | `false` | No screenshot is taken during the tests execution (faster if the browser command supports screenshot) |
 | -- | | Parameters given right after `--` are directly added to the browser instantiation arguments *(see below)* |
 | parallel | `2` | Number of parallel tests executions (`0` to ignore tests and keep alive) |
 | tstReportDir | `'report'` | Directory to output test reports *(relative to `cwd`)* |
@@ -187,10 +192,24 @@ For instance :
 
 > Structure of the `libs` parameter
 
+## Tips & tricks
+
+* The runner takes a screenshot for **every** OPA assertion (`Opa5.assert.ok`)
+* To benefit from **parallelization**, split the OPA test pages per journey.<br> An example pattern :
+  - **Declare** the list of journeys in a json file : [`AllJourneys.json`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/integration/AllJourneys.json)
+  - **Enumerate** `AllJourneys.json` in [`testsuite.qunit.html`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/testsuite.qunit.html#L17) to declare as many pages as journeys
+  - **Modify** [`AllJourneys.js`](https://github.com/ArnaudBuchholz/training-ui5con18-opa/blob/master/webapp/test/integration/AllJourneys.js#L26) to support a `journey=` parameter and list all declared journeys
+
 ## Building a custom browser instantiation command
 
 * You may follow the pattern being used by [`chromium.js`](https://github.com/ArnaudBuchholz/ui5-test-runner/blob/main/defaults/chromium.js)
 * It is **mandatory** to ensure that the child process explicitly exits at some point *(see this [thread](https://github.com/nodejs/node-v0.x-archive/issues/2605) explaining the fork behavior with Node.js)*
+* The child process will receive messages that must be handled appropriately :
+  - `message.command === 'stop'` : the browser must be closed and the command line must exit
+  - `message.command === 'capabilities'` : a message must be sent back to indicate if the following features are supported *(boolean)*
+    - `screenshot` : the browser can take screenshots (in the `__REPORT__` folder, name is provided when needed)
+    - `consoleLog` : the browser serializes the console traces (in the `__REPORT__` folder with the name `console.txt`)
+  - `message.command === 'screenshot'` : should generate a screenshot (the message contains a `filename` member). To indicate that the screenshot is done, the command line must send back the same message (`process.send(message)`).
 
 ## License
 [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FArnaudBuchholz%2Fui5-test-runner?ref=badge_large)

package/defaults/chromium.js

@@ -16,20 +16,35 @@ if (reportDir) {
 }
 
 process.on('message', async message => {
-  if (message.command === 'stop') {
-    if (reportDir && page) {
-      await page.screenshot({ path: join(reportDir, 'screenshot.png') })
+  try {
+    if (message.command === 'stop') {
+      await browser.close()
+      process.exit(0)
+    } else if (message.command === 'screenshot') {
+      if (reportDir && page) {
+        await page.screenshot({ path: join(reportDir, message.filename) })
+        process.send(message)
+      }
+    } else if (message.command === 'capabilities') {
+      process.send({
+        command: 'capabilities',
+        screenshot: true,
+        consoleLog: true
+      })
     }
-    await browser.close()
-    process.exit(0)
+  } catch (e) {
+    console.error(e)
+    process.exit(-2)
   }
 })
 
 async function main () {
   browser = await puppeteer.launch({
     headless,
+    defaultViewport: null,
     args: [
       url,
+      '--start-maximized',
       '--no-sandbox',
       '--disable-gpu',
       '--disable-extensions'

package/index.js

@@ -2,6 +2,7 @@
 
 'use strict'
 
+const output = require('./src/output')
 const { log, serve } = require('reserve')
 const jobFactory = require('./src/job')
 const reserveConfigurationFactory = require('./src/reserve')
@@ -23,7 +24,7 @@ async function notifyAndExecuteTests (job) {
     await executeTests(job)
     send({ msg: 'end', status: job.failed || 0 })
   } catch (error) {
-    console.error('ERROR', error)
+    output.genericError(error)
     send({ msg: 'error', error })
   }
 }
@@ -40,15 +41,16 @@ async function main () {
       job.port = port
       send({ msg: 'ready', port: job.port })
       if (!job.logServer) {
-        console.log(`Server running at ${url}`)
+        output.serving(url)
       }
+      output.report(job)
       await notifyAndExecuteTests(job)
       if (job.watch) {
         delete job.start
         if (!job.watching) {
-          console.log('Watching changes on', job.webapp)
+          output.watching(job.webapp)
           watch(job.webapp, { recursive: true }, (eventType, filename) => {
-            console.log(eventType, filename)
+            output.changeDetected(eventType, filename)
             if (!job.start) {
               notifyAndExecuteTests(job)
             }
@@ -57,15 +59,14 @@ async function main () {
         }
       } else if (job.keepAlive) {
         job.status = 'Serving'
-        console.log('Keeping alive.')
       } else {
         process.exit(job.failed || 0)
       }
     })
     .on('error', error => {
-      console.error('ERROR', error)
+      output.genericError(error)
       send({ msg: 'error', error })
     })
 }
 
-main()
+main().catch(reason => output.genericError(reason))

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ui5-test-runner",
-  "version": "1.0.7",
+  "version": "1.1.3",
   "description": "Standalone test runner for UI5",
   "main": "index.js",
   "files": [
@@ -39,16 +39,15 @@
   },
   "homepage": "https://github.com/ArnaudBuchholz/ui5-test-runner#readme",
   "dependencies": {
-    "colors": "^1.4.0",
-    "mime": "^2.5.2",
+    "mime": "^3.0.0",
     "nyc": "^15.1.0",
-    "puppeteer": "^10.2.0",
-    "reserve": "^1.11.7"
+    "puppeteer": "^11.0.0",
+    "reserve": "^1.12.1"
   },
   "devDependencies": {
-    "jest": "^27.1.0",
-    "nock": "^13.1.3",
-    "standard": "^16.0.3"
+    "jest": "^27.3.1",
+    "nock": "^13.2.1",
+    "standard": "^16.0.4"
   },
   "standard": {
     "env": [
@@ -59,10 +58,16 @@
     ]
   },
   "jest": {
+    "setupFilesAfterEnv": [
+      "./__mocks__/setup.js"
+    ],
     "collectCoverage": true,
     "collectCoverageFrom": [
       "src/*.js"
     ],
+    "coveragePathIgnorePatterns": [
+      "output\\.js"
+    ],
     "coverageThreshold": {
       "global": {
         "branches": 80,

package/src/browsers.js

@@ -4,12 +4,16 @@ const { fork } = require('child_process')
 const { join } = require('path')
 const { recreateDir, filename } = require('./tools')
 const { getPageTimeout } = require('./timeout')
+const output = require('./output')
+
+let lastScreenshotId = 0
+const screenshots = {}
 
 function start (job, relativeUrl) {
   if (!job.browsers) {
     job.browsers = {}
   }
-  console.log('>>', relativeUrl)
+  output.browserStart(relativeUrl)
   const reportDir = join(job.tstReportDir, filename(relativeUrl))
   const args = job.args.split(' ')
     .map(arg => arg
@@ -28,35 +32,72 @@ function start (job, relativeUrl) {
   job.browsers[relativeUrl] = pageBrowser
   run(job, pageBrowser)
   return promise.then(() => {
-    console.log('<<', relativeUrl)
+    output.browserStopped(relativeUrl)
   })
 }
 
 async function run (job, pageBrowser) {
   const { relativeUrl } = pageBrowser
   if (pageBrowser.retry) {
-    console.log('>> RETRY', pageBrowser.retry, relativeUrl)
+    output.browserRetry(relativeUrl, pageBrowser.retry)
   }
   await recreateDir(pageBrowser.reportDir)
   delete pageBrowser.stopped
-  const childProcess = fork(job.browser, pageBrowser.args.map(arg => arg.replace('__RETRY__', pageBrowser.retry)), { stdio: 'inherit' })
+  const childProcess = fork(job.browser, pageBrowser.args.map(arg => arg.replace('__RETRY__', pageBrowser.retry)), { stdio: 'pipe' })
+  output.monitor(childProcess)
   pageBrowser.childProcess = childProcess
   const timeout = getPageTimeout(job)
   if (timeout) {
     pageBrowser.timeoutId = setTimeout(() => {
-      console.log('!! TIMEOUT', relativeUrl)
+      output.browserTimeout(relativeUrl)
       stop(job, relativeUrl)
     }, timeout)
   }
+  childProcess.on('message', message => {
+    if (message.command === 'screenshot') {
+      const { id } = message
+      screenshots[id]()
+      delete screenshots[id]
+    } else /* istanbul ignore else */ if (message.command === 'capabilities') {
+      job.browserCapabilities = { ...message }
+      delete job.browserCapabilities.command
+      output.browserCapabilities(job.browserCapabilities)
+    }
+  })
   childProcess.on('close', () => {
     if (!pageBrowser.stopped) {
-      console.log('!! BROWSER CLOSED', relativeUrl)
+      output.browserClosed(relativeUrl)
       stop(job, relativeUrl, true)
     }
   })
+  if (!job.browserCapabilities) {
+    childProcess.send({ command: 'capabilities' })
+  }
+}
+
+async function screenshot (job, relativeUrl, filename) {
+  if (job.noScreenshot || !job.browserCapabilities || !job.browserCapabilities.screenshot) {
+    return
+  }
+  const pageBrowser = job.browsers[relativeUrl]
+  if (pageBrowser) {
+    const { childProcess } = pageBrowser
+    if (childProcess.connected) {
+      const id = ++lastScreenshotId
+      const promise = new Promise(resolve => {
+        screenshots[id] = resolve
+      })
+      childProcess.send({
+        id,
+        command: 'screenshot',
+        filename
+      })
+      await promise
+    }
+  }
 }
 
-function stop (job, relativeUrl, retry = false) {
+async function stop (job, relativeUrl, retry = false) {
   const pageBrowser = job.browsers[relativeUrl]
   if (pageBrowser) {
     pageBrowser.stopped = true
@@ -76,4 +117,4 @@ function stop (job, relativeUrl, retry = false) {
   }
 }
 
-module.exports = { start, stop }
+module.exports = { start, screenshot, stop }

package/src/coverage.js

@@ -5,14 +5,14 @@ const { fork } = require('child_process')
 const { cleanDir, createDir } = require('./tools')
 const { readdir, readFile, stat, writeFile } = require('fs').promises
 const { Readable } = require('stream')
+const output = require('./output')
 
 const nycScript = require.resolve('nyc/bin/nyc.js')
 
 function nyc (...args) {
-  console.log('nyc', ...args)
-  const childProcess = fork(nycScript, args, {
-    stdio: 'inherit'
-  })
+  output.nyc(...args)
+  const childProcess = fork(nycScript, args, { stdio: 'pipe' })
+  output.monitor(childProcess)
   let done
   const promise = new Promise(resolve => { done = resolve })
   childProcess.on('close', done)
@@ -44,6 +44,15 @@ async function instrument (job) {
   await createDir(join(job.covTempDir, 'settings'))
   const settings = JSON.parse((await readFile(job.covSettings)).toString())
   settings.cwd = job.cwd
+  if (!settings.exclude) {
+    settings.exclude = []
+  }
+  settings.exclude.push(join(job.covTempDir, '**'))
+  if (job.cache) {
+    settings.exclude.push(join(job.cache, '**'))
+  }
+  settings.exclude.push(join(job.tstReportDir, '**'))
+  settings.exclude.push(join(job.covReportDir, '**'))
   await writeFile(job.nycSettingsPath, JSON.stringify(settings))
   await nyc('instrument', job.webapp, join(job.covTempDir, 'instrumented'), '--nycrc-path', job.nycSettingsPath)
 }

package/src/endpoints.js

@@ -2,29 +2,51 @@
 
 const { join } = require('path')
 const { body } = require('reserve')
-const { stop } = require('./browsers')
+const { screenshot, stop } = require('./browsers')
 const { writeFile } = require('fs').promises
 const { extractUrl, filename } = require('./tools')
 const { Request, Response } = require('reserve')
+const output = require('./output')
 
 module.exports = job => {
+  async function endpointImpl (implementation, request) {
+    const url = extractUrl(request.headers)
+    const data = JSON.parse(await body(request))
+    if (job.parallel === -1) {
+      output.endpoint(url, data)
+    }
+    try {
+      await implementation.call(this, url, data)
+    } catch (e) {
+      output.endpointError(url, data, e)
+    }
+  }
+
+  function synchronousEndpoint (implementation) {
+    return async function (request, response) {
+      await endpointImpl(implementation, request)
+      response.writeHead(200)
+      response.end()
+    }
+  }
+
   function endpoint (implementation) {
     return async function (request, response) {
       response.writeHead(200)
       response.end()
-      const url = extractUrl(request.headers)
-      const data = JSON.parse(await body(request))
-      if (job.parallel === -1) {
-        console.log(url, data)
-      }
-      try {
-        await implementation.call(this, url, data)
-      } catch (e) {
-        console.error(`Exception when processing ${url}`)
-        console.error(data)
-        console.error(e)
+      await endpointImpl(implementation, request)
+    }
+  }
+
+  function getPageTest (page, testId) {
+    const { tests, order } = page
+    if (!tests[testId]) {
+      tests[testId] = {
+        timestamps: []
       }
+      order.push(testId)
     }
+    return tests[testId]
   }
 
   return job.parallel
@@ -94,41 +116,60 @@ module.exports = job => {
       // Endpoint to receive QUnit.begin
         match: '^/_/QUnit/begin',
         custom: endpoint((url, details) => {
-          job.testPages[url] = {
+          const page = {
+            isOpa: details.isOpa,
             total: details.totalTests,
             failed: 0,
             passed: 0,
-            tests: []
+            tests: {},
+            order: []
+          }
+          details.modules.forEach(module => {
+            module.tests.forEach(test => getPageTest(page, test.testId))
+          })
+          job.testPages[url] = page
+        })
+      }, {
+      // Endpoint to receive QUnit.log
+        match: '^/_/QUnit/log',
+        custom: synchronousEndpoint(async (url, report) => {
+          const page = job.testPages[url]
+          if (page.isOpa) {
+            const { testId, runtime } = report
+            getPageTest(page, testId).timestamps.push(runtime)
+            await screenshot(job, url, `${testId}-${runtime}.png`)
           }
         })
       }, {
       // Endpoint to receive QUnit.testDone
         match: '^/_/QUnit/testDone',
-        custom: endpoint((url, report) => {
+        custom: synchronousEndpoint(async (url, report) => {
           const page = job.testPages[url]
+          const { testId } = report
           if (report.failed) {
+            await screenshot(job, url, `${testId}.png`)
             job.failed = true
             ++page.failed
           } else {
             ++page.passed
           }
-          page.tests.push(report)
+          getPageTest(page, testId).report = report
         })
       }, {
       // Endpoint to receive QUnit.done
         match: '^/_/QUnit/done',
-        custom: endpoint((url, report) => {
-          let promise = Promise.resolve()
+        custom: endpoint(async (url, report) => {
           const page = job.testPages[url]
           if (page) {
+            await screenshot(job, url, 'screenshot.png')
             if (report.__coverage__) {
               const coverageFileName = join(job.covTempDir, `${filename(url)}.json`)
-              promise = writeFile(coverageFileName, JSON.stringify(report.__coverage__))
+              await writeFile(coverageFileName, JSON.stringify(report.__coverage__))
               delete report.__coverage__
             }
             page.report = report
           }
-          promise.then(() => stop(job, url))
+          stop(job, url)
         })
       }, {
       // UI to follow progress
@@ -139,14 +180,17 @@ module.exports = job => {
         match: '^/_/progress',
         custom: async (request, response) => {
           const json = JSON.stringify(job, (key, value) => {
-            if (key === 'tests' && Array.isArray(value)) {
+            if (((key === 'tests' || key === 'browsers') && typeof value === 'object') ||
+                (key === 'order' && Array.isArray(value))
+            ) {
               return undefined // Filter out
             }
             return value
           })
+          const length = (new TextEncoder().encode(json)).length
           response.writeHead(200, {
             'Content-Type': 'application/json',
-            'Content-Length': json.length
+            'Content-Length': length
           })
           response.end(json)
         }

package/src/inject/qunit-hooks.js

@@ -2,24 +2,73 @@
 (function () {
   'use strict'
 
+  if (window['ui5-test-runner/qunit-hooks']) {
+    return // already installed
+  }
+  window['ui5-test-runner/qunit-hooks'] = true
+
   function post (url, data) {
-    const xhr = new XMLHttpRequest()
-    xhr.open('POST', '/_/' + url)
-    xhr.send(JSON.stringify(data))
+    return new Promise(function (resolve, reject) {
+      const xhr = new XMLHttpRequest()
+      xhr.open('POST', '/_/' + url)
+      xhr.send(JSON.stringify(data))
+      xhr.onreadystatechange = function () {
+        if (xhr.readyState === 4) {
+          if (xhr.status === 200) {
+            resolve(xhr.responseText)
+          } else {
+            reject(xhr.statusText)
+          }
+        }
+      }
+    })
+  }
+
+  function isOpa () {
+    try {
+      return !!window.sap.ui.test.Opa5
+    } catch (e) {
+      return false
+    }
   }
 
   QUnit.begin(function (details) {
-    post('QUnit/begin', details)
+    details.isOpa = isOpa()
+    return post('QUnit/begin', details)
+  })
+
+  QUnit.log(function (report) {
+    let ready = false
+    const log = {
+      testId: report.testId,
+      runtime: report.runtime
+    }
+    post('QUnit/log', log)
+      .then(undefined, function () {
+        console.error('Failed to POST to QUnit/log (no timestamp)', log)
+      })
+      .then(function () {
+        ready = true
+      })
+    if (isOpa()) {
+      window.sap.ui.test.Opa5.prototype.waitFor({
+        timeout: 10,
+        autoWait: false, // Ignore interactable constraint
+        check: function () {
+          return ready
+        }
+      })
+    }
   })
 
   QUnit.testDone(function (report) {
-    post('QUnit/testDone', report)
+    return post('QUnit/testDone', report)
   })
 
   QUnit.done(function (report) {
     if (window.__coverage__) {
       report.__coverage__ = window.__coverage__
     }
-    post('QUnit/done', report)
+    return post('QUnit/done', report)
   })
 }())

package/src/job.js

@@ -1,6 +1,8 @@
 'use strict'
 
+const { accessSync } = require('fs')
 const { join, isAbsolute } = require('path')
+const output = require('./output')
 
 function allocate (cwd) {
   return {
@@ -11,6 +13,7 @@ function allocate (cwd) {
     libs: [],
     cache: '',
     webapp: 'webapp',
+    testsuite: 'test/testsuite.qunit.html',
     pageFilter: '',
     pageParams: '',
     pageTimeout: 0,
@@ -22,6 +25,7 @@ function allocate (cwd) {
 
     browser: join(__dirname, '../defaults/chromium.js'),
     browserRetry: 1,
+    noScreenshot: false,
     args: '__URL__ __REPORT__',
 
     parallel: 2,
@@ -35,6 +39,14 @@ function allocate (cwd) {
   }
 }
 
+function checkAccess (path, label) {
+  try {
+    accessSync(path)
+  } catch (error) {
+    throw new Error(`Unable to access ${label}, check your settings`)
+  }
+}
+
 function finalize (job) {
   Object.keys(job)
     .filter(name => name.startsWith('!'))
@@ -55,9 +67,15 @@ function finalize (job) {
   'webapp,browser,tstReportDir,covSettings,covTempDir,covReportDir'
     .split(',')
     .forEach(setting => updateToAbsolute(setting))
+  checkAccess(job.webapp, 'webapp folder')
+  checkAccess(job.browser, 'browser command')
+
+  const testsuitePath = toAbsolute(job.testsuite, job.webapp)
+  checkAccess(testsuitePath, 'testsuite')
 
   job.libs.forEach(libMapping => {
     libMapping.source = toAbsolute(libMapping.source)
+    checkAccess(libMapping.source, `lib mapping of ${libMapping.relative}`)
   })
 
   if (job.parallel <= 0) {
@@ -65,7 +83,7 @@ function finalize (job) {
   }
 
   if (job.browserRetry < 0) {
-    console.warn('Invalid browserRetry value, defaulting to 1')
+    output.unexpectedOptionValue('browserRetry', 'defaulting to 1')
     job.browserRetry = 1
   }
 }
@@ -97,7 +115,7 @@ function parseJobParam (job, arg) {
       try {
         job[name] = valueParser(value, job[name])
       } catch (error) {
-        console.error(`Unexpected value for option ${name} : ${error.message}`)
+        output.unexpectedOptionValue(name, error.message)
       }
     }
   }

package/src/output.js

@@ -0,0 +1,199 @@
+'use strict'
+
+const nativeConsole = console
+const mockConsole = {}
+const interactive = process.stdout.columns !== undefined
+let lastTick = 0
+const ticks = ['\u280b', '\u2819', '\u2839', '\u2838', '\u283c', '\u2834', '\u2826', '\u2827', '\u2807', '\u280f']
+let job
+let lines = 1
+
+function write (...parts) {
+  parts.forEach(part => process.stdout.write(part))
+}
+
+function clean () {
+  write(`\x1b[${lines.toString()}F`)
+  for (let line = 0; line < lines; ++line) {
+    if (line > 1) {
+      write('\x1b[1E')
+    }
+    write(''.padEnd(process.stdout.columns, ' '))
+  }
+  if (lines > 1) {
+    write(`\x1b[${(lines - 1).toString()}F`)
+  } else {
+    write('\x1b[1G')
+  }
+}
+
+const width = 10
+
+function bar (ratio, msg) {
+  write('[')
+  if (typeof ratio === 'string') {
+    if (ratio.length > width) {
+      write(ratio.substring(0, width - 3), '...')
+    } else {
+      const padded = ratio.padStart(width - Math.floor((width - ratio.length) / 2), '-').padEnd(width, '-')
+      write(padded)
+    }
+    write(']     ')
+  } else {
+    const filled = Math.floor(width * ratio)
+    write(''.padEnd(filled, '\u2588'), ''.padEnd(width - filled, '\u2591'))
+    const percent = Math.floor(100 * ratio).toString().padStart(3, ' ')
+    write('] ', percent, '%')
+  }
+  write(' ', msg, '\n')
+}
+
+function progress (cleanFirst = true) {
+  if (cleanFirst) {
+    clean()
+  }
+  lines = 1
+  let progressRatio
+  if (job.testPageUrls && job.testPages && job.parallel > 0) {
+    const total = job.testPageUrls.length
+    const done = Object.keys(job.testPages)
+      .filter(pageUrl => !!job.testPages[pageUrl].report)
+      .length
+    if (done < total) {
+      progressRatio = done / total
+    }
+  }
+  if (job.browsers) {
+    const runningPages = Object.keys(job.browsers)
+    lines += runningPages.length
+    runningPages.forEach(pageUrl => {
+      let starting = true
+      if (job.testPages) {
+        const page = job.testPages[pageUrl]
+        if (page) {
+          const { total, passed, failed } = page
+          if (total) {
+            const progress = passed + failed
+            bar(progress / total, pageUrl)
+            starting = false
+          }
+        }
+      }
+      if (starting) {
+        bar('starting', pageUrl)
+      }
+    })
+  }
+  const status = `${ticks[++lastTick % ticks.length]} ${job.status}`
+  if (progressRatio !== undefined) {
+    bar(progressRatio, status)
+  } else {
+    write(status, '\n')
+  }
+}
+
+function wrap (write) {
+  if (job) {
+    clean()
+  }
+  write()
+  if (job) {
+    progress(false)
+  }
+}
+
+Object.getOwnPropertyNames(console).forEach(name => {
+  mockConsole[name] = function (...args) {
+    wrap(() => nativeConsole[name](...args))
+  }
+})
+
+if (interactive) {
+  global.console = mockConsole
+}
+
+module.exports = {
+  serving (url) {
+    console.log(`Server running at ${url}`)
+  },
+  watching (path) {
+    console.log('Watching changes on', path)
+  },
+  changeDetected (eventType, filename) {
+    console.log(eventType, filename)
+  },
+  report (newJob) {
+    job = newJob
+    if (interactive) {
+      process.stdout.write('\n')
+      setInterval(progress, 250)
+    }
+  },
+  browserStart (url) {
+    if (!interactive) {
+      console.log('>>', url)
+    }
+  },
+  browserCapabilities (capabilities) {
+    console.log('Browser capabilities :', capabilities)
+  },
+  browserStopped (url) {
+    if (!interactive) {
+      console.log('<<', url)
+    }
+  },
+  browserClosed (url) {
+    console.log('!! BROWSER CLOSED', url)
+  },
+  browserRetry (url, retry) {
+    console.log('>> RETRY', retry, url)
+  },
+  browserTimeout (url) {
+    console.log('!! TIMEOUT', url)
+  },
+  monitor (childProcess) {
+    ['stdout', 'stderr'].forEach(channel => {
+      childProcess[channel].on('data', chunk => {
+        wrap(() => process[channel].write(chunk))
+      })
+    })
+  },
+  nyc (...args) {
+    console.log('nyc', ...args)
+  },
+  endpoint (url, data) {
+    console.log(url, data)
+  },
+  endpointError (url, data, error) {
+    console.error(`Exception when processing ${url}`)
+    console.error(data)
+    console.error(error)
+  },
+  globalTimeout (url) {
+    console.log('!! TIMEOUT', url)
+  },
+  failFast (url) {
+    console.log('!! FAILFAST', url)
+  },
+  timeSpent (start, end = new Date()) {
+    console.log(`Time spent: ${end - start}ms`)
+  },
+  unexpectedOptionValue (optionName, message) {
+    console.error(`Unexpected value for option ${optionName} : ${message}`)
+  },
+  noTestPageFound () {
+    console.error('No test page found (or all filtered out)')
+  },
+  failedToCacheUI5resource (path, statusCode) {
+    console.error(`Unable to cache '${path}' (status ${statusCode})`)
+  },
+  genericError (error) {
+    console.error('An unexpected error occurred :', error.message || error)
+  },
+  unhandled () {
+    console.warn('Some requests are not handled properly, check the unhandled.txt report for more info')
+  },
+  results (pages) {
+    console.table(pages)
+  }
+}

package/src/report.html

@@ -1,9 +1,67 @@
 <html>
   <head>
     <title>ui5-test-runner report</title>
+    <style>
+
+body {
+  font-family: Verdana, Geneva, Tahoma, sans-serif;
+}
+
+.testId {
+  padding-right: .5rem;
+  font-size: x-small;
+  color: gray;
+}
+
+.runtime {
+  padding-left: .5rem;
+  padding-right: .5rem;
+  font-size: x-small;
+}
+
+.runtime::after {
+  content: "ms"
+}
+
+.timestamp {
+  font-size: x-small;
+}
+
+.timestamp::before {
+  content: "@"
+}
+
+.timestamp::after {
+  content: "ms"
+}
+
+img.log {
+  width: 50%;
+}
+
+table.job {
+  font-size: small;
+}
+
+table.job.object {
+  font-size: x-small;
+}
+
+table.job th {
+  text-align: right;
+  padding-right: .5rem;
+}
+
+pre {
+  margin-top: 0;
+}
+
+    </style>
   </head>
   <body>
     <script>
+const ok = '\u2714\ufe0f'
+const ko = '\u274c'
 
 function tx (content) {
   return document.createTextNode(content)
@@ -14,13 +72,15 @@ function el (name, attributes = {}, ...children) {
   Object.keys(attributes).forEach(attribute => {
     element.setAttribute(attribute, attributes[attribute])
   })
-  children.forEach(child => {
+  children
+    .filter(child => !!child)
+    .forEach(child => {
       if (typeof child !== 'object') {
         element.appendChild(tx(child.toString()))
       } else {
         element.appendChild(child)
       }
-  })
+    })
   return element
 }
 
@@ -28,62 +88,115 @@ async function json (url) {
   return (await fetch(url)).json()
 }
 
-json('pages.json')
-  .then(pages => Object.keys(pages).forEach(page => {
-    const path = pages[page]
-    const details = document.body.appendChild(el('details'))
-    const summary = details.appendChild(el('summary'))
-    const ok = '\u2714\ufe0f'
-    const ko = '\u274c'
-    summary.appendChild(el('b', {}, page))
+async function generate () {
+  const job = await json('job.json')
+  let screenshot = false
+  let consoleLog = false
+  if (job.browserCapabilities) {
+    screenshot = job.browserCapabilities.screenshot
+    consoleLog = job.browserCapabilities.consoleLog
+  }
+  if (job.noScreenshot) {
+    screenshot = false
+  }
 
-    json(`${path}.json`)
-      .then(report => {
-        if (!report.report || report.failed) {
-          summary.appendChild(tx(ko))
-        } else {
-          summary.appendChild(tx(ok))
-        }
-        const table = details.appendChild(el('table', { width: '100%' }))
-        const row = (module, name = ' ') => {
-          const tr = table.appendChild(el('tr'))
-          tr.innerHTML = `<td width="15%">${module}</td><td width="85%">${name}</td>`
-          return tr
-        }
-        let lastModule
-        report.tests.forEach(test => {
-          let { module, name, failed } = test
-          if (module !== lastModule) {
-            lastModule = module
+  const jobDetails = document.body.appendChild(el('details'))
+  const jobSummary = jobDetails.appendChild(el('summary'))
+  jobSummary.appendChild(el('span', {}, 'job details'))
+  const jobParams = jobDetails.appendChild(el('table', { class: 'job' }))
+  
+  function dump (obj, level) {
+    Object.keys(obj).forEach(property => {
+      const value = obj[property]
+      const tr = level.appendChild(el('tr'))
+      tr.appendChild(el('th', {}, property))
+      if (typeof value !== 'object') {
+        let stringifiedValue
+        if (typeof value === 'string') {
+          if (value.match(/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z/)) {
+            stringifiedValue = new Date(value).toString()
           } else {
-            module = ' '
+            stringifiedValue = value
           }
-          if (failed) {
-            const tr = row(module)
-            const td = tr.querySelectorAll('td')[1]
-            const assertions = td.appendChild(el('details'))
-            assertions.appendChild(el('summary', {}, name + ko))
-            test.assertions.forEach(assertion => {
-              const bullet = assertion.result && ok || ko
-              assertions.appendChild(el('div', {}, el('pre', {}, bullet + ' ' + assertion.message)))
-            })
-          } else {
-            row(module, name)
-          }
-        })
-        if (report.report) {
-          row(`${report.report.runtime} ms`)
         } else {
-          row(`Timed out ${ko}`)
+          stringifiedValue = JSON.stringify(value)
         }
-        details.appendChild(el('a', { href: `${path}/screenshot.png`, target: '_blank' }, 'screenshot'))
-        details.appendChild(el('span', {}, ' '))
-        details.appendChild(el('a', { href: `${path}/console.txt`, target: '_blank' }, 'console'))
-      }, () => {
-        summary.appendChild(tx(ko))
-        details.appendChild(el('p', {}, 'No report found ', el('i', {}, '(fatal error or timeout)')))
+        tr.appendChild(el('td', {}, stringifiedValue))
+      } else {
+        const subLevel = tr.appendChild(el('td')).appendChild(el('table', { class: 'job object' }))
+        dump(value, subLevel)
+      }
+    })
+  }
+  dump(job, jobParams)
+
+  const pages = await json('pages.json')
+  Object.keys(pages).forEach(async page => {
+    const pagePath = pages[page]
+    const pageDetails = document.body.appendChild(el('details'))
+    const pageSummary = pageDetails.appendChild(el('summary'))
+    pageSummary.appendChild(el('b', {}, page))
+
+    const pageReport = await json(`${pagePath}.json`)
+    const { isOpa, report, failed } = pageReport
+    if (!report || failed) {
+      pageSummary.appendChild(tx(ko))
+    } else {
+      pageSummary.appendChild(tx(ok))
+    }
+
+    const pageDetailsTable = pageDetails.appendChild(el('table', { width: '100%' }))
+    const addPageDetailsRow = module => {
+      const tr = pageDetailsTable.appendChild(el('tr'))
+      tr.innerHTML = `<td width="15%" valign="top">${module}</td><td width="85%"> </td>`
+      return tr
+    }
+
+    let lastModule
+    pageReport.order.forEach(testId => {
+      const { timestamps, report } = pageReport.tests[testId]
+      const { module, name, failed, assertions, runtime } = report
+
+      let displayModule = module
+      if (module !== lastModule) {
+        lastModule = module
+      } else {
+        displayModule = ' '
+      }
+      const pageAssertions = addPageDetailsRow(displayModule).querySelectorAll('td')[1].appendChild(el('details'))
+      pageAssertions.appendChild(el('summary', {}, name, el('span', { class: 'runtime' }, runtime), el('span', { class: 'testId' }, testId), failed ? ko : ''))
+      const validTimestamps = assertions.length === timestamps.length
+      assertions.forEach((assertion, index) => {
+        const bullet = assertion.result && ok || ko
+        const timestamp = timestamps[index]
+        let preview
+        if (isOpa && screenshot && validTimestamps) {
+          preview = el('img', { loading: 'lazy', src: `${pagePath}/${testId}-${timestamp}.png`, class: 'log' })
+        }
+        pageAssertions.appendChild(el('div', {}, el('div', { class: 'timestamp' }, timestamp), el('pre', {}, bullet + ' ' + assertion.message), preview))
       })
-  }))
+    })
+
+    if (report) {
+      addPageDetailsRow(`${report.runtime} ms`)
+    } else {
+      addPageDetailsRow(`Timed out ${ko}`)
+    }
+    if (screenshot) {
+      pageDetails.appendChild(el('a', { href: `${pagePath}/screenshot.png`, target: '_blank' }, 'screenshot'))
+      pageDetails.appendChild(el('span', {}, ' '))
+    }
+    if (consoleLog) {
+      pageDetails.appendChild(el('a', { href: `${pagePath}/console.txt`, target: '_blank' }, 'console'))
+    }
+  }, () => {
+    pageSummary.appendChild(tx(ko))
+    pageDetails.appendChild(el('p', {}, 'No report found ', el('i', {}, '(fatal error or timeout)')))
+  })
+}
+
+generate()
+
     </script>
   </body>
 </html>

package/src/reserve.js

@@ -19,6 +19,6 @@ module.exports = job => check({
       strict: true,
       'ignore-if-not-found': true
     },
-    ...unhandled
+    ...unhandled(job)
   ]
 })

package/src/tests.js

@@ -6,16 +6,29 @@ const { filename, recreateDir } = require('./tools')
 const { join } = require('path')
 const { copyFile, writeFile } = require('fs').promises
 const { globallyTimedOut } = require('./timeout')
+const output = require('./output')
+
+async function saveJob (job) {
+  await writeFile(join(job.tstReportDir, 'job.json'), JSON.stringify({
+    ...job,
+    // Following members are useless because already serialized or not relevant
+    status: undefined,
+    testPageUrls: undefined,
+    browsers: undefined,
+    testPages: undefined
+  }))
+}
 
 async function extractTestPages (job) {
   job.start = new Date()
   await instrument(job)
-  job.status = 'Extracting test pages'
   await recreateDir(job.tstReportDir)
+  await saveJob(job)
+  job.status = 'Extracting test pages'
   job.testPageUrls = []
-  await start(job, '/test/testsuite.qunit.html')
+  await start(job, '/' + job.testsuite)
   if (job.testPageUrls.length === 0) {
-    console.log('No test page found')
+    output.noTestPageFound()
     job.failed = true
     return Promise.resolve()
   }
@@ -42,9 +55,9 @@ async function runTestPage (job) {
   const index = job.testPagesStarted++
   const url = job.testPageUrls[index]
   if (globallyTimedOut(job)) {
-    console.log('!! TIMEOUT', url)
+    output.globalTimeout(url)
   } else if (job.failFast && job.failed) {
-    console.log('!! FAILFAST', url)
+    output.failFast(url)
   } else {
     await start(job, url)
     const page = job.testPages[url]
@@ -78,10 +91,11 @@ async function generateReport (job) {
       job.failed += 1
     }
   }
-  console.table(pages)
+  output.results(pages)
+  await saveJob(job)
   await copyFile(join(__dirname, 'report.html'), join(job.tstReportDir, 'report.html'))
   await generateCoverageReport(job)
-  console.log(`Time spent: ${new Date() - job.start}ms`)
+  output.timeSpent(job.start)
   job.status = 'Done'
 }
 

package/src/ui5.js

@@ -4,6 +4,7 @@ const { dirname, join } = require('path')
 const { createWriteStream } = require('fs')
 const { mkdir, unlink } = require('fs').promises
 const { capture } = require('reserve')
+const output = require('./output')
 
 module.exports = job => {
   const [, hostName] = /https?:\/\/([^/]*)/.exec(job.ui5)
@@ -58,7 +59,7 @@ module.exports = job => {
           file.end()
           uncachable[path] = true
           if (response.statusCode !== 404) {
-            console.error(`Unable to cache '${path}' (status ${response.statusCode})`)
+            output.failedToCacheUI5resource(path, response.statusCode)
           }
           return unlink(cachePath)
         })

package/src/unhandled.js

@@ -1,17 +1,32 @@
 'use strict'
 
 const { extractUrl } = require('./tools')
+const { join } = require('path')
+const { writeFile } = require('fs')
+const output = require('./output')
 
-module.exports = [{
-  custom: ({ headers, method, url }, response) => {
-    if (method === 'GET' && url.match(/favicon\.ico$|-preload\.js$|-dbg(\.[^.]+)*\.js$|i18n_\w+\.properties$/)) {
-      return 404 // expected
+module.exports = job => {
+  const unhandled = join(job.tstReportDir, 'unhandled.txt')
+  let outputUnhandled = true
+  return [{
+    custom: ({ headers, method, url }) => {
+      if (method === 'GET' && url.match(/favicon\.ico$|-preload\.js$|-dbg(\.[^.]+)*\.js$|i18n_\w+\.properties$/)) {
+        return 404 // expected
+      }
+      let status
+      if (method === 'GET') {
+        status = 404
+      } else {
+        status = 500
+      }
+      if (outputUnhandled) {
+        output.unhandled()
+        outputUnhandled = false
+      }
+      writeFile(unhandled, `${extractUrl(headers)} ${status} ${method} ${url}\n`, {
+        flag: 'a'
+      }, () => {})
+      return status
     }
-    if (method === 'GET') {
-      console.warn(`?? ${extractUrl(headers)} 404 ${method} ${url}`)
-      return 404
-    }
-    console.error(`!! ${extractUrl(headers)} 500 ${method} ${url}`)
-    return 500
-  }
-}]
+  }]
+}