sitespeed.io 36.4.1 → 37.0.1

package/CHANGELOG.md

@@ -1,6 +1,27 @@
 
 # CHANGELOG - sitespeed.io  (we use [semantic versioning](https://semver.org))
 
+## 37.0.1 - 2025-03-05
+### Fixed
+* There was a bug in the InfluxDB plugin and loading the cli parameters failed [#4463](https://github.com/sitespeedio/sitespeed.io/pull/4463).
+
+
+## 37.0.0 - 2025-03-05
+### Breaking change
+* If you are a InfluxDB user the InfluxDB functionality been moved to a standalone plugin [plugin-influxdb](https://github.com/sitespeedio/plugin-influxdb). This means if that sitespeed.io using NodeJS and the default Docker container will not include the InfluxDB plugin. The +1 container will still include the plugin but you need to add it with `--plugins.add @sitespeed.io/plugin-influxdb` for it to be able to run.
+
+The InfluxDB plugin has never gotten the love it deserves, moving it out, it means that you easier can do your own modification and get the data the way that you need.
+
+### Added
+* Support for getting CLI options from plugins as long as you use `--help --plugins.add`. This is start to separate CLI options from the monsterous CLI file and instead have them in each plugin [#4450](https://github.com/sitespeedio/sitespeed.io/pull/4450), [#4452](https://github.com/sitespeedio/sitespeed.io/pull/4452) [#4455](https://github.com/sitespeedio/sitespeed.io/pull/4455).
+* You can now set the exact minimum log level using `--logLevel`[#4459](https://github.com/sitespeedio/sitespeed.io/pull/4459).
+
+### Fixed
+* Replace the junit-report-builder package [#4448](https://github.com/sitespeedio/sitespeed.io/pull/4448).
+* Remove Tape dependencies [#4447](https://github.com/sitespeedio/sitespeed.io/pull/4447).
+* Catch if the browser fails to open a broken page [#4457](https://github.com/sitespeedio/sitespeed.io/pull/4457).
+
+
 ## 36.4.1 - 2025-02-17
 ### Fixed
 * The Docker container for last release was never push. The reason is that our GitHub action that do the release automatically was upgraded to Ubuntu 24 and there its a problem building containers for ARM. With this release we use Ubuntu 22 instead.

package/HELP.md

@@ -2,7 +2,7 @@
 We want to make sitespeed.io one of the best web performance tool in the world and we hope you can help us!
 
 ## Developers
-We love to have more people involved in improving sitespeed.io. We are constantly working on adding more documentation and trying to write more information in the issues so its easier to help out. If there's an [issue](https://github.com/sitespeedio/sitespeed.io/issues) that you want to take on, ping the the issue and we can help you get started. You can also [join our Slack channel](https://join.slack.com/t/sitespeedio/shared_invite/zt-296jzr7qs-d6DId2KpEnMPJSQ8_R~WFw) if you need help!
+We love to have more people involved in improving sitespeed.io. We are constantly working on adding more documentation and trying to write more information in the issues so its easier to help out. If there's an [issue](https://github.com/sitespeedio/sitespeed.io/issues) that you want to take on, ping the the issue and we can help you get started. You can also [join our Slack channel](https://join.slack.com/t/sitespeedio/shared_invite/zt-296jzr7qs-d6DId2KpEnMPJSQ8_R~WFw) if you need help! You can start by reading the [developer documentation](https://www.sitespeed.io/documentation/sitespeed.io/developers/).
 
 ## Designers
 As a designer there's a lot you can do: You can help us improve the HTML result pages. Maybe we should restructure the metrics  ? Or could the header/footer look better? You could also have look at [https://www.sitespeed.io](https://www.sitespeed.io/) where we have all the documentation. You can pretty much help us with everything, no one in the core team got design skills :)
@@ -10,13 +10,6 @@ As a designer there's a lot you can do: You can help us improve the HTML result
 ## Documentation
 Documentation is fun and it is the core of making sitespeed.io easy to use. We got a [special documentation tag for issues](https://github.com/sitespeedio/sitespeed.io/issues?q=is%3Aissue+is%3Aopen+label%3Adocumentation) that you can use to find where we know we lack documentation. Fixing spelling mistakes is great. Or rewrite parts that you think is too complicated. You can find what you need to send a PR to the documentation [here](https://github.com/sitespeedio/sitespeed.io/tree/main/docs).
 
-## Tests
-We lack unit tests. You can read about [our testing pipeline](https://www.sitespeed.io/releasing-with-confidence/) that works good for us but more unit tests are always good. A good start is adding support for the code in the [support library](https://github.com/sitespeedio/sitespeed.io/tree/main/lib/support).
-
 
 ## Companies
-Do you use sitespeed.io in your everyday work? Then we have a perfect proposition for you! Have a hack day with focus on sitespeed.io for your team and contribute back. Pick one the things in the **Help wanted** section or make your plugin Open Source and tell us about it. Or maybe there's a something you think is missing? Create it. Contribute back.
-
-# Help wanted
-You can help us:
-* Help us improve the documentation. We love your feedback and help [https://github.com/sitespeedio/sitespeed.io/tree/main/docs](https://github.com/sitespeedio/sitespeed.io/tree/main/docs).
+Do you use sitespeed.io in your everyday work? Then go to the [sponsor page](https://www.sitespeed.io/sponsor/).

package/lib/cli/cli.js

@@ -18,6 +18,7 @@ import { config as slackConfig } from '../plugins/slack/index.js';
 import { config as htmlConfig } from '../plugins/html/index.js';
 import { messageTypes as matrixMessageTypes } from '../plugins/matrix/index.js';
 import { findUpSync } from '../support/fileUtil.js';
+import { registerPluginOptions } from './pluginOptions.js';
 
 const metricList = Object.keys(friendlynames);
 const require = createRequire(import.meta.url);
@@ -25,6 +26,13 @@ const version = require('../../package.json').version;
 
 const configFiles = ['.sitespeed.io.json'];
 
+const addedPlugins = yargs(hideBin(process.argv))
+  .option('plugins.add', { type: 'array' })
+  .help(false)
+  .version(false)
+  .parse();
+const globalPluginsToAdd = addedPlugins.plugins?.add || [];
+
 function fixAndroidArgs(args) {
   return args.map(arg => (arg === '--android' ? '--android.enabled' : arg));
 }
@@ -220,7 +228,7 @@ function validateInput(argv) {
 export async function parseCommandLine() {
   const fixedArgs = fixAndroidArgs(hideBin(process.argv));
   const yargsInstance = yargs(fixedArgs);
-  let parsed = yargsInstance
+  yargsInstance
     .parserConfiguration({
       'camel-case-expansion': false,
       'deep-merge-config': true
@@ -421,7 +429,7 @@ export async function parseCommandLine() {
     })
     .option('browsertime.script', {
       describe:
-        'Add custom Javascript that collect metrics and run after the page has finished loading. Note that --script can be passed multiple times if you want to collect multiple metrics. The metrics will automatically be pushed to the summary/detailed summary and each individual page + sent to Graphite/InfluxDB.',
+        'Add custom Javascript that collect metrics and run after the page has finished loading. Note that --script can be passed multiple times if you want to collect multiple metrics. The metrics will automatically be pushed to the summary/detailed summary and each individual page + sent to Graphite',
       alias: ['script'],
       group: 'Browser'
     })
@@ -513,7 +521,7 @@ export async function parseCommandLine() {
     .option('browsertime.scriptInput.visualElements', {
       alias: ['scriptInput.visualElements'],
       describe:
-        'Include specific elements in visual elements. Give the element a name and select it with document.body.querySelector. Use like this: --scriptInput.visualElements name:domSelector . Add multiple instances to measure multiple elements. Visual Metrics will use these elements and calculate when they are visible and fully rendered.',
+        'Include specific elements in visual elements. Give the element a name and select it with document.body.querySelector. Use like this: --scriptInput.visualElements name:domSelector . If you want to measure multiple elements, use a configuration file with an array for the input. Visual Metrics will use these elements and calculate when they are visible and fully rendered.',
       group: 'Browser'
     })
     .option('browsertime.scriptInput.longTask', {
@@ -1160,7 +1168,6 @@ export async function parseCommandLine() {
       describe: 'Ignore robots.txt rules of the crawled domain.',
       group: 'Crawler'
     })
-
     .option('scp.host', {
       describe: 'The host.',
       group: 'scp'
@@ -1347,80 +1354,6 @@ export async function parseCommandLine() {
         'Define which messages to send to Graphite. By default we do not send data per run, but you can change that by adding run as one of the options',
       group: 'Graphite'
     })
-
-    .option('influxdb.protocol', {
-      describe: 'The protocol used to store connect to the InfluxDB host.',
-      default: 'http',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.host', {
-      describe: 'The InfluxDB host used to store captured metrics.',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.port', {
-      default: 8086,
-      describe: 'The InfluxDB port used to store captured metrics.',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.username', {
-      describe:
-        'The InfluxDB username for your InfluxDB instance (only for InfluxDB v1)',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.password', {
-      describe:
-        'The InfluxDB password for your InfluxDB instance (only for InfluxDB v1).',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.organisation', {
-      describe:
-        'The InfluxDB organisation for your InfluxDB instance (only for InfluxDB v2)',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.token', {
-      describe:
-        'The InfluxDB token for your InfluxDB instance (only for InfluxDB v2)',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.version', {
-      default: 1,
-      describe: 'The InfluxDB version of your InfluxDB instance.',
-      type: 'integer',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.database', {
-      default: 'sitespeed',
-      describe: 'The database name used to store captured metrics.',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.tags', {
-      default: 'category=default',
-      describe:
-        'A comma separated list of tags and values added to each metric',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.includeQueryParams', {
-      default: false,
-      describe:
-        'Whether to include query parameters from the URL in the InfluxDB keys or not',
-      type: 'boolean',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.groupSeparator', {
-      default: '_',
-      describe:
-        'Choose which character that will separate a group/domain. Default is underscore, set it to a dot if you wanna keep the original domain name.',
-      group: 'InfluxDB'
-    })
-    .option('influxdb.annotationScreenshot', {
-      default: false,
-      type: 'boolean',
-      describe:
-        'Include screenshot (from Browsertime) in the annotation. You need to specify a --resultBaseURL for this to work.',
-      group: 'InfluxDB'
-    });
-
-  parsed
     /** Plugins */
     .option('plugins.list', {
       describe: 'List all configured plugins in the log.',
@@ -1526,13 +1459,7 @@ export async function parseCommandLine() {
       describe: 'The max size of the screenshot (width and height).',
       default: browsertimeConfig.screenshotParams.maxSize,
       group: 'Screenshot'
-    });
-  /**
-   InfluxDB cli option
-   */
-
-  parsed
-    // Metrics
+    })
     .option('metrics.list', {
       describe: 'List all possible metrics in the data folder (metrics.txt).',
       type: 'boolean',
@@ -1877,9 +1804,7 @@ export async function parseCommandLine() {
       describe:
         'Instead of using the local copy of the hosting database, you can use the latest version through the Green Web Foundation API. This means sitespeed.io will make HTTP GET to the the hosting info.',
       group: 'Sustainable'
-    });
-
-  parsed
+    })
     .option('api.key', {
       describe: 'The API key to use.',
       group: 'API'
@@ -1930,9 +1855,7 @@ export async function parseCommandLine() {
     .option('api.json', {
       describe: 'Output the result as JSON.',
       group: 'API'
-    });
-
-  parsed
+    })
     .option('compare.id', {
       type: 'string',
       describe:
@@ -1993,8 +1916,7 @@ export async function parseCommandLine() {
         'Selects the method for calculating the Mann-Whitney U test. auto automatically selects between exact and asymptotic based on sample size, exact uses the exact distribution of U, and symptotic uses a normal approximation.',
       default: 'auto',
       group: 'compare'
-    });
-  parsed
+    })
     .option('mobile', {
       describe:
         'Access pages as mobile a fake mobile device. Set UA and width/height. For Chrome it will use device Moto G4.',
@@ -2028,12 +1950,12 @@ export async function parseCommandLine() {
     })
     .option('urlAlias', {
       describe:
-        'Use an alias for the URL (if you feed URLs from a file you can instead have the alias in the file). You need to pass on the same amount of alias as URLs. The alias is used as the name of the URL on the HTML report and in Graphite/InfluxDB. Pass on multiple --urlAlias for multiple alias/URLs. This will override alias in a file.',
+        'Use an alias for the URL (if you feed URLs from a file you can instead have the alias in the file). You need to pass on the same amount of alias as URLs. The alias is used as the name of the URL on the HTML report and in Graphite. Pass on multiple --urlAlias for multiple alias/URLs. This will override alias in a file.',
       type: 'string'
     })
     .option('groupAlias', {
       describe:
-        'Use an alias for the group/domain. You need to pass on the same amount of alias as URLs. The alias is used as the name of the group in Graphite/InfluxDB. Pass on multiple --groupAlias for multiple alias/groups. This do not work for scripting at the moment.',
+        'Use an alias for the group/domain. You need to pass on the same amount of alias as URLs. The alias is used as the name of the group in Graphite. Pass on multiple --groupAlias for multiple alias/groups. This do not work for scripting at the moment.',
       type: 'string'
     })
     .option('utc', {
@@ -2056,6 +1978,11 @@ export async function parseCommandLine() {
     .option('name', {
       describe: 'Give your test a name.'
     })
+    .option('logLevel', {
+      type: 'string',
+      choices: ['trace', 'verbose', 'debug', 'info', 'warning', 'error'],
+      describe: 'Manually set the min log level'
+    })
     .option('open', {
       alias: ['o', 'view'],
       describe:
@@ -2124,8 +2051,12 @@ export async function parseCommandLine() {
         }
         return plugins;
       }
-    })
-    //     .describe('browser', 'Specify browser')
+    });
+  //     .describe('browser', 'Specify browser')
+
+  await registerPluginOptions(yargsInstance, globalPluginsToAdd);
+
+  let parsed = yargsInstance
     .wrap(yargsInstance.terminalWidth())
     //  .check(validateInput)
     .epilog(

package/lib/cli/pluginOptions.js

@@ -0,0 +1,34 @@
+import { importGlobalSilent } from 'import-global';
+
+/**
+ * Dynamically load and register CLI options from plugins.
+ *
+ * @param {import('yargs').Argv} yargsInstance - The yargs instance to extend.
+ * @param {string[]} plugins - Array of plugin module names.
+ * @returns {Promise<void>}
+ */
+export async function registerPluginOptions(yargsInstance, plugins) {
+  for (const pluginName of plugins) {
+    try {
+      // Dynamically import the plugin
+      let { default: plugin } = await importGlobalSilent(pluginName);
+      // If the plugin exports a function to get CLI options, merge them
+      if (plugin && typeof plugin.getCliOptions === 'function') {
+        const options = plugin.getCliOptions();
+        yargsInstance.options(options);
+      } else {
+        try {
+          const { default: plugin } = await import(pluginName);
+          if (plugin && typeof plugin.getCliOptions === 'function') {
+            const options = plugin.getCliOptions();
+            yargsInstance.options(options);
+          }
+        } catch {
+          // Swallow this silent
+        }
+      }
+    } catch {
+      // Swallow this silent
+    }
+  }
+}

package/lib/core/logging.js

@@ -2,6 +2,7 @@ import { configureLog } from '@sitespeed.io/log';
 
 export function configure(options = {}) {
   configureLog({
+    level: options.logLevel ?? undefined,
     verbose: options.verbose ?? 0,
     silent: options.silent ?? false
   });

package/lib/core/pluginLoader.js

@@ -81,12 +81,18 @@ export async function loadPlugins(pluginNames, options, context, queue) {
           plugins.push(p);
         } catch (error) {
           // try global
-          let { default: plugin } = await importGlobalSilent(name);
-          if (plugin) {
-            let p = new plugin(options, context, queue);
-            plugins.push(p);
-          } else {
-            console.error("Couldn't load plugin %s: %s", name, error_);
+          try {
+            let { default: plugin } = await importGlobalSilent(name);
+            if (plugin) {
+              let p = new plugin(options, context, queue);
+              plugins.push(p);
+            } else {
+              console.error("Couldn't load plugin %s: %s", name, error_);
+              // if it fails here, let it fail hard
+              throw error;
+            }
+          } catch {
+            console.error("Couldn't find/load plugin %s", name);
             // if it fails here, let it fail hard
             throw error;
           }

package/lib/plugins/browsertime/index.js

@@ -3,6 +3,8 @@ import { parse } from 'node:url';
 import { default as _merge } from 'lodash.merge';
 
 import { getLogger } from '@sitespeed.io/log';
+import { configureLogging } from 'browsertime';
+
 const log = getLogger('plugin.browsertime');
 
 import dayjs from 'dayjs';
@@ -88,10 +90,10 @@ export default class BrowsertimePlugin extends SitespeedioPlugin {
       'browsertime.run'
     );
     this.axeAggregatorTotal = new AxeAggregator(this.options);
+    configureLogging(options);
   }
+
   async processMessage(message) {
-    const { configureLogging } = await import('browsertime');
-    configureLogging(this.options);
     const options = this.options;
     switch (message.type) {
       // When sistespeed.io starts, a setup messages is posted on the queue

package/lib/plugins/budget/junit.js

@@ -1,69 +1,95 @@
 import path from 'node:path';
-import { parse } from 'node:url';
-
-import jrp from 'junit-report-builder';
-
+import fs from 'node:fs';
+import merge from 'lodash.merge';
 import { getLogger } from '@sitespeed.io/log';
+
 const log = getLogger('sitespeedio.plugin.budget');
 
-import merge from 'lodash.merge';
+/**
+ * Escapes XML special characters.
+ *
+ * @param {string} str - The text to escape.
+ * @returns {string} The escaped text.
+ */
+function xmlEscape(str) {
+  return String(str)
+    .replaceAll('&', '&')
+    .replaceAll('<', '&lt;')
+    .replaceAll('>', '&gt;')
+    .replaceAll('"', '&quot;')
+    .replaceAll("'", '&apos;');
+}
+
+/**
+ * Wraps a string in a CDATA block.
+ *
+ * @param {string} str - The string to wrap.
+ * @returns {string} The CDATA-wrapped string.
+ */
+function cdata(str) {
+  return `<![CDATA[${str}]]>`;
+}
 
+/**
+ * Writes a JUnit XML report mimicking the original output.
+ *
+ * @param {object} results - Object containing `failing` and `working` results.
+ * @param {string} dir - Directory where `junit.xml` will be written.
+ * @param {object} options - Options (expects `options.budget.friendlyName`).
+ */
 export function writeJunit(results, dir, options) {
-  // lets have one suite per URL
-  const urls = Object.keys(merge({}, results.failing, results.working));
+  const failing = results.failing || {};
+  const working = results.working || {};
+  const urls = Object.keys(merge({}, failing, working));
 
-  for (const url of urls) {
-    // The URL can be an alias
-    let name = url;
-    if (url.startsWith('http')) {
-      const parsedUrl = parse(url);
-      name = url.startsWith('http') ? url : url;
-      parsedUrl.hostname.replaceAll('.', '_') +
-        '.' +
-        parsedUrl.path.replaceAll('.', '_').replaceAll('/', '_');
-    }
+  let totalTests = 0;
+  let totalFailures = 0;
+  let suitesXml = '';
 
-    const suite = jrp
-      .testSuite()
-      .name(options.budget.friendlyName || 'sitespeed.io' + '.' + name);
+  for (const url of urls) {
+    const suiteName = `${options.budget.friendlyName || 'sitespeed.io'}.${url}`;
+    let suiteTests = 0;
+    let suiteFailures = 0;
+    let testCasesXml = '';
 
-    if (results.failing[url]) {
-      for (const result of results.failing[url]) {
-        suite
-          .testCase()
-          .className(name)
-          .name(result.type + '.' + result.metric)
-          .failure(
-            result.metric + ' is ' + result.friendlyValue ||
-              result.value +
-                ' and limit ' +
-                result.limitType +
-                ' ' +
-                result.friendlyLimit ||
-              result.limit + ' ' + url
-          );
+    if (failing[url]) {
+      for (const result of failing[url]) {
+        suiteTests++;
+        totalTests++;
+        suiteFailures++;
+        totalFailures++;
+        const testCaseName = `${result.type}.${result.metric}`;
+        const failureMessage = `${result.metric} is ${result.friendlyValue || result.value}`;
+        testCasesXml += `    <testcase classname="${xmlEscape(url)}" name="${xmlEscape(testCaseName)}">\n`;
+        testCasesXml += `      <failure message="${xmlEscape(failureMessage)}"/>\n`;
+        testCasesXml += `    </testcase>\n`;
       }
     }
 
-    if (results.working[url]) {
-      for (const result of results.working[url]) {
-        suite
-          .testCase()
-          .className(name)
-          .name(result.type + '.' + result.metric)
-          .standardOutput(
-            result.metric + ' is ' + result.friendlyValue ||
-              result.value +
-                ' and limit ' +
-                result.limitType +
-                ' ' +
-                result.friendlyLimit ||
-              result.limit + ' ' + url
-          );
+    if (working[url]) {
+      for (const result of working[url]) {
+        suiteTests++;
+        totalTests++;
+        const testCaseName = `${result.type}.${result.metric}`;
+        const systemOutMessage = `${result.metric} is ${result.friendlyValue || result.value}`;
+        testCasesXml += `    <testcase classname="${xmlEscape(url)}" name="${xmlEscape(testCaseName)}">\n`;
+        testCasesXml += `      <system-out>${cdata(systemOutMessage)}</system-out>\n`;
+        testCasesXml += `    </testcase>\n`;
       }
     }
+
+    suitesXml += `  <testsuite name="${xmlEscape(suiteName)}" tests="${suiteTests}" failures="${suiteFailures}" errors="0" skipped="0">\n`;
+    suitesXml += testCasesXml;
+    suitesXml += `  </testsuite>\n`;
   }
+
+  const xml =
+    `<?xml version="1.0" encoding="UTF-8"?>\n` +
+    `<testsuites tests="${totalTests}" failures="${totalFailures}" errors="0" skipped="0">\n` +
+    suitesXml +
+    `</testsuites>\n`;
+
   const file = path.join(dir, 'junit.xml');
   log.info('Write junit budget to %s', path.resolve(file));
-  jrp.writeTo(file);
+  fs.writeFileSync(file, xml);
 }

package/lib/plugins/budget/tap.js

@@ -1,36 +1,44 @@
 import path from 'node:path';
 import fs from 'node:fs';
 import { EOL } from 'node:os';
-import tap from 'tape';
 import { getLogger } from '@sitespeed.io/log';
+
 const log = getLogger('sitespeedio.plugin.budget');
 
 export function writeTap(results, dir) {
   const file = path.join(dir, 'budget.tap');
   log.info('Write budget to %s', path.resolve(file));
-  const tapOutput = fs.createWriteStream(file);
-  tap.createStream().pipe(tapOutput);
 
-  for (const resultType of Object.keys(results)) {
-    const urls = Object.keys(results.failing);
+  const lines = [];
+  lines.push('TAP version 13');
+  let testCount = 0;
 
+  // Iterate over each result group (e.g. "passing" and "failing")
+  for (const resultType of Object.keys(results)) {
+    const group = results[resultType];
+    if (!group) {
+      continue;
+    }
+    const urls = Object.keys(group);
     for (const url of urls) {
-      for (const result of results.failing[url]) {
-        tap(result.type + '.' + result.metric + ' ' + url, function (t) {
-          let extra = '';
-          if (resultType === 'failing') {
-            extra =
-              ' limit ' + result.limitType + ' ' + result.friendlyLimit ||
-              result.limit + EOL;
-          }
-          t.ok(
-            resultType === 'failing' ? false : true,
-            result.type + '.' + result.metric + ' ' + result.friendlyValue ||
-              result.value + ' ' + extra + ' ' + url
-          );
-          t.end();
-        });
+      for (const result of group[url]) {
+        testCount += 1;
+        const testTitle = `${result.type}.${result.metric} ${url}`;
+        let extra = '';
+        if (resultType === 'failing') {
+          extra = ` limit ${result.limitType} ${result.friendlyLimit || result.limit}`;
+        }
+        const valueDisplay = result.friendlyValue || result.value;
+
+        lines.push(`# ${testTitle}`);
+        const status = resultType === 'failing' ? 'not ok' : 'ok';
+        lines.push(
+          `${status} ${testCount} ${testTitle} ${valueDisplay}${extra ? ` ${extra}` : ''}`
+        );
       }
     }
   }
+
+  lines.push(`1..${testCount}`);
+  fs.writeFileSync(file, lines.join(EOL) + EOL);
 }

package/lib/plugins/grafana/send-annotation.js

@@ -41,9 +41,7 @@ export function send(
     options,
     group,
     url,
-    tsdbType === 'graphite'
-      ? options.graphite.includeQueryParams
-      : options.influxdb.includeQueryParams,
+    options.graphite.includeQueryParams,
     alias
   ).split('.');
 

package/lib/plugins/pagexray/index.js

@@ -128,15 +128,17 @@ export default class PageXrayPlugin extends SitespeedioPlugin {
               );
             }
           } else {
-            pageSummary[0].statistics = this.pageXrayAggregator.summarizePerUrl(
-              message.url
-            );
-            queue.postMessage(
-              make('pagexray.pageSummary', pageSummary[0], {
-                url: message.url,
-                group // TODO get the group from the URL?
-              })
-            );
+            // Check that we actually have one tested page
+            if (pageSummary.length > 0) {
+              pageSummary[0].statistics =
+                this.pageXrayAggregator.summarizePerUrl(message.url);
+              queue.postMessage(
+                make('pagexray.pageSummary', pageSummary[0], {
+                  url: message.url,
+                  group // TODO get the group from the URL?
+                })
+              );
+            }
             let iteration = 1;
             for (let summary of pageSummary) {
               queue.postMessage(

package/lib/plugins/pagexray/pagexrayAggregator.js

@@ -140,7 +140,8 @@ export class PageXrayAggregator {
     }
   }
   summarizePerUrl(url) {
-    return this.summarizePerObject(this.urls[url]);
+    // If we do not have a tested working page there's nothing to summarize
+    return this.urls[url] ? this.summarizePerObject(this.urls[url]) : {};
   }
   summarize() {
     if (Object.keys(this.stats).length === 0) {