newrelic 4.6.0 → 4.9.0

package/NEWS.md

@@ -1,3 +1,150 @@
+### 4.9.0 (2018-10-01):
+
+* Updated DT payload creation to use `primary_application_id` from connect response.
+
+* Added protection against functions with modified prototypes in `shim.applySegment`.
+
+* Replaced SQL ID hash generation algorithm with SHA1 instead of MD5 to allow usage
+  in FIPS compliant environments.
+
+* Leveraged 16 hex digits for creation of SQL ID.
+
+* Fixed `codec.decode()` callback bug that would re-call a callback with an error
+  thrown within the callback.
+
+* Added `superagent` as built-in instrumentation.
+
+  This instrumentation just maintains transaction state when using the `superagent`
+  module to make HTTP requests with either callbacks or promises.
+
+* Updated `noticeError` API method to be partially functional in High Security Mode.
+
+  In HSM, any custom attributes will be ignored, but the error will still be tracked.
+  This brings the Node agent in line with the behavior of other language agents.
+
+* Upgraded ejs module to get rid of Github security warnings. The ejs module was
+  only used for tests and not in main agent code.
+
+* Fixed bug requiring Cross Application Tracing (CAT) to be enabled for Distributed
+  Tracing (DT) `createDistributedTracePayload` and `acceptDistributedTracePayload`
+  APIs to function. DT configuration will no longer consider CAT configuration.
+
+* Changes DT payload configuration log messages to debug level as it is not uncommon
+  for calls to occur before server configuration has been retrieved.
+
+* Converted `net` instrumentation to use shim api.
+
+* Converted child_process instrumentation to newer shim style.
+
+* Converted Timers instrumentation to newer shim style.
+
+* Fixed bug in wrap() that would fail to wrap callbacks if the callback index was 0.
+
+* Added `PromiseShim` class for instrumenting promise libraries.
+
+* Support for setting `transaction_tracer.transaction_threshold` to 0 has been added.
+
+* The agent now respects `NEW_RELIC_TRACER_THRESHOLD`.
+
+  Previously, this environment variable was stored as a string. The environment
+  variable is now stored as a float.
+
+* Converted zlib instrumentation to use shim API.
+
+### 4.8.1 (2018-08-27):
+
+* Converted File System instrumentation to use newer shim style.
+
+* Agent instrumentation will no longer interfere with promisification of core
+  methods.
+
+  Some core methods expose pre-promisified versions of the methods as a reference
+  on the method itself. When instrumenting these methods, it neglected to forward
+  these references onto the wrapper function. Now the instrumentation will properly
+  forward property look ups to the original method.
+
+* Converted DNS instrumentation to newer shim style.
+
+* Added tracking of callbacks to DNS instrumentation.
+
+* Converted crypto instrumentation to newer shim style.
+
+* Updated domains instrumentation to use an instrumentation shim.
+
+* Refactored the global instrumentation to use the shim API.
+
+* Ported inspector instrumentation to use an instrumentation shim.
+
+* Ported async_hooks based promise instrumentation over to using shims.
+
+* Added shim types for core instrumentations.
+
+* Fixed outbound https call to use example.com to resolve integration test issue.
+
+* Fixed tests for ioredis 4.0.0 and above.
+
+* Improved benchmark comparison output.
+
+* Added `http` benchmark tests.
+
+### 4.8.0 (2018-08-13):
+
+* Added JSON-formatted output to benchmarks to enable automated benchmark comparison.
+
+* Updated the benchmark runner to measure specifically userland CPU overhead.
+
+* Added DatastoreShim benchmarks.
+
+* Fixed MongoDB instrumentation for driver versions greater than 3.0.6.
+
+  Mongo 3.0.6 removed metadata the Agent relied upon to instrument the driver. This
+  fixes that by going back to the old method of manually listing all objects and
+  methods to instrument.
+
+* Implemented enforcement of `max_payload_size_in_bytes` config value.
+
+  Any payload during the harvest sequence that exceeds the configured limit will
+  be discarded.
+
+* Updated MySQL versioned tests to run against the latest release.
+
+### 4.7.0 (2018-07-31):
+
+* Added support for distributed tracing.
+
+  Distributed tracing lets you see the path that a request takes as it travels
+  through your distributed system. By showing the distributed activity through a
+  unified view, you can troubleshoot and understand a complex system better than
+  ever before.
+
+  Distributed tracing is available with an APM Pro or equivalent subscription.
+  To see a complete distributed trace, you need to enable the feature on a set of
+  neighboring services. Enabling distributed tracing changes the behavior of some
+  New Relic features, so carefully consult the [transition guide](https://docs.newrelic.com/docs/transition-guide-distributed-tracing) before
+  you enable this feature.
+
+  To enable distributed tracing, set `distributed_tracing.enabled` to `true` in
+  your `newrelic.js` file, or set `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED` in your
+  environment.
+
+* Added a warning for too-new versions of Node.js during agent startup.
+
+* Appropriately obfuscated SQL statements will now be included in all transaction
+  traces.
+
+  Previously, the agent would only include the SQL statements if the corresponding
+  query was sufficiently slow.
+
+* Added ability to execute instrumentation functions in the context of the segment
+  the segment descriptor is describing.
+
+  All `record*` methods supplied by all instrumentation shim classes now allow for
+  a function to be executed under the context of the segment the record call will
+  produce. This may be done by supplying a function in the `inContext` key for the
+  segment descriptor passed to the record method.
+
+* Reservoirs will now respect setting their size to 0.
+
 ### 4.6.0 (2018-07-24):
 
 * Added full support for Node v10.
@@ -11,6 +158,8 @@
 * Updated tests to use `asyncResource.runInAsyncScope` instead of `emitBefore` and
   `emitAfter`
 
+* Pulled `distributed_tracing` config value from behind `feature_flag`.
+
 ### 4.5.1 (2018-07-18):
 
 - The agent will now properly remerge event data on collection failure.
@@ -33,6 +182,12 @@
 
 * Updated Hapi v17 instrumentation to wrap `server` export, in addition to `Server`.
 
+* `ROOT` segment no longer turns into a span event.
+
+* Fixed span collection when transactions are `sampled=false`.
+
+* Removed `grandparentId` from spans.
+
 ### 4.4.0 (2018-07-12):
 
 * Added config `utilization` env vars to the `BOOLEAN_VARS` set.
@@ -44,6 +199,29 @@
 
 * Added node v10 to the test matrix.
 
+* Converted distributed trace `x-newrelic-trace` header name to `newrelic`.
+
+* Added support for different transport types in distributed tracing.
+
+* Added more tests around priority/sampled attributes on traces and events.
+
+* Lazily calculate transaction priority only when needed.
+
+* Transaction priority is now truncated to 6 decimal places on generation.
+
+* Adaptive sampling now uses the `sampling_target` and
+  `sampling_target_period_in_seconds` configuration values.
+
+  With these configurations, the adaptive sampling window is separated from the
+  harvest window.
+
+* Removed `nr.tripId` attribute from distributed trace intrinsics.
+
+* Default span events to enabled.
+
+  These are still protected behind `feature_flag.distributed_tracing` which defaults
+  to `false`.
+
 ### 4.3.0 (2018-07-09):
 
 * Added `nonce` option for `newrelic.getBrowserTimingHeader()`
@@ -65,7 +243,7 @@
 
 * Fixed issue with tracking external requests to default ports.
 
- Special thanks to Ryan King for pinpointing the cause of this issue.
+  Special thanks to Ryan King for pinpointing the cause of this issue.
 
 * Added extra check for handling arrays of functions when wrapping middleware
   mounters.
@@ -103,6 +281,25 @@
 
 * No longer download gcc on test suites that do not require it.
 
+* Added `url` parameter to `http` external segments.
+
+* Renamed request parameters on external segments.
+
+  Previously these were named just the parameter name (e.g. `/foo?bar=baz` would
+  become the parameter `"bar": "baz"`). Now they are prefixed with
+  `request.parameter`. (e.g. `"request.parameter.bar": "baz"`).
+
+* Added `EventAggregator` base class.
+
+  The `ErrorAggregator` class was refactored and most generic event aggregation
+  logic was moved to the new `EventAggregator` class.
+
+* Added `SpanEvent` and `SpanAggregator` classes.
+
+* Added Span event generation to the trace `end` method.
+
+* Added Span events to harvest cycle steps.
+
 ### 4.1.5 (2018-06-11):
 
 * Make `require()` statements explicitly reference `package.json` as a `.json` file.
@@ -145,6 +342,11 @@
   result in a false `uninstrumented` status, because the agent would interpret
   `redis.js` as the module itself.
 
+* Moved `computeSampled` call to `Transaction` constructor.
+
+  Previously it was only called in `createDistributedTracePayload`, but this
+  gives all transactions a `sampled` value, and potentially a boosted priority.
+
 ### 4.1.4 (2018-06-04):
 
 * Transaction stubs are now created properly in `api#getTransaction`
@@ -168,6 +370,25 @@
 
 * Modularlized configuration constants to improve readability.
 
+* Added `distributed_tracing` feature flag.
+
+* Added `acceptDistributedTracePayload` method to `Transaction`.
+
+* Added `createDistributedTracePayload` method to `Transaction`.
+
+* Updated `Agent#recordSupportability` to not include `Nodejs/` in the default metric name.
+
+* Added distributed tracing methods to `TransactionHandle`.
+
+* Added distributed tracing cases for `http` and `other` metric recorders.
+
+* Implemented `_addDistributedTraceInstrinsics` on `Transaction`.
+
+  If the `distributed_tracing` feature flag is enabled, the agent will ignore old
+  CAT attributes in favor of distributed trace–related ones.
+
+* Added integration tests around better CAT functionality.
+
 ### 4.1.2 (2018-05-22):
 
 * Fixed access to properties on promisified methods.

package/api.js

@@ -397,29 +397,35 @@ API.prototype.setIgnoreTransaction = function setIgnoreTransaction(ignored) {
  *  Optional. Any custom attributes to be displayed in the New Relic UI.
  */
 API.prototype.noticeError = function noticeError(error, customAttributes) {
-  var metric = this.agent.metrics.getOrCreateMetric(
+  const metric = this.agent.metrics.getOrCreateMetric(
     NAMES.SUPPORTABILITY.API + '/noticeError'
   )
   metric.incrementCallCount()
 
-  // If high security mode is on, noticeError is disabled.
-  if (this.agent.config.high_security === true) {
-    logger.warnOnce(
-      'Notice Error',
-      'Notice error API is disabled by high security mode.'
-    )
-    return false
-  } else if (!this.agent.config.api.notice_error_enabled) {
+  if (!this.agent.config.api.notice_error_enabled) {
     logger.debug(
       'Config.api.notice_error_enabled set to false, not collecting error'
     )
     return false
   }
 
+  // If high security mode is on or custom attributes are disabled,
+  // noticeError does not collect custom attributes.
+  if (this.agent.config.high_security === true) {
+    logger.debug(
+      'Passing custom attributes to notice error API is disabled in high security mode.'
+    )
+  } else if (!this.agent.config.api.custom_attributes_enabled) {
+    logger.debug(
+      'Config.api.custom_attributes_enabled set to false, ' +
+      'ignoring custom error attributes.'
+    )
+  }
+
   if (typeof error === 'string') {
     error = new Error(error)
   }
-  var transaction = this.agent.tracer.getTransaction()
+  const transaction = this.agent.tracer.getTransaction()
 
   this.agent.errors.addUserError(transaction, error, customAttributes)
 }
@@ -702,6 +708,13 @@ function createTracer(name, callback) {
   return arity.fixArity(callback, tracer.bindFunction(callback, segment, true))
 }
 
+/**
+ * @callback startSegmentCallback
+ * @param {function} cb
+ *   The function to time with the created segment.
+ * @return {Promise=} Returns a promise if cb returns a promise.
+ */
+
 /**
  * Wraps the given handler in a segment which may optionally be turned into a
  * metric.
@@ -720,7 +733,7 @@ function createTracer(name, callback) {
  *  up on the transaction breakdown table and server breakdown graph. Segments
  *  just show up in transaction traces.
  *
- * @param {function(cb) -> ?Promise} handler
+ * @param {startSegmentCallback} handler
  *  The function to track as a segment.
  *
  * @param {function} [callback]

package/bin/compare-bench-results.js

@@ -0,0 +1,167 @@
+'use strict'
+/* eslint-disable no-console */
+
+const async = require('async')
+const fs = require('fs')
+
+if (process.argv.length !== 4) {
+  console.log('Usage: %s %s <baseline> <upstream>', process.argv[0], process.argv[1])
+  console.log('  <baseline>   JSON file containing benchmark results for upstream')
+  console.log('               upstream comparison.')
+  console.log('  <downstream> JSON file containing benchmark results from')
+  console.log('               downstream branch to compare against base.')
+  process.exit(1)
+}
+
+async.map(process.argv.slice(2), (file, cb) => {
+  fs.readFile(file, {encoding: 'utf8'}, (err, data) => {
+    if (err) {
+      return cb(err)
+    }
+
+    let parsed = null
+    try {
+      parsed = JSON.parse(data)
+    } catch (parseError) {
+      return cb(parseError)
+    }
+
+    cb(null, parsed)
+  })
+}, (err, resultFiles) => {
+  if (err) {
+    console.log('Failed to load files.')
+    console.log(err)
+    process.exit(2)
+  }
+
+  const baseline = resultFiles[0]
+  const downstream = resultFiles[1]
+
+  const baselineFiles = Object.keys(baseline)
+  const downstreamFiles = Object.keys(downstream)
+  const warnings = []
+
+  diffArrays(baselineFiles, downstreamFiles).forEach((file) => {
+    warnings.push(`- **WARNING**: File "${file}" in base but not branch.`)
+  })
+  diffArrays(downstreamFiles, baselineFiles).forEach((file) => {
+    warnings.push(`- **NOTE**: File "${file}" in branch but not base.`)
+  })
+
+  let allPassing = true
+  const details = baselineFiles.sort().map((testFile) => {
+    const base = baseline[testFile]
+    const down = downstream[testFile]
+
+    if (!down) {
+      return [
+        '<details>',
+        `<summary>${testFile} file missing from branch</summary>`,
+        '',
+        '</details>'
+      ].join('\n')
+    }
+
+    let filePassing = true
+    const baseTests = Object.keys(base)
+    const downTests = Object.keys(down)
+
+    diffArrays(baseTests, downTests).forEach((test) => {
+      warnings.push(`- **WARNING**: Test "${test}" in base but not branch.`)
+    })
+    diffArrays(downTests, baseTests).forEach((test) => {
+      warnings.push(`- **NOTE**: Test "${test}" in branch but not base.`)
+    })
+
+    const results = baseTests.sort().map((test) => {
+      const passes = compareResults(base[test], down[test])
+      filePassing = filePassing && passes
+
+      return [
+        '<details>',
+        `<summary>${test}: ${passMark(passes)}</summary>`,
+        '',
+        formatResults(base[test], down[test]),
+        '</details>',
+        ''
+      ].join('\n')
+    }).join('\n')
+    allPassing = allPassing && filePassing
+
+    return [
+      '<details>',
+      `<summary>${testFile}: ${passMark(filePassing)}</summary>`,
+      '',
+      results,
+      '',
+      '-----------------------------------------------------------------------',
+      '</details>'
+    ].join('\n')
+  }).join('\n\n')
+
+  if (warnings.length) {
+    console.log('### WARNINGS')
+    console.log(warnings.join('\n'))
+    console.log('')
+  }
+
+  console.log(`### Benchmark Results: ${passMark(allPassing)}`)
+  console.log('')
+  console.log('### Details')
+  console.log('_Lower is better._')
+  console.log(details)
+
+  if (!allPassing) {
+    process.exitCode = -1
+  }
+})
+
+function diffArrays(a, b) {
+  return a.filter((elem) => !b.includes(elem))
+}
+
+function compareResults(base, down) {
+  const delta = down.mean - base.mean
+  const deltaPercent = delta / base.mean
+  if (Math.abs(delta) < 0.1) {
+    return deltaPercent < 100
+  }
+  return deltaPercent < 2
+}
+
+function passMark(passes) {
+  return passes ? '✔' : '✘'
+}
+
+function formatResults(base, down) {
+  return [
+    'Field | Upstream (ms) | Downstream (ms) | Delta (ms) | Delta (%)',
+    '----- | ------------: | --------------: | ---------: | --------:',
+    formatField('numSamples'),
+    formatField('mean'),
+    formatField('stdDev'),
+    formatField('max'),
+    formatField('min'),
+    formatField('5thPercentile'),
+    formatField('95thPercentile'),
+    formatField('median')
+  ].join('\n')
+
+  function formatField(field) {
+    const baseValue = base[field]
+    const downValue = down[field]
+    const diffValue = downValue - baseValue
+    const diffPercent = (100 * diffValue / baseValue).toFixed(2)
+    const prefix = diffValue >= 0 ? '+' : ''
+
+    return (
+      `${field} | ${fixValue(baseValue)} | ${fixValue(downValue)} |` +
+      ` ${fixValue(diffValue)} | ${prefix}${diffPercent}%`
+    )
+  }
+
+  function fixValue(value) {
+    return value % 1 ? value.toFixed(5) : value
+  }
+}

package/bin/run-bench.js

@@ -36,54 +36,97 @@ if (tests.length === 0 && globs.length === 0) {
   )
 }
 
-a.series([
-  function resolveGlobs(cb) {
-    if (!globs.length) {
-      return cb()
-    }
+class ConsolePrinter {
+  /* eslint-disable no-console */
+  addTest(name, child) {
+    console.log(name)
+    child.stdout.on('data', (d) => process.stdout.write(d))
+    child.stderr.on('data', (d) => process.stderr.write(d))
+    child.once('exit', () => console.log(''))
+  }
+
+  finish() {
+    console.log('')
+  }
+  /* eslint-enable no-console */
+}
+
+class JSONPrinter {
+  constructor() {
+    this._tests = Object.create(null)
+  }
+
+  addTest(name, child) {
+    let output = ''
+    this._tests[name] = null
+    child.stdout.on('data', (d) => output += d.toString())
+    child.stdout.on('end', () => this._tests[name] = JSON.parse(output))
+    child.stderr.on('data', (d) => process.stderr.write(d))
+  }
+
+  finish() {
+    /* eslint-disable no-console */
+    console.log(JSON.stringify(this._tests, null, 2))
+    /* eslint-enable no-console */
+  }
+}
+
+run()
+
+function run() {
+  const printer = opts.json ? new JSONPrinter() : new ConsolePrinter()
 
-    a.map(globs, glob, function afterGlobbing(err, resolved) {
-      if (err) {
-        console.error('Failed to glob:', err)
-        process.exitCode = -1
-        return cb(err)
+  a.series([
+    function resolveGlobs(cb) {
+      if (!globs.length) {
+        return cb()
       }
-      resolved.forEach(function mergeResolved(files) {
-        files.forEach(function mergeFile(file) {
-          if (tests.indexOf(file) === -1) {
-            tests.push(file)
-          }
+
+      a.map(globs, glob, function afterGlobbing(err, resolved) {
+        if (err) {
+          console.error('Failed to glob:', err)
+          process.exitCode = -1
+          return cb(err)
+        }
+        resolved.forEach(function mergeResolved(files) {
+          files.forEach(function mergeFile(file) {
+            if (tests.indexOf(file) === -1) {
+              tests.push(file)
+            }
+          })
         })
+        cb()
       })
-      cb()
-    })
-  },
-  function runBenchmarks(cb) {
-    tests.sort()
-    a.eachSeries(tests, function spawnEachFile(file, cb) {
-      var test = path.relative(benchpath, file)
-
-      console.log(test)
-      var args = ['--expose-gc', file]
-      if (opts.inspect) {
-        args.unshift('--inspect-brk')
-      }
-      var child = cp.spawn('node', args, {cwd: cwd, stdio: 'inherit'})
-      child.on('error', cb)
-      child.on('exit', function onChildExit(code) {
-        console.log('')
-        if (code) {
-          return cb(new Error('Benchmark exited with code ' + code))
+    },
+    function runBenchmarks(cb) {
+      tests.sort()
+      a.eachSeries(tests, function spawnEachFile(file, cb) {
+        var test = path.relative(benchpath, file)
+
+        var args = [file]
+        if (opts.inspect) {
+          args.unshift('--inspect-brk')
+        }
+        var child = cp.spawn('node', args, {cwd: cwd, stdio: 'pipe'})
+        printer.addTest(test, child)
+
+        child.on('error', cb)
+        child.on('exit', function onChildExit(code) {
+          if (code) {
+            return cb(new Error('Benchmark exited with code ' + code))
+          }
+          cb()
+        })
+      }, function afterSpawnEachFile(err) {
+        if (err) {
+          console.error('Spawning failed:', err)
+          process.exitCode = -2
+          return cb(err)
         }
         cb()
       })
-    }, function afterSpawnEachFile(err) {
-      if (err) {
-        console.error('Spawning failed:', err)
-        process.exitCode = -2
-        return cb(err)
-      }
-      cb()
-    })
-  }
-])
+    }
+  ], () => {
+    printer.finish()
+  })
+}

package/bin/travis-setup.sh

@@ -1,9 +1,10 @@
 #! /bin/bash
 
-function get_gcc_version {
-  local gcc_version_match='[[:digit:]]\.[[:digit:]]\.[[:digit:]]'
-  local gcc_version=`$CC --version 2>/dev/null | grep -o "$gcc_version_match" | head -1`
-  echo $gcc_version | grep -o '[[:digit:]]' | head -1
+
+function get_version {
+  local num='[[:digit:]][[:digit:]]*' # Grep doesn't have `+` operator.
+  local version=`$1 --version 2>/dev/null | grep -o "$num\.$num\.$num" | head -1`
+  echo $version | grep -o "$num" | head -1
 }
 
 TOOLCHAIN_ADDED="false"
@@ -15,15 +16,25 @@ function add_toolchain {
   TOOLCHAIN_ADDED="true"
 }
 
-# Only upgrade GCC if we need to.
 
 if [ "$SUITE" = "versioned" ]; then
-  if [ "$(get_gcc_version)" != "5" ]; then
+
+  # GCC 5 is the lowest version of GCC we can use.
+  if [ "$(get_version gcc)" == "4" ]; then
     echo " --- upgrading GCC --- "
     add_toolchain
     ./bin/travis-install-gcc5.sh > /dev/null
   else
-    echo " --- not upgrading GCC --- "
+    echo " --- not upgrading GCC ($(gcc --version)) --- "
+  fi
+
+  # npm 2 has an issue installing correctly for the superagent versioned tests.
+  # TODO: Remove this check when deprecating Node <5.
+  if [ "$(get_version npm)" == "2" ]; then
+    echo " -- upgrading npm --- "
+    npm install -g npm@3
+  else
+    echo " --- not upgrading npm ($(npm --version)) --- "
   fi
 
   echo " --- installing $SUITE requirements --- "