newrelic 4.5.0 → 4.8.0

package/NEWS.md

@@ -1,3 +1,83 @@
+### 4.8.0 (2018-08-13):
+
+* Added JSON-formatted output to benchmarks to enable automated benchmark comparison.
+
+* Updated the benchmark runner to measure specifically userland CPU overhead.
+
+* Added DatastoreShim benchmarks.
+
+* Fixed MongoDB instrumentation for driver versions greater than 3.0.6.
+
+  Mongo 3.0.6 removed metadata the Agent relied upon to instrument the driver. This
+  fixes that by going back to the old method of manually listing all objects and
+  methods to instrument.
+
+* Implemented enforcement of `max_payload_size_in_bytes` config value.
+
+  Any payload during the harvest sequence that exceeds the configured limit will
+  be discarded.
+
+* Updated MySQL versioned tests to run against the latest release.
+
+### 4.7.0 (2018-07-31):
+
+* Added support for distributed tracing.
+
+  Distributed tracing lets you see the path that a request takes as it travels
+  through your distributed system. By showing the distributed activity through a
+  unified view, you can troubleshoot and understand a complex system better than
+  ever before.
+
+  Distributed tracing is available with an APM Pro or equivalent subscription.
+  To see a complete distributed trace, you need to enable the feature on a set of
+  neighboring services. Enabling distributed tracing changes the behavior of some
+  New Relic features, so carefully consult the [transition guide](https://docs.newrelic.com/docs/transition-guide-distributed-tracing) before
+  you enable this feature.
+
+  To enable distributed tracing, set `distributed_tracing.enabled` to `true` in
+  your `newrelic.js` file, or set `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED` in your
+  environment.
+
+* Added a warning for too-new versions of Node.js during agent startup.
+
+* Appropriately obfuscated SQL statements will now be included in all transaction
+  traces.
+
+  Previously, the agent would only include the SQL statements if the corresponding
+  query was sufficiently slow.
+
+* Added ability to execute instrumentation functions in the context of the segment
+  the segment descriptor is describing.
+
+  All `record*` methods supplied by all instrumentation shim classes now allow for
+  a function to be executed under the context of the segment the record call will
+  produce. This may be done by supplying a function in the `inContext` key for the
+  segment descriptor passed to the record method.
+
+* Reservoirs will now respect setting their size to 0.
+
+### 4.6.0 (2018-07-24):
+
+* Added full support for Node v10.
+
+* Added instrumentation for `crypto.scrypt`.
+
+* Added instrumentation for `fs.realpath.native`.
+
+* Added instrumentation for `process.setUncaughtExceptionCaptureCallback`.
+
+* Updated tests to use `asyncResource.runInAsyncScope` instead of `emitBefore` and
+  `emitAfter`
+
+* Pulled `distributed_tracing` config value from behind `feature_flag`.
+
+### 4.5.1 (2018-07-18):
+
+- The agent will now properly remerge event data on collection failure.
+
+  Previously, the agent wouldn't observe the correct format for remerging, causing
+  undefined events to be pushed into the reservoir.
+
 ### 4.5.0 (2018-07-16):
 
 * Feature flags may now be set from environment variables.
@@ -13,6 +93,12 @@
 
 * Updated Hapi v17 instrumentation to wrap `server` export, in addition to `Server`.
 
+* `ROOT` segment no longer turns into a span event.
+
+* Fixed span collection when transactions are `sampled=false`.
+
+* Removed `grandparentId` from spans.
+
 ### 4.4.0 (2018-07-12):
 
 * Added config `utilization` env vars to the `BOOLEAN_VARS` set.
@@ -24,6 +110,29 @@
 
 * Added node v10 to the test matrix.
 
+* Converted distributed trace `x-newrelic-trace` header name to `newrelic`.
+
+* Added support for different transport types in distributed tracing.
+
+* Added more tests around priority/sampled attributes on traces and events.
+
+* Lazily calculate transaction priority only when needed.
+
+* Transaction priority is now truncated to 6 decimal places on generation.
+
+* Adaptive sampling now uses the `sampling_target` and
+  `sampling_target_period_in_seconds` configuration values.
+
+  With these configurations, the adaptive sampling window is separated from the
+  harvest window.
+
+* Removed `nr.tripId` attribute from distributed trace intrinsics.
+
+* Default span events to enabled.
+
+  These are still protected behind `feature_flag.distributed_tracing` which defaults
+  to `false`.
+
 ### 4.3.0 (2018-07-09):
 
 * Added `nonce` option for `newrelic.getBrowserTimingHeader()`
@@ -45,7 +154,7 @@
 
 * Fixed issue with tracking external requests to default ports.
 
- Special thanks to Ryan King for pinpointing the cause of this issue.
+  Special thanks to Ryan King for pinpointing the cause of this issue.
 
 * Added extra check for handling arrays of functions when wrapping middleware
   mounters.
@@ -83,6 +192,25 @@
 
 * No longer download gcc on test suites that do not require it.
 
+* Added `url` parameter to `http` external segments.
+
+* Renamed request parameters on external segments.
+
+  Previously these were named just the parameter name (e.g. `/foo?bar=baz` would
+  become the parameter `"bar": "baz"`). Now they are prefixed with
+  `request.parameter`. (e.g. `"request.parameter.bar": "baz"`).
+
+* Added `EventAggregator` base class.
+
+  The `ErrorAggregator` class was refactored and most generic event aggregation
+  logic was moved to the new `EventAggregator` class.
+
+* Added `SpanEvent` and `SpanAggregator` classes.
+
+* Added Span event generation to the trace `end` method.
+
+* Added Span events to harvest cycle steps.
+
 ### 4.1.5 (2018-06-11):
 
 * Make `require()` statements explicitly reference `package.json` as a `.json` file.
@@ -125,6 +253,11 @@
   result in a false `uninstrumented` status, because the agent would interpret
   `redis.js` as the module itself.
 
+* Moved `computeSampled` call to `Transaction` constructor.
+
+  Previously it was only called in `createDistributedTracePayload`, but this
+  gives all transactions a `sampled` value, and potentially a boosted priority.
+
 ### 4.1.4 (2018-06-04):
 
 * Transaction stubs are now created properly in `api#getTransaction`
@@ -148,6 +281,25 @@
 
 * Modularlized configuration constants to improve readability.
 
+* Added `distributed_tracing` feature flag.
+
+* Added `acceptDistributedTracePayload` method to `Transaction`.
+
+* Added `createDistributedTracePayload` method to `Transaction`.
+
+* Updated `Agent#recordSupportability` to not include `Nodejs/` in the default metric name.
+
+* Added distributed tracing methods to `TransactionHandle`.
+
+* Added distributed tracing cases for `http` and `other` metric recorders.
+
+* Implemented `_addDistributedTraceInstrinsics` on `Transaction`.
+
+  If the `distributed_tracing` feature flag is enabled, the agent will ignore old
+  CAT attributes in favor of distributed trace–related ones.
+
+* Added integration tests around better CAT functionality.
+
 ### 4.1.2 (2018-05-22):
 
 * Fixed access to properties on promisified methods.

package/api.js

@@ -702,6 +702,13 @@ function createTracer(name, callback) {
   return arity.fixArity(callback, tracer.bindFunction(callback, segment, true))
 }
 
+/**
+ * @callback startSegmentCallback
+ * @param {function} cb
+ *   The function to time with the created segment.
+ * @return {Promise=} Returns a promise if cb returns a promise.
+ */
+
 /**
  * Wraps the given handler in a segment which may optionally be turned into a
  * metric.
@@ -720,7 +727,7 @@ function createTracer(name, callback) {
  *  up on the transaction breakdown table and server breakdown graph. Segments
  *  just show up in transaction traces.
  *
- * @param {function(cb) -> ?Promise} handler
+ * @param {startSegmentCallback} handler
  *  The function to track as a segment.
  *
  * @param {function} [callback]

package/bin/compare-bench-results.js

@@ -0,0 +1,159 @@
+'use strict'
+/* eslint-disable no-console */
+
+const async = require('async')
+const fs = require('fs')
+
+if (process.argv.length !== 4) {
+  console.log('Usage: %s %s <baseline> <upstream>', process.argv[0], process.argv[1])
+  console.log('  <baseline>   JSON file containing benchmark results for upstream')
+  console.log('               upstream comparison.')
+  console.log('  <downstream> JSON file containing benchmark results from')
+  console.log('               downstream branch to compare against base.')
+  process.exit(1)
+}
+
+async.map(process.argv.slice(2), (file, cb) => {
+  fs.readFile(file, {encoding: 'utf8'}, (err, data) => {
+    if (err) {
+      return cb(err)
+    }
+
+    let parsed = null
+    try {
+      parsed = JSON.parse(data)
+    } catch (parseError) {
+      return cb(parseError)
+    }
+
+    cb(null, parsed)
+  })
+}, (err, resultFiles) => {
+  if (err) {
+    console.log('Failed to load files.')
+    console.log(err)
+    process.exit(2)
+  }
+
+  const baseline = resultFiles[0]
+  const downstream = resultFiles[1]
+
+  const baselineFiles = Object.keys(baseline)
+  const downstreamFiles = Object.keys(downstream)
+  const warnings = []
+
+  diffArrays(baselineFiles, downstreamFiles).forEach((file) => {
+    warnings.push(`- **WARNING**: File "${file}" in base but not branch.`)
+  })
+  diffArrays(downstreamFiles, baselineFiles).forEach((file) => {
+    warnings.push(`- **NOTE**: File "${file}" in branch but not base.`)
+  })
+
+  let allPassing = true
+  const details = baselineFiles.sort().map((testFile) => {
+    const base = baseline[testFile]
+    const down = downstream[testFile]
+
+    if (!down) {
+      return [
+        '<details>',
+        `<summary>${testFile} file missing from branch</summary>`,
+        '',
+        '</details>'
+      ].join('\n')
+    }
+
+    let filePassing = true
+    const baseTests = Object.keys(base)
+    const downTests = Object.keys(down)
+
+    diffArrays(baseTests, downTests).forEach((test) => {
+      warnings.push(`- **WARNING**: Test "${test}" in base but not branch.`)
+    })
+    diffArrays(downTests, baseTests).forEach((test) => {
+      warnings.push(`- **NOTE**: Test "${test}" in branch but not base.`)
+    })
+
+    const results = baseTests.sort().map((test) => {
+      const passes = compareResults(base[test], down[test])
+      filePassing = filePassing && passes
+
+      return [
+        '<details>',
+        `<summary>${test}: ${passMark(passes)}</summary>`,
+        '',
+        formatResults(base[test], down[test]),
+        '</details>',
+        ''
+      ].join('\n')
+    }).join('\n')
+    allPassing = allPassing && filePassing
+
+    return [
+      '<details>',
+      `<summary>${testFile}: ${passMark(filePassing)}</summary>`,
+      '',
+      results,
+      '</details>'
+    ].join('\n')
+  }).join('\n\n-----------------------------------------------------------\n\n')
+
+  if (warnings.length) {
+    console.log('### WARNINGS')
+    console.log(warnings.join('\n'))
+    console.log('')
+  }
+
+  console.log(`### Benchmark Results: ${passMark(allPassing)}`)
+  console.log('')
+  console.log('### Details')
+  console.log(details)
+
+  if (!allPassing) {
+    process.exitCode = -1
+  }
+})
+
+function diffArrays(a, b) {
+  return a.filter((elem) => !b.includes(elem))
+}
+
+function compareResults(base, down) {
+  const delta = base.mean - down.mean
+  const deltaPercent = delta / base.mean
+  return delta < 1 || deltaPercent < 2
+}
+
+function passMark(passes) {
+  return passes ? '✔' : '✘'
+}
+
+function formatResults(base, down) {
+  return [
+    `- ${formatField('numSamples', base, down)}`,
+    `- ${formatField('mean', base, down)}`,
+    `- ${formatField('stdDev', base, down)}`,
+    `- ${formatField('max', base, down)}`,
+    `- ${formatField('min', base, down)}`,
+    `- ${formatField('5thPercentile', base, down)}`,
+    `- ${formatField('95thPercentile', base, down)}`,
+    `- ${formatField('median', base, down)}`,
+  ].join('\n')
+}
+
+function formatField(field, base, down) {
+  const baseValue = base[field]
+  const downValue = down[field]
+  const diffValue = baseValue - downValue
+  const diffPercent = (100 * diffValue / baseValue).toFixed(2)
+  const prefix = diffValue >= 0 ? '+' : ''
+
+  return (
+    `${field}: ${fixValue(baseValue)} - ${fixValue(downValue)} =` +
+    ` ${fixValue(diffValue)} (${prefix}${diffPercent}%)`
+  )
+}
+
+function fixValue(value) {
+  return value % 1 ? value.toFixed(5) : value
+}

package/bin/run-bench.js

@@ -36,54 +36,97 @@ if (tests.length === 0 && globs.length === 0) {
   )
 }
 
-a.series([
-  function resolveGlobs(cb) {
-    if (!globs.length) {
-      return cb()
-    }
+class ConsolePrinter {
+  /* eslint-disable no-console */
+  addTest(name, child) {
+    console.log(name)
+    child.stdout.on('data', (d) => process.stdout.write(d))
+    child.stderr.on('data', (d) => process.stderr.write(d))
+    child.once('exit', () => console.log(''))
+  }
+
+  finish() {
+    console.log('')
+  }
+  /* eslint-enable no-console */
+}
+
+class JSONPrinter {
+  constructor() {
+    this._tests = Object.create(null)
+  }
+
+  addTest(name, child) {
+    let output = ''
+    this._tests[name] = null
+    child.stdout.on('data', (d) => output += d.toString())
+    child.stdout.on('end', () => this._tests[name] = JSON.parse(output))
+    child.stderr.on('data', (d) => process.stderr.write(d))
+  }
+
+  finish() {
+    /* eslint-disable no-console */
+    console.log(JSON.stringify(this._tests, null, 2))
+    /* eslint-enable no-console */
+  }
+}
+
+run()
+
+function run() {
+  const printer = opts.json ? new JSONPrinter() : new ConsolePrinter()
 
-    a.map(globs, glob, function afterGlobbing(err, resolved) {
-      if (err) {
-        console.error('Failed to glob:', err)
-        process.exitCode = -1
-        return cb(err)
+  a.series([
+    function resolveGlobs(cb) {
+      if (!globs.length) {
+        return cb()
       }
-      resolved.forEach(function mergeResolved(files) {
-        files.forEach(function mergeFile(file) {
-          if (tests.indexOf(file) === -1) {
-            tests.push(file)
-          }
+
+      a.map(globs, glob, function afterGlobbing(err, resolved) {
+        if (err) {
+          console.error('Failed to glob:', err)
+          process.exitCode = -1
+          return cb(err)
+        }
+        resolved.forEach(function mergeResolved(files) {
+          files.forEach(function mergeFile(file) {
+            if (tests.indexOf(file) === -1) {
+              tests.push(file)
+            }
+          })
         })
+        cb()
       })
-      cb()
-    })
-  },
-  function runBenchmarks(cb) {
-    tests.sort()
-    a.eachSeries(tests, function spawnEachFile(file, cb) {
-      var test = path.relative(benchpath, file)
-
-      console.log(test)
-      var args = ['--expose-gc', file]
-      if (opts.inspect) {
-        args.unshift('--inspect-brk')
-      }
-      var child = cp.spawn('node', args, {cwd: cwd, stdio: 'inherit'})
-      child.on('error', cb)
-      child.on('exit', function onChildExit(code) {
-        console.log('')
-        if (code) {
-          return cb(new Error('Benchmark exited with code ' + code))
+    },
+    function runBenchmarks(cb) {
+      tests.sort()
+      a.eachSeries(tests, function spawnEachFile(file, cb) {
+        var test = path.relative(benchpath, file)
+
+        var args = [file]
+        if (opts.inspect) {
+          args.unshift('--inspect-brk')
+        }
+        var child = cp.spawn('node', args, {cwd: cwd, stdio: 'pipe'})
+        printer.addTest(test, child)
+
+        child.on('error', cb)
+        child.on('exit', function onChildExit(code) {
+          if (code) {
+            return cb(new Error('Benchmark exited with code ' + code))
+          }
+          cb()
+        })
+      }, function afterSpawnEachFile(err) {
+        if (err) {
+          console.error('Spawning failed:', err)
+          process.exitCode = -2
+          return cb(err)
         }
         cb()
       })
-    }, function afterSpawnEachFile(err) {
-      if (err) {
-        console.error('Spawning failed:', err)
-        process.exitCode = -2
-        return cb(err)
-      }
-      cb()
-    })
-  }
-])
+    }
+  ], () => {
+    printer.finish()
+  })
+}

package/index.js

@@ -1,21 +1,22 @@
 'use strict'
 
 // Record opening times before loading any other files.
-var preAgentTime = process.uptime()
-var agentStart = Date.now()
+const preAgentTime = process.uptime()
+const agentStart = Date.now()
 
 // Load unwrapped core now to ensure it gets the freshest properties.
 require('./lib/util/unwrapped-core')
 
-var featureFlags = require('./lib/feature_flags').prerelease
-var logger = require('./lib/logger')
-var psemver = require('./lib/util/process-version')
+const featureFlags = require('./lib/feature_flags').prerelease
+const psemver = require('./lib/util/process-version')
+let logger = require('./lib/logger') // Gets re-loaded after initialization.
 
 
-var agentVersion = require('./package.json').version
+const pkgJSON = require('./package.json')
 logger.info(
-  "Using New Relic for Node.js. Agent version: %s; Node version: %s.",
-  agentVersion, process.version
+  'Using New Relic for Node.js. Agent version: %s; Node version: %s.',
+  pkgJSON.version,
+  process.version
 )
 
 if (require.cache.__NR_cache) {
@@ -49,6 +50,13 @@ function initialize() {
 
       logger.error(message)
       throw new Error(message)
+    } else if (!psemver.satisfies(pkgJSON.engines.node)) {
+      logger.warn(
+        'New Relic for Node.js %s has not been tested on Node.js %s. Please ' +
+        'update the agent or downgrade your version of Node.js',
+        pkgJSON.version,
+        process.version
+      )
     }
 
     logger.debug("Current working directory at module load is %s.", process.cwd())

package/lib/agent.js

@@ -516,7 +516,7 @@ function _addIntrinsicAttrsFromTransaction(transaction) {
 
   // If CAT is enabled, extract the data gleened from the CAT headers.
   if (this.config.cross_application_tracer.enabled) {
-    if (this.config.feature_flag.distributed_tracing) {
+    if (this.config.distributed_tracing.enabled) {
       transaction.addDistributedTraceIntrinsics(intrinsicAttributes)
       if (transaction.parentSpanId) {
         intrinsicAttributes.parentSpanId = transaction.parentSpanId

package/lib/collector/remote-method.js

@@ -10,6 +10,7 @@ var stringify = require('json-stringify-safe')
 var Sink = require('../util/stream-sink')
 var agents = require('./http-agents')
 var certificates = require('./ssl/certificates')
+const isValidLength = require('../util/byte-limit').isValidLength
 
 /*
  *
@@ -147,8 +148,19 @@ RemoteMethod.prototype._safeRequest = function _safeRequest(options) {
   var protocol = 'https'
   var logConfig = this._config.logging
   var auditLog = this._config.audit_log
+  const maxPayloadSize = this._config.max_payload_size_in_bytes
   var level = 'trace'
 
+  if (!isValidLength(options.body, maxPayloadSize + 1)) {
+    logger.warn(
+      'The payload size %d being sent to method %s exceeded the maximum size of %d',
+      Buffer.byteLength(options.body, 'utf8'),
+      this.name,
+      maxPayloadSize
+    )
+    throw new Error('Maximum payload size exceeded')
+  }
+
   // If trace level is not explicity enabled check to see if the audit log is
   // enabled.
   if (logConfig != null && logConfig.level !== 'trace' && auditLog.enabled) {

package/lib/config/default.js

@@ -791,6 +791,23 @@ exports.config = {
     enabled: true
   },
 
+  /**
+   * Controls the method of cross agent tracing in the agent.
+   * Distributed tracing lets you see the path that a request takes through your
+   * distributed system. Enabling distributed tracing changes the behavior of some
+   * New Relic features, so carefully consult the transition guide before you enable
+   * this feature: https://docs.newrelic.com/docs/transition-guide-distributed-tracing
+   * Default is false.
+   */
+  distributed_tracing: {
+    /**
+     * Enables/disables distributed tracing.
+     *
+     * @env NEW_RELIC_DISTRIBUTED_TRACING_ENABLED
+     */
+    enabled: false
+  },
+
   /**
    * Controls the use of cross-application tracing.
    *

package/lib/config/env.js

@@ -114,6 +114,9 @@ const ENV_MAPPING = {
   span_events: {
     enabled: 'NEW_RELIC_SPAN_EVENTS_ENABLED'
   },
+  distributed_tracing: {
+    enabled: 'NEW_RELIC_DISTRIBUTED_TRACING_ENABLED'
+  },
   message_tracer: {
     segment_parameters: {
       enabled: 'NEW_RELIC_MESSAGE_TRACER_SEGMENT_PARAMETERS_ENABLED'
@@ -167,6 +170,7 @@ const BOOLEAN_VARS = new Set([
   'NEW_RELIC_HIGH_SECURITY',
   'NEW_RELIC_SLOW_SQL_ENABLED',
   'NEW_RELIC_SPAN_EVENTS_ENABLED',
+  'NEW_RELIC_DISTRIBUTED_TRACING_ENABLED',
   'NEW_RELIC_LOG_ENABLED',
   'NEW_RELIC_AUDIT_LOG_ENABLED',
   'NEW_RELIC_DATASTORE_DATABASE_NAME_REPORTING_ENABLED',

package/lib/config/index.js

@@ -21,6 +21,7 @@ var logger = null // Lazy-loaded in `initialize`.
 /**
  * CONSTANTS -- we gotta lotta 'em
  */
+const DEFAULT_MAX_PAYLOAD_SIZE_IN_BYTES = 1000000
 const DEFAULT_CONFIG_PATH = require.resolve('./default')
 const DEFAULT_CONFIG = require('./default').config
 const DEFAULT_FILENAME = 'newrelic.js'
@@ -126,6 +127,7 @@ function Config(config) {
   this.trusted_account_key = null
   this.sampling_target = 10
   this.sampling_target_period_in_seconds = 60
+  this.max_payload_size_in_bytes = DEFAULT_MAX_PAYLOAD_SIZE_IN_BYTES
 
   // how frequently harvester runs
   this.data_report_period = 60
@@ -273,6 +275,7 @@ Config.prototype._fromServer = function _fromServer(params, key) {
     case 'collect_errors':
     case 'collect_traces':
     case 'product_level':
+    case 'max_payload_size_in_bytes':
     case 'sampling_target':
     case 'sampling_target_period_in_seconds':
     case 'trusted_account_ids':