newrelic 4.6.0 → 4.7.0

package/NEWS.md

@@ -1,3 +1,40 @@
+### 4.7.0 (2018-07-31):
+
+* Added support for distributed tracing.
+
+  Distributed tracing lets you see the path that a request takes as it travels
+  through your distributed system. By showing the distributed activity through a
+  unified view, you can troubleshoot and understand a complex system better than
+  ever before.
+
+  Distributed tracing is available with an APM Pro or equivalent subscription.
+  To see a complete distributed trace, you need to enable the feature on a set of
+  neighboring services. Enabling distributed tracing changes the behavior of some
+  New Relic features, so carefully consult the [transition guide](https://docs.newrelic.com/docs/transition-guide-distributed-tracing) before
+  you enable this feature.
+
+  To enable distributed tracing, set `distributed_tracing.enabled` to `true` in
+  your `newrelic.js` file, or set `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED` in your
+  environment.
+
+* Added a warning for too-new versions of Node.js during agent startup.
+
+* Appropriately obfuscated SQL statements will now be included in all transaction
+  traces.
+
+  Previously, the agent would only include the SQL statements if the corresponding
+  query was sufficiently slow.
+
+* Added ability to execute instrumentation functions in the context of the segment
+  the segment descriptor is describing.
+
+  All `record*` methods supplied by all instrumentation shim classes now allow for
+  a function to be executed under the context of the segment the record call will
+  produce. This may be done by supplying a function in the `inContext` key for the
+  segment descriptor passed to the record method.
+
+* Reservoirs will now respect setting their size to 0.
+
 ### 4.6.0 (2018-07-24):
 
 * Added full support for Node v10.
@@ -11,6 +48,8 @@
 * Updated tests to use `asyncResource.runInAsyncScope` instead of `emitBefore` and
   `emitAfter`
 
+* Pulled `distributed_tracing` config value from behind `feature_flag`.
+
 ### 4.5.1 (2018-07-18):
 
 - The agent will now properly remerge event data on collection failure.
@@ -33,6 +72,12 @@
 
 * Updated Hapi v17 instrumentation to wrap `server` export, in addition to `Server`.
 
+* `ROOT` segment no longer turns into a span event.
+
+* Fixed span collection when transactions are `sampled=false`.
+
+* Removed `grandparentId` from spans.
+
 ### 4.4.0 (2018-07-12):
 
 * Added config `utilization` env vars to the `BOOLEAN_VARS` set.
@@ -44,6 +89,29 @@
 
 * Added node v10 to the test matrix.
 
+* Converted distributed trace `x-newrelic-trace` header name to `newrelic`.
+
+* Added support for different transport types in distributed tracing.
+
+* Added more tests around priority/sampled attributes on traces and events.
+
+* Lazily calculate transaction priority only when needed.
+
+* Transaction priority is now truncated to 6 decimal places on generation.
+
+* Adaptive sampling now uses the `sampling_target` and
+  `sampling_target_period_in_seconds` configuration values.
+
+  With these configurations, the adaptive sampling window is separated from the
+  harvest window.
+
+* Removed `nr.tripId` attribute from distributed trace intrinsics.
+
+* Default span events to enabled.
+
+  These are still protected behind `feature_flag.distributed_tracing` which defaults
+  to `false`.
+
 ### 4.3.0 (2018-07-09):
 
 * Added `nonce` option for `newrelic.getBrowserTimingHeader()`
@@ -65,7 +133,7 @@
 
 * Fixed issue with tracking external requests to default ports.
 
- Special thanks to Ryan King for pinpointing the cause of this issue.
+  Special thanks to Ryan King for pinpointing the cause of this issue.
 
 * Added extra check for handling arrays of functions when wrapping middleware
   mounters.
@@ -103,6 +171,25 @@
 
 * No longer download gcc on test suites that do not require it.
 
+* Added `url` parameter to `http` external segments.
+
+* Renamed request parameters on external segments.
+
+  Previously these were named just the parameter name (e.g. `/foo?bar=baz` would
+  become the parameter `"bar": "baz"`). Now they are prefixed with
+  `request.parameter`. (e.g. `"request.parameter.bar": "baz"`).
+
+* Added `EventAggregator` base class.
+
+  The `ErrorAggregator` class was refactored and most generic event aggregation
+  logic was moved to the new `EventAggregator` class.
+
+* Added `SpanEvent` and `SpanAggregator` classes.
+
+* Added Span event generation to the trace `end` method.
+
+* Added Span events to harvest cycle steps.
+
 ### 4.1.5 (2018-06-11):
 
 * Make `require()` statements explicitly reference `package.json` as a `.json` file.
@@ -145,6 +232,11 @@
   result in a false `uninstrumented` status, because the agent would interpret
   `redis.js` as the module itself.
 
+* Moved `computeSampled` call to `Transaction` constructor.
+
+  Previously it was only called in `createDistributedTracePayload`, but this
+  gives all transactions a `sampled` value, and potentially a boosted priority.
+
 ### 4.1.4 (2018-06-04):
 
 * Transaction stubs are now created properly in `api#getTransaction`
@@ -168,6 +260,25 @@
 
 * Modularlized configuration constants to improve readability.
 
+* Added `distributed_tracing` feature flag.
+
+* Added `acceptDistributedTracePayload` method to `Transaction`.
+
+* Added `createDistributedTracePayload` method to `Transaction`.
+
+* Updated `Agent#recordSupportability` to not include `Nodejs/` in the default metric name.
+
+* Added distributed tracing methods to `TransactionHandle`.
+
+* Added distributed tracing cases for `http` and `other` metric recorders.
+
+* Implemented `_addDistributedTraceInstrinsics` on `Transaction`.
+
+  If the `distributed_tracing` feature flag is enabled, the agent will ignore old
+  CAT attributes in favor of distributed trace–related ones.
+
+* Added integration tests around better CAT functionality.
+
 ### 4.1.2 (2018-05-22):
 
 * Fixed access to properties on promisified methods.

package/api.js

@@ -702,6 +702,13 @@ function createTracer(name, callback) {
   return arity.fixArity(callback, tracer.bindFunction(callback, segment, true))
 }
 
+/**
+ * @callback startSegmentCallback
+ * @param {function} cb
+ *   The function to time with the created segment.
+ * @return {Promise=} Returns a promise if cb returns a promise.
+ */
+
 /**
  * Wraps the given handler in a segment which may optionally be turned into a
  * metric.
@@ -720,7 +727,7 @@ function createTracer(name, callback) {
  *  up on the transaction breakdown table and server breakdown graph. Segments
  *  just show up in transaction traces.
  *
- * @param {function(cb) -> ?Promise} handler
+ * @param {startSegmentCallback} handler
  *  The function to track as a segment.
  *
  * @param {function} [callback]

package/index.js

@@ -1,21 +1,22 @@
 'use strict'
 
 // Record opening times before loading any other files.
-var preAgentTime = process.uptime()
-var agentStart = Date.now()
+const preAgentTime = process.uptime()
+const agentStart = Date.now()
 
 // Load unwrapped core now to ensure it gets the freshest properties.
 require('./lib/util/unwrapped-core')
 
-var featureFlags = require('./lib/feature_flags').prerelease
-var logger = require('./lib/logger')
-var psemver = require('./lib/util/process-version')
+const featureFlags = require('./lib/feature_flags').prerelease
+const psemver = require('./lib/util/process-version')
+let logger = require('./lib/logger') // Gets re-loaded after initialization.
 
 
-var agentVersion = require('./package.json').version
+const pkgJSON = require('./package.json')
 logger.info(
-  "Using New Relic for Node.js. Agent version: %s; Node version: %s.",
-  agentVersion, process.version
+  'Using New Relic for Node.js. Agent version: %s; Node version: %s.',
+  pkgJSON.version,
+  process.version
 )
 
 if (require.cache.__NR_cache) {
@@ -49,6 +50,13 @@ function initialize() {
 
       logger.error(message)
       throw new Error(message)
+    } else if (!psemver.satisfies(pkgJSON.engines.node)) {
+      logger.warn(
+        'New Relic for Node.js %s has not been tested on Node.js %s. Please ' +
+        'update the agent or downgrade your version of Node.js',
+        pkgJSON.version,
+        process.version
+      )
     }
 
     logger.debug("Current working directory at module load is %s.", process.cwd())

package/lib/config/default.js

@@ -793,6 +793,11 @@ exports.config = {
 
   /**
    * Controls the method of cross agent tracing in the agent.
+   * Distributed tracing lets you see the path that a request takes through your
+   * distributed system. Enabling distributed tracing changes the behavior of some
+   * New Relic features, so carefully consult the transition guide before you enable
+   * this feature: https://docs.newrelic.com/docs/transition-guide-distributed-tracing
+   * Default is false.
    */
   distributed_tracing: {
     /**

package/lib/db/tracer.js

@@ -46,27 +46,36 @@ QueryTracer.prototype.merge = function merge(tracer) {
 
 QueryTracer.prototype.addQuery = function addQuery(segment, type, query, trace) {
   const ttConfig = this.config.transaction_tracer
-  if (segment.getDurationInMillis() < ttConfig.explain_threshold) {
-    return
-  }
-  const slowQuery = new SlowQuery(segment, type, query, trace)
 
+  // If DT is enabled and the segment is part of a sampled transaction
+  // (i.e. we are creating a span event for this segment), then we need
+  // to collect the sql trace.
+  var slowQuery
   switch (ttConfig.record_sql) {
     case 'raw':
+      slowQuery = new SlowQuery(segment, type, query, trace)
       logger.trace('recording raw sql')
       segment.parameters.sql = slowQuery.query
       break
     case 'obfuscated':
+      slowQuery = new SlowQuery(segment, type, query, trace)
       logger.trace('recording obfuscated sql')
       segment.parameters.sql_obfuscated = slowQuery.obfuscated
       break
     default:
       logger.trace(
-        'not collecting slow-query because transaction_tracer.record_sql was set to %s',
+        'not recording sql statement, transaction_tracer.record_sql was set to %s',
         ttConfig.record_sql
       )
       return
   }
+
+  if (segment.getDurationInMillis() < ttConfig.explain_threshold) {
+    return
+  }
+
+  slowQuery = slowQuery || new SlowQuery(segment, type, query, trace)
+
   segment.parameters.backtrace = slowQuery.trace
 
   if (!this.config.slow_sql.enabled) {

package/lib/environment.js

@@ -305,7 +305,7 @@ function getGlobalPackages(cb) {
  * package appears at most once, with all the versions joined into a
  * comma-delimited list.
  *
- * @return {Array.<String>[]} Sorted list of [name, version] pairs.
+ * @return {Array.<string[]>} Sorted list of [name, version] pairs.
  */
 function flattenVersions(packages) {
   var info = Object.create(null)

package/lib/instrumentation/core/http.js

@@ -693,7 +693,7 @@ module.exports = function initialize(agent, http, moduleName) {
  *
  * @param {string} header - The raw X-NewRelic-Synthetics header
  * @param {string} encKey - Encoding key handed down from the server
- * @param {̄Array} trustedIds - Array of accounts to trust the header from.
+ * @param {Array.<number>} trustedIds - Array of accounts to trust the header from.
  * @param {Transaction} transaction - Where the synthetics data is attached to.
  */
 function handleSyntheticsHeader(header, encKey, trustedIds, transaction) {
@@ -711,8 +711,8 @@ function handleSyntheticsHeader(header, encKey, trustedIds, transaction) {
  *
  * @param {string} header - The raw X-NewRelic-Synthetics header
  * @param {string} encKey - Encoding key handed down from the server
- * @param {̄Array} trustedIds - Array of accounts to trust the header from.
- * @return {Object or null} - On successful parse and verification an object of
+ * @param {Array.<number>} trustedIds - Array of accounts to trust the header from.
+ * @return {Object|null} - On successful parse and verification an object of
  *                            synthetics data is returned, otherwise null is
  *                            returned.
  */

package/lib/priority-queue.js

@@ -3,7 +3,7 @@
 var Heap = require('@tyriar/fibonacci-heap').FibonacciHeap
 
 function PriorityQueue(limit) {
-  this.limit = limit || 10
+  this.limit = limit == null ? 10 : limit
   this.seen = 0
   this._data = new Heap()
 
@@ -24,8 +24,11 @@ PriorityQueue.prototype.getMinimumPriority = function getMinimumPriority() {
 }
 
 PriorityQueue.prototype.add = function add(value, priority) {
-  priority = priority || Math.random()
   this.seen++
+  if (this.limit <= 0) {
+    return false
+  }
+  priority = priority || Math.random()
   if (this.length === this.limit) {
     return this._replace(value, priority)
   }

package/lib/shim/message-shim.js

@@ -333,10 +333,6 @@ function recordProduce(nodule, properties, recordNamer) {
     }
 
     var name = _nameMessageSegment(shim, msgDesc, shim._metrics.PRODUCE)
-    if (msgDesc.headers) {
-      shim.insertCATRequestHeaders(msgDesc.headers, true)
-    }
-
     if (!shim.agent.config.message_tracer.segment_parameters.enabled) {
       delete msgDesc.parameters
     } else if (msgDesc.routingKey) {
@@ -350,6 +346,11 @@ function recordProduce(nodule, properties, recordNamer) {
       promise: msgDesc.promise || false,
       callback: msgDesc.callback || null,
       recorder: genericRecorder,
+      inContext: function generateCATHeaders() {
+        if (msgDesc.headers) {
+          shim.insertCATRequestHeaders(msgDesc.headers, true)
+        }
+      },
       parameters: msgDesc.parameters || null
     }
   })

package/lib/shim/shim.js

@@ -946,7 +946,7 @@ function record(nodule, properties, recordNamer) {
       var promised = false
       var ret
       try {
-        ret = shim.applySegment(fn, segment, true, ctx, args)
+        ret = shim.applySegment(fn, segment, true, ctx, args, segDesc.inContext)
         if (segDesc.after && segDesc.promise && shim.isPromise(ret)) {
           promised = true
           return ret.then(function onThen(val) {
@@ -1286,7 +1286,7 @@ function storeSegment(obj, segment) {
  * Sets the given segment as the active one for the duration of the function's
  * execution.
  *
- * - `applySegment(func, segment, full, context, args)`
+ * - `applySegment(func, segment, full, context, args[, inContextCB])`
  *
  * @memberof Shim.prototype
  *
@@ -1305,9 +1305,15 @@ function storeSegment(obj, segment) {
  * @param {Array.<*>} args
  *  The arguments to be passed into the function.
  *
+ * @param {Function} [inContextCB]
+ *  The function used to do more instrumentation work. This function is
+ *  guaranteed to be executed with the segment associated with.
+ *
+ *
  * @return {*} Whatever value `func` returned.
  */
-function applySegment(func, segment, full, context, args) {
+/* eslint-disable max-params */
+function applySegment(func, segment, full, context, args, inContextCB) {
   // Exist fast for bad arguments.
   if (!this.isFunction(func)) {
     return
@@ -1327,6 +1333,10 @@ function applySegment(func, segment, full, context, args) {
     segment.start()
   }
 
+  if (typeof inContextCB === 'function') {
+    inContextCB()
+  }
+
   // Execute the function and then return the tracer segment to the old one.
   try {
     return func.apply(context, args)
@@ -1337,6 +1347,7 @@ function applySegment(func, segment, full, context, args) {
     tracer.segment = prevSegment
   }
 }
+/* eslint-enable max-params */
 
 /**
  * Creates a new segment.

package/lib/shim/specs/index.js

@@ -45,6 +45,7 @@ function WrapSpec(spec) {
 function SegmentSpec(spec) {
   this.name       = hasOwnProperty(spec, 'name') ? spec.name : null
   this.recorder   = hasOwnProperty(spec, 'recorder') ? spec.recorder : null
+  this.inContext  = hasOwnProperty(spec, 'inContext') ? spec.inContext : null
   this.parent     = hasOwnProperty(spec, 'parent') ? spec.parent : null
   this.parameters = hasOwnProperty(spec, 'parameters') ? spec.parameters : null
   this.internal   = hasOwnProperty(spec, 'internal') ? spec.internal : false

package/lib/transaction/dt-payload.js

@@ -7,6 +7,12 @@ const DT_VERSION_MAJOR = 0
 const DT_VERSION_MINOR = 1
 
 module.exports = class DistributedTracePayload {
+  /**
+   * The class reponsible for producing distributed trace payloads.
+   * Created by calling {@link TransactionHandle#createDistributedTracePayload}.
+   *
+   * @constructor
+   */
   constructor(payload) {
     logger.trace('DistributedTracePayload created with %s', payload)
     this.plainTextPayload = JSON.stringify({
@@ -16,11 +22,21 @@ module.exports = class DistributedTracePayload {
     this.base64Payload = null
   }
 
+  /**
+   * @returns {String} The base64 encoded JSON representation of the
+   * distributed trace payload.
+   */
   text() {
     logger.trace('DistributedTracePayload text: %s', this.plainTextPayload)
     return this.plainTextPayload
   }
 
+  /**
+   * Construct a payload suitable for HTTP transport.
+   *
+   * @returns {String} The base64 encoded JSON representation of the
+   * distributed trace payload.
+   */
   httpSafe() {
     if (!this.base64Payload) {
       this.base64Payload = makeBuffer(this.plainTextPayload, 'utf-8').toString('base64')

package/lib/transaction/handle.js

@@ -33,7 +33,7 @@ module.exports = class TransactionHandle {
   *
   * Proxy method for Transaction#createDistrubtedTracePayload.
   *
-  * @returns {DistributedTracePayload} The created payload.
+  * @returns {DistributedTracePayload} The created payload object.
   *
   */
   createDistributedTracePayload() {

package/lib/transaction/index.js

@@ -193,6 +193,10 @@ Transaction.prototype.end = function end(done) {
       transaction.record()
     }
 
+    // This is function currently must be called after all recorders have been fired due
+    // to some of the recorders (namely the db recorders) add parameters to the segments.
+    transaction.trace.generateSpanEvents()
+
     transaction.agent.emit('transactionFinished', transaction)
     if (typeof done === 'function') {
       done(transaction)

package/lib/transaction/trace/index.js

@@ -49,30 +49,48 @@ function Trace(transaction) {
  * segments that support recording.
  */
 Trace.prototype.end = function end() {
+  var segments = [this.root]
+
+  while (segments.length) {
+    var segment = segments.pop()
+    _softEndSegment(segment)
+    Array.prototype.push.apply(segments, segment.getChildren())
+  }
+}
+
+/**
+ * Iterates over the trace tree and generates a span event for each segment.
+ */
+Trace.prototype.generateSpanEvents = function generateSpanEvents() {
   var config = this.transaction.agent.config
-  var segments = []
-  var processFn
 
   if (
     this.transaction.sampled &&
     config.span_events.enabled &&
     config.distributed_tracing.enabled
   ) {
+    var toProcess = []
     // Root segment does not become a span, so we need to process it separately.
-    _softEndSegment(this.root)
+    const spanAggregator = this.transaction.agent.spans
     const children = this.root.getChildren()
     for (let i = 0; i < children.length; ++i) {
-      segments.push(new DTTraceNode(children[i], this.transaction.parentSpanId))
+      toProcess.push(new DTTraceNode(children[i], this.transaction.parentSpanId))
     }
-    processFn = makeDTNodeProcessor(this.transaction.agent.spans)
-  } else {
-    segments.push(this.root)
-    processFn = processNode
-  }
 
-  while (segments.length) {
-    var segment = segments.pop()
-    processFn(segment, segments)
+    while (toProcess.length) {
+      var segmentInfo = toProcess.pop()
+      var segment = segmentInfo.segment
+
+      // Even though at some point we might want to stop adding events
+      // because all the priorities should be the same, we need to count
+      // the spans as seen.
+      spanAggregator.addSegment(segment, segmentInfo.parentId)
+
+      Array.prototype.push.apply(
+        toProcess,
+        segment.getChildren().map(child => new DTTraceNode(child, segment.id))
+      )
+    }
   }
 }
 
@@ -90,31 +108,6 @@ function _softEndSegment(segment) {
   }
 }
 
-function makeDTNodeProcessor(spanAggregator) {
-  var addMoreSpans = true
-  return function processNodeDT(segmentInfo, segments) {
-    var segment = segmentInfo.segment
-    _softEndSegment(segment)
-
-    // This assumes that spans will have their transaction's priority. If insertion of
-    // one event fails, all following will fail due to having the same priority.
-    if (addMoreSpans && segment.name !== 'ROOT') {
-      addMoreSpans = spanAggregator.addSegment(segment, segmentInfo.parentId)
-    }
-
-    var children = segment.getChildren()
-    Array.prototype.push.apply(
-      segments,
-      children.map(child => new DTTraceNode(child, segment.id))
-    )
-  }
-}
-
-function processNode(segment, segments) {
-  _softEndSegment(segment)
-  Array.prototype.push.apply(segments, segment.getChildren())
-}
-
 /**
  * Add a child to the list of segments.
  *

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "newrelic",
-  "version": "4.6.0",
+  "version": "4.7.0",
   "author": "New Relic Node.js agent team <nodejs@newrelic.com>",
   "licenses": [
     {
@@ -96,7 +96,7 @@
   ],
   "homepage": "http://github.com/newrelic/node-newrelic",
   "engines": {
-    "node": ">=4.0.0",
+    "node": ">=4.0.0 <11.0.0",
     "npm": ">=2.0.0"
   },
   "directories": {