newrelic 9.0.3 → 9.1.0

package/NEWS.md

@@ -1,6 +1,24 @@
+### v9.1.0 (2022-09-22)
+
+* Added [experimental loader](https://nodejs.org/api/esm.html#loaders) to support instrumentation of CommonJS packages in ECMAScript Module(ESM) applications.
+  * It only supports versions of Node.js >= `16.12.0`.
+  * It is subject to change due to its experimental stability.
+
+* Enhanced supportability metrics for ESM support.
+  * Added new metrics to track usage of ESM loader(`Supportability/Features/ESM/Loader` and `Supportability/Features/ESM/UnsupportedLoader`).
+  * Updated instrumentation map to include an optional "friendly name" for tracking metrics.
+
+* Enabled re-throwing ESM import errors of `newrelic.js` so that the user is informed to rename it to `newrelic.cjs`
+
+* Fixed an issue with mongodb instrumentation where IPv6 address([::1]) was not getting mapped to localhost when setting the host attribute on the segment.
+
+* Added a test ESM loader to properly mock out agent in versioned tests.
+
+* Added ESM versioned tests for: `express`, `pg`, `mongodb`, and `@grpc/grpc-js`.
+
 ### v9.0.3 (2022-09-06)
 
-* Updated gRPC client instrumenation to respect `grpc.record_errors` when deciding to log errors on gRPC client requests.
+* Updated gRPC client instrumentation to respect `grpc.record_errors` when deciding to log errors on gRPC client requests.
 
 * Fixed transaction name finalization to properly copy the appropriate transaction name to root segment.
 

package/README.md

@@ -20,11 +20,15 @@ To use New Relic's Node.js agent entails these three steps, which are described
 
 1. To install the agent for performance monitoring, use your favorite npm-based package manager and install the `newrelic` package into your application:
 
-    `$ npm install newrelic`
+    ```sh
+    $ npm install newrelic
+    ```
 
 2. Then, copy the stock configuration file to your program's base folder:
 
-    `$ cp ./node_modules/newrelic/newrelic.js ./<your destination>`
+    ```sh
+    $ cp ./node_modules/newrelic/newrelic.js ./<your destination>
+    ```
 
 3. Now, add your New Relic license key and application/service name to that file:
 
@@ -58,6 +62,28 @@ If you cannot control how your program is run, you can load the `newrelic` modul
     /* ... the rest of your program ... */
 ```
 
+## ECMAScript Modules
+
+If your application is written with `import` and `export` statements in javascript, you are using [ES Modules](https://nodejs.org/api/esm.html#modules-ecmascript-modules) and must bootstrap the agent in a different way.
+
+The New Relic Node.js agent includes ***_experimental_*** support for ES Modules. The agent is reliant on an experimental feature in Node.js in order to appropriately register instrumentation. Until the Node.js API for [ES Module Loaders](https://nodejs.org/api/esm.html#loaders) is stable, breaking changes may occur when updating Node.js. Lastly, the ESM loader does not follow the same [supported Node.js versions](https://docs.newrelic.com/docs/apm/agents/nodejs-agent/getting-started/compatibility-requirements-nodejs-agent#system) as the agent. The minimum supported version of Node.js is `v16.12.0`.
+
+### Setup
+
+ 1. If you rely on a configuration file to run the agent, you must rename the file from `newrelic.js` to `newrelic.cjs` so it can be properly loaded.  All the contents of the configuration file will behave the same once you rename. See [CommonJS modules in ESM](https://nodejs.org/api/modules.html#enabling) for more details.
+
+```sh
+$ mv newrelic.js newrelic.cjs
+```
+
+ 2. To use the newrelic ESM loader, start your program with node and use the `--experimental-loader` flag and a path to the loader file, like this:
+
+```sh
+$ node --experimental-loader newrelic/esm-loader.mjs your-program.js
+```
+
+**Note**: Unlike the CommonJS methods listed above, there are no alternatives to running the agent without the `--experimental-loader` flag.
+
 ## Getting Started
 
 For more information on getting started, [check the Node.js docs](https://docs.newrelic.com/docs/agents/nodejs-agent/getting-started/introduction-new-relic-nodejs).

package/THIRD_PARTY_NOTICES.md

@@ -64,6 +64,7 @@ code, the source code can be found at [https://github.com/newrelic/node-newrelic
 * [sinon](#sinon)
 * [tap](#tap)
 * [temp](#temp)
+* [testdouble](#testdouble)
 * [when](#when)
 
 **[optionalDependencies](#optionalDependencies)**
@@ -1333,7 +1334,7 @@ SOFTWARE.
 
 ### @newrelic/eslint-config
 
-This product includes source derived from [@newrelic/eslint-config](https://github.com/newrelic/eslint-config-newrelic) ([v0.0.3](https://github.com/newrelic/eslint-config-newrelic/tree/v0.0.3)), distributed under the [Apache-2.0 License](https://github.com/newrelic/eslint-config-newrelic/blob/v0.0.3/LICENSE):
+This product includes source derived from [@newrelic/eslint-config](https://github.com/newrelic/eslint-config-newrelic) ([v0.1.0](https://github.com/newrelic/eslint-config-newrelic/tree/v0.1.0)), distributed under the [Apache-2.0 License](https://github.com/newrelic/eslint-config-newrelic/blob/v0.1.0/LICENSE):
 
 ```
                                  Apache License
@@ -1766,7 +1767,7 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI
 
 ### @newrelic/test-utilities
 
-This product includes source derived from [@newrelic/test-utilities](https://github.com/newrelic/node-test-utilities) ([v7.0.0](https://github.com/newrelic/node-test-utilities/tree/v7.0.0)), distributed under the [Apache-2.0 License](https://github.com/newrelic/node-test-utilities/blob/v7.0.0/LICENSE):
+This product includes source derived from [@newrelic/test-utilities](https://github.com/newrelic/node-test-utilities) ([v7.1.1](https://github.com/newrelic/node-test-utilities/tree/v7.1.1)), distributed under the [Apache-2.0 License](https://github.com/newrelic/node-test-utilities/blob/v7.1.1/LICENSE):
 
 ```
                                  Apache License
@@ -8977,6 +8978,35 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
 ```
 
+### testdouble
+
+This product includes source derived from [testdouble](https://github.com/testdouble/testdouble.js) ([v3.16.6](https://github.com/testdouble/testdouble.js/tree/v3.16.6)), distributed under the [MIT License](https://github.com/testdouble/testdouble.js/blob/v3.16.6/LICENSE.txt):
+
+```
+The MIT License (MIT)
+
+Copyright (c) 2015 Test Double, LLC.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+```
+
 ### when
 
 This product includes source derived from [when](https://github.com/cujojs/when) ([v3.7.8](https://github.com/cujojs/when/tree/v3.7.8)), distributed under the [MIT License](https://github.com/cujojs/when/blob/v3.7.8/LICENSE.txt):

package/esm-loader.mjs

@@ -0,0 +1,94 @@
+/*
+ * Copyright 2022 New Relic Corporation. All rights reserved.
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+import newrelic from './index.js'
+import shimmer from './lib/shimmer.js'
+import loggingModule from './lib/logger.js'
+import NAMES from './lib/metrics/names.js'
+import semver from 'semver'
+
+const isSupportedVersion = () => semver.gte(process.version, 'v16.12.0')
+const logger = loggingModule.child({ component: 'esm-loader' })
+
+if (newrelic.agent) {
+  addESMSupportabilityMetrics(newrelic.agent)
+}
+
+/**
+ * Hook chain responsible for resolving a file URL for a given module specifier
+ *
+ * Our loader has to be the last user-supplied loader if chaining is happening,
+ * as we rely on `nextResolve` being the default Node.js resolve hook to get our URL
+ *
+ * Docs: https://nodejs.org/api/esm.html#resolvespecifier-context-nextresolve
+ *
+ * @param {string} specifier
+ *        String identifier in an import statement or import() expression
+ * @param {object} context
+ *        Metadata about the specifier, including url of the parent module and any import assertions
+ *        Optional argument that only needs to be passed when changed
+ * @param {Function} nextResolve
+ *        The Node.js default resolve hook
+ * @returns {Promise} Promise object representing the resolution of a given specifier
+ */
+export async function resolve(specifier, context, nextResolve) {
+  if (!newrelic.agent || !isSupportedVersion()) {
+    return nextResolve(specifier, context, nextResolve)
+  }
+
+  /**
+   * We manually call the default Node.js resolve hook so
+   * that we can get the fully qualified URL path and the
+   * package type (commonjs/module/builtin) without
+   * duplicating the logic of the Node.js hook
+   */
+  const resolvedModule = await nextResolve(specifier, context, nextResolve)
+  const instrumentationName = shimmer.getInstrumentationNameFromModuleName(specifier)
+  const instrumentationDefinition = shimmer.registeredInstrumentations[instrumentationName]
+
+  if (instrumentationDefinition) {
+    logger.debug(`Instrumentation exists for ${specifier}`)
+
+    if (resolvedModule.format === 'commonjs') {
+      // ES Modules translate import statements into fully qualified filepaths, so we create a copy of our instrumentation under this filepath
+      const instrumentationDefinitionCopy = Object.assign({}, instrumentationDefinition)
+
+      // Stripping the prefix is necessary because the code downstream gets this url without it
+      instrumentationDefinitionCopy.moduleName = resolvedModule.url.replace('file://', '')
+
+      // Added to keep our Supportability metrics from exploding/including customer info via full filepath
+      instrumentationDefinitionCopy.specifier = specifier
+
+      shimmer.registerInstrumentation(instrumentationDefinitionCopy)
+
+      logger.debug(
+        `Registered CommonJS instrumentation for ${specifier} under ${instrumentationDefinitionCopy.moduleName}`
+      )
+    } else {
+      logger.debug(`${specifier} is not a CommonJS module, skipping for now`)
+    }
+  }
+
+  return resolvedModule
+}
+
+/**
+ * Helper function for determining which of our Supportability metrics to use for the current loader invocation
+ *
+ * @param {object} agent
+ *        instantiation of the New Relic agent
+ * @returns {void}
+ */
+export function addESMSupportabilityMetrics(agent) {
+  if (isSupportedVersion()) {
+    agent.metrics.getOrCreateMetric(NAMES.FEATURES.ESM.LOADER).incrementCallCount()
+  } else {
+    logger.warn(
+      'New Relic for Node.js ESM loader requires a version of Node >= v16.12.0; your version is %s.  Instrumentation will not be registered.',
+      process.version
+    )
+    agent.metrics.getOrCreateMetric(NAMES.FEATURES.ESM.UNSUPPORTED_LOADER).incrementCallCount()
+  }
+}

package/index.js

@@ -134,7 +134,7 @@ function createAgent(config) {
   if (appNames.length < 1) {
     const message =
       'New Relic requires that you name this application!\n' +
-      'Set app_name in your newrelic.js file or set environment variable\n' +
+      'Set app_name in your newrelic.js or newrelic.cjs file or set environment variable\n' +
       'NEW_RELIC_APP_NAME. Not starting!'
     logger.error(message)
     throw new Error(message)

package/lib/config/index.js

@@ -1647,6 +1647,10 @@ function initialize(config) {
   try {
     userConf = require(filepath).config
   } catch (error) {
+    if (error.code === 'ERR_REQUIRE_ESM') {
+      // Attempted to import newrelic.js or similar from an ESM module, error out early.
+      throw error
+    }
     logger.error(error)
 
     logger.warn(

package/lib/instrumentation/mongodb/v4-mongo.js

@@ -45,7 +45,7 @@ function cmdStartedHandler(shim, evnt) {
       if (hosts && hosts.length && hosts[0].socketPath) {
         port = hosts[0].socketPath
       }
-    } else if (['127.0.0.1', '::1'].includes(host)) {
+    } else if (['127.0.0.1', '::1', '[::1]'].includes(host)) {
       host = 'localhost'
     }
 

package/lib/metrics/names.js

@@ -257,6 +257,10 @@ const INFINITE_TRACING = {
 }
 
 const FEATURES = {
+  ESM: {
+    LOADER: `${SUPPORTABILITY.FEATURES}/ESM/Loader`,
+    UNSUPPORTED_LOADER: `${SUPPORTABILITY.FEATURES}/ESM/UnsupportedLoader`
+  },
   CERTIFICATES: SUPPORTABILITY.FEATURES + '/Certificates',
   INSTRUMENTATION: {
     ON_RESOLVED: SUPPORTABILITY.FEATURES + '/Instrumentation/OnResolved',

package/lib/shimmer.js

@@ -517,7 +517,12 @@ function instrumentPostLoad(agent, nodule, moduleName, resolvedName) {
   const shim = shims.createShimFromType(instrumentation.type, agent, moduleName, resolvedName)
 
   applyDebugState(shim, nodule)
-  trackInstrumentationUsage(agent, shim, moduleName, NAMES.FEATURES.INSTRUMENTATION.ON_REQUIRE)
+  trackInstrumentationUsage(
+    agent,
+    shim,
+    instrumentation.specifier || moduleName,
+    NAMES.FEATURES.INSTRUMENTATION.ON_REQUIRE
+  )
 
   try {
     if (instrumentation.onRequire(shim, nodule, moduleName) !== false) {
@@ -613,7 +618,12 @@ function _instrumentOnResolved(agent, moduleName, resolvedFilepath) {
 
   const shim = shims.createShimFromType(instrumentation.type, agent, moduleName, resolvedFilepath)
 
-  trackInstrumentationUsage(agent, shim, moduleName, NAMES.FEATURES.INSTRUMENTATION.ON_RESOLVED)
+  trackInstrumentationUsage(
+    agent,
+    shim,
+    instrumentation.specifier || moduleName,
+    NAMES.FEATURES.INSTRUMENTATION.ON_RESOLVED
+  )
 
   try {
     instrumentation.onResolved(shim, moduleName, resolvedFilepath)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "newrelic",
-  "version": "9.0.3",
+  "version": "9.1.0",
   "author": "New Relic Node.js agent team <nodejs@newrelic.com>",
   "license": "Apache-2.0",
   "contributors": [
@@ -113,6 +113,16 @@
       "name": "Jordi Gutiérrez Hermoso",
       "email": "jgutierrezhermoso@newrelic.com",
       "web": "https://newrelic.com"
+    },
+    {
+      "name": "Maurice Rickard",
+      "email": "mrickard@newrelic.com",
+      "web": "https://newrelic.com"
+    },
+    {
+      "name": "Jessica Lopatta",
+      "email": "jlopatta@newrelic.com",
+      "web": "https://newrelic.com"
     }
   ],
   "description": "New Relic agent",
@@ -138,17 +148,18 @@
     "docs": "npm ci && jsdoc -c ./jsdoc-conf.json --private -r .",
     "integration": "npm run prepare-test && npm run sub-install && time tap test/integration/**/**/*.tap.js --timeout=180 --no-coverage --reporter classic",
     "prepare-test": "npm run ssl && npm run docker-env",
-    "lint": "eslint ./*.js lib test bin examples",
-    "lint:fix": "eslint --fix, ./*.js lib test bin examples",
+    "lint": "eslint ./*.{js,mjs} lib test bin examples",
+    "lint:fix": "eslint --fix, ./*.{js,mjs} lib test bin examples",
     "public-docs": "jsdoc -c ./jsdoc-conf.json --tutorials examples/shim api.js lib/shim/ lib/transaction/handle.js && cp examples/shim/*.png out/",
     "publish-docs": "./bin/publish-docs.sh",
     "services": "./bin/docker-services.sh",
     "smoke": "npm run ssl && time tap test/smoke/**/**/*.tap.js --timeout=180 --no-coverage",
     "ssl": "./bin/ssl.sh",
     "sub-install": "node test/bin/install_sub_deps",
-    "test": "npm run integration && npm run unit",
+    "test": "npm run integration && npm run unit && npm run unit:esm",
     "third-party-updates": "oss third-party manifest --includeOptDeps && oss third-party notices --includeOptDeps && git add THIRD_PARTY_NOTICES.md third_party_manifest.json",
     "unit": "rm -f newrelic_agent.log && time tap --test-regex='(\\/|^test\\/unit\\/.*\\.test\\.js)$' --timeout=180 --no-coverage --reporter classic",
+    "unit:esm": "rm -f newrelic_agent.log && time tap --test-regex='(\\/|^test\\/unit\\/.*\\.test\\.mjs)$' --timeout=180 --no-coverage --reporter classic --node-arg=--experimental-loader=testdouble",
     "update-cross-agent-tests": "./bin/update-cats.sh",
     "versioned-tests": "./bin/run-versioned-tests.sh",
     "update-changelog-version": "node ./bin/update-changelog-version",
@@ -180,10 +191,10 @@
     "@newrelic/native-metrics": "^9.0.0"
   },
   "devDependencies": {
-    "@newrelic/eslint-config": "^0.0.3",
+    "@newrelic/eslint-config": "^0.1.0",
     "@newrelic/newrelic-oss-cli": "^0.1.2",
     "@newrelic/proxy": "^2.0.0",
-    "@newrelic/test-utilities": "^7.0.0",
+    "@newrelic/test-utilities": "^7.1.1",
     "@octokit/rest": "^18.0.15",
     "@slack/bolt": "^3.7.0",
     "ajv": "^6.12.6",
@@ -215,6 +226,7 @@
     "sinon": "^4.5.0",
     "tap": "^16.0.1",
     "temp": "^0.8.1",
+    "testdouble": "^3.16.6",
     "when": "*"
   },
   "repository": {
@@ -232,6 +244,7 @@
     "THIRD_PARTY_NOTICES.md",
     "lib/",
     "bin/tracetractor",
-    "bin/test-naming-rules.js"
+    "bin/test-naming-rules.js",
+    "esm-loader.mjs"
   ]
 }