newrelic_rpm 6.15.0 → 8.13.1

data/lib/tasks/instrumentation_generator/README.md

@@ -0,0 +1,63 @@
+# Instrumentation Generator
+
+## Usage
+
+`thor list` can be passed to get a list of all available thor commands
+
+### instrumentation:scaffold
+
+This task requires one parameter by default: the name of the library or class you are instrumenting. This task generates the basic file structure needed to add new instrumentation to the Ruby agent.
+
+Run it using:
+
+`thor instrumentation:scaffold gem_name`
+
+It accepts two optional parameters:
+
+- `--method`: the name of a method to instrument. Only one method can be accepted at this time. Defaults to `method_to_instrument`
+- `--args`: the arguments for the instrumented method. A comma-separated list can be given if there are multiple arguments. Defaults to `*args`.
+
+With the optional parameters, the task can be run like so:
+
+`thor instrumentation:scaffold gem_name --method=method_to_instrument --args=arg1,arg2,arg3`
+
+### WIP: instrumentation:add_new_method
+
+Are you trying add instrumentation for a method within library we already instrument? This task is for you! Instead of building the entire scaffold, this task inserts only the changes needed within the `instrumentation.rb` and test files to instrument the new method. It requires one argument, the name of the existing instrumentation to add the method to. If the instrumented library does not already exist, it will create a new scaffold for that library.
+
+Run it using:
+
+`thor instrumentation:scaffold gem_name`
+
+It accepts two optional parameters:
+
+- `--method`: the name of a method to instrument. Only one method can be accepted at this time. Defaults to `method_to_instrument`
+- `--args`: the arguments for the instrumented method. A comma-separated list can be given if there are multiple arguments. Defaults to `*args`.
+
+With the optional parameters, the task can be run like so:
+
+`thor instrumentation:add_new_method gem_name --method=method_to_instrument --args=arg1,arg2,arg3`
+
+## Idea
+
+Create a CLI, similar to Rails’ generators and scaffold scripts, to create the required files and basic classes for instrumenting new libraries. The hope is that the parts of adding new instrumentation that is repetitive can be eliminated and/or reduced so that it takes less time to add new instrumentation and we can spend development time considering the best attributes to collect and interactions to measure. I also hope this project will reduce the overall toil related to adding new instrumentation.
+
+The files we create with little variation besides library and method names are:
+
+- Lib
+  - Dependency detection
+  - Chain
+  - Prepend
+  - Instrumentation
+- Tests
+  - configuration/newrelic.yml
+  - Envfile
+  - Test file
+
+Furthermore, we also create very similar snippets inside the default source configuration file that could be generated by this project as well.
+
+## Outcome
+
+A prototype outside the agent was first created. This prototype generated the required files to add instrumentation. The prototype accepted three arguments: name (name of the library), method (method to instrument), args (arguments for the method). This prototype has evolved into the current directory.
+
+This project leverages the Ruby gem Thor, a toolkit for building powerful command-line interfaces used in Bundler, Vagrant, Rails and others powers this CLI.

data/lib/tasks/instrumentation_generator/TODO.md

@@ -0,0 +1,33 @@
+# TODO
+
+# - [X] Create instrumentation file
+
+# - [X] Create chain file
+
+# - [X] Create prepend file
+
+# - [X] Create dependency detection file
+
+# - [X] Add config to default source
+
+# - [X] Create multiverse suite
+
+# - [X] Create Envfile
+
+# - [X] Create test file with examples
+
+# - [X] Create option for method names to instrument
+
+# - [X] Add entry to newrelic.yml
+
+# - [ ] Append a new method to instrument to an existing instrumentation class (with tests?)
+
+# - [ ] Documentation: examples of what to add in each gap (Good examples of tests, instrumentation, etc. as comments to guide users)
+
+# - [ ] Handle multi-word gem names (camel case for classes, handle hyphens, concurrent-ruby as example)
+
+# - [ ] Allow multiple method arguments to be passed to the command line
+
+# - [ ] Add tests for the instrumentation_generator code
+
+# - [ ] See if instrumentation PRs can get automatically generated when an already instrumented library adds methods to its codebase

data/lib/tasks/instrumentation_generator/instrumentation.thor

@@ -0,0 +1,124 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+require_relative '../../new_relic/language_support'
+require 'thor'
+
+class Instrumentation < Thor
+  include Thor::Actions
+
+  INSTRUMENTATION_ROOT = 'lib/new_relic/agent/instrumentation/'
+  MULTIVERSE_SUITE_ROOT = 'test/multiverse/suites/'
+  DEFAULT_SOURCE_LOCATION = 'lib/new_relic/agent/configuration/default_source.rb'
+  NEWRELIC_YML_LOCATION = 'newrelic.yml'
+
+  desc('scaffold NAME', 'Scaffold the required files for adding new instrumentation')
+  long_desc <<-LONGDESC
+    `instrumentation scaffold` requires one parameter by default: the name of the
+    library or class you are instrumenting. This task generates the basic
+    file structure needed to add new instrumentation to the Ruby agent.
+  LONGDESC
+
+  source_root(File.dirname(__FILE__))
+
+  option :method,
+    default: 'method_to_instrument',
+    desc: 'The method you would like to prepend or chain instrumentation onto'
+  option :args,
+    default: '*args',
+    desc: 'The arguments associated with the original method'
+
+  def scaffold(name)
+    @name = name
+    @method = options[:method] if options[:method]
+    @args = options[:args] if options[:args]
+    @class_name = ::NewRelic::LanguageSupport.camelize(name)
+    base_path = "#{INSTRUMENTATION_ROOT}#{name.downcase}"
+
+    empty_directory(base_path)
+    create_instrumentation_files(base_path)
+    append_to_default_source(name)
+    append_to_newrelic_yml(name)
+    create_tests(name)
+  end
+
+  desc 'add_new_method NAME', 'Inserts a new method into an existing piece of instrumentation'
+
+  option :method, required: true, desc: 'The name of the method to instrument'
+  option :args, default: '*args', desc: 'The arguments associated with the instrumented method'
+
+  def add_new_method(name, method_name)
+    # Verify that existing instrumentation exists
+    # if it doesn't, should we just call the #scaffold method instead since we have all the stuff
+    # otherwise, inject the new method into the instrumentation matching the first arg
+    # add to only chain, instrumentation, prepend
+    # move the method content to a partial
+  end
+
+  private
+
+  def create_instrumentation_files(base_path)
+    %w[chain instrumentation prepend].each do |file|
+      template("templates/#{file}.tt", "#{base_path}/#{file}.rb")
+    end
+
+    template('templates/dependency_detection.tt', "#{base_path}.rb")
+  end
+
+  def create_tests(name)
+    @name = name
+    @instrumentation_method_global_erb_snippet = '<%= $instrumentation_method %>'
+    base_path = "#{MULTIVERSE_SUITE_ROOT}#{@name.downcase}"
+    empty_directory(base_path)
+    template('templates/Envfile.tt', "#{base_path}/Envfile")
+    template('templates/test.tt', "#{base_path}/#{@name.downcase}_instrumentation_test.rb")
+
+    empty_directory("#{base_path}/config")
+    template('templates/newrelic.yml.tt', "#{base_path}/config/newrelic.yml")
+  end
+
+  def append_to_default_source(name)
+    insert_into_file(
+      DEFAULT_SOURCE_LOCATION,
+      config_block(name.downcase),
+      after: ":description => 'Controls auto-instrumentation of bunny at start up.  May be one of [auto|prepend|chain|disabled].'
+        },\n"
+    )
+  end
+
+  def append_to_newrelic_yml(name)
+    insert_into_file(
+      NEWRELIC_YML_LOCATION,
+      yaml_block(name),
+      after: "# instrumentation.bunny: auto\n"
+    )
+  end
+
+  def config_block(name)
+    <<-CONFIG
+        :'instrumentation.#{name.downcase}' => {
+          :default => 'auto',
+          :public => true,
+          :type => String,
+          :dynamic_name => true,
+          :allowed_from_server => false,
+          :description => 'Controls auto-instrumentation of the #{name} library at start up. May be one of [auto|prepend|chain|disabled].'
+        },
+    CONFIG
+  end
+
+  # TODO: OLD RUBIES - RUBY_VERSION 2.2
+  # When we only support 2.3+ this can be changed to a sqiggle heredoc (<<~)
+  # and use standard indentation
+  def yaml_block(name)
+<<HEREDOC # rubocop:disable Layout/IndentationWidth
+
+  # Controls auto-instrumentation of #{name} at start up.
+  # May be one of [auto|prepend|chain|disabled]
+  # instrumentation.#{name.downcase}: auto
+HEREDOC
+  end
+end
+
+Instrumentation.start(ARGV)

data/lib/tasks/instrumentation_generator/templates/Envfile.tt

@@ -0,0 +1,9 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+instrumentation_methods :chain, :prepend
+
+gemfile <<-RB
+  gem '<%= @name.downcase %>'
+RB

data/lib/tasks/instrumentation_generator/templates/chain.tt

@@ -0,0 +1,22 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+module NewRelic::Agent::Instrumentation
+  module <%= @class_name %>
+    def self.instrument!
+      ::<%= @class_name %>.class_eval do
+        include NewRelic::Agent::Instrumentation::<%= @class_name %>
+
+        alias_method(:<%= @method.downcase %>_without_new_relic, :<%= @method.downcase %>)
+        alias_method(:<%= @method.downcase %>, :<%= @method.downcase %>_with_new_relic)
+
+        def <%= @method.downcase %><%= "(#{@args})" unless @args.empty? %>
+          <%= @method.downcase %>_with_new_relic<%= "(#{@args})" unless @args.empty? %> do
+            <%= @method.downcase %>_without_new_relic<%= "(#{@args})" unless @args.empty? %>
+          end
+        end
+      end
+    end
+  end
+end

data/lib/tasks/instrumentation_generator/templates/chain_method.tt

@@ -0,0 +1,8 @@
+alias_method(:<%= @method.downcase %>_without_new_relic, :<%= @method.downcase %>)
+alias_method(:<%= @method.downcase %>, :<%= @method.downcase %>_with_new_relic)
+
+def <%= @method.downcase %><%= "(#{@args})" unless @args.empty? %>
+  <%= @method.downcase %>_with_new_relic<%= "(#{@args})" unless @args.empty? %> do
+    <%= @method.downcase %>_without_new_relic<%= "(#{@args})" unless @args.empty? %>
+  end
+end

data/lib/tasks/instrumentation_generator/templates/dependency_detection.tt

@@ -0,0 +1,29 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+require_relative '<%= @name.downcase %>/instrumentation'
+require_relative '<%= @name.downcase %>/chain'
+require_relative '<%= @name.downcase %>/prepend'
+
+DependencyDetection.defer do
+  named :<%= @name.match?(/\-|\_/) ? "'#{@name.downcase}'" : @name.downcase %>
+
+  depends_on do
+    # The class that needs to be defined to prepend/chain onto. This can be used
+    # to determine whether the library is installed.
+    defined?(::<%= @class_name %>)
+    # Add any additional requirements to verify whether this instrumentation
+    # should be installed
+  end
+
+  executes do
+    ::NewRelic::Agent.logger.info('Installing <%= @name.downcase %> instrumentation')
+
+    if use_prepend?
+      prepend_instrument ::<%= @class_name %>, NewRelic::Agent::Instrumentation::<%= @class_name %>::Prepend
+    else
+      chain_instrument NewRelic::Agent::Instrumentation::<%= @class_name %>
+    end
+  end
+end

data/lib/tasks/instrumentation_generator/templates/instrumentation.tt

@@ -0,0 +1,13 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+module NewRelic::Agent::Instrumentation
+  module <%= @class_name %>
+
+    def <%= @method.downcase %>_with_new_relic<%= "(#{@args})" unless @args.empty? %>
+      # add instrumentation content here
+      yield
+    end
+  end
+end

data/lib/tasks/instrumentation_generator/templates/instrumentation_method.tt

@@ -0,0 +1,3 @@
+def <%= @method.downcase %>_with_new_relic<%= "(#{@args})" unless @args.empty? %>
+  # add instrumentation content here
+end

data/lib/tasks/instrumentation_generator/templates/newrelic.yml.tt

@@ -0,0 +1,19 @@
+---
+development:
+  error_collector:
+    enabled: true
+  apdex_t: 0.5
+  monitor_mode: true
+  license_key: bootstrap_newrelic_admin_license_key_000
+  instrumentation:
+    <%= @name.downcase %>: <%= @instrumentation_method_global_erb_snippet %>
+  app_name: test
+  log_level: debug
+  host: 127.0.0.1
+  api_host: 127.0.0.1
+  transaction_trace:
+    record_sql: obfuscated
+    enabled: true
+    stack_trace_threshold: 0.5
+    transaction_threshold: 1.0
+  capture_params: false

data/lib/tasks/instrumentation_generator/templates/prepend.tt

@@ -0,0 +1,13 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+module NewRelic::Agent::Instrumentation
+  module <%= @class_name %>::Prepend
+    include NewRelic::Agent::Instrumentation::<%= @class_name %>
+
+    def <%= @method.downcase %><%= "(#{@args})" unless @args.empty? %>
+      <%= @method.downcase %>_with_new_relic<%= "(#{@args})" unless @args.empty? %> { super }
+    end
+  end
+end

data/lib/tasks/instrumentation_generator/templates/prepend_method.tt

@@ -0,0 +1,3 @@
+def <%= @method.downcase %><%= "(#{@args})" unless @args.empty? %>
+  <%= @method.downcase %>_with_new_relic<%= "(#{@args})" unless @args.empty? %> { super }
+end

data/lib/tasks/instrumentation_generator/templates/test.tt

@@ -0,0 +1,15 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+class <%= @class_name %>InstrumentationTest < Minitest::Test
+  def setup
+    @stats_engine = NewRelic::Agent.instance.stats_engine
+  end
+
+  def teardown
+    NewRelic::Agent.instance.stats_engine.clear_stats
+  end
+
+  # Add tests here
+end

data/lib/tasks/multiverse.rake

@@ -1,2 +1,6 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
 # Guts of task are in a .rb file to allow requiring from external gems
 require "#{File.join(File.dirname(__FILE__))}/multiverse"

data/lib/tasks/multiverse.rb

@@ -1,6 +1,6 @@
-# encoding: utf-8
 # This file is distributed under New Relic's license terms.
 # See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
 #
 # Rake task for running Ruby agent multiverse tests. This file may be required
 # from third party gems. It is also used by the agent itself to run multiverse.
@@ -35,6 +35,9 @@
 #   # Runs with a specific test seed
 #   bundle exec rake test:multiverse[my_gem,seed=1337]
 
+require_relative 'helpers/removers'
+include Removers
+
 namespace :test do
   desc "Run functional test suite for New Relic"
   task :multiverse, [:suite, :param1, :param2, :param3, :param4] => ['multiverse:env'] do |_, args|
@@ -42,35 +45,6 @@ namespace :test do
   end
 
   namespace :multiverse do
-
-    def remove_local_multiverse_databases
-      list_databases_command = %{echo "show databases" | mysql -u root}
-      databases = `#{list_databases_command}`.chomp!.split("\n").select{|s| s =~ /multiverse/}
-      databases.each do |database|
-        puts "Dropping #{database}"
-        `echo "drop database #{database}" | mysql -u root`
-      end
-    rescue => error
-      puts "ERROR: Cannot get MySQL databases..."
-      puts error.message
-    end
-
-    def remove_generated_gemfiles
-      file_path = File.expand_path "test/multiverse/suites"
-      Dir.glob(File.join file_path, "**", "Gemfile*").each do |fn|
-        puts "Removing #{fn.gsub(file_path,'.../suites')}"
-        FileUtils.rm fn
-      end
-    end
-
-    def remove_generated_gemfile_lockfiles
-      file_path = File.expand_path "test/environments"
-      Dir.glob(File.join file_path, "**", "Gemfile.lock").each do |fn|
-        puts "Removing #{fn.gsub(file_path,'.../environments')}"
-        FileUtils.rm fn
-      end
-    end
-
     task :env do
       # ENV['SUITES_DIRECTORY'] = File.expand_path('../../test/multiverse/suites', __FILE__)
       require File.expand_path('../../../test/multiverse/lib/multiverse', __FILE__)
@@ -84,7 +58,7 @@ namespace :test do
 
     desc "Clean cached gemfiles from Bundler.bundle_path"
     task :clean_gemfile_cache do
-      glob = File.expand_path 'multiverse-cache/Gemfile.*.lock', Bundler.bundle_path
+      glob = File.expand_path('multiverse-cache/Gemfile.*.lock', Bundler.bundle_path)
       File.delete(*Dir[glob])
     end
 
@@ -99,6 +73,5 @@ namespace :test do
     task :prime, [:suite] => [:env] do |_, args|
       Multiverse::Runner.prime(args.suite, Multiverse::Runner.parse_args(args))
     end
-
   end
 end

data/lib/tasks/newrelic.rb

@@ -1,6 +1,6 @@
-# encoding: utf-8
 # This file is distributed under New Relic's license terms.
 # See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
 #
 # If you are looking to instrument rake tasks in a Rails environment, but the
 # task doesn't depend on :environment, this task may be included to ensure that

data/lib/tasks/tests.rake

@@ -1,3 +1,10 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+require_relative 'helpers/matches'
+include Matches
+
 begin
   require 'rake/testtask'
 rescue LoadError
@@ -5,37 +12,25 @@ end
 
 if defined? Rake::TestTask
   namespace :test do
-    def look_for_seed(tasks)
-      matches = tasks.map { |t| /(seed=.*?)[,\]]/.match(t) }.compact
-      if matches.any?
-        matches.first[1]
-      else
-        nil
-      end
-    end
-
     tasks = Rake.application.top_level_tasks
     ENV["TESTOPTS"] ||= ""
-    if tasks.any? { |t| t.include?("verbose")}
+    if tasks.any? { |t| t.include?("verbose") }
       ENV["TESTOPTS"] += " -v"
     end
     if seed = look_for_seed(tasks)
       ENV["TESTOPTS"] += " --" + seed
     end
 
-    agent_home = File.expand_path(File.join(File.dirname(__FILE__),'..','..'))
+    agent_home = File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
 
     Rake::TestTask.new(:newrelic) do |t|
-      
       file_pattern = ENV["file"]
-      file_pattern = file_pattern.split(",").map{|f| "#{agent_home}/#{f}".gsub("//", "/")} if file_pattern
+      file_pattern = file_pattern.split(",").map { |f| "#{agent_home}/#{f}".gsub("//", "/") } if file_pattern
       file_pattern ||= "#{agent_home}/test/new_relic/**/*_test.rb"
 
       t.libs << "#{agent_home}/test"
       t.libs << "#{agent_home}/lib"
       t.pattern = Array(file_pattern)
-      t.verbose = true
     end
-
   end
 end