newrelic_rpm 3.12.0.288 → 9.10.2

data/test/performance/README.md

@@ -1,182 +0,0 @@
-# Ruby Agent Performance Tests
-
-This is a performance testing framework for the Ruby Agent.
-
-## Motivation
-
-There are two main goals driving the development of this framework:
-
-1. Add a way for automated performance tests to be run against the Ruby Agent
-   and ingested into a system for tracking these results over time.
-2. Provide a tool for Ruby Agent engineers to use while working on performance
-   improvements.
-
-## Examples
-
-### Invoking via rake task
-
-Basic performance test invocations can be done using a rake task provided in the
-newrelic_rpm Rakefile.
-
-Run all performance tests, reporting results to the console:
-
-```
-$ rake test:performance
-```
-
-Run one specific suite:
-
-```
-$ rake test:performance[TransactionTracingPerfTests]
-```
-
-Run one specific suite and test (test name matching is via regex):
-
-```
-$ rake test:performance[TransactionTracingPerfTests,test_short_transactions]
-```
-
-### Invoking via the runner directly
-
-More advanced options can be specified by invoking the runner script directly.
-See `./test/performance/script/runner -h` for a full list of options.
-
-Run all tests, report detailed results in a human-readable form
-
-```
-$ ./test/performance/script/runner
-```
-
-List all available test suites and names:
-
-```
-$ ./test/performance/script/runner -l
-```
-
-Run a specific test (test name matching is via regex):
-
-```
-$ ./test/performance/script/runner -n short
-```
-
-To compare results for a specific test between two versions of the code, use the
-`-B` (for Baseline) and `-C` for (for Compare) switches:
-
-```
-$ ./test/performance/script/runner -n short -B
-1 tests, 0 failures, 8.199975 s total
-Saved 1 results as baseline.
-
-... switch to another branch and run again with -C ...
-
-$ ./test/performance/script/runner -n short -C
-1 tests, 0 failures, 8.220509 s total
-+-----------------------------------------------------+-----------+-----------+-------+---------------+--------------+--------------+
-| name                                                | before    | after     | delta | allocs_before | allocs_after | allocs_delta |
-|-----------------------------------------------------+-----------+-----------+-------+---------------+--------------+--------------|
-| TransactionTracingPerfTests#test_short_transactions | 214.27 µs | 210.31 µs | -1.8% |            97 |           97 |         0.0% |
-+-----------------------------------------------------+-----------+-----------+-------+---------------+--------------+--------------+
-```
-
-Run all the tests, produce machine readable JSON output (for eventual ingestion into a storage system):
-
-```
-$ ./test/performance/script/runner -j | json_reformat
-```
-
-Run a specific test under a profiler (either stackprof or perftools.rb, depending on your Ruby version):
-
-```
-$ ./test/performance/script/runner -n short --profile
-```
-
-Run with a set number of iterations, and do object allocation profiling (again to a call-graph dot file):
-
-```
-$ ./test/performance/script/runner -n short -a -N 1000
-```
-
-## Pointing at a different copy of the agent
-
-If you want to run performance tests against an older copy of the agent that
-doesn't have the performance test framework embedded within it, you can do that
-by specifying the path to the agent you want to test against by passing the `-A`
-flag to the `runner` script, or by setting the `AGENT_PATH` environment variable
-when using the rake task.
-
-## Sending results to Hako
-
-This is currently considered experimental, but you can send results
-automatically to Hako by passing `-R HakoReporter` on the command line to the
-`runner`. This will produce JSON-formatted versions of each test result, and
-submit them to Hako. You will need to set the `HAKO_TOKEN` environment variable
-to a valid Hako access token when doing this.
-
-## Writing tests
-
-Performance tests are written in the style of `test/unit`: create a `.rb` file
-under `test/performance/suites`, subclass `Performance::TestCase`, and write
-test methods that start with `test_`. You can also write `setup` and `teardown`
-methods that will be run before/after each test.
-
-Within your `test_` method, you must call `measure` and pass it a block
-containing the code that you'd like to actually measure the timing of. This
-allows you to do test-specific setup that doesn't get counted towards your
-test timing.
-
-The block that you pass to `measure` will automatically be run in a loop for a
-fixed amount of time by the performance runner harness (5s by default), and the
-number of iterations performed will be recorded so that measurements can be
-normalized to per-iteration values.
-
-You can look at the [existing tests](suites) for examples.
-
-## Test Isolation
-
-Initial testing suggested that certain kinds of tests would have a large impact
-on tests run later on in the same process (e.g. tests that create lots of
-long-lived objects will slow down all future GC runs for as long as those
-objects remain live).
-
-In order to address this problem, the test runner will attempt to isolate each
-test to its own process by re-spawning itself for each test invocation. This is
-done using `IO.popen` rather than `Process.fork` in order to maintain
-compatibility with JRuby.
-
-Additionally, when operating in this mode, the `newrelic_rpm` gem will not be
-loaded until *after* the fork call. This means that your **test cases must be
-loadable (though not necessarily runnable) without the `newrelic_rpm` gem
-avaiable**.
-
-Not all command-line options to the runner work with this test isolation yet.
-You can disable it by passing the `-I/--inline` flag to the runner.
-
-## Adding instrumentation layers
-
-The GC stats that are collected with each test run, and the perftools.rb
-profiling are examples of Instrumentors which can be wrapped around each test run.
-
-The basic idea is that each instrumentor gets callbacks before and after each
-test, and add information to the test results, or attach artifacts (a fancy name
-for file paths, currently) to the result.
-
-Instrumentors inherit from `Performance::Instrumentation::Instrumentor`, and may
-constrain themselves to running only on certain platforms (see `instrumentor.rb`
-for a list) by calling the `platforms` method in their class definitions. They
-may also signal that they should be used by default by calling `on_by_default`.
-They should imlement the `before`, `after` and `results` methods as follows:
-
-The `before` method is called before each test is run. The test class and test
-name are passed as arguments.
-
-The `after` method is called after each test is run. The test class, and test
-name are passed as arguments. Artifacts may be attached to the result here by
-appending to the `@artifacts` array. You may obtain paths to store artifacts at
-by calling `Performance::Instrumentation::Instrumentor#artifact_path` (see
-perf_tools.rb for an example).
-
-The `results` method must return a Hash of key-value pairs to be attached to the
-`Result` object produced by running each test.
-
-If your instrumentation layer needs to do one-time setup (requiring a gem, for
-example), implement the `setup` class method to do this setup.

data/test/performance/lib/performance/baseline.rb

@@ -1,36 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  class Baseline
-    PERSIST_PATH = File.expand_path("~/.newrelic_rpm_baseline")
-
-    attr_reader :results
-
-    def self.load!
-      self.new.load!
-    end
-
-    def self.save!(results)
-      baseline = self.new
-      results.each { |r| baseline.results << r }
-      baseline.save!
-    end
-
-    def initialize
-      @results = []
-    end
-
-    def load!
-      result_hashes = JSON.parse(File.read(PERSIST_PATH))
-      @results = result_hashes.map { |h| Result.from_hash(h) }
-    end
-
-    def save!
-      File.open(PERSIST_PATH, "w") do |f|
-        f.write(JSON.dump(@results.map(&:to_h)))
-      end
-    end
-  end
-end

data/test/performance/lib/performance/baseline_compare_reporter.rb

@@ -1,103 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  class BaselineCompareReporter
-    include Reporting
-
-    def initialize(results, elapsed, options={})
-      @results = results
-      @elapsed = elapsed
-      @options = options
-    end
-
-    def report
-      report_summary
-
-      begin
-        baseline = Baseline.load!
-      rescue => e
-        puts "Failed to load baseline results: #{e}\n#{e.backtrace.join("\n\t")}"
-        return
-      end
-
-      report_successful_results(baseline, successes) unless successes.empty?
-      report_failed_results
-    end
-
-    def report_successful_results(baseline, results)
-      baseline_identifiers  = baseline.map(&:identifier)
-      new_identifiers       = results.map(&:identifier)
-      missing_from_baseline = new_identifiers - baseline_identifiers
-      missing_from_new      = baseline_identifiers - new_identifiers
-      common_identifiers    = new_identifiers & baseline_identifiers
-
-      if !missing_from_baseline.empty?
-        puts "The following tests were not found in the baseline results:\n"
-        missing_from_baseline.each do |identifier|
-          puts "  #{identifier}"
-        end
-        puts ""
-      end
-
-      if !missing_from_baseline.empty?
-        puts "The following tests were not found in the new results:\n"
-        missing_from_new.each do |identifier|
-          puts "  #{identifier}"
-        end
-        puts ""
-      end
-
-      rows = []
-
-      common_identifiers.each do |identifier|
-        old_result = baseline.find { |r| r.identifier == identifier}
-        new_result = results.find  { |r| r.identifier == identifier }
-
-        delta = new_result.time_per_iteration - old_result.time_per_iteration
-        percent_delta = delta / old_result.time_per_iteration * 100.0
-
-        allocations_before = old_result.measurements[:allocations]
-        allocations_after  = new_result.measurements[:allocations]
-        if allocations_before && allocations_after
-          # normalize allocation counts to be per-iteration
-          allocations_before /= old_result.iterations
-          allocations_after  /= new_result.iterations
-
-          allocations_delta  = allocations_after - allocations_before
-          allocations_delta_percent = allocations_delta.to_f / allocations_before * 100
-        else
-          allocations_delta_percent = 0
-        end
-
-        rows << [
-          identifier,
-          old_result.time_per_iteration,
-          new_result.time_per_iteration,
-          percent_delta,
-          allocations_before,
-          allocations_after,
-          allocations_delta_percent
-        ]
-      end
-
-      format_percent_delta = Proc.new { |v|
-        prefix = v > 0 ? "+" : ""
-        sprintf("#{prefix}%.1f%%", v)
-      }
-
-      table = Table.new(rows) do
-        column :name
-        column :before,        &(FormattingHelpers.method(:format_duration))
-        column :after,         &(FormattingHelpers.method(:format_duration))
-        column :delta,         &format_percent_delta
-        column :allocs_before
-        column :allocs_after
-        column :allocs_delta,  &format_percent_delta
-      end
-
-      puts table.render
-    end
-  end
-end

data/test/performance/lib/performance/baseline_save_reporter.rb

@@ -1,24 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  class BaselineSaveReporter
-    include Reporting
-
-    def initialize(results, elapsed, options={})
-      @results = results
-      @elapsed = elapsed
-      @options = options
-    end
-
-    def report
-      report_summary
-
-      Baseline.save!(successes)
-      puts "Saved #{successes.size} results as baseline."
-
-      report_failed_results
-    end
-  end
-end

data/test/performance/lib/performance/console_reporter.rb

@@ -1,66 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-require 'pathname'
-
-module Performance
-  class ConsoleReporter
-    include Reporting
-
-    def initialize(results, elapsed, options={})
-      @results = results
-      @elapsed = elapsed
-      @options = options
-    end
-
-    def report
-      report_summary
-      report_successful_results(successes) unless successes.empty?
-      report_failed_results
-    end
-
-    def format_measurements(result)
-      measurements = result.measurements.merge(:iterations => result.iterations)
-
-      key_width        = measurements.keys.map(&:size).max
-      formatted_values = measurements.values.map { |v| sprintf("%g", v) }
-      value_width      = formatted_values.map(&:size).max
-
-      rows = measurements.map do |key, value|
-        if key == :iterations
-          "  %#{key_width}s: %#{value_width}g" % [key, value]
-        else
-          per_iteration = value / result.iterations.to_f
-          "  %#{key_width}s: %#{value_width}g (%.2f / iter)" % [key, value, per_iteration]
-        end
-      end
-
-      rows.join("\n") + "\n"
-    end
-
-    def report_successful_results(results)
-      return if successes.empty?
-
-      puts ''
-      results.each do |result|
-        formatted_duration = FormattingHelpers.format_duration(result.time_per_iteration)
-        puts "#{result.identifier}: %.3f ips (#{formatted_duration} / iteration)" % [result.ips]
-        unless @options[:brief]
-          puts format_measurements(result)
-        end
-        unless result.artifacts.empty?
-          puts "  artifacts:"
-          result.artifacts.each do |artifact|
-            puts "    #{make_relative(artifact)}"
-          end
-        end
-        puts '' if !@options[:brief] || !result.artifacts.empty?
-      end
-    end
-
-    def make_relative(path)
-      "./#{Pathname.new(path).relative_path_from(Pathname.getwd)}"
-    end
-  end
-end

data/test/performance/lib/performance/formatting_helpers.rb

@@ -1,22 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  module FormattingHelpers
-    def self.format_duration(d)
-      if d < 0.001
-        ds = d * 1000 * 1000
-        unit = "µs"
-      elsif d < 1.0
-        ds = d * 1000
-        unit = "ms"
-      else
-        ds = d
-        unit = "s"
-      end
-
-      sprintf("%.2f %s", ds, unit)
-    end
-  end
-end

data/test/performance/lib/performance/hako_client.rb

@@ -1,31 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-require 'json'
-require 'net/http'
-require 'uri'
-
-module Performance
-  class HakoClient
-    BASE_URI = 'http://hako.pdx.vm.datanerd.us'
-
-    def initialize(token)
-      @token = token
-    end
-
-    def submit(result)
-      body = JSON.dump('result' => result.to_h)
-
-      uri = URI(BASE_URI + "/api/results")
-      req = Net::HTTP::Post.new(uri.to_s)
-      req.body = body
-      req.content_type = 'application/json'
-      req['Authorization'] = "Token token=\"#{@token}\""
-
-      Net::HTTP.start(uri.host, uri.port) do |conn|
-        conn.request(req)
-      end
-    end
-  end
-end

data/test/performance/lib/performance/hako_reporter.rb

@@ -1,26 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  class HakoReporter
-    def initialize(results, elapsed, options={})
-      @results = results
-      @token = ENV['HAKO_TOKEN'] || options[:hako_token]
-    end
-
-    def report
-      Performance.logger.info("Uploading #{@results.size} results to Hako")
-      client = HakoClient.new(@token)
-      @results.each do |result|
-        rsp = client.submit(result)
-        case rsp
-        when Net::HTTPSuccess
-          Performance.logger.debug("Successfully posted result to Hako")
-        else
-          Performance.logger.error("Failed to post results to Hako: #{rsp.inspect}")
-        end
-      end
-    end
-  end
-end

data/test/performance/lib/performance/instrumentation/cpu_usage.rb

@@ -1,26 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  module Instrumentation
-    class CpuUsage < Instrumentor
-      on_by_default
-
-      def before(*)
-        @times_before = Process.times
-      end
-
-      def after(*)
-        @times_after = Process.times
-      end
-
-      def results
-        {
-          :cpu_time_user   => @times_after.utime - @times_before.utime,
-          :cpu_time_system => @times_after.stime - @times_before.stime
-        }
-      end
-    end
-  end
-end

data/test/performance/lib/performance/instrumentation/gc_stats.rb

@@ -1,58 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  module Instrumentation
-    class MRIGCStats < Instrumentor
-      platforms :mri_193, :mri_20, :mri_21, :mri_22
-      on_by_default
-
-      def before(*)
-        @stats_before = GC.stat
-      end
-
-      def after(*)
-        @stats_after = GC.stat
-      end
-
-      def results
-        heap_live_before = @stats_before[:heap_live_slots] || @stats_before[:heap_live_num] || @stats_before[:heap_live_slot]
-        heap_live_after  = @stats_after[:heap_live_slots]  || @stats_after[:heap_live_num]  || @stats_after[:heap_live_slot]
-
-        res = {
-          :gc_runs      => @stats_after[:count] - @stats_before[:count],
-          :live_objects => heap_live_after      - heap_live_before
-        }
-        if RUBY_VERSION >= "2.0.0"
-          allocs_before = @stats_before[:total_allocated_objects] || @stats_before[:total_allocated_object]
-          allocs_after  = @stats_after[:total_allocated_objects]  || @stats_after[:total_allocated_object]
-          res[:allocations] = allocs_after - allocs_before
-        end
-        res
-      end
-    end
-
-    class REEGCStats < Instrumentor
-      platforms :ree
-      on_by_default
-
-      def before(*)
-        @allocations_before = ObjectSpace.allocated_objects
-        @live_objects_before = ObjectSpace.live_objects
-      end
-
-      def after(*)
-        @allocations_after = ObjectSpace.allocated_objects
-        @live_objects_after = ObjectSpace.live_objects
-      end
-
-      def results
-        {
-          :allocations => @allocations_after - @allocations_before,
-          :live_objects => @live_objects_after - @live_objects_before,
-        }
-      end
-    end
-  end
-end

data/test/performance/lib/performance/instrumentation/perf_tools.rb

@@ -1,30 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  module Instrumentation
-    class PerfToolsProfile < Instrumentor
-      platforms :mri_19, :mri_20, :mri_21
-
-      def self.setup
-        require 'tmpdir'
-        require 'perftools'
-      end
-
-      def before(test, test_name)
-        @profile_dir = Dir.mktmpdir('profile')
-        @profile_path = File.join(@profile_dir, "profile")
-        PerfTools::CpuProfiler.start(@profile_path)
-      end
-
-      def after(test, test_name)
-        PerfTools::CpuProfiler.stop
-        output_profile_path = artifact_path(test, test_name, "dot")
-        system("pprof.rb --dot #{@profile_path} >#{output_profile_path}")
-        @artifacts << output_profile_path
-        FileUtils.remove_entry_secure(@profile_dir)
-      end
-    end
-  end
-end

data/test/performance/lib/performance/instrumentation/stackprof.rb

@@ -1,46 +0,0 @@
-# encoding: utf-8
-# This file is distributed under New Relic's license terms.
-# See https://github.com/newrelic/rpm/blob/master/LICENSE for complete details.
-
-module Performance
-  module Instrumentation
-    class StackProfProfile < Instrumentor
-      platforms :mri_21, :mri_22
-
-      def self.setup
-        require 'tmpdir'
-        require 'stackprof'
-      end
-
-      def mode
-        :wall
-      end
-
-      def before(test, test_name)
-        StackProf.start(:mode => mode)
-      end
-
-      def after(test, test_name)
-        StackProf.stop
-
-        output_dump_path = artifact_path(test, test_name, "dump")
-        StackProf.results(output_dump_path)
-        @artifacts << output_dump_path
-
-        results = Marshal.load(File.read(output_dump_path))
-        output_dot_path = artifact_path(test, test_name, "dot")
-        report = StackProf::Report.new(results)
-        File.open(output_dot_path, "w") do |f|
-          report.print_graphviz(nil, f)
-        end
-        @artifacts << output_dot_path
-      end
-    end
-
-    class StackProfAllocationProfile < StackProfProfile
-      def mode
-        :object
-      end
-    end
-  end
-end