mmtrix_rpm 1.0.0.1

data/test/performance/README.md

@@ -0,0 +1,182 @@
+# Ruby Agent Performance Tests
+
+This is a performance testing framework for the Ruby Agent.
+
+## Motivation
+
+There are two main goals driving the development of this framework:
+
+1. Add a way for automated performance tests to be run against the Ruby Agent
+   and ingested into a system for tracking these results over time.
+2. Provide a tool for Ruby Agent engineers to use while working on performance
+   improvements.
+
+## Examples
+
+### Invoking via rake task
+
+Basic performance test invocations can be done using a rake task provided in the
+mmtrix_rpm Rakefile.
+
+Run all performance tests, reporting results to the console:
+
+```
+$ rake test:performance
+```
+
+Run one specific suite:
+
+```
+$ rake test:performance[TransactionTracingPerfTests]
+```
+
+Run one specific suite and test (test name matching is via regex):
+
+```
+$ rake test:performance[TransactionTracingPerfTests,test_short_transactions]
+```
+
+### Invoking via the runner directly
+
+More advanced options can be specified by invoking the runner script directly.
+See `./test/performance/script/runner -h` for a full list of options.
+
+Run all tests, report detailed results in a human-readable form
+
+```
+$ ./test/performance/script/runner
+```
+
+List all available test suites and names:
+
+```
+$ ./test/performance/script/runner -l
+```
+
+Run a specific test (test name matching is via regex):
+
+```
+$ ./test/performance/script/runner -n short
+```
+
+To compare results for a specific test between two versions of the code, use the
+`-B` (for Baseline) and `-C` for (for Compare) switches:
+
+```
+$ ./test/performance/script/runner -n short -B
+1 tests, 0 failures, 8.199975 s total
+Saved 1 results as baseline.
+
+... switch to another branch and run again with -C ...
+
+$ ./test/performance/script/runner -n short -C
+1 tests, 0 failures, 8.220509 s total
++-----------------------------------------------------+-----------+-----------+-------+---------------+--------------+--------------+
+| name                                                | before    | after     | delta | allocs_before | allocs_after | allocs_delta |
+|-----------------------------------------------------+-----------+-----------+-------+---------------+--------------+--------------|
+| TransactionTracingPerfTests#test_short_transactions | 214.27 µs | 210.31 µs | -1.8% |            97 |           97 |         0.0% |
++-----------------------------------------------------+-----------+-----------+-------+---------------+--------------+--------------+
+```
+
+Run all the tests, produce machine readable JSON output (for eventual ingestion into a storage system):
+
+```
+$ ./test/performance/script/runner -j | json_reformat
+```
+
+Run a specific test under a profiler (either stackprof or perftools.rb, depending on your Ruby version):
+
+```
+$ ./test/performance/script/runner -n short --profile
+```
+
+Run with a set number of iterations, and do object allocation profiling (again to a call-graph dot file):
+
+```
+$ ./test/performance/script/runner -n short -a -N 1000
+```
+
+## Pointing at a different copy of the agent
+
+If you want to run performance tests against an older copy of the agent that
+doesn't have the performance test framework embedded within it, you can do that
+by specifying the path to the agent you want to test against by passing the `-A`
+flag to the `runner` script, or by setting the `AGENT_PATH` environment variable
+when using the rake task.
+
+## Sending results to Hako
+
+This is currently considered experimental, but you can send results
+automatically to Hako by passing `-R HakoReporter` on the command line to the
+`runner`. This will produce JSON-formatted versions of each test result, and
+submit them to Hako. You will need to set the `HAKO_TOKEN` environment variable
+to a valid Hako access token when doing this.
+
+## Writing tests
+
+Performance tests are written in the style of `test/unit`: create a `.rb` file
+under `test/performance/suites`, subclass `Performance::TestCase`, and write
+test methods that start with `test_`. You can also write `setup` and `teardown`
+methods that will be run before/after each test.
+
+Within your `test_` method, you must call `measure` and pass it a block
+containing the code that you'd like to actually measure the timing of. This
+allows you to do test-specific setup that doesn't get counted towards your
+test timing.
+
+The block that you pass to `measure` will automatically be run in a loop for a
+fixed amount of time by the performance runner harness (5s by default), and the
+number of iterations performed will be recorded so that measurements can be
+normalized to per-iteration values.
+
+You can look at the [existing tests](suites) for examples.
+
+## Test Isolation
+
+Initial testing suggested that certain kinds of tests would have a large impact
+on tests run later on in the same process (e.g. tests that create lots of
+long-lived objects will slow down all future GC runs for as long as those
+objects remain live).
+
+In order to address this problem, the test runner will attempt to isolate each
+test to its own process by re-spawning itself for each test invocation. This is
+done using `IO.popen` rather than `Process.fork` in order to maintain
+compatibility with JRuby.
+
+Additionally, when operating in this mode, the `mmtrix_rpm` gem will not be
+loaded until *after* the fork call. This means that your **test cases must be
+loadable (though not necessarily runnable) without the `mmtrix_rpm` gem
+avaiable**.
+
+Not all command-line options to the runner work with this test isolation yet.
+You can disable it by passing the `-I/--inline` flag to the runner.
+
+## Adding instrumentation layers
+
+The GC stats that are collected with each test run, and the perftools.rb
+profiling are examples of Instrumentors which can be wrapped around each test run.
+
+The basic idea is that each instrumentor gets callbacks before and after each
+test, and add information to the test results, or attach artifacts (a fancy name
+for file paths, currently) to the result.
+
+Instrumentors inherit from `Performance::Instrumentation::Instrumentor`, and may
+constrain themselves to running only on certain platforms (see `instrumentor.rb`
+for a list) by calling the `platforms` method in their class definitions. They
+may also signal that they should be used by default by calling `on_by_default`.
+They should imlement the `before`, `after` and `results` methods as follows:
+
+The `before` method is called before each test is run. The test class and test
+name are passed as arguments.
+
+The `after` method is called after each test is run. The test class, and test
+name are passed as arguments. Artifacts may be attached to the result here by
+appending to the `@artifacts` array. You may obtain paths to store artifacts at
+by calling `Performance::Instrumentation::Instrumentor#artifact_path` (see
+perf_tools.rb for an example).
+
+The `results` method must return a Hash of key-value pairs to be attached to the
+`Result` object produced by running each test.
+
+If your instrumentation layer needs to do one-time setup (requiring a gem, for
+example), implement the `setup` class method to do this setup.

data/test/performance/lib/performance/baseline.rb

@@ -0,0 +1,36 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  class Baseline
+    PERSIST_PATH = File.expand_path("~/.mmtrix_rpm_baseline")
+
+    attr_reader :results
+
+    def self.load!
+      self.new.load!
+    end
+
+    def self.save!(results)
+      baseline = self.new
+      results.each { |r| baseline.results << r }
+      baseline.save!
+    end
+
+    def initialize
+      @results = []
+    end
+
+    def load!
+      result_hashes = JSON.parse(File.read(PERSIST_PATH))
+      @results = result_hashes.map { |h| Result.from_hash(h) }
+    end
+
+    def save!
+      File.open(PERSIST_PATH, "w") do |f|
+        f.write(JSON.dump(@results.map(&:to_h)))
+      end
+    end
+  end
+end

data/test/performance/lib/performance/baseline_compare_reporter.rb

@@ -0,0 +1,103 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  class BaselineCompareReporter
+    include Reporting
+
+    def initialize(results, elapsed, options={})
+      @results = results
+      @elapsed = elapsed
+      @options = options
+    end
+
+    def report
+      report_summary
+
+      begin
+        baseline = Baseline.load!
+      rescue => e
+        puts "Failed to load baseline results: #{e}\n#{e.backtrace.join("\n\t")}"
+        return
+      end
+
+      report_successful_results(baseline, successes) unless successes.empty?
+      report_failed_results
+    end
+
+    def report_successful_results(baseline, results)
+      baseline_identifiers  = baseline.map(&:identifier)
+      new_identifiers       = results.map(&:identifier)
+      missing_from_baseline = new_identifiers - baseline_identifiers
+      missing_from_new      = baseline_identifiers - new_identifiers
+      common_identifiers    = new_identifiers & baseline_identifiers
+
+      if !missing_from_baseline.empty?
+        puts "The following tests were not found in the baseline results:\n"
+        missing_from_baseline.each do |identifier|
+          puts "  #{identifier}"
+        end
+        puts ""
+      end
+
+      if !missing_from_baseline.empty?
+        puts "The following tests were not found in the new results:\n"
+        missing_from_new.each do |identifier|
+          puts "  #{identifier}"
+        end
+        puts ""
+      end
+
+      rows = []
+
+      common_identifiers.each do |identifier|
+        old_result = baseline.find { |r| r.identifier == identifier}
+        new_result = results.find  { |r| r.identifier == identifier }
+
+        delta = new_result.time_per_iteration - old_result.time_per_iteration
+        percent_delta = delta / old_result.time_per_iteration * 100.0
+
+        allocations_before = old_result.measurements[:allocations]
+        allocations_after  = new_result.measurements[:allocations]
+        if allocations_before && allocations_after
+          # normalize allocation counts to be per-iteration
+          allocations_before /= old_result.iterations
+          allocations_after  /= new_result.iterations
+
+          allocations_delta  = allocations_after - allocations_before
+          allocations_delta_percent = allocations_delta.to_f / allocations_before * 100
+        else
+          allocations_delta_percent = 0
+        end
+
+        rows << [
+          identifier,
+          old_result.time_per_iteration,
+          new_result.time_per_iteration,
+          percent_delta,
+          allocations_before,
+          allocations_after,
+          allocations_delta_percent
+        ]
+      end
+
+      format_percent_delta = Proc.new { |v|
+        prefix = v > 0 ? "+" : ""
+        sprintf("#{prefix}%.1f%%", v)
+      }
+
+      table = Table.new(rows) do
+        column :name
+        column :before,        &(FormattingHelpers.method(:format_duration))
+        column :after,         &(FormattingHelpers.method(:format_duration))
+        column :delta,         &format_percent_delta
+        column :allocs_before
+        column :allocs_after
+        column :allocs_delta,  &format_percent_delta
+      end
+
+      puts table.render
+    end
+  end
+end

data/test/performance/lib/performance/baseline_save_reporter.rb

@@ -0,0 +1,24 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  class BaselineSaveReporter
+    include Reporting
+
+    def initialize(results, elapsed, options={})
+      @results = results
+      @elapsed = elapsed
+      @options = options
+    end
+
+    def report
+      report_summary
+
+      Baseline.save!(successes)
+      puts "Saved #{successes.size} results as baseline."
+
+      report_failed_results
+    end
+  end
+end

data/test/performance/lib/performance/console_reporter.rb

@@ -0,0 +1,66 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+require 'pathname'
+
+module Performance
+  class ConsoleReporter
+    include Reporting
+
+    def initialize(results, elapsed, options={})
+      @results = results
+      @elapsed = elapsed
+      @options = options
+    end
+
+    def report
+      report_summary
+      report_successful_results(successes) unless successes.empty?
+      report_failed_results
+    end
+
+    def format_measurements(result)
+      measurements = result.measurements.merge(:iterations => result.iterations)
+
+      key_width        = measurements.keys.map(&:size).max
+      formatted_values = measurements.values.map { |v| sprintf("%g", v) }
+      value_width      = formatted_values.map(&:size).max
+
+      rows = measurements.map do |key, value|
+        if key == :iterations
+          "  %#{key_width}s: %#{value_width}g" % [key, value]
+        else
+          per_iteration = value / result.iterations.to_f
+          "  %#{key_width}s: %#{value_width}g (%.2f / iter)" % [key, value, per_iteration]
+        end
+      end
+
+      rows.join("\n") + "\n"
+    end
+
+    def report_successful_results(results)
+      return if successes.empty?
+
+      puts ''
+      results.each do |result|
+        formatted_duration = FormattingHelpers.format_duration(result.time_per_iteration)
+        puts "#{result.identifier}: %.3f ips (#{formatted_duration} / iteration)" % [result.ips]
+        unless @options[:brief]
+          puts format_measurements(result)
+        end
+        unless result.artifacts.empty?
+          puts "  artifacts:"
+          result.artifacts.each do |artifact|
+            puts "    #{make_relative(artifact)}"
+          end
+        end
+        puts '' if !@options[:brief] || !result.artifacts.empty?
+      end
+    end
+
+    def make_relative(path)
+      "./#{Pathname.new(path).relative_path_from(Pathname.getwd)}"
+    end
+  end
+end

data/test/performance/lib/performance/formatting_helpers.rb

@@ -0,0 +1,22 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  module FormattingHelpers
+    def self.format_duration(d)
+      if d < 0.001
+        ds = d * 1000 * 1000
+        unit = "µs"
+      elsif d < 1.0
+        ds = d * 1000
+        unit = "ms"
+      else
+        ds = d
+        unit = "s"
+      end
+
+      sprintf("%.2f %s", ds, unit)
+    end
+  end
+end

data/test/performance/lib/performance/hako_client.rb

@@ -0,0 +1,31 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+require 'json'
+require 'net/http'
+require 'uri'
+
+module Performance
+  class HakoClient
+    BASE_URI = 'http://hako.pdx.vm.datanerd.us'
+
+    def initialize(token)
+      @token = token
+    end
+
+    def submit(result)
+      body = JSON.dump('result' => result.to_h)
+
+      uri = URI(BASE_URI + "/api/results")
+      req = Net::HTTP::Post.new(uri.to_s)
+      req.body = body
+      req.content_type = 'application/json'
+      req['Authorization'] = "Token token=\"#{@token}\""
+
+      Net::HTTP.start(uri.host, uri.port) do |conn|
+        conn.request(req)
+      end
+    end
+  end
+end

data/test/performance/lib/performance/hako_reporter.rb

@@ -0,0 +1,26 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  class HakoReporter
+    def initialize(results, elapsed, options={})
+      @results = results
+      @token = ENV['HAKO_TOKEN'] || options[:hako_token]
+    end
+
+    def report
+      Performance.logger.info("Uploading #{@results.size} results to Hako")
+      client = HakoClient.new(@token)
+      @results.each do |result|
+        rsp = client.submit(result)
+        case rsp
+        when Net::HTTPSuccess
+          Performance.logger.debug("Successfully posted result to Hako")
+        else
+          Performance.logger.error("Failed to post results to Hako: #{rsp.inspect}")
+        end
+      end
+    end
+  end
+end

data/test/performance/lib/performance/instrumentation/cpu_usage.rb

@@ -0,0 +1,26 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  module Instrumentation
+    class CpuUsage < Instrumentor
+      on_by_default
+
+      def before(*)
+        @times_before = Process.times
+      end
+
+      def after(*)
+        @times_after = Process.times
+      end
+
+      def results
+        {
+          :cpu_time_user   => @times_after.utime - @times_before.utime,
+          :cpu_time_system => @times_after.stime - @times_before.stime
+        }
+      end
+    end
+  end
+end

data/test/performance/lib/performance/instrumentation/gc_stats.rb

@@ -0,0 +1,58 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  module Instrumentation
+    class MRIGCStats < Instrumentor
+      platforms :mri_193, :mri_20, :mri_21, :mri_22
+      on_by_default
+
+      def before(*)
+        @stats_before = GC.stat
+      end
+
+      def after(*)
+        @stats_after = GC.stat
+      end
+
+      def results
+        heap_live_before = @stats_before[:heap_live_slots] || @stats_before[:heap_live_num] || @stats_before[:heap_live_slot]
+        heap_live_after  = @stats_after[:heap_live_slots]  || @stats_after[:heap_live_num]  || @stats_after[:heap_live_slot]
+
+        res = {
+          :gc_runs      => @stats_after[:count] - @stats_before[:count],
+          :live_objects => heap_live_after      - heap_live_before
+        }
+        if RUBY_VERSION >= "2.0.0"
+          allocs_before = @stats_before[:total_allocated_objects] || @stats_before[:total_allocated_object]
+          allocs_after  = @stats_after[:total_allocated_objects]  || @stats_after[:total_allocated_object]
+          res[:allocations] = allocs_after - allocs_before
+        end
+        res
+      end
+    end
+
+    class REEGCStats < Instrumentor
+      platforms :ree
+      on_by_default
+
+      def before(*)
+        @allocations_before = ObjectSpace.allocated_objects
+        @live_objects_before = ObjectSpace.live_objects
+      end
+
+      def after(*)
+        @allocations_after = ObjectSpace.allocated_objects
+        @live_objects_after = ObjectSpace.live_objects
+      end
+
+      def results
+        {
+          :allocations => @allocations_after - @allocations_before,
+          :live_objects => @live_objects_after - @live_objects_before,
+        }
+      end
+    end
+  end
+end

data/test/performance/lib/performance/instrumentation/perf_tools.rb

@@ -0,0 +1,30 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  module Instrumentation
+    class PerfToolsProfile < Instrumentor
+      platforms :mri_19, :mri_20, :mri_21
+
+      def self.setup
+        require 'tmpdir'
+        require 'perftools'
+      end
+
+      def before(test, test_name)
+        @profile_dir = Dir.mktmpdir('profile')
+        @profile_path = File.join(@profile_dir, "profile")
+        PerfTools::CpuProfiler.start(@profile_path)
+      end
+
+      def after(test, test_name)
+        PerfTools::CpuProfiler.stop
+        output_profile_path = artifact_path(test, test_name, "dot")
+        system("pprof.rb --dot #{@profile_path} >#{output_profile_path}")
+        @artifacts << output_profile_path
+        FileUtils.remove_entry_secure(@profile_dir)
+      end
+    end
+  end
+end

data/test/performance/lib/performance/instrumentation/stackprof.rb

@@ -0,0 +1,46 @@
+# encoding: utf-8
+# This file is distributed under Mmtrix's license terms.
+# See https://github.com/mmtrix/rpm/blob/master/LICENSE for complete details.
+
+module Performance
+  module Instrumentation
+    class StackProfProfile < Instrumentor
+      platforms :mri_21, :mri_22
+
+      def self.setup
+        require 'tmpdir'
+        require 'stackprof'
+      end
+
+      def mode
+        :wall
+      end
+
+      def before(test, test_name)
+        StackProf.start(:mode => mode)
+      end
+
+      def after(test, test_name)
+        StackProf.stop
+
+        output_dump_path = artifact_path(test, test_name, "dump")
+        StackProf.results(output_dump_path)
+        @artifacts << output_dump_path
+
+        results = Marshal.load(File.read(output_dump_path))
+        output_dot_path = artifact_path(test, test_name, "dot")
+        report = StackProf::Report.new(results)
+        File.open(output_dot_path, "w") do |f|
+          report.print_graphviz(nil, f)
+        end
+        @artifacts << output_dot_path
+      end
+    end
+
+    class StackProfAllocationProfile < StackProfProfile
+      def mode
+        :object
+      end
+    end
+  end
+end