derailed_benchmarks 1.3.6 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cc052693e6702c9fae168ebd5af966409edfd862cb10f493eeeef6f2b56c702a
-  data.tar.gz: e68aa2d95326f0d462b6f531318277180f2f155098c930fec1f7d9d5055a90c1
+  metadata.gz: cefb45902f24e4e4b0d8083b59ad9733005943db3df8a8b97b30552a27ff400b
+  data.tar.gz: 9d5921ac19df4db40e78ec2ca6396593f1f5235c3caaae84c332163ed3c5e4ec
 SHA512:
-  metadata.gz: da2a2ddb042b1d7d50e2d6ef70dc281fbe09bac107b763e64fe26bb47b30a3a0b46580e618f7c97c29c689b3b0981333a902dbb32f3c8df340a6cc2c54db4cf9
-  data.tar.gz: 113502784b03d89cafc13ed4856db140352a83ab5e68393e5a09bd9f7942505aa0aec33d8b35858e97d873c2e6435fe1cdb140ae1915dbd60a4d7d2c82129310
+  metadata.gz: 3335a72cc6b965b76f6804c3dd71ae5cf61d56f5203447870f0ad31a30c6cef5c47b28cf569311d7c033fe2898f2653ab041d7c7dceefe85e5501c4b79ab06b4
+  data.tar.gz: f591582236f863286b65d9a96b33b481e0a2c9cb05a6985ecb4123d121452ebda81a1b5d123f1d0a77c297939e8f6f2de16ca14dad892436109d819ff2b7ee91

data/.travis.yml

@@ -8,6 +8,7 @@ gemfile:
   - gemfiles/rails_5_1.gemfile
   - gemfiles/rails_5_2.gemfile
   - gemfiles/rails_6_0.gemfile
+  - gemfiles/rails_git.gemfile
 
 before_install:
   - gem install bundler

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 # A Log of Changes!
 
+## 1.4.0
+
+- Allow configuration of `perf:ips` benchmark.
+- Fix bug with `require_relative` [#142](https://github.com/schneems/derailed_benchmarks/pull/142)
+- Introduce `perf:library` to profile patches to libraries (like Rails) [#135](https://github.com/schneems/derailed_benchmarks/pull/135), [#139](https://github.com/schneems/derailed_benchmarks/pull/139), [#140](https://github.com/schneems/derailed_benchmarks/pull/140), [#141](https://github.com/schneems/derailed_benchmarks/pull/141)
+
 ## 1.3.6
 
 - `require_relative` is now measured [commit](https://github.com/schneems/derailed_benchmarks/commit/af11bcc46a4fa24f79e4897a51034927a56e077e)
@@ -67,4 +73,4 @@
 
 ## [0.0.0] - 2014-08-15
 
-- Initial release
+- Initial release

data/README.md

@@ -317,7 +317,7 @@ TOP: 54.1836 MiB
     action_view/base: 0.4336 MiB
 ```
 
-You can use `CUT_OFF=0.3` to only show files that have above a certain memory useage, this can be used to help eliminate noise.
+You can use `CUT_OFF=0.3` to only show files that have above a certain memory usage, this can be used to help eliminate noise.
 
 If your application code is extremely large at boot consider using `$ derailed exec perf:objects` to debug low level object creation.
 
@@ -388,12 +388,80 @@ $ bundle exec derailed exec perf:test
 
 But I wouldn't, benchmark-ips is a better measure.
 
+### Configuring `benchmark-ips`
+
+The `benchmark-ips` gem allows for a number of test run customizations, and `derailed_benchmarks` exposes a few of them via environment variables.
+
+- `IPS_WARMUP`: number of seconds spent warming up the app, defaullt is `2`
+- `IPS_TIME`: number of seconds to run ips benchmark for after warm up, defaullt is `5`
+- `IPS_SUITE`: custom suite to use to run test
+- `IPS_ITERATIONS`: number of times to run the test, displaying that last result, defaullt is `1`
+
+## I made a patch to to Rails how can I tell if it made my Rails app faster and test for statistical significance
+
+When you're trying to submit a performance patch to rails/rails then they'll likely ask you for a benchmark. While you can sometimes provide a microbenchmark, a real world full stack request/response test is the gold standard.
+
+That's what this section is about. You'll need a rails app, ideally one you can open source (see [example apps](http://codetriage.com/example_app) if you need inspiration for extracting your private code into something external).
+
+Then you'll need to fork rails and make a branch. Then point your rails app to your branch in your gemfile
+
+```
+gem 'rails', github: "<github username>/rails", branch: "<your branch name>"
+```
+
+or point it at your local copy:
+
+```
+gem 'rails', path: "<path/to/your/local/copy/rails>"
+```
+
+To run your test:
+
+```
+$ bundle exec derailed exec perf:library
+```
+
+This will automatically test the two latest commits of Rails (or the library you've specified). If you would like to compare against different SHAs you can manually specify them:
+
+```
+$ SHAS_TO_TEST="7b4d80cb373e,13d6aa3a7b70" bundle exec derailed exec perf:library
+```
+
+Use a comma to seperate your branch names with the `SHAS_TO_TEST` env var, or omit the env var to use the last 2 git commits.
+
+If you only include one SHA, then derailed will grab the latest commit and compare it to that SHA.
+
+These tests might take a along time to run so the output is stored on disk incase you want to see them in the future, they're at `tmp/library_branches/<timestamp>` and labeled with the same names as your commits.
+
+When the test is done it will output which commit "won" and by how much:
+
+```
+❤️ ❤️ ❤️  (Statistically Significant) ❤️ ❤️ ❤️
+
+[7b4d80cb37] "1.8x Faster Partial Caching - Faster Cache Keys" - (10.9711965 seconds)
+  FASTER by:
+    1.0870x [older/newer]
+    8.0026% [(older - newer) / older * 100]
+[13d6aa3a7b] "Merge pull request #36284 from kamipo/fix_eager_loading_with_string_joins" - (11.9255485 seconds)
+
+P-value: 4.635595463712749e-05
+Is significant? (P-value < 0.05): true
+```
+
+You can provide this to the Rails team along with the example app you used to benchmark (so they can independently verify if needed).
+
+Generally performance patches have to be weighted in terms of how much they help versus how large/difficult/gnarly the patch is. If the above example was a really tiny patch and it was in a common component, then half a percent might be a justafiable increase. If it was a huge re-write then it's likely going to be closed. In general I tend to not submit patches unless I'm seeing `>= 1%` performance increases.
+
+You can use this to test changes in other libraries that aren't rails, you just have to tell it the path to the library you want to test against with the `DERAILED_PATH_TO_LIBRARY` env var.
+
+> To get the best results before running tests you should close all programs on your laptop, turn on a program to prevent your laptop from going to sleep (or increase your sleep timer). Make sure it's plugged into a power outlet and  go grab a cup of coffee. If you do anything on your laptop while this test is running you risk the chance of skewing your results.
+
+By default derailed will stop once statistical signficance has been detected, you can tune this behavior by setting `DERAILED_STOP_VALID_COUNT` env var. Setting this to a positive number, will increase the number of iterations required that are detected to be statistically significant. For example setting it to 100 might result in 120 runs if it takes 20 runs to detect significance. Generally the more runs you have, the more accurate your averages will be. You can disable this all together by setting `DERAILED_STOP_VALID_COUNT=0` which will force derailed to run all iterations.
 
 ## Environment Variables
 
 All the tasks accept configuration in the form of environment variables.
 
-
 ### Increasing or decreasing test count `TEST_COUNT`
 
 For tasks that are run a number of times you can set the number using `TEST_COUNT` for example:

data/Rakefile

@@ -1,4 +1,6 @@
 # encoding: UTF-8
+# frozen_string_literal: true
+
 require 'rubygems'
 require 'bundler'
 require "bundler/gem_tasks"

data/bin/derailed

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 unless File.respond_to? :realpath
   class File #:nodoc:

data/derailed_benchmarks.gemspec

@@ -1,4 +1,6 @@
 # -*- encoding: utf-8 -*-
+# frozen_string_literal: true
+
 lib = File.expand_path('../lib', __FILE__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 'derailed_benchmarks/version'
@@ -27,10 +29,11 @@ Gem::Specification.new do |gem|
   gem.add_dependency "rack",            ">= 1"
   gem.add_dependency "rake",            "> 10", "< 13"
   gem.add_dependency "thor",            "~> 0.19"
+  gem.add_dependency "ruby-statistics", ">= 2.1"
 
   gem.add_development_dependency "capybara",  "~> 2"
   gem.add_development_dependency "m"
   gem.add_development_dependency "rails",     "> 3", "<= 6"
-  gem.add_development_dependency "devise",    "> 3", "< 5"
+  gem.add_development_dependency "devise",    "> 3", "< 6"
   gem.add_development_dependency "appraisal", "2.2.0"
 end

data/gemfiles/rails_5_1.gemfile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 # This file was generated by Appraisal
 
 source "https://rubygems.org"

data/gemfiles/rails_5_2.gemfile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 # This file was generated by Appraisal
 
 source "https://rubygems.org"

data/gemfiles/rails_6_0.gemfile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 # This file was generated by Appraisal
 
 source "https://rubygems.org"

data/gemfiles/rails_git.gemfile

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+# $ BUNDLE_GEMFILE="$(pwd)/gemfiles/rails_git.gemfile" bundle exec m test/integration/tasks_test.rb:30
+
+source "https://rubygems.org"
+
+gem "rails", github: "rails/rails", ref: "3054e1d584e7eca110c69a1f8423f2e0866abbf9"
+
+gem 'devise', github: "plataformatec/devise"
+
+group :development, :test do
+  gem "sqlite3", platform: [:ruby, :mswin, :mingw]
+  gem "activerecord-jdbcsqlite3-adapter", "~> 1.3.13", platform: :jruby
+  gem "test-unit", "~> 3.0"
+end
+
+gemspec path: "../"
+
+ENV['USING_RAILS_GIT'] = "1"

data/lib/derailed_benchmarks.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require 'time'
 require 'bundler'
 
@@ -13,6 +15,21 @@ module DerailedBenchmarks
     attr_accessor :auth
   end
 
+  def self.rails_path_on_disk
+    require 'rails/version'
+    rails_version_file = Rails.method(:version).source_location[0]
+    path = Pathname.new(rails_version_file).expand_path.parent.parent
+
+    while path != Pathname.new("/")
+      basename = path.expand_path.basename.to_s
+
+      break if basename.start_with?("rails") && basename != "railties"
+      path = path.parent
+    end
+    raise "Could not find rails folder on a folder in #{rails_version_file}"  if path == Pathname.new("/")
+    path.expand_path
+  end
+
   def self.add_auth(app)
     if use_auth = ENV['USE_AUTH']
       puts "Auth: #{use_auth}"
@@ -26,6 +43,9 @@ end
 require 'derailed_benchmarks/require_tree'
 require 'derailed_benchmarks/auth_helper'
 
+require 'derailed_benchmarks/stats_in_file'
+require 'derailed_benchmarks/stats_from_dir'
+
 if DerailedBenchmarks.gem_is_bundled?("devise")
   DerailedBenchmarks.auth = DerailedBenchmarks::AuthHelpers::Devise.new
 end

data/lib/derailed_benchmarks/auth_helper.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require 'securerandom'
 
 module DerailedBenchmarks

data/lib/derailed_benchmarks/auth_helpers/devise.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 module DerailedBenchmarks
   class AuthHelpers
     # Devise helper for authenticating requests

data/lib/derailed_benchmarks/core_ext/kernel_require.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require 'get_process_mem'
 require 'derailed_benchmarks/require_tree'
 
@@ -20,8 +22,11 @@ module Kernel
   end
 
   def require_relative(file)
-    # Kernel.require_relative(file)
-    require File.expand_path("../#{file}", caller_locations(1, 1)[0].absolute_path)
+    if Pathname.new(file).absolute?
+      require file
+    else
+      require File.expand_path("../#{file}", caller_locations(1, 1)[0].absolute_path)
+    end
   end
 
   class << self

data/lib/derailed_benchmarks/load_tasks.rb

@@ -0,0 +1,138 @@
+# frozen_string_literal: true
+
+namespace :perf do
+  task :rails_load do
+    ENV["RAILS_ENV"] ||= "production"
+    ENV['RACK_ENV']  = ENV["RAILS_ENV"]
+    ENV["DISABLE_SPRING"] = "true"
+
+    ENV["SECRET_KEY_BASE"] ||= "foofoofoo"
+
+    ENV['LOG_LEVEL'] ||= "FATAL"
+
+    require 'rails'
+
+    puts "Booting: #{Rails.env}"
+
+    %W{ . lib test config }.each do |file|
+      $LOAD_PATH << File.expand_path(file)
+    end
+
+    require 'application'
+
+    Rails.env = ENV["RAILS_ENV"]
+
+    DERAILED_APP = Rails.application
+
+    if DERAILED_APP.respond_to?(:initialized?)
+      DERAILED_APP.initialize! unless DERAILED_APP.initialized?
+    else
+      DERAILED_APP.initialize! unless DERAILED_APP.instance_variable_get(:@initialized)
+    end
+
+    if  ENV["DERAILED_SKIP_ACTIVE_RECORD"] && defined? ActiveRecord
+      if defined? ActiveRecord::Tasks::DatabaseTasks
+        ActiveRecord::Tasks::DatabaseTasks.create_current
+      else # Rails 3.2
+        raise "No valid database for #{ENV['RAILS_ENV']}, please create one" unless ActiveRecord::Base.connection.active?.inspect
+      end
+
+      ActiveRecord::Migrator.migrations_paths = DERAILED_APP.paths['db/migrate'].to_a
+      ActiveRecord::Migration.verbose         = true
+      ActiveRecord::Migrator.migrate(ActiveRecord::Migrator.migrations_paths, nil)
+    end
+
+    DERAILED_APP.config.consider_all_requests_local = true
+  end
+
+  task :rack_load do
+    puts "You're not using Rails"
+    puts "You need to tell derailed how to boot your app"
+    puts "In your perf.rake add:"
+    puts
+    puts "namespace :perf do"
+    puts "  task :rack_load do"
+    puts "    # DERAILED_APP = your code here"
+    puts "  end"
+    puts "end"
+  end
+
+  task :setup do
+    if DerailedBenchmarks.gem_is_bundled?("railties")
+      Rake::Task["perf:rails_load"].invoke
+    else
+      Rake::Task["perf:rack_load"].invoke
+    end
+
+    WARM_COUNT  = (ENV['WARM_COUNT'] || 0).to_i
+    TEST_COUNT  = (ENV['TEST_COUNT'] || ENV['CNT'] || 1_000).to_i
+    PATH_TO_HIT = ENV["PATH_TO_HIT"] || ENV['ENDPOINT'] || "/"
+    puts "Endpoint: #{ PATH_TO_HIT.inspect }"
+
+    HTTP_HEADER_PREFIX = "HTTP_".freeze
+    RACK_HTTP_HEADERS = ENV.select { |key| key.start_with?(HTTP_HEADER_PREFIX) }
+
+    HTTP_HEADERS = RACK_HTTP_HEADERS.keys.inject({}) do |hash, rack_header_name|
+      # e.g. "HTTP_ACCEPT_CHARSET" -> "Accept-Charset"
+      header_name = rack_header_name[HTTP_HEADER_PREFIX.size..-1].split("_").map(&:downcase).map(&:capitalize).join("-")
+      hash[header_name] = RACK_HTTP_HEADERS[rack_header_name]
+      hash
+    end
+    puts "HTTP headers: #{HTTP_HEADERS}" unless HTTP_HEADERS.empty?
+
+    CURL_HTTP_HEADER_ARGS = HTTP_HEADERS.map { |http_header_name, value| "-H \"#{http_header_name}: #{value}\"" }.join(" ")
+
+    require 'rack/test'
+    require 'rack/file'
+
+    DERAILED_APP = DerailedBenchmarks.add_auth(Object.class_eval { remove_const(:DERAILED_APP) })
+    if server = ENV["USE_SERVER"]
+      @port = (3000..3900).to_a.sample
+      puts "Port: #{ @port.inspect }"
+      puts "Server: #{ server.inspect }"
+      thread = Thread.new do
+        Rack::Server.start(app: DERAILED_APP, :Port => @port, environment: "none", server: server)
+      end
+      sleep 1
+
+      def call_app(path = File.join("/", PATH_TO_HIT))
+        cmd = "curl #{CURL_HTTP_HEADER_ARGS} 'http://localhost:#{@port}#{path}' -s --fail 2>&1"
+        response = `#{cmd}`
+        unless $?.success?
+          STDERR.puts "Couldn't call app."
+          STDERR.puts "Bad request to #{cmd.inspect} \n\n***RESPONSE***:\n\n#{ response.inspect }"
+
+          FileUtils.mkdir_p("tmp")
+          File.open("tmp/fail.html", "w+") {|f| f.write response.body }
+
+          `open #{File.expand_path("tmp/fail.html")}` if ENV["DERAILED_DEBUG"]
+
+          exit(1)
+        end
+      end
+    else
+      @app = Rack::MockRequest.new(DERAILED_APP)
+
+      def call_app
+        response = @app.get(PATH_TO_HIT, RACK_HTTP_HEADERS)
+        if response.status != 200
+          STDERR.puts "Couldn't call app. Bad request to #{PATH_TO_HIT}! Resulted in #{response.status} status."
+          STDERR.puts "\n\n***RESPONSE BODY***\n\n"
+          STDERR.puts response.body
+
+          FileUtils.mkdir_p("tmp")
+          File.open("tmp/fail.html", "w+") {|f| f.write response.body }
+
+          `open #{File.expand_path("tmp/fail.html")}` if ENV["DERAILED_DEBUG"]
+
+          exit(1)
+        end
+        response
+      end
+    end
+    if WARM_COUNT > 0
+      puts "Warming up app: #{WARM_COUNT} times"
+      WARM_COUNT.times { call_app }
+    end
+  end
+end

data/lib/derailed_benchmarks/require_tree.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 # Tree structure used to store and sort require memory costs
 # RequireTree.new('get_process_mem')
 module DerailedBenchmarks
@@ -38,7 +40,7 @@ module DerailedBenchmarks
     end
 
     def to_string
-      str = "#{name}: #{cost.round(4)} MiB"
+      str = +"#{name}: #{cost.round(4)} MiB"
       if parent && REQUIRED_BY[self.name.to_s]
         names = REQUIRED_BY[self.name.to_s].uniq - [parent.name.to_s]
         if names.any?

data/lib/derailed_benchmarks/stats_from_dir.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+require 'bigdecimal'
+require 'statistics'
+
+module DerailedBenchmarks
+  # A class used to read several benchmark files
+  # it will parse each file, then sort by average
+  # time of benchmarks. It can be used to find
+  # the fastest and slowest examples and give information
+  # about them such as what the percent difference is
+  # and if the results are statistically significant
+  #
+  # Example:
+  #
+  #   branch_info = {}
+  #   branch_info["loser"]  = { desc: "Old commit", time: Time.now, file: dir.join("loser.bench.txt"), name: "loser" }
+  #   branch_info["winner"] = { desc: "I am the new commit", time: Time.now + 1, file: dir.join("winner.bench.txt"), name: "winner" }
+  #   stats = DerailedBenchmarks::StatsFromDir.new(branch_info)
+  #
+  #   stats.newest.average  # => 10.5
+  #   stats.oldest.average  # => 11.0
+  #   stats.significant?    # => true
+  #   stats.x_faster        # => "1.0476"
+  class StatsFromDir
+    FORMAT = "%0.4f"
+    attr_reader :stats, :oldest, :newest
+
+    def initialize(hash)
+      @files = []
+
+      hash.each do |branch, info_hash|
+        file = info_hash.fetch(:file)
+        desc = info_hash.fetch(:desc)
+        time = info_hash.fetch(:time)
+        @files << StatsForFile.new(file: file, desc: desc, time: time, name: branch)
+      end
+      @files.sort_by! { |f| f.time }
+      @oldest = @files.first
+      @newest = @files.last
+    end
+
+    def call
+      @files.each(&:call)
+      @stats = students_t_test
+      self
+    end
+
+    def students_t_test(series_1=oldest.values, series_2=newest.values)
+      StatisticalTest::TTest.perform(
+        alpha = 0.05,
+        :two_tail,
+        series_1,
+        series_2
+      )
+    end
+
+    def significant?
+      @stats[:alternative]
+    end
+
+    def p_value
+      @stats[:p_value].to_f
+    end
+
+    def x_faster
+      FORMAT % (oldest.average/newest.average).to_f
+    end
+
+    def percent_faster
+      FORMAT % (((oldest.average - newest.average) / oldest.average).to_f  * 100)
+    end
+
+    def change_direction
+      newest.average < oldest.average ? "FASTER" : "SLOWER"
+    end
+
+    def banner(io = Kernel)
+      io.puts
+      if significant?
+        io.puts "❤️ ❤️ ❤️  (Statistically Significant) ❤️ ❤️ ❤️"
+      else
+        io.puts "👎👎👎(NOT Statistically Significant) 👎👎👎"
+      end
+      io.puts
+      io.puts "[#{newest.name}] #{newest.desc.inspect} - (#{newest.average} seconds)"
+      io.puts "  #{change_direction} by:"
+      io.puts "    #{x_faster}x [older/newer]"
+      io.puts "    #{percent_faster}\% [(older - newer) / older * 100]"
+      io.puts "[#{oldest.name}] #{oldest.desc.inspect} - (#{oldest.average} seconds)"
+      io.puts
+      io.puts "Iterations per sample: #{ENV["TEST_COUNT"]}"
+      io.puts "Samples: #{newest.values.length}"
+      io.puts "P-value: #{p_value}"
+      io.puts "Is significant? (P-value < 0.05): #{significant?}"
+      io.puts
+    end
+  end
+end