derailed_benchmarks 1.4.2 → 1.8.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2ed7fac91722a804da9125b03501653c1e1d78993b9c7d874e5e597656cb3c12
4
- data.tar.gz: 5c36d946b6e0533d195f9b285d1d6541e4890a2f2eb37f2ca5d26b4c9434b655
3
+ metadata.gz: 24d0d07556d2d6676ea8b2c06d58583071fd9302c056d36db05d252480889308
4
+ data.tar.gz: 0c783db16c1612173975c12760b101fc30cbd5765d296b2fd66506992305ac69
5
5
  SHA512:
6
- metadata.gz: be591fafddeff784a41a817c51f1f0f4c6a7f0b14b80f16b4792c61564af97f22a38f8cc999d5e7431cae3393cbfe427eebb291e2bc2efac7c578a802a870ec7
7
- data.tar.gz: 4e0fceb32c0acbc09f74c91959121a9033b5ac188700eb225c7d51a4cee796665640302e5e7eed5508c870c63fc26a4370ed1ed6e88b15757fc8c291a70a3c88
6
+ metadata.gz: e37aa667a5fe031a384a343de999a34ef9f1e0ba98f8190d6f86d326ee30f968e3555344fbaf8841f99da85354c2b4383fe1b75ab166eb0df9134d20ee48fd95
7
+ data.tar.gz: 3f581637e835d808b8264786ffd70159fe534c4147e72e65f5649fe7ecb0b92b9edc215c9edd43e2ca51a510793ae475da3e0318262103bb741823746d9f3421
@@ -1,10 +1,13 @@
1
1
  name: Check Changelog
2
- on: [pull_request]
2
+
3
+ on:
4
+ pull_request:
5
+ types: [opened, reopened, edited, synchronize]
3
6
  jobs:
4
- build:
5
- runs-on: ubuntu-latest
6
- steps:
7
- - uses: actions/checkout@v1
8
- - name: Check that CHANGELOG is touched
9
- run: |
10
- cat $GITHUB_EVENT_PATH | jq .pull_request.title | grep -i '\[\(\(changelog skip\)\|\(ci skip\)\)\]' || git diff remotes/origin/${{ github.base_ref }} --name-only | grep CHANGELOG.md
7
+ build:
8
+ runs-on: ubuntu-latest
9
+ steps:
10
+ - uses: actions/checkout@v1
11
+ - name: Check that CHANGELOG is touched
12
+ run: |
13
+ cat $GITHUB_EVENT_PATH | jq .pull_request.title | grep -i '\[\(\(changelog skip\)\|\(ci skip\)\)\]' || git diff remotes/origin/${{ github.base_ref }} --name-only | grep CHANGELOG.md
@@ -1,18 +1,20 @@
1
1
  language: ruby
2
2
  rvm:
3
- - 2.5.5
4
- - 2.6.3
5
- - ruby-head
3
+ - 2.2.10
4
+ - 2.5.8
5
+ - 2.7.1
6
6
 
7
7
  gemfile:
8
8
  - gemfiles/rails_5_1.gemfile
9
- - gemfiles/rails_5_2.gemfile
10
9
  - gemfiles/rails_6_0.gemfile
11
10
  - gemfiles/rails_git.gemfile
12
11
 
13
12
  before_install:
14
- - gem install bundler
13
+ - gem install bundler -v 1.17.3
15
14
 
16
- matrix:
15
+ jobs:
17
16
  allow_failures:
18
- - rvm: ruby-head
17
+ - rvm: 2.2.10
18
+ gemfile: gemfiles/rails_6_0.gemfile
19
+ - rvm: 2.2.10
20
+ gemfile: gemfiles/rails_git.gemfile
@@ -1,4 +1,31 @@
1
- ## master (unreleased)
1
+ ## HEAD
2
+
3
+ ## 1.8.0
4
+
5
+ - Ruby 2.2 is now officialy supported and tested (https://github.com/schneems/derailed_benchmarks/pull/177)
6
+
7
+ ## 1.7.0
8
+
9
+ - Add histogram support to `perf:library` (https://github.com/schneems/derailed_benchmarks/pull/169)
10
+ - Fix bug with `Kernel#require` patch when Zeitwerk is enabled (https://github.com/schneems/derailed_benchmarks/pull/170)
11
+
12
+ ## 1.6.0
13
+
14
+ - Added the `perf:app` command to compare commits within the same application. (https://github.com/schneems/derailed_benchmarks/pull/157)
15
+ - Allow Rails < 7 and 1.0 <= Thor < 2 (https://github.com/schneems/derailed_benchmarks/pull/168)
16
+
17
+ ## 1.5.0
18
+
19
+ - Test `perf:library` results against 99% confidence interval in addition to 95% (https://github.com/schneems/derailed_benchmarks/pull/165)
20
+ - Change default, `perf:library` tests do not stop automatically any more (https://github.com/schneems/derailed_benchmarks/pull/164)
21
+
22
+ ## 1.4.4
23
+
24
+ - Fix alignment of deicmals in output (https://github.com/schneems/derailed_benchmarks/pull/161)
25
+
26
+ ## 1.4.3
27
+
28
+ - perf:library now uses median instead of average (https://github.com/schneems/derailed_benchmarks/pull/160)
2
29
 
3
30
  ## 1.4.2
4
31
 
data/README.md CHANGED
@@ -10,7 +10,7 @@ A series of things you can use to benchmark a Rails or Ruby app.
10
10
  ## Compatibility/Requirements
11
11
 
12
12
  This gem has been tested and is known to work with Rails 3.2+ using Ruby
13
- 2.1+. Some commands __may__ work on older versions of Ruby, but not all commands are supported.
13
+ 2.2+. Some commands __may__ work on older versions of Ruby, but not all commands are supported.
14
14
 
15
15
  For some benchmarks, not all, you'll need to verify you have a working version of curl on your OS:
16
16
 
@@ -415,13 +415,23 @@ or point it at your local copy:
415
415
  gem 'rails', path: "<path/to/your/local/copy/rails>"
416
416
  ```
417
417
 
418
- To run your test:
418
+ To run your tests within the context of your current app/repo:
419
+
420
+ ```
421
+ $ bundle exec derailed exec perf:app
422
+ ```
423
+
424
+ This will automatically test the two latest commits of your library/current directory.
425
+
426
+ If you'd like to test the Rails library instead, make sure that `ENV[DERAILED_PATH_TO_LIBRARY]` is unset.
419
427
 
420
428
  ```
421
429
  $ bundle exec derailed exec perf:library
422
430
  ```
423
431
 
424
- This will automatically test the two latest commits of Rails (or the library you've specified). If you would like to compare against different SHAs you can manually specify them:
432
+ This will automatically test the two latest commits of Rails.
433
+
434
+ If you would also like to compare against different SHAs you can manually specify them:
425
435
 
426
436
  ```
427
437
  $ SHAS_TO_TEST="7b4d80cb373e,13d6aa3a7b70" bundle exec derailed exec perf:library
@@ -431,7 +441,7 @@ Use a comma to seperate your branch names with the `SHAS_TO_TEST` env var, or om
431
441
 
432
442
  If you only include one SHA, then derailed will grab the latest commit and compare it to that SHA.
433
443
 
434
- These tests might take a along time to run so the output is stored on disk incase you want to see them in the future, they're at `tmp/library_branches/<timestamp>` and labeled with the same names as your commits.
444
+ These tests might take a along time to run so the output is stored on disk incase you want to see them in the future, they're at `tmp/compare_branches/<timestamp>` and labeled with the same names as your commits.
435
445
 
436
446
  When the test is done it will output which commit "won" and by how much:
437
447
 
@@ -453,7 +463,7 @@ You can use this to test changes in other libraries that aren't rails, you just
453
463
 
454
464
  > To get the best results before running tests you should close all programs on your laptop, turn on a program to prevent your laptop from going to sleep (or increase your sleep timer). Make sure it's plugged into a power outlet and go grab a cup of coffee. If you do anything on your laptop while this test is running you risk the chance of skewing your results.
455
465
 
456
- By default derailed will stop once statistical signficance has been detected, you can tune this behavior by setting `DERAILED_STOP_VALID_COUNT` env var. Setting this to a positive number, will increase the number of iterations required that are detected to be statistically significant. For example setting it to 100 might result in 120 runs if it takes 20 runs to detect significance. Generally the more runs you have, the more accurate your averages will be. You can disable this all together by setting `DERAILED_STOP_VALID_COUNT=0` which will force derailed to run all iterations.
466
+ As the test is executing, intermediate results will be printed every 50 iterations.
457
467
 
458
468
  ## Environment Variables
459
469
 
@@ -20,7 +20,7 @@ Gem::Specification.new do |gem|
20
20
  gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
21
21
  gem.require_paths = ["lib"]
22
22
 
23
- gem.required_ruby_version = ">= 2.1.0"
23
+ gem.required_ruby_version = ">= 2.2.0"
24
24
 
25
25
  gem.add_dependency "heapy", "~> 0"
26
26
  gem.add_dependency "memory_profiler", "~> 0"
@@ -28,12 +28,13 @@ Gem::Specification.new do |gem|
28
28
  gem.add_dependency "benchmark-ips", "~> 2"
29
29
  gem.add_dependency "rack", ">= 1"
30
30
  gem.add_dependency "rake", "> 10", "< 14"
31
- gem.add_dependency "thor", "~> 0.19"
31
+ gem.add_dependency "thor", ">= 0.19", "< 2"
32
32
  gem.add_dependency "ruby-statistics", ">= 2.1"
33
+ gem.add_dependency "mini_histogram", ">= 0.2.1"
33
34
 
34
35
  gem.add_development_dependency "capybara", "~> 2"
35
36
  gem.add_development_dependency "m"
36
- gem.add_development_dependency "rails", "> 3", "<= 6"
37
+ gem.add_development_dependency "rails", "> 3", "<= 7"
37
38
  gem.add_development_dependency "devise", "> 3", "< 6"
38
39
  gem.add_development_dependency "appraisal", "2.2.0"
39
40
  end
@@ -43,8 +43,9 @@ end
43
43
  require 'derailed_benchmarks/require_tree'
44
44
  require 'derailed_benchmarks/auth_helper'
45
45
 
46
- require 'derailed_benchmarks/stats_in_file'
46
+ require 'derailed_benchmarks/stats_for_file'
47
47
  require 'derailed_benchmarks/stats_from_dir'
48
+ require 'derailed_benchmarks/git/switch_project'
48
49
 
49
50
  if DerailedBenchmarks.gem_is_bundled?("devise")
50
51
  DerailedBenchmarks.auth = DerailedBenchmarks::AuthHelpers::Devise.new
@@ -11,14 +11,20 @@ ENV['CUT_OFF'] ||= "0.3"
11
11
  # Monkey patch kernel to ensure that all `require` calls call the same
12
12
  # method
13
13
  module Kernel
14
+ REQUIRE_STACK = []
14
15
 
15
- private
16
+ module_function
16
17
 
17
- alias :original_require :require
18
- REQUIRE_STACK = []
18
+ alias_method :original_require, :require
19
+ alias_method :original_require_relative, :require_relative
19
20
 
20
21
  def require(file)
21
- Kernel.require(file)
22
+ measure_memory_impact(file) do |file|
23
+ # "source_annotation_extractor" is deprecated in Rails 6
24
+ # # if we don't skip the library it leads to a crash
25
+ # next if file == "rails/source_annotation_extractor" && Rails.version >= '6.0'
26
+ original_require(file)
27
+ end
22
28
  end
23
29
 
24
30
  def require_relative(file)
@@ -29,10 +35,7 @@ module Kernel
29
35
  end
30
36
  end
31
37
 
32
- class << self
33
- alias :original_require :require
34
- alias :original_require_relative :require_relative
35
- end
38
+ private
36
39
 
37
40
  # The core extension we use to measure require time of all requires
38
41
  # When a file is required we create a tree node with its file name.
@@ -46,7 +49,7 @@ module Kernel
46
49
  # When a require returns we remove it from the require stack so we don't
47
50
  # accidentally push additional children nodes to it. We then store the
48
51
  # memory cost of the require in the tree node.
49
- def self.measure_memory_impact(file, &block)
52
+ def measure_memory_impact(file, &block)
50
53
  mem = GetProcessMem.new
51
54
  node = DerailedBenchmarks::RequireTree.new(file)
52
55
 
@@ -68,15 +71,6 @@ end
68
71
  TOP_REQUIRE = DerailedBenchmarks::RequireTree.new("TOP")
69
72
  REQUIRE_STACK.push(TOP_REQUIRE)
70
73
 
71
- Kernel.define_singleton_method(:require) do |file|
72
- measure_memory_impact(file) do |file|
73
- # "source_annotation_extractor" is deprecated in Rails 6
74
- # # if we don't skip the library it leads to a crash
75
- # next if file == "rails/source_annotation_extractor" && Rails.version >= '6.0'
76
- original_require(file)
77
- end
78
- end
79
-
80
74
  class Object
81
75
  private
82
76
 
@@ -0,0 +1,36 @@
1
+ module DerailedBenchmarks
2
+ # Represents a specific commit in a git repo
3
+ #
4
+ # Can be used to get information from the commit or to check it out
5
+ #
6
+ # commit = GitCommit.new(path: "path/to/repo", ref: "6e642963acec0ff64af51bd6fba8db3c4176ed6e")
7
+ # commit.short_sha # => "6e64296"
8
+ # commit.checkout! # Will check out the current commit at the repo in the path
9
+ class Git::Commit
10
+ attr_reader :ref, :description, :time, :short_sha, :log
11
+
12
+ def initialize(path: , ref: , log_dir: Pathname.new("/dev/null"))
13
+ @in_git_path = Git::InPath.new(path)
14
+ @ref = ref
15
+ @log = log_dir.join("#{file_safe_ref}.bench.txt")
16
+
17
+ Dir.chdir(path) do
18
+ checkout!
19
+ @description = @in_git_path.description
20
+ @short_sha = @in_git_path.short_sha
21
+ @time = @in_git_path.time
22
+ end
23
+ end
24
+
25
+ alias :desc :description
26
+ alias :file :log
27
+
28
+ def checkout!
29
+ @in_git_path.checkout!(ref)
30
+ end
31
+
32
+ private def file_safe_ref
33
+ ref.gsub('/', ':')
34
+ end
35
+ end
36
+ end
@@ -0,0 +1,59 @@
1
+ module DerailedBenchmarks
2
+ # A class for running commands in a git directory
3
+ #
4
+ # It's faster to check if we're already in that directory instead
5
+ # of having to `cd` into each time. https://twitter.com/schneems/status/1305196730170961920
6
+ #
7
+ # Example:
8
+ #
9
+ # in_git_path = InGitPath.new(`bundle info heapy --path`.strip)
10
+ # in_git_path.checkout!("f0f92b06156f2274021aa42f15326da041ee9009")
11
+ # in_git_path.short_sha # => "f0f92b0"
12
+ class Git::InPath
13
+ attr_reader :path
14
+
15
+ def initialize(path)
16
+ @path = path
17
+ end
18
+
19
+ def description
20
+ run!("git log --oneline --format=%B -n 1 HEAD | head -n 1")
21
+ end
22
+
23
+ def short_sha
24
+ run!("git rev-parse --short HEAD")
25
+ end
26
+
27
+ def time_stamp_string
28
+ run!("git log -n 1 --pretty=format:%ci") # https://stackoverflow.com/a/25921837/147390
29
+ end
30
+
31
+ def branch
32
+ branch = run!("git rev-parse --abbrev-ref HEAD")
33
+ branch == "HEAD" ? nil : branch
34
+ end
35
+
36
+ def checkout!(ref)
37
+ run!("git checkout '#{ref}' 2>&1")
38
+ end
39
+
40
+ def time
41
+ DateTime.parse(time_stamp_string)
42
+ end
43
+
44
+ def run(cmd)
45
+ if Dir.pwd == path
46
+ out = `#{cmd}`.strip
47
+ else
48
+ out = `cd #{path} && #{cmd}`.strip
49
+ end
50
+ out
51
+ end
52
+
53
+ def run!(cmd)
54
+ out = run(cmd)
55
+ raise "Error while running #{cmd.inspect}: #{out}" unless $?.success?
56
+ out
57
+ end
58
+ end
59
+ end
@@ -0,0 +1,128 @@
1
+ module DerailedBenchmarks
2
+ class Git
3
+ end
4
+ end
5
+ require_relative "in_path.rb"
6
+ require_relative "commit.rb"
7
+
8
+ module DerailedBenchmarks
9
+ # Wraps two or more git commits in a specific location
10
+ #
11
+ # Returns an array of GitCommit objects that can be used to manipulate
12
+ # and checkout the repo
13
+ #
14
+ # Example:
15
+ #
16
+ # `git clone https://sharpstone/default_ruby tmp/default_ruby`
17
+ #
18
+ # project = GitSwitchProject.new(path: "tmp/default_ruby")
19
+ #
20
+ # By default it will represent the last two commits:
21
+ #
22
+ # project.commits.length # => 2
23
+ #
24
+ # You can pass in explicit REFs in an array:
25
+ #
26
+ # ref_array = ["da748a59340be8b950e7bbbfb32077eb67d70c3c", "9b19275a592f148e2a53b87ead4ccd8c747539c9"]
27
+ # project = GitSwitchProject.new(path: "tmp/default_ruby", ref_array: ref_array)
28
+ #
29
+ # puts project.commits.map(&:ref) == ref_array # => true
30
+ #
31
+ #
32
+ # It knows the current branch or sha:
33
+ #
34
+ # `cd tmp/ruby && git checkout -b mybranch`
35
+ # project.current_branch_or_sha #=> "mybranch"
36
+ #
37
+ # It can be used for safely wrapping checkouts to ensure the project returns to it's original branch:
38
+ #
39
+ # project.restore_branch_on_return do
40
+ # project.commits.first.checkout!
41
+ # project.current_branch_or_sha # => "da748a593"
42
+ # end
43
+ #
44
+ # project.current_branch_or_sha # => "mybranch"
45
+ class Git::SwitchProject
46
+ attr_reader :commits
47
+
48
+ def initialize(path: , ref_array: [], io: STDOUT, log_dir: "/dev/null")
49
+ @path = Pathname.new(path)
50
+
51
+ @in_git_path = Git::InPath.new(@path.expand_path)
52
+
53
+ raise "Must be a path with a .git directory '#{@path}'" if !@path.join(".git").exist?
54
+ @io = io
55
+ @commits = []
56
+ log_dir = Pathname(log_dir)
57
+
58
+ expand_refs(ref_array).each do |ref|
59
+ restore_branch_on_return(quiet: true) do
60
+ @commits << Git::Commit.new(path: @path, ref: ref, log_dir: log_dir)
61
+ end
62
+ end
63
+
64
+ if (duplicate = @commits.group_by(&:short_sha).detect {|(k, v)| v.length > 1})
65
+ raise "Duplicate SHA resolved #{duplicate[0].inspect}: #{duplicate[1].map {|c| "'#{c.ref}' => '#{c.short_sha}'"}.join(", ") } at #{@path}"
66
+ end
67
+ end
68
+
69
+ def current_branch_or_sha
70
+ branch_or_sha = @in_git_path.branch
71
+ branch_or_sha ||= @in_git_path.short_sha
72
+ branch_or_sha
73
+ end
74
+
75
+ def dirty?
76
+ !clean?
77
+ end
78
+
79
+ # https://stackoverflow.com/a/3879077/147390
80
+ def clean?
81
+ @in_git_path.run("git diff-index --quiet HEAD --") && $?.success?
82
+ end
83
+
84
+ private def status(pattern: "*.gemspec")
85
+ @in_git_path.run("git status #{pattern}")
86
+ end
87
+
88
+ def restore_branch_on_return(quiet: false)
89
+ if dirty? && status.include?("gemspec")
90
+ dirty_gemspec = true
91
+ unless quiet
92
+ @io.puts "Working tree at #{@path} is dirty, stashing. This will be popped on return"
93
+ @io.puts "Bundler modifies gemspec files on git install, this is normal"
94
+ @io.puts "Original status:\n#{status}"
95
+ end
96
+ @in_git_path.run!("git stash")
97
+ end
98
+ branch_or_sha = self.current_branch_or_sha
99
+ yield
100
+ ensure
101
+ return unless branch_or_sha
102
+ @io.puts "Resetting git dir of '#{@path.to_s}' to #{branch_or_sha.inspect}" unless quiet
103
+
104
+ @in_git_path.checkout!(branch_or_sha)
105
+ if dirty_gemspec
106
+ out = @in_git_path.run!("git stash apply 2>&1")
107
+ @io.puts "Applying stash of '#{@path.to_s}':\n#{out}" unless quiet
108
+ end
109
+ end
110
+
111
+ # case ref_array.length
112
+ # when >= 2
113
+ # returns original array
114
+ # when 1
115
+ # returns the given ref plus the one before it
116
+ # when 0
117
+ # returns the most recent 2 refs
118
+ private def expand_refs(ref_array)
119
+ return ref_array if ref_array.length >= 2
120
+
121
+ @in_git_path.checkout!(ref_array.first) if ref_array.first
122
+
123
+ branches_string = @in_git_path.run!("git log --format='%H' -n 2")
124
+ ref_array = branches_string.split($/)
125
+ return ref_array
126
+ end
127
+ end
128
+ end