minitest-heat 0.0.3 → 0.0.7

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: dc3d70999fca25317d7ff7a96b67cd2c6ee6f59970054b0eaec88c051581b855
4
- data.tar.gz: 1197c05fe06b12737b2c73c27d7266e986fcae22cd3957b3e64cd14e2e70295f
3
+ metadata.gz: 7e33245f206aa23bcbad541459db13fb8f6f088df51777850bb1276369dc0b3b
4
+ data.tar.gz: f66fd4f6ba49d1f80be0f97e0d54fd5fa5a2f4fa2063a57f20d9f46520d17ed2
5
5
  SHA512:
6
- metadata.gz: 207f670e0e27dedb872c8251e3890043955c60b6dd6e7dbafce1ba11fd9de5dde20ecf902ac0afd7fa5338d846afa9bc0c2f498df1ca5cdf83b964865792075f
7
- data.tar.gz: c7d3219eb7c7d357a6bdeb82695356f2b56f1bc0f617ed9a9719c3c8227ca9b9bbf0c4929c14d75eed60ce6eade8552f693ad689d6b71c057fc08015a3d38828
6
+ metadata.gz: 24897c2218d77cae17eba80b0e28f24df8f051d13f3f2cddeb47ffabb4730a19b45601f34119749f7fa3d7828fbe617313821d6939cf41653e6559e91bf74526
7
+ data.tar.gz: 38b9e2b060ba864243e384ef260bfb9f771b607bc2786668d462abee84142d5505f3de63182e8238e94bc8ba01932f991125a4db1fcceb4252d764afcc7a62f8
data/.gitignore CHANGED
@@ -6,3 +6,4 @@
6
6
  /pkg/
7
7
  /spec/reports/
8
8
  /tmp/
9
+ /rubocop_cache/
data/.rubocop.yml ADDED
@@ -0,0 +1,23 @@
1
+ AllCops:
2
+ NewCops: enable
3
+ UseCache: true
4
+ CacheRootDirectory: './'
5
+ Exclude:
6
+ - 'bin/**/*'
7
+ - 'test/files/source.rb' # An example test file for reading source code
8
+
9
+ # Let's aim for 80, but we don't need to be nagged if we judiciously go over.
10
+ Layout/LineLength:
11
+ Enabled: false
12
+
13
+ # One case statement in a single method isn't complex.
14
+ Metrics/CyclomaticComplexity:
15
+ IgnoredMethods: ['case']
16
+
17
+ # 10 is a good goal but a little draconian
18
+ Metrics/MethodLength:
19
+ CountAsOne: ['array', 'hash', 'heredoc']
20
+ Max: 15
21
+
22
+ Style/ClassAndModuleChildren:
23
+ Enabled: false
data/Gemfile CHANGED
@@ -1,7 +1,9 @@
1
- source "https://rubygems.org"
1
+ # frozen_string_literal: true
2
+
3
+ source 'https://rubygems.org'
2
4
 
3
5
  # Specify your gem's dependencies in minitest-heat.gemspec
4
6
  gemspec
5
7
 
6
- gem "rake", "~> 12.0"
7
- gem "minitest", "~> 5.0"
8
+ gem 'minitest', '~> 5.0'
9
+ gem 'rake', '~> 12.0'
data/Gemfile.lock CHANGED
@@ -1,37 +1,66 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- minitest-heat (0.0.3)
4
+ minitest-heat (0.0.7)
5
5
  minitest
6
6
 
7
7
  GEM
8
8
  remote: https://rubygems.org/
9
9
  specs:
10
+ ast (2.4.2)
11
+ awesome_print (1.9.2)
10
12
  coderay (1.1.3)
11
13
  dead_end (1.1.7)
12
14
  docile (1.4.0)
13
15
  method_source (1.0.0)
14
16
  minitest (5.14.4)
17
+ parallel (1.21.0)
18
+ parser (3.0.2.0)
19
+ ast (~> 2.4.1)
15
20
  pry (0.14.1)
16
21
  coderay (~> 1.1)
17
22
  method_source (~> 1.0)
23
+ rainbow (3.0.0)
18
24
  rake (12.3.3)
25
+ regexp_parser (2.1.1)
26
+ rexml (3.2.5)
27
+ rubocop (1.22.1)
28
+ parallel (~> 1.10)
29
+ parser (>= 3.0.0.0)
30
+ rainbow (>= 2.2.2, < 4.0)
31
+ regexp_parser (>= 1.8, < 3.0)
32
+ rexml
33
+ rubocop-ast (>= 1.12.0, < 2.0)
34
+ ruby-progressbar (~> 1.7)
35
+ unicode-display_width (>= 1.4.0, < 3.0)
36
+ rubocop-ast (1.12.0)
37
+ parser (>= 3.0.1.1)
38
+ rubocop-minitest (0.15.1)
39
+ rubocop (>= 0.90, < 2.0)
40
+ rubocop-rake (0.6.0)
41
+ rubocop (~> 1.0)
42
+ ruby-progressbar (1.11.0)
19
43
  simplecov (0.21.2)
20
44
  docile (~> 1.1)
21
45
  simplecov-html (~> 0.11)
22
46
  simplecov_json_formatter (~> 0.1)
23
47
  simplecov-html (0.12.3)
24
48
  simplecov_json_formatter (0.1.3)
49
+ unicode-display_width (2.1.0)
25
50
 
26
51
  PLATFORMS
27
52
  ruby
28
53
 
29
54
  DEPENDENCIES
55
+ awesome_print
30
56
  dead_end
31
57
  minitest (~> 5.0)
32
58
  minitest-heat!
33
59
  pry
34
60
  rake (~> 12.0)
61
+ rubocop
62
+ rubocop-minitest
63
+ rubocop-rake
35
64
  simplecov
36
65
 
37
66
  BUNDLED WITH
data/README.md CHANGED
@@ -1,8 +1,9 @@
1
1
  # Minitest::Heat
2
+ **Important:** As of September 13, 2021, Minitest::Heat is an early work-in-progress. It's usable, but it can still ocasionally be buggy as it takes shape.
2
3
 
3
- Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/minitest/heat`. To experiment with that code, run `bin/console` for an interactive prompt.
4
+ Minitest::Heat aims to surface context around test failures to help you more efficiently identify and prioritize fixing failed tests to help save time.
4
5
 
5
- TODO: Delete this and the text above, and describe your gem
6
+ For some early insight about priorities and how it works, this [Twitter thread](https://twitter.com/garrettdimon/status/1432703746526560266) is currently the best place to start.
6
7
 
7
8
  ## Installation
8
9
 
@@ -20,9 +21,17 @@ Or install it yourself as:
20
21
 
21
22
  $ gem install minitest-heat
22
23
 
24
+ And add this line to your `test/test_helper.rb` file:
25
+
26
+ ```ruby
27
+ require 'minitest/heat'
28
+ ```
29
+
23
30
  ## Usage
24
31
 
25
- TODO: Write usage instructions here
32
+ **Important:** In its current state, `Minitest::Heat` replaces any other reporter plugins you may have. Long-term, it should play nicer with other reporters, but during the initial heavy development cycle, it's been easier to have a high confidence that other reporters aren't the source of unexpected behavior.
33
+
34
+ Otherwise, once it's bundled and added to your `test_helper`, it shold "just work" whenever you run your test suite.
26
35
 
27
36
  ## Development
28
37
 
data/Rakefile CHANGED
@@ -1,10 +1,12 @@
1
- require "bundler/gem_tasks"
2
- require "rake/testtask"
1
+ # frozen_string_literal: true
2
+
3
+ require 'bundler/gem_tasks'
4
+ require 'rake/testtask'
3
5
 
4
6
  Rake::TestTask.new(:test) do |t|
5
- t.libs << "test"
6
- t.libs << "lib"
7
- t.test_files = FileList["test/**/*_test.rb"]
7
+ t.libs << 'test'
8
+ t.libs << 'lib'
9
+ t.test_files = FileList['test/**/*_test.rb']
8
10
  end
9
11
 
10
- task :default => :test
12
+ task default: :test
@@ -4,20 +4,6 @@ module Minitest
4
4
  module Heat
5
5
  # Wrapper for separating backtrace into component parts
6
6
  class Backtrace
7
- Line = Struct.new(:path, :file, :number, :container, :mtime, keyword_init: true) do
8
- def to_s
9
- "#{path}/#{file}:#{line} in `#{container}` #{age_in_seconds}"
10
- end
11
-
12
- def to_file
13
- "#{path}/#{file}"
14
- end
15
-
16
- def age_in_seconds
17
- (Time.now - mtime).to_i
18
- end
19
- end
20
-
21
7
  attr_reader :raw_backtrace
22
8
 
23
9
  def initialize(raw_backtrace)
@@ -29,64 +15,55 @@ module Minitest
29
15
  end
30
16
 
31
17
  def final_location
32
- parsed.first
18
+ parsed_entries.first
33
19
  end
34
20
 
35
21
  def final_project_location
36
- project.first
22
+ project_entries.first
37
23
  end
38
24
 
39
- def final_test_location
40
- tests.first
25
+ def freshest_project_location
26
+ recently_modified_entries.first
41
27
  end
42
28
 
43
- def freshest_project_location
44
- recently_modified.first
29
+ def final_source_code_location
30
+ source_code_entries.first
45
31
  end
46
32
 
47
- def project
48
- @project ||= parsed.select { |line| line[:path].include?(Dir.pwd) }
33
+ def final_test_location
34
+ test_entries.first
49
35
  end
50
36
 
51
- def tests
52
- @tests ||= project.select { |line| test_file?(line) }
37
+ def project_entries
38
+ @project_entries ||= parsed_entries.select { |entry| entry.path.to_s.include?(Dir.pwd) }
53
39
  end
54
40
 
55
- def recently_modified
56
- @recently_modified ||= project.sort_by { |line| line[:mtime] }.reverse
41
+ def recently_modified_entries
42
+ @recently_modified_entries ||= project_entries.sort_by(&:mtime).reverse
57
43
  end
58
44
 
59
- def parsed
60
- return [] if raw_backtrace.nil?
45
+ def test_entries
46
+ @tests_entries ||= project_entries.select { |entry| test_file?(entry) }
47
+ end
61
48
 
62
- @parsed ||= raw_backtrace.map { |line| parse(line) }
49
+ def source_code_entries
50
+ @source_code_entries ||= project_entries - test_entries
63
51
  end
64
52
 
65
- private
53
+ def parsed_entries
54
+ return [] if raw_backtrace.nil?
66
55
 
67
- def reduce_container(container)
68
- container.delete_prefix('in `').delete_suffix("'")
56
+ @parsed_entries ||= raw_backtrace.map { |entry| Line.parse_backtrace(entry) }
69
57
  end
70
58
 
71
- def parse(line)
72
- Line.new(line_attributes(line))
73
- end
59
+ private
74
60
 
75
- def line_attributes(line)
76
- parts = line.split(':')
77
- pathname = Pathname.new(parts[0])
78
-
79
- {
80
- path: pathname.dirname.to_s,
81
- file: pathname.basename.to_s,
82
- number: parts[1],
83
- container: reduce_container(parts[2]),
84
- mtime: pathname.exist? ? pathname.mtime : nil
85
- }
61
+ def parse(entry)
62
+ Line.parse_backtrace(entry)
86
63
  end
87
64
 
88
- def test_file?(line)
89
- line[:file].end_with?('_test.rb') || line[:file].start_with?('test_')
65
+ def test_file?(entry)
66
+ entry.file.to_s.end_with?('_test.rb') || entry.file.to_s.start_with?('test_')
90
67
  end
91
68
  end
92
69
  end
@@ -0,0 +1,83 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'forwardable'
4
+
5
+ module Minitest
6
+ module Heat
7
+ # Kind of like an issue, but instead of focusing on a failing test, it covers all issues for a
8
+ # given file
9
+ class Hit
10
+ # So we can sort hot spots by liklihood of being the most important spot to check out before
11
+ # trying to fix something. These are ranked based on the possibility they represent ripple
12
+ # effects where fixing one problem could potentially fix multiple other failures.
13
+ #
14
+ # For example, if there's an exception in the file, start there. Broken code can't run. If a
15
+ # test is broken (i.e. raising an exception), that's a special sort of failure that would be
16
+ # misleading. It doesn't represent a proper failure, but rather a test that doesn't work.
17
+ WEIGHTS = {
18
+ error: 3, # exceptions from source code have the highest likelihood of a ripple effect
19
+ broken: 2, # broken tests won't have ripple effects but can't help if they can't run
20
+ failure: 1, # failures are kind of the whole point, and they could have ripple effects
21
+ skipped: 0, # skips aren't failures, but they shouldn't go ignored
22
+ painful: 0, # slow tests aren't failures, but they shouldn't be ignored
23
+ slow: 0
24
+ }.freeze
25
+
26
+ attr_reader :pathname, :issues
27
+
28
+ def initialize(pathname)
29
+ @pathname = Pathname(pathname)
30
+ @issues = {}
31
+ end
32
+
33
+ def log(type, line_number)
34
+ @issues[type] ||= []
35
+ @issues[type] << line_number
36
+ end
37
+
38
+ def mtime
39
+ pathname.mtime
40
+ end
41
+
42
+ def age_in_seconds
43
+ (Time.now - mtime).to_i
44
+ end
45
+
46
+ def critical_issues?
47
+ issues[:error].any? || issues[:broken].any? || issues[:failure].any?
48
+ end
49
+
50
+ def issue_count
51
+ count = 0
52
+ Issue::TYPES.each do |issue_type|
53
+ count += issues.fetch(issue_type) { [] }.size
54
+ end
55
+ count
56
+ end
57
+
58
+ def weight
59
+ weight = 0
60
+ issues.each_pair do |type, values|
61
+ weight += values.size * WEIGHTS.fetch(type, 0)
62
+ end
63
+ weight
64
+ end
65
+
66
+ def count
67
+ count = 0
68
+ issues.each_pair do |_type, values|
69
+ count += values.size
70
+ end
71
+ count
72
+ end
73
+
74
+ def line_numbers
75
+ line_numbers = []
76
+ issues.each_pair do |_type, values|
77
+ line_numbers += values
78
+ end
79
+ line_numbers.uniq.sort
80
+ end
81
+ end
82
+ end
83
+ end
@@ -8,17 +8,19 @@ module Minitest
8
8
  class Issue
9
9
  extend Forwardable
10
10
 
11
- SLOW_THRESHOLD = 0.5
11
+ TYPES = %i[error broken failure skipped painful slow].freeze
12
12
 
13
- SHARED_SYMBOLS = {
14
- spacer: ' · ',
15
- arrow: ' > '
13
+ # Long-term, these could be configurable so that people can determine their own thresholds of
14
+ # pain for slow tests
15
+ SLOW_THRESHOLDS = {
16
+ slow: 1.0,
17
+ painful: 3.0
16
18
  }.freeze
17
19
 
18
20
  attr_reader :result, :location, :failure
19
21
 
20
22
  def_delegators :@result, :passed?, :error?, :skipped?
21
- def_delegators :@location, :backtrace
23
+ def_delegators :@location, :backtrace, :test_definition_line, :test_failure_line
22
24
 
23
25
  def initialize(result)
24
26
  @result = result
@@ -27,23 +29,36 @@ module Minitest
27
29
  @location = Location.new(result.source_location, @failure&.backtrace)
28
30
  end
29
31
 
32
+ # Returns the primary location of the issue with the present working directory removed from
33
+ # the string for conciseness
34
+ #
35
+ # @return [String] the pathname for the file relative to the present working directory
36
+ def short_location
37
+ location.to_s.delete_prefix("#{Dir.pwd}/")
38
+ end
39
+
40
+ # Converts an issue to the key attributes for recording a 'hit'
41
+ #
42
+ # @return [Array] the filename, failure line, and issue type for categorizing a 'hit' to
43
+ # support generating the heat map
30
44
  def to_hit
31
45
  [
32
- location.source_file,
33
- location.source_failure_line,
46
+ location.project_file.to_s,
47
+ location.project_failure_line,
34
48
  type
35
49
  ]
36
50
  end
37
51
 
38
- def spacer
39
- SHARED_SYMBOLS[:spacer]
40
- end
41
-
42
- def arrow
43
- SHARED_SYMBOLS[:arrow]
44
- end
45
-
46
- def type # rubocop:disable Metrics/MethodLength
52
+ # Classifies different issue types so they can be categorized, organized, and prioritized.
53
+ # Primarily helps add some nuance to issue types. For example, an exception that arises from
54
+ # the project's source code is a genuine exception. But if the exception arose directly from
55
+ # the test, then it's more likely that there's just a simple syntax issue in the test.
56
+ # Similarly, the difference between a moderately slow test and a painfully slow test can be
57
+ # significant. A test that takes half a second is slow, but a test that takes 10 seconds is
58
+ # painfully slow and should get more attention.
59
+ #
60
+ # @return [Symbol] issue type for classifying issues and reporting
61
+ def type
47
62
  if error? && in_test?
48
63
  :broken
49
64
  elsif error?
@@ -52,6 +67,8 @@ module Minitest
52
67
  :skipped
53
68
  elsif !passed?
54
69
  :failure
70
+ elsif painful?
71
+ :painful
55
72
  elsif slow?
56
73
  :slow
57
74
  else
@@ -59,20 +76,49 @@ module Minitest
59
76
  end
60
77
  end
61
78
 
79
+ # Determines if the issue is a proper 'hit' which is anything that doesn't pass or is slow.
80
+ # (Because slow tests still pass and wouldn't otherwise be considered an issue.)
81
+ #
82
+ # @return [Boolean] true if the test did not pass or if it was slow
83
+ def hit?
84
+ !passed? || slow?
85
+ end
86
+
87
+ # Determines if a test should be considered slow by comparing it to the low end definition of
88
+ # what is considered slow.
89
+ #
90
+ # @return [Boolean] true if the test took longer to run than `SLOW_THRESHOLDS[:slow]`
62
91
  def slow?
63
- time > SLOW_THRESHOLD
92
+ time >= SLOW_THRESHOLDS[:slow]
64
93
  end
65
94
 
95
+ # Determines if a test should be considered painfully slow by comparing it to the high end
96
+ # definition of what is considered slow.
97
+ #
98
+ # @return [Boolean] true if the test took longer to run than `SLOW_THRESHOLDS[:painful]`
99
+ def painful?
100
+ time >= SLOW_THRESHOLDS[:painful]
101
+ end
102
+
103
+ # Determines if the issue is an exception that was raised from directly within a test
104
+ # definition. In these cases, it's more likely to be a quick fix.
105
+ #
106
+ # @return [Boolean] true if the final location of the stacktrace was a test file
66
107
  def in_test?
67
- location.failure_in_test?
108
+ location.broken_test?
68
109
  end
69
110
 
111
+ # Determines if the issue is an exception that was raised from directly within the project
112
+ # codebase.
113
+ #
114
+ # @return [Boolean] true if the final location of the stacktrace was a file from the project
115
+ # (as opposed to a dependency or Ruby library)
70
116
  def in_source?
71
- location.failure_in_source?
117
+ location.proper_failure?
72
118
  end
73
119
 
74
120
  def test_class
75
- result.klass.delete_prefix('Minitest::')
121
+ result.klass
76
122
  end
77
123
 
78
124
  def test_identifier
@@ -100,24 +146,13 @@ module Minitest
100
146
  # When the exception came out of the test itself, that's a different kind of exception
101
147
  # that really only indicates there's a problem with the code in the test. It's kind of
102
148
  # between an error and a test.
103
- 'Test Error'
149
+ 'Broken Test'
104
150
  elsif error? || !passed?
105
151
  failure.result_label
152
+ elsif painful?
153
+ 'Passed but Very Slow'
106
154
  elsif slow?
107
155
  'Passed but Slow'
108
- else
109
-
110
- end
111
- end
112
-
113
- def marker
114
- case type
115
- when :broken then 'B'
116
- when :error then 'E'
117
- when :skipped then 'S'
118
- when :failure then 'F'
119
- when :slow then '_'
120
- else '.'
121
156
  end
122
157
  end
123
158
 
@@ -0,0 +1,74 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'forwardable'
4
+
5
+ module Minitest
6
+ module Heat
7
+ # Represents a line of code from the project and provides convenient access to information about
8
+ # the line for displaying in test results
9
+ class Line
10
+ attr_accessor :pathname, :number, :container
11
+ alias line_number number
12
+
13
+ def initialize(pathname:, number:, container: nil)
14
+ @pathname = Pathname(pathname)
15
+ @number = number.to_i
16
+ @container = container.to_s
17
+ end
18
+
19
+ # Convenient interface to read a line from a backtrace convert it to usable components
20
+ def self.parse_backtrace(raw_text)
21
+ raw_pathname, raw_line_number, raw_container = raw_text.split(':')
22
+ raw_container = raw_container.delete_prefix('in `').delete_suffix("'")
23
+
24
+ new(pathname: raw_pathname, number: raw_line_number, container: raw_container)
25
+ end
26
+
27
+ def to_s
28
+ "#{location} in `#{container}`"
29
+ end
30
+
31
+ def path
32
+ pathname.exist? ? pathname.dirname : UNRECOGNIZED
33
+ end
34
+
35
+ def file
36
+ pathname.exist? ? pathname.basename : UNRECOGNIZED
37
+ end
38
+
39
+ def mtime
40
+ pathname.exist? ? pathname.mtime : UNKNOWN_MODIFICATION_TIME
41
+ end
42
+
43
+ def age_in_seconds
44
+ pathname.exist? ? seconds_ago : UNKNOWN_MODIFICATION_SECONDS
45
+ end
46
+
47
+ def location
48
+ "#{pathname}:#{number}"
49
+ end
50
+
51
+ def short_location
52
+ "#{file}:#{number}"
53
+ end
54
+
55
+ def source_code(max_line_count: 1)
56
+ Minitest::Heat::Source.new(
57
+ pathname.to_s,
58
+ line_number: line_number,
59
+ max_line_count: max_line_count
60
+ )
61
+ end
62
+
63
+ private
64
+
65
+ UNRECOGNIZED = '(Unrecognized File)'
66
+ UNKNOWN_MODIFICATION_TIME = Time.at(0)
67
+ UNKNOWN_MODIFICATION_SECONDS = -1
68
+
69
+ def seconds_ago
70
+ (Time.now - mtime).to_i
71
+ end
72
+ end
73
+ end
74
+ end