minitest-heat 0.0.5 → 0.0.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c760c15a45811d5773dee18e32aadc063ed61c3eb04447c637719d5d3d46178a
4
- data.tar.gz: 90c345c56cbd2589719eef8a0264a5973f77a8104c91149f4e4ad2320067a49a
3
+ metadata.gz: 1b06865dbf64c7b926b600aa635f27e00a0ac65cd08c25cc35ac2ee4de1768a0
4
+ data.tar.gz: 79ba518f1b7137992f35ac2489ac09c6d5687ae5a8c68e166995a63277e02f99
5
5
  SHA512:
6
- metadata.gz: 47be296804a2171023bfcdb5c4192376698349bd76c4c3a2991d9accd9f631b88d2ffb867d821f2479c41bd6fd2cf919af325ecde975f98ee4592277b3b09432
7
- data.tar.gz: 79a180260e9a1d8c55d898883f13969f2576ceb6450dc9e080050e55674c73fc7979202a2c8a78542958491fce91fee2a3cb1b66fd2705edc4a8d432f5e28fcc
6
+ metadata.gz: 3e5450a68e065e1573a21acba73537e2a003afdc098e7aab6eed2b41638c66a82d74a97faf3d94c9cb79c6279ebda7567dee4bd6a5bec75599e11c170c07e5dc
7
+ data.tar.gz: 8386cc9f07313cec0686e5efce192a9a7763b78a628ea0eb2b428890a7f5d43e30c472fe4dbcc65497fc2d908d7fc79457a5784890e583951b33dec964259cbf
data/.gitignore CHANGED
@@ -6,3 +6,4 @@
6
6
  /pkg/
7
7
  /spec/reports/
8
8
  /tmp/
9
+ /rubocop_cache/
data/.rubocop.yml ADDED
@@ -0,0 +1,23 @@
1
+ AllCops:
2
+ NewCops: enable
3
+ UseCache: true
4
+ CacheRootDirectory: './'
5
+ Exclude:
6
+ - 'bin/**/*'
7
+ - 'test/files/source.rb' # An example test file for reading source code
8
+
9
+ # Let's aim for 80, but we don't need to be nagged if we judiciously go over.
10
+ Layout/LineLength:
11
+ Enabled: false
12
+
13
+ # One case statement in a single method isn't complex.
14
+ Metrics/CyclomaticComplexity:
15
+ IgnoredMethods: ['case']
16
+
17
+ # 10 is a good goal but a little draconian
18
+ Metrics/MethodLength:
19
+ CountAsOne: ['array', 'hash', 'heredoc']
20
+ Max: 15
21
+
22
+ Style/ClassAndModuleChildren:
23
+ Enabled: false
data/Gemfile CHANGED
@@ -1,7 +1,9 @@
1
- source "https://rubygems.org"
1
+ # frozen_string_literal: true
2
+
3
+ source 'https://rubygems.org'
2
4
 
3
5
  # Specify your gem's dependencies in minitest-heat.gemspec
4
6
  gemspec
5
7
 
6
- gem "rake", "~> 12.0"
7
- gem "minitest", "~> 5.0"
8
+ gem 'minitest', '~> 5.0'
9
+ gem 'rake', '~> 12.0'
data/Gemfile.lock CHANGED
@@ -1,37 +1,66 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- minitest-heat (0.0.5)
4
+ minitest-heat (0.0.9)
5
5
  minitest
6
6
 
7
7
  GEM
8
8
  remote: https://rubygems.org/
9
9
  specs:
10
+ ast (2.4.2)
11
+ awesome_print (1.9.2)
10
12
  coderay (1.1.3)
11
13
  dead_end (1.1.7)
12
14
  docile (1.4.0)
13
15
  method_source (1.0.0)
14
16
  minitest (5.14.4)
17
+ parallel (1.21.0)
18
+ parser (3.0.2.0)
19
+ ast (~> 2.4.1)
15
20
  pry (0.14.1)
16
21
  coderay (~> 1.1)
17
22
  method_source (~> 1.0)
23
+ rainbow (3.0.0)
18
24
  rake (12.3.3)
25
+ regexp_parser (2.1.1)
26
+ rexml (3.2.5)
27
+ rubocop (1.22.1)
28
+ parallel (~> 1.10)
29
+ parser (>= 3.0.0.0)
30
+ rainbow (>= 2.2.2, < 4.0)
31
+ regexp_parser (>= 1.8, < 3.0)
32
+ rexml
33
+ rubocop-ast (>= 1.12.0, < 2.0)
34
+ ruby-progressbar (~> 1.7)
35
+ unicode-display_width (>= 1.4.0, < 3.0)
36
+ rubocop-ast (1.12.0)
37
+ parser (>= 3.0.1.1)
38
+ rubocop-minitest (0.15.1)
39
+ rubocop (>= 0.90, < 2.0)
40
+ rubocop-rake (0.6.0)
41
+ rubocop (~> 1.0)
42
+ ruby-progressbar (1.11.0)
19
43
  simplecov (0.21.2)
20
44
  docile (~> 1.1)
21
45
  simplecov-html (~> 0.11)
22
46
  simplecov_json_formatter (~> 0.1)
23
47
  simplecov-html (0.12.3)
24
48
  simplecov_json_formatter (0.1.3)
49
+ unicode-display_width (2.1.0)
25
50
 
26
51
  PLATFORMS
27
52
  ruby
28
53
 
29
54
  DEPENDENCIES
55
+ awesome_print
30
56
  dead_end
31
57
  minitest (~> 5.0)
32
58
  minitest-heat!
33
59
  pry
34
60
  rake (~> 12.0)
61
+ rubocop
62
+ rubocop-minitest
63
+ rubocop-rake
35
64
  simplecov
36
65
 
37
66
  BUNDLED WITH
data/README.md CHANGED
@@ -1,8 +1,9 @@
1
1
  # Minitest::Heat
2
+ **Important:** As of September 13, 2021, Minitest::Heat is an early work-in-progress. It's usable, but it can still ocasionally be buggy as it takes shape.
2
3
 
3
- Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/minitest/heat`. To experiment with that code, run `bin/console` for an interactive prompt.
4
+ Minitest::Heat aims to surface context around test failures to help you more efficiently identify and prioritize fixing failed tests to help save time.
4
5
 
5
- TODO: Delete this and the text above, and describe your gem
6
+ For some early insight about priorities and how it works, this [Twitter thread](https://twitter.com/garrettdimon/status/1432703746526560266) is currently the best place to start.
6
7
 
7
8
  ## Installation
8
9
 
@@ -20,9 +21,17 @@ Or install it yourself as:
20
21
 
21
22
  $ gem install minitest-heat
22
23
 
24
+ And add this line to your `test/test_helper.rb` file:
25
+
26
+ ```ruby
27
+ require 'minitest/heat'
28
+ ```
29
+
23
30
  ## Usage
24
31
 
25
- TODO: Write usage instructions here
32
+ **Important:** In its current state, `Minitest::Heat` replaces any other reporter plugins you may have. Long-term, it should play nicer with other reporters, but during the initial heavy development cycle, it's been easier to have a high confidence that other reporters aren't the source of unexpected behavior.
33
+
34
+ Otherwise, once it's bundled and added to your `test_helper`, it shold "just work" whenever you run your test suite.
26
35
 
27
36
  ## Development
28
37
 
data/Rakefile CHANGED
@@ -1,10 +1,12 @@
1
- require "bundler/gem_tasks"
2
- require "rake/testtask"
1
+ # frozen_string_literal: true
2
+
3
+ require 'bundler/gem_tasks'
4
+ require 'rake/testtask'
3
5
 
4
6
  Rake::TestTask.new(:test) do |t|
5
- t.libs << "test"
6
- t.libs << "lib"
7
- t.test_files = FileList["test/**/*_test.rb"]
7
+ t.libs << 'test'
8
+ t.libs << 'lib'
9
+ t.test_files = FileList['test/**/*_test.rb']
8
10
  end
9
11
 
10
- task :default => :test
12
+ task default: :test
@@ -4,42 +4,6 @@ module Minitest
4
4
  module Heat
5
5
  # Wrapper for separating backtrace into component parts
6
6
  class Backtrace
7
- Line = Struct.new(:path, :file, :number, :container, :mtime, keyword_init: true) do
8
- def to_s
9
- "#{location} in `#{container}`"
10
- end
11
-
12
- def pathname
13
- Pathname("#{path}/#{file}")
14
- end
15
-
16
- def location
17
- "#{pathname.to_s}:#{number}"
18
- end
19
-
20
- def short_pathname
21
- pathname.delete_prefix(Dir.pwd)
22
- end
23
-
24
- def short_location
25
- "#{pathname.basename.to_s}:#{number}"
26
- end
27
-
28
- def age_in_seconds
29
- (Time.now - mtime).to_i
30
- end
31
- end
32
-
33
- UNREADABLE_FILE_ATTRIBUTES = {
34
- path: '(Unknown Path)',
35
- file: '(Unknown File)',
36
- number: '(Unknown Line Number)',
37
- container: '(Unknown Method)',
38
- mtime: '(Unknown Modification Time)'
39
- }
40
-
41
- UNREADABLE_LINE = Line.new(UNREADABLE_FILE_ATTRIBUTES)
42
-
43
7
  attr_reader :raw_backtrace
44
8
 
45
9
  def initialize(raw_backtrace)
@@ -51,74 +15,55 @@ module Minitest
51
15
  end
52
16
 
53
17
  def final_location
54
- parsed_lines.first
18
+ parsed_entries.first
55
19
  end
56
20
 
57
21
  def final_project_location
58
- project_lines.first
22
+ project_entries.first
59
23
  end
60
24
 
61
25
  def freshest_project_location
62
- recently_modified_lines.first
26
+ recently_modified_entries.first
63
27
  end
64
28
 
65
29
  def final_source_code_location
66
- source_code_lines.first
30
+ source_code_entries.first
67
31
  end
68
32
 
69
33
  def final_test_location
70
- test_lines.first
34
+ test_entries.first
71
35
  end
72
36
 
73
- def project_lines
74
- @project_lines ||= parsed_lines.select { |line| line[:path].include?(Dir.pwd) }
37
+ def project_entries
38
+ @project_entries ||= parsed_entries.select { |entry| entry.path.to_s.include?(Dir.pwd) }
75
39
  end
76
40
 
77
- def recently_modified_lines
78
- @recently_modified_lines ||= project_lines.sort_by { |line| line[:mtime] }.reverse
41
+ def recently_modified_entries
42
+ @recently_modified_entries ||= project_entries.sort_by(&:mtime).reverse
79
43
  end
80
44
 
81
- def test_lines
82
- @tests_lines ||= project_lines.select { |line| test_file?(line) }
45
+ def test_entries
46
+ @tests_entries ||= project_entries.select { |entry| test_file?(entry) }
83
47
  end
84
48
 
85
- def source_code_lines
86
- @source_code_lines ||= project_lines - test_lines
49
+ def source_code_entries
50
+ @source_code_entries ||= project_entries - test_entries
87
51
  end
88
52
 
89
- def parsed_lines
53
+ def parsed_entries
90
54
  return [] if raw_backtrace.nil?
91
55
 
92
- @parsed_lines ||= raw_backtrace.map { |line| parse(line) }
56
+ @parsed_entries ||= raw_backtrace.map { |entry| Line.parse_backtrace(entry) }
93
57
  end
94
58
 
95
59
  private
96
60
 
97
- def reduce_container(container)
98
- container.delete_prefix('in `').delete_suffix("'")
99
- end
100
-
101
- def parse(line)
102
- Line.new(line_attributes(line))
103
- end
104
-
105
- def line_attributes(line)
106
- parts = line.split(':')
107
- pathname = Pathname.new(parts[0])
108
-
109
- {
110
- path: pathname.dirname.to_s,
111
- file: pathname.basename.to_s,
112
- number: parts[1],
113
- container: reduce_container(parts[2]),
114
- mtime: pathname.exist? ? pathname.mtime : nil
115
- }
116
- rescue
117
- UNREADABLE_FILE_ATTRIBUTES
61
+ def parse(entry)
62
+ Line.parse_backtrace(entry)
118
63
  end
119
64
 
120
- def test_file?(line)
121
- line[:file].end_with?('_test.rb') || line[:file].start_with?('test_')
65
+ def test_file?(entry)
66
+ entry.file.to_s.end_with?('_test.rb') || entry.file.to_s.start_with?('test_')
122
67
  end
123
68
  end
124
69
  end
@@ -0,0 +1,79 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'forwardable'
4
+
5
+ module Minitest
6
+ module Heat
7
+ # Kind of like an issue, but instead of focusing on a failing test, it covers all issues for a
8
+ # given file
9
+ class Hit
10
+ # So we can sort hot spots by liklihood of being the most important spot to check out before
11
+ # trying to fix something. These are ranked based on the possibility they represent ripple
12
+ # effects where fixing one problem could potentially fix multiple other failures.
13
+ #
14
+ # For example, if there's an exception in the file, start there. Broken code can't run. If a
15
+ # test is broken (i.e. raising an exception), that's a special sort of failure that would be
16
+ # misleading. It doesn't represent a proper failure, but rather a test that doesn't work.
17
+ WEIGHTS = {
18
+ error: 5, # exceptions from source code have the highest likelihood of a ripple effect
19
+ broken: 4, # broken tests won't have ripple effects but can't help if they can't run
20
+ failure: 3, # failures are kind of the whole point, and they could have ripple effects
21
+ skipped: 2, # skips aren't failures, but they shouldn't go ignored
22
+ painful: 1, # slow tests aren't failures, but they shouldn't be ignored
23
+ slow: 0
24
+ }.freeze
25
+
26
+ attr_reader :pathname, :issues
27
+
28
+ def initialize(pathname)
29
+ @pathname = Pathname(pathname)
30
+ @issues = {}
31
+ end
32
+
33
+ def log(type, line_number)
34
+ @issues[type] ||= []
35
+ @issues[type] << line_number
36
+ end
37
+
38
+ def mtime
39
+ pathname.mtime
40
+ end
41
+
42
+ def age_in_seconds
43
+ (Time.now - mtime).to_i
44
+ end
45
+
46
+ def issue_count
47
+ count = 0
48
+ Issue::TYPES.each do |issue_type|
49
+ count += issues.fetch(issue_type) { [] }.size
50
+ end
51
+ count
52
+ end
53
+
54
+ def weight
55
+ weight = 0
56
+ issues.each_pair do |type, values|
57
+ weight += values.size * WEIGHTS.fetch(type, 0)
58
+ end
59
+ weight
60
+ end
61
+
62
+ def count
63
+ count = 0
64
+ issues.each_pair do |_type, values|
65
+ count += values.size
66
+ end
67
+ count
68
+ end
69
+
70
+ def line_numbers
71
+ line_numbers = []
72
+ issues.each_pair do |_type, values|
73
+ line_numbers += values
74
+ end
75
+ line_numbers.uniq.sort
76
+ end
77
+ end
78
+ end
79
+ end
@@ -8,30 +8,19 @@ module Minitest
8
8
  class Issue
9
9
  extend Forwardable
10
10
 
11
+ TYPES = %i[error broken failure skipped painful slow].freeze
12
+
13
+ # Long-term, these could be configurable so that people can determine their own thresholds of
14
+ # pain for slow tests
11
15
  SLOW_THRESHOLDS = {
12
16
  slow: 1.0,
13
17
  painful: 3.0
14
- }
15
-
16
- MARKERS = {
17
- success: '·',
18
- slow: '–',
19
- painful: '—',
20
- broken: 'B',
21
- error: 'E',
22
- skipped: 'S',
23
- failure: 'F',
24
- }
25
-
26
- SHARED_SYMBOLS = {
27
- spacer: ' · ',
28
- arrow: ' > '
29
18
  }.freeze
30
19
 
31
20
  attr_reader :result, :location, :failure
32
21
 
33
22
  def_delegators :@result, :passed?, :error?, :skipped?
34
- def_delegators :@location, :backtrace
23
+ def_delegators :@location, :backtrace, :test_definition_line, :test_failure_line
35
24
 
36
25
  def initialize(result)
37
26
  @result = result
@@ -40,27 +29,36 @@ module Minitest
40
29
  @location = Location.new(result.source_location, @failure&.backtrace)
41
30
  end
42
31
 
32
+ # Returns the primary location of the issue with the present working directory removed from
33
+ # the string for conciseness
34
+ #
35
+ # @return [String] the pathname for the file relative to the present working directory
43
36
  def short_location
44
- location.to_s.delete_prefix(Dir.pwd)
37
+ location.to_s.delete_prefix("#{Dir.pwd}/")
45
38
  end
46
39
 
40
+ # Converts an issue to the key attributes for recording a 'hit'
41
+ #
42
+ # @return [Array] the filename, failure line, and issue type for categorizing a 'hit' to
43
+ # support generating the heat map
47
44
  def to_hit
48
45
  [
49
46
  location.project_file.to_s,
50
- location.project_failure_line,
47
+ Integer(location.project_failure_line),
51
48
  type
52
49
  ]
53
50
  end
54
51
 
55
- def spacer
56
- SHARED_SYMBOLS[:spacer]
57
- end
58
-
59
- def arrow
60
- SHARED_SYMBOLS[:arrow]
61
- end
62
-
63
- def type # rubocop:disable Metrics/MethodLength
52
+ # Classifies different issue types so they can be categorized, organized, and prioritized.
53
+ # Primarily helps add some nuance to issue types. For example, an exception that arises from
54
+ # the project's source code is a genuine exception. But if the exception arose directly from
55
+ # the test, then it's more likely that there's just a simple syntax issue in the test.
56
+ # Similarly, the difference between a moderately slow test and a painfully slow test can be
57
+ # significant. A test that takes half a second is slow, but a test that takes 10 seconds is
58
+ # painfully slow and should get more attention.
59
+ #
60
+ # @return [Symbol] issue type for classifying issues and reporting
61
+ def type
64
62
  if error? && in_test?
65
63
  :broken
66
64
  elsif error?
@@ -78,22 +76,43 @@ module Minitest
78
76
  end
79
77
  end
80
78
 
79
+ # Determines if the issue is a proper 'hit' which is anything that doesn't pass or is slow.
80
+ # (Because slow tests still pass and wouldn't otherwise be considered an issue.)
81
+ #
82
+ # @return [Boolean] true if the test did not pass or if it was slow
81
83
  def hit?
82
84
  !passed? || slow?
83
85
  end
84
86
 
87
+ # Determines if a test should be considered slow by comparing it to the low end definition of
88
+ # what is considered slow.
89
+ #
90
+ # @return [Boolean] true if the test took longer to run than `SLOW_THRESHOLDS[:slow]`
85
91
  def slow?
86
92
  time >= SLOW_THRESHOLDS[:slow]
87
93
  end
88
94
 
95
+ # Determines if a test should be considered painfully slow by comparing it to the high end
96
+ # definition of what is considered slow.
97
+ #
98
+ # @return [Boolean] true if the test took longer to run than `SLOW_THRESHOLDS[:painful]`
89
99
  def painful?
90
100
  time >= SLOW_THRESHOLDS[:painful]
91
101
  end
92
102
 
103
+ # Determines if the issue is an exception that was raised from directly within a test
104
+ # definition. In these cases, it's more likely to be a quick fix.
105
+ #
106
+ # @return [Boolean] true if the final location of the stacktrace was a test file
93
107
  def in_test?
94
108
  location.broken_test?
95
109
  end
96
110
 
111
+ # Determines if the issue is an exception that was raised from directly within the project
112
+ # codebase.
113
+ #
114
+ # @return [Boolean] true if the final location of the stacktrace was a file from the project
115
+ # (as opposed to a dependency or Ruby library)
97
116
  def in_source?
98
117
  location.proper_failure?
99
118
  end
@@ -127,20 +146,16 @@ module Minitest
127
146
  # When the exception came out of the test itself, that's a different kind of exception
128
147
  # that really only indicates there's a problem with the code in the test. It's kind of
129
148
  # between an error and a test.
130
- 'Test Error'
149
+ 'Broken Test'
131
150
  elsif error? || !passed?
132
151
  failure.result_label
152
+ elsif painful?
153
+ 'Passed but Very Slow'
133
154
  elsif slow?
134
155
  'Passed but Slow'
135
- else
136
-
137
156
  end
138
157
  end
139
158
 
140
- def marker
141
- MARKERS.fetch(type.to_sym)
142
- end
143
-
144
159
  def summary
145
160
  error? ? exception_parts[0] : exception.message
146
161
  end
@@ -0,0 +1,74 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'forwardable'
4
+
5
+ module Minitest
6
+ module Heat
7
+ # Represents a line of code from the project and provides convenient access to information about
8
+ # the line for displaying in test results
9
+ class Line
10
+ attr_accessor :pathname, :number, :container
11
+ alias line_number number
12
+
13
+ def initialize(pathname:, number:, container: nil)
14
+ @pathname = Pathname(pathname)
15
+ @number = number.to_i
16
+ @container = container.to_s
17
+ end
18
+
19
+ # Convenient interface to read a line from a backtrace convert it to usable components
20
+ def self.parse_backtrace(raw_text)
21
+ raw_pathname, raw_line_number, raw_container = raw_text.split(':')
22
+ raw_container = raw_container.delete_prefix('in `').delete_suffix("'")
23
+
24
+ new(pathname: raw_pathname, number: raw_line_number, container: raw_container)
25
+ end
26
+
27
+ def to_s
28
+ "#{location} in `#{container}`"
29
+ end
30
+
31
+ def path
32
+ pathname.exist? ? pathname.dirname : UNRECOGNIZED
33
+ end
34
+
35
+ def file
36
+ pathname.exist? ? pathname.basename : UNRECOGNIZED
37
+ end
38
+
39
+ def mtime
40
+ pathname.exist? ? pathname.mtime : UNKNOWN_MODIFICATION_TIME
41
+ end
42
+
43
+ def age_in_seconds
44
+ pathname.exist? ? seconds_ago : UNKNOWN_MODIFICATION_SECONDS
45
+ end
46
+
47
+ def location
48
+ "#{pathname}:#{number}"
49
+ end
50
+
51
+ def short_location
52
+ "#{file}:#{number}"
53
+ end
54
+
55
+ def source_code(max_line_count: 1)
56
+ Minitest::Heat::Source.new(
57
+ pathname.to_s,
58
+ line_number: line_number,
59
+ max_line_count: max_line_count
60
+ )
61
+ end
62
+
63
+ private
64
+
65
+ UNRECOGNIZED = '(Unrecognized File)'
66
+ UNKNOWN_MODIFICATION_TIME = Time.at(0)
67
+ UNKNOWN_MODIFICATION_SECONDS = -1
68
+
69
+ def seconds_ago
70
+ (Time.now - mtime).to_i
71
+ end
72
+ end
73
+ end
74
+ end