minitest-heat 0.0.10 → 0.0.14

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6c60c717971a6a1858de4337247b8bfd37f63a63f31a1783de4ecaaad8ca5a8c
-  data.tar.gz: c491cb526bd8a243810fe88da7d15679a6c5182b256a5187db4c457bcc9b7a7a
+  metadata.gz: 4009f20d5e5d4d92d8e0796e853df4da059e3ec7a6b49cae98a8ce71352316fd
+  data.tar.gz: adf5692ab8af5f08ec97607f43ac8f454ad91fb62d15d9f867ee354d920a7a45
 SHA512:
-  metadata.gz: abf3a4c476c1041f75fedfa3eac1235f134bc99bbbf1c909e6e407e461380e9028079e3ad927180d7bec4f039c54247172c9a12bee579dec5eab53b444cf3f68
-  data.tar.gz: fb7d2373153fdaaf6fe8b3a6ec693baf3e04a8fa8dc5636f75d1b0afb45e5a7b25365f3dbe44b0aa3c53bb79b8ed7b0551c3798ceaa3477ce1d9e122159e116a
+  metadata.gz: 8e0d422fbc88d79598464f1640ffb7b6ea3e8ab7324e85f9a059a61f1d14e6c8792f277525dadf6cf81ee493168708dd0eb1a3f9b836b7efd0af3e73b959f25b
+  data.tar.gz: dfa2aa6267e1c026da29b93e1e7ab73f4a1fc65d9d1132228efff315622b742ebe40074eec7a639e6246b7bad27f14a31feee647c9c1274b77d8a57b8d8ed22b

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    minitest-heat (0.0.10)
+    minitest-heat (0.0.14)
       minitest
 
 GEM
@@ -10,7 +10,7 @@ GEM
     ast (2.4.2)
     awesome_print (1.9.2)
     coderay (1.1.3)
-    dead_end (1.1.7)
+    dead_end (3.0.2)
     docile (1.4.0)
     method_source (1.0.0)
     minitest (5.14.4)
@@ -24,7 +24,7 @@ GEM
     rake (12.3.3)
     regexp_parser (2.1.1)
     rexml (3.2.5)
-    rubocop (1.22.1)
+    rubocop (1.22.3)
       parallel (~> 1.10)
       parser (>= 3.0.0.0)
       rainbow (>= 2.2.2, < 4.0)
@@ -33,9 +33,9 @@ GEM
       rubocop-ast (>= 1.12.0, < 2.0)
       ruby-progressbar (~> 1.7)
       unicode-display_width (>= 1.4.0, < 3.0)
-    rubocop-ast (1.12.0)
+    rubocop-ast (1.13.0)
       parser (>= 3.0.1.1)
-    rubocop-minitest (0.15.1)
+    rubocop-minitest (0.15.2)
       rubocop (>= 0.90, < 2.0)
     rubocop-rake (0.6.0)
       rubocop (~> 1.0)
@@ -64,4 +64,4 @@ DEPENDENCIES
   simplecov
 
 BUNDLED WITH
-   2.1.4
+   2.2.30

data/README.md

@@ -1,12 +1,41 @@
 # Minitest::Heat
-**Important:** As of September 13, 2021, Minitest::Heat is an early work-in-progress. It's usable, but it can still ocasionally be buggy as it takes shape.
+Minitest::Heat helps you identify problems faster so you can more efficiently resolve test failures. It does this through a few different methods.
 
-Minitest::Heat aims to surface context around test failures to help you more efficiently identify and prioritize fixing failed tests to help save time.
+It collects failures and inspects backtraces to identify patterns and provide a heat map summary of the files and line numbers that most frequently appear to be the causes of issues.
 
-For some early insight about priorities and how it works, this [Twitter thread](https://twitter.com/garrettdimon/status/1432703746526560266) is currently the best place to start.
+![Example Heat Map Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/map.png)
 
-## Installation
+It suppresses less critical issues like skips or slows when there are legitimate failures. It won't display information about slow tests unless all tests are passing (meaning no errors, failures, or skips)
+
+It presents failures differently depending on the context of failure. For instance, it treats exceptions differently based on whether they arose directly from a test or from source code. It also treats extremely slow tests differently from moderately slow tests.
+
+Markers get some nuance so that slow tests receive different markers than standard passing tests, and exception-triggered failures get different markers for source-code triggered exceptions (E) and test-triggered exceptions ('B' for 'Broken Test').
+
+![Example Markers Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/markers.png)
+
+It also formats the failure details and backtraces to make them more scannable by emphasizing the project-relates lines from the backtrace.
+
+It intelligently recognizes when an exception was raised from a test defintion vs. when an exception is genuinely triggered from the source code in order to help focus on fixing deeper exceptions first.
+
+![Example Exceptions Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/exceptions.png)
+
+Failures are displayed ina  fairly predictable manner but formatted to show the source code from the test so you can see the assertion that failed in addition to the summary of values that didn't satisfy the assertion.
+
+![Example Failures Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/failures.png)
+
+Skipped tests are displayed in a simple manner as well so that it's easy to see the source of the skipped test as well as the reason it was skipped.
+
+![Example Skips Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/skips.png)
+
+Slow tests get slightly more informative labels to indicate that they did pass, but they could use performance improvements. Tests that are particularly slow are called out with a little more emphasis so it's easier to focus on really slow tests first as they frequently represent the most potential for performance gains.
 
+![Example Slows Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/slows.png)
+
+It also always displays the most significant issues at the bottom of the list in order to reduce the need to scroll up through the test failures. As you fix issues, the list becomes shorter, and the less significant issues will make there way to the bottom and be visible without scrolling.
+
+For some additional insight about priorities and how it works, this [Twitter thread](https://twitter.com/garrettdimon/status/1432703746526560266) is currently the best place to start.
+
+## Installation
 Add this line to your application's Gemfile:
 
 ```ruby
@@ -27,27 +56,33 @@ And depending on your usage, you may need to require Minitest Heat in your test
 require 'minitest/heat'
 ```
 
-## Usage
-
-**Important:** In its current state, `Minitest::Heat` replaces any other reporter plugins you may have. Long-term, it should play nicer with other reporters, but during the initial heavy development cycle, it's been easier to have a high confidence that other reporters aren't the source of unexpected behavior.
-
-Otherwise, once it's bundled and added to your `test_helper`, it shold "just work" whenever you run your test suite.
+## Configuration
+Minitest Heat doesn't currently offer a significant set of configuration options, but it will eventually support customizing the thresholds for "Slow" and "Painfully Slow". By default, it considers anything over 1.0s to be 'slow' and anything over 3.0s to be 'painfully slow'.
 
 ## Development
-
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-## Contributing
+### Forcing Test Failures
+In order to easily see how Minitest Heat handles different combinations of different types of failures, the following environment variables can be used to force failures.
+
+```bash
+IMPLODE=true           # Every possible type of failure, skip, and slow is generated
+FORCE_EXCEPTIONS=true  # Only exception-triggered failures
+FORCE_FAILURES=true    # Only standard assertion failures
+FORCE_SKIPS=true       # No errors, just the skipped tests
+FORCE_SLOWS=true       # No errors or skipped tests, just slow tests
+```
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/minitest-heat. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/minitest-heat/blob/master/CODE_OF_CONDUCT.md).
+So to see the full context of a test suite, `IMPLODE=true bundle exec rake` will work its magic.
 
+## Contributing
+Bug reports and pull requests are welcome on GitHub at https://github.com/garrettdimon/minitest-heat. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/minitest-heat/blob/master/CODE_OF_CONDUCT.md).
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
 ## Code of Conduct
-
 Everyone interacting in the Minitest::Heat project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/minitest-heat/blob/master/CODE_OF_CONDUCT.md).

data/lib/minitest/heat/backtrace/line_parser.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+require 'forwardable'
+
+module Minitest
+  module Heat
+    class Backtrace
+      # Represents a line from a backtrace to provide more convenient access to information about
+      #   the relevant file and line number for displaying in test results
+      module LineParser
+        # Parses a line from a backtrace in order to convert it to usable components
+        def self.read(raw_text)
+          raw_pathname, raw_line_number, raw_container = raw_text.split(':')
+          raw_container = raw_container.delete_prefix('in `').delete_suffix("'")
+
+          ::Minitest::Heat::Location.new(
+            pathname: raw_pathname,
+            line_number: raw_line_number,
+            container: raw_container
+          )
+        end
+      end
+    end
+  end
+end

data/lib/minitest/heat/backtrace.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require_relative 'backtrace/line'
+require_relative 'backtrace/line_parser'
 
 module Minitest
   module Heat
@@ -9,7 +9,7 @@ module Minitest
       attr_reader :raw_backtrace
 
       # Creates a more flexible backtrace data structure by parsing the lines of the backtrace to
-      #   extract individual elements for investigating the offending files and line numbers
+      #   extract specific subsets of lines for investigating the offending files and line numbers
       # @param raw_backtrace [Array] the array of lines from the backtrace
       #
       # @return [self]
@@ -24,80 +24,42 @@ module Minitest
         raw_backtrace.empty?
       end
 
-      # The final location exposed in the backtrace. Could be a line from the project or from a
-      #   dependency or the Ruby core libraries
+      # All lines of the backtrace converted to Backtrace::LineParser's
       #
-      # @return [Line] the final location from the backtrace parsed as a Backtrace::Line
-      def final_location
-        parsed_entries.first
-      end
-
-      # The final location from within the project exposed in the backtrace. Could be test files or
-      #   source code files
-      #
-      # @return [Line] the final project location from the backtrace parsed as a Backtrace::Line
-      def final_project_location
-        project_entries.first
-      end
-
-      # The most recently modified location from within the project
-      #
-      # @return [Line] the most recently modified project location from the backtrace parsed as a
-      #   Backtrace::Line
-      def freshest_project_location
-        recently_modified_entries.first
-      end
+      # @return [Array<Location>] the full set of backtrace lines parsed as Location instances
+      def locations
+        return [] if raw_backtrace.nil?
 
-      # The final location from within the project source code (i.e. excluding tests)
-      #
-      # @return [Line] the final source code location from the backtrace parsed as a Backtrace::Line
-      def final_source_code_location
-        source_code_entries.first
+        @locations ||= raw_backtrace.map { |entry| Backtrace::LineParser.read(entry) }
       end
 
-      # The final location from within the project's tests (i.e. excluding source code)
+      # All entries from the backtrace within the project and sorted with the most recently modified
+      #   files at the beginning
       #
-      # @return [Line] the final test location from the backtrace parsed as a Backtrace::Line
-      def final_test_location
-        test_entries.first
+      # @return [Array<Location>] the sorted backtrace lines from the project
+      def recently_modified_locations
+        @recently_modified_locations ||= project_locations.sort_by(&:mtime).reverse
       end
 
       # All entries from the backtrace that are files within the project
       #
-      # @return [Line] the backtrace lines from within the project parsed as Backtrace::Line's
-      def project_entries
-        @project_entries ||= parsed_entries.select { |entry| entry.path.to_s.include?(Dir.pwd) }
-      end
-
-      # All entries from the backtrace within the project and sorted with the most recently modified
-      #   files at the beginning
-      #
-      # @return [Line] the sorted backtrace lines from the project parsed as Backtrace::Line's
-      def recently_modified_entries
-        @recently_modified_entries ||= project_entries.sort_by(&:mtime).reverse
+      # @return [Array<Location>] the backtrace lines from within the project
+      def project_locations
+        @project_locations ||= locations.select(&:project_file?)
       end
 
       # All entries from the backtrace within the project tests
       #
-      # @return [Line] the backtrace lines from within the project tests parsed as Backtrace::Line's
-      def test_entries
-        @test_entries ||= project_entries.select(&:test_file?)
+      # @return [Array<Location>] the backtrace lines from within the tests
+      def test_locations
+        @test_locations ||= project_locations.select(&:test_file?)
       end
 
       # All source code entries from the backtrace (i.e. excluding tests)
       #
-      # @return [Line] the backtrace lines from within the source code parsed as Backtrace::Line's
-      def source_code_entries
-        @source_code_entries ||= project_entries - test_entries
-      end
-
-      # All lines of the backtrace converted to Backtrace::Line's
-      #
-      # @return [Line] the full set of backtrace lines parsed as Backtrace::Line instances
-      def parsed_entries
-        return [] if raw_backtrace.nil?
-
-        @parsed_entries ||= raw_backtrace.map { |entry| Backtrace::Line.parse_backtrace(entry) }
+      # @return [Array<Location>] the backtrace lines from within the source code
+      def source_code_locations
+        @source_code_locations ||= project_locations.select(&:source_code_file?)
       end
     end
   end

data/lib/minitest/heat/hit.rb

@@ -5,8 +5,10 @@ require 'forwardable'
 module Minitest
   module Heat
     # Kind of like an issue, but instead of focusing on a failing test, it covers all issues for a
-    #   given file to build a heat map of the affected files
+    #   given file to build a heat map of the affected files and line numbers
     class Hit
+      Trace = Struct.new(:type, :line_number, :locations)
+
       # So we can sort hot spots by liklihood of being the most important spot to check out before
       #   trying to fix something. These are ranked based on the possibility they represent ripple
       #   effects where fixing one problem could potentially fix multiple other failures.
@@ -23,7 +25,7 @@ module Minitest
         slow: 0
       }.freeze
 
-      attr_reader :pathname, :issues
+      attr_reader :pathname, :issues, :lines
 
       # Creates an instance of a Hit for the given pathname. It must be the full pathname to
       #   uniquely identify the file or we could run into collisions that muddy the water and
@@ -34,16 +36,26 @@ module Minitest
       def initialize(pathname)
         @pathname = Pathname(pathname)
         @issues = {}
+        @lines = {}
       end
 
       # Adds a record of a given issue type for the line number
       # @param type [Symbol] one of Issue::TYPES
       # @param line_number [Integer,String] the line number to record the issue on
+      # @param backtrace: nil [Array<Location>] the project locations from the backtrace
       #
-      # @return [type] [description]
-      def log(type, line_number)
-        @issues[type] ||= []
-        @issues[type] << Integer(line_number)
+      # @return [void]
+      def log(type, line_number, backtrace: [])
+        line_number = Integer(line_number)
+        issue_type = type.to_sym
+
+        # Store issues by issue type with an array of line numbers
+        @issues[issue_type] ||= []
+        @issues[issue_type] << line_number
+
+        # Store issues by line number with an array of Traces
+        @lines[line_number.to_s] ||= []
+        @lines[line_number.to_s] << Trace.new(issue_type, line_number, backtrace)
       end
 
       # Calcuates an approximate weight to serve as a proxy for which files are most likely to be

data/lib/minitest/heat/issue.rb

@@ -18,7 +18,7 @@ module Minitest
       }.freeze
 
       attr_reader :assertions,
-                  :location,
+                  :locations,
                   :message,
                   :test_class,
                   :test_identifier,
@@ -27,7 +27,7 @@ module Minitest
                   :error,
                   :skipped
 
-      def_delegators :@location, :backtrace, :test_definition_line, :test_failure_line
+      def_delegators :@locations, :backtrace, :test_definition_line, :test_failure_line
 
       # Extracts the necessary data from result.
       # @param result [Minitest::Result] the instance of Minitest::Result to examine
@@ -39,7 +39,7 @@ module Minitest
 
         new(
           assertions: result.assertions,
-          location: result.source_location,
+          test_location: result.source_location,
           test_class: result.klass,
           test_identifier: result.name,
           execution_time: result.time,
@@ -47,7 +47,7 @@ module Minitest
           error: result.error?,
           skipped: result.skipped?,
           message: exception&.message,
-          backtrace: exception&.backtrace,
+          backtrace: exception&.backtrace
         )
       end
 
@@ -57,7 +57,7 @@ module Minitest
       # @param assertions: 1 [Integer] the number of assertions in the result
       # @param message: nil [String] exception if there is one
       # @param backtrace: [] [Array<String>] the array of backtrace lines from an exception
-      # @param location: nil [String] the location identifier for a test
+      # @param test_location: nil [Array<String, Integer>] the locations identifier for a test
       # @param test_class: nil [String] the class name for the test result's containing class
       # @param test_identifier: nil [String] the name of the test
       # @param execution_time: nil [Float] the time it took to run the test
@@ -66,11 +66,11 @@ module Minitest
       # @param skipped: false [Boolean] true if the test was skipped
       #
       # @return [type] [description]
-      def initialize(assertions: 1, location: ['unknown', 1], backtrace: [], execution_time: 0.0, message: nil, test_class: nil, test_identifier: nil, passed: false, error: false, skipped: false)
+      def initialize(assertions: 1, test_location: ['Unrecognized Test File', 1], backtrace: [], execution_time: 0.0, message: nil, test_class: nil, test_identifier: nil, passed: false, error: false, skipped: false)
         @message = message
 
         @assertions = Integer(assertions)
-        @location = Location.new(location, backtrace)
+        @locations = Locations.new(test_location, backtrace)
 
         @test_class = test_class
         @test_identifier = test_identifier
@@ -99,9 +99,9 @@ module Minitest
           :skipped
         elsif !passed?
           :failure
-        elsif painful?
+        elsif passed? && painful?
           :painful
-        elsif slow?
+        elsif passed? && slow?
           :slow
         else
           :success
@@ -116,37 +116,55 @@ module Minitest
         !passed? || slow? || painful?
       end
 
+      # The number, in seconds, for a test to be considered "slow"
+      #
+      # @return [Float] number of seconds after which a test is considered slow
+      def slow_threshold
+        # Using a method here so that this can eventually be configurable such that the constant is
+        # only a fallback value if it's not specified anywhere else
+        SLOW_THRESHOLDS[:slow]
+      end
+
+      # The number, in seconds, for a test to be considered "painfully slow"
+      #
+      # @return [Float] number of seconds after which a test is considered painfully slow
+      def painfully_slow_threshold
+        # Using a method here so that this can eventually be configurable such that the constant is
+        # only a fallback value if it's not specified anywhere else
+        SLOW_THRESHOLDS[:painful]
+      end
+
       # Determines if a test should be considered slow by comparing it to the low end definition of
       #   what is considered slow.
       #
-      # @return [Boolean] true if the test took longer to run than `SLOW_THRESHOLDS[:slow]`
+      # @return [Boolean] true if the test took longer to run than `slow_threshold`
       def slow?
-        execution_time >= SLOW_THRESHOLDS[:slow] && execution_time < SLOW_THRESHOLDS[:painful]
+        execution_time >= slow_threshold && execution_time < painfully_slow_threshold
       end
 
       # Determines if a test should be considered painfully slow by comparing it to the high end
       #   definition of what is considered slow.
       #
-      # @return [Boolean] true if the test took longer to run than `SLOW_THRESHOLDS[:painful]`
+      # @return [Boolean] true if the test took longer to run than `painfully_slow_threshold`
       def painful?
-        execution_time >= SLOW_THRESHOLDS[:painful]
+        execution_time >= painfully_slow_threshold
       end
 
       # Determines if the issue is an exception that was raised from directly within a test
       #   definition. In these cases, it's more likely to be a quick fix.
       #
-      # @return [Boolean] true if the final location of the stacktrace was a test file
+      # @return [Boolean] true if the final locations of the stacktrace was a test file
       def in_test?
-        location.broken_test?
+        locations.broken_test?
       end
 
       # Determines if the issue is an exception that was raised from directly within the project
       #   codebase.
       #
-      # @return [Boolean] true if the final location of the stacktrace was a file from the project
+      # @return [Boolean] true if the final locations of the stacktrace was a file from the project
       #   (as opposed to a dependency or Ruby library)
       def in_source?
-        location.proper_failure?
+        locations.proper_failure?
       end
 
       # Was the result a pass? i.e. Skips aren't passes or failures. Slows are still passes. So this