minitest-heat 0.0.9 → 0.0.13

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1b06865dbf64c7b926b600aa635f27e00a0ac65cd08c25cc35ac2ee4de1768a0
-  data.tar.gz: 79ba518f1b7137992f35ac2489ac09c6d5687ae5a8c68e166995a63277e02f99
+  metadata.gz: c523abe811fe169d2a919fe4495cad61c8f282f0d76809d302504f94b6030eb4
+  data.tar.gz: fffa2dcf19662bd6aa2b4ab46243cb797f9de5fa32723428008496d6942ad11e
 SHA512:
-  metadata.gz: 3e5450a68e065e1573a21acba73537e2a003afdc098e7aab6eed2b41638c66a82d74a97faf3d94c9cb79c6279ebda7567dee4bd6a5bec75599e11c170c07e5dc
-  data.tar.gz: 8386cc9f07313cec0686e5efce192a9a7763b78a628ea0eb2b428890a7f5d43e30c472fe4dbcc65497fc2d908d7fc79457a5784890e583951b33dec964259cbf
+  metadata.gz: 7efa01e486a94c5506b6e68b0eb7c200243575909546f871dc2a905e7477d124f85c79d4f0160b9f101fc2d42f01828500565c6fb3434f207d5699b7adabd715
+  data.tar.gz: 114ee328cacda9818932c188368f7d808587374012af973998dcd1ee777f30c6fd878f97b3093e2fadb87010e6db5d7f3a0b739a2c805dd162ff41b51a05f99c

data/.rubocop.yml

@@ -2,6 +2,7 @@ AllCops:
   NewCops: enable
   UseCache: true
   CacheRootDirectory: './'
+  TargetRubyVersion: 2.5.9
   Exclude:
     - 'bin/**/*'
     - 'test/files/source.rb' # An example test file for reading source code

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    minitest-heat (0.0.9)
+    minitest-heat (0.0.13)
       minitest
 
 GEM
@@ -10,7 +10,7 @@ GEM
     ast (2.4.2)
     awesome_print (1.9.2)
     coderay (1.1.3)
-    dead_end (1.1.7)
+    dead_end (3.0.2)
     docile (1.4.0)
     method_source (1.0.0)
     minitest (5.14.4)
@@ -24,7 +24,7 @@ GEM
     rake (12.3.3)
     regexp_parser (2.1.1)
     rexml (3.2.5)
-    rubocop (1.22.1)
+    rubocop (1.22.3)
       parallel (~> 1.10)
       parser (>= 3.0.0.0)
       rainbow (>= 2.2.2, < 4.0)
@@ -33,9 +33,9 @@ GEM
       rubocop-ast (>= 1.12.0, < 2.0)
       ruby-progressbar (~> 1.7)
       unicode-display_width (>= 1.4.0, < 3.0)
-    rubocop-ast (1.12.0)
+    rubocop-ast (1.13.0)
       parser (>= 3.0.1.1)
-    rubocop-minitest (0.15.1)
+    rubocop-minitest (0.15.2)
       rubocop (>= 0.90, < 2.0)
     rubocop-rake (0.6.0)
       rubocop (~> 1.0)
@@ -64,4 +64,4 @@ DEPENDENCIES
   simplecov
 
 BUNDLED WITH
-   2.1.4
+   2.2.30

data/README.md

@@ -1,12 +1,41 @@
 # Minitest::Heat
-**Important:** As of September 13, 2021, Minitest::Heat is an early work-in-progress. It's usable, but it can still ocasionally be buggy as it takes shape.
+Minitest::Heat helps you identify problems faster so you can more efficiently resolve test failures. It does this through a few different methods.
 
-Minitest::Heat aims to surface context around test failures to help you more efficiently identify and prioritize fixing failed tests to help save time.
+It collects failures and inspects backtraces to identify patterns and provide a heat map summary of the files and line numbers that most frequently appear to be the causes of issues.
 
-For some early insight about priorities and how it works, this [Twitter thread](https://twitter.com/garrettdimon/status/1432703746526560266) is currently the best place to start.
+![Example Heat Map Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/map.png)
 
-## Installation
+It suppresses less critical issues like skips or slows when there are legitimate failures. It won't display information about slow tests unless all tests are passing (meaning no errors, failures, or skips)
+
+It presents failures differently depending on the context of failure. For instance, it treats exceptions differently based on whether they arose directly from a test or from source code. It also treats extremely slow tests differently from moderately slow tests.
+
+Markers get some nuance so that slow tests receive different markers than standard passing tests, and exception-triggered failures get different markers for source-code triggered exceptions (E) and test-triggered exceptions ('B' for 'Broken Test').
+
+![Example Markers Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/markers.png)
+
+It also formats the failure details and backtraces to make them more scannable by emphasizing the project-relates lines from the backtrace.
+
+It intelligently recognizes when an exception was raised from a test defintion vs. when an exception is genuinely triggered from the source code in order to help focus on fixing deeper exceptions first.
+
+![Example Exceptions Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/exceptions.png)
+
+Failures are displayed ina  fairly predictable manner but formatted to show the source code from the test so you can see the assertion that failed in addition to the summary of values that didn't satisfy the assertion.
+
+![Example Failures Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/failures.png)
+
+Skipped tests are displayed in a simple manner as well so that it's easy to see the source of the skipped test as well as the reason it was skipped.
+
+![Example Skips Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/skips.png)
+
+Slow tests get slightly more informative labels to indicate that they did pass, but they could use performance improvements. Tests that are particularly slow are called out with a little more emphasis so it's easier to focus on really slow tests first as they frequently represent the most potential for performance gains.
 
+![Example Slows Displayed by Minitest Heat](https://raw.githubusercontent.com/garrettdimon/minitest-heat/main/examples/slows.png)
+
+It also always displays the most significant issues at the bottom of the list in order to reduce the need to scroll up through the test failures. As you fix issues, the list becomes shorter, and the less significant issues will make there way to the bottom and be visible without scrolling.
+
+For some additional insight about priorities and how it works, this [Twitter thread](https://twitter.com/garrettdimon/status/1432703746526560266) is currently the best place to start.
+
+## Installation
 Add this line to your application's Gemfile:
 
 ```ruby
@@ -21,33 +50,39 @@ Or install it yourself as:
 
     $ gem install minitest-heat
 
-And add this line to your `test/test_helper.rb` file:
+And depending on your usage, you may need to require Minitest Heat in your test suite:
 
 ```ruby
 require 'minitest/heat'
 ```
 
-## Usage
-
-**Important:** In its current state, `Minitest::Heat` replaces any other reporter plugins you may have. Long-term, it should play nicer with other reporters, but during the initial heavy development cycle, it's been easier to have a high confidence that other reporters aren't the source of unexpected behavior.
-
-Otherwise, once it's bundled and added to your `test_helper`, it shold "just work" whenever you run your test suite.
+## Configuration
+Minitest Heat doesn't currently offer a significant set of configuration options, but it will eventually support customizing the thresholds for "Slow" and "Painfully Slow". By default, it considers anything over 1.0s to be 'slow' and anything over 3.0s to be 'painfully slow'.
 
 ## Development
-
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-## Contributing
+### Forcing Test Failures
+In order to easily see how Minitest Heat handles different combinations of different types of failures, the following environment variables can be used to force failures.
+
+```bash
+IMPLODE=true           # Every possible type of failure, skip, and slow is generated
+FORCE_EXCEPTIONS=true  # Only exception-triggered failures
+FORCE_FAILURES=true    # Only standard assertion failures
+FORCE_SKIPS=true       # No errors, just the skipped tests
+FORCE_SLOWS=true       # No errors or skipped tests, just slow tests
+```
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/minitest-heat. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/minitest-heat/blob/master/CODE_OF_CONDUCT.md).
+So to see the full context of a test suite, `IMPLODE=true bundle exec rake` will work its magic.
 
+## Contributing
+Bug reports and pull requests are welcome on GitHub at https://github.com/garrettdimon/minitest-heat. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/minitest-heat/blob/master/CODE_OF_CONDUCT.md).
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
 ## Code of Conduct
-
 Everyone interacting in the Minitest::Heat project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/minitest-heat/blob/master/CODE_OF_CONDUCT.md).

data/lib/minitest/heat/backtrace/line_parser.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+require 'forwardable'
+
+module Minitest
+  module Heat
+    class Backtrace
+      # Represents a line from a backtrace to provide more convenient access to information about
+      #   the relevant file and line number for displaying in test results
+      module LineParser
+        # Parses a line from a backtrace in order to convert it to usable components
+        def self.read(raw_text)
+          raw_pathname, raw_line_number, raw_container = raw_text.split(':')
+          raw_container = raw_container.delete_prefix('in `').delete_suffix("'")
+
+          ::Minitest::Heat::Location.new(
+            pathname: raw_pathname,
+            line_number: raw_line_number,
+            container: raw_container
+          )
+        end
+      end
+    end
+  end
+end

data/lib/minitest/heat/backtrace.rb

@@ -1,69 +1,65 @@
 # frozen_string_literal: true
 
+require_relative 'backtrace/line_parser'
+
 module Minitest
   module Heat
     # Wrapper for separating backtrace into component parts
     class Backtrace
       attr_reader :raw_backtrace
 
+      # Creates a more flexible backtrace data structure by parsing the lines of the backtrace to
+      #   extract specific subsets of lines for investigating the offending files and line numbers
+      # @param raw_backtrace [Array] the array of lines from the backtrace
+      #
+      # @return [self]
       def initialize(raw_backtrace)
-        @raw_backtrace = raw_backtrace
+        @raw_backtrace = Array(raw_backtrace)
       end
 
+      # Determines if the raw backtrace has values in it
+      #
+      # @return [Boolean] true if there's no backtrace or it's empty
       def empty?
-        raw_backtrace.nil? || raw_backtrace.empty?
-      end
-
-      def final_location
-        parsed_entries.first
-      end
-
-      def final_project_location
-        project_entries.first
-      end
-
-      def freshest_project_location
-        recently_modified_entries.first
-      end
-
-      def final_source_code_location
-        source_code_entries.first
+        raw_backtrace.empty?
       end
 
-      def final_test_location
-        test_entries.first
-      end
-
-      def project_entries
-        @project_entries ||= parsed_entries.select { |entry| entry.path.to_s.include?(Dir.pwd) }
-      end
-
-      def recently_modified_entries
-        @recently_modified_entries ||= project_entries.sort_by(&:mtime).reverse
-      end
+      # All lines of the backtrace converted to Backtrace::LineParser's
+      #
+      # @return [Array<Location>] the full set of backtrace lines parsed as Location instances
+      def locations
+        return [] if raw_backtrace.nil?
 
-      def test_entries
-        @tests_entries ||= project_entries.select { |entry| test_file?(entry) }
+        @locations ||= raw_backtrace.map { |entry| Backtrace::LineParser.read(entry) }
       end
 
-      def source_code_entries
-        @source_code_entries ||= project_entries - test_entries
+      # All entries from the backtrace within the project and sorted with the most recently modified
+      #   files at the beginning
+      #
+      # @return [Array<Location>] the sorted backtrace lines from the project
+      def recently_modified_locations
+        @recently_modified_locations ||= project_locations.sort_by(&:mtime).reverse
       end
 
-      def parsed_entries
-        return [] if raw_backtrace.nil?
-
-        @parsed_entries ||= raw_backtrace.map { |entry| Line.parse_backtrace(entry) }
+      # All entries from the backtrace that are files within the project
+      #
+      # @return [Array<Location>] the backtrace lines from within the project
+      def project_locations
+        @project_locations ||= locations.select(&:project_file?)
       end
 
-      private
-
-      def parse(entry)
-        Line.parse_backtrace(entry)
+      # All entries from the backtrace within the project tests
+      #
+      # @return [Array<Location>] the backtrace lines from within the tests
+      def test_locations
+        @test_locations ||= project_locations.select(&:test_file?)
       end
 
-      def test_file?(entry)
-        entry.file.to_s.end_with?('_test.rb') || entry.file.to_s.start_with?('test_')
+      # All source code entries from the backtrace (i.e. excluding tests)
+      #
+      # @return [Array<Location>] the backtrace lines from within the source code
+      def source_code_locations
+        @source_code_locations ||= project_locations.select(&:source_code_file?)
       end
     end
   end

data/lib/minitest/heat/hit.rb

@@ -5,8 +5,10 @@ require 'forwardable'
 module Minitest
   module Heat
     # Kind of like an issue, but instead of focusing on a failing test, it covers all issues for a
-    #   given file
+    #   given file to build a heat map of the affected files and line numbers
     class Hit
+      Trace = Struct.new(:type, :line_number, :locations)
+
       # So we can sort hot spots by liklihood of being the most important spot to check out before
       #   trying to fix something. These are ranked based on the possibility they represent ripple
       #   effects where fixing one problem could potentially fix multiple other failures.
@@ -23,34 +25,43 @@ module Minitest
         slow: 0
       }.freeze
 
-      attr_reader :pathname, :issues
+      attr_reader :pathname, :issues, :lines
 
+      # Creates an instance of a Hit for the given pathname. It must be the full pathname to
+      #   uniquely identify the file or we could run into collisions that muddy the water and
+      #   obscure which files had which errors on which line numbers
+      # @param pathname [Pathname,String] the full pathname to the file
+      #
+      # @return [self]
       def initialize(pathname)
         @pathname = Pathname(pathname)
         @issues = {}
+        @lines = {}
       end
 
-      def log(type, line_number)
-        @issues[type] ||= []
-        @issues[type] << line_number
-      end
+      # Adds a record of a given issue type for the line number
+      # @param type [Symbol] one of Issue::TYPES
+      # @param line_number [Integer,String] the line number to record the issue on
+      # @param backtrace: nil [Array<Location>] the project locations from the backtrace
+      #
+      # @return [void]
+      def log(type, line_number, backtrace: [])
+        line_number = Integer(line_number)
+        issue_type = type.to_sym
 
-      def mtime
-        pathname.mtime
-      end
+        # Store issues by issue type with an array of line numbers
+        @issues[issue_type] ||= []
+        @issues[issue_type] << line_number
 
-      def age_in_seconds
-        (Time.now - mtime).to_i
-      end
-
-      def issue_count
-        count = 0
-        Issue::TYPES.each do |issue_type|
-          count += issues.fetch(issue_type) { [] }.size
-        end
-        count
+        # Store issues by line number with an array of Traces
+        @lines[line_number.to_s] ||= []
+        @lines[line_number.to_s] << Trace.new(issue_type, line_number, backtrace)
       end
 
+      # Calcuates an approximate weight to serve as a proxy for which files are most likely to be
+      #   the most problematic across the various issue types
+      #
+      # @return [Integer] the problem weight for the file
       def weight
         weight = 0
         issues.each_pair do |type, values|
@@ -59,6 +70,9 @@ module Minitest
         weight
       end
 
+      # The total issue count for the file across all issue types. Includes duplicates if they exist
+      #
+      # @return [Integer] the sum of the counts for all line numbers for all issue types
       def count
         count = 0
         issues.each_pair do |_type, values|
@@ -67,6 +81,9 @@ module Minitest
         count
       end
 
+      # The full set of unique line numbers across all issue types
+      #
+      # @return [Array<Integer>] the full set of unique offending line numbers for the hit
       def line_numbers
         line_numbers = []
         issues.each_pair do |_type, values|