tldr 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 05c0bd6cc3f9a95769cf36122dc0b15d3b8991fd1e52ebe631e96e27ca0a20d3
-  data.tar.gz: ad535be63a614c6154c8e8d20317b666641e127e244dff4e6c3196bafd65b485
+  metadata.gz: 71122b11466d7a2682ace7dbba2258d2ead4a4aec723882b0c543bcbc3c12f39
+  data.tar.gz: efe76c0ab2ee09ba20921d56922630557ffb11704e3839a8a7322329b8fb5889
 SHA512:
-  metadata.gz: 31300d441a674c0cb5bf997126a4e49206ba78b25d4320047506ec078d2e804733251d12117d4e70b0a02d3bd8dbb116908d6ddab845ff5c3b2bcaca66fed95e
-  data.tar.gz: cf7583cdcb753e2b12178a5db25f001acf0f81088ae35e14df0089ede4e9dc3cbeb95b6d69d0ea1d3add9ce7070c6d7d3e63ba95460765060f428e55068f93cd
+  metadata.gz: 8a3dbacc10c845d976b89775ad762b5e37d2b6c2dbd657dc02422f7d356f98770b16ea485f242c7065ba688d8d69db078801735b41812f1f91a139bcd6d1ac31
+  data.tar.gz: ccc10568c6c506d7656fee1619f678dc32da38298d0ef33c7e2b613e849f9a323dc3105c49aea7ecf108bd705c92fd90dd6cab88fb57e69f0fb4a53e119144ab

data/CHANGELOG.md

@@ -1,5 +1,10 @@
 ## [Unreleased]
 
+## [0.3.0]
+
+* Add `TLDR.run_these_together!` method to allow tests that can't safely be run
+concurrently to be grouped and run together
+
 ## [0.2.1]
 
 * Define a default empty setup/teardown in the base class, to guard against

data/README.md

@@ -225,10 +225,55 @@ You could then run the task with:
 $ TLDR_OPTS="--no-parallel" bundle exec rake tldr
 ```
 
+### Parallel-by-default was a bold choice—also my tests are failing now, thanks
+
+**Read this before you add `--no-parallel` because some tests are failing when
+you run `tldr`.**
+
+The vast majority of test suites in the wild are not parallelized and the vast
+majority of _those_ will only parallelize by forking processes as opposed to
+using a thread pool. We wanted to encourage more people to save time (after all,
+you only get 1.8 seconds here) by making your test suite run as fast as it can,
+so your tests run in parallel by default.
+
+If you're writing new code and tests with TLDR and dutifully running `tldr`
+constantly for fast feedback, odds are that this will help you catch thread
+safety issues early—this is a good thing, because it gives you a chance to fix
+them! But maybe you're porting an existing test suite to TLDR and running in
+parallel for the first time, or maybe you need to test something that simply
+_can't_ be exercised in a thread-safe way. For those cases, TLDR's goal is to
+give you some tools to prevent you from giving up and adding `--no-parallel` to
+your entire test suite and **slowing everything down for the sake of a few
+tests**.
+
+So, when you see a test that is failing when run in parallel with the rest of your
+suite, here is what we recommend doing, in priority order:
+
+1. Figure out a way to redesign the test (or the code under test) to be
+thread-safe.  Modern versions of Ruby provide a number of tools to make this
+easier than it used to be, and it may be as simple as making an instance
+variable thread-local
+2. If the problem is that a subset of your tests depend on the same resource,
+try using [TLDR.run_these_together!](/lib/tldr/run_these_together.rb) class to
+group the tests together. This will ensure that those tests run in the same
+thread in sequence (here's a a [simple
+example](/tests/fixture/run_these_together.rb))
+3. Give up and make the whole suite `--no-parallel`. If you find that you need
+to resort to this, you might save some keystrokes by adding `parallel: false` in
+a [.tldr.yml](#setting-defaults-in-tldryml) file
+
+We have a couple other ideas of ways to incorporate non-thread-safe tests into
+your suite without slowing down the rest of your tests, so stay tuned!
+
 ### How will I run all my tests in CI without the time bomb going off?
 
 TLDR will run all your tests in CI without the time bomb going off.
 
+### What about mocking?
+
+TLDR is laser-focused on running tests. Might we interest you in a refreshing
+[mocktail](https://github.com/testdouble/mocktail), instead?
+
 ## Acknowledgements
 
 Thanks to [George Sheppard](https://github.com/fuzzmonkey) for freeing up the

data/lib/tldr/parallelizer.rb

@@ -0,0 +1,36 @@
+class TLDR
+  class Parallelizer
+    def initialize
+      @strategizer = Strategizer.new
+    end
+
+    def parallelize tests, parallel, &blk
+      return tests.map(&blk) if tests.size < 2 || !parallel
+      tldr_pool = Concurrent::ThreadPoolExecutor.new(
+        name: "tldr",
+        auto_terminate: true
+      )
+
+      strategy = @strategizer.strategize tests, GROUPED_TESTS
+
+      strategy.tests_and_groups.map { |test_or_group|
+        tests_to_run = if test_or_group.group?
+          test_or_group.tests.select { |test| tests.include? test }
+        else
+          [test_or_group]
+        end
+
+        unless tests_to_run.empty?
+          Concurrent::Promises.future_on(tldr_pool) {
+            tests_to_run.map(&blk)
+          }
+        end
+      }.compact.flat_map(&:value)
+    end
+
+    private
+
+    def substitute_tests_grouped_by_run_these_together! tests
+    end
+  end
+end

data/lib/tldr/planner.rb

@@ -50,8 +50,7 @@ class TLDR
     def gather_tests
       gather_descendants(TLDR).flat_map { |subklass|
         subklass.instance_methods.grep(/^test_/).sort.map { |method|
-          file, line = SorbetCompatibility.unwrap_method(subklass.instance_method(method)).source_location
-          Test.new subklass, method, file, line
+          Test.new subklass, method
         }
       }
     end

data/lib/tldr/run_these_together.rb

@@ -0,0 +1,23 @@
+class TLDR
+  GROUPED_TESTS = Concurrent::Array.new
+
+  # If it's not safe to run a set of tests in parallel, you can force them to
+  # run in a group together (in a single worker) with `run_these_together!` in
+  # your test.
+  #
+  # This method takes an array of tuples, where the first element is the class
+  # (or its name as a string, if the class is not yet defined in the current
+  # file) and the second element is the method name. If the second element is
+  # nil, then all the tests on the class will be run together.
+  #
+  # Examples:
+  #   - `run_these_together!` will run all the tests defined on the current
+  #     class to be run in a group
+  #   - `run_these_together!([[ClassA, nil], ["ClassB", :test_1], [ClassB, :test_2]])`
+  #    will run all the tests defined on ClassA, and test_1 and test_2 from ClassB
+  #   (nil in the second position means "all the tests on this class")
+  #
+  def self.run_these_together! klass_method_tuples = [[self, nil]]
+    GROUPED_TESTS << TestGroup.new(klass_method_tuples)
+  end
+end

data/lib/tldr/runner.rb

@@ -1,9 +1,9 @@
 require "irb"
-require "concurrent"
 
 class TLDR
   class Runner
     def initialize
+      @parallelizer = Parallelizer.new
       @wip = Concurrent::Array.new
       @results = Concurrent::Array.new
       @run_aborted = Concurrent::AtomicBoolean.new false
@@ -34,7 +34,7 @@ class TLDR
         end
       }
 
-      results = parallelize(plan.tests, config.parallel) { |test|
+      results = @parallelizer.parallelize(plan.tests, config.parallel) { |test|
         e = nil
         start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC, :microsecond)
         wip_test = WIPTest.new test, start_time
@@ -65,20 +65,6 @@ class TLDR
 
     private
 
-    def parallelize tests, parallel, &blk
-      return tests.map(&blk) if tests.size < 2 || !parallel
-      tldr_pool = Concurrent::ThreadPoolExecutor.new(
-        name: "tldr",
-        auto_terminate: true
-      )
-
-      tests.map { |test|
-        Concurrent::Promises.future_on(tldr_pool) {
-          blk.call test
-        }
-      }.flat_map(&:value)
-    end
-
     def fail_fast reporter, plan, fast_failed_result
       unless @run_aborted.true?
         @run_aborted.make_true

data/lib/tldr/strategizer.rb

@@ -0,0 +1,31 @@
+class TLDR
+  class Strategizer
+    Strategy = Struct.new :tests, :tests_and_groups
+
+    # Combine all discovered test methods with any methods grouped by run_these_together!
+    #
+    # Priorities:
+    #   - Map over tests to build out groups in order to retain shuffle order
+    #     (group will run in position of first test in the group)
+    #   - If a test is in multiple groups, only run it once
+    def strategize tests, grouped_tests
+      already_included_groups = []
+
+      Strategy.new tests, tests.map { |test|
+        if (group = grouped_tests.find { |group| group.tests.include? test })
+          if already_included_groups.include? group
+            next
+          elsif (other = already_included_groups.find { |other| (group.tests & other.tests).any? })
+            other.tests |= group.tests
+            next
+          else
+            already_included_groups << group
+            group
+          end
+        else
+          test
+        end
+      }.compact
+    end
+  end
+end

data/lib/tldr/value/config.rb

@@ -1,5 +1,3 @@
-require "concurrent"
-
 class TLDR
   CONFLAGS = {
     seed: "--seed",

data/lib/tldr/value/test.rb

@@ -1,10 +1,11 @@
 class TLDR
-  Test = Struct.new :klass, :method, :file, :line do
-    attr_reader :location
+  Test = Struct.new :klass, :method do
+    attr_reader :file, :line, :location
 
     def initialize(*args)
       super
-      @location = Location.new(file, line)
+      @file, @line = SorbetCompatibility.unwrap_method(klass.instance_method(method)).source_location
+      @location = Location.new file, line
     end
 
     # Memoizing at call time, because re-parsing isn't free and isn't usually necessary
@@ -19,5 +20,9 @@ class TLDR
     def covers_line? l
       line == l || (l >= line && l <= end_line)
     end
+
+    def group?
+      false
+    end
   end
 end

data/lib/tldr/value/test_group.rb

@@ -0,0 +1,22 @@
+class TLDR
+  TestGroup = Struct.new :configuration do
+    attr_writer :tests
+
+    def tests
+      @tests ||= configuration.flat_map { |(klass, method)|
+        klass = Kernel.const_get(klass) if klass.is_a? String
+        if method.nil?
+          klass.instance_methods.grep(/^test_/).sort.map { |method|
+            Test.new klass, method
+          }
+        else
+          Test.new klass, method
+        end
+      }
+    end
+
+    def group?
+      true
+    end
+  end
+end

data/lib/tldr/value.rb

@@ -4,3 +4,4 @@ require "tldr/value/test"
 require "tldr/value/wip_test"
 require "tldr/value/test_result"
 require "tldr/value/location"
+require "tldr/value/test_group"

data/lib/tldr/version.rb

@@ -1,3 +1,3 @@
 class TLDR
-  VERSION = "0.2.1"
+  VERSION = "0.3.0"
 end

data/lib/tldr.rb

@@ -1,14 +1,19 @@
-require_relative "tldr/version"
+require "concurrent-ruby"
+
+require_relative "tldr/argv_parser"
+require_relative "tldr/assertions"
 require_relative "tldr/backtrace_filter"
 require_relative "tldr/error"
-require_relative "tldr/value"
-require_relative "tldr/reporters"
-require_relative "tldr/argv_parser"
+require_relative "tldr/parallelizer"
 require_relative "tldr/planner"
+require_relative "tldr/reporters"
+require_relative "tldr/run_these_together"
 require_relative "tldr/runner"
-require_relative "tldr/assertions"
 require_relative "tldr/skippable"
 require_relative "tldr/sorbet_compatibility"
+require_relative "tldr/strategizer"
+require_relative "tldr/value"
+require_relative "tldr/version"
 
 class TLDR
   include Assertions

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: tldr
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - Justin Searls
@@ -60,20 +60,24 @@ files:
 - lib/tldr/assertions/minitest_compatibility.rb
 - lib/tldr/backtrace_filter.rb
 - lib/tldr/error.rb
+- lib/tldr/parallelizer.rb
 - lib/tldr/planner.rb
 - lib/tldr/rake.rb
 - lib/tldr/reporters.rb
 - lib/tldr/reporters/base.rb
 - lib/tldr/reporters/default.rb
 - lib/tldr/reporters/icon_provider.rb
+- lib/tldr/run_these_together.rb
 - lib/tldr/runner.rb
 - lib/tldr/skippable.rb
 - lib/tldr/sorbet_compatibility.rb
+- lib/tldr/strategizer.rb
 - lib/tldr/value.rb
 - lib/tldr/value/config.rb
 - lib/tldr/value/location.rb
 - lib/tldr/value/plan.rb
 - lib/tldr/value/test.rb
+- lib/tldr/value/test_group.rb
 - lib/tldr/value/test_result.rb
 - lib/tldr/value/wip_test.rb
 - lib/tldr/version.rb