tldr 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 71122b11466d7a2682ace7dbba2258d2ead4a4aec723882b0c543bcbc3c12f39
-  data.tar.gz: efe76c0ab2ee09ba20921d56922630557ffb11704e3839a8a7322329b8fb5889
+  metadata.gz: 91190a0094836b21df8aaeccb57684848ada68707a4c52241ebd74437fb514c4
+  data.tar.gz: a6cc8f1573bdb317f6fd5f337a9670bbeef95f735bdc1003774f4bca0a7b294b
 SHA512:
-  metadata.gz: 8a3dbacc10c845d976b89775ad762b5e37d2b6c2dbd657dc02422f7d356f98770b16ea485f242c7065ba688d8d69db078801735b41812f1f91a139bcd6d1ac31
-  data.tar.gz: ccc10568c6c506d7656fee1619f678dc32da38298d0ef33c7e2b613e849f9a323dc3105c49aea7ecf108bd705c92fd90dd6cab88fb57e69f0fb4a53e119144ab
+  metadata.gz: 93b9fdf0d61dcaba8c3c6dff464700ca260bbdef433eaf799e33130baf5ef63d0e7dec1c6dc3558e5d0f1bc580b53fbacc31c466da1f4a9803533470bb1b5ec5
+  data.tar.gz: f2a8de04ed3e9b84c277db21f111177849138c1b1b692a43d96492b0707c0e9a8c255c26a9991d899ed27312a04ebe13887a6db7d4417a4878b4e2cfc542acc4

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 ## [Unreleased]
 
+## [0.4.0]
+
+* Add `TLDR.dont_run_these_in_parallel!` method to allow tests to indicate that they
+must be run in isolation and not concurrently with any other tests
+* Add support for `around` hooks (similar to [minitest-around](https://github.com/splattael/minitest-around))
+
 ## [0.3.0]
 
 * Add `TLDR.run_these_together!` method to allow tests that can't safely be run

data/README.md

@@ -1,6 +1,8 @@
 # TLDR - for people who don't have time for slow tests
 
-**tl;dr, this is a very nice test runner for Ruby that fails after 1.8 seconds**
+Okay, you might need to sit down for this:
+
+**tl;dr, TLDR is a Ruby test framework that stops running your tests after 1.8 seconds.**
 
 We initially meant this as a joke [while
 pairin'](https://www.youtube.com/live/bmi-SWeH4MA?si=p5g1j1FQZrbYEOCg&t=63), but
@@ -9,16 +11,28 @@ in addition to being funny, it was also a pretty good idea. So we fleshed out
 [Minitest-compatible](#minitest-compatibility), and downright pleasant test
 framework for Ruby.
 
+The "big idea" here is TLDR is designed for users to run the `tldr` command
+repeatedly as they work—as opposed to only running the tests for whatever is
+being worked on. Even if the suite run over the 1.8 second time limit. Because
+TLDR shuffles and runs in parallel and is guaranteed to take less than two
+seconds,
+**you'll actually wind up running _all_ of your tests quite often as you work**,
+catching any problems much earlier than if you had waited until the end of the
+day to push your work and let a continuous integration server run the full
+suite.
+
 Some stuff you might like:
 
 * A CLI that can run tests by line number(s) (e.g. `foo.rb:5 bar.rb:3:10`) and
 by names or patterns (e.g. `--name test_fail,test_error --name "/_\d/"`)
-* Everything is **parallel by default**, and seems pretty fast (you can disable with `--no-parallel`)
-* Surprisingly delightful color diff output when two things fail to equal one another, care of [@mcmire's super_diff gem](https://github.com/mcmire/super_diff)
+* Everything is **parallel by default**, and seems pretty darn fast; TLDR
+also provides [several escape hatches to sequester tests that aren't thread-safe](#parallel-by-default-is-nice-in-theory-but-half-my-tests-are-failing-wat)
+* Surprisingly delightful color diff output when two things fail to equal one
+another, care of [@mcmire's super_diff gem](https://github.com/mcmire/super_diff)
 * By default, the CLI will prepend your most-recently-edited test file to the
-front of your suite so its tests will run first. The tests you're working on are
-the most likely you care about running, so TLDR runs them first (see the
-`--prepend` option)
+front of your suite so its tests will run first. The test you worked on most recently
+is the one you most likely want to ensure runs, so TLDR runs it first (see the
+`--prepend` option for how to control this behavior)
 * And, of course, our signature feature: your test suite will never grow into
 a glacially slow, soul-sucking albatross around your neck, because **after 1.8
 seconds, it stops running your tests**, with a report on what it _was_ able to
@@ -50,7 +64,7 @@ end
 ```
 
 A TLDR subclass defines its tests with instance methods that begin with
-`_test`. They can define `setup` and/or `teardown` methods which will run before
+`test_`. They can define `setup` and/or `teardown` methods which will run before
 and after each test, respectively.
 
 If you place your tests in `test/**/*_test.rb` (and/or `test/**/test_*.rb`)
@@ -82,68 +96,9 @@ flags:
 $ tldr --name FooTest#test_foo -n test_bar,test_baz -n /_qux/
 ```
 
-(The above will translate to this array of name fiilters internally:
+(The above will translate to this array of name filters internally:
  `["FooTest#test_foo", "test_bar", "test_baz", "/_qux/"]`.)
 
-### Running tests without the CLI
-
-If you'd rather use TLDR by running Ruby files instead of the `tldr` CLI
-(similar to `require "minitest/autorun"`), here's how to do it!
-
-Given a file `test/some_test.rb`:
-
-```ruby
-require "tldr"
-TLDR::Run.at_exit! TLDR::Config.new(no_emoji: true)
-
-class SomeTest < TLDR
-  def test_truth
-    assert true
-  end
-end
-```
-
-You could run the test with:
-
-```
-$ ruby test/some_test.rb
-```
-
-To maximize control and to avoid running code accidentally (and _unlike_ the
-`tldr` CLI), running `at_exit!` will not set default values to the `paths`,
-`helper`, `load_paths`, and `prepend_tests` config properties. You'll have to
-pass any values you want to set on a [Config object](/lib/tldr/value/config.rb)
-and pass it to `at_exit!`.
-
-To avoid running multiple suites accidentally, if `TLDR::Run.at_exit!` is
-encountered multiple times, only the first hook will be registered. If the
-`tldr` CLI is running and encounters a call to `at_exit!`, it will be ignored.
-
-#### Setting up the load path
-
-When running TLDR from a Ruby script, one thing the framework can't help you with
-is setting up load paths for you.
-
-If you want to require code in `test/` or `lib/` without using
-`require_relative`, you'll need to add those directories to the load path. You
-can do this programmatically by prepending the path to `$LOAD_PATH`, like
-this:
-
-```ruby
-$LOAD_PATH.unshift "test"
-
-require "tldr"
-TLDR::Run.at_exit! TLDR::Config.new(no_emoji: true)
-
-require "helper"
-```
-
-Or by using Ruby's `-I` flag to include it:
-
-```
-$ ruby -Itest test/some_test.rb
-```
-
 ### Options
 
 Here is the full list of CLI options:
@@ -192,7 +147,8 @@ well as some [internal tests](/tests/dotfile_test.rb) demonstrating its behavior
 Tests you write with tldr are designed to be mostly-compatible with
 [Minitest](https://github.com/minitest/minitest) tests. Some notes:
 
-* `setup` and `teardown` hook methods should work as you expect
+* `setup` and `teardown` hook methods should work as you expect. (We even threw
+in [an `around` hook](https://github.com/splattael/minitest-around) as a bonus!)
 * All of Minitest's assertions (e.g. `assert`, `assert_equals`) are provided,
 with these caveats:
   * To retain the `expected, actual` argument ordering, `tldr` defines
@@ -201,7 +157,7 @@ with these caveats:
   * If you want to maximize compatibility and mix in `assert_includes` and the
   deprecated `assert_send`, just `include
   TLDR::Assertions::MinitestCompatibility` into the `TLDR` base class or
-  individual test classes
+  individual test classesJust set it
 
 ### Running TLDR with Rake
 
@@ -225,7 +181,71 @@ You could then run the task with:
 $ TLDR_OPTS="--no-parallel" bundle exec rake tldr
 ```
 
-### Parallel-by-default was a bold choice—also my tests are failing now, thanks
+### Running tests without the CLI
+
+If you'd rather use TLDR by running Ruby files instead of the `tldr` CLI
+(similar to `require "minitest/autorun"`), here's how to do it!
+
+Given a file `test/some_test.rb`:
+
+```ruby
+require "tldr"
+TLDR::Run.at_exit! TLDR::Config.new(no_emoji: true)
+
+class SomeTest < TLDR
+  def test_truth
+    assert true
+  end
+end
+```
+
+You could run the test with:
+
+```
+$ ruby test/some_test.rb
+```
+
+To maximize control and to avoid running code accidentally (and _unlike_ the
+`tldr` CLI), running `at_exit!` will not set default values to the `paths`,
+`helper`, `load_paths`, and `prepend_tests` config properties. You'll have to
+pass any values you want to set on a [Config object](/lib/tldr/value/config.rb)
+and pass it to `at_exit!`.
+
+To avoid running multiple suites accidentally, if `TLDR::Run.at_exit!` is
+encountered multiple times, only the first hook will be registered. If the
+`tldr` CLI is running and encounters a call to `at_exit!`, it will be ignored.
+
+#### Setting up the load path
+
+When running TLDR from a Ruby script, one thing the framework can't help you with
+is setting up load paths for you.
+
+If you want to require code in `test/` or `lib/` without using
+`require_relative`, you'll need to add those directories to the load path. You
+can do this programmatically by prepending the path to `$LOAD_PATH`, like
+this:
+
+```ruby
+$LOAD_PATH.unshift "test"
+
+require "tldr"
+TLDR::Run.at_exit! TLDR::Config.new(no_emoji: true)
+
+require "helper"
+```
+
+Or by using Ruby's `-I` flag to include it:
+
+```
+$ ruby -Itest test/some_test.rb
+```
+
+## Questions you might be asking
+
+TLDR is very similar to Minitest in API, but different in enough ways that you
+probably have some questions.
+
+### Parallel-by-default is nice in theory but half my tests are failing. Wat?
 
 **Read this before you add `--no-parallel` because some tests are failing when
 you run `tldr`.**
@@ -234,17 +254,17 @@ The vast majority of test suites in the wild are not parallelized and the vast
 majority of _those_ will only parallelize by forking processes as opposed to
 using a thread pool. We wanted to encourage more people to save time (after all,
 you only get 1.8 seconds here) by making your test suite run as fast as it can,
-so your tests run in parallel by default.
+so your tests run in parallel threads by default.
 
 If you're writing new code and tests with TLDR and dutifully running `tldr`
 constantly for fast feedback, odds are that this will help you catch thread
-safety issues early—this is a good thing, because it gives you a chance to fix
-them! But maybe you're porting an existing test suite to TLDR and running in
-parallel for the first time, or maybe you need to test something that simply
-_can't_ be exercised in a thread-safe way. For those cases, TLDR's goal is to
-give you some tools to prevent you from giving up and adding `--no-parallel` to
-your entire test suite and **slowing everything down for the sake of a few
-tests**.
+safety issues early—this is a good thing, because it gives you a chance to
+address them before they're too hard to fix! But maybe you're porting an
+existing test suite to TLDR and running in parallel for the first time, or maybe
+you need to test something that simply _can't_ be exercised in a thread-safe
+way. For those cases, TLDR's goal is to give you some tools to prevent you from
+giving up and adding `--no-parallel` to your entire test suite and **slowing
+everything down for the sake of a few tests**.
 
 So, when you see a test that is failing when run in parallel with the rest of your
 suite, here is what we recommend doing, in priority order:
@@ -254,11 +274,17 @@ thread-safe.  Modern versions of Ruby provide a number of tools to make this
 easier than it used to be, and it may be as simple as making an instance
 variable thread-local
 2. If the problem is that a subset of your tests depend on the same resource,
-try using [TLDR.run_these_together!](/lib/tldr/run_these_together.rb) class to
+try using [TLDR.run_these_together!](lib/tldr/parallel_controls.rb) class to
 group the tests together. This will ensure that those tests run in the same
-thread in sequence (here's a a [simple
+thread in sequence (here's a [simple
 example](/tests/fixture/run_these_together.rb))
-3. Give up and make the whole suite `--no-parallel`. If you find that you need
+3. For tests that affect process-wide resources like setting the system clock or
+changing the process's working directory (i.e. `Dir.chdir`), you can sequester
+them to run sequentially _after_ all parallel tests in your suite have run with
+[TLDR.dont_run_these_in_parallel!](lib/tldr/parallel_controls.rb), which takes
+the same arguments as `run_these_together!`
+([example](/tests/fixture/dont_run_these_in_parallel.rb))
+4. Give up and make the whole suite `--no-parallel`. If you find that you need
 to resort to this, you might save some keystrokes by adding `parallel: false` in
 a [.tldr.yml](#setting-defaults-in-tldryml) file
 
@@ -267,11 +293,23 @@ your suite without slowing down the rest of your tests, so stay tuned!
 
 ### How will I run all my tests in CI without the time bomb going off?
 
-TLDR will run all your tests in CI without the time bomb going off.
+TLDR will run all your tests in CI without the time bomb going off. If
+`tldr` is run in a non-interactive shell and a `CI` environment variable is set
+(as it is on virtually every CI service), then the bomb will be defused.
+
+### Is there a plugin system?
+
+There is not.
+
+Currently, the only pluggable aspect of TLDR are reporters, which can be set
+with the `--reporter` command line option. It can be set to any fully-qualified
+class name that extends from
+[TLDR::Reporters::Base](/lib/tldr/reporters/base.rb).
 
 ### What about mocking?
 
-TLDR is laser-focused on running tests. Might we interest you in a refreshing
+TLDR is laser-focused on running tests, so it doesn't provide a built-in mocking
+facility. Might we interest you in a refreshing
 [mocktail](https://github.com/testdouble/mocktail), instead?
 
 ## Acknowledgements

data/lib/tldr/parallel_controls.rb

@@ -0,0 +1,47 @@
+class TLDR
+  # If it's not safe to run a set of tests in parallel, you can force them to
+  # run in a group together (in a single worker) with `run_these_together!` in
+  # your test.
+  #
+  # This method takes an array of tuples, where the first element is the class
+  # (or its name as a string, if the class is not yet defined in the current
+  # file) and the second element is the method name. If the second element is
+  # nil, then all the tests on the class will be run together.
+  #
+  # Examples:
+  #   - `run_these_together!` will run all the tests defined on the current
+  #     class to be run in a group
+  #   - `run_these_together!([[ClassA, nil], ["ClassB", :test_1], [ClassB, :test_2]])`
+  #    will run all the tests defined on ClassA, and test_1 and test_2 from ClassB
+  #
+  GROUPED_TESTS = Concurrent::Array.new
+  def self.run_these_together! klass_method_tuples = [[self, nil]]
+    GROUPED_TESTS << TestGroup.new(klass_method_tuples)
+  end
+
+  # This is a similar API to run_these_together! but its effect is more drastic
+  # Rather than running the provided (class, method) tuples in a group within a
+  # thread as part of a parallel run, it will reserve all tests specified by
+  # all calls to `dont_run_these_in_parallel!` to be run after all parallel tests have
+  # finished.
+  #
+  # This has an important implication! If your test suite is over TLDR's time
+  # limit, it means that these tests will never be run outside of CI unless you
+  # run them manually.
+  #
+  # Like `run_these_together!`, `dont_run_these_in_parallel!` takes an array of
+  # tuples, where the first element is the class (or its fully-qualified name as
+  # a string) and the second element is `nil` (matching all the class's test
+  # methods) or else one of the methods on the class.
+  #
+  # Examples:
+  #   - `dont_run_these_in_parallel!` will run all the tests defined on the current
+  #     class after all parallel tests have finished
+  #   - `dont_run_these_in_parallel!([[ClassA, nil], ["ClassB", :test_1], [ClassB, :test_2]])`
+  #    will run all the tests defined on ClassA, and test_1 and test_2 from ClassB
+  #
+  THREAD_UNSAFE_TESTS = Concurrent::Array.new
+  def self.dont_run_these_in_parallel! klass_method_tuples = [[self, nil]]
+    THREAD_UNSAFE_TESTS << TestGroup.new(klass_method_tuples)
+  end
+end

data/lib/tldr/parallelizer.rb

@@ -2,35 +2,41 @@ class TLDR
   class Parallelizer
     def initialize
       @strategizer = Strategizer.new
-    end
-
-    def parallelize tests, parallel, &blk
-      return tests.map(&blk) if tests.size < 2 || !parallel
-      tldr_pool = Concurrent::ThreadPoolExecutor.new(
+      @thread_pool = Concurrent::ThreadPoolExecutor.new(
         name: "tldr",
         auto_terminate: true
       )
+    end
+
+    def parallelize all_tests, parallel, &blk
+      return run_in_sequence(all_tests, &blk) if all_tests.size < 2 || !parallel
+
+      strategy = @strategizer.strategize all_tests, GROUPED_TESTS, THREAD_UNSAFE_TESTS
+
+      run_in_parallel(strategy.parallel_tests_and_groups, &blk) +
+        run_in_sequence(strategy.thread_unsafe_tests, &blk)
+    end
 
-      strategy = @strategizer.strategize tests, GROUPED_TESTS
+    private
+
+    def run_in_sequence tests, &blk
+      tests.map(&blk)
+    end
 
-      strategy.tests_and_groups.map { |test_or_group|
+    def run_in_parallel tests_and_groups, &blk
+      tests_and_groups.map { |test_or_group|
         tests_to_run = if test_or_group.group?
-          test_or_group.tests.select { |test| tests.include? test }
+          test_or_group.tests
         else
           [test_or_group]
         end
 
         unless tests_to_run.empty?
-          Concurrent::Promises.future_on(tldr_pool) {
+          Concurrent::Promises.future_on(@thread_pool) {
             tests_to_run.map(&blk)
           }
         end
       }.compact.flat_map(&:value)
     end
-
-    private
-
-    def substitute_tests_grouped_by_run_these_together! tests
-    end
   end
 end

data/lib/tldr/reporters/default.rb

@@ -17,12 +17,17 @@ class TLDR
       end
 
       def after_test result
-        @out.print case result.type
+        output = case result.type
         when :success then @icons.success
         when :skip then @icons.skip
         when :failure then @icons.failure
         when :error then @icons.error
         end
+        if @config.verbose
+          @out.puts "#{output} #{result.type.capitalize} - #{describe(result.test, result.relevant_location)}"
+        else
+          @out.print output
+        end
       end
 
       def time_diff start, stop = Process.clock_gettime(Process::CLOCK_MONOTONIC, :microsecond)

data/lib/tldr/runner.rb

@@ -42,7 +42,16 @@ class TLDR
         runtime = time_it(start_time) do
           instance = test.klass.new
           instance.setup if instance.respond_to? :setup
-          instance.send(test.method)
+          if instance.respond_to? :around
+            did_run = false
+            instance.around {
+              did_run = true
+              instance.send(test.method)
+            }
+            raise Error, "#{test.klass}#around failed to yield or call the passed test block" unless did_run
+          else
+            instance.send(test.method)
+          end
           instance.teardown if instance.respond_to? :teardown
         rescue Skip, Failure, StandardError => e
         end

data/lib/tldr/strategizer.rb

@@ -1,6 +1,6 @@
 class TLDR
   class Strategizer
-    Strategy = Struct.new :tests, :tests_and_groups
+    Strategy = Struct.new :parallel_tests_and_groups, :thread_unsafe_tests
 
     # Combine all discovered test methods with any methods grouped by run_these_together!
     #
@@ -8,10 +8,13 @@ class TLDR
     #   - Map over tests to build out groups in order to retain shuffle order
     #     (group will run in position of first test in the group)
     #   - If a test is in multiple groups, only run it once
-    def strategize tests, grouped_tests
+    def strategize all_tests, run_these_together_groups, thread_unsafe_test_groups
+      thread_unsafe_tests, thread_safe_tests = partition_unsafe(all_tests, thread_unsafe_test_groups)
+
+      grouped_tests = prepare_run_together_groups run_these_together_groups, thread_safe_tests, thread_unsafe_tests
       already_included_groups = []
 
-      Strategy.new tests, tests.map { |test|
+      Strategy.new thread_safe_tests.map { |test|
         if (group = grouped_tests.find { |group| group.tests.include? test })
           if already_included_groups.include? group
             next
@@ -25,7 +28,27 @@ class TLDR
         else
           test
         end
-      }.compact
+      }.compact, thread_unsafe_tests
+    end
+
+    private
+
+    def partition_unsafe tests, thread_unsafe_test_groups
+      tests.partition { |test|
+        thread_unsafe_test_groups.any? { |group| group.tests.include? test }
+      }
+    end
+
+    def prepare_run_together_groups run_these_together_groups, thread_safe_tests, thread_unsafe_tests
+      grouped_tests = run_these_together_groups.map(&:dup)
+
+      grouped_tests.each do |group|
+        group.tests = group.tests.select { |test|
+          thread_safe_tests.include?(test) && !thread_unsafe_tests.include?(test)
+        }
+      end
+
+      grouped_tests.reject { |group| group.tests.size < 2 }
     end
   end
 end

data/lib/tldr/version.rb

@@ -1,3 +1,3 @@
 class TLDR
-  VERSION = "0.3.0"
+  VERSION = "0.4.0"
 end

data/lib/tldr.rb

@@ -4,10 +4,10 @@ require_relative "tldr/argv_parser"
 require_relative "tldr/assertions"
 require_relative "tldr/backtrace_filter"
 require_relative "tldr/error"
+require_relative "tldr/parallel_controls"
 require_relative "tldr/parallelizer"
 require_relative "tldr/planner"
 require_relative "tldr/reporters"
-require_relative "tldr/run_these_together"
 require_relative "tldr/runner"
 require_relative "tldr/skippable"
 require_relative "tldr/sorbet_compatibility"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: tldr
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Justin Searls
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-09-26 00:00:00.000000000 Z
+date: 2023-09-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: super_diff
@@ -60,6 +60,7 @@ files:
 - lib/tldr/assertions/minitest_compatibility.rb
 - lib/tldr/backtrace_filter.rb
 - lib/tldr/error.rb
+- lib/tldr/parallel_controls.rb
 - lib/tldr/parallelizer.rb
 - lib/tldr/planner.rb
 - lib/tldr/rake.rb
@@ -67,7 +68,6 @@ files:
 - lib/tldr/reporters/base.rb
 - lib/tldr/reporters/default.rb
 - lib/tldr/reporters/icon_provider.rb
-- lib/tldr/run_these_together.rb
 - lib/tldr/runner.rb
 - lib/tldr/skippable.rb
 - lib/tldr/sorbet_compatibility.rb
@@ -105,7 +105,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.6
+rubygems_version: 3.4.17
 signing_key:
 specification_version: 4
 summary: TLDR will run your tests, but only for 1.8 seconds.

data/lib/tldr/run_these_together.rb

@@ -1,23 +0,0 @@
-class TLDR
-  GROUPED_TESTS = Concurrent::Array.new
-
-  # If it's not safe to run a set of tests in parallel, you can force them to
-  # run in a group together (in a single worker) with `run_these_together!` in
-  # your test.
-  #
-  # This method takes an array of tuples, where the first element is the class
-  # (or its name as a string, if the class is not yet defined in the current
-  # file) and the second element is the method name. If the second element is
-  # nil, then all the tests on the class will be run together.
-  #
-  # Examples:
-  #   - `run_these_together!` will run all the tests defined on the current
-  #     class to be run in a group
-  #   - `run_these_together!([[ClassA, nil], ["ClassB", :test_1], [ClassB, :test_2]])`
-  #    will run all the tests defined on ClassA, and test_1 and test_2 from ClassB
-  #   (nil in the second position means "all the tests on this class")
-  #
-  def self.run_these_together! klass_method_tuples = [[self, nil]]
-    GROUPED_TESTS << TestGroup.new(klass_method_tuples)
-  end
-end