tldr 0.10.1 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +26 -0
- data/README.md +237 -247
- data/exe/tldr +2 -0
- data/exe/tldt +2 -0
- data/lib/tldr/argv_parser.rb +81 -25
- data/lib/tldr/argv_reconstructor.rb +107 -0
- data/lib/tldr/assertions.rb +2 -3
- data/lib/tldr/autorun.rb +2 -0
- data/lib/tldr/minitest_compatibility.rb +35 -0
- data/lib/tldr/reporters/default.rb +11 -11
- data/lib/tldr/reporters/icon_provider.rb +9 -0
- data/lib/tldr/runner.rb +37 -12
- data/lib/tldr/value/config.rb +60 -134
- data/lib/tldr/value/test.rb +1 -1
- data/lib/tldr/version.rb +1 -1
- data/lib/tldr/yaml_parser.rb +29 -0
- data/lib/tldr.rb +3 -0
- data/script/test +3 -3
- metadata +25 -4
- data/lib/tldr/assertions/minitest_compatibility.rb +0 -55
data/README.md
CHANGED
@@ -1,114 +1,51 @@
|
|
1
|
-
# TLDR - for
|
1
|
+
# TLDR - Testing for rubyists who want fast feedback
|
2
2
|
|
3
|
-
|
3
|
+
TLDR is a suspiciously-delightful testing framework for Ruby.
|
4
4
|
|
5
|
-
|
5
|
+
As a test library, TLDR is largely [API-compatible with Minitest](#minitest-compatibility). As a test runner, TLDR boasts a few features RSpec's CLI still doesn't have.
|
6
6
|
|
7
|
-
|
8
|
-
pairin'](https://www.youtube.com/live/bmi-SWeH4MA?si=p5g1j1FQZrbYEOCg&t=63), but
|
9
|
-
in addition to being funny, it was also a pretty good idea. So we fleshed out
|
10
|
-
`tldr` to be a full-featured, mostly
|
11
|
-
[Minitest-compatible](#minitest-compatibility), and dare-we-say pleasant test
|
12
|
-
framework for Ruby.
|
7
|
+
The library, command line interface, and every decision in-between was prioritized to maximize productivity to promote fast feedback loops. Some highlights:
|
13
8
|
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
day to push your work and let a continuous integration server run the full
|
22
|
-
suite.
|
9
|
+
* Numerous ways to run specific tests: path (`foo_test.rb`), line number (`foo_test.rb:13`), name (`--name test_foo`), or regex pattern (`-n /_foo/`)
|
10
|
+
* Parallel test execution by default, as well as [controls for serial execution of thread-unsafe tests](#parallel-by-default-is-nice-in-theory-but-half-my-tests-are-failing-wat)
|
11
|
+
* Continuously run [after every file change](#running-tests-continuously-with---watch) with `--watch`
|
12
|
+
* An optional timer to [enforce your tests never get slow](#enforcing-a-testing---timeout) with `--timeout`
|
13
|
+
* A `--fail-fast` flag that aborts the run [as soon as a failure is encountered](#failing-fast-and-first)
|
14
|
+
* Running your most-recently-edited test before all the others (see `--prepend`)
|
15
|
+
* Delightful diffs when assertions fail, care of [super_diff](https://github.com/splitwise/super_diff)
|
23
16
|
|
24
|
-
|
17
|
+
We hope you'll give it a try!
|
25
18
|
|
26
|
-
|
27
|
-
and by names or patterns (e.g. `--name test_fail,test_error --name "/_\d/"`)
|
28
|
-
* Everything is **parallel by default**, and seems pretty darn fast; TLDR
|
29
|
-
also provides [several escape hatches to sequester tests that aren't thread-safe](#parallel-by-default-is-nice-in-theory-but-half-my-tests-are-failing-wat)
|
30
|
-
* Surprisingly delightful color diff output when two things fail to equal one
|
31
|
-
another, care of [@mcmire's super_diff gem](https://github.com/mcmire/super_diff)
|
32
|
-
* By default, the CLI will prepend your most-recently-edited test file to the
|
33
|
-
front of your suite so its tests will run first. The test you worked on most recently
|
34
|
-
is the one you most likely want to ensure runs, so TLDR runs it first (see the
|
35
|
-
`--prepend` option for how to control this behavior)
|
36
|
-
* And, of course, our signature feature: your test suite will never grow into
|
37
|
-
a glacially slow, soul-sucking albatross around your neck, because **after 1.8
|
38
|
-
seconds, it stops running your tests**, with a report on what it _was_ able to
|
39
|
-
run and where your slowest tests are
|
19
|
+
## Getting started
|
40
20
|
|
41
|
-
|
21
|
+
You can either read the rest of this README and learn about TLDR passively, or you can just clone this [TLDR demo repo](https://github.com/searls/tldr_demo) and work through its README as you play with its tests and run them with various options.
|
42
22
|
|
43
|
-
|
44
|
-
* That bit about your test suite exploding after 1.8 seconds
|
23
|
+
**JUST IN CASE YOU'RE ALREADY SKIMMING THIS**, I said stop reading and [clone this interactive repo](https://github.com/searls/tldr_demo) if you're a hands-on learner.
|
45
24
|
|
46
|
-
|
25
|
+
### Install
|
47
26
|
|
48
|
-
|
27
|
+
You know the drill. `gem install tldr` or add it to your Gemfile:
|
49
28
|
|
50
29
|
```
|
51
30
|
gem "tldr"
|
52
31
|
```
|
53
32
|
|
54
|
-
|
33
|
+
### Project setup
|
55
34
|
|
56
|
-
|
35
|
+
By default, TLDR expects your tests to be in `test/` with filenames that match `test_*.rb` or `*_test.rb` and will require a `test/helper.rb` if you define one.
|
57
36
|
|
58
|
-
|
59
|
-
class MathTest < TLDR
|
60
|
-
def test_adding
|
61
|
-
assert_equal 1 + 1, 2
|
62
|
-
end
|
63
|
-
end
|
64
|
-
```
|
65
|
-
|
66
|
-
A TLDR subclass defines its tests with instance methods that begin with
|
67
|
-
`test_`. They can define `setup` and/or `teardown` methods which will run before
|
68
|
-
and after each test, respectively.
|
69
|
-
|
70
|
-
If you place your tests in `test/**/*_test.rb` (and/or `test/**/test_*.rb`)
|
71
|
-
files, the `tldr` executable will find them automatically. And if you define a
|
72
|
-
`test/helper.rb` file, it will be loaded prior to your tests.
|
73
|
-
|
74
|
-
Running the CLI is pretty straightforward:
|
75
|
-
|
76
|
-
```
|
77
|
-
$ tldr
|
78
|
-
```
|
79
|
-
|
80
|
-
You can, of course, also just run a specific test file or glob:
|
81
|
-
|
82
|
-
```
|
83
|
-
$ tldr test/this/one/in/particular.rb
|
84
|
-
```
|
85
|
-
|
86
|
-
Or specify the line numbers of tests to run by appending them after a `:`
|
87
|
-
|
88
|
-
```
|
89
|
-
$ tldr test/fixture/line_number.rb:3:10
|
90
|
-
```
|
37
|
+
## Configuring TLDR
|
91
38
|
|
92
|
-
|
93
|
-
flags:
|
94
|
-
|
95
|
-
```
|
96
|
-
$ tldr --name FooTest#test_foo -n test_bar,test_baz -n /_qux/
|
97
|
-
```
|
98
|
-
|
99
|
-
(The above will translate to this array of name filters internally:
|
100
|
-
`["FooTest#test_foo", "test_bar", "test_baz", "/_qux/"]`.)
|
101
|
-
|
102
|
-
### Options
|
103
|
-
|
104
|
-
Here is the full list of CLI options:
|
39
|
+
### CLI Options
|
105
40
|
|
106
41
|
```
|
107
42
|
$ tldr --help
|
108
43
|
Usage: tldr [options] some_tests/**/*.rb some/path.rb:13 ...
|
44
|
+
-t, --[no-]timeout [TIMEOUT] Timeout (in seconds) before timer aborts the run (Default: 1.8)
|
45
|
+
--watch Run your tests continuously on file save (requires 'fswatch' to be installed)
|
109
46
|
--fail-fast Stop running tests as soon as one fails
|
110
|
-
-s, --seed SEED Seed for randomization
|
111
47
|
--[no-]parallel Parallelize tests (Default: true)
|
48
|
+
-s, --seed SEED Random seed for test order (setting --seed disables parallelization by default)
|
112
49
|
-n, --name PATTERN One or more names or /patterns/ of tests to run (like: foo_test, /test_foo.*/, Foo#foo_test)
|
113
50
|
--exclude-name PATTERN One or more names or /patterns/ NOT to run
|
114
51
|
--exclude-path PATH One or more paths NOT to run (like: foo.rb, "test/bar/**", baz.rb:3)
|
@@ -117,96 +54,95 @@ Usage: tldr [options] some_tests/**/*.rb some/path.rb:13 ...
|
|
117
54
|
--prepend PATH Prepend one or more paths to run before the rest (Default: most recently modified test)
|
118
55
|
--no-prepend Don't prepend any tests before the rest of the suite
|
119
56
|
-l, --load-path PATH Add one or more paths to the $LOAD_PATH (Default: ["lib", "test"])
|
120
|
-
-r, --reporter REPORTER Set a custom reporter class (Default: "TLDR::Reporters::Default")
|
121
57
|
--base-path PATH Change the working directory for all relative paths (Default: current working directory)
|
122
|
-
|
123
|
-
|
58
|
+
-c, --[no-]config PATH The YAML configuration file to load (Default: '.tldr.yml')
|
59
|
+
-r, --reporter REPORTER Set a custom reporter class (Default: "TLDR::Reporters::Default")
|
60
|
+
--[no-]emoji Enable emoji output for the default reporter (Default: false)
|
61
|
+
--[no-]warnings Print Ruby warnings (Default: true)
|
124
62
|
-v, --verbose Print stack traces for errors
|
63
|
+
--yes-i-know Suppress TLDR report when suite runs beyond any configured --timeout
|
64
|
+
--exit-0-on-timeout Exit with status code 0 when suite times out instead of 3
|
65
|
+
--exit-2-on-failure Exit with status code 2 (normally for errors) for both failures and errors
|
125
66
|
--print-interrupted-test-backtraces
|
126
|
-
Print stack traces
|
127
|
-
--[no-]warnings Print Ruby warnings (Default: true)
|
128
|
-
--watch Run your tests continuously on file save (requires 'fswatch' to be installed)
|
129
|
-
--yes-i-know Suppress TLDR report when suite runs over 1.8s
|
130
|
-
--i-am-being-watched [INTERNAL] Signals to tldr it is being invoked under --watch mode
|
131
|
-
--comment COMMENT [INTERNAL] No-op; used for multi-line execution instructions
|
67
|
+
Print stack traces of tests interrupted after a timeout
|
132
68
|
```
|
133
69
|
|
134
|
-
After being parsed, all the CLI options are converted into a
|
135
|
-
[TLDR::Config](/lib/tldr/value/config.rb) object.
|
136
|
-
|
137
70
|
### Setting defaults in .tldr.yml
|
138
71
|
|
139
|
-
The `tldr` CLI will look for a `.tldr.yml` file in your project
|
140
|
-
working directory or whatever `--base-path` you set), which can contain values
|
141
|
-
for any properties on [TLDR::Config](/lib/tldr/value/config.rb) (with the
|
142
|
-
exception of `--base-path` itself).
|
72
|
+
The `tldr` CLI will look for a `.tldr.yml` file in the root of your project that can set all the same options supported by the CLI. You can specify a custom YAML location with `--config some/path.yml` if you want it to live someplace else.
|
143
73
|
|
144
|
-
Any
|
145
|
-
still be specified by the `tldr` CLI or a `TLDR::Config` object passed to
|
146
|
-
[TLDR::Run.at_exit!](#running-tests-without-the-cli).
|
74
|
+
Any options found in the dotfile will override TLDR's defaults, but can still be overridden by the `tldr` CLI or a `TLDR::Config` object when running tests programmatically.
|
147
75
|
|
148
|
-
Here's an [example project](/example/c) that specifies a `.tldr.yml` file as
|
149
|
-
well as some [internal tests](/tests/dotfile_test.rb) demonstrating its behavior.
|
76
|
+
Here's an [example project](/example/c) that specifies a `.tldr.yml` file as well as some [internal tests](/tests/dotfile_test.rb) demonstrating its behavior.
|
150
77
|
|
151
|
-
|
78
|
+
## Writing your tests
|
152
79
|
|
153
|
-
|
154
|
-
[Minitest](https://github.com/minitest/minitest) tests. Some notes:
|
155
|
-
|
156
|
-
* `setup` and `teardown` hook methods should work as you expect. (We even threw
|
157
|
-
in [an `around` hook](https://github.com/splattael/minitest-around) as a bonus!)
|
158
|
-
* All of Minitest's assertions (e.g. `assert`, `assert_equals`) are provided,
|
159
|
-
with these caveats:
|
160
|
-
* To retain the `expected, actual` argument ordering, `tldr` defines
|
161
|
-
`assert_include?(element, container)` instead of
|
162
|
-
`assert_includes(container, element)`
|
163
|
-
* If you want to maximize compatibility and mix in `assert_includes` and the
|
164
|
-
deprecated `assert_send`, just `include
|
165
|
-
TLDR::Assertions::MinitestCompatibility` into the `TLDR` base class or
|
166
|
-
individual test classesJust set it
|
80
|
+
If you've ever seen a [Minitest](https://github.com/minitest/minitest?tab=readme-ov-file#synopsis-) test, then you already know how to write TLDR tests. Rather than document how to write tests, this section just highlights the ways TLDR tests differ from Minitest tests.
|
167
81
|
|
168
|
-
|
82
|
+
First, instead of inheriting from `Minitest::Test`, TLDR test classes should descend from (wait for it) the `TLDR` base class:
|
169
83
|
|
170
|
-
|
171
|
-
|
172
|
-
|
173
|
-
|
84
|
+
```ruby
|
85
|
+
class MyTest < TLDR
|
86
|
+
def test_looks_familiar
|
87
|
+
assert true
|
88
|
+
end
|
89
|
+
end
|
90
|
+
```
|
174
91
|
|
175
|
-
|
176
|
-
[fswatch](https://github.com/emcrisostomo/fswatch) installed and on your `PATH`
|
92
|
+
Second, if your tests depend on a test helper, it will be automatically loaded by TLDR _if_ you name it `test/helper.rb`. That means you don't need to add `require "helper"` to the top of every test. If you want to name the helper something else, you can do so with the `--helper` option:
|
177
93
|
|
178
|
-
|
94
|
+
```
|
95
|
+
tldr --helper test/test_helper.rb
|
96
|
+
```
|
179
97
|
|
180
|
-
|
98
|
+
Third, TLDR offers fewer features:
|
181
99
|
|
182
|
-
|
100
|
+
* No built-in mock library ([use mocktail](https://justin.searls.co/posts/a-real-world-mocktail-example-test/), maybe!)
|
101
|
+
* No "spec" API
|
102
|
+
* No benchmark tool
|
103
|
+
* No bisect script
|
183
104
|
|
184
|
-
|
185
|
-
out to the `tldr` CLI. If you want to run TLDR with Rake, you can configure
|
186
|
-
the test run by setting flags on an env var named `TLDR_OPTS` or else in
|
187
|
-
the [.tldr.yml](#setting-defaults-in-tldryml).
|
105
|
+
And that's it! You officially know how to write TLDR tests.
|
188
106
|
|
189
|
-
|
107
|
+
## Running your tests
|
190
108
|
|
191
|
-
|
192
|
-
|
193
|
-
|
109
|
+
Because TLDR ships with a CLI, it offers a veritable _plethora_ of ways to run your tests.
|
110
|
+
|
111
|
+
### Running your tests
|
194
112
|
|
195
|
-
|
113
|
+
Once installed, running all your tests is just five keystrokes away:
|
114
|
+
|
115
|
+
```
|
116
|
+
tldr
|
196
117
|
```
|
197
118
|
|
198
|
-
|
119
|
+
This assumes your tests are stored in `test/`. It will also add `lib/` to Ruby's load paths and require `test/helper.rb` before your tests, if it exists.
|
120
|
+
|
121
|
+
### Running TLDR with Rake
|
122
|
+
|
123
|
+
TLDR ships with a minimal [rake task](lib/tldr/rake.rb) that simply shells out to the `tldr` CLI by default. If you want to run TLDR with Rake, you can configure the task by setting flags on an env var named `TLDR_OPTS` or in a [.tldr.yml file](#setting-defaults-in-tldryml).
|
124
|
+
|
125
|
+
All your Rakefile needs is `require "tldr/rake"` and you can run the task individually like this:
|
199
126
|
|
200
127
|
```
|
201
|
-
$
|
128
|
+
$ rake tldr
|
129
|
+
|
130
|
+
# Or, with options in TLDR_OPTS
|
131
|
+
$ TLDR_OPTS="--no-parallel" rake tldr
|
202
132
|
```
|
203
133
|
|
204
|
-
|
205
|
-
|
206
|
-
|
207
|
-
|
134
|
+
Here's an example Rakefile that runs both TLDR and [Standard Ruby](https://github.com/standardrb/standard) as the default task:
|
135
|
+
|
136
|
+
```ruby
|
137
|
+
require "standard/rake"
|
138
|
+
require "tldr/rake"
|
139
|
+
|
140
|
+
task default: ["tldr", "standard:fix"]
|
141
|
+
```
|
208
142
|
|
209
|
-
|
143
|
+
One situation where you'd want to invoke TLDR with Rake is when you have multiple test suites that you want to be able to easily run separately ([this talk](https://blog.testdouble.com/talks/2014-05-25-breaking-up-with-your-test-suite/) discussed a few reasons why this can be useful).
|
144
|
+
|
145
|
+
To create a custom TLDR Rake task, you can instantiate `TLDR::Task` like this, which allows you to define its [TLDR::Config](/lib/tldr/value/config.rb) configuration in code:
|
210
146
|
|
211
147
|
```ruby
|
212
148
|
require "tldr/rake"
|
@@ -218,20 +154,27 @@ TLDR::Task.new(name: :safe_tests, config: TLDR::Config.new(
|
|
218
154
|
))
|
219
155
|
```
|
220
156
|
|
221
|
-
The above will create a second Rake task named `safe_tests` running a different
|
222
|
-
|
223
|
-
|
157
|
+
The above will create a second Rake task named `safe_tests` running a different set of tests than the default `tldr` task. Here's [an example](/example/b/Rakefile) from TLDR's test suite.
|
158
|
+
|
159
|
+
### Running tests continuously with --watch
|
160
|
+
|
161
|
+
The `tldr` CLI includes a `--watch` option that will watch for changes in any of the configured load paths (`["test", "lib"]` by default) and then execute your tests each time a file is changed. To keep the output up-to-date and easy
|
162
|
+
to scan, it will also clear your console before each run.
|
163
|
+
|
164
|
+
Note that this feature requires you have [fswatch](https://github.com/emcrisostomo/fswatch) installed and on your `PATH`
|
224
165
|
|
225
|
-
|
166
|
+
Here's what that might look like with the `--emoji` flag enabled:
|
167
|
+
|
168
|
+

|
226
169
|
|
227
|
-
|
228
|
-
|
170
|
+
### Running tests programmatically
|
171
|
+
|
172
|
+
If you'd rather use TLDR by running Ruby files instead of the `tldr` CLI, you can simply `require "tldr/autorun"` (just like `require "minitest/autorun"`).
|
229
173
|
|
230
174
|
Given a file `test/some_test.rb`:
|
231
175
|
|
232
176
|
```ruby
|
233
|
-
require "tldr"
|
234
|
-
TLDR::Run.at_exit! TLDR::Config.new(no_emoji: true)
|
177
|
+
require "tldr/autorun"
|
235
178
|
|
236
179
|
class SomeTest < TLDR
|
237
180
|
def test_truth
|
@@ -240,112 +183,166 @@ class SomeTest < TLDR
|
|
240
183
|
end
|
241
184
|
```
|
242
185
|
|
243
|
-
You
|
186
|
+
You can then run the test by passing `ruby` the file:
|
244
187
|
|
245
188
|
```
|
246
189
|
$ ruby test/some_test.rb
|
247
190
|
```
|
248
191
|
|
249
|
-
|
250
|
-
`tldr` CLI), running `at_exit!` will not set default values to the `paths`,
|
251
|
-
`helper`, `load_paths`, and `prepend_paths` config properties. You'll have to
|
252
|
-
pass any values you want to set on a [Config object](/lib/tldr/value/config.rb)
|
253
|
-
and pass it to `at_exit!`.
|
192
|
+
Any CLI options you add will still be parsed, as well (e.g. `ruby my_test.rb --emoji` will work).
|
254
193
|
|
255
|
-
|
256
|
-
encountered multiple times, only the first hook will be registered. If the
|
257
|
-
`tldr` CLI is running and encounters a call to `at_exit!`, it will be ignored.
|
194
|
+
If you want to be explicit about setting the `Kernel.at_exit` hook, or if you want to configure TLDR with code, you can invoke `TLDR.at_exit!` directly:
|
258
195
|
|
259
|
-
|
196
|
+
```ruby
|
197
|
+
require "tldr"
|
198
|
+
TLDR::Run.at_exit! TLDR::Config.new(emoji: true)
|
199
|
+
```
|
260
200
|
|
261
|
-
|
262
|
-
for you, but when running TLDR from a Ruby script, it doesn't set those up for
|
263
|
-
you.
|
201
|
+
## Failing with style
|
264
202
|
|
265
|
-
|
266
|
-
`require_relative`, you'll need to add those directories to the load path. You
|
267
|
-
can do this programmatically by prepending the path to `$LOAD_PATH`, like
|
268
|
-
this:
|
203
|
+
### Failing fast and first
|
269
204
|
|
270
|
-
|
271
|
-
$LOAD_PATH.unshift "test"
|
205
|
+
If we just want to know if the build passes, we want to know as fast as possible. Ever see a test fail and then sit around waiting for the whole suite to finish running anyway? Why wait? Turn on `--fail-fast` and abort the test run the instant a failure is encountered:
|
272
206
|
|
273
|
-
|
274
|
-
|
207
|
+
```
|
208
|
+
tldr --fail-fast
|
209
|
+
```
|
210
|
+
|
211
|
+
Additionally, you might notice the top of each run will show you a command you can use to execute the same run, like this:
|
212
|
+
|
213
|
+
```
|
214
|
+
Command: bundle exec tldr --fail-fast --prepend "test/calculator_test.rb"
|
215
|
+
```
|
216
|
+
|
217
|
+
That's because TLDR will look at the file system and move your most-recently-edited test file to the front of the queue with `--prepend`. When used in conjunction with `--fail-fast`, you'll fail _extra fast_, because the most likely test to fail is the one you're actively working on.
|
218
|
+
|
219
|
+
### Enforcing a testing --timeout
|
220
|
+
|
221
|
+
We initially developed TLDR because we wanted a test runner that supported suite-wide time limits as a first-class feature. When test suites become slow, people run them (_much_) less often. And once a developer gets in the habit of only running tests occasionally, it's not long before they only run them before push, and then only the ones they're immediately working on, and then they just wait for CI to run them in a pull request. And if you don't run all your tests very often, you don't feel any pain when you make your tests (or the code its testing) slower.
|
222
|
+
|
223
|
+
Each time you write code and don't run your tests, you're making an assumption that whatever code you just wrote _works flawlessly_. If that assumption is correct, you saved however much time it takes to run your tests. But every time that assumption is incorrect, you've just extended the amount of time before discovering that you broke something. If that's a few minutes later, that might only cost a few minutes of rework. **If you only run tests once or twice a day, you might have to undo hours of work to fix it.**
|
275
224
|
|
276
|
-
|
225
|
+
We originally came up with the idea of TLDR [on a livestream](https://www.youtube.com/live/bmi-SWeH4MA?si=p5g1j1FQZrbYEOCg&t=63), joking that only an unconfigurable 1.8 second time limit would prevent test suites from ballooning in duration over time. As of 1.0, we've made the timeout configurable, but we still think it's a good idea to enable it with the `--timeout` option:
|
226
|
+
|
227
|
+
```
|
228
|
+
tldr --timeout
|
277
229
|
```
|
278
230
|
|
279
|
-
|
231
|
+
Not only does a timeout keep us running the whole suite frequently (it'll never take more than 1.8 seconds, after all), but even if the suite begins to exceed our self-imposed timeout, TLDR's random test order and parallel execution means that—so long as you keep running that partial suite frequently—you'll still be running ALL your tests _many more times_ than if they waited until some arbitrary checkpoint to run them.
|
232
|
+
|
233
|
+
When enabled, `--timeout` will set the timer to 1.8 seconds. But you can set whatever time limit you like. The right value is depends on your individual capacity for paying attention. Basically, "however long you're willing to wait without caving and running your tests less often."
|
234
|
+
|
235
|
+
Examples:
|
280
236
|
|
281
237
|
```
|
282
|
-
|
238
|
+
# A badass 200ms timeout
|
239
|
+
tldr --timeout 0.2
|
240
|
+
|
241
|
+
# A miserable-sounding 20 second timeout
|
242
|
+
tldr --timeout 20
|
243
|
+
```
|
244
|
+
|
245
|
+
If we've won you over towards this way of working, we suggest creating a [.tldr.yml file](#setting-defaults-in-tldryml) in the root of your project and specifying your desired timeout.
|
246
|
+
|
247
|
+
For TLDR Classic™ and a 1.8s timeout:
|
248
|
+
|
249
|
+
```yaml
|
250
|
+
# .tldr.yml
|
251
|
+
timeout: true
|
283
252
|
```
|
284
253
|
|
254
|
+
Or any number of seconds you like:
|
255
|
+
|
256
|
+
```yaml
|
257
|
+
# .tldr.yml
|
258
|
+
timeout: 0.01
|
259
|
+
```
|
260
|
+
|
261
|
+
And if you're running with the timeout enabled this way, you can still disable it for any given test run by adding the `--no-timeout` flag.
|
262
|
+
|
263
|
+
#### Consider timeouts as a success with exit code 0
|
264
|
+
|
265
|
+
By default, when TLDR times out it exits with status code 3 to indicate the test suite was aborted. However, if you know that your full test suite is slower than whatever duration you consider to be "fast enough for fast feedback", you can use the `--exit-0-on-timeout` flag and simply use the timeout to enforce whatever feedback loop budget you set with `--timeout`.
|
266
|
+
|
267
|
+
For example:
|
268
|
+
|
269
|
+
```
|
270
|
+
# Get feedback within 400ms but don't consider a timeout a failure
|
271
|
+
tldr --timeout 0.4 --exit-0-on-timeout
|
272
|
+
```
|
273
|
+
|
274
|
+
This is particularly useful for:
|
275
|
+
- [Claude Code hooks](https://docs.anthropic.com/en/docs/claude-code/hooks) where you want fast feedback without failing the command
|
276
|
+
- [Git pre-commit hooks](https://git-scm.com/docs/githooks#_pre_commit) where you want quick test results without blocking commits
|
277
|
+
- Development workflows where timeouts are informational rather than failures
|
278
|
+
- Any situation where you want a time budget without hard failures
|
279
|
+
|
280
|
+
## Using TLDR as a Claude Code hook
|
281
|
+
|
282
|
+
In AI agent-based coding workflows, it's common to [configure hooks](https://docs.anthropic.com/en/docs/claude-code/hooks) that will block the agent from proceeding whenever tests or linters fail. With Claude Code specifically, a hook will only block if it exits with status code 2. As a result, you can use `--exit-2-on-failure` so that assertion failures will block the agent from continuing.
|
283
|
+
|
284
|
+
So you might configure a Claude Code hook with a 250ms budget by setting timeouts to exit code 0 and failures to exit code 2:
|
285
|
+
|
286
|
+
```
|
287
|
+
# Get feedback within 250ms, don't block the agent on timeouts, but do block it on assertion failures
|
288
|
+
tldr --timeout 0.25 --exit-0-on-timeout --exit-2-on-failure
|
289
|
+
```
|
290
|
+
|
291
|
+
|
285
292
|
## Questions you might be asking
|
286
293
|
|
287
294
|
TLDR is very similar to Minitest in API, but different in enough ways that you
|
288
295
|
probably have some questions.
|
289
296
|
|
297
|
+
## Wait, isn't this the one that blows up after 1.8 seconds?
|
298
|
+
|
299
|
+
The `tldr` gem was initially developed and released after we did a [lighthearted pairing session]() and imagined a Ruby test runner with a CLI and an unorthodox, unconfigurable rule: a hard-and-fast 1.8 second time limit on every test suite.
|
300
|
+
|
301
|
+
In the 18 months since, and to our utter surprise, TLDR did not immediately rise to the top of the charts and dominate the Ruby testing world.
|
302
|
+
|
303
|
+
While we still contend the mandatory time limit was a Very Good Idea (even if your tests are necessarily slower than 1.8 seconds), we believe there is a very remote, almost certainly wrong possibility that it was slowing adoption of this otherwise very capable test runner. As a result, **as of 1.0.0, the 1.8s timeout is disabled by default, and can be re-enabled (and even set to a specific value) with the `--timeout` option.
|
304
|
+
|
305
|
+
### Minitest compatibility
|
306
|
+
|
307
|
+
Tests you write with tldr are designed to be mostly-compatible with [Minitest](https://github.com/minitest/minitest) tests.
|
308
|
+
|
309
|
+
Details:
|
310
|
+
|
311
|
+
* We [implemented all](/lib/tldr/assertions.rb) of Minitest's built-in assertions (e.g. `assert`, `assert_equals`)
|
312
|
+
* `setup` and `teardown` hook methods should work as you expect. (We even threw in [our own `around` hook](https://github.com/search?q=repo%3Atendersearls/tldr%20%3Aaround&type=code), free of charge!)
|
313
|
+
* If you need anything else from Minitest as you port tests to TLDR, try `include TLDR::MinitestCompatibility`, and if that doesn't do the trick [add whatever you need in a pull request](/lib/tldr/minitest_compatibility.rb)
|
314
|
+
|
290
315
|
### Parallel-by-default is nice in theory but half my tests are failing. Wat?
|
291
316
|
|
292
|
-
**Read this before you add `--no-parallel` because some tests are failing when
|
293
|
-
you run `tldr`.**
|
294
|
-
|
295
|
-
The vast majority of test suites in the wild are not parallelized and the vast
|
296
|
-
majority of _those_ will only parallelize by forking processes as opposed to
|
297
|
-
using a thread pool. We wanted to encourage more people to save time (after all,
|
298
|
-
you only get 1.8 seconds here) by making your test suite run as fast as it can,
|
299
|
-
so your tests run in parallel threads by default.
|
300
|
-
|
301
|
-
If you're writing new code and tests with TLDR and dutifully running `tldr`
|
302
|
-
constantly for fast feedback, odds are that this will help you catch thread
|
303
|
-
safety issues early—this is a good thing, because it gives you a chance to
|
304
|
-
address them before they're too hard to fix! But maybe you're porting an
|
305
|
-
existing test suite to TLDR and running in parallel for the first time, or maybe
|
306
|
-
you need to test something that simply _can't_ be exercised in a thread-safe
|
307
|
-
way. For those cases, TLDR's goal is to give you some tools to prevent you from
|
308
|
-
giving up and adding `--no-parallel` to your entire test suite and **slowing
|
309
|
-
everything down for the sake of a few tests**.
|
310
|
-
|
311
|
-
So, when you see a test that is failing when run in parallel with the rest of your
|
312
|
-
suite, here is what we recommend doing, in priority order:
|
313
|
-
|
314
|
-
1. Figure out a way to redesign the test (or the code under test) to be
|
315
|
-
thread-safe. Modern versions of Ruby provide a number of tools to make this
|
316
|
-
easier than it used to be, and it may be as simple as making an instance
|
317
|
-
variable thread-local
|
318
|
-
2. If the problem is that a subset of your tests depend on the same resource,
|
319
|
-
try using [TLDR.run_these_together!](lib/tldr/parallel_controls.rb) class to
|
320
|
-
group the tests together. This will ensure that those tests run in the same
|
321
|
-
thread in sequence (here's a [simple
|
322
|
-
example](/tests/fixture/run_these_together.rb))
|
323
|
-
3. For tests that affect process-wide resources like setting the system clock or
|
324
|
-
changing the process's working directory (i.e. `Dir.chdir`), you can sequester
|
325
|
-
them to run sequentially _after_ all parallel tests in your suite have run with
|
326
|
-
[TLDR.dont_run_these_in_parallel!](lib/tldr/parallel_controls.rb), which takes
|
327
|
-
the same arguments as `run_these_together!`
|
328
|
-
([example](/tests/fixture/dont_run_these_in_parallel.rb))
|
329
|
-
4. Give up and make the whole suite `--no-parallel`. If you find that you need
|
330
|
-
to resort to this, you might save some keystrokes by adding `parallel: false` in
|
331
|
-
a [.tldr.yml](#setting-defaults-in-tldryml) file
|
332
|
-
|
333
|
-
We have a couple other ideas of ways to incorporate non-thread-safe tests into
|
334
|
-
your suite without slowing down the rest of your tests, so stay tuned!
|
335
|
-
|
336
|
-
### How will I run all my tests in CI without the time bomb going off?
|
337
|
-
|
338
|
-
TLDR will run all your tests in CI without the time bomb going off. If
|
339
|
-
`tldr` is run in a non-interactive shell and a `CI` environment variable is set
|
340
|
-
(as it is on virtually every CI service), then the bomb will be defused.
|
317
|
+
**Read this before you add `--no-parallel` because some tests are failing when you run `tldr`.**
|
341
318
|
|
342
|
-
|
319
|
+
The vast majority of test suites in the wild are run sequentially even though they'd work perfectly fine in parallel. Why? Because test runners tend not to enable it by default. Moreover, when they do, they usually rely on forking processes, which is slower and more resource-intensive than using threads or [Ractors](https://docs.ruby-lang.org/en/3.4/ractor_md.html). For this reason, TLDR is optimistic by default and will multi-thread your test suite.
|
320
|
+
|
321
|
+
So, what do you do when you run into a situation where a test has good reason not to run in parallel? Either because of resource contention or because the order of its test cases actually matters? Here's what we'd do:
|
322
|
+
|
323
|
+
1. Start by challenging the assumption that the test can't be run in a thread-safe way (it may be as simple as changing a globally-edited instance variable to a [thread-local](https://docs.ruby-lang.org/en/master/Thread.html#class-Thread-label-Fiber-local+vs.+Thread-local))
|
324
|
+
2. If the problem is that a subset of your tests depend on the same resource, try using [TLDR.run_these_together!](/lib/tldr/parallel_controls.rb) class to group the tests together. This will ensure that those tests run in the same thread in sequence (here's a [simple example](/tests/fixture/run_these_together.rb))
|
325
|
+
3. For tests that affect process-wide resources like setting the system clock or changing the process's working directory (i.e. `Dir.chdir`), you can sequester them to run sequentially _after_ all parallel tests in your suite have run with [TLDR.dont_run_these_in_parallel!](lib/tldr/parallel_controls.rb), which takes the same arguments as `run_these_together!` ([example](/tests/fixture/dont_run_these_in_parallel.rb))
|
326
|
+
4. Give up and make the whole suite `--no-parallel`. If you find that you need to resort to this, you might save some keystrokes by adding `parallel: false` in a [.tldr.yml](#setting-defaults-in-tldryml) file
|
327
|
+
|
328
|
+
### Any help porting from Minitest?
|
329
|
+
|
330
|
+
If you're currently using Minitest, you can take a stab at dropping your dependency on the minitest gem and replace references to `Minitest::Test` dynamically in a test helper, like I did for my [todo_or_die gem](https://github.com/searls/todo_or_die/blob/b50beb0166307d901393435594508f5142976c93/test/helper.rb#L7-L16):
|
331
|
+
|
332
|
+
```
|
333
|
+
require "tldr"
|
334
|
+
if defined?(Minitest::Test)
|
335
|
+
TLDR::MinitestTestBackup = Minitest::Test
|
336
|
+
Minitest.send(:remove_const, "Test")
|
337
|
+
end
|
338
|
+
module Minitest
|
339
|
+
class Test < TLDR
|
340
|
+
include TLDR::MinitestCompatibility
|
341
|
+
end
|
342
|
+
end
|
343
|
+
```
|
343
344
|
|
344
|
-
|
345
|
-
with this gem's binary in your PATH. If that's the case you could change your
|
346
|
-
path, invoke `bundle exec tldr`, run [with Rake](#running-tldr-with-rake), or
|
347
|
-
use the `tldt` ("too long; didn't test") executable alias that ships with this
|
348
|
-
gem.
|
345
|
+
This probably won't work for complex projects, but it might for simple ones!
|
349
346
|
|
350
347
|
### Is there a plugin system?
|
351
348
|
|
@@ -353,21 +350,14 @@ There is not.
|
|
353
350
|
|
354
351
|
Currently, the only pluggable aspect of TLDR are reporters, which can be set
|
355
352
|
with the `--reporter` command line option. It can be set to any fully-qualified
|
356
|
-
class name that
|
353
|
+
class name that responds to the same methods defined in
|
357
354
|
[TLDR::Reporters::Base](/lib/tldr/reporters/base.rb).
|
358
355
|
|
359
|
-
|
356
|
+
If you define a custom reporter, be sure to require it from your test helper, so TLDR can instantiate it!
|
360
357
|
|
361
|
-
|
362
|
-
huge summary at the end of each test run can be distracting and make it harder
|
363
|
-
to spot test failures. If you know your test suite is too slow, you can simply
|
364
|
-
add the `--yes-i-know` flag
|
365
|
-
|
366
|
-
### What about mocking?
|
358
|
+
### What if I already have another `tldr` executable on my path?
|
367
359
|
|
368
|
-
|
369
|
-
facility. Might we interest you in a refreshing
|
370
|
-
[mocktail](https://github.com/testdouble/mocktail), instead?
|
360
|
+
There's a [command-line utility named tldr](https://tldr.sh) that might conflict with this gem's executable in your PATH. If that's the case, you could either change your path, invoke `bundle exec tldr`, run [with Rake](#running-tldr-with-rake), or use the `tldt` ("too long; didn't test") executable alias that ships with this gem.
|
371
361
|
|
372
362
|
## Contributing to TLDR
|
373
363
|
|
data/exe/tldr
CHANGED