minitest-distributed 0.2.4 → 0.2.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/dependabot.yml +20 -0
- data/.rubocop.yml +2 -0
- data/Gemfile +1 -0
- data/README.md +32 -27
- data/Rakefile +1 -0
- data/lib/minitest/distributed/configuration.rb +1 -1
- data/lib/minitest/distributed/coordinators/redis_coordinator.rb +24 -23
- data/lib/minitest/distributed/reporters/distributed_progress_reporter.rb +1 -1
- data/lib/minitest/distributed/result_type.rb +2 -2
- data/lib/minitest/distributed/test_selector.rb +1 -0
- data/lib/minitest/distributed/version.rb +1 -1
- data/sorbet/rbi/redis.rbi +23 -2
- metadata +4 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: '0208858e32026a813e488ae30c72ae1ab9e274574aa55f5eb873f50db170a9d4'
|
4
|
+
data.tar.gz: 2e12cb8be3eb47009eb6aa30a5d58b9e7ef796a4e6851b02754fed3d3e261fdb
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e18ffe94425ae0726468df99f69d748a4c4d6744960d7f731079bd9dd2a927a924108d6a26e9e6383abe3d91bf311ae7619c666ae563605cacfd7f41c1b7de02
|
7
|
+
data.tar.gz: 83ad27805dd3285a398e9266f7ed9d3a0f7e67468d3a9b8020300c7cc81b89b94222b424aa0ec2e5cb3dd231563d75d1c342f55c9c4918435a490697a326318e
|
@@ -0,0 +1,20 @@
|
|
1
|
+
version: 2
|
2
|
+
registries:
|
3
|
+
rubygems-server-pkgs-shopify-io:
|
4
|
+
type: rubygems-server
|
5
|
+
url: https://pkgs.shopify.io
|
6
|
+
username: ${{secrets.RUBYGEMS_SERVER_PKGS_SHOPIFY_IO_USERNAME}}
|
7
|
+
password: ${{secrets.RUBYGEMS_SERVER_PKGS_SHOPIFY_IO_PASSWORD}}
|
8
|
+
github-com:
|
9
|
+
type: git
|
10
|
+
url: https://github.com
|
11
|
+
username: ${{secrets.DEPENDENCIES_GITHUB_USER}}
|
12
|
+
password: ${{secrets.DEPENDENCIES_GITHUB_TOKEN}}
|
13
|
+
updates:
|
14
|
+
- package-ecosystem: bundler
|
15
|
+
directory: "/"
|
16
|
+
schedule:
|
17
|
+
interval: daily
|
18
|
+
open-pull-requests-limit: 100
|
19
|
+
insecure-external-code-execution: allow
|
20
|
+
registries: "*"
|
data/.rubocop.yml
CHANGED
data/Gemfile
CHANGED
data/README.md
CHANGED
@@ -1,5 +1,12 @@
|
|
1
1
|
# minitest-distributed
|
2
2
|
|
3
|
+
[](LICENSE.md)
|
4
|
+
|
5
|
+
[About this repo](#about-this-repo) | [Commands](#commands) | [How to use this repo](#how-to-use-this-repo) | [Contribute to this repo](#contribute-to-this-repo) | [License](#license)
|
6
|
+
|
7
|
+
## About this repo
|
8
|
+
**Introduction:**
|
9
|
+
|
3
10
|
`minitest-distributed` is a plugin for [minitest](https://github.com/seattlerb/minitest)
|
4
11
|
for executing tests on a distributed set of unreliable workers.
|
5
12
|
|
@@ -16,14 +23,14 @@ flakiness. To combat flakiness, minitest-distributed implements resiliency
|
|
16
23
|
patterns, like re-running a test on a different worker on failure, and a
|
17
24
|
circuit breaker for misbehaving workers.
|
18
25
|
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
also be set using environment variables.
|
26
|
+
| | |
|
27
|
+
|----------------|--------------------------------------------------------------------------------------------------------------------------------------|
|
28
|
+
| Current status | Ongoing |
|
29
|
+
| Owner | [@Shopify/test-infra](https://github.com/orgs/Shopify/teams/test-infra) |
|
30
|
+
| Help | [#team-test-infra](https://shopify.slack.com/archives/team-test-infra) |
|
25
31
|
|
26
|
-
|
32
|
+
## Commands
|
33
|
+
**Distributed invocation**
|
27
34
|
|
28
35
|
To actually run tests with multiple workers, you have to point every worker to
|
29
36
|
a Redis coordinator, and use the same run identifier.
|
@@ -52,7 +59,7 @@ Rake::TestTask.new(:test) do |t|
|
|
52
59
|
end
|
53
60
|
```
|
54
61
|
|
55
|
-
|
62
|
+
**Worker retries**
|
56
63
|
|
57
64
|
Many CI systems offer the options to retry jobs that fail. When jobs are
|
58
65
|
retried that were previously part of a worker cluster, all the retried jobs
|
@@ -61,7 +68,7 @@ during the previous run attempt. This is to make it faster to re-run tests
|
|
61
68
|
that failed due to flakiness, or confirm that it was not flakiness that caused
|
62
69
|
them to fail.
|
63
70
|
|
64
|
-
|
71
|
+
**Other optional command line arguments**
|
65
72
|
|
66
73
|
- `--test-timeout=SECONDS` or `ENV[MINITEST_TEST_TIMEOUT_SECONDS]` (default: 30s):
|
67
74
|
the maximum amount a test is allowed to run before it times out. In a distributed
|
@@ -87,7 +94,7 @@ them to fail.
|
|
87
94
|
the test run. The file should include test identifiers seperated by
|
88
95
|
newlines.
|
89
96
|
|
90
|
-
|
97
|
+
**Limitations**
|
91
98
|
|
92
99
|
**Parallel tests not supported:** Minitest comes bundled with a parallel test
|
93
100
|
executor, which will run tests that are specifically tagged as such in
|
@@ -96,7 +103,20 @@ in parallel using separate processes, generally on different VMs. For this
|
|
96
103
|
reason, tests marked as `parallel` will not be treated any differently than
|
97
104
|
other tests.
|
98
105
|
|
99
|
-
##
|
106
|
+
## How to use this repo
|
107
|
+
Add `minitest-distributed` to your `Gemfile`, and run `bundle install`. The
|
108
|
+
plugin will be loaded by minitest automatically. The plugin exposes some
|
109
|
+
command line arguments that you can use to influence its behavior. They can
|
110
|
+
also be set using environment variables.
|
111
|
+
|
112
|
+
## Contribute to this repo
|
113
|
+
Bug reports and pull requests are welcome on GitHub at
|
114
|
+
https://github.com/Shopify/minitest-distributed. This project is intended to
|
115
|
+
be a safe, welcoming space for collaboration, and contributors are expected to
|
116
|
+
adhere to the [code of
|
117
|
+
conduct](https://github.com/Shopify/minitest-distributed/blob/master/CODE_OF_CONDUCT.md).
|
118
|
+
|
119
|
+
**Development**
|
100
120
|
|
101
121
|
To bootstrap a local development environment:
|
102
122
|
|
@@ -109,7 +129,7 @@ To bootstrap a local development environment:
|
|
109
129
|
- You can also run `bin/console` for an interactive prompt that will allow you
|
110
130
|
to experiment.
|
111
131
|
|
112
|
-
|
132
|
+
**Releasing a new version**
|
113
133
|
|
114
134
|
- To install this gem onto your local machine, run `bin/rake install`.
|
115
135
|
- Only people at Shopify can release a new version to
|
@@ -117,21 +137,6 @@ To bootstrap a local development environment:
|
|
117
137
|
in `version.rb`, and merge to master. Shipit will take care of building the
|
118
138
|
`.gem` bundle, and pushing it to rubygems.org.
|
119
139
|
|
120
|
-
## Contributing
|
121
|
-
|
122
|
-
Bug reports and pull requests are welcome on GitHub at
|
123
|
-
https://github.com/Shopify/minitest-distributed. This project is intended to
|
124
|
-
be a safe, welcoming space for collaboration, and contributors are expected to
|
125
|
-
adhere to the [code of
|
126
|
-
conduct](https://github.com/Shopify/minitest-distributed/blob/master/CODE_OF_CONDUCT.md).
|
127
|
-
|
128
140
|
## License
|
129
|
-
|
130
141
|
The gem is available as open source under the terms of the [MIT
|
131
142
|
License](https://opensource.org/licenses/MIT).
|
132
|
-
|
133
|
-
## Code of Conduct
|
134
|
-
|
135
|
-
Everyone interacting in the `minitest-distributed` project's codebases, issue
|
136
|
-
trackers, chat rooms and mailing lists is expected to follow the [code of
|
137
|
-
conduct](https://github.com/Shopify/minitest-distributed/blob/master/CODE_OF_CONDUCT.md).
|
data/Rakefile
CHANGED
@@ -62,7 +62,7 @@ module Minitest
|
|
62
62
|
|
63
63
|
opts.on(
|
64
64
|
"--[no-]retry-failures", "Retry failed and errored tests from a previous run attempt " \
|
65
|
-
|
65
|
+
"with the same run ID (default: enabled)"
|
66
66
|
) do |enabled|
|
67
67
|
configuration.retry_failures = enabled
|
68
68
|
end
|
@@ -2,6 +2,7 @@
|
|
2
2
|
# frozen_string_literal: true
|
3
3
|
|
4
4
|
require "redis"
|
5
|
+
require "set"
|
5
6
|
|
6
7
|
module Minitest
|
7
8
|
module Distributed
|
@@ -128,7 +129,6 @@ module Minitest
|
|
128
129
|
argv: [group_name],
|
129
130
|
)
|
130
131
|
keys_deleted == 0
|
131
|
-
|
132
132
|
rescue Redis::CommandError => ce
|
133
133
|
if ce.message.include?("BUSYGROUP")
|
134
134
|
# If Redis returns a BUSYGROUP error, it means that the consumer group already
|
@@ -165,10 +165,10 @@ module Minitest
|
|
165
165
|
adjust_combined_results(ResultAggregate.new(size: 0))
|
166
166
|
T.let([], T::Array[Minitest::Runnable])
|
167
167
|
else
|
168
|
-
previous_failures, previous_errors, _deleted = redis.multi do
|
169
|
-
|
170
|
-
|
171
|
-
|
168
|
+
previous_failures, previous_errors, _deleted = redis.multi do |pipeline|
|
169
|
+
pipeline.lrange(list_key(ResultType::Failed.serialize), 0, -1)
|
170
|
+
pipeline.lrange(list_key(ResultType::Error.serialize), 0, -1)
|
171
|
+
pipeline.del(list_key(ResultType::Failed.serialize), list_key(ResultType::Error.serialize))
|
172
172
|
end
|
173
173
|
|
174
174
|
# We set the `size` key to the number of tests we are planning to schedule.
|
@@ -199,9 +199,9 @@ module Minitest
|
|
199
199
|
T.let([], T::Array[Minitest::Runnable])
|
200
200
|
end
|
201
201
|
|
202
|
-
redis.pipelined do
|
202
|
+
redis.pipelined do |pipeline|
|
203
203
|
tests.each do |test|
|
204
|
-
|
204
|
+
pipeline.xadd(stream_key, { class_name: T.must(test.class.name), method_name: test.name })
|
205
205
|
end
|
206
206
|
end
|
207
207
|
end
|
@@ -292,6 +292,7 @@ module Minitest
|
|
292
292
|
end
|
293
293
|
def xclaim_messages(pending_messages, max_idle_time_ms:)
|
294
294
|
return [] if pending_messages.empty?
|
295
|
+
|
295
296
|
claimed = redis.xclaim(stream_key, group_name, configuration.worker_id,
|
296
297
|
max_idle_time_ms, pending_messages.keys)
|
297
298
|
|
@@ -383,9 +384,9 @@ module Minitest
|
|
383
384
|
# timeout. If the worker crashes between removing an item from the retry setm the test
|
384
385
|
# will eventually be picked up by another worker.
|
385
386
|
messages_in_retry_set = {}
|
386
|
-
redis.multi do
|
387
|
+
redis.multi do |pipeline|
|
387
388
|
active_messages.each do |key, message|
|
388
|
-
messages_in_retry_set[key] =
|
389
|
+
messages_in_retry_set[key] = pipeline.srem(key("retry_set"), message.attempt_id)
|
389
390
|
end
|
390
391
|
end
|
391
392
|
|
@@ -407,17 +408,17 @@ module Minitest
|
|
407
408
|
|
408
409
|
sig { params(results: ResultAggregate).void }
|
409
410
|
def adjust_combined_results(results)
|
410
|
-
updated = redis.multi do
|
411
|
-
|
412
|
-
|
413
|
-
|
414
|
-
|
415
|
-
|
416
|
-
|
417
|
-
|
418
|
-
|
419
|
-
|
420
|
-
|
411
|
+
updated = redis.multi do |pipeline|
|
412
|
+
pipeline.incrby(key("runs"), results.runs)
|
413
|
+
pipeline.incrby(key("assertions"), results.assertions)
|
414
|
+
pipeline.incrby(key("passes"), results.passes)
|
415
|
+
pipeline.incrby(key("failures"), results.failures)
|
416
|
+
pipeline.incrby(key("errors"), results.errors)
|
417
|
+
pipeline.incrby(key("skips"), results.skips)
|
418
|
+
pipeline.incrby(key("requeues"), results.requeues)
|
419
|
+
pipeline.incrby(key("discards"), results.discards)
|
420
|
+
pipeline.incrby(key("acks"), results.acks)
|
421
|
+
pipeline.incrby(key("size"), results.size)
|
421
422
|
end
|
422
423
|
|
423
424
|
@combined_results = ResultAggregate.new(max_failures: configuration.max_failures,
|
@@ -450,14 +451,14 @@ module Minitest
|
|
450
451
|
|
451
452
|
# Try to commit all the results of this batch to Redis
|
452
453
|
runnable_results = []
|
453
|
-
redis.multi do
|
454
|
+
redis.multi do |pipeline|
|
454
455
|
results.each do |enqueued_runnable, initial_result|
|
455
456
|
runnable_results << enqueued_runnable.commit_result(initial_result) do |result_to_commit|
|
456
457
|
if ResultType.of(result_to_commit) == ResultType::Requeued
|
457
|
-
sadd_future =
|
458
|
+
sadd_future = pipeline.sadd(key("retry_set"), enqueued_runnable.attempt_id)
|
458
459
|
EnqueuedRunnable::Result::Commit.new { sadd_future.value }
|
459
460
|
else
|
460
|
-
xack_future =
|
461
|
+
xack_future = pipeline.xack(stream_key, group_name, enqueued_runnable.entry_id)
|
461
462
|
EnqueuedRunnable::Result::Commit.new { xack_future.value == 1 }
|
462
463
|
end
|
463
464
|
end
|
@@ -29,7 +29,7 @@ module Minitest
|
|
29
29
|
super
|
30
30
|
end
|
31
31
|
|
32
|
-
#
|
32
|
+
# NOTE: due to batching and parallel tests, we have no guarantee that `prerecord`
|
33
33
|
# and `record` will be called in succession for the same test without calls to
|
34
34
|
# either method being interjected for other tests.
|
35
35
|
#
|
@@ -25,8 +25,8 @@ module Minitest
|
|
25
25
|
if result.time > test_timeout_seconds
|
26
26
|
message << format(
|
27
27
|
"\n\nThe test took %0.3fs to run, longer than the test timeout which is configured to be %0.1fs.\n" \
|
28
|
-
|
29
|
-
|
28
|
+
"Another worker likely claimed ownership of this test, and will commit the result instead.\n" \
|
29
|
+
"For best results, make sure that all your tests finish within %0.1fs.",
|
30
30
|
result.time, test_timeout_seconds, test_timeout_seconds
|
31
31
|
)
|
32
32
|
end
|
@@ -44,6 +44,7 @@ module Minitest
|
|
44
44
|
sig { params(tests: T::Array[Minitest::Runnable]).returns(T::Array[Minitest::Runnable]) }
|
45
45
|
def select_tests(tests)
|
46
46
|
return tests if filters.empty?
|
47
|
+
|
47
48
|
tests.flat_map do |runnable_method|
|
48
49
|
filters.flat_map do |filter|
|
49
50
|
filter.call(runnable_method)
|
data/sorbet/rbi/redis.rbi
CHANGED
@@ -1,4 +1,5 @@
|
|
1
1
|
# typed: true
|
2
|
+
# frozen_string_literal: true
|
2
3
|
|
3
4
|
class Redis
|
4
5
|
class Error < StandardError
|
@@ -21,10 +22,10 @@ class Redis
|
|
21
22
|
sig { void }
|
22
23
|
def flushdb; end
|
23
24
|
|
24
|
-
sig { params(block: T.proc.void).returns(T::Array[T.untyped]) }
|
25
|
+
sig { params(block: T.proc.params(arg0: Redis::PipelinedConnection).void).returns(T::Array[T.untyped]) }
|
25
26
|
def pipelined(&block); end
|
26
27
|
|
27
|
-
sig { params(block: T.proc.void).returns(T::Array[T.untyped]) }
|
28
|
+
sig { params(block: T.proc.params(arg0: Redis::PipelinedConnection).void).returns(T::Array[T.untyped]) }
|
28
29
|
def multi(&block); end
|
29
30
|
|
30
31
|
sig { params(script: String, keys: T::Array[String], argv: T::Array[String]).returns(T.untyped) }
|
@@ -83,3 +84,23 @@ class Redis
|
|
83
84
|
def xclaim(*); end
|
84
85
|
def xinfo(*); end
|
85
86
|
end
|
87
|
+
|
88
|
+
class Redis::PipelinedConnection
|
89
|
+
sig { params(key: String, value: T.untyped).returns(T.untyped) }
|
90
|
+
def sadd(key, value); end
|
91
|
+
|
92
|
+
sig { params(key: String, amount: Integer).returns(Integer) }
|
93
|
+
def incrby(key, amount); end
|
94
|
+
|
95
|
+
sig { params(key: String, value: T.untyped).returns(T::Boolean) }
|
96
|
+
def srem(key, value); end
|
97
|
+
|
98
|
+
sig { params(keys: String).void }
|
99
|
+
def del(*keys); end
|
100
|
+
|
101
|
+
sig { params(key: String, start: Integer, stop: Integer).void }
|
102
|
+
def lrange(key, start, stop); end
|
103
|
+
|
104
|
+
def xack(stream_key, group_name, *entry_ids); end
|
105
|
+
def xadd(key, value); end
|
106
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: minitest-distributed
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.2.
|
4
|
+
version: 0.2.5
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Willem van Bergen
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2022-03-21 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: minitest
|
@@ -74,6 +74,7 @@ executables: []
|
|
74
74
|
extensions: []
|
75
75
|
extra_rdoc_files: []
|
76
76
|
files:
|
77
|
+
- ".github/dependabot.yml"
|
77
78
|
- ".github/workflows/ruby.yml"
|
78
79
|
- ".gitignore"
|
79
80
|
- ".rubocop.yml"
|
@@ -139,7 +140,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
139
140
|
- !ruby/object:Gem::Version
|
140
141
|
version: '0'
|
141
142
|
requirements: []
|
142
|
-
rubygems_version: 3.
|
143
|
+
rubygems_version: 3.2.20
|
143
144
|
signing_key:
|
144
145
|
specification_version: 4
|
145
146
|
summary: Distributed test executor plugin for Minitest
|