specbandit 0.7.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +141 -15
- data/lib/specbandit/adapter.rb +20 -0
- data/lib/specbandit/cli.rb +87 -24
- data/lib/specbandit/cli_adapter.rb +66 -0
- data/lib/specbandit/configuration.rb +12 -1
- data/lib/specbandit/rspec_adapter.rb +111 -0
- data/lib/specbandit/version.rb +1 -1
- data/lib/specbandit/worker.rb +168 -119
- data/lib/specbandit.rb +3 -0
- metadata +8 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: bde3099f7c42b0d8c294590f4523aaaddee09f4057c42d368c4f5cc4f6855a58
|
|
4
|
+
data.tar.gz: a8c5648df12c5f5103937f89fd9869c19a4d94aae0124ae116467438346434c9
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e87c4f5952982592627f71b22c2b433289cc28a5a94bd6138d67e411fce4d4609708da129fd649d4d2675c2c082990872642f359527edb6100906cf68bce4158
|
|
7
|
+
data.tar.gz: ffc87b7b36a8249f62d104545aac53ab192d7f815711fd98e179441dcb20e2742e15ae6e6c95cbebb4b37f783c2d3f7034bf2fa8325b77083e34ae4e855d3794
|
data/README.md
CHANGED
|
@@ -4,13 +4,13 @@
|
|
|
4
4
|
|
|
5
5
|
# specbandit
|
|
6
6
|
|
|
7
|
-
Distributed
|
|
7
|
+
Distributed test runner using Redis as a work queue. One process pushes test file paths to a Redis list; multiple CI runners atomically steal batches and execute them via a pluggable adapter.
|
|
8
8
|
|
|
9
9
|
```
|
|
10
10
|
CI Job 1 (push): RPUSH key f1 f2 f3 ... fN --> [Redis List]
|
|
11
|
-
CI Job 2 (worker): LPOP key 5 <-- [Redis List] -->
|
|
12
|
-
CI Job 3 (worker): LPOP key 5 <-- [Redis List] -->
|
|
13
|
-
CI Job N (worker): LPOP key 5 <-- [Redis List] -->
|
|
11
|
+
CI Job 2 (worker): LPOP key 5 <-- [Redis List] --> adapter (cli/rspec)
|
|
12
|
+
CI Job 3 (worker): LPOP key 5 <-- [Redis List] --> adapter (cli/rspec)
|
|
13
|
+
CI Job N (worker): LPOP key 5 <-- [Redis List] --> adapter (cli/rspec)
|
|
14
14
|
```
|
|
15
15
|
|
|
16
16
|
`LPOP` with a count argument (Redis 6.2+) is atomic -- multiple workers calling it concurrently will never receive the same file.
|
|
@@ -31,11 +31,70 @@ gem install specbandit
|
|
|
31
31
|
|
|
32
32
|
**Requirements**: Ruby >= 3.0, Redis >= 6.2
|
|
33
33
|
|
|
34
|
+
## Adapters
|
|
35
|
+
|
|
36
|
+
specbandit v0.7.0 introduces a pluggable adapter architecture. Two adapters ship out of the box:
|
|
37
|
+
|
|
38
|
+
| Adapter | Default? | How it runs | Best for |
|
|
39
|
+
|---------|----------|-------------|----------|
|
|
40
|
+
| `cli` | Yes | Spawns a shell command per batch | Any test runner (Jest, pytest, Go test, etc.) |
|
|
41
|
+
| `rspec` | No | Runs `RSpec::Core::Runner` in-process | RSpec (fastest, richest reporting) |
|
|
42
|
+
|
|
43
|
+
### CLI adapter (default)
|
|
44
|
+
|
|
45
|
+
The CLI adapter spawns a shell command for each batch, appending file paths as arguments. It works with any test runner.
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
# Run RSpec via CLI adapter
|
|
49
|
+
specbandit work --key KEY --command "bundle exec rspec"
|
|
50
|
+
|
|
51
|
+
# Run with extra options
|
|
52
|
+
specbandit work --key KEY --command "bundle exec rspec" --command-opts "--format documentation"
|
|
53
|
+
|
|
54
|
+
# Run Jest
|
|
55
|
+
specbandit work --key KEY --command "npx jest"
|
|
56
|
+
|
|
57
|
+
# Forward args after -- (merged with --command-opts)
|
|
58
|
+
specbandit work --key KEY --command "bundle exec rspec" -- --format documentation
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### RSpec adapter
|
|
62
|
+
|
|
63
|
+
The RSpec adapter runs `RSpec::Core::Runner.run` in-process with `RSpec.clear_examples` between batches. No subprocess forking overhead. Provides rich reporting with per-example details, failure messages, and JSON accumulation.
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
specbandit work --key KEY --adapter rspec
|
|
67
|
+
|
|
68
|
+
# With RSpec options
|
|
69
|
+
specbandit work --key KEY --adapter rspec --rspec-opts "--format documentation"
|
|
70
|
+
|
|
71
|
+
# JSON output for CI artifact collection
|
|
72
|
+
specbandit work --key KEY --adapter rspec -- --format json --out results.json
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### Migration from v0.6.x
|
|
76
|
+
|
|
77
|
+
In v0.6.x, RSpec was the only execution method and was always used implicitly. In v0.7.0, the default adapter changed to `cli`. To keep the previous behavior, add `--adapter rspec`:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
# v0.6.x
|
|
81
|
+
specbandit work --key KEY
|
|
82
|
+
|
|
83
|
+
# v0.7.0 equivalent
|
|
84
|
+
specbandit work --key KEY --adapter rspec
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Or set the environment variable:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
export SPECBANDIT_ADAPTER=rspec
|
|
91
|
+
```
|
|
92
|
+
|
|
34
93
|
## Usage
|
|
35
94
|
|
|
36
|
-
### 1. Push
|
|
95
|
+
### 1. Push test files to Redis
|
|
37
96
|
|
|
38
|
-
A single CI job enqueues all
|
|
97
|
+
A single CI job enqueues all test file paths before workers start.
|
|
39
98
|
|
|
40
99
|
```bash
|
|
41
100
|
# Via glob pattern (resolved in Ruby, avoids shell ARG_MAX limits)
|
|
@@ -55,12 +114,16 @@ File input priority: **stdin > --pattern > direct args**.
|
|
|
55
114
|
Each CI runner steals batches and runs them. Start as many runners as you want -- they'll divide the work automatically.
|
|
56
115
|
|
|
57
116
|
```bash
|
|
58
|
-
|
|
117
|
+
# Using CLI adapter (default) -- works with any test runner
|
|
118
|
+
specbandit work --key pr-123-run-456 --command "bundle exec rspec" --batch-size 10
|
|
119
|
+
|
|
120
|
+
# Using RSpec adapter -- in-process, fastest for RSpec
|
|
121
|
+
specbandit work --key pr-123-run-456 --adapter rspec --batch-size 10
|
|
59
122
|
```
|
|
60
123
|
|
|
61
124
|
Each worker loops:
|
|
62
125
|
1. `LPOP` N file paths from Redis (atomic)
|
|
63
|
-
2.
|
|
126
|
+
2. Execute them via the configured adapter
|
|
64
127
|
3. Repeat until the queue is empty
|
|
65
128
|
4. Exit 0 if all batches passed, 1 if any failed
|
|
66
129
|
|
|
@@ -75,13 +138,20 @@ specbandit push [options] [files...]
|
|
|
75
138
|
--redis-url URL Redis URL (default: redis://localhost:6379)
|
|
76
139
|
--key-ttl SECONDS TTL for the Redis key (default: 21600 / 6 hours)
|
|
77
140
|
|
|
78
|
-
specbandit work [options]
|
|
141
|
+
specbandit work [options] [-- extra-opts...]
|
|
79
142
|
--key KEY Redis queue key (required)
|
|
143
|
+
--adapter TYPE Adapter type: 'cli' (default) or 'rspec'
|
|
144
|
+
--command CMD Command to run (required for cli adapter)
|
|
145
|
+
--command-opts OPTS Extra options forwarded to the command (space-separated)
|
|
146
|
+
--rspec-opts OPTS Extra options forwarded to RSpec (for rspec adapter)
|
|
80
147
|
--batch-size N Files per batch (default: 5)
|
|
81
148
|
--redis-url URL Redis URL (default: redis://localhost:6379)
|
|
82
|
-
--rspec-opts OPTS Extra options forwarded to RSpec
|
|
83
149
|
--key-rerun KEY Per-runner rerun key for re-run support (see below)
|
|
84
150
|
--key-rerun-ttl SECS TTL for rerun key (default: 604800 / 1 week)
|
|
151
|
+
--verbose Show per-batch file list and full command output
|
|
152
|
+
|
|
153
|
+
Arguments after -- are forwarded to the adapter. They are merged with
|
|
154
|
+
--command-opts (cli adapter) or --rspec-opts (rspec adapter).
|
|
85
155
|
```
|
|
86
156
|
|
|
87
157
|
### Environment variables
|
|
@@ -92,11 +162,15 @@ All CLI options can be set via environment variables:
|
|
|
92
162
|
|---|---|---|
|
|
93
163
|
| `SPECBANDIT_KEY` | Redis queue key | _(required)_ |
|
|
94
164
|
| `SPECBANDIT_REDIS_URL` | Redis connection URL | `redis://localhost:6379` |
|
|
165
|
+
| `SPECBANDIT_ADAPTER` | Adapter type (`cli` or `rspec`) | `cli` |
|
|
166
|
+
| `SPECBANDIT_COMMAND` | Command to run (cli adapter) | _(none)_ |
|
|
167
|
+
| `SPECBANDIT_COMMAND_OPTS` | Space-separated command options | _(none)_ |
|
|
95
168
|
| `SPECBANDIT_BATCH_SIZE` | Files per steal | `5` |
|
|
96
169
|
| `SPECBANDIT_KEY_TTL` | Key expiry in seconds | `21600` (6 hours) |
|
|
97
|
-
| `SPECBANDIT_RSPEC_OPTS` | Space-separated RSpec options | _(none)_ |
|
|
170
|
+
| `SPECBANDIT_RSPEC_OPTS` | Space-separated RSpec options (rspec adapter) | _(none)_ |
|
|
98
171
|
| `SPECBANDIT_KEY_RERUN` | Per-runner rerun key | _(none)_ |
|
|
99
172
|
| `SPECBANDIT_KEY_RERUN_TTL` | Rerun key expiry in seconds | `604800` (1 week) |
|
|
173
|
+
| `SPECBANDIT_VERBOSE` | Enable verbose output (`1`/`true`/`yes`) | _(false)_ |
|
|
100
174
|
|
|
101
175
|
CLI flags take precedence over environment variables.
|
|
102
176
|
|
|
@@ -112,20 +186,68 @@ Specbandit.configure do |c|
|
|
|
112
186
|
c.key_ttl = 7200 # 2 hours (default: 21600 / 6 hours)
|
|
113
187
|
c.key_rerun = "pr-123-run-456-runner-3"
|
|
114
188
|
c.key_rerun_ttl = 604_800 # 1 week (default)
|
|
115
|
-
c.rspec_opts = ["--format", "documentation"]
|
|
116
189
|
end
|
|
117
190
|
|
|
118
191
|
# Push
|
|
119
192
|
publisher = Specbandit::Publisher.new
|
|
120
193
|
publisher.publish(pattern: "spec/**/*_spec.rb")
|
|
121
194
|
|
|
122
|
-
# Work
|
|
123
|
-
|
|
195
|
+
# Work with CLI adapter (default)
|
|
196
|
+
adapter = Specbandit::CliAdapter.new(
|
|
197
|
+
command: "bundle exec rspec",
|
|
198
|
+
command_opts: ["--format", "documentation"]
|
|
199
|
+
)
|
|
200
|
+
worker = Specbandit::Worker.new(adapter: adapter)
|
|
201
|
+
exit_code = worker.run
|
|
202
|
+
|
|
203
|
+
# Work with RSpec adapter (in-process)
|
|
204
|
+
adapter = Specbandit::RspecAdapter.new(
|
|
205
|
+
rspec_opts: ["--format", "documentation"]
|
|
206
|
+
)
|
|
207
|
+
worker = Specbandit::Worker.new(adapter: adapter)
|
|
208
|
+
exit_code = worker.run
|
|
209
|
+
|
|
210
|
+
# Legacy: passing rspec_opts directly still works (auto-creates RspecAdapter)
|
|
211
|
+
worker = Specbandit::Worker.new(rspec_opts: ["--format", "documentation"])
|
|
124
212
|
exit_code = worker.run
|
|
125
213
|
```
|
|
126
214
|
|
|
127
215
|
## Example: GitHub Actions (basic)
|
|
128
216
|
|
|
217
|
+
### Using RSpec adapter (in-process)
|
|
218
|
+
|
|
219
|
+
```yaml
|
|
220
|
+
jobs:
|
|
221
|
+
push-specs:
|
|
222
|
+
runs-on: ubuntu-latest
|
|
223
|
+
steps:
|
|
224
|
+
- uses: actions/checkout@v4
|
|
225
|
+
- run: bundle install
|
|
226
|
+
- run: |
|
|
227
|
+
specbandit push \
|
|
228
|
+
--key "pr-${{ github.event.number }}-${{ github.run_id }}" \
|
|
229
|
+
--redis-url "${{ secrets.REDIS_URL }}" \
|
|
230
|
+
--pattern 'spec/**/*_spec.rb'
|
|
231
|
+
|
|
232
|
+
run-specs:
|
|
233
|
+
runs-on: ubuntu-latest
|
|
234
|
+
needs: push-specs
|
|
235
|
+
strategy:
|
|
236
|
+
matrix:
|
|
237
|
+
runner: [1, 2, 3, 4]
|
|
238
|
+
steps:
|
|
239
|
+
- uses: actions/checkout@v4
|
|
240
|
+
- run: bundle install
|
|
241
|
+
- run: |
|
|
242
|
+
specbandit work \
|
|
243
|
+
--key "pr-${{ github.event.number }}-${{ github.run_id }}" \
|
|
244
|
+
--redis-url "${{ secrets.REDIS_URL }}" \
|
|
245
|
+
--adapter rspec \
|
|
246
|
+
--batch-size 10
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
### Using CLI adapter (any test runner)
|
|
250
|
+
|
|
129
251
|
```yaml
|
|
130
252
|
jobs:
|
|
131
253
|
push-specs:
|
|
@@ -152,6 +274,7 @@ jobs:
|
|
|
152
274
|
specbandit work \
|
|
153
275
|
--key "pr-${{ github.event.number }}-${{ github.run_id }}" \
|
|
154
276
|
--redis-url "${{ secrets.REDIS_URL }}" \
|
|
277
|
+
--command "bundle exec rspec" \
|
|
155
278
|
--batch-size 10
|
|
156
279
|
```
|
|
157
280
|
|
|
@@ -250,6 +373,7 @@ jobs:
|
|
|
250
373
|
--key "pr-${{ github.event.number }}-${{ github.run_id }}" \
|
|
251
374
|
--key-rerun "pr-${{ github.event.number }}-${{ github.run_id }}-runner-${{ matrix.runner }}" \
|
|
252
375
|
--redis-url "${{ secrets.REDIS_URL }}" \
|
|
376
|
+
--adapter rspec \
|
|
253
377
|
--batch-size 10
|
|
254
378
|
```
|
|
255
379
|
|
|
@@ -297,7 +421,9 @@ specbandit work \
|
|
|
297
421
|
- **Steal** uses `LPOP key count` (Redis 6.2+), which atomically pops up to N elements. No Lua scripts, no locks, no race conditions.
|
|
298
422
|
- **Record** (when `--key-rerun` is set): after each steal, the batch is also `RPUSH`ed to the per-runner rerun key with its own TTL (default: 1 week).
|
|
299
423
|
- **Replay** (when `--key-rerun` has data): reads all files from the rerun key via `LRANGE` (non-destructive), splits into batches, and runs them locally. The shared queue is never touched.
|
|
300
|
-
- **Run**
|
|
424
|
+
- **Run** delegates to the configured adapter:
|
|
425
|
+
- **CLI adapter**: spawns a shell command per batch via `Open3`, appending file paths as arguments. Works with any test runner.
|
|
426
|
+
- **RSpec adapter**: uses `RSpec::Core::Runner.run` in-process with `RSpec.clear_examples` between batches to reset example state while preserving configuration. No subprocess forking overhead.
|
|
301
427
|
- **Exit code** is 0 if every batch passed (or the queue was already empty), 1 if any batch had failures.
|
|
302
428
|
|
|
303
429
|
## Development
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Specbandit
|
|
4
|
+
# Result of running a single batch of test files.
|
|
5
|
+
# Adapters return instances of this (or a subclass) from #run_batch.
|
|
6
|
+
BatchResult = Struct.new(:batch_num, :files, :exit_code, :duration, keyword_init: true)
|
|
7
|
+
|
|
8
|
+
# Adapter interface for executing test batches.
|
|
9
|
+
#
|
|
10
|
+
# specbandit supports pluggable execution strategies:
|
|
11
|
+
# - CliAdapter: spawns a shell command for each batch (works with any test runner)
|
|
12
|
+
# - RspecAdapter: runs RSpec programmatically in-process (maximum performance)
|
|
13
|
+
#
|
|
14
|
+
# To implement a custom adapter, define a class that responds to:
|
|
15
|
+
# #setup - One-time initialization before any batches run
|
|
16
|
+
# #run_batch(files, batch_num) - Execute a batch, return a BatchResult
|
|
17
|
+
# #teardown - Cleanup after all batches are done
|
|
18
|
+
module Adapter
|
|
19
|
+
end
|
|
20
|
+
end
|
data/lib/specbandit/cli.rb
CHANGED
|
@@ -85,12 +85,24 @@ module Specbandit
|
|
|
85
85
|
|
|
86
86
|
def run_work
|
|
87
87
|
parser = OptionParser.new do |opts|
|
|
88
|
-
opts.banner = 'Usage: specbandit work [options]'
|
|
88
|
+
opts.banner = 'Usage: specbandit work [options] [-- extra-opts...]'
|
|
89
89
|
|
|
90
90
|
opts.on('--key KEY', 'Redis queue key (required, or set SPECBANDIT_KEY)') do |v|
|
|
91
91
|
Specbandit.configuration.key = v
|
|
92
92
|
end
|
|
93
93
|
|
|
94
|
+
opts.on('--adapter TYPE', 'Adapter type: cli (default) or rspec') do |v|
|
|
95
|
+
Specbandit.configuration.adapter = v
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
opts.on('--command CMD', 'Command to run with file paths (required for cli adapter)') do |v|
|
|
99
|
+
Specbandit.configuration.command = v
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
opts.on('--command-opts OPTS', 'Extra options forwarded to the command (space-separated)') do |v|
|
|
103
|
+
Specbandit.configuration.command_opts = v.split
|
|
104
|
+
end
|
|
105
|
+
|
|
94
106
|
opts.on('--batch-size N', Integer, 'Number of files to steal per batch (default: 5)') do |v|
|
|
95
107
|
Specbandit.configuration.batch_size = v
|
|
96
108
|
end
|
|
@@ -99,7 +111,7 @@ module Specbandit
|
|
|
99
111
|
Specbandit.configuration.redis_url = v
|
|
100
112
|
end
|
|
101
113
|
|
|
102
|
-
opts.on('--rspec-opts OPTS', 'Extra options to pass to RSpec (space-separated)') do |v|
|
|
114
|
+
opts.on('--rspec-opts OPTS', 'Extra options to pass to RSpec (for rspec adapter, space-separated)') do |v|
|
|
103
115
|
Specbandit.configuration.rspec_opts = v.split
|
|
104
116
|
end
|
|
105
117
|
|
|
@@ -111,7 +123,7 @@ module Specbandit
|
|
|
111
123
|
Specbandit.configuration.key_rerun_ttl = v
|
|
112
124
|
end
|
|
113
125
|
|
|
114
|
-
opts.on('--verbose', 'Show per-batch file list and full
|
|
126
|
+
opts.on('--verbose', 'Show per-batch file list and full command output (default: quiet)') do
|
|
115
127
|
Specbandit.configuration.verbose = true
|
|
116
128
|
end
|
|
117
129
|
|
|
@@ -122,46 +134,91 @@ module Specbandit
|
|
|
122
134
|
end
|
|
123
135
|
|
|
124
136
|
parser.parse!(argv)
|
|
125
|
-
Specbandit.configuration.validate!
|
|
126
137
|
|
|
127
|
-
# Remaining args after `--` are
|
|
128
|
-
#
|
|
129
|
-
|
|
138
|
+
# Remaining args after `--` are forwarded to the adapter.
|
|
139
|
+
# They are merged with --command-opts or --rspec-opts depending on the adapter.
|
|
140
|
+
extra_opts = argv.any? ? argv : []
|
|
141
|
+
|
|
142
|
+
config = Specbandit.configuration
|
|
143
|
+
adapter = build_adapter(config, extra_opts)
|
|
130
144
|
|
|
131
|
-
|
|
145
|
+
config.validate!
|
|
146
|
+
|
|
147
|
+
worker = Worker.new(adapter: adapter)
|
|
132
148
|
worker.run
|
|
133
149
|
end
|
|
134
150
|
|
|
151
|
+
# Build the appropriate adapter based on configuration.
|
|
152
|
+
#
|
|
153
|
+
# --adapter rspec -> RspecAdapter (runs RSpec programmatically in-process)
|
|
154
|
+
# --adapter cli -> CliAdapter (default, spawns shell commands)
|
|
155
|
+
# (no --adapter) -> CliAdapter (backward compatible with specbanditjs)
|
|
156
|
+
def build_adapter(config, extra_opts)
|
|
157
|
+
adapter_type = config.adapter.downcase
|
|
158
|
+
|
|
159
|
+
case adapter_type
|
|
160
|
+
when 'rspec'
|
|
161
|
+
rspec_opts = config.rspec_opts + extra_opts
|
|
162
|
+
RspecAdapter.new(
|
|
163
|
+
rspec_opts: rspec_opts,
|
|
164
|
+
verbose: config.verbose,
|
|
165
|
+
output: $stdout
|
|
166
|
+
)
|
|
167
|
+
when 'cli'
|
|
168
|
+
unless config.command
|
|
169
|
+
raise Error, 'command is required for CLI adapter (set via --command or SPECBANDIT_COMMAND)'
|
|
170
|
+
end
|
|
171
|
+
|
|
172
|
+
command_opts = config.command_opts + extra_opts
|
|
173
|
+
CliAdapter.new(
|
|
174
|
+
command: config.command,
|
|
175
|
+
command_opts: command_opts,
|
|
176
|
+
verbose: config.verbose,
|
|
177
|
+
output: $stdout
|
|
178
|
+
)
|
|
179
|
+
else
|
|
180
|
+
raise Error, "Unknown adapter: #{adapter_type}. Supported: cli, rspec"
|
|
181
|
+
end
|
|
182
|
+
end
|
|
183
|
+
|
|
135
184
|
def print_usage
|
|
136
185
|
puts <<~USAGE
|
|
137
|
-
specbandit v#{VERSION} - Distributed
|
|
186
|
+
specbandit v#{VERSION} - Distributed test runner using Redis
|
|
138
187
|
|
|
139
188
|
Usage:
|
|
140
|
-
specbandit push [options] [files...]
|
|
141
|
-
specbandit work [options]
|
|
189
|
+
specbandit push [options] [files...] Enqueue test files into Redis
|
|
190
|
+
specbandit work [options] [-- extra-opts...] Steal and run test file batches
|
|
142
191
|
|
|
143
192
|
Push options:
|
|
144
|
-
--key KEY
|
|
145
|
-
--pattern PATTERN
|
|
146
|
-
--redis-url URL
|
|
147
|
-
--key-ttl SECONDS
|
|
193
|
+
--key KEY Redis queue key (required, or set SPECBANDIT_KEY)
|
|
194
|
+
--pattern PATTERN Glob pattern for file discovery (e.g. 'spec/**/*_spec.rb')
|
|
195
|
+
--redis-url URL Redis URL (default: redis://localhost:6379)
|
|
196
|
+
--key-ttl SECONDS TTL for the Redis key (default: 21600 / 6 hours)
|
|
148
197
|
|
|
149
198
|
Work options:
|
|
150
|
-
--key KEY
|
|
151
|
-
--
|
|
152
|
-
--
|
|
153
|
-
--
|
|
154
|
-
--
|
|
155
|
-
--
|
|
156
|
-
--
|
|
157
|
-
--
|
|
199
|
+
--key KEY Redis queue key (required, or set SPECBANDIT_KEY)
|
|
200
|
+
--adapter TYPE Adapter type: 'cli' (default) or 'rspec'
|
|
201
|
+
--command CMD Command to run with file paths (required for cli adapter)
|
|
202
|
+
--command-opts OPTS Extra options forwarded to the command (space-separated)
|
|
203
|
+
--rspec-opts OPTS Extra options forwarded to RSpec (for rspec adapter)
|
|
204
|
+
--batch-size N Files per batch (default: 5, or set SPECBANDIT_BATCH_SIZE)
|
|
205
|
+
--redis-url URL Redis URL (default: redis://localhost:6379)
|
|
206
|
+
--key-rerun KEY Per-runner rerun key for re-run support
|
|
207
|
+
--key-rerun-ttl N TTL for rerun key (default: 604800 / 1 week)
|
|
208
|
+
--verbose Show per-batch file list and full command output
|
|
209
|
+
|
|
210
|
+
Arguments after -- are forwarded to the adapter (rspec opts, command opts, etc.).
|
|
211
|
+
They are merged with --command-opts or --rspec-opts if both are provided.
|
|
158
212
|
|
|
159
213
|
Environment variables:
|
|
160
214
|
SPECBANDIT_KEY Queue key
|
|
161
215
|
SPECBANDIT_REDIS_URL Redis URL
|
|
216
|
+
SPECBANDIT_ADAPTER Adapter type (cli/rspec)
|
|
217
|
+
SPECBANDIT_COMMAND Command to run (cli adapter)
|
|
218
|
+
SPECBANDIT_COMMAND_OPTS Command options (space-separated)
|
|
162
219
|
SPECBANDIT_BATCH_SIZE Batch size
|
|
163
220
|
SPECBANDIT_KEY_TTL Key TTL in seconds (default: 21600)
|
|
164
|
-
SPECBANDIT_RSPEC_OPTS RSpec options
|
|
221
|
+
SPECBANDIT_RSPEC_OPTS RSpec options (rspec adapter)
|
|
165
222
|
SPECBANDIT_KEY_RERUN Per-runner rerun key
|
|
166
223
|
SPECBANDIT_KEY_RERUN_TTL Rerun key TTL in seconds (default: 604800)
|
|
167
224
|
SPECBANDIT_VERBOSE Enable verbose output (1/true/yes)
|
|
@@ -170,6 +227,12 @@ module Specbandit
|
|
|
170
227
|
1. stdin (piped) echo "spec/a_spec.rb" | specbandit push --key KEY
|
|
171
228
|
2. --pattern specbandit push --key KEY --pattern 'spec/**/*_spec.rb'
|
|
172
229
|
3. direct args specbandit push --key KEY spec/a_spec.rb spec/b_spec.rb
|
|
230
|
+
|
|
231
|
+
Adapters:
|
|
232
|
+
cli (default) Spawns a shell command for each batch. Works with any test runner.
|
|
233
|
+
Requires --command.
|
|
234
|
+
rspec Runs RSpec programmatically in-process. No process startup overhead per batch.
|
|
235
|
+
Requires rspec-core ~> 3.0.
|
|
173
236
|
USAGE
|
|
174
237
|
end
|
|
175
238
|
end
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'open3'
|
|
4
|
+
|
|
5
|
+
module Specbandit
|
|
6
|
+
# CLI adapter: spawns a shell command for each batch.
|
|
7
|
+
#
|
|
8
|
+
# Works with any test runner. The command string is split on whitespace,
|
|
9
|
+
# and file paths are appended as arguments:
|
|
10
|
+
#
|
|
11
|
+
# <executable> [...command_args] [...command_opts] [...file_paths]
|
|
12
|
+
#
|
|
13
|
+
# Example: command="bundle exec rspec", command_opts=["--format", "documentation"]
|
|
14
|
+
# -> system("bundle", "exec", "rspec", "--format", "documentation", "file1.rb", "file2.rb")
|
|
15
|
+
class CliAdapter
|
|
16
|
+
include Adapter
|
|
17
|
+
|
|
18
|
+
attr_reader :command, :command_opts, :verbose, :output
|
|
19
|
+
|
|
20
|
+
def initialize(command:, command_opts: [], verbose: false, output: $stdout)
|
|
21
|
+
@command = command
|
|
22
|
+
@command_opts = Array(command_opts)
|
|
23
|
+
@verbose = verbose
|
|
24
|
+
@output = output
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
# No-op for CLI adapter.
|
|
28
|
+
def setup; end
|
|
29
|
+
|
|
30
|
+
# Spawn the command with file paths appended as arguments.
|
|
31
|
+
# Returns a BatchResult with the exit code and timing.
|
|
32
|
+
def run_batch(files, batch_num)
|
|
33
|
+
command_parts = command.split(/\s+/)
|
|
34
|
+
args = command_parts + command_opts + files
|
|
35
|
+
|
|
36
|
+
start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
37
|
+
|
|
38
|
+
if verbose
|
|
39
|
+
# Inherit stdio so user sees output in real-time
|
|
40
|
+
system(*args)
|
|
41
|
+
exit_code = $?.exitstatus || 1
|
|
42
|
+
else
|
|
43
|
+
stdout, stderr, status = Open3.capture3(*args)
|
|
44
|
+
exit_code = status.exitstatus || 1
|
|
45
|
+
|
|
46
|
+
# Print stderr on failure
|
|
47
|
+
output.puts stderr if exit_code != 0 && stderr && !stderr.strip.empty?
|
|
48
|
+
|
|
49
|
+
# Print stdout if any
|
|
50
|
+
output.print(stdout) if stdout && !stdout.empty?
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time
|
|
54
|
+
|
|
55
|
+
BatchResult.new(
|
|
56
|
+
batch_num: batch_num,
|
|
57
|
+
files: files,
|
|
58
|
+
exit_code: exit_code,
|
|
59
|
+
duration: duration
|
|
60
|
+
)
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# No-op for CLI adapter.
|
|
64
|
+
def teardown; end
|
|
65
|
+
end
|
|
66
|
+
end
|
|
@@ -3,12 +3,14 @@
|
|
|
3
3
|
module Specbandit
|
|
4
4
|
class Configuration
|
|
5
5
|
attr_accessor :redis_url, :batch_size, :key, :rspec_opts, :key_ttl,
|
|
6
|
-
:key_rerun, :key_rerun_ttl, :verbose
|
|
6
|
+
:key_rerun, :key_rerun_ttl, :verbose,
|
|
7
|
+
:adapter, :command, :command_opts
|
|
7
8
|
|
|
8
9
|
DEFAULT_REDIS_URL = 'redis://localhost:6379'
|
|
9
10
|
DEFAULT_BATCH_SIZE = 5
|
|
10
11
|
DEFAULT_KEY_TTL = 21_600 # 6 hours in seconds
|
|
11
12
|
DEFAULT_KEY_RERUN_TTL = 604_800 # 1 week in seconds
|
|
13
|
+
DEFAULT_ADAPTER = 'cli'
|
|
12
14
|
|
|
13
15
|
def initialize
|
|
14
16
|
@redis_url = ENV.fetch('SPECBANDIT_REDIS_URL', DEFAULT_REDIS_URL)
|
|
@@ -19,6 +21,9 @@ module Specbandit
|
|
|
19
21
|
@key_rerun = ENV.fetch('SPECBANDIT_KEY_RERUN', nil)
|
|
20
22
|
@key_rerun_ttl = Integer(ENV.fetch('SPECBANDIT_KEY_RERUN_TTL', DEFAULT_KEY_RERUN_TTL))
|
|
21
23
|
@verbose = env_truthy?('SPECBANDIT_VERBOSE')
|
|
24
|
+
@adapter = ENV.fetch('SPECBANDIT_ADAPTER', DEFAULT_ADAPTER)
|
|
25
|
+
@command = ENV.fetch('SPECBANDIT_COMMAND', nil)
|
|
26
|
+
@command_opts = parse_space_separated(ENV.fetch('SPECBANDIT_COMMAND_OPTS', nil))
|
|
22
27
|
end
|
|
23
28
|
|
|
24
29
|
def validate!
|
|
@@ -36,6 +41,12 @@ module Specbandit
|
|
|
36
41
|
opts.split
|
|
37
42
|
end
|
|
38
43
|
|
|
44
|
+
def parse_space_separated(value)
|
|
45
|
+
return [] if value.nil? || value.empty?
|
|
46
|
+
|
|
47
|
+
value.split
|
|
48
|
+
end
|
|
49
|
+
|
|
39
50
|
def env_truthy?(name)
|
|
40
51
|
%w[1 true yes].include?(ENV.fetch(name, '').downcase)
|
|
41
52
|
end
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'stringio'
|
|
4
|
+
require 'json'
|
|
5
|
+
require 'tempfile'
|
|
6
|
+
require 'rspec/core'
|
|
7
|
+
|
|
8
|
+
module Specbandit
|
|
9
|
+
# RSpec-specific batch result that extends BatchResult with the path
|
|
10
|
+
# to the JSON output file. The Worker uses this to accumulate
|
|
11
|
+
# per-example results for rich reporting (failed spec details, etc.).
|
|
12
|
+
class RspecBatchResult < BatchResult
|
|
13
|
+
attr_accessor :json_path
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
# RSpec adapter: runs RSpec programmatically in-process.
|
|
17
|
+
#
|
|
18
|
+
# Each batch calls RSpec::Core::Runner.run with careful state cleanup
|
|
19
|
+
# between batches. This avoids process startup overhead and is the
|
|
20
|
+
# fastest way to run RSpec batches in a single process.
|
|
21
|
+
#
|
|
22
|
+
# The adapter injects a JSON formatter writing to a tempfile so the
|
|
23
|
+
# Worker can accumulate structured results. User-provided rspec_opts
|
|
24
|
+
# (e.g. --format documentation) are preserved and prepended.
|
|
25
|
+
class RspecAdapter
|
|
26
|
+
include Adapter
|
|
27
|
+
|
|
28
|
+
attr_reader :rspec_opts, :verbose, :output
|
|
29
|
+
|
|
30
|
+
def initialize(rspec_opts: [], verbose: false, output: $stdout)
|
|
31
|
+
@rspec_opts = Array(rspec_opts)
|
|
32
|
+
@verbose = verbose
|
|
33
|
+
@output = output
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# No-op for RSpec adapter — RSpec is already loaded.
|
|
37
|
+
def setup; end
|
|
38
|
+
|
|
39
|
+
# Run a batch of spec files via RSpec::Core::Runner.run.
|
|
40
|
+
#
|
|
41
|
+
# Returns an RspecBatchResult with exit_code, duration, and json_path
|
|
42
|
+
# pointing to a tempfile containing the JSON output. The caller is
|
|
43
|
+
# responsible for reading and cleaning up the tempfile.
|
|
44
|
+
def run_batch(files, batch_num)
|
|
45
|
+
reset_rspec_state
|
|
46
|
+
|
|
47
|
+
batch_json = Tempfile.new(['specbandit-batch', '.json'])
|
|
48
|
+
args = files + rspec_opts + ['--format', 'json', '--out', batch_json.path]
|
|
49
|
+
|
|
50
|
+
err = StringIO.new
|
|
51
|
+
out = StringIO.new
|
|
52
|
+
|
|
53
|
+
start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
54
|
+
exit_code = RSpec::Core::Runner.run(args, err, out)
|
|
55
|
+
duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time
|
|
56
|
+
|
|
57
|
+
result = RspecBatchResult.new(
|
|
58
|
+
batch_num: batch_num,
|
|
59
|
+
files: files,
|
|
60
|
+
exit_code: exit_code,
|
|
61
|
+
duration: duration
|
|
62
|
+
)
|
|
63
|
+
result.json_path = batch_json.path
|
|
64
|
+
result
|
|
65
|
+
ensure
|
|
66
|
+
# Print RSpec output through our output stream
|
|
67
|
+
rspec_output = out&.string
|
|
68
|
+
output.print(rspec_output) if verbose && rspec_output && !rspec_output.empty?
|
|
69
|
+
|
|
70
|
+
rspec_err = err&.string
|
|
71
|
+
output.print(rspec_err) if verbose && rspec_err && !rspec_err.empty?
|
|
72
|
+
|
|
73
|
+
# Don't unlink the tempfile here — the Worker needs to read it.
|
|
74
|
+
# The Worker is responsible for cleanup after accumulation.
|
|
75
|
+
batch_json&.close
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# No-op for RSpec adapter.
|
|
79
|
+
def teardown; end
|
|
80
|
+
|
|
81
|
+
private
|
|
82
|
+
|
|
83
|
+
# Reset RSpec state between batches so each batch runs cleanly.
|
|
84
|
+
#
|
|
85
|
+
# RSpec.clear_examples resets example groups, the reporter, filters, and
|
|
86
|
+
# the start-time clock -- but it leaves three critical pieces of state
|
|
87
|
+
# that cause cascading failures when running multiple batches in the
|
|
88
|
+
# same process:
|
|
89
|
+
#
|
|
90
|
+
# 1. output_stream -- After batch #1, Runner#configure sets
|
|
91
|
+
# output_stream to a StringIO. On batch #2+, the guard
|
|
92
|
+
# `if output_stream == $stdout` is permanently false, so the new
|
|
93
|
+
# `out` is never used. All RSpec output silently goes to the stale
|
|
94
|
+
# batch-1 StringIO.
|
|
95
|
+
#
|
|
96
|
+
# 2. wants_to_quit -- If any batch triggers a load error or fail-fast,
|
|
97
|
+
# this flag is set to true. On subsequent batches, Runner#setup
|
|
98
|
+
# returns immediately and Runner#run does exit_early -- specs are
|
|
99
|
+
# never loaded or run.
|
|
100
|
+
#
|
|
101
|
+
# 3. non_example_failure -- Once set, exit_code() unconditionally
|
|
102
|
+
# returns the failure exit code, even if all examples passed.
|
|
103
|
+
#
|
|
104
|
+
def reset_rspec_state
|
|
105
|
+
RSpec.clear_examples
|
|
106
|
+
RSpec.world.wants_to_quit = false
|
|
107
|
+
RSpec.world.non_example_failure = false
|
|
108
|
+
RSpec.configuration.output_stream = $stdout
|
|
109
|
+
end
|
|
110
|
+
end
|
|
111
|
+
end
|
data/lib/specbandit/version.rb
CHANGED
data/lib/specbandit/worker.rb
CHANGED
|
@@ -2,41 +2,51 @@
|
|
|
2
2
|
|
|
3
3
|
require 'stringio'
|
|
4
4
|
require 'json'
|
|
5
|
-
require 'tempfile'
|
|
6
|
-
require 'rspec/core'
|
|
7
5
|
|
|
8
6
|
module Specbandit
|
|
9
7
|
class Worker
|
|
10
|
-
attr_reader :queue, :key, :batch_size, :
|
|
8
|
+
attr_reader :queue, :key, :batch_size, :adapter, :key_rerun, :key_rerun_ttl, :output, :verbose
|
|
11
9
|
|
|
12
10
|
def initialize(
|
|
13
11
|
key: Specbandit.configuration.key,
|
|
14
12
|
batch_size: Specbandit.configuration.batch_size,
|
|
15
|
-
|
|
13
|
+
adapter: nil,
|
|
16
14
|
key_rerun: Specbandit.configuration.key_rerun,
|
|
17
15
|
key_rerun_ttl: Specbandit.configuration.key_rerun_ttl,
|
|
18
16
|
verbose: Specbandit.configuration.verbose,
|
|
19
17
|
queue: nil,
|
|
20
|
-
output: $stdout
|
|
18
|
+
output: $stdout,
|
|
19
|
+
# Legacy parameter for backward compatibility.
|
|
20
|
+
# When adapter is not provided, rspec_opts is used to build an RspecAdapter.
|
|
21
|
+
rspec_opts: nil
|
|
21
22
|
)
|
|
22
23
|
@key = key
|
|
23
24
|
@batch_size = batch_size
|
|
24
|
-
@rspec_opts = Array(rspec_opts)
|
|
25
25
|
@key_rerun = key_rerun
|
|
26
26
|
@key_rerun_ttl = key_rerun_ttl
|
|
27
27
|
@verbose = verbose
|
|
28
28
|
@queue = queue || RedisQueue.new
|
|
29
29
|
@output = output
|
|
30
|
-
@
|
|
30
|
+
@batch_results = []
|
|
31
31
|
@accumulated_examples = []
|
|
32
32
|
@accumulated_summary = { duration: 0.0, example_count: 0, failure_count: 0, pending_count: 0,
|
|
33
33
|
errors_outside_of_examples_count: 0 }
|
|
34
|
+
|
|
35
|
+
# Support both new adapter-based and legacy rspec_opts-based construction.
|
|
36
|
+
# If no adapter is provided, fall back to RspecAdapter for backward compatibility.
|
|
37
|
+
@adapter = adapter || RspecAdapter.new(
|
|
38
|
+
rspec_opts: rspec_opts || Specbandit.configuration.rspec_opts,
|
|
39
|
+
verbose: verbose,
|
|
40
|
+
output: output
|
|
41
|
+
)
|
|
34
42
|
end
|
|
35
43
|
|
|
36
44
|
# Main entry point. Detects the operating mode and dispatches accordingly.
|
|
37
45
|
#
|
|
38
46
|
# Returns 0 if all batches passed (or nothing to do), 1 if any batch failed.
|
|
39
47
|
def run
|
|
48
|
+
adapter.setup
|
|
49
|
+
|
|
40
50
|
exit_code = if key_rerun
|
|
41
51
|
rerun_files = queue.read_all(key_rerun)
|
|
42
52
|
if rerun_files.any?
|
|
@@ -48,11 +58,13 @@ module Specbandit
|
|
|
48
58
|
run_steal(record: false)
|
|
49
59
|
end
|
|
50
60
|
|
|
51
|
-
print_summary if @
|
|
61
|
+
print_summary if @batch_results.any?
|
|
52
62
|
merge_json_results
|
|
53
63
|
write_github_step_summary if ENV['GITHUB_STEP_SUMMARY']
|
|
54
64
|
|
|
55
65
|
exit_code
|
|
66
|
+
ensure
|
|
67
|
+
adapter.teardown
|
|
56
68
|
end
|
|
57
69
|
|
|
58
70
|
private
|
|
@@ -72,9 +84,11 @@ module Specbandit
|
|
|
72
84
|
output.puts "[specbandit] Batch ##{batch_num}: running #{batch.size} files"
|
|
73
85
|
batch.each { |f| output.puts " #{f}" } if verbose
|
|
74
86
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
87
|
+
result = adapter.run_batch(batch, batch_num)
|
|
88
|
+
process_batch_result(result)
|
|
89
|
+
|
|
90
|
+
if result.exit_code != 0
|
|
91
|
+
output.puts "[specbandit] Batch ##{batch_num} FAILED (exit code: #{result.exit_code})"
|
|
78
92
|
failed = true
|
|
79
93
|
else
|
|
80
94
|
output.puts "[specbandit] Batch ##{batch_num} passed."
|
|
@@ -111,9 +125,11 @@ module Specbandit
|
|
|
111
125
|
output.puts "[specbandit] Batch ##{batch_num}: running #{files.size} files"
|
|
112
126
|
files.each { |f| output.puts " #{f}" } if verbose
|
|
113
127
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
128
|
+
result = adapter.run_batch(files, batch_num)
|
|
129
|
+
process_batch_result(result)
|
|
130
|
+
|
|
131
|
+
if result.exit_code != 0
|
|
132
|
+
output.puts "[specbandit] Batch ##{batch_num} FAILED (exit code: #{result.exit_code})"
|
|
117
133
|
failed = true
|
|
118
134
|
else
|
|
119
135
|
output.puts "[specbandit] Batch ##{batch_num} passed."
|
|
@@ -129,71 +145,42 @@ module Specbandit
|
|
|
129
145
|
failed ? 1 : 0
|
|
130
146
|
end
|
|
131
147
|
|
|
132
|
-
|
|
133
|
-
|
|
148
|
+
# Process a BatchResult: store it, and for RSpec batches,
|
|
149
|
+
# read the JSON output for rich reporting.
|
|
150
|
+
def process_batch_result(result)
|
|
151
|
+
@batch_results << result
|
|
134
152
|
|
|
135
|
-
#
|
|
136
|
-
#
|
|
137
|
-
|
|
138
|
-
args = files + rspec_opts + ['--format', 'json', '--out', batch_json.path]
|
|
153
|
+
# If the adapter returned an RspecBatchResult with a json_path,
|
|
154
|
+
# accumulate the structured results for rich reporting.
|
|
155
|
+
return unless result.is_a?(RspecBatchResult) && result.json_path
|
|
139
156
|
|
|
140
|
-
|
|
141
|
-
out = StringIO.new
|
|
157
|
+
accumulate_json_results(result.json_path)
|
|
142
158
|
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
exit_code
|
|
150
|
-
ensure
|
|
151
|
-
# Print RSpec output through our output stream
|
|
152
|
-
rspec_output = out&.string
|
|
153
|
-
output.print(rspec_output) if verbose && rspec_output && !rspec_output.empty?
|
|
159
|
+
# Clean up the tempfile now that we've read it
|
|
160
|
+
File.delete(result.json_path) if File.exist?(result.json_path)
|
|
161
|
+
rescue StandardError
|
|
162
|
+
# Never fail because of tempfile cleanup
|
|
163
|
+
nil
|
|
164
|
+
end
|
|
154
165
|
|
|
155
|
-
|
|
156
|
-
output.print(rspec_err) if verbose && rspec_err && !rspec_err.empty?
|
|
166
|
+
# --- Reporting helpers ---
|
|
157
167
|
|
|
158
|
-
|
|
159
|
-
|
|
168
|
+
def batch_durations
|
|
169
|
+
@batch_results.map(&:duration)
|
|
160
170
|
end
|
|
161
171
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
# RSpec.clear_examples resets example groups, the reporter, filters, and
|
|
165
|
-
# the start-time clock -- but it leaves three critical pieces of state
|
|
166
|
-
# that cause cascading failures when running multiple batches in the
|
|
167
|
-
# same process:
|
|
168
|
-
#
|
|
169
|
-
# 1. output_stream -- After batch #1, Runner#configure sets
|
|
170
|
-
# output_stream to a StringIO. On batch #2+, the guard
|
|
171
|
-
# `if output_stream == $stdout` is permanently false, so the new
|
|
172
|
-
# `out` is never used. All RSpec output silently goes to the stale
|
|
173
|
-
# batch-1 StringIO.
|
|
174
|
-
#
|
|
175
|
-
# 2. wants_to_quit -- If any batch triggers a load error or fail-fast,
|
|
176
|
-
# this flag is set to true. On subsequent batches, Runner#setup
|
|
177
|
-
# returns immediately and Runner#run does exit_early -- specs are
|
|
178
|
-
# never loaded or run.
|
|
179
|
-
#
|
|
180
|
-
# 3. non_example_failure -- Once set, exit_code() unconditionally
|
|
181
|
-
# returns the failure exit code, even if all examples passed.
|
|
182
|
-
#
|
|
183
|
-
def reset_rspec_state
|
|
184
|
-
RSpec.clear_examples
|
|
185
|
-
RSpec.world.wants_to_quit = false
|
|
186
|
-
RSpec.world.non_example_failure = false
|
|
187
|
-
RSpec.configuration.output_stream = $stdout
|
|
172
|
+
def has_rspec_results?
|
|
173
|
+
@accumulated_examples.any? || @accumulated_summary[:example_count] > 0
|
|
188
174
|
end
|
|
189
175
|
|
|
190
|
-
#
|
|
191
|
-
|
|
192
|
-
# Extract the --out file path from rspec_opts.
|
|
176
|
+
# Extract the --out file path from rspec_opts (when using RspecAdapter).
|
|
193
177
|
# RSpec accepts: --out FILE or -o FILE
|
|
194
178
|
def json_output_path
|
|
195
|
-
|
|
196
|
-
|
|
179
|
+
return nil unless adapter.is_a?(RspecAdapter)
|
|
180
|
+
|
|
181
|
+
opts = adapter.rspec_opts
|
|
182
|
+
opts.each_with_index do |opt, i|
|
|
183
|
+
return opts[i + 1] if ['--out', '-o'].include?(opt) && opts[i + 1]
|
|
197
184
|
end
|
|
198
185
|
nil
|
|
199
186
|
end
|
|
@@ -238,11 +225,11 @@ module Specbandit
|
|
|
238
225
|
'summary_line' => summary_line,
|
|
239
226
|
'examples' => @accumulated_examples,
|
|
240
227
|
'batch_timings' => {
|
|
241
|
-
'count' =>
|
|
242
|
-
'min' =>
|
|
243
|
-
'avg' =>
|
|
244
|
-
'max' =>
|
|
245
|
-
'all' =>
|
|
228
|
+
'count' => batch_durations.size,
|
|
229
|
+
'min' => batch_durations.min&.round(2),
|
|
230
|
+
'avg' => batch_durations.empty? ? 0 : (batch_durations.sum / batch_durations.size).round(2),
|
|
231
|
+
'max' => batch_durations.max&.round(2),
|
|
232
|
+
'all' => batch_durations.map { |d| d.round(2) }
|
|
246
233
|
}
|
|
247
234
|
}
|
|
248
235
|
|
|
@@ -255,33 +242,54 @@ module Specbandit
|
|
|
255
242
|
output.puts '=' * 60
|
|
256
243
|
output.puts '[specbandit] Summary'
|
|
257
244
|
output.puts '=' * 60
|
|
258
|
-
output.puts " Batches: #{
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
245
|
+
output.puts " Batches: #{batch_durations.size}"
|
|
246
|
+
|
|
247
|
+
if has_rspec_results?
|
|
248
|
+
# Rich RSpec-specific summary
|
|
249
|
+
output.puts " Examples: #{@accumulated_summary[:example_count]}"
|
|
250
|
+
output.puts " Failures: #{@accumulated_summary[:failure_count]}"
|
|
251
|
+
output.puts " Pending: #{@accumulated_summary[:pending_count]}"
|
|
252
|
+
else
|
|
253
|
+
# Generic batch-level summary (CLI adapter or no JSON data)
|
|
254
|
+
total_files = @batch_results.sum { |r| r.files.size }
|
|
255
|
+
failed_batches = @batch_results.count { |r| r.exit_code != 0 }
|
|
256
|
+
output.puts " Files: #{total_files}"
|
|
257
|
+
output.puts " Failed batches: #{failed_batches}"
|
|
258
|
+
end
|
|
262
259
|
|
|
263
260
|
output.puts ''
|
|
264
261
|
output.puts format(
|
|
265
262
|
' Batch timing: min %.1fs | avg %.1fs | max %.1fs',
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
263
|
+
batch_durations.min || 0,
|
|
264
|
+
batch_durations.empty? ? 0 : batch_durations.sum / batch_durations.size,
|
|
265
|
+
batch_durations.max || 0
|
|
269
266
|
)
|
|
270
267
|
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
268
|
+
if has_rspec_results?
|
|
269
|
+
failed_examples = @accumulated_examples.select { |e| e['status'] == 'failed' }
|
|
270
|
+
if failed_examples.any?
|
|
271
|
+
output.puts ''
|
|
272
|
+
output.puts " Failed specs (#{failed_examples.size}):"
|
|
273
|
+
failed_examples.each do |ex|
|
|
274
|
+
location = ex.dig('file_path') || 'unknown'
|
|
275
|
+
line = ex.dig('line_number')
|
|
276
|
+
location = "#{location}:#{line}" if line
|
|
277
|
+
desc = ex.dig('full_description') || ex.dig('description') || ''
|
|
278
|
+
message = ex.dig('exception', 'message') || ''
|
|
279
|
+
# Truncate long messages
|
|
280
|
+
message = "#{message[0, 120]}..." if message.length > 120
|
|
281
|
+
output.puts " #{location} - #{desc}"
|
|
282
|
+
output.puts " #{message}" unless message.empty?
|
|
283
|
+
end
|
|
284
|
+
end
|
|
285
|
+
else
|
|
286
|
+
failed_batch_results = @batch_results.select { |r| r.exit_code != 0 }
|
|
287
|
+
if failed_batch_results.any?
|
|
288
|
+
output.puts ''
|
|
289
|
+
output.puts " Failed batches (#{failed_batch_results.size}):"
|
|
290
|
+
failed_batch_results.each do |r|
|
|
291
|
+
output.puts " Batch ##{r.batch_num} (exit code #{r.exit_code}): #{r.files.join(', ')}"
|
|
292
|
+
end
|
|
285
293
|
end
|
|
286
294
|
end
|
|
287
295
|
|
|
@@ -302,44 +310,85 @@ module Specbandit
|
|
|
302
310
|
return unless path
|
|
303
311
|
|
|
304
312
|
md = StringIO.new
|
|
305
|
-
|
|
313
|
+
|
|
314
|
+
if has_rspec_results?
|
|
315
|
+
write_rspec_github_summary(md)
|
|
316
|
+
else
|
|
317
|
+
write_generic_github_summary(md)
|
|
318
|
+
end
|
|
319
|
+
|
|
320
|
+
File.open(path, 'a') { |f| f.write(md.string) }
|
|
321
|
+
rescue StandardError
|
|
322
|
+
# Never fail the build because of summary writing
|
|
323
|
+
nil
|
|
324
|
+
end
|
|
325
|
+
|
|
326
|
+
def write_rspec_github_summary(md)
|
|
327
|
+
md.puts '### Specbandit Results'
|
|
306
328
|
md.puts ''
|
|
307
329
|
md.puts '| Metric | Value |'
|
|
308
330
|
md.puts '|--------|-------|'
|
|
309
|
-
md.puts "| Batches | #{
|
|
331
|
+
md.puts "| Batches | #{batch_durations.size} |"
|
|
310
332
|
md.puts "| Examples | #{@accumulated_summary[:example_count]} |"
|
|
311
333
|
md.puts "| Failures | #{@accumulated_summary[:failure_count]} |"
|
|
312
334
|
md.puts "| Pending | #{@accumulated_summary[:pending_count]} |"
|
|
313
335
|
|
|
314
|
-
md.puts format('| Batch time (min) | %.1fs |',
|
|
336
|
+
md.puts format('| Batch time (min) | %.1fs |', batch_durations.min || 0)
|
|
315
337
|
md.puts format('| Batch time (avg) | %.1fs |',
|
|
316
|
-
|
|
317
|
-
md.puts format('| Batch time (max) | %.1fs |',
|
|
338
|
+
batch_durations.empty? ? 0 : batch_durations.sum / batch_durations.size)
|
|
339
|
+
md.puts format('| Batch time (max) | %.1fs |', batch_durations.max || 0)
|
|
318
340
|
md.puts ''
|
|
319
341
|
|
|
320
342
|
failed_examples = @accumulated_examples.select { |e| e['status'] == 'failed' }
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
md.puts ''
|
|
336
|
-
md.puts '</details>'
|
|
343
|
+
return unless failed_examples.any?
|
|
344
|
+
|
|
345
|
+
md.puts "<details><summary>#{failed_examples.size} failed specs</summary>"
|
|
346
|
+
md.puts ''
|
|
347
|
+
md.puts '| Location | Description | Error |'
|
|
348
|
+
md.puts '|----------|-------------|-------|'
|
|
349
|
+
failed_examples.each do |ex|
|
|
350
|
+
location = ex['file_path'] || 'unknown'
|
|
351
|
+
line = ex['line_number']
|
|
352
|
+
location = "#{location}:#{line}" if line
|
|
353
|
+
desc = (ex['full_description'] || ex['description'] || '').gsub('|', '\\|')
|
|
354
|
+
message = (ex.dig('exception', 'message') || '').gsub('|', '\\|').gsub("\n", ' ')
|
|
355
|
+
message = "#{message[0, 100]}..." if message.length > 100
|
|
356
|
+
md.puts "| `#{location}` | #{desc} | #{message} |"
|
|
337
357
|
end
|
|
358
|
+
md.puts ''
|
|
359
|
+
md.puts '</details>'
|
|
360
|
+
end
|
|
338
361
|
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
362
|
+
def write_generic_github_summary(md)
|
|
363
|
+
total_files = @batch_results.sum { |r| r.files.size }
|
|
364
|
+
failed_batch_results = @batch_results.select { |r| r.exit_code != 0 }
|
|
365
|
+
|
|
366
|
+
md.puts '### Specbandit Results'
|
|
367
|
+
md.puts ''
|
|
368
|
+
md.puts '| Metric | Value |'
|
|
369
|
+
md.puts '|--------|-------|'
|
|
370
|
+
md.puts "| Batches | #{batch_durations.size} |"
|
|
371
|
+
md.puts "| Files | #{total_files} |"
|
|
372
|
+
md.puts "| Failed batches | #{failed_batch_results.size} |"
|
|
373
|
+
|
|
374
|
+
md.puts format('| Batch time (min) | %.1fs |', batch_durations.min || 0)
|
|
375
|
+
md.puts format('| Batch time (avg) | %.1fs |',
|
|
376
|
+
batch_durations.empty? ? 0 : batch_durations.sum / batch_durations.size)
|
|
377
|
+
md.puts format('| Batch time (max) | %.1fs |', batch_durations.max || 0)
|
|
378
|
+
md.puts ''
|
|
379
|
+
|
|
380
|
+
return unless failed_batch_results.any?
|
|
381
|
+
|
|
382
|
+
md.puts "<details><summary>#{failed_batch_results.size} failed batches</summary>"
|
|
383
|
+
md.puts ''
|
|
384
|
+
md.puts '| Batch | Exit Code | Files |'
|
|
385
|
+
md.puts '|-------|-----------|-------|'
|
|
386
|
+
failed_batch_results.each do |r|
|
|
387
|
+
files_str = r.files.map { |f| "`#{f}`" }.join(', ')
|
|
388
|
+
md.puts "| ##{r.batch_num} | #{r.exit_code} | #{files_str} |"
|
|
389
|
+
end
|
|
390
|
+
md.puts ''
|
|
391
|
+
md.puts '</details>'
|
|
343
392
|
end
|
|
344
393
|
end
|
|
345
394
|
end
|
data/lib/specbandit.rb
CHANGED
|
@@ -2,6 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
require 'specbandit/version'
|
|
4
4
|
require 'specbandit/configuration'
|
|
5
|
+
require 'specbandit/adapter'
|
|
6
|
+
require 'specbandit/rspec_adapter'
|
|
7
|
+
require 'specbandit/cli_adapter'
|
|
5
8
|
require 'specbandit/redis_queue'
|
|
6
9
|
require 'specbandit/publisher'
|
|
7
10
|
require 'specbandit/worker'
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: specbandit
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.9.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ferran Basora
|
|
@@ -71,9 +71,9 @@ dependencies:
|
|
|
71
71
|
- - "~>"
|
|
72
72
|
- !ruby/object:Gem::Version
|
|
73
73
|
version: '3.0'
|
|
74
|
-
description: A work-stealing distributed
|
|
75
|
-
list, then multiple CI runners atomically steal batches and execute them
|
|
76
|
-
|
|
74
|
+
description: A work-stealing distributed test runner. Push test file paths to a Redis
|
|
75
|
+
list, then multiple CI runners atomically steal batches and execute them via a pluggable
|
|
76
|
+
adapter (CLI for any test runner, or in-process RSpec for maximum performance).
|
|
77
77
|
executables:
|
|
78
78
|
- specbandit
|
|
79
79
|
extensions: []
|
|
@@ -82,10 +82,13 @@ files:
|
|
|
82
82
|
- README.md
|
|
83
83
|
- bin/specbandit
|
|
84
84
|
- lib/specbandit.rb
|
|
85
|
+
- lib/specbandit/adapter.rb
|
|
85
86
|
- lib/specbandit/cli.rb
|
|
87
|
+
- lib/specbandit/cli_adapter.rb
|
|
86
88
|
- lib/specbandit/configuration.rb
|
|
87
89
|
- lib/specbandit/publisher.rb
|
|
88
90
|
- lib/specbandit/redis_queue.rb
|
|
91
|
+
- lib/specbandit/rspec_adapter.rb
|
|
89
92
|
- lib/specbandit/version.rb
|
|
90
93
|
- lib/specbandit/worker.rb
|
|
91
94
|
homepage: https://github.com/factorialco/specbandit
|
|
@@ -108,5 +111,5 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
108
111
|
requirements: []
|
|
109
112
|
rubygems_version: 4.0.6
|
|
110
113
|
specification_version: 4
|
|
111
|
-
summary: Distributed
|
|
114
|
+
summary: Distributed test runner using Redis as a work queue
|
|
112
115
|
test_files: []
|