rspec-signal 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +68 -0
- data/LICENSE +21 -0
- data/README.md +672 -0
- data/exe/rspec-signal +11 -0
- data/exe/rspec-signal-parallel +52 -0
- data/lib/rspec/signal/backtrace/classifier.rb +88 -0
- data/lib/rspec/signal/backtrace/frame.rb +31 -0
- data/lib/rspec/signal/backtrace/parser.rb +61 -0
- data/lib/rspec/signal/backtrace/reducer.rb +244 -0
- data/lib/rspec/signal/cluster.rb +68 -0
- data/lib/rspec/signal/clusterer.rb +49 -0
- data/lib/rspec/signal/configuration.rb +144 -0
- data/lib/rspec/signal/failure.rb +62 -0
- data/lib/rspec/signal/failure_builder.rb +224 -0
- data/lib/rspec/signal/fingerprint.rb +56 -0
- data/lib/rspec/signal/formatter.rb +207 -0
- data/lib/rspec/signal/group.rb +61 -0
- data/lib/rspec/signal/grouper.rb +28 -0
- data/lib/rspec/signal/html_summary.rb +224 -0
- data/lib/rspec/signal/integrations/capybara.rb +102 -0
- data/lib/rspec/signal/message.rb +161 -0
- data/lib/rspec/signal/parallel_merger.rb +125 -0
- data/lib/rspec/signal/parallel_run.rb +56 -0
- data/lib/rspec/signal/project.rb +158 -0
- data/lib/rspec/signal/redactor.rb +91 -0
- data/lib/rspec/signal/report.rb +88 -0
- data/lib/rspec/signal/reporters/full_output.rb +38 -0
- data/lib/rspec/signal/reporters/json_report.rb +20 -0
- data/lib/rspec/signal/reporters/markdown.rb +271 -0
- data/lib/rspec/signal/reporters/related_failures.rb +101 -0
- data/lib/rspec/signal/symptom.rb +22 -0
- data/lib/rspec/signal/symptoms/exception_class.rb +46 -0
- data/lib/rspec/signal/symptoms/http_status.rb +98 -0
- data/lib/rspec/signal/symptoms/record.rb +56 -0
- data/lib/rspec/signal/symptoms/route.rb +45 -0
- data/lib/rspec/signal/symptoms/ruby_error.rb +55 -0
- data/lib/rspec/signal/symptoms/selector.rb +72 -0
- data/lib/rspec/signal/symptoms.rb +42 -0
- data/lib/rspec/signal/version.rb +7 -0
- data/lib/rspec/signal/writer.rb +88 -0
- data/lib/rspec/signal.rb +156 -0
- data/lib/rspec-signal.rb +3 -0
- metadata +117 -0
data/README.md
ADDED
|
@@ -0,0 +1,672 @@
|
|
|
1
|
+
# rspec-signal
|
|
2
|
+
|
|
3
|
+
[](https://github.com/SilenceDogood1984/rspec-signal/actions/workflows/ci.yml)
|
|
4
|
+
[](https://rubygems.org/gems/rspec-signal)
|
|
5
|
+
|
|
6
|
+
**In one real-world run, 42 failures produced roughly 22,000 lines of raw RSpec output. `rspec-signal` reduced that to roughly 1,100 lines of focused diagnostics.**
|
|
7
|
+
|
|
8
|
+
RSpec's failure output is written for a human staring at a terminal, scrolling to the
|
|
9
|
+
one line they recognise. Hand the same output to an AI coding agent and you spend
|
|
10
|
+
thousands of tokens on `rspec-core/hooks.rb`, Bundler, and Thor before the agent
|
|
11
|
+
reaches a single line of your code — and you pay that cost repeatedly,
|
|
12
|
+
even when many failures stem from one bug.
|
|
13
|
+
|
|
14
|
+
`rspec-signal` is a deterministic context-reduction layer between RSpec and a coding
|
|
15
|
+
agent. It writes `tmp/rspec-signal/signal.md`: a compact, model-neutral report you
|
|
16
|
+
can paste into any assistant or point an agent at.
|
|
17
|
+
|
|
18
|
+
In normal mode your terminal output stays as it was. In opt-in agent mode it also becomes a quiet formatter that prevents verbose failures from entering an autonomous agent's context.
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## Before and after
|
|
23
|
+
|
|
24
|
+
A run with 17 failures caused by 2 bugs. This is one failure out of the seventeen,
|
|
25
|
+
as RSpec prints it:
|
|
26
|
+
|
|
27
|
+
```text
|
|
28
|
+
1) Shelf renders row set 0
|
|
29
|
+
Failure/Error: @rows.map { |r| r.fetch(:title) }.join(", ")
|
|
30
|
+
|
|
31
|
+
KeyError:
|
|
32
|
+
key not found: :title
|
|
33
|
+
# ./app/models/shelf.rb:4:in `fetch'
|
|
34
|
+
# ./app/models/shelf.rb:4:in `block in render'
|
|
35
|
+
# ./app/models/shelf.rb:4:in `map'
|
|
36
|
+
# ./app/models/shelf.rb:4:in `render'
|
|
37
|
+
# ./spec/models/shelf_spec.rb:5:in `block (3 levels) in <top (required)>'
|
|
38
|
+
# /usr/local/bundle/gems/rspec-core-3.13.6/lib/rspec/core/example.rb:263:in `instance_exec'
|
|
39
|
+
# /usr/local/bundle/gems/rspec-core-3.13.6/lib/rspec/core/example.rb:263:in `block in run'
|
|
40
|
+
# /usr/local/bundle/gems/rspec-core-3.13.6/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'
|
|
41
|
+
# /usr/local/bundle/gems/rspec-core-3.13.6/lib/rspec/core/hooks.rb:486:in `block in run'
|
|
42
|
+
# ... 22 more frames of rspec-core, Bundler, Thor and bin/bundle ...
|
|
43
|
+
|
|
44
|
+
2) Shelf renders row set 1
|
|
45
|
+
... the same 31 lines again ...
|
|
46
|
+
|
|
47
|
+
3) Shelf renders row set 2
|
|
48
|
+
... and again, fifteen more times ...
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
677 lines. 72 KB. And here is what `rspec-signal` writes for the whole run:
|
|
52
|
+
|
|
53
|
+
````markdown
|
|
54
|
+
# RSpec Signal
|
|
55
|
+
|
|
56
|
+
**17 examples | 17 failures | 2 distinct signatures**
|
|
57
|
+
Backtraces reduced from 535 to 65 frames (470 framework/library frames omitted).
|
|
58
|
+
|
|
59
|
+
## 1. KeyError -- 12 examples
|
|
60
|
+
|
|
61
|
+
> Shelf renders row set 0
|
|
62
|
+
|
|
63
|
+
```text
|
|
64
|
+
Failure/Error: @rows.map { |r| r.fetch(:title) }.join(", ")
|
|
65
|
+
|
|
66
|
+
KeyError:
|
|
67
|
+
key not found: :title
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
- Example `spec/models/shelf_spec.rb:4`
|
|
71
|
+
- Your code `app/models/shelf.rb:4`
|
|
72
|
+
|
|
73
|
+
**Trace**
|
|
74
|
+
|
|
75
|
+
```text
|
|
76
|
+
app/models/shelf.rb:4 in `fetch'
|
|
77
|
+
app/models/shelf.rb:4 in `block in render'
|
|
78
|
+
app/models/shelf.rb:4 in `map'
|
|
79
|
+
app/models/shelf.rb:4 in `render'
|
|
80
|
+
spec/models/shelf_spec.rb:5 in `block (3 levels) in <top (required)>'
|
|
81
|
+
[25 framework/runtime frames omitted]
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
**Rerun**
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
bundle exec rspec spec/models/shelf_spec.rb:4
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
**Also failing identically (11)**
|
|
91
|
+
|
|
92
|
+
```text
|
|
93
|
+
spec/models/shelf_spec.rb:4 (11 examples)
|
|
94
|
+
```
|
|
95
|
+
````
|
|
96
|
+
|
|
97
|
+
105 lines. 3 KB. Nothing an agent needs was removed: the failing expression, the
|
|
98
|
+
exception, the message, every frame of your own code, the exact file to open, and
|
|
99
|
+
the command to reproduce it.
|
|
100
|
+
|
|
101
|
+
Those numbers come from one synthetic run. Your ratio depends on how noisy your
|
|
102
|
+
backtraces are and how much your failures repeat — a suite of genuinely distinct
|
|
103
|
+
failures reduces far less, and that is correct behaviour.
|
|
104
|
+
|
|
105
|
+
## Installation
|
|
106
|
+
|
|
107
|
+
```ruby
|
|
108
|
+
# Gemfile
|
|
109
|
+
group :test do
|
|
110
|
+
gem "rspec-signal"
|
|
111
|
+
end
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
Then require it from `spec_helper.rb` (or `rails_helper.rb`):
|
|
115
|
+
|
|
116
|
+
```ruby
|
|
117
|
+
require "rspec/signal"
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
That is the whole setup. Requiring the gem registers the formatter and puts RSpec's
|
|
121
|
+
default formatter back, so your terminal output is unchanged.
|
|
122
|
+
|
|
123
|
+
If you would rather be explicit, or you need it only in CI, skip the require and
|
|
124
|
+
name the formatter instead:
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
bundle exec rspec --require rspec/signal --format progress --format RSpec::Signal::Formatter
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
## Autonomous-agent workflow (quiet mode)
|
|
131
|
+
|
|
132
|
+
Run the gem's quiet wrapper for an autonomous agent. It forwards every argument to
|
|
133
|
+
RSpec and exits with RSpec's status:
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
bundle exec rspec-signal
|
|
137
|
+
bundle exec rspec-signal spec/models/user_spec.rb:42
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
The wrapper selects Signal as RSpec's only formatter, so no verbose failure formatter
|
|
141
|
+
is added. On an interactive terminal, a bounded single-line progress bar is repainted
|
|
142
|
+
in place; redirected stdout contains no live progress or control sequences. A failing
|
|
143
|
+
suite remains non-zero; a passing suite remains zero. The
|
|
144
|
+
low-level `bundle exec rspec --format RSpec::Signal::Formatter` interface remains
|
|
145
|
+
available after requiring `rspec/signal` from the spec helper.
|
|
146
|
+
|
|
147
|
+
```text
|
|
148
|
+
RSpec runs the full suite
|
|
149
|
+
↓
|
|
150
|
+
verbose failure bodies and framework backtraces are not printed
|
|
151
|
+
↓
|
|
152
|
+
rspec-signal writes tmp/rspec-signal/signal.md
|
|
153
|
+
↓
|
|
154
|
+
the agent reads signal.md
|
|
155
|
+
↓
|
|
156
|
+
the agent reruns individual failures from the report as needed
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
### Parallel suites (`parallel_tests`)
|
|
160
|
+
|
|
161
|
+
For suites normally launched with `parallel_rspec`, use the dedicated wrapper:
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
bundle exec rspec-signal-parallel spec -n 8
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
All arguments are forwarded unchanged to `parallel_rspec`, so its file selection,
|
|
168
|
+
process-count, grouping, and runtime-log options continue to work. The wrapper adds
|
|
169
|
+
Signal as the quiet RSpec formatter through `SPEC_OPTS`; individual workers therefore
|
|
170
|
+
do not print failure bodies or backtraces. `parallel_tests` may still print its own
|
|
171
|
+
bounded process and progress output.
|
|
172
|
+
|
|
173
|
+
The wrapper gives every invocation a random run ID. Workers use
|
|
174
|
+
`TEST_ENV_NUMBER` (with the first process numbered `1`) and write structured files to:
|
|
175
|
+
|
|
176
|
+
```text
|
|
177
|
+
tmp/rspec-signal/workers/<run-id>/<test-process-number>/signal.json
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
After `parallel_rspec` returns, the parent reads those worker files, reconstructs all
|
|
181
|
+
failures, and performs exact grouping and related-failure clustering globally. It then
|
|
182
|
+
writes the usual top-level `signal.md` and `signal.json` (subject to the normal
|
|
183
|
+
artifact configuration). Thus identical or related failures in different
|
|
184
|
+
partitions appear together in the final report. If `write_full` is enabled, the parent
|
|
185
|
+
deterministically builds one top-level `full.txt` from the worker payloads; workers
|
|
186
|
+
never share that file.
|
|
187
|
+
|
|
188
|
+
Workers also record the aggregation-relevant configuration used to reduce and render
|
|
189
|
+
their results. The parent requires these settings to match across every worker rather
|
|
190
|
+
than silently applying whichever worker happens to finish first. A mismatch fails the
|
|
191
|
+
aggregation with a concise error, which usually indicates conditional configuration in
|
|
192
|
+
the project's spec helper.
|
|
193
|
+
|
|
194
|
+
Each run has a new ID and the merger only reads paths registered for that invocation,
|
|
195
|
+
so abandoned or stale worker reports cannot enter a later report. Once a run's worker
|
|
196
|
+
payloads have been merged into the top-level report, that run's worker directory is
|
|
197
|
+
removed; a run whose merge fails leaves its worker directory in place for diagnosis.
|
|
198
|
+
Missing registered artifacts and merge errors produce concise warnings. A merge error
|
|
199
|
+
makes an otherwise successful command fail, while a failing `parallel_rspec` status is
|
|
200
|
+
always preserved.
|
|
201
|
+
|
|
202
|
+
This first implementation supports local filesystem execution through
|
|
203
|
+
`parallel_tests`. Other parallel RSpec runners, multiple hosts, and distributed CI
|
|
204
|
+
workers are not yet guaranteed. Add `parallel_tests` to the application's test bundle;
|
|
205
|
+
it is deliberately a development/test dependency of this gem rather than a runtime
|
|
206
|
+
dependency.
|
|
207
|
+
|
|
208
|
+
In that observed run, 2,085 examples produced 42 failures, 35 exact signatures,
|
|
209
|
+
4 related clusters, and omitted 2,767 backtrace frames. The raw output was roughly
|
|
210
|
+
22,000 lines; the final compact parallel artifact was roughly 1,100 lines. These
|
|
211
|
+
figures describe that run, not a universal benchmark. Its quiet terminal summary
|
|
212
|
+
ended like this:
|
|
213
|
+
|
|
214
|
+
```text
|
|
215
|
+
2085 examples, 42 failures, 6 pending
|
|
216
|
+
|
|
217
|
+
rspec-signal: 42 failures in 35 distinct signatures, 4 related clusters (2767 backtrace frames omitted)
|
|
218
|
+
Report: tmp/rspec-signal/signal.md
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
### Human and CI modes
|
|
222
|
+
|
|
223
|
+
For a human, run RSpec normally:
|
|
224
|
+
|
|
225
|
+
```bash
|
|
226
|
+
bundle exec rspec
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
Requiring the gem auto-installs its collector and restores RSpec's default formatter.
|
|
230
|
+
Normal progress, failure bodies, backtraces, and summary remain visible, followed by
|
|
231
|
+
the short rspec-signal artifact notice. This preserves existing behavior.
|
|
232
|
+
|
|
233
|
+
For CI, choose deliberately: use normal mode when logs are the primary diagnostic,
|
|
234
|
+
or use the quiet formatter above when `signal.md` and `signal.json` are uploaded as
|
|
235
|
+
artifacts. In both modes RSpec owns the process status and test execution semantics.
|
|
236
|
+
|
|
237
|
+
Then hand the primary artifact to an agent, for example:
|
|
238
|
+
|
|
239
|
+
```bash
|
|
240
|
+
claude "fix the failures in tmp/rspec-signal/signal.md"
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
## Generated artifacts
|
|
244
|
+
|
|
245
|
+
```text
|
|
246
|
+
tmp/rspec-signal/
|
|
247
|
+
signal.md primary compact report -- hand this to the agent
|
|
248
|
+
signal.json the same data, machine readable, for CI and tooling
|
|
249
|
+
(`signatures` and `related` mirror the two grouping layers)
|
|
250
|
+
full.txt optional original output; off by default
|
|
251
|
+
.gitignore written automatically; artifacts can contain application data
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
A run with **no** failures deletes these files. A stale report describing failures
|
|
255
|
+
you already fixed is worse than no report at all, because an agent will go and
|
|
256
|
+
"fix" them again.
|
|
257
|
+
|
|
258
|
+
The large `full.txt` artifact is off by default. Enable it only when you need the
|
|
259
|
+
original unreduced formatter rendering:
|
|
260
|
+
|
|
261
|
+
```ruby
|
|
262
|
+
RSpec::Signal.configure do |config|
|
|
263
|
+
config.write_full = true
|
|
264
|
+
end
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
Individual per-failure files are deliberately **not** written. With failures
|
|
268
|
+
collapsed into a handful of signatures, `signal.md` is already the unit you want
|
|
269
|
+
to paste, and a directory of near-duplicate fragments is just more to sift through.
|
|
270
|
+
|
|
271
|
+
## Filtering philosophy
|
|
272
|
+
|
|
273
|
+
The goal is not the shortest possible output. It is **the minimum context that
|
|
274
|
+
preserves diagnostic usefulness**. A 50-line report containing the clue beats a
|
|
275
|
+
10-line report that removed it.
|
|
276
|
+
|
|
277
|
+
Every backtrace frame is classified as one of three things:
|
|
278
|
+
|
|
279
|
+
| Kind | What it is | What happens to it |
|
|
280
|
+
|------|-----------|--------------------|
|
|
281
|
+
| **project** | Code in your repository, plus Bundler `path:` gems and local engines | Always kept |
|
|
282
|
+
| **library** | Third-party code: Capybara, ActiveRecord, Rack, Net::HTTP, the standard library | Kept only where it touches your code |
|
|
283
|
+
| **framework** | The test runner, the loader, the CLI: rspec-core, rspec-expectations, Bundler, Thor, Rake, binstubs, `<internal:>` | Always dropped |
|
|
284
|
+
|
|
285
|
+
The distinction that matters is the middle row. Library frames are not noise — a
|
|
286
|
+
Capybara `find` or an ActiveRecord `save!` is often the only thing that tells you
|
|
287
|
+
*what operation failed*. So for each run of library frames that directly touches
|
|
288
|
+
first-party code, `rspec-signal` keeps:
|
|
289
|
+
|
|
290
|
+
- the **entry point**: the outermost library frame with a real method name, which is
|
|
291
|
+
the call your code actually made. Delegation shims whose frames are anonymous
|
|
292
|
+
blocks (`capybara/dsl.rb:52 in block (2 levels) in <module:DSL>`) are skipped in
|
|
293
|
+
favour of the frame that names the operation (`capybara/node/finders.rb:60 in find`).
|
|
294
|
+
- the **raise site**: the innermost frame of that run.
|
|
295
|
+
- one or two neighbours, budget permitting.
|
|
296
|
+
|
|
297
|
+
Library frames that touch no first-party code at all — a gem calling another gem
|
|
298
|
+
five layers down — are dropped and counted.
|
|
299
|
+
|
|
300
|
+
A worked example. Sixty-four frames in, six lines out:
|
|
301
|
+
|
|
302
|
+
```text
|
|
303
|
+
capybara/node/finders.rb:312 in `synced_resolve'
|
|
304
|
+
[2 library frames omitted]
|
|
305
|
+
capybara/node/finders.rb:60 in `find'
|
|
306
|
+
[1 library frame omitted]
|
|
307
|
+
capybara/dsl.rb:52 in `block (2 levels) in <module:DSL>'
|
|
308
|
+
spec/system/reader_self_reading_integrity_spec.rb:104 in `block (3 levels) in <top (required)>'
|
|
309
|
+
[47 framework/runtime frames omitted]
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
Two more rules keep the reduction honest:
|
|
313
|
+
|
|
314
|
+
- **First-party beats everything.** A frame in your repository is never classified as
|
|
315
|
+
framework plumbing, whatever it is named. A checkout in `~/src/rspec-signal-demo/`
|
|
316
|
+
does not lose all its own frames to a pattern match. The only exception is
|
|
317
|
+
binstubs (`bin/rspec`, `bin/bundle`), which are plumbing wherever they live.
|
|
318
|
+
- **The trace is never empty.** If a failure happens entirely inside a gem, or
|
|
319
|
+
entirely inside RSpec itself, the report falls back to the innermost non-framework
|
|
320
|
+
frames and says so: *"No first-party frames in this backtrace; innermost frames
|
|
321
|
+
shown instead."* Filtering that hides the only available evidence is a bug.
|
|
322
|
+
|
|
323
|
+
Two other things RSpec loses that `rspec-signal` keeps:
|
|
324
|
+
|
|
325
|
+
- **Exception causes.** RSpec puts the `Caused by:` chain in the *backtrace*, which
|
|
326
|
+
is exactly what gets reduced away. A `PG::UniqueViolation` behind a bland
|
|
327
|
+
`RuntimeError` is usually the answer, so the chain is folded into the message,
|
|
328
|
+
with the first-party frame it came from.
|
|
329
|
+
- **Aggregated failures.** For `:aggregate_failures`, RSpec deliberately empties the
|
|
330
|
+
message and moves the sub-failures into formatter-only callbacks, and the
|
|
331
|
+
exception's own backtrace is the aggregator's internals. `rspec-signal` recovers
|
|
332
|
+
both the sub-failures and the real backtrace from the first sub-exception.
|
|
333
|
+
|
|
334
|
+
## Grouping behaviour
|
|
335
|
+
|
|
336
|
+
Forty-three failures that are one bug should read as one bug. Failures are collapsed
|
|
337
|
+
by a deterministic fingerprint of four components:
|
|
338
|
+
|
|
339
|
+
| Component | What it is |
|
|
340
|
+
|-----------|-----------|
|
|
341
|
+
| **exception class** | `Capybara::ElementNotFound`, `ActiveRecord::RecordInvalid`, ... |
|
|
342
|
+
| **normalized message** | The message with volatile parts masked: object addresses, UUIDs, timestamps, record ids, temp paths. Small numbers are left alone — `expected 3, got 4` and `expected 7, got 2` are different failures. |
|
|
343
|
+
| **culprit** | The innermost frame that is not test-runner plumbing: the code that actually raised. |
|
|
344
|
+
| **app context** | The innermost first-party frame outside your spec suite. `nil` for a plain matcher failure; decisive when the same error comes from two different call sites. |
|
|
345
|
+
|
|
346
|
+
Deliberately **not** in the fingerprint: the example description, and the example's
|
|
347
|
+
own location. Fourteen specs in six files that all trip over the same missing DOM
|
|
348
|
+
node are one problem, not fourteen.
|
|
349
|
+
|
|
350
|
+
The four components are what stop over-collapsing:
|
|
351
|
+
|
|
352
|
+
- Two `Capybara::ElementNotFound` failures for **different selectors** stay apart —
|
|
353
|
+
the message differs.
|
|
354
|
+
- Two `ActiveRecord::RecordInvalid` failures with the **same message** from
|
|
355
|
+
`SubscriptionCreator` and `InviteCreator` stay apart — the app context differs.
|
|
356
|
+
- Two identical `expect(x).to be true` failures in **different specs** stay apart —
|
|
357
|
+
a matcher failure raises at the spec line, so the culprit differs.
|
|
358
|
+
- The same missing record id in twenty specs collapses to one — ids are masked.
|
|
359
|
+
|
|
360
|
+
Each group renders one full trace, from the failure carrying the most first-party
|
|
361
|
+
frames, plus every affected example's location. Groups are ordered largest first,
|
|
362
|
+
with ties broken by run order, so two runs of the same suite produce byte-identical
|
|
363
|
+
reports.
|
|
364
|
+
|
|
365
|
+
Failures that are clearly connected but correctly *not* identical are handled by a
|
|
366
|
+
separate, looser layer — see [Related failure clustering](#related-failure-clustering).
|
|
367
|
+
|
|
368
|
+
## Related failure clustering
|
|
369
|
+
|
|
370
|
+
Exact signatures are conservative on purpose: they never claim two failures are
|
|
371
|
+
the same failure unless they really are. That leaves a gap the real world fills
|
|
372
|
+
constantly — a dozen request specs that each wanted a different thing and all got
|
|
373
|
+
a 404, or one missing `data-testid` reported by `find` in one spec and by
|
|
374
|
+
`have_css` in another. Different signatures, obviously one problem.
|
|
375
|
+
|
|
376
|
+
So there is a second, deliberately looser layer. Each failure is scanned for **one
|
|
377
|
+
strong diagnostic symptom**, and failures sharing a symptom are reported together:
|
|
378
|
+
|
|
379
|
+
```markdown
|
|
380
|
+
## Related failures
|
|
381
|
+
|
|
382
|
+
Failures sharing one diagnostic symptom across more than one signature. Weaker than
|
|
383
|
+
a signature: a likely common cause, not a proven identical failure. The signatures
|
|
384
|
+
below remain authoritative.
|
|
385
|
+
|
|
386
|
+
### R1. Unexpected 404 (Not Found) responses -- 12 examples across 12 signatures
|
|
387
|
+
|
|
388
|
+
- Symptoms: `expected 200, got 404` (6), `expected redirect, got 404` (6)
|
|
389
|
+
- Specs: `spec/requests/checkpoint_responses_controller_spec.rb`, ... and 6 more
|
|
390
|
+
- Signatures: #3, #4, #5, #6, #7, #8, #9, #10, and 4 more
|
|
391
|
+
|
|
392
|
+
### R2. Missing css selector `[data-testid="reader-progress"] span` -- 4 examples across 2 signatures
|
|
393
|
+
|
|
394
|
+
- Specs: `spec/system/reader_progress_spec.rb`, `spec/system/reader_layout_spec.rb`, ...
|
|
395
|
+
- Signatures: #1, #2
|
|
396
|
+
```
|
|
397
|
+
|
|
398
|
+
The symptoms, in the order they are tried:
|
|
399
|
+
|
|
400
|
+
| Symptom | Cluster key | Example |
|
|
401
|
+
|---------|-------------|---------|
|
|
402
|
+
| **HTTP status** | the status that actually came back | `expected 200, got 404` and `expected redirect, got 404` cluster together; `got 500` does not |
|
|
403
|
+
| **Route** | the path with identifiers masked, or the `_path` helper | `No route matches [GET] "/readers/:id/progress"` |
|
|
404
|
+
| **Selector / page text** | the selector, compared exactly | `Unable to find css "x"` and `expected to find css "x" but there were no matches` cluster together |
|
|
405
|
+
| **ActiveRecord** | the model, the validation sentence, or the missing column/table | `Couldn't find User` clusters with `Couldn't find User`, never with `Couldn't find Order` |
|
|
406
|
+
| **Ruby error** | the method *and* its receiver, or the constant | `undefined method 'progress' for nil` |
|
|
407
|
+
| **Exception class** | the class, only when namespaced | `PG::ConnectionBad`, `Errno::ECONNREFUSED` |
|
|
408
|
+
|
|
409
|
+
### What stops it over-clustering
|
|
410
|
+
|
|
411
|
+
Clusters are a hint, and a hint that fires too often is worse than none. Five rules
|
|
412
|
+
hold the line:
|
|
413
|
+
|
|
414
|
+
1. **A cluster needs two or more failures.**
|
|
415
|
+
2. **A cluster needs two or more exact signatures.** If everything sharing a symptom
|
|
416
|
+
is already one signature, the signature section said it better, and repeating it
|
|
417
|
+
is pure noise. This is what keeps the section short.
|
|
418
|
+
3. **Each failure joins at most one cluster** — the first symptom that matches, in
|
|
419
|
+
the order above. Nothing appears twice.
|
|
420
|
+
4. **No similarity, ever.** Every symptom is anchored on a specific phrase from a
|
|
421
|
+
specific library. There is no fuzzy matching, no embedding, no threshold. A
|
|
422
|
+
failure that matches nothing has no symptom and joins nothing, which is the safe
|
|
423
|
+
direction to be wrong in.
|
|
424
|
+
5. **Exception class is a last resort, and a narrow one.** It fires only for a
|
|
425
|
+
*namespaced* class, never for `RuntimeError`, `ArgumentError` or anything under
|
|
426
|
+
`RSpec::`, and never for a class an earlier symptom is responsible for — so a
|
|
427
|
+
`Capybara::ElementNotFound` whose message could not be parsed can never drag
|
|
428
|
+
unrelated selectors into one cluster.
|
|
429
|
+
|
|
430
|
+
Ordering is deterministic: largest cluster first, then most signatures spanned, then
|
|
431
|
+
run order.
|
|
432
|
+
|
|
433
|
+
## Large HTML responses
|
|
434
|
+
|
|
435
|
+
A request spec expecting one sentence and receiving a Rails exception page produces
|
|
436
|
+
a diff several thousand lines long, and its first hundred lines are the exception
|
|
437
|
+
page's own CSS — the one part of the response guaranteed to be identical for every
|
|
438
|
+
failure in the suite. Capping it still spends the report's opening on stylesheet.
|
|
439
|
+
|
|
440
|
+
When the actual value is bulk HTML, `rspec-signal` replaces it:
|
|
441
|
+
|
|
442
|
+
```text
|
|
443
|
+
Failure/Error: expect(response.body).to include("You've finished this document.")
|
|
444
|
+
|
|
445
|
+
expected [HTML document] to include "You've finished this document."
|
|
446
|
+
|
|
447
|
+
[HTML document: 6,371 lines, 284 KB -- markup omitted]
|
|
448
|
+
Title: Action Controller: Exception caught
|
|
449
|
+
Heading: NoMethodError in ReaderController#show
|
|
450
|
+
Message: undefined method 'progress' for nil
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
- The **expected value is never touched** — it is the small, useful half.
|
|
454
|
+
- Detection is by shape: a `<!doctype html>`/`<html>` opening, or five or more tags
|
|
455
|
+
around structural elements. Both the one-enormous-inspected-line form and the
|
|
456
|
+
unified-diff form are handled.
|
|
457
|
+
- `<script>`, `<style>` and comments are stripped **before** any text is read, so
|
|
458
|
+
CSS can never be mistaken for content.
|
|
459
|
+
- Signals extracted: `<title>`, `<h1>`, `<h2>` or the first `<pre>`, falling back to
|
|
460
|
+
the leading visible text when a page has neither. On a Rails error page those are
|
|
461
|
+
the exception class and its message. Capybara's own `status_code` still appears
|
|
462
|
+
under **Browser state**.
|
|
463
|
+
- HTML small enough to read is left exactly as it was (`config.max_html_chars`,
|
|
464
|
+
default 1500 characters).
|
|
465
|
+
- No DOM parser, and no new dependency: regex only. It never has to be correct, only
|
|
466
|
+
useful, and it is handed broken markup by definition.
|
|
467
|
+
- If `write_full` is enabled, the untouched original is written to `full.txt`.
|
|
468
|
+
|
|
469
|
+
## Configuration
|
|
470
|
+
|
|
471
|
+
None is required. Everything below is optional, in `spec_helper.rb`:
|
|
472
|
+
|
|
473
|
+
```ruby
|
|
474
|
+
RSpec::Signal.configure do |config|
|
|
475
|
+
# Where artifacts go (relative to the project root, or absolute).
|
|
476
|
+
config.output_dir = "tmp/rspec-signal"
|
|
477
|
+
|
|
478
|
+
# Reduction budgets.
|
|
479
|
+
config.max_frames = 12 # frames kept per trace
|
|
480
|
+
config.max_external_context = 3 # library frames kept per adjacent run
|
|
481
|
+
config.max_project_frames = 8 # first-party frames kept per trace
|
|
482
|
+
config.fallback_frames = 6 # frames shown when nothing else survives
|
|
483
|
+
|
|
484
|
+
# Message budgets. Diffs are the biggest source of bloat.
|
|
485
|
+
config.max_message_lines = 30
|
|
486
|
+
config.max_diff_lines = 20
|
|
487
|
+
|
|
488
|
+
# Large HTML responses (see "Large HTML responses").
|
|
489
|
+
config.reduce_html = true
|
|
490
|
+
config.max_html_chars = 1500 # smallest HTML blob replaced by a summary
|
|
491
|
+
|
|
492
|
+
# Report budgets.
|
|
493
|
+
config.max_affected_examples = 25 # locations listed per group
|
|
494
|
+
config.max_groups = nil # signatures rendered in full (nil = all)
|
|
495
|
+
|
|
496
|
+
# Related failure clustering (see "Related failure clustering").
|
|
497
|
+
config.relate_failures = true
|
|
498
|
+
config.max_clusters = 10 # clusters rendered in full (nil = all)
|
|
499
|
+
config.max_cluster_specs = 6 # spec files listed per cluster
|
|
500
|
+
|
|
501
|
+
# Artifacts.
|
|
502
|
+
config.write_json = true
|
|
503
|
+
config.write_full = true
|
|
504
|
+
config.write_gitignore = true
|
|
505
|
+
|
|
506
|
+
# Behaviour.
|
|
507
|
+
config.enabled = true
|
|
508
|
+
config.terminal_summary = true
|
|
509
|
+
|
|
510
|
+
# Secret scrubbing (see Privacy).
|
|
511
|
+
config.redact = true
|
|
512
|
+
config.redaction_patterns = [/INTERNAL-[A-Z0-9]+/]
|
|
513
|
+
config.redaction_filter = ->(text) { text.gsub(customer_name, "[CUSTOMER]") }
|
|
514
|
+
|
|
515
|
+
# Classification.
|
|
516
|
+
config.project_root = Rails.root.to_s
|
|
517
|
+
config.extra_first_party = ["../billing-engine"] # a sibling checkout
|
|
518
|
+
config.ignore_patterns << %r{/lib/my_test_harness/} # your code, but plumbing
|
|
519
|
+
config.framework_patterns << /vendor_test_runner-/ # a gem, but plumbing
|
|
520
|
+
|
|
521
|
+
# Capybara.
|
|
522
|
+
config.capture_capybara = true
|
|
523
|
+
config.capture_page_html = false
|
|
524
|
+
end
|
|
525
|
+
```
|
|
526
|
+
|
|
527
|
+
Two environment variables are honoured, which is usually what you want in CI:
|
|
528
|
+
|
|
529
|
+
```bash
|
|
530
|
+
RSPEC_SIGNAL_DISABLE=1 # turn the gem off entirely
|
|
531
|
+
RSPEC_SIGNAL_OUTPUT_DIR=... # override the output directory
|
|
532
|
+
```
|
|
533
|
+
|
|
534
|
+
Note the two classification lists. `ignore_patterns` is checked **before**
|
|
535
|
+
first-party detection — it is how you say "this is my code, but treat it as
|
|
536
|
+
plumbing". `framework_patterns` is checked **after** — it is for third-party code
|
|
537
|
+
and cannot accidentally swallow your repository.
|
|
538
|
+
|
|
539
|
+
## Rails and Capybara
|
|
540
|
+
|
|
541
|
+
`rspec-signal` depends on `rspec-core` and nothing else. Rails, Capybara and
|
|
542
|
+
ActiveRecord are all optional; the gem gets richer when they are present.
|
|
543
|
+
|
|
544
|
+
**Rails.** The project root defaults to `Rails.root` when Rails is loaded, and the
|
|
545
|
+
report header records the Rails version. `app/`, `lib/`, `spec/`, and local engines
|
|
546
|
+
are first-party; `vendor/bundle` is not.
|
|
547
|
+
|
|
548
|
+
**System and feature specs.** For a failing example with `type: :system`,
|
|
549
|
+
`type: :feature`, or `js: true`, an `after(:each)` hook captures browser state into
|
|
550
|
+
the report:
|
|
551
|
+
|
|
552
|
+
```markdown
|
|
553
|
+
**Browser state**
|
|
554
|
+
|
|
555
|
+
- URL: `https://app.test/library`
|
|
556
|
+
- Page title: `Library`
|
|
557
|
+
- Status: `200`
|
|
558
|
+
- Console: `SEVERE: Uncaught TypeError: shelf.render is not a function`
|
|
559
|
+
- Screenshot: `tmp/screenshots/failures_reader_shelf.png`
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
Rails writes the screenshot itself on a system-test failure; `rspec-signal` finds the
|
|
563
|
+
path in `metadata[:extra_failure_lines]` and surfaces it as a link rather than
|
|
564
|
+
leaving it buried in the message. Browser console output is small and frequently
|
|
565
|
+
contains the real cause of a JavaScript-driven failure.
|
|
566
|
+
|
|
567
|
+
The whole capture is best effort and wrapped in rescues: a driver that cannot report
|
|
568
|
+
a status code just contributes less detail. It never asks Capybara for
|
|
569
|
+
`current_session`, which would *create* a session and possibly boot a browser — it
|
|
570
|
+
only reads a session that the test already opened.
|
|
571
|
+
|
|
572
|
+
Page HTML is **not** saved by default. It is frequently the clue, and it is also
|
|
573
|
+
100 KB of exactly the bloat this gem exists to remove. Enable it deliberately with
|
|
574
|
+
`config.capture_page_html = true`, and the report will link to the saved file.
|
|
575
|
+
|
|
576
|
+
## Privacy
|
|
577
|
+
|
|
578
|
+
Failure output contains whatever your tests put in it: fixture data, request bodies,
|
|
579
|
+
headers, environment values. These artifacts are explicitly meant to be handed to
|
|
580
|
+
external AI systems, so `rspec-signal` scrubs obvious credentials by default —
|
|
581
|
+
`Authorization` headers, credential-shaped assignments (`password:`, `api_key=`,
|
|
582
|
+
`?access_token=`), URL userinfo, and well-known token formats (AWS, GitHub, Slack,
|
|
583
|
+
Stripe, GitLab, Google, JWTs, PEM private key blocks).
|
|
584
|
+
|
|
585
|
+
It targets shapes that are unambiguous and deliberately does not guess at arbitrary
|
|
586
|
+
values, because false positives destroy the diagnostic value of a report:
|
|
587
|
+
`expected 3 items, got 4` and `undefined method 'password_digest' for nil` are left
|
|
588
|
+
exactly as they are.
|
|
589
|
+
|
|
590
|
+
**This is a safety net, not a guarantee. Review artifacts before sending them
|
|
591
|
+
somewhere you do not control.** `tmp/rspec-signal/.gitignore` is written
|
|
592
|
+
automatically so they do not reach version control by accident.
|
|
593
|
+
|
|
594
|
+
You can add your own patterns with `config.redaction_patterns`, post-process
|
|
595
|
+
everything with `config.redaction_filter`, or turn scrubbing off with
|
|
596
|
+
`config.redact = false`.
|
|
597
|
+
|
|
598
|
+
## Limitations
|
|
599
|
+
|
|
600
|
+
- **Errors outside examples.** A `before(:suite)` blow-up, or a spec file that fails
|
|
601
|
+
to load, produces no failed examples. RSpec reports those only through its message
|
|
602
|
+
stream, which a formatter cannot subscribe to without swallowing every other
|
|
603
|
+
message. `rspec-signal` reports the count and says where to look, but cannot
|
|
604
|
+
capture the text.
|
|
605
|
+
- **Parallel test runners.** Use `rspec-signal-parallel` for suites launched with
|
|
606
|
+
`parallel_tests` (see [Parallel suites](#parallel-suites-parallel_tests)); it isolates
|
|
607
|
+
each worker and merges the results deterministically. Do not set a per-worker
|
|
608
|
+
`RSPEC_SIGNAL_OUTPUT_DIR` with the wrapper -- it requires every worker to report the
|
|
609
|
+
same configuration and fails the merge if the output directory differs between them.
|
|
610
|
+
Running `parallel_rspec` directly, with the gem only required from `spec_helper.rb`
|
|
611
|
+
and no wrapper, is not supported: every worker writes to the same `signal.md` with a
|
|
612
|
+
non-atomic write, and a worker that finishes green can delete a sibling worker's
|
|
613
|
+
still-relevant report.
|
|
614
|
+
- **Grouping is a heuristic.** It is deterministic and explained above, but two
|
|
615
|
+
genuinely different bugs that raise the same exception with the same message from
|
|
616
|
+
the same line will collapse into one signature. The affected-example list always
|
|
617
|
+
shows you everything that was merged.
|
|
618
|
+
- **Related clusters are a hint, not a diagnosis.** They say two failures share a
|
|
619
|
+
symptom, never that they share a cause; the report says so in those words. The
|
|
620
|
+
symptom list is finite, so a suite whose failures are worded in a way no extractor
|
|
621
|
+
recognises simply gets no clusters — a silence, not a wrong answer.
|
|
622
|
+
- **HTML reduction reads markup with regexes.** It extracts the title, headings and
|
|
623
|
+
leading text of a page; it does not understand the page. A response whose useful
|
|
624
|
+
content is buried deep in the body will be summarised as an HTML document and
|
|
625
|
+
little more. Enable `write_full` temporarily if you need the original markup.
|
|
626
|
+
- **No source packaging.** `rspec-signal` names files and lines; it does not embed
|
|
627
|
+
source code. Agents working inside a repository can open the files themselves, and
|
|
628
|
+
in v1 that is a better division of labour than guessing which snippets to inline.
|
|
629
|
+
- **Reduction depends on your suite.** Failures that are already distinct and already
|
|
630
|
+
shallow reduce very little. That is the correct outcome, not a failure of the gem.
|
|
631
|
+
|
|
632
|
+
## Compatibility
|
|
633
|
+
|
|
634
|
+
- Ruby 3.1+
|
|
635
|
+
- RSpec 3.10+ (via `rspec-core`)
|
|
636
|
+
- Rails, Capybara, ActiveRecord: optional
|
|
637
|
+
|
|
638
|
+
The only runtime dependency is `rspec-core`.
|
|
639
|
+
|
|
640
|
+
## Development
|
|
641
|
+
|
|
642
|
+
```bash
|
|
643
|
+
bin/setup # or: bundle install
|
|
644
|
+
bundle exec rake # specs + RuboCop
|
|
645
|
+
bundle exec rspec
|
|
646
|
+
bundle exec rubocop
|
|
647
|
+
```
|
|
648
|
+
|
|
649
|
+
The test suite runs `rspec-signal` on itself, so a failing run writes its own
|
|
650
|
+
report to `tmp/rspec-signal/signal.md`.
|
|
651
|
+
|
|
652
|
+
Most of the gem is plain Ruby with no RSpec dependency. Only `Formatter` and
|
|
653
|
+
`FailureBuilder` touch RSpec's notification API, which is what lets the reduction,
|
|
654
|
+
grouping and rendering stages be tested directly against synthetic fixtures in
|
|
655
|
+
`spec/fixtures/backtraces.rb`. `spec/integration/end_to_end_spec.rb` shells out to
|
|
656
|
+
real `rspec` processes in generated throwaway projects.
|
|
657
|
+
|
|
658
|
+
Tests deliberately assert on output *size* as well as content, so a future change
|
|
659
|
+
cannot quietly let noise back in.
|
|
660
|
+
|
|
661
|
+
## Contributing
|
|
662
|
+
|
|
663
|
+
Bug reports and pull requests are welcome at
|
|
664
|
+
<https://github.com/SilenceDogood1984/rspec-signal>.
|
|
665
|
+
|
|
666
|
+
If you are reporting a case where reduction removed something important, the most
|
|
667
|
+
useful thing you can attach is the raw backtrace — a new fixture in
|
|
668
|
+
`spec/fixtures/backtraces.rb` is the ideal shape for it.
|
|
669
|
+
|
|
670
|
+
## License
|
|
671
|
+
|
|
672
|
+
MIT. See [LICENSE](LICENSE).
|
data/exe/rspec-signal
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
ENV["RSPEC_SIGNAL_QUIET"] = "1"
|
|
5
|
+
arguments = ["--require", "rspec/signal", "--format", "RSpec::Signal::Formatter", *ARGV]
|
|
6
|
+
|
|
7
|
+
# Run the same executable as `bundle exec rspec`. Besides avoiding a parallel
|
|
8
|
+
# implementation of RSpec's CLI, this is important because the executable uses
|
|
9
|
+
# Runner.invoke, which applies RSpec's default spec discovery when no paths are
|
|
10
|
+
# supplied. Runner.run is a lower-level API and does not provide that behavior.
|
|
11
|
+
exec Gem.ruby, Gem.bin_path("rspec-core", "rspec"), *arguments
|