rspec-signal 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +68 -0
  3. data/LICENSE +21 -0
  4. data/README.md +672 -0
  5. data/exe/rspec-signal +11 -0
  6. data/exe/rspec-signal-parallel +52 -0
  7. data/lib/rspec/signal/backtrace/classifier.rb +88 -0
  8. data/lib/rspec/signal/backtrace/frame.rb +31 -0
  9. data/lib/rspec/signal/backtrace/parser.rb +61 -0
  10. data/lib/rspec/signal/backtrace/reducer.rb +244 -0
  11. data/lib/rspec/signal/cluster.rb +68 -0
  12. data/lib/rspec/signal/clusterer.rb +49 -0
  13. data/lib/rspec/signal/configuration.rb +144 -0
  14. data/lib/rspec/signal/failure.rb +62 -0
  15. data/lib/rspec/signal/failure_builder.rb +224 -0
  16. data/lib/rspec/signal/fingerprint.rb +56 -0
  17. data/lib/rspec/signal/formatter.rb +207 -0
  18. data/lib/rspec/signal/group.rb +61 -0
  19. data/lib/rspec/signal/grouper.rb +28 -0
  20. data/lib/rspec/signal/html_summary.rb +224 -0
  21. data/lib/rspec/signal/integrations/capybara.rb +102 -0
  22. data/lib/rspec/signal/message.rb +161 -0
  23. data/lib/rspec/signal/parallel_merger.rb +125 -0
  24. data/lib/rspec/signal/parallel_run.rb +56 -0
  25. data/lib/rspec/signal/project.rb +158 -0
  26. data/lib/rspec/signal/redactor.rb +91 -0
  27. data/lib/rspec/signal/report.rb +88 -0
  28. data/lib/rspec/signal/reporters/full_output.rb +38 -0
  29. data/lib/rspec/signal/reporters/json_report.rb +20 -0
  30. data/lib/rspec/signal/reporters/markdown.rb +271 -0
  31. data/lib/rspec/signal/reporters/related_failures.rb +101 -0
  32. data/lib/rspec/signal/symptom.rb +22 -0
  33. data/lib/rspec/signal/symptoms/exception_class.rb +46 -0
  34. data/lib/rspec/signal/symptoms/http_status.rb +98 -0
  35. data/lib/rspec/signal/symptoms/record.rb +56 -0
  36. data/lib/rspec/signal/symptoms/route.rb +45 -0
  37. data/lib/rspec/signal/symptoms/ruby_error.rb +55 -0
  38. data/lib/rspec/signal/symptoms/selector.rb +72 -0
  39. data/lib/rspec/signal/symptoms.rb +42 -0
  40. data/lib/rspec/signal/version.rb +7 -0
  41. data/lib/rspec/signal/writer.rb +88 -0
  42. data/lib/rspec/signal.rb +156 -0
  43. data/lib/rspec-signal.rb +3 -0
  44. metadata +117 -0
data/README.md ADDED
@@ -0,0 +1,672 @@
1
+ # rspec-signal
2
+
3
+ [![CI](https://github.com/SilenceDogood1984/rspec-signal/actions/workflows/ci.yml/badge.svg)](https://github.com/SilenceDogood1984/rspec-signal/actions/workflows/ci.yml)
4
+ [![Gem Version](https://badge.fury.io/rb/rspec-signal.svg)](https://rubygems.org/gems/rspec-signal)
5
+
6
+ **In one real-world run, 42 failures produced roughly 22,000 lines of raw RSpec output. `rspec-signal` reduced that to roughly 1,100 lines of focused diagnostics.**
7
+
8
+ RSpec's failure output is written for a human staring at a terminal, scrolling to the
9
+ one line they recognise. Hand the same output to an AI coding agent and you spend
10
+ thousands of tokens on `rspec-core/hooks.rb`, Bundler, and Thor before the agent
11
+ reaches a single line of your code — and you pay that cost repeatedly,
12
+ even when many failures stem from one bug.
13
+
14
+ `rspec-signal` is a deterministic context-reduction layer between RSpec and a coding
15
+ agent. It writes `tmp/rspec-signal/signal.md`: a compact, model-neutral report you
16
+ can paste into any assistant or point an agent at.
17
+
18
+ In normal mode your terminal output stays as it was. In opt-in agent mode it also becomes a quiet formatter that prevents verbose failures from entering an autonomous agent's context.
19
+
20
+ ---
21
+
22
+ ## Before and after
23
+
24
+ A run with 17 failures caused by 2 bugs. This is one failure out of the seventeen,
25
+ as RSpec prints it:
26
+
27
+ ```text
28
+ 1) Shelf renders row set 0
29
+ Failure/Error: @rows.map { |r| r.fetch(:title) }.join(", ")
30
+
31
+ KeyError:
32
+ key not found: :title
33
+ # ./app/models/shelf.rb:4:in `fetch'
34
+ # ./app/models/shelf.rb:4:in `block in render'
35
+ # ./app/models/shelf.rb:4:in `map'
36
+ # ./app/models/shelf.rb:4:in `render'
37
+ # ./spec/models/shelf_spec.rb:5:in `block (3 levels) in <top (required)>'
38
+ # /usr/local/bundle/gems/rspec-core-3.13.6/lib/rspec/core/example.rb:263:in `instance_exec'
39
+ # /usr/local/bundle/gems/rspec-core-3.13.6/lib/rspec/core/example.rb:263:in `block in run'
40
+ # /usr/local/bundle/gems/rspec-core-3.13.6/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'
41
+ # /usr/local/bundle/gems/rspec-core-3.13.6/lib/rspec/core/hooks.rb:486:in `block in run'
42
+ # ... 22 more frames of rspec-core, Bundler, Thor and bin/bundle ...
43
+
44
+ 2) Shelf renders row set 1
45
+ ... the same 31 lines again ...
46
+
47
+ 3) Shelf renders row set 2
48
+ ... and again, fifteen more times ...
49
+ ```
50
+
51
+ 677 lines. 72 KB. And here is what `rspec-signal` writes for the whole run:
52
+
53
+ ````markdown
54
+ # RSpec Signal
55
+
56
+ **17 examples | 17 failures | 2 distinct signatures**
57
+ Backtraces reduced from 535 to 65 frames (470 framework/library frames omitted).
58
+
59
+ ## 1. KeyError -- 12 examples
60
+
61
+ > Shelf renders row set 0
62
+
63
+ ```text
64
+ Failure/Error: @rows.map { |r| r.fetch(:title) }.join(", ")
65
+
66
+ KeyError:
67
+ key not found: :title
68
+ ```
69
+
70
+ - Example `spec/models/shelf_spec.rb:4`
71
+ - Your code `app/models/shelf.rb:4`
72
+
73
+ **Trace**
74
+
75
+ ```text
76
+ app/models/shelf.rb:4 in `fetch'
77
+ app/models/shelf.rb:4 in `block in render'
78
+ app/models/shelf.rb:4 in `map'
79
+ app/models/shelf.rb:4 in `render'
80
+ spec/models/shelf_spec.rb:5 in `block (3 levels) in <top (required)>'
81
+ [25 framework/runtime frames omitted]
82
+ ```
83
+
84
+ **Rerun**
85
+
86
+ ```bash
87
+ bundle exec rspec spec/models/shelf_spec.rb:4
88
+ ```
89
+
90
+ **Also failing identically (11)**
91
+
92
+ ```text
93
+ spec/models/shelf_spec.rb:4 (11 examples)
94
+ ```
95
+ ````
96
+
97
+ 105 lines. 3 KB. Nothing an agent needs was removed: the failing expression, the
98
+ exception, the message, every frame of your own code, the exact file to open, and
99
+ the command to reproduce it.
100
+
101
+ Those numbers come from one synthetic run. Your ratio depends on how noisy your
102
+ backtraces are and how much your failures repeat — a suite of genuinely distinct
103
+ failures reduces far less, and that is correct behaviour.
104
+
105
+ ## Installation
106
+
107
+ ```ruby
108
+ # Gemfile
109
+ group :test do
110
+ gem "rspec-signal"
111
+ end
112
+ ```
113
+
114
+ Then require it from `spec_helper.rb` (or `rails_helper.rb`):
115
+
116
+ ```ruby
117
+ require "rspec/signal"
118
+ ```
119
+
120
+ That is the whole setup. Requiring the gem registers the formatter and puts RSpec's
121
+ default formatter back, so your terminal output is unchanged.
122
+
123
+ If you would rather be explicit, or you need it only in CI, skip the require and
124
+ name the formatter instead:
125
+
126
+ ```bash
127
+ bundle exec rspec --require rspec/signal --format progress --format RSpec::Signal::Formatter
128
+ ```
129
+
130
+ ## Autonomous-agent workflow (quiet mode)
131
+
132
+ Run the gem's quiet wrapper for an autonomous agent. It forwards every argument to
133
+ RSpec and exits with RSpec's status:
134
+
135
+ ```bash
136
+ bundle exec rspec-signal
137
+ bundle exec rspec-signal spec/models/user_spec.rb:42
138
+ ```
139
+
140
+ The wrapper selects Signal as RSpec's only formatter, so no verbose failure formatter
141
+ is added. On an interactive terminal, a bounded single-line progress bar is repainted
142
+ in place; redirected stdout contains no live progress or control sequences. A failing
143
+ suite remains non-zero; a passing suite remains zero. The
144
+ low-level `bundle exec rspec --format RSpec::Signal::Formatter` interface remains
145
+ available after requiring `rspec/signal` from the spec helper.
146
+
147
+ ```text
148
+ RSpec runs the full suite
149
+
150
+ verbose failure bodies and framework backtraces are not printed
151
+
152
+ rspec-signal writes tmp/rspec-signal/signal.md
153
+
154
+ the agent reads signal.md
155
+
156
+ the agent reruns individual failures from the report as needed
157
+ ```
158
+
159
+ ### Parallel suites (`parallel_tests`)
160
+
161
+ For suites normally launched with `parallel_rspec`, use the dedicated wrapper:
162
+
163
+ ```bash
164
+ bundle exec rspec-signal-parallel spec -n 8
165
+ ```
166
+
167
+ All arguments are forwarded unchanged to `parallel_rspec`, so its file selection,
168
+ process-count, grouping, and runtime-log options continue to work. The wrapper adds
169
+ Signal as the quiet RSpec formatter through `SPEC_OPTS`; individual workers therefore
170
+ do not print failure bodies or backtraces. `parallel_tests` may still print its own
171
+ bounded process and progress output.
172
+
173
+ The wrapper gives every invocation a random run ID. Workers use
174
+ `TEST_ENV_NUMBER` (with the first process numbered `1`) and write structured files to:
175
+
176
+ ```text
177
+ tmp/rspec-signal/workers/<run-id>/<test-process-number>/signal.json
178
+ ```
179
+
180
+ After `parallel_rspec` returns, the parent reads those worker files, reconstructs all
181
+ failures, and performs exact grouping and related-failure clustering globally. It then
182
+ writes the usual top-level `signal.md` and `signal.json` (subject to the normal
183
+ artifact configuration). Thus identical or related failures in different
184
+ partitions appear together in the final report. If `write_full` is enabled, the parent
185
+ deterministically builds one top-level `full.txt` from the worker payloads; workers
186
+ never share that file.
187
+
188
+ Workers also record the aggregation-relevant configuration used to reduce and render
189
+ their results. The parent requires these settings to match across every worker rather
190
+ than silently applying whichever worker happens to finish first. A mismatch fails the
191
+ aggregation with a concise error, which usually indicates conditional configuration in
192
+ the project's spec helper.
193
+
194
+ Each run has a new ID and the merger only reads paths registered for that invocation,
195
+ so abandoned or stale worker reports cannot enter a later report. Once a run's worker
196
+ payloads have been merged into the top-level report, that run's worker directory is
197
+ removed; a run whose merge fails leaves its worker directory in place for diagnosis.
198
+ Missing registered artifacts and merge errors produce concise warnings. A merge error
199
+ makes an otherwise successful command fail, while a failing `parallel_rspec` status is
200
+ always preserved.
201
+
202
+ This first implementation supports local filesystem execution through
203
+ `parallel_tests`. Other parallel RSpec runners, multiple hosts, and distributed CI
204
+ workers are not yet guaranteed. Add `parallel_tests` to the application's test bundle;
205
+ it is deliberately a development/test dependency of this gem rather than a runtime
206
+ dependency.
207
+
208
+ In that observed run, 2,085 examples produced 42 failures, 35 exact signatures,
209
+ 4 related clusters, and omitted 2,767 backtrace frames. The raw output was roughly
210
+ 22,000 lines; the final compact parallel artifact was roughly 1,100 lines. These
211
+ figures describe that run, not a universal benchmark. Its quiet terminal summary
212
+ ended like this:
213
+
214
+ ```text
215
+ 2085 examples, 42 failures, 6 pending
216
+
217
+ rspec-signal: 42 failures in 35 distinct signatures, 4 related clusters (2767 backtrace frames omitted)
218
+ Report: tmp/rspec-signal/signal.md
219
+ ```
220
+
221
+ ### Human and CI modes
222
+
223
+ For a human, run RSpec normally:
224
+
225
+ ```bash
226
+ bundle exec rspec
227
+ ```
228
+
229
+ Requiring the gem auto-installs its collector and restores RSpec's default formatter.
230
+ Normal progress, failure bodies, backtraces, and summary remain visible, followed by
231
+ the short rspec-signal artifact notice. This preserves existing behavior.
232
+
233
+ For CI, choose deliberately: use normal mode when logs are the primary diagnostic,
234
+ or use the quiet formatter above when `signal.md` and `signal.json` are uploaded as
235
+ artifacts. In both modes RSpec owns the process status and test execution semantics.
236
+
237
+ Then hand the primary artifact to an agent, for example:
238
+
239
+ ```bash
240
+ claude "fix the failures in tmp/rspec-signal/signal.md"
241
+ ```
242
+
243
+ ## Generated artifacts
244
+
245
+ ```text
246
+ tmp/rspec-signal/
247
+ signal.md primary compact report -- hand this to the agent
248
+ signal.json the same data, machine readable, for CI and tooling
249
+ (`signatures` and `related` mirror the two grouping layers)
250
+ full.txt optional original output; off by default
251
+ .gitignore written automatically; artifacts can contain application data
252
+ ```
253
+
254
+ A run with **no** failures deletes these files. A stale report describing failures
255
+ you already fixed is worse than no report at all, because an agent will go and
256
+ "fix" them again.
257
+
258
+ The large `full.txt` artifact is off by default. Enable it only when you need the
259
+ original unreduced formatter rendering:
260
+
261
+ ```ruby
262
+ RSpec::Signal.configure do |config|
263
+ config.write_full = true
264
+ end
265
+ ```
266
+
267
+ Individual per-failure files are deliberately **not** written. With failures
268
+ collapsed into a handful of signatures, `signal.md` is already the unit you want
269
+ to paste, and a directory of near-duplicate fragments is just more to sift through.
270
+
271
+ ## Filtering philosophy
272
+
273
+ The goal is not the shortest possible output. It is **the minimum context that
274
+ preserves diagnostic usefulness**. A 50-line report containing the clue beats a
275
+ 10-line report that removed it.
276
+
277
+ Every backtrace frame is classified as one of three things:
278
+
279
+ | Kind | What it is | What happens to it |
280
+ |------|-----------|--------------------|
281
+ | **project** | Code in your repository, plus Bundler `path:` gems and local engines | Always kept |
282
+ | **library** | Third-party code: Capybara, ActiveRecord, Rack, Net::HTTP, the standard library | Kept only where it touches your code |
283
+ | **framework** | The test runner, the loader, the CLI: rspec-core, rspec-expectations, Bundler, Thor, Rake, binstubs, `<internal:>` | Always dropped |
284
+
285
+ The distinction that matters is the middle row. Library frames are not noise — a
286
+ Capybara `find` or an ActiveRecord `save!` is often the only thing that tells you
287
+ *what operation failed*. So for each run of library frames that directly touches
288
+ first-party code, `rspec-signal` keeps:
289
+
290
+ - the **entry point**: the outermost library frame with a real method name, which is
291
+ the call your code actually made. Delegation shims whose frames are anonymous
292
+ blocks (`capybara/dsl.rb:52 in block (2 levels) in <module:DSL>`) are skipped in
293
+ favour of the frame that names the operation (`capybara/node/finders.rb:60 in find`).
294
+ - the **raise site**: the innermost frame of that run.
295
+ - one or two neighbours, budget permitting.
296
+
297
+ Library frames that touch no first-party code at all — a gem calling another gem
298
+ five layers down — are dropped and counted.
299
+
300
+ A worked example. Sixty-four frames in, six lines out:
301
+
302
+ ```text
303
+ capybara/node/finders.rb:312 in `synced_resolve'
304
+ [2 library frames omitted]
305
+ capybara/node/finders.rb:60 in `find'
306
+ [1 library frame omitted]
307
+ capybara/dsl.rb:52 in `block (2 levels) in <module:DSL>'
308
+ spec/system/reader_self_reading_integrity_spec.rb:104 in `block (3 levels) in <top (required)>'
309
+ [47 framework/runtime frames omitted]
310
+ ```
311
+
312
+ Two more rules keep the reduction honest:
313
+
314
+ - **First-party beats everything.** A frame in your repository is never classified as
315
+ framework plumbing, whatever it is named. A checkout in `~/src/rspec-signal-demo/`
316
+ does not lose all its own frames to a pattern match. The only exception is
317
+ binstubs (`bin/rspec`, `bin/bundle`), which are plumbing wherever they live.
318
+ - **The trace is never empty.** If a failure happens entirely inside a gem, or
319
+ entirely inside RSpec itself, the report falls back to the innermost non-framework
320
+ frames and says so: *"No first-party frames in this backtrace; innermost frames
321
+ shown instead."* Filtering that hides the only available evidence is a bug.
322
+
323
+ Two other things RSpec loses that `rspec-signal` keeps:
324
+
325
+ - **Exception causes.** RSpec puts the `Caused by:` chain in the *backtrace*, which
326
+ is exactly what gets reduced away. A `PG::UniqueViolation` behind a bland
327
+ `RuntimeError` is usually the answer, so the chain is folded into the message,
328
+ with the first-party frame it came from.
329
+ - **Aggregated failures.** For `:aggregate_failures`, RSpec deliberately empties the
330
+ message and moves the sub-failures into formatter-only callbacks, and the
331
+ exception's own backtrace is the aggregator's internals. `rspec-signal` recovers
332
+ both the sub-failures and the real backtrace from the first sub-exception.
333
+
334
+ ## Grouping behaviour
335
+
336
+ Forty-three failures that are one bug should read as one bug. Failures are collapsed
337
+ by a deterministic fingerprint of four components:
338
+
339
+ | Component | What it is |
340
+ |-----------|-----------|
341
+ | **exception class** | `Capybara::ElementNotFound`, `ActiveRecord::RecordInvalid`, ... |
342
+ | **normalized message** | The message with volatile parts masked: object addresses, UUIDs, timestamps, record ids, temp paths. Small numbers are left alone — `expected 3, got 4` and `expected 7, got 2` are different failures. |
343
+ | **culprit** | The innermost frame that is not test-runner plumbing: the code that actually raised. |
344
+ | **app context** | The innermost first-party frame outside your spec suite. `nil` for a plain matcher failure; decisive when the same error comes from two different call sites. |
345
+
346
+ Deliberately **not** in the fingerprint: the example description, and the example's
347
+ own location. Fourteen specs in six files that all trip over the same missing DOM
348
+ node are one problem, not fourteen.
349
+
350
+ The four components are what stop over-collapsing:
351
+
352
+ - Two `Capybara::ElementNotFound` failures for **different selectors** stay apart —
353
+ the message differs.
354
+ - Two `ActiveRecord::RecordInvalid` failures with the **same message** from
355
+ `SubscriptionCreator` and `InviteCreator` stay apart — the app context differs.
356
+ - Two identical `expect(x).to be true` failures in **different specs** stay apart —
357
+ a matcher failure raises at the spec line, so the culprit differs.
358
+ - The same missing record id in twenty specs collapses to one — ids are masked.
359
+
360
+ Each group renders one full trace, from the failure carrying the most first-party
361
+ frames, plus every affected example's location. Groups are ordered largest first,
362
+ with ties broken by run order, so two runs of the same suite produce byte-identical
363
+ reports.
364
+
365
+ Failures that are clearly connected but correctly *not* identical are handled by a
366
+ separate, looser layer — see [Related failure clustering](#related-failure-clustering).
367
+
368
+ ## Related failure clustering
369
+
370
+ Exact signatures are conservative on purpose: they never claim two failures are
371
+ the same failure unless they really are. That leaves a gap the real world fills
372
+ constantly — a dozen request specs that each wanted a different thing and all got
373
+ a 404, or one missing `data-testid` reported by `find` in one spec and by
374
+ `have_css` in another. Different signatures, obviously one problem.
375
+
376
+ So there is a second, deliberately looser layer. Each failure is scanned for **one
377
+ strong diagnostic symptom**, and failures sharing a symptom are reported together:
378
+
379
+ ```markdown
380
+ ## Related failures
381
+
382
+ Failures sharing one diagnostic symptom across more than one signature. Weaker than
383
+ a signature: a likely common cause, not a proven identical failure. The signatures
384
+ below remain authoritative.
385
+
386
+ ### R1. Unexpected 404 (Not Found) responses -- 12 examples across 12 signatures
387
+
388
+ - Symptoms: `expected 200, got 404` (6), `expected redirect, got 404` (6)
389
+ - Specs: `spec/requests/checkpoint_responses_controller_spec.rb`, ... and 6 more
390
+ - Signatures: #3, #4, #5, #6, #7, #8, #9, #10, and 4 more
391
+
392
+ ### R2. Missing css selector `[data-testid="reader-progress"] span` -- 4 examples across 2 signatures
393
+
394
+ - Specs: `spec/system/reader_progress_spec.rb`, `spec/system/reader_layout_spec.rb`, ...
395
+ - Signatures: #1, #2
396
+ ```
397
+
398
+ The symptoms, in the order they are tried:
399
+
400
+ | Symptom | Cluster key | Example |
401
+ |---------|-------------|---------|
402
+ | **HTTP status** | the status that actually came back | `expected 200, got 404` and `expected redirect, got 404` cluster together; `got 500` does not |
403
+ | **Route** | the path with identifiers masked, or the `_path` helper | `No route matches [GET] "/readers/:id/progress"` |
404
+ | **Selector / page text** | the selector, compared exactly | `Unable to find css "x"` and `expected to find css "x" but there were no matches` cluster together |
405
+ | **ActiveRecord** | the model, the validation sentence, or the missing column/table | `Couldn't find User` clusters with `Couldn't find User`, never with `Couldn't find Order` |
406
+ | **Ruby error** | the method *and* its receiver, or the constant | `undefined method 'progress' for nil` |
407
+ | **Exception class** | the class, only when namespaced | `PG::ConnectionBad`, `Errno::ECONNREFUSED` |
408
+
409
+ ### What stops it over-clustering
410
+
411
+ Clusters are a hint, and a hint that fires too often is worse than none. Five rules
412
+ hold the line:
413
+
414
+ 1. **A cluster needs two or more failures.**
415
+ 2. **A cluster needs two or more exact signatures.** If everything sharing a symptom
416
+ is already one signature, the signature section said it better, and repeating it
417
+ is pure noise. This is what keeps the section short.
418
+ 3. **Each failure joins at most one cluster** — the first symptom that matches, in
419
+ the order above. Nothing appears twice.
420
+ 4. **No similarity, ever.** Every symptom is anchored on a specific phrase from a
421
+ specific library. There is no fuzzy matching, no embedding, no threshold. A
422
+ failure that matches nothing has no symptom and joins nothing, which is the safe
423
+ direction to be wrong in.
424
+ 5. **Exception class is a last resort, and a narrow one.** It fires only for a
425
+ *namespaced* class, never for `RuntimeError`, `ArgumentError` or anything under
426
+ `RSpec::`, and never for a class an earlier symptom is responsible for — so a
427
+ `Capybara::ElementNotFound` whose message could not be parsed can never drag
428
+ unrelated selectors into one cluster.
429
+
430
+ Ordering is deterministic: largest cluster first, then most signatures spanned, then
431
+ run order.
432
+
433
+ ## Large HTML responses
434
+
435
+ A request spec expecting one sentence and receiving a Rails exception page produces
436
+ a diff several thousand lines long, and its first hundred lines are the exception
437
+ page's own CSS — the one part of the response guaranteed to be identical for every
438
+ failure in the suite. Capping it still spends the report's opening on stylesheet.
439
+
440
+ When the actual value is bulk HTML, `rspec-signal` replaces it:
441
+
442
+ ```text
443
+ Failure/Error: expect(response.body).to include("You've finished this document.")
444
+
445
+ expected [HTML document] to include "You've finished this document."
446
+
447
+ [HTML document: 6,371 lines, 284 KB -- markup omitted]
448
+ Title: Action Controller: Exception caught
449
+ Heading: NoMethodError in ReaderController#show
450
+ Message: undefined method 'progress' for nil
451
+ ```
452
+
453
+ - The **expected value is never touched** — it is the small, useful half.
454
+ - Detection is by shape: a `<!doctype html>`/`<html>` opening, or five or more tags
455
+ around structural elements. Both the one-enormous-inspected-line form and the
456
+ unified-diff form are handled.
457
+ - `<script>`, `<style>` and comments are stripped **before** any text is read, so
458
+ CSS can never be mistaken for content.
459
+ - Signals extracted: `<title>`, `<h1>`, `<h2>` or the first `<pre>`, falling back to
460
+ the leading visible text when a page has neither. On a Rails error page those are
461
+ the exception class and its message. Capybara's own `status_code` still appears
462
+ under **Browser state**.
463
+ - HTML small enough to read is left exactly as it was (`config.max_html_chars`,
464
+ default 1500 characters).
465
+ - No DOM parser, and no new dependency: regex only. It never has to be correct, only
466
+ useful, and it is handed broken markup by definition.
467
+ - If `write_full` is enabled, the untouched original is written to `full.txt`.
468
+
469
+ ## Configuration
470
+
471
+ None is required. Everything below is optional, in `spec_helper.rb`:
472
+
473
+ ```ruby
474
+ RSpec::Signal.configure do |config|
475
+ # Where artifacts go (relative to the project root, or absolute).
476
+ config.output_dir = "tmp/rspec-signal"
477
+
478
+ # Reduction budgets.
479
+ config.max_frames = 12 # frames kept per trace
480
+ config.max_external_context = 3 # library frames kept per adjacent run
481
+ config.max_project_frames = 8 # first-party frames kept per trace
482
+ config.fallback_frames = 6 # frames shown when nothing else survives
483
+
484
+ # Message budgets. Diffs are the biggest source of bloat.
485
+ config.max_message_lines = 30
486
+ config.max_diff_lines = 20
487
+
488
+ # Large HTML responses (see "Large HTML responses").
489
+ config.reduce_html = true
490
+ config.max_html_chars = 1500 # smallest HTML blob replaced by a summary
491
+
492
+ # Report budgets.
493
+ config.max_affected_examples = 25 # locations listed per group
494
+ config.max_groups = nil # signatures rendered in full (nil = all)
495
+
496
+ # Related failure clustering (see "Related failure clustering").
497
+ config.relate_failures = true
498
+ config.max_clusters = 10 # clusters rendered in full (nil = all)
499
+ config.max_cluster_specs = 6 # spec files listed per cluster
500
+
501
+ # Artifacts.
502
+ config.write_json = true
503
+ config.write_full = true
504
+ config.write_gitignore = true
505
+
506
+ # Behaviour.
507
+ config.enabled = true
508
+ config.terminal_summary = true
509
+
510
+ # Secret scrubbing (see Privacy).
511
+ config.redact = true
512
+ config.redaction_patterns = [/INTERNAL-[A-Z0-9]+/]
513
+ config.redaction_filter = ->(text) { text.gsub(customer_name, "[CUSTOMER]") }
514
+
515
+ # Classification.
516
+ config.project_root = Rails.root.to_s
517
+ config.extra_first_party = ["../billing-engine"] # a sibling checkout
518
+ config.ignore_patterns << %r{/lib/my_test_harness/} # your code, but plumbing
519
+ config.framework_patterns << /vendor_test_runner-/ # a gem, but plumbing
520
+
521
+ # Capybara.
522
+ config.capture_capybara = true
523
+ config.capture_page_html = false
524
+ end
525
+ ```
526
+
527
+ Two environment variables are honoured, which is usually what you want in CI:
528
+
529
+ ```bash
530
+ RSPEC_SIGNAL_DISABLE=1 # turn the gem off entirely
531
+ RSPEC_SIGNAL_OUTPUT_DIR=... # override the output directory
532
+ ```
533
+
534
+ Note the two classification lists. `ignore_patterns` is checked **before**
535
+ first-party detection — it is how you say "this is my code, but treat it as
536
+ plumbing". `framework_patterns` is checked **after** — it is for third-party code
537
+ and cannot accidentally swallow your repository.
538
+
539
+ ## Rails and Capybara
540
+
541
+ `rspec-signal` depends on `rspec-core` and nothing else. Rails, Capybara and
542
+ ActiveRecord are all optional; the gem gets richer when they are present.
543
+
544
+ **Rails.** The project root defaults to `Rails.root` when Rails is loaded, and the
545
+ report header records the Rails version. `app/`, `lib/`, `spec/`, and local engines
546
+ are first-party; `vendor/bundle` is not.
547
+
548
+ **System and feature specs.** For a failing example with `type: :system`,
549
+ `type: :feature`, or `js: true`, an `after(:each)` hook captures browser state into
550
+ the report:
551
+
552
+ ```markdown
553
+ **Browser state**
554
+
555
+ - URL: `https://app.test/library`
556
+ - Page title: `Library`
557
+ - Status: `200`
558
+ - Console: `SEVERE: Uncaught TypeError: shelf.render is not a function`
559
+ - Screenshot: `tmp/screenshots/failures_reader_shelf.png`
560
+ ```
561
+
562
+ Rails writes the screenshot itself on a system-test failure; `rspec-signal` finds the
563
+ path in `metadata[:extra_failure_lines]` and surfaces it as a link rather than
564
+ leaving it buried in the message. Browser console output is small and frequently
565
+ contains the real cause of a JavaScript-driven failure.
566
+
567
+ The whole capture is best effort and wrapped in rescues: a driver that cannot report
568
+ a status code just contributes less detail. It never asks Capybara for
569
+ `current_session`, which would *create* a session and possibly boot a browser — it
570
+ only reads a session that the test already opened.
571
+
572
+ Page HTML is **not** saved by default. It is frequently the clue, and it is also
573
+ 100 KB of exactly the bloat this gem exists to remove. Enable it deliberately with
574
+ `config.capture_page_html = true`, and the report will link to the saved file.
575
+
576
+ ## Privacy
577
+
578
+ Failure output contains whatever your tests put in it: fixture data, request bodies,
579
+ headers, environment values. These artifacts are explicitly meant to be handed to
580
+ external AI systems, so `rspec-signal` scrubs obvious credentials by default —
581
+ `Authorization` headers, credential-shaped assignments (`password:`, `api_key=`,
582
+ `?access_token=`), URL userinfo, and well-known token formats (AWS, GitHub, Slack,
583
+ Stripe, GitLab, Google, JWTs, PEM private key blocks).
584
+
585
+ It targets shapes that are unambiguous and deliberately does not guess at arbitrary
586
+ values, because false positives destroy the diagnostic value of a report:
587
+ `expected 3 items, got 4` and `undefined method 'password_digest' for nil` are left
588
+ exactly as they are.
589
+
590
+ **This is a safety net, not a guarantee. Review artifacts before sending them
591
+ somewhere you do not control.** `tmp/rspec-signal/.gitignore` is written
592
+ automatically so they do not reach version control by accident.
593
+
594
+ You can add your own patterns with `config.redaction_patterns`, post-process
595
+ everything with `config.redaction_filter`, or turn scrubbing off with
596
+ `config.redact = false`.
597
+
598
+ ## Limitations
599
+
600
+ - **Errors outside examples.** A `before(:suite)` blow-up, or a spec file that fails
601
+ to load, produces no failed examples. RSpec reports those only through its message
602
+ stream, which a formatter cannot subscribe to without swallowing every other
603
+ message. `rspec-signal` reports the count and says where to look, but cannot
604
+ capture the text.
605
+ - **Parallel test runners.** Use `rspec-signal-parallel` for suites launched with
606
+ `parallel_tests` (see [Parallel suites](#parallel-suites-parallel_tests)); it isolates
607
+ each worker and merges the results deterministically. Do not set a per-worker
608
+ `RSPEC_SIGNAL_OUTPUT_DIR` with the wrapper -- it requires every worker to report the
609
+ same configuration and fails the merge if the output directory differs between them.
610
+ Running `parallel_rspec` directly, with the gem only required from `spec_helper.rb`
611
+ and no wrapper, is not supported: every worker writes to the same `signal.md` with a
612
+ non-atomic write, and a worker that finishes green can delete a sibling worker's
613
+ still-relevant report.
614
+ - **Grouping is a heuristic.** It is deterministic and explained above, but two
615
+ genuinely different bugs that raise the same exception with the same message from
616
+ the same line will collapse into one signature. The affected-example list always
617
+ shows you everything that was merged.
618
+ - **Related clusters are a hint, not a diagnosis.** They say two failures share a
619
+ symptom, never that they share a cause; the report says so in those words. The
620
+ symptom list is finite, so a suite whose failures are worded in a way no extractor
621
+ recognises simply gets no clusters — a silence, not a wrong answer.
622
+ - **HTML reduction reads markup with regexes.** It extracts the title, headings and
623
+ leading text of a page; it does not understand the page. A response whose useful
624
+ content is buried deep in the body will be summarised as an HTML document and
625
+ little more. Enable `write_full` temporarily if you need the original markup.
626
+ - **No source packaging.** `rspec-signal` names files and lines; it does not embed
627
+ source code. Agents working inside a repository can open the files themselves, and
628
+ in v1 that is a better division of labour than guessing which snippets to inline.
629
+ - **Reduction depends on your suite.** Failures that are already distinct and already
630
+ shallow reduce very little. That is the correct outcome, not a failure of the gem.
631
+
632
+ ## Compatibility
633
+
634
+ - Ruby 3.1+
635
+ - RSpec 3.10+ (via `rspec-core`)
636
+ - Rails, Capybara, ActiveRecord: optional
637
+
638
+ The only runtime dependency is `rspec-core`.
639
+
640
+ ## Development
641
+
642
+ ```bash
643
+ bin/setup # or: bundle install
644
+ bundle exec rake # specs + RuboCop
645
+ bundle exec rspec
646
+ bundle exec rubocop
647
+ ```
648
+
649
+ The test suite runs `rspec-signal` on itself, so a failing run writes its own
650
+ report to `tmp/rspec-signal/signal.md`.
651
+
652
+ Most of the gem is plain Ruby with no RSpec dependency. Only `Formatter` and
653
+ `FailureBuilder` touch RSpec's notification API, which is what lets the reduction,
654
+ grouping and rendering stages be tested directly against synthetic fixtures in
655
+ `spec/fixtures/backtraces.rb`. `spec/integration/end_to_end_spec.rb` shells out to
656
+ real `rspec` processes in generated throwaway projects.
657
+
658
+ Tests deliberately assert on output *size* as well as content, so a future change
659
+ cannot quietly let noise back in.
660
+
661
+ ## Contributing
662
+
663
+ Bug reports and pull requests are welcome at
664
+ <https://github.com/SilenceDogood1984/rspec-signal>.
665
+
666
+ If you are reporting a case where reduction removed something important, the most
667
+ useful thing you can attach is the raw backtrace — a new fixture in
668
+ `spec/fixtures/backtraces.rb` is the ideal shape for it.
669
+
670
+ ## License
671
+
672
+ MIT. See [LICENSE](LICENSE).
data/exe/rspec-signal ADDED
@@ -0,0 +1,11 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ ENV["RSPEC_SIGNAL_QUIET"] = "1"
5
+ arguments = ["--require", "rspec/signal", "--format", "RSpec::Signal::Formatter", *ARGV]
6
+
7
+ # Run the same executable as `bundle exec rspec`. Besides avoiding a parallel
8
+ # implementation of RSpec's CLI, this is important because the executable uses
9
+ # Runner.invoke, which applies RSpec's default spec discovery when no paths are
10
+ # supplied. Runner.run is a lower-level API and does not provide that behavior.
11
+ exec Gem.ruby, Gem.bin_path("rspec-core", "rspec"), *arguments