specguard-rspec 0.2.1 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +437 -79
- data/bin/specguard-ingest +40 -0
- data/lib/specguard/rspec/annotation_scanner.rb +50 -4
- data/lib/specguard/rspec/cli.rb +26 -11
- data/lib/specguard/rspec/finding.rb +16 -5
- data/lib/specguard/rspec/formatter.rb +173 -12
- data/lib/specguard/rspec/ingest_cli.rb +942 -0
- data/lib/specguard/rspec/ingest_reporter.rb +280 -0
- data/lib/specguard/rspec/json_reporter.rb +1 -2
- data/lib/specguard/rspec/linter.rb +33 -28
- data/lib/specguard/rspec/payload_normalizer.rb +6 -2
- data/lib/specguard/rspec/scanner.rb +142 -124
- data/lib/specguard/rspec/schema.rb +17 -12
- data/lib/specguard/rspec/transport.rb +143 -4
- data/lib/specguard/rspec/validator_backend.rb +58 -43
- data/lib/specguard/rspec/version.rb +1 -1
- data/lib/specguard/rspec/violation_renderer.rb +37 -33
- data/script/bump-version.sh +19 -7
- metadata +25 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 544f8fefb100531adc58a42d6ee9018572605253fd84f76d213ae8ca55f1c687
|
|
4
|
+
data.tar.gz: 4856ee3ffe6237ea7d6eac2d49a3f0d5ff7a27d1b867c0d380b8118997e991b7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: bc93bc698eaa81c4b3ab43507371d990ef079d3f9df0f816af1e69d440cb1db91201dedb5298ab385a047b14b6453ca13d856b4618400822bf69d7c13e87a0ac
|
|
7
|
+
data.tar.gz: 5f0fe5d59ec173fd3a91980de3c097766e77f650f5601da9501d4bb5f739450b35c88476edb78a3d1e51008398e10847fc6902c3447ef4b8868f5108a0768a91
|
data/README.md
CHANGED
|
@@ -4,6 +4,8 @@
|
|
|
4
4
|
> and a CLI linter that validates `@intent` annotations.
|
|
5
5
|
|
|
6
6
|
Two independent tools, one dependency — the [OpenTestIntent](https://github.com/yatfa-ai/open-test-intent) annotation format.
|
|
7
|
+
A third command, [`specguard-ingest`](#replaying-a-saved-run--specguard-ingest), belongs to the first of them: it replays a
|
|
8
|
+
run the formatter saved when the endpoint could not be reached.
|
|
7
9
|
|
|
8
10
|
## Install
|
|
9
11
|
|
|
@@ -26,17 +28,17 @@ bundle exec specguard-lint # one-off audit: every *_spec.rb
|
|
|
26
28
|
```
|
|
27
29
|
|
|
28
30
|
Files are **positional** (`specguard-lint spec/order_spec.rb`); there is no `--source` flag —
|
|
29
|
-
that belongs to
|
|
30
|
-
|
|
31
|
-
The linter is an independent implementation of the same protocol
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
A handful of differences are ratified as such
|
|
37
|
-
*still* differ; everything else matches byte for
|
|
38
|
-
the backend below — two read failures, a
|
|
39
|
-
wording at all.
|
|
31
|
+
that belongs to `validate-intent`, not to this one.
|
|
32
|
+
|
|
33
|
+
The linter is an independent implementation of the same protocol — written against
|
|
34
|
+
[`PROTOCOL.md`](https://github.com/yatfa-ai/open-test-intent/blob/main/PROTOCOL.md) and the
|
|
35
|
+
canonical schema, which are what decide whether an annotation is valid. Its agreement with
|
|
36
|
+
`validate-intent` is checked by replaying that tool's own recorded reports through this CLI and
|
|
37
|
+
comparing findings, ordering and exit codes, in
|
|
38
|
+
`spec/specguard/rspec/validator_backend_spec.rb`. A handful of differences are ratified as such
|
|
39
|
+
there, each with its reason and each asserted to *still* differ; everything else matches byte for
|
|
40
|
+
byte. They are the same ones described under the backend below — two read failures, a
|
|
41
|
+
parse-failure message, and one that is not about wording at all.
|
|
40
42
|
|
|
41
43
|
### Machine-readable output (`--json`)
|
|
42
44
|
|
|
@@ -64,8 +66,8 @@ bundle exec specguard-lint --json spec/models/order_spec.rb
|
|
|
64
66
|
|
|
65
67
|
This is the **same document** `validate-intent --json --source` emits, key for key — the gem
|
|
66
68
|
already *consumes* it when the Go backend is on, and a consumer of both tools should not need two
|
|
67
|
-
parsers for one protocol. It is not byte-identical
|
|
68
|
-
|
|
69
|
+
parsers for one protocol. It is not byte-identical; it is key-, type- and value-identical, which is
|
|
70
|
+
what a parser sees.
|
|
69
71
|
|
|
70
72
|
| field | meaning |
|
|
71
73
|
| --------------------- | ------- |
|
|
@@ -285,77 +287,68 @@ is worth stating precisely rather than reassuringly.
|
|
|
285
287
|
For every payload both JSON parsers accept, the two backends agree completely: the same finding
|
|
286
288
|
against the same file at the same line, the same classification, the same counts and the same exit
|
|
287
289
|
code. **The messages in the table below differ in their trailing text only** — the rows *are* the
|
|
288
|
-
enumeration, not a sample of it, and
|
|
289
|
-
suite rather than leaving a stale claim here — in
|
|
290
|
-
`spec/specguard/rspec/validator_backend_spec.rb
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
| a
|
|
296
|
-
| a
|
|
297
|
-
| a path that
|
|
290
|
+
enumeration, not a sample of it, and all four are asserted in both directions, so closing one fails
|
|
291
|
+
the suite rather than leaving a stale claim here — in
|
|
292
|
+
`spec/specguard/rspec/validator_backend_spec.rb`, where each row's comparison is labelled
|
|
293
|
+
`ENUMERATED DIFFERENCE n of 4` under the number it carries here:
|
|
294
|
+
|
|
295
|
+
| # | input | Ruby path | Go backend |
|
|
296
|
+
|---|---|---|---|
|
|
297
|
+
| 1 | a payload that is still not JSON after normalisation | Ruby's `JSON::ParserError` text | `expected a JSON value (line 1, column 102)` |
|
|
298
|
+
| 2 | a file that is not valid UTF-8 | `invalid UTF-8 byte sequence` | `input is not well-formed UTF-8 (PROTOCOL.md §1.1 requires it)` |
|
|
299
|
+
| 3 | a path that does not exist | `No such file or directory @ rb_sysopen - …` | `no file at this path` |
|
|
300
|
+
| 4 | a path that is not a regular file | `Is a directory @ io_fread - …` | `no file at this path` |
|
|
298
301
|
|
|
299
302
|
The first row is the one you are most likely to actually see: `parse` is one of the three things
|
|
300
303
|
the linter reports, and **every** malformed-JSON annotation renders differently under the backend.
|
|
301
|
-
The Ruby path interpolates Ruby's `JSON::ParserError`; the validator
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
304
|
+
The Ruby path interpolates Ruby's `JSON::ParserError`; the validator has its own prose, and the
|
|
305
|
+
backend passes it through unaltered rather than inventing a third spelling. `PROTOCOL.md` specifies
|
|
306
|
+
the accepted JSON *language*, not the words a validator refuses in, so two spellings of one refusal
|
|
307
|
+
are both conformant. Both agree on which annotation broke, and on the line and column — only the
|
|
308
|
+
prose moves. The other rows are read failures and need an unreadable path to reach at all.
|
|
309
|
+
|
|
310
|
+
Rows 3 and 4 share a Go column, and that is the substance of row 4 rather than a typo. The
|
|
311
|
+
binary's arguments are glob *patterns* and a match is filtered to regular files, so a directory and
|
|
312
|
+
a name matching nothing reach it as the same answer; the Ruby path opens the path it was given, so
|
|
313
|
+
it has an errno and names which one. Ruby tells the two apart, the backend cannot, and neither
|
|
314
|
+
pretends otherwise.
|
|
305
315
|
|
|
306
316
|
#### The difference that is not about wording
|
|
307
317
|
|
|
308
|
-
The two JSON parsers do not accept the same language
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
1. the
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
On such a payload the backend does not word the failure differently; it
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
the
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
`max_nesting: false` would close the non-finite case, but the surrogate case has no such option and
|
|
343
|
-
would mean porting CPython's string decoder into this gem — and both would change what the
|
|
344
|
-
**default** Ruby path does, which the slice that added this backend deliberately holds fixed. See
|
|
345
|
-
`Scanner#parse` for the full reasoning; whether the gem should adopt CPython's acceptance grammar
|
|
346
|
-
is left open there rather than settled.
|
|
347
|
-
|
|
348
|
-
**If you go and count them.** The parity harness numbers this same enumeration two ways, and both
|
|
349
|
-
numbers are correct, so it is worth saying which is which before you follow the link. It *asserts*
|
|
350
|
-
them as six entries, `(i)`–`(vi)`, under "8b. the six enumerated backend differences" — the rows of
|
|
351
|
-
the table above, plus the two shapes the acceptance set takes. It *groups* them in its header as
|
|
352
|
-
four mechanisms, `(a)`–`(d)`, because the two unreadable-path rows share one cause and the two
|
|
353
|
-
acceptance-set shapes share another. Six entries, four mechanisms, one enumeration.
|
|
354
|
-
|
|
355
|
-
Those two totals live in that harness, which asserts them. This page does not restate them beside
|
|
356
|
-
the table, because a count kept in two places is a count that will eventually disagree with itself
|
|
357
|
-
— which is exactly how an earlier revision of this section came to announce three messages directly
|
|
358
|
-
above four rows.
|
|
318
|
+
The two JSON parsers do not accept quite the same language, and the difference is now small and
|
|
319
|
+
runs the *opposite* way from how it used to.
|
|
320
|
+
|
|
321
|
+
`PROTOCOL.md` §1.1 states the accepted language — an RFC 8259 JSON text, with the three points that
|
|
322
|
+
RFC leaves to the implementation settled explicitly. The validator refuses the non-finite literals
|
|
323
|
+
(§1.1(b)), unpaired surrogate escapes (§1.1(a)) and nesting past 100 (§1.1(c)). Ruby's `JSON.parse`
|
|
324
|
+
refuses or limits all three too, so on those three the two now **agree**. (They did not before: the
|
|
325
|
+
validator's parser used to reproduce a foreign runtime's grammar, which the protocol had never
|
|
326
|
+
specified. Removing that is what SPGD-403 did.)
|
|
327
|
+
|
|
328
|
+
**What survives is this gem being the more permissive side**, in two places:
|
|
329
|
+
|
|
330
|
+
1. **A lone LOW surrogate escape** (`\udc00`–`\udfff`). `JSON.parse` accepts it; §1.1(a) refuses it,
|
|
331
|
+
because a surrogate escape must form a pair. Ruby refuses only the HIGH half, which is why the
|
|
332
|
+
rule is narrower than "surrogate escapes diverge".
|
|
333
|
+
2. **The nesting boundary**, which sits a little deeper here than §1.1(c)'s 100.
|
|
334
|
+
|
|
335
|
+
The first one is not only a verdict difference. What `JSON.parse` returns for `"\udc00"` is a
|
|
336
|
+
String whose `valid_encoding?` is **false**: it cannot be re-serialised and cannot cross the ingest
|
|
337
|
+
transport, so a payload this gem calls valid is one it cannot send. That is the cost §1.1(a) exists
|
|
338
|
+
to remove, and it is still paid on the Ruby path.
|
|
339
|
+
|
|
340
|
+
On such a payload the backend does not word the failure differently; it *has* one where the Ruby
|
|
341
|
+
path does not. And because the only surviving member lives inside a string — a slot the schema
|
|
342
|
+
declares — the payload is otherwise schema-valid, so the backend **exits 1 where the Ruby path
|
|
343
|
+
exits 0**. This is the one input on which the two backends disagree about whether your suite passes.
|
|
344
|
+
It is enumerated and asserted from both sides in
|
|
345
|
+
`spec/specguard/rspec/validator_backend_spec.rb`, along with the convergence above, so a validator
|
|
346
|
+
that went back to accepting a superset of JSON fails there by name.
|
|
347
|
+
|
|
348
|
+
Both are ratified rather than fixed, and the reason is scope: this gem's hand-rolled validation
|
|
349
|
+
logic is slated for **removal** by the roadmap that owns the validator rather than for repair, and
|
|
350
|
+
closing the gap here would change what the **default** Ruby path does, which the slice that added
|
|
351
|
+
this backend deliberately holds fixed. See `Scanner#parse` for the full reasoning.
|
|
359
352
|
|
|
360
353
|
Every way the backend can fail — the binary is missing, will not execute, exits with something that
|
|
361
354
|
is not a verdict, or emits output that is not a report — is **exit 2**, the linter's "could not do
|
|
@@ -498,13 +491,34 @@ rather than like a broken build.
|
|
|
498
491
|
the run (a `401` from a rotated key, a `400`, a `500`) or cannot be reached at
|
|
499
492
|
all (connection refused, DNS failure, timeout), the formatter prints **one**
|
|
500
493
|
line to stderr naming the status or the error, and writes the payload to
|
|
501
|
-
`log/test_results.jsonl` so the run can be replayed later
|
|
494
|
+
`log/test_results.jsonl` so the run can be replayed later with
|
|
495
|
+
[`specguard-ingest`](#replaying-a-saved-run--specguard-ingest):
|
|
502
496
|
|
|
503
497
|
```
|
|
504
498
|
SpecGuard: could not deliver test telemetry (HTTP 401 — the API key was not
|
|
505
499
|
accepted). Falling back to log/test_results.jsonl; the test run is unaffected.
|
|
506
500
|
```
|
|
507
501
|
|
|
502
|
+
**That line carries the endpoint's own words when it has any.** A `400` refusal
|
|
503
|
+
names the offending spec by index, file and line, so a rejected payload is a
|
|
504
|
+
thing you can fix from the CI log rather than one you have to reproduce
|
|
505
|
+
locally:
|
|
506
|
+
|
|
507
|
+
```
|
|
508
|
+
SpecGuard: could not deliver test telemetry (HTTP 400 — the endpoint rejected
|
|
509
|
+
the payload — spec 3 (spec/orders_spec.rb:9): line_number is required and must
|
|
510
|
+
be a positive integer; spec 7 (spec/orders_spec.rb:31): outcome must be one of
|
|
511
|
+
passed, failed, pending). Falling back to log/test_results.jsonl; the test run
|
|
512
|
+
is unaffected.
|
|
513
|
+
```
|
|
514
|
+
|
|
515
|
+
It stays **one** line whatever comes back. A systemic problem can have the
|
|
516
|
+
endpoint refusing every spec in the suite, so at most three reasons are spelled
|
|
517
|
+
out and the rest are counted (`… and 497 more`); anything that arrives without
|
|
518
|
+
a reason it can read — an empty body, or the HTML a proxy answers a `413` with
|
|
519
|
+
— prints the bare status line above and is still reported as a refusal, not as
|
|
520
|
+
an error.
|
|
521
|
+
|
|
508
522
|
There are **no retries**, and the whole delivery is bounded by `timeout`
|
|
509
523
|
(10 seconds by default, against `Net::HTTP`'s own 60): telemetry is explicitly
|
|
510
524
|
allowed to be lost, and a retry would only double what a hung endpoint can cost
|
|
@@ -525,8 +539,27 @@ same corruption, deferred until something replays it — and says so once:
|
|
|
525
539
|
SpecGuard: skipped test telemetry for a dry run (rspec --dry-run executes no
|
|
526
540
|
example bodies, so this run's durations and outcomes would not be
|
|
527
541
|
measurements). Nothing was sent or written; the test run is unaffected.
|
|
542
|
+
Annotation coverage is a fact about source, not about execution, so it
|
|
543
|
+
survives the refusal — this working tree: 6 examples, 3 annotated,
|
|
544
|
+
3 unannotated (50% annotated).
|
|
545
|
+
the 3 unannotated examples, by definition site:
|
|
546
|
+
spec/orders_spec.rb:7 Order has no annotation
|
|
547
|
+
spec/orders_spec.rb:16 Order has a malformed annotation
|
|
548
|
+
spec/orders_spec.rb:21 Order has a schema-invalid annotation
|
|
528
549
|
```
|
|
529
550
|
|
|
551
|
+
The refusal throws away less than it used to. `duration` and `outcome` are
|
|
552
|
+
fabricated by a dry run, which is what makes them unpublishable — but the
|
|
553
|
+
third field the formatter computes per example, `annotated` / `unannotated`,
|
|
554
|
+
comes from scanning the **spec file's source text** and is identical whether or
|
|
555
|
+
not a body ran. Since a dry run still builds every example, it holds the exact
|
|
556
|
+
numerator and denominator of the annotation-coverage metric, so `--dry-run` is
|
|
557
|
+
also the way to ask *"where are we?"* without a commit, a push, and a CI round
|
|
558
|
+
trip. The figure describes **your working tree right now**, so it will differ
|
|
559
|
+
from the dashboard's the moment you edit a spec — that difference is the point.
|
|
560
|
+
|
|
561
|
+
Nothing is published either way: the report goes to stderr and to nowhere else.
|
|
562
|
+
|
|
530
563
|
This matters most where you are least likely to look for it: an API key is
|
|
531
564
|
usually an environment-level secret rather than a job-level one, so a lint job
|
|
532
565
|
that runs `rspec --dry-run` to catch an unparseable spec file inherits the key
|
|
@@ -540,6 +573,331 @@ alone. A non-2xx response gets the same treatment: `Net::HTTP` returns those as
|
|
|
540
573
|
ordinary values rather than raising, so they are checked for explicitly instead
|
|
541
574
|
of being left to a `rescue` that would never see them.
|
|
542
575
|
|
|
576
|
+
### Replaying a saved run — `specguard-ingest`
|
|
577
|
+
|
|
578
|
+
The suite is over by the time you see the `401`, and re-running it to recover
|
|
579
|
+
the telemetry costs you the whole suite again. So the file the formatter wrote
|
|
580
|
+
is the run: each line is byte-for-byte the body the endpoint refused, and
|
|
581
|
+
`specguard-ingest` is the command that sends it.
|
|
582
|
+
|
|
583
|
+
```bash
|
|
584
|
+
export SPECGUARD_API_KEY=… # the key that was rotated, fixed
|
|
585
|
+
bundle exec specguard-ingest log/test_results.jsonl
|
|
586
|
+
```
|
|
587
|
+
|
|
588
|
+
```
|
|
589
|
+
line 1: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
|
|
590
|
+
line 2: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
|
|
591
|
+
specguard-ingest: delivered 2 of 2 runs from log/test_results.jsonl
|
|
592
|
+
specguard-ingest: lines 1, 2 carried ci_run_id 17442 and each came back with
|
|
593
|
+
test_run_id 41f2c9b8 — the endpoint folded them onto one run
|
|
594
|
+
```
|
|
595
|
+
|
|
596
|
+
It reads the same `SPECGUARD_ENDPOINT`, `SPECGUARD_API_KEY` and
|
|
597
|
+
`SPECGUARD_TIMEOUT` the formatter does, and sends each line through the same
|
|
598
|
+
transport — so a run that was refused for a rotated key appears on the platform
|
|
599
|
+
once the secret is fixed, with the shard folding described in
|
|
600
|
+
[If you shard your suite](#if-you-shard-your-suite) applying exactly as it would
|
|
601
|
+
have during the run.
|
|
602
|
+
|
|
603
|
+
> **It re-delivers *every* line in the file you give it.** The formatter writes
|
|
604
|
+
> to `log/test_results.jsonl` when a delivery **failed** *and* when no API key
|
|
605
|
+
> was configured at all — and the two are indistinguishable on the line, because
|
|
606
|
+
> nothing in the payload records which sink it was destined for. So a laptop's
|
|
607
|
+
> file is a file of ordinary local runs, and this command will send all of them.
|
|
608
|
+
> There is no filter and no heuristic: guessing which lines "were failures" from
|
|
609
|
+
> data that does not say would be confidently wrong about which of your runs
|
|
610
|
+
> reach the platform. Check the file first with
|
|
611
|
+
> [`--list`](#checking-a-file-before-you-send-it----list), and check it before
|
|
612
|
+
> you replay one you did not write.
|
|
613
|
+
|
|
614
|
+
#### Checking a file before you send it — `--list`
|
|
615
|
+
|
|
616
|
+
`--list` prints one row per line and **delivers nothing**:
|
|
617
|
+
|
|
618
|
+
```bash
|
|
619
|
+
bundle exec specguard-ingest --list log/test_results.jsonl
|
|
620
|
+
```
|
|
621
|
+
|
|
622
|
+
```
|
|
623
|
+
line 1: branch main, commit_sha 0d4a1f2c9b8e7d6a5f4c3b2a1908f7e6d5c4b3a2, ci_run_id 17442, 412 examples, 93.4s
|
|
624
|
+
line 2: branch main, commit_sha 0d4a1f2c9b8e7d6a5f4c3b2a1908f7e6d5c4b3a2, ci_run_id 17442, 388 examples, 91.2s
|
|
625
|
+
line 3: branch spike/local, commit_sha 9c2e7a10b4d3, no ci_run_id, 6 examples, 0.4s
|
|
626
|
+
line 4: unparseable — could not parse the line as JSON: unexpected end of input
|
|
627
|
+
specguard-ingest: listed 4 lines from log/test_results.jsonl; nothing was delivered
|
|
628
|
+
```
|
|
629
|
+
|
|
630
|
+
Every field on the row is already on the line — nothing is guessed at, and a
|
|
631
|
+
line the command cannot parse is listed **as unparseable** rather than quietly
|
|
632
|
+
dropped from the preview. `no ci_run_id` is the one to read for: that line has
|
|
633
|
+
no identity for SpecGuard to fold a redelivery onto, so sending it creates a new
|
|
634
|
+
run rather than joining an existing one.
|
|
635
|
+
|
|
636
|
+
Reading the file yourself is not the alternative. One line is one whole run, and
|
|
637
|
+
at 20,000 examples that is megabytes of JSON on a single physical line.
|
|
638
|
+
|
|
639
|
+
**It needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`** — deliberately.
|
|
640
|
+
The file most worth checking is the one written *because* no API key was set, so
|
|
641
|
+
requiring a key to look at it would withdraw the instrument in exactly the
|
|
642
|
+
situation that produces the hazard. It composes with `--from-line` and `--lines`
|
|
643
|
+
too, so you can list the exact set you are about to send:
|
|
644
|
+
|
|
645
|
+
```bash
|
|
646
|
+
bundle exec specguard-ingest --list --from-line 7 log/test_results.jsonl
|
|
647
|
+
bundle exec specguard-ingest --list --lines 3,7,12-15 log/test_results.jsonl
|
|
648
|
+
```
|
|
649
|
+
|
|
650
|
+
A listing under either selector previews **exactly** the lines the same command
|
|
651
|
+
without `--list` would deliver, by the same numbers.
|
|
652
|
+
|
|
653
|
+
Listing sends nothing, so it can never be a verdict about a run: it exits `0`
|
|
654
|
+
when it listed the file and `2` when it could not read it or the flags were
|
|
655
|
+
wrong. **`1` is unreachable with `--list`.**
|
|
656
|
+
|
|
657
|
+
**Each line is reported by its line number**, and `--from-line N` starts at one —
|
|
658
|
+
so a file that was only partly accepted is resumed from the line the report
|
|
659
|
+
named, rather than blindly re-sent:
|
|
660
|
+
|
|
661
|
+
```bash
|
|
662
|
+
bundle exec specguard-ingest --from-line 7 log/test_results.jsonl
|
|
663
|
+
```
|
|
664
|
+
|
|
665
|
+
The numbering never shifts: line 7 is line 7 of the file you gave it, both times.
|
|
666
|
+
Re-sending a line that already landed is harmless *only* when it carries a
|
|
667
|
+
`ci_run_id` — that is the identity SpecGuard folds a redelivery onto. A line
|
|
668
|
+
**without** one has nothing to fold onto and becomes a second run, and a keyless
|
|
669
|
+
local file is made entirely of those, so `--from-line` is worth the two seconds
|
|
670
|
+
it takes to read the previous report.
|
|
671
|
+
|
|
672
|
+
#### Sending a set rather than a suffix — `--lines`
|
|
673
|
+
|
|
674
|
+
`--from-line` can only express a **suffix**, and the set a per-line report points
|
|
675
|
+
at is a suffix at most once. `--lines` takes the set itself — comma-separated
|
|
676
|
+
numbers and ranges, over the file's own numbering:
|
|
677
|
+
|
|
678
|
+
```bash
|
|
679
|
+
bundle exec specguard-ingest --lines 3,7,12-15 log/test_results.jsonl
|
|
680
|
+
```
|
|
681
|
+
|
|
682
|
+
```
|
|
683
|
+
line 3: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
|
|
684
|
+
line 7: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
|
|
685
|
+
line 12: accepted — HTTP 202, test_run_id 5a3d0e91, ci_run_id 17443
|
|
686
|
+
…
|
|
687
|
+
specguard-ingest: delivered 6 of 6 runs from log/test_results.jsonl; 34 lines not selected by --lines
|
|
688
|
+
```
|
|
689
|
+
|
|
690
|
+
Two things want it. The first is an **interior line that will never be accepted**:
|
|
691
|
+
an HTTP `400` is the one response SpecGuard forms an opinion about your payload
|
|
692
|
+
in, so a line it refuses is refused every time it is offered. Sitting at line 3
|
|
693
|
+
of a 40-line file, no `--from-line` can step over it — the file can never be
|
|
694
|
+
replayed to completion, and the command can never exit `0` over it. Naming the
|
|
695
|
+
set around it can:
|
|
696
|
+
|
|
697
|
+
```bash
|
|
698
|
+
bundle exec specguard-ingest --lines 1-2,4-40 log/test_results.jsonl
|
|
699
|
+
```
|
|
700
|
+
|
|
701
|
+
The second is that the sink is **append-only and mixes both sources**, so
|
|
702
|
+
ordinary keyless laptop runs keep landing *after* the CI failures you want to
|
|
703
|
+
replay. Every unwanted keyless line a too-early `--from-line` sweeps up is a
|
|
704
|
+
spurious run on the platform, because a line with no `ci_run_id` has nothing to
|
|
705
|
+
fold onto.
|
|
706
|
+
|
|
707
|
+
Carving the file up first (`sed -n '21,24p' file > tmp.jsonl`) is not the
|
|
708
|
+
alternative: a carved file **renumbers**, and the whole value of acting on a
|
|
709
|
+
per-line report is that line 12 is still line 12.
|
|
710
|
+
|
|
711
|
+
The held-back lines are **counted and reported**, exactly as `--from-line`'s and
|
|
712
|
+
the blank ones are — a summary that quietly narrowed what it was summarising
|
|
713
|
+
would be worse than no summary.
|
|
714
|
+
|
|
715
|
+
A spec is read strictly, and a bad one is a `2` rather than a fallback to the
|
|
716
|
+
whole file — which is the one outcome a selector exists to prevent. `--lines 0`,
|
|
717
|
+
`--lines 5-2`, `--lines abc`, `--lines 12-`, an empty spec and an empty entry
|
|
718
|
+
(`3,,5`) are all refused, naming what was wrong. Whitespace *between* entries is
|
|
719
|
+
fine (`3, 7`); inside one it is a typo, not a range (`5 - 7` is refused).
|
|
720
|
+
|
|
721
|
+
**`--lines` and `--from-line` do not combine** — giving both is a `2`:
|
|
722
|
+
|
|
723
|
+
```
|
|
724
|
+
specguard-ingest: error: --from-line and --lines both choose which lines to send; give one or the other
|
|
725
|
+
```
|
|
726
|
+
|
|
727
|
+
They answer the same question, and intersecting them would silently drop a
|
|
728
|
+
number you typed: `--from-line 5 --lines 3,7` would send only line 7, and the 3
|
|
729
|
+
would vanish without a word. Refusing the pair is the same discipline as the
|
|
730
|
+
rest of this command — it will not quietly narrow what it was asked for.
|
|
731
|
+
|
|
732
|
+
Repeating **one** selector is a different case and is allowed: the last one
|
|
733
|
+
wins. `--lines 1,2 --lines 4` sends line 4, and `--from-line 2 --from-line 5`
|
|
734
|
+
starts at 5. A repeat replaces rather than intersects, so the set delivered is
|
|
735
|
+
exactly the last one you typed — nothing is combined into something smaller than
|
|
736
|
+
you asked for, which is the objection to the pair above. It is also what lets
|
|
737
|
+
you override a selector baked into a wrapper script or shell alias by appending
|
|
738
|
+
a new one.
|
|
739
|
+
|
|
740
|
+
Nothing about a line's **content** is consulted by either flag. The numbers come
|
|
741
|
+
from you, after reading `--list`; that is what keeps this an explicit selector
|
|
742
|
+
rather than the heuristic this command refuses to grow.
|
|
743
|
+
|
|
744
|
+
Each line is delivered **once** — the command runs out of band and costs your CI
|
|
745
|
+
nothing, but a retry loop cannot see *why* an attempt failed and you can, so
|
|
746
|
+
re-running the command is the retry.
|
|
747
|
+
|
|
748
|
+
**A dry run is never in the file**, so nothing here can replay one: the
|
|
749
|
+
formatter refuses both sinks for `rspec --dry-run` (see above), which means this
|
|
750
|
+
command inherits that guarantee rather than re-checking it.
|
|
751
|
+
|
|
752
|
+
The exit code is the contract, and it is `specguard-lint`'s:
|
|
753
|
+
|
|
754
|
+
| Code | Meaning |
|
|
755
|
+
| --- | --- |
|
|
756
|
+
| `0` | every line was accepted |
|
|
757
|
+
| `1` | at least one line was **refused by the endpoint** — it read the payload and said no, with its own reasons rendered exactly as the formatter renders them |
|
|
758
|
+
| `2` | the command could not do its job — no endpoint or API key, an unreadable file, an unparseable line, a bad flag, or a delivery the platform never stored |
|
|
759
|
+
|
|
760
|
+
`1` is reachable only by the endpoint having read a payload and said no, which
|
|
761
|
+
is an **HTTP 400** and nothing else: that is the one response SpecGuard forms an
|
|
762
|
+
opinion about your run in. A `401` is answered before the request reaches the
|
|
763
|
+
code that would read the payload; a `404`, `429` or `5xx` never gets that far
|
|
764
|
+
either. **Nothing was stored in any of them**, so all of them are a `2` — as are
|
|
765
|
+
a connection refused, a DNS failure and a timeout. Reporting any of these as a
|
|
766
|
+
`1` would be the command telling you your suite is bad on the strength of a
|
|
767
|
+
rotated key, a typo in `SPECGUARD_ENDPOINT`, or a bad afternoon at the platform.
|
|
768
|
+
Note what that buys you: `1` means *fix the payload*, `2` means *fix the setup
|
|
769
|
+
or try again later*, and a `404` and an unset `SPECGUARD_ENDPOINT` — the same
|
|
770
|
+
mistake — give you the same code. When a file produces both, `2` wins, and every
|
|
771
|
+
line is still printed either way.
|
|
772
|
+
|
|
773
|
+
`--list` sits outside that table's `1`, and outside most of its `2`: it makes no
|
|
774
|
+
request, so no endpoint has read anything and there is no verdict to report. A
|
|
775
|
+
listing exits `0` or `2` only, and the only `2`s it can reach are a bad flag and
|
|
776
|
+
a file it could not read. The other causes in that row are delivery's, not
|
|
777
|
+
listing's — listing needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`,
|
|
778
|
+
and an unparseable line becomes a row in the listing that names it rather than
|
|
779
|
+
an exit code.
|
|
780
|
+
|
|
781
|
+
**What it will not tell you** is whether a replayed line *created* a new run or
|
|
782
|
+
*folded into* an existing one. The ingest endpoint's `202` carries the run's id
|
|
783
|
+
but no created-versus-updated flag, so the command reports what it can see: the
|
|
784
|
+
`test_run_id` that came back, and whether the line carried a `ci_run_id` of its
|
|
785
|
+
own. Two lines that went out with the same `ci_run_id` and came back with the
|
|
786
|
+
same `test_run_id` landed on one record — that is the sentence above, and it is
|
|
787
|
+
an observation rather than an inference.
|
|
788
|
+
|
|
789
|
+
Bulk-importing an aged archive is **not** what this is for. The payload carries
|
|
790
|
+
no execution timestamp and SpecGuard orders a repository's runs by when they
|
|
791
|
+
were ingested, so a replayed run becomes the repository's latest. For the case
|
|
792
|
+
this exists to serve — replay the run that just failed, right after fixing the
|
|
793
|
+
credential — that is correct.
|
|
794
|
+
|
|
795
|
+
#### Machine-readable output — `--json`
|
|
796
|
+
|
|
797
|
+
An HTTP `400` is the one **permanent** verdict in the table above: a refused line
|
|
798
|
+
is refused every time it is offered, so the only way to land the run is to learn
|
|
799
|
+
which specs SpecGuard objected to and fix the payload. It names **every** one of
|
|
800
|
+
them — one error per offending spec, by index, file and line — and the human
|
|
801
|
+
report has room for three:
|
|
802
|
+
|
|
803
|
+
```
|
|
804
|
+
line 3: refused — HTTP 400 — the endpoint rejected the payload — specs[417] spec/models/user_spec.rb:88: duration must be a non-negative number when present; specs[418] …; specs[419] … and 19997 more
|
|
805
|
+
```
|
|
806
|
+
|
|
807
|
+
That cap is right where it is: it exists for the **one stderr line** an in-run CI
|
|
808
|
+
warning is allowed, and the formatter still has to fit inside it. It is a cap on
|
|
809
|
+
a *line*, though, and `--json` is the other channel — stdout carries one JSON
|
|
810
|
+
document instead of the human report, with the whole list in it:
|
|
811
|
+
|
|
812
|
+
```bash
|
|
813
|
+
bundle exec specguard-ingest --json log/test_results.jsonl
|
|
814
|
+
```
|
|
815
|
+
```json
|
|
816
|
+
{
|
|
817
|
+
"tool": "specguard-ingest",
|
|
818
|
+
"mode": "deliver",
|
|
819
|
+
"file": "log/test_results.jsonl",
|
|
820
|
+
"summary": { "lines": 3, "attempted": 3, "accepted": 2, "refused": 1,
|
|
821
|
+
"undelivered": 0, "unparseable": 0, "blank": 0, "skipped": 0,
|
|
822
|
+
"selector": null },
|
|
823
|
+
"lines": [
|
|
824
|
+
{ "number": 1, "status": "accepted", "code": 202, "reasons": [],
|
|
825
|
+
"test_run_id": "41f2c9b8", "ci_run_id": "17442" },
|
|
826
|
+
{ "number": 2, "status": "accepted", "code": 202, "reasons": [],
|
|
827
|
+
"test_run_id": "41f2c9b8", "ci_run_id": "17442" },
|
|
828
|
+
{ "number": 3, "status": "refused", "code": 400, "test_run_id": null,
|
|
829
|
+
"ci_run_id": "17443",
|
|
830
|
+
"reasons": [
|
|
831
|
+
"specs[417] spec/models/user_spec.rb:88: duration must be a non-negative number when present",
|
|
832
|
+
"specs[418] spec/models/user_spec.rb:96: duration must be a non-negative number when present"
|
|
833
|
+
] }
|
|
834
|
+
],
|
|
835
|
+
"foldings": [
|
|
836
|
+
{ "ci_run_id": "17442", "test_run_id": "41f2c9b8", "lines": [1, 2] }
|
|
837
|
+
]
|
|
838
|
+
}
|
|
839
|
+
```
|
|
840
|
+
|
|
841
|
+
| field | meaning |
|
|
842
|
+
| --- | --- |
|
|
843
|
+
| `tool` | always `"specguard-ingest"`. Deliberately **not** a schema id: this document is about deliveries, and `specguard-lint --json` is the one that mirrors `validate-intent`'s |
|
|
844
|
+
| `mode` | `"deliver"` or `"list"` — whether the lines were sent or only shown |
|
|
845
|
+
| `file` | the path you gave it, echoed back |
|
|
846
|
+
| `summary.lines` | rows in `lines`: the lines that carried a payload and were not held back by a selector |
|
|
847
|
+
| `summary.attempted` | how many of those were offered to the endpoint — always `0` under `--list`, and `lines` minus the unparseable ones otherwise |
|
|
848
|
+
| `summary.accepted` / `refused` / `undelivered` / `unparseable` | the same four counts the text summary line states, computed once for both renderers so they cannot disagree |
|
|
849
|
+
| `summary.blank` / `skipped` | the two ways a line of the file is not a row here, counted rather than dropped |
|
|
850
|
+
| `summary.selector` | `"--lines"`, `"--from-line"`, or `null` when nothing was held back |
|
|
851
|
+
| `lines[]` | one entry per row, in the file's order |
|
|
852
|
+
| `foldings[]` | folding, **observed**: the lines that went out with one `ci_run_id` and came back with one `test_run_id`. The same statement the text report makes as a sentence |
|
|
853
|
+
|
|
854
|
+
Every delivered line has the same six keys:
|
|
855
|
+
|
|
856
|
+
| field | meaning |
|
|
857
|
+
| --- | --- |
|
|
858
|
+
| `number` | its 1-based line number in the file **as given**, blank lines counted — so it is the number `--from-line` and `--lines` take |
|
|
859
|
+
| `status` | `accepted`, `refused`, `undelivered` or `unparseable`. The tool's own vocabulary, not the report's wording (`undelivered`, where the row prints `not delivered`) |
|
|
860
|
+
| `code` | the HTTP status, or **`null`** where there is not one: a line that was never a run, and a delivery that got no answer at all (connection refused, DNS, TLS, a timeout) |
|
|
861
|
+
| `reasons` | why the line did not land — **always** a list of strings, never null and never a bare string, so a consumer never branches on its type. SpecGuard's own per-spec errors on a refusal (all of them, in its order), the parse problem where the line was not a run, the error where nothing reached the endpoint, and `[]` where it landed or where the refusal's body said nothing readable |
|
|
862
|
+
| `test_run_id` | the run the line landed on, as the endpoint reported it; `null` where that cannot be said honestly |
|
|
863
|
+
| `ci_run_id` | the run identity the line carried, or `null` — the field to read for, because a line without one has nothing for SpecGuard to fold a redelivery onto |
|
|
864
|
+
|
|
865
|
+
Every listed line has the same eight keys — `number`, `status` and `reasons`, as
|
|
866
|
+
on a delivered line, and then the five envelope facts the text row prints
|
|
867
|
+
instead of a delivery's outcome:
|
|
868
|
+
|
|
869
|
+
| field | meaning |
|
|
870
|
+
| --- | --- |
|
|
871
|
+
| `number` | its 1-based line number in the file **as given**, blank lines counted — the same number as on a delivered line, and the one `--from-line` and `--lines` take |
|
|
872
|
+
| `status` | `listed`, or `unparseable` where the line could not be parsed as a run — the two outcomes a preview has, since nothing was sent |
|
|
873
|
+
| `reasons` | the parse problem on an `unparseable` row, and `[]` on a `listed` one — **always** a list of strings, never null and never a bare string, so a consumer never branches on its type. It is the only field that says *why* a previewed line is unusable |
|
|
874
|
+
| `branch` | the branch the line carried, or **`null`** where the row says `no branch` |
|
|
875
|
+
| `commit_sha` | the commit the line carried, or **`null`** |
|
|
876
|
+
| `ci_run_id` | the run identity the line carried, or **`null`** where the row says `no ci_run_id` — the field to read for here too |
|
|
877
|
+
| `examples` | how many examples the line carried, or **`null`** where the row says `no specs`. `0` and `null` stay different facts, exactly as `0 examples` and `no specs` do |
|
|
878
|
+
| `duration_seconds` | the run's duration, or **`null`** where the row says `no duration_seconds` |
|
|
879
|
+
|
|
880
|
+
`--list --json` needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`,
|
|
881
|
+
exactly as `--list` does, and it previews the same set by the same numbers a
|
|
882
|
+
delivery would send.
|
|
883
|
+
|
|
884
|
+
Four things worth knowing:
|
|
885
|
+
|
|
886
|
+
- **The exit code is identical with and without the flag**, and the default
|
|
887
|
+
output is unchanged. `--json` is a second renderer over the same lines, the
|
|
888
|
+
same statuses and the same counts — pinned that way, byte for byte, in
|
|
889
|
+
`spec/specguard/rspec/regression_targets_spec.rb`.
|
|
890
|
+
- **`--json` does not lift the cap on the human line.** The two channels render
|
|
891
|
+
the same refusal at different lengths on purpose; nothing about the formatter's
|
|
892
|
+
in-run warning moves.
|
|
893
|
+
- **A run that never got as far as reading the file emits no document.** A bad
|
|
894
|
+
flag, `--from-line` with `--lines`, no endpoint or API key, a file that cannot
|
|
895
|
+
be read — all still exit `2` with prose on stderr and **nothing** on stdout,
|
|
896
|
+
because there is nothing yet to be a document about. A file the command *did*
|
|
897
|
+
read always gets one, whatever the exit code, including an empty one.
|
|
898
|
+
- **Warnings stay on stderr**, in both renderers. A run that delivered nothing is
|
|
899
|
+
still loud there; stdout is the document and nothing else.
|
|
900
|
+
|
|
543
901
|
### If you shard your suite
|
|
544
902
|
|
|
545
903
|
`parallel_tests`, Knapsack and a CI matrix all run the suite as several
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
# specguard-ingest — replay a saved run through SpecGuard's ingest endpoint.
|
|
5
|
+
#
|
|
6
|
+
# The RSpec formatter writes each undelivered run to `log/test_results.jsonl`,
|
|
7
|
+
# one whole run per line and byte-for-byte the body the endpoint was offered.
|
|
8
|
+
# This is the command that sends it: fix the rotated key, then replay the run
|
|
9
|
+
# the suite already finished, instead of re-running the suite.
|
|
10
|
+
#
|
|
11
|
+
# Exit codes are the contract, and they are `specguard-lint`'s reasoning
|
|
12
|
+
# transferred (see SpecGuard::RSpec::IngestCLI):
|
|
13
|
+
#
|
|
14
|
+
# 0 every line was accepted
|
|
15
|
+
# 1 at least one line was refused by the endpoint
|
|
16
|
+
# 2 this tool could not do its job — misuse, or the tool itself broken
|
|
17
|
+
#
|
|
18
|
+
# SpecGuard::RSpec::IngestCLI#run is written to *return* one of those three and
|
|
19
|
+
# never to raise. The `require`s below are the one thing outside its reach, so
|
|
20
|
+
# they get the same treatment here: a gem that cannot even load is the tool
|
|
21
|
+
# being broken, which is a 2. Left bare, Ruby would exit 1 and report that the
|
|
22
|
+
# platform refused a run it was never offered.
|
|
23
|
+
|
|
24
|
+
# Allow running from a source checkout without bundler by putting the local
|
|
25
|
+
# lib/ on the load path. When the gem is installed (or run under
|
|
26
|
+
# `bundle exec`), specguard/rspec resolves through the normal load path.
|
|
27
|
+
source_lib = File.expand_path("../lib", __dir__)
|
|
28
|
+
$LOAD_PATH.unshift(source_lib) unless $LOAD_PATH.include?(source_lib)
|
|
29
|
+
|
|
30
|
+
begin
|
|
31
|
+
require "specguard/rspec"
|
|
32
|
+
# By its own path, deliberately. `net/http`, `uri` and `zlib` are not on the
|
|
33
|
+
# linter's load chain and this command is the reason to pay for them.
|
|
34
|
+
require "specguard/rspec/ingest_cli"
|
|
35
|
+
rescue ScriptError, StandardError => e
|
|
36
|
+
warn "specguard-ingest: error: could not load specguard-rspec: #{e.class}: #{e.message}"
|
|
37
|
+
exit 2
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
exit SpecGuard::RSpec::IngestCLI.new.run(ARGV)
|