specguard-rspec 0.2.1 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8df0db27fb3830ed9d13feb79f169746ca21b6985bc62b84a69d7285425e1cb6
4
- data.tar.gz: a2e9c814cac4514eb74eda9e0ae56fafb0e6aa2bf2ff204ee5eae1fd5af92729
3
+ metadata.gz: 544f8fefb100531adc58a42d6ee9018572605253fd84f76d213ae8ca55f1c687
4
+ data.tar.gz: 4856ee3ffe6237ea7d6eac2d49a3f0d5ff7a27d1b867c0d380b8118997e991b7
5
5
  SHA512:
6
- metadata.gz: 26c40b33f81916fc72014d2b96f34d65326af41583e0095c12db91759b64d742c766db8e138bc127b298002f067176ad19753d28f538c6b544d5d38fcf1dc4d1
7
- data.tar.gz: 010c652c2f9ef9fc44fc5deeca844d0704da92b4cb719a35e28b85faab0456c0db4c7c16bf79a7408d8911b00dd4377bcdcc2953c5f0bff04a51de628175f2f0
6
+ metadata.gz: bc93bc698eaa81c4b3ab43507371d990ef079d3f9df0f816af1e69d440cb1db91201dedb5298ab385a047b14b6453ca13d856b4618400822bf69d7c13e87a0ac
7
+ data.tar.gz: 5f0fe5d59ec173fd3a91980de3c097766e77f650f5601da9501d4bb5f739450b35c88476edb78a3d1e51008398e10847fc6902c3447ef4b8868f5108a0768a91
data/README.md CHANGED
@@ -4,6 +4,8 @@
4
4
  > and a CLI linter that validates `@intent` annotations.
5
5
 
6
6
  Two independent tools, one dependency — the [OpenTestIntent](https://github.com/yatfa-ai/open-test-intent) annotation format.
7
+ A third command, [`specguard-ingest`](#replaying-a-saved-run--specguard-ingest), belongs to the first of them: it replays a
8
+ run the formatter saved when the endpoint could not be reached.
7
9
 
8
10
  ## Install
9
11
 
@@ -26,17 +28,17 @@ bundle exec specguard-lint # one-off audit: every *_spec.rb
26
28
  ```
27
29
 
28
30
  Files are **positional** (`specguard-lint spec/order_spec.rb`); there is no `--source` flag —
29
- that belongs to the reference tool, not to this one.
30
-
31
- The linter is an independent implementation of the same protocol, and its agreement with the
32
- reference is checked by running the two side by side rather than asserted in a comment:
33
- `tests/parity/run_ruby_parity.sh` in
34
- [open-test-intent](https://github.com/yatfa-ai/open-test-intent) runs `specguard-lint` and the
35
- Go validator over a shared corpus and requires identical findings, ordering and exit codes.
36
- A handful of differences are ratified as such there, each with its reason and each asserted to
37
- *still* differ; everything else matches byte for byte. They are the same ones described under
38
- the backend below — two read failures, a parse-failure message, and one that is not about
39
- wording at all.
31
+ that belongs to `validate-intent`, not to this one.
32
+
33
+ The linter is an independent implementation of the same protocol written against
34
+ [`PROTOCOL.md`](https://github.com/yatfa-ai/open-test-intent/blob/main/PROTOCOL.md) and the
35
+ canonical schema, which are what decide whether an annotation is valid. Its agreement with
36
+ `validate-intent` is checked by replaying that tool's own recorded reports through this CLI and
37
+ comparing findings, ordering and exit codes, in
38
+ `spec/specguard/rspec/validator_backend_spec.rb`. A handful of differences are ratified as such
39
+ there, each with its reason and each asserted to *still* differ; everything else matches byte for
40
+ byte. They are the same ones described under the backend below — two read failures, a
41
+ parse-failure message, and one that is not about wording at all.
40
42
 
41
43
  ### Machine-readable output (`--json`)
42
44
 
@@ -64,8 +66,8 @@ bundle exec specguard-lint --json spec/models/order_spec.rb
64
66
 
65
67
  This is the **same document** `validate-intent --json --source` emits, key for key — the gem
66
68
  already *consumes* it when the Go backend is on, and a consumer of both tools should not need two
67
- parsers for one protocol. It is not byte-identical (Ruby does not escape non-ASCII where Python's
68
- `json.dumps` does); it is key-, type- and value-identical, which is what a parser sees.
69
+ parsers for one protocol. It is not byte-identical; it is key-, type- and value-identical, which is
70
+ what a parser sees.
69
71
 
70
72
  | field | meaning |
71
73
  | --------------------- | ------- |
@@ -285,77 +287,68 @@ is worth stating precisely rather than reassuringly.
285
287
  For every payload both JSON parsers accept, the two backends agree completely: the same finding
286
288
  against the same file at the same line, the same classification, the same counts and the same exit
287
289
  code. **The messages in the table below differ in their trailing text only** — the rows *are* the
288
- enumeration, not a sample of it, and each is asserted in both directions, so closing one fails the
289
- suite rather than leaving a stale claim here — in
290
- `spec/specguard/rspec/validator_backend_spec.rb` and in the parity harness above:
291
-
292
- | input | Ruby path | Go backend |
293
- |---|---|---|
294
- | a payload that is still not JSON after normalisation | `unexpected token at '{ "entity": …'` | `Expecting value: line 1 column 102 (char 101)` |
295
- | a file that is not valid UTF-8 | `invalid UTF-8 byte sequence` | the validator's own decoder message |
296
- | a path that does not exist | `No such file or directory @ rb_sysopen - …` | `no file at this path` |
297
- | a path that is not a regular file | `Is a directory @ io_fread - …` | `no file at this path` |
290
+ enumeration, not a sample of it, and all four are asserted in both directions, so closing one fails
291
+ the suite rather than leaving a stale claim here — in
292
+ `spec/specguard/rspec/validator_backend_spec.rb`, where each row's comparison is labelled
293
+ `ENUMERATED DIFFERENCE n of 4` under the number it carries here:
294
+
295
+ | # | input | Ruby path | Go backend |
296
+ |---|---|---|---|
297
+ | 1 | a payload that is still not JSON after normalisation | Ruby's `JSON::ParserError` text | `expected a JSON value (line 1, column 102)` |
298
+ | 2 | a file that is not valid UTF-8 | `invalid UTF-8 byte sequence` | `input is not well-formed UTF-8 (PROTOCOL.md §1.1 requires it)` |
299
+ | 3 | a path that does not exist | `No such file or directory @ rb_sysopen - …` | `no file at this path` |
300
+ | 4 | a path that is not a regular file | `Is a directory @ io_fread - …` | `no file at this path` |
298
301
 
299
302
  The first row is the one you are most likely to actually see: `parse` is one of the three things
300
303
  the linter reports, and **every** malformed-JSON annotation renders differently under the backend.
301
- The Ruby path interpolates Ruby's `JSON::ParserError`; the validator reproduces CPython's `json`
302
- diagnostic, and the backend passes that through unaltered rather than inventing a third spelling.
303
- Both agree on which annotation broke, and on the line and column only the prose moves. The other
304
- rows are read failures and need an unreadable path to reach at all.
304
+ The Ruby path interpolates Ruby's `JSON::ParserError`; the validator has its own prose, and the
305
+ backend passes it through unaltered rather than inventing a third spelling. `PROTOCOL.md` specifies
306
+ the accepted JSON *language*, not the words a validator refuses in, so two spellings of one refusal
307
+ are both conformant. Both agree on which annotation broke, and on the line and column only the
308
+ prose moves. The other rows are read failures and need an unreadable path to reach at all.
309
+
310
+ Rows 3 and 4 share a Go column, and that is the substance of row 4 rather than a typo. The
311
+ binary's arguments are glob *patterns* and a match is filtered to regular files, so a directory and
312
+ a name matching nothing reach it as the same answer; the Ruby path opens the path it was given, so
313
+ it has an errno and names which one. Ruby tells the two apart, the backend cannot, and neither
314
+ pretends otherwise.
305
315
 
306
316
  #### The difference that is not about wording
307
317
 
308
- The two JSON parsers do not accept the same language. CPython's `json` which the validator
309
- reproduces deliberately accepts two things Ruby's `JSON.parse` refuses:
310
-
311
- 1. the non-finite literals `NaN`, `Infinity` and `-Infinity`;
312
- 2. a high surrogate escape (`\ud800`–`\udbff`) with **nothing escaped after it** either last in
313
- the string, or followed by a literal character. Ruby rescues it as soon as another `\uXXXX`
314
- escape follows, without requiring that escape to be a genuine low surrogate.
315
-
316
- The list was three entries until the `json` gem changed underneath it, and the third is worth
317
- recording rather than deleting: Ruby used to refuse nesting past `max_nesting: 100` at a depth
318
- CPython accepted, and used to require a real low surrogate. Both narrowed, so the set shrank —
319
- which is the direction it is expected to move, and the specs pin the new boundary exactly (see
320
- `spec/specguard/rspec/validator_backend_spec.rb`) so the next shift is caught the same way.
321
-
322
- The membership was derived rather than guessed: 89,108 documents, including every
323
- one of the 65,536 single `\uXXXX` escapes, through both parsers. (A *lone low* surrogate is fine on
324
- both — the rule is narrower than "surrogate escapes".) The sweep is also **symmetric**: the other
325
- direction documents Ruby accepts and CPython refuses was swept too and is **empty**, so the
326
- list above is the whole difference between the two parsers and not just the half we went looking
327
- for. That matters, because a member of the reverse set would show up as the mirror of the first
328
- shape below: the Ruby path reporting a schema violation where the backend reports a parse failure.
329
-
330
- On such a payload the backend does not word the failure differently; it does not have the same
331
- failure. It parses the payload and validates it, so you get a schema violation with `-> ` reason
332
- lines where the Ruby path gives you one `— could not parse annotation:` line. The file, the line,
333
- the counts and the exit code still agree.
334
-
335
- And if that payload is otherwise **schema-valid** reachable only through the surrogate case,
336
- since a number and a container cannot fill a slot the schema declares as a string — the backend
337
- finds nothing wrong and **exits 0 where the Ruby path exits 1**. This is the one input on which
338
- the two backends disagree about whether your suite passes. It is enumerated and asserted from both
339
- sides, in the same places as the rows above.
340
-
341
- Both are ratified rather than fixed, and the reason is scope: `allow_nan: true` and
342
- `max_nesting: false` would close the non-finite case, but the surrogate case has no such option and
343
- would mean porting CPython's string decoder into this gem — and both would change what the
344
- **default** Ruby path does, which the slice that added this backend deliberately holds fixed. See
345
- `Scanner#parse` for the full reasoning; whether the gem should adopt CPython's acceptance grammar
346
- is left open there rather than settled.
347
-
348
- **If you go and count them.** The parity harness numbers this same enumeration two ways, and both
349
- numbers are correct, so it is worth saying which is which before you follow the link. It *asserts*
350
- them as six entries, `(i)`–`(vi)`, under "8b. the six enumerated backend differences" — the rows of
351
- the table above, plus the two shapes the acceptance set takes. It *groups* them in its header as
352
- four mechanisms, `(a)`–`(d)`, because the two unreadable-path rows share one cause and the two
353
- acceptance-set shapes share another. Six entries, four mechanisms, one enumeration.
354
-
355
- Those two totals live in that harness, which asserts them. This page does not restate them beside
356
- the table, because a count kept in two places is a count that will eventually disagree with itself
357
- — which is exactly how an earlier revision of this section came to announce three messages directly
358
- above four rows.
318
+ The two JSON parsers do not accept quite the same language, and the difference is now small and
319
+ runs the *opposite* way from how it used to.
320
+
321
+ `PROTOCOL.md` §1.1 states the accepted language an RFC 8259 JSON text, with the three points that
322
+ RFC leaves to the implementation settled explicitly. The validator refuses the non-finite literals
323
+ (§1.1(b)), unpaired surrogate escapes (§1.1(a)) and nesting past 100 (§1.1(c)). Ruby's `JSON.parse`
324
+ refuses or limits all three too, so on those three the two now **agree**. (They did not before: the
325
+ validator's parser used to reproduce a foreign runtime's grammar, which the protocol had never
326
+ specified. Removing that is what SPGD-403 did.)
327
+
328
+ **What survives is this gem being the more permissive side**, in two places:
329
+
330
+ 1. **A lone LOW surrogate escape** (`\udc00`–`\udfff`). `JSON.parse` accepts it; §1.1(a) refuses it,
331
+ because a surrogate escape must form a pair. Ruby refuses only the HIGH half, which is why the
332
+ rule is narrower than "surrogate escapes diverge".
333
+ 2. **The nesting boundary**, which sits a little deeper here than §1.1(c)'s 100.
334
+
335
+ The first one is not only a verdict difference. What `JSON.parse` returns for `"\udc00"` is a
336
+ String whose `valid_encoding?` is **false**: it cannot be re-serialised and cannot cross the ingest
337
+ transport, so a payload this gem calls valid is one it cannot send. That is the cost §1.1(a) exists
338
+ to remove, and it is still paid on the Ruby path.
339
+
340
+ On such a payload the backend does not word the failure differently; it *has* one where the Ruby
341
+ path does not. And because the only surviving member lives inside a string a slot the schema
342
+ declares the payload is otherwise schema-valid, so the backend **exits 1 where the Ruby path
343
+ exits 0**. This is the one input on which the two backends disagree about whether your suite passes.
344
+ It is enumerated and asserted from both sides in
345
+ `spec/specguard/rspec/validator_backend_spec.rb`, along with the convergence above, so a validator
346
+ that went back to accepting a superset of JSON fails there by name.
347
+
348
+ Both are ratified rather than fixed, and the reason is scope: this gem's hand-rolled validation
349
+ logic is slated for **removal** by the roadmap that owns the validator rather than for repair, and
350
+ closing the gap here would change what the **default** Ruby path does, which the slice that added
351
+ this backend deliberately holds fixed. See `Scanner#parse` for the full reasoning.
359
352
 
360
353
  Every way the backend can fail — the binary is missing, will not execute, exits with something that
361
354
  is not a verdict, or emits output that is not a report — is **exit 2**, the linter's "could not do
@@ -498,13 +491,34 @@ rather than like a broken build.
498
491
  the run (a `401` from a rotated key, a `400`, a `500`) or cannot be reached at
499
492
  all (connection refused, DNS failure, timeout), the formatter prints **one**
500
493
  line to stderr naming the status or the error, and writes the payload to
501
- `log/test_results.jsonl` so the run can be replayed later:
494
+ `log/test_results.jsonl` so the run can be replayed later with
495
+ [`specguard-ingest`](#replaying-a-saved-run--specguard-ingest):
502
496
 
503
497
  ```
504
498
  SpecGuard: could not deliver test telemetry (HTTP 401 — the API key was not
505
499
  accepted). Falling back to log/test_results.jsonl; the test run is unaffected.
506
500
  ```
507
501
 
502
+ **That line carries the endpoint's own words when it has any.** A `400` refusal
503
+ names the offending spec by index, file and line, so a rejected payload is a
504
+ thing you can fix from the CI log rather than one you have to reproduce
505
+ locally:
506
+
507
+ ```
508
+ SpecGuard: could not deliver test telemetry (HTTP 400 — the endpoint rejected
509
+ the payload — spec 3 (spec/orders_spec.rb:9): line_number is required and must
510
+ be a positive integer; spec 7 (spec/orders_spec.rb:31): outcome must be one of
511
+ passed, failed, pending). Falling back to log/test_results.jsonl; the test run
512
+ is unaffected.
513
+ ```
514
+
515
+ It stays **one** line whatever comes back. A systemic problem can have the
516
+ endpoint refusing every spec in the suite, so at most three reasons are spelled
517
+ out and the rest are counted (`… and 497 more`); anything that arrives without
518
+ a reason it can read — an empty body, or the HTML a proxy answers a `413` with
519
+ — prints the bare status line above and is still reported as a refusal, not as
520
+ an error.
521
+
508
522
  There are **no retries**, and the whole delivery is bounded by `timeout`
509
523
  (10 seconds by default, against `Net::HTTP`'s own 60): telemetry is explicitly
510
524
  allowed to be lost, and a retry would only double what a hung endpoint can cost
@@ -525,8 +539,27 @@ same corruption, deferred until something replays it — and says so once:
525
539
  SpecGuard: skipped test telemetry for a dry run (rspec --dry-run executes no
526
540
  example bodies, so this run's durations and outcomes would not be
527
541
  measurements). Nothing was sent or written; the test run is unaffected.
542
+ Annotation coverage is a fact about source, not about execution, so it
543
+ survives the refusal — this working tree: 6 examples, 3 annotated,
544
+ 3 unannotated (50% annotated).
545
+ the 3 unannotated examples, by definition site:
546
+ spec/orders_spec.rb:7 Order has no annotation
547
+ spec/orders_spec.rb:16 Order has a malformed annotation
548
+ spec/orders_spec.rb:21 Order has a schema-invalid annotation
528
549
  ```
529
550
 
551
+ The refusal throws away less than it used to. `duration` and `outcome` are
552
+ fabricated by a dry run, which is what makes them unpublishable — but the
553
+ third field the formatter computes per example, `annotated` / `unannotated`,
554
+ comes from scanning the **spec file's source text** and is identical whether or
555
+ not a body ran. Since a dry run still builds every example, it holds the exact
556
+ numerator and denominator of the annotation-coverage metric, so `--dry-run` is
557
+ also the way to ask *"where are we?"* without a commit, a push, and a CI round
558
+ trip. The figure describes **your working tree right now**, so it will differ
559
+ from the dashboard's the moment you edit a spec — that difference is the point.
560
+
561
+ Nothing is published either way: the report goes to stderr and to nowhere else.
562
+
530
563
  This matters most where you are least likely to look for it: an API key is
531
564
  usually an environment-level secret rather than a job-level one, so a lint job
532
565
  that runs `rspec --dry-run` to catch an unparseable spec file inherits the key
@@ -540,6 +573,331 @@ alone. A non-2xx response gets the same treatment: `Net::HTTP` returns those as
540
573
  ordinary values rather than raising, so they are checked for explicitly instead
541
574
  of being left to a `rescue` that would never see them.
542
575
 
576
+ ### Replaying a saved run — `specguard-ingest`
577
+
578
+ The suite is over by the time you see the `401`, and re-running it to recover
579
+ the telemetry costs you the whole suite again. So the file the formatter wrote
580
+ is the run: each line is byte-for-byte the body the endpoint refused, and
581
+ `specguard-ingest` is the command that sends it.
582
+
583
+ ```bash
584
+ export SPECGUARD_API_KEY=… # the key that was rotated, fixed
585
+ bundle exec specguard-ingest log/test_results.jsonl
586
+ ```
587
+
588
+ ```
589
+ line 1: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
590
+ line 2: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
591
+ specguard-ingest: delivered 2 of 2 runs from log/test_results.jsonl
592
+ specguard-ingest: lines 1, 2 carried ci_run_id 17442 and each came back with
593
+ test_run_id 41f2c9b8 — the endpoint folded them onto one run
594
+ ```
595
+
596
+ It reads the same `SPECGUARD_ENDPOINT`, `SPECGUARD_API_KEY` and
597
+ `SPECGUARD_TIMEOUT` the formatter does, and sends each line through the same
598
+ transport — so a run that was refused for a rotated key appears on the platform
599
+ once the secret is fixed, with the shard folding described in
600
+ [If you shard your suite](#if-you-shard-your-suite) applying exactly as it would
601
+ have during the run.
602
+
603
+ > **It re-delivers *every* line in the file you give it.** The formatter writes
604
+ > to `log/test_results.jsonl` when a delivery **failed** *and* when no API key
605
+ > was configured at all — and the two are indistinguishable on the line, because
606
+ > nothing in the payload records which sink it was destined for. So a laptop's
607
+ > file is a file of ordinary local runs, and this command will send all of them.
608
+ > There is no filter and no heuristic: guessing which lines "were failures" from
609
+ > data that does not say would be confidently wrong about which of your runs
610
+ > reach the platform. Check the file first with
611
+ > [`--list`](#checking-a-file-before-you-send-it----list), and check it before
612
+ > you replay one you did not write.
613
+
614
+ #### Checking a file before you send it — `--list`
615
+
616
+ `--list` prints one row per line and **delivers nothing**:
617
+
618
+ ```bash
619
+ bundle exec specguard-ingest --list log/test_results.jsonl
620
+ ```
621
+
622
+ ```
623
+ line 1: branch main, commit_sha 0d4a1f2c9b8e7d6a5f4c3b2a1908f7e6d5c4b3a2, ci_run_id 17442, 412 examples, 93.4s
624
+ line 2: branch main, commit_sha 0d4a1f2c9b8e7d6a5f4c3b2a1908f7e6d5c4b3a2, ci_run_id 17442, 388 examples, 91.2s
625
+ line 3: branch spike/local, commit_sha 9c2e7a10b4d3, no ci_run_id, 6 examples, 0.4s
626
+ line 4: unparseable — could not parse the line as JSON: unexpected end of input
627
+ specguard-ingest: listed 4 lines from log/test_results.jsonl; nothing was delivered
628
+ ```
629
+
630
+ Every field on the row is already on the line — nothing is guessed at, and a
631
+ line the command cannot parse is listed **as unparseable** rather than quietly
632
+ dropped from the preview. `no ci_run_id` is the one to read for: that line has
633
+ no identity for SpecGuard to fold a redelivery onto, so sending it creates a new
634
+ run rather than joining an existing one.
635
+
636
+ Reading the file yourself is not the alternative. One line is one whole run, and
637
+ at 20,000 examples that is megabytes of JSON on a single physical line.
638
+
639
+ **It needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`** — deliberately.
640
+ The file most worth checking is the one written *because* no API key was set, so
641
+ requiring a key to look at it would withdraw the instrument in exactly the
642
+ situation that produces the hazard. It composes with `--from-line` and `--lines`
643
+ too, so you can list the exact set you are about to send:
644
+
645
+ ```bash
646
+ bundle exec specguard-ingest --list --from-line 7 log/test_results.jsonl
647
+ bundle exec specguard-ingest --list --lines 3,7,12-15 log/test_results.jsonl
648
+ ```
649
+
650
+ A listing under either selector previews **exactly** the lines the same command
651
+ without `--list` would deliver, by the same numbers.
652
+
653
+ Listing sends nothing, so it can never be a verdict about a run: it exits `0`
654
+ when it listed the file and `2` when it could not read it or the flags were
655
+ wrong. **`1` is unreachable with `--list`.**
656
+
657
+ **Each line is reported by its line number**, and `--from-line N` starts at one —
658
+ so a file that was only partly accepted is resumed from the line the report
659
+ named, rather than blindly re-sent:
660
+
661
+ ```bash
662
+ bundle exec specguard-ingest --from-line 7 log/test_results.jsonl
663
+ ```
664
+
665
+ The numbering never shifts: line 7 is line 7 of the file you gave it, both times.
666
+ Re-sending a line that already landed is harmless *only* when it carries a
667
+ `ci_run_id` — that is the identity SpecGuard folds a redelivery onto. A line
668
+ **without** one has nothing to fold onto and becomes a second run, and a keyless
669
+ local file is made entirely of those, so `--from-line` is worth the two seconds
670
+ it takes to read the previous report.
671
+
672
+ #### Sending a set rather than a suffix — `--lines`
673
+
674
+ `--from-line` can only express a **suffix**, and the set a per-line report points
675
+ at is a suffix at most once. `--lines` takes the set itself — comma-separated
676
+ numbers and ranges, over the file's own numbering:
677
+
678
+ ```bash
679
+ bundle exec specguard-ingest --lines 3,7,12-15 log/test_results.jsonl
680
+ ```
681
+
682
+ ```
683
+ line 3: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
684
+ line 7: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
685
+ line 12: accepted — HTTP 202, test_run_id 5a3d0e91, ci_run_id 17443
686
+
687
+ specguard-ingest: delivered 6 of 6 runs from log/test_results.jsonl; 34 lines not selected by --lines
688
+ ```
689
+
690
+ Two things want it. The first is an **interior line that will never be accepted**:
691
+ an HTTP `400` is the one response SpecGuard forms an opinion about your payload
692
+ in, so a line it refuses is refused every time it is offered. Sitting at line 3
693
+ of a 40-line file, no `--from-line` can step over it — the file can never be
694
+ replayed to completion, and the command can never exit `0` over it. Naming the
695
+ set around it can:
696
+
697
+ ```bash
698
+ bundle exec specguard-ingest --lines 1-2,4-40 log/test_results.jsonl
699
+ ```
700
+
701
+ The second is that the sink is **append-only and mixes both sources**, so
702
+ ordinary keyless laptop runs keep landing *after* the CI failures you want to
703
+ replay. Every unwanted keyless line a too-early `--from-line` sweeps up is a
704
+ spurious run on the platform, because a line with no `ci_run_id` has nothing to
705
+ fold onto.
706
+
707
+ Carving the file up first (`sed -n '21,24p' file > tmp.jsonl`) is not the
708
+ alternative: a carved file **renumbers**, and the whole value of acting on a
709
+ per-line report is that line 12 is still line 12.
710
+
711
+ The held-back lines are **counted and reported**, exactly as `--from-line`'s and
712
+ the blank ones are — a summary that quietly narrowed what it was summarising
713
+ would be worse than no summary.
714
+
715
+ A spec is read strictly, and a bad one is a `2` rather than a fallback to the
716
+ whole file — which is the one outcome a selector exists to prevent. `--lines 0`,
717
+ `--lines 5-2`, `--lines abc`, `--lines 12-`, an empty spec and an empty entry
718
+ (`3,,5`) are all refused, naming what was wrong. Whitespace *between* entries is
719
+ fine (`3, 7`); inside one it is a typo, not a range (`5 - 7` is refused).
720
+
721
+ **`--lines` and `--from-line` do not combine** — giving both is a `2`:
722
+
723
+ ```
724
+ specguard-ingest: error: --from-line and --lines both choose which lines to send; give one or the other
725
+ ```
726
+
727
+ They answer the same question, and intersecting them would silently drop a
728
+ number you typed: `--from-line 5 --lines 3,7` would send only line 7, and the 3
729
+ would vanish without a word. Refusing the pair is the same discipline as the
730
+ rest of this command — it will not quietly narrow what it was asked for.
731
+
732
+ Repeating **one** selector is a different case and is allowed: the last one
733
+ wins. `--lines 1,2 --lines 4` sends line 4, and `--from-line 2 --from-line 5`
734
+ starts at 5. A repeat replaces rather than intersects, so the set delivered is
735
+ exactly the last one you typed — nothing is combined into something smaller than
736
+ you asked for, which is the objection to the pair above. It is also what lets
737
+ you override a selector baked into a wrapper script or shell alias by appending
738
+ a new one.
739
+
740
+ Nothing about a line's **content** is consulted by either flag. The numbers come
741
+ from you, after reading `--list`; that is what keeps this an explicit selector
742
+ rather than the heuristic this command refuses to grow.
743
+
744
+ Each line is delivered **once** — the command runs out of band and costs your CI
745
+ nothing, but a retry loop cannot see *why* an attempt failed and you can, so
746
+ re-running the command is the retry.
747
+
748
+ **A dry run is never in the file**, so nothing here can replay one: the
749
+ formatter refuses both sinks for `rspec --dry-run` (see above), which means this
750
+ command inherits that guarantee rather than re-checking it.
751
+
752
+ The exit code is the contract, and it is `specguard-lint`'s:
753
+
754
+ | Code | Meaning |
755
+ | --- | --- |
756
+ | `0` | every line was accepted |
757
+ | `1` | at least one line was **refused by the endpoint** — it read the payload and said no, with its own reasons rendered exactly as the formatter renders them |
758
+ | `2` | the command could not do its job — no endpoint or API key, an unreadable file, an unparseable line, a bad flag, or a delivery the platform never stored |
759
+
760
+ `1` is reachable only by the endpoint having read a payload and said no, which
761
+ is an **HTTP 400** and nothing else: that is the one response SpecGuard forms an
762
+ opinion about your run in. A `401` is answered before the request reaches the
763
+ code that would read the payload; a `404`, `429` or `5xx` never gets that far
764
+ either. **Nothing was stored in any of them**, so all of them are a `2` — as are
765
+ a connection refused, a DNS failure and a timeout. Reporting any of these as a
766
+ `1` would be the command telling you your suite is bad on the strength of a
767
+ rotated key, a typo in `SPECGUARD_ENDPOINT`, or a bad afternoon at the platform.
768
+ Note what that buys you: `1` means *fix the payload*, `2` means *fix the setup
769
+ or try again later*, and a `404` and an unset `SPECGUARD_ENDPOINT` — the same
770
+ mistake — give you the same code. When a file produces both, `2` wins, and every
771
+ line is still printed either way.
772
+
773
+ `--list` sits outside that table's `1`, and outside most of its `2`: it makes no
774
+ request, so no endpoint has read anything and there is no verdict to report. A
775
+ listing exits `0` or `2` only, and the only `2`s it can reach are a bad flag and
776
+ a file it could not read. The other causes in that row are delivery's, not
777
+ listing's — listing needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`,
778
+ and an unparseable line becomes a row in the listing that names it rather than
779
+ an exit code.
780
+
781
+ **What it will not tell you** is whether a replayed line *created* a new run or
782
+ *folded into* an existing one. The ingest endpoint's `202` carries the run's id
783
+ but no created-versus-updated flag, so the command reports what it can see: the
784
+ `test_run_id` that came back, and whether the line carried a `ci_run_id` of its
785
+ own. Two lines that went out with the same `ci_run_id` and came back with the
786
+ same `test_run_id` landed on one record — that is the sentence above, and it is
787
+ an observation rather than an inference.
788
+
789
+ Bulk-importing an aged archive is **not** what this is for. The payload carries
790
+ no execution timestamp and SpecGuard orders a repository's runs by when they
791
+ were ingested, so a replayed run becomes the repository's latest. For the case
792
+ this exists to serve — replay the run that just failed, right after fixing the
793
+ credential — that is correct.
794
+
795
+ #### Machine-readable output — `--json`
796
+
797
+ An HTTP `400` is the one **permanent** verdict in the table above: a refused line
798
+ is refused every time it is offered, so the only way to land the run is to learn
799
+ which specs SpecGuard objected to and fix the payload. It names **every** one of
800
+ them — one error per offending spec, by index, file and line — and the human
801
+ report has room for three:
802
+
803
+ ```
804
+ line 3: refused — HTTP 400 — the endpoint rejected the payload — specs[417] spec/models/user_spec.rb:88: duration must be a non-negative number when present; specs[418] …; specs[419] … and 19997 more
805
+ ```
806
+
807
+ That cap is right where it is: it exists for the **one stderr line** an in-run CI
808
+ warning is allowed, and the formatter still has to fit inside it. It is a cap on
809
+ a *line*, though, and `--json` is the other channel — stdout carries one JSON
810
+ document instead of the human report, with the whole list in it:
811
+
812
+ ```bash
813
+ bundle exec specguard-ingest --json log/test_results.jsonl
814
+ ```
815
+ ```json
816
+ {
817
+ "tool": "specguard-ingest",
818
+ "mode": "deliver",
819
+ "file": "log/test_results.jsonl",
820
+ "summary": { "lines": 3, "attempted": 3, "accepted": 2, "refused": 1,
821
+ "undelivered": 0, "unparseable": 0, "blank": 0, "skipped": 0,
822
+ "selector": null },
823
+ "lines": [
824
+ { "number": 1, "status": "accepted", "code": 202, "reasons": [],
825
+ "test_run_id": "41f2c9b8", "ci_run_id": "17442" },
826
+ { "number": 2, "status": "accepted", "code": 202, "reasons": [],
827
+ "test_run_id": "41f2c9b8", "ci_run_id": "17442" },
828
+ { "number": 3, "status": "refused", "code": 400, "test_run_id": null,
829
+ "ci_run_id": "17443",
830
+ "reasons": [
831
+ "specs[417] spec/models/user_spec.rb:88: duration must be a non-negative number when present",
832
+ "specs[418] spec/models/user_spec.rb:96: duration must be a non-negative number when present"
833
+ ] }
834
+ ],
835
+ "foldings": [
836
+ { "ci_run_id": "17442", "test_run_id": "41f2c9b8", "lines": [1, 2] }
837
+ ]
838
+ }
839
+ ```
840
+
841
+ | field | meaning |
842
+ | --- | --- |
843
+ | `tool` | always `"specguard-ingest"`. Deliberately **not** a schema id: this document is about deliveries, and `specguard-lint --json` is the one that mirrors `validate-intent`'s |
844
+ | `mode` | `"deliver"` or `"list"` — whether the lines were sent or only shown |
845
+ | `file` | the path you gave it, echoed back |
846
+ | `summary.lines` | rows in `lines`: the lines that carried a payload and were not held back by a selector |
847
+ | `summary.attempted` | how many of those were offered to the endpoint — always `0` under `--list`, and `lines` minus the unparseable ones otherwise |
848
+ | `summary.accepted` / `refused` / `undelivered` / `unparseable` | the same four counts the text summary line states, computed once for both renderers so they cannot disagree |
849
+ | `summary.blank` / `skipped` | the two ways a line of the file is not a row here, counted rather than dropped |
850
+ | `summary.selector` | `"--lines"`, `"--from-line"`, or `null` when nothing was held back |
851
+ | `lines[]` | one entry per row, in the file's order |
852
+ | `foldings[]` | folding, **observed**: the lines that went out with one `ci_run_id` and came back with one `test_run_id`. The same statement the text report makes as a sentence |
853
+
854
+ Every delivered line has the same six keys:
855
+
856
+ | field | meaning |
857
+ | --- | --- |
858
+ | `number` | its 1-based line number in the file **as given**, blank lines counted — so it is the number `--from-line` and `--lines` take |
859
+ | `status` | `accepted`, `refused`, `undelivered` or `unparseable`. The tool's own vocabulary, not the report's wording (`undelivered`, where the row prints `not delivered`) |
860
+ | `code` | the HTTP status, or **`null`** where there is not one: a line that was never a run, and a delivery that got no answer at all (connection refused, DNS, TLS, a timeout) |
861
+ | `reasons` | why the line did not land — **always** a list of strings, never null and never a bare string, so a consumer never branches on its type. SpecGuard's own per-spec errors on a refusal (all of them, in its order), the parse problem where the line was not a run, the error where nothing reached the endpoint, and `[]` where it landed or where the refusal's body said nothing readable |
862
+ | `test_run_id` | the run the line landed on, as the endpoint reported it; `null` where that cannot be said honestly |
863
+ | `ci_run_id` | the run identity the line carried, or `null` — the field to read for, because a line without one has nothing for SpecGuard to fold a redelivery onto |
864
+
865
+ Every listed line has the same eight keys — `number`, `status` and `reasons`, as
866
+ on a delivered line, and then the five envelope facts the text row prints
867
+ instead of a delivery's outcome:
868
+
869
+ | field | meaning |
870
+ | --- | --- |
871
+ | `number` | its 1-based line number in the file **as given**, blank lines counted — the same number as on a delivered line, and the one `--from-line` and `--lines` take |
872
+ | `status` | `listed`, or `unparseable` where the line could not be parsed as a run — the two outcomes a preview has, since nothing was sent |
873
+ | `reasons` | the parse problem on an `unparseable` row, and `[]` on a `listed` one — **always** a list of strings, never null and never a bare string, so a consumer never branches on its type. It is the only field that says *why* a previewed line is unusable |
874
+ | `branch` | the branch the line carried, or **`null`** where the row says `no branch` |
875
+ | `commit_sha` | the commit the line carried, or **`null`** |
876
+ | `ci_run_id` | the run identity the line carried, or **`null`** where the row says `no ci_run_id` — the field to read for here too |
877
+ | `examples` | how many examples the line carried, or **`null`** where the row says `no specs`. `0` and `null` stay different facts, exactly as `0 examples` and `no specs` do |
878
+ | `duration_seconds` | the run's duration, or **`null`** where the row says `no duration_seconds` |
879
+
880
+ `--list --json` needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`,
881
+ exactly as `--list` does, and it previews the same set by the same numbers a
882
+ delivery would send.
883
+
884
+ Four things worth knowing:
885
+
886
+ - **The exit code is identical with and without the flag**, and the default
887
+ output is unchanged. `--json` is a second renderer over the same lines, the
888
+ same statuses and the same counts — pinned that way, byte for byte, in
889
+ `spec/specguard/rspec/regression_targets_spec.rb`.
890
+ - **`--json` does not lift the cap on the human line.** The two channels render
891
+ the same refusal at different lengths on purpose; nothing about the formatter's
892
+ in-run warning moves.
893
+ - **A run that never got as far as reading the file emits no document.** A bad
894
+ flag, `--from-line` with `--lines`, no endpoint or API key, a file that cannot
895
+ be read — all still exit `2` with prose on stderr and **nothing** on stdout,
896
+ because there is nothing yet to be a document about. A file the command *did*
897
+ read always gets one, whatever the exit code, including an empty one.
898
+ - **Warnings stay on stderr**, in both renderers. A run that delivered nothing is
899
+ still loud there; stdout is the document and nothing else.
900
+
543
901
  ### If you shard your suite
544
902
 
545
903
  `parallel_tests`, Knapsack and a CI matrix all run the suite as several
@@ -0,0 +1,40 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ # specguard-ingest — replay a saved run through SpecGuard's ingest endpoint.
5
+ #
6
+ # The RSpec formatter writes each undelivered run to `log/test_results.jsonl`,
7
+ # one whole run per line and byte-for-byte the body the endpoint was offered.
8
+ # This is the command that sends it: fix the rotated key, then replay the run
9
+ # the suite already finished, instead of re-running the suite.
10
+ #
11
+ # Exit codes are the contract, and they are `specguard-lint`'s reasoning
12
+ # transferred (see SpecGuard::RSpec::IngestCLI):
13
+ #
14
+ # 0 every line was accepted
15
+ # 1 at least one line was refused by the endpoint
16
+ # 2 this tool could not do its job — misuse, or the tool itself broken
17
+ #
18
+ # SpecGuard::RSpec::IngestCLI#run is written to *return* one of those three and
19
+ # never to raise. The `require`s below are the one thing outside its reach, so
20
+ # they get the same treatment here: a gem that cannot even load is the tool
21
+ # being broken, which is a 2. Left bare, Ruby would exit 1 and report that the
22
+ # platform refused a run it was never offered.
23
+
24
+ # Allow running from a source checkout without bundler by putting the local
25
+ # lib/ on the load path. When the gem is installed (or run under
26
+ # `bundle exec`), specguard/rspec resolves through the normal load path.
27
+ source_lib = File.expand_path("../lib", __dir__)
28
+ $LOAD_PATH.unshift(source_lib) unless $LOAD_PATH.include?(source_lib)
29
+
30
+ begin
31
+ require "specguard/rspec"
32
+ # By its own path, deliberately. `net/http`, `uri` and `zlib` are not on the
33
+ # linter's load chain and this command is the reason to pay for them.
34
+ require "specguard/rspec/ingest_cli"
35
+ rescue ScriptError, StandardError => e
36
+ warn "specguard-ingest: error: could not load specguard-rspec: #{e.class}: #{e.message}"
37
+ exit 2
38
+ end
39
+
40
+ exit SpecGuard::RSpec::IngestCLI.new.run(ARGV)