specguard-ruby 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md ADDED
@@ -0,0 +1,1036 @@
1
+ # specguard-ruby
2
+
3
+ > The Ruby client for [SpecGuard](https://github.com/yatfa-ai/specguard), every Ruby test framework
4
+ > in one gem: an RSpec formatter and a Minitest reporter that ship test-run telemetry, and a CLI
5
+ > linter that validates `@intent` annotations.
6
+
7
+ Two independent tools, one dependency — the [OpenTestIntent](https://github.com/yatfa-ai/open-test-intent) annotation format.
8
+ A third command, [`specguard-ingest`](#replaying-a-saved-run--specguard-ingest), belongs to the first of them: it replays a
9
+ run a reporter saved when the endpoint could not be reached.
10
+
11
+ This gem was named `specguard-rspec` while RSpec was its only adapter; it was renamed when the
12
+ Minitest reporter joined. Same environment variables, same wire contract, same tools.
13
+
14
+ ## Install
15
+
16
+ ```ruby
17
+ # Gemfile
18
+ group :test do
19
+ gem "specguard-ruby", require: false
20
+ end
21
+ ```
22
+
23
+ ## The Minitest reporter
24
+
25
+ Minitest discovers the reporter as a plugin — the gem on the load path is the whole integration,
26
+ no spec file changes, no flags:
27
+
28
+ ```bash
29
+ SPECGUARD_ENDPOINT=https://specguard.example SPECGUARD_API_KEY=sgk_… bundle exec ruby test/your_suite.rb
30
+ ```
31
+
32
+ The plugin rides alongside Minitest's own reporters (the suite's output is unchanged), posts one
33
+ envelope per process with the same field names the RSpec formatter sends, and never fails the run:
34
+ a refused or unreachable delivery costs one line on stderr and a line in the local sink. Without
35
+ `SPECGUARD_API_KEY` nothing is sent — the run is appended to the local development record, exactly
36
+ as the RSpec formatter behaves. For a deterministic CI attachment where plugin discovery must not
37
+ be assumed, require it explicitly before the run:
38
+
39
+ ```bash
40
+ bundle exec ruby -rminitest/specguard_plugin -e 'Minitest.extensions << "specguard"; load ARGV[0]' test/your_suite.rb
41
+ ```
42
+
43
+ ## The linter — `specguard-lint`
44
+
45
+ Validates `# @intent:` annotations in changed (or all) `*_spec.rb` files against the OpenTestIntent
46
+ JSON Schema. Exits `1` on a malformed annotation; **never** fails on a *missing* one (adoption is
47
+ opt-in and gradual).
48
+
49
+ ```bash
50
+ bundle exec specguard-lint --changed # CI mode: only files in the current diff
51
+ bundle exec specguard-lint # one-off audit: every *_spec.rb
52
+ ```
53
+
54
+ Files are **positional** (`specguard-lint spec/order_spec.rb`); there is no `--source` flag —
55
+ that belongs to `validate-intent`, not to this one.
56
+
57
+ The linter is a thin CLI over the [`validate-intent`](https://github.com/yatfa-ai/open-test-intent)
58
+ binary — the protocol's own reference implementation, which decides whether an annotation is valid
59
+ against [`PROTOCOL.md`](https://github.com/yatfa-ai/open-test-intent/blob/main/PROTOCOL.md) and the
60
+ canonical schema compiled into it. Since SPGD-867 that is the only validator: the gem selects the
61
+ files, resolves the binary, renders the report and owns the 0/1/2 exit contract, and the verdicts
62
+ are the binary's. Its rendering is checked by replaying that tool's own recorded reports through
63
+ this CLI in `spec/specguard/rspec/validator_backend_spec.rb` — findings, ordering and exit codes,
64
+ byte for byte.
65
+
66
+ ### Machine-readable output (`--json`)
67
+
68
+ The human report is for humans. `--json` emits **one JSON document on stdout** instead, so a CI
69
+ step or an agent gets *which file, which line, which rule* as data rather than a prose format to
70
+ regex and a 3-valued exit code. The flag can go anywhere on the command line.
71
+
72
+ ```bash
73
+ bundle exec specguard-lint --json spec/models/order_spec.rb
74
+ ```
75
+ ```json
76
+ {
77
+ "schema": "open-test-intent.v1.json",
78
+ "mode": "source",
79
+ "ok": false,
80
+ "summary": { "files": 1, "annotations": 2, "failed": 2 },
81
+ "findings": [
82
+ { "file": "spec/models/order_spec.rb", "line": 24, "ok": false, "kind": "schema",
83
+ "errors": ["<root>: additional property 'entiity' is not allowed"],
84
+ "intent": { "entiity": "Order", "action": "checkout", "behavior": "...", "layer": "request" } },
85
+ { "file": "spec/models/order_spec.rb", "line": 31, "ok": false, "kind": "extraction",
86
+ "errors": ["unterminated object literal (an annotation must fit on one line)"],
87
+ "intent": null }
88
+ ]
89
+ }
90
+ ```
91
+
92
+ This is the **same document** `validate-intent --json --source` emits, key for key — the gem
93
+ already *consumes* it when the Go backend is on, and a consumer of both tools should not need two
94
+ parsers for one protocol. It is not byte-identical; it is key-, type- and value-identical, which is
95
+ what a parser sees.
96
+
97
+ | field | meaning |
98
+ | --------------------- | ------- |
99
+ | `schema` | the OpenTestIntent schema version the payloads were validated against |
100
+ | `mode` | always `"source"` — annotations in spec sources, the port's name for what this tool does. Not the *selection* mode (`--changed` vs named files), which the document has no field for |
101
+ | `ok` | whether the run passed — derived from the exit code, not recomputed, so the two renderers cannot disagree |
102
+ | `summary.files` | spec files selected: the number the text report's leading `checked N spec file(s)` line states |
103
+ | `summary.annotations` | annotation sites examined: the number its trailing summary line states. A site whose payload could not be captured or parsed still counts; a file that could not be read contributes none |
104
+ | `summary.failed` | findings with `"ok": false`, read failures included. Note this is **not** the text summary's `M malformed`, which counts malformed annotations and reports unread files in its own clause |
105
+
106
+ Every finding has the same six keys:
107
+
108
+ | field | meaning |
109
+ | -------- | ------- |
110
+ | `file` | the path, echoed back exactly as it was given |
111
+ | `line` | the annotation's line number; **`null`** where the finding is not line-scoped — a read failure saw no line of the file, and `file:0` would point CI annotations and editor quickfix at a line that does not exist |
112
+ | `ok` | whether this finding passed |
113
+ | `kind` | *how* it failed; `null` when it passed |
114
+ | `errors` | every violated rule, or the single problem — **always** a list of strings, never null and never a bare string, so a consumer never branches on its type |
115
+ | `intent` | what the payload **parsed to** — the annotation as data, after the permissive syntax (unquoted keys, single quotes) has been normalized away. `null` where there was no payload to parse: a read, extraction or parse failure. A **schema-rejected** annotation still carries it — it parsed, and `ok` already reports the verdict, so this field answers *what does this say*, not *is this good* |
116
+
117
+ > `intent` is `null` in one case that is not about your annotation: a payload holding a lone
118
+ > surrogate escape (`"\ud800"`) is valid to the Go/Python validator and cannot be represented in
119
+ > Ruby at all — `JSON.parse` refuses it and `JSON.generate` cannot re-emit it. Rather than repair
120
+ > it into something you did not write, this tool drops the payload and keeps the verdict, so the
121
+ > finding is still reported and the exit code is unchanged.
122
+
123
+ `kind` is the field the prose renderer destroys: a failed extraction and an unparseable payload
124
+ both read as one sentence after `FAIL … — `, and only `kind` tells them apart.
125
+
126
+ | `kind` | means |
127
+ | ------ | ----- |
128
+ | `schema` | parsed fine, violated the OpenTestIntent schema |
129
+ | `extraction` | an `@intent:` token whose object literal could not be captured (missing or unbalanced braces, or spread across lines) |
130
+ | `parse` | the payload was captured but is not JSON even after normalisation |
131
+ | `read` | the file could not be read at all (missing, unopenable, or not valid UTF-8), so no annotation in it was ever seen |
132
+
133
+ The port has a fifth kind, `no-match`, that cannot appear here: its arguments are globs and this
134
+ tool's are paths, so a path that matches nothing is a `read` failure of that path.
135
+
136
+ Three things worth knowing:
137
+
138
+ - **Exit codes are identical with and without the flag**, and the default output is unchanged.
139
+ `--json` is a second renderer over the same checks, not a second code path — it is pinned that
140
+ way in `spec/specguard/rspec/exit_contract_spec.rb` and
141
+ `spec/specguard/rspec/regression_targets_spec.rb`.
142
+ - **A run that could not produce verdicts emits no document.** Bad flags, `--changed` outside a
143
+ repository, a validator that could not be resolved — all still exit `2` with prose on stderr. Those runs checked nothing, and `{"ok": false, "findings": []}` is exactly how a
144
+ gate that checked nothing gets mistaken for one that found nothing.
145
+ - **The provenance line stays on stderr** and is deliberately *not* duplicated into the document
146
+ (see below). Redirect `2>` to keep it; stdout is the document and nothing else.
147
+
148
+ ### The validator: `validate-intent`, resolved on every run
149
+
150
+ `specguard-lint` validates through the Go `validate-intent` binary and **only** through it — the
151
+ Ruby hand-rolled validation path was removed at SPGD-867, completing the SPGD-96 cutover. The
152
+ formatter's annotation half asks the same binary the linter does, so what CI ratified and what the
153
+ platform receives come from one implementation.
154
+
155
+ **Resolution order** (in `ValidatorBackend.resolve`):
156
+
157
+ 1. `SPECGUARD_VALIDATE_INTENT` names a binary → that one, verified before anything is selected or
158
+ checked. A blank value counts as unset.
159
+ 2. Otherwise → the platform-matched prebuilt binary from open-test-intent's GitHub release
160
+ (`v0.1.3`: `validate-intent-{linux,darwin}-{amd64,arm64}`, static, schema compiled in), fetched
161
+ on first run, verified against the release's `SHA256SUMS` manifest, installed atomically under
162
+ the user cache dir, and reused from there afterwards. Redirect the cache with
163
+ `SPECGUARD_CACHE_DIR` (XDG_CACHE_HOME is honoured too).
164
+
165
+ **When no binary can be resolved** — first run with no network, an unsupported platform — the run
166
+ exits `2` with a message naming **both** remediations, before anything is selected or checked:
167
+
168
+ ```
169
+ specguard-lint: error: could not obtain validate-intent from https://github.com/yatfa-ai/open-test-intent/releases/download/v0.1.3: ... ; set SPECGUARD_VALIDATE_INTENT to a validate-intent binary, or install one with: curl -fsSL https://raw.githubusercontent.com/yatfa-ai/open-test-intent/v0.1.3/scripts/install.sh | sh
170
+ ```
171
+
172
+ It never falls back to Ruby, because there is no Ruby validation left to fall back to — that is the
173
+ cutover. `--require-validator` is gone with the same stroke: its assertion ("fail unless the binary
174
+ actually ran") is now the always-on failure mode, so passing the retired flag is an ordinary usage
175
+ error (exit `2`, `invalid option`).
176
+
177
+ **CI without network access** should pin the install explicitly with open-test-intent's own
178
+ `install.sh` (the same verified release artifact) and point the env var at the result:
179
+
180
+ ```bash
181
+ curl -fsSL https://raw.githubusercontent.com/yatfa-ai/open-test-intent/v0.1.3/scripts/install.sh | sh
182
+ SPECGUARD_VALIDATE_INTENT="$(command -v validate-intent)" bundle exec specguard-lint --changed
183
+ ```
184
+
185
+ #### Every run says which validator ran
186
+
187
+ `specguard-lint` states it, in one line on **stderr**, on every run:
188
+
189
+ ```
190
+ specguard-lint: validated by validate-intent 1.4.0 (go1.22.12 linux/arm64) schema sha256:3760d8f7… at /path/to/validate-intent (SPECGUARD_VALIDATE_INTENT) — it reports enforcing the schema this gem vendors, loaded from /usr/local/schemas/open-test-intent.v1.json
191
+ ```
192
+
193
+ Three things worth knowing about it:
194
+
195
+ * **stdout is untouched.** The line is on stderr, beside the other diagnostics about the linter
196
+ itself, so the findings and the two `checked …` lines stay safe to pipe. Under `--json` it stays
197
+ exactly where it is and is **not** copied into the document.
198
+ * **The identity is the binary's own.** It comes from `<binary> --version`, asked once per run
199
+ before any file is selected, and is passed through verbatim rather than reworded — that is the
200
+ only thing that can tell two builds of the validator apart.
201
+ * **A binary that cannot answer still validates.** `--version` and `--schema-source` each arrived in
202
+ a later slice of the validator; a build without one reads the flag as a filename and exits 1. That
203
+ costs nothing — same findings, same exit code, same stdout — and the line says which question went
204
+ unanswered, in words (`… which could not report its identity, so the schema contract it carries
205
+ could not be checked`) rather than going missing.
206
+
207
+ #### Which schema the run enforces
208
+
209
+ The identity line is not only printed. `specguard-lint` compares the schema *this gem* vendors —
210
+ digested from the file at runtime — against the schema the binary reports, before any file is
211
+ selected or checked, and refuses the run when the two differ.
212
+
213
+ **Which digest it asks for is the whole of this check.** `validate-intent --version` ends
214
+ `schema sha256:<64-hex>`, the digest of the JSON Schema **compiled into** that binary — and that is
215
+ not the schema a run necessarily *loads*. A `schemas/open-test-intent.v1.json` sitting beside the
216
+ executable takes precedence over the compiled-in copy, and `--version` answers above that decision
217
+ and never reaches it; the binary's own `--help` says the digest "is not a claim about what a given
218
+ run enforced". So `specguard-lint` asks `validate-intent --schema-source`, which runs the real
219
+ loader and reports the origin and digest of the bytes a verdict run would enforce, and compares
220
+ *that*. Both questions are asked once per run, before any file is selected or checked.
221
+
222
+ This is the one thing about the pair that neither half can check by itself. Both sides already pin
223
+ their own schema against their own tree, and both stay green while disagreeing with each other: the
224
+ gem is installed from RubyGems, the binary is resolved or fetched by version separately, and nothing
225
+ ties the two vintages together. What that produces is a run that succeeds under a contract other
226
+ than the one this gem ships.
227
+
228
+ **The run enforces the schema this gem vendors.** It proceeds, and the line names where that schema
229
+ came from — an absolute path when a file beside the binary won, or `<embedded schema>` when the
230
+ compiled-in copy did:
231
+
232
+ ```
233
+ specguard-lint: validated by validate-intent 1.4.0 (…) at /path/to/validate-intent (SPECGUARD_VALIDATE_INTENT) — it reports enforcing the schema this gem vendors, loaded from <embedded schema>
234
+ ```
235
+
236
+ **It enforces a different one.** Exit `2`, before any file is selected or checked:
237
+
238
+ ```
239
+ specguard-lint: error: the validator backend at /path/to/validate-intent (SPECGUARD_VALIDATE_INTENT) reports enforcing schema sha256:9c1e…, loaded from /usr/local/schemas/open-test-intent.v1.json, but this gem vendors sha256:3760… — the two halves would enforce different contracts, so this run would produce a verdict this gem cannot stand behind; the binary identifies itself as validate-intent 1.5.0 (go1.22.12 linux/arm64) schema sha256:3760…
240
+ ```
241
+
242
+ Both digests are printed in full (elided above only to fit), because one of them lives inside a
243
+ binary and the other inside an installed gem and neither is inspectable from where the other lives.
244
+ The origin is there because it says *which half to move*: a stale `<embedded schema>` is fixed by
245
+ rebuilding or reinstalling the binary, and a path on this host by replacing or deleting that file.
246
+ To fix a divergence, move whichever half is stale so the two agree.
247
+
248
+ **The binary is too old to be asked.** `--schema-source` arrived in a later slice of the validator;
249
+ an older build reads it as a filename and exits 1, and a schema that exists beside the binary and
250
+ will not load exits 2 with its own "could not load schema" diagnostic (which the run reaches a
251
+ moment later anyway, from the path that owns it). Neither costs a verdict. The comparison falls back
252
+ to the *carried* digest — the same two outcomes, proceed or exit `2` — and the line keeps the hedge
253
+ that belongs to that weaker question, because on that path it is still true:
254
+
255
+ ```
256
+ specguard-lint: validated by validate-intent 1.2.0 (…) at /path/to/validate-intent (SPECGUARD_VALIDATE_INTENT), which reports carrying the schema this gem vendors — the contract it carries, not necessarily the one this run enforced
257
+ ```
258
+
259
+ **No digest to compare.** Never a refusal — same findings, same exit code, same stdout — and the
260
+ provenance line says which kind of "could not check" it was, in its own words:
261
+
262
+ * `…, which reports no schema digest, so the contract it carries could not be checked` — a build
263
+ older than the slice that added the token.
264
+ * `…, which could not report its identity, so the schema contract it carries could not be checked` —
265
+ a build too old to answer `--version` at all.
266
+ * `…, whose schema contract could not be checked: this gem could not read its own vendored copy` —
267
+ the gem's own installation is missing or unreadable. Not fatal on purpose: the binary's run does
268
+ not otherwise read that file, and a missing operand is an unanswered question, not a
269
+ disagreement.
270
+
271
+ #### Read-failure wording
272
+
273
+ The gem's arguments are paths; the binary's are glob patterns. A path that matches nothing —
274
+ missing, or not a regular file — reaches the gem as the same answer the binary gives both shapes
275
+ (`no-match`), which the CLI re-words as `could not read file: no file at this path` and folds into
276
+ the `read` kind. The binary's own UTF-8 refusal prose (`input is not well-formed UTF-8
277
+ (PROTOCOL.md §1.1 requires it)`) and parse-failure prose are passed through unaltered.
278
+
279
+ Every way the backend can fail — the binary is missing, will not execute, exits with something that
280
+ is not a verdict, or emits output that is not a report — is **exit 2**, the linter's "could not do
281
+ my job" code. It never becomes exit 1, which means "an annotation is malformed" and nothing else.
282
+
283
+ ## The formatter — `SpecGuard::RSpecFormatter`
284
+
285
+ An **additive** RSpec formatter: it runs alongside your usual one (`progress`, `documentation`, …)
286
+ rather than replacing it, and records every example that finished — annotated or not — as one JSON
287
+ object per run, POSTed to SpecGuard (or written to `log/test_results.jsonl` when there is no API
288
+ key).
289
+
290
+ ```ruby
291
+ # spec/spec_helper.rb
292
+ require "specguard/rspec/formatter"
293
+ RSpec.configure do |config|
294
+ config.add_formatter(SpecGuard::RSpecFormatter)
295
+ end
296
+ ```
297
+
298
+ ```
299
+ # ...or in .rspec — the --require is not optional, RSpec cannot guess this path
300
+ --require specguard/rspec/formatter
301
+ --format SpecGuard::RSpecFormatter
302
+ ```
303
+
304
+ The two forms are equivalent, and neither needs you to name a human formatter. Additive is meant
305
+ literally, in both directions: if you chose a formatter that reports the run to a human
306
+ (`progress`, `documentation`, `--format failures`, `--format json`, …), it is left alone and
307
+ SpecGuard adds nothing to your output; if you chose none, you get RSpec's default (`progress`)
308
+ exactly as you would without this gem — same dots, same failures, same summary, byte for byte.
309
+
310
+ The qualifier on that first half is deliberate. SpecGuard restores the default when no *other*
311
+ registered formatter would give a human an account of the run, and it judges that by the formatter
312
+ protocol — whether anything answers to `example_started`, `example_passed`, `example_failed`,
313
+ `example_pending` or `dump_summary`. So if the only other formatter you registered is a silent one
314
+ (another telemetry gem, a custom notifier that writes elsewhere), SpecGuard reads the run as
315
+ unserved and restores `progress`, and you get output you did not have before. That is the error
316
+ direction chosen on purpose — noisy beats silent, which is the whole point of this behaviour — but
317
+ if you want a genuinely quiet run, name a formatter that reports the run and says little:
318
+ `--format failures` prints one line per failure and nothing else, so a green suite stays at zero
319
+ bytes and the restore does not fire.
320
+
321
+ That second half is not free, because RSpec installs its default formatter only when *no* formatter
322
+ was registered at all — so a gem that registers one silently suppresses it, and a failing suite
323
+ prints nothing. SpecGuard restores it on the first notification of the run, once RSpec has finished
324
+ deciding. If you want something other than `progress`, name it the usual way (`--format
325
+ documentation`, or `config.default_formatter = "doc"`) and that is what you will get, on its own.
326
+
327
+ "Byte for byte" is checked rather than asserted: `spec/specguard/rspec/formatter_run_spec.rb` runs
328
+ each wiring and the same suite with no SpecGuard at all, and diffs the two streams end to end with
329
+ only the two wall-clock numbers erased. Both a failing suite and a suite that reports through
330
+ `reporter.message` — an error in an `after(:context)` hook — are compared that way, because they
331
+ travel through different formatters and an addition that is invisible in one shows up in the other.
332
+
333
+ Each example contributes its `id`, `spec_file_path`, `file_path`, `line_number`, `name` (the composed
334
+ `describe`/`context`/`it` string), `duration`, `outcome`, `status` (`"annotated"` or
335
+ `"unannotated"`) and `intent` — the parsed annotation when there is one, `null` when there is not;
336
+ the run envelope carries `commit_sha`, `branch` and `duration_seconds`.
337
+
338
+ `id` is RSpec's own example id — `./spec/orders_spec.rb[1:2]`, the argument that re-runs that one
339
+ example — and it is the key that distinguishes examples a coordinate cannot. A table-driven loop
340
+ writes its `it` once, so all of its examples share a `line_number`; a shared example group reports
341
+ the coordinate of `spec/support/shared.rb` from every file that includes it. `spec_file_path` is the
342
+ spec file that actually **ran** the example, which is the same as `file_path` for an ordinary example
343
+ and the *including* file for a shared one — so duration-by-file adds up against the file you would
344
+ have named, not against a `spec/support/` helper.
345
+
346
+ > `id` is unique within a run, not stable across refactors: it is positional, so reordering examples
347
+ > changes it, exactly as inserting a line changes `line_number`. Matching one test across runs is
348
+ > `name` plus file.
349
+
350
+ `file_path` and `line_number` keep meaning the **definition** site — that is the line the `@intent:`
351
+ annotation is read from.
352
+
353
+ An example counts as **annotated** when an `@intent:` sits on its `it` line, or on the comment line
354
+ immediately above it:
355
+
356
+ ```ruby
357
+ # @intent: { entity: "Order", action: "refund", behavior: "restores stock levels on refund", layer: "unit" }
358
+ it "restores stock on refund" do
359
+
360
+ it "surfaces the decline reason" do # @intent: { entity: "Order", action: "checkout", ... }
361
+ ```
362
+
363
+ One line of lookback, no more — and a trailing annotation belongs to its own example only, never to
364
+ the one on the next line.
365
+
366
+ > **A malformed or schema-invalid annotation is recorded as `unannotated`, with a `null` intent, and
367
+ > the formatter says nothing about it.** That is deliberate: telemetry must never block CI, and the
368
+ > platform validates a run's payload as a whole — so shipping one bad annotation would cost the
369
+ > *entire run* its telemetry rather than one row its metadata. `specguard-lint` is the half of this
370
+ > gem that tells you about a bad annotation, loudly, with exit code `1`. Run it in CI and the
371
+ > formatter never has anything to hide.
372
+
373
+ ```ruby
374
+ # optional — the defaults read the commit, branch, CI run id and shard index
375
+ # from whichever provider is running you (GitHub Actions, GitLab CI, CircleCI,
376
+ # Buildkite, Jenkins), and when none of them named the commit or the branch,
377
+ # ask git directly for both — so a laptop run and a hand-rolled container
378
+ # report their checkout too, without being configured to. A detached checkout
379
+ # reports no branch rather than the string "HEAD".
380
+ # SPECGUARD_COMMIT_SHA / SPECGUARD_BRANCH / SPECGUARD_RUN_ID /
381
+ # SPECGUARD_SHARD_ID / SPECGUARD_OUTPUT_PATH / SPECGUARD_LOCAL_OUTPUT_PATH
382
+ # override any of it.
383
+ #
384
+ # Assign a value here only when it is one neither source can know:
385
+ SpecGuard::RSpec.configure do |config|
386
+ config.branch = "release/2.0"
387
+ end
388
+ ```
389
+
390
+ ## Shipping the run to SpecGuard
391
+
392
+ Set an API key and an endpoint and the run is POSTed to
393
+ `<endpoint>/api/v1/ingest` — once per process, as a single request (see
394
+ [If you shard your suite](#if-you-shard-your-suite) for what happens when there
395
+ is more than one process):
396
+
397
+ ```bash
398
+ export SPECGUARD_ENDPOINT=https://specguard.example.com
399
+ export SPECGUARD_API_KEY=… # from your repository's settings
400
+ export SPECGUARD_TIMEOUT=10 # optional; seconds, applied to connect and read
401
+ ```
402
+
403
+ ```ruby
404
+ # ...or in Ruby, if you would rather not use the environment
405
+ SpecGuard::RSpec.configure do |config|
406
+ config.endpoint = "https://specguard.example.com"
407
+ config.api_key = ENV["SPECGUARD_API_KEY"]
408
+ config.timeout = 10
409
+ end
410
+ ```
411
+
412
+ **The API key is the switch.** With no key nothing is sent anywhere and the run
413
+ is written to `log/test_results.local.jsonl` — the local development record,
414
+ kept apart from the replay queue — so local development needs no opt-out, and a
415
+ fork with no secret configured behaves like a laptop rather than like a broken
416
+ build. The local file's name is configurable via `SPECGUARD_LOCAL_OUTPUT_PATH`
417
+ (or `SpecGuard::RSpec.configure { |c| c.local_output_path = ... }`).
418
+
419
+ **A failed delivery is never silent, and never lost.** If the endpoint refuses
420
+ the run (a `401` from a rotated key, a `400`, a `500`) or cannot be reached at
421
+ all (connection refused, DNS failure, timeout), the formatter prints **one**
422
+ line to stderr naming the status or the error, and writes the payload to
423
+ `log/test_results.jsonl` — the **replay queue**: runs offered to the endpoint
424
+ and not accepted — so the run can be replayed later with
425
+ [`specguard-ingest`](#replaying-a-saved-run--specguard-ingest):
426
+
427
+ ```
428
+ SpecGuard: could not deliver test telemetry (HTTP 401 — the API key was not
429
+ accepted). Falling back to log/test_results.jsonl; the test run is unaffected.
430
+ ```
431
+
432
+ **That line carries the endpoint's own words when it has any.** A `400` refusal
433
+ names the offending spec by index, file and line, so a rejected payload is a
434
+ thing you can fix from the CI log rather than one you have to reproduce
435
+ locally:
436
+
437
+ ```
438
+ SpecGuard: could not deliver test telemetry (HTTP 400 — the endpoint rejected
439
+ the payload — spec 3 (spec/orders_spec.rb:9): line_number is required and must
440
+ be a positive integer; spec 7 (spec/orders_spec.rb:31): outcome must be one of
441
+ passed, failed, pending). Falling back to log/test_results.jsonl; the test run
442
+ is unaffected.
443
+ ```
444
+
445
+ It stays **one** line whatever comes back. A systemic problem can have the
446
+ endpoint refusing every spec in the suite, so at most three reasons are spelled
447
+ out and the rest are counted (`… and 497 more`); anything that arrives without
448
+ a reason it can read — an empty body, or the HTML a proxy answers a `413` with
449
+ — prints the bare status line above and is still reported as a refusal, not as
450
+ an error.
451
+
452
+ There are **no retries**, and the whole delivery is bounded by `timeout`
453
+ (10 seconds by default, against `Net::HTTP`'s own 60): telemetry is explicitly
454
+ allowed to be lost, and a retry would only double what a hung endpoint can cost
455
+ your CI run.
456
+
457
+ **A dry run is refused, to both sinks.** `rspec --dry-run` builds and reports
458
+ every example without executing a single body, so its per-example `duration` is
459
+ the cost of *constructing* an example (single-digit microseconds — a
460
+ `sleep 0.05` example understates its own runtime by three to four orders of
461
+ magnitude) and its `outcome` is `passed` for code that never
462
+ ran. Nothing downstream can tell the difference, and an all-green, near-instant
463
+ run is exactly the shape that poisons both the numbers SpecGuard reports. So
464
+ when RSpec is in dry-run mode the formatter makes no POST **and** writes no line
465
+ to `log/test_results.jsonl` — a file full of zero-duration green runs is the
466
+ same corruption, deferred until something replays it — and says so once:
467
+
468
+ ```
469
+ SpecGuard: skipped test telemetry for a dry run (rspec --dry-run executes no
470
+ example bodies, so this run's durations and outcomes would not be
471
+ measurements). Nothing was sent or written; the test run is unaffected.
472
+ Annotation coverage is a fact about source, not about execution, so it
473
+ survives the refusal — this working tree: 6 examples, 3 annotated,
474
+ 3 unannotated (50% annotated).
475
+ the 3 unannotated examples, by definition site:
476
+ spec/orders_spec.rb:7 Order has no annotation
477
+ spec/orders_spec.rb:16 Order has a malformed annotation
478
+ spec/orders_spec.rb:21 Order has a schema-invalid annotation
479
+ ```
480
+
481
+ The refusal throws away less than it used to. `duration` and `outcome` are
482
+ fabricated by a dry run, which is what makes them unpublishable — but the
483
+ third field the formatter computes per example, `annotated` / `unannotated`,
484
+ comes from scanning the **spec file's source text** and is identical whether or
485
+ not a body ran. Since a dry run still builds every example, it holds the exact
486
+ numerator and denominator of the annotation-coverage metric, so `--dry-run` is
487
+ also the way to ask *"where are we?"* without a commit, a push, and a CI round
488
+ trip. The figure describes **your working tree right now**, so it will differ
489
+ from the dashboard's the moment you edit a spec — that difference is the point.
490
+
491
+ Nothing is published either way: the report goes to stderr and to nowhere else.
492
+
493
+ This matters most where you are least likely to look for it: an API key is
494
+ usually an environment-level secret rather than a job-level one, so a lint job
495
+ that runs `rspec --dry-run` to catch an unparseable spec file inherits the key
496
+ and would otherwise overwrite your suite's real duration and pass/fail picture
497
+ with zeroes and green.
498
+
499
+ It **never blocks CI.** RSpec does not sandbox formatters — an exception raised
500
+ in one escapes the runner and takes RSpec's own exit code with it — so every
501
+ hook rescues, warns once on stderr, and leaves the exit status to your suite
502
+ alone. A non-2xx response gets the same treatment: `Net::HTTP` returns those as
503
+ ordinary values rather than raising, so they are checked for explicitly instead
504
+ of being left to a `rescue` that would never see them.
505
+
506
+ ### Replaying a saved run — `specguard-ingest`
507
+
508
+ The suite is over by the time you see the `401`, and re-running it to recover
509
+ the telemetry costs you the whole suite again. So the file the formatter wrote
510
+ is the run: each line is byte-for-byte the body the endpoint refused, and
511
+ `specguard-ingest` is the command that sends it.
512
+
513
+ ```bash
514
+ export SPECGUARD_API_KEY=… # the key that was rotated, fixed
515
+ bundle exec specguard-ingest log/test_results.jsonl
516
+ ```
517
+
518
+ ```
519
+ line 1: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
520
+ line 2: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
521
+ specguard-ingest: delivered 2 of 2 runs from log/test_results.jsonl
522
+ specguard-ingest: lines 1, 2 carried ci_run_id 17442 and each came back with
523
+ test_run_id 41f2c9b8 — the endpoint folded them onto one run
524
+ ```
525
+
526
+ It reads the same `SPECGUARD_ENDPOINT`, `SPECGUARD_API_KEY` and
527
+ `SPECGUARD_TIMEOUT` the formatter does, and sends each line through the same
528
+ transport — so a run that was refused for a rotated key appears on the platform
529
+ once the secret is fixed, with the shard folding described in
530
+ [If you shard your suite](#if-you-shard-your-suite) applying exactly as it would
531
+ have during the run.
532
+
533
+ > **It re-delivers *every* line in the file you give it.** Two kinds of line can
534
+ > be sitting in the file you point it at. In files written by **an earlier
535
+ > version of the gem**, and in `log/test_results.local.jsonl` — the local
536
+ > development record the formatter now writes when no API key is configured —
537
+ > ordinary local runs and genuinely failed deliveries are indistinguishable on
538
+ > the line, because nothing in the payload records which sink it was destined
539
+ > for: the writer split into two files precisely so this could not happen on
540
+ > new replay-queue files, and a line from before that split carries no marker.
541
+ > So this command will send every line in such a file, and no filter or
542
+ > heuristic can change that: guessing which lines "were failures" from data
543
+ > that does not say would be confidently wrong about which of your runs reach
544
+ > the platform. Check the file first with
545
+ > [`--list`](#checking-a-file-before-you-send-it----list), and check it before
546
+ > you replay one you did not write. A `log/test_results.jsonl` written entirely
547
+ > by this version or later holds only genuine failed deliveries by
548
+ > construction — but verify that before you rely on it. If you deliberately
549
+ > want one file for both roles, set `local_output_path` to the same value as
550
+ > `output_path` and the formatter reproduces the old single-file behaviour.
551
+
552
+ #### Checking a file before you send it — `--list`
553
+
554
+ `--list` prints one row per line and **delivers nothing**:
555
+
556
+ ```bash
557
+ bundle exec specguard-ingest --list log/test_results.jsonl
558
+ ```
559
+
560
+ ```
561
+ line 1: branch main, commit_sha 0d4a1f2c9b8e7d6a5f4c3b2a1908f7e6d5c4b3a2, ci_run_id 17442, 412 examples, 93.4s
562
+ line 2: branch main, commit_sha 0d4a1f2c9b8e7d6a5f4c3b2a1908f7e6d5c4b3a2, ci_run_id 17442, 388 examples, 91.2s
563
+ line 3: branch spike/local, commit_sha 9c2e7a10b4d3, no ci_run_id, 6 examples, 0.4s
564
+ line 4: unparseable — could not parse the line as JSON: unexpected end of input
565
+ specguard-ingest: listed 4 lines from log/test_results.jsonl; nothing was delivered
566
+ ```
567
+
568
+ Every field on the row is already on the line — nothing is guessed at, and a
569
+ line the command cannot parse is listed **as unparseable** rather than quietly
570
+ dropped from the preview. `no ci_run_id` is the one to read for: that line has
571
+ no identity for SpecGuard to fold a redelivery onto, so sending it creates a new
572
+ run rather than joining an existing one.
573
+
574
+ Reading the file yourself is not the alternative. One line is one whole run, and
575
+ at 20,000 examples that is megabytes of JSON on a single physical line.
576
+
577
+ **It needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`** — deliberately.
578
+ The file most worth checking is the one written *because* no API key was set, so
579
+ requiring a key to look at it would withdraw the instrument in exactly the
580
+ situation that produces the hazard. It composes with `--from-line` and `--lines`
581
+ too, so you can list the exact set you are about to send:
582
+
583
+ ```bash
584
+ bundle exec specguard-ingest --list --from-line 7 log/test_results.jsonl
585
+ bundle exec specguard-ingest --list --lines 3,7,12-15 log/test_results.jsonl
586
+ ```
587
+
588
+ A listing under either selector previews **exactly** the lines the same command
589
+ without `--list` would deliver, by the same numbers.
590
+
591
+ Listing sends nothing, so it can never be a verdict about a run: it exits `0`
592
+ when it listed the file and `2` when it could not read it or the flags were
593
+ wrong. **`1` is unreachable with `--list`.**
594
+
595
+ **Each line is reported by its line number**, and `--from-line N` starts at one —
596
+ so a file that was only partly accepted is resumed from the line the report
597
+ named, rather than blindly re-sent:
598
+
599
+ ```bash
600
+ bundle exec specguard-ingest --from-line 7 log/test_results.jsonl
601
+ ```
602
+
603
+ The numbering never shifts: line 7 is line 7 of the file you gave it, both times.
604
+ Re-sending a line that already landed is harmless *only* when it carries a
605
+ `ci_run_id` — that is the identity SpecGuard folds a redelivery onto. A line
606
+ **without** one has nothing to fold onto and becomes a second run, and a keyless
607
+ local file is made entirely of those, so `--from-line` is worth the two seconds
608
+ it takes to read the previous report.
609
+
610
+ #### Sending a set rather than a suffix — `--lines`
611
+
612
+ `--from-line` can only express a **suffix**, and the set a per-line report points
613
+ at is a suffix at most once. `--lines` takes the set itself — comma-separated
614
+ numbers and ranges, over the file's own numbering:
615
+
616
+ ```bash
617
+ bundle exec specguard-ingest --lines 3,7,12-15 log/test_results.jsonl
618
+ ```
619
+
620
+ ```
621
+ line 3: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
622
+ line 7: accepted — HTTP 202, test_run_id 41f2c9b8, ci_run_id 17442
623
+ line 12: accepted — HTTP 202, test_run_id 5a3d0e91, ci_run_id 17443
624
+
625
+ specguard-ingest: delivered 6 of 6 runs from log/test_results.jsonl; 34 lines not selected by --lines
626
+ ```
627
+
628
+ Two things want it. The first is an **interior line that will never be accepted**:
629
+ an HTTP `400` is the one response SpecGuard forms an opinion about your payload
630
+ in, so a line it refuses is refused every time it is offered. Sitting at line 3
631
+ of a 40-line file, no `--from-line` can step over it — the file can never be
632
+ replayed to completion, and the command can never exit `0` over it. Naming the
633
+ set around it can:
634
+
635
+ ```bash
636
+ bundle exec specguard-ingest --lines 1-2,4-40 log/test_results.jsonl
637
+ ```
638
+
639
+ The second is that the sink is **append-only and mixes both sources**, so
640
+ ordinary keyless laptop runs keep landing *after* the CI failures you want to
641
+ replay. Every unwanted keyless line a too-early `--from-line` sweeps up is a
642
+ spurious run on the platform, because a line with no `ci_run_id` has nothing to
643
+ fold onto.
644
+
645
+ Carving the file up first (`sed -n '21,24p' file > tmp.jsonl`) is not the
646
+ alternative: a carved file **renumbers**, and the whole value of acting on a
647
+ per-line report is that line 12 is still line 12.
648
+
649
+ The held-back lines are **counted and reported**, exactly as `--from-line`'s and
650
+ the blank ones are — a summary that quietly narrowed what it was summarising
651
+ would be worse than no summary.
652
+
653
+ A spec is read strictly, and a bad one is a `2` rather than a fallback to the
654
+ whole file — which is the one outcome a selector exists to prevent. `--lines 0`,
655
+ `--lines 5-2`, `--lines abc`, `--lines 12-`, an empty spec and an empty entry
656
+ (`3,,5`) are all refused, naming what was wrong. Whitespace *between* entries is
657
+ fine (`3, 7`); inside one it is a typo, not a range (`5 - 7` is refused).
658
+
659
+ **`--lines` and `--from-line` do not combine** — giving both is a `2`:
660
+
661
+ ```
662
+ specguard-ingest: error: --from-line and --lines both choose which lines to send; give one or the other
663
+ ```
664
+
665
+ They answer the same question, and intersecting them would silently drop a
666
+ number you typed: `--from-line 5 --lines 3,7` would send only line 7, and the 3
667
+ would vanish without a word. Refusing the pair is the same discipline as the
668
+ rest of this command — it will not quietly narrow what it was asked for.
669
+
670
+ Repeating **one** selector is a different case and is allowed: the last one
671
+ wins. `--lines 1,2 --lines 4` sends line 4, and `--from-line 2 --from-line 5`
672
+ starts at 5. A repeat replaces rather than intersects, so the set delivered is
673
+ exactly the last one you typed — nothing is combined into something smaller than
674
+ you asked for, which is the objection to the pair above. It is also what lets
675
+ you override a selector baked into a wrapper script or shell alias by appending
676
+ a new one.
677
+
678
+ Nothing about a line's **content** is consulted by either flag. The numbers come
679
+ from you, after reading `--list`; that is what keeps this an explicit selector
680
+ rather than the heuristic this command refuses to grow.
681
+
682
+ Each line is delivered **once** — the command runs out of band and costs your CI
683
+ nothing, but a retry loop cannot see *why* an attempt failed and you can, so
684
+ re-running the command is the retry.
685
+
686
+ **A dry run is never in the file**, so nothing here can replay one: the
687
+ formatter refuses both sinks for `rspec --dry-run` (see above), which means this
688
+ command inherits that guarantee rather than re-checking it.
689
+
690
+ The exit code is the contract, and it is `specguard-lint`'s:
691
+
692
+ | Code | Meaning |
693
+ | --- | --- |
694
+ | `0` | every line was accepted |
695
+ | `1` | at least one line was **refused by the endpoint** — it read the payload and said no, with its own reasons rendered exactly as the formatter renders them |
696
+ | `2` | the command could not do its job — no endpoint or API key, an unreadable file, an unparseable line, a bad flag, or a delivery the platform never stored |
697
+
698
+ `1` is reachable only by the endpoint having read a payload and said no, which
699
+ is an **HTTP 400** and nothing else: that is the one response SpecGuard forms an
700
+ opinion about your run in. A `401` is answered before the request reaches the
701
+ code that would read the payload; a `404`, `429` or `5xx` never gets that far
702
+ either. **Nothing was stored in any of them**, so all of them are a `2` — as are
703
+ a connection refused, a DNS failure and a timeout. Reporting any of these as a
704
+ `1` would be the command telling you your suite is bad on the strength of a
705
+ rotated key, a typo in `SPECGUARD_ENDPOINT`, or a bad afternoon at the platform.
706
+ Note what that buys you: `1` means *fix the payload*, `2` means *fix the setup
707
+ or try again later*, and a `404` and an unset `SPECGUARD_ENDPOINT` — the same
708
+ mistake — give you the same code. When a file produces both, `2` wins, and every
709
+ line is still printed either way.
710
+
711
+ `--list` sits outside that table's `1`, and outside most of its `2`: it makes no
712
+ request, so no endpoint has read anything and there is no verdict to report. A
713
+ listing exits `0` or `2` only, and the only `2`s it can reach are a bad flag and
714
+ a file it could not read. The other causes in that row are delivery's, not
715
+ listing's — listing needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`,
716
+ and an unparseable line becomes a row in the listing that names it rather than
717
+ an exit code.
718
+
719
+ **What it will not tell you** is whether a replayed line *created* a new run or
720
+ *folded into* an existing one. The ingest endpoint's `202` carries the run's id
721
+ but no created-versus-updated flag, so the command reports what it can see: the
722
+ `test_run_id` that came back, and whether the line carried a `ci_run_id` of its
723
+ own. Two lines that went out with the same `ci_run_id` and came back with the
724
+ same `test_run_id` landed on one record — that is the sentence above, and it is
725
+ an observation rather than an inference.
726
+
727
+ Bulk-importing an aged archive is **not** what this is for. The payload carries
728
+ no execution timestamp and SpecGuard orders a repository's runs by when they
729
+ were ingested, so a replayed run becomes the repository's latest. For the case
730
+ this exists to serve — replay the run that just failed, right after fixing the
731
+ credential — that is correct.
732
+
733
+ #### Machine-readable output — `--json`
734
+
735
+ An HTTP `400` is the one **permanent** verdict in the table above: a refused line
736
+ is refused every time it is offered, so the only way to land the run is to learn
737
+ which specs SpecGuard objected to and fix the payload. It names **every** one of
738
+ them — one error per offending spec, by index, file and line — and the human
739
+ report has room for three:
740
+
741
+ ```
742
+ line 3: refused — HTTP 400 — the endpoint rejected the payload — specs[417] spec/models/user_spec.rb:88: duration must be a non-negative number when present; specs[418] …; specs[419] … and 19997 more
743
+ ```
744
+
745
+ That cap is right where it is: it exists for the **one stderr line** an in-run CI
746
+ warning is allowed, and the formatter still has to fit inside it. It is a cap on
747
+ a *line*, though, and `--json` is the other channel — stdout carries one JSON
748
+ document instead of the human report, with the whole list in it:
749
+
750
+ ```bash
751
+ bundle exec specguard-ingest --json log/test_results.jsonl
752
+ ```
753
+ ```json
754
+ {
755
+ "tool": "specguard-ingest",
756
+ "mode": "deliver",
757
+ "file": "log/test_results.jsonl",
758
+ "summary": { "lines": 3, "attempted": 3, "accepted": 2, "refused": 1,
759
+ "undelivered": 0, "unparseable": 0, "blank": 0, "skipped": 0,
760
+ "selector": null },
761
+ "lines": [
762
+ { "number": 1, "status": "accepted", "code": 202, "reasons": [],
763
+ "test_run_id": "41f2c9b8", "ci_run_id": "17442" },
764
+ { "number": 2, "status": "accepted", "code": 202, "reasons": [],
765
+ "test_run_id": "41f2c9b8", "ci_run_id": "17442" },
766
+ { "number": 3, "status": "refused", "code": 400, "test_run_id": null,
767
+ "ci_run_id": "17443",
768
+ "reasons": [
769
+ "specs[417] spec/models/user_spec.rb:88: duration must be a non-negative number when present",
770
+ "specs[418] spec/models/user_spec.rb:96: duration must be a non-negative number when present"
771
+ ] }
772
+ ],
773
+ "foldings": [
774
+ { "ci_run_id": "17442", "test_run_id": "41f2c9b8", "lines": [1, 2] }
775
+ ]
776
+ }
777
+ ```
778
+
779
+ | field | meaning |
780
+ | --- | --- |
781
+ | `tool` | always `"specguard-ingest"`. Deliberately **not** a schema id: this document is about deliveries, and `specguard-lint --json` is the one that mirrors `validate-intent`'s |
782
+ | `mode` | `"deliver"` or `"list"` — whether the lines were sent or only shown |
783
+ | `file` | the path you gave it, echoed back |
784
+ | `summary.lines` | rows in `lines`: the lines that carried a payload and were not held back by a selector |
785
+ | `summary.attempted` | how many of those were offered to the endpoint — always `0` under `--list`, and `lines` minus the unparseable ones otherwise |
786
+ | `summary.accepted` / `refused` / `undelivered` / `unparseable` | the same four counts the text summary line states, computed once for both renderers so they cannot disagree |
787
+ | `summary.blank` / `skipped` | the two ways a line of the file is not a row here, counted rather than dropped |
788
+ | `summary.selector` | `"--lines"`, `"--from-line"`, or `null` when nothing was held back |
789
+ | `lines[]` | one entry per row, in the file's order |
790
+ | `foldings[]` | folding, **observed**: the lines that went out with one `ci_run_id` and came back with one `test_run_id`. The same statement the text report makes as a sentence |
791
+
792
+ Every delivered line has the same six keys:
793
+
794
+ | field | meaning |
795
+ | --- | --- |
796
+ | `number` | its 1-based line number in the file **as given**, blank lines counted — so it is the number `--from-line` and `--lines` take |
797
+ | `status` | `accepted`, `refused`, `undelivered` or `unparseable`. The tool's own vocabulary, not the report's wording (`undelivered`, where the row prints `not delivered`) |
798
+ | `code` | the HTTP status, or **`null`** where there is not one: a line that was never a run, and a delivery that got no answer at all (connection refused, DNS, TLS, a timeout) |
799
+ | `reasons` | why the line did not land — **always** a list of strings, never null and never a bare string, so a consumer never branches on its type. SpecGuard's own per-spec errors on a refusal (all of them, in its order), the parse problem where the line was not a run, the error where nothing reached the endpoint, and `[]` where it landed or where the refusal's body said nothing readable |
800
+ | `test_run_id` | the run the line landed on, as the endpoint reported it; `null` where that cannot be said honestly |
801
+ | `ci_run_id` | the run identity the line carried, or `null` — the field to read for, because a line without one has nothing for SpecGuard to fold a redelivery onto |
802
+
803
+ Every listed line has the same eight keys — `number`, `status` and `reasons`, as
804
+ on a delivered line, and then the five envelope facts the text row prints
805
+ instead of a delivery's outcome:
806
+
807
+ | field | meaning |
808
+ | --- | --- |
809
+ | `number` | its 1-based line number in the file **as given**, blank lines counted — the same number as on a delivered line, and the one `--from-line` and `--lines` take |
810
+ | `status` | `listed`, or `unparseable` where the line could not be parsed as a run — the two outcomes a preview has, since nothing was sent |
811
+ | `reasons` | the parse problem on an `unparseable` row, and `[]` on a `listed` one — **always** a list of strings, never null and never a bare string, so a consumer never branches on its type. It is the only field that says *why* a previewed line is unusable |
812
+ | `branch` | the branch the line carried, or **`null`** where the row says `no branch` |
813
+ | `commit_sha` | the commit the line carried, or **`null`** |
814
+ | `ci_run_id` | the run identity the line carried, or **`null`** where the row says `no ci_run_id` — the field to read for here too |
815
+ | `examples` | how many examples the line carried, or **`null`** where the row says `no specs`. `0` and `null` stay different facts, exactly as `0 examples` and `no specs` do |
816
+ | `duration_seconds` | the run's duration, or **`null`** where the row says `no duration_seconds` |
817
+
818
+ `--list --json` needs no `SPECGUARD_ENDPOINT` and no `SPECGUARD_API_KEY`,
819
+ exactly as `--list` does, and it previews the same set by the same numbers a
820
+ delivery would send.
821
+
822
+ Four things worth knowing:
823
+
824
+ - **The exit code is identical with and without the flag**, and the default
825
+ output is unchanged. `--json` is a second renderer over the same lines, the
826
+ same statuses and the same counts — pinned that way, byte for byte, in
827
+ `spec/specguard/rspec/regression_targets_spec.rb`.
828
+ - **`--json` does not lift the cap on the human line.** The two channels render
829
+ the same refusal at different lengths on purpose; nothing about the formatter's
830
+ in-run warning moves.
831
+ - **A run that never got as far as reading the file emits no document.** A bad
832
+ flag, `--from-line` with `--lines`, no endpoint or API key, a file that cannot
833
+ be read — all still exit `2` with prose on stderr and **nothing** on stdout,
834
+ because there is nothing yet to be a document about. A file the command *did*
835
+ read always gets one, whatever the exit code, including an empty one.
836
+ - **Warnings stay on stderr**, in both renderers. A run that delivered nothing is
837
+ still loud there; stdout is the document and nothing else.
838
+
839
+ ### If you shard your suite
840
+
841
+ `parallel_tests`, Knapsack and a CI matrix all run the suite as several
842
+ processes, and each one loads this formatter and POSTs its own slice. The run id
843
+ is what tells SpecGuard those POSTs are **one run**: shards that share it are
844
+ accumulated onto a single record, so a 20,000-example suite reports a 20,000
845
+ denominator instead of one record per shard holding a quarter of it — and a
846
+ quarter is what the dashboard showed before, attributed to the right commit,
847
+ with nothing to mark it as partial.
848
+
849
+ Every supported provider publishes an id for the build (`GITHUB_RUN_ID`,
850
+ `CI_PIPELINE_ID`, `CIRCLE_WORKFLOW_ID`, `BUILDKITE_BUILD_ID`, `BUILD_TAG`), so a
851
+ sharded job on any of them needs no configuration. If you shard somewhere else,
852
+ export one yourself — any value that every shard of the run shares and no other
853
+ run repeats:
854
+
855
+ ```bash
856
+ export SPECGUARD_RUN_ID="$MY_CI_BUILD_ID"
857
+ ```
858
+
859
+ Unset is not an error. A run with no id is treated as a run of its own, which is
860
+ exactly right for `bundle exec rspec` on a laptop. A genuinely different run — a
861
+ nightly, a later push — gets a different id from its provider and stays a
862
+ separate record, which is why the commit alone cannot do this job.
863
+
864
+ #### Re-runs, and why each shard also names itself
865
+
866
+ A CI run id does **not** change when you re-run the build. GitHub's own wording
867
+ for `GITHUB_RUN_ID` is *"This number does not change if you re-run the workflow
868
+ run"*; Buildkite retries a job inside the same `BUILDKITE_BUILD_ID` and GitLab
869
+ inside the same `CI_PIPELINE_ID`. That is the behaviour SpecGuard wants — press
870
+ "re-run failed jobs" on a sharded suite and only the failed shards run again, so
871
+ they need to land back on the run they came from rather than forming a new run
872
+ holding a fifth of the suite.
873
+
874
+ For that to be right, a shard has to be able to *replace* its own earlier
875
+ numbers instead of adding to them, which means naming itself. SpecGuard reads
876
+ the shard index your runner already exports:
877
+
878
+ | Runner | Variable |
879
+ | --- | --- |
880
+ | `parallel_tests` | `TEST_ENV_NUMBER` (its blank first process is read as shard `1`) |
881
+ | GitLab `parallel:`, Knapsack Pro | `CI_NODE_INDEX` |
882
+ | CircleCI `parallelism:` | `CIRCLE_NODE_INDEX` |
883
+ | Buildkite `parallelism:` | `BUILDKITE_PARALLEL_JOB` |
884
+
885
+ **GitHub Actions `matrix:` is the one that needs a line of config.** It exports
886
+ no per-leg index — `GITHUB_JOB` is the job's id in your YAML and is identical
887
+ across every leg — so set it from the matrix value:
888
+
889
+ ```yaml
890
+ strategy:
891
+ matrix:
892
+ shard: [1, 2, 3, 4]
893
+ steps:
894
+ - run: bundle exec rspec
895
+ env:
896
+ SPECGUARD_SHARD_ID: ${{ matrix.shard }}
897
+ ```
898
+
899
+ The value only has to be unique *within one run*; it is never compared across
900
+ runs. Do the same if you nest — `parallel_tests` inside a matrix leg repeats
901
+ `TEST_ENV_NUMBER` across legs, so give `SPECGUARD_SHARD_ID` something that
902
+ composes both.
903
+
904
+ Leaving it unset is not an error and does not lose the slice: an unnamed shard
905
+ is still counted into the run. What it cannot do is be recognised on a second
906
+ delivery, so if that shard is retried its numbers are added again rather than
907
+ replacing what was there. If your suite shards and you re-run it, name the
908
+ shards.
909
+
910
+ ## What SpecGuard collects
911
+
912
+ Everything below is the whole of it. `Transport#deliver` sends the payload
913
+ verbatim — there is no filtering layer between what the formatter captures and
914
+ what leaves the machine — so this list *is* the request body, and a spec pins
915
+ it so that adding a field without updating this section fails the build. A run
916
+ big enough to be worth compressing is gzipped in transit; that changes how the
917
+ body is encoded on the wire, never what is in it.
918
+
919
+ **The run envelope — six fields, once per process:**
920
+
921
+ | Field | What it holds | Where it comes from |
922
+ | --- | --- | --- |
923
+ | `commit_sha` | the commit the suite ran against | `SPECGUARD_COMMIT_SHA` if you set it, else `GITHUB_SHA`, `CI_COMMIT_SHA`, `CIRCLE_SHA1`, `BUILDKITE_COMMIT`, `GIT_COMMIT`, else `git rev-parse HEAD` |
924
+ | `branch` | the branch name; `null` on a detached checkout | `SPECGUARD_BRANCH` if you set it, else `GITHUB_REF_NAME`, `CI_COMMIT_REF_NAME`, `CIRCLE_BRANCH`, `BUILDKITE_BRANCH`, `GIT_BRANCH`, else `git symbolic-ref --short -q HEAD` |
925
+ | `ci_run_id` | your provider's build id, so shards of one run fold together; `null` on a laptop | `SPECGUARD_RUN_ID` if you set it, else `GITHUB_RUN_ID`, `CI_PIPELINE_ID`, `CIRCLE_WORKFLOW_ID`, `BUILDKITE_BUILD_ID`, `BUILD_TAG` |
926
+ | `shard_id` | which slice of that run this process is; `null` when unsharded | `SPECGUARD_SHARD_ID` if you set it, else `TEST_ENV_NUMBER`, `CI_NODE_INDEX`, `CIRCLE_NODE_INDEX`, `BUILDKITE_PARALLEL_JOB` |
927
+ | `duration_seconds` | wall clock for the whole run | measured by the formatter |
928
+ | `specs` | one object per example that finished — the nine fields below | the run |
929
+
930
+ **Each example — nine fields, one object per example, annotated or not:**
931
+
932
+ | Field | What it holds | Where it comes from |
933
+ | --- | --- | --- |
934
+ | `id` | RSpec's own example id, `./spec/orders_spec.rb[1:2]` — the re-run argument | `example.id` |
935
+ | `spec_file_path` | the spec file that **ran** the example, relative to the project root when it lives under it — an absolute path when it does not, because a spec outside the working directory has no relative name | `metadata[:rerun_file_path]` |
936
+ | `file_path` | the spec file the example is **defined** in, on the same terms | `metadata[:file_path]` |
937
+ | `line_number` | the line it is defined on | `metadata[:line_number]` |
938
+ | `name` | the composed `describe`/`context`/`it` string | `example.full_description` |
939
+ | `duration` | seconds that one example took | `execution_result.run_time` |
940
+ | `outcome` | `passed`, `failed` or `pending` | `execution_result.status` |
941
+ | `status` | `annotated` or `unannotated` | whether an `@intent:` was found for that line |
942
+ | `intent` | the parsed `@intent:` annotation; `null` when there is none | the annotation you wrote in the spec file |
943
+
944
+ Two request headers say something about you rather than about the request: the
945
+ API key travels as a bearer token in `Authorization`, and `User-Agent` names
946
+ this gem and its version (`specguard-ruby/<version>`), so the platform can tell
947
+ its clients apart. The rest are ordinary HTTP plumbing that describe the message
948
+ itself and carry nothing about your code or your suite — `Content-Type`,
949
+ `Accept`, `Content-Length`, `Host`, `Accept-Encoding`, and `Content-Encoding:
950
+ gzip` on a run large enough to be compressed. A spec pins that header set too,
951
+ so a header added later cannot quietly slip past this paragraph.
952
+
953
+ ### Test names and annotations are free text, and that is the point
954
+
955
+ `name` and `intent` are written by your developers, in prose; `file_path` and
956
+ `spec_file_path` are the names they gave the files. They **will** carry internal
957
+ product detail — feature names, customer names, the shape of work you have not
958
+ shipped — because a suite describes the system it tests.
959
+
960
+ The paths are the one part of this that is not authored but **machine-derived**,
961
+ and it is worth knowing where that can go further than you meant. A spec under
962
+ the project root reports a project-relative name and nothing more. A spec run
963
+ from *outside* it has no relative name, so its real location is what travels —
964
+ `/home/build-agent-07/…`, `/var/lib/jenkins/workspace/acme-payments-nightly/…` —
965
+ in `spec_file_path`, in `file_path`, and in `id`, which is that same path plus a
966
+ position. Ordinary suites never hit this; a spec vendored outside the tree, a
967
+ shard splitter that expands its arguments to absolute paths, or an IDE runner
968
+ will. That discloses a build machine's directory layout, which is a different
969
+ category from prose, so it is named here rather than folded into the paragraph
970
+ above.
971
+
972
+ SpecGuard is built on that and cannot be built without it. The product answers
973
+ "what does this suite actually cover, and where are the gaps" — a question whose
974
+ entire input is what your tests say they cover. A mode that shipped anonymised
975
+ coordinates would not be a lighter SpecGuard; it would be a SpecGuard that
976
+ cannot answer anything. So there is no opt-out, no field-level redaction and no
977
+ name-scrubbing switch, and none is planned. This is a deliberate product
978
+ decision, stated here so you can make yours.
979
+
980
+ ### If this cannot leave your perimeter, run SpecGuard inside it
981
+
982
+ Self-hosting is the supported answer, and it needs no code change — point
983
+ `SPECGUARD_ENDPOINT` at your own deployment and every byte described above goes
984
+ there instead:
985
+
986
+ ```bash
987
+ export SPECGUARD_ENDPOINT=https://specguard.internal.example.com
988
+ ```
989
+
990
+ ### What is never collected
991
+
992
+ - **No source code.** Not your application's, and not your tests' — no example
993
+ body, no `let`, no fixture, no diff of any of it.
994
+ - **No failure messages and no backtraces.** A failing example contributes the
995
+ string `failed` and nothing else; the exception, its message and its stack
996
+ stay on your machine.
997
+ - **No test output.** Nothing your suite printed to stdout or stderr, and
998
+ nothing any other formatter wrote, is read or forwarded.
999
+ - **No environment.** SpecGuard's own code reads a fixed list of variables and
1000
+ no others: the ones named in the envelope table above, which fill
1001
+ `commit_sha`, `branch`, `ci_run_id` and `shard_id`; plus four that configure
1002
+ the gem itself rather than describing your suite — `SPECGUARD_ENDPOINT` (where
1003
+ to send the run), `SPECGUARD_OUTPUT_PATH` (where to write the replay queue —
1004
+ runs the endpoint refused or could not be reached for), `SPECGUARD_LOCAL_OUTPUT_PATH`
1005
+ (where to write the local development record when there is no key),
1006
+ `SPECGUARD_TIMEOUT` (how long to wait), and
1007
+ `SPECGUARD_API_KEY`, which leaves the machine only as the bearer token
1008
+ described above. The other three are never sent, and there is no general
1009
+ environment capture to be caught by. (The linter is a separate program that
1010
+ sends nothing at all; it reads two variables of its own —
1011
+ `SPECGUARD_VALIDATE_INTENT` to name a validator binary, and
1012
+ `SPECGUARD_CACHE_DIR` to redirect its download cache — documented above. On a
1013
+ first run with neither cache nor variable set it fetches the validator
1014
+ binary from open-test-intent's GitHub release, over HTTPS, and nothing else.)
1015
+ - **One exception, and it is about the route rather than the contents: your
1016
+ proxy settings are read.** Sending the run goes through Ruby's `Net::HTTP`,
1017
+ which resolves a proxy from the environment the way every Ruby HTTP client
1018
+ does — so if your network requires a proxy, the run takes it, without
1019
+ SpecGuard being told about it. `http_proxy` (or `HTTP_PROXY`) is the variable
1020
+ that does it, **including for an `https://` endpoint**: `Net::HTTP` resolves
1021
+ the proxy against an `http` URL whatever the transport, which means setting
1022
+ only `https_proxy` will *not* proxy your run. `no_proxy` (or `NO_PROXY`)
1023
+ suppresses it per host. In a CGI environment (`REQUEST_METHOD` set)
1024
+ `CGI_HTTP_PROXY` is read instead and the uppercase spelling is ignored. None
1025
+ of these is ever transmitted, and none of them changes a byte of what is sent
1026
+ — they decide only **where it goes**, which is worth knowing alongside
1027
+ *"If this cannot leave your perimeter, run SpecGuard inside it"* above, since
1028
+ `SPECGUARD_ENDPOINT` is not the only thing that determines the destination.
1029
+
1030
+ ---
1031
+
1032
+ <p align="center">
1033
+ <a href="https://yatfa.com">
1034
+ <img src="assets/built-with-yatfa.png" alt="Built with yatfa — a team of AI agents that plans, builds &amp; ships software." width="100%">
1035
+ </a>
1036
+ </p>