specshield 3.5.0 → 3.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +55 -0
- package/README.md +95 -2
- package/package.json +1 -1
- package/src/cli.js +4 -0
- package/src/commands/gate.js +13 -0
- package/src/commands/prove.js +149 -0
- package/src/commands/replay.js +195 -0
- package/src/core/configLoader.js +5 -2
- package/src/core/gate/runGate.js +10 -1
- package/src/core/prove/contract.js +241 -0
- package/src/core/prove/execute.js +135 -0
- package/src/core/prove/http.js +104 -0
- package/src/core/prove/index.js +84 -0
- package/src/core/prove/instance.js +400 -0
- package/src/core/prove/relate.js +23 -0
- package/src/core/prove/render.js +169 -0
- package/src/core/prove/repair.js +109 -0
- package/src/core/prove/schema.js +261 -0
- package/src/core/prove/subschema.js +257 -0
- package/src/core/prove/witness.js +702 -0
- package/src/core/replay/runReplay.js +169 -0
- package/src/core/replay/shape.js +100 -0
- package/src/core/replay/traffic.js +168 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,60 @@
|
|
|
1
1
|
# SpecShield CLI changelog
|
|
2
2
|
|
|
3
|
+
## Unreleased — proof, not opinion: `specshield prove` and `specshield replay`
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- **`specshield prove <base> <target>`**: for every breaking change, the exact request that
|
|
8
|
+
worked before and fails now. For a response change, it gives a body the new spec allows that
|
|
9
|
+
clients of the old spec reject.
|
|
10
|
+
- **How a witness is found.** Candidate values for the changed location are enums, bounds just
|
|
11
|
+
past the new limits, pattern matches, every `oneOf` branch, and absence. Each is validated
|
|
12
|
+
with ajv against both full specs.
|
|
13
|
+
- **Isolation.** The rest of the witness is repaired to satisfy both specs. Each request
|
|
14
|
+
witness carries a control that differs only at the change, which proves the cause.
|
|
15
|
+
- **Results.** Each change is one of: proven, example (real but not rejectable: a removed
|
|
16
|
+
optional field), compatible (with a structural reason), or no proof (with the reason why).
|
|
17
|
+
- **A sound subschema check** behind the *compatible* verdicts. It recognizes added optional
|
|
18
|
+
properties, extended enums, relaxed bounds, widened types and numeric formats, and `oneOf`
|
|
19
|
+
branches the engine reports as changed only because they were reordered.
|
|
20
|
+
- Recursive schemas are compared coinductively.
|
|
21
|
+
- It assumes a closed world (clients send only documented fields).
|
|
22
|
+
- **`prove --against <url>`** sends every witness to a running build and reports what it does.
|
|
23
|
+
- *confirmed*: the witness is rejected while its control is accepted.
|
|
24
|
+
- *not reproduced*: the build still accepts it, so the spec and the code disagree.
|
|
25
|
+
- *inconclusive*: credentials or test data, not the change.
|
|
26
|
+
- Writes are sent only with `--include-mutating`; remote hosts only with `--allow-remote`.
|
|
27
|
+
- **`specshield replay`** replays a seeded sample of real traffic against the new build.
|
|
28
|
+
- **Sources:** HAR files, or access logs (nginx/Apache, ALB, JSON lines).
|
|
29
|
+
- **Sampling:** stratified per operation.
|
|
30
|
+
- **References:** the old build side by side (`--baseline`), the recorded responses, or the
|
|
31
|
+
old spec as the oracle (`--old-spec`, which works from status-only access logs).
|
|
32
|
+
- **Verdicts:** broken (worked before, fails now), changed (lost fields or changed types), or
|
|
33
|
+
inconclusive (auth, throttling, a 404 that missing test data could explain).
|
|
34
|
+
- **Safety:** captured credentials are never replayed, query values are hidden unless
|
|
35
|
+
`--show-values`, and writes are never rebuilt from body-less logs.
|
|
36
|
+
- **`gate --prove`** (also `gate.prove: true` in `.specshield.yml`) attaches the proof to every
|
|
37
|
+
blocking finding, in the terminal and as a collapsible *Proof* section in the GitHub job summary.
|
|
38
|
+
- Form-encoded request bodies (OpenAPI 3 `application/x-www-form-urlencoded`, Swagger 2.0
|
|
39
|
+
`formData`) are proven on their parsed values and sent bracket-encoded.
|
|
40
|
+
|
|
41
|
+
### Tested on real data
|
|
42
|
+
|
|
43
|
+
- **GitHub REST API**, Apr → Oct 2026, 112 findings: 35 proven, 11 by example, 41 compatible
|
|
44
|
+
(reordered `oneOf` rule branches, widened numeric formats, narrowed responses), 25 without
|
|
45
|
+
proof (removed status codes and shared schemas). 1.1 s.
|
|
46
|
+
- **Stripe**, Mar → Sep 2026, 112 findings: 89 proven, 19 by example, 4 without proof. Its
|
|
47
|
+
1,972 per-endpoint changes ran in 16 s, after two fixes:
|
|
48
|
+
- spec components are now registered with ajv once, instead of recompiled per schema;
|
|
49
|
+
- the subschema check now compares lazily with memoization, instead of inlining.
|
|
50
|
+
- **Kubernetes** (Swagger 2.0), Jul → Oct 2026, 103 findings: 58 proven. 384 per-endpoint
|
|
51
|
+
"field became required" findings on responses are shown compatible.
|
|
52
|
+
- Every proof on all three was isolated to its own change.
|
|
53
|
+
- **A Spring Boot service**, old and new builds side by side: all 6 proofs confirmed on the new
|
|
54
|
+
build, and all 6 *not reproduced* on a build whose code hadn't changed. Replay from a HAR,
|
|
55
|
+
and from an nginx log with `--old-spec`, found the removed field, the type change and the
|
|
56
|
+
rejected writes.
|
|
57
|
+
|
|
3
58
|
## 3.5.0 — 2026-10-29 — gate, fix, safe removal
|
|
4
59
|
|
|
5
60
|
### A gate teams can leave blocking: `specshield gate`
|
package/README.md
CHANGED
|
@@ -42,7 +42,7 @@ What the CLI does:
|
|
|
42
42
|
- [vs. Alternatives](#vs-alternatives) · [Pricing](#pricing)
|
|
43
43
|
|
|
44
44
|
**Commands — local & hosted compare**
|
|
45
|
-
- [Local Compare](#local-compare) · [**Who does a change break? (usage-aware compare)**](#who-does-a-change-break--usage-aware-compare) · [**Safe removal: deprecate → drain → remove**](#safe-removal-deprecate--watch-traffic-drain--remove) · [**Break → fix: `specshield fix`**](#break--fix-generate-the-non-breaking-version) · [**CI gate: `specshield gate`**](#a-gate-you-can-leave-blocking-specshield-gate) · [Remote Compare](#remote-compare) · [Comparison History](#comparison-history) · [Share a Comparison](#share-a-comparison) · [GitHub App PR Checks](#github-app--pr-checks)
|
|
45
|
+
- [Local Compare](#local-compare) · [**Who does a change break? (usage-aware compare)**](#who-does-a-change-break--usage-aware-compare) · [**Safe removal: deprecate → drain → remove**](#safe-removal-deprecate--watch-traffic-drain--remove) · [**Break → fix: `specshield fix`**](#break--fix-generate-the-non-breaking-version) · [**CI gate: `specshield gate`**](#a-gate-you-can-leave-blocking-specshield-gate) · [**Proof: `prove` & `replay`**](#proof-not-opinion-specshield-prove-and-specshield-replay) · [Remote Compare](#remote-compare) · [Comparison History](#comparison-history) · [Share a Comparison](#share-a-comparison) · [GitHub App PR Checks](#github-app--pr-checks)
|
|
46
46
|
|
|
47
47
|
**Governance — lint & gate**
|
|
48
48
|
- [`specshield govern`](#governance--lint-a-spec-and-gate-the-pr)
|
|
@@ -578,6 +578,99 @@ In GitHub Actions the gate also writes a job summary and annotations (`--no-summ
|
|
|
578
578
|
|
|
579
579
|
---
|
|
580
580
|
|
|
581
|
+
## Proof, not opinion: `specshield prove` and `specshield replay`
|
|
582
|
+
|
|
583
|
+
A diff says *"breaking change"*. A reviewer asks *"does it actually break anything?"* — and a gate that can't answer gets overruled. `prove` answers with the exact request that worked before and fails now; `replay` answers with real traffic.
|
|
584
|
+
|
|
585
|
+
```bash
|
|
586
|
+
specshield prove openapi-v1.yaml openapi-v2.yaml
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
```
|
|
590
|
+
PROOF Field "requestBody.channel" was added to POST /payments (required)
|
|
591
|
+
worked before POST /payments {"amount":1.5,"currency":"USD","customerId":"cus_1"}
|
|
592
|
+
fails now body is missing required field "channel"
|
|
593
|
+
reproduce curl -X POST "$BASE_URL/payments" -H 'Content-Type: application/json' -d '{"amount":1.5,…}'
|
|
594
|
+
|
|
595
|
+
PROOF Response field "amount" changed type — 3 endpoints: GET /payments, POST /payments, GET /payments/{paymentId}
|
|
596
|
+
request GET /payments/PAY-1
|
|
597
|
+
new spec allows {"paymentId":"abc","amount":"abc","currency":"abc","status":"abc"} (200)
|
|
598
|
+
old clients reject it response.amount must be number
|
|
599
|
+
|
|
600
|
+
COMPATIBLE Field "responses.200.message" became required in PATCH /budgets/{id}
|
|
601
|
+
Everything the new schema lets the server send here, the old schema accepted too (checked schema against schema), so old clients can't break on this change.
|
|
602
|
+
```
|
|
603
|
+
|
|
604
|
+
**How a proof is made.** For each breaking change, `prove` generates values the old contract accepts at the changed location — enum values, boundaries just past the new limits, strings matching the old pattern, every `oneOf` branch, leaving the field out — builds a whole request around each, and validates it with ajv against **both** full specs. A *request* witness is accepted by the old spec and rejected by the new one; a *response* witness is a body the new spec lets the server send that a client built on the old spec rejects.
|
|
605
|
+
|
|
606
|
+
Two rules keep proofs honest:
|
|
607
|
+
|
|
608
|
+
- **Isolation.** Every other part of the witness is repaired until both specs accept it, so it fails for this change alone — not for the three other changes in the same release. Where no value can satisfy both (a field that went from integer to string next door), the conflict is listed separately as *also*.
|
|
609
|
+
- **A control.** Each request witness ships with its control: the same request with only the changed value swapped for one the new spec accepts. When the control passes, the change is provably the cause.
|
|
610
|
+
|
|
611
|
+
Every change gets one of four results:
|
|
612
|
+
|
|
613
|
+
| Result | Meaning |
|
|
614
|
+
|---|---|
|
|
615
|
+
| **PROOF** | A witness, validated against both specs as above |
|
|
616
|
+
| **EXAMPLE** | The change is real but schemas can't reject anything — a removed optional response field, a parameter the server will now ignore. Shown as a concrete before/after |
|
|
617
|
+
| **COMPATIBLE** | No witness exists, and a structural argument says why: the new schema accepts everything the old one did (a sound subschema check), only a numeric format changed, `oneOf` branches were reordered, the change only narrows what a server sends |
|
|
618
|
+
| **NO PROOF** | No witness found; the reason says why (a removed status code has no payload to show) |
|
|
619
|
+
|
|
620
|
+
**Against a running build.** `--against http://localhost:8080` sends every witness to the new build and reports what it does: *confirmed* (rejected, while the control got a 2xx), *not reproduced* (the build still accepts it — the spec and the code disagree), *inconclusive* (both rejected: credentials or test data, not the change). Only `GET`/`HEAD`/`OPTIONS` are sent unless you pass `--include-mutating`; only local and private hosts are allowed unless you pass `--allow-remote`; `--header "Authorization: Bearer $TOKEN"` adds credentials.
|
|
621
|
+
|
|
622
|
+
```bash
|
|
623
|
+
specshield prove v1.yaml v2.yaml --against http://localhost:8080 --include-mutating
|
|
624
|
+
# On http://localhost:8080: 6 confirmed
|
|
625
|
+
# … on the build ✔ confirmed — HTTP 400: rejected, while the same request valid under the new spec got HTTP 201
|
|
626
|
+
```
|
|
627
|
+
|
|
628
|
+
**In the gate.** `specshield gate --prove` (or `gate.prove: true`) attaches the proof to every blocking change — in the terminal, and as a collapsible *Proof* section with the request, the reason and a `curl` command in the GitHub job summary.
|
|
629
|
+
|
|
630
|
+
### Replay real traffic: `specshield replay`
|
|
631
|
+
|
|
632
|
+
Synthetic witnesses show what *can* break; replay shows what *does*, for the requests your clients actually send.
|
|
633
|
+
|
|
634
|
+
```bash
|
|
635
|
+
# Side by side: the old build and the new one, same data — the strongest comparison
|
|
636
|
+
specshield replay --from traffic.har --against http://localhost:8082 --baseline http://localhost:8081 --spec v2.yaml
|
|
637
|
+
|
|
638
|
+
# Access logs (nginx/Apache, ALB, JSON lines; .gz ok) record no bodies: use the old spec as the oracle
|
|
639
|
+
specshield replay --from access.log --against http://localhost:8082 --old-spec v1.yaml
|
|
640
|
+
```
|
|
641
|
+
|
|
642
|
+
```
|
|
643
|
+
✖ 10 requests that worked before break on the new build
|
|
644
|
+
✔ 1 ok ✖ 2 broken ▲ 8 changed
|
|
645
|
+
|
|
646
|
+
POST /payments 2 replayed · 2 broken
|
|
647
|
+
✖ POST /payments
|
|
648
|
+
worked before (HTTP 201), fails now (HTTP 400)
|
|
649
|
+
GET /payments/{paymentId} 5 replayed · 4 changed
|
|
650
|
+
▲ GET /payments/PAY-1
|
|
651
|
+
missing receiptUrl; amount number → string
|
|
652
|
+
```
|
|
653
|
+
|
|
654
|
+
- **Sampling.** Up to `--per-operation` requests (default 5) from every operation, `--sample` (default 200) in total, taken round-robin so rare operations survive the cut. Reservoir sampling with a `--seed`: the same seed replays the same requests.
|
|
655
|
+
- **What is compared.** With `--baseline <old build>`, each request goes to both builds — only the code differs. Without it, the recorded HAR response is the reference, and a `404` is *inconclusive* (missing test data explains it as well as a removed route). Response shapes are compared, not values: a field the old response had and the new one lacks, or a type that changed, is *changed*; different data is not. `--old-spec` checks every response against the old contract; `--spec` groups traffic by operation, spots removed routes and flags responses that don't match the new spec.
|
|
656
|
+
- **Safe by default.** Writes are replayed only with `--include-mutating`, and never from access logs (they don't record bodies). Captured credentials — `Authorization`, cookies, anything named like a key or token — are never replayed; pass your own with `--header`. Query values are hidden in the report unless `--show-values`. Only local and private hosts without `--allow-remote`.
|
|
657
|
+
|
|
658
|
+
Exit codes for both: `1` when something breaks (`prove`: breaking changes, as `compare`; `replay`: a request that worked before fails now), `0` otherwise, `2` on errors. `--json` prints everything, including each witness, its control and, with `--against`, what the build did.
|
|
659
|
+
|
|
660
|
+
**Tested on real data.**
|
|
661
|
+
|
|
662
|
+
| Spec pair | Findings | Proven | Example | Compatible | No proof | Time |
|
|
663
|
+
|---|---|---|---|---|---|---|
|
|
664
|
+
| GitHub REST API, Apr → Oct 2026 | 112 | 35 | 11 | 41 | 25 | 1.1 s |
|
|
665
|
+
| Stripe API, Mar → Sep 2026 (form bodies) | 112 | 89 | 19 | 0 | 4 | 16 s |
|
|
666
|
+
| Kubernetes (Swagger 2.0), Jul → Oct 2026 | 103 | 58 | 0 | 0 | 45 | 2.6 s |
|
|
667
|
+
|
|
668
|
+
Every proof was isolated to its own change. The *compatible* findings on GitHub are rule branches GitHub reordered inside a `oneOf` (the engine compares branches by position), numeric formats that only widened, and response fields that became required. *No proof* is almost entirely removed status codes and removed shared schemas, which are proven through the endpoints that used them. On Kubernetes, 384 of the per-endpoint "field became required" findings sit on responses and are shown compatible, while the same fields on requests are proven. Against a Spring Boot service whose old and new builds ran side by side, all 6 proofs were confirmed on the new build. All 6 were *not reproduced* on a build whose code hadn't changed.
|
|
669
|
+
|
|
670
|
+
**Limits.** Witnesses are built for JSON and form bodies. The subschema check assumes a closed world — clients send only fields the spec documents — which is what every compatibility check assumes; it is sound (a *compatible* verdict holds for every instance) but incomplete: regex inclusion, `not` and `if/then` are reported as *no proof* rather than guessed. Pattern strings are generated for the regex features API specs use; lookarounds and backreferences are skipped.
|
|
671
|
+
|
|
672
|
+
---
|
|
673
|
+
|
|
581
674
|
## Remote Compare
|
|
582
675
|
|
|
583
676
|
Sends your specs to SpecShield's hosted diff engine. Useful for keeping the
|
|
@@ -1706,7 +1799,7 @@ Per-subcommand flags:
|
|
|
1706
1799
|
| Code | Meaning |
|
|
1707
1800
|
|---|---|
|
|
1708
1801
|
| `0` | Clean — no breaking changes / deployable / all probes pass |
|
|
1709
|
-
| `1` | Breaking changes found / not deployable / one or more probes failed |
|
|
1802
|
+
| `1` | Breaking changes found / not deployable / one or more probes failed / a replayed request that worked before fails now |
|
|
1710
1803
|
| `2` | Config error, missing token, or runtime error |
|
|
1711
1804
|
|
|
1712
1805
|
---
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "specshield",
|
|
3
|
-
"version": "3.5.
|
|
3
|
+
"version": "3.5.1",
|
|
4
4
|
"description": "Contract compatibility testing for APIs — catch breaking OpenAPI changes before they reach your consumers, with can-i-deploy deploy gating and GitHub PR checks. (a.k.a. bidirectional contract testing.)",
|
|
5
5
|
"main": "src/cli.js",
|
|
6
6
|
"homepage": "https://specshield.io",
|
package/src/cli.js
CHANGED
|
@@ -16,6 +16,8 @@ const deprecationsCommand = require('./commands/deprecations');
|
|
|
16
16
|
const fixCommand = require('./commands/fix');
|
|
17
17
|
const gateCommand = require('./commands/gate');
|
|
18
18
|
const waiversCommand = require('./commands/waivers');
|
|
19
|
+
const proveCommand = require('./commands/prove');
|
|
20
|
+
const replayCommand = require('./commands/replay');
|
|
19
21
|
|
|
20
22
|
const program = new Command();
|
|
21
23
|
|
|
@@ -42,6 +44,8 @@ program.addCommand(deprecationsCommand);
|
|
|
42
44
|
program.addCommand(fixCommand);
|
|
43
45
|
program.addCommand(gateCommand);
|
|
44
46
|
program.addCommand(waiversCommand);
|
|
47
|
+
program.addCommand(proveCommand);
|
|
48
|
+
program.addCommand(replayCommand);
|
|
45
49
|
|
|
46
50
|
program.parseAsync(process.argv).catch((err) => {
|
|
47
51
|
const logger = require('./utils/logger');
|
package/src/commands/gate.js
CHANGED
|
@@ -10,6 +10,7 @@ const { usageOptions } = require('../core/usage/applyUsage');
|
|
|
10
10
|
const W = require('../core/gate/waivers');
|
|
11
11
|
const G = require('../core/gate/git');
|
|
12
12
|
const { plural } = require('../core/usage/format');
|
|
13
|
+
const R = require('../core/prove/render');
|
|
13
14
|
const { uploadGateRun } = require('../api/bdctClient');
|
|
14
15
|
const { getStoredApiKey } = require('../config/localConfig');
|
|
15
16
|
const logger = require('../utils/logger');
|
|
@@ -25,6 +26,7 @@ const gate = new Command('gate')
|
|
|
25
26
|
.option('--max-waiver-days <n>', `Longest a waiver may run before it must be renewed (default: ${W.DEFAULT_MAX_DAYS})`)
|
|
26
27
|
.option('--no-fetch', 'Never run git fetch, even when the baseline commit is missing from a shallow clone')
|
|
27
28
|
.option('--usage <file>', 'usage.json from "specshield usage import" — breaking changes no client uses stop blocking')
|
|
29
|
+
.option('--prove', 'Attach a proof to every breaking change: the request that worked at the baseline and fails now')
|
|
28
30
|
.option('--no-summary', 'Skip the GitHub Actions job summary and annotations')
|
|
29
31
|
.option('--upload', 'Record this run on your SpecShield dashboard, for the shadow-mode graduation report (Team plan)')
|
|
30
32
|
.option('--org <key>', 'Org key for --upload (default: gate.org or bdct.org in .specshield.yml)')
|
|
@@ -50,6 +52,7 @@ const gate = new Command('gate')
|
|
|
50
52
|
maxWaiverDays,
|
|
51
53
|
ignore: config.ignore || [],
|
|
52
54
|
usage: usage.usage ? usage : null,
|
|
55
|
+
prove: Boolean(opts.prove || g.prove),
|
|
53
56
|
});
|
|
54
57
|
|
|
55
58
|
if (opts.upload || g.upload) r.upload = await upload(r, { ...opts, org: opts.org || g.org });
|
|
@@ -180,6 +183,7 @@ function render(r) {
|
|
|
180
183
|
if (c.expiredWaiver) {
|
|
181
184
|
lines.push(chalk.yellow(` waiver expired ${c.expiredWaiver.expires} (owner ${c.expiredWaiver.owner}) — renew it or fix the change`));
|
|
182
185
|
}
|
|
186
|
+
if (c.proof) lines.push(...R.lines(c.proof, { pad: ' ' }));
|
|
183
187
|
lines.push(chalk.gray(` intended? ${waiveCommand(c, r.waivers.file)}`));
|
|
184
188
|
}
|
|
185
189
|
}
|
|
@@ -278,6 +282,15 @@ function markdown(r) {
|
|
|
278
282
|
const { where, what } = label(c);
|
|
279
283
|
md.push(`| ${cell(what)} | ${cell(where || '—')} | ${c.expiredWaiver ? `waiver expired ${c.expiredWaiver.expires}` : ''} |`);
|
|
280
284
|
}
|
|
285
|
+
const proven = r.blocking.filter((c) => c.proof && (c.proof.status === 'proven' || c.proof.status === 'example'));
|
|
286
|
+
if (proven.length) {
|
|
287
|
+
md.push('');
|
|
288
|
+
md.push(`<details><summary>Proof for ${plural(proven.length, 'change')}</summary>`);
|
|
289
|
+
for (const c of proven) {
|
|
290
|
+
md.push('', `#### ${cell(label(c).what)}`, '', R.markdown(c.proof));
|
|
291
|
+
}
|
|
292
|
+
md.push('', '</details>');
|
|
293
|
+
}
|
|
281
294
|
md.push('');
|
|
282
295
|
md.push(`Intended? Add a waiver with an owner, a reason and an expiry: \`${waiveCommand(r.blocking[0], r.waivers.file)}\``);
|
|
283
296
|
}
|
|
@@ -0,0 +1,149 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
const { Command } = require('commander');
|
|
4
|
+
const chalk = require('chalk');
|
|
5
|
+
const { loadSpec } = require('../core/loadSpec');
|
|
6
|
+
const { parseSpec } = require('../core/parseSpec');
|
|
7
|
+
const { prove } = require('../core/prove');
|
|
8
|
+
const { confirmAll } = require('../core/prove/execute');
|
|
9
|
+
const R = require('../core/prove/render');
|
|
10
|
+
const H = require('../core/prove/http');
|
|
11
|
+
const { plural } = require('../core/usage/format');
|
|
12
|
+
const logger = require('../utils/logger');
|
|
13
|
+
const exitAfterFlush = require('../utils/exitAfterFlush');
|
|
14
|
+
|
|
15
|
+
const prove_ = new Command('prove')
|
|
16
|
+
.description('Proof, not opinion: for every breaking change, the exact request that worked before and fails now — checked against both specs, and optionally against a running build')
|
|
17
|
+
.argument('<base>', 'The spec your clients use today')
|
|
18
|
+
.argument('<target>', 'The new spec')
|
|
19
|
+
.option('--against <url>', 'Send each witness to a running build of the new API and report what it does')
|
|
20
|
+
.option('--header <header...>', 'Header for --against requests, e.g. "Authorization: Bearer $TOKEN" (repeatable)')
|
|
21
|
+
.option('--include-mutating', 'With --against, also send POST/PUT/PATCH/DELETE witnesses (they may change data)')
|
|
22
|
+
.option('--allow-remote', 'Allow --against to target a host that is not local or private')
|
|
23
|
+
.option('--timeout <ms>', 'Per-request timeout for --against', '10000')
|
|
24
|
+
.option('--base-url <url>', 'Base URL to print in the curl commands (default: $BASE_URL, or --against)')
|
|
25
|
+
.option('--warnings', 'Also prove constraint warnings (tightened min/max/length/pattern), which never fail the build')
|
|
26
|
+
.option('--json', 'Output machine-readable JSON')
|
|
27
|
+
.action(async (basePath, targetPath, opts) => {
|
|
28
|
+
try {
|
|
29
|
+
const baseDoc = parseSpec(await loadSpec(basePath), basePath);
|
|
30
|
+
const targetDoc = parseSpec(await loadSpec(targetPath), targetPath);
|
|
31
|
+
const r = prove(baseDoc, targetDoc);
|
|
32
|
+
const shown = opts.warnings ? [...r.breaking, ...r.warnings] : r.breaking;
|
|
33
|
+
|
|
34
|
+
let against = null;
|
|
35
|
+
if (opts.against) {
|
|
36
|
+
against = H.targetUrl(opts.against, { allowRemote: opts.allowRemote });
|
|
37
|
+
const items = shown.filter((c) => c.proof).map((c) => ({ diff: c.proof.diff, witness: c.proof }));
|
|
38
|
+
await confirmAll(items, {
|
|
39
|
+
baseDoc, targetDoc, against,
|
|
40
|
+
headers: H.parseHeaders(opts.header),
|
|
41
|
+
includeMutating: opts.includeMutating,
|
|
42
|
+
timeoutMs: Number(opts.timeout) || 10000,
|
|
43
|
+
});
|
|
44
|
+
const sent = items.filter((i) => i.witness.run.outcome !== 'skipped');
|
|
45
|
+
if (sent.length && sent.every((i) => i.witness.run.outcome === 'error')) {
|
|
46
|
+
throw new Error(`Can't reach ${against}: ${sent[0].witness.run.detail}`);
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
if (opts.json) {
|
|
51
|
+
process.stdout.write(JSON.stringify(toJson(r, { baseDoc, targetDoc, against, warnings: opts.warnings }), null, 2) + '\n');
|
|
52
|
+
} else {
|
|
53
|
+
process.stdout.write(render(r, { baseDoc, targetDoc, against, base: opts.baseUrl || against, warnings: opts.warnings }) + '\n');
|
|
54
|
+
}
|
|
55
|
+
exitAfterFlush(r.breaking.length > 0 ? 1 : 0);
|
|
56
|
+
} catch (err) {
|
|
57
|
+
logger.error(`Error: ${err.message}`);
|
|
58
|
+
process.exit(2);
|
|
59
|
+
}
|
|
60
|
+
});
|
|
61
|
+
|
|
62
|
+
function title(doc) {
|
|
63
|
+
const t = (doc.info && doc.info.title) || 'API';
|
|
64
|
+
return { title: t, version: doc.info && doc.info.version ? String(doc.info.version) : null };
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
function toJson(r, { baseDoc, targetDoc, against, warnings }) {
|
|
68
|
+
const item = (c) => {
|
|
69
|
+
const { proof, ...change } = c;
|
|
70
|
+
return { ...change, proof: proof ? strip(proof) : null };
|
|
71
|
+
};
|
|
72
|
+
return {
|
|
73
|
+
base: title(baseDoc),
|
|
74
|
+
target: title(targetDoc),
|
|
75
|
+
against,
|
|
76
|
+
summary: { ...r.summary, ...(against ? { onTheBuild: runCounts(r.breaking) } : {}) },
|
|
77
|
+
breakingChanges: r.breaking.map(item),
|
|
78
|
+
warnings: warnings ? r.warnings.map(item) : r.warnings.map(({ proof, ...c }) => c),
|
|
79
|
+
nonBreaking: r.nonBreaking,
|
|
80
|
+
};
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
function strip(p) {
|
|
84
|
+
const { diff, ...rest } = p;
|
|
85
|
+
return { ...rest, at: diff };
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
function runCounts(changes) {
|
|
89
|
+
const out = {};
|
|
90
|
+
for (const c of changes) if (c.proof && c.proof.run) out[c.proof.run.outcome] = (out[c.proof.run.outcome] || 0) + 1;
|
|
91
|
+
return out;
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
function render(r, { baseDoc, targetDoc, against, base, warnings }) {
|
|
95
|
+
const b = title(baseDoc);
|
|
96
|
+
const t = title(targetDoc);
|
|
97
|
+
const lines = [''];
|
|
98
|
+
lines.push(chalk.bold(` SpecShield™ Prove — ${t.title}${b.version ? ` ${b.version}` : ''} →${t.version ? ` ${t.version}` : ''}`));
|
|
99
|
+
lines.push(chalk.gray(' ─────────────────────────────────────────'));
|
|
100
|
+
if (r.breaking.length === 0) {
|
|
101
|
+
lines.push(chalk.green(' ✔ No breaking changes — nothing to prove.'));
|
|
102
|
+
} else {
|
|
103
|
+
const s = r.summary;
|
|
104
|
+
const parts = [`${chalk.green(`${s.proven} proven`)} with a failing request`];
|
|
105
|
+
if (s.example) parts.push(chalk.yellow(`${s.example} shown by example`));
|
|
106
|
+
if (s.compatible) parts.push(chalk.cyan(`${s.compatible} compatible on their own`));
|
|
107
|
+
if (s.unproven) parts.push(chalk.gray(`${s.unproven} without proof`));
|
|
108
|
+
lines.push(` ${plural(s.total, 'breaking change')} · ${parts.join(' · ')}`);
|
|
109
|
+
lines.push(chalk.gray(' Every proof is checked against both specs: the old one accepts it, the new one rejects it.'));
|
|
110
|
+
if (against) {
|
|
111
|
+
const counts = runCounts(r.breaking);
|
|
112
|
+
lines.push(` On ${against}: ` + Object.entries(counts).map(([k, n]) => `${n} ${k}`).join(' · '));
|
|
113
|
+
}
|
|
114
|
+
for (const c of r.breaking) block(lines, c, base);
|
|
115
|
+
}
|
|
116
|
+
const provable = r.warnings.filter((c) => c.proof && c.proof.status === 'proven');
|
|
117
|
+
if (warnings && r.warnings.length) {
|
|
118
|
+
lines.push('');
|
|
119
|
+
lines.push(chalk.yellow.bold(` Warnings (${r.warnings.length}) — reported, never fail the build`));
|
|
120
|
+
for (const c of r.warnings) block(lines, c, base);
|
|
121
|
+
} else if (provable.length) {
|
|
122
|
+
lines.push('');
|
|
123
|
+
lines.push(chalk.gray(` ${plural(provable.length, 'constraint warning')} also ${provable.length === 1 ? 'has' : 'have'} a failing request — see them with --warnings.`));
|
|
124
|
+
}
|
|
125
|
+
if (!base && r.breaking.some((c) => c.proof && c.proof.request)) {
|
|
126
|
+
lines.push('');
|
|
127
|
+
lines.push(chalk.gray(' Set BASE_URL to run the curl commands, e.g. export BASE_URL=http://localhost:8080'));
|
|
128
|
+
}
|
|
129
|
+
lines.push('');
|
|
130
|
+
return lines.join('\n');
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
/** A merged change's description lists every endpoint (with repeats); name a few, count them all. */
|
|
134
|
+
function headline(c) {
|
|
135
|
+
const eps = [...new Set(c.affectedEndpoints || [])];
|
|
136
|
+
if (eps.length < 2) return c.description;
|
|
137
|
+
const head = c.description.replace(/ \(affects \d+ endpoints?:.*\)$/, '');
|
|
138
|
+
return `${head} — ${eps.length} endpoints: ${eps.slice(0, 3).join(', ')}${eps.length > 3 ? ` and ${eps.length - 3} more` : ''}`;
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
function block(lines, c, base) {
|
|
142
|
+
lines.push('');
|
|
143
|
+
lines.push(` ${R.badge(c.proof)} ${headline(c)}`);
|
|
144
|
+
// A merged change already names its endpoints in the description.
|
|
145
|
+
if (c.method && !(c.affectedEndpoints && c.affectedEndpoints.length > 1)) lines.push(chalk.gray(` ${c.method.toUpperCase()} ${c.path}`));
|
|
146
|
+
lines.push(...R.lines(c.proof, { base }));
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
module.exports = prove_;
|
|
@@ -0,0 +1,195 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
const { Command } = require('commander');
|
|
4
|
+
const chalk = require('chalk');
|
|
5
|
+
const fs = require('fs');
|
|
6
|
+
const { loadSpec } = require('../core/loadSpec');
|
|
7
|
+
const { parseSpec } = require('../core/parseSpec');
|
|
8
|
+
const { loadTraffic } = require('../core/replay/traffic');
|
|
9
|
+
const { runReplay, summarize } = require('../core/replay/runReplay');
|
|
10
|
+
const H = require('../core/prove/http');
|
|
11
|
+
const { plural } = require('../core/usage/format');
|
|
12
|
+
const logger = require('../utils/logger');
|
|
13
|
+
const exitAfterFlush = require('../utils/exitAfterFlush');
|
|
14
|
+
|
|
15
|
+
const replay = new Command('replay')
|
|
16
|
+
.description('Replay a sample of real captured requests (HAR or access logs) against a build of the new API, and report every request that worked before and fails now')
|
|
17
|
+
.requiredOption('--against <url>', 'The new build, e.g. http://localhost:8080')
|
|
18
|
+
.requiredOption('--from <file...>', 'Captured traffic: .har files and/or access logs (nginx/Apache, ALB, JSON lines; .gz ok)')
|
|
19
|
+
.option('--baseline <url>', 'The old build: send each request there too and compare side by side (recommended)')
|
|
20
|
+
.option('--spec <file>', 'The new spec: groups traffic by operation, spots removed routes, and checks responses against it')
|
|
21
|
+
.option('--old-spec <file>', 'The old spec: every response must still satisfy it — catches shape breaks even from access logs, which record no bodies')
|
|
22
|
+
.option('--base-path <path>', 'Path prefix in the captured URLs that the spec\'s paths leave out, e.g. /v1')
|
|
23
|
+
.option('--host <host...>', 'Only replay requests captured for these hosts')
|
|
24
|
+
.option('--sample <n>', 'At most this many requests in total', '200')
|
|
25
|
+
.option('--per-operation <n>', 'At most this many requests per operation', '5')
|
|
26
|
+
.option('--seed <n>', 'Sampling seed — the same seed replays the same requests', '1')
|
|
27
|
+
.option('--include-mutating', 'Also replay POST/PUT/PATCH/DELETE (they change data on the build)')
|
|
28
|
+
.option('--allow-remote', 'Allow --against/--baseline hosts that are not local or private')
|
|
29
|
+
.option('--header <header...>', 'Header added to every request, e.g. "Authorization: Bearer $TOKEN" (captured credentials are never replayed)')
|
|
30
|
+
.option('--concurrency <n>', 'Requests in flight', '4')
|
|
31
|
+
.option('--timeout <ms>', 'Per-request timeout', '10000')
|
|
32
|
+
.option('--format <format>', 'Access log format: auto, combined, alb, json', 'auto')
|
|
33
|
+
.option('--show-values', 'Print query values in the report (hidden by default; they can be personal data)')
|
|
34
|
+
.option('--no-summary', 'Skip the GitHub Actions job summary')
|
|
35
|
+
.option('--json', 'Output machine-readable JSON')
|
|
36
|
+
.action(async (opts) => {
|
|
37
|
+
try {
|
|
38
|
+
const against = H.targetUrl(opts.against, { allowRemote: opts.allowRemote });
|
|
39
|
+
const baseline = opts.baseline ? H.targetUrl(opts.baseline, { allowRemote: opts.allowRemote, flag: '--baseline' }) : null;
|
|
40
|
+
const spec = opts.spec ? parseSpec(await loadSpec(opts.spec), opts.spec) : null;
|
|
41
|
+
const oldSpec = opts.oldSpec ? parseSpec(await loadSpec(opts.oldSpec), opts.oldSpec) : null;
|
|
42
|
+
for (const f of opts.from) if (f !== '-' && !fs.existsSync(f)) throw new Error(`Traffic file not found: ${f}`);
|
|
43
|
+
|
|
44
|
+
const traffic = await loadTraffic(opts.from, {
|
|
45
|
+
format: opts.format,
|
|
46
|
+
spec: spec || oldSpec,
|
|
47
|
+
basePath: opts.basePath,
|
|
48
|
+
perOperation: positive(opts.perOperation, '--per-operation'),
|
|
49
|
+
limit: positive(opts.sample, '--sample'),
|
|
50
|
+
seed: Number(opts.seed) || 1,
|
|
51
|
+
includeMutating: opts.includeMutating,
|
|
52
|
+
hosts: opts.host ? new Set(opts.host.map((h) => h.toLowerCase())) : null,
|
|
53
|
+
});
|
|
54
|
+
if (traffic.records.length === 0) throw new Error(emptyReason(traffic.stats, opts));
|
|
55
|
+
|
|
56
|
+
const results = await runReplay(traffic.records, {
|
|
57
|
+
against, baseline,
|
|
58
|
+
headers: H.parseHeaders(opts.header),
|
|
59
|
+
timeoutMs: Number(opts.timeout) || 10000,
|
|
60
|
+
concurrency: positive(opts.concurrency, '--concurrency'),
|
|
61
|
+
spec,
|
|
62
|
+
oldSpec,
|
|
63
|
+
basePath: opts.basePath,
|
|
64
|
+
});
|
|
65
|
+
if (results.every((r) => r.verdict === 'error')) throw new Error(`Can't reach the build: ${results[0].detail}`);
|
|
66
|
+
|
|
67
|
+
const s = summarize(results);
|
|
68
|
+
const bodies = Boolean(baseline) || traffic.records.some((r) => r.recorded.json !== undefined);
|
|
69
|
+
const report = { against, baseline, reference: baseline ? 'baseline' : 'recorded', bodies, oldSpec: Boolean(oldSpec), traffic: traffic.stats, totals: s.totals, operations: s.operations };
|
|
70
|
+
if (opts.json) process.stdout.write(JSON.stringify(toJson(report, opts), null, 2) + '\n');
|
|
71
|
+
else process.stdout.write(render(report, opts) + '\n');
|
|
72
|
+
|
|
73
|
+
if (opts.summary !== false && process.env.GITHUB_ACTIONS === 'true' && process.env.GITHUB_STEP_SUMMARY) {
|
|
74
|
+
try { fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, markdown(report, opts) + '\n'); } catch { /* best-effort */ }
|
|
75
|
+
}
|
|
76
|
+
exitAfterFlush(s.totals.broken + s.totals.changed > 0 ? 1 : 0);
|
|
77
|
+
} catch (err) {
|
|
78
|
+
logger.error(`Error: ${err.message}`);
|
|
79
|
+
process.exit(2);
|
|
80
|
+
}
|
|
81
|
+
});
|
|
82
|
+
|
|
83
|
+
function positive(v, flag) {
|
|
84
|
+
const n = Number(v);
|
|
85
|
+
if (!Number.isInteger(n) || n < 1) throw new Error(`${flag} must be a whole number ≥ 1`);
|
|
86
|
+
return n;
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
function emptyReason(st, opts) {
|
|
90
|
+
if (st.read === 0) return 'No requests found in the traffic files';
|
|
91
|
+
if (st.bodilessWrites === st.read) return `All ${st.read} logged requests change data, and access logs don't record request bodies — record a HAR to replay writes`;
|
|
92
|
+
if (st.mutatingSkipped + st.bodilessWrites === st.read) return `All ${st.read} captured requests change data (POST/PUT/PATCH/DELETE); pass --include-mutating to replay them`;
|
|
93
|
+
if (st.otherHosts) return `No requests for ${opts.host.join(', ')} (${st.otherHosts} for other hosts)`;
|
|
94
|
+
return 'Nothing to replay after filtering';
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
/** "/orders?status=open&limit=5" → "/orders?status=…&limit=…" unless --show-values. */
|
|
98
|
+
function target(rec, showValues) {
|
|
99
|
+
if (!rec.query) return rec.path;
|
|
100
|
+
if (showValues) return `${rec.path}?${rec.query}`;
|
|
101
|
+
return `${rec.path}?${rec.query.split('&').map((p) => `${p.split('=')[0]}=…`).join('&')}`;
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
function toJson(r, opts) {
|
|
105
|
+
const req = (x) => ({
|
|
106
|
+
method: x.rec.method, target: target(x.rec, opts.showValues), operation: x.rec.operation,
|
|
107
|
+
reference: x.ref, build: x.cand, verdict: x.verdict, detail: x.detail,
|
|
108
|
+
...(x.diff ? { missing: x.diff.missing.map(field), changed: x.diff.changed.map((c) => ({ ...c, path: field(c.path) })) } : {}),
|
|
109
|
+
...(x.specDrift ? { specDrift: x.specDrift } : {}),
|
|
110
|
+
...(x.oldContract ? { oldContract: x.oldContract } : {}),
|
|
111
|
+
...(x.excerpt && opts.showValues ? { response: x.excerpt } : {}),
|
|
112
|
+
});
|
|
113
|
+
return {
|
|
114
|
+
against: r.against,
|
|
115
|
+
baseline: r.baseline,
|
|
116
|
+
reference: r.reference,
|
|
117
|
+
comparedBodies: r.bodies,
|
|
118
|
+
checkedAgainstOldSpec: r.oldSpec,
|
|
119
|
+
traffic: r.traffic,
|
|
120
|
+
totals: r.totals,
|
|
121
|
+
operations: r.operations.map(({ examples, drift, ...o }) => ({ ...o, examples: examples.map(req), specDrift: drift.map(req) })),
|
|
122
|
+
};
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/** ".items[].id" → "items[].id" */
|
|
126
|
+
function field(p) {
|
|
127
|
+
return p.replace(/^\./, '') || '(root)';
|
|
128
|
+
}
|
|
129
|
+
|
|
130
|
+
const MARK = {
|
|
131
|
+
broken: chalk.red('✖'), changed: chalk.red('▲'), error: chalk.red('!'), inconclusive: chalk.yellow('?'), improved: chalk.cyan('↑'), ok: chalk.green('✔'),
|
|
132
|
+
};
|
|
133
|
+
|
|
134
|
+
function render(r, opts) {
|
|
135
|
+
const t = r.totals;
|
|
136
|
+
const lines = [''];
|
|
137
|
+
lines.push(chalk.bold(` SpecShield™ Replay — ${r.against}`));
|
|
138
|
+
lines.push(chalk.gray(' ─────────────────────────────────────────'));
|
|
139
|
+
const sources = r.traffic.files.map((f) => `${f.file} (${f.kind}, ${f.requests})`).join(', ');
|
|
140
|
+
lines.push(` ${chalk.gray('Traffic')} ${plural(r.traffic.sampled, 'request')} sampled from ${plural(r.traffic.considered, 'captured request')} across ${plural(r.traffic.operations, 'operation')}`);
|
|
141
|
+
lines.push(chalk.gray(` ${sources}`));
|
|
142
|
+
if (r.traffic.mutatingSkipped) lines.push(chalk.gray(` ${plural(r.traffic.mutatingSkipped, 'write')} not replayed (--include-mutating to replay them)`));
|
|
143
|
+
if (r.traffic.bodilessWrites) lines.push(chalk.gray(` ${plural(r.traffic.bodilessWrites, 'write')} from access logs not replayed: logs don't record request bodies`));
|
|
144
|
+
lines.push(` ${chalk.gray('Compared')} ${r.baseline ? `with the old build at ${r.baseline}, side by side` : 'with the recorded responses — use --baseline <old build> to rule out data differences'}`);
|
|
145
|
+
if (r.oldSpec) lines.push(chalk.gray(' and every response checked against the old spec'));
|
|
146
|
+
else if (!r.bodies) lines.push(chalk.yellow(' status codes only: access logs record no bodies — add --old-spec (or --baseline) to compare responses'));
|
|
147
|
+
lines.push('');
|
|
148
|
+
|
|
149
|
+
const bad = t.broken + t.changed;
|
|
150
|
+
if (bad === 0) lines.push(chalk.green(` ✔ No request that worked before fails now (${plural(results(t), 'request')}${!r.bodies && !r.oldSpec ? ', by status code' : ''})`));
|
|
151
|
+
else lines.push(chalk.red.bold(` ✖ ${plural(bad, 'request')} that worked before ${bad === 1 ? 'breaks' : 'break'} on the new build`));
|
|
152
|
+
const counts = Object.entries(t).filter(([, n]) => n > 0).map(([k, n]) => `${MARK[k]} ${n} ${k}`);
|
|
153
|
+
lines.push(` ${counts.join(' ')}`);
|
|
154
|
+
|
|
155
|
+
for (const o of r.operations) {
|
|
156
|
+
if (o.broken + o.changed + o.error + o.inconclusive === 0 && !o.drift.length) continue;
|
|
157
|
+
lines.push('');
|
|
158
|
+
const head = [o.broken && chalk.red(`${o.broken} broken`), o.changed && chalk.red(`${o.changed} changed`), o.inconclusive && chalk.yellow(`${o.inconclusive} inconclusive`), o.error && chalk.red(`${o.error} error`)].filter(Boolean).join(', ');
|
|
159
|
+
lines.push(` ${chalk.bold(o.operation)} ${chalk.gray(`${o.total} replayed`)}${head ? ` · ${head}` : ''}`);
|
|
160
|
+
for (const x of o.examples) {
|
|
161
|
+
lines.push(` ${MARK[x.verdict]} ${x.rec.method} ${target(x.rec, opts.showValues)}`);
|
|
162
|
+
lines.push(chalk.gray(` ${x.detail}`));
|
|
163
|
+
if (x.excerpt && opts.showValues) lines.push(chalk.gray(` response: ${x.excerpt}`));
|
|
164
|
+
}
|
|
165
|
+
for (const x of o.drift) lines.push(chalk.yellow(` ⚠ the build's response doesn't match the new spec: ${x.specDrift.join('; ')}`));
|
|
166
|
+
}
|
|
167
|
+
if (!opts.showValues && r.operations.some((o) => o.examples.some((x) => x.rec.query))) {
|
|
168
|
+
lines.push('');
|
|
169
|
+
lines.push(chalk.gray(' Query values are hidden (they can be personal data); --show-values prints them.'));
|
|
170
|
+
}
|
|
171
|
+
lines.push('');
|
|
172
|
+
return lines.join('\n');
|
|
173
|
+
}
|
|
174
|
+
|
|
175
|
+
function results(t) {
|
|
176
|
+
return Object.values(t).reduce((a, b) => a + b, 0);
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
function markdown(r, opts) {
|
|
180
|
+
const t = r.totals;
|
|
181
|
+
const bad = t.broken + t.changed;
|
|
182
|
+
const out = ['### SpecShield replay', ''];
|
|
183
|
+
out.push(bad ? `❌ **${plural(bad, 'request')} that worked before ${bad === 1 ? 'breaks' : 'break'} on the new build**` : `✅ No request that worked before fails now (${plural(results(t), 'request')} replayed)`);
|
|
184
|
+
out.push('', `Compared with ${r.baseline ? 'the old build, side by side' : 'the recorded responses'} · ${Object.entries(t).filter(([, n]) => n).map(([k, n]) => `${n} ${k}`).join(' · ')}`);
|
|
185
|
+
const rows = r.operations.filter((o) => o.examples.length);
|
|
186
|
+
if (rows.length) {
|
|
187
|
+
out.push('', '| Operation | Request | Result |', '|---|---|---|');
|
|
188
|
+
for (const o of rows) {
|
|
189
|
+
for (const x of o.examples) out.push(`| \`${o.operation}\` | \`${x.rec.method} ${target(x.rec, opts.showValues).replace(/\|/g, '\\|')}\` | ${x.verdict}: ${x.detail.replace(/\|/g, '\\|')} |`);
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
return out.join('\n');
|
|
193
|
+
}
|
|
194
|
+
|
|
195
|
+
module.exports = replay;
|
package/src/core/configLoader.js
CHANGED
|
@@ -158,8 +158,10 @@ function validateConfig(config, filePath) {
|
|
|
158
158
|
throw new Error(`Config error in "${filePath}": gate.${key} must be a string`);
|
|
159
159
|
}
|
|
160
160
|
}
|
|
161
|
-
|
|
162
|
-
|
|
161
|
+
for (const key of ['upload', 'prove']) {
|
|
162
|
+
if (g[key] !== undefined && typeof g[key] !== 'boolean') {
|
|
163
|
+
throw new Error(`Config error in "${filePath}": gate.${key} must be a boolean`);
|
|
164
|
+
}
|
|
163
165
|
}
|
|
164
166
|
valid.gate = {
|
|
165
167
|
spec: g.spec,
|
|
@@ -167,6 +169,7 @@ function validateConfig(config, filePath) {
|
|
|
167
169
|
waivers: g.waivers,
|
|
168
170
|
maxWaiverDays: g.maxWaiverDays !== undefined ? Number(g.maxWaiverDays) : undefined,
|
|
169
171
|
upload: g.upload,
|
|
172
|
+
prove: g.prove,
|
|
170
173
|
org: g.org,
|
|
171
174
|
};
|
|
172
175
|
}
|