testguard-cli 0.18.3 → 0.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +104 -0
- package/README.md +84 -1097
- package/package.json +1 -1
- package/spec/GATE-SEMANTICS.md +90 -24
- package/spec/README.md +4 -2
- package/spec/lib/origins.mjs +19 -0
- package/spec/lib/validate.mjs +22 -1
- package/spec/schemas/claims.schema.json +1 -15
- package/spec/schemas/common.schema.json +16 -0
- package/spec/schemas/evidence.schema.json +3 -16
- package/spec/schemas/ignore.schema.json +2 -2
- package/spec/schemas/status.schema.json +23 -0
- package/src/brief/brief.mjs +18 -0
- package/src/cli.mjs +34 -10
- package/src/commands/admit.mjs +8 -1
- package/src/commands/baseline.mjs +16 -2
- package/src/commands/brief.mjs +2 -2
- package/src/commands/claims.mjs +1 -1
- package/src/commands/concerns.mjs +2 -10
- package/src/commands/gate.mjs +2 -4
- package/src/commands/init.mjs +12 -2
- package/src/commands/probe.mjs +13 -5
- package/src/commands/replay.mjs +9 -2
- package/src/commands/runner-options.mjs +28 -0
- package/src/commands/sweep.mjs +8 -1
- package/src/gate/changed.mjs +11 -0
- package/src/git.mjs +12 -2
- package/src/init/init.mjs +169 -74
- package/src/probe/attribution.mjs +5 -0
- package/src/probe/contention.mjs +14 -3
- package/src/probe/cost.mjs +7 -1
- package/src/probe/policy-freshness.mjs +5 -1
- package/src/probe/probe.mjs +31 -5
- package/src/probe/runners/python.mjs +19 -5
- package/src/probe/universe.mjs +14 -2
- package/src/replay/replay.mjs +9 -8
- package/src/supply/concerns.mjs +14 -8
- package/src/sweep/sweep.mjs +2 -0
package/README.md
CHANGED
|
@@ -7,8 +7,10 @@
|
|
|
7
7
|
[](https://www.npmjs.com/package/testguard-cli)
|
|
8
8
|
[](https://pypi.org/project/testguard-cli/)
|
|
9
9
|
[](https://nodejs.org)
|
|
10
|
-
[](
|
|
11
|
-
[](
|
|
10
|
+
[](https://github.com/raccioly/testguard/blob/main/LICENSE)
|
|
11
|
+
[](https://github.com/raccioly/testguard/blob/main/package.json)
|
|
12
|
+
|
|
13
|
+
**English** · [Português (Brasil)](https://github.com/raccioly/testguard/blob/main/docs/i18n/pt-BR/README.md) · [Español](https://github.com/raccioly/testguard/blob/main/docs/i18n/es/README.md) · [简体中文](https://github.com/raccioly/testguard/blob/main/docs/i18n/zh-CN/README.md)
|
|
12
14
|
|
|
13
15
|
> Breaks your code on purpose and reports every promise your tests did not
|
|
14
16
|
> notice breaking.
|
|
@@ -80,7 +82,7 @@ risk. TestGuard binds every fault to a **stated claim**, so the output is not
|
|
|
80
82
|
a score but a finding: *"Your project says a missing scope fails closed.
|
|
81
83
|
Nothing checks that."* That is the difference between a metric and an audit.
|
|
82
84
|
|
|
83
|
-
📄 **[Read the six-page technical brief (PDF)](docs/testguard-explained.pdf)**
|
|
85
|
+
📄 **[Read the six-page technical brief (PDF)](https://github.com/raccioly/testguard/blob/main/docs/testguard-explained.pdf)**
|
|
84
86
|
— the idea on page one, then the field evidence, the anatomy of a run, the
|
|
85
87
|
verdicts, the loop and its calibration, and the prior art.
|
|
86
88
|
|
|
@@ -127,1153 +129,138 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
|
|
|
127
129
|
| npm | `npm i -D testguard-cli` then `npx testguard probe` |
|
|
128
130
|
| pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
|
|
129
131
|
| Homebrew | `brew tap raccioly/tap && brew install testguard` |
|
|
130
|
-
| GitHub Action | `uses: raccioly/testguard@v0.
|
|
131
|
-
| pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
|
|
132
|
-
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.
|
|
132
|
+
| GitHub Action | `uses: raccioly/testguard@v0.20.0` |
|
|
133
|
+
| pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-gate`, `testguard-probe` |
|
|
134
|
+
| GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.20.0/packaging/gitlab/testguard.gitlab-ci.yml` |
|
|
133
135
|
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
`npm i -D testguard-cli --min-release-age=0`.
|
|
136
|
+
Requirements, offline installs, `ENOVERSIONS` and every CI option are in the
|
|
137
|
+
[installation guide](https://github.com/raccioly/testguard/blob/main/docs/installation.md).
|
|
137
138
|
|
|
138
|
-
##
|
|
139
|
+
## Quickstart
|
|
139
140
|
|
|
140
141
|
```bash
|
|
141
142
|
npx testguard-cli init # install the agent layer at the git root
|
|
142
143
|
npx testguard-cli status --json # where the project is, and the ONE next action
|
|
143
|
-
npx testguard-cli claims # what does this project claim, and is each claim probeable?
|
|
144
|
-
npx testguard-cli claims --check-anchors # fail fast if an exact anchor moved or a replacement no longer parses
|
|
145
|
-
npx testguard-cli claims --annotate --claim AUTH-ADMIN --json # read-only file-header placement preview
|
|
146
144
|
npx testguard-cli scaffold src/auth.ts # propose faults for a file, as a draft to keep or drop
|
|
145
|
+
npx testguard-cli claims --check-anchors # every claim is probeable before anything runs
|
|
147
146
|
npx testguard-cli probe # try to falsify each claim; report what the tests missed
|
|
148
|
-
npx testguard-cli admit test/auth.test.ts --claim AUTH-ADMIN # does this test
|
|
147
|
+
npx testguard-cli admit test/auth.test.ts --claim AUTH-ADMIN # does this test fail when the claim is false?
|
|
149
148
|
npx testguard-cli baseline # freeze today's unproven findings; only new ones gate
|
|
150
149
|
npx testguard-cli gate --changed origin/main # fail when a changed source file carries no claim
|
|
151
|
-
npx testguard-cli sweep --changed origin/main # no claims yet?
|
|
150
|
+
npx testguard-cli sweep --changed origin/main # no claims yet? report what nothing noticed in the changed files
|
|
152
151
|
npx testguard-cli brief --text # tell the agent where the suite is blind, before it writes
|
|
153
|
-
npx testguard-cli replay --since
|
|
154
|
-
npx testguard-cli mcp # serve the read-only loop over MCP, on stdio
|
|
152
|
+
npx testguard-cli replay --since HEAD~50..HEAD # would this suite have caught the bugs that escaped?
|
|
155
153
|
```
|
|
156
154
|
|
|
157
155
|
Exit codes are the contract: `0` nothing new to prove, `1` unproven claims,
|
|
158
156
|
unclaimed changes or invalid fault anchors, `2` a precondition failed and
|
|
159
|
-
nothing was probed, `3` usage.
|
|
160
|
-
Every command accepts `--json`.
|
|
161
|
-
|
|
162
|
-
### Intent-first drafts
|
|
163
|
-
|
|
164
|
-
Scaffold drafts propose mechanical faults, not intended behavior. New claims
|
|
165
|
-
declare `inferred` origin even with an annotation. Supply the intended observable
|
|
166
|
-
behavior from a requirement, ADR, bug or incident independently of the code:
|
|
167
|
-
inputs, expected outcome and forbidden outcome. Record the supplied source
|
|
168
|
-
kind/ref; when unavailable keep inferred. References are not authenticated or
|
|
169
|
-
fetched, and passing tests do not establish independent intent. Review proposals
|
|
170
|
-
against the supplied intent before adopting and probing; existing `--claim`
|
|
171
|
-
metadata is preserved.
|
|
172
|
-
|
|
173
|
-
### Optional annotation placement
|
|
174
|
-
|
|
175
|
-
Ordinary `claims` output and status notes identify missing scanned source links
|
|
176
|
-
as advisories. Counts are distinct claim IDs, not faults. They do not change
|
|
177
|
-
verification state, its next action or exit codes; annotation-sourced missing
|
|
178
|
-
IDs still produce the existing claims drift error. Scanned text is a lexical
|
|
179
|
-
link, not authenticated intent or source ownership.
|
|
180
|
-
Status advice uses a bounded scan (10,000 entries, 1,000 files, 64 directory
|
|
181
|
-
levels, 256 KiB per file, 2 MiB total); incomplete scans report unavailable
|
|
182
|
-
advice without concluding an ID is absent. Keep the original `--claims`
|
|
183
|
-
argument for preview when inspecting a custom claims selection.
|
|
184
|
-
|
|
185
|
-
`testguard claims --annotate` previews JavaScript/TypeScript or Python
|
|
186
|
-
file-header links for declared claims; repeat `--claim` or use comma-separated
|
|
187
|
-
IDs to select claims. Add `--apply` only after reviewing the selection to
|
|
188
|
-
explicitly write annotations. Each invocation recomputes its plan: an earlier
|
|
189
|
-
preview is not a saved approval token. No `--out` or unrelated inspection
|
|
190
|
-
switches are accepted with authoring. JSON conforms to the `annotations`
|
|
191
|
-
authoring schema; it contains no source/proposed contents or probe verdicts.
|
|
192
|
-
|
|
193
|
-
This is file-level placement, not symbol ownership or proof of correctness.
|
|
194
|
-
Missing annotations do not disable exact fault-anchor validation, and placement
|
|
195
|
-
does not change claim origins, evidence or baselines. Unsupported, ambiguous or
|
|
196
|
-
unsafe targets block the complete selection. Original/proposed files are capped
|
|
197
|
-
at 2 MiB and the aggregate at 16 MiB. Preserve BOM, shebang, Python encoding
|
|
198
|
-
cookie, newline style and permissions. A preview exits 0 when applicable, 2 when
|
|
199
|
-
refused; apply exits 0 only after verified completion, 2 for refusal/partial
|
|
200
|
-
failure; malformed options exit 3.
|
|
201
|
-
|
|
202
|
-
Apply uses an exclusive owned lock and retains fsynced originals in an owner-only
|
|
203
|
-
recovery directory outside the checkout (the result reports its location).
|
|
204
|
-
External claims files can be previewed but cannot authorize writes. It never
|
|
205
|
-
reclaims an existing lock or automatically rolls back over editor changes.
|
|
206
|
-
This is recoverable, not a cross-file atomic transaction: interruption or
|
|
207
|
-
arbitrary concurrent edits can leave partial content. Compare retained originals
|
|
208
|
-
manually before recovery; inspect ownership before removing a leftover lock.
|
|
209
|
-
|
|
210
|
-
## How it works
|
|
211
|
-
|
|
212
|
-
```bash
|
|
213
|
-
npx testguard-cli init # install the agent layer: skill, session-start hook, AGENTS.md section
|
|
214
|
-
npx testguard-cli status --json # where the project is and the ONE next action — the machine entry point
|
|
215
|
-
npx testguard-cli claims # what does this project claim, and is every claim probeable?
|
|
216
|
-
npx testguard-cli probe # try to falsify each claim; report what the tests missed
|
|
217
|
-
npx testguard-cli baseline # freeze today's unproven findings; from now on only new ones gate
|
|
218
|
-
npx testguard-cli brief # tell the agent where the suite is blind, before it writes
|
|
219
|
-
npx testguard-cli gate --changed origin/main # fail when a changed source file carries no claim at all
|
|
220
|
-
npx testguard-cli scaffold src/x.ts # propose faults for a file, as a draft to keep or drop
|
|
221
|
-
npx testguard-cli admit test/x.test.ts --claim X # is this test green on HEAD and does it fail on every fault of X? ADMITTED or NOT ADMITTED
|
|
222
|
-
```
|
|
223
|
-
|
|
224
|
-
1. **Claims** live in `testguard.claims.json` (editors validate it against
|
|
225
|
-
`"$schema": "./node_modules/testguard-cli/spec/schemas/claims.schema.json"`):
|
|
226
|
-
a statement, where it comes from, which tests supposedly defend it, and
|
|
227
|
-
one or more *faults* — each a deterministic source change that would make
|
|
228
|
-
the statement false. Every
|
|
229
|
-
claim and every fault records who produced it. `testguard claims`
|
|
230
|
-
validates the file and reports drift against `@claim <ID>` annotations in
|
|
231
|
-
source. Test files are deliberately not scanned — a claim asserted by a test is the authorship trap the tool exists for — and annotation ids must contain a hyphen so prose is never mistaken for one.
|
|
232
|
-
`claims --check-anchors` additionally locates every exact anchor and parses
|
|
233
|
-
each in-memory JavaScript or Python replacement. It runs no tests, creates
|
|
234
|
-
no worktree and changes no file, so a moved anchor is reported before an
|
|
235
|
-
expensive probe. It reports and stops; it never guesses a new anchor.
|
|
236
|
-
2. **Probe** confirms the defenders are green N times unmodified, applies
|
|
237
|
-
each fault in a scratch git worktree (your tree is never touched), runs
|
|
238
|
-
the defenders N times, re-runs survivors against the whole suite with
|
|
239
|
-
N-run attribution (up to N: it stops as soon as no test has failed in
|
|
240
|
-
every run, because from there none can be named a killer), restores, and
|
|
241
|
-
classifies. It warns when another test runner is already running — a
|
|
242
|
-
contended machine turns a slow suite into a `TIMEOUT` verdict about the
|
|
243
|
-
load, not the claim — records that on the evidence, and offers `--serial`
|
|
244
|
-
to run one test file at a time. Verdicts are a closed set:
|
|
245
|
-
|
|
246
|
-
| Verdict | Meaning |
|
|
247
|
-
|---|---|
|
|
248
|
-
| `killed` | a test body rejected the behaviour, N/N — the only pass |
|
|
249
|
-
| `SURVIVED` | the defenders stayed green while the claim was false |
|
|
250
|
-
| `NOCOVER` | no test file defends the claim at all |
|
|
251
|
-
| `UNVERIFIABLE` | the fault's anchor is missing or ambiguous — loud, never a skip |
|
|
252
|
-
| `TIMEOUT` | the defenders hung; a hang is not a detection |
|
|
253
|
-
| `FAULT-INVALID` | the replacement does not load — a bad fault, not a finding |
|
|
254
|
-
| `FLAKY-DEFENDER` | the defenders are not reliably green, or disagreed across runs — `detail.flakeRate` says how often (`failures` of `runs` on unmodified source) |
|
|
255
|
-
|
|
256
|
-
Never a single score. Findings are ranked by severity, claim provenance
|
|
257
|
-
and blast radius (relative imports, `tsconfig` path aliases and
|
|
258
|
-
`package.json#imports` are resolved; bare package names are not), and
|
|
259
|
-
written to `.testguard/evidence.json` — validated against the spec before
|
|
260
|
-
it is written.
|
|
261
|
-
|
|
262
|
-
Worktree mode probes a **commit**. If a defender or target file has
|
|
263
|
-
uncommitted changes and you asked for the implicit HEAD, `probe` refuses
|
|
264
|
-
and says so — otherwise your new tests would be silently absent and the
|
|
265
|
-
same survivors would come back with no hint why. `--include-dirty`
|
|
266
|
-
snapshots the working tree (tracked edits and new files) into a throwaway
|
|
267
|
-
commit and probes that; your tree, HEAD and index are never touched. An
|
|
268
|
-
**explicit `--ref`** (a pre-fix commit in a post-mortem, say) is honoured
|
|
269
|
-
over a dirty tree, as is `--ignore-dirty`: a warning names the files and
|
|
270
|
-
the evidence records them as `repo.ignoredDirty`, so the run says what it
|
|
271
|
-
did not look at. Every summary names the commit probed.
|
|
272
|
-
|
|
273
|
-
A fault may declare its own `defendedBy` to override the claim's set.
|
|
274
|
-
Omit the fault field to inherit; set it to `[]` to request discovery rather
|
|
275
|
-
than inheritance. Evidence names the selection origin and selected files.
|
|
276
|
-
Changing either invalidates reuse and makes status stale. A narrowed fault
|
|
277
|
-
set after a prior kill is warned about by `claims`; re-probe before trusting
|
|
278
|
-
it. For example, a unit-level fault can select `["test/unit.test.mjs"]`
|
|
279
|
-
while an integration fault inherits the claim's broader set.
|
|
280
|
-
|
|
281
|
-
A claim with no `defendedBy` has its defenders **discovered**: the test
|
|
282
|
-
files that import the fault's target, by relative path or resolved alias
|
|
283
|
-
(tsconfig `paths`, through `extends` and `references`, vite/vitest
|
|
284
|
-
`resolve.alias`, package.json `imports`) — **minus the files that mock
|
|
285
|
-
it**. A test that `vi.mock`s / `jest.mock`s the target cannot detect any
|
|
286
|
-
fault in it; counting it would make `NOCOVER` under-report and waste runs.
|
|
287
|
-
`NOCOVER` therefore means exactly "no test file imports this source
|
|
288
|
-
without mocking it". `claims` prints the split (`18 import · 16 mock · 2
|
|
289
|
-
can detect`), the evidence lists the mocking files, and a mocking file
|
|
290
|
-
that never `expect(...)`s anything imported from the target carries the
|
|
291
|
-
static signal `mocked-never-asserted` — the cheapest blind-spot signal
|
|
292
|
-
there is, and the exact signature of one escaped bug in the field
|
|
293
|
-
reports. Silence it, visibly, with `// unasserted: <why>` above the mock.
|
|
294
|
-
Its sibling `persistence-payload-unasserted` is attached to a **survivor
|
|
295
|
-
on a write path** whose defender mocks the database: the test can prove
|
|
296
|
-
the call shape and never that the row landed, and the signal names the
|
|
297
|
-
mocked layer and the file's assertion mix (exact, partial, argument-free).
|
|
298
|
-
|
|
299
|
-
Runners: **vitest**, **jest**, **Playwright** and **Python** (`--runner
|
|
300
|
-
auto` picks the first of vitest, jest, python that resolves). A JavaScript
|
|
301
|
-
runner is resolved from the **project's own package** first — its pinned
|
|
302
|
-
version, its own bin script — and only then from an executable on PATH,
|
|
303
|
-
which the evidence records as `runner.source: "path"`; `npx` is never
|
|
304
|
-
asked, because its cache answers for packages a project does not have.
|
|
305
|
-
**Node's built-in framework** uses explicit `--runner node-test` with plain
|
|
306
|
-
JavaScript test files and default Node collection. It uses the Node executable
|
|
307
|
-
running TestGuard, records `runner.source: "builtin"` and its actual version,
|
|
308
|
-
and installs nothing. Discovery loads test modules with test bodies filtered
|
|
309
|
-
out, so top-level code can run. Custom loaders, `NODE_OPTIONS`, `NODE_PATH`
|
|
310
|
-
and module-mocking configurations are outside this initial adapter's scope.
|
|
311
|
-
`auto` keeps its existing runner order. Each confirmation uses fresh isolated
|
|
312
|
-
processes and the same worker, timeout and confirmation rules.
|
|
313
|
-
|
|
314
|
-
To reproduce the narrow native-runner comparison, run
|
|
315
|
-
`node .github/scripts/benchmark-node-runner.mjs --out /absolute/receipt-dir`
|
|
316
|
-
from this repository after installing its pinned development dependencies.
|
|
317
|
-
It compares 17 identical classifier inputs and four existing faults through
|
|
318
|
-
complete probes, with three confirmations, fresh worker receipts and
|
|
319
|
-
reversed execution order. Results describe that workload and runtime only.
|
|
320
|
-
|
|
321
|
-
Other frameworks go through `--runner-cmd`. Each runner is proven against
|
|
322
|
-
its own copy of the known-answer fixture.
|
|
323
|
-
|
|
324
|
-
**Python** runs under `pytest` when the project's interpreter can import
|
|
325
|
-
it and stdlib `unittest` when it cannot, and the evidence records the
|
|
326
|
-
engine that actually ran — `--runner pytest` or `--runner unittest` pins
|
|
327
|
-
the choice and fails rather than falling back. Nothing is installed into
|
|
328
|
-
the project: the reporters ship inside TestGuard and reach the interpreter
|
|
329
|
-
through `PYTHONPATH`, so a codebase whose test dependencies are the
|
|
330
|
-
standard library stays that way. The interpreter is the project's
|
|
331
|
-
`.venv`/`venv`, the active `VIRTUAL_ENV`, or `--python <path>`. A `.py`
|
|
332
|
-
defender runs under Python whatever the project runner is, so one claim
|
|
333
|
-
can be defended by a vitest test and a pytest test at once.
|
|
334
|
-
|
|
335
|
-
Two things Python forces that JavaScript does not. **`patch("pkg.mod.fn")`
|
|
336
|
-
is not `vi.mock`**: it replaces one attribute, so the file is still a
|
|
337
|
-
defender of every other fault in that module — it keeps its place and
|
|
338
|
-
carries a `target-attribute-patched` signal naming the attributes. And
|
|
339
|
-
**the fault must be the code that ran**: a strict editable install puts an
|
|
340
|
-
import hook ahead of `sys.path`, so Python can load the original file while
|
|
341
|
-
TestGuard faults the copy — a green baseline, every claim `SURVIVED`, and a
|
|
342
|
-
report that reads as a devastating finding while being entirely false. With
|
|
343
|
-
the fault applied, TestGuard asks which file was actually imported and
|
|
344
|
-
refuses the run when it is outside the tree being probed. When no defender
|
|
345
|
-
imported the subject at all, the run continues and the record says so
|
|
346
|
-
(`detail.targetNotImported`), because that is a fact about the defenders'
|
|
347
|
-
reach and not about their assertions.
|
|
348
|
-
|
|
349
|
-
The two-gate rule, as one verb: `testguard admit <test-file> --claim <ID>`
|
|
350
|
-
runs the claim's faults against your uncommitted test and answers
|
|
351
|
-
`ADMITTED` (exit 0: the test is green on unmodified HEAD and fails on
|
|
352
|
-
every fault, N/N) or `NOT ADMITTED` (exit 1: the first blocking fault and
|
|
353
|
-
what to do about it). The test must be a declared or discovered defender
|
|
354
|
-
of the claim. It is sugar over `probe --claim <ID> --include-dirty`, so
|
|
355
|
-
it can never disagree with the gate. Every generator on the market admits
|
|
356
|
-
a test because it compiles, passes and raises coverage; `admit` admits it
|
|
357
|
-
because it fails when the claim is false.
|
|
358
|
-
|
|
359
|
-
Practical loop: first pass `--no-escalate` (escalation re-runs the whole
|
|
360
|
-
suite N times per survivor); iterate on one claim with `--claim <ID>`, or
|
|
361
|
-
select a batch with commas or repeated flags (`--claim A,B --claim C`).
|
|
362
|
-
Every occurrence is retained in first-seen order, duplicates are collapsed,
|
|
363
|
-
and partial human/JSON output names both the requested and probed counts, so
|
|
364
|
-
a green subset cannot masquerade as the selection the operator asked for.
|
|
365
|
-
Use either `--include-dirty` or `--in-place` (only fault target files must be
|
|
366
|
-
clean there; test files may be dirty), optionally `--confirm 1` for a fast
|
|
367
|
-
**provisional** signal — verdicts print with a `?`, evidence goes to
|
|
368
|
-
`evidence-provisional.json`, and `baseline` refuses it; final pass with
|
|
369
|
-
defaults. By default
|
|
370
|
-
the stream shows only unproven faults plus a killed count — `--verbose`
|
|
371
|
-
shows every fault. A custom
|
|
372
|
-
runner (`pnpm --filter`, a specific config) goes in
|
|
373
|
-
`--runner-cmd "<cmd> {files} … {out}"`; if the scratch worktree cannot
|
|
374
|
-
see your `node_modules`, pass `--node-modules <dir>`.
|
|
375
|
-
3. **Baseline** freezes every non-passing fingerprint. Later probes suppress
|
|
376
|
-
what was already known and exit non-zero only on what is new. Prior verdicts
|
|
377
|
-
can be reused when their recorded inputs and runner policy still match.
|
|
378
|
-
Cold runs, changed discovery/configuration or `--no-reuse` can require a
|
|
379
|
-
full probe; unchanged target and defender files alone do not guarantee reuse.
|
|
380
|
-
A baseline frozen from `--include-dirty`
|
|
381
|
-
evidence records the snapshot and points at the *parent* of the commit
|
|
382
|
-
that will carry your tests; after you commit, a clean `probe` plus
|
|
383
|
-
`baseline --restamp` moves it to that commit — only when the fingerprints
|
|
384
|
-
are identical, never otherwise. `status` notes a baseline that predates
|
|
385
|
-
HEAD without making it a state.
|
|
386
|
-
4. **Brief** turns evidence plus baseline into a ranked, capped
|
|
387
|
-
`## TEST BLINDSPOT CONTEXT` block, printed and also written to
|
|
388
|
-
`.testguard/brief.json` (`--text` prints only; `--markdown` prints the same brief as a
|
|
389
|
-
merge-request note for the human reviewer, unclaimed changes first). Wire it into an agent's session start
|
|
390
|
-
— for Claude Code, in `.claude/settings.json`:
|
|
391
|
-
|
|
392
|
-
```json
|
|
393
|
-
{ "hooks": { "SessionStart": [ { "hooks": [
|
|
394
|
-
{ "type": "command", "command": "node_modules/.bin/testguard brief --text 2>/dev/null || { command -v testguard >/dev/null 2>&1 && testguard brief --text 2>/dev/null; } || true" }
|
|
395
|
-
] } ] } }
|
|
396
|
-
```
|
|
397
|
-
|
|
398
|
-
`--text` prints only, and exits 0 silently when there is no evidence yet;
|
|
399
|
-
the command resolves the project's own install, then a `testguard` on
|
|
400
|
-
`PATH`, and ends in `true` — so the hook can never break a session. There
|
|
401
|
-
is no `npx` in it, in any form: `--no-install` resolves the package from
|
|
402
|
-
the registry before declining to install it, so it is quiet rather than
|
|
403
|
-
offline. The brief's first line says which install answered
|
|
404
|
-
(`local install` or `global`), so a stale one is visible.
|
|
405
|
-
|
|
406
|
-
### Advisory updates for AI sessions
|
|
407
|
-
|
|
408
|
-
New integrations created by `testguard init` tell the AI to identify the actual
|
|
409
|
-
CLI (project-local before global) and, once per session when network policy
|
|
410
|
-
permits, read the registry's stable version:
|
|
411
|
-
|
|
412
|
-
```bash
|
|
413
|
-
npm view testguard-cli dist-tags.latest --json --fetch-retries=0 --fetch-timeout=5000
|
|
414
|
-
```
|
|
415
|
-
|
|
416
|
-
This is an optional check performed by the AI, not by TestGuard's CLI or hook.
|
|
417
|
-
A newer stable version produces a suggestion and a request for approval, never
|
|
418
|
-
an automatic install. Project pins and the original package manager still
|
|
419
|
-
apply; prereleases and development checkouts are not replaced. Offline or
|
|
420
|
-
failed checks do not change verdicts, evidence or exit codes, and do not prove
|
|
421
|
-
that an installation is current. AI instructions are guidance, not a guarantee
|
|
422
|
-
that every harness performs the check.
|
|
423
|
-
|
|
424
|
-
Upgrading the package does **not** refresh an existing copied skill or an
|
|
425
|
-
existing managed section already listing the project. Compare the upgraded
|
|
426
|
-
package's `src/init/templates/SKILL.md` with
|
|
427
|
-
`.claude/skills/testguard/SKILL.md` and merge approved changes, preserving local
|
|
428
|
-
customizations. For an existing `AGENTS.md` section, add the same advisory
|
|
429
|
-
policy after reviewing it. `testguard init --force` replaces the whole skill
|
|
430
|
-
but does not refresh an already-listed managed section; use it only with
|
|
431
|
-
explicit overwrite approval and review the diff. No new automatic refresh
|
|
432
|
-
mechanism is introduced.
|
|
433
|
-
|
|
434
|
-
### Would this suite have caught the bugs that already escaped?
|
|
435
|
-
|
|
436
|
-
An injected fault is a fault somebody thought of. A bug that actually shipped
|
|
437
|
-
is ground truth: a human already confirmed it was a defect, and there is no
|
|
438
|
-
equivalent-mutant argument to have about it.
|
|
439
|
-
|
|
440
|
-
```bash
|
|
441
|
-
npx testguard-cli replay --since HEAD~50..HEAD --max 10
|
|
442
|
-
```
|
|
443
|
-
|
|
444
|
-
For each fix commit in the range — one that changes source **and** a test
|
|
445
|
-
together — `replay` checks it out in a scratch worktree, reverts only its
|
|
446
|
-
source files to the parent, **deletes the test the fix shipped** (that test
|
|
447
|
-
proves nothing about what the suite knew before it existed), and runs the
|
|
448
|
-
tests that import the reverted code:
|
|
449
|
-
|
|
450
|
-
| Verdict | Meaning |
|
|
451
|
-
|---|---|
|
|
452
|
-
| `caught` | a remaining test failed by assertion, every run. The suite knew. |
|
|
453
|
-
| `blind` | the suite stayed green on known-broken code. |
|
|
454
|
-
| `nocover` | no test imports the reverted files — worse than blind. |
|
|
455
|
-
| `flaky` | the runs disagreed, so nothing can be concluded. A flaky failure reads as detection, which biases this metric *optimistically*; mixed runs are never `caught`. |
|
|
456
|
-
| `unverifiable` | the revert did not apply, or the suite could not load. |
|
|
457
|
-
|
|
458
|
-
One patch counts once (`git patch-id`), because a dual-branch topology
|
|
459
|
-
carries the same fix under two or three shas. **`replay` reports and never
|
|
460
|
-
gates**: a bug that escaped is history, not a regression in this change.
|
|
461
|
-
|
|
462
|
-
Each replayed bug is also labelled with the injected-fault class its diff
|
|
463
|
-
most resembles, which is the join key that turns an uninterpretable mutation
|
|
464
|
-
score into a statement with a sample size:
|
|
465
|
-
|
|
466
|
-
```
|
|
467
|
-
guard-removed missed 7/9 p=0.78 ci [0.45, 0.94]
|
|
468
|
-
field-dropped missed 4/4 p=1.00 ci [0.51, 1.00]
|
|
469
|
-
```
|
|
470
|
-
|
|
471
|
-
Read as *"when a real bug of this class escapes, how often does the suite
|
|
472
|
-
miss it"* — a miss rate, so a high `p` is bad. It is written as a
|
|
473
|
-
`calibration` document beside the replay one. `caught`, `blind` and
|
|
474
|
-
`nocover` are measurements and enter the ratio, `nocover` as a miss: leaving
|
|
475
|
-
it out would score a project with no tests at all for a subsystem *better*
|
|
476
|
-
than one with weak tests. `flaky` and `unverifiable` are failed measurements
|
|
477
|
-
and enter neither side. Every `p` and `ci` is recomputable from `n` and the
|
|
478
|
-
miss count, and the validator recomputes them.
|
|
479
|
-
|
|
480
|
-
**The open question this exists to answer.** Does a calibration learned on a
|
|
481
|
-
repository *with* history transfer to a greenfield one that has none? AI
|
|
482
|
-
authored code has no history, so replay cannot help it directly and
|
|
483
|
-
calibration is the only bridge. That transfer is unproven, it is the core
|
|
484
|
-
product bet, and this is the instrument for testing it — not the answer.
|
|
485
|
-
|
|
486
|
-
### Read CI's evidence locally
|
|
157
|
+
nothing was probed, `3` usage. Every command except `mcp` accepts `--json`.
|
|
487
158
|
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
159
|
+
The [quickstart](https://github.com/raccioly/testguard/blob/main/docs/quickstart.md)
|
|
160
|
+
walks through a first claim end to end. Adding TestGuard to a codebase that
|
|
161
|
+
already has a large suite has its own guide:
|
|
162
|
+
[adopting TestGuard in an existing project](https://github.com/raccioly/testguard/blob/main/docs/guides/existing-projects.md).
|
|
492
163
|
|
|
493
|
-
|
|
494
|
-
testguard status . --evidence .testguard/ci/ci-self-evidence.json
|
|
495
|
-
testguard brief . --text --evidence .testguard/ci/ci-self-evidence.json
|
|
496
|
-
```
|
|
497
|
-
|
|
498
|
-
A foreign document is not trusted blindly. `status` marks it
|
|
499
|
-
`evidenceSource: provided`, prints the commit it describes next to the
|
|
500
|
-
commit in your tree, and still computes staleness from the recorded input
|
|
501
|
-
hashes — so a file you have edited since CI probed it goes `evidence-stale`
|
|
502
|
-
for exactly those claims.
|
|
503
|
-
|
|
504
|
-
`testguard init --ci-evidence github` (or `gitlab`) writes
|
|
505
|
-
`.testguard/fetch-ci-evidence.sh`, which downloads the branch-named artifact
|
|
506
|
-
with the platform CLI you already have and then briefs from it. **It is an
|
|
507
|
-
on-demand helper, not a hook**: the session-start hook never touches the
|
|
508
|
-
network, and the helper exits 0 with a message when the CLI or the artifact
|
|
509
|
-
is missing. TestGuard itself still makes no network calls.
|
|
510
|
-
|
|
511
|
-
### Starting from zero claims
|
|
512
|
-
|
|
513
|
-
During gradual adoption, use `testguard probe --allow-empty` to skip a valid
|
|
514
|
-
empty claims file without blocking a push. It explicitly reports that no
|
|
515
|
-
verification was performed and writes no evidence; `--json` retains the
|
|
516
|
-
`no-claims` state. Missing or invalid claims files still fail. Run `gate`
|
|
517
|
-
separately to enforce changed-file coverage. For the pre-commit probe hook,
|
|
518
|
-
set `args: [--allow-empty]`; for the GitHub Action, set `allow-empty: 'true'`.
|
|
519
|
-
The default probe still exits 2 when no claims exist.
|
|
520
|
-
|
|
521
|
-
In a monorepo, a valid nested `testguard.claims.json` marks a separate project.
|
|
522
|
-
The parent gate reports its changed files as delegated and does not count them
|
|
523
|
-
as parent coverage. Run `gate` in each child project as well; a passing parent
|
|
524
|
-
gate does not establish that child coverage passed. Invalid child claims fail
|
|
525
|
-
evaluation instead of hiding files.
|
|
164
|
+
## Languages and test runners
|
|
526
165
|
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
166
|
+
| Language | Runners | Faults proposed by `scaffold` |
|
|
167
|
+
|---|---|---|
|
|
168
|
+
| JavaScript | vitest, jest, Playwright, Node's built-in test runner | seven shapes: forced guards, deleted statements, altered returns, weakened literals, removed calls, dropped payload fields, swapped arguments |
|
|
169
|
+
| TypeScript | vitest, jest, Playwright (through the project's own transform) | the same seven |
|
|
170
|
+
| JSX, TSX (UI) | vitest, jest, Playwright | the seven, plus removed elements and dropped event handlers |
|
|
171
|
+
| Python ≥ 3.8 | pytest, or stdlib `unittest` when pytest is absent | the same seven, in Python syntax |
|
|
172
|
+
| Anything else | `--runner-cmd` with a jest-compatible JSON report | hand-written faults |
|
|
531
173
|
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
```bash
|
|
536
|
-
npx testguard-cli sweep --changed origin/main # propose, probe a bounded selection, report
|
|
537
|
-
npx testguard-cli sweep --changed HEAD --include-dirty --cap 20
|
|
538
|
-
```
|
|
539
|
-
|
|
540
|
-
It takes the files `gate` just called uncovered, proposes faults with the same
|
|
541
|
-
mechanical producers `scaffold` uses, probes a bounded selection, and reports
|
|
542
|
-
what a green suite did not notice. A fault that survives needs no claim to be
|
|
543
|
-
alarming: something was deliberately broken and not one test failed.
|
|
544
|
-
|
|
545
|
-
```
|
|
546
|
-
swept 13 of 14 unclaimed changed files against origin/main.
|
|
547
|
-
proposed 510 faults, probed 10 (cap 10), deferred 350.
|
|
548
|
-
The 350 deferred are not a verdict: raise --cap, or sweep a smaller change.
|
|
549
|
-
150 presentational elements (icons, static wrappers) set aside: nobody writes a test for those, and a survivor there would teach the ordering the wrong lesson.
|
|
174
|
+
One claim can be defended by a vitest test and a pytest test at once. Runner
|
|
175
|
+
resolution, discovery, mocks and limits are in the
|
|
176
|
+
[languages and runners reference](https://github.com/raccioly/testguard/blob/main/docs/reference/languages-and-runners.md).
|
|
550
177
|
|
|
551
|
-
|
|
178
|
+
## Use it with your AI agent
|
|
552
179
|
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
payload fails only against an assertion that names that field exactly.
|
|
558
|
-
Assert the whole object written by the call this fault changes, including
|
|
559
|
-
`password_hash` — or read the record back and assert on what was stored.
|
|
560
|
-
```
|
|
561
|
-
|
|
562
|
-
Three rules make it safe to run on a repository that has never seen this tool:
|
|
563
|
-
|
|
564
|
-
Pooled draft IDs are unique even when source files share basenames/functions or
|
|
565
|
-
annotation IDs. Sweep reserves original IDs and suffixes collisions before
|
|
566
|
-
selection, keeping every fault, its metadata and defenders attached to its own
|
|
567
|
-
draft. These are proposal identities, not authenticated claims.
|
|
568
|
-
|
|
569
|
-
- **It never writes `testguard.claims.json`**, and never will. A claim is a
|
|
570
|
-
sentence someone is willing to stand behind; a sentence nobody wrote is not
|
|
571
|
-
one. Its drafts land beside the evidence to keep or drop.
|
|
572
|
-
- **Its evidence never replaces `.testguard/evidence.json`.** A sweep probes
|
|
573
|
-
faults nobody stated, under TODO statements; folding that into the document
|
|
574
|
-
`status` and `baseline` read would corrupt the record of what the project
|
|
575
|
-
actually claims.
|
|
576
|
-
- **It does not fail on its own bad guess.** Only `survived` and `nocover`
|
|
577
|
-
exit 1. A proposal that would not compile, or an anchor that did not locate,
|
|
578
|
-
is reported and never gates — a tool that fails because its own guess was bad
|
|
579
|
-
is a tool people switch off.
|
|
580
|
-
|
|
581
|
-
The ordering **learns from your own runs**. Every evidence record already
|
|
582
|
-
carries a fault class and a verdict, so "how often does a fault of this class
|
|
583
|
-
survive here" is a tally, not a new thing to collect — a sweep persists its
|
|
584
|
-
evidence and the next one reads it. A project with no evidence falls back to
|
|
585
|
-
the shipped prior and says so; one with its own records overrides it in
|
|
586
|
-
proportion to how many it has. The report names which, because an ordering
|
|
587
|
-
nobody can trace is a number nobody should trust.
|
|
588
|
-
|
|
589
|
-
The cap and the ordering are Google's: their mutation service surfaces at most
|
|
590
|
-
7 × |files| mutants per change and orders candidates on the measured
|
|
591
|
-
productivity of their operator in similar context, which took their productive
|
|
592
|
-
rate from 15% to 89%
|
|
593
|
-
([Petrović et al., *Practical Mutation Testing at Scale*, 2021](https://arxiv.org/abs/2102.11378)).
|
|
594
|
-
TestGuard's ordering prior is measured the same way — from probed faults on a
|
|
595
|
-
real AI-authored codebase — and shrunk toward a neutral value, so one
|
|
596
|
-
observation of a class surviving never dominates a sweep. The cap is spread
|
|
597
|
-
across files: a single file with many candidates cannot make the others look
|
|
598
|
-
clean.
|
|
599
|
-
|
|
600
|
-
A sweep is weaker evidence than a probe, and says so. Nobody stated that the
|
|
601
|
-
behaviour mattered. It is stronger than nothing, which is what a repository
|
|
602
|
-
with no claims has — and the survivors worth defending become the first claims.
|
|
603
|
-
|
|
604
|
-
#### Concerns: one sentence, two hundred screens
|
|
605
|
-
|
|
606
|
-
A claim names one promise precisely, which is why a repository with two hundred
|
|
607
|
-
screens never finishes writing them. A **concern** names a *kind* of promise and
|
|
608
|
-
says where to look for it:
|
|
609
|
-
|
|
610
|
-
```bash
|
|
611
|
-
npx testguard-cli concerns # what this project can be swept by
|
|
612
|
-
npx testguard-cli sweep --concern ADMIN-GUARDS # aim a sweep at one
|
|
613
|
-
```
|
|
180
|
+
TestGuard is meant to be driven by an agent. `testguard init` installs a skill
|
|
181
|
+
that teaches the operating loop, an `AGENTS.md` section, and a session-start
|
|
182
|
+
hook that briefs every session with the ranked blind spots before it writes
|
|
183
|
+
code. For Claude Code, the hook in `.claude/settings.json` is:
|
|
614
184
|
|
|
615
185
|
```json
|
|
616
|
-
{
|
|
617
|
-
"
|
|
618
|
-
|
|
619
|
-
{
|
|
620
|
-
"id": "ADMIN-GUARDS",
|
|
621
|
-
"statement": "Every admin route refuses a caller without the admin role.",
|
|
622
|
-
"severity": "critical",
|
|
623
|
-
"targets": { "kind": "glob", "globs": ["src/app/api/admin/**"] },
|
|
624
|
-
"faultClasses": ["guard-removed", "condition-forced", "return-altered"]
|
|
625
|
-
}
|
|
626
|
-
]
|
|
627
|
-
}
|
|
186
|
+
{ "hooks": { "SessionStart": [ { "hooks": [
|
|
187
|
+
{ "type": "command", "command": "node_modules/.bin/testguard brief --text 2>/dev/null || { command -v testguard >/dev/null 2>&1 && testguard brief --text 2>/dev/null; } || true" }
|
|
188
|
+
] } ] } }
|
|
628
189
|
```
|
|
629
190
|
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
|
|
636
|
-
SURVIVED src/app/api/admin/claims/[id]/route.ts
|
|
637
|
-
[line 41] Guard never triggers: `if (!membership || (membership.role !== 'owner'
|
|
638
|
-
&& membership.role !== 'manager'))` becomes `if (false)`.
|
|
639
|
-
```
|
|
640
|
-
|
|
641
|
-
The idea is Meta's: ACH has an engineer describe an area of concern in plain
|
|
642
|
-
text and gates every generated step by execution. What is taken here is the
|
|
643
|
-
**unit** — the concern as the thing a human writes — not their generator, and
|
|
644
|
-
**no model is required**. A concern's useful core is a named scope plus a
|
|
645
|
-
producer selection, which the mechanical producers already satisfy, so every
|
|
646
|
-
verifying command stays offline.
|
|
647
|
-
|
|
648
|
-
Two concerns ship built in, and only two. An authorization concern would need a
|
|
649
|
-
heuristic for *"which files check permissions"*, and a guess there is exactly
|
|
650
|
-
the noise that gets a check switched off — so `SAVE-PERSISTS` and
|
|
651
|
-
`CHANGED-CODE` are the defaults, and a project that knows its own auth layer
|
|
652
|
-
says so in three lines. Declaring a concern with a built-in's id replaces it,
|
|
653
|
-
and the replacement is reported rather than silent.
|
|
654
|
-
|
|
655
|
-
A concern is **not** a claim and never becomes one by itself. It says where to
|
|
656
|
-
look; the probe says what it found; a human states the sentence worth defending.
|
|
657
|
-
|
|
658
|
-
#### Sweeping the save surface instead of the diff
|
|
659
|
-
|
|
660
|
-
A diff cannot answer *"when the user clicks save, does it actually save?"* — it
|
|
661
|
-
only knows what changed today. `--save-paths` points the same sweep at every
|
|
662
|
-
file that writes to storage:
|
|
663
|
-
|
|
664
|
-
```bash
|
|
665
|
-
npx testguard-cli sweep --save-paths --cap 20
|
|
666
|
-
```
|
|
667
|
-
|
|
668
|
-
```
|
|
669
|
-
119 writes to storage across 37 files, carrying 226 payload fields. Swept 36.
|
|
670
|
-
proposed 762 faults, probed 20 (cap 20), deferred 742.
|
|
671
|
-
|
|
672
|
-
SURVIVED src/app/actions/register.ts [write path]
|
|
673
|
-
[line 59] Field dropped: `password_hash` is no longer written.
|
|
674
|
-
src/tests/actions/register.test.ts mocks @/lib/prisma; 1 exact, 8 partial,
|
|
675
|
-
31 argument-free call assertions in the file. A field dropped from the write
|
|
676
|
-
payload fails only against an assertion that names that field exactly.
|
|
677
|
-
```
|
|
678
|
-
|
|
679
|
-
The first line is the point. A report that lists findings without saying how
|
|
680
|
-
much surface it looked at invites you to assume the rest is fine, so the
|
|
681
|
-
denominator comes first and the document carries `writeSites` and
|
|
682
|
-
`payloadFields` whether or not anything survived. Measured on a real
|
|
683
|
-
AI-authored application: **12 of 26 probed persistence faults survived a fully
|
|
684
|
-
green suite, and all 12 were a dropped payload field.**
|
|
685
|
-
|
|
686
|
-
It also states, before any verdict, **what the defenders could prove at all**:
|
|
687
|
-
|
|
688
|
-
```
|
|
689
|
-
of those 37 files: 32 defended only by tests that mock the persistence layer,
|
|
690
|
-
1 with an unmocked defender, 4 with no defender at all.
|
|
691
|
-
```
|
|
692
|
-
|
|
693
|
-
That line is the read-back answer. A test that replaced the database can prove
|
|
694
|
-
the *call shape* and nothing beyond it — a `where` that matches no rows, a
|
|
695
|
-
rolled-back transaction and a rejected constraint all pass against a mock that
|
|
696
|
-
recorded the arguments and returned a plausible object. So rather than pretend
|
|
697
|
-
to measure persistence, TestGuard states the limit, the same way `nocover` says
|
|
698
|
-
"no test imports this" instead of guessing.
|
|
699
|
-
|
|
700
|
-
When every defender mocks the layer, the report says so outright: *nothing in
|
|
701
|
-
this suite can prove a write reached storage*. A clean probe there is not
|
|
702
|
-
evidence of persistence — it is evidence that nothing could have measured it.
|
|
703
|
-
|
|
704
|
-
`unmocked` is deliberately weaker than "proves persistence": a test that does
|
|
705
|
-
not replace the database may simply never reach it. Possible is the honest word.
|
|
706
|
-
|
|
707
|
-
### Every change needs a claim
|
|
191
|
+
It resolves the project's own install, then a `testguard` on `PATH`, and ends
|
|
192
|
+
in `true`, so it can never break a session or reach the network. Any other
|
|
193
|
+
harness can use `testguard mcp` (five read-only tools over stdio) or
|
|
194
|
+
`testguard status --json`. Setup for Claude Code, Codex, Cursor and others is
|
|
195
|
+
in the [AI agents guide](https://github.com/raccioly/testguard/blob/main/docs/guides/ai-agents.md).
|
|
708
196
|
|
|
709
|
-
|
|
710
|
-
reports so far was a **claim gap**: the feature shipped green with zero
|
|
711
|
-
claims, and a verifier with no claim about a feature is silent about it by
|
|
712
|
-
construction. `gate` closes that hole on the delta:
|
|
197
|
+
## Documentation
|
|
713
198
|
|
|
714
|
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
unclaimed file exits `1`; there is no percentage. Every reliance on an ignore
|
|
723
|
-
entry is printed, so a reviewer sees *why* the gate passed; an expired entry
|
|
724
|
-
excuses nothing. Non-source files and documented never-claimed patterns
|
|
725
|
-
(`*.d.ts`, `*.config.*`, fixtures, mocks; `--explain` lists them) are
|
|
726
|
-
excluded and said so; `--strict` fails a change that evaluated nothing.
|
|
727
|
-
|
|
728
|
-
An ignore file is a list of `entries`. The field is `pattern`, and `reason` is
|
|
729
|
-
not optional — it is the whole point of the mechanism:
|
|
730
|
-
|
|
731
|
-
```json
|
|
732
|
-
{
|
|
733
|
-
"$schema": "./node_modules/testguard-cli/spec/schemas/ignore.schema.json",
|
|
734
|
-
"schemaVersion": 1,
|
|
735
|
-
"entries": [
|
|
736
|
-
{
|
|
737
|
-
"kind": "path",
|
|
738
|
-
"pattern": "src/commands/**",
|
|
739
|
-
"reason": "Thin wrappers: read flags, call the module, print. The logic is claimed in the modules they call.",
|
|
740
|
-
"by": "maintainer",
|
|
741
|
-
"at": "2026-09-17T00:00:00Z"
|
|
742
|
-
},
|
|
743
|
-
{
|
|
744
|
-
"kind": "path",
|
|
745
|
-
"pattern": "src/legacy/billing.ts",
|
|
746
|
-
"reason": "Scheduled for deletion in Q4; claims would outlive the file.",
|
|
747
|
-
"by": "maintainer",
|
|
748
|
-
"at": "2026-09-17T00:00:00Z",
|
|
749
|
-
"expires": "2026-12-31T00:00:00Z"
|
|
750
|
-
}
|
|
751
|
-
]
|
|
752
|
-
}
|
|
753
|
-
```
|
|
754
|
-
|
|
755
|
-
`kind: "claim"` entries take the same shape and excuse a *removed claim*
|
|
756
|
-
instead of an unclaimed file. An entry past its `expires` excuses nothing and
|
|
757
|
-
is reported as `EXPIRED`.
|
|
758
|
-
|
|
759
|
-
With a reference known, `status --changed <ref>` reports `unclaimed-changes`
|
|
760
|
-
**before** any evidence state and makes the claim the next action; the brief
|
|
761
|
-
lists the unclaimed files first. The claim is written before more code.
|
|
762
|
-
After claim coverage, `status` reports `invalid-anchors` before any evidence
|
|
763
|
-
state when an exact fault anchor is missing or ambiguous, and points at the
|
|
764
|
-
fault definition to repair. Repair preserves the fault's meaning and requires
|
|
765
|
-
a re-probe; there is deliberately no `--fix`.
|
|
766
|
-
|
|
767
|
-
**The base is detected when Git records it.** An explicit `--changed` wins,
|
|
768
|
-
then `TESTGUARD_CHANGED_REF`, GitHub Actions (`GITHUB_BASE_REF`) and GitLab
|
|
769
|
-
merge request pipelines (`CI_MERGE_REQUEST_DIFF_BASE_SHA`, then
|
|
770
|
-
`CI_MERGE_REQUEST_TARGET_BRANCH_NAME`). In an ordinary clone on a feature
|
|
771
|
-
branch, bare `testguard gate` uses the configured remote's symbolic default
|
|
772
|
-
branch (normally `origin/main`) and says how it chose it. If that symbolic ref
|
|
773
|
-
is absent, a configured upstream is used only when its branch name differs
|
|
774
|
-
from the current branch. A feature branch's same-name tracking ref and the
|
|
775
|
-
default branch itself are never selected: either could produce a misleading
|
|
776
|
-
empty diff. In those ambiguous cases, pass `--changed` explicitly.
|
|
777
|
-
|
|
778
|
-
The base must exist locally: GitHub — `actions/checkout` with `fetch-depth: 0`;
|
|
779
|
-
GitLab — the diff base sha needs nothing extra on a merge request pipeline,
|
|
780
|
-
the branch name needs `GIT_DEPTH: 0` or a `git fetch origin <target>`. A
|
|
781
|
-
detected base that does not resolve is a warning for `status` and `brief`
|
|
782
|
-
(they keep working) and an error for `gate` (its whole job is the measurement).
|
|
783
|
-
|
|
784
|
-
```yaml
|
|
785
|
-
# GitHub Actions
|
|
786
|
-
- uses: actions/checkout@v4
|
|
787
|
-
with: { fetch-depth: 0 }
|
|
788
|
-
- uses: raccioly/testguard@v0.18.3
|
|
789
|
-
with: { command: gate }
|
|
790
|
-
|
|
791
|
-
# GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
|
|
792
|
-
include:
|
|
793
|
-
- remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.18.3/packaging/gitlab/testguard.gitlab-ci.yml'
|
|
794
|
-
inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
|
|
795
|
-
```
|
|
796
|
-
|
|
797
|
-
The template carries `spec: inputs:` (`version`, `dir`, `image`, `severity`,
|
|
798
|
-
`confirm`, `budget`, `no_escalate`, `strict`, `post_note`, `stage`), so
|
|
799
|
-
mirrored into a GitLab project it is a catalog component that a compliance
|
|
800
|
-
framework can require on every project in a group. The CLI itself never
|
|
801
|
-
talks to the network; only the job posts, and only when told to.
|
|
802
|
-
|
|
803
|
-
### Properties
|
|
804
|
-
|
|
805
|
-
- **Deterministic measurement.** No model decides a verdict. Faults are
|
|
806
|
-
string substitutions from a reviewed file; verdicts come from the test
|
|
807
|
-
runner's structured report, confirmed N times. The same inputs give the
|
|
808
|
-
same evidence, which is what makes it replayable for an auditor.
|
|
809
|
-
- **No network, no telemetry.** The CLI never opens a socket. The
|
|
810
|
-
session-start hook resolves an installed binary and never fetches one.
|
|
811
|
-
Nothing leaves the machine unless a CI job you configured posts it.
|
|
812
|
-
- **Artifacts are data in the repository.** Claims, evidence, baseline and
|
|
813
|
-
brief are JSON validated against a published schema before they are
|
|
814
|
-
written. Nothing is healed, regenerated or re-targeted at run time; a
|
|
815
|
-
claims file is code, and is reviewed like code.
|
|
816
|
-
- **Never optimistic.** A timeout, a load failure, a mixed N-run result, a
|
|
817
|
-
flaky defender or a missing anchor is reported as unproven, never rounded
|
|
818
|
-
toward green. The one pass is `killed`; everything else gates.
|
|
819
|
-
- **Per-run and whole-command time are separate.** `--budget` limits each
|
|
820
|
-
runner invocation. `--command-budget` is a cooperative deadline for the
|
|
821
|
-
complete `probe`, `sweep`, or `replay` measurement: asynchronous children
|
|
822
|
-
receive only the remaining time, and stage/write boundaries re-check it.
|
|
823
|
-
Synchronous setup may overrun the nominal deadline, but once expiry is
|
|
824
|
-
observed TestGuard exits `2` and writes no new result document; a completed
|
|
825
|
-
prefix is not presented as though the unattempted suffix were clean.
|
|
826
|
-
- **Configured discovery is evidence.** Vitest, Jest, and Playwright enumerate
|
|
827
|
-
their own configured files; TestGuard validates and hashes that immutable
|
|
828
|
-
universe instead of guessing with filename globs. Automatic JavaScript and
|
|
829
|
-
TypeScript discovery follows the imported symbol through barrels. Ambiguous
|
|
830
|
-
or unbounded resolution is `unverifiable`, never a flattering `nocover`.
|
|
831
|
-
- **One exact-pinned runtime dependency** (`ajv`), Node ≥ 20, MIT.
|
|
832
|
-
|
|
833
|
-
### Prior art, and what was taken from where
|
|
834
|
-
|
|
835
|
-
Breaking code to test the tests is mutation testing, and it dates to the 1970s.
|
|
836
|
-
Two industrial programmes solved the parts that make it usable, and this tool
|
|
837
|
-
takes from both.
|
|
838
|
-
|
|
839
|
-
**Google** made it affordable: mutate only *changed* lines, cap what is
|
|
840
|
-
surfaced at seven per file, and order candidates by the measured productivity
|
|
841
|
-
of the operator in similar context — developer feedback took their productive
|
|
842
|
-
rate from 15% to 89%
|
|
843
|
-
([Petrović et al., 2021](https://arxiv.org/abs/2102.11378)). `sweep`'s diff
|
|
844
|
-
scope, its cap and its ordering are theirs; the productivity prior here is
|
|
845
|
-
measured on real probed faults rather than assumed.
|
|
846
|
-
|
|
847
|
-
**Meta** made it specific: an engineer writes a *concern* in plain text, a
|
|
848
|
-
model drafts faults for it, and every step is gated by execution before a
|
|
849
|
-
human ever sees it — 73% of the resulting tests were accepted
|
|
850
|
-
([Foster et al., FSE 2025](https://arxiv.org/abs/2501.12862)). The rule that
|
|
851
|
-
nothing a model proposes reaches the claims file without passing `probe` and a
|
|
852
|
-
human keep is theirs.
|
|
853
|
-
|
|
854
|
-
What neither does is bind a fault to a **stated promise**, which is what this
|
|
855
|
-
tool is for. The full record — including what was deliberately *not* taken, the
|
|
856
|
-
measured reason no model decides a verdict, and the gaps in what we borrowed —
|
|
857
|
-
is in [`docs-canonical/PRIOR-ART.md`](docs-canonical/PRIOR-ART.md).
|
|
199
|
+
| | |
|
|
200
|
+
|---|---|
|
|
201
|
+
| **Get started** | [Quickstart](https://github.com/raccioly/testguard/blob/main/docs/quickstart.md) · [Installation](https://github.com/raccioly/testguard/blob/main/docs/installation.md) · [Upgrading](https://github.com/raccioly/testguard/blob/main/docs/upgrade.md) |
|
|
202
|
+
| **Guides** | [Existing projects](https://github.com/raccioly/testguard/blob/main/docs/guides/existing-projects.md) · [New projects](https://github.com/raccioly/testguard/blob/main/docs/guides/new-projects.md) · [Writing claims](https://github.com/raccioly/testguard/blob/main/docs/guides/writing-claims.md) · [AI agents](https://github.com/raccioly/testguard/blob/main/docs/guides/ai-agents.md) · [Python](https://github.com/raccioly/testguard/blob/main/docs/guides/python.md) · [Monorepos](https://github.com/raccioly/testguard/blob/main/docs/guides/monorepo.md) · [Performance](https://github.com/raccioly/testguard/blob/main/docs/guides/performance.md) · [Replaying escaped bugs](https://github.com/raccioly/testguard/blob/main/docs/guides/replay.md) |
|
|
203
|
+
| **CI** | [GitHub Actions](https://github.com/raccioly/testguard/blob/main/docs/guides/ci/github-actions.md) · [GitLab CI](https://github.com/raccioly/testguard/blob/main/docs/guides/ci/gitlab.md) · [pre-commit](https://github.com/raccioly/testguard/blob/main/docs/guides/ci/pre-commit.md) |
|
|
204
|
+
| **Reference** | [CLI](https://github.com/raccioly/testguard/blob/main/docs/reference/cli.md) · [Languages and runners](https://github.com/raccioly/testguard/blob/main/docs/reference/languages-and-runners.md) · [Configuration](https://github.com/raccioly/testguard/blob/main/docs/reference/configuration.md) · [Verdicts](https://github.com/raccioly/testguard/blob/main/docs/reference/verdicts.md) · [Artifacts](https://github.com/raccioly/testguard/blob/main/docs/reference/artifacts.md) · [MCP](https://github.com/raccioly/testguard/blob/main/docs/reference/mcp.md) · [Gate semantics](https://github.com/raccioly/testguard/blob/main/spec/GATE-SEMANTICS.md) |
|
|
205
|
+
| **Understand** | [How it works](https://github.com/raccioly/testguard/blob/main/docs/concepts/how-it-works.md) · [Glossary](https://github.com/raccioly/testguard/blob/main/docs/glossary.md) · [FAQ](https://github.com/raccioly/testguard/blob/main/docs/faq.md) · [Troubleshooting](https://github.com/raccioly/testguard/blob/main/docs/troubleshooting.md) · [Prior art](https://github.com/raccioly/testguard/blob/main/docs-canonical/PRIOR-ART.md) |
|
|
206
|
+
| **Translations** | [Português (Brasil)](https://github.com/raccioly/testguard/blob/main/docs/i18n/pt-BR/README.md) · [Español](https://github.com/raccioly/testguard/blob/main/docs/i18n/es/README.md) · [简体中文](https://github.com/raccioly/testguard/blob/main/docs/i18n/zh-CN/README.md) |
|
|
858
207
|
|
|
859
|
-
|
|
208
|
+
## What TestGuard is not
|
|
860
209
|
|
|
861
210
|
- **Not a test generator.** It judges a test the agent wrote (`admit`); the
|
|
862
|
-
generating half stays with the agent
|
|
211
|
+
generating half stays with the agent.
|
|
863
212
|
- **Not a mutation-score dashboard.** No blanket mutants, no single score, no
|
|
864
213
|
threshold. Faults are few and bound to stated claims; findings are ranked,
|
|
865
214
|
never summed.
|
|
866
|
-
- **Not a self-healing runner.** A defender that adapts itself to a change is
|
|
867
|
-
a defender that did not observe it; TestGuard reports that, it does not do
|
|
868
|
-
it.
|
|
869
215
|
- **Not a coverage tool.** A line executed says nothing about whether an
|
|
870
|
-
assertion would notice. `NOCOVER` here means no test even imports the
|
|
871
|
-
|
|
872
|
-
|
|
873
|
-
A claim that **disappeared** is invisible to all of the above: `probe`
|
|
874
|
-
verifies the claims that exist, `gate` sees the source file covered by some
|
|
875
|
-
other claim, and `status` says clean. Deleting a claim is therefore cheaper
|
|
876
|
-
than weakening its fault — and `changedFaults` exists precisely because
|
|
877
|
-
weakening was the cheap escape. So:
|
|
878
|
-
|
|
879
|
-
```bash
|
|
880
|
-
npx testguard-cli claims --since origin/main # what existed there and does not now
|
|
881
|
-
```
|
|
882
|
-
|
|
883
|
-
`removed-claim` and `removed-fault` gate; a rename that keeps the statement
|
|
884
|
-
verbatim is reported as `renamed-claim` and does not. Where evidence is
|
|
885
|
-
present, the line names the verdict the claim last had, because *"it was
|
|
886
|
-
SURVIVED when it was removed"* is the sentence that matters. The escape hatch
|
|
887
|
-
is the same one as everywhere else: a `claim` entry in
|
|
888
|
-
`testguard.ignore.json` with a reason, expiring if you give it an `expires`.
|
|
889
|
-
Removal is allowed — claims can be wrong, superseded or split — and it is
|
|
890
|
-
never silent.
|
|
891
|
-
|
|
892
|
-
### Probing a Python project
|
|
893
|
-
|
|
894
|
-
Nothing is installed into your project. TestGuard carries its own reporters and
|
|
895
|
-
puts them on the interpreter's `PYTHONPATH`, so a codebase whose test
|
|
896
|
-
dependencies are the standard library keeps none.
|
|
897
|
-
|
|
898
|
-
```bash
|
|
899
|
-
npx testguard-cli probe . # auto: pytest if importable, else stdlib unittest
|
|
900
|
-
npx testguard-cli probe . --runner unittest # pin the stdlib engine
|
|
901
|
-
npx testguard-cli probe . --python .venv/bin/python
|
|
902
|
-
```
|
|
903
|
-
|
|
904
|
-
The evidence records the engine that actually ran, and `--runner pytest` fails
|
|
905
|
-
rather than quietly using `unittest` — which engine ran changes what the
|
|
906
|
-
evidence means.
|
|
907
|
-
|
|
908
|
-
Three things behave differently from JavaScript, and each one is a place a
|
|
909
|
-
naive port would have produced a confident wrong answer:
|
|
910
|
-
|
|
911
|
-
- **Defenders are matched by module name**, the way Python itself matches, so
|
|
912
|
-
a test that reaches your package through `sys.path`, a `conftest.py` or an
|
|
913
|
-
installed distribution is found either way.
|
|
914
|
-
- **`patch("pkg.mod.fn")` is not `vi.mock`.** It replaces one attribute, so the
|
|
915
|
-
file still detects a fault anywhere else in that module: it stays a defender
|
|
916
|
-
and carries a `target-attribute-patched` signal naming the attributes. Only a
|
|
917
|
-
patch of the module *itself* removes a defender.
|
|
918
|
-
- **The fault has to be the code that ran.** A strict editable install puts an
|
|
919
|
-
import hook ahead of `sys.path`, so Python can load your original file while
|
|
920
|
-
TestGuard faults its copy — green baseline, every claim `SURVIVED`, and a
|
|
921
|
-
report that reads as a catastrophe while being entirely false. With the fault
|
|
922
|
-
applied TestGuard asks which file was really imported and **refuses the run**
|
|
923
|
-
if it came from outside the tree being probed. If no defender imported the
|
|
924
|
-
subject at all, the run continues and the record says so
|
|
925
|
-
(`detail.targetNotImported`): that is a fact about the tests' reach, not
|
|
926
|
-
about their assertions, and the two must not read alike.
|
|
927
|
-
|
|
928
|
-
`testguard scaffold path/to/module.py` proposes Python faults the same way it
|
|
929
|
-
does for JavaScript.
|
|
930
|
-
|
|
931
|
-
### Run it from any harness
|
|
932
|
-
|
|
933
|
-
The operating loop above lives in a Claude Code skill and a session-start
|
|
934
|
-
hook. Both vanish the moment the harness is Cursor, Codex, Devin or whatever
|
|
935
|
-
comes next — and the harness is exactly the layer most likely to change. So
|
|
936
|
-
the same documents are also served over the Model Context Protocol:
|
|
937
|
-
|
|
938
|
-
```bash
|
|
939
|
-
npx testguard-cli mcp # JSON-RPC 2.0 on stdio
|
|
940
|
-
npx testguard-cli init --mcp # prints the config for Claude Code, Cursor and Codex
|
|
941
|
-
```
|
|
942
|
-
|
|
943
|
-
Five tools, all **read-only**: `testguard_status`, `testguard_brief`,
|
|
944
|
-
`testguard_claims`, `testguard_evidence`, `testguard_next_command`.
|
|
945
|
-
|
|
946
|
-
**No tool runs a probe.** A probe is long-running, budgeted, and the person
|
|
947
|
-
should see it happen, so `next_command` hands back the exact shell line for
|
|
948
|
-
the agent to run in its own terminal. Nothing here writes a file either —
|
|
949
|
-
editing a claims file through a connector would defeat the point of recording
|
|
950
|
-
every fault edit. The server declares only a `tools` capability: no prompts,
|
|
951
|
-
no resources, no sampling.
|
|
952
|
-
|
|
953
|
-
It is hand-written against a pinned protocol version rather than built on the
|
|
954
|
-
official SDK, because this tool has one exact-pinned runtime dependency and
|
|
955
|
-
keeps it that way. The surface is three methods and five tools and will not
|
|
956
|
-
grow; tests speak the wire format to a real child process and check each
|
|
957
|
-
tool against the CLI's own `--json` output, so the two cannot drift.
|
|
958
|
-
|
|
959
|
-
### Built for agents to run
|
|
960
|
-
|
|
961
|
-
TestGuard is meant to be driven by an AI agent, not typed by a person. Three
|
|
962
|
-
things make that safe:
|
|
963
|
-
|
|
964
|
-
- **One source of truth.** `testguard status --json` computes `state` and the
|
|
965
|
-
one `next` action from the claims file, the evidence, the baseline and the
|
|
966
|
-
working tree. Every human rendering — the CLI text, the session-start
|
|
967
|
-
brief, the skill — derives from it, so they cannot disagree. Every command
|
|
968
|
-
accepts `--json`.
|
|
969
|
-
- **An installable operating loop.** `testguard init [dir]` writes the
|
|
970
|
-
**agent layer at the git root**, where agent sessions run —
|
|
971
|
-
`.claude/skills/testguard/SKILL.md` (state → action, verdict → the only
|
|
972
|
-
acceptable fix, the two-gate rule for any test the agent writes), the
|
|
973
|
-
`brief --text` session-start hook, an `AGENTS.md` section — and the
|
|
974
|
-
**project layer** (`.gitignore` lines) beside the claims file. A second
|
|
975
|
-
project in the same repository adds a hook line and an `AGENTS.md`
|
|
976
|
-
bullet; `--here` keeps everything in the subdirectory. The hook
|
|
977
|
-
resolves the project's `node_modules/.bin/testguard`, then the git root's,
|
|
978
|
-
then a `testguard` on `PATH` via `command -v`, then does nothing — **no
|
|
979
|
-
`npx` in any form, so no path to the network**; a pre-0.6 `npx -y` or
|
|
980
|
-
`npx --no-install` hook is replaced. A written file that
|
|
981
|
-
`.gitignore` swallows is reported, not offered for commit. Idempotent.
|
|
982
|
-
- **Independence is recorded (L3).** `probe` measures *power* — would this
|
|
983
|
-
test notice if the code were wrong. It also records, on every kill, whether
|
|
984
|
-
the killing test was last touched by the same change (or the same author) as
|
|
985
|
-
the code it guards: `detail.independence` is `co-authored`,
|
|
986
|
-
`separate-change` or `unknown`. A co-authored kill is legitimate — a fix
|
|
987
|
-
*should* ship with its regression test — but a repository where every kill
|
|
988
|
-
is co-authored has no independent verification, whatever its claim
|
|
989
|
-
verification rate says. It is a **signal**: ranking reads it, verdicts never
|
|
990
|
-
do, and `probe`, `status` and the brief all say it in one line. Nothing else in this category
|
|
991
|
-
measures it, and agents saturate the tests they can see
|
|
992
|
-
([SpecBench](https://arxiv.org/abs/2605.21384)).
|
|
993
|
-
- **Claimed surface is visible.** A clean probe is only about claims that
|
|
994
|
-
exist. `status` therefore reports claimed source modules over the current
|
|
995
|
-
module denominator, how many of the 20 highest-churn modules are claimed,
|
|
996
|
-
and the concrete unclaimed modules worth examining next over a bounded
|
|
997
|
-
200-commit window. Churn and path-risk signals stay separate; there is no
|
|
998
|
-
synthetic health score. When most modules remain unclaimed and the existing
|
|
999
|
-
evidence is clean, the next action expands the denominator instead of
|
|
1000
|
-
congratulating the same small set of claims again.
|
|
1001
|
-
- **Gaming is visible.** The cheapest way to make a survivor disappear is to
|
|
1002
|
-
weaken its fault, not to write a test. Evidence records every fault's
|
|
1003
|
-
content hash; `status` lists any fault edited after it survived, with its
|
|
1004
|
-
previous verdict, and makes reviewing that edit the next action. Editing a
|
|
1005
|
-
claim is allowed — claims can be wrong — but it is never invisible.
|
|
1006
|
-
|
|
1007
|
-
### Watching a probe that is still running
|
|
1008
|
-
|
|
1009
|
-
A probe runs the defenders N times clean and N times per fault, which takes
|
|
1010
|
-
minutes. It used to print its stage line only when `stderr` was a terminal —
|
|
1011
|
-
so CI, a redirected log and an agent harness saw **nothing at all** for the
|
|
1012
|
-
whole run, which is indistinguishable from a hang. A gate you cannot tell
|
|
1013
|
-
from a hang is a gate people start killing.
|
|
1014
|
-
|
|
1015
|
-
The rewriting is a rendering choice, not a reason to withhold the
|
|
1016
|
-
information:
|
|
1017
|
-
|
|
1018
|
-
| `--progress` | What you get |
|
|
1019
|
-
|---|---|
|
|
1020
|
-
| `auto` (default) | `tty` at a terminal, `plain` everywhere else |
|
|
1021
|
-
| `tty` | one line, rewritten in place — for a human watching |
|
|
1022
|
-
| `plain` | one append-only line per stage — a log file, CI, an agent |
|
|
1023
|
-
| `ndjson` | one JSON object per line, stages **and** verdicts as they land |
|
|
1024
|
-
| `none` | silence |
|
|
1025
|
-
|
|
1026
|
-
```bash
|
|
1027
|
-
npx testguard-cli probe . > probe.log 2>&1 # now reports; used to be silent
|
|
1028
|
-
npx testguard-cli probe . --progress ndjson 2>events.ndjson
|
|
1029
|
-
```
|
|
1030
|
-
|
|
1031
|
-
```
|
|
1032
|
-
{"event":"stage","claim":"TG-COST-NAMES-SHARED-DEFENDERS","fault":"F1","stage":"baseline","run":1,"of":3,"at":"…"}
|
|
1033
|
-
{"event":"verdict","claim":"TG-COST-NAMES-SHARED-DEFENDERS","fault":"F1","verdict":"killed","severity":"medium","file":"src/probe/cost.mjs","at":"…"}
|
|
1034
|
-
```
|
|
1035
|
-
|
|
1036
|
-
**Progress always goes to stderr**, so `--json` leaves stdout as one document
|
|
1037
|
-
and nothing else. `--quiet` and `--json` imply `none`, and an explicit
|
|
1038
|
-
`--progress` overrides both: an operator who asks for a stream of events has
|
|
1039
|
-
said what they want.
|
|
1040
|
-
|
|
1041
|
-
### What the probe costs, and why a gate gets slow
|
|
1042
|
-
|
|
1043
|
-
A fresh probe grows with the selected faults, confirmation count and the cost
|
|
1044
|
-
of their defenders. Shared baselines and valid verdict reuse avoid some repeated
|
|
1045
|
-
work. Roughly, the fresh work is
|
|
1046
|
-
|
|
1047
|
-
```
|
|
1048
|
-
fresh baseline runs × defender-set cost
|
|
1049
|
-
+ Σ (fault confirmations × that fault's defender-set cost)
|
|
1050
|
-
+ survivor diagnostics and setup/discovery work
|
|
1051
|
-
```
|
|
1052
|
-
|
|
1053
|
-
One slow acceptance test selected by several faults can therefore be executed
|
|
1054
|
-
many times. The symptom can be a gate that takes twenty minutes; the selected
|
|
1055
|
-
defender sets determine how much each fault costs. Historical attributed totals
|
|
1056
|
-
are reported separately from measured fresh elapsed time and runner invocations.
|
|
1057
|
-
|
|
1058
|
-
```bash
|
|
1059
|
-
npx testguard-cli claims . --cost # read back from the last probe's evidence
|
|
1060
|
-
npx testguard-cli probe . --cost # what the run you just made spent
|
|
1061
|
-
```
|
|
1062
|
-
|
|
1063
|
-
```
|
|
1064
|
-
63 fault records cost 24m across 378 defender runs.
|
|
1065
|
-
|
|
1066
|
-
most expensive claims
|
|
1067
|
-
4m 58s TG-FAULT-EDIT-VISIBLE 1 fault · 6 runs
|
|
1068
|
-
|
|
1069
|
-
defender files, by the time of the runs that included them
|
|
1070
|
-
(a fault runs its selected defender set at once, so these overlap and do not sum to the total)
|
|
1071
|
-
24m 3s test/probe.fixture.test.mjs 5 claims
|
|
1072
|
-
|
|
1073
|
-
shared defenders — each fault selects the file; durations include recorded baseline sharing
|
|
1074
|
-
24m 3s test/probe.fixture.test.mjs
|
|
1075
|
-
named by TG-ESCALATION-N-RUNS/F1, TG-FAULT-EDIT-VISIBLE/F1, …
|
|
1076
|
-
```
|
|
1077
|
-
|
|
1078
|
-
Every number is read back out of the `durationMs` the probe already records.
|
|
1079
|
-
`--cost` re-measures nothing, spawns nothing, and costs the price of reading
|
|
1080
|
-
one JSON file.
|
|
1081
|
-
|
|
1082
|
-
Cost output also lists individual faults, their selected defenders and selection
|
|
1083
|
-
origin. Sharing is counted across distinct fault identities, including siblings
|
|
1084
|
-
within one claim; repeated confirmation runs alone are not sharing.
|
|
1085
|
-
|
|
1086
|
-
**The per-file figures overlap.** A run executes a fault's whole defender set
|
|
1087
|
-
at once, so the runner never says how much of it belonged to which file. A
|
|
1088
|
-
file's `ms` is the time of every run that *included* it: an upper bound on
|
|
1089
|
-
what removing it could save, which is the decision you are making. Only
|
|
1090
|
-
`totalMs` is additive.
|
|
1091
|
-
|
|
1092
|
-
**What to do about a shared defender.** Extract the decision it proves into a
|
|
1093
|
-
pure function, unit-test that, and point the claim at the unit test. This is
|
|
1094
|
-
not a testing trick — it is the same reason `classify()` in this codebase is
|
|
1095
|
-
pure. A rule you can state in three lines should not need a fifty-second
|
|
1096
|
-
acceptance test to falsify it. When the extraction is right the claim still
|
|
1097
|
-
fails on the fault; when it is wrong the claim SURVIVES, and the probe tells
|
|
1098
|
-
you so before you have shipped a weaker gate. Do not narrow `defendedBy` and
|
|
1099
|
-
assume — re-probe the claim and read the verdict.
|
|
1100
|
-
|
|
1101
|
-
### Authoring faults mechanically
|
|
1102
|
-
|
|
1103
|
-
Writing faults by hand means reading the code to find exact anchors. Two
|
|
1104
|
-
field reports found that ~80% of hand-written faults are one of nine shapes,
|
|
1105
|
-
so `scaffold` proposes them for you:
|
|
1106
|
-
|
|
1107
|
-
```bash
|
|
1108
|
-
npx testguard-cli scaffold src/auth.ts # → .testguard/scaffold-auth.json (a draft, never your claims file)
|
|
1109
|
-
npx testguard-cli scaffold src/auth.ts --claim AUTH-ADMIN # every proposal under one claim; copies it if it exists
|
|
1110
|
-
npx testguard-cli scaffold src/auth.ts src/session.ts --into draft.json --claim AUTH-ADMIN --json # read-only merged preview
|
|
1111
|
-
npx testguard-cli scaffold src/auth.ts src/session.ts --into draft.json --claim AUTH-ADMIN # explicit recoverable draft update
|
|
1112
|
-
npx testguard-cli scaffold --from-document requirements.md --json # read-only input metadata, no document text
|
|
1113
|
-
npx testguard-cli scaffold --from-fix <full-commit-ID> --json # read-only fixed historical inventory
|
|
1114
|
-
```
|
|
1115
|
-
|
|
1116
|
-
Input inspection uses exactly one `--from-document` or `--from-fix`, with no
|
|
1117
|
-
positional sources or other options except JSON/help/version. Fix IDs must be
|
|
1118
|
-
full lowercase SHA-1/SHA-256 commit IDs with exactly one parent. Plain-text
|
|
1119
|
-
documents are bounded to 2 MiB; local Git inspection is offline and bounded.
|
|
1120
|
-
Both output modes are stdout-only and say verification was not performed.
|
|
1121
|
-
Metadata retains historical exclusions, not private/delegated filenames.
|
|
1122
|
-
Input text and commit subjects are untrusted data, not instructions or an
|
|
1123
|
-
authenticated claim. Supply independent intended behavior, review faults and
|
|
1124
|
-
probe afterward; inspection does not create claims or prove coverage.
|
|
1125
|
-
|
|
1126
|
-
Without a supplied ID or annotation, generated groups use unfinished
|
|
1127
|
-
`TODO-CLAIM-N` placeholders rather than module/function names. Replace them with
|
|
1128
|
-
intent-shaped identities when authoring; mechanical grouping does not establish
|
|
1129
|
-
an invariant boundary. Existing claims/evidence are never automatically renamed.
|
|
1130
|
-
|
|
1131
|
-
`--into` requires an existing disposable draft and exactly one existing claim
|
|
1132
|
-
ID. It preserves supplied intent, metadata and prior faults; multiple source
|
|
1133
|
-
paths append in deterministic order. `--json` is a read-only preview. Without
|
|
1134
|
-
it, the tool keeps private recovery outside the checkout and reports updated
|
|
1135
|
-
or unchanged **unproven** draft status. Review and probe are still required.
|
|
1136
|
-
Canonical claims/evidence/baseline paths, unsafe aliases and conflicting options
|
|
1137
|
-
are refused; partial writes never become success or trigger automatic rollback.
|
|
1138
|
-
Limits: 32 sources, 2 MiB per input/output draft or source, 16 MiB source total,
|
|
1139
|
-
4096 generated proposals. These limits are not a performance guarantee.
|
|
1140
|
-
|
|
1141
|
-
| Shape | What it proposes |
|
|
1142
|
-
|---|---|
|
|
1143
|
-
| `condition-forced` | `if (<guard>) {` → `if (false) {` — a guard is a `!…` condition or one whose body returns, throws or 4xx-es |
|
|
1144
|
-
| `statement-deleted` | a single-line guard (`if (…) return …;`) or a state change (`x = …;`) removed |
|
|
1145
|
-
| `return-altered` | `return <check>;` (`===`, `.includes(`, `&&`, …) → `return true;` |
|
|
1146
|
-
| `literal-changed` | `httpOnly`/`secure` flipped, `sameSite` → `none`, a cost/rounds → `1`, a ttl/tolerance/window/limit ×1000 |
|
|
1147
|
-
| `call-removed` | a bare `verify…()` / `validate…()` / `check…()` / `authorize…()` call removed |
|
|
1148
|
-
| `field-dropped` | a field removed from an object that is returned, built by an arrow, assigned to a payload-ish name, passed to a `save`/`update`/`send`/`write`/… call, or is a `z.object({…})`-style schema; a string entry removed from an allow-list array; a `...base,` line or inline `{ ...base, … }` merge dropped. Only a line that can go on its own (balanced, comma-terminated or followed by the closer); never inside tests, fixtures or migrations |
|
|
1149
|
-
| `argument-swapped` | a call kept, its first argument swapped for `undefined` (and `{}` when the argument is itself a call) — only when that argument is derived from a parameter of the enclosing function or a request-like value (`req`, `ctx`, `event`, …). The seam fault: `decide(deriveFrom(input), now)` → `decide(undefined, now)` |
|
|
1150
|
-
| `element-removed` | a one-line JSX element — self-closing (`<Toggle … />`) or paired (`<button …>Save</button>`) — removed; the UI shape behind "the toggle is invisible", killable by a browser-layer defender |
|
|
1151
|
-
| `handler-dropped` | an `on<Event>={…}` prop removed, whether it is its own line or inline in the tag; the control renders and does nothing |
|
|
1152
|
-
|
|
1153
|
-
The table above is JavaScript. `scaffold` reads Python too, and proposes the
|
|
1154
|
-
same fault classes in Python syntax: `if <guard>:` → `if False:`,
|
|
1155
|
-
`return <check>` → `return True`, `verify=True` → `verify=False`, a
|
|
1156
|
-
parameter-derived argument swapped for `None`, a key dropped from a payload
|
|
1157
|
-
`dict` or an allow-list. Three differences are deliberate. A statement is
|
|
1158
|
-
removed by replacing it with `pass`, never by deleting the line, because a
|
|
1159
|
-
block whose only statement is gone is an `IndentationError`; and a line that
|
|
1160
|
-
leaves a bracket open (`COLOURS = {`), or sits inside one opened on an earlier
|
|
1161
|
-
line (`help="…"` in an `add_argument(`), is never removed at all. Both exist
|
|
1162
|
-
because a fault that cannot compile is a `fault-invalid` verdict — a probe run
|
|
1163
|
-
spent saying nothing about the tests. Third, a **module-level dunder
|
|
1164
|
-
assignment** (`__version__`, `__all__`, `__author__`) is not proposed. Deleting
|
|
1165
|
-
one does change behaviour, so it is not an equivalent mutant; it is something
|
|
1166
|
-
nobody writes a test for, which is the disqualification an icon already gets on
|
|
1167
|
-
the JSX side — a survivor there teaches the survival-learned ordering to prefer
|
|
1168
|
-
a barren class. The rule is narrow on purpose: a module-level *constant*
|
|
1169
|
-
(`DEFAULT_MAX_SIZE = 100`) is still proposed, because removing a real default
|
|
1170
|
-
changes a real default. The two JSX shapes (`element-removed`,
|
|
1171
|
-
`handler-dropped`) have no Python meaning and are absent rather than faked.
|
|
1172
|
-
|
|
1173
|
-
Every proposal's `find` is the exact line with `expectHits`/`occurrence`
|
|
1174
|
-
computed from the file, so it is verifiable by construction; provenance is
|
|
1175
|
-
`producer: derived`; `defendedBy` is prefilled from the tests that import
|
|
1176
|
-
the module; proposals are grouped under a preceding `@claim <ID>` annotation
|
|
1177
|
-
or by enclosing function. Statements are `TODO:` placeholders — a proposal
|
|
1178
|
-
becomes a claim only when a human states what it defends. Deterministic
|
|
1179
|
-
heuristics, no AST, no LLM; a proposal the tool cannot anchor is never
|
|
1180
|
-
emitted.
|
|
1181
|
-
|
|
1182
|
-
**Commit `.testguard/baseline.json`; ignore `evidence.json` and `brief.json`.**
|
|
1183
|
-
The baseline is the frozen contract; the other two are regenerated per run.
|
|
1184
|
-
|
|
1185
|
-
The fault model is the auditable artifact. You never reach 100% of
|
|
1186
|
-
correctness; you reach **100% of stated claims verified**, and the statement
|
|
1187
|
-
of claims is what an assessor reads. A claims file is code — its `replace`
|
|
1188
|
-
strings run under your test runner — so review it like code.
|
|
216
|
+
assertion would notice. `NOCOVER` here means no test even imports the file.
|
|
217
|
+
- **Not a network client.** The CLI never opens a socket and collects no
|
|
218
|
+
telemetry. One exact-pinned runtime dependency (`ajv`), Node ≥ 20, MIT.
|
|
1189
219
|
|
|
1190
220
|
## Try it
|
|
1191
221
|
|
|
1192
222
|
The repository ships a known-answer fixture with a real blind spot:
|
|
1193
223
|
|
|
1194
224
|
```bash
|
|
1195
|
-
git clone
|
|
225
|
+
git clone https://github.com/raccioly/testguard && cd testguard && npm install
|
|
1196
226
|
npm test # includes probing the fixture end to end
|
|
1197
227
|
```
|
|
1198
228
|
|
|
1199
229
|
`fixtures/known-answer/` is a tiny project whose audit-row test asserts with
|
|
1200
230
|
`expect.objectContaining({...})` and omits the `content` key. Swap the
|
|
1201
231
|
redacted text for the raw input and the test stays green. `probe` reports it
|
|
1202
|
-
as `SURVIVED`; the fixture's [README](fixtures/known-answer/README.md)
|
|
1203
|
-
through every verdict.
|
|
1204
|
-
|
|
1205
|
-
|
|
1206
|
-
|
|
1207
|
-
run under **both** stdlib `unittest` and `pytest`, which have different report
|
|
1208
|
-
shapes and different notions of failure and must still agree on all sixteen
|
|
1209
|
-
verdicts.
|
|
232
|
+
as `SURVIVED`; the fixture's [README](https://github.com/raccioly/testguard/blob/main/fixtures/known-answer/README.md)
|
|
233
|
+
walks through every verdict. The same blind spot in Python is
|
|
234
|
+
[`fixtures/known-answer-python/`](https://github.com/raccioly/testguard/blob/main/fixtures/known-answer-python/README.md),
|
|
235
|
+
run under both stdlib `unittest` and `pytest`, which must agree on every
|
|
236
|
+
verdict.
|
|
1210
237
|
|
|
1211
238
|
## Status
|
|
1212
239
|
|
|
1213
|
-
|
|
1214
|
-
|
|
1215
|
-
|
|
1216
|
-
|
|
1217
|
-
|
|
1218
|
-
|
|
1219
|
-
|
|
1220
|
-
|
|
1221
|
-
|
|
1222
|
-
|
|
1223
|
-
|
|
1224
|
-
|
|
1225
|
-
|
|
1226
|
-
current bindings exit 2. The command rechecks native runner/configuration
|
|
1227
|
-
membership, current claims, targets, defenders and discovery dependencies before
|
|
1228
|
-
publishing. Custom/static runner universes cannot certify this boundary. This is
|
|
1229
|
-
a declaration check, not authenticated independence or an atomic disk snapshot.
|
|
1230
|
-
Offline status recomputes a current-declaration audit using the recorded eligible
|
|
1231
|
-
selection, always marking native freshness unavailable without launching a
|
|
1232
|
-
runner. It does not implicitly activate a new gate or replace status's next
|
|
1233
|
-
action. Briefs preserve the recorded-run policy before caps and baseline
|
|
1234
|
-
filtering, explicitly not current freshness. A zero-item cap never turns
|
|
1235
|
-
survivors into an all-defended message.
|
|
1236
|
-
|
|
1237
|
-
**v0.5.** Eleven commands (`status`, `init`, `claims`, `probe`, `admit`,
|
|
1238
|
-
`replay`, `baseline`, `brief`, `gate`, `scaffold`, `sweep`, `mcp`), vitest, jest,
|
|
1239
|
-
Playwright and Python (pytest / stdlib unittest) runners, hand-authored faults
|
|
1240
|
-
plus a mechanical scaffold for JavaScript and Python, an agent operating layer
|
|
1241
|
-
(`status`, `init`) and a change gate (`gate`). The contract spine — eight JSON
|
|
1242
|
-
Schemas shared with the other Guard tools — is under [`spec/`](spec/). One
|
|
1243
|
-
exact-pinned runtime dependency (`ajv`, for schema validation); Node ≥ 20, and
|
|
1244
|
-
Python ≥ 3.8 only when probing Python.
|
|
1245
|
-
|
|
1246
|
-
Not yet: test generation (the acceptance half, `admit`, exists; the generating
|
|
1247
|
-
half stays the agent's), runners beyond those four, AST-aware producers, and
|
|
1248
|
-
the transfer of a calibration between repositories — `replay` measures it now;
|
|
240
|
+
Thirteen commands (`status`, `init`, `claims`, `probe`, `admit`, `baseline`,
|
|
241
|
+
`brief`, `gate`, `scaffold`, `sweep`, `concerns`, `replay`, `mcp`). Runners
|
|
242
|
+
for vitest, jest, Playwright, Node's built-in test runner, pytest and stdlib
|
|
243
|
+
`unittest`, plus `--runner-cmd` for anything that writes a jest-compatible
|
|
244
|
+
report. Hand-authored faults and a mechanical scaffold for JavaScript,
|
|
245
|
+
TypeScript and Python. The contract is fourteen JSON Schemas under
|
|
246
|
+
[`spec/`](https://github.com/raccioly/testguard/blob/main/spec/README.md),
|
|
247
|
+
shared with the other Guard tools. Every release is in the
|
|
248
|
+
[changelog](https://github.com/raccioly/testguard/blob/main/CHANGELOG.md).
|
|
249
|
+
|
|
250
|
+
Not yet: test generation (the acceptance half, `admit`, exists; the
|
|
251
|
+
generating half stays the agent's), AST-aware producers, and the transfer of
|
|
252
|
+
a calibration between repositories. `replay` measures that transfer now;
|
|
1249
253
|
whether it carries to a repository with no history is unproven. Each is
|
|
1250
254
|
designed for; none is claimed.
|
|
1251
255
|
|
|
1252
|
-
|
|
1253
|
-
|
|
1254
|
-
Built-in probe, sweep, replay and admission runners default to one worker. Use
|
|
1255
|
-
`--workers 2` for a larger Vitest/Jest/Playwright worker ceiling; `--serial` always
|
|
1256
|
-
uses one. Python remains serial. Faults and mixed-runner groups run sequentially.
|
|
1257
|
-
Custom `--runner-cmd` commands, browser subprocesses, native library threads and
|
|
1258
|
-
test-created processes are not an OS resource sandbox. The regression suite also
|
|
1259
|
-
caps outer workers at two, or one when fewer than three CPUs are available
|
|
1260
|
-
in an interactive run; dedicated CI uses at most two because its acceptance tests start nested runners.
|
|
1261
|
-
|
|
1262
|
-
`--budget` bounds each runner invocation; `--command-budget` bounds a complete
|
|
1263
|
-
command cooperatively and refuses partial result publication. Cancellation kills
|
|
1264
|
-
owned runners before restoring mutations and removing scratch worktrees. Already
|
|
1265
|
-
reparented daemons remain outside the portable cleanup guarantee. No runner output
|
|
1266
|
-
limit can count as an assertion failure or detection.
|
|
1267
|
-
|
|
1268
|
-
New probe evidence includes actual elapsed time, unique fresh runner time and
|
|
1269
|
-
setup/discovery/other overhead. `probe --cost` and `claims --cost` display this
|
|
1270
|
-
separately from historical attributed record costs. Existing cost ceilings are
|
|
1271
|
-
unchanged; old evidence does not supply a measured breakdown.
|
|
256
|
+
## Contributing
|
|
1272
257
|
|
|
1273
|
-
|
|
1274
|
-
|
|
1275
|
-
|
|
1276
|
-
|
|
258
|
+
Bug reports, field reports and documentation fixes are welcome. Read
|
|
259
|
+
[CONTRIBUTING.md](https://github.com/raccioly/testguard/blob/main/CONTRIBUTING.md)
|
|
260
|
+
and [AGENTS.md](https://github.com/raccioly/testguard/blob/main/AGENTS.md)
|
|
261
|
+
first; for help, see [SUPPORT.md](https://github.com/raccioly/testguard/blob/main/SUPPORT.md),
|
|
262
|
+
and report vulnerabilities privately per
|
|
263
|
+
[SECURITY.md](https://github.com/raccioly/testguard/blob/main/SECURITY.md).
|
|
1277
264
|
|
|
1278
265
|
## Licence
|
|
1279
266
|
|