testguard-cli 0.18.2 → 0.19.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -7,8 +7,10 @@
7
7
  [![npm](https://img.shields.io/npm/v/testguard-cli.svg)](https://www.npmjs.com/package/testguard-cli)
8
8
  [![PyPI](https://img.shields.io/pypi/v/testguard-cli.svg)](https://pypi.org/project/testguard-cli/)
9
9
  [![node](https://img.shields.io/node/v/testguard-cli.svg)](https://nodejs.org)
10
- [![license](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
11
- [![deps](https://img.shields.io/badge/runtime%20deps-1%20pinned-brightgreen.svg)](./package.json)
10
+ [![license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/raccioly/testguard/blob/main/LICENSE)
11
+ [![deps](https://img.shields.io/badge/runtime%20deps-1%20pinned-brightgreen.svg)](https://github.com/raccioly/testguard/blob/main/package.json)
12
+
13
+ **English** · [Português (Brasil)](https://github.com/raccioly/testguard/blob/main/docs/i18n/pt-BR/README.md) · [Español](https://github.com/raccioly/testguard/blob/main/docs/i18n/es/README.md) · [简体中文](https://github.com/raccioly/testguard/blob/main/docs/i18n/zh-CN/README.md)
12
14
 
13
15
  > Breaks your code on purpose and reports every promise your tests did not
14
16
  > notice breaking.
@@ -80,7 +82,7 @@ risk. TestGuard binds every fault to a **stated claim**, so the output is not
80
82
  a score but a finding: *"Your project says a missing scope fails closed.
81
83
  Nothing checks that."* That is the difference between a metric and an audit.
82
84
 
83
- 📄 **[Read the six-page technical brief (PDF)](docs/testguard-explained.pdf)**
85
+ 📄 **[Read the six-page technical brief (PDF)](https://github.com/raccioly/testguard/blob/main/docs/testguard-explained.pdf)**
84
86
  — the idea on page one, then the field evidence, the anatomy of a run, the
85
87
  verdicts, the loop and its calibration, and the prior art.
86
88
 
@@ -127,1153 +129,138 @@ raises coverage. TestGuard admits it because it fails when the claim is false.
127
129
  | npm | `npm i -D testguard-cli` then `npx testguard probe` |
128
130
  | pip | `pip install testguard-cli` then `testguard probe` (needs Node ≥ 20) |
129
131
  | Homebrew | `brew tap raccioly/tap && brew install testguard` |
130
- | GitHub Action | `uses: raccioly/testguard@v0.18.2` — see [`action.yml`](./action.yml) |
131
- | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-probe` |
132
- | GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.18.2/packaging/gitlab/testguard.gitlab-ci.yml` with `inputs:` — see [`packaging/gitlab/`](./packaging/gitlab/testguard.gitlab-ci.yml) |
132
+ | GitHub Action | `uses: raccioly/testguard@v0.19.0` |
133
+ | pre-commit | `repo: https://github.com/raccioly/testguard`, hooks `testguard-claims`, `testguard-gate`, `testguard-probe` |
134
+ | GitLab CI | `include: - remote: https://raw.githubusercontent.com/raccioly/testguard/v0.19.0/packaging/gitlab/testguard.gitlab-ci.yml` |
133
135
 
134
- Projects that set `min-release-age` in `.npmrc` cannot see a version published
135
- less than that many days ago (`ENOVERSIONS`); install that one with
136
- `npm i -D testguard-cli --min-release-age=0`.
136
+ Requirements, offline installs, `ENOVERSIONS` and every CI option are in the
137
+ [installation guide](https://github.com/raccioly/testguard/blob/main/docs/installation.md).
137
138
 
138
- ## Usage
139
+ ## Quickstart
139
140
 
140
141
  ```bash
141
142
  npx testguard-cli init # install the agent layer at the git root
142
143
  npx testguard-cli status --json # where the project is, and the ONE next action
143
- npx testguard-cli claims # what does this project claim, and is each claim probeable?
144
- npx testguard-cli claims --check-anchors # fail fast if an exact anchor moved or a replacement no longer parses
145
- npx testguard-cli claims --annotate --claim AUTH-ADMIN --json # read-only file-header placement preview
146
144
  npx testguard-cli scaffold src/auth.ts # propose faults for a file, as a draft to keep or drop
145
+ npx testguard-cli claims --check-anchors # every claim is probeable before anything runs
147
146
  npx testguard-cli probe # try to falsify each claim; report what the tests missed
148
- npx testguard-cli admit test/auth.test.ts --claim AUTH-ADMIN # does this test satisfy the two-gate rule?
147
+ npx testguard-cli admit test/auth.test.ts --claim AUTH-ADMIN # does this test fail when the claim is false?
149
148
  npx testguard-cli baseline # freeze today's unproven findings; only new ones gate
150
149
  npx testguard-cli gate --changed origin/main # fail when a changed source file carries no claim
151
- npx testguard-cli sweep --changed origin/main # no claims yet? propose faults for the changed files and report what nothing noticed
150
+ npx testguard-cli sweep --changed origin/main # no claims yet? report what nothing noticed in the changed files
152
151
  npx testguard-cli brief --text # tell the agent where the suite is blind, before it writes
153
- npx testguard-cli replay --since v1.0..HEAD # would this suite have caught the bugs that escaped?
154
- npx testguard-cli mcp # serve the read-only loop over MCP, on stdio
152
+ npx testguard-cli replay --since HEAD~50..HEAD # would this suite have caught the bugs that escaped?
155
153
  ```
156
154
 
157
155
  Exit codes are the contract: `0` nothing new to prove, `1` unproven claims,
158
156
  unclaimed changes or invalid fault anchors, `2` a precondition failed and
159
- nothing was probed, `3` usage.
160
- Every command accepts `--json`.
161
-
162
- ### Intent-first drafts
163
-
164
- Scaffold drafts propose mechanical faults, not intended behavior. New claims
165
- declare `inferred` origin even with an annotation. Supply the intended observable
166
- behavior from a requirement, ADR, bug or incident independently of the code:
167
- inputs, expected outcome and forbidden outcome. Record the supplied source
168
- kind/ref; when unavailable keep inferred. References are not authenticated or
169
- fetched, and passing tests do not establish independent intent. Review proposals
170
- against the supplied intent before adopting and probing; existing `--claim`
171
- metadata is preserved.
172
-
173
- ### Optional annotation placement
174
-
175
- Ordinary `claims` output and status notes identify missing scanned source links
176
- as advisories. Counts are distinct claim IDs, not faults. They do not change
177
- verification state, its next action or exit codes; annotation-sourced missing
178
- IDs still produce the existing claims drift error. Scanned text is a lexical
179
- link, not authenticated intent or source ownership.
180
- Status advice uses a bounded scan (10,000 entries, 1,000 files, 64 directory
181
- levels, 256 KiB per file, 2 MiB total); incomplete scans report unavailable
182
- advice without concluding an ID is absent. Keep the original `--claims`
183
- argument for preview when inspecting a custom claims selection.
184
-
185
- `testguard claims --annotate` previews JavaScript/TypeScript or Python
186
- file-header links for declared claims; repeat `--claim` or use comma-separated
187
- IDs to select claims. Add `--apply` only after reviewing the selection to
188
- explicitly write annotations. Each invocation recomputes its plan: an earlier
189
- preview is not a saved approval token. No `--out` or unrelated inspection
190
- switches are accepted with authoring. JSON conforms to the `annotations`
191
- authoring schema; it contains no source/proposed contents or probe verdicts.
192
-
193
- This is file-level placement, not symbol ownership or proof of correctness.
194
- Missing annotations do not disable exact fault-anchor validation, and placement
195
- does not change claim origins, evidence or baselines. Unsupported, ambiguous or
196
- unsafe targets block the complete selection. Original/proposed files are capped
197
- at 2 MiB and the aggregate at 16 MiB. Preserve BOM, shebang, Python encoding
198
- cookie, newline style and permissions. A preview exits 0 when applicable, 2 when
199
- refused; apply exits 0 only after verified completion, 2 for refusal/partial
200
- failure; malformed options exit 3.
201
-
202
- Apply uses an exclusive owned lock and retains fsynced originals in an owner-only
203
- recovery directory outside the checkout (the result reports its location).
204
- External claims files can be previewed but cannot authorize writes. It never
205
- reclaims an existing lock or automatically rolls back over editor changes.
206
- This is recoverable, not a cross-file atomic transaction: interruption or
207
- arbitrary concurrent edits can leave partial content. Compare retained originals
208
- manually before recovery; inspect ownership before removing a leftover lock.
209
-
210
- ## How it works
211
-
212
- ```bash
213
- npx testguard-cli init # install the agent layer: skill, session-start hook, AGENTS.md section
214
- npx testguard-cli status --json # where the project is and the ONE next action — the machine entry point
215
- npx testguard-cli claims # what does this project claim, and is every claim probeable?
216
- npx testguard-cli probe # try to falsify each claim; report what the tests missed
217
- npx testguard-cli baseline # freeze today's unproven findings; from now on only new ones gate
218
- npx testguard-cli brief # tell the agent where the suite is blind, before it writes
219
- npx testguard-cli gate --changed origin/main # fail when a changed source file carries no claim at all
220
- npx testguard-cli scaffold src/x.ts # propose faults for a file, as a draft to keep or drop
221
- npx testguard-cli admit test/x.test.ts --claim X # is this test green on HEAD and does it fail on every fault of X? ADMITTED or NOT ADMITTED
222
- ```
223
-
224
- 1. **Claims** live in `testguard.claims.json` (editors validate it against
225
- `"$schema": "./node_modules/testguard-cli/spec/schemas/claims.schema.json"`):
226
- a statement, where it comes from, which tests supposedly defend it, and
227
- one or more *faults* — each a deterministic source change that would make
228
- the statement false. Every
229
- claim and every fault records who produced it. `testguard claims`
230
- validates the file and reports drift against `@claim <ID>` annotations in
231
- source. Test files are deliberately not scanned — a claim asserted by a test is the authorship trap the tool exists for — and annotation ids must contain a hyphen so prose is never mistaken for one.
232
- `claims --check-anchors` additionally locates every exact anchor and parses
233
- each in-memory JavaScript or Python replacement. It runs no tests, creates
234
- no worktree and changes no file, so a moved anchor is reported before an
235
- expensive probe. It reports and stops; it never guesses a new anchor.
236
- 2. **Probe** confirms the defenders are green N times unmodified, applies
237
- each fault in a scratch git worktree (your tree is never touched), runs
238
- the defenders N times, re-runs survivors against the whole suite with
239
- N-run attribution (up to N: it stops as soon as no test has failed in
240
- every run, because from there none can be named a killer), restores, and
241
- classifies. It warns when another test runner is already running — a
242
- contended machine turns a slow suite into a `TIMEOUT` verdict about the
243
- load, not the claim — records that on the evidence, and offers `--serial`
244
- to run one test file at a time. Verdicts are a closed set:
245
-
246
- | Verdict | Meaning |
247
- |---|---|
248
- | `killed` | a test body rejected the behaviour, N/N — the only pass |
249
- | `SURVIVED` | the defenders stayed green while the claim was false |
250
- | `NOCOVER` | no test file defends the claim at all |
251
- | `UNVERIFIABLE` | the fault's anchor is missing or ambiguous — loud, never a skip |
252
- | `TIMEOUT` | the defenders hung; a hang is not a detection |
253
- | `FAULT-INVALID` | the replacement does not load — a bad fault, not a finding |
254
- | `FLAKY-DEFENDER` | the defenders are not reliably green, or disagreed across runs — `detail.flakeRate` says how often (`failures` of `runs` on unmodified source) |
255
-
256
- Never a single score. Findings are ranked by severity, claim provenance
257
- and blast radius (relative imports, `tsconfig` path aliases and
258
- `package.json#imports` are resolved; bare package names are not), and
259
- written to `.testguard/evidence.json` — validated against the spec before
260
- it is written.
261
-
262
- Worktree mode probes a **commit**. If a defender or target file has
263
- uncommitted changes and you asked for the implicit HEAD, `probe` refuses
264
- and says so — otherwise your new tests would be silently absent and the
265
- same survivors would come back with no hint why. `--include-dirty`
266
- snapshots the working tree (tracked edits and new files) into a throwaway
267
- commit and probes that; your tree, HEAD and index are never touched. An
268
- **explicit `--ref`** (a pre-fix commit in a post-mortem, say) is honoured
269
- over a dirty tree, as is `--ignore-dirty`: a warning names the files and
270
- the evidence records them as `repo.ignoredDirty`, so the run says what it
271
- did not look at. Every summary names the commit probed.
272
-
273
- A fault may declare its own `defendedBy` to override the claim's set.
274
- Omit the fault field to inherit; set it to `[]` to request discovery rather
275
- than inheritance. Evidence names the selection origin and selected files.
276
- Changing either invalidates reuse and makes status stale. A narrowed fault
277
- set after a prior kill is warned about by `claims`; re-probe before trusting
278
- it. For example, a unit-level fault can select `["test/unit.test.mjs"]`
279
- while an integration fault inherits the claim's broader set.
280
-
281
- A claim with no `defendedBy` has its defenders **discovered**: the test
282
- files that import the fault's target, by relative path or resolved alias
283
- (tsconfig `paths`, through `extends` and `references`, vite/vitest
284
- `resolve.alias`, package.json `imports`) — **minus the files that mock
285
- it**. A test that `vi.mock`s / `jest.mock`s the target cannot detect any
286
- fault in it; counting it would make `NOCOVER` under-report and waste runs.
287
- `NOCOVER` therefore means exactly "no test file imports this source
288
- without mocking it". `claims` prints the split (`18 import · 16 mock · 2
289
- can detect`), the evidence lists the mocking files, and a mocking file
290
- that never `expect(...)`s anything imported from the target carries the
291
- static signal `mocked-never-asserted` — the cheapest blind-spot signal
292
- there is, and the exact signature of one escaped bug in the field
293
- reports. Silence it, visibly, with `// unasserted: <why>` above the mock.
294
- Its sibling `persistence-payload-unasserted` is attached to a **survivor
295
- on a write path** whose defender mocks the database: the test can prove
296
- the call shape and never that the row landed, and the signal names the
297
- mocked layer and the file's assertion mix (exact, partial, argument-free).
298
-
299
- Runners: **vitest**, **jest**, **Playwright** and **Python** (`--runner
300
- auto` picks the first of vitest, jest, python that resolves). A JavaScript
301
- runner is resolved from the **project's own package** first — its pinned
302
- version, its own bin script — and only then from an executable on PATH,
303
- which the evidence records as `runner.source: "path"`; `npx` is never
304
- asked, because its cache answers for packages a project does not have.
305
- **Node's built-in framework** uses explicit `--runner node-test` with plain
306
- JavaScript test files and default Node collection. It uses the Node executable
307
- running TestGuard, records `runner.source: "builtin"` and its actual version,
308
- and installs nothing. Discovery loads test modules with test bodies filtered
309
- out, so top-level code can run. Custom loaders, `NODE_OPTIONS`, `NODE_PATH`
310
- and module-mocking configurations are outside this initial adapter's scope.
311
- `auto` keeps its existing runner order. Each confirmation uses fresh isolated
312
- processes and the same worker, timeout and confirmation rules.
313
-
314
- To reproduce the narrow native-runner comparison, run
315
- `node .github/scripts/benchmark-node-runner.mjs --out /absolute/receipt-dir`
316
- from this repository after installing its pinned development dependencies.
317
- It compares 17 identical classifier inputs and four existing faults through
318
- complete probes, with three confirmations, fresh worker receipts and
319
- reversed execution order. Results describe that workload and runtime only.
320
-
321
- Other frameworks go through `--runner-cmd`. Each runner is proven against
322
- its own copy of the known-answer fixture.
323
-
324
- **Python** runs under `pytest` when the project's interpreter can import
325
- it and stdlib `unittest` when it cannot, and the evidence records the
326
- engine that actually ran — `--runner pytest` or `--runner unittest` pins
327
- the choice and fails rather than falling back. Nothing is installed into
328
- the project: the reporters ship inside TestGuard and reach the interpreter
329
- through `PYTHONPATH`, so a codebase whose test dependencies are the
330
- standard library stays that way. The interpreter is the project's
331
- `.venv`/`venv`, the active `VIRTUAL_ENV`, or `--python <path>`. A `.py`
332
- defender runs under Python whatever the project runner is, so one claim
333
- can be defended by a vitest test and a pytest test at once.
334
-
335
- Two things Python forces that JavaScript does not. **`patch("pkg.mod.fn")`
336
- is not `vi.mock`**: it replaces one attribute, so the file is still a
337
- defender of every other fault in that module — it keeps its place and
338
- carries a `target-attribute-patched` signal naming the attributes. And
339
- **the fault must be the code that ran**: a strict editable install puts an
340
- import hook ahead of `sys.path`, so Python can load the original file while
341
- TestGuard faults the copy — a green baseline, every claim `SURVIVED`, and a
342
- report that reads as a devastating finding while being entirely false. With
343
- the fault applied, TestGuard asks which file was actually imported and
344
- refuses the run when it is outside the tree being probed. When no defender
345
- imported the subject at all, the run continues and the record says so
346
- (`detail.targetNotImported`), because that is a fact about the defenders'
347
- reach and not about their assertions.
348
-
349
- The two-gate rule, as one verb: `testguard admit <test-file> --claim <ID>`
350
- runs the claim's faults against your uncommitted test and answers
351
- `ADMITTED` (exit 0: the test is green on unmodified HEAD and fails on
352
- every fault, N/N) or `NOT ADMITTED` (exit 1: the first blocking fault and
353
- what to do about it). The test must be a declared or discovered defender
354
- of the claim. It is sugar over `probe --claim <ID> --include-dirty`, so
355
- it can never disagree with the gate. Every generator on the market admits
356
- a test because it compiles, passes and raises coverage; `admit` admits it
357
- because it fails when the claim is false.
358
-
359
- Practical loop: first pass `--no-escalate` (escalation re-runs the whole
360
- suite N times per survivor); iterate on one claim with `--claim <ID>`, or
361
- select a batch with commas or repeated flags (`--claim A,B --claim C`).
362
- Every occurrence is retained in first-seen order, duplicates are collapsed,
363
- and partial human/JSON output names both the requested and probed counts, so
364
- a green subset cannot masquerade as the selection the operator asked for.
365
- Use either `--include-dirty` or `--in-place` (only fault target files must be
366
- clean there; test files may be dirty), optionally `--confirm 1` for a fast
367
- **provisional** signal — verdicts print with a `?`, evidence goes to
368
- `evidence-provisional.json`, and `baseline` refuses it; final pass with
369
- defaults. By default
370
- the stream shows only unproven faults plus a killed count — `--verbose`
371
- shows every fault. A custom
372
- runner (`pnpm --filter`, a specific config) goes in
373
- `--runner-cmd "<cmd> {files} … {out}"`; if the scratch worktree cannot
374
- see your `node_modules`, pass `--node-modules <dir>`.
375
- 3. **Baseline** freezes every non-passing fingerprint. Later probes suppress
376
- what was already known and exit non-zero only on what is new. Prior verdicts
377
- can be reused when their recorded inputs and runner policy still match.
378
- Cold runs, changed discovery/configuration or `--no-reuse` can require a
379
- full probe; unchanged target and defender files alone do not guarantee reuse.
380
- A baseline frozen from `--include-dirty`
381
- evidence records the snapshot and points at the *parent* of the commit
382
- that will carry your tests; after you commit, a clean `probe` plus
383
- `baseline --restamp` moves it to that commit — only when the fingerprints
384
- are identical, never otherwise. `status` notes a baseline that predates
385
- HEAD without making it a state.
386
- 4. **Brief** turns evidence plus baseline into a ranked, capped
387
- `## TEST BLINDSPOT CONTEXT` block, printed and also written to
388
- `.testguard/brief.json` (`--text` prints only; `--markdown` prints the same brief as a
389
- merge-request note for the human reviewer, unclaimed changes first). Wire it into an agent's session start
390
- — for Claude Code, in `.claude/settings.json`:
391
-
392
- ```json
393
- { "hooks": { "SessionStart": [ { "hooks": [
394
- { "type": "command", "command": "node_modules/.bin/testguard brief --text 2>/dev/null || { command -v testguard >/dev/null 2>&1 && testguard brief --text 2>/dev/null; } || true" }
395
- ] } ] } }
396
- ```
397
-
398
- `--text` prints only, and exits 0 silently when there is no evidence yet;
399
- the command resolves the project's own install, then a `testguard` on
400
- `PATH`, and ends in `true` — so the hook can never break a session. There
401
- is no `npx` in it, in any form: `--no-install` resolves the package from
402
- the registry before declining to install it, so it is quiet rather than
403
- offline. The brief's first line says which install answered
404
- (`local install` or `global`), so a stale one is visible.
405
-
406
- ### Advisory updates for AI sessions
407
-
408
- New integrations created by `testguard init` tell the AI to identify the actual
409
- CLI (project-local before global) and, once per session when network policy
410
- permits, read the registry's stable version:
411
-
412
- ```bash
413
- npm view testguard-cli dist-tags.latest --json --fetch-retries=0 --fetch-timeout=5000
414
- ```
415
-
416
- This is an optional check performed by the AI, not by TestGuard's CLI or hook.
417
- A newer stable version produces a suggestion and a request for approval, never
418
- an automatic install. Project pins and the original package manager still
419
- apply; prereleases and development checkouts are not replaced. Offline or
420
- failed checks do not change verdicts, evidence or exit codes, and do not prove
421
- that an installation is current. AI instructions are guidance, not a guarantee
422
- that every harness performs the check.
423
-
424
- Upgrading the package does **not** refresh an existing copied skill or an
425
- existing managed section already listing the project. Compare the upgraded
426
- package's `src/init/templates/SKILL.md` with
427
- `.claude/skills/testguard/SKILL.md` and merge approved changes, preserving local
428
- customizations. For an existing `AGENTS.md` section, add the same advisory
429
- policy after reviewing it. `testguard init --force` replaces the whole skill
430
- but does not refresh an already-listed managed section; use it only with
431
- explicit overwrite approval and review the diff. No new automatic refresh
432
- mechanism is introduced.
433
-
434
- ### Would this suite have caught the bugs that already escaped?
435
-
436
- An injected fault is a fault somebody thought of. A bug that actually shipped
437
- is ground truth: a human already confirmed it was a defect, and there is no
438
- equivalent-mutant argument to have about it.
439
-
440
- ```bash
441
- npx testguard-cli replay --since HEAD~50..HEAD --max 10
442
- ```
443
-
444
- For each fix commit in the range — one that changes source **and** a test
445
- together — `replay` checks it out in a scratch worktree, reverts only its
446
- source files to the parent, **deletes the test the fix shipped** (that test
447
- proves nothing about what the suite knew before it existed), and runs the
448
- tests that import the reverted code:
449
-
450
- | Verdict | Meaning |
451
- |---|---|
452
- | `caught` | a remaining test failed by assertion, every run. The suite knew. |
453
- | `blind` | the suite stayed green on known-broken code. |
454
- | `nocover` | no test imports the reverted files — worse than blind. |
455
- | `flaky` | the runs disagreed, so nothing can be concluded. A flaky failure reads as detection, which biases this metric *optimistically*; mixed runs are never `caught`. |
456
- | `unverifiable` | the revert did not apply, or the suite could not load. |
457
-
458
- One patch counts once (`git patch-id`), because a dual-branch topology
459
- carries the same fix under two or three shas. **`replay` reports and never
460
- gates**: a bug that escaped is history, not a regression in this change.
461
-
462
- Each replayed bug is also labelled with the injected-fault class its diff
463
- most resembles, which is the join key that turns an uninterpretable mutation
464
- score into a statement with a sample size:
465
-
466
- ```
467
- guard-removed missed 7/9 p=0.78 ci [0.45, 0.94]
468
- field-dropped missed 4/4 p=1.00 ci [0.51, 1.00]
469
- ```
470
-
471
- Read as *"when a real bug of this class escapes, how often does the suite
472
- miss it"* — a miss rate, so a high `p` is bad. It is written as a
473
- `calibration` document beside the replay one. `caught`, `blind` and
474
- `nocover` are measurements and enter the ratio, `nocover` as a miss: leaving
475
- it out would score a project with no tests at all for a subsystem *better*
476
- than one with weak tests. `flaky` and `unverifiable` are failed measurements
477
- and enter neither side. Every `p` and `ci` is recomputable from `n` and the
478
- miss count, and the validator recomputes them.
479
-
480
- **The open question this exists to answer.** Does a calibration learned on a
481
- repository *with* history transfer to a greenfield one that has none? AI
482
- authored code has no history, so replay cannot help it directly and
483
- calibration is the only bridge. That transfer is unproven, it is the core
484
- product bet, and this is the instrument for testing it — not the answer.
485
-
486
- ### Read CI's evidence locally
157
+ nothing was probed, `3` usage. Every command except `mcp` accepts `--json`.
487
158
 
488
- Verdict reuse makes a probe cheap, but the evidence lives where `probe` ran
489
- and is gitignored. On a fresh clone, or on a laptop where CI does the
490
- probing, the session-start brief is empty and `status` says `unprobed` while
491
- the default branch has full evidence. Point either command at CI's document:
159
+ The [quickstart](https://github.com/raccioly/testguard/blob/main/docs/quickstart.md)
160
+ walks through a first claim end to end. Adding TestGuard to a codebase that
161
+ already has a large suite has its own guide:
162
+ [adopting TestGuard in an existing project](https://github.com/raccioly/testguard/blob/main/docs/guides/existing-projects.md).
492
163
 
493
- ```bash
494
- testguard status . --evidence .testguard/ci/ci-self-evidence.json
495
- testguard brief . --text --evidence .testguard/ci/ci-self-evidence.json
496
- ```
497
-
498
- A foreign document is not trusted blindly. `status` marks it
499
- `evidenceSource: provided`, prints the commit it describes next to the
500
- commit in your tree, and still computes staleness from the recorded input
501
- hashes — so a file you have edited since CI probed it goes `evidence-stale`
502
- for exactly those claims.
503
-
504
- `testguard init --ci-evidence github` (or `gitlab`) writes
505
- `.testguard/fetch-ci-evidence.sh`, which downloads the branch-named artifact
506
- with the platform CLI you already have and then briefs from it. **It is an
507
- on-demand helper, not a hook**: the session-start hook never touches the
508
- network, and the helper exits 0 with a message when the CLI or the artifact
509
- is missing. TestGuard itself still makes no network calls.
510
-
511
- ### Starting from zero claims
512
-
513
- During gradual adoption, use `testguard probe --allow-empty` to skip a valid
514
- empty claims file without blocking a push. It explicitly reports that no
515
- verification was performed and writes no evidence; `--json` retains the
516
- `no-claims` state. Missing or invalid claims files still fail. Run `gate`
517
- separately to enforce changed-file coverage. For the pre-commit probe hook,
518
- set `args: [--allow-empty]`; for the GitHub Action, set `allow-empty: 'true'`.
519
- The default probe still exits 2 when no claims exist.
520
-
521
- In a monorepo, a valid nested `testguard.claims.json` marks a separate project.
522
- The parent gate reports its changed files as delegated and does not count them
523
- as parent coverage. Run `gate` in each child project as well; a passing parent
524
- gate does not establish that child coverage passed. Invalid child claims fail
525
- evaluation instead of hiding files.
164
+ ## Languages and test runners
526
165
 
527
- `gate` names the changed files that carry no claim, and stops there —
528
- correctly, because stating a claim is a human act. But a project adopting this
529
- tool reads that list, has nothing to compare it against, and closes the tab.
530
- The bottleneck was never verification; it is **oracle supply**.
166
+ | Language | Runners | Faults proposed by `scaffold` |
167
+ |---|---|---|
168
+ | JavaScript | vitest, jest, Playwright, Node's built-in test runner | seven shapes: forced guards, deleted statements, altered returns, weakened literals, removed calls, dropped payload fields, swapped arguments |
169
+ | TypeScript | vitest, jest, Playwright (through the project's own transform) | the same seven |
170
+ | JSX, TSX (UI) | vitest, jest, Playwright | the seven, plus removed elements and dropped event handlers |
171
+ | Python ≥ 3.8 | pytest, or stdlib `unittest` when pytest is absent | the same seven, in Python syntax |
172
+ | Anything else | `--runner-cmd` with a jest-compatible JSON report | hand-written faults |
531
173
 
532
- `sweep` is the lowest rung of that supply. No claim, no concern, nothing
533
- written by anybody:
534
-
535
- ```bash
536
- npx testguard-cli sweep --changed origin/main # propose, probe a bounded selection, report
537
- npx testguard-cli sweep --changed HEAD --include-dirty --cap 20
538
- ```
539
-
540
- It takes the files `gate` just called uncovered, proposes faults with the same
541
- mechanical producers `scaffold` uses, probes a bounded selection, and reports
542
- what a green suite did not notice. A fault that survives needs no claim to be
543
- alarming: something was deliberately broken and not one test failed.
544
-
545
- ```
546
- swept 13 of 14 unclaimed changed files against origin/main.
547
- proposed 510 faults, probed 10 (cap 10), deferred 350.
548
- The 350 deferred are not a verdict: raise --cap, or sweep a smaller change.
549
- 150 presentational elements (icons, static wrappers) set aside: nobody writes a test for those, and a survivor there would teach the ordering the wrong lesson.
174
+ One claim can be defended by a vitest test and a pytest test at once. Runner
175
+ resolution, discovery, mocks and limits are in the
176
+ [languages and runners reference](https://github.com/raccioly/testguard/blob/main/docs/reference/languages-and-runners.md).
550
177
 
551
- 9 findings — a deliberate break that no test noticed:
178
+ ## Use it with your AI agent
552
179
 
553
- SURVIVED src/app/actions/register.ts:118 [write path]
554
- [line 118] Field dropped: `password_hash` is no longer written.
555
- src/tests/actions/register.test.ts mocks @/lib/prisma; 1 exact, 8 partial,
556
- 31 argument-free call assertions in the file. A field dropped from the write
557
- payload fails only against an assertion that names that field exactly.
558
- Assert the whole object written by the call this fault changes, including
559
- `password_hash` — or read the record back and assert on what was stored.
560
- ```
561
-
562
- Three rules make it safe to run on a repository that has never seen this tool:
563
-
564
- Pooled draft IDs are unique even when source files share basenames/functions or
565
- annotation IDs. Sweep reserves original IDs and suffixes collisions before
566
- selection, keeping every fault, its metadata and defenders attached to its own
567
- draft. These are proposal identities, not authenticated claims.
568
-
569
- - **It never writes `testguard.claims.json`**, and never will. A claim is a
570
- sentence someone is willing to stand behind; a sentence nobody wrote is not
571
- one. Its drafts land beside the evidence to keep or drop.
572
- - **Its evidence never replaces `.testguard/evidence.json`.** A sweep probes
573
- faults nobody stated, under TODO statements; folding that into the document
574
- `status` and `baseline` read would corrupt the record of what the project
575
- actually claims.
576
- - **It does not fail on its own bad guess.** Only `survived` and `nocover`
577
- exit 1. A proposal that would not compile, or an anchor that did not locate,
578
- is reported and never gates — a tool that fails because its own guess was bad
579
- is a tool people switch off.
580
-
581
- The ordering **learns from your own runs**. Every evidence record already
582
- carries a fault class and a verdict, so "how often does a fault of this class
583
- survive here" is a tally, not a new thing to collect — a sweep persists its
584
- evidence and the next one reads it. A project with no evidence falls back to
585
- the shipped prior and says so; one with its own records overrides it in
586
- proportion to how many it has. The report names which, because an ordering
587
- nobody can trace is a number nobody should trust.
588
-
589
- The cap and the ordering are Google's: their mutation service surfaces at most
590
- 7 × |files| mutants per change and orders candidates on the measured
591
- productivity of their operator in similar context, which took their productive
592
- rate from 15% to 89%
593
- ([Petrović et al., *Practical Mutation Testing at Scale*, 2021](https://arxiv.org/abs/2102.11378)).
594
- TestGuard's ordering prior is measured the same way — from probed faults on a
595
- real AI-authored codebase — and shrunk toward a neutral value, so one
596
- observation of a class surviving never dominates a sweep. The cap is spread
597
- across files: a single file with many candidates cannot make the others look
598
- clean.
599
-
600
- A sweep is weaker evidence than a probe, and says so. Nobody stated that the
601
- behaviour mattered. It is stronger than nothing, which is what a repository
602
- with no claims has — and the survivors worth defending become the first claims.
603
-
604
- #### Concerns: one sentence, two hundred screens
605
-
606
- A claim names one promise precisely, which is why a repository with two hundred
607
- screens never finishes writing them. A **concern** names a *kind* of promise and
608
- says where to look for it:
609
-
610
- ```bash
611
- npx testguard-cli concerns # what this project can be swept by
612
- npx testguard-cli sweep --concern ADMIN-GUARDS # aim a sweep at one
613
- ```
180
+ TestGuard is meant to be driven by an agent. `testguard init` installs a skill
181
+ that teaches the operating loop, an `AGENTS.md` section, and a session-start
182
+ hook that briefs every session with the ranked blind spots before it writes
183
+ code. For Claude Code, the hook in `.claude/settings.json` is:
614
184
 
615
185
  ```json
616
- {
617
- "schemaVersion": 1,
618
- "concerns": [
619
- {
620
- "id": "ADMIN-GUARDS",
621
- "statement": "Every admin route refuses a caller without the admin role.",
622
- "severity": "critical",
623
- "targets": { "kind": "glob", "globs": ["src/app/api/admin/**"] },
624
- "faultClasses": ["guard-removed", "condition-forced", "return-altered"]
625
- }
626
- ]
627
- }
186
+ { "hooks": { "SessionStart": [ { "hooks": [
187
+ { "type": "command", "command": "node_modules/.bin/testguard brief --text 2>/dev/null || { command -v testguard >/dev/null 2>&1 && testguard brief --text 2>/dev/null; } || true" }
188
+ ] } ] } }
628
189
  ```
629
190
 
630
- Five lines, and the sweep reports:
631
-
632
- ```
633
- concern ADMIN-GUARDS matches 24 files. Swept 23.
634
- 158 further proposals were outside this concern's fault classes.
635
-
636
- SURVIVED src/app/api/admin/claims/[id]/route.ts
637
- [line 41] Guard never triggers: `if (!membership || (membership.role !== 'owner'
638
- && membership.role !== 'manager'))` becomes `if (false)`.
639
- ```
640
-
641
- The idea is Meta's: ACH has an engineer describe an area of concern in plain
642
- text and gates every generated step by execution. What is taken here is the
643
- **unit** — the concern as the thing a human writes — not their generator, and
644
- **no model is required**. A concern's useful core is a named scope plus a
645
- producer selection, which the mechanical producers already satisfy, so every
646
- verifying command stays offline.
647
-
648
- Two concerns ship built in, and only two. An authorization concern would need a
649
- heuristic for *"which files check permissions"*, and a guess there is exactly
650
- the noise that gets a check switched off — so `SAVE-PERSISTS` and
651
- `CHANGED-CODE` are the defaults, and a project that knows its own auth layer
652
- says so in three lines. Declaring a concern with a built-in's id replaces it,
653
- and the replacement is reported rather than silent.
654
-
655
- A concern is **not** a claim and never becomes one by itself. It says where to
656
- look; the probe says what it found; a human states the sentence worth defending.
657
-
658
- #### Sweeping the save surface instead of the diff
659
-
660
- A diff cannot answer *"when the user clicks save, does it actually save?"* — it
661
- only knows what changed today. `--save-paths` points the same sweep at every
662
- file that writes to storage:
663
-
664
- ```bash
665
- npx testguard-cli sweep --save-paths --cap 20
666
- ```
667
-
668
- ```
669
- 119 writes to storage across 37 files, carrying 226 payload fields. Swept 36.
670
- proposed 762 faults, probed 20 (cap 20), deferred 742.
671
-
672
- SURVIVED src/app/actions/register.ts [write path]
673
- [line 59] Field dropped: `password_hash` is no longer written.
674
- src/tests/actions/register.test.ts mocks @/lib/prisma; 1 exact, 8 partial,
675
- 31 argument-free call assertions in the file. A field dropped from the write
676
- payload fails only against an assertion that names that field exactly.
677
- ```
678
-
679
- The first line is the point. A report that lists findings without saying how
680
- much surface it looked at invites you to assume the rest is fine, so the
681
- denominator comes first and the document carries `writeSites` and
682
- `payloadFields` whether or not anything survived. Measured on a real
683
- AI-authored application: **12 of 26 probed persistence faults survived a fully
684
- green suite, and all 12 were a dropped payload field.**
685
-
686
- It also states, before any verdict, **what the defenders could prove at all**:
687
-
688
- ```
689
- of those 37 files: 32 defended only by tests that mock the persistence layer,
690
- 1 with an unmocked defender, 4 with no defender at all.
691
- ```
692
-
693
- That line is the read-back answer. A test that replaced the database can prove
694
- the *call shape* and nothing beyond it — a `where` that matches no rows, a
695
- rolled-back transaction and a rejected constraint all pass against a mock that
696
- recorded the arguments and returned a plausible object. So rather than pretend
697
- to measure persistence, TestGuard states the limit, the same way `nocover` says
698
- "no test imports this" instead of guessing.
699
-
700
- When every defender mocks the layer, the report says so outright: *nothing in
701
- this suite can prove a write reached storage*. A clean probe there is not
702
- evidence of persistence — it is evidence that nothing could have measured it.
703
-
704
- `unmocked` is deliberately weaker than "proves persistence": a test that does
705
- not replace the database may simply never reach it. Possible is the honest word.
706
-
707
- ### Every change needs a claim
191
+ It resolves the project's own install, then a `testguard` on `PATH`, and ends
192
+ in `true`, so it can never break a session or reach the network. Any other
193
+ harness can use `testguard mcp` (five read-only tools over stdio) or
194
+ `testguard status --json`. Setup for Claude Code, Codex, Cursor and others is
195
+ in the [AI agents guide](https://github.com/raccioly/testguard/blob/main/docs/guides/ai-agents.md).
708
196
 
709
- `probe` can only verify claims that exist. Every escaped defect in the field
710
- reports so far was a **claim gap**: the feature shipped green with zero
711
- claims, and a verifier with no claim about a feature is silent about it by
712
- construction. `gate` closes that hole on the delta:
197
+ ## Documentation
713
198
 
714
- ```bash
715
- npx testguard-cli gate --changed origin/main # in a PR: the files changed since the base branch
716
- npx testguard-cli gate --changed HEAD --include-dirty # before a commit: the working tree, staged or not
717
- ```
718
-
719
- Every changed source file must carry a fault, resolve as a defender of a
720
- claim (test files), or be excused by an unexpired `path` entry in
721
- `testguard.ignore.json` — with a reason a reviewer will accept. One
722
- unclaimed file exits `1`; there is no percentage. Every reliance on an ignore
723
- entry is printed, so a reviewer sees *why* the gate passed; an expired entry
724
- excuses nothing. Non-source files and documented never-claimed patterns
725
- (`*.d.ts`, `*.config.*`, fixtures, mocks; `--explain` lists them) are
726
- excluded and said so; `--strict` fails a change that evaluated nothing.
727
-
728
- An ignore file is a list of `entries`. The field is `pattern`, and `reason` is
729
- not optional — it is the whole point of the mechanism:
730
-
731
- ```json
732
- {
733
- "$schema": "./node_modules/testguard-cli/spec/schemas/ignore.schema.json",
734
- "schemaVersion": 1,
735
- "entries": [
736
- {
737
- "kind": "path",
738
- "pattern": "src/commands/**",
739
- "reason": "Thin wrappers: read flags, call the module, print. The logic is claimed in the modules they call.",
740
- "by": "maintainer",
741
- "at": "2026-09-17T00:00:00Z"
742
- },
743
- {
744
- "kind": "path",
745
- "pattern": "src/legacy/billing.ts",
746
- "reason": "Scheduled for deletion in Q4; claims would outlive the file.",
747
- "by": "maintainer",
748
- "at": "2026-09-17T00:00:00Z",
749
- "expires": "2026-12-31T00:00:00Z"
750
- }
751
- ]
752
- }
753
- ```
754
-
755
- `kind: "claim"` entries take the same shape and excuse a *removed claim*
756
- instead of an unclaimed file. An entry past its `expires` excuses nothing and
757
- is reported as `EXPIRED`.
758
-
759
- With a reference known, `status --changed <ref>` reports `unclaimed-changes`
760
- **before** any evidence state and makes the claim the next action; the brief
761
- lists the unclaimed files first. The claim is written before more code.
762
- After claim coverage, `status` reports `invalid-anchors` before any evidence
763
- state when an exact fault anchor is missing or ambiguous, and points at the
764
- fault definition to repair. Repair preserves the fault's meaning and requires
765
- a re-probe; there is deliberately no `--fix`.
766
-
767
- **The base is detected when Git records it.** An explicit `--changed` wins,
768
- then `TESTGUARD_CHANGED_REF`, GitHub Actions (`GITHUB_BASE_REF`) and GitLab
769
- merge request pipelines (`CI_MERGE_REQUEST_DIFF_BASE_SHA`, then
770
- `CI_MERGE_REQUEST_TARGET_BRANCH_NAME`). In an ordinary clone on a feature
771
- branch, bare `testguard gate` uses the configured remote's symbolic default
772
- branch (normally `origin/main`) and says how it chose it. If that symbolic ref
773
- is absent, a configured upstream is used only when its branch name differs
774
- from the current branch. A feature branch's same-name tracking ref and the
775
- default branch itself are never selected: either could produce a misleading
776
- empty diff. In those ambiguous cases, pass `--changed` explicitly.
777
-
778
- The base must exist locally: GitHub — `actions/checkout` with `fetch-depth: 0`;
779
- GitLab — the diff base sha needs nothing extra on a merge request pipeline,
780
- the branch name needs `GIT_DEPTH: 0` or a `git fetch origin <target>`. A
781
- detected base that does not resolve is a warning for `status` and `brief`
782
- (they keep working) and an error for `gate` (its whole job is the measurement).
783
-
784
- ```yaml
785
- # GitHub Actions
786
- - uses: actions/checkout@v4
787
- with: { fetch-depth: 0 }
788
- - uses: raccioly/testguard@v0.18.2
789
- with: { command: gate }
790
-
791
- # GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
792
- include:
793
- - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.18.2/packaging/gitlab/testguard.gitlab-ci.yml'
794
- inputs: { dir: backend, post_note: true } # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place
795
- ```
796
-
797
- The template carries `spec: inputs:` (`version`, `dir`, `image`, `severity`,
798
- `confirm`, `budget`, `no_escalate`, `strict`, `post_note`, `stage`), so
799
- mirrored into a GitLab project it is a catalog component that a compliance
800
- framework can require on every project in a group. The CLI itself never
801
- talks to the network; only the job posts, and only when told to.
802
-
803
- ### Properties
804
-
805
- - **Deterministic measurement.** No model decides a verdict. Faults are
806
- string substitutions from a reviewed file; verdicts come from the test
807
- runner's structured report, confirmed N times. The same inputs give the
808
- same evidence, which is what makes it replayable for an auditor.
809
- - **No network, no telemetry.** The CLI never opens a socket. The
810
- session-start hook resolves an installed binary and never fetches one.
811
- Nothing leaves the machine unless a CI job you configured posts it.
812
- - **Artifacts are data in the repository.** Claims, evidence, baseline and
813
- brief are JSON validated against a published schema before they are
814
- written. Nothing is healed, regenerated or re-targeted at run time; a
815
- claims file is code, and is reviewed like code.
816
- - **Never optimistic.** A timeout, a load failure, a mixed N-run result, a
817
- flaky defender or a missing anchor is reported as unproven, never rounded
818
- toward green. The one pass is `killed`; everything else gates.
819
- - **Per-run and whole-command time are separate.** `--budget` limits each
820
- runner invocation. `--command-budget` is a cooperative deadline for the
821
- complete `probe`, `sweep`, or `replay` measurement: asynchronous children
822
- receive only the remaining time, and stage/write boundaries re-check it.
823
- Synchronous setup may overrun the nominal deadline, but once expiry is
824
- observed TestGuard exits `2` and writes no new result document; a completed
825
- prefix is not presented as though the unattempted suffix were clean.
826
- - **Configured discovery is evidence.** Vitest, Jest, and Playwright enumerate
827
- their own configured files; TestGuard validates and hashes that immutable
828
- universe instead of guessing with filename globs. Automatic JavaScript and
829
- TypeScript discovery follows the imported symbol through barrels. Ambiguous
830
- or unbounded resolution is `unverifiable`, never a flattering `nocover`.
831
- - **One exact-pinned runtime dependency** (`ajv`), Node ≥ 20, MIT.
832
-
833
- ### Prior art, and what was taken from where
834
-
835
- Breaking code to test the tests is mutation testing, and it dates to the 1970s.
836
- Two industrial programmes solved the parts that make it usable, and this tool
837
- takes from both.
838
-
839
- **Google** made it affordable: mutate only *changed* lines, cap what is
840
- surfaced at seven per file, and order candidates by the measured productivity
841
- of the operator in similar context — developer feedback took their productive
842
- rate from 15% to 89%
843
- ([Petrović et al., 2021](https://arxiv.org/abs/2102.11378)). `sweep`'s diff
844
- scope, its cap and its ordering are theirs; the productivity prior here is
845
- measured on real probed faults rather than assumed.
846
-
847
- **Meta** made it specific: an engineer writes a *concern* in plain text, a
848
- model drafts faults for it, and every step is gated by execution before a
849
- human ever sees it — 73% of the resulting tests were accepted
850
- ([Foster et al., FSE 2025](https://arxiv.org/abs/2501.12862)). The rule that
851
- nothing a model proposes reaches the claims file without passing `probe` and a
852
- human keep is theirs.
853
-
854
- What neither does is bind a fault to a **stated promise**, which is what this
855
- tool is for. The full record — including what was deliberately *not* taken, the
856
- measured reason no model decides a verdict, and the gaps in what we borrowed —
857
- is in [`docs-canonical/PRIOR-ART.md`](docs-canonical/PRIOR-ART.md).
199
+ | | |
200
+ |---|---|
201
+ | **Get started** | [Quickstart](https://github.com/raccioly/testguard/blob/main/docs/quickstart.md) · [Installation](https://github.com/raccioly/testguard/blob/main/docs/installation.md) · [Upgrading](https://github.com/raccioly/testguard/blob/main/docs/upgrade.md) |
202
+ | **Guides** | [Existing projects](https://github.com/raccioly/testguard/blob/main/docs/guides/existing-projects.md) · [New projects](https://github.com/raccioly/testguard/blob/main/docs/guides/new-projects.md) · [Writing claims](https://github.com/raccioly/testguard/blob/main/docs/guides/writing-claims.md) · [AI agents](https://github.com/raccioly/testguard/blob/main/docs/guides/ai-agents.md) · [Python](https://github.com/raccioly/testguard/blob/main/docs/guides/python.md) · [Monorepos](https://github.com/raccioly/testguard/blob/main/docs/guides/monorepo.md) · [Performance](https://github.com/raccioly/testguard/blob/main/docs/guides/performance.md) · [Replaying escaped bugs](https://github.com/raccioly/testguard/blob/main/docs/guides/replay.md) |
203
+ | **CI** | [GitHub Actions](https://github.com/raccioly/testguard/blob/main/docs/guides/ci/github-actions.md) · [GitLab CI](https://github.com/raccioly/testguard/blob/main/docs/guides/ci/gitlab.md) · [pre-commit](https://github.com/raccioly/testguard/blob/main/docs/guides/ci/pre-commit.md) |
204
+ | **Reference** | [CLI](https://github.com/raccioly/testguard/blob/main/docs/reference/cli.md) · [Languages and runners](https://github.com/raccioly/testguard/blob/main/docs/reference/languages-and-runners.md) · [Configuration](https://github.com/raccioly/testguard/blob/main/docs/reference/configuration.md) · [Verdicts](https://github.com/raccioly/testguard/blob/main/docs/reference/verdicts.md) · [Artifacts](https://github.com/raccioly/testguard/blob/main/docs/reference/artifacts.md) · [MCP](https://github.com/raccioly/testguard/blob/main/docs/reference/mcp.md) · [Gate semantics](https://github.com/raccioly/testguard/blob/main/spec/GATE-SEMANTICS.md) |
205
+ | **Understand** | [How it works](https://github.com/raccioly/testguard/blob/main/docs/concepts/how-it-works.md) · [Glossary](https://github.com/raccioly/testguard/blob/main/docs/glossary.md) · [FAQ](https://github.com/raccioly/testguard/blob/main/docs/faq.md) · [Troubleshooting](https://github.com/raccioly/testguard/blob/main/docs/troubleshooting.md) · [Prior art](https://github.com/raccioly/testguard/blob/main/docs-canonical/PRIOR-ART.md) |
206
+ | **Translations** | [Português (Brasil)](https://github.com/raccioly/testguard/blob/main/docs/i18n/pt-BR/README.md) · [Español](https://github.com/raccioly/testguard/blob/main/docs/i18n/es/README.md) · [简体中文](https://github.com/raccioly/testguard/blob/main/docs/i18n/zh-CN/README.md) |
858
207
 
859
- ### What TestGuard is not
208
+ ## What TestGuard is not
860
209
 
861
210
  - **Not a test generator.** It judges a test the agent wrote (`admit`); the
862
- generating half stays with the agent, where the market is putting it.
211
+ generating half stays with the agent.
863
212
  - **Not a mutation-score dashboard.** No blanket mutants, no single score, no
864
213
  threshold. Faults are few and bound to stated claims; findings are ranked,
865
214
  never summed.
866
- - **Not a self-healing runner.** A defender that adapts itself to a change is
867
- a defender that did not observe it; TestGuard reports that, it does not do
868
- it.
869
215
  - **Not a coverage tool.** A line executed says nothing about whether an
870
- assertion would notice. `NOCOVER` here means no test even imports the
871
- file; everything above that is measured by injecting the fault.
872
-
873
- A claim that **disappeared** is invisible to all of the above: `probe`
874
- verifies the claims that exist, `gate` sees the source file covered by some
875
- other claim, and `status` says clean. Deleting a claim is therefore cheaper
876
- than weakening its fault — and `changedFaults` exists precisely because
877
- weakening was the cheap escape. So:
878
-
879
- ```bash
880
- npx testguard-cli claims --since origin/main # what existed there and does not now
881
- ```
882
-
883
- `removed-claim` and `removed-fault` gate; a rename that keeps the statement
884
- verbatim is reported as `renamed-claim` and does not. Where evidence is
885
- present, the line names the verdict the claim last had, because *"it was
886
- SURVIVED when it was removed"* is the sentence that matters. The escape hatch
887
- is the same one as everywhere else: a `claim` entry in
888
- `testguard.ignore.json` with a reason, expiring if you give it an `expires`.
889
- Removal is allowed — claims can be wrong, superseded or split — and it is
890
- never silent.
891
-
892
- ### Probing a Python project
893
-
894
- Nothing is installed into your project. TestGuard carries its own reporters and
895
- puts them on the interpreter's `PYTHONPATH`, so a codebase whose test
896
- dependencies are the standard library keeps none.
897
-
898
- ```bash
899
- npx testguard-cli probe . # auto: pytest if importable, else stdlib unittest
900
- npx testguard-cli probe . --runner unittest # pin the stdlib engine
901
- npx testguard-cli probe . --python .venv/bin/python
902
- ```
903
-
904
- The evidence records the engine that actually ran, and `--runner pytest` fails
905
- rather than quietly using `unittest` — which engine ran changes what the
906
- evidence means.
907
-
908
- Three things behave differently from JavaScript, and each one is a place a
909
- naive port would have produced a confident wrong answer:
910
-
911
- - **Defenders are matched by module name**, the way Python itself matches, so
912
- a test that reaches your package through `sys.path`, a `conftest.py` or an
913
- installed distribution is found either way.
914
- - **`patch("pkg.mod.fn")` is not `vi.mock`.** It replaces one attribute, so the
915
- file still detects a fault anywhere else in that module: it stays a defender
916
- and carries a `target-attribute-patched` signal naming the attributes. Only a
917
- patch of the module *itself* removes a defender.
918
- - **The fault has to be the code that ran.** A strict editable install puts an
919
- import hook ahead of `sys.path`, so Python can load your original file while
920
- TestGuard faults its copy — green baseline, every claim `SURVIVED`, and a
921
- report that reads as a catastrophe while being entirely false. With the fault
922
- applied TestGuard asks which file was really imported and **refuses the run**
923
- if it came from outside the tree being probed. If no defender imported the
924
- subject at all, the run continues and the record says so
925
- (`detail.targetNotImported`): that is a fact about the tests' reach, not
926
- about their assertions, and the two must not read alike.
927
-
928
- `testguard scaffold path/to/module.py` proposes Python faults the same way it
929
- does for JavaScript.
930
-
931
- ### Run it from any harness
932
-
933
- The operating loop above lives in a Claude Code skill and a session-start
934
- hook. Both vanish the moment the harness is Cursor, Codex, Devin or whatever
935
- comes next — and the harness is exactly the layer most likely to change. So
936
- the same documents are also served over the Model Context Protocol:
937
-
938
- ```bash
939
- npx testguard-cli mcp # JSON-RPC 2.0 on stdio
940
- npx testguard-cli init --mcp # prints the config for Claude Code, Cursor and Codex
941
- ```
942
-
943
- Five tools, all **read-only**: `testguard_status`, `testguard_brief`,
944
- `testguard_claims`, `testguard_evidence`, `testguard_next_command`.
945
-
946
- **No tool runs a probe.** A probe is long-running, budgeted, and the person
947
- should see it happen, so `next_command` hands back the exact shell line for
948
- the agent to run in its own terminal. Nothing here writes a file either —
949
- editing a claims file through a connector would defeat the point of recording
950
- every fault edit. The server declares only a `tools` capability: no prompts,
951
- no resources, no sampling.
952
-
953
- It is hand-written against a pinned protocol version rather than built on the
954
- official SDK, because this tool has one exact-pinned runtime dependency and
955
- keeps it that way. The surface is three methods and five tools and will not
956
- grow; tests speak the wire format to a real child process and check each
957
- tool against the CLI's own `--json` output, so the two cannot drift.
958
-
959
- ### Built for agents to run
960
-
961
- TestGuard is meant to be driven by an AI agent, not typed by a person. Three
962
- things make that safe:
963
-
964
- - **One source of truth.** `testguard status --json` computes `state` and the
965
- one `next` action from the claims file, the evidence, the baseline and the
966
- working tree. Every human rendering — the CLI text, the session-start
967
- brief, the skill — derives from it, so they cannot disagree. Every command
968
- accepts `--json`.
969
- - **An installable operating loop.** `testguard init [dir]` writes the
970
- **agent layer at the git root**, where agent sessions run —
971
- `.claude/skills/testguard/SKILL.md` (state → action, verdict → the only
972
- acceptable fix, the two-gate rule for any test the agent writes), the
973
- `brief --text` session-start hook, an `AGENTS.md` section — and the
974
- **project layer** (`.gitignore` lines) beside the claims file. A second
975
- project in the same repository adds a hook line and an `AGENTS.md`
976
- bullet; `--here` keeps everything in the subdirectory. The hook
977
- resolves the project's `node_modules/.bin/testguard`, then the git root's,
978
- then a `testguard` on `PATH` via `command -v`, then does nothing — **no
979
- `npx` in any form, so no path to the network**; a pre-0.6 `npx -y` or
980
- `npx --no-install` hook is replaced. A written file that
981
- `.gitignore` swallows is reported, not offered for commit. Idempotent.
982
- - **Independence is recorded (L3).** `probe` measures *power* — would this
983
- test notice if the code were wrong. It also records, on every kill, whether
984
- the killing test was last touched by the same change (or the same author) as
985
- the code it guards: `detail.independence` is `co-authored`,
986
- `separate-change` or `unknown`. A co-authored kill is legitimate — a fix
987
- *should* ship with its regression test — but a repository where every kill
988
- is co-authored has no independent verification, whatever its claim
989
- verification rate says. It is a **signal**: ranking reads it, verdicts never
990
- do, and `probe`, `status` and the brief all say it in one line. Nothing else in this category
991
- measures it, and agents saturate the tests they can see
992
- ([SpecBench](https://arxiv.org/abs/2605.21384)).
993
- - **Claimed surface is visible.** A clean probe is only about claims that
994
- exist. `status` therefore reports claimed source modules over the current
995
- module denominator, how many of the 20 highest-churn modules are claimed,
996
- and the concrete unclaimed modules worth examining next over a bounded
997
- 200-commit window. Churn and path-risk signals stay separate; there is no
998
- synthetic health score. When most modules remain unclaimed and the existing
999
- evidence is clean, the next action expands the denominator instead of
1000
- congratulating the same small set of claims again.
1001
- - **Gaming is visible.** The cheapest way to make a survivor disappear is to
1002
- weaken its fault, not to write a test. Evidence records every fault's
1003
- content hash; `status` lists any fault edited after it survived, with its
1004
- previous verdict, and makes reviewing that edit the next action. Editing a
1005
- claim is allowed — claims can be wrong — but it is never invisible.
1006
-
1007
- ### Watching a probe that is still running
1008
-
1009
- A probe runs the defenders N times clean and N times per fault, which takes
1010
- minutes. It used to print its stage line only when `stderr` was a terminal —
1011
- so CI, a redirected log and an agent harness saw **nothing at all** for the
1012
- whole run, which is indistinguishable from a hang. A gate you cannot tell
1013
- from a hang is a gate people start killing.
1014
-
1015
- The rewriting is a rendering choice, not a reason to withhold the
1016
- information:
1017
-
1018
- | `--progress` | What you get |
1019
- |---|---|
1020
- | `auto` (default) | `tty` at a terminal, `plain` everywhere else |
1021
- | `tty` | one line, rewritten in place — for a human watching |
1022
- | `plain` | one append-only line per stage — a log file, CI, an agent |
1023
- | `ndjson` | one JSON object per line, stages **and** verdicts as they land |
1024
- | `none` | silence |
1025
-
1026
- ```bash
1027
- npx testguard-cli probe . > probe.log 2>&1 # now reports; used to be silent
1028
- npx testguard-cli probe . --progress ndjson 2>events.ndjson
1029
- ```
1030
-
1031
- ```
1032
- {"event":"stage","claim":"TG-COST-NAMES-SHARED-DEFENDERS","fault":"F1","stage":"baseline","run":1,"of":3,"at":"…"}
1033
- {"event":"verdict","claim":"TG-COST-NAMES-SHARED-DEFENDERS","fault":"F1","verdict":"killed","severity":"medium","file":"src/probe/cost.mjs","at":"…"}
1034
- ```
1035
-
1036
- **Progress always goes to stderr**, so `--json` leaves stdout as one document
1037
- and nothing else. `--quiet` and `--json` imply `none`, and an explicit
1038
- `--progress` overrides both: an operator who asks for a stream of events has
1039
- said what they want.
1040
-
1041
- ### What the probe costs, and why a gate gets slow
1042
-
1043
- A fresh probe grows with the selected faults, confirmation count and the cost
1044
- of their defenders. Shared baselines and valid verdict reuse avoid some repeated
1045
- work. Roughly, the fresh work is
1046
-
1047
- ```
1048
- fresh baseline runs × defender-set cost
1049
- + Σ (fault confirmations × that fault's defender-set cost)
1050
- + survivor diagnostics and setup/discovery work
1051
- ```
1052
-
1053
- One slow acceptance test selected by several faults can therefore be executed
1054
- many times. The symptom can be a gate that takes twenty minutes; the selected
1055
- defender sets determine how much each fault costs. Historical attributed totals
1056
- are reported separately from measured fresh elapsed time and runner invocations.
1057
-
1058
- ```bash
1059
- npx testguard-cli claims . --cost # read back from the last probe's evidence
1060
- npx testguard-cli probe . --cost # what the run you just made spent
1061
- ```
1062
-
1063
- ```
1064
- 63 fault records cost 24m across 378 defender runs.
1065
-
1066
- most expensive claims
1067
- 4m 58s TG-FAULT-EDIT-VISIBLE 1 fault · 6 runs
1068
-
1069
- defender files, by the time of the runs that included them
1070
- (a fault runs its selected defender set at once, so these overlap and do not sum to the total)
1071
- 24m 3s test/probe.fixture.test.mjs 5 claims
1072
-
1073
- shared defenders — each fault selects the file; durations include recorded baseline sharing
1074
- 24m 3s test/probe.fixture.test.mjs
1075
- named by TG-ESCALATION-N-RUNS/F1, TG-FAULT-EDIT-VISIBLE/F1, …
1076
- ```
1077
-
1078
- Every number is read back out of the `durationMs` the probe already records.
1079
- `--cost` re-measures nothing, spawns nothing, and costs the price of reading
1080
- one JSON file.
1081
-
1082
- Cost output also lists individual faults, their selected defenders and selection
1083
- origin. Sharing is counted across distinct fault identities, including siblings
1084
- within one claim; repeated confirmation runs alone are not sharing.
1085
-
1086
- **The per-file figures overlap.** A run executes a fault's whole defender set
1087
- at once, so the runner never says how much of it belonged to which file. A
1088
- file's `ms` is the time of every run that *included* it: an upper bound on
1089
- what removing it could save, which is the decision you are making. Only
1090
- `totalMs` is additive.
1091
-
1092
- **What to do about a shared defender.** Extract the decision it proves into a
1093
- pure function, unit-test that, and point the claim at the unit test. This is
1094
- not a testing trick — it is the same reason `classify()` in this codebase is
1095
- pure. A rule you can state in three lines should not need a fifty-second
1096
- acceptance test to falsify it. When the extraction is right the claim still
1097
- fails on the fault; when it is wrong the claim SURVIVES, and the probe tells
1098
- you so before you have shipped a weaker gate. Do not narrow `defendedBy` and
1099
- assume — re-probe the claim and read the verdict.
1100
-
1101
- ### Authoring faults mechanically
1102
-
1103
- Writing faults by hand means reading the code to find exact anchors. Two
1104
- field reports found that ~80% of hand-written faults are one of nine shapes,
1105
- so `scaffold` proposes them for you:
1106
-
1107
- ```bash
1108
- npx testguard-cli scaffold src/auth.ts # → .testguard/scaffold-auth.json (a draft, never your claims file)
1109
- npx testguard-cli scaffold src/auth.ts --claim AUTH-ADMIN # every proposal under one claim; copies it if it exists
1110
- npx testguard-cli scaffold src/auth.ts src/session.ts --into draft.json --claim AUTH-ADMIN --json # read-only merged preview
1111
- npx testguard-cli scaffold src/auth.ts src/session.ts --into draft.json --claim AUTH-ADMIN # explicit recoverable draft update
1112
- npx testguard-cli scaffold --from-document requirements.md --json # read-only input metadata, no document text
1113
- npx testguard-cli scaffold --from-fix <full-commit-ID> --json # read-only fixed historical inventory
1114
- ```
1115
-
1116
- Input inspection uses exactly one `--from-document` or `--from-fix`, with no
1117
- positional sources or other options except JSON/help/version. Fix IDs must be
1118
- full lowercase SHA-1/SHA-256 commit IDs with exactly one parent. Plain-text
1119
- documents are bounded to 2 MiB; local Git inspection is offline and bounded.
1120
- Both output modes are stdout-only and say verification was not performed.
1121
- Metadata retains historical exclusions, not private/delegated filenames.
1122
- Input text and commit subjects are untrusted data, not instructions or an
1123
- authenticated claim. Supply independent intended behavior, review faults and
1124
- probe afterward; inspection does not create claims or prove coverage.
1125
-
1126
- Without a supplied ID or annotation, generated groups use unfinished
1127
- `TODO-CLAIM-N` placeholders rather than module/function names. Replace them with
1128
- intent-shaped identities when authoring; mechanical grouping does not establish
1129
- an invariant boundary. Existing claims/evidence are never automatically renamed.
1130
-
1131
- `--into` requires an existing disposable draft and exactly one existing claim
1132
- ID. It preserves supplied intent, metadata and prior faults; multiple source
1133
- paths append in deterministic order. `--json` is a read-only preview. Without
1134
- it, the tool keeps private recovery outside the checkout and reports updated
1135
- or unchanged **unproven** draft status. Review and probe are still required.
1136
- Canonical claims/evidence/baseline paths, unsafe aliases and conflicting options
1137
- are refused; partial writes never become success or trigger automatic rollback.
1138
- Limits: 32 sources, 2 MiB per input/output draft or source, 16 MiB source total,
1139
- 4096 generated proposals. These limits are not a performance guarantee.
1140
-
1141
- | Shape | What it proposes |
1142
- |---|---|
1143
- | `condition-forced` | `if (<guard>) {` → `if (false) {` — a guard is a `!…` condition or one whose body returns, throws or 4xx-es |
1144
- | `statement-deleted` | a single-line guard (`if (…) return …;`) or a state change (`x = …;`) removed |
1145
- | `return-altered` | `return <check>;` (`===`, `.includes(`, `&&`, …) → `return true;` |
1146
- | `literal-changed` | `httpOnly`/`secure` flipped, `sameSite` → `none`, a cost/rounds → `1`, a ttl/tolerance/window/limit ×1000 |
1147
- | `call-removed` | a bare `verify…()` / `validate…()` / `check…()` / `authorize…()` call removed |
1148
- | `field-dropped` | a field removed from an object that is returned, built by an arrow, assigned to a payload-ish name, passed to a `save`/`update`/`send`/`write`/… call, or is a `z.object({…})`-style schema; a string entry removed from an allow-list array; a `...base,` line or inline `{ ...base, … }` merge dropped. Only a line that can go on its own (balanced, comma-terminated or followed by the closer); never inside tests, fixtures or migrations |
1149
- | `argument-swapped` | a call kept, its first argument swapped for `undefined` (and `{}` when the argument is itself a call) — only when that argument is derived from a parameter of the enclosing function or a request-like value (`req`, `ctx`, `event`, …). The seam fault: `decide(deriveFrom(input), now)` → `decide(undefined, now)` |
1150
- | `element-removed` | a one-line JSX element — self-closing (`<Toggle … />`) or paired (`<button …>Save</button>`) — removed; the UI shape behind "the toggle is invisible", killable by a browser-layer defender |
1151
- | `handler-dropped` | an `on<Event>={…}` prop removed, whether it is its own line or inline in the tag; the control renders and does nothing |
1152
-
1153
- The table above is JavaScript. `scaffold` reads Python too, and proposes the
1154
- same fault classes in Python syntax: `if <guard>:` → `if False:`,
1155
- `return <check>` → `return True`, `verify=True` → `verify=False`, a
1156
- parameter-derived argument swapped for `None`, a key dropped from a payload
1157
- `dict` or an allow-list. Three differences are deliberate. A statement is
1158
- removed by replacing it with `pass`, never by deleting the line, because a
1159
- block whose only statement is gone is an `IndentationError`; and a line that
1160
- leaves a bracket open (`COLOURS = {`), or sits inside one opened on an earlier
1161
- line (`help="…"` in an `add_argument(`), is never removed at all. Both exist
1162
- because a fault that cannot compile is a `fault-invalid` verdict — a probe run
1163
- spent saying nothing about the tests. Third, a **module-level dunder
1164
- assignment** (`__version__`, `__all__`, `__author__`) is not proposed. Deleting
1165
- one does change behaviour, so it is not an equivalent mutant; it is something
1166
- nobody writes a test for, which is the disqualification an icon already gets on
1167
- the JSX side — a survivor there teaches the survival-learned ordering to prefer
1168
- a barren class. The rule is narrow on purpose: a module-level *constant*
1169
- (`DEFAULT_MAX_SIZE = 100`) is still proposed, because removing a real default
1170
- changes a real default. The two JSX shapes (`element-removed`,
1171
- `handler-dropped`) have no Python meaning and are absent rather than faked.
1172
-
1173
- Every proposal's `find` is the exact line with `expectHits`/`occurrence`
1174
- computed from the file, so it is verifiable by construction; provenance is
1175
- `producer: derived`; `defendedBy` is prefilled from the tests that import
1176
- the module; proposals are grouped under a preceding `@claim <ID>` annotation
1177
- or by enclosing function. Statements are `TODO:` placeholders — a proposal
1178
- becomes a claim only when a human states what it defends. Deterministic
1179
- heuristics, no AST, no LLM; a proposal the tool cannot anchor is never
1180
- emitted.
1181
-
1182
- **Commit `.testguard/baseline.json`; ignore `evidence.json` and `brief.json`.**
1183
- The baseline is the frozen contract; the other two are regenerated per run.
1184
-
1185
- The fault model is the auditable artifact. You never reach 100% of
1186
- correctness; you reach **100% of stated claims verified**, and the statement
1187
- of claims is what an assessor reads. A claims file is code — its `replace`
1188
- strings run under your test runner — so review it like code.
216
+ assertion would notice. `NOCOVER` here means no test even imports the file.
217
+ - **Not a network client.** The CLI never opens a socket and collects no
218
+ telemetry. One exact-pinned runtime dependency (`ajv`), Node ≥ 20, MIT.
1189
219
 
1190
220
  ## Try it
1191
221
 
1192
222
  The repository ships a known-answer fixture with a real blind spot:
1193
223
 
1194
224
  ```bash
1195
- git clone <this repo> && cd testguard && npm install
225
+ git clone https://github.com/raccioly/testguard && cd testguard && npm install
1196
226
  npm test # includes probing the fixture end to end
1197
227
  ```
1198
228
 
1199
229
  `fixtures/known-answer/` is a tiny project whose audit-row test asserts with
1200
230
  `expect.objectContaining({...})` and omits the `content` key. Swap the
1201
231
  redacted text for the raw input and the test stays green. `probe` reports it
1202
- as `SURVIVED`; the fixture's [README](fixtures/known-answer/README.md) walks
1203
- through every verdict.
1204
-
1205
- The same blind spot, in Python, is
1206
- [`fixtures/known-answer-python/`](fixtures/known-answer-python/README.md) —
1207
- run under **both** stdlib `unittest` and `pytest`, which have different report
1208
- shapes and different notions of failure and must still agree on all sixteen
1209
- verdicts.
232
+ as `SURVIVED`; the fixture's [README](https://github.com/raccioly/testguard/blob/main/fixtures/known-answer/README.md)
233
+ walks through every verdict. The same blind spot in Python is
234
+ [`fixtures/known-answer-python/`](https://github.com/raccioly/testguard/blob/main/fixtures/known-answer-python/README.md),
235
+ run under both stdlib `unittest` and `pytest`, which must agree on every
236
+ verdict.
1210
237
 
1211
238
  ## Status
1212
239
 
1213
- Origin summaries are informational: status counts current declared claim
1214
- origins even when evidence is stale; probe evidence and briefs count recorded
1215
- origins across all records before filtering or truncation. Distinct claims and
1216
- records have separate totals; conflicting recorded kinds or references count
1217
- as mixed claims. Labels such as `spec`, `bug` or `incident` do not authenticate
1218
- independence and never waive a failed measurement or change default gates.
1219
- References remain opaque and are not echoed in these summaries. Older strict
1220
- readers reject these additive fields; use the matching candidate reader for
1221
- unreleased formats. The candidate `probe --require-origin spec,adr,bug,incident`
1222
- option selects declared kinds for a complete confirmed current-project probe.
1223
- Repeat the option or use comma-separated kinds; partial, provisional, explicit
1224
- historical and ignore-dirty combinations are refused. Failed policy exits 1
1225
- even when baseline debt or a severity floor hides ordinary findings; unavailable
1226
- current bindings exit 2. The command rechecks native runner/configuration
1227
- membership, current claims, targets, defenders and discovery dependencies before
1228
- publishing. Custom/static runner universes cannot certify this boundary. This is
1229
- a declaration check, not authenticated independence or an atomic disk snapshot.
1230
- Offline status recomputes a current-declaration audit using the recorded eligible
1231
- selection, always marking native freshness unavailable without launching a
1232
- runner. It does not implicitly activate a new gate or replace status's next
1233
- action. Briefs preserve the recorded-run policy before caps and baseline
1234
- filtering, explicitly not current freshness. A zero-item cap never turns
1235
- survivors into an all-defended message.
1236
-
1237
- **v0.5.** Eleven commands (`status`, `init`, `claims`, `probe`, `admit`,
1238
- `replay`, `baseline`, `brief`, `gate`, `scaffold`, `sweep`, `mcp`), vitest, jest,
1239
- Playwright and Python (pytest / stdlib unittest) runners, hand-authored faults
1240
- plus a mechanical scaffold for JavaScript and Python, an agent operating layer
1241
- (`status`, `init`) and a change gate (`gate`). The contract spine — eight JSON
1242
- Schemas shared with the other Guard tools — is under [`spec/`](spec/). One
1243
- exact-pinned runtime dependency (`ajv`, for schema validation); Node ≥ 20, and
1244
- Python ≥ 3.8 only when probing Python.
1245
-
1246
- Not yet: test generation (the acceptance half, `admit`, exists; the generating
1247
- half stays the agent's), runners beyond those four, AST-aware producers, and
1248
- the transfer of a calibration between repositories — `replay` measures it now;
240
+ Thirteen commands (`status`, `init`, `claims`, `probe`, `admit`, `baseline`,
241
+ `brief`, `gate`, `scaffold`, `sweep`, `concerns`, `replay`, `mcp`). Runners
242
+ for vitest, jest, Playwright, Node's built-in test runner, pytest and stdlib
243
+ `unittest`, plus `--runner-cmd` for anything that writes a jest-compatible
244
+ report. Hand-authored faults and a mechanical scaffold for JavaScript,
245
+ TypeScript and Python. The contract is fourteen JSON Schemas under
246
+ [`spec/`](https://github.com/raccioly/testguard/blob/main/spec/README.md),
247
+ shared with the other Guard tools. Every release is in the
248
+ [changelog](https://github.com/raccioly/testguard/blob/main/CHANGELOG.md).
249
+
250
+ Not yet: test generation (the acceptance half, `admit`, exists; the
251
+ generating half stays the agent's), AST-aware producers, and the transfer of
252
+ a calibration between repositories. `replay` measures that transfer now;
1249
253
  whether it carries to a repository with no history is unproven. Each is
1250
254
  designed for; none is claimed.
1251
255
 
1252
- ### Keeping developer machines responsive
1253
-
1254
- Built-in probe, sweep, replay and admission runners default to one worker. Use
1255
- `--workers 2` for a larger Vitest/Jest/Playwright worker ceiling; `--serial` always
1256
- uses one. Python remains serial. Faults and mixed-runner groups run sequentially.
1257
- Custom `--runner-cmd` commands, browser subprocesses, native library threads and
1258
- test-created processes are not an OS resource sandbox. The regression suite also
1259
- caps outer workers at two, or one when fewer than three CPUs are available
1260
- in an interactive run; dedicated CI uses at most two because its acceptance tests start nested runners.
1261
-
1262
- `--budget` bounds each runner invocation; `--command-budget` bounds a complete
1263
- command cooperatively and refuses partial result publication. Cancellation kills
1264
- owned runners before restoring mutations and removing scratch worktrees. Already
1265
- reparented daemons remain outside the portable cleanup guarantee. No runner output
1266
- limit can count as an assertion failure or detection.
1267
-
1268
- New probe evidence includes actual elapsed time, unique fresh runner time and
1269
- setup/discovery/other overhead. `probe --cost` and `claims --cost` display this
1270
- separately from historical attributed record costs. Existing cost ceilings are
1271
- unchanged; old evidence does not supply a measured breakdown.
256
+ ## Contributing
1272
257
 
1273
- Playwright confirmations run the same project-resolved CLI used for discovery,
1274
- avoiding an npm subprocess per run. Python projects whose configuration requires
1275
- pytest-xdist must support serial execution; an incompatible configuration remains
1276
- unproven rather than bypassing the worker policy.
258
+ Bug reports, field reports and documentation fixes are welcome. Read
259
+ [CONTRIBUTING.md](https://github.com/raccioly/testguard/blob/main/CONTRIBUTING.md)
260
+ and [AGENTS.md](https://github.com/raccioly/testguard/blob/main/AGENTS.md)
261
+ first; for help, see [SUPPORT.md](https://github.com/raccioly/testguard/blob/main/SUPPORT.md),
262
+ and report vulnerabilities privately per
263
+ [SECURITY.md](https://github.com/raccioly/testguard/blob/main/SECURITY.md).
1277
264
 
1278
265
  ## Licence
1279
266