testguard-cli 0.5.0__tar.gz → 0.7.0__tar.gz

testguard_cli-0.7.0/.agents/skills/testguard/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: testguard
+description: Prove the test suite defends this project's claims. Use before writing or changing tests, after changing guarded code, and whenever a session starts with a TEST BLINDSPOT CONTEXT block.
+---
+
+# TestGuard — how an agent runs it
+
+TestGuard injects the faults a project's claims forbid and reports every one
+the tests fail to detect. It is a **claim verifier**, not a test generator.
+You operate it through one loop and one source of truth.
+
+## The one source of truth
+
+```bash
+testguard status --json
+```
+
+Read `state` and `next`. Do what `next.command` says. Never infer state from
+which files exist. `state` is one of:
+
+| state | meaning | you do |
+|---|---|---|
+| `no-claims` | no `testguard.claims.json` | `testguard scaffold <file>` for a file with guards; replace every `TODO:` statement with what the code guarantees; keep or drop each proposal; move the claims into `testguard.claims.json` |
+| `unprobed` | claims never probed | `testguard probe` |
+| `evidence-stale` | code, tests or claims changed since the evidence | `testguard probe --include-dirty` (or `--claim <ID>` for one) |
+| `provisional-only` | only `--confirm 1` evidence exists | `testguard probe --confirm 3` |
+| `unproven` | a finding is not covered by the baseline | `next.target` names it; see the verdict table |
+| `clean` | everything killed or baselined | `testguard baseline` if `next` says so; otherwise nothing |
+
+Exit codes: `0` clean · `1` unproven claims (or drift) · `2` precondition failed / nothing to do yet · `3` usage.
+
+## Verdict → action
+
+| verdict | what it means | the only acceptable fix |
+|---|---|---|
+| `SURVIVED` | the defenders stayed green while the claim was false | **write a test** in the defender file that fails with the fault applied and passes on HEAD |
+| `SURVIVED [killed-by-undeclared-tests: …]` | other tests catch it | add those files to the claim's `defendedBy` |
+| `NOCOVER` | no test file imports the target | write a new test file that imports it and asserts the claim |
+| `UNVERIFIABLE` | the fault's anchor no longer matches, or the defenders fail to load | fix the **fault definition** (or the defenders' import); never the code |
+| `FAULT-INVALID` | the replacement does not compile | fix the fault definition |
+| `TIMEOUT` | the fault makes the code hang | write an assertion that fails on it; a hang is not a detection |
+| `FLAKY-DEFENDER` | defenders not green 3/3, or disagreed across runs | fix the flake first; no verdict is trustworthy until then |
+
+## The fix loop
+
+1. `testguard status --json` → take `next.target`.
+2. Write the test. It must **fail when the fault is applied and pass on HEAD**. To check the first half by hand: apply `find` → `replace` in the target file, run the defender, restore.
+3. `testguard probe --claim <ID> --include-dirty` — probes your uncommitted test without touching the tree. `--confirm 1` gives a fast **provisional** signal; a `?` on a verdict means unconfirmed.
+4. When it is `killed` at `--confirm 3`, commit the test.
+5. When everything is killed or baselined, `testguard baseline` if `next` says so, and commit `.testguard/baseline.json`.
+
+## Rules that are not yours to bend
+
+- **Never make a fault die by editing the claims file.** Editing a claim is allowed when the claim was wrong — but the evidence records every fault's content hash, and `status` lists any fault edited after it survived, with its previous verdict, as `changedFaults`. The change is permitted; it is never invisible. Say why in the commit.
+- **A test that asserts current behaviour is not a fix.** If the test would pass with the fault applied, it defends nothing.
+- **Never re-implement the code under test inside the test.** Drive the real function; assert the outcome, not that a function was called.
+- `--confirm 1` results are provisional: they print with `?`, go to `evidence-provisional.json`, and cannot be baselined.
+- Worktree mode probes a **commit**. If `probe` refuses because defenders are dirty, use `--include-dirty`; do not commit half-written tests to satisfy it.
+- Do not add `.testguard/evidence.json` or `brief.json` to git; do commit `baseline.json`.
+
+## Reading the numbers
+
+`N unproven faults across M claims` counts faults and the claims they belong to. There is no single score on purpose: blindness is concentrated, and one number hides where.

testguard_cli-0.7.0/.docguard.json

@@ -0,0 +1,36 @@
+{
+  "$schema": "https://raccioly.github.io/docguard/schemas/docguard-config.schema.json",
+  "projectName": "testguard-cli",
+  "version": "0.6",
+  "profile": "enterprise",
+  "projectType": "cli",
+  "projectTypeConfig": {
+    "needsEnvVars": false,
+    "needsEnvExample": false,
+    "needsE2E": false,
+    "needsDatabase": false
+  },
+  "requiredFiles": {
+    "canonical": [
+      "docs-canonical/ARCHITECTURE.md",
+      "docs-canonical/DATA-MODEL.md",
+      "docs-canonical/SECURITY.md",
+      "docs-canonical/TEST-SPEC.md",
+      "docs-canonical/ENVIRONMENT.md",
+      "docs-canonical/REQUIREMENTS.md"
+    ]
+  },
+  "validators": {
+    "structure": true,
+    "docsSync": true,
+    "drift": true,
+    "changelog": true,
+    "architecture": true,
+    "testSpec": true,
+    "security": true,
+    "environment": true,
+    "freshness": true
+  },
+  "severity": {},
+  "findingSeverity": {}
+}

testguard_cli-0.7.0/.docguardignore

@@ -0,0 +1,30 @@
+# .docguardignore — paths to exclude from DocGuard validation.
+# Gitignore-style syntax: one pattern per line, # for comments.
+# Merged into config.ignore (in .docguard.json) at runtime.
+#
+# Common examples:
+#   build/                   # exclude a directory
+#   **/__generated__/**      # exclude anything in any __generated__ dir
+#   vendor/legacy.ts         # exclude a single file
+#   **/*.snap                # exclude all files matching a glob
+#
+# Build outputs, vendored libs, generated code are good candidates here.
+
+# Vendored / generated code that's not yours to document
+**/__generated__/**
+**/generated/**
+**/*.generated.*
+
+# Migrations and lock files
+**/migrations/**
+**/*.lock
+package-lock.json
+yarn.lock
+pnpm-lock.yaml
+Cargo.lock
+poetry.lock
+
+# Common build artifacts (defaults also cover these, but listing here is clearer)
+# dist/
+# build/
+# coverage/

testguard_cli-0.7.0/.github/scripts/build-one-pager.mjs

@@ -0,0 +1,49 @@
+#!/usr/bin/env node
+/**
+ * Render docs/testguard-explained.html to PDF with headless Chrome.
+ *
+ * Chrome rather than a library because this repository has one pinned runtime
+ * dependency and keeps it that way: a PDF renderer would be the second, and it
+ * would be needed only to build a document nobody imports.
+ *
+ * Set CHROME to override the binary. Exits 0 with a notice when no Chrome is
+ * present, so a checkout without one can still run every other script.
+ */
+import { existsSync } from 'node:fs';
+import { spawnSync } from 'node:child_process';
+import { dirname, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const root = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..');
+const src = join(root, 'docs', 'testguard-explained.html');
+const out = join(root, 'docs', 'testguard-explained.pdf');
+
+const candidates = [
+  process.env.CHROME,
+  '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
+  '/Applications/Chromium.app/Contents/MacOS/Chromium',
+  '/usr/bin/google-chrome',
+  '/usr/bin/chromium',
+  '/usr/bin/chromium-browser',
+].filter(Boolean);
+
+const chrome = candidates.find((c) => existsSync(c));
+if (!chrome) {
+  console.log('no Chrome found; skipping PDF build (set CHROME=<path> to force)');
+  process.exit(0);
+}
+
+const r = spawnSync(chrome, [
+  '--headless',
+  '--disable-gpu',
+  '--no-sandbox',
+  '--no-pdf-header-footer',
+  `--print-to-pdf=${out}`,
+  `file://${src}`,
+], { encoding: 'utf8' });
+
+if (r.status !== 0 || !existsSync(out)) {
+  console.error(r.stderr || 'chrome produced no PDF');
+  process.exit(1);
+}
+console.log(`docs/testguard-explained.pdf written with ${chrome.split('/').pop()}`);

testguard_cli-0.7.0/.github/scripts/check-python-fixture.mjs

@@ -0,0 +1,53 @@
+#!/usr/bin/env node
+/**
+ * Compare a probe of `fixtures/known-answer-python` against its oracle.
+ *
+ * Shared by both CI legs so the stdlib engine and pytest are held to exactly
+ * the same expectations — the point of the fixture is that two engines with
+ * different report shapes must not disagree about a single verdict.
+ */
+import { readFileSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const [evidencePath, expectedEngine] = process.argv.slice(2);
+if (!evidencePath) {
+  console.error('usage: check-python-fixture.mjs <evidence.json> [engine]');
+  process.exit(2);
+}
+
+const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
+const oracle = JSON.parse(readFileSync(join(ROOT, 'fixtures', 'known-answer-python', 'expected.json'), 'utf8')).expected;
+const evidence = JSON.parse(readFileSync(evidencePath, 'utf8'));
+
+const problems = [];
+if (expectedEngine && evidence.run.runner.name !== expectedEngine) {
+  problems.push(`runner was ${evidence.run.runner.name}, expected ${expectedEngine}`);
+}
+
+const seen = new Set();
+for (const r of evidence.records) {
+  const key = `${r.claim.id}/${r.subject.id}`;
+  seen.add(key);
+  const want = oracle[key];
+  if (!want) { problems.push(`${key}: not in expected.json`); continue; }
+  if (want.verdict !== r.verdict) problems.push(`${key}: ${r.verdict}, expected ${want.verdict}`);
+  if (want.reason && want.reason !== r.detail.reason) problems.push(`${key}: reason ${r.detail.reason}, expected ${want.reason}`);
+  if (Boolean(want.targetNotImported) !== Boolean(r.detail.targetNotImported)) {
+    problems.push(`${key}: targetNotImported ${Boolean(r.detail.targetNotImported)}, expected ${Boolean(want.targetNotImported)}`);
+  }
+  // The negative control is pinned too, not merely tolerated: a fixture that
+  // accepted either answer would stop defending the one behaviour it exists
+  // to prove, and the verdict alone cannot tell "the defenders reached this
+  // file and said nothing" from "they never reached it".
+  if ((want.negativeControl ?? null) !== (r.detail.negativeControl ?? null)) {
+    problems.push(`${key}: negativeControl ${r.detail.negativeControl ?? 'absent'}, expected ${want.negativeControl ?? 'absent'}`);
+  }
+}
+for (const key of Object.keys(oracle)) if (!seen.has(key)) problems.push(`${key}: expected but never probed`);
+
+if (problems.length) {
+  for (const p of problems) console.error(`::error::${p}`);
+  process.exit(1);
+}
+console.log(`python fixture verdicts match expected.json under ${evidence.run.runner.name}: ${evidence.records.length} records`);

{testguard_cli-0.5.0 → testguard_cli-0.7.0}/.github/scripts/sync-release-version.mjs

@@ -5,7 +5,15 @@
  *   - pyproject.toml         `version = "x.y.z"`
  *   - action.yml             default of the `version` input
  *   - packaging/homebrew/*.rb the tarball URL (sha256 is set after publish)
- *   - packaging/gitlab/*.yml  the include URL tag and TESTGUARD_VERSION
+ *   - packaging/gitlab/*.yml  the include URL tag and the `version` input default
+ *   - README.md               the GitHub Action reference AND the GitLab include URL
+ *
+ * A file may carry more than one form, and each needs its own entry: README
+ * holds both the `raccioly/testguard@vX.Y.Z` action reference and the
+ * `testguard/vX.Y.Z/packaging` template URL. Only the first was listed, so
+ * v0.6.0 shipped with both README copies of the GitLab URL left at v0.5.0
+ * while `--check` reported "all version surfaces at 0.6.0" — a surface this
+ * script was never told to look at cannot drift in its eyes.
  *
  * `--check` verifies instead of writing (used by release.yml).
  */
@@ -22,8 +30,9 @@ const surfaces = [
   ['action.yml', /(\n  version:\n    description: [^\n]*\n    required: false\n    default: ')[^']*(')/, `$1${version}$2`],
   ['packaging/homebrew/testguard.rb', /testguard-cli-\d+\.\d+\.\d+\.tgz/g, `testguard-cli-${version}.tgz`],
   ['README.md', /raccioly\/testguard@v\d+\.\d+\.\d+/g, `raccioly/testguard@v${version}`],
+  ['README.md', /testguard\/v\d+\.\d+\.\d+\/packaging/g, `testguard/v${version}/packaging`],
   ['packaging/gitlab/testguard.gitlab-ci.yml', /testguard\/v\d+\.\d+\.\d+\/packaging/g, `testguard/v${version}/packaging`],
-  ['packaging/gitlab/testguard.gitlab-ci.yml', /TESTGUARD_VERSION: "\d+\.\d+\.\d+"/, `TESTGUARD_VERSION: "${version}"`],
+  ['packaging/gitlab/testguard.gitlab-ci.yml', /(\n    version:\n      description: [^\n]*\n      default: ")\d+\.\d+\.\d+(")/, `$1${version}$2`],
 ];
 
 let drift = 0;

testguard_cli-0.7.0/.github/workflows/ci.yml

@@ -0,0 +1,360 @@
+name: TestGuard CI
+
+on:
+  workflow_dispatch: # manual recovery for pushes that don't auto-trigger (e.g. release PRs opened by GITHUB_TOKEN)
+  push:
+    branches: [main]
+    tags: ['v*'] # publish-check below runs on tags only
+  pull_request:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        # 20 is the floor (node:util.parseArgs). 24 ships npm 11 — keep it so
+        # `npm ci` exercises the npm that treats some config as fatal where
+        # npm 10 is silent. Do not drop 24.
+        node-version: [20, 22, 24]
+
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0 # probe builds a git worktree from HEAD; the self-probe below needs history
+
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: ${{ matrix.node-version }}
+
+      - name: Install
+        run: npm ci
+
+      # ── Syntax-check EVERY source, not just what the entry point eagerly
+      #    imports; a lazily loaded module's parse error would otherwise slip. ──
+      - name: Syntax-check all sources
+        run: find cli src spec -name '*.mjs' -print0 | xargs -0 -n1 node --check
+
+      # ── Install-from-tarball smoke: the package must run as INSTALLED, with
+      #    production dependencies only. 0.1.0 shipped unrunnable because the
+      #    schema validator's dependency was a devDependency; the suite, which
+      #    runs from the checkout, could not see it. ──
+      - name: Install smoke (pack → install --omit=dev → run)
+        run: npm run test:install
+
+      - name: Python wrapper imports (PyPI surface)
+        run: python3 -c "import testguard_cli.wrapper; print('testguard_cli.wrapper OK')"
+
+      # ── Test suite with a RUNTIME BUDGET. A test that walks the wrong tree
+      #    or probes far more than it should still passes — it just takes
+      #    forever. Only a duration gate catches that class. Normal: ~15s
+      #    locally, ~30–60s here. Budget is loose on purpose. ──
+      - name: Run tests (with runtime budget)
+        shell: bash
+        env:
+          TEST_BUDGET_MS: 240000
+        run: |
+          START=$(date +%s%3N)
+          npm test
+          DURATION=$(( $(date +%s%3N) - START ))
+          echo "Test suite duration: ${DURATION}ms (budget: ${TEST_BUDGET_MS}ms)"
+          if [ "$DURATION" -gt "$TEST_BUDGET_MS" ]; then
+            echo "::error::Test suite took ${DURATION}ms, over the ${TEST_BUDGET_MS}ms budget — something is probing far more than it should."
+            exit 1
+          fi
+
+      # ── Dogfood: the real binary against the known-answer fixture, as a
+      #    user would run it (the suite drives main() in-process). ──
+      - name: CLI version
+        run: node cli/testguard.mjs --version
+
+      - name: claims — fixture lists cleanly, no drift
+        run: node cli/testguard.mjs claims fixtures/known-answer
+
+      - name: probe → baseline → probe → brief on the fixture (through the bin)
+        shell: bash
+        run: |
+          set -o pipefail
+          S=$(mktemp -d)
+          cp -R fixtures/known-answer/. "$S/"
+          rm -rf "$S/node_modules" "$S/.flake-counter" "$S/.testguard"
+          ln -s "$GITHUB_WORKSPACE/node_modules" "$S/node_modules"
+          git -C "$S" init -q
+          git -C "$S" -c user.email=ci@example.invalid -c user.name=ci add -A
+          git -C "$S" -c user.email=ci@example.invalid -c user.name=ci commit -qm fixture
+          B="$GITHUB_WORKSPACE/cli/testguard.mjs"
+          status=0; node "$B" probe "$S" --budget 60000 --quiet || status=$?
+          [ "$status" -eq 1 ] || { echo "::error::expected exit 1 (unproven claims), got $status"; exit 1; }
+          node "$B" baseline "$S"
+          status=0; node "$B" probe "$S" --budget 60000 --quiet || status=$?
+          [ "$status" -eq 0 ] || { echo "::error::expected exit 0 after baseline, got $status"; exit 1; }
+          node "$B" brief "$S" --text | head -12
+          # every verdict must appear exactly once per expected.json
+          node --input-type=module -e "
+            import { readFileSync } from 'node:fs';
+            const ev = JSON.parse(readFileSync('$S/.testguard/evidence.json','utf8'));
+            const exp = JSON.parse(readFileSync('fixtures/known-answer/expected.json','utf8')).expected;
+            const bad = ev.records.filter(r => exp[r.claim.id+'/'+r.subject.id].verdict !== r.verdict);
+            if (bad.length) { console.error('MISMATCH', bad.map(r => r.claim.id+'/'+r.subject.id+':'+r.verdict)); process.exit(1); }
+            console.log('fixture verdicts match expected.json:', ev.records.length);
+          "
+
+      - name: probe the jest fixture (through the bin, --runner jest)
+        shell: bash
+        run: |
+          S=$(mktemp -d)
+          cp -R fixtures/known-answer-jest/. "$S/"
+          rm -rf "$S/node_modules" "$S/.flake-counter" "$S/.testguard"
+          ln -s "$GITHUB_WORKSPACE/node_modules" "$S/node_modules"
+          git -C "$S" init -q && git -C "$S" -c user.email=ci@example.invalid -c user.name=ci add -A && git -C "$S" -c user.email=ci@example.invalid -c user.name=ci commit -qm fixture
+          status=0; node cli/testguard.mjs probe "$S" --runner jest --budget 60000 --quiet --no-escalate || status=$?
+          [ "$status" -eq 1 ] || { echo "::error::expected exit 1, got $status"; exit 1; }
+          node --input-type=module -e "
+            import { readFileSync } from 'node:fs';
+            const ev = JSON.parse(readFileSync('$S/.testguard/evidence.json','utf8'));
+            const exp = JSON.parse(readFileSync('fixtures/known-answer-jest/expected.json','utf8')).expected;
+            if (ev.run.runner.name !== 'jest') { console.error('runner was', ev.run.runner.name); process.exit(1); }
+            const bad = ev.records.filter(r => exp[r.claim.id+'/'+r.subject.id].verdict !== r.verdict);
+            if (bad.length) { console.error('MISMATCH', bad.map(r => r.claim.id+'/'+r.subject.id+':'+r.verdict)); process.exit(1); }
+            console.log('jest fixture verdicts match expected.json:', ev.records.length);
+          "
+
+      # ── The MCP server, driven as a client would: handshake, list, call.
+      #    A harness-independent surface is only worth having if it actually
+      #    speaks the protocol from a cold start. ──
+      - name: mcp — handshake, tools/list, tools/call over stdio
+        shell: bash
+        run: |
+          printf '%s\n' \
+            '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
+            '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
+            '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
+            '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"testguard_status","arguments":{"dir":"."}}}' \
+            | node cli/testguard.mjs mcp > "$RUNNER_TEMP/mcp.jsonl"
+          node --input-type=module -e "
+            import { readFileSync } from 'node:fs';
+            const msgs = readFileSync(process.env.RUNNER_TEMP + '/mcp.jsonl', 'utf8').trim().split('\n').map((l) => JSON.parse(l));
+            if (msgs.length !== 3) { console.error('expected 3 replies (the notification is never answered), got', msgs.length); process.exit(1); }
+            const [init, list, call] = msgs;
+            if (!init.result.protocolVersion) { console.error('no protocolVersion'); process.exit(1); }
+            if (list.result.tools.length !== 5) { console.error('expected 5 tools, got', list.result.tools.length); process.exit(1); }
+            if (!list.result.tools.every((t) => t.annotations.readOnlyHint)) { console.error('a tool is not marked read-only'); process.exit(1); }
+            if (call.result.isError) { console.error('status call failed:', call.result.content[0].text); process.exit(1); }
+            if (!call.result.structuredContent.next.action) { console.error('status carried no next action'); process.exit(1); }
+            console.log('mcp:', list.result.tools.map((t) => t.name).join(', '), '→ state', call.result.structuredContent.state);
+          "
+
+      # ── Documentation is canon: DocGuard validates this repository against
+      #    docs-canonical/. This is the authoritative check — a machine-local
+      #    .git/hooks file is neither shared with contributors nor reviewable,
+      #    so CI is where doc drift is actually caught. `guard` exits 2 for
+      #    warnings, which we surface without failing; errors fail the build. ──
+      - name: DocGuard — documentation matches the code
+        run: |
+          # Capture the exit code explicitly. A pipeline under the default
+          # `bash -e` aborts the step before any branch on its status can run,
+          # and `|| code=$?` is the one form errexit does not trip on.
+          code=0
+          npx -y docguard-cli@0.41.5 guard > "$RUNNER_TEMP/docguard-guard.txt" 2>&1 || code=$?
+          cat "$RUNNER_TEMP/docguard-guard.txt"
+          if [ "$code" -eq 2 ]; then
+            echo "::warning::DocGuard reported warnings (exit 2); see the output above."
+          elif [ "$code" -ne 0 ]; then
+            echo "::error::DocGuard guard failed (exit $code) — documentation contradicts the code."
+            exit 1
+          fi
+
+      - name: DocGuard — maturity score (reported, not gated)
+        if: always()
+        run: npx -y docguard-cli@0.41.5 score || true
+
+      # ── Every change needs a claim: the gate runs on this repository's own
+      #    pull requests. A new source file without a claim (or an excusing
+      #    ignore entry with a reason) turns the PR red. ──
+      # A claim that disappeared is invisible to every other check: `probe`
+      # verifies what is there, `gate` sees the file covered by another claim.
+      # Removal is allowed; it is never silent.
+      - name: claims — no claim or fault removed without a reason
+        if: github.event_name == 'pull_request'
+        run: node cli/testguard.mjs claims . --since "origin/${{ github.base_ref }}"
+
+      - name: gate — every changed source file carries a claim
+        if: github.event_name == 'pull_request'
+        run: node cli/testguard.mjs gate . --changed "origin/${{ github.base_ref }}"
+
+      # ── Self-verification: TestGuard is governed by the standard it defines.
+      #    testguard.claims.json states invariants of the tool itself; every
+      #    fault must be killed by the tool's own tests. Exit 0 = all defended. ──
+      #    Runs on ONE Node leg: the verdicts do not depend on the Node minor,
+      #    and a from-scratch probe is ~10 minutes (it was ~24 before the
+    #    defender split in #67; `--cost` below prints where the time goes,
+    #    so a regression in gate cost is visible in the log). The previous run's
+      #    evidence is restored from the cache so claims whose target and
+      #    defenders did not change reuse their verdict (the tool's own
+      #    reuse rule); only what changed is re-probed. No escalation in CI:
+      #    a survivor here is a failure either way. ──
+      - name: Restore previous self-probe evidence (verdict reuse)
+        if: matrix.node-version == 22
+        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+        with:
+          path: .testguard/ci-self-evidence.json
+          key: self-evidence-${{ hashFiles('testguard.claims.json') }}-${{ github.sha }}
+          restore-keys: |
+            self-evidence-${{ hashFiles('testguard.claims.json') }}-
+            self-evidence-
+
+      - name: probe TestGuard's own claims
+        if: matrix.node-version == 22
+        run: node cli/testguard.mjs probe . --budget 180000 --no-escalate --cost --out .testguard/ci-self-evidence.json
+
+      # A stable, branch-named artifact so a developer can fetch the evidence
+      # CI produced without knowing a run id (README: read CI's evidence
+      # locally). One Node leg is enough; the verdicts are identical.
+      - name: Upload evidence under a stable, branch-named artifact
+        if: always() && matrix.node-version == 22 && github.event_name == 'push'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: testguard-evidence-${{ github.ref_name }}
+          path: .testguard/ci-self-evidence.json
+          if-no-files-found: warn
+
+      - name: Upload self-probe evidence
+        if: always() && matrix.node-version == 22
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: testguard-self-evidence
+          path: .testguard/ci-self-evidence.json
+          if-no-files-found: warn
+
+  # ── Python: two engines, one oracle. The stdlib leg runs FIRST and with no
+  #    pip install of any kind, because "a project with no test dependencies
+  #    still probes" is a claim about the tool and the only way to keep it true
+  #    is to prove it in an environment that has none. The pytest leg then
+  #    installs pytest into a venv and must reach the SAME sixteen verdicts
+  #    through a different report shape. ──
+  python:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 22
+          cache: npm
+      - name: Install (tool only)
+        run: npm ci
+
+      # A bare virtualenv, not the system interpreter: the claim under test is
+      # "a project whose test dependencies are the standard library still
+      # probes", and asserting that the runner image happens to lack pytest
+      # would make this job fail for a reason that has nothing to do with the
+      # product the day that image changes. Guarantee the condition instead.
+      - name: The stdlib engine must need nothing installed
+        shell: bash
+        run: |
+          set -o pipefail
+          python3 -m venv "$RUNNER_TEMP/bare"
+          if "$RUNNER_TEMP/bare/bin/python" -c 'import pytest' 2>/dev/null; then
+            echo "::error::the bare venv can import pytest; it is not bare and this leg proves nothing"
+            exit 1
+          fi
+          TESTGUARD_PYTHON="$RUNNER_TEMP/bare/bin/python" \
+            npx vitest run test/runner-python.test.mjs test/pyimports.test.mjs test/scaffold.python.test.mjs test/probe.python.test.mjs
+
+      - name: probe the Python fixture through the bin, stdlib engine
+        shell: bash
+        run: |
+          set -o pipefail
+          S=$(mktemp -d)
+          cp -R fixtures/known-answer-python/. "$S/"
+          rm -rf "$S/.flake-counter" "$S/.testguard" "$S/.pytest_cache"
+          find "$S" -name __pycache__ -type d -exec rm -rf {} + 2>/dev/null || true
+          git -C "$S" init -q
+          git -C "$S" -c user.email=ci@example.invalid -c user.name=ci add -A
+          git -C "$S" -c user.email=ci@example.invalid -c user.name=ci commit -qm fixture
+          status=0; node cli/testguard.mjs probe "$S" --budget 20000 --quiet --no-escalate --python "$RUNNER_TEMP/bare/bin/python" || status=$?
+          [ "$status" -eq 1 ] || { echo "::error::expected exit 1, got $status"; exit 1; }
+          node .github/scripts/check-python-fixture.mjs "$S/.testguard/evidence.json" unittest
+
+      - name: install pytest into a venv, and reach the same verdicts through it
+        shell: bash
+        run: |
+          set -o pipefail
+          python3 -m venv "$RUNNER_TEMP/pyenv"
+          "$RUNNER_TEMP/pyenv/bin/pip" install -q pytest
+          S=$(mktemp -d)
+          cp -R fixtures/known-answer-python/. "$S/"
+          rm -rf "$S/.flake-counter" "$S/.testguard" "$S/.pytest_cache"
+          find "$S" -name __pycache__ -type d -exec rm -rf {} + 2>/dev/null || true
+          git -C "$S" init -q
+          git -C "$S" -c user.email=ci@example.invalid -c user.name=ci add -A
+          git -C "$S" -c user.email=ci@example.invalid -c user.name=ci commit -qm fixture
+          status=0; node cli/testguard.mjs probe "$S" --budget 20000 --quiet --no-escalate \
+            --runner pytest --python "$RUNNER_TEMP/pyenv/bin/python" || status=$?
+          [ "$status" -eq 1 ] || { echo "::error::expected exit 1, got $status"; exit 1; }
+          node .github/scripts/check-python-fixture.mjs "$S/.testguard/evidence.json" pytest
+
+      - name: the acceptance suite again, now against an interpreter that HAS pytest
+        shell: bash
+        run: |
+          TESTGUARD_PYTHON="$RUNNER_TEMP/pyenv/bin/python" npx vitest run test/probe.python.test.mjs
+
+  # ── Playwright: the per-file runner is proven against its own browserless
+  #    fixture, whose dependencies live in the fixture (never in the tool's
+  #    package.json). Installed and run here only. ──
+  playwright:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 22
+          cache: npm
+          cache-dependency-path: |
+            package-lock.json
+            fixtures/known-answer-playwright/package-lock.json
+      - name: Install (tool + fixture)
+        run: |
+          npm ci
+          npm ci --prefix fixtures/known-answer-playwright
+      - name: Playwright adapter units + mixed-defender acceptance
+        run: npx vitest run test/runner-playwright.test.mjs test/probe.playwright.test.mjs
+      - name: probe the playwright fixture through the bin (mixed defenders, auto runner)
+        shell: bash
+        run: |
+          set -o pipefail
+          S=$(mktemp -d)
+          cp -R fixtures/known-answer-playwright/. "$S/"
+          rm -rf "$S/node_modules" "$S/.flake-counter" "$S/.testguard" "$S/test-results"
+          ln -s "$GITHUB_WORKSPACE/fixtures/known-answer-playwright/node_modules" "$S/node_modules"
+          git -C "$S" init -q && git -C "$S" -c user.email=ci@example.invalid -c user.name=ci add -A && git -C "$S" -c user.email=ci@example.invalid -c user.name=ci commit -qm fixture
+          status=0; node cli/testguard.mjs probe "$S" --budget 60000 --quiet --no-escalate || status=$?
+          [ "$status" -eq 1 ] || { echo "::error::expected exit 1, got $status"; exit 1; }
+          node --input-type=module -e "
+            import { readFileSync } from 'node:fs';
+            const ev = JSON.parse(readFileSync('$S/.testguard/evidence.json','utf8'));
+            const exp = JSON.parse(readFileSync('fixtures/known-answer-playwright/expected.json','utf8')).expected;
+            const names = (ev.run.runners ?? []).map(r => r.name).sort().join(',');
+            if (names !== 'playwright,vitest') { console.error('runners were', names); process.exit(1); }
+            const bad = ev.records.filter(r => exp[r.claim.id+'/'+r.subject.id].verdict !== r.verdict);
+            if (bad.length) { console.error('MISMATCH', bad.map(r => r.claim.id+'/'+r.subject.id+':'+r.verdict)); process.exit(1); }
+            console.log('playwright fixture verdicts match expected.json:', ev.records.length);
+          "
+
+  # ── npm publish dry-run on tags ──
+  publish-check:
+    runs-on: ubuntu-latest
+    if: startsWith(github.ref, 'refs/tags/v')
+    needs: test
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 24
+      - run: npm publish --dry-run

{testguard_cli-0.5.0 → testguard_cli-0.7.0}/.github/workflows/release.yml

@@ -80,7 +80,7 @@ jobs:
       - name: Install smoke — the tarball must run with production deps only
         run: npm run test:install
       - name: Self-probe must be fully defended before anything ships
-        run: node cli/testguard.mjs probe . --budget 180000 --out "$RUNNER_TEMP/self-evidence.json" --quiet
+        run: node cli/testguard.mjs probe . --budget 180000 --out "$RUNNER_TEMP/self-evidence.json" --quiet --cost
       - name: Version surfaces are in sync
         run: node .github/scripts/sync-release-version.mjs --check
 

{testguard_cli-0.5.0 → testguard_cli-0.7.0}/.gitignore

@@ -9,10 +9,12 @@ coverage/
 .testguard/evidence-partial.json
 .testguard/brief.json
 .testguard/gate.json
+.testguard/ci-self-evidence.json
 .testguard/scaffold-*.json
 # Local agent tooling — not part of the public project
 .wolf/
 .claude/
+.agent/
 CLAUDE.local.md
 graphify-out/
 .vitest/
@@ -21,3 +23,8 @@ dist/
 build/
 *.egg-info/
 __pycache__/
+
+# Never commit secrets
+.env
+.env.*
+!.env.example