testguard-cli 0.6.0__tar.gz → 0.7.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- testguard_cli-0.7.0/.agents/skills/testguard/SKILL.md +63 -0
- testguard_cli-0.7.0/.docguard.json +36 -0
- testguard_cli-0.7.0/.docguardignore +30 -0
- testguard_cli-0.7.0/.github/scripts/build-one-pager.mjs +49 -0
- testguard_cli-0.7.0/.github/scripts/check-python-fixture.mjs +53 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/scripts/sync-release-version.mjs +9 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/workflows/ci.yml +102 -2
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/workflows/release.yml +1 -1
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.gitignore +6 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.npmignore +4 -0
- testguard_cli-0.7.0/.pre-commit-config.yaml +40 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/AGENTS.md +6 -1
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/CHANGELOG.md +373 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/CLAUDE.md +2 -0
- testguard_cli-0.7.0/DRIFT-LOG.md +32 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/PKG-INFO +280 -30
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/README.md +277 -27
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/action.yml +12 -2
- testguard_cli-0.7.0/docs/testguard-explained.html +774 -0
- testguard_cli-0.7.0/docs/testguard-explained.pdf +0 -0
- testguard_cli-0.7.0/docs-canonical/ARCHITECTURE.md +184 -0
- testguard_cli-0.7.0/docs-canonical/DATA-MODEL.md +121 -0
- testguard_cli-0.7.0/docs-canonical/ENVIRONMENT.md +72 -0
- testguard_cli-0.7.0/docs-canonical/REQUIREMENTS.md +115 -0
- testguard_cli-0.7.0/docs-canonical/SECURITY.md +103 -0
- testguard_cli-0.7.0/docs-canonical/TEST-SPEC.md +126 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/.gitignore +4 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/README.md +91 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/demo/__init__.py +0 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/demo/export.py +5 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/demo/orphan.py +5 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/demo/pipeline.py +7 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/demo/redact.py +50 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/expected.json +80 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/testguard.claims.json +425 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/tests/test_export_mocked.py +17 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/tests/test_flaky.py +27 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/tests/test_patched.py +18 -0
- testguard_cli-0.7.0/fixtures/known-answer-python/tests/test_redact.py +40 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/package.json +2 -2
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/packaging/gitlab/testguard.gitlab-ci.yml +9 -2
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/packaging/homebrew/testguard.rb +1 -1
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/pyproject.toml +3 -3
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/GATE-SEMANTICS.md +112 -5
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/README.md +28 -10
- testguard_cli-0.7.0/spec/conformance/examples/claims-methods.json +96 -0
- testguard_cli-0.7.0/spec/conformance/examples/evidence-disturbed-run.json +244 -0
- testguard_cli-0.7.0/spec/conformance/examples/evidence-negative-control.json +335 -0
- testguard_cli-0.7.0/spec/conformance/examples/evidence-python.json +189 -0
- testguard_cli-0.7.0/spec/conformance/examples/evidence-scan-method.json +52 -0
- testguard_cli-0.7.0/spec/conformance/invalid/claims.fault-injection-without-a-substitution.json +33 -0
- testguard_cli-0.7.0/spec/conformance/invalid/evidence.attribute-signal-not-a-defender.json +169 -0
- testguard_cli-0.7.0/spec/conformance/invalid/evidence.attribute-signal-without-attributes.json +168 -0
- testguard_cli-0.7.0/spec/conformance/invalid/evidence.fault-injection-without-its-evidence.json +54 -0
- testguard_cli-0.7.0/spec/conformance/invalid/evidence.negative-control-not-reached-but-survived.json +190 -0
- testguard_cli-0.7.0/spec/conformance/invalid/evidence.negative-control-reached-on-a-kill.json +190 -0
- testguard_cli-0.7.0/spec/conformance/invalid/evidence.probe-error-without-message.json +243 -0
- testguard_cli-0.7.0/spec/conformance/invalid/evidence.restore-skipped-in-place.json +234 -0
- testguard_cli-0.7.0/spec/conformance/invalid/evidence.target-not-imported-but-killed.json +163 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/schemas.test.mjs +4 -1
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/lib/validate.mjs +84 -8
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/schemas/baseline.schema.json +7 -7
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/schemas/brief.schema.json +12 -12
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/schemas/calibration.schema.json +5 -5
- testguard_cli-0.7.0/spec/schemas/claims.schema.json +200 -0
- testguard_cli-0.7.0/spec/schemas/common.schema.json +223 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/schemas/evidence.schema.json +169 -73
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/schemas/gate.schema.json +12 -12
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/schemas/ignore.schema.json +4 -4
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/schemas/replay.schema.json +10 -10
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/schemas/status.schema.json +25 -25
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/cli.mjs +20 -5
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/claims.mjs +22 -2
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/probe.mjs +24 -4
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/gate/changed.mjs +9 -2
- testguard_cli-0.7.0/src/probe/attribution.mjs +123 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/classify.mjs +18 -2
- testguard_cli-0.7.0/src/probe/cost.mjs +145 -0
- testguard_cli-0.7.0/src/probe/discover.mjs +29 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/inject.mjs +41 -9
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/mocks.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/probe.mjs +235 -52
- testguard_cli-0.7.0/src/probe/progress.mjs +87 -0
- testguard_cli-0.7.0/src/probe/pyimports.mjs +287 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/rank.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/runners/index.mjs +38 -11
- testguard_cli-0.7.0/src/probe/runners/jest.mjs +21 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/runners/playwright.mjs +9 -0
- testguard_cli-0.7.0/src/probe/runners/python/_testguard_pytest_plugin.py +132 -0
- testguard_cli-0.7.0/src/probe/runners/python/_testguard_report.py +141 -0
- testguard_cli-0.7.0/src/probe/runners/python/_testguard_unittest_main.py +168 -0
- testguard_cli-0.7.0/src/probe/runners/python.mjs +316 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/runners/shared.mjs +10 -1
- testguard_cli-0.7.0/src/probe/runners/vitest.mjs +20 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/replay/replay.mjs +3 -2
- testguard_cli-0.7.0/src/scaffold/producers.python.mjs +209 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/scaffold/scaffold.mjs +86 -32
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/util/glob.mjs +7 -1
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/admit.test.mjs +2 -0
- testguard_cli-0.7.0/test/attribution.test.mjs +198 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/baseline.test.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/brief.test.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/classify.test.mjs +4 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/contention.test.mjs +2 -0
- testguard_cli-0.7.0/test/cost.test.mjs +127 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/discover.test.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/gate.test.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/gitlab-template.test.mjs +1 -1
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/init.test.mjs +2 -0
- testguard_cli-0.7.0/test/inject.test.mjs +93 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/mcp.test.mjs +3 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/mocks.test.mjs +2 -0
- testguard_cli-0.7.0/test/negative-control.test.mjs +184 -0
- testguard_cli-0.7.0/test/probe-error.test.mjs +182 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/probe-preconditions.test.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/probe.fixture.test.mjs +35 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/probe.jest.test.mjs +2 -0
- testguard_cli-0.7.0/test/probe.python.test.mjs +223 -0
- testguard_cli-0.7.0/test/progress.test.mjs +96 -0
- testguard_cli-0.7.0/test/pyimports.test.mjs +210 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/release-sync.test.mjs +61 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/replay.test.mjs +2 -0
- testguard_cli-0.7.0/test/runner-argv.test.mjs +129 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/runner-command.test.mjs +17 -1
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/runner-playwright.test.mjs +6 -1
- testguard_cli-0.7.0/test/runner-python.test.mjs +202 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/runner-vitest.test.mjs +27 -0
- testguard_cli-0.7.0/test/samples/python-report-collect-error.json +10 -0
- testguard_cli-0.7.0/test/samples/python-report.json +16 -0
- testguard_cli-0.7.0/test/scaffold.python.test.mjs +153 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/scaffold.test.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/status.test.mjs +2 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/worktree.test.mjs +51 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/writer.test.mjs +3 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/testguard.claims.json +974 -38
- testguard_cli-0.7.0/testguard.ignore.json +62 -0
- testguard_cli-0.6.0/spec/schemas/claims.schema.json +0 -103
- testguard_cli-0.6.0/spec/schemas/common.schema.json +0 -125
- testguard_cli-0.6.0/src/probe/discover.mjs +0 -21
- testguard_cli-0.6.0/src/probe/runners/jest.mjs +0 -9
- testguard_cli-0.6.0/src/probe/runners/vitest.mjs +0 -8
- testguard_cli-0.6.0/test/inject.test.mjs +0 -46
- testguard_cli-0.6.0/testguard.ignore.json +0 -41
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.codex/hooks.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.gitattributes +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/dependabot.yml +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/scripts/install-smoke.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/workflows/auto-merge.yml +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/workflows/scheduled-release.yml +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.github/workflows/supply-chain.yml +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.npmrc +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/.pre-commit-hooks.yaml +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/CODE_OF_CONDUCT.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/CONTRIBUTING.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/LICENSE +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/PRIVACY.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/SECURITY.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/SUPPORT.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/bench/README.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/cli/testguard.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/.gitignore +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/README.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/expected.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/package.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/src/export.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/src/redact.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/test/export-mocked.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/test/flaky.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/test/redact.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/testguard.claims.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer/vitest.config.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/.gitignore +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/README.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/expected.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/jest.config.js +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/package.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/src/export.js +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/src/redact.js +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/test/export-mocked.test.js +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/test/flaky.test.js +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/test/redact.test.js +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-jest/testguard.claims.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/.gitignore +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/README.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/e2e/flaky.spec.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/e2e/toggle.spec.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/expected.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/package-lock.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/package.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/playwright.config.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/src/toggle.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/test/toggle.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/test-results/.last-run.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/testguard.claims.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/fixtures/known-answer-playwright/vitest.config.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/package-lock.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/baseline.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/brief.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/calibration.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/claims.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/evidence-provisional.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/evidence.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/gate.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/ignore.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/replay.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/examples/status.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/baseline.bad-fingerprint-key.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/baseline.zero-count.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/brief.text-without-heading.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/brief.unknown-verdict-key.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/calibration.p-outside-ci.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/calibration.positives-exceed-n.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/claims.absolute-path.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/claims.duplicate-fault-id.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/claims.missing-provenance.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/claims.noop-fault.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/claims.occurrence-exceeds-hits.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/claims.parent-traversal.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.annotated-signal-without-reason.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.byrunner-file-not-resolved.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.contention-without-runners.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.discovered-defender-mocks-subject.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.flaky-defender-without-flakiness.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.ignored-dirty-on-snapshot.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.killed-by-non-assertion.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.killed-on-red-baseline.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.killed-without-n-runs.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.provisional-flag-on-confirmed-run.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.rank-extra-property.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.single-run-without-provisional.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.undeclared-killers-without-reason.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.unknown-verdict.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.unverifiable-without-reason.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/evidence.wrong-fingerprint.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/gate.counts-disagree.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/gate.exit-zero-with-uncovered.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/gate.ignore-cover-without-pattern.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/ignore.missing-reason.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/ignore.short-reason.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/replay.blind-with-a-failing-run.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/replay.duplicate-patch-id.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/status.claim-without-file.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/status.clean-with-new-findings.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/status.unclaimed-hidden-behind-unproven.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/conformance/invalid/status.write-test-without-target.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/spec/lib/fingerprint.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/admit/admit.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/baseline/baseline.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/brief/brief.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/claims/annotations.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/claims/load.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/claims/removed.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/admit.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/baseline.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/brief.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/gate.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/init.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/mcp.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/replay.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/scaffold.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/commands/status.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/evidence/writer.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/git.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/init/init.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/init/templates/SKILL.md +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/mcp/server.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/mcp/tools.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/contention.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/independence.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/probe/worktree.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/render.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/replay/label.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/scaffold/producers.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/status/status.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/src/util/hash.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/annotations.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/claims-since.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/glob.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/independence.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/probe.playwright.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/rank-aliases.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/samples/playwright-report.json +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/test/snapshot.test.mjs +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/testguard_cli/__init__.py +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/testguard_cli/wrapper.py +0 -0
- {testguard_cli-0.6.0 → testguard_cli-0.7.0}/vitest.config.mjs +0 -0
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: testguard
|
|
3
|
+
description: Prove the test suite defends this project's claims. Use before writing or changing tests, after changing guarded code, and whenever a session starts with a TEST BLINDSPOT CONTEXT block.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# TestGuard — how an agent runs it
|
|
7
|
+
|
|
8
|
+
TestGuard injects the faults a project's claims forbid and reports every one
|
|
9
|
+
the tests fail to detect. It is a **claim verifier**, not a test generator.
|
|
10
|
+
You operate it through one loop and one source of truth.
|
|
11
|
+
|
|
12
|
+
## The one source of truth
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
testguard status --json
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Read `state` and `next`. Do what `next.command` says. Never infer state from
|
|
19
|
+
which files exist. `state` is one of:
|
|
20
|
+
|
|
21
|
+
| state | meaning | you do |
|
|
22
|
+
|---|---|---|
|
|
23
|
+
| `no-claims` | no `testguard.claims.json` | `testguard scaffold <file>` for a file with guards; replace every `TODO:` statement with what the code guarantees; keep or drop each proposal; move the claims into `testguard.claims.json` |
|
|
24
|
+
| `unprobed` | claims never probed | `testguard probe` |
|
|
25
|
+
| `evidence-stale` | code, tests or claims changed since the evidence | `testguard probe --include-dirty` (or `--claim <ID>` for one) |
|
|
26
|
+
| `provisional-only` | only `--confirm 1` evidence exists | `testguard probe --confirm 3` |
|
|
27
|
+
| `unproven` | a finding is not covered by the baseline | `next.target` names it; see the verdict table |
|
|
28
|
+
| `clean` | everything killed or baselined | `testguard baseline` if `next` says so; otherwise nothing |
|
|
29
|
+
|
|
30
|
+
Exit codes: `0` clean · `1` unproven claims (or drift) · `2` precondition failed / nothing to do yet · `3` usage.
|
|
31
|
+
|
|
32
|
+
## Verdict → action
|
|
33
|
+
|
|
34
|
+
| verdict | what it means | the only acceptable fix |
|
|
35
|
+
|---|---|---|
|
|
36
|
+
| `SURVIVED` | the defenders stayed green while the claim was false | **write a test** in the defender file that fails with the fault applied and passes on HEAD |
|
|
37
|
+
| `SURVIVED [killed-by-undeclared-tests: …]` | other tests catch it | add those files to the claim's `defendedBy` |
|
|
38
|
+
| `NOCOVER` | no test file imports the target | write a new test file that imports it and asserts the claim |
|
|
39
|
+
| `UNVERIFIABLE` | the fault's anchor no longer matches, or the defenders fail to load | fix the **fault definition** (or the defenders' import); never the code |
|
|
40
|
+
| `FAULT-INVALID` | the replacement does not compile | fix the fault definition |
|
|
41
|
+
| `TIMEOUT` | the fault makes the code hang | write an assertion that fails on it; a hang is not a detection |
|
|
42
|
+
| `FLAKY-DEFENDER` | defenders not green 3/3, or disagreed across runs | fix the flake first; no verdict is trustworthy until then |
|
|
43
|
+
|
|
44
|
+
## The fix loop
|
|
45
|
+
|
|
46
|
+
1. `testguard status --json` → take `next.target`.
|
|
47
|
+
2. Write the test. It must **fail when the fault is applied and pass on HEAD**. To check the first half by hand: apply `find` → `replace` in the target file, run the defender, restore.
|
|
48
|
+
3. `testguard probe --claim <ID> --include-dirty` — probes your uncommitted test without touching the tree. `--confirm 1` gives a fast **provisional** signal; a `?` on a verdict means unconfirmed.
|
|
49
|
+
4. When it is `killed` at `--confirm 3`, commit the test.
|
|
50
|
+
5. When everything is killed or baselined, `testguard baseline` if `next` says so, and commit `.testguard/baseline.json`.
|
|
51
|
+
|
|
52
|
+
## Rules that are not yours to bend
|
|
53
|
+
|
|
54
|
+
- **Never make a fault die by editing the claims file.** Editing a claim is allowed when the claim was wrong — but the evidence records every fault's content hash, and `status` lists any fault edited after it survived, with its previous verdict, as `changedFaults`. The change is permitted; it is never invisible. Say why in the commit.
|
|
55
|
+
- **A test that asserts current behaviour is not a fix.** If the test would pass with the fault applied, it defends nothing.
|
|
56
|
+
- **Never re-implement the code under test inside the test.** Drive the real function; assert the outcome, not that a function was called.
|
|
57
|
+
- `--confirm 1` results are provisional: they print with `?`, go to `evidence-provisional.json`, and cannot be baselined.
|
|
58
|
+
- Worktree mode probes a **commit**. If `probe` refuses because defenders are dirty, use `--include-dirty`; do not commit half-written tests to satisfy it.
|
|
59
|
+
- Do not add `.testguard/evidence.json` or `brief.json` to git; do commit `baseline.json`.
|
|
60
|
+
|
|
61
|
+
## Reading the numbers
|
|
62
|
+
|
|
63
|
+
`N unproven faults across M claims` counts faults and the claims they belong to. There is no single score on purpose: blindness is concentrated, and one number hides where.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://raccioly.github.io/docguard/schemas/docguard-config.schema.json",
|
|
3
|
+
"projectName": "testguard-cli",
|
|
4
|
+
"version": "0.6",
|
|
5
|
+
"profile": "enterprise",
|
|
6
|
+
"projectType": "cli",
|
|
7
|
+
"projectTypeConfig": {
|
|
8
|
+
"needsEnvVars": false,
|
|
9
|
+
"needsEnvExample": false,
|
|
10
|
+
"needsE2E": false,
|
|
11
|
+
"needsDatabase": false
|
|
12
|
+
},
|
|
13
|
+
"requiredFiles": {
|
|
14
|
+
"canonical": [
|
|
15
|
+
"docs-canonical/ARCHITECTURE.md",
|
|
16
|
+
"docs-canonical/DATA-MODEL.md",
|
|
17
|
+
"docs-canonical/SECURITY.md",
|
|
18
|
+
"docs-canonical/TEST-SPEC.md",
|
|
19
|
+
"docs-canonical/ENVIRONMENT.md",
|
|
20
|
+
"docs-canonical/REQUIREMENTS.md"
|
|
21
|
+
]
|
|
22
|
+
},
|
|
23
|
+
"validators": {
|
|
24
|
+
"structure": true,
|
|
25
|
+
"docsSync": true,
|
|
26
|
+
"drift": true,
|
|
27
|
+
"changelog": true,
|
|
28
|
+
"architecture": true,
|
|
29
|
+
"testSpec": true,
|
|
30
|
+
"security": true,
|
|
31
|
+
"environment": true,
|
|
32
|
+
"freshness": true
|
|
33
|
+
},
|
|
34
|
+
"severity": {},
|
|
35
|
+
"findingSeverity": {}
|
|
36
|
+
}
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# .docguardignore — paths to exclude from DocGuard validation.
|
|
2
|
+
# Gitignore-style syntax: one pattern per line, # for comments.
|
|
3
|
+
# Merged into config.ignore (in .docguard.json) at runtime.
|
|
4
|
+
#
|
|
5
|
+
# Common examples:
|
|
6
|
+
# build/ # exclude a directory
|
|
7
|
+
# **/__generated__/** # exclude anything in any __generated__ dir
|
|
8
|
+
# vendor/legacy.ts # exclude a single file
|
|
9
|
+
# **/*.snap # exclude all files matching a glob
|
|
10
|
+
#
|
|
11
|
+
# Build outputs, vendored libs, generated code are good candidates here.
|
|
12
|
+
|
|
13
|
+
# Vendored / generated code that's not yours to document
|
|
14
|
+
**/__generated__/**
|
|
15
|
+
**/generated/**
|
|
16
|
+
**/*.generated.*
|
|
17
|
+
|
|
18
|
+
# Migrations and lock files
|
|
19
|
+
**/migrations/**
|
|
20
|
+
**/*.lock
|
|
21
|
+
package-lock.json
|
|
22
|
+
yarn.lock
|
|
23
|
+
pnpm-lock.yaml
|
|
24
|
+
Cargo.lock
|
|
25
|
+
poetry.lock
|
|
26
|
+
|
|
27
|
+
# Common build artifacts (defaults also cover these, but listing here is clearer)
|
|
28
|
+
# dist/
|
|
29
|
+
# build/
|
|
30
|
+
# coverage/
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* Render docs/testguard-explained.html to PDF with headless Chrome.
|
|
4
|
+
*
|
|
5
|
+
* Chrome rather than a library because this repository has one pinned runtime
|
|
6
|
+
* dependency and keeps it that way: a PDF renderer would be the second, and it
|
|
7
|
+
* would be needed only to build a document nobody imports.
|
|
8
|
+
*
|
|
9
|
+
* Set CHROME to override the binary. Exits 0 with a notice when no Chrome is
|
|
10
|
+
* present, so a checkout without one can still run every other script.
|
|
11
|
+
*/
|
|
12
|
+
import { existsSync } from 'node:fs';
|
|
13
|
+
import { spawnSync } from 'node:child_process';
|
|
14
|
+
import { dirname, join, resolve } from 'node:path';
|
|
15
|
+
import { fileURLToPath } from 'node:url';
|
|
16
|
+
|
|
17
|
+
const root = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
|
18
|
+
const src = join(root, 'docs', 'testguard-explained.html');
|
|
19
|
+
const out = join(root, 'docs', 'testguard-explained.pdf');
|
|
20
|
+
|
|
21
|
+
const candidates = [
|
|
22
|
+
process.env.CHROME,
|
|
23
|
+
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
|
|
24
|
+
'/Applications/Chromium.app/Contents/MacOS/Chromium',
|
|
25
|
+
'/usr/bin/google-chrome',
|
|
26
|
+
'/usr/bin/chromium',
|
|
27
|
+
'/usr/bin/chromium-browser',
|
|
28
|
+
].filter(Boolean);
|
|
29
|
+
|
|
30
|
+
const chrome = candidates.find((c) => existsSync(c));
|
|
31
|
+
if (!chrome) {
|
|
32
|
+
console.log('no Chrome found; skipping PDF build (set CHROME=<path> to force)');
|
|
33
|
+
process.exit(0);
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
const r = spawnSync(chrome, [
|
|
37
|
+
'--headless',
|
|
38
|
+
'--disable-gpu',
|
|
39
|
+
'--no-sandbox',
|
|
40
|
+
'--no-pdf-header-footer',
|
|
41
|
+
`--print-to-pdf=${out}`,
|
|
42
|
+
`file://${src}`,
|
|
43
|
+
], { encoding: 'utf8' });
|
|
44
|
+
|
|
45
|
+
if (r.status !== 0 || !existsSync(out)) {
|
|
46
|
+
console.error(r.stderr || 'chrome produced no PDF');
|
|
47
|
+
process.exit(1);
|
|
48
|
+
}
|
|
49
|
+
console.log(`docs/testguard-explained.pdf written with ${chrome.split('/').pop()}`);
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* Compare a probe of `fixtures/known-answer-python` against its oracle.
|
|
4
|
+
*
|
|
5
|
+
* Shared by both CI legs so the stdlib engine and pytest are held to exactly
|
|
6
|
+
* the same expectations — the point of the fixture is that two engines with
|
|
7
|
+
* different report shapes must not disagree about a single verdict.
|
|
8
|
+
*/
|
|
9
|
+
import { readFileSync } from 'node:fs';
|
|
10
|
+
import { dirname, join } from 'node:path';
|
|
11
|
+
import { fileURLToPath } from 'node:url';
|
|
12
|
+
|
|
13
|
+
const [evidencePath, expectedEngine] = process.argv.slice(2);
|
|
14
|
+
if (!evidencePath) {
|
|
15
|
+
console.error('usage: check-python-fixture.mjs <evidence.json> [engine]');
|
|
16
|
+
process.exit(2);
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
|
20
|
+
const oracle = JSON.parse(readFileSync(join(ROOT, 'fixtures', 'known-answer-python', 'expected.json'), 'utf8')).expected;
|
|
21
|
+
const evidence = JSON.parse(readFileSync(evidencePath, 'utf8'));
|
|
22
|
+
|
|
23
|
+
const problems = [];
|
|
24
|
+
if (expectedEngine && evidence.run.runner.name !== expectedEngine) {
|
|
25
|
+
problems.push(`runner was ${evidence.run.runner.name}, expected ${expectedEngine}`);
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
const seen = new Set();
|
|
29
|
+
for (const r of evidence.records) {
|
|
30
|
+
const key = `${r.claim.id}/${r.subject.id}`;
|
|
31
|
+
seen.add(key);
|
|
32
|
+
const want = oracle[key];
|
|
33
|
+
if (!want) { problems.push(`${key}: not in expected.json`); continue; }
|
|
34
|
+
if (want.verdict !== r.verdict) problems.push(`${key}: ${r.verdict}, expected ${want.verdict}`);
|
|
35
|
+
if (want.reason && want.reason !== r.detail.reason) problems.push(`${key}: reason ${r.detail.reason}, expected ${want.reason}`);
|
|
36
|
+
if (Boolean(want.targetNotImported) !== Boolean(r.detail.targetNotImported)) {
|
|
37
|
+
problems.push(`${key}: targetNotImported ${Boolean(r.detail.targetNotImported)}, expected ${Boolean(want.targetNotImported)}`);
|
|
38
|
+
}
|
|
39
|
+
// The negative control is pinned too, not merely tolerated: a fixture that
|
|
40
|
+
// accepted either answer would stop defending the one behaviour it exists
|
|
41
|
+
// to prove, and the verdict alone cannot tell "the defenders reached this
|
|
42
|
+
// file and said nothing" from "they never reached it".
|
|
43
|
+
if ((want.negativeControl ?? null) !== (r.detail.negativeControl ?? null)) {
|
|
44
|
+
problems.push(`${key}: negativeControl ${r.detail.negativeControl ?? 'absent'}, expected ${want.negativeControl ?? 'absent'}`);
|
|
45
|
+
}
|
|
46
|
+
}
|
|
47
|
+
for (const key of Object.keys(oracle)) if (!seen.has(key)) problems.push(`${key}: expected but never probed`);
|
|
48
|
+
|
|
49
|
+
if (problems.length) {
|
|
50
|
+
for (const p of problems) console.error(`::error::${p}`);
|
|
51
|
+
process.exit(1);
|
|
52
|
+
}
|
|
53
|
+
console.log(`python fixture verdicts match expected.json under ${evidence.run.runner.name}: ${evidence.records.length} records`);
|
|
@@ -6,6 +6,14 @@
|
|
|
6
6
|
* - action.yml default of the `version` input
|
|
7
7
|
* - packaging/homebrew/*.rb the tarball URL (sha256 is set after publish)
|
|
8
8
|
* - packaging/gitlab/*.yml the include URL tag and the `version` input default
|
|
9
|
+
* - README.md the GitHub Action reference AND the GitLab include URL
|
|
10
|
+
*
|
|
11
|
+
* A file may carry more than one form, and each needs its own entry: README
|
|
12
|
+
* holds both the `raccioly/testguard@vX.Y.Z` action reference and the
|
|
13
|
+
* `testguard/vX.Y.Z/packaging` template URL. Only the first was listed, so
|
|
14
|
+
* v0.6.0 shipped with both README copies of the GitLab URL left at v0.5.0
|
|
15
|
+
* while `--check` reported "all version surfaces at 0.6.0" — a surface this
|
|
16
|
+
* script was never told to look at cannot drift in its eyes.
|
|
9
17
|
*
|
|
10
18
|
* `--check` verifies instead of writing (used by release.yml).
|
|
11
19
|
*/
|
|
@@ -22,6 +30,7 @@ const surfaces = [
|
|
|
22
30
|
['action.yml', /(\n version:\n description: [^\n]*\n required: false\n default: ')[^']*(')/, `$1${version}$2`],
|
|
23
31
|
['packaging/homebrew/testguard.rb', /testguard-cli-\d+\.\d+\.\d+\.tgz/g, `testguard-cli-${version}.tgz`],
|
|
24
32
|
['README.md', /raccioly\/testguard@v\d+\.\d+\.\d+/g, `raccioly/testguard@v${version}`],
|
|
33
|
+
['README.md', /testguard\/v\d+\.\d+\.\d+\/packaging/g, `testguard/v${version}/packaging`],
|
|
25
34
|
['packaging/gitlab/testguard.gitlab-ci.yml', /testguard\/v\d+\.\d+\.\d+\/packaging/g, `testguard/v${version}/packaging`],
|
|
26
35
|
['packaging/gitlab/testguard.gitlab-ci.yml', /(\n version:\n description: [^\n]*\n default: ")\d+\.\d+\.\d+(")/, `$1${version}$2`],
|
|
27
36
|
];
|
|
@@ -147,6 +147,30 @@ jobs:
|
|
|
147
147
|
console.log('mcp:', list.result.tools.map((t) => t.name).join(', '), '→ state', call.result.structuredContent.state);
|
|
148
148
|
"
|
|
149
149
|
|
|
150
|
+
# ── Documentation is canon: DocGuard validates this repository against
|
|
151
|
+
# docs-canonical/. This is the authoritative check — a machine-local
|
|
152
|
+
# .git/hooks file is neither shared with contributors nor reviewable,
|
|
153
|
+
# so CI is where doc drift is actually caught. `guard` exits 2 for
|
|
154
|
+
# warnings, which we surface without failing; errors fail the build. ──
|
|
155
|
+
- name: DocGuard — documentation matches the code
|
|
156
|
+
run: |
|
|
157
|
+
# Capture the exit code explicitly. A pipeline under the default
|
|
158
|
+
# `bash -e` aborts the step before any branch on its status can run,
|
|
159
|
+
# and `|| code=$?` is the one form errexit does not trip on.
|
|
160
|
+
code=0
|
|
161
|
+
npx -y docguard-cli@0.41.5 guard > "$RUNNER_TEMP/docguard-guard.txt" 2>&1 || code=$?
|
|
162
|
+
cat "$RUNNER_TEMP/docguard-guard.txt"
|
|
163
|
+
if [ "$code" -eq 2 ]; then
|
|
164
|
+
echo "::warning::DocGuard reported warnings (exit 2); see the output above."
|
|
165
|
+
elif [ "$code" -ne 0 ]; then
|
|
166
|
+
echo "::error::DocGuard guard failed (exit $code) — documentation contradicts the code."
|
|
167
|
+
exit 1
|
|
168
|
+
fi
|
|
169
|
+
|
|
170
|
+
- name: DocGuard — maturity score (reported, not gated)
|
|
171
|
+
if: always()
|
|
172
|
+
run: npx -y docguard-cli@0.41.5 score || true
|
|
173
|
+
|
|
150
174
|
# ── Every change needs a claim: the gate runs on this repository's own
|
|
151
175
|
# pull requests. A new source file without a claim (or an excusing
|
|
152
176
|
# ignore entry with a reason) turns the PR red. ──
|
|
@@ -165,7 +189,9 @@ jobs:
|
|
|
165
189
|
# testguard.claims.json states invariants of the tool itself; every
|
|
166
190
|
# fault must be killed by the tool's own tests. Exit 0 = all defended. ──
|
|
167
191
|
# Runs on ONE Node leg: the verdicts do not depend on the Node minor,
|
|
168
|
-
# and
|
|
192
|
+
# and a from-scratch probe is ~10 minutes (it was ~24 before the
|
|
193
|
+
# defender split in #67; `--cost` below prints where the time goes,
|
|
194
|
+
# so a regression in gate cost is visible in the log). The previous run's
|
|
169
195
|
# evidence is restored from the cache so claims whose target and
|
|
170
196
|
# defenders did not change reuse their verdict (the tool's own
|
|
171
197
|
# reuse rule); only what changed is re-probed. No escalation in CI:
|
|
@@ -182,7 +208,7 @@ jobs:
|
|
|
182
208
|
|
|
183
209
|
- name: probe TestGuard's own claims
|
|
184
210
|
if: matrix.node-version == 22
|
|
185
|
-
run: node cli/testguard.mjs probe . --budget 180000 --no-escalate --out .testguard/ci-self-evidence.json
|
|
211
|
+
run: node cli/testguard.mjs probe . --budget 180000 --no-escalate --cost --out .testguard/ci-self-evidence.json
|
|
186
212
|
|
|
187
213
|
# A stable, branch-named artifact so a developer can fetch the evidence
|
|
188
214
|
# CI produced without knowing a run id (README: read CI's evidence
|
|
@@ -203,6 +229,80 @@ jobs:
|
|
|
203
229
|
path: .testguard/ci-self-evidence.json
|
|
204
230
|
if-no-files-found: warn
|
|
205
231
|
|
|
232
|
+
# ── Python: two engines, one oracle. The stdlib leg runs FIRST and with no
|
|
233
|
+
# pip install of any kind, because "a project with no test dependencies
|
|
234
|
+
# still probes" is a claim about the tool and the only way to keep it true
|
|
235
|
+
# is to prove it in an environment that has none. The pytest leg then
|
|
236
|
+
# installs pytest into a venv and must reach the SAME sixteen verdicts
|
|
237
|
+
# through a different report shape. ──
|
|
238
|
+
python:
|
|
239
|
+
runs-on: ubuntu-latest
|
|
240
|
+
steps:
|
|
241
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
242
|
+
with:
|
|
243
|
+
fetch-depth: 0
|
|
244
|
+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
|
|
245
|
+
with:
|
|
246
|
+
node-version: 22
|
|
247
|
+
cache: npm
|
|
248
|
+
- name: Install (tool only)
|
|
249
|
+
run: npm ci
|
|
250
|
+
|
|
251
|
+
# A bare virtualenv, not the system interpreter: the claim under test is
|
|
252
|
+
# "a project whose test dependencies are the standard library still
|
|
253
|
+
# probes", and asserting that the runner image happens to lack pytest
|
|
254
|
+
# would make this job fail for a reason that has nothing to do with the
|
|
255
|
+
# product the day that image changes. Guarantee the condition instead.
|
|
256
|
+
- name: The stdlib engine must need nothing installed
|
|
257
|
+
shell: bash
|
|
258
|
+
run: |
|
|
259
|
+
set -o pipefail
|
|
260
|
+
python3 -m venv "$RUNNER_TEMP/bare"
|
|
261
|
+
if "$RUNNER_TEMP/bare/bin/python" -c 'import pytest' 2>/dev/null; then
|
|
262
|
+
echo "::error::the bare venv can import pytest; it is not bare and this leg proves nothing"
|
|
263
|
+
exit 1
|
|
264
|
+
fi
|
|
265
|
+
TESTGUARD_PYTHON="$RUNNER_TEMP/bare/bin/python" \
|
|
266
|
+
npx vitest run test/runner-python.test.mjs test/pyimports.test.mjs test/scaffold.python.test.mjs test/probe.python.test.mjs
|
|
267
|
+
|
|
268
|
+
- name: probe the Python fixture through the bin, stdlib engine
|
|
269
|
+
shell: bash
|
|
270
|
+
run: |
|
|
271
|
+
set -o pipefail
|
|
272
|
+
S=$(mktemp -d)
|
|
273
|
+
cp -R fixtures/known-answer-python/. "$S/"
|
|
274
|
+
rm -rf "$S/.flake-counter" "$S/.testguard" "$S/.pytest_cache"
|
|
275
|
+
find "$S" -name __pycache__ -type d -exec rm -rf {} + 2>/dev/null || true
|
|
276
|
+
git -C "$S" init -q
|
|
277
|
+
git -C "$S" -c user.email=ci@example.invalid -c user.name=ci add -A
|
|
278
|
+
git -C "$S" -c user.email=ci@example.invalid -c user.name=ci commit -qm fixture
|
|
279
|
+
status=0; node cli/testguard.mjs probe "$S" --budget 20000 --quiet --no-escalate --python "$RUNNER_TEMP/bare/bin/python" || status=$?
|
|
280
|
+
[ "$status" -eq 1 ] || { echo "::error::expected exit 1, got $status"; exit 1; }
|
|
281
|
+
node .github/scripts/check-python-fixture.mjs "$S/.testguard/evidence.json" unittest
|
|
282
|
+
|
|
283
|
+
- name: install pytest into a venv, and reach the same verdicts through it
|
|
284
|
+
shell: bash
|
|
285
|
+
run: |
|
|
286
|
+
set -o pipefail
|
|
287
|
+
python3 -m venv "$RUNNER_TEMP/pyenv"
|
|
288
|
+
"$RUNNER_TEMP/pyenv/bin/pip" install -q pytest
|
|
289
|
+
S=$(mktemp -d)
|
|
290
|
+
cp -R fixtures/known-answer-python/. "$S/"
|
|
291
|
+
rm -rf "$S/.flake-counter" "$S/.testguard" "$S/.pytest_cache"
|
|
292
|
+
find "$S" -name __pycache__ -type d -exec rm -rf {} + 2>/dev/null || true
|
|
293
|
+
git -C "$S" init -q
|
|
294
|
+
git -C "$S" -c user.email=ci@example.invalid -c user.name=ci add -A
|
|
295
|
+
git -C "$S" -c user.email=ci@example.invalid -c user.name=ci commit -qm fixture
|
|
296
|
+
status=0; node cli/testguard.mjs probe "$S" --budget 20000 --quiet --no-escalate \
|
|
297
|
+
--runner pytest --python "$RUNNER_TEMP/pyenv/bin/python" || status=$?
|
|
298
|
+
[ "$status" -eq 1 ] || { echo "::error::expected exit 1, got $status"; exit 1; }
|
|
299
|
+
node .github/scripts/check-python-fixture.mjs "$S/.testguard/evidence.json" pytest
|
|
300
|
+
|
|
301
|
+
- name: the acceptance suite again, now against an interpreter that HAS pytest
|
|
302
|
+
shell: bash
|
|
303
|
+
run: |
|
|
304
|
+
TESTGUARD_PYTHON="$RUNNER_TEMP/pyenv/bin/python" npx vitest run test/probe.python.test.mjs
|
|
305
|
+
|
|
206
306
|
# ── Playwright: the per-file runner is proven against its own browserless
|
|
207
307
|
# fixture, whose dependencies live in the fixture (never in the tool's
|
|
208
308
|
# package.json). Installed and run here only. ──
|
|
@@ -80,7 +80,7 @@ jobs:
|
|
|
80
80
|
- name: Install smoke — the tarball must run with production deps only
|
|
81
81
|
run: npm run test:install
|
|
82
82
|
- name: Self-probe must be fully defended before anything ships
|
|
83
|
-
run: node cli/testguard.mjs probe . --budget 180000 --out "$RUNNER_TEMP/self-evidence.json" --quiet
|
|
83
|
+
run: node cli/testguard.mjs probe . --budget 180000 --out "$RUNNER_TEMP/self-evidence.json" --quiet --cost
|
|
84
84
|
- name: Version surfaces are in sync
|
|
85
85
|
run: node .github/scripts/sync-release-version.mjs --check
|
|
86
86
|
|
|
@@ -14,6 +14,7 @@ coverage/
|
|
|
14
14
|
# Local agent tooling — not part of the public project
|
|
15
15
|
.wolf/
|
|
16
16
|
.claude/
|
|
17
|
+
.agent/
|
|
17
18
|
CLAUDE.local.md
|
|
18
19
|
graphify-out/
|
|
19
20
|
.vitest/
|
|
@@ -22,3 +23,8 @@ dist/
|
|
|
22
23
|
build/
|
|
23
24
|
*.egg-info/
|
|
24
25
|
__pycache__/
|
|
26
|
+
|
|
27
|
+
# Never commit secrets
|
|
28
|
+
.env
|
|
29
|
+
.env.*
|
|
30
|
+
!.env.example
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# Local hooks for contributors. Opt in once per clone:
|
|
2
|
+
#
|
|
3
|
+
# pre-commit install # doc + claim checks on commit
|
|
4
|
+
# pre-commit install --hook-type pre-push # the probe, before you push
|
|
5
|
+
#
|
|
6
|
+
# These are the same checks CI runs. CI is authoritative; this is the fast
|
|
7
|
+
# feedback loop. Nothing here reaches the network.
|
|
8
|
+
repos:
|
|
9
|
+
- repo: local
|
|
10
|
+
hooks:
|
|
11
|
+
- id: docguard-guard
|
|
12
|
+
name: DocGuard — docs match code
|
|
13
|
+
description: Validates this repository against docs-canonical/. Fails on errors, passes on warnings (exit 2). Needs docguard on PATH.
|
|
14
|
+
entry: bash -c 'code=0; docguard guard || code=$?; if [ "$code" -eq 2 ]; then exit 0; fi; exit "$code"'
|
|
15
|
+
language: system
|
|
16
|
+
pass_filenames: false
|
|
17
|
+
always_run: true
|
|
18
|
+
|
|
19
|
+
- id: testguard-claims
|
|
20
|
+
name: TestGuard — claims file valid, no annotation drift
|
|
21
|
+
entry: node cli/testguard.mjs claims .
|
|
22
|
+
language: system
|
|
23
|
+
pass_filenames: false
|
|
24
|
+
always_run: true
|
|
25
|
+
|
|
26
|
+
- id: testguard-gate
|
|
27
|
+
name: TestGuard — every changed source file carries a claim
|
|
28
|
+
entry: node cli/testguard.mjs gate . --changed HEAD --include-dirty --quiet
|
|
29
|
+
language: system
|
|
30
|
+
pass_filenames: false
|
|
31
|
+
always_run: true
|
|
32
|
+
|
|
33
|
+
- id: testguard-probe
|
|
34
|
+
name: TestGuard — probe the claims (pre-push)
|
|
35
|
+
description: Slow by design. Runs the faults and fails on any the tests miss.
|
|
36
|
+
entry: node cli/testguard.mjs probe . --quiet
|
|
37
|
+
language: system
|
|
38
|
+
pass_filenames: false
|
|
39
|
+
always_run: true
|
|
40
|
+
stages: [pre-push]
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# AI Agent Instructions — TestGuard
|
|
2
2
|
|
|
3
|
+
<!-- docguard:last-reviewed 2026-09-18 -->
|
|
4
|
+
|
|
3
5
|
Read this before changing anything. It is short on purpose.
|
|
4
6
|
|
|
5
7
|
## What this is
|
|
@@ -32,7 +34,10 @@ node cli/testguard.mjs scaffold src/probe/classify.mjs --json # what the produ
|
|
|
32
34
|
- `fixtures/known-answer/expected.json` — the oracle. Never edit it to match output.
|
|
33
35
|
- `testguard.claims.json` — claims about this codebase, probed in CI.
|
|
34
36
|
- `src/scaffold/producers.mjs` — the nine fault shapes. Deterministic line heuristics; no AST, no LLM. A new shape needs a synthetic-file test and a README row.
|
|
35
|
-
- `src/probe/runners/` — one module per runner (`name`, `testGlobs`, `check`, `tests`, `run`, optionally `owns` for a per-file runner) over `shared.mjs` (jest-compatible report parsing, budgeted process runner, pluggable `parse`/`env`). `playwright.mjs`
|
|
37
|
+
- `src/probe/runners/` — one module per runner (`name`, `testGlobs`, `check`, `tests`, `run`, optionally `owns` for a per-file runner) over `shared.mjs` (jest-compatible report parsing, budgeted process runner, pluggable `parse`/`env`). `check`/`run` also receive `sourceDir` (the real project directory, for a toolchain the scratch worktree does not contain) and `targets` (the fault's files); `check` may return `engine`, and the engine is what the evidence records as `runner.name`. `playwright.mjs` and `python.mjs` are per-file runners: Playwright owns the files under its `testDir`, Python owns every `.py`; `index.mjs` partitions defenders by runner and merges their runs pessimistically. A new runner needs its own fixture copy with its own `expected.json`, verified by hand.
|
|
38
|
+
- `src/probe/runners/python/` — the injected Python reporters (`_testguard_report.py`, a pytest plugin, a stdlib `unittest` main). They reach the interpreter through `PYTHONPATH`, never through an install into the project under test; changing their report shape is a change to `python.mjs`'s `parseReport` in the same PR. They also report **import provenance**, which `probe.mjs` checks once the fault is live — see `checkProvenance` and `GATE-SEMANTICS.md` rule 9.
|
|
39
|
+
- `src/probe/pyimports.mjs` — Python defender discovery, patch-awareness and blast radius, matched by module name. A `patch("pkg.mod.fn")` is NOT a `vi.mock`: it leaves the file a defender. Only a patch of the module itself removes one.
|
|
40
|
+
- `src/scaffold/producers.python.mjs` — the Python fault shapes. A statement is removed with `pass`, never by deleting the line, and a line that leaves a bracket open is never removed: a fault that cannot compile is a probe run that says nothing.
|
|
36
41
|
- `src/gate/changed.mjs` — claim coverage of a change (`gate --changed`). File-level, delta-only; an uncovered file exits 1; every ignore reliance is reported. Its rules are a section of `GATE-SEMANTICS.md`.
|
|
37
42
|
- `src/status/status.mjs` — the state machine every rendering derives from. A new state or action is a spec change (`status.schema.json`) and a skill-template change (`src/init/templates/SKILL.md`) in the same PR.
|
|
38
43
|
|