assertledger 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +31 -0
- package/LICENSE +21 -0
- package/README.fr.md +236 -0
- package/README.md +224 -0
- package/SECURITY.md +51 -0
- package/benchmarks/agentic-profile/README.md +15 -0
- package/benchmarks/agentic-profile/public/README.md +5 -0
- package/benchmarks/self-hosted-core/README.md +113 -0
- package/benchmarks/self-hosted-core/adapter.mjs +293 -0
- package/benchmarks/self-hosted-core/builder.ts +193 -0
- package/benchmarks/self-hosted-core/campaign.ts +233 -0
- package/benchmarks/self-hosted-core/existing-tests-builder.ts +217 -0
- package/benchmarks/self-hosted-core/existing-tests.ts +146 -0
- package/benchmarks/self-hosted-core/liveness.test.mjs +8 -0
- package/conformance/v1/bundle.json +104 -0
- package/conformance/v1/expected/canonical-order-a.json +4 -0
- package/conformance/v1/expected/canonical-order-b.json +4 -0
- package/conformance/v1/expected/create-benchmark-v1-measured.json +575 -0
- package/conformance/v1/expected/create-profile-v1-qualified.json +280 -0
- package/conformance/v1/expected/decide-collection-failure-non-kill.json +192 -0
- package/conformance/v1/expected/decide-compile-failure-non-kill.json +192 -0
- package/conformance/v1/expected/decide-infra-error-non-kill.json +192 -0
- package/conformance/v1/expected/decide-no-test-discovered-non-kill.json +192 -0
- package/conformance/v1/expected/decide-process-crash-non-kill.json +192 -0
- package/conformance/v1/expected/decide-timeout-non-kill.json +192 -0
- package/conformance/v1/expected/decide-verified.json +192 -0
- package/conformance/v1/expected/replay-benchmark-v1-resealed-summary-forgery.json +12 -0
- package/conformance/v1/expected/replay-evidence-raw-tamper.json +6 -0
- package/conformance/v1/expected/replay-evidence-resealed-semantic-forgery.json +6 -0
- package/conformance/v1/inputs/canonical-order-a.json +8 -0
- package/conformance/v1/inputs/canonical-order-b.json +8 -0
- package/conformance/v1/inputs/create-benchmark-v1-measured.json +459 -0
- package/conformance/v1/inputs/create-profile-v1-qualified.json +228 -0
- package/conformance/v1/inputs/decide-collection-failure-non-kill.json +143 -0
- package/conformance/v1/inputs/decide-compile-failure-non-kill.json +143 -0
- package/conformance/v1/inputs/decide-infra-error-non-kill.json +143 -0
- package/conformance/v1/inputs/decide-no-test-discovered-non-kill.json +143 -0
- package/conformance/v1/inputs/decide-process-crash-non-kill.json +143 -0
- package/conformance/v1/inputs/decide-timeout-non-kill.json +143 -0
- package/conformance/v1/inputs/decide-verified.json +143 -0
- package/conformance/v1/inputs/replay-benchmark-v1-resealed-summary-forgery.json +575 -0
- package/conformance/v1/inputs/replay-evidence-raw-tamper.json +201 -0
- package/conformance/v1/inputs/replay-evidence-resealed-semantic-forgery.json +192 -0
- package/conformance/v1/schemas/expected-digests.json +175 -0
- package/dist/cli.d.ts +9 -0
- package/dist/cli.d.ts.map +1 -0
- package/dist/cli.js +951 -0
- package/dist/cli.js.map +1 -0
- package/dist/contracts/diagnostics.d.ts +18 -0
- package/dist/contracts/diagnostics.d.ts.map +1 -0
- package/dist/contracts/diagnostics.js +13 -0
- package/dist/contracts/diagnostics.js.map +1 -0
- package/dist/contracts/index.d.ts +3908 -0
- package/dist/contracts/index.d.ts.map +1 -0
- package/dist/contracts/index.js +2569 -0
- package/dist/contracts/index.js.map +1 -0
- package/dist/contracts/runtime-doctor.d.ts +107 -0
- package/dist/contracts/runtime-doctor.d.ts.map +1 -0
- package/dist/contracts/runtime-doctor.js +91 -0
- package/dist/contracts/runtime-doctor.js.map +1 -0
- package/dist/core/index.d.ts +200 -0
- package/dist/core/index.d.ts.map +1 -0
- package/dist/core/index.js +2587 -0
- package/dist/core/index.js.map +1 -0
- package/dist/diagnostics.d.ts +7 -0
- package/dist/diagnostics.d.ts.map +1 -0
- package/dist/diagnostics.js +252 -0
- package/dist/diagnostics.js.map +1 -0
- package/dist/engine/adapters/node-test-profile.d.ts +14 -0
- package/dist/engine/adapters/node-test-profile.d.ts.map +1 -0
- package/dist/engine/adapters/node-test-profile.js +14 -0
- package/dist/engine/adapters/node-test-profile.js.map +1 -0
- package/dist/engine/adapters/node-test-runtime.d.ts +39 -0
- package/dist/engine/adapters/node-test-runtime.d.ts.map +1 -0
- package/dist/engine/adapters/node-test-runtime.js +173 -0
- package/dist/engine/adapters/node-test-runtime.js.map +1 -0
- package/dist/engine/adapters/runtime-facts.d.ts +26 -0
- package/dist/engine/adapters/runtime-facts.d.ts.map +1 -0
- package/dist/engine/adapters/runtime-facts.js +73 -0
- package/dist/engine/adapters/runtime-facts.js.map +1 -0
- package/dist/engine/connection.d.ts +22 -0
- package/dist/engine/connection.d.ts.map +1 -0
- package/dist/engine/connection.js +343 -0
- package/dist/engine/connection.js.map +1 -0
- package/dist/engine/git-regression.d.ts +25 -0
- package/dist/engine/git-regression.d.ts.map +1 -0
- package/dist/engine/git-regression.js +803 -0
- package/dist/engine/git-regression.js.map +1 -0
- package/dist/engine/index.d.ts +55 -0
- package/dist/engine/index.d.ts.map +1 -0
- package/dist/engine/index.js +2782 -0
- package/dist/engine/index.js.map +1 -0
- package/dist/engine/node-test-reporter.d.ts +2 -0
- package/dist/engine/node-test-reporter.d.ts.map +1 -0
- package/dist/engine/node-test-reporter.js +70 -0
- package/dist/engine/node-test-reporter.js.map +1 -0
- package/dist/engine/runtime-doctor.d.ts +16 -0
- package/dist/engine/runtime-doctor.d.ts.map +1 -0
- package/dist/engine/runtime-doctor.js +100 -0
- package/dist/engine/runtime-doctor.js.map +1 -0
- package/dist/evaluation/agentic-corpus.d.ts +161 -0
- package/dist/evaluation/agentic-corpus.d.ts.map +1 -0
- package/dist/evaluation/agentic-corpus.js +710 -0
- package/dist/evaluation/agentic-corpus.js.map +1 -0
- package/dist/index.d.ts +8 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +8 -0
- package/dist/index.js.map +1 -0
- package/dist/mcp/index.d.ts +13 -0
- package/dist/mcp/index.d.ts.map +1 -0
- package/dist/mcp/index.js +391 -0
- package/dist/mcp/index.js.map +1 -0
- package/dist/mcp/stdio.d.ts +3 -0
- package/dist/mcp/stdio.d.ts.map +1 -0
- package/dist/mcp/stdio.js +13 -0
- package/dist/mcp/stdio.js.map +1 -0
- package/dist/sdk/index.d.ts +52 -0
- package/dist/sdk/index.d.ts.map +1 -0
- package/dist/sdk/index.js +224 -0
- package/dist/sdk/index.js.map +1 -0
- package/dist/version.d.ts +2 -0
- package/dist/version.d.ts.map +1 -0
- package/dist/version.js +10 -0
- package/dist/version.js.map +1 -0
- package/docs/adapter-protocol.md +196 -0
- package/docs/agentic-benchmark.md +118 -0
- package/docs/agentic-corpus-experiment-h3.md +89 -0
- package/docs/agentic-corpus-plan.md +105 -0
- package/docs/agentic-corpus-provenance.md +59 -0
- package/docs/agentic-test-profile-pilot.md +57 -0
- package/docs/agentic-test-profile-v2.md +116 -0
- package/docs/agentic-test-profile.md +274 -0
- package/docs/architecture.md +157 -0
- package/docs/ci.md +37 -0
- package/docs/client-connections.md +61 -0
- package/docs/conformance-v1.md +72 -0
- package/docs/decisions/0001-typescript-runtime.md +24 -0
- package/docs/developer-experience.md +55 -0
- package/docs/diagnostics.md +35 -0
- package/docs/distribution.md +40 -0
- package/docs/git-regression.md +39 -0
- package/docs/migration-repository-validation-order.md +35 -0
- package/docs/migration-testforge-to-assertledger.md +64 -0
- package/docs/project-intent.md +173 -0
- package/docs/proof-model.md +116 -0
- package/docs/reference.md +336 -0
- package/docs/release-1.0.md +63 -0
- package/docs/repository-audit.md +52 -0
- package/docs/repository-init.md +60 -0
- package/docs/research-basis.md +27 -0
- package/docs/roadmap.md +74 -0
- package/docs/runtime-doctor.md +65 -0
- package/docs/testexplora-calibration.md +71 -0
- package/examples/agentic-benchmark/benchmark-request.mjs +19 -0
- package/examples/agentic-benchmark/structured-phase-adapter-fixture.mjs +35 -0
- package/examples/agentic-profile/profile-benchmark.mjs +34 -0
- package/examples/agentic-profile/profile-manifest.mjs +28 -0
- package/examples/git-history/README.md +44 -0
- package/examples/git-history/create-demo.mjs +128 -0
- package/examples/git-history/escape-string-regexp/LICENSE +9 -0
- package/examples/git-history/escape-string-regexp/before.cjs.txt +11 -0
- package/examples/git-history/escape-string-regexp/fixed.cjs.txt +13 -0
- package/examples/git-history/escape-string-regexp/provenance.json +28 -0
- package/examples/node-test/repository/package.json +5 -0
- package/examples/node-test/repository/src/is-even.js +3 -0
- package/examples/node-test/repository/tests/base.test.js +6 -0
- package/examples/node-test/request.json +93 -0
- package/integrations/skill/SKILL.md +51 -0
- package/package.json +88 -0
- package/schemas/agentic-benchmark-acquisition-replay-result.v1.json +70 -0
- package/schemas/agentic-benchmark-acquisition-request.v1.json +564 -0
- package/schemas/agentic-benchmark-acquisition-result.v1.json +1409 -0
- package/schemas/agentic-benchmark-artifact.v1.json +1251 -0
- package/schemas/agentic-benchmark-replay-result.v1.json +84 -0
- package/schemas/agentic-benchmark-request.v1.json +1034 -0
- package/schemas/agentic-corpus-allocation-commitment-replay-result.v1.json +58 -0
- package/schemas/agentic-corpus-allocation-commitment.v1.json +141 -0
- package/schemas/agentic-corpus-allocation-replay-result.v1.json +34 -0
- package/schemas/agentic-corpus-allocation-request.v1.json +65 -0
- package/schemas/agentic-corpus-allocation-reveal.v1.json +66 -0
- package/schemas/agentic-corpus-allocation.v1.json +167 -0
- package/schemas/agentic-corpus-experiment-artifact.v1.json +329 -0
- package/schemas/agentic-corpus-experiment-plan-replay-result.v1.json +50 -0
- package/schemas/agentic-corpus-experiment-plan.v1.json +424 -0
- package/schemas/agentic-corpus-experiment-replay-request.v1.json +336 -0
- package/schemas/agentic-corpus-experiment-replay-result.v1.json +106 -0
- package/schemas/agentic-corpus-experiment-request.v1.json +204 -0
- package/schemas/agentic-corpus-provenance.v1.json +143 -0
- package/schemas/agentic-corpus-trust-policy.v1.json +133 -0
- package/schemas/agentic-profile-replay-result.v1.json +56 -0
- package/schemas/agentic-profile-replay-result.v2.json +63 -0
- package/schemas/agentic-profile-report.v1.json +961 -0
- package/schemas/agentic-profile-report.v2.json +1674 -0
- package/schemas/agentic-profile-request.v1.json +671 -0
- package/schemas/agentic-profile-request.v2.json +1338 -0
- package/schemas/evidence-manifest.v1.json +636 -0
- package/schemas/replay-result.v1.json +49 -0
- package/schemas/repository-analysis.v1.json +119 -0
- package/schemas/repository-audit.v1.json +811 -0
- package/schemas/repository-init-config.v1.json +183 -0
- package/schemas/repository-init-lock.v1.json +162 -0
- package/schemas/repository-init-result.v1.json +212 -0
- package/schemas/verification-request.v1.json +389 -0
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Runtime doctor
|
|
2
|
+
|
|
3
|
+
AssertLedger's default doctor is static. It reads repository metadata and configuration without
|
|
4
|
+
starting an executable:
|
|
5
|
+
|
|
6
|
+
```text
|
|
7
|
+
assertledger doctor .
|
|
8
|
+
assertledger doctor . --json
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
Use runtime doctor only after reviewing the repository and creating its current
|
|
12
|
+
`assertledger.config.json` and `assertledger.lock.json` files:
|
|
13
|
+
|
|
14
|
+
```text
|
|
15
|
+
assertledger doctor . --runtime --allow-unsafe-execution
|
|
16
|
+
assertledger doctor . --runtime --allow-unsafe-execution --json
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Runtime doctor is explicitly **UNSANDBOXED trusted-local**. The authorization flag is required to
|
|
20
|
+
start processes; it does not create a sandbox. Omitting it returns a stable `BLOCKED` result before
|
|
21
|
+
configuration inspection or executable probes. Unknown flags, repeated flags, and using
|
|
22
|
+
`--allow-unsafe-execution` without `--runtime` are usage errors.
|
|
23
|
+
|
|
24
|
+
## What it checks
|
|
25
|
+
|
|
26
|
+
Version 1.0 supports the generated official `node:test` adapter. It fails closed for other
|
|
27
|
+
frameworks and operator-supplied adapters. In order, it checks:
|
|
28
|
+
|
|
29
|
+
1. explicit trusted-local authorization;
|
|
30
|
+
2. a current AssertLedger configuration and evidence lock;
|
|
31
|
+
3. the supported generated `node:test` adapter;
|
|
32
|
+
4. a stable Node.js executable at version 22.15 or newer;
|
|
33
|
+
5. availability of Node's built-in `node:test` module;
|
|
34
|
+
6. permission to create, write, and clean up an operating-system temporary workspace;
|
|
35
|
+
7. controlled reporter discovery, liveness, and attribution probes.
|
|
36
|
+
|
|
37
|
+
The final probes use disposable synthetic tests. One assertion failure must be attributed to the
|
|
38
|
+
candidate; a generic throw with a nested assertion cause must remain a process crash and must not
|
|
39
|
+
be attributed. Malformed reporter output, missing discovery, cleanup failure, timeout, or process
|
|
40
|
+
failure blocks readiness. AssertLedger does not write to the repository during runtime doctor.
|
|
41
|
+
|
|
42
|
+
## Result contract
|
|
43
|
+
|
|
44
|
+
The SDK exposes the same strict additive contract:
|
|
45
|
+
|
|
46
|
+
```ts
|
|
47
|
+
import { AssertLedger, parseRuntimeDoctorResult } from "assertledger";
|
|
48
|
+
|
|
49
|
+
const result = await new AssertLedger().doctorRuntime(repositoryRoot, {
|
|
50
|
+
allowUnsafeExecution: true,
|
|
51
|
+
});
|
|
52
|
+
parseRuntimeDoctorResult(result);
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
`schemaVersion` is `1.0.0`. Each check has `PASS`, `BLOCKED`, or `LIMITATION`, a stable reason code
|
|
56
|
+
when relevant, a concise summary, and a safe next action. The top-level status is `READY` only when
|
|
57
|
+
every supported runtime boundary passes. JSON results contain no subprocess output, environment
|
|
58
|
+
values, credentials, or repository source.
|
|
59
|
+
|
|
60
|
+
Runtime doctor always records a limitation: it does not run the repository's tests, evaluate a
|
|
61
|
+
candidate, or prove campaign evidence. `READY` means the supported synthetic runtime boundary is
|
|
62
|
+
operational. It is not a test-health or regression-verification verdict. Run an explicitly
|
|
63
|
+
authorized `check` or `verify` workflow for evidence.
|
|
64
|
+
|
|
65
|
+
The CLI exits with `0` for `READY`, `3` for `BLOCKED`, and `64` for malformed arguments.
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# TestExplora curated calibration evidence
|
|
2
|
+
|
|
3
|
+
TestExplora is the third real-fault source selected after BugsJS, ManyBugs, and BugSwarm failed the
|
|
4
|
+
declared reproducibility gates. Its admission status is `ADMITTED_CURATED_CALIBRATION`: eight cases
|
|
5
|
+
are suitable for threshold and protocol calibration, but they are not a hidden holdout, a
|
|
6
|
+
population-rate estimate, or evidence that H3 is ready.
|
|
7
|
+
|
|
8
|
+
## Immutable source inputs
|
|
9
|
+
|
|
10
|
+
- harness: `microsoft/TestExplora@11e6952261f58d3ceb9f4d2571e03aa35dad9523`;
|
|
11
|
+
- harness tree: `98bb0effbcb55b9d15c442660a9de145a7268e14`;
|
|
12
|
+
- harness MIT license bytes: `sha256:98a96c38494df4963951e32b81ec4effebe4c8a812ab8d9e7d185d59e86fe2fc`;
|
|
13
|
+
- dataset revision: `91d8edfb851331bc77eddff40c794336b16c79fe`;
|
|
14
|
+
- source parquet: 19,131,078 bytes,
|
|
15
|
+
`sha256:0d107eb22a082a3549adaf3ea52473d07ffee13b8d9eec908a243916de171c0a`.
|
|
16
|
+
|
|
17
|
+
The pinned parquet contains 1,552 rows from 482 repositories. Its released schema exposes
|
|
18
|
+
`instance_id`, `repo`, `pull_number`, `base_commit`, `pr_patch`, `code_patch`, `test_patch`,
|
|
19
|
+
`documentation`, and `test_invokes`. It does not expose the `FAIL_TO_PASS` or `test_invoke_path`
|
|
20
|
+
fields described by earlier material. `test_invokes` is dependency metadata, not an execution
|
|
21
|
+
command or an outcome oracle.
|
|
22
|
+
|
|
23
|
+
## Qualification protocol
|
|
24
|
+
|
|
25
|
+
Each admitted case uses the upstream production patch and test patch, while keeping the test patch
|
|
26
|
+
outside any future generator context. The environment was reconstructed from pinned base images,
|
|
27
|
+
wheel or source distributions, and subject commits. Dependencies were downloaded before the lock;
|
|
28
|
+
project images were built and all observations executed with networking disabled.
|
|
29
|
+
|
|
30
|
+
Every case required three fresh buggy observations with an assertion failure attributable to the
|
|
31
|
+
introduced test, three fresh fixed observations that passed, and a green positive sentinel. A
|
|
32
|
+
compilation, collection, process, timeout, infrastructure, generic-exception, or positive-suite
|
|
33
|
+
failure was inconclusive and never counted as detection.
|
|
34
|
+
|
|
35
|
+
The eight admitted cases are:
|
|
36
|
+
|
|
37
|
+
- `almarklein__timetagger-117`;
|
|
38
|
+
- `rigetti__pyquil-1758`;
|
|
39
|
+
- `cloud-custodian__cloud-custodian-9970`;
|
|
40
|
+
- `pyinfra-dev__pyinfra-930`;
|
|
41
|
+
- `reata__sqllineage-431`;
|
|
42
|
+
- `PyCQA__pyflakes-438`;
|
|
43
|
+
- `adrienverge__yamllint-422`;
|
|
44
|
+
- `googleapis__python-genai-739`.
|
|
45
|
+
|
|
46
|
+
## Evidence index
|
|
47
|
+
|
|
48
|
+
The external evidence root is `E:\testforge-evidence\corpus-sources`. Subject sources and images
|
|
49
|
+
are not redistributed from this repository.
|
|
50
|
+
|
|
51
|
+
| Campaign | Admitted | Manifest SHA-256 |
|
|
52
|
+
| --- | ---: | --- |
|
|
53
|
+
| `testexplora-preflight-v1` | 5 | `e461cd48f252b9c16705656f42ca3cb8a6389c70c9be0bedac3ce4aafe5f50ad` |
|
|
54
|
+
| `testexplora-replacements-v1` | 1 | `9faf9acec63b96f9e3849a9b37066607394bc5c5899ecc63cf312a3171470e76` |
|
|
55
|
+
| `testexplora-replacements-v2` | 1 | `ca545ccb83841b929181cbac6cf508974ba2d856f008b4922277b42bc85ef9d5` |
|
|
56
|
+
| `testexplora-replacements-v3` | 1 | `121078b7d80ce677148ae5fc0bb79a44e45297631765059b2189122b77c82b09` |
|
|
57
|
+
|
|
58
|
+
The combined admission document is
|
|
59
|
+
`testexplora-replacements-v3/COMBINED_ADMISSION.json`, SHA-256
|
|
60
|
+
`05695fb7afa269eb740c092e438561b54aa55f89857ce58d2d842b5a79603560`.
|
|
61
|
+
It binds the prior campaign manifests and the final campaign's selection, plan, pre-run lock, and
|
|
62
|
+
results. The final recursive manifest lists 61 payloads with zero mismatch and zero unlisted file.
|
|
63
|
+
|
|
64
|
+
## Proof boundary
|
|
65
|
+
|
|
66
|
+
The environments are AssertLedger reconstructions, not reference environments supplied by
|
|
67
|
+
TestExplora. The multi-campaign selection is intentionally calibrated from prior outcomes. These
|
|
68
|
+
receipts establish eight stable real-fault calibration cases, not sampling representativeness,
|
|
69
|
+
generalization, independent temporal custody, or holdout integrity. TestExplora can contribute to
|
|
70
|
+
mature H3 only through a new unseen selection, a two-party committed allocation, signed provenance,
|
|
71
|
+
an externally pinned experiment plan, and independently executed receipts.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
import { readFile } from "node:fs/promises";
|
|
2
|
+
import { AssertLedger } from "assertledger";
|
|
3
|
+
|
|
4
|
+
const requestPath = process.argv[2];
|
|
5
|
+
if (requestPath === undefined) {
|
|
6
|
+
console.error("Usage: node benchmark-request.mjs <agentic-benchmark-request.json>");
|
|
7
|
+
process.exitCode = 64;
|
|
8
|
+
} else {
|
|
9
|
+
const request = JSON.parse(await readFile(requestPath, "utf8"));
|
|
10
|
+
const assertLedger = new AssertLedger();
|
|
11
|
+
const artifact = assertLedger.benchmark(request);
|
|
12
|
+
process.stdout.write(`${JSON.stringify(artifact)}\n`);
|
|
13
|
+
const statuses = artifact.summaries.map((summary) => summary.status);
|
|
14
|
+
process.exitCode = statuses.every((status) => status === "MEASURED")
|
|
15
|
+
? 0
|
|
16
|
+
: statuses.some((status) => status === "OBSERVED_RUN_FAILURE")
|
|
17
|
+
? 2
|
|
18
|
+
: 3;
|
|
19
|
+
}
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
// Deterministic wire-protocol fixture. Fixed durations are illustrative, not measurements.
|
|
2
|
+
import { writeFile } from "node:fs/promises";
|
|
3
|
+
|
|
4
|
+
const candidateFiles = JSON.parse(process.env.TESTFORGE_CANDIDATE_FILES ?? "[]");
|
|
5
|
+
const benchmarkResultFile = process.env.TESTFORGE_BENCHMARK_RESULT_FILE;
|
|
6
|
+
|
|
7
|
+
if (benchmarkResultFile === undefined) {
|
|
8
|
+
await writeFile(
|
|
9
|
+
process.env.TESTFORGE_RESULT_FILE,
|
|
10
|
+
JSON.stringify({
|
|
11
|
+
protocolVersion: "1.0.0",
|
|
12
|
+
outcome: "PASS",
|
|
13
|
+
testsDiscovered: candidateFiles.length === 0 ? 1 : 2,
|
|
14
|
+
candidateTestsDiscovered: candidateFiles.length === 0 ? 0 : 1,
|
|
15
|
+
attributed: candidateFiles.length > 0,
|
|
16
|
+
}),
|
|
17
|
+
);
|
|
18
|
+
} else {
|
|
19
|
+
await writeFile(
|
|
20
|
+
benchmarkResultFile,
|
|
21
|
+
JSON.stringify({
|
|
22
|
+
protocolVersion: "1.0.0",
|
|
23
|
+
outcome: "PASS",
|
|
24
|
+
testsDiscovered: 1,
|
|
25
|
+
candidateTestsDiscovered: 1,
|
|
26
|
+
attributed: true,
|
|
27
|
+
phases: [
|
|
28
|
+
{ phase: "STARTUP", durationUs: 10 },
|
|
29
|
+
{ phase: "COMPILE_OR_COLLECTION", durationUs: 20 },
|
|
30
|
+
{ phase: "EXECUTION", durationUs: 30 },
|
|
31
|
+
],
|
|
32
|
+
cpuTimeUs: null,
|
|
33
|
+
}),
|
|
34
|
+
);
|
|
35
|
+
}
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
import { readFile } from "node:fs/promises";
|
|
2
|
+
import { AssertLedger } from "../../dist/index.js";
|
|
3
|
+
|
|
4
|
+
const artifactPath = process.argv[2];
|
|
5
|
+
if (artifactPath === undefined) {
|
|
6
|
+
throw new Error("Usage: node profile-benchmark.mjs <benchmark-artifact.json>");
|
|
7
|
+
}
|
|
8
|
+
|
|
9
|
+
const benchmarkArtifact = JSON.parse(await readFile(artifactPath, "utf8"));
|
|
10
|
+
const assertLedger = new AssertLedger();
|
|
11
|
+
const report = assertLedger.profileV2({
|
|
12
|
+
schemaVersion: "2.0.0",
|
|
13
|
+
benchmarkArtifact,
|
|
14
|
+
policy: {
|
|
15
|
+
profileVersion: "2.0.0",
|
|
16
|
+
profileId: "example/hardening-v2",
|
|
17
|
+
mode: "HARDENING",
|
|
18
|
+
requiredComparisonScopeDigest: benchmarkArtifact.comparisonScopeDigest,
|
|
19
|
+
costBasis: {
|
|
20
|
+
regime: "WARM",
|
|
21
|
+
measure: "WALL",
|
|
22
|
+
aggregation: "TOTAL",
|
|
23
|
+
statistic: "P95",
|
|
24
|
+
unit: "MICROSECOND",
|
|
25
|
+
portfolioAggregation: "SUM_OF_INDIVIDUAL_P95",
|
|
26
|
+
},
|
|
27
|
+
lanes: [
|
|
28
|
+
{ id: "instant", maximumWarmTotalWallP95Us: 2_000_000 },
|
|
29
|
+
{ id: "loop", maximumWarmTotalWallP95Us: 10_000_000 },
|
|
30
|
+
],
|
|
31
|
+
},
|
|
32
|
+
});
|
|
33
|
+
|
|
34
|
+
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
import { readFile } from "node:fs/promises";
|
|
2
|
+
import { AssertLedger } from "assertledger";
|
|
3
|
+
|
|
4
|
+
const manifestPath = process.argv[2];
|
|
5
|
+
if (manifestPath === undefined) {
|
|
6
|
+
console.error("Usage: node profile-manifest.mjs <manifest.json> [profile-id]");
|
|
7
|
+
process.exitCode = 64;
|
|
8
|
+
} else {
|
|
9
|
+
const manifest = JSON.parse(await readFile(manifestPath, "utf8"));
|
|
10
|
+
const assertLedger = new AssertLedger();
|
|
11
|
+
const report = assertLedger.profile({
|
|
12
|
+
schemaVersion: "1.0.0",
|
|
13
|
+
manifest,
|
|
14
|
+
policy: {
|
|
15
|
+
profileVersion: "1.0.0",
|
|
16
|
+
profileId: process.argv[3] ?? "example/default",
|
|
17
|
+
mode: "HARDENING",
|
|
18
|
+
minimumTimingSamples: 3,
|
|
19
|
+
lanes: [
|
|
20
|
+
{ id: "instant", maximumReferenceP95Ms: 2_000 },
|
|
21
|
+
{ id: "loop", maximumReferenceP95Ms: 10_000 },
|
|
22
|
+
{ id: "gate", maximumReferenceP95Ms: 60_000 },
|
|
23
|
+
],
|
|
24
|
+
},
|
|
25
|
+
});
|
|
26
|
+
process.stdout.write(`${JSON.stringify(report)}\n`);
|
|
27
|
+
process.exitCode = report.status === "QUALIFIED" ? 0 : 2;
|
|
28
|
+
}
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# A historical bug, exercised from the installed package
|
|
2
|
+
|
|
3
|
+
This example uses byte-exact `index.js` snapshots from the MIT-licensed
|
|
4
|
+
[`escape-string-regexp` Unicode-hyphen correction](https://github.com/sindresorhus/escape-string-regexp/commit/732905da074f0220487ad6a27590f89bd0819374).
|
|
5
|
+
The upstream commit and blob IDs, SHA-256 hashes and license are bundled beside them.
|
|
6
|
+
|
|
7
|
+
The demo creates a **new projected Git history** with an AssertLedger-authored `node:test`
|
|
8
|
+
harness. It does not claim to run the upstream repository's original AVA tests. No network
|
|
9
|
+
access or dependency installation is needed once AssertLedger is installed; Node and Git
|
|
10
|
+
must be available.
|
|
11
|
+
|
|
12
|
+
Run the following from your project after installing `assertledger`:
|
|
13
|
+
|
|
14
|
+
```sh
|
|
15
|
+
node node_modules/assertledger/examples/git-history/create-demo.mjs regression-demo
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The JSON output gives `before`, `after`, `neutral`, the test path and the declared neutral
|
|
19
|
+
reason. Copy those exact values into `assertledger check`; see the
|
|
20
|
+
[Git command reference](../../docs/git-regression.md). The target directory must not exist.
|
|
21
|
+
|
|
22
|
+
| Candidate | Expected result | Why |
|
|
23
|
+
| --- | --- | --- |
|
|
24
|
+
| `strong.test.mjs` | `VERIFIED` | An assertion detects the invalid Unicode pattern on the buggy code. |
|
|
25
|
+
| `weak.test.mjs` | `REJECTED` | Checking only the return type passes on both versions. |
|
|
26
|
+
| `crash.test.mjs` | `REJECTED` | A generic error on buggy code is not an attributed assertion. |
|
|
27
|
+
|
|
28
|
+
Use a new `--out` directory for each candidate. Base tests stay identical across all three
|
|
29
|
+
worlds. The neutral commit changes only documentation, so it is a declared control, not an
|
|
30
|
+
independent demonstration of robustness. `trusted-local` execution remains unsandboxed.
|
|
31
|
+
|
|
32
|
+
The package smoke runs all three candidates, replays every manifest, checks source and index
|
|
33
|
+
preservation, and retains the manifests, executed requests and summaries as CI artifacts.
|
|
34
|
+
|
|
35
|
+
## Français
|
|
36
|
+
|
|
37
|
+
Le cas reproduit une correction réelle : l’ancienne fonction échappait le tiret avec une forme
|
|
38
|
+
invalide dans une expression régulière Unicode. Les deux fichiers sources sont conservés à
|
|
39
|
+
l’octet près, avec leurs références Git et leur licence.
|
|
40
|
+
|
|
41
|
+
L’historique de démonstration et les tests `node:test` sont créés par AssertLedger. Le test fort
|
|
42
|
+
vérifie que la construction de l’expression régulière ne lève pas d’erreur ; le test faible
|
|
43
|
+
vérifie seulement le type de retour. Une exception générique ne suffit pas pour prouver la
|
|
44
|
+
détection du défaut.
|
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
import assert from "node:assert/strict";
|
|
3
|
+
import { spawnSync } from "node:child_process";
|
|
4
|
+
import { createHash } from "node:crypto";
|
|
5
|
+
import { copyFileSync, mkdirSync, readFileSync, realpathSync, writeFileSync } from "node:fs";
|
|
6
|
+
import path from "node:path";
|
|
7
|
+
import { fileURLToPath } from "node:url";
|
|
8
|
+
|
|
9
|
+
// The snapshots retain their upstream bytes and MIT notice. Only the harness is authored here.
|
|
10
|
+
const sources = path.join(path.dirname(fileURLToPath(import.meta.url)), "escape-string-regexp");
|
|
11
|
+
const provenance = JSON.parse(readFileSync(path.join(sources, "provenance.json"), "utf8"));
|
|
12
|
+
for (const source of provenance.sources) {
|
|
13
|
+
const bytes = readFileSync(path.join(sources, source.file));
|
|
14
|
+
assert.equal(createHash("sha256").update(bytes).digest("hex"), source.sha256);
|
|
15
|
+
assert.equal(
|
|
16
|
+
createHash("sha1").update(`blob ${bytes.length}\0`).update(bytes).digest("hex"),
|
|
17
|
+
source.gitBlob,
|
|
18
|
+
);
|
|
19
|
+
}
|
|
20
|
+
assert.equal(process.argv.length, 3, "Usage: node create-demo.mjs NEW_DIRECTORY");
|
|
21
|
+
const requested = path.resolve(process.argv[2]);
|
|
22
|
+
mkdirSync(requested); // Exclusive creation: never overwrite a user's repository.
|
|
23
|
+
const root = realpathSync(requested);
|
|
24
|
+
const environment = {
|
|
25
|
+
PATH: process.env.PATH ?? "",
|
|
26
|
+
SystemRoot: process.env.SystemRoot ?? "",
|
|
27
|
+
TEMP: process.env.TEMP ?? "",
|
|
28
|
+
TMP: process.env.TMP ?? "",
|
|
29
|
+
GIT_CONFIG_NOSYSTEM: "1",
|
|
30
|
+
GIT_CONFIG_GLOBAL: path.join(root, ".absent-global-config"),
|
|
31
|
+
GIT_AUTHOR_DATE: "2000-01-01T00:00:00Z",
|
|
32
|
+
GIT_COMMITTER_DATE: "2000-01-01T00:00:00Z",
|
|
33
|
+
};
|
|
34
|
+
function git(args) {
|
|
35
|
+
const result = spawnSync(
|
|
36
|
+
"git",
|
|
37
|
+
[
|
|
38
|
+
"-c",
|
|
39
|
+
"core.autocrlf=false",
|
|
40
|
+
"-c",
|
|
41
|
+
"commit.gpgSign=false",
|
|
42
|
+
"-c",
|
|
43
|
+
`core.hooksPath=${path.join(root, ".git/no-hooks")}`,
|
|
44
|
+
"-c",
|
|
45
|
+
"user.name=AssertLedger Demo",
|
|
46
|
+
"-c",
|
|
47
|
+
"user.email=demo@example.invalid",
|
|
48
|
+
...args,
|
|
49
|
+
],
|
|
50
|
+
{
|
|
51
|
+
cwd: root,
|
|
52
|
+
env: environment,
|
|
53
|
+
shell: false,
|
|
54
|
+
windowsHide: true,
|
|
55
|
+
encoding: "utf8",
|
|
56
|
+
timeout: 10_000,
|
|
57
|
+
maxBuffer: 1_048_576,
|
|
58
|
+
},
|
|
59
|
+
);
|
|
60
|
+
assert.ifError(result.error);
|
|
61
|
+
assert.equal(result.signal, null);
|
|
62
|
+
assert.equal(result.status, 0, result.stderr);
|
|
63
|
+
return result.stdout.trim();
|
|
64
|
+
}
|
|
65
|
+
function commit(message) {
|
|
66
|
+
git(["add", "."]);
|
|
67
|
+
git(["commit", "-m", message]);
|
|
68
|
+
return git(["rev-parse", "HEAD"]);
|
|
69
|
+
}
|
|
70
|
+
git(["init", "--template=", "--initial-branch=main"]);
|
|
71
|
+
writeFileSync(
|
|
72
|
+
path.join(root, "package.json"),
|
|
73
|
+
'{"name":"assertledger-historical-demo","private":true}\n',
|
|
74
|
+
);
|
|
75
|
+
copyFileSync(path.join(sources, "LICENSE"), path.join(root, "LICENSE"));
|
|
76
|
+
copyFileSync(path.join(sources, "before.cjs.txt"), path.join(root, "subject.cjs"));
|
|
77
|
+
writeFileSync(
|
|
78
|
+
path.join(root, "README.md"),
|
|
79
|
+
"Projection of a real upstream correction; see AssertLedger example provenance.\n",
|
|
80
|
+
);
|
|
81
|
+
const prelude =
|
|
82
|
+
'import assert from "node:assert/strict";\nimport test from "node:test";\nimport escape from "./subject.cjs";\n';
|
|
83
|
+
writeFileSync(
|
|
84
|
+
path.join(root, "base.test.mjs"),
|
|
85
|
+
`${prelude}test("ordinary characters", () => assert.equal(escape("hello"), "hello"));\n`,
|
|
86
|
+
);
|
|
87
|
+
writeFileSync(
|
|
88
|
+
path.join(root, "strong.test.mjs"),
|
|
89
|
+
`${prelude}test("Unicode hyphen regression", () => {
|
|
90
|
+
assert.doesNotThrow(() => new RegExp(escape("-"), "u"));
|
|
91
|
+
assert.equal(new RegExp(escape("-"), "u").test("-"), true);
|
|
92
|
+
});\n`,
|
|
93
|
+
);
|
|
94
|
+
writeFileSync(
|
|
95
|
+
path.join(root, "weak.test.mjs"),
|
|
96
|
+
`${prelude}test("weak check", () => assert.equal(typeof escape("-"), "string"));\n`,
|
|
97
|
+
);
|
|
98
|
+
writeFileSync(
|
|
99
|
+
path.join(root, "crash.test.mjs"),
|
|
100
|
+
`${prelude}test("generic throw is not evidence", () => {
|
|
101
|
+
if (escape("-") === "\\\\-") throw new Error("generic failure");
|
|
102
|
+
assert.equal(typeof escape("-"), "string");
|
|
103
|
+
});\n`,
|
|
104
|
+
);
|
|
105
|
+
const before = commit(
|
|
106
|
+
`Upstream buggy module ${provenance.sources[0].commit}, with node:test harness`,
|
|
107
|
+
);
|
|
108
|
+
copyFileSync(path.join(sources, "fixed.cjs.txt"), path.join(root, "subject.cjs"));
|
|
109
|
+
const after = commit(`Upstream corrected module ${provenance.sources[1].commit}, byte-exact`);
|
|
110
|
+
writeFileSync(
|
|
111
|
+
path.join(root, "README.md"),
|
|
112
|
+
"Projection of a real upstream correction; documentation-only neutral control.\n",
|
|
113
|
+
);
|
|
114
|
+
const neutral = commit("Declared neutral: documentation-only edit to corrected tree");
|
|
115
|
+
console.log(
|
|
116
|
+
JSON.stringify({
|
|
117
|
+
repository: root,
|
|
118
|
+
before,
|
|
119
|
+
after,
|
|
120
|
+
neutral,
|
|
121
|
+
neutralReason:
|
|
122
|
+
"Documentation-only change to the corrected tree; no additional behavioral robustness claim",
|
|
123
|
+
test: "strong.test.mjs",
|
|
124
|
+
baseTests: ["base.test.mjs"],
|
|
125
|
+
out: "evidence-strong",
|
|
126
|
+
provenance,
|
|
127
|
+
}),
|
|
128
|
+
);
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
|
6
|
+
|
|
7
|
+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
|
8
|
+
|
|
9
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
module.exports = string => {
|
|
4
|
+
if (typeof string !== 'string') {
|
|
5
|
+
throw new TypeError('Expected a string');
|
|
6
|
+
}
|
|
7
|
+
|
|
8
|
+
// Escape characters with special meaning either inside or outside character sets.
|
|
9
|
+
// Use a simple backslash escape when it’s always valid, and a \unnnn escape when the simpler form would be disallowed by Unicode patterns’ stricter grammar.
|
|
10
|
+
return string
|
|
11
|
+
.replace(/[|\\{}()[\]^$+*?.]/g, '\\$&')
|
|
12
|
+
.replace(/-/g, '\\u002d');
|
|
13
|
+
};
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"repository": "sindresorhus/escape-string-regexp",
|
|
3
|
+
"fixUrl": "https://github.com/sindresorhus/escape-string-regexp/commit/732905da074f0220487ad6a27590f89bd0819374",
|
|
4
|
+
"sources": [
|
|
5
|
+
{
|
|
6
|
+
"file": "before.cjs.txt",
|
|
7
|
+
"upstreamPath": "index.js",
|
|
8
|
+
"commit": "5085b257c801507460270b747f645276fbc1d937",
|
|
9
|
+
"gitBlob": "58217a4efa3c835c532499a3dad887017dc70a6b",
|
|
10
|
+
"sha256": "48b8be4119e6f09b8942c490397fc047da012e0cc223d75a76363856af68fce4"
|
|
11
|
+
},
|
|
12
|
+
{
|
|
13
|
+
"file": "fixed.cjs.txt",
|
|
14
|
+
"upstreamPath": "index.js",
|
|
15
|
+
"commit": "732905da074f0220487ad6a27590f89bd0819374",
|
|
16
|
+
"gitBlob": "e5bb9db7933b7230327c7d99cc8459575f090dd4",
|
|
17
|
+
"sha256": "44f81777dbee24c245fc220d9e019e031da31a5722743a74272f218b7ffed563"
|
|
18
|
+
},
|
|
19
|
+
{
|
|
20
|
+
"file": "LICENSE",
|
|
21
|
+
"upstreamPath": "license",
|
|
22
|
+
"commit": "732905da074f0220487ad6a27590f89bd0819374",
|
|
23
|
+
"gitBlob": "e7af2f77107d73046421ef56c4684cbfdd3c1e89",
|
|
24
|
+
"sha256": "48da2f39e100d4085767e94966b43f4fa95ff6a0698fba57ed460914e35f94a0"
|
|
25
|
+
}
|
|
26
|
+
],
|
|
27
|
+
"scope": "Byte-exact upstream module snapshots, projected into an AssertLedger-authored node:test Git fixture. This does not qualify the complete upstream repository or its original AVA suite."
|
|
28
|
+
}
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schemaVersion": "1.0.0",
|
|
3
|
+
"repository": {
|
|
4
|
+
"root": "./examples/node-test/repository",
|
|
5
|
+
"exclude": [".git", ".testforge", "node_modules"]
|
|
6
|
+
},
|
|
7
|
+
"adapter": {
|
|
8
|
+
"kind": "node-test",
|
|
9
|
+
"executable": "node",
|
|
10
|
+
"baseTestFiles": ["tests/base.test.js"]
|
|
11
|
+
},
|
|
12
|
+
"isolation": {
|
|
13
|
+
"kind": "trusted-local",
|
|
14
|
+
"acknowledgedUnsafeExecution": true,
|
|
15
|
+
"environmentAllowlist": ["PATH", "SystemRoot", "WINDIR", "TEMP", "TMP"]
|
|
16
|
+
},
|
|
17
|
+
"candidateRoots": ["tests/candidates"],
|
|
18
|
+
"budgets": {
|
|
19
|
+
"maximumCandidates": 4,
|
|
20
|
+
"maximumWorlds": 4,
|
|
21
|
+
"maximumExecutions": 24,
|
|
22
|
+
"maximumRepositoryFiles": 10000,
|
|
23
|
+
"maximumRepositoryBytes": 100000000,
|
|
24
|
+
"maximumWorldOverlayBytes": 1000000,
|
|
25
|
+
"maximumCandidateBytes": 16384,
|
|
26
|
+
"maximumTotalCandidateBytes": 65536,
|
|
27
|
+
"timeoutMsPerExecution": 5000,
|
|
28
|
+
"maximumOutputBytes": 65536
|
|
29
|
+
},
|
|
30
|
+
"policy": {
|
|
31
|
+
"policyVersion": "1.0.0",
|
|
32
|
+
"requiredAttempts": 2,
|
|
33
|
+
"minimumTargetWeightPermille": 1000,
|
|
34
|
+
"maximumSelectedCandidates": 1,
|
|
35
|
+
"acceptedTargetOutcomes": ["ASSERTION_FAILURE"]
|
|
36
|
+
},
|
|
37
|
+
"worlds": [
|
|
38
|
+
{
|
|
39
|
+
"id": "reference",
|
|
40
|
+
"kind": "REFERENCE",
|
|
41
|
+
"required": true,
|
|
42
|
+
"weight": 0,
|
|
43
|
+
"provenance": "example:correct-implementation",
|
|
44
|
+
"files": []
|
|
45
|
+
},
|
|
46
|
+
{
|
|
47
|
+
"id": "target-parity-inversion",
|
|
48
|
+
"kind": "TARGET",
|
|
49
|
+
"required": true,
|
|
50
|
+
"weight": 1,
|
|
51
|
+
"provenance": "example:known-fault",
|
|
52
|
+
"files": [
|
|
53
|
+
{
|
|
54
|
+
"path": "src/is-even.js",
|
|
55
|
+
"content": "export function isEven(value) {\n return value % 2 === 1;\n}\n"
|
|
56
|
+
}
|
|
57
|
+
]
|
|
58
|
+
},
|
|
59
|
+
{
|
|
60
|
+
"id": "neutral-bitwise-equivalence",
|
|
61
|
+
"kind": "NEUTRAL",
|
|
62
|
+
"required": true,
|
|
63
|
+
"weight": 0,
|
|
64
|
+
"provenance": "example:reviewed-equivalent",
|
|
65
|
+
"files": [
|
|
66
|
+
{
|
|
67
|
+
"path": "src/is-even.js",
|
|
68
|
+
"content": "export function isEven(value) {\n return (value & 1) === 0;\n}\n"
|
|
69
|
+
}
|
|
70
|
+
]
|
|
71
|
+
}
|
|
72
|
+
],
|
|
73
|
+
"candidates": [
|
|
74
|
+
{
|
|
75
|
+
"id": "strong",
|
|
76
|
+
"files": [
|
|
77
|
+
{
|
|
78
|
+
"path": "tests/candidates/strong.test.js",
|
|
79
|
+
"content": "import assert from \"node:assert/strict\";\nimport test from \"node:test\";\nimport { isEven } from \"../../src/is-even.js\";\ntest(\"two is even\", () => assert.equal(isEven(2), true));\n"
|
|
80
|
+
}
|
|
81
|
+
]
|
|
82
|
+
},
|
|
83
|
+
{
|
|
84
|
+
"id": "weak",
|
|
85
|
+
"files": [
|
|
86
|
+
{
|
|
87
|
+
"path": "tests/candidates/weak.test.js",
|
|
88
|
+
"content": "import assert from \"node:assert/strict\";\nimport test from \"node:test\";\nimport { isEven } from \"../../src/is-even.js\";\ntest(\"returns a boolean\", () => assert.equal(typeof isEven(2), \"boolean\"));\n"
|
|
89
|
+
}
|
|
90
|
+
]
|
|
91
|
+
}
|
|
92
|
+
]
|
|
93
|
+
}
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: assertledger
|
|
3
|
+
description: Analyze a repository, submit candidate tests, and accept only deterministic AssertLedger evidence.
|
|
4
|
+
metadata:
|
|
5
|
+
version: "1.0.0"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# AssertLedger skill
|
|
9
|
+
|
|
10
|
+
Use this skill when an agent is asked to generate tests whose actual evidence must be checked.
|
|
11
|
+
|
|
12
|
+
AssertLedger was formerly named TestForge. The `assertledger` CLI is the preferred binary; the
|
|
13
|
+
`testforge` binary remains a legacy-compatible alias for the same commands during the compatibility
|
|
14
|
+
window.
|
|
15
|
+
|
|
16
|
+
## Preferred path: a committed regression
|
|
17
|
+
|
|
18
|
+
1. Run `assertledger doctor <repo> --json` for static readiness. Runtime probes require the separate
|
|
19
|
+
`doctor --runtime --allow-unsafe-execution` command and prior operator authorization.
|
|
20
|
+
2. Ask the operator or harness for the buggy, corrected and neutral revisions, the neutral reason,
|
|
21
|
+
candidate path and unchanged base tests. Do not invent a neutral control's meaning.
|
|
22
|
+
3. For committed dependency-free JavaScript `node:test`, use `assertledger check` or the MCP
|
|
23
|
+
`assertledger_check` tool. The server operator must enable execution; a tool payload cannot grant it.
|
|
24
|
+
4. Preserve the manifest and run `assertledger replay`. Use `assertledger explain CODE` or the
|
|
25
|
+
read-only `assertledger_explain` tool to understand a refusal without changing the policy.
|
|
26
|
+
5. The configuration, skill and server do not grant trust. Never treat a crash, timeout, compilation
|
|
27
|
+
error or missing test as detection. The returned limitations remain part of the result.
|
|
28
|
+
|
|
29
|
+
## Lower-level verification requests
|
|
30
|
+
|
|
31
|
+
1. Run `assertledger init <repo> --dry-run --json`, review conflicts and planned bytes, then run
|
|
32
|
+
`assertledger init <repo> --json`. Initialization is static and manages only the two AssertLedger
|
|
33
|
+
root files; it never executes a command or generates worlds or candidates.
|
|
34
|
+
2. Run `assertledger audit <repo> --json`, then `assertledger analyze <repo> --json` for generation
|
|
35
|
+
context. Audit may report `VERIFICATION_REQUEST_UNAVAILABLE` until the harness supplies the two
|
|
36
|
+
required operator inputs: `worlds` and `candidates`.
|
|
37
|
+
3. Propose candidate files only under the config's `candidateRoots`.
|
|
38
|
+
4. Do not let candidate content choose worlds, expected outcomes, policy, or budgets. Those belong to
|
|
39
|
+
the operator or harness.
|
|
40
|
+
5. Submit all candidates in one versioned verification request.
|
|
41
|
+
6. Run `assertledger verify <request> --allow-unsafe-execution --json` only when the controlling human
|
|
42
|
+
or CI policy has authorized the `UNSANDBOXED` local backend.
|
|
43
|
+
7. Preserve the returned manifest, then run `assertledger replay <manifest> --json`.
|
|
44
|
+
8. Treat only `VERIFIED` plus a replay result whose `valid` field is true as positive evidence. This
|
|
45
|
+
requires valid schema, decision digest, artifact digest, and decision semantics. `REJECTED`,
|
|
46
|
+
`INCONCLUSIVE`, invalid input, engine errors, and invalid replay results are stop conditions, not
|
|
47
|
+
soft passes.
|
|
48
|
+
9. Preserve the emitted manifest as the auditable artifact. Never summarize away failed attempts,
|
|
49
|
+
controls, limitations, or isolation level.
|
|
50
|
+
|
|
51
|
+
Never claim that AssertLedger proves overall program correctness or universal non-flakiness.
|