assertledger 1.0.0

package/docs/agentic-corpus-experiment-h3.md

@@ -0,0 +1,89 @@
+# Agentic corpus experiment artifact H3 v1
+
+H3 evaluates historical-fault detection only after three independent operator pins are supplied:
+the corpus trust-policy digest, the two-party allocation-commitment digest, and the pre-declared
+experiment-plan digest. None may fall back to a digest embedded in the artifact.
+
+## Allocation commitment and reveal
+
+Allocation Commitment/Reveal v1 binds the exact case identity, provenance, source ID and
+source-identity digests plus a calibration count for every source stratum. Each source with at
+least two cases contributes at least one calibration and one holdout case. Distinct Ed25519 AUTHOR
+and REVIEWER principals each commit a
+32-byte secret share and sign the same domain-separated commitment projection containing both
+share commitments. The reveal must contain both pre-committed shares. AssertLedger sorts them by key
+ID, length-prefixes their decoded bytes, derives the final seed, then recomputes and replays Corpus
+Allocation v1. This prevents either principal from choosing or replacing a share after the joint
+commitment; it does not prove that a principal avoided grinding before committing its own share.
+
+## Pre-declared plan
+
+The externally pinned Experiment Plan v1 contains no outcomes, logs, durations or results. It binds
+the allocation and trust anchors, H3 protocol, detailed subject revisions and qualification
+receipts, adapter identities, a shared content-addressed candidate universe, each arm's
+content-addressed selection and derived suite digest, executable identities plus argument arrays,
+repository state, test-suite digests, time/output caps, and the exact schedule. Every
+case/arm/attempt/state tuple appears exactly once. Replay derives both equal
+`PLANNED_PROCESS_EXECUTION` counts and equal total planned timeout ceilings by summing each
+scheduled command's `timeoutMs`. This is comparable pre-declared time allowance, not observed
+wall-time or CPU equality.
+
+## Run evidence and derivation
+
+Each scheduled process has one canonical run receipt binding its plan, run and command digests,
+case, arm, attempt, subject state, repository and test-suite digests, the applied timeout limit,
+process exit/timed-out state,
+and SHA-256 digests for stdout, stderr and a structured result. Replay resolves the actual receipt,
+stdout, stderr and structured-result bytes and recomputes every digest. The strict
+`TESTFORGE_H3_STRUCTURED_RESULT_V1` parser derives normalized outcome and attribution; no outcome
+field in the receipt is trusted.
+
+Replay resolves every `candidateDigest` in the frozen universe against the supplied evidence
+contents, recomputes its raw SHA-256 digest, and checks the artifact's derived candidate-set digest.
+Missing or mismatched candidate bytes invalidate replay. This proves the identity of the bytes
+provided to AssertLedger, not that the runner actually executed those bytes.
+
+Process consistency is exact: `PASS` requires `timedOut=false` and exit code zero;
+`PROCESS_CRASH` requires `timedOut=false` and a null exit code; `TIMEOUT` requires
+`timedOut=true`; every assertion, compile, collection, infrastructure, or no-test failure requires
+`timedOut=false` and a nonzero non-null exit code. A contradiction invalidates the artifact rather
+than becoming insufficient evidence.
+
+Detection requires an attributed assertion failure on the buggy revision and PASS on the fixed
+revision and required suite for every stable attempt. Buggy PASS is a valid non-detection. Compile,
+collection, timeout, crash, infrastructure and no-test outcomes are valid operational evidence but
+make H3 `INSUFFICIENT`; contradictory attempts do the same. They never become detections.
+Non-inferiority must hold both in aggregate and independently for every source stratum, so gains
+on one source cannot compensate for regressions on another.
+
+The SDK requires the artifact and replay options as separate arguments. The CLI requires separate
+files and flags for all three external pins, policy, commitment/reveal, allocation, plan, subject
+evidence and evidence bytes. H3 tools are intentionally absent from the default agent-facing MCP
+server because an input JSON document cannot establish those operator-owned roots of trust.
+
+## Compatibility and non-claims
+
+Existing AgenticCorpusCase v1 `h1`-`h4` members remain wire-compatible as
+`LEGACY_CASE_METRICS`, but any such member makes a subject inadmissible to mature H3 replay. This
+contract establishes deterministic consistency relative to supplied external anchors and bytes. It
+does not authenticate the machine that executed commands, prove private custody, or establish that
+the corpus represents all real defects. Qualification receipt digests are pre-declared identities;
+H3 v1 does not resolve their bytes. A run receipt's applied-timeout field binds the runner's
+attestation to the pre-declared limit; it is not an independent execution authority. H1 latency
+and H4 mutation strength remain separate and cannot
+compensate for H3 failure.
+
+## Migration note
+
+The pre-release v1 allocation and H3 projections were tightened before a stable release. Global
+`calibrationCount` plus `caseIds` inputs are rejected; callers must provide source strata. Candidate
+IDs became content-addressed references, plan budgets gained derived timeout ceilings, receipts
+gained `appliedTimeoutMs`, and results gained per-source strata. Existing stored drafts must be
+regenerated and externally repinned. Artifacts also gained a derived candidate-set digest and
+replay gained candidate-byte resolution. Redigesting an old document is insufficient.
+
+Recent research reinforces those boundaries: coverage and mutation can be context-dependent
+proxies when the code under test may already be buggy ([arXiv:2607.22880](https://arxiv.org/abs/2607.22880)),
+while a strong recent baseline and evaluation granularity can materially change measured cost
+([arXiv:2601.09695](https://arxiv.org/abs/2601.09695)). AssertLedger therefore requires a frozen shared
+candidate universe rather than comparison against a conveniently weak historical prompt.

package/docs/agentic-corpus-plan.md

@@ -0,0 +1,105 @@
+# Agentic profile corpus plan
+
+This document defines the first empirical calibration corpus for the Agentic Test Profile. It is a
+reconstruction plan, not collected evidence. A case counts only after source-native reproduction,
+dual-signed provenance, and admission to the physical corpus.
+
+## Frozen pilot scope
+
+The planned pilot targets 24 independently qualified historical-fault cases: eight from each
+source. Four cases per source are intended for public calibration and four for holdout. Case
+identifiers are frozen only after the buggy/fixed pair reproduces on the pinned environment. At
+present, only the eight TestExplora cases below have qualified, and only for curated calibration.
+
+| Source | Pin | Cases | Primary contribution |
+| --- | --- | ---: | --- |
+| Defects4J 3.0.1 | `rjust/defects4j@8c16da8230843cdc918eaf4ddb449637f02b83c6` | 8 | H1, H3, and the pilot's only native H4 mutation baseline |
+| TestExplora | harness `microsoft/TestExplora@11e6952261f58d3ceb9f4d2571e03aa35dad9523`; dataset `91d8edfb851331bc77eddff40c794336b16c79fe` | 8 | Curated H1/H3 calibration candidates from real Python pull requests; not a holdout |
+| SWE-bench Verified | dataset `78f471bf655a3137b2e8a75af1501690ec009ec3`; harness `7a21e05772954cc81471ae19d56f436cecf43c54` | 8 | H1 and H3 for recent Python issue/PR regressions |
+
+TestExplora is admitted only as `ADMITTED_CURATED_CALIBRATION`. Eight cases reproduced with stable
+attributable assertion failures on the buggy state and stable passes on the fixed state, but the
+selection was calibrated across multiple pre-declared campaigns. It is therefore neither a hidden
+holdout nor evidence that the corpus is `READY_H3`. The harness also does not publish a frozen
+per-task environment; AssertLedger reconstructed dependency images from pinned inputs and executed
+them without network access. See [TestExplora calibration evidence](testexplora-calibration.md).
+Subject repositories and container images must be recorded by immutable commit or image digest in
+each execution receipt.
+
+## Allocation freeze boundary
+
+Calibration and holdout membership must be created once before any profile threshold, portfolio,
+or prompt is tuned. A mature freeze uses Allocation Commitment/Reveal v1: distinct authorized
+AUTHOR and REVIEWER principals pre-commit independent secret shares while signing the exact case
+identity/provenance/source set and the calibration count for every source stratum. Every source
+with at least two admitted cases must retain non-empty calibration and holdout partitions. After
+both shares are revealed, AssertLedger derives a
+domain-separated seed, applies `SHA256_ASCENDING_SPLIT_V1`, and replays the resulting allocation.
+The operator must independently pin the trust-policy and commitment digests.
+
+Real allocation seeds, membership, score ordering, and allocation digests remain in private
+holdout custody outside the repository. Public tests use an unrelated synthetic allocation only;
+they establish algorithm conformance without revealing any real holdout membership. The admitted
+cases remain pending until their complete provenance sidecars and required evidence are present.
+
+The earlier single-nonce pilot commitment is invalidated: one party could choose the seed and the
+receipt did not bind dual authorization, case provenance, or the allocation protocol. It is not a
+mature freeze and must not be used for an H3 claim. No valid real holdout commitment, seed,
+allocation digest, case identifier, or membership is published in this repository.
+
+The existing optional `h1` through `h4` members of an AgenticCorpusCase v1 are retained unchanged
+for wire compatibility and are designated `LEGACY_CASE_METRICS`. They are self-contained summary
+claims, not replayable experiment artifacts, and therefore cannot support a mature profile,
+sweet-spot, or hypothesis claim. Mature H3 evaluation uses the separate versioned
+[experiment artifact](agentic-corpus-experiment-h3.md), which requires an externally pinned
+two-party allocation commitment and pre-declared experiment plan. H1, H2, and H4 remain blocked on
+their own mature experiment artifacts.
+
+## Admission protocol
+
+Every case must record the benchmark pin, exact subject buggy and fixed revisions, environment and
+tool versions, an executable-plus-arguments command, input and output digests, and at least three
+repeated source-native runs. The expected fault must fail on the buggy revision and pass on the
+fixed revision together with the source-required regression suite.
+
+Timeout, setup, compilation, collection, process, and infrastructure failures are inconclusive.
+They never count as historical-fault detections or mutant kills. Source files are reconstructed from
+their upstream repositories by default; AssertLedger does not assume that a benchmark framework's
+license grants redistribution rights for every subject repository.
+
+After reproduction, an authorized author and a different authorized reviewer sign the canonical
+case sidecar described in [Agentic corpus provenance v1](agentic-corpus-provenance.md). The expected
+trust-policy digest must be distributed outside the corpus. Pending or unsigned cases stay outside
+the active `public/` and `private/` splits.
+
+## Hypothesis boundaries
+
+- H1 uses warmed repeated execution p95 for the selected portfolio and the declared full eligible
+  suite at equal target strength.
+- H2 requires separately constructed, language-specific behavior-preserving neutral worlds and an
+  independent preservation review. The three historical-fault datasets do not supply this proof.
+  Formatting-only controls may test integrity, but cannot establish substantive neutral robustness.
+- H3 uses source-stratified historical-fault holdouts that remain unseen during policy calibration.
+- H4 is scoped to the Defects4J tranche in the pilot because it alone exposes a native mutation
+  interface. No cross-language H4 claim follows from this tranche.
+
+The corpus becomes mechanically `READY` only after the existing minimum-case, minimum-source,
+public/holdout, H1-H4 evaluability, and signed-provenance gates pass. `READY` means the corpus may be
+evaluated; it does not mean the hypotheses are supported. A public sweet-spot claim additionally
+requires the predeclared public and holdout evaluation criteria to succeed.
+
+## Primary sources
+
+- [Defects4J pinned README](https://raw.githubusercontent.com/rjust/defects4j/8c16da8230843cdc918eaf4ddb449637f02b83c6/README.md)
+  and [command reference](https://defects4j.org/html_doc/index.html)
+- [TestExplora pinned harness](https://github.com/microsoft/TestExplora/tree/11e6952261f58d3ceb9f4d2571e03aa35dad9523)
+  and [dataset](https://huggingface.co/datasets/microsoft/TestExplora/tree/91d8edfb851331bc77eddff40c794336b16c79fe)
+- [SWE-bench dataset fields](https://www.swebench.com/SWE-bench/guides/datasets/),
+  [pinned harness](https://github.com/SWE-bench/SWE-bench/tree/7a21e05772954cc81471ae19d56f436cecf43c54),
+  and [benchmark description](https://www.swebench.com/original.html)
+
+Defects4J, TestExplora, and the SWE-bench harness use MIT licenses at the listed pins. Each
+TestExplora subject keeps its own license, verified at the selected base commit; the dataset license
+does not replace subject licensing. BugsInPy and Bugs.jar are excluded because their official
+repositories do not provide a usable declared license. GitBug-Java remains a later expansion due
+to its substantially larger documented storage footprint.

package/docs/agentic-corpus-provenance.md

@@ -0,0 +1,59 @@
+# Agentic corpus provenance v1
+
+An Agentic Test Profile corpus case is not trusted because it names a plausible `sourceId`.
+Every `<name>.case.json` must retain the unchanged case-v1 format and be paired with a canonical
+`<name>.provenance.json`. Readiness additionally requires an operator-supplied trust policy and its
+expected digest.
+
+## Trust root
+
+The trust policy registers Ed25519 SPKI public keys, their subjects and AUTHOR or REVIEWER roles,
+then binds each source ID to one immutable source-identity digest and explicit authorized subjects.
+The `policyDigest` is the canonical SHA-256 digest of the policy without that field. Callers must
+pin the same value independently with `--trust-policy-digest`; a policy bundled beside a corpus is
+not its own root of trust.
+
+Key IDs are SHA-256 digests of canonical SPKI DER bytes. AssertLedger imports and re-exports every key,
+requires byte-for-byte equality with the supplied DER, and compares canonical fingerprints. This
+rejects alternate encodings such as a valid SPKI followed by ignored suffix bytes. A case requires
+different author and reviewer keys, canonical fingerprints, and subjects. Both principals sign the
+same domain-separated canonical projection, including both key IDs, the case digest, source identity
+and revision, and execution receipt. `provenanceDigest` then binds that projection and both
+signatures.
+
+`keyId` values are globally unique within a policy. `subjectId` values intentionally are not: one
+principal may rotate across multiple trusted keys. That rotation does not create reviewer
+independence. For a case replay, the selected author and reviewer must still have different key IDs,
+canonical key fingerprints, and subject IDs.
+
+Sidecars use canonical JSON exactly: `canonicalize(parsed) + "\n"`. Formatting variants and JSON
+with duplicate members are rejected before evaluation. A sidecar must be a regular, non-symlink
+file whose resolved parent is the physical split directory. Case identifiers and paired basenames
+also use AssertLedger's NFC and case-folded portable path key, so host-dependent collisions fail closed.
+
+## CLI
+
+```sh
+tsx scripts/evaluate-agentic-profile-corpus.ts status benchmarks/agentic-profile \
+  --trust-policy operator-policy.json \
+  --trust-policy-digest sha256:...
+
+tsx scripts/evaluate-agentic-profile-corpus.ts replay-provenance \
+  public/example.case.json public/example.provenance.json \
+  --trust-policy operator-policy.json \
+  --trust-policy-digest sha256:...
+```
+
+`status`, `evaluate-public`, and `evaluate-holdout` return exit 3 when not ready. Invalid policy or
+provenance returns exit 4. Valid replay or evaluation returns exit 0. AssertLedger intentionally has no
+private-key or signing command.
+
+## Migration and limits
+
+Unsigned v1 case files remain parse-compatible, but can never make a corpus READY. Producers must
+retain each case unchanged, create its paired sidecar, and distribute the policy digest through an
+independent trusted channel.
+
+The signatures prove that the named authorized principals approved the signed bytes. They do not
+prove that execution actually occurred. Until `evidenceDigests` resolve to independently replayable
+artifacts, the EXECUTED receipt remains an attestation rather than execution proof.

package/docs/agentic-test-profile-pilot.md

@@ -0,0 +1,57 @@
+# Agentic Test Profile pilot
+
+This pilot exercises the implemented profile against two real AssertLedger campaigns from
+`dofus-battlebot`. It is compatibility and local repeatability evidence, not a benchmark of broad
+fault-detection effectiveness.
+
+## Frozen inputs
+
+| Campaign | Source artifact digest | Observations | Attempts | Source decision |
+| --- | --- | ---: | ---: | --- |
+| `recover-v1` | `sha256:e696766ae1effc96203d2077cf51e9e7f731f708dad1f0c94b9c6a9876d1cca9` | 90 | 3 | `VERIFIED` |
+| `recover-v2` | `sha256:ed03d03d2892d7b945ef0af0506bb44ccb84eff73ea7f99a990335312137ba9f` | 90 | 3 | `VERIFIED` |
+
+Both campaigns contain one reference world, four required target worlds, one required neutral
+world, and four candidate tests. They use the unsandboxed trusted-local backend.
+
+## Policy
+
+Both manifests were profiled with three local lanes: `instant` at 2,000 ms, `loop` at 10,000 ms,
+and `gate` at 60,000 ms. The minimum timing sample count was three.
+
+## Results
+
+| Campaign | Profile | Qualified candidate | Target weight | Reference p95 | Pareto | Replay |
+| --- | --- | --- | ---: | ---: | --- | --- |
+| `recover-v1` | `QUALIFIED` | `boundary-complete` | 1000/1000 | 159 ms | yes | valid |
+| `recover-v2` | `QUALIFIED` | `boundary-complete` | 1000/1000 | 165 ms | yes | valid |
+
+Every lane selected the same one-candidate portfolio. The other candidates remained
+`NOT_QUALIFIED`, including a fast candidate that killed all targets but failed the reference gate.
+This is the intended non-compensation behavior: speed and mutant kills do not rescue an invalid
+oracle.
+
+## Reproduction
+
+After `pnpm build`, run the packaged example against either manifest:
+
+```sh
+node examples/agentic-profile/profile-manifest.mjs /path/to/manifest.json campaign/profile-id
+```
+
+Replay the emitted JSON with `assertledger profile-replay`.
+
+## What this pilot establishes
+
+- existing v1 evidence manifests remain accepted without migration;
+- the profile derives the same qualitative decision across two independently sealed campaigns;
+- local timing evidence is sufficient for the declared policy;
+- report replay validates source evidence, policy digest, report digest, and recomputed semantics;
+- an invalid fast candidate cannot enter the Pareto frontier or a selected portfolio.
+
+## What remains unproven
+
+The campaigns exercise the same repository behavior, candidate family, and target portfolio. They
+do not establish H1 through H4, compare against a coverage-guided baseline, estimate confidence
+intervals, or test a hidden holdout. They also do not separate cold, warm, collection, compilation,
+and execution phases. A public sweet-spot claim remains blocked on a broader versioned corpus.

package/docs/agentic-test-profile-v2.md

@@ -0,0 +1,116 @@
+# Agentic Test Profile v2
+
+Status: implemented additive public contract for `testforge-agentic-profile/2.0.0`.
+
+## Why v2 exists
+
+Profile v1 derives latency from wall times embedded in a verification campaign. Those observations
+are useful local evidence, but they do not distinguish cold startup, compilation or collection,
+warm execution, or the measurement environment. Profile v2 consumes a replay-valid Agentic
+Benchmark Artifact and uses one fixed cost basis:
+
+```text
+regime:                WARM
+measure:               WALL
+aggregation:           TOTAL
+statistic:             P95
+unit:                  MICROSECOND
+portfolio aggregation: SUM_OF_INDIVIDUAL_P95
+```
+
+No CPU time, phase timing, cold timing, candidate size, APFDc, or composite score may compensate
+for this cost basis or enter the Pareto decision.
+
+`benchmark-acquire` is the direct path to a replay-valid artifact for this profile. Profile v2 still
+evaluates only the source-selected eligible universe. A successful acquisition establishes neither
+a repository-global optimum nor a universal sweet spot, and timing failures never weaken or revise
+the source `VERIFIED` verdict.
+
+## Preconditions and status precedence
+
+The request embeds the complete benchmark artifact and names its required
+`comparisonScopeDigest`. AssertLedger first replays that artifact. Invalid benchmark evidence produces
+no report.
+
+For a valid artifact, report status follows this exact precedence:
+
+1. `COMPARISON_SCOPE_MISMATCH` when the policy and artifact scopes differ;
+2. `OBSERVED_BENCHMARK_FAILURE` when any warm measurement failed;
+3. `INSUFFICIENT_TIMING_EVIDENCE` when any warm summary lacks enough accepted samples;
+4. `QUALIFIED` when at least one declared lane admits a non-zero-strength candidate;
+5. `BUDGET_MISSED` otherwise.
+
+The first three states suppress the whole cohort's Pareto frontier and portfolios. A cold failure
+remains visible in the embedded benchmark but does not block the warm-cost profile.
+
+## Candidate universe
+
+Profile v2 deliberately evaluates only candidates that the source manifest both classified
+`ELIGIBLE` and selected. The report states this as `SOURCE_SELECTED_ELIGIBLE`, lists every included
+identifier, and separately lists eligible candidates excluded by source selection.
+
+This boundary matters: the report does not claim to find the best candidate among every submitted
+or eligible candidate. Broader comparison requires a benchmark artifact whose source selection
+contains that broader cohort.
+
+## Pareto frontier
+
+Candidate A dominates B only when A is no worse on all three dimensions and strictly better on at
+least one:
+
+- target-world weight killed: greater is better;
+- required target worlds killed: greater is better;
+- warm total wall p95: lower is better.
+
+The report does not collapse these dimensions into a single score. Tie-breaking in portfolio
+selection uses exact integer cross-products with `BigInt`, followed by marginal weight, cost,
+candidate size, digest, and identifier. Candidate size is only a deterministic tie-breaker; it is
+not a quality dimension.
+
+## Portfolio semantics
+
+For each strictly increasing lane budget, AssertLedger repeatedly selects the remaining candidate with
+the greatest new target weight per unit of warm p95 cost, recomputing marginal gain after every
+step. Zero-cost positive gain ranks first. Each step records its marginal targets, marginal weight,
+individual cost, and cumulative totals.
+
+`sumIndividualWarmTotalWallP95Us` is exactly the sum of individual candidate p95 values. It is not
+a measured portfolio p95: shared startup, caching, parallelism, contention, and test-runner
+scheduling could make an executed suite differ materially.
+
+## Replay and compatibility
+
+The report embeds its benchmark, policy, explicit cohort, candidate rows, portfolios, limitations,
+and canonical digest. Replay independently verifies:
+
+- benchmark replay;
+- benchmark digest and comparison-scope binding;
+- policy digest;
+- report digest; and
+- a complete deterministic semantic recomputation.
+
+Profile v2 is additive. Profile v1 schemas, SDK methods, CLI commands, MCP tools, and report digests
+remain unchanged. There is no automatic v1-to-v2 conversion because v1 lacks the fingerprinted,
+cold/warm, phase-aware raw evidence required by v2.
+
+## Usage
+
+```sh
+assertledger benchmark benchmark-request.json --json > benchmark-artifact.json
+assertledger benchmark-replay benchmark-artifact.json --json
+assertledger profile-v2 profile-v2-request.json --json > profile-v2-report.json
+assertledger profile-v2-replay profile-v2-report.json --json
+```
+
+The SDK methods are `profileV2()` and `replayProfileV2()`. MCP exposes the preferred
+`assertledger_profile_v2` and `assertledger_profile_v2_replay` tools alongside the legacy
+`testforge_profile_v2` and `testforge_profile_v2_replay` aliases.
+
+## Non-claims
+
+Profile v2 does not prove program correctness, world completeness, permanent stability, provenance
+authenticity beyond the externally pinned corpus policy, cross-machine portability, global subset optimality, or an empirically optimal
+coverage-versus-speed sweet spot. The last claim remains blocked until the public/private corpus
+passes its H1-H4 readiness and holdout gates on enough independently reviewed projects. Corpus
+readiness now requires canonical dual-signed sidecars; see
+[Agentic corpus provenance v1](agentic-corpus-provenance.md).