@tangle-network/agent-eval 0.205.2 → 0.207.0

package/docs/changes/jev-evidence-review.md

@@ -9,7 +9,7 @@ The optional `examples/jev-behavior-review.ts` recipe covers reward gaming, eval
 policy/authorization/data-boundary breaches, injection-following, contradicted completion,
 oversight bypass, matched-baseline anomalies and cross-run behavior. It composes the existing
 `jevAnalyst`; the taxonomy is not installed in the generic evaluator and is not a calibrated
-safety detector. See [behavioral reviews](../jev-behavior-review.md) for evidence and limits.
+safety detector. See [behavioral reviews](../systemone-behavior-review.md) for evidence and limits.
 
 No dependency, transport, scheduler, package entrypoint, default model, production action or
 spending policy changes. Existing evaluator, judge, analyst and protocol APIs remain unchanged.

package/docs/changes/systemone-provider-neutral.md

@@ -0,0 +1,36 @@
+# System One is provider-neutral
+
+The Jev surface is renamed to System One, because the same typed decision protocol is now served by
+TypeSafe (Jev), Cloudflare Workers AI (Clef, Clef-flash) and Perplexity (pplx-decider). The wire
+format does not change, so recorded requests and answers stay readable. There are no aliases: each
+consumer renames in the pull request that bumps its pin.
+
+| Before | After |
+| --- | --- |
+| `@tangle-network/agent-eval/jev`, `/jev/protocol` | `/systemone`, `/systemone/protocol` |
+| `Jev*` types (`JevRequest`, `JevResult`, `JevQuestions`, `JevRecordedRequest`, ...) | `SystemOne*` |
+| `parseJev{Request,RecordedRequest,Questions,Result}`, `jevUsage` | `parseSystemOne*`, `systemOneUsage` |
+| `JevResponseError` (message prefix `Jev:`) | `SystemOneResponseError` (prefix `System One:`) |
+| `jevEvaluator`, `jevJudge`, `jevAnalyst`, `normalizedJevScore` | `systemOneEvaluator`, `systemOneJudge`, `systemOneAnalyst`, `normalizedSystemOneScore` |
+| `prepareJevReview`, `assessJevReview`, `jevReviewFindings` | `prepareSystemOneReview`, `assessSystemOneReview`, `systemOneReviewFindings` |
+| `claimIntegrityJev{Questions,Review}`, `jevSignal`, signal id `jev`, version prefix `claim-integrity-jev-` | `claimIntegritySystemOne*`, `systemOneSignal`, id `systemone`, prefix `claim-integrity-systemone-` |
+
+Historical records keep their old signal ids and versions; a reader that filters on `jev` must also
+accept `systemone`.
+
+New in `/systemone/protocol`:
+
+- `normalizeSystemOneResult(raw, request, profile)` removes a provider's declared rounding before
+  the strict parser and gives every provider the same `confidence`. The route owner supplies the
+  profile (`{ decimals?, confidence: canonical | provider-defined }`). Answers that already pass
+  the strict checks are unchanged.
+- `systemOneConfidence(answer)` is that one definition, TypeSafe's published formulas.
+
+New in `/meta-eval`: `brierScore(pairs)`. Equal-mass ECE and the reliability table already come from
+`calibrationFromPairs(pairs, ..., { binning: equal-frequency, range: { lo: 0, hi: 1 } })`.
+
+Without normalization the strict parser refuses a recorded TypeSafe score answer whose rounded score
+is not the expectation of its rounded probabilities (8.79 against 8.86), so a transport that passes
+TypeSafe bytes straight to `parseSystemOneResult` should normalize with `{ decimals: 2, confidence:
+canonical }`. See [System One evaluations](../systemone.md#providers-rounding-and-confidence) for
+the recorded provider profiles.

package/docs/public-api.md

@@ -7,11 +7,11 @@ Generated by `pnpm api:census` on demand — this is a dated reading, not a gate
 | measure | count |
 | --- | --- |
 | export subpaths | 33 |
-| published value exports (subpath x symbol) | 1589 |
-| distinct symbols | 1401 |
-| production | 1048 |
-| planned | 278 |
-| none | 263 |
+| published value exports (subpath x symbol) | 1615 |
+| distinct symbols | 1423 |
+| production | 1053 |
+| planned | 296 |
+| none | 266 |
 
 Type-only exports are not listed: a type binds no runtime surface, and removing one cannot break a caller at run time.
 
@@ -84,7 +84,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 ### `.`
 
-360 value exports — 279 production, 40 planned, 41 none.
+371 value exports — 281 production, 44 planned, 46 none.
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
@@ -150,6 +150,8 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `checkCanaries` | production | blueprint-agent:scripts/experiments/lib/contamination-preflight.ts |
 | `checkServedModel` | production | this package: src/integrity/preflight.ts:33 |
 | `checkTraceContracts` | production | creative-agent:eval/lib/trace-contracts.ts |
+| `chiSquared` | none | only this package's tests: src/statistics/inference.test.ts:2 |
+| `chiSquaredPValue` | none | only this package's tests: src/statistics/inference.test.ts:2 |
 | `clamp01` | production | agent-dev-container:products/intelligence/api/src/lib/consultant/executor.ts |
 | `classifyFailure` | production | phony:products/builder/api/src/eval/autoresearch/trace-analyst.ts |
 | `classifyFailureReason` | production | this package: src/diagnosis/first-failure.ts:39 |
@@ -246,6 +248,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `FileSystemTraceStore` | production | agent-builder:scripts/eval.ts |
 | `fileVerdictCache` | production | blueprint-agent:scripts/experiments/calibrate-instrument.ts |
 | `FindingsStore` | production | agent-builder:scripts/run-canonical-analyst-loop.ts |
+| `fisherExactTwoSided` | none | only this package's tests: src/statistics/inference.test.ts:2 |
 | `formatScorecardDiff` | production | ai-trading-blueprint:evals/src/trading/scorecard-integration.ts |
 | `gainHistogram` | production | blueprint-agent:scripts/experiments/vb-runrecord-analysis.ts |
 | `gateTreatmentApplied` | production | blueprint-agent:scripts/experiments/lib/validity-gates.ts |
@@ -286,6 +289,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `judgeSpans` | production | this package: src/builder-eval/three-layer-eval.ts:25 |
 | `knowledgeReadinessTracePayload` | production | agent-dev-container:products/intelligence/api/src/routes/project-engines.ts |
 | `leaderboard` | production | blueprint-agent:scripts/experiments/lib/cross-harness-leaderboard.ts |
+| `linearRegressionSlope` | none | only this package's tests: src/statistics/inference.test.ts:2 |
 | `lintTraceContractSpec` | planned | doc: docs/trace-contracts.md |
 | `LlmCallError` | production | agent-dev-container:products/intelligence/api/src/lib/router-model-owner.ts |
 | `llmJudge` | production | agent-runtime:examples/agentic-data-creation/offline-fixtures.ts |
@@ -314,6 +318,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `mulberry32` | production | agent-knowledge:src/memory/holdout.ts |
 | `MultiLayerVerifier` | production | agent-builder:src/lib/.server/eval/loops/forge-refinement.ts |
 | `NoopRawProviderSink` | production | agent-dev-container:products/intelligence/api/src/lib/eval-engine.ts |
+| `normalTwoSidedPValue` | none | only this package's tests: src/statistics/inference.test.ts:2 |
 | `notBlocked` | none | only this package's tests: src/oracle.test.ts:2 |
 | `NotFoundError` | production | agent-runtime:src/errors.ts |
 | `objectiveEval` | production | agent-knowledge:src/research-loop.ts |
@@ -398,6 +403,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `runSemanticConceptJudge` | production | agent-dev-container:products/intelligence/api/src/lib/optimization-engine.ts |
 | `runsForScenario` | production | phony:products/builder/api/src/eval/gate.ts |
 | `runTaskScore` | production | discovery-lab:tools/strict-screen.mjs |
+| `sampleMedian` | planned | doc: docs/concepts.md |
 | `scoreKnowledgeReadiness` | production | agent-builder:src/lib/.server/runtime/agent-task-readiness.ts |
 | `scorePolicyEditReadiness` | none | — |
 | `scoreRedTeamOutput` | production | agent-dev-container:products/intelligence/api/src/lib/optimization-gate.ts |
@@ -409,12 +415,15 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `ServedCrossFamilyError` | none | only this package's tests: src/integrity/served-model.test.ts:2 |
 | `servedModelAcceptable` | production | this package: src/integrity/preflight.ts:33 |
 | `shareAllowed` | planned | doc: docs/redaction.md |
+| `spanRecordsToTraceAnalysisStore` | planned | doc: docs/trace-analysis.md |
 | `spearmanR` | production | blueprint-agent:scripts/experiments/lib/benchmark-validity-report.mjs |
 | `stripFencedJson` | production | blueprint-agent:scripts/experiments/compare-analysis-reports.ts |
+| `studentTTwoSidedPValue` | production | this package: src/baseline.ts:19 |
 | `subjectiveEval` | production | creative-agent:eval/control/creative-onboarding.ts |
 | `summarizeBackendIntegrity` | production | agent-dev-container:products/intelligence/api/src/lib/run-provenance.ts |
 | `summarizeNumberSeries` | production | this package: src/campaign/cell-aggregates.ts:12 |
 | `summarizePreferenceMemory` | production | agent-builder:src/lib/.server/eval/loops/auto-research-runner.ts |
+| `summarizeSample` | planned | doc: docs/concepts.md |
 | `summaryTable` | production | blueprint-agent:scripts/experiments/vb-runrecord-analysis.ts |
 | `textInSnapshot` | none | only this package's tests: src/oracle.test.ts:2 |
 | `toAgentProfileJson` | production | blueprint-agent:scripts/experiments/lib/agent-profile-cell.ts |
@@ -425,6 +434,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `traceContract` | production | creative-agent:eval/lib/trace-contracts.ts |
 | `TraceEmitter` | production | agent-builder:src/lib/.server/runtime/trace-runtime.ts |
 | `transientDispatchFailure` | planned | doc: docs/eval-surface-map.md |
+| `twoByTwoPValue` | planned | doc: docs/concepts.md |
 | `UNKNOWN_MODEL` | production | this package: src/campaign/presets/run-profile-matrix.ts:54 |
 | `urlContains` | none | only this package's tests: src/oracle.test.ts:2 |
 | `userQuestionsForKnowledgeGaps` | production | agent-dev-container:products/intelligence/api/src/routes/project-engines.ts |
@@ -441,6 +451,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `viteDeployRunner` | production | blueprint-agent:scripts/experiments/lib/deploy-runner.ts |
 | `weightedComposite` | production | agent-app:src/eval/index.ts |
 | `weightedMean` | production | blueprint-agent:scripts/experiments/lib/synthesis-quality-judge.ts |
+| `welchsTTest` | production | this package: src/contract/analyze-runs.ts:25 |
 | `WILCOXON_EXACT_MAX_N` | none | — |
 | `wilcoxonSignedRank` | production | agent-builder:src/lib/.server/eval/loops/differential-eval.ts |
 | `wilson` | production | agent-dev-container:products/intelligence/api/src/lib/project-analysis-coverage-recommendations.ts |
@@ -489,7 +500,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `AnalystRegistry` | production | blueprint-agent:scripts/experiments/vb-multi-analyst.ts |
 | `analystUsageReceiptFromPrimeUsage` | production | this package: src/analyst/benchmark-runner-prime.ts:19 |
 | `appendVerificationArtifactsToOtlp` | production | this package: src/analyst/benchmark-public-data.ts:32 |
-| `askTraceQuestions` | planned | doc: docs/jev-behavior-review.md |
+| `askTraceQuestions` | planned | doc: docs/systemone-behavior-review.md |
 | `assertProposalFindings` | production | agent-runtime:src/improvement/code-execution.ts |
 | `behavioralAnalyst` | production | agent-dev-container:products/intelligence/api/src/lib/deep-trace-analyst.ts |
 | `bindAnalyst` | planned | doc: docs/trace-analysis.md |
@@ -644,7 +655,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 ### `./campaign`
 
-166 value exports — 117 production, 22 planned, 27 none.
+169 value exports — 120 production, 22 planned, 27 none.
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
@@ -665,6 +676,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `buildTraceAnalystSurfaceDispatch` | none | only this package's tests: src/campaign/analyst-surface.test.ts:4 |
 | `campaignBreakdown` | production | this package: src/campaign/external-text-evaluation.ts:11 |
 | `campaignCellSearchResult` | production | this package: src/campaign/presets/run-optimization.ts:51 |
+| `campaignCoverage` | production | this package: src/campaign/presets/run-optimization.ts:32 |
 | `campaignMeanComposite` | production | agent-dev-container:products/intelligence/api/src/lib/optimization-engine.ts |
 | `campaignMeasurementDigest` | production | this package: src/contract/self-improve-method.ts:14 |
 | `campaignScenarioIdentity` | production | agent-runtime:src/improvement/method-identity.ts |
@@ -681,6 +693,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `costFromLedgerSummary` | production | supervisor-lab:bench/drain/seat.ts |
 | `createProfileMatrixPlan` | production | discovery-lab:tools/run-profile-confirmation.mjs |
 | `createReferenceEquivalenceJudge` | none | only this package's tests: src/reference-equivalence-judge.test.ts:13 |
+| `createRetainedDispatch` | production | this package: scripts/prove-retained-dispatch-integrity.ts:9 |
 | `createRunCostLedger` | production | agent-dev-container:products/intelligence/api/src/lib/eval-engine.ts |
 | `createSearchHistoryReceipt` | production | this package: src/campaign/search-ledger-recording.ts:27 |
 | `crowdedFrontierParent` | production | this package: scripts/search-sim.ts:146 |
@@ -713,13 +726,13 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `inMemoryCampaignStorage` | production | agent-dev-container:products/sandbox/evals/src/auto-optimization.ts |
 | `isProposedCandidate` | production | supervisor-lab:bench/sequential-arena.ts |
 | `isTransientTransportFailure` | production | gtm-agent:eval/suites/persona.ts |
-| `LabeledScenarioStoreError` | none | only this package's tests: tests/campaign/run-campaign.test.ts:6 |
+| `LabeledScenarioStoreError` | none | only this package's tests: tests/campaign/run-campaign.test.ts:7 |
 | `labelTrustRank` | production | this package: src/campaign/labeled-store/fs-adapter.ts:40 |
 | `llmJudge` | production | agent-runtime:examples/agentic-data-creation/offline-fixtures.ts |
 | `loadEvalFixture` | planned | doc: docs/eval-fixtures.md |
 | `loadEvalFixtureScenarios` | planned | example: examples/eval-fixtures-quickstart/index.ts:11 |
 | `loopProvenanceArgsFromResult` | production | this package: src/contract/self-improve.ts:36 |
-| `loopProvenanceSpans` | none | only this package's tests: tests/campaign/presets.test.ts:6 |
+| `loopProvenanceSpans` | none | only this package's tests: tests/campaign/presets.test.ts:7 |
 | `makePlaybackDispatch` | none | only this package's tests: src/campaign/presets/playback.test.ts:6 |
 | `makeProposalFinding` | production | agent-runtime:examples/improve/improve.ts |
 | `neutralizationGate` | planned | consumer tests: tax-agent:tests/eval/benchmarks/taxcalc/evolve.ts |
@@ -774,6 +787,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `searchClaimReserveUsd` | production | this package: scripts/search-meta-sim.ts:43 |
 | `searchDivergence` | planned | doc: docs/search-ledger.md |
 | `searchEdgeId` | planned | doc: docs/search-ledger.md |
+| `searchEstimateMethod` | production | this package: src/search/lenses/edit-credit.ts:34 |
 | `searchExpansionIndex` | production | this package: scripts/search-sim.ts:122 |
 | `searchHistoryCoverageRow` | production | this package: src/campaign/optimization-method.ts:14 |
 | `SearchHistoryRequiredError` | none | — |
@@ -907,9 +921,9 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
-| `asAnalyst` | production | this package: src/jev.ts:5 |
-| `asJudge` | production | this package: src/jev.ts:5 |
-| `createEvaluator` | production | this package: src/jev.ts:5 |
+| `asAnalyst` | production | this package: src/systemone.ts:5 |
+| `asJudge` | production | this package: src/systemone.ts:5 |
+| `createEvaluator` | production | this package: src/systemone.ts:5 |
 
 ### `./experiment`
 
@@ -1061,40 +1075,6 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `TraceSpanEventSchema` | planned | example: examples/hosted-ingest-server/server.ts:33 |
 | `UnixNanoTimestampSchema` | none | — |
 
-### `./jev`
-
-15 value exports — 8 production, 6 planned, 1 none.
-
-| symbol | consumer | evidence |
-| --- | --- | --- |
-| `asAnalyst` | production | this package: src/jev.ts:5 |
-| `asJudge` | production | this package: src/jev.ts:5 |
-| `assessJevReview` | planned | doc: docs/changes/jev-evidence-review.md |
-| `createEvaluator` | production | this package: src/jev.ts:5 |
-| `jevAnalyst` | planned | example: examples/jev-behavior-review.ts:3 |
-| `jevEvaluator` | planned | doc: docs/jev-behavior-review.md |
-| `jevJudge` | planned | example: examples/jev-product-integrity.ts:4 |
-| `JevResponseError` | none | — |
-| `jevReviewFindings` | planned | example: examples/jev-behavior-review.ts:3 |
-| `jevUsage` | production | this package: src/jev.ts:7 |
-| `normalizedJevScore` | planned | doc: docs/jev.md |
-| `parseJevQuestions` | production | this package: src/jev.ts:7 |
-| `parseJevRequest` | production | this package: src/jev-review.ts:9 |
-| `parseJevResult` | production | this package: src/jev-review.ts:9 |
-| `prepareJevReview` | production | this package: src/rl/claim-integrity.ts:39 |
-
-### `./jev/protocol`
-
-5 value exports — 4 production, 0 planned, 1 none.
-
-| symbol | consumer | evidence |
-| --- | --- | --- |
-| `JevResponseError` | none | — |
-| `jevUsage` | production | this package: src/jev.ts:7 |
-| `parseJevQuestions` | production | this package: src/jev.ts:7 |
-| `parseJevRequest` | production | this package: src/jev-review.ts:9 |
-| `parseJevResult` | production | this package: src/jev-review.ts:9 |
-
 ### `./ledger-core`
 
 20 value exports — 19 production, 0 planned, 1 none.
@@ -1116,7 +1096,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `readTrustedHeadFile` | production | this package: src/ledger-core/journal.ts:55 |
 | `replayLedgerText` | production | this package: src/campaign/search-ledger.ts:23 |
 | `trustedHeadPathFor` | production | this package: src/ledger-core/journal.ts:55 |
-| `tryAcquireAtomicFileLock` | production | this package: src/campaign/single-run-lock.ts:17 |
+| `tryAcquireAtomicFileLock` | production | this package: scripts/prove-retained-dispatch.ts:10 |
 | `tryWithLedgerFileLock` | production | this package: src/campaign/search-ledger-file.ts:3 |
 | `verifyEntriesAgainstTrustedHead` | production | this package: src/ledger-core/journal.ts:55 |
 | `withLedgerFileLock` | production | this package: src/analyst/benchmark-response-cache.ts:6 |
@@ -1134,21 +1114,24 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 ### `./meta-eval`
 
-28 value exports — 13 production, 14 planned, 1 none.
+32 value exports — 13 production, 18 planned, 1 none.
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
 | `auditEvaluator` | planned | example: examples/evaluation-integrity/index.ts:15 |
 | `auditProbabilityPolicy` | planned | doc: docs/changes/probability-policy-audit.md |
+| `brierScore` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `calibrateJudge` | production | creative-agent:eval/calibrate-judges.ts |
 | `calibrateJudgeContinuous` | planned | consumer tests: tax-agent:tests/eval/lib/judge-sentinel.ts |
 | `calibrationCurve` | planned | doc: docs/eval-surface-map.md |
-| `calibrationFromPairs` | planned | doc: docs/eval-surface-map.md |
+| `calibrationFromPairs` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `catchRate` | planned | doc: docs/eval-surface-map.md |
 | `continuousAgreement` | production | agent-builder:frontier/judges/eval-validity.ts |
 | `correlationStudy` | production | phony:products/builder/api/src/eval/outcomes.ts |
 | `definePlant` | planned | doc: docs/eval-surface-map.md |
 | `evalHealthStamp` | production | creative-agent:eval/lib/judge-sentinel.ts |
+| `evaluatorAdmissionPolicySchema` | planned | doc: docs/evaluator-construction.md |
+| `evaluatorAuditObservationSchema` | planned | doc: docs/evaluator-construction.md |
 | `fileSentinelStore` | production | creative-agent:eval/lib/judge-sentinel.ts |
 | `FileSystemOutcomeStore` | production | phony:products/builder/api/src/eval/outcomes.ts |
 | `InMemoryOutcomeStore` | planned | consumer tests: phony:products/builder/api/src/eval/outcomes-correlate.test.ts |
@@ -1156,6 +1139,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `judgeSentinelReport` | production | creative-agent:eval/lib/judge-sentinel.ts |
 | `OutcomeStoreError` | planned | doc: docs/outcome-validity.md |
 | `perturbEvidence` | planned | doc: docs/plants.md |
+| `planEvaluatorAudit` | planned | doc: docs/evaluator-construction.md |
 | `plantByPerturbation` | planned | doc: docs/plants.md |
 | `positionalBias` | planned | doc: docs/concepts.md |
 | `rubricPredictiveValidity` | production | physim:apps/server/src/scripts/agent-eval/validate-rubrics.ts |
@@ -1294,14 +1278,14 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `buildRlDataset` | production | this package: src/rl/corpus.ts:24 |
 | `buildVerifiedFindingRow` | none | only this package's tests: src/rl/verified-findings-dataset.test.ts:5 |
 | `campaignToRunRecords` | production | agent-builder:src/lib/.server/eval/loops/auto-research-runner.ts |
-| `checkerReadSignal` | planned | doc: docs/jev-behavior-review.md |
+| `checkerReadSignal` | planned | doc: docs/systemone-behavior-review.md |
 | `CLAIM_INTEGRITY_VERSION` | none | — |
 | `claimBrief` | none | — |
 | `claimIntegrityDigest` | none | only this package's tests: src/rl/claim-integrity.test.ts:4 |
-| `claimIntegrityJevQuestions` | none | — |
-| `claimIntegrityJevReview` | planned | doc: docs/jev-behavior-review.md |
-| `claimIntegrityTraceQuestions` | planned | doc: docs/jev-behavior-review.md |
-| `claimIntegrityVerdict` | planned | doc: docs/jev-behavior-review.md |
+| `claimIntegritySystemOneQuestions` | none | — |
+| `claimIntegritySystemOneReview` | planned | doc: docs/systemone-behavior-review.md |
+| `claimIntegrityTraceQuestions` | planned | doc: docs/systemone-behavior-review.md |
+| `claimIntegrityVerdict` | planned | doc: docs/systemone-behavior-review.md |
 | `compareAdaptationCurves` | planned | doc: docs/design/statistics-decisions.md |
 | `datasheetToMarkdown` | none | only this package's tests: src/rl/dataset.test.ts:7 |
 | `defaultBehaviorFeatures` | none | only this package's tests: src/rl/sim-fidelity.test.ts:3 |
@@ -1312,29 +1296,28 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `extractStepRewards` | none | only this package's tests: tests/rl-process-reward.test.ts:3 |
 | `extractVerifiableReward` | planned | named in gtm-agent (bind not in the import graph) |
 | `extractVerifiableRewardsFromRecords` | production | creative-agent:eval/canonical-export.ts |
-| `farmingSignal` | planned | doc: docs/jev-behavior-review.md |
+| `farmingSignal` | planned | doc: docs/systemone-behavior-review.md |
 | `FileSystemOutcomeStore` | production | phony:products/builder/api/src/eval/outcomes.ts |
 | `filterDeterministicallyRewarded` | production | this package: src/rl/reward-hacking.ts:43 |
 | `firstPassK` | planned | doc: docs/statistical-evidence.md |
 | `fitBradleyTerry` | none | only this package's tests: tests/rl-tournament.test.ts:2 |
 | `GRADER_REFERENCE_PHRASES` | none | — |
-| `graderReferenceSignal` | planned | doc: docs/jev-behavior-review.md |
+| `graderReferenceSignal` | planned | doc: docs/systemone-behavior-review.md |
 | `injectIrrelevantClause` | none | only this package's tests: tests/rl-contamination.test.ts:2 |
 | `InMemoryOutcomeStore` | planned | consumer tests: phony:products/builder/api/src/eval/outcomes-correlate.test.ts |
 | `inverseProbabilityWeighting` | planned | consumer tests: agent-knowledge:tests/memory-holdout.test.ts |
-| `jevSignal` | planned | doc: docs/jev-behavior-review.md |
 | `jsDivergence` | none | only this package's tests: src/rl/sim-fidelity.test.ts:3 |
 | `loadVerifiedFindingsDataset` | planned | doc: docs/verified-labels-flywheel.md |
-| `monitorSignal` | planned | doc: docs/jev-behavior-review.md |
+| `monitorSignal` | planned | doc: docs/systemone-behavior-review.md |
 | `offPolicyEstimateAll` | none | only this package's tests: tests/rl-off-policy.test.ts:3 |
 | `paretoFrontier` | production | agent-dev-container:products/sandbox/evals/run.ts |
-| `parseSourceScopeReferee` | planned | doc: docs/jev-behavior-review.md |
+| `parseSourceScopeReferee` | planned | doc: docs/systemone-behavior-review.md |
 | `PredictiveValidityResearcher` | planned | doc: docs/design/mlbenchmarks-book-review.md |
 | `prmTrainingPairs` | planned | doc: examples/fine-tune-with-prime-rl/README.md |
 | `quantileEdges` | none | only this package's tests: src/rl/sim-fidelity.test.ts:3 |
 | `readCorpus` | planned | named in agent-runtime (bind not in the import graph) |
 | `REFEREE_VERDICTS` | none | — |
-| `refereeSignal` | planned | doc: docs/jev-behavior-review.md |
+| `refereeSignal` | planned | doc: docs/systemone-behavior-review.md |
 | `renameVariables` | none | only this package's tests: tests/rl-contamination.test.ts:2 |
 | `ruleHolds` | none | only this package's tests: src/rl/claim-integrity.test.ts:4 |
 | `runAdaptationCurve` | planned | doc: docs/statistical-evidence.md |
@@ -1343,14 +1326,15 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `runEvalCampaign` | production | agent-builder:scripts/eval.ts |
 | `runRLCampaign` | planned | doc: docs/outcome-validity.md |
 | `runwiseStepRewardSummary` | none | only this package's tests: tests/rl-process-reward.test.ts:3 |
-| `scopeSignals` | planned | doc: docs/jev-behavior-review.md |
+| `scopeSignals` | planned | doc: docs/systemone-behavior-review.md |
 | `selfConsistency` | none | only this package's tests: tests/rl-adversarial-and-compute.test.ts:2 |
 | `selfNormalizedImportanceWeighting` | planned | consumer tests: agent-knowledge:tests/memory-holdout.test.ts |
 | `shuffleOrder` | none | — |
 | `simFidelityReport` | none | only this package's tests: src/rl/sim-fidelity.test.ts:3 |
-| `sourceScopeRefereeMessages` | planned | doc: docs/jev-behavior-review.md |
+| `sourceScopeRefereeMessages` | planned | doc: docs/systemone-behavior-review.md |
 | `stepRewardsToJsonl` | none | only this package's tests: src/rollout/gate-properties.test.ts:38 |
 | `summarizeVerifiedFindings` | none | only this package's tests: src/rl/verified-findings-dataset.test.ts:5 |
+| `systemOneSignal` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `toAnthropicFormat` | production | blueprint-agent:scripts/experiments/lib/steering-dataset.ts |
 | `toDpoJsonl` | production | this package: src/rl/dataset.ts:27 |
 | `toDpoRows` | production | this package: src/rl/dataset.ts:27 |
@@ -1369,7 +1353,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 ### `./rollout`
 
-67 value exports — 42 production, 15 planned, 10 none.
+68 value exports — 42 production, 16 planned, 10 none.
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
@@ -1401,13 +1385,14 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `gateGamedOutcome` | none | only this package's tests: src/rollout/gate-properties.test.ts:56 |
 | `HARBOR_IMPORT_GAP` | none | only this package's tests: src/rollout/interchange/harbor.test.ts:6 |
 | `isRealnessGated` | production | starter-foundry:src/lib/held-out-gate.ts |
-| `isRolloutLine` | production | this package: src/supervisor-run/rollout-nodes.ts:26 |
+| `isRolloutLine` | production | this package: src/supervisor-run/rollout-nodes.ts:27 |
 | `isTrainableSplit` | production | this package: src/rollout/release/hf-dataset.ts:24 |
 | `measureFormatGate` | production | this package: src/rollout/release/hf-dataset.ts:26 |
 | `mintRolloutRows` | production | agent-builder:src/lib/.server/eval/loops/auto-research-runner.ts |
 | `observedScore` | production | this package: src/rl/reward-hacking.ts:40 |
 | `observedSplitScore` | production | starter-foundry:src/lib/held-out-gate.ts |
 | `openOpencodeDb` | production | discovery-lab:dashboard/lib/transcript.mjs |
+| `parseClaudeTranscript` | planned | doc: docs/rollout.md |
 | `parseRolloutReleaseArgs` | none | only this package's tests: src/rollout/release/hf-dataset.test.ts:9 |
 | `planPushCommand` | none | only this package's tests: src/rollout/release/hf-dataset.test.ts:9 |
 | `readClaudeTranscript` | planned | doc: docs/rollout.md |
@@ -1430,7 +1415,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `toHarborTrajectory` | planned | doc: docs/rollout.md |
 | `toJsonl` | production | supervisor-lab:bench/vertical-rollout-package.ts |
 | `toRewardRows` | planned | doc: docs/rollout.md |
-| `toRftItem` | none | only this package's tests: src/rollout/reward-invariant.test.ts:33 |
+| `toRftItem` | none | only this package's tests: src/rollout/mint.test.ts:9 |
 | `toRftItems` | production | this package: src/rollout/release/hf-dataset.ts:22 |
 | `toSftRows` | production | this package: src/rl/dataset.ts:27 |
 | `toVerifiersRolloutOutput` | none | only this package's tests: src/rollout/exporters.test.ts:2 |
@@ -1482,7 +1467,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `summarizeSamples` | production | this package: src/search/lenses/meta-search.ts:55 |
 | `surfaceDigestEdits` | none | — |
 | `surfaceTextEdits` | production | this package: src/search-command.ts:41 |
-| `taskMatrix` | production | this package: scripts/prove-task-matrix.mts:8 |
+| `taskMatrix` | production | this package: scripts/prove-task-matrix.mts:19 |
 | `tree` | production | this package: src/search-command.ts:47 |
 | `vectorEmbedding` | planned | doc: docs/search-ledger.md |
 
@@ -1539,6 +1524,46 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `writeSupervisorRunReport` | planned | named in agent-runtime (bind not in the import graph) |
 | `writeSupervisorRunReportSafe` | planned | named in agent-runtime (bind not in the import graph) |
 
+### `./systemone`
+
+18 value exports — 8 production, 10 planned, 0 none.
+
+| symbol | consumer | evidence |
+| --- | --- | --- |
+| `asAnalyst` | production | this package: src/systemone.ts:5 |
+| `asJudge` | production | this package: src/systemone.ts:5 |
+| `assessSystemOneReview` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `createEvaluator` | production | this package: src/systemone.ts:5 |
+| `normalizedSystemOneScore` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `normalizeSystemOneResult` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `parseSystemOneQuestions` | production | this package: src/systemone.ts:14 |
+| `parseSystemOneRecordedRequest` | planned | example: examples/systemone-replay.ts:7 |
+| `parseSystemOneRequest` | production | this package: src/systemone-review.ts:11 |
+| `parseSystemOneResult` | production | this package: src/systemone-review.ts:11 |
+| `prepareSystemOneReview` | production | this package: src/rl/claim-integrity.ts:47 |
+| `systemOneAnalyst` | planned | example: examples/systemone-behavior-review.ts:3 |
+| `systemOneConfidence` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `systemOneEvaluator` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `systemOneJudge` | planned | example: examples/systemone-product-integrity.ts:4 |
+| `SystemOneResponseError` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `systemOneReviewFindings` | planned | example: examples/systemone-behavior-review.ts:3 |
+| `systemOneUsage` | production | this package: src/systemone.ts:14 |
+
+### `./systemone/protocol`
+
+8 value exports — 4 production, 4 planned, 0 none.
+
+| symbol | consumer | evidence |
+| --- | --- | --- |
+| `normalizeSystemOneResult` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `parseSystemOneQuestions` | production | this package: src/systemone.ts:14 |
+| `parseSystemOneRecordedRequest` | planned | example: examples/systemone-replay.ts:7 |
+| `parseSystemOneRequest` | production | this package: src/systemone-review.ts:11 |
+| `parseSystemOneResult` | production | this package: src/systemone-review.ts:11 |
+| `systemOneConfidence` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `SystemOneResponseError` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `systemOneUsage` | production | this package: src/systemone.ts:14 |
+
 ### `./trace-attributes`
 
 28 value exports — 26 production, 1 planned, 1 none.
@@ -1679,7 +1704,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 ### `./traces`
 
-124 value exports — 96 production, 8 planned, 20 none.
+125 value exports — 96 production, 9 planned, 20 none.
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
@@ -1702,7 +1727,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `convertTraceStoresToOtlp` | planned | consumer tests: legal-agent:tests/eval/lib/traces-to-otlp.ts |
 | `createBoundedTraceAnalysisStore` | production | braid:src/adapters/analysis/trace-store.ts |
 | `DEFAULT_TRACE_ANALYST_BUDGETS` | production | agent-builder:src/lib/.server/eval/stores/d1-trace-analysis-store-adapter.ts |
-| `defaultProviderRedactor` | production | this package: src/llm-client.ts:53 |
+| `defaultProviderRedactor` | production | this package: src/llm-client.ts:54 |
 | `defaultTraceInsightPanel` | none | only this package's tests: src/trace-analyst/insights.test.ts:3 |
 | `describeTraceInsightScope` | production | blueprint-agent:scripts/experiments/lib/analyze-vb-run/report-html.ts |
 | `detectCredential` | production | this package: scripts/redaction-corpus.ts:9 |
@@ -1763,7 +1788,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `OUTPUT_VALUE` | production | agent-runtime:src/runtime/supervise-surface.ts |
 | `planTraceInsightQuestions` | none | only this package's tests: src/trace-analyst/insights.test.ts:3 |
 | `projectOtlpFlatLine` | production | this package: src/trace-analyst/otlp-to-run-records.ts:60 |
-| `providerFromBaseUrl` | production | this package: src/llm-client.ts:53 |
+| `providerFromBaseUrl` | production | this package: src/llm-client.ts:54 |
 | `redact` | production | this package: scripts/redaction-corpus.ts:9 |
 | `redactForShare` | production | this package: scripts/redaction-corpus.ts:9 |
 | `REDACTION_PROFILES` | production | this package: scripts/redaction-corpus.ts:9 |
@@ -1779,6 +1804,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `shareAllowed` | planned | doc: docs/redaction.md |
 | `SPAN_KIND_ATTR_KEYS` | production | traces:src/run-span-tree.ts |
 | `SpanNotFoundError` | production | this package: src/trace-analyst/store-boundary.ts:1 |
+| `spanRecordsToTraceAnalysisStore` | planned | doc: docs/trace-analysis.md |
 | `stringField` | planned | named in agent-dev-container (bind not in the import graph) |
 | `throwIfRunIncomplete` | none | only this package's tests: tests/run-integrity.test.ts:3 |
 | `tokenizeDomainWords` | planned | consumer tests: blueprint-agent:scripts/experiments/__tests__/analyze-vb-run.test.ts |

package/docs/search-ledger.md

@@ -208,18 +208,24 @@ The two `--wired` behaviors above (`incumbentWithOperatorBandit`, `draftOnPlatea
 | `tree(state)` | a tidy tree of nodes and edges — the base view every other lens sits beside; a node with unknown-cost cells shows its known spend, its proven floor and how many cells are unknown, never a total | `tree.nodeCount` (drives no policy) |
 | `operatorYield(state, { split? })` | each edge operator's proposals, re-proposals, outcome counts and yield: a node's gain over the root from `searchPosterior` (`estimateNode`'s tree-wide contrast) divided by the known cost of producing and screening it — its even share of the proposal operation's cost over that operation's child edges, plus its cells allocated before its first decision (the first rung and the train feedback); cells of later rungs and the claim are the allocator's and the claim's choice, so they are not charged to the operator; a node counts once, under the operator of the edge that registered it, and a re-proposal adds a proposal but no outcome or sample; a node is excluded from yield when it was decided invalid, shares fewer than 2 units with the root, or its proposal or a screen cell has an unknown, unrecorded or zero cost | `operatorYield.weights`: an operator's yield mean once it has 6 or more yield-eligible nodes (`MIN_OUTCOMES_FOR_WEIGHT`), else `null` |
 | `front(state, { split?, axes? })` | the Pareto frontier over per-unit mean score and known cost per attempted cell on the split (reusing `paretoFrontier`), with room for caller-declared extra axes; a node's total spend depends on how far the allocator measured it, so it is shown but is not an axis; a node decided invalid, scored on fewer than 2 units, with an unknown-cost cell on the split, or with a non-finite extra axis is excluded from every frontier pass, with the reason | `front.membership`: 1 for a node on the frontier, 0 otherwise (including an excluded node) |
-| `taskMatrix(state, { split? })` | nodes and units, each single-linkage clustered on the root-mean-square difference over their shared scores, cutoff at the data's own median pairwise distance | `taskMatrix.specialistGain`: per unit cluster, the best node's mean minus the mean of node means, in the objective's direction, among nodes not decided invalid that scored every unit of the cluster; `null` with the reason, and omitted from the signal, below 2 such nodes or 6 units |
+| `taskMatrix(state, { split? })` | at most 200 nodes, evenly sampled in registration order when the ledger is larger, and their observed units; each axis is single-linkage clustered on the root-mean-square difference over shared scores, cutoff at the sample's own median pairwise distance; sampled results report `data.sampling` counts and the cap in `data.method` | `taskMatrix.specialistGain`: per unit cluster, the best sampled node's mean minus the mean of sampled node means, in the objective's direction, among nodes not decided invalid that scored every unit of the cluster; `null` with the reason, and omitted from the signal, below 2 such nodes or 6 units |
 
 `agent-eval search show <ledger> [--tree] [--operator-yield] [--front] [--task-matrix]` prints each lens's text form below the search summary, so an agent reading the CLI sees the same numbers Intelligence, discovery lab, VerticalBench and agent-runtime `improve()` would render from the same JSON.
 
 The task matrix evaluates each node pair and each unit pair at most twice.
-Clustering preserves the median cutoff, missing-score rules, deterministic ordering, and specialist gains.
+Its 200-node cap bounds node-distance calls to 39,800, under 1% of the 3,998,000 calls for 2,000 nodes.
+It reuses the same `evenSample` helper as `landscape`; the matrix cells, unit clusters, and specialist gains all use those selected nodes.
+Units observed only on omitted nodes are omitted too. The unit count itself is not capped.
+For a larger ledger, `data.sampling` is `{ nodesClustered: 200, nodesTotal, cap: 200 }`, `data.method` states the cap and scope, and the text CLI prints how many nodes were clustered out of the total.
+Ledgers with at most 200 nodes retain their complete output, including the original method text and absence of sampling metadata.
+Within that node set, clustering preserves the median cutoff, missing-score rules, deterministic ordering, and specialist-gain eligibility.
 Piped CLI output includes the complete JSON before the process exits.
 
-Run the proof against retained search ledgers:
+Run the self-contained proof, optionally adding retained search ledgers:
 
 ```sh
 pnpm build
+pnpm exec tsx scripts/prove-task-matrix.mts
 pnpm exec tsx scripts/prove-task-matrix.mts /path/to/search-ledger.jsonl
 ```