@tangle-network/agent-eval 0.206.0 → 0.207.0

package/docs/public-api.md

@@ -7,9 +7,9 @@ Generated by `pnpm api:census` on demand — this is a dated reading, not a gate
 | measure | count |
 | --- | --- |
 | export subpaths | 33 |
-| published value exports (subpath x symbol) | 1614 |
-| distinct symbols | 1422 |
-| production | 1052 |
+| published value exports (subpath x symbol) | 1615 |
+| distinct symbols | 1423 |
+| production | 1053 |
 | planned | 296 |
 | none | 266 |
 
@@ -655,7 +655,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 ### `./campaign`
 
-168 value exports — 119 production, 22 planned, 27 none.
+169 value exports — 120 production, 22 planned, 27 none.
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
@@ -787,6 +787,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `searchClaimReserveUsd` | production | this package: scripts/search-meta-sim.ts:43 |
 | `searchDivergence` | planned | doc: docs/search-ledger.md |
 | `searchEdgeId` | planned | doc: docs/search-ledger.md |
+| `searchEstimateMethod` | production | this package: src/search/lenses/edit-credit.ts:34 |
 | `searchExpansionIndex` | production | this package: scripts/search-sim.ts:122 |
 | `searchHistoryCoverageRow` | production | this package: src/campaign/optimization-method.ts:14 |
 | `SearchHistoryRequiredError` | none | — |
@@ -1119,11 +1120,11 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | --- | --- | --- |
 | `auditEvaluator` | planned | example: examples/evaluation-integrity/index.ts:15 |
 | `auditProbabilityPolicy` | planned | doc: docs/changes/probability-policy-audit.md |
-| `brierScore` | planned | doc: docs/systemone.md |
+| `brierScore` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `calibrateJudge` | production | creative-agent:eval/calibrate-judges.ts |
 | `calibrateJudgeContinuous` | planned | consumer tests: tax-agent:tests/eval/lib/judge-sentinel.ts |
 | `calibrationCurve` | planned | doc: docs/eval-surface-map.md |
-| `calibrationFromPairs` | planned | doc: docs/eval-surface-map.md |
+| `calibrationFromPairs` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `catchRate` | planned | doc: docs/eval-surface-map.md |
 | `continuousAgreement` | production | agent-builder:frontier/judges/eval-validity.ts |
 | `correlationStudy` | production | phony:products/builder/api/src/eval/outcomes.ts |
@@ -1333,7 +1334,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `sourceScopeRefereeMessages` | planned | doc: docs/systemone-behavior-review.md |
 | `stepRewardsToJsonl` | none | only this package's tests: src/rollout/gate-properties.test.ts:38 |
 | `summarizeVerifiedFindings` | none | only this package's tests: src/rl/verified-findings-dataset.test.ts:5 |
-| `systemOneSignal` | planned | doc: docs/systemone-behavior-review.md |
+| `systemOneSignal` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `toAnthropicFormat` | production | blueprint-agent:scripts/experiments/lib/steering-dataset.ts |
 | `toDpoJsonl` | production | this package: src/rl/dataset.ts:27 |
 | `toDpoRows` | production | this package: src/rl/dataset.ts:27 |
@@ -1466,7 +1467,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `summarizeSamples` | production | this package: src/search/lenses/meta-search.ts:55 |
 | `surfaceDigestEdits` | none | — |
 | `surfaceTextEdits` | production | this package: src/search-command.ts:41 |
-| `taskMatrix` | production | this package: scripts/prove-task-matrix.mts:8 |
+| `taskMatrix` | production | this package: scripts/prove-task-matrix.mts:19 |
 | `tree` | production | this package: src/search-command.ts:47 |
 | `vectorEmbedding` | planned | doc: docs/search-ledger.md |
 
@@ -1531,20 +1532,20 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | --- | --- | --- |
 | `asAnalyst` | production | this package: src/systemone.ts:5 |
 | `asJudge` | production | this package: src/systemone.ts:5 |
-| `assessSystemOneReview` | planned | doc: docs/systemone-behavior-review.md |
+| `assessSystemOneReview` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `createEvaluator` | production | this package: src/systemone.ts:5 |
-| `normalizedSystemOneScore` | planned | doc: docs/systemone.md |
-| `normalizeSystemOneResult` | planned | doc: docs/systemone.md |
+| `normalizedSystemOneScore` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `normalizeSystemOneResult` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `parseSystemOneQuestions` | production | this package: src/systemone.ts:14 |
 | `parseSystemOneRecordedRequest` | planned | example: examples/systemone-replay.ts:7 |
 | `parseSystemOneRequest` | production | this package: src/systemone-review.ts:11 |
 | `parseSystemOneResult` | production | this package: src/systemone-review.ts:11 |
 | `prepareSystemOneReview` | production | this package: src/rl/claim-integrity.ts:47 |
 | `systemOneAnalyst` | planned | example: examples/systemone-behavior-review.ts:3 |
-| `systemOneConfidence` | planned | doc: docs/systemone.md |
-| `systemOneEvaluator` | planned | doc: docs/systemone-behavior-review.md |
+| `systemOneConfidence` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `systemOneEvaluator` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `systemOneJudge` | planned | example: examples/systemone-product-integrity.ts:4 |
-| `SystemOneResponseError` | planned | doc: docs/systemone.md |
+| `SystemOneResponseError` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `systemOneReviewFindings` | planned | example: examples/systemone-behavior-review.ts:3 |
 | `systemOneUsage` | production | this package: src/systemone.ts:14 |
 
@@ -1554,13 +1555,13 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
-| `normalizeSystemOneResult` | planned | doc: docs/systemone.md |
+| `normalizeSystemOneResult` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `parseSystemOneQuestions` | production | this package: src/systemone.ts:14 |
 | `parseSystemOneRecordedRequest` | planned | example: examples/systemone-replay.ts:7 |
 | `parseSystemOneRequest` | production | this package: src/systemone-review.ts:11 |
 | `parseSystemOneResult` | production | this package: src/systemone-review.ts:11 |
-| `systemOneConfidence` | planned | doc: docs/systemone.md |
-| `SystemOneResponseError` | planned | doc: docs/systemone.md |
+| `systemOneConfidence` | planned | doc: docs/changes/systemone-provider-neutral.md |
+| `SystemOneResponseError` | planned | doc: docs/changes/systemone-provider-neutral.md |
 | `systemOneUsage` | production | this package: src/systemone.ts:14 |
 
 ### `./trace-attributes`

package/docs/search-ledger.md

@@ -208,18 +208,24 @@ The two `--wired` behaviors above (`incumbentWithOperatorBandit`, `draftOnPlatea
 | `tree(state)` | a tidy tree of nodes and edges — the base view every other lens sits beside; a node with unknown-cost cells shows its known spend, its proven floor and how many cells are unknown, never a total | `tree.nodeCount` (drives no policy) |
 | `operatorYield(state, { split? })` | each edge operator's proposals, re-proposals, outcome counts and yield: a node's gain over the root from `searchPosterior` (`estimateNode`'s tree-wide contrast) divided by the known cost of producing and screening it — its even share of the proposal operation's cost over that operation's child edges, plus its cells allocated before its first decision (the first rung and the train feedback); cells of later rungs and the claim are the allocator's and the claim's choice, so they are not charged to the operator; a node counts once, under the operator of the edge that registered it, and a re-proposal adds a proposal but no outcome or sample; a node is excluded from yield when it was decided invalid, shares fewer than 2 units with the root, or its proposal or a screen cell has an unknown, unrecorded or zero cost | `operatorYield.weights`: an operator's yield mean once it has 6 or more yield-eligible nodes (`MIN_OUTCOMES_FOR_WEIGHT`), else `null` |
 | `front(state, { split?, axes? })` | the Pareto frontier over per-unit mean score and known cost per attempted cell on the split (reusing `paretoFrontier`), with room for caller-declared extra axes; a node's total spend depends on how far the allocator measured it, so it is shown but is not an axis; a node decided invalid, scored on fewer than 2 units, with an unknown-cost cell on the split, or with a non-finite extra axis is excluded from every frontier pass, with the reason | `front.membership`: 1 for a node on the frontier, 0 otherwise (including an excluded node) |
-| `taskMatrix(state, { split? })` | nodes and units, each single-linkage clustered on the root-mean-square difference over their shared scores, cutoff at the data's own median pairwise distance | `taskMatrix.specialistGain`: per unit cluster, the best node's mean minus the mean of node means, in the objective's direction, among nodes not decided invalid that scored every unit of the cluster; `null` with the reason, and omitted from the signal, below 2 such nodes or 6 units |
+| `taskMatrix(state, { split? })` | at most 200 nodes, evenly sampled in registration order when the ledger is larger, and their observed units; each axis is single-linkage clustered on the root-mean-square difference over shared scores, cutoff at the sample's own median pairwise distance; sampled results report `data.sampling` counts and the cap in `data.method` | `taskMatrix.specialistGain`: per unit cluster, the best sampled node's mean minus the mean of sampled node means, in the objective's direction, among nodes not decided invalid that scored every unit of the cluster; `null` with the reason, and omitted from the signal, below 2 such nodes or 6 units |
 
 `agent-eval search show <ledger> [--tree] [--operator-yield] [--front] [--task-matrix]` prints each lens's text form below the search summary, so an agent reading the CLI sees the same numbers Intelligence, discovery lab, VerticalBench and agent-runtime `improve()` would render from the same JSON.
 
 The task matrix evaluates each node pair and each unit pair at most twice.
-Clustering preserves the median cutoff, missing-score rules, deterministic ordering, and specialist gains.
+Its 200-node cap bounds node-distance calls to 39,800, under 1% of the 3,998,000 calls for 2,000 nodes.
+It reuses the same `evenSample` helper as `landscape`; the matrix cells, unit clusters, and specialist gains all use those selected nodes.
+Units observed only on omitted nodes are omitted too. The unit count itself is not capped.
+For a larger ledger, `data.sampling` is `{ nodesClustered: 200, nodesTotal, cap: 200 }`, `data.method` states the cap and scope, and the text CLI prints how many nodes were clustered out of the total.
+Ledgers with at most 200 nodes retain their complete output, including the original method text and absence of sampling metadata.
+Within that node set, clustering preserves the median cutoff, missing-score rules, deterministic ordering, and specialist-gain eligibility.
 Piped CLI output includes the complete JSON before the process exits.
 
-Run the proof against retained search ledgers:
+Run the self-contained proof, optionally adding retained search ledgers:
 
 ```sh
 pnpm build
+pnpm exec tsx scripts/prove-task-matrix.mts
 pnpm exec tsx scripts/prove-task-matrix.mts /path/to/search-ledger.jsonl
 ```
 

package/docs/systemone.md

@@ -91,9 +91,21 @@ recorded TypeSafe score answers and both recorded Clef score answers.
 
 Probabilities are not made equivalent across models. A Clef 0.8 and a Jev 0.8 are different
 statements, so a threshold belongs to one model: change `{ model, threshold }` together and audit
-the pair with `auditProbabilityPolicy` against independent outcomes. `brierScore` and
-`calibrationFromPairs` (with `range: { lo: 0, hi: 1 }`; `binning: 'equal-frequency'` for
-equal-mass bins) in `/meta-eval` report calibration on the same pairs.
+the pair with `auditProbabilityPolicy` against independent outcomes.
+
+Score each question type with its proper scoring rule from `/meta-eval`, on `probabilities` and
+`noul`, never on `confidence`:
+
+| Question | Rule |
+| --- | --- |
+| `noul` | `brierScore` (binary) |
+| `choice` | `multiclassBrierScore` (sum form, range 0 to 2) |
+| `score` | `rankedProbabilityScore` (levels in order, range 0 to 1) |
+
+Top-label ECE comes from `calibrationFromPairs` on (top probability, top answer correct) pairs, with
+`range: { lo: 0, hi: 1 }` and `binning: 'equal-frequency'` for equal-mass bins.
+When items share a source (one task, one pull request), interval estimates resample whole sources:
+`clusterBootstrapMean` takes per-item values or per-item paired differences with their cluster id.
 
 ## Any classifier, not only System One
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.206.0",
+  "version": "0.207.0",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {