@tangle-network/agent-eval 0.215.0 → 0.216.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/dist/adapters/http.d.ts +3 -3
- package/dist/{agent-profile-BGyt2z8U.d.ts → agent-profile-CIA2Atnt.d.ts} +5 -5
- package/dist/{agent-profile-BGyt2z8U.d.ts.map → agent-profile-CIA2Atnt.d.ts.map} +1 -1
- package/dist/agreement-irr-DsTURxdr.js +186 -0
- package/dist/agreement-irr-DsTURxdr.js.map +1 -0
- package/dist/analyst/index.d.ts +487 -14
- package/dist/analyst/index.d.ts.map +1 -1
- package/dist/analyst/index.js +1545 -10
- package/dist/analyst/index.js.map +1 -1
- package/dist/{analyze-runs-BlslTkOJ.d.ts → analyze-runs-CSDAh8A7.d.ts} +4 -4
- package/dist/{analyze-runs-BlslTkOJ.d.ts.map → analyze-runs-CSDAh8A7.d.ts.map} +1 -1
- package/dist/{benchmark-command-CMmeV-_Z.js → benchmark-command-CByl9dxG.js} +18 -34
- package/dist/benchmark-command-CByl9dxG.js.map +1 -0
- package/dist/benchmarks/index.d.ts +5 -5
- package/dist/benchmarks/index.js +3 -3
- package/dist/campaign/index.d.ts +7 -7
- package/dist/campaign/index.js +5 -5
- package/dist/{campaign-Cuw3DQKF.js → campaign-BQ-Jujuj.js} +7 -7
- package/dist/{campaign-Cuw3DQKF.js.map → campaign-BQ-Jujuj.js.map} +1 -1
- package/dist/{chat-client-Bnx8NZe4.js → chat-client-DthPOYGd.js} +2 -2
- package/dist/{chat-client-Bnx8NZe4.js.map → chat-client-DthPOYGd.js.map} +1 -1
- package/dist/{chat-json-call-CZI1NoGZ.js → chat-json-call-BrKsyspg.js} +2 -2
- package/dist/{chat-json-call-CZI1NoGZ.js.map → chat-json-call-BrKsyspg.js.map} +1 -1
- package/dist/cli.js +4 -4
- package/dist/cluster-bootstrap-B3JywQQn.js +66 -0
- package/dist/cluster-bootstrap-B3JywQQn.js.map +1 -0
- package/dist/contract/index.d.ts +11 -11
- package/dist/contract/index.js +8 -8
- package/dist/{cost-ledger-DbQdN3nO.d.ts → cost-ledger-CO7UyEcQ.d.ts} +3 -8
- package/dist/{cost-ledger-DbQdN3nO.d.ts.map → cost-ledger-CO7UyEcQ.d.ts.map} +1 -1
- package/dist/{cost-ledger-DFOeLeJi.js → cost-ledger-D2FnJPBF.js} +27 -23
- package/dist/cost-ledger-D2FnJPBF.js.map +1 -0
- package/dist/{default-registry-BAiTePcB.d.ts → default-registry-BrWh62zf.d.ts} +6 -6
- package/dist/{default-registry-BAiTePcB.d.ts.map → default-registry-BrWh62zf.d.ts.map} +1 -1
- package/dist/{default-registry-BhYGMVbV.js → default-registry-Cn1LQ2Ve.js} +742 -757
- package/dist/default-registry-Cn1LQ2Ve.js.map +1 -0
- package/dist/{define-agent-eval-CZdBPUkk.d.ts → define-agent-eval-CEWkX5wZ.d.ts} +8 -8
- package/dist/{define-agent-eval-CZdBPUkk.d.ts.map → define-agent-eval-CEWkX5wZ.d.ts.map} +1 -1
- package/dist/{define-agent-eval-FGtpqnh4.js → define-agent-eval-Ch3MWX6k.js} +3 -3
- package/dist/{define-agent-eval-FGtpqnh4.js.map → define-agent-eval-Ch3MWX6k.js.map} +1 -1
- package/dist/diagnosis/index.d.ts +2 -2
- package/dist/{dspy-rlm-engine-OMBXVN0u.js → dspy-rlm-engine-D-KX_e64.js} +3 -3
- package/dist/{dspy-rlm-engine-OMBXVN0u.js.map → dspy-rlm-engine-D-KX_e64.js.map} +1 -1
- package/dist/{engine-BzOR5Ygn.d.ts → engine-C_zNWfM8.d.ts} +5 -5
- package/dist/{engine-BzOR5Ygn.d.ts.map → engine-C_zNWfM8.d.ts.map} +1 -1
- package/dist/evaluation-Cn2dRZIz.d.ts +75 -0
- package/dist/evaluation-Cn2dRZIz.d.ts.map +1 -0
- package/dist/evaluation.d.ts +2 -74
- package/dist/evaluation.js +1 -1
- package/dist/{exact-types-DjhfZglz.d.ts → exact-types-CD0lf5Qd.d.ts} +2 -2
- package/dist/{exact-types-DjhfZglz.d.ts.map → exact-types-CD0lf5Qd.d.ts.map} +1 -1
- package/dist/experiment/index.d.ts +1 -1
- package/dist/{external-optimizer-process-DCZx5PdG.js → external-optimizer-process-nnsY5D_4.js} +2 -2
- package/dist/{external-optimizer-process-DCZx5PdG.js.map → external-optimizer-process-nnsY5D_4.js.map} +1 -1
- package/dist/{external-optimizer-subprocess-CA3aXT8I.js → external-optimizer-subprocess-DcerlZSG.js} +263 -4
- package/dist/external-optimizer-subprocess-DcerlZSG.js.map +1 -0
- package/dist/{feedback-trajectory-CX01700m.d.ts → feedback-trajectory-DkaDj0dj.d.ts} +3 -3
- package/dist/{feedback-trajectory-CX01700m.d.ts.map → feedback-trajectory-DkaDj0dj.d.ts.map} +1 -1
- package/dist/fuzz.d.ts +1 -1
- package/dist/fuzz.js +1 -1
- package/dist/{index-D0_l25We.d.ts → index-CPJJr98o.d.ts} +2 -2
- package/dist/{index-D0_l25We.d.ts.map → index-CPJJr98o.d.ts.map} +1 -1
- package/dist/{index-DNgf5gyG.d.ts → index-Ch9VxCpK.d.ts} +2 -2
- package/dist/{index-DNgf5gyG.d.ts.map → index-Ch9VxCpK.d.ts.map} +1 -1
- package/dist/{index-BTwxP7L0.d.ts → index-DlH0Y9gI.d.ts} +10 -10
- package/dist/{index-BTwxP7L0.d.ts.map → index-DlH0Y9gI.d.ts.map} +1 -1
- package/dist/{index-D_rfXep9.d.ts → index-NvGBiO8H.d.ts} +10 -10
- package/dist/{index-D_rfXep9.d.ts.map → index-NvGBiO8H.d.ts.map} +1 -1
- package/dist/{index-BWiAAWrk.d.ts → index-oQ4aHziK.d.ts} +9 -9
- package/dist/{index-BWiAAWrk.d.ts.map → index-oQ4aHziK.d.ts.map} +1 -1
- package/dist/index.d.ts +25 -25
- package/dist/index.js +16 -15
- package/dist/index.js.map +1 -1
- package/dist/{integrity-zloavsM_.d.ts → integrity-CCAxGVK6.d.ts} +2 -2
- package/dist/{integrity-zloavsM_.d.ts.map → integrity-CCAxGVK6.d.ts.map} +1 -1
- package/dist/{judge-gate-B1_O0QIs.d.ts → judge-gate-DNcUrRFA.d.ts} +2 -2
- package/dist/{judge-gate-B1_O0QIs.d.ts.map → judge-gate-DNcUrRFA.d.ts.map} +1 -1
- package/dist/{kind-factory-CMlOx08-.js → kind-factory-CbZ-6tvu.js} +3 -3
- package/dist/{kind-factory-CMlOx08-.js.map → kind-factory-CbZ-6tvu.js.map} +1 -1
- package/dist/{llm-client-C_o62yG1.js → llm-client-DQveoxSM.js} +2 -2
- package/dist/{llm-client-C_o62yG1.js.map → llm-client-DQveoxSM.js.map} +1 -1
- package/dist/{llm-judge-Dw0rDpve.js → llm-judge-PXV7Xljl.js} +5 -5
- package/dist/{llm-judge-Dw0rDpve.js.map → llm-judge-PXV7Xljl.js.map} +1 -1
- package/dist/matrix/index.d.ts +2 -2
- package/dist/{matrix-CQwid3kj.d.ts → matrix-CHMiebFx.d.ts} +4 -4
- package/dist/{matrix-CQwid3kj.d.ts.map → matrix-CHMiebFx.d.ts.map} +1 -1
- package/dist/meta-eval/index.d.ts +2 -2
- package/dist/meta-eval/index.js +1 -62
- package/dist/meta-eval/index.js.map +1 -1
- package/dist/{mint-CDy-brUR.d.ts → mint-DCGoBYhB.d.ts} +2 -2
- package/dist/{mint-CDy-brUR.d.ts.map → mint-DCGoBYhB.d.ts.map} +1 -1
- package/dist/multishot/golden/index.d.ts +1 -1
- package/dist/multishot/index.d.ts +2 -2
- package/dist/openapi.json +1 -1
- package/dist/pipelines/index.js +1 -1
- package/dist/{prime-protocol-B7et3tI7.d.ts → prime-protocol-CEE4BR9N.d.ts} +3 -3
- package/dist/{prime-protocol-B7et3tI7.d.ts.map → prime-protocol-CEE4BR9N.d.ts.map} +1 -1
- package/dist/{produced-state-C1cTYK6w.js → produced-state-ZCtDBxAE.js} +4 -4
- package/dist/{produced-state-C1cTYK6w.js.map → produced-state-ZCtDBxAE.js.map} +1 -1
- package/dist/{proposal-findings-CXblsskH.js → proposal-findings-BZiMTzAF.js} +2 -36
- package/dist/proposal-findings-BZiMTzAF.js.map +1 -0
- package/dist/{registry-BL7H33tj.d.ts → registry-JYfPxcVz.d.ts} +11 -13
- package/dist/registry-JYfPxcVz.d.ts.map +1 -0
- package/dist/{release-confidence-C4kI7lQi.d.ts → release-confidence-BK2QCB5M.d.ts} +3 -3
- package/dist/{release-confidence-C4kI7lQi.d.ts.map → release-confidence-BK2QCB5M.d.ts.map} +1 -1
- package/dist/{report-command-CtIIsQKP.js → report-command-y8H6RSX8.js} +23 -7
- package/dist/report-command-y8H6RSX8.js.map +1 -0
- package/dist/reporting.d.ts +3 -3
- package/dist/{researcher-DuIJ9IaA.d.ts → researcher-f6mZhh58.d.ts} +5 -5
- package/dist/{researcher-DuIJ9IaA.d.ts.map → researcher-f6mZhh58.d.ts.map} +1 -1
- package/dist/{reward-hacking-C6Kp54Av.d.ts → reward-hacking-DHA7c1l8.d.ts} +2 -2
- package/dist/{reward-hacking-C6Kp54Av.d.ts.map → reward-hacking-DHA7c1l8.d.ts.map} +1 -1
- package/dist/rl.d.ts +8 -8
- package/dist/rl.js +1 -1
- package/dist/rollout/index.d.ts +2 -2
- package/dist/{rubric-predictive-validity-BiNNEjpG.d.ts → rubric-predictive-validity-2SgHxoa5.d.ts} +2 -2
- package/dist/{rubric-predictive-validity-BiNNEjpG.d.ts.map → rubric-predictive-validity-2SgHxoa5.d.ts.map} +1 -1
- package/dist/{run-record-p2AYTrCn.d.ts → run-record-D1Hw-Jml.d.ts} +2 -2
- package/dist/{run-record-p2AYTrCn.d.ts.map → run-record-D1Hw-Jml.d.ts.map} +1 -1
- package/dist/search/index.d.ts +2 -2
- package/dist/{search-kernel-BWp7hfJn.d.ts → search-kernel-CpcKH5Ih.d.ts} +4 -4
- package/dist/{search-kernel-BWp7hfJn.d.ts.map → search-kernel-CpcKH5Ih.d.ts.map} +1 -1
- package/dist/{semantic-concept-judge-sDBNg_mI.js → semantic-concept-judge-zQBG9hHU.js} +6 -6
- package/dist/{semantic-concept-judge-sDBNg_mI.js.map → semantic-concept-judge-zQBG9hHU.js.map} +1 -1
- package/dist/{server-BRpadnVt.js → server-DbcC7xMl.js} +3 -3
- package/dist/{server-BRpadnVt.js.map → server-DbcC7xMl.js.map} +1 -1
- package/dist/{skillopt-optimization-method-CvaFGm95.js → skillopt-optimization-method-CCtzralq.js} +5 -5
- package/dist/{skillopt-optimization-method-CvaFGm95.js.map → skillopt-optimization-method-CCtzralq.js.map} +1 -1
- package/dist/{statistical-heldout-DZaTo3gZ.d.ts → statistical-heldout-BeR4bOdg.d.ts} +3 -3
- package/dist/{statistical-heldout-DZaTo3gZ.d.ts.map → statistical-heldout-BeR4bOdg.d.ts.map} +1 -1
- package/dist/{store-otlp-0AiS29kg.js → store-otlp-DtB4Slsr.js} +2 -2
- package/dist/{store-otlp-0AiS29kg.js.map → store-otlp-DtB4Slsr.js.map} +1 -1
- package/dist/{store-tool-spans-DuvZ4YWb.d.ts → store-tool-spans-DR1SrCLK.d.ts} +7 -7
- package/dist/{store-tool-spans-DuvZ4YWb.d.ts.map → store-tool-spans-DR1SrCLK.d.ts.map} +1 -1
- package/dist/{store-tool-spans-BAcBBSM8.js → store-tool-spans-fZIigrpP.js} +3 -3
- package/dist/{store-tool-spans-BAcBBSM8.js.map → store-tool-spans-fZIigrpP.js.map} +1 -1
- package/dist/{summary-report-CK0ueoJd.d.ts → summary-report-C-CtItCN.d.ts} +2 -2
- package/dist/{summary-report-CK0ueoJd.d.ts.map → summary-report-C-CtItCN.d.ts.map} +1 -1
- package/dist/supervisor-run/index.d.ts +1 -1
- package/dist/supervisor-run/index.d.ts.map +1 -1
- package/dist/supervisor-run/index.js +1 -1
- package/dist/systemone-CpTLAlqb.d.ts +135 -0
- package/dist/systemone-CpTLAlqb.d.ts.map +1 -0
- package/dist/{systemone-review-DTClwgmE.js → systemone-review-BZWfrtq1.js} +2 -2
- package/dist/{systemone-review-DTClwgmE.js.map → systemone-review-BZWfrtq1.js.map} +1 -1
- package/dist/{systemone-review-BReh3mvm.d.ts → systemone-review-B_yo1xsd.d.ts} +2 -2
- package/dist/{systemone-review-BReh3mvm.d.ts.map → systemone-review-B_yo1xsd.d.ts.map} +1 -1
- package/dist/systemone.d.ts +4 -134
- package/dist/systemone.js +1 -1
- package/dist/terminal-record-BHocwFw1.js.map +1 -1
- package/dist/{tool-groups-DOriCbkR.d.ts → tool-groups-mcxOXVlb.d.ts} +3 -3
- package/dist/tool-groups-mcxOXVlb.d.ts.map +1 -0
- package/dist/{tool-waste-BYt5rdt8.js → tool-waste-D52PqU9Q.js} +3 -156
- package/dist/tool-waste-D52PqU9Q.js.map +1 -0
- package/dist/{trace-questions-RCaYLkwH.d.ts → trace-questions-DG7nqLQ7.d.ts} +4 -4
- package/dist/{trace-questions-RCaYLkwH.d.ts.map → trace-questions-DG7nqLQ7.d.ts.map} +1 -1
- package/dist/trace-repair/index.d.ts +3 -3
- package/dist/traces.d.ts +6 -6
- package/dist/traces.js +3 -3
- package/dist/{types-CMnPcbW5.d.ts → types-B4C5nFw6.d.ts} +6 -6
- package/dist/{types-CMnPcbW5.d.ts.map → types-B4C5nFw6.d.ts.map} +1 -1
- package/dist/{types-DjOMwGY_.d.ts → types-BqkVmHfv.d.ts} +7 -1
- package/dist/types-BqkVmHfv.d.ts.map +1 -0
- package/dist/{types-Dj0Qv2qi.d.ts → types-CfZMIzB_.d.ts} +3 -3
- package/dist/{types-Dj0Qv2qi.d.ts.map → types-CfZMIzB_.d.ts.map} +1 -1
- package/dist/{types-Lub7o6bN.d.ts → types-DBZ7keha.d.ts} +2 -2
- package/dist/{types-Lub7o6bN.d.ts.map → types-DBZ7keha.d.ts.map} +1 -1
- package/dist/{types-CW5b90JH.js → types-IXD2HJzu.js} +16 -3
- package/dist/types-IXD2HJzu.js.map +1 -0
- package/dist/{types-KEkKkETv.d.ts → types-oKRhQAde.d.ts} +347 -6
- package/dist/types-oKRhQAde.d.ts.map +1 -0
- package/dist/wire/index.d.ts +3 -3
- package/dist/wire/index.js +1 -1
- package/docs/campaign-proposers.md +3 -0
- package/docs/concepts.md +1 -0
- package/docs/finding-layers.md +69 -0
- package/package.json +1 -1
- package/dist/benchmark-command-CMmeV-_Z.js.map +0 -1
- package/dist/cost-ledger-DFOeLeJi.js.map +0 -1
- package/dist/default-registry-BhYGMVbV.js.map +0 -1
- package/dist/evaluation.d.ts.map +0 -1
- package/dist/external-optimizer-subprocess-CA3aXT8I.js.map +0 -1
- package/dist/proposal-findings-CXblsskH.js.map +0 -1
- package/dist/registry-BL7H33tj.d.ts.map +0 -1
- package/dist/report-command-CtIIsQKP.js.map +0 -1
- package/dist/systemone.d.ts.map +0 -1
- package/dist/tool-groups-DOriCbkR.d.ts.map +0 -1
- package/dist/tool-waste-BYt5rdt8.js.map +0 -1
- package/dist/types-CW5b90JH.js.map +0 -1
- package/dist/types-DjOMwGY_.d.ts.map +0 -1
- package/dist/types-KEkKkETv.d.ts.map +0 -1
|
@@ -1,6 +1,7 @@
|
|
|
1
|
-
import { c as CostLedgerHandle, p as CostProvenance } from "./cost-ledger-
|
|
2
|
-
import { a as RunRecord, d as RunTokenUsage } from "./run-record-
|
|
3
|
-
import { c as JudgeInput, x as ChatClient } from "./types-
|
|
1
|
+
import { c as CostLedgerHandle, p as CostProvenance } from "./cost-ledger-CO7UyEcQ.js";
|
|
2
|
+
import { a as RunRecord, d as RunTokenUsage } from "./run-record-D1Hw-Jml.js";
|
|
3
|
+
import { c as JudgeInput, x as ChatClient } from "./types-DBZ7keha.js";
|
|
4
|
+
import { z } from "zod";
|
|
4
5
|
import { SpanKind } from "@tangle-network/agent-trace-contract";
|
|
5
6
|
import { RE2JS } from "re2js";
|
|
6
7
|
//#region src/trace-analyst/types.d.ts
|
|
@@ -289,13 +290,326 @@ interface BoundedTraceAnalysisStoreOptions {
|
|
|
289
290
|
/** Apply the public validation, cancellation, not-found, and size rules to any adapter. */
|
|
290
291
|
declare function createBoundedTraceAnalysisStore(source: TraceAnalysisStore, options?: BoundedTraceAnalysisStoreOptions): TraceAnalysisStore;
|
|
291
292
|
//#endregion
|
|
293
|
+
//#region src/analyst/finding-scope.d.ts
|
|
294
|
+
/**
|
|
295
|
+
* The layer ladder, lowest first. `problem` is what discovery-lab calls a
|
|
296
|
+
* theatre. `meta` sits outside the ladder: it is about the instrument, not the
|
|
297
|
+
* agents, so it never promotes and never bounds another layer.
|
|
298
|
+
*/
|
|
299
|
+
declare const FINDING_LAYERS: readonly ['event', 'turn', 'agent', 'team', 'run', 'play', 'problem', 'domain', 'lab', 'meta'];
|
|
300
|
+
type FindingLayer = (typeof FINDING_LAYERS)[number];
|
|
301
|
+
/** Rank on the ladder; `meta` has none. */
|
|
302
|
+
declare function findingLayerRank(layer: FindingLayer): number | null;
|
|
303
|
+
interface FindingScope {
|
|
304
|
+
/**
|
|
305
|
+
* Ids of the measured units. A population is stored resolved, never as a
|
|
306
|
+
* query: the ids are what `n`, `evidence_units` and promotion's lineage
|
|
307
|
+
* check are compared against, and a query re-run later can select a
|
|
308
|
+
* different set. Select populations with `queryFindings` (layer, subject,
|
|
309
|
+
* domain path, analyst) and store the ids it returned.
|
|
310
|
+
*/
|
|
311
|
+
population: string[];
|
|
312
|
+
/** Units measured. Below 2, readers label the finding an anecdote. */
|
|
313
|
+
n: number;
|
|
314
|
+
/** ISO-8601 bounds of the measured activity. */
|
|
315
|
+
window: {
|
|
316
|
+
from: string;
|
|
317
|
+
to: string;
|
|
318
|
+
};
|
|
319
|
+
/** Domain taxonomy path of any depth, broadest first. */
|
|
320
|
+
domain_path?: string[];
|
|
321
|
+
}
|
|
322
|
+
interface FindingCalibration {
|
|
323
|
+
/** Gold items the judge was scored against. */
|
|
324
|
+
gold_n: number;
|
|
325
|
+
/** Share of gold items where the judge matched the gold label. */
|
|
326
|
+
agreement: number;
|
|
327
|
+
/** Chance-corrected agreement (Cohen's κ). */
|
|
328
|
+
kappa: number;
|
|
329
|
+
}
|
|
330
|
+
interface FindingMethod {
|
|
331
|
+
kind: 'computed' | 'measured' | 'judged';
|
|
332
|
+
name: string;
|
|
333
|
+
version: string;
|
|
334
|
+
calibration?: FindingCalibration;
|
|
335
|
+
}
|
|
336
|
+
interface FindingMeasure {
|
|
337
|
+
name: string;
|
|
338
|
+
value: number;
|
|
339
|
+
unit: string;
|
|
340
|
+
interval?: [number, number];
|
|
341
|
+
/** Confidence level of `interval`, in (0, 1). */
|
|
342
|
+
level?: number;
|
|
343
|
+
}
|
|
344
|
+
/**
|
|
345
|
+
* One measured unit. `derived_from` is the unit this one inherited state from
|
|
346
|
+
* (run 2 built on run 1's code): `null` records an independent start, and an
|
|
347
|
+
* absent field records lineage nobody captured. Unknown lineage counts as
|
|
348
|
+
* shared, so it can never manufacture independent support for a claim.
|
|
349
|
+
*/
|
|
350
|
+
interface FindingEvidenceUnit {
|
|
351
|
+
id: string;
|
|
352
|
+
layer: FindingLayer;
|
|
353
|
+
derived_from?: string | null;
|
|
354
|
+
}
|
|
355
|
+
/** The one validator for stored findings of either schema version. Unknown fields fail. */
|
|
356
|
+
declare const AnalystFindingSchema: z.ZodObject<{
|
|
357
|
+
schema_version: z.ZodEnum<{
|
|
358
|
+
"1.0.0": "1.0.0";
|
|
359
|
+
"1.1.0": "1.1.0";
|
|
360
|
+
}>;
|
|
361
|
+
finding_id: z.ZodString;
|
|
362
|
+
analyst_id: z.ZodString;
|
|
363
|
+
produced_at: z.ZodString;
|
|
364
|
+
severity: z.ZodEnum<{
|
|
365
|
+
critical: "critical";
|
|
366
|
+
high: "high";
|
|
367
|
+
info: "info";
|
|
368
|
+
low: "low";
|
|
369
|
+
medium: "medium";
|
|
370
|
+
}>;
|
|
371
|
+
area: z.ZodString;
|
|
372
|
+
claim: z.ZodString;
|
|
373
|
+
rationale: z.ZodOptional<z.ZodString>;
|
|
374
|
+
evidence_refs: z.ZodArray<z.ZodObject<{
|
|
375
|
+
kind: z.ZodEnum<{
|
|
376
|
+
artifact: "artifact";
|
|
377
|
+
event: "event";
|
|
378
|
+
finding: "finding";
|
|
379
|
+
metric: "metric";
|
|
380
|
+
span: "span";
|
|
381
|
+
}>;
|
|
382
|
+
uri: z.ZodString;
|
|
383
|
+
excerpt: z.ZodOptional<z.ZodString>;
|
|
384
|
+
}, z.core.$strict>>;
|
|
385
|
+
recommended_action: z.ZodOptional<z.ZodString>;
|
|
386
|
+
validation_plan: z.ZodOptional<z.ZodString>;
|
|
387
|
+
confidence: z.ZodNumber;
|
|
388
|
+
subject: z.ZodOptional<z.ZodString>;
|
|
389
|
+
derived_from_judge: z.ZodOptional<z.ZodBoolean>;
|
|
390
|
+
metadata: z.ZodOptional<z.ZodRecord<z.ZodString, z.ZodUnknown>>;
|
|
391
|
+
layer: z.ZodOptional<z.ZodEnum<{
|
|
392
|
+
agent: "agent";
|
|
393
|
+
domain: "domain";
|
|
394
|
+
event: "event";
|
|
395
|
+
lab: "lab";
|
|
396
|
+
meta: "meta";
|
|
397
|
+
play: "play";
|
|
398
|
+
problem: "problem";
|
|
399
|
+
run: "run";
|
|
400
|
+
team: "team";
|
|
401
|
+
turn: "turn";
|
|
402
|
+
}>>;
|
|
403
|
+
scope: z.ZodOptional<z.ZodObject<{
|
|
404
|
+
population: z.ZodArray<z.ZodString>;
|
|
405
|
+
n: z.ZodNumber;
|
|
406
|
+
window: z.ZodObject<{
|
|
407
|
+
from: z.ZodString;
|
|
408
|
+
to: z.ZodString;
|
|
409
|
+
}, z.core.$strict>;
|
|
410
|
+
domain_path: z.ZodOptional<z.ZodArray<z.ZodString>>;
|
|
411
|
+
}, z.core.$strict>>;
|
|
412
|
+
method: z.ZodOptional<z.ZodObject<{
|
|
413
|
+
kind: z.ZodEnum<{
|
|
414
|
+
computed: "computed";
|
|
415
|
+
judged: "judged";
|
|
416
|
+
measured: "measured";
|
|
417
|
+
}>;
|
|
418
|
+
name: z.ZodString;
|
|
419
|
+
version: z.ZodString;
|
|
420
|
+
calibration: z.ZodOptional<z.ZodObject<{
|
|
421
|
+
gold_n: z.ZodNumber;
|
|
422
|
+
agreement: z.ZodNumber;
|
|
423
|
+
kappa: z.ZodNumber;
|
|
424
|
+
}, z.core.$strict>>;
|
|
425
|
+
}, z.core.$strict>>;
|
|
426
|
+
measure: z.ZodOptional<z.ZodObject<{
|
|
427
|
+
name: z.ZodString;
|
|
428
|
+
value: z.ZodNumber;
|
|
429
|
+
unit: z.ZodString;
|
|
430
|
+
interval: z.ZodOptional<z.ZodTuple<[z.ZodNumber, z.ZodNumber], null>>;
|
|
431
|
+
level: z.ZodOptional<z.ZodNumber>;
|
|
432
|
+
}, z.core.$strict>>;
|
|
433
|
+
confounders: z.ZodOptional<z.ZodArray<z.ZodString>>;
|
|
434
|
+
valid_for: z.ZodOptional<z.ZodString>;
|
|
435
|
+
claim_id: z.ZodOptional<z.ZodString>;
|
|
436
|
+
evidence_units: z.ZodOptional<z.ZodArray<z.ZodObject<{
|
|
437
|
+
id: z.ZodString;
|
|
438
|
+
layer: z.ZodEnum<{
|
|
439
|
+
agent: "agent";
|
|
440
|
+
domain: "domain";
|
|
441
|
+
event: "event";
|
|
442
|
+
lab: "lab";
|
|
443
|
+
meta: "meta";
|
|
444
|
+
play: "play";
|
|
445
|
+
problem: "problem";
|
|
446
|
+
run: "run";
|
|
447
|
+
team: "team";
|
|
448
|
+
turn: "turn";
|
|
449
|
+
}>;
|
|
450
|
+
derived_from: z.ZodOptional<z.ZodNullable<z.ZodString>>;
|
|
451
|
+
}, z.core.$strict>>>;
|
|
452
|
+
}, z.core.$strict>;
|
|
453
|
+
/** A finding admitted as candidate-generation input; same rules plus its origin. */
|
|
454
|
+
declare const ProposalFindingSchema: z.ZodObject<{
|
|
455
|
+
schema_version: z.ZodEnum<{
|
|
456
|
+
"1.0.0": "1.0.0";
|
|
457
|
+
"1.1.0": "1.1.0";
|
|
458
|
+
}>;
|
|
459
|
+
finding_id: z.ZodString;
|
|
460
|
+
analyst_id: z.ZodString;
|
|
461
|
+
produced_at: z.ZodString;
|
|
462
|
+
severity: z.ZodEnum<{
|
|
463
|
+
critical: "critical";
|
|
464
|
+
high: "high";
|
|
465
|
+
info: "info";
|
|
466
|
+
low: "low";
|
|
467
|
+
medium: "medium";
|
|
468
|
+
}>;
|
|
469
|
+
area: z.ZodString;
|
|
470
|
+
claim: z.ZodString;
|
|
471
|
+
rationale: z.ZodOptional<z.ZodString>;
|
|
472
|
+
evidence_refs: z.ZodArray<z.ZodObject<{
|
|
473
|
+
kind: z.ZodEnum<{
|
|
474
|
+
artifact: "artifact";
|
|
475
|
+
event: "event";
|
|
476
|
+
finding: "finding";
|
|
477
|
+
metric: "metric";
|
|
478
|
+
span: "span";
|
|
479
|
+
}>;
|
|
480
|
+
uri: z.ZodString;
|
|
481
|
+
excerpt: z.ZodOptional<z.ZodString>;
|
|
482
|
+
}, z.core.$strict>>;
|
|
483
|
+
recommended_action: z.ZodOptional<z.ZodString>;
|
|
484
|
+
validation_plan: z.ZodOptional<z.ZodString>;
|
|
485
|
+
confidence: z.ZodNumber;
|
|
486
|
+
subject: z.ZodOptional<z.ZodString>;
|
|
487
|
+
derived_from_judge: z.ZodOptional<z.ZodBoolean>;
|
|
488
|
+
metadata: z.ZodOptional<z.ZodRecord<z.ZodString, z.ZodUnknown>>;
|
|
489
|
+
layer: z.ZodOptional<z.ZodEnum<{
|
|
490
|
+
agent: "agent";
|
|
491
|
+
domain: "domain";
|
|
492
|
+
event: "event";
|
|
493
|
+
lab: "lab";
|
|
494
|
+
meta: "meta";
|
|
495
|
+
play: "play";
|
|
496
|
+
problem: "problem";
|
|
497
|
+
run: "run";
|
|
498
|
+
team: "team";
|
|
499
|
+
turn: "turn";
|
|
500
|
+
}>>;
|
|
501
|
+
scope: z.ZodOptional<z.ZodObject<{
|
|
502
|
+
population: z.ZodArray<z.ZodString>;
|
|
503
|
+
n: z.ZodNumber;
|
|
504
|
+
window: z.ZodObject<{
|
|
505
|
+
from: z.ZodString;
|
|
506
|
+
to: z.ZodString;
|
|
507
|
+
}, z.core.$strict>;
|
|
508
|
+
domain_path: z.ZodOptional<z.ZodArray<z.ZodString>>;
|
|
509
|
+
}, z.core.$strict>>;
|
|
510
|
+
method: z.ZodOptional<z.ZodObject<{
|
|
511
|
+
kind: z.ZodEnum<{
|
|
512
|
+
computed: "computed";
|
|
513
|
+
judged: "judged";
|
|
514
|
+
measured: "measured";
|
|
515
|
+
}>;
|
|
516
|
+
name: z.ZodString;
|
|
517
|
+
version: z.ZodString;
|
|
518
|
+
calibration: z.ZodOptional<z.ZodObject<{
|
|
519
|
+
gold_n: z.ZodNumber;
|
|
520
|
+
agreement: z.ZodNumber;
|
|
521
|
+
kappa: z.ZodNumber;
|
|
522
|
+
}, z.core.$strict>>;
|
|
523
|
+
}, z.core.$strict>>;
|
|
524
|
+
measure: z.ZodOptional<z.ZodObject<{
|
|
525
|
+
name: z.ZodString;
|
|
526
|
+
value: z.ZodNumber;
|
|
527
|
+
unit: z.ZodString;
|
|
528
|
+
interval: z.ZodOptional<z.ZodTuple<[z.ZodNumber, z.ZodNumber], null>>;
|
|
529
|
+
level: z.ZodOptional<z.ZodNumber>;
|
|
530
|
+
}, z.core.$strict>>;
|
|
531
|
+
confounders: z.ZodOptional<z.ZodArray<z.ZodString>>;
|
|
532
|
+
valid_for: z.ZodOptional<z.ZodString>;
|
|
533
|
+
claim_id: z.ZodOptional<z.ZodString>;
|
|
534
|
+
evidence_units: z.ZodOptional<z.ZodArray<z.ZodObject<{
|
|
535
|
+
id: z.ZodString;
|
|
536
|
+
layer: z.ZodEnum<{
|
|
537
|
+
agent: "agent";
|
|
538
|
+
domain: "domain";
|
|
539
|
+
event: "event";
|
|
540
|
+
lab: "lab";
|
|
541
|
+
meta: "meta";
|
|
542
|
+
play: "play";
|
|
543
|
+
problem: "problem";
|
|
544
|
+
run: "run";
|
|
545
|
+
team: "team";
|
|
546
|
+
turn: "turn";
|
|
547
|
+
}>;
|
|
548
|
+
derived_from: z.ZodOptional<z.ZodNullable<z.ZodString>>;
|
|
549
|
+
}, z.core.$strict>>>;
|
|
550
|
+
proposal_origin: z.ZodEnum<{
|
|
551
|
+
production: "production";
|
|
552
|
+
search: "search";
|
|
553
|
+
}>;
|
|
554
|
+
}, z.core.$strict>;
|
|
555
|
+
type FindingParseOutcome = {
|
|
556
|
+
succeeded: true;
|
|
557
|
+
value: AnalystFinding;
|
|
558
|
+
} | {
|
|
559
|
+
succeeded: false;
|
|
560
|
+
error: {
|
|
561
|
+
path: string;
|
|
562
|
+
message: string;
|
|
563
|
+
};
|
|
564
|
+
};
|
|
565
|
+
/** Validate one stored finding of schema 1.0.0 or 1.1.0. */
|
|
566
|
+
declare function parseAnalystFinding(value: unknown): FindingParseOutcome;
|
|
567
|
+
/**
|
|
568
|
+
* A finding's scope as a reader sees it. A 1.0.0 finding states neither layer
|
|
569
|
+
* nor population, so both read `unknown` — never a guessed value.
|
|
570
|
+
*/
|
|
571
|
+
interface FindingScopeReading {
|
|
572
|
+
layer: FindingLayer | 'unknown';
|
|
573
|
+
n: number | 'unknown';
|
|
574
|
+
/** Units with distinct lineage roots; unknown lineage counts as one shared root. */
|
|
575
|
+
independent_n: number | 'unknown';
|
|
576
|
+
domain_path: readonly string[] | null;
|
|
577
|
+
labels: FindingReaderLabel[];
|
|
578
|
+
}
|
|
579
|
+
/**
|
|
580
|
+
* - `unscoped`: a 1.0.0 finding with no layer or population.
|
|
581
|
+
* - `anecdote`: measured on fewer than two units.
|
|
582
|
+
* - `uncalibrated`: judged, with no calibration against gold labels.
|
|
583
|
+
* - `lineage-unknown`: at least one unit has no recorded lineage.
|
|
584
|
+
*/
|
|
585
|
+
type FindingReaderLabel = 'unscoped' | 'anecdote' | 'uncalibrated' | 'lineage-unknown';
|
|
586
|
+
declare function readFindingScope(finding: AnalystFinding): FindingScopeReading;
|
|
587
|
+
/** Root token shared by every unit whose lineage was not recorded. */
|
|
588
|
+
declare const UNKNOWN_LINEAGE = "?unknown-lineage";
|
|
589
|
+
/**
|
|
590
|
+
* Lineage root of each unit, keyed by unit id. A unit derived from another
|
|
591
|
+
* unit in the list shares that unit's root; a unit derived from an id outside
|
|
592
|
+
* the list takes that id as its root; unknown lineage maps to one shared root.
|
|
593
|
+
*/
|
|
594
|
+
declare function lineageRoots(units: ReadonlyArray<FindingEvidenceUnit>): Map<string, string>;
|
|
595
|
+
/** Units with distinct lineage roots. */
|
|
596
|
+
declare function independentUnitCount(units: ReadonlyArray<FindingEvidenceUnit>): number;
|
|
597
|
+
/** One entry per unit id, sorted; a later copy that records lineage fills an earlier unknown. */
|
|
598
|
+
declare function mergeEvidenceUnits(units: readonly FindingEvidenceUnit[]): FindingEvidenceUnit[];
|
|
599
|
+
//#endregion
|
|
292
600
|
//#region src/analyst/types.d.ts
|
|
293
601
|
/**
|
|
294
602
|
* Unified envelope every analyst emits. Schema-versioned so renderers
|
|
295
603
|
* and time-series diffs survive future field additions.
|
|
604
|
+
*
|
|
605
|
+
* 1.1.0 adds what a finding is about and what it was measured on: `layer`,
|
|
606
|
+
* `scope`, `method`, `measure`, `confounders`, `valid_for`, `claim_id` and
|
|
607
|
+
* `evidence_units`. They are all optional, and a 1.0.0 finding carries none of
|
|
608
|
+
* them; `readFindingScope` reads such a finding as layer and n unknown.
|
|
609
|
+
* `parseAnalystFinding` (finding-scope.ts) is the one validator for both.
|
|
296
610
|
*/
|
|
297
611
|
interface AnalystFinding {
|
|
298
|
-
schema_version:
|
|
612
|
+
schema_version: AnalystFindingSchemaVersion;
|
|
299
613
|
/**
|
|
300
614
|
* Stable hash over identity-defining fields (analyst_id + canonical
|
|
301
615
|
* claim + area + optional subject). Two findings from two runs that
|
|
@@ -332,7 +646,32 @@ interface AnalystFinding {
|
|
|
332
646
|
derived_from_judge?: boolean;
|
|
333
647
|
/** Analyst-private extras; renderers ignore unless they know the analyst. */
|
|
334
648
|
metadata?: Record<string, unknown>;
|
|
649
|
+
/** 1.1.0: the level the claim is about, from one event up to the whole lab. */
|
|
650
|
+
layer?: FindingLayer;
|
|
651
|
+
/** 1.1.0: the population the claim was measured on. */
|
|
652
|
+
scope?: FindingScope;
|
|
653
|
+
/** 1.1.0: how the claim was produced, and the judge's calibration when judged. */
|
|
654
|
+
method?: FindingMethod;
|
|
655
|
+
/** 1.1.0: the number behind the claim, when it has one. */
|
|
656
|
+
measure?: FindingMeasure;
|
|
657
|
+
/** 1.1.0: known differences between the compared units besides the treatment. */
|
|
658
|
+
confounders?: string[];
|
|
659
|
+
/** 1.1.0: the conditions the claim was measured under, in words. */
|
|
660
|
+
valid_for?: string;
|
|
661
|
+
/**
|
|
662
|
+
* 1.1.0: stable key for the claim across subjects and layers. `promote`
|
|
663
|
+
* groups children by it; a finding without one never lifts.
|
|
664
|
+
*/
|
|
665
|
+
claim_id?: string;
|
|
666
|
+
/**
|
|
667
|
+
* 1.1.0: the measured units with their lineage. When present, `scope.n` and
|
|
668
|
+
* `scope.population` must match them and no unit may sit above `layer`.
|
|
669
|
+
*/
|
|
670
|
+
evidence_units?: FindingEvidenceUnit[];
|
|
335
671
|
}
|
|
672
|
+
/** Every finding schema this package reads. Writers emit 1.1.0 only when a 1.1.0 field is set. */
|
|
673
|
+
declare const ANALYST_FINDING_SCHEMA_VERSIONS: readonly ['1.0.0', '1.1.0'];
|
|
674
|
+
type AnalystFindingSchemaVersion = (typeof ANALYST_FINDING_SCHEMA_VERSIONS)[number];
|
|
336
675
|
/** Finding severity. `AnalystSeverity` derives from this array, so the type
|
|
337
676
|
* and the schema that validates a finding cannot name different levels. */
|
|
338
677
|
declare const ANALYST_SEVERITIES: readonly ['critical', 'high', 'medium', 'low', 'info'];
|
|
@@ -549,6 +888,8 @@ declare function computeFindingId(input: {
|
|
|
549
888
|
/** Override the claim for hashing — use when the displayed claim has run-specific bits. */
|
|
550
889
|
id_basis?: string;
|
|
551
890
|
}): string;
|
|
891
|
+
/** Fields that exist only in schema 1.1.0. */
|
|
892
|
+
declare const FINDING_SCOPE_FIELDS: readonly ["layer", "scope", "method", "measure", "confounders", "valid_for", "claim_id", "evidence_units"];
|
|
552
893
|
/**
|
|
553
894
|
* Convenience factory: produce a fully-formed AnalystFinding with the
|
|
554
895
|
* id computed automatically. Analyst code stays terse.
|
|
@@ -629,5 +970,5 @@ type AnalystRunEvent = {
|
|
|
629
970
|
result: AnalystRunResult;
|
|
630
971
|
};
|
|
631
972
|
//#endregion
|
|
632
|
-
export {
|
|
633
|
-
//# sourceMappingURL=types-
|
|
973
|
+
export { DatasetOverview as $, FindingMeasure as A, lineageRoots as B, makeFinding as C, FindingCalibration as D, FINDING_LAYERS as E, FindingScopeReading as F, BoundedTraceAnalysisStoreOptions as G, parseAnalystFinding as H, ProposalFindingSchema as I, SpanSourceReader as J, ReadSpanSourceInput as K, UNKNOWN_LINEAGE as L, FindingParseOutcome as M, FindingReaderLabel as N, FindingEvidenceUnit as O, FindingScope as P, DEFAULT_TRACE_ANALYST_BUDGETS as Q, findingLayerRank as R, computeFindingId as S, AnalystFindingSchema as T, readFindingScope as U, mergeEvidenceUnits as V, createBoundedTraceAnalysisStore as W, TraceAnalysisStore as X, TRACE_ANALYSIS_LIMITS as Y, TraceAnalysisStoreContext as Z, ExecutionProbeOutcome as _, AnalystFinding as a, TRACE_ANALYST_TRUNCATION_MARKER_PREFIX as at, ProposalFinding as b, AnalystRequirements as c, TraceAnalystSpan as ct, AnalystRunResult as d, TraceAnalystTraceSummary as dt, ErrorCluster as et, AnalystRunSummary as f, ViewSpansResult as ft, ExecutionProbe as g, EvidenceRef as h, AnalystCost as i, SpanMatchRecord as it, FindingMethod as j, FindingLayer as k, AnalystRunEvent as l, TraceAnalystSpanKind as lt, AnalystUsageReceipt as m, ViewTraceResult as mt, Analyst as n, SearchSpanResult as nt, AnalystFindingSchemaVersion as o, TraceAnalystByteBudgets as ot, AnalystSeverity as p, ViewTraceOversized as pt, ReadSpanSourceResult as q, AnalystContext as r, SearchTraceResult as rt, AnalystInputKind as s, TraceAnalystFilters as st, ANALYST_FINDING_SCHEMA_VERSIONS as t, QueryTracesPage as tt, AnalystRunInputs as u, TraceAnalystSpanStatus as ut, ExecutionProbeRequest as v, makeProposalFinding as w, ProposalFindingOrigin as x, FINDING_SCOPE_FIELDS as y, independentUnitCount as z };
|
|
974
|
+
//# sourceMappingURL=types-oKRhQAde.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"types-oKRhQAde.d.ts","names":[],"sources":["../src/trace-analyst/types.ts","../src/trace-analyst/store-contract.ts","../src/trace-analyst/store-boundary.ts","../src/analyst/finding-scope.ts","../src/analyst/types.ts"],"mappings":";;;;;;;;;;;;KAsBY,uBAAuB;KAEvB;;;;UAKK;EACf;EACA;EACA;EACA;EACA,MAAM;EACN;EACA;EACA;EACA,QAAQ;EACR;EACA;EACA;EACA;EACA;;;EAGA,YAAY;;UAGG;EACf;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;;UAGe;;EAEf;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA;;;EAGA;;;;;;;UAQe;;EAEf;;EAEA;;EAEA;;EAEA;EACA;EACA;;EAEA;;EAEA;;EAEA;;UAGe;EACf;EACA;EACA;EACA;EACA;EACA;;EAEA;EACA;IAAU;IAAqB;;;;;;EAK/B,gBAAgB;EAChB;IAAc;IAAkB;;;UAGjB;EACf,QAAQ;EACR;EACA;;;;;UAMe;EACf;EACA,QAAQ;EACR,YAAY;;UAGG;EACf;;EAEA,gBAAgB;;EAEhB;EACA;;UAGe;EACf;EACA,OAAO;;EAEP;;EAEA;;EAEA;;EAEA;;UAGe;EACf;EACA;EACA;EACA,WAAW;;;EAGX;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA,MAAM;EACN;;UAGe;EACf;EACA;EACA,MAAM;EACN;;;UAIe;;;EAGf;;;EAGA;;;EAGA;;;EAGA;;cAGW,+BAA+B;;;cAS/B;;;cC3MA;WACX;WACA;WACA;WACA;WACA;WACA;WACA;WACA;;UAGe;EACf,SAAS;;;UAIM;EACf;EACA;EACA;EACA;EACA;EACA;;KAGU;EAEN;EACA;EACA;EACA;EACA;EACA;;EAGA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;IACE;IACA;IACA;IACA;IACA;;;;KAKI,oBACV,OAAO,qBACP,UAAU,8BACP,QAAQ;;;;;;;;UASI;;EAEf,iBAAiB;EAEjB,SAAS,kBAAkB,UAAU,4BAA4B;EAEjE,SACE;IAAS;IAAkB;KAC3B,UAAU,4BACT;EAEH,YACE,UAAU,qBACV,UAAU,4BACT,QAAQ;EAEX,YACE;IAAS,UAAU;IAAqB;IAAe;KACvD,UAAU,4BACT,QAAQ;EAEX,YAAY,UAAU,qBAAqB,UAAU,4BAA4B;EAEjF,UACE;IACE;IACA;KAEF,UAAU,4BACT,QAAQ;EAEX,UACE;IACE;IACA;IACA;KAEF,UAAU,4BACT,QAAQ;EAEX,YACE;IACE;IACA;IACA;KAEF,UAAU,4BACT,QAAQ;EAEX,WACE;IACE;IACA;IACA;IACA;KAEF,UAAU,4BACT,QAAQ;;UAGI;EACf,UAAU,QAAQ;;;;;iBC1GJ,gCACd,QAAQ,oBACR,UAAS,mCACR;;;;;;;;cCCU;KAaD,uBAAuB;;iBAGnB,iBAAiB,OAAO;UAIvB;;;;;;;;EAQf;;EAEA;;EAEA;IAAU;IAAc;;;EAExB;;UAGe;;EAEf;;EAEA;;EAEA;;UAGe;EACf;EACA;EACA;EACA,cAAc;;UAGC;EACf;EACA;EACA;EACA;;EAEA;;;;;;;;UASe;EACf;EACA,OAAO;EACP;;;cAoKW,sBAAoB,EAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAA0C,EAAA,KAAA;;cAG9D,uBAAqB,EAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAGN,EAAA,KAAA;KAQhB;EACN;EAAiB,OAAO;;EACxB;EAAkB;IAAS;IAAc;;;;iBAG/B,oBAAoB,iBAAiB;;;;;UAiBpC;EACf,OAAO;EACP;;EAEA;EACA;EACA,QAAQ;;;;;;;;KASE;iBAEI,iBAAiB,SAAS,iBAAiB;;cA2B9C;;;;;;iBAOG,aAAa,OAAO,cAAc,uBAAuB;;iBA0BzD,qBAAqB,OAAO,cAAc;;iBAK1C,mBAAmB,gBAAgB,wBAAwB;;;;;;;;;;;;;UC5V1D;EACf,gBAAgB;;;;;;;EAOhB;EACA;EACA;EACA,UAAU;;;;;;;EAOV;EACA;EACA;EACA,eAAe;EACf;EACA;;EAEA;;;;;;EAMA;;;;EAIA;;EAEA,WAAW;;EAEX,QAAQ;;EAER,QAAQ;;EAER,SAAS;;EAET,UAAU;;EAEV;;EAEA;;;;;EAKA;;;;;EAKA,iBAAiB;;;cAIN;KAED,sCAAsC;;;cAIrC;KAED,0BAA0B;;KAG1B;;KAGA,kBAAkB;WACnB,iBAAiB;;UAGX;;;;;;;EAOf;EACA;EACA;;;;;;;;KAWU;UAOK;;EAEf;;EAEA;;EAEA;;EAEA;;UAGe;;EAEf;;EAEA;;;;;;UAOe;EACf,aAAa;EACb;EACA,YAAY;EACZ,aAAa;;EAEb,SAAS;;UAGM;EACf;;EAEA;;EAEA;;EAEA;;EAEA,aAAa;;EAEb;;;;;;;EAOA,OAAO;;;;;;;;;;EAUP,gBAAgB,cAAc;;;;;;;EAO9B,mBAAmB,cAAc;;;;;EAKjC,eAAe,SAAS;;EAExB,OAAO;;EAEP,OAAO,aAAa,SAAS;;EAE7B,SAAS;;;;;;;;EAQT,QAAQ;;;UAMO;EACf;;EAEA;;EAEA;;EAEA;EACA,SAAS;;;;;;;KAQC;EAEN;EACA;EACA;EACA;EACA;;EAEA;;EAEA;EAAkB;IAAS;IAAe;;;;;;;;UAO/B;;WAEN;EACT,QAAQ,SAAS,wBAAwB,QAAQ;;;;;;;;UASlC,QAAQ;;WAEd;;WAEA;WACA,WAAW;WACX,MAAM;WACN,WAAW;;WAEX;EACT,QAAQ,OAAO,QAAQ,KAAK,iBAAiB,QAAQ;;;UAItC;;EAEf;;EAEA,QAAQ;;EAER,MAAM;;EAEN;;;;;;;;EAQA;IAAkB;IAAsB;;;;;;;;EAOxC;;;;;;;;;;iBAac,iBAAiB;EAC/B;EACA;EACA;EACA;;EAEA;;;cAsBW;;;;;iBAeG,YACd,MAAM,KAAK;EACT;EACA;IAED;;iBAmBa,oBACd,MAAM,KAAK;EACT;EACA;IAED;UAOc;EACf;EACA;;EAEA;EACA;EACA;;EAEA,OAAO;;EAEP;IAAU;IAAe;;;UAGV;EACf;EACA;EACA;EACA;EACA,UAAU;EACV,aAAa;;EAEb;;;;;EAKA,wBAAwB;;;;;;;;;;;;;;;KAkBd;EAEN;EACA;EACA;EACA;;EAEA,aAAa;;EAGb;EACA,SAAS;;EAGT;EACA;EACA;;EAGA;;EAEA,SAAS;EACT,UAAU,cAAc;;EAGxB;EACA,QAAQ"}
|
package/dist/wire/index.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
|
-
import { b as CustomTokenPricing, c as CostLedgerHandle } from "../cost-ledger-
|
|
2
|
-
import { x as ChatClient } from "../types-
|
|
1
|
+
import { b as CustomTokenPricing, c as CostLedgerHandle } from "../cost-ledger-CO7UyEcQ.js";
|
|
2
|
+
import { x as ChatClient } from "../types-DBZ7keha.js";
|
|
3
3
|
import { s as TraceStore } from "../store-C4avFjIi.js";
|
|
4
|
-
import { m as FeedbackTrajectoryStore } from "../feedback-trajectory-
|
|
4
|
+
import { m as FeedbackTrajectoryStore } from "../feedback-trajectory-DkaDj0dj.js";
|
|
5
5
|
import { z } from "zod";
|
|
6
6
|
import { ServerType } from "@hono/node-server";
|
|
7
7
|
import { Hono } from "hono";
|
package/dist/wire/index.js
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
import { A as WIRE_VERSION, C as RUBRIC_VERSION_SCHEME, D as TracesIngestRequestSchema, E as TraceEventSchema, O as TracesIngestResponseSchema, S as ListRubricsResponseSchema, T as RubricSchema, _ as FeedbackIngestResponseSchema, a as runRpcBatch, b as JudgeRequestSchema, c as WireError, d as handleListRubrics, f as handleTracesIngest, g as ErrorResponseSchema, h as listBuiltinRubrics, i as dispatchRpc, j as hashRubric, k as VersionResponseSchema, l as handleFeedbackIngest, m as getBuiltinRubric, n as startServer, o as runRpcOnce, p as handleVersion, r as startServerAsync, s as buildOpenApi, t as createApp, u as handleJudge, v as FeedbackTrajectorySchema, w as RubricDimensionSchema, x as JudgeResultSchema, y as HealthResponseSchema } from "../server-
|
|
1
|
+
import { A as WIRE_VERSION, C as RUBRIC_VERSION_SCHEME, D as TracesIngestRequestSchema, E as TraceEventSchema, O as TracesIngestResponseSchema, S as ListRubricsResponseSchema, T as RubricSchema, _ as FeedbackIngestResponseSchema, a as runRpcBatch, b as JudgeRequestSchema, c as WireError, d as handleListRubrics, f as handleTracesIngest, g as ErrorResponseSchema, h as listBuiltinRubrics, i as dispatchRpc, j as hashRubric, k as VersionResponseSchema, l as handleFeedbackIngest, m as getBuiltinRubric, n as startServer, o as runRpcOnce, p as handleVersion, r as startServerAsync, s as buildOpenApi, t as createApp, u as handleJudge, v as FeedbackTrajectorySchema, w as RubricDimensionSchema, x as JudgeResultSchema, y as HealthResponseSchema } from "../server-DbcC7xMl.js";
|
|
2
2
|
export { ErrorResponseSchema, FeedbackIngestResponseSchema, FeedbackTrajectorySchema, HealthResponseSchema, JudgeRequestSchema, JudgeResultSchema, ListRubricsResponseSchema, RUBRIC_VERSION_SCHEME, RubricDimensionSchema, RubricSchema, TraceEventSchema, TracesIngestRequestSchema, TracesIngestResponseSchema, VersionResponseSchema, WIRE_VERSION, WireError, buildOpenApi, createApp, dispatchRpc, getBuiltinRubric, handleFeedbackIngest, handleJudge, handleListRubrics, handleTracesIngest, handleVersion, hashRubric, listBuiltinRubrics, runRpcBatch, runRpcOnce, startServer, startServerAsync };
|
|
@@ -127,6 +127,9 @@ Pass `context.evaluate` to the optimizer so every candidate is executed and scor
|
|
|
127
127
|
Unknown case IDs and calls beyond `maxEvaluations` are rejected.
|
|
128
128
|
|
|
129
129
|
Optimizer-owned model or service calls must use `context.cost.runPaidCall()`.
|
|
130
|
+
Admission derives unresolved calls, reserved amounts, conservative settled spend, and maximum breaches in one record-order pass without constructing reporting summaries.
|
|
131
|
+
Unknown settled costs retain any reserved maximum in the admission total, and unresolved pending calls block new paid work.
|
|
132
|
+
|
|
130
133
|
The example below assumes the upstream package enforces `maxCostUsd` for the complete run and returns aggregate usage.
|
|
131
134
|
If the package exposes a model callback instead, wrap each model call separately with the same cost ledger and phase.
|
|
132
135
|
|
package/docs/concepts.md
CHANGED
|
@@ -418,6 +418,7 @@ release decision.
|
|
|
418
418
|
- **Need the feature map?** Read [feature-guide.md](./feature-guide.md) for integration patterns and operational limits.
|
|
419
419
|
- **Scoring a string from another language?** Read [wire-protocol.md](./wire-protocol.md) for the HTTP/RPC interface.
|
|
420
420
|
- **Diagnosing failures across a batch of runs?** Read [trace-analysis.md](./trace-analysis.md) for the recursive DSPy-RLM analyst and the public benchmark it's calibrated against.
|
|
421
|
+
- **Stating what a finding holds for?** Read [finding-layers.md](./finding-layers.md) for layers, scope, lineage, promotion and the findings ledger.
|
|
421
422
|
- **Building a driver and worker loop?** Read [control-runtime.md](./control-runtime.md) for coding, browser, computer-use, and research patterns.
|
|
422
423
|
- **Turning review feedback into reusable data?** Read [feedback-trajectories.md](./feedback-trajectories.md) for dataset, optimization, and preference-memory examples.
|
|
423
424
|
- **Building a code-generator eval?** → Start with `BuilderSession`, `SandboxHarness`, and `MultiLayerVerifier`.
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Finding layers
|
|
2
|
+
|
|
3
|
+
Every analyst finding can say what it is about and what it was measured on.
|
|
4
|
+
A claim measured on 4 runs of one play stays a play-level result until independent evidence lifts it.
|
|
5
|
+
|
|
6
|
+
Source of truth: `src/analyst/finding-scope.ts`, `domain-taxonomy.ts`, `promote.ts`, `findings-ledger.ts`, `team-shape.ts`, `failure-modes.ts` and `rollup.ts`, all exported from `@tangle-network/agent-eval/analyst`.
|
|
7
|
+
|
|
8
|
+
## The ladder
|
|
9
|
+
|
|
10
|
+
| Layer | A claim about |
|
|
11
|
+
| --- | --- |
|
|
12
|
+
| `event`, `turn` | One step of one agent |
|
|
13
|
+
| `agent` | One agent across its work |
|
|
14
|
+
| `team` | How a supervision tree was organized |
|
|
15
|
+
| `run` | One run |
|
|
16
|
+
| `play` | Runs of one play |
|
|
17
|
+
| `problem` | Plays of one problem (discovery-lab's theatre) |
|
|
18
|
+
| `domain` | Problems under one domain node, at any depth |
|
|
19
|
+
| `lab` | Everything |
|
|
20
|
+
| `meta` | The instrument itself: a judge, a ruler, an analyst |
|
|
21
|
+
|
|
22
|
+
## Schema 1.1.0
|
|
23
|
+
|
|
24
|
+
AnalystFinding 1.1.0 adds `layer`, `scope`, `method`, `measure`, `confounders`, `valid_for`, `claim_id` and `evidence_units`.
|
|
25
|
+
All are optional on the type, and `makeFinding` writes 1.1.0 only when one is set, so existing writers and stored 1.0.0 findings are unchanged.
|
|
26
|
+
`parseAnalystFinding` reads both versions with one schema:
|
|
27
|
+
|
|
28
|
+
- A 1.0.0 finding may not carry 1.1.0 fields; it reads as layer and n `unknown`.
|
|
29
|
+
- A 1.1.0 finding needs `layer`, `scope` and `method`.
|
|
30
|
+
- With `evidence_units`, `scope.n` and `scope.population` must match the units, and no unit may sit above the finding's layer.
|
|
31
|
+
|
|
32
|
+
`readFindingScope` labels a finding `anecdote` when n < 2, `uncalibrated` when it was judged without calibration, `lineage-unknown` when a unit's lineage was not recorded, and `unscoped` for 1.0.0.
|
|
33
|
+
|
|
34
|
+
`scope.population` stores resolved unit ids, never a query: `n`, `evidence_units` and promotion's lineage check compare against those ids, and a query re-run later can select a different set. Select a population with `queryFindings` and store the ids it returned.
|
|
35
|
+
|
|
36
|
+
## Lineage
|
|
37
|
+
|
|
38
|
+
An evidence unit's `derived_from` names the unit it inherited state from, `null` records an independent start, and an absent field records lineage nobody captured.
|
|
39
|
+
Units are grouped by lineage root.
|
|
40
|
+
Unknown lineage shares one root, so it can never count as independent support.
|
|
41
|
+
|
|
42
|
+
## Taxonomy and promotion
|
|
43
|
+
|
|
44
|
+
`defineDomainTaxonomy` builds one versioned tree of domain nodes; `attachToTaxonomy` hangs a problem on one primary domain path plus tags, a play on a problem, and a run on a play.
|
|
45
|
+
Everything sits under the implicit `lab` root.
|
|
46
|
+
|
|
47
|
+
`promote(findings, taxonomy, { k })` is the only way a claim moves up:
|
|
48
|
+
|
|
49
|
+
1. A finding sits at the lowest common ancestor of its population. That node's layer must equal the finding's layer, so a lab-level claim measured on one play is refused, not lifted.
|
|
50
|
+
2. Siblings with the same `claim_id` are grouped under their parent.
|
|
51
|
+
3. Only children with `measure.interval` vote, on the same measure and unit, and their intervals must share a point.
|
|
52
|
+
4. Children whose units share a lineage root count once. At least `k` independent children (default 2) are required.
|
|
53
|
+
5. A lifted finding cites its children as `finding://<finding_id>`; the rest are returned as untested above.
|
|
54
|
+
|
|
55
|
+
Event, turn, agent, team and meta findings are never placed on the ladder.
|
|
56
|
+
|
|
57
|
+
## Ledger
|
|
58
|
+
|
|
59
|
+
`openFindingsLedger` stores findings in the ledger-core journal: canonical, hash-chained, fsynced, pinned to a trusted head, and validated by the finding schema on write and read.
|
|
60
|
+
`read()` returns the rows and their Merkle root.
|
|
61
|
+
`queryFindings` filters by layer, subject, analyst, run, claim and domain-path prefix.
|
|
62
|
+
`diffFindingsOverTime` runs `diffFindings` between consecutive runs in ledger order.
|
|
63
|
+
`FindingsStore` remains the unchained JSONL store for its existing consumers.
|
|
64
|
+
|
|
65
|
+
## Analysts
|
|
66
|
+
|
|
67
|
+
- `team-shape` (computed, `team`) reads a Runtime spawn journal and coordination log: `teamShapeInputFromSources` takes them from `readRuntimeSupervisorRun`, and the same parser reads the structure-only reductions `extract.py` writes. E (command) edges come from the id path (`R:s1:s0` is a child of `R:s1`), G (governance) edges from the journal holding each `settled` row plus each delivered steer's sender, and C (communication) edges from coordination rows naming a receiver, sent by `from`/`by` or else the log owner's digest. Classes, version 1, checked in order: single, governed swarm, fleet, partition, `hive depth N`, star with sub-leads, flat star (`root integrates`, `root does not integrate` or `integration unknown`, from the caller's optional `integrations`). The 50% and 20% thresholds are provisional and carried in `method.version`. Every class check is reported as yes, no or `not observable from this record`; an owner digest is not an agent id, so lateral C and steer governance are usually not observable, and the record never reads them as absent. The four recorded Magnetar runs under `tests/fixtures/team-shape`, with the lead's gold labels, are the acceptance fixtures.
|
|
68
|
+
- `failure-modes` (judged, `agent`, `team` or `run`) asks fixed-choice questions through `systemOneAnalyst`: a MAST-adapted failure mode, delegation-brief completeness and steer handling. `createFailureModesAnalyst` refuses a rubric whose recorded calibration κ is below `FAILURE_MODES_MIN_KAPPA` (0.6). Calibration is a live step after release: `loadFailureModeGold` reads the public AgentRx label file through `loadPublicBenchmarkRows` (the file `analyst-benchmark --dataset agentrx --labels` takes) plus the caller's gold file, the judge labels the same items, and `calibrateFailureModes` scores it. Disco runs that on the Router and commits the receipt; the AgentRx file is not stored in this repository.
|
|
69
|
+
- `rollup` (computed) pools run findings per play with a lineage-clustered bootstrap, then lets `promote` decide what climbs. Each lifted finding gets the pooled interval over every run beneath it and its observation and cluster counts.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tangle-network/agent-eval",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.216.0",
|
|
4
4
|
"description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
|
|
5
5
|
"homepage": "https://github.com/tangle-network/agent-eval#readme",
|
|
6
6
|
"repository": {
|