@tangle-network/agent-eval 0.213.0 → 0.215.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/dist/adapters/http.d.ts +2 -2
- package/dist/analyst/index.d.ts +6 -6
- package/dist/analyst/index.d.ts.map +1 -1
- package/dist/analyst/index.js +3 -3
- package/dist/{benchmark-command-IMtRf9eK.js → benchmark-command-CMmeV-_Z.js} +7 -6
- package/dist/benchmark-command-CMmeV-_Z.js.map +1 -0
- package/dist/benchmarks/index.d.ts +4 -4
- package/dist/benchmarks/index.js +3 -3
- package/dist/campaign/index.d.ts +4 -4
- package/dist/campaign/index.js +8 -8
- package/dist/{campaign-CrMj5nYM.js → campaign-Cuw3DQKF.js} +9 -9
- package/dist/{campaign-CrMj5nYM.js.map → campaign-Cuw3DQKF.js.map} +1 -1
- package/dist/cli.js +6 -6
- package/dist/contract/index.d.ts +7 -7
- package/dist/contract/index.js +4 -4
- package/dist/{default-registry-Crk3QEy1.d.ts → default-registry-BAiTePcB.d.ts} +3 -3
- package/dist/{default-registry-Crk3QEy1.d.ts.map → default-registry-BAiTePcB.d.ts.map} +1 -1
- package/dist/{define-agent-eval-B_zZY2AZ.d.ts → define-agent-eval-CZdBPUkk.d.ts} +5 -5
- package/dist/{define-agent-eval-B_zZY2AZ.d.ts.map → define-agent-eval-CZdBPUkk.d.ts.map} +1 -1
- package/dist/{define-agent-eval-BmF0c78X.js → define-agent-eval-FGtpqnh4.js} +4 -4
- package/dist/{define-agent-eval-BmF0c78X.js.map → define-agent-eval-FGtpqnh4.js.map} +1 -1
- package/dist/{dspy-rlm-engine-D7BIsKyZ.js → dspy-rlm-engine-OMBXVN0u.js} +2 -2
- package/dist/{dspy-rlm-engine-D7BIsKyZ.js.map → dspy-rlm-engine-OMBXVN0u.js.map} +1 -1
- package/dist/{engine-D-YZ32ZJ.d.ts → engine-BzOR5Ygn.d.ts} +2 -2
- package/dist/{engine-D-YZ32ZJ.d.ts.map → engine-BzOR5Ygn.d.ts.map} +1 -1
- package/dist/evaluation.d.ts +1 -1
- package/dist/experiment/index.d.ts +1 -1
- package/dist/{external-optimizer-process-CB3RGyr7.js → external-optimizer-process-DCZx5PdG.js} +2 -2
- package/dist/{external-optimizer-process-CB3RGyr7.js.map → external-optimizer-process-DCZx5PdG.js.map} +1 -1
- package/dist/{external-optimizer-subprocess-DitxECD3.js → external-optimizer-subprocess-CA3aXT8I.js} +2 -2
- package/dist/{external-optimizer-subprocess-DitxECD3.js.map → external-optimizer-subprocess-CA3aXT8I.js.map} +1 -1
- package/dist/hosted/index.js +1 -1
- package/dist/{index-DnZdBxCL.d.ts → index-BTwxP7L0.d.ts} +6 -6
- package/dist/{index-DnZdBxCL.d.ts.map → index-BTwxP7L0.d.ts.map} +1 -1
- package/dist/{index-UxNTvn-w.d.ts → index-BWiAAWrk.d.ts} +3 -3
- package/dist/{index-UxNTvn-w.d.ts.map → index-BWiAAWrk.d.ts.map} +1 -1
- package/dist/{index-Bj96W0jZ.d.ts → index-DHKEAv3s.d.ts} +44 -2
- package/dist/index-DHKEAv3s.d.ts.map +1 -0
- package/dist/{index-xnhRdqik.d.ts → index-D_rfXep9.d.ts} +4 -4
- package/dist/{index-xnhRdqik.d.ts.map → index-D_rfXep9.d.ts.map} +1 -1
- package/dist/index.d.ts +9 -9
- package/dist/index.js +6 -6
- package/dist/ledger-core/index.d.ts +2 -2
- package/dist/ledger-core/index.js +2 -1
- package/dist/ledger-core-B2hahzrt.js +248 -0
- package/dist/ledger-core-B2hahzrt.js.map +1 -0
- package/dist/{llm-judge-BtMxSe1e.js → llm-judge-Dw0rDpve.js} +5 -5
- package/dist/{llm-judge-BtMxSe1e.js.map → llm-judge-Dw0rDpve.js.map} +1 -1
- package/dist/{matrix-jHTsHEkR.d.ts → matrix-CQwid3kj.d.ts} +2 -2
- package/dist/{matrix-jHTsHEkR.d.ts.map → matrix-CQwid3kj.d.ts.map} +1 -1
- package/dist/multishot/golden/index.d.ts +1 -1
- package/dist/multishot/index.d.ts +2 -2
- package/dist/openapi.json +1 -1
- package/dist/{produced-state-CYhim2We.js → produced-state-C1cTYK6w.js} +2 -2
- package/dist/{produced-state-CYhim2We.js.map → produced-state-C1cTYK6w.js.map} +1 -1
- package/dist/rl.d.ts +2 -2
- package/dist/search/index.d.ts +2 -2
- package/dist/search/index.js +2 -2
- package/dist/{search-kernel-ah2GESe3.d.ts → search-kernel-BWp7hfJn.d.ts} +3 -3
- package/dist/{search-kernel-ah2GESe3.d.ts.map → search-kernel-BWp7hfJn.d.ts.map} +1 -1
- package/dist/{search-kernel-BIXWM7YV.js → search-kernel-BztlERLv.js} +3 -3
- package/dist/{search-kernel-BIXWM7YV.js.map → search-kernel-BztlERLv.js.map} +1 -1
- package/dist/{search-ledger-aMTNQn_m.js → search-ledger-d7V8o9Lk.js} +3 -3
- package/dist/{search-ledger-aMTNQn_m.js.map → search-ledger-d7V8o9Lk.js.map} +1 -1
- package/dist/{search-ledger-file-DY0AXX3W.js → search-ledger-file-BqOo-bVo.js} +2 -2
- package/dist/{search-ledger-file-DY0AXX3W.js.map → search-ledger-file-BqOo-bVo.js.map} +1 -1
- package/dist/{search-shipper-DKXWuLnp.js → search-shipper-er4pK5_Y.js} +3 -3
- package/dist/{search-shipper-DKXWuLnp.js.map → search-shipper-er4pK5_Y.js.map} +1 -1
- package/dist/{search-summary-nwrtXxC_.js → search-summary-CBvz3u_S.js} +3 -3
- package/dist/{search-summary-nwrtXxC_.js.map → search-summary-CBvz3u_S.js.map} +1 -1
- package/dist/{semantic-concept-judge-B4zwq9Uy.js → semantic-concept-judge-sDBNg_mI.js} +2 -2
- package/dist/{semantic-concept-judge-B4zwq9Uy.js.map → semantic-concept-judge-sDBNg_mI.js.map} +1 -1
- package/dist/{skillopt-optimization-method-Ds4Ck2Dn.js → skillopt-optimization-method-CvaFGm95.js} +5 -5
- package/dist/{skillopt-optimization-method-Ds4Ck2Dn.js.map → skillopt-optimization-method-CvaFGm95.js.map} +1 -1
- package/dist/{statistical-heldout-8RGPIeyo.d.ts → statistical-heldout-DZaTo3gZ.d.ts} +2 -2
- package/dist/{statistical-heldout-8RGPIeyo.d.ts.map → statistical-heldout-DZaTo3gZ.d.ts.map} +1 -1
- package/dist/{store-tool-spans-wQ9idsnn.d.ts → store-tool-spans-DuvZ4YWb.d.ts} +3 -3
- package/dist/{store-tool-spans-wQ9idsnn.d.ts.map → store-tool-spans-DuvZ4YWb.d.ts.map} +1 -1
- package/dist/systemone.d.ts +76 -1
- package/dist/systemone.d.ts.map +1 -1
- package/dist/systemone.js +185 -2
- package/dist/systemone.js.map +1 -1
- package/dist/{tool-groups-v28A-ZAF.d.ts → tool-groups-DOriCbkR.d.ts} +2 -2
- package/dist/tool-groups-DOriCbkR.d.ts.map +1 -0
- package/dist/{trace-questions-C26UvI3D.d.ts → trace-questions-RCaYLkwH.d.ts} +2 -2
- package/dist/{trace-questions-C26UvI3D.d.ts.map → trace-questions-RCaYLkwH.d.ts.map} +1 -1
- package/dist/trace-repair/index.d.ts +1 -1
- package/dist/traces.d.ts +2 -2
- package/dist/{tree-kc1d80yF.js → tree-rXLW3M_4.js} +3 -3
- package/dist/{tree-kc1d80yF.js.map → tree-rXLW3M_4.js.map} +1 -1
- package/dist/{types-Dv9sYwBB.d.ts → types-CMnPcbW5.d.ts} +2 -2
- package/dist/{types-Dv9sYwBB.d.ts.map → types-CMnPcbW5.d.ts.map} +1 -1
- package/dist/{types-tWFY6bON.d.ts → types-Dj0Qv2qi.d.ts} +2 -2
- package/dist/{types-tWFY6bON.d.ts.map → types-Dj0Qv2qi.d.ts.map} +1 -1
- package/docs/public-api.md +47 -19
- package/docs/systemone-integrations.md +1 -0
- package/docs/systemone.md +34 -0
- package/docs/verified-labels-flywheel.md +2 -2
- package/package.json +1 -1
- package/dist/benchmark-command-IMtRf9eK.js.map +0 -1
- package/dist/index-Bj96W0jZ.d.ts.map +0 -1
- package/dist/tool-groups-v28A-ZAF.d.ts.map +0 -1
|
@@ -4,7 +4,7 @@ import { _ as ProposalFinding } from "./types-KEkKkETv.js";
|
|
|
4
4
|
import { M as LlmCallMetadata } from "./types-Lub7o6bN.js";
|
|
5
5
|
import { t as SeriesDistribution } from "./descriptive-B2iPaT9J.js";
|
|
6
6
|
import { A as SearchLedgerHash, G as SearchSplit, I as SearchNodeStatus, L as SearchOpenedEvent, N as SearchNodeDecision, O as SearchLedgerEntry, P as SearchNodeRef, Q as SearchTaskOutcome, R as SearchOperationKind, U as SearchReservation, Z as SearchTask, b as SearchClosedEvent, d as SearchCellCancelledEvent, i as SearchArtifactRef, l as SearchCandidateSurface, o as SearchAudit, p as SearchCellStage, t as NodeEstimate, w as SearchEdgeRecordedEvent, z as SearchOperationRecordedEvent } from "./search-ledger-types-p2xKfiDJ.js";
|
|
7
|
-
import {
|
|
7
|
+
import { w as LedgerProjector } from "./index-DHKEAv3s.js";
|
|
8
8
|
import { i as RolloutSearchLineage } from "./mint-CDy-brUR.js";
|
|
9
9
|
//#region src/campaign/search-state.d.ts
|
|
10
10
|
/** Deterministic node id: identical content in one search is one node. */
|
|
@@ -1015,4 +1015,4 @@ interface CampaignResult<TArtifact = unknown, TScenario extends Scenario = Scena
|
|
|
1015
1015
|
}
|
|
1016
1016
|
//#endregion
|
|
1017
1017
|
export { SearchNode as $, LabeledScenarioWrite as A, ScoredSurfaceOutcome as B, JudgeDimension as C, LabeledScenarioSampleArgs as D, LabeledScenarioRecord as E, ProposeContext as F, labelTrustRank as G, SurfaceProposer as H, ProposedCandidate as I, renderSearchSummary as J, SearchProposerView as K, RedactionStatus as L, OptimizerConfig as M, ParetoParent as N, LabeledScenarioSource as O, ProposalTrackContext as P, SearchDecisionRecord as Q, Scenario as R, JudgeConfig as S, LabelTrust as T, TraceSpan as U, SessionScript as V, isProposedCandidate as W, SearchCell as X, searchProposerView as Y, SearchCompletion as Z, GateDecision as _, CampaignResult as a, SearchUnitScore as at, GenerationRecord as b, CampaignTraceWriter as c, searchTaskSetDigest as ct, DispatchContext as d, SearchOperation as et, DispatchFn as f, GateContribution as g, GateContext as h, CampaignCostMeter as i, SearchStateView as it, MutableSurface as j, LabeledScenarioStore as k, CodeSurface as l, searchUnitScores as lt, GateCheckStatus as m, CampaignArtifactWriter as n, SearchSpend as nt, CampaignScenarioIdentity as o, searchCellId as ot, Gate as p, SearchSummaryOptions as q, CampaignCellResult as r, SearchState as rt, CampaignTokenUsage as s, searchNodeId as st, CampaignAggregates as t, SearchScoredCell as tt, ComponentSurface as u, GateResult as v, JudgeScore as w, JudgeAggregate as x, GenerationCandidate as y, ScenarioAggregate as z };
|
|
1018
|
-
//# sourceMappingURL=types-
|
|
1018
|
+
//# sourceMappingURL=types-CMnPcbW5.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types-
|
|
1
|
+
{"version":3,"file":"types-CMnPcbW5.d.ts","names":[],"sources":["../src/campaign/search-state.ts","../src/campaign/search-summary.ts","../src/campaign/types.ts"],"mappings":";;;;;;;;;;iBAwDgB,aAAa,kBAAkB;;iBAK/B,aACd,kBACA,gBACA,gBACA,OAAO,aACP;;iBAMc,oBAAoB,gBAAgB,eAAe;UAkBlD;EACf;EACA,UAAU;EACV,OAAO;EACP;EACA;;UAGe;EACf;EACA;EACA;EACA;;;;UAKe;EACf;EACA;EACA;EACA;;;UAIe;EACf;EACA;;EAEA;EACA;;;;;;;;iBASc,iBAAiB,gBAAgB,qBAAqB;UAoBrD;EACf;;EAEA;EACA,gBAAgB;EAChB,UAAU;EACV,mBAAmB;;;EAGnB;;EAEA;;;;EAIA,kBAAkB;EAClB;EACA;EACA,QAAQ;EACR;;EAEA;EACA,oBAAoB;EACpB;EACA,OAAO;EACP;EACA;;UAGe;EACf;EACA;EACA;EACA;EACA,OAAO;EACP;EACA,OAAO;EACP;EACA,aAAa;EACb;EACA;EACA,SAAS;;EAET;;EAEA;EACA,WAAW;;EAEX;;EAEA;EACA;EACA;EACA;;UAGe;EACf;EACA,eAAe;EACf,aAAa;EACb;;EAEA,SAAS;;;EAGT,WAAW;;;EAGX,oBAAoB;;EAEpB;;EAEA;;UAGe;EACf;EACA;;;;;;;cA6BW,uBAAuB,gBAAgB,mBAAmB;WAC5D;UACD;UACA;UACA;UACA;UACA;UACA;mBACS;mBACA;mBACA;mBACA;mBACA;mBACA;mBACA;mBACA;UACT;UACA;;mBAES;;UAET;mBACS;EAEjB,YAAY;EAgCZ,MAAM,OAAO,mBAAmB;EA6BhC,YAAY;;EAKZ,KAAK;UAIG;UAgCA;UA6CA;UAsBA;UA8BA;UA8CA;UAsHA;UAmEA;UA4EA;UAoBA;UAmDA;UAaA;;;UA2EA;;EASR;;;UAmBQ;;;UAkBA;;;;;;;;;;;UAoBA;;;EAsBR;IAAgB;IAA4B;;;EAa5C,cAAc;;EAKd,cAAc;;EAKd;IAAc;IAAkB,WAAW;;;EAK3C;;EAKA,aAAa;;EAcb,SAAS,iBAAiB;;EAM1B;;EAKA,YAAY;;EAKZ,oBAAoB;;EAKpB,SAAS,iBAAiB;;EAK1B;;EAKA,SAAS,iBAAiB;;EAM1B,YAAY;;EAMZ,cAAc,sBAAsB;;EAMpC,gBAAgB,gBAAgB,OAAO,cAAc;;;;;;;cAiB1C;WACF;WACA,QAAQ;WACR,QAAQ;WACR;IAAQ;IAAkB,WAAW;;;WAErC;WACA,OAAO;WACP,YAAY;;;;;;;WAOZ;IAAU;IAA4B;;mBAC9B;mBACA;EAEjB,YAAY,OAAO,aAAa;;MAgB5B;EAIJ,KAAK,iBAAiB;EAItB,QAAQ;;EAKR;;EAKA,SAAS;EAKT,gBAAgB;EAIhB,KAAK,iBAAiB;;EAKtB,SAAS;EAKT,KAAK,iBAAiB;;EAKtB,MAAM;IAAU;MAAyB;EAKzC,UAAU,sBAAsB;;;EAMhC,YAAY,gBAAgB,OAAO,cAAc;;EAKjD,WAAW,gBAAgB,OAAO,cAAc;;EAKhD,QAAQ,aAAa,uBAAuB;UAmCpC;;;;;;UCluCO;WACN;WACA;;;EAGT,YAAY,0BAA0B;;EAEtC,WAAW,0BAA0B;;;iBAIvB,mBAAmB,OAAO,kBAAkB;UAW3C;;;;;;;;EAQf,OAAO;;EAEP;;;;;;;;;;;iBAYc,oBAAoB,OAAO,iBAAiB,SAAS;;;;;UCtCpD;EACf;EACA;EACA;;;;;EAKA;;;UAIe,iCAAiC,KAAK;EACrD;;;;;UAMe;EACf;;EAEA;EACA;EACA;EACA;EACA,QAAQ;EACR,OAAO;EACP,WAAW;EACX,MAAM;;EAEN;;EAEA;;;;;;;;EAQA;;;;KAKU,WAAW,kBAAkB,UAAU,cACjD,UAAU,WACV,KAAK,oBACF,QAAQ;;;;UAOI,cAAc,WAAW;EACxC;EACA;EACA;;EAEA;;;EAGA,sBAAsB,UAAU,WAAW,sBAAsB,UAAU,cAAc;;UAK1E;;EAEf;;EAEA;;;;;;;;;UAUe,YAAY,WAAW,kBAAkB,WAAW;EACnE;EACA,YAAY;;;;EAIZ;;;EAGA,MAAM;IACJ,UAAU;IACV,UAAU;IACV,QAAQ;;IAER,aAAa;IACb;IACA,WAAW;MACT,aAAa,QAAQ;EACzB,aAAa,UAAU;;;;;;;;;;;UAYR;EACf,YAAY;EACZ;EACA;;EAEA,UAAU;;;;;;;EAOV;;;;EAIA,eAAe,eAAe;IAAQ;IAAe;;;;;EAIrD;;;EAGA;;EAEA;;EAEA,WAAW,eAAe;;;;;;;UAUX;WACN;;;WAGA;;WAEA;;WAEA;;WAEA;;WAEA;;WAEA;;;WAGA;aACE;aACA;aACA;;;WAGF;;;UAIM;WACN;WACA,YAAY,SAAS;;;;;;;;;;;KAYpB,0BAA0B,mBAAmB;;;;;;;UAQxC;EACf,SAAS;;EAET;;;;EAIA;;;;;;EAMA,cAAc,SAAS;;;;iBAKT,oBACd,OAAO,iBAAiB,oBACvB,SAAS;;;;;;;;;UAkBK;EACf,SAAS;EACT;;;EAGA,YAAY;;;EAGZ;;EAEA;EACA;EACA;;;UAIe;EACf;EACA;EACA;EACA;EACA;;;;;UAMe;;;EAGf;;EAEA;EACA;EACA;EACA,YAAY;;;;;EAKZ,WAAW;IAAQ;IAAoB;IAAmB;IAAgB;;EAC1E;IACE;IACA;;;;;UAMa,eAAe,YAAY;;;;WAIjC,gBAAgB;;;;;WAKhB;;;;WAIA,UAAU;IAAgB;IAAgB,UAAU;;;;WAGpD,QAAQ;;;;WAIR;;;WAGA,SAAS,cAAc;WACvB,UAAU,cAAc;;WAExB;WACA;WACA,QAAQ;;WAER,QAAQ;;;WAGR,kBAAkB;;;WAGlB,mBAAmB;;;;WAInB,gBAAgB;;;;WAIhB;;;;;;;WAOA,gBAAgB,cAAc;;WAE9B,aAAa;WACb;;;;;;;;;;;;;;UAeM,gBAAgB,YAAY;EAC3C;;;;;EAKA,QAAQ,KAAK,eAAe,aAAa,QAAQ,MAAM,iBAAiB;;;EAGxE,QAAQ;IAAQ,SAAS,cAAc;;IAAwB;IAAe;;;UAG/D;EACf;EACA;EACA,mBAAmB,qBAAqB;;UAGzB,wBAAwB;EACvC,UAAU;;;KAMA;;KAGA;UAEK;EACf;EACA,QAAQ;EACR;;UAGe,YAAY,WAAW,kBAAkB;EACxD,oBAAoB,YAAY;EAChC,oBAAoB,YAAY;;EAEhC,aAAa,YAAY,eAAe;;;;;EAKxC,sBAAsB,YAAY,eAAe;;;;;;;EAOjD,yBAAyB,YAAY,eAAe;;;EAGpD,uBAAuB,YAAY;EACnC,WAAW;EACX;IAAQ;IAAmB;;;EAE3B,aAAa;EACb;EACA,QAAQ;;UAGO;EACf,UAAU;EACV;EACA,mBAAmB;EACnB;;;UAIe,KAAK,qBAAqB,kBAAkB,WAAW;EACtE;EACA,OAAO,KAAK,YAAY,WAAW,aAAa,QAAQ;;;;UAOzC;EACf,KAAK,cAAc,aAAa,0BAA0B;EAC1D,SAAS;;UAGM;EACf,IAAI,aAAa;EACjB,aAAa,aAAa;;;;;UAMX;EACf,MAAM,cAAc,kBAAkB,aAAa;EACnD,UAAU,cAAc,iBAAiB;;;;;;KAO/B,qBAAqB;;;;;UAMhB;;EAEf,YAAY,GACV,OAAO,KAAK,iBAAiB;IAC3B,UAAU;MAEX,QAAQ,eAAe;;;;;KAQhB;KAOA;;;;;;;;;;;;;;;KAgBA;;iBASI,eAAe,OAAO;;;;UAOrB,qBAAqB,kBAAkB,WAAW,UAAU;EAC3E,UAAU;EACV,UAAU;EACV,aAAa,eAAe;EAC5B,QAAQ;EACR;EACA;EACA,iBAAiB;;;;;EAKjB,aAAa;;EAEb;;UAGe,sBAAsB,kBAAkB,WAAW,UAAU,6BACpE,qBAAqB,WAAW;;EAExC;;;EAGA;;UAGe;EACf;;EAEA;;;;EAIA;EACA;IACE;IACA,SAAS,wBAAwB;IACjC;IACA;;;;;IAKA,WAAW;;;UAIE;EACf,QAAQ,OAAO,uBAAuB;EACtC,OAAO,MAAM,4BAA4B,QAAQ;EACjD,QAAQ;IACN;IACA;IACA,UAAU;;;IAGV,SAAS,OAAO;;;UAMH,mBAAmB;;;EAGlC;EACA;;EAEA;IAAY;IAAsB;;EAClC;EACA;EACA;EACA,UAAU;EACV,aAAa,eAAe;;;EAG5B;;EAEA,gBAAgB;;EAEhB;;;EAGA,YAAY;;;EAGZ;;;EAGA;EACA;EACA;EACA;;;;EAIA;;EAEA;;EAEA;EACA;;UAGe;EACf;EACA;EACA;EACA;;;;;;EAMA,cAAc;;UAGC;EACf;EACA;EACA;;;EAGA,cAAc;;UAGC;EACf;EACA,YAAY;EACZ;;;;;;UAOe;EACf;;;;EAIA;;;;EAIA;;;;EAIA;IACE;IACA;IACA,iBAAiB;MAAQ;MAAgB;;;;;EAI3C,YAAY;;;;;;;;;;EAUZ,WAAW;IAAQ;IAAoB;IAAmB;IAAgB;;;;EAG1E;;;;EAIA;;;;EAIA,cAAc,SAAS;;UAGR;EACf,SAAS,eAAe;EACxB,YAAY,eAAe;;EAE3B,MAAM;;EAEN;EACA;EACA;;EAEA;;EAEA;;EAEA;;EAEA;;UAGe,eAAe,qBAAqB,kBAAkB,WAAW;;EAEhF;;EAEA;EACA;;EAEA;EACA;EACA;EACA;EACA,OAAO,MAAM,mBAAmB;EAChC,YAAY;EACZ;IACE,aAAa;IACb;;EAEF,OAAO;EACP;EACA;EACA,iBAAiB;;;EAGjB,WAAW,MAAM,2BAA2B,KAAK"}
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { c as RunSplitTag } from "./run-record-p2AYTrCn.js";
|
|
2
|
-
import { R as Scenario, d as DispatchContext } from "./types-
|
|
2
|
+
import { R as Scenario, d as DispatchContext } from "./types-CMnPcbW5.js";
|
|
3
3
|
//#region src/benchmarks/types.d.ts
|
|
4
4
|
type BenchmarkTaskKind = 'retrieval' | 'rag-answer' | 'hallucination' | 'kb-improvement' | 'routing' | 'custom';
|
|
5
5
|
type BenchmarkFamily = 'beir' | 'mteb-retrieval' | 'msmarco' | 'trec-dl' | 'miracl' | 'lotte' | 'bright' | 'crag' | 'hotpotqa' | 'kilt' | 'ragtruth' | 'faithbench' | 'first-party' | 'custom';
|
|
@@ -99,4 +99,4 @@ declare const BENCHMARK_SPLIT_SEED = "agent-eval-v1";
|
|
|
99
99
|
declare function deterministicSplit(itemId: string, seed?: string): RunSplitTag;
|
|
100
100
|
//#endregion
|
|
101
101
|
export { BenchmarkFamily as a, BenchmarkSource as c, BenchmarkEvaluation as i, BenchmarkTaskKind as l, BenchmarkAdapter as n, BenchmarkResponder as o, BenchmarkDatasetItem as r, BenchmarkScenario as s, BENCHMARK_SPLIT_SEED as t, deterministicSplit as u };
|
|
102
|
-
//# sourceMappingURL=types-
|
|
102
|
+
//# sourceMappingURL=types-Dj0Qv2qi.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types-
|
|
1
|
+
{"version":3,"file":"types-Dj0Qv2qi.d.ts","names":[],"sources":["../src/benchmarks/types.ts"],"mappings":";;;KASY;KAQA;UAgBK,qBAAqB;;;EAGpC;;EAEA,SAAS;;EAET,QAAQ;;EAER,SAAS;;EAET,WAAW;;EAEX;;EAEA,SAAS;EACT,WAAW;;UAGI;;;;EAIf;;EAEA;;EAEA,aAAa;;;EAGb,KAAK;EACL;;UAGe;EACf;EACA;EACA;;EAEA;EACA;EACA;;EAEA;;;;;EAKA;;;UAQe,iBAAiB,kBAAkB,oBAAoB;;EAEtE;EACA,SAAS;EACT,WAAW;EACX;EACA,SAAS;EACT;;;;;EAKA,YAAY,OAAO,cAAc,QAAQ,qBAAqB;;EAE9D,SAAS,MAAM,qBAAqB,WAAW,UAAU,YAAY,QAAQ;;;;EAI7E,YAAY,iBAAiB;;UAGd,kBAAkB,4BAA4B;EAC7D;EACA;EACA,QAAQ;EACR,UAAU;EACV,UAAU;EACV,MAAM,qBAAqB;;KAGjB,mBAAmB,oBAAoB,uBAAuB;EACxE,UAAU,kBAAkB;EAC5B,MAAM,qBAAqB;EAC3B,SAAS;MACL,QAAQ,aAAa;;;cAoBd;;;;;;;;;iBAUG,mBACd,gBACA,gBACC"}
|
package/docs/public-api.md
CHANGED
|
@@ -7,11 +7,11 @@ Generated by `pnpm api:census` on demand — this is a dated reading, not a gate
|
|
|
7
7
|
| measure | count |
|
|
8
8
|
| --- | --- |
|
|
9
9
|
| export subpaths | 33 |
|
|
10
|
-
| published value exports (subpath x symbol) |
|
|
11
|
-
| distinct symbols |
|
|
10
|
+
| published value exports (subpath x symbol) | 1612 |
|
|
11
|
+
| distinct symbols | 1413 |
|
|
12
12
|
| production | 1104 |
|
|
13
|
-
| planned |
|
|
14
|
-
| none |
|
|
13
|
+
| planned | 284 |
|
|
14
|
+
| none | 224 |
|
|
15
15
|
|
|
16
16
|
Type-only exports are not listed: a type binds no runtime surface, and removing one cannot break a caller at run time.
|
|
17
17
|
|
|
@@ -90,11 +90,12 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
90
90
|
|
|
91
91
|
### `.`
|
|
92
92
|
|
|
93
|
-
|
|
93
|
+
366 value exports — 291 production, 43 planned, 32 none.
|
|
94
94
|
|
|
95
95
|
| symbol | consumer | evidence |
|
|
96
96
|
| --- | --- | --- |
|
|
97
97
|
| `acquisitionPlansForKnowledgeGaps` | production | agent-builder:src/lib/.server/eval/loops/data-acquisition-engine.ts |
|
|
98
|
+
| `addCalibrationVerdicts` | planned | doc: docs/concepts.md |
|
|
98
99
|
| `admitPolicyEdit` | none | only this package's tests: src/analyst/policy-edit.test.ts:2 |
|
|
99
100
|
| `AGENT_PROFILE_KINDS` | production | blueprint-agent:scripts/experiments/lib/agent-profile-cell.ts |
|
|
100
101
|
| `AgentEvalError` | production | agent-runtime:src/errors.ts |
|
|
@@ -117,6 +118,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
117
118
|
| `assertCapabilityHeadroom` | production | blueprint-agent:scripts/experiments/lib/validity-gates.ts |
|
|
118
119
|
| `assertCrossFamily` | production | agent-builder:frontier/judges/artifact-head-to-head.ts |
|
|
119
120
|
| `assertCrossFamilyServed` | planned | doc: docs/building-doctrine.md |
|
|
121
|
+
| `assertJudgeMayGate` | planned | doc: docs/concepts.md |
|
|
120
122
|
| `assertNoHiddenLeak` | planned | doc: docs/design/statistics-decisions.md |
|
|
121
123
|
| `assertNoJudgeVerdict` | production | this package: src/analyst/policy-edit.ts:6 |
|
|
122
124
|
| `assertProductBenchmarkRun` | production | creative-agent:eval/research-package.ts |
|
|
@@ -151,7 +153,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
151
153
|
| `canonicalJson` | production | agent-knowledge:src/benchmarks/memory-recovery.ts |
|
|
152
154
|
| `capabilityHeadroom` | production | blueprint-agent:scripts/experiments/lib/validity-gates.ts |
|
|
153
155
|
| `captureFetchToRawSink` | production | creative-agent:eval/canonical-runner.ts |
|
|
154
|
-
| `certificationEvidenceDigest` | production | this package: src/completion-verifier.ts:
|
|
156
|
+
| `certificationEvidenceDigest` | production | this package: src/completion-verifier.ts:37 |
|
|
155
157
|
| `checkCanaries` | production | blueprint-agent:scripts/experiments/lib/contamination-preflight.ts |
|
|
156
158
|
| `checkServedModel` | production | this package: src/integrity/preflight.ts:33 |
|
|
157
159
|
| `checkTraceContracts` | production | creative-agent:eval/lib/trace-contracts.ts |
|
|
@@ -202,6 +204,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
202
204
|
| `DECISION_PAIRED_DELTA_STATISTIC` | production | this package: src/campaign/estimate-node.ts:19 |
|
|
203
205
|
| `DEFAULT_BOUNDED_PROCESS_TIMEOUT_MS` | production | this package: src/command-runner.ts:28 |
|
|
204
206
|
| `DEFAULT_FAILURE_REASON_RULES` | none | only this package's tests: tests/failure-taxonomy.test.ts:2 |
|
|
207
|
+
| `DEFAULT_JUDGE_GATE_POLICY` | none | — |
|
|
205
208
|
| `DEFAULT_RED_TEAM_CORPUS` | production | starter-foundry:registry/layers/agent-eval/redteam/files/src/eval/redteam/runner.ts |
|
|
206
209
|
| `DEFAULT_TRACE_ANALYST_BUDGETS` | production | agent-builder:src/lib/.server/eval/stores/d1-trace-analysis-store-adapter.ts |
|
|
207
210
|
| `DEFAULT_TRACE_ANALYST_KINDS` | production | agent-dev-container:products/intelligence/api/src/lib/deep-trace-analyst.ts |
|
|
@@ -236,7 +239,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
236
239
|
| `exportRunAsOtlp` | production | agent-builder:src/lib/.server/eval/analysts/canonical-trace-analyst.ts |
|
|
237
240
|
| `extractErrorCount` | production | blueprint-agent:scripts/experiments/lib/error-count-extractor.ts |
|
|
238
241
|
| `extractProducedState` | production | agent-app:src/eval/index.ts |
|
|
239
|
-
| `extractUsage` | production | this package: src/contract/intake/code-agent-session.ts:
|
|
242
|
+
| `extractUsage` | production | this package: src/contract/intake/code-agent-session.ts:13 |
|
|
240
243
|
| `extractUsageFromSse` | production | this package: src/trace/capture-fetch.ts:8 |
|
|
241
244
|
| `FAILURE_BLAME` | none | only this package's tests: tests/failure-taxonomy.test.ts:2 |
|
|
242
245
|
| `FAILURE_CLASSES` | production | agent-dev-container:products/intelligence/api/src/lib/deep-trace-analyst.ts |
|
|
@@ -288,6 +291,8 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
288
291
|
| `judgeAgreementView` | production | insurance-agent:scripts/analyze-agent-eval-evidence.ts |
|
|
289
292
|
| `JudgeError` | production | agent-runtime:src/errors.ts |
|
|
290
293
|
| `judgeFamily` | production | agent-builder:frontier/judges/llm.ts |
|
|
294
|
+
| `judgeGateDecision` | planned | doc: docs/concepts.md |
|
|
295
|
+
| `JudgeGateRefusedError` | planned | doc: docs/concepts.md |
|
|
291
296
|
| `judgeSpans` | production | this package: src/builder-eval/three-layer-eval.ts:25 |
|
|
292
297
|
| `knowledgeReadinessTracePayload` | production | agent-dev-container:products/intelligence/api/src/routes/project-engines.ts |
|
|
293
298
|
| `leaderboard` | production | blueprint-agent:scripts/experiments/lib/cross-harness-leaderboard.ts |
|
|
@@ -309,6 +314,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
309
314
|
| `mcnemar` | production | blueprint-agent:scripts/experiments/analyze-convergence.ts |
|
|
310
315
|
| `mcnemarPower` | production | discovery:research/e1-topology-20260920/gate/gate.mjs |
|
|
311
316
|
| `mcnemarRequiredN` | production | blueprint-agent:scripts/experiments/analyze-convergence.ts |
|
|
317
|
+
| `measureJudgeAgreement` | planned | doc: docs/concepts.md |
|
|
312
318
|
| `minimumPairsForPairedDeltaTest` | production | agent-dev-container:products/intelligence/api/src/routes/optimizations.ts |
|
|
313
319
|
| `mintRolloutRows` | production | blueprint-agent:scripts/experiments/lib/vb-climb.ts |
|
|
314
320
|
| `MODEL_PRICING` | production | supervisor-lab:archive/bench/profile-arena.ts |
|
|
@@ -373,6 +379,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
373
379
|
| `redTeamDataset` | production | creative-agent:eval/red-team.ts |
|
|
374
380
|
| `redTeamReport` | production | creative-agent:eval/red-team.ts |
|
|
375
381
|
| `regexMatches` | none | only this package's tests: src/oracle.test.ts:2 |
|
|
382
|
+
| `registerCalibrationSet` | planned | doc: docs/concepts.md |
|
|
376
383
|
| `renderPreferenceMemoryMarkdown` | production | agent-builder:src/lib/.server/eval/loops/auto-research-runner.ts |
|
|
377
384
|
| `repeatedActionDetector` | production | agent-runtime:src/runtime/supervise/detector-monitor.ts |
|
|
378
385
|
| `requiredPairedSampleSize` | production | discovery:research/e1-topology-20260920/gate/gate.mjs |
|
|
@@ -409,6 +416,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
409
416
|
| `ServedCrossFamilyError` | none | only this package's tests: src/integrity/served-model.test.ts:2 |
|
|
410
417
|
| `servedModelAcceptable` | production | this package: src/integrity/preflight.ts:33 |
|
|
411
418
|
| `shareAllowed` | production | agent-dev-container:products/intelligence/api/src/lib/search-blobs.ts |
|
|
419
|
+
| `snapshotFromJudgeAgreement` | planned | doc: docs/concepts.md |
|
|
412
420
|
| `spanRecordsToTraceAnalysisStore` | production | agent-dev-container:products/intelligence/api/src/lib/pg-trace-analyst-store.ts |
|
|
413
421
|
| `spearmanR` | production | this package: scripts/synthetic-search-ledger.ts:71 |
|
|
414
422
|
| `stripFencedJson` | production | agent-builder:src/lib/.server/eval/router-llm-client.ts |
|
|
@@ -824,7 +832,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
824
832
|
|
|
825
833
|
### `./contract`
|
|
826
834
|
|
|
827
|
-
|
|
835
|
+
63 value exports — 41 production, 15 planned, 7 none.
|
|
828
836
|
|
|
829
837
|
| symbol | consumer | evidence |
|
|
830
838
|
| --- | --- | --- |
|
|
@@ -848,6 +856,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
848
856
|
| `fromKimiCodeSession` | planned | doc: docs/code-agent-intake.md |
|
|
849
857
|
| `fromOpenCodeSession` | planned | doc: docs/code-agent-intake.md |
|
|
850
858
|
| `fromOtelSpans` | production | agent-dev-container:packages/intelligence-kernel/src/deterministic.ts |
|
|
859
|
+
| `fromPiGraphSession` | none | only this package's tests: tests/contract-code-agent-intake.test.ts:6 |
|
|
851
860
|
| `fromPiSession` | planned | doc: docs/code-agent-intake.md |
|
|
852
861
|
| `fromRunRecordDir` | production | this package: src/contract/eval-reporting-suite.ts:25 |
|
|
853
862
|
| `fsCampaignStorage` | production | agent-dev-container:products/intelligence/api/src/lib/eval-engine.ts |
|
|
@@ -859,7 +868,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
859
868
|
| `makeProposalFinding` | production | agent-lab:projects/playproof/code-surface.ts |
|
|
860
869
|
| `measuredComparisonFromAgentProfileImprovementExperiment` | production | agent-runtime:src/intelligence/authored-profile-improvement.ts |
|
|
861
870
|
| `measuredComparisonFromCandidateExperiment` | production | agent-runtime:src/intelligence/improvement-cycle.ts |
|
|
862
|
-
| `observeCodeAgentSession` | production | this package: src/contract/intake/code-agent-session.ts:
|
|
871
|
+
| `observeCodeAgentSession` | production | this package: src/contract/intake/code-agent-session.ts:14 |
|
|
863
872
|
| `observeCodeAgentStore` | none | only this package's tests: tests/contract-code-agent-store.test.ts:5 |
|
|
864
873
|
| `paretoPolicy` | planned | named in discovery-lab (bind not in the import graph) |
|
|
865
874
|
| `paretoSignificanceGate` | production | agent-app:src/eval-campaign/index.ts |
|
|
@@ -946,7 +955,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
946
955
|
| `eProcess` | production | this package: src/campaign/gates/sequential.ts:37 |
|
|
947
956
|
| `evaluateCondition` | planned | named in agent-dev-container (bind not in the import graph) |
|
|
948
957
|
| `evaluateHaltRule` | production | this package: src/experiment/define.ts:20 |
|
|
949
|
-
| `evaluateHypothesis` | none | only this package's tests: tests/tier2.test.ts:
|
|
958
|
+
| `evaluateHypothesis` | none | only this package's tests: tests/tier2.test.ts:4 |
|
|
950
959
|
| `evaluateIdentityGate` | production | this package: src/experiment/define.ts:20 |
|
|
951
960
|
| `evaluateOracleDeterminismGate` | production | this package: src/experiment/define.ts:20 |
|
|
952
961
|
| `evaluatePopulationReproducibilityGate` | production | this package: src/experiment/define.ts:20 |
|
|
@@ -1066,7 +1075,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1066
1075
|
|
|
1067
1076
|
### `./ledger-core`
|
|
1068
1077
|
|
|
1069
|
-
|
|
1078
|
+
28 value exports — 19 production, 1 planned, 8 none.
|
|
1070
1079
|
|
|
1071
1080
|
| symbol | consumer | evidence |
|
|
1072
1081
|
| --- | --- | --- |
|
|
@@ -1079,6 +1088,12 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1079
1088
|
| `LEDGER_HASH_PATTERN` | production | blueprint-agent:scripts/experiments/lib/causal-analytics.ts |
|
|
1080
1089
|
| `LedgerCanonicalizationError` | planned | named in blueprint-agent (bind not in the import graph) |
|
|
1081
1090
|
| `MemoryLedgerJournal` | production | this package: src/campaign/search-ledger.ts:23 |
|
|
1091
|
+
| `MERKLE_EMPTY_ROOT` | none | only this package's tests: src/ledger-core/merkle.test.ts:4 |
|
|
1092
|
+
| `merkleConsistencyProof` | none | only this package's tests: src/ledger-core/merkle.test.ts:4 |
|
|
1093
|
+
| `merkleInclusionProof` | none | only this package's tests: src/ledger-core/merkle.test.ts:4 |
|
|
1094
|
+
| `merkleLeafHash` | none | only this package's tests: src/ledger-core/merkle.test.ts:4 |
|
|
1095
|
+
| `merkleRoot` | none | only this package's tests: src/ledger-core/merkle.test.ts:4 |
|
|
1096
|
+
| `merkleSalt` | none | only this package's tests: src/ledger-core/merkle.test.ts:4 |
|
|
1082
1097
|
| `probeAtomicFileLock` | production | this package: src/campaign/single-run-lock.ts:17 |
|
|
1083
1098
|
| `projectorSnapshotPathFor` | production | this package: src/ledger-core/journal.ts:48 |
|
|
1084
1099
|
| `readProjectorSnapshotFile` | production | this package: src/ledger-core/journal.ts:48 |
|
|
@@ -1088,6 +1103,8 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1088
1103
|
| `tryAcquireAtomicFileLock` | production | discovery-lab:runner/evidence-review.mjs |
|
|
1089
1104
|
| `tryWithLedgerFileLock` | production | discovery-lab:experiments/diu-campaign-v1/launch-wave-cloud.mjs |
|
|
1090
1105
|
| `verifyEntriesAgainstTrustedHead` | production | this package: src/ledger-core/journal.ts:55 |
|
|
1106
|
+
| `verifyMerkleConsistency` | none | only this package's tests: src/ledger-core/merkle.test.ts:4 |
|
|
1107
|
+
| `verifyMerkleInclusion` | none | only this package's tests: src/ledger-core/merkle.test.ts:4 |
|
|
1091
1108
|
| `withLedgerFileLock` | production | this package: src/analyst/benchmark-response-cache.ts:6 |
|
|
1092
1109
|
| `writeLedgerFileAtomically` | production | discovery-lab:experiments/diu-campaign-v1/launch-wave-cloud.mjs |
|
|
1093
1110
|
|
|
@@ -1103,10 +1120,12 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1103
1120
|
|
|
1104
1121
|
### `./meta-eval`
|
|
1105
1122
|
|
|
1106
|
-
|
|
1123
|
+
43 value exports — 27 production, 14 planned, 2 none.
|
|
1107
1124
|
|
|
1108
1125
|
| symbol | consumer | evidence |
|
|
1109
1126
|
| --- | --- | --- |
|
|
1127
|
+
| `addCalibrationVerdicts` | planned | doc: docs/concepts.md |
|
|
1128
|
+
| `assertJudgeMayGate` | planned | doc: docs/concepts.md |
|
|
1110
1129
|
| `auditEvaluator` | production | agent-dev-container:products/intelligence/api/src/lib/measurement-factory.ts |
|
|
1111
1130
|
| `auditProbabilityPolicy` | planned | doc: docs/changes/probability-policy-audit.md |
|
|
1112
1131
|
| `brierScore` | production | discovery-lab:experiments/decision-bench-v1/score.mjs |
|
|
@@ -1118,6 +1137,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1118
1137
|
| `clusterBootstrapMean` | production | discovery-lab:experiments/decision-bench-v1/score.mjs |
|
|
1119
1138
|
| `continuousAgreement` | production | agent-builder:frontier/judges/eval-validity.ts |
|
|
1120
1139
|
| `correlationStudy` | production | ph0ny:products/builder/api/src/eval/outcomes.ts |
|
|
1140
|
+
| `DEFAULT_JUDGE_GATE_POLICY` | none | — |
|
|
1121
1141
|
| `definePlant` | production | discovery-lab:experiments/e1-checker/score.mjs |
|
|
1122
1142
|
| `evalHealthStamp` | production | creative-agent:eval/lib/judge-sentinel.ts |
|
|
1123
1143
|
| `evaluatorAdmissionPolicySchema` | production | agent-dev-container:products/intelligence/api/src/lib/measurement-factory.ts |
|
|
@@ -1125,8 +1145,11 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1125
1145
|
| `fileSentinelStore` | production | creative-agent:eval/lib/judge-sentinel.ts |
|
|
1126
1146
|
| `FileSystemOutcomeStore` | production | ph0ny:products/builder/api/src/eval/outcomes.ts |
|
|
1127
1147
|
| `InMemoryOutcomeStore` | planned | consumer tests: ph0ny:products/builder/api/src/eval/outcomes-correlate.test.ts |
|
|
1128
|
-
| `inMemorySentinelStore` | none | only this package's tests:
|
|
1148
|
+
| `inMemorySentinelStore` | none | only this package's tests: src/meta-eval/judge-gate.test.ts:2 |
|
|
1149
|
+
| `judgeGateDecision` | planned | doc: docs/concepts.md |
|
|
1150
|
+
| `JudgeGateRefusedError` | planned | doc: docs/concepts.md |
|
|
1129
1151
|
| `judgeSentinelReport` | production | creative-agent:eval/lib/judge-sentinel.ts |
|
|
1152
|
+
| `measureJudgeAgreement` | planned | doc: docs/concepts.md |
|
|
1130
1153
|
| `multiclassBrierScore` | production | discovery-lab:experiments/decision-bench-v1/score.mjs |
|
|
1131
1154
|
| `OutcomeStoreError` | planned | doc: docs/outcome-validity.md |
|
|
1132
1155
|
| `perturbEvidence` | planned | doc: docs/plants.md |
|
|
@@ -1134,11 +1157,13 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1134
1157
|
| `plantByPerturbation` | planned | doc: docs/plants.md |
|
|
1135
1158
|
| `positionalBias` | production | ph0ny:products/builder/api/src/eval/judge-bias.ts |
|
|
1136
1159
|
| `rankedProbabilityScore` | production | discovery-lab:experiments/decision-bench-v1/score.mjs |
|
|
1160
|
+
| `registerCalibrationSet` | planned | doc: docs/concepts.md |
|
|
1137
1161
|
| `rubricPredictiveValidity` | production | physim:apps/server/src/scripts/agent-eval/validate-rubrics.ts |
|
|
1138
1162
|
| `seedPlants` | production | discovery-lab:experiments/e1-checker/score.mjs |
|
|
1139
1163
|
| `selfPreference` | production | ph0ny:products/builder/api/src/eval/judge-bias.ts |
|
|
1140
1164
|
| `snapshotFromAgreement` | production | physim:apps/server/src/scripts/agent-eval/judge-sentinel.ts |
|
|
1141
1165
|
| `snapshotFromCalibration` | production | gtm-agent:eval/calibrate-judges.ts |
|
|
1166
|
+
| `snapshotFromJudgeAgreement` | planned | doc: docs/concepts.md |
|
|
1142
1167
|
| `snapshotFromSentinelSet` | production | creative-agent:eval/lib/judge-sentinel.ts |
|
|
1143
1168
|
| `validateSentinelSnapshot` | production | creative-agent:eval/lib/judge-sentinel.ts |
|
|
1144
1169
|
| `verbosityBias` | production | blueprint-agent:scripts/experiments/calibration/judge-agreement.ts |
|
|
@@ -1341,7 +1366,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1341
1366
|
|
|
1342
1367
|
### `./rollout`
|
|
1343
1368
|
|
|
1344
|
-
|
|
1369
|
+
69 value exports — 45 production, 13 planned, 11 none.
|
|
1345
1370
|
|
|
1346
1371
|
| symbol | consumer | evidence |
|
|
1347
1372
|
| --- | --- | --- |
|
|
@@ -1354,13 +1379,13 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1354
1379
|
| `ATIF_SCHEMA_VERSION` | none | only this package's tests: src/rollout/interchange/harbor.test.ts:6 |
|
|
1355
1380
|
| `buildDatasetCard` | production | this package: src/rollout/release/hf-dataset.ts:25 |
|
|
1356
1381
|
| `buildHfDataset` | production | supervisor-lab:archive/bench/vertical-rollout-package.ts |
|
|
1357
|
-
| `claudeProjectSlug` | none | only this package's tests: src/rollout/readers/claude-jsonl.test.ts:
|
|
1382
|
+
| `claudeProjectSlug` | none | only this package's tests: src/rollout/readers/claude-jsonl.test.ts:11 |
|
|
1358
1383
|
| `DEFAULT_CLAUDE_PROJECTS_DIR` | planned | named in discovery-lab (bind not in the import graph) |
|
|
1359
1384
|
| `DEFAULT_OPENCODE_DB` | planned | named in discovery-lab (bind not in the import graph) |
|
|
1360
1385
|
| `defaultRolloutScrubber` | production | supervisor-lab:archive/bench/vertical-rollout-package.ts |
|
|
1361
1386
|
| `emptyScrubCounts` | production | this package: src/rollout/release/hf-dataset.ts:34 |
|
|
1362
1387
|
| `findClaudeTranscripts` | planned | doc: docs/rollout.md |
|
|
1363
|
-
| `findOpencodeSessionsByDirectory` | none | only this package's tests: src/rollout/readers/opencode-sqlite.test.ts:
|
|
1388
|
+
| `findOpencodeSessionsByDirectory` | none | only this package's tests: src/rollout/readers/opencode-sqlite.test.ts:7 |
|
|
1364
1389
|
| `FORMAT_FILES` | production | this package: src/rollout/release/hf-dataset.ts:25 |
|
|
1365
1390
|
| `FORMAT_GATE_DISPOSITION` | production | this package: src/rollout/release/card.ts:12 |
|
|
1366
1391
|
| `fromHarborTrajectory` | planned | doc: docs/rollout.md |
|
|
@@ -1379,11 +1404,11 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1379
1404
|
| `mintRolloutRows` | production | blueprint-agent:scripts/experiments/lib/vb-climb.ts |
|
|
1380
1405
|
| `observedScore` | production | this package: src/rl/reward-hacking.ts:40 |
|
|
1381
1406
|
| `observedSplitScore` | production | starter-foundry:src/lib/held-out-gate.ts |
|
|
1382
|
-
| `openOpencodeDb` | production | super-agent:evals/conversation-application.ts |
|
|
1383
1407
|
| `parseClaudeTranscript` | planned | doc: docs/rollout.md |
|
|
1384
1408
|
| `parseRolloutReleaseArgs` | none | only this package's tests: src/rollout/release/hf-dataset.test.ts:9 |
|
|
1385
1409
|
| `planPushCommand` | none | only this package's tests: src/rollout/release/hf-dataset.test.ts:9 |
|
|
1386
1410
|
| `readClaudeTranscript` | production | blueprint-agent:scripts/experiments/lib/vb-climb-coder.ts |
|
|
1411
|
+
| `readOpencodeSession` | none | only this package's tests: src/rollout/readers/opencode-sqlite.test.ts:7 |
|
|
1387
1412
|
| `readOpencodeSessionMessages` | production | super-agent:evals/conversation-application.ts |
|
|
1388
1413
|
| `readRolloutJournal` | none | only this package's tests: src/rollout/ledger.test.ts:6 |
|
|
1389
1414
|
| `readRolloutLedger` | production | this package: src/rollout/release/hf-dataset.ts:23 |
|
|
@@ -1410,6 +1435,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1410
1435
|
| `toVerifiersRolloutOutputs` | production | this package: src/rollout/release/hf-dataset.ts:22 |
|
|
1411
1436
|
| `trainingReward` | production | supervisor-lab:archive/bench/vertical-gate-proof.ts |
|
|
1412
1437
|
| `trainingScore` | production | this package: src/eval-trace-store.ts:20 |
|
|
1438
|
+
| `transcriptFromSession` | production | this package: src/supervisor-run/claude-code-reader.ts:67 |
|
|
1413
1439
|
| `unmintableReasons` | production | supervisor-lab:archive/bench/mint-audit.ts |
|
|
1414
1440
|
| `validateRolloutLine` | production | this package: src/supervisor-run/integrity-tree.ts:1 |
|
|
1415
1441
|
| `writeRolloutLedger` | production | blueprint-agent:scripts/experiments/lib/vb-climb.ts |
|
|
@@ -1507,12 +1533,13 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1507
1533
|
|
|
1508
1534
|
### `./systemone`
|
|
1509
1535
|
|
|
1510
|
-
|
|
1536
|
+
20 value exports — 10 production, 10 planned, 0 none.
|
|
1511
1537
|
|
|
1512
1538
|
| symbol | consumer | evidence |
|
|
1513
1539
|
| --- | --- | --- |
|
|
1514
1540
|
| `asAnalyst` | production | this package: src/systemone.ts:5 |
|
|
1515
1541
|
| `asJudge` | production | this package: src/systemone.ts:5 |
|
|
1542
|
+
| `assessSystemOneDomainMatch` | planned | doc: docs/systemone-integrations.md |
|
|
1516
1543
|
| `assessSystemOneReview` | planned | doc: docs/changes/systemone-provider-neutral.md |
|
|
1517
1544
|
| `createEvaluator` | production | blueprint-agent:scripts/experiments/app-grade/drive.ts |
|
|
1518
1545
|
| `normalizedSystemOneScore` | planned | doc: docs/changes/systemone-provider-neutral.md |
|
|
@@ -1521,6 +1548,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1521
1548
|
| `parseSystemOneRecordedRequest` | planned | example: examples/systemone-replay.ts:7 |
|
|
1522
1549
|
| `parseSystemOneRequest` | production | discovery-lab:experiments/diu-campaign-v1/jev-review.mjs |
|
|
1523
1550
|
| `parseSystemOneResult` | production | discovery-lab:experiments/decision-bench-v1/score.mjs |
|
|
1551
|
+
| `prepareSystemOneDomainMatch` | planned | doc: docs/systemone-integrations.md |
|
|
1524
1552
|
| `prepareSystemOneReview` | production | this package: src/rl/claim-integrity.ts:47 |
|
|
1525
1553
|
| `systemOneAnalyst` | planned | example: examples/systemone-behavior-review.ts:3 |
|
|
1526
1554
|
| `systemOneConfidence` | planned | doc: docs/changes/systemone-provider-neutral.md |
|
|
@@ -1716,7 +1744,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
|
|
|
1716
1744
|
| `emptyRedactionReport` | production | agent-runtime:src/candidate-execution/protected-redaction.ts |
|
|
1717
1745
|
| `exportRunAsOtlp` | production | agent-builder:src/lib/.server/eval/analysts/canonical-trace-analyst.ts |
|
|
1718
1746
|
| `extractOtlpAttributes` | production | this package: src/trace-analyst/store-otlp.ts:40 |
|
|
1719
|
-
| `extractUsage` | production | this package: src/contract/intake/code-agent-session.ts:
|
|
1747
|
+
| `extractUsage` | production | this package: src/contract/intake/code-agent-session.ts:13 |
|
|
1720
1748
|
| `extractUsageFromResponse` | none | only this package's tests: src/trace/extract-usage.test.ts:2 |
|
|
1721
1749
|
| `extractUsageFromSse` | production | this package: src/trace/capture-fetch.ts:8 |
|
|
1722
1750
|
| `FAILURE_CLASSES` | production | agent-dev-container:products/intelligence/api/src/lib/deep-trace-analyst.ts |
|
|
@@ -16,6 +16,7 @@ asJudge and asAnalyst adapt the same operation to existing consumers.
|
|
|
16
16
|
| Next verification | Choose a test, retrieval or review that resolves uncertainty | Existing graph checkpoints and awaited boundaries, not new permissions. |
|
|
17
17
|
| Candidate selection | Choose among generated plans or artifacts | Existing matrix/graph, exact checks first, semantic assessment second. |
|
|
18
18
|
| Configuration experiments | Compare questions, context, models and thresholds | examples/systemone-decision-benchmark.ts using runAgentMatrix and cost receipts. |
|
|
19
|
+
| Domain classification | Match bounded evidence to a product intent and caller-owned categories | prepareSystemOneDomainMatch plus assessSystemOneDomainMatch; positive matches require a qualified category and exact source. |
|
|
19
20
|
|
|
20
21
|
These are compositions, not ten new frameworks. Product triage and the benchmark recipe are concrete
|
|
21
22
|
adoption work; other rows reuse existing APIs or identify application policies to configure. An API
|
package/docs/systemone.md
CHANGED
|
@@ -62,6 +62,40 @@ add their own limits: Clef requires non-null `instructions`, 1–64 questions wi
|
|
|
62
62
|
exact answer names and choice-label unions. Runtime JSON must be validated; its
|
|
63
63
|
labels cannot become compile-time literals automatically.
|
|
64
64
|
|
|
65
|
+
## DomainMatch
|
|
66
|
+
|
|
67
|
+
`prepareSystemOneDomainMatch` builds a frozen three-question classification for products that need
|
|
68
|
+
relevance, a caller-owned category, and an exact source segment. `assessSystemOneDomainMatch`
|
|
69
|
+
revalidates the retained answer and returns `match`, `no_match`, or `uncertain` without another
|
|
70
|
+
model call. It does not choose a domain taxonomy, threshold, credential, endpoint, or effect.
|
|
71
|
+
|
|
72
|
+
```ts
|
|
73
|
+
const prepared = prepareSystemOneDomainMatch({
|
|
74
|
+
version: 'health-v1',
|
|
75
|
+
model: selectedModel,
|
|
76
|
+
intent: 'Find personal health correspondence, not general wellness marketing.',
|
|
77
|
+
categories: { lab_result: 'A medical laboratory result', appointment: 'An appointment' },
|
|
78
|
+
evidence: [{ id: 'email-1', text: emailText }],
|
|
79
|
+
coverage: 'complete',
|
|
80
|
+
thresholds: {
|
|
81
|
+
matchAtLeast: 0.8,
|
|
82
|
+
noMatchAtLeast: 0.8,
|
|
83
|
+
categoryAtLeast: 0.7,
|
|
84
|
+
sourceAtLeast: 0.7,
|
|
85
|
+
minimumMargin: 0.2,
|
|
86
|
+
},
|
|
87
|
+
})
|
|
88
|
+
const observation = await evaluate(prepared.request, { signal, idempotencyKey })
|
|
89
|
+
const decision = assessSystemOneDomainMatch(prepared, observation.value)
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
A positive match requires the match, category, and source questions to pass their thresholds and
|
|
93
|
+
margins. Missing evidence stays unresolved; partial evidence can support a match but cannot refute
|
|
94
|
+
one. The report retains native probabilities and an exact host-supplied evidence segment. Those
|
|
95
|
+
probabilities are not calibrated domain confidence, and a result never authorizes storage, hiding,
|
|
96
|
+
sending, deletion, or another effect. Re-assessing the same prepared input and retained answer is
|
|
97
|
+
an offline replay, not new inference.
|
|
98
|
+
|
|
65
99
|
## Providers, rounding and confidence
|
|
66
100
|
|
|
67
101
|
The wire format is the same for every provider; how each reports its numbers is not. A
|
|
@@ -22,7 +22,7 @@ Derived from what the run2 replay batch actually consumed (its enumeration exclu
|
|
|
22
22
|
| Claude Code transcripts `~/.claude/projects` | 474 projects, 7,579 session JSONLs, 4.6 GB | no image; `cwd` + `gitBranch` + harness `version` per message | yes — full tool calls + results | **no** — Bash `toolUseResult` records `stdout`/`stderr`/`interrupted` only, no exit code field | no (host env unpinned, no rc) |
|
|
23
23
|
| Codex sessions `~/.codex/sessions` | 4,332 rollout files, 113 GB | no image; `session_meta` has `cwd` + git `commit_hash`/`branch`/`repository_url` + `cli_version` | yes | shell events carry exit codes in payloads (format varies by version) | no (host env unpinned) |
|
|
24
24
|
| traces CLI normalized envelope (`~/code/traces`, 18 harness adapters: claude, codex, amp, gemini, opencode, pi, copilot, factory, forge, qwen, …) | imports the two stores above | `cwd` filter exists; **no image/sandbox field in the envelope** | yes | adapter-dependent | no — the schema itself cannot express environment identity |
|
|
25
|
-
| Sandbox sessions (agent-
|
|
25
|
+
| Sandbox sessions (agent-container `PersistedSession`) | remote fleet; none stored on this host | runtime knows the image — `runtime.ready` event carries `image?` + `sandboxId` + backend — but `PersistedSession` persists only `workspaceRoot` + free-form `metadata`; `image?` is optional even on the event | yes (message store) | via tool parts, not normalized | **almost** — the image is in hand at runtime and dropped at persistence time |
|
|
26
26
|
| mini-SWE / CodeTraceBench benchmark trajectories (`~/bench-cache/ctb-20260801`) | 133 labeled, 22 replayable | yes — `mswebench/*` images + cwd | yes | yes — `<returncode>N</returncode>` in every observation | **yes — the only eligible class; run2 proved 16/22 reproduce** |
|
|
27
27
|
|
|
28
28
|
Conclusion: today only benchmark-imported trajectories are replay-eligible.
|
|
@@ -31,7 +31,7 @@ The sandbox class is one persistence field away from eligibility — the runtime
|
|
|
31
31
|
|
|
32
32
|
## Phase-2 capture changes (ranked by unlock per line of code)
|
|
33
33
|
|
|
34
|
-
1. **Persist the sandbox image at session start** (agent-
|
|
34
|
+
1. **Persist the sandbox image at session start** (agent-container): copy `runtime.ready`'s `image` (as a digest, not a tag) + `sandboxId` into `PersistedSession` as first-class fields, and make `image` required on the event.
|
|
35
35
|
This single change makes every future fleet sandbox session satisfy requirements 1–2 — the highest-leverage line in the flywheel.
|
|
36
36
|
2. **Add environment identity to the traces envelope** (traces repo): an optional `environment: { image?, imageDigest?, cwd, gitCommit? }` block on the normalized session, populated by adapters where known.
|
|
37
37
|
Without it, eligible sandbox sessions lose their eligibility at import time.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tangle-network/agent-eval",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.215.0",
|
|
4
4
|
"description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
|
|
5
5
|
"homepage": "https://github.com/tangle-network/agent-eval#readme",
|
|
6
6
|
"repository": {
|