@tangle-network/agent-eval 0.150.0 → 0.150.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/dist/analyst/index.d.ts +13 -13
- package/dist/analyst/index.js +4 -4
- package/dist/{backend-integrity-DaeroaEs.d.ts → backend-integrity-DOCa_QrR.d.ts} +2 -2
- package/dist/{backend-integrity-DaeroaEs.d.ts.map → backend-integrity-DOCa_QrR.d.ts.map} +1 -1
- package/dist/{benchmark-A0E-JZ1H.d.ts → benchmark-BtAWA8nT.d.ts} +3 -3
- package/dist/{benchmark-A0E-JZ1H.d.ts.map → benchmark-BtAWA8nT.d.ts.map} +1 -1
- package/dist/{benchmark-command-CqGA0SJQ.js → benchmark-command-CAFwbH0L.js} +7 -7
- package/dist/{benchmark-command-CqGA0SJQ.js.map → benchmark-command-CAFwbH0L.js.map} +1 -1
- package/dist/benchmarks/index.d.ts +4 -4
- package/dist/benchmarks/index.js +3 -3
- package/dist/campaign/index.d.ts +8 -8
- package/dist/campaign/index.js +5 -5
- package/dist/{campaign-CR6Yt5ZS.js → campaign-CN_7xJdV.js} +18 -9
- package/dist/campaign-CN_7xJdV.js.map +1 -0
- package/dist/{capture-fetch-Im0UN1i6.d.ts → capture-fetch-BBVFzhkk.d.ts} +2 -2
- package/dist/{capture-fetch-Im0UN1i6.d.ts.map → capture-fetch-BBVFzhkk.d.ts.map} +1 -1
- package/dist/{chat-client-sFiClniQ.js → chat-client-2bVfrzhN.js} +2 -2
- package/dist/{chat-client-sFiClniQ.js.map → chat-client-2bVfrzhN.js.map} +1 -1
- package/dist/cli.js +2 -2
- package/dist/{client-w6vVJhSp.d.ts → client-kPQYT_56.d.ts} +2 -2
- package/dist/{client-w6vVJhSp.d.ts.map → client-kPQYT_56.d.ts.map} +1 -1
- package/dist/contract/index.d.ts +10 -10
- package/dist/contract/index.js +5 -5
- package/dist/{default-registry-BhRNsP-D.d.ts → default-registry-Cw0Ohdoj.d.ts} +6 -6
- package/dist/{default-registry-BhRNsP-D.d.ts.map → default-registry-Cw0Ohdoj.d.ts.map} +1 -1
- package/dist/{define-agent-eval-Cw3mkB4-.d.ts → define-agent-eval-CEQWL9Hy.d.ts} +5 -5
- package/dist/{define-agent-eval-Cw3mkB4-.d.ts.map → define-agent-eval-CEQWL9Hy.d.ts.map} +1 -1
- package/dist/{define-agent-eval-ChrtErmg.js → define-agent-eval-rqNyVhVV.js} +4 -4
- package/dist/{define-agent-eval-ChrtErmg.js.map → define-agent-eval-rqNyVhVV.js.map} +1 -1
- package/dist/{dspy-rlm-engine-Csi3CUAT.js → dspy-rlm-engine-DHI0WrUU.js} +2 -2
- package/dist/{dspy-rlm-engine-Csi3CUAT.js.map → dspy-rlm-engine-DHI0WrUU.js.map} +1 -1
- package/dist/{engine-CgQdUh-r.d.ts → engine-BLzhNzoY.d.ts} +4 -4
- package/dist/{engine-CgQdUh-r.d.ts.map → engine-BLzhNzoY.d.ts.map} +1 -1
- package/dist/{eval-campaign-Czc2TcW5.js → eval-campaign-C4jmuM-b.js} +2 -2
- package/dist/{eval-campaign-Czc2TcW5.js.map → eval-campaign-C4jmuM-b.js.map} +1 -1
- package/dist/{exact-types-D9_T83Ol.d.ts → exact-types-ccQAyut1.d.ts} +2 -2
- package/dist/{exact-types-D9_T83Ol.d.ts.map → exact-types-ccQAyut1.d.ts.map} +1 -1
- package/dist/experiment/index.d.ts +3 -3
- package/dist/{experiment-tracker-Cp9ougHs.d.ts → experiment-tracker-0MhuPArU.d.ts} +2 -2
- package/dist/{experiment-tracker-Cp9ougHs.d.ts.map → experiment-tracker-0MhuPArU.d.ts.map} +1 -1
- package/dist/{external-optimizer-contracts-mbhR9Ldu.d.ts → external-optimizer-contracts-DbLsm4Po.d.ts} +2 -2
- package/dist/{external-optimizer-contracts-mbhR9Ldu.d.ts.map → external-optimizer-contracts-DbLsm4Po.d.ts.map} +1 -1
- package/dist/{external-optimizer-process-tDe72VRO.js → external-optimizer-process-Bhmzngf-.js} +2 -2
- package/dist/{external-optimizer-process-tDe72VRO.js.map → external-optimizer-process-Bhmzngf-.js.map} +1 -1
- package/dist/{external-optimizer-subprocess-BepyNwtj.js → external-optimizer-subprocess-Dn90UqN2.js} +273 -42
- package/dist/external-optimizer-subprocess-Dn90UqN2.js.map +1 -0
- package/dist/{feedback-trajectory-Cc9M2tNF.d.ts → feedback-trajectory-DpTTjo0q.d.ts} +3 -3
- package/dist/{feedback-trajectory-Cc9M2tNF.d.ts.map → feedback-trajectory-DpTTjo0q.d.ts.map} +1 -1
- package/dist/hosted/index.d.ts +2 -2
- package/dist/index-C1ravkGA.d.ts +1 -0
- package/dist/{index-DL4MJuJL.d.ts → index-CTKpu9ry.d.ts} +7 -7
- package/dist/{index-DL4MJuJL.d.ts.map → index-CTKpu9ry.d.ts.map} +1 -1
- package/dist/{index-dh0kp2GM.d.ts → index-IQccV3Ou.d.ts} +15 -10
- package/dist/index-IQccV3Ou.d.ts.map +1 -0
- package/dist/index.d.ts +19 -19
- package/dist/index.js +8 -8
- package/dist/{integrity-D2MAlC_K.d.ts → integrity-B0dZ96EO.d.ts} +2 -2
- package/dist/{integrity-D2MAlC_K.d.ts.map → integrity-B0dZ96EO.d.ts.map} +1 -1
- package/dist/{llm-client-CxyJ8Oft.js → llm-client-Bg32RW0j.js} +47 -2
- package/dist/llm-client-Bg32RW0j.js.map +1 -0
- package/dist/{llm-judge-D7UuMI2t.js → llm-judge-CVq33oz1.js} +4 -4
- package/dist/{llm-judge-D7UuMI2t.js.map → llm-judge-CVq33oz1.js.map} +1 -1
- package/dist/{matrix-D3hL68uV.d.ts → matrix-DrVnRp4G.d.ts} +2 -2
- package/dist/{matrix-D3hL68uV.d.ts.map → matrix-DrVnRp4G.d.ts.map} +1 -1
- package/dist/meta-eval/index.d.ts +1 -1
- package/dist/multishot/golden/index.d.ts +1 -1
- package/dist/multishot/index.d.ts +2 -2
- package/dist/openapi.json +1 -1
- package/dist/{produced-state-WJs_enbZ.js → produced-state-jfk8Du3b.js} +3 -3
- package/dist/{produced-state-WJs_enbZ.js.map → produced-state-jfk8Du3b.js.map} +1 -1
- package/dist/{promotion-policy-DYRzXdTV.d.ts → promotion-policy-DLOUkYhI.d.ts} +2 -2
- package/dist/{promotion-policy-DYRzXdTV.d.ts.map → promotion-policy-DLOUkYhI.d.ts.map} +1 -1
- package/dist/{provenance-Bh9k3sjl.d.ts → provenance-oA4-zUqm.d.ts} +6 -6
- package/dist/{provenance-Bh9k3sjl.d.ts.map → provenance-oA4-zUqm.d.ts.map} +1 -1
- package/dist/{registry-DhpX9Bsg.d.ts → registry-BQwrSYpC.d.ts} +4 -4
- package/dist/{registry-DhpX9Bsg.d.ts.map → registry-BQwrSYpC.d.ts.map} +1 -1
- package/dist/{researcher-CbCK-ppF.d.ts → researcher-DJnoUE8c.d.ts} +3 -3
- package/dist/{researcher-CbCK-ppF.d.ts.map → researcher-DJnoUE8c.d.ts.map} +1 -1
- package/dist/{reward-hacking-DFo2FU5J.js → reward-hacking-DKI9T52l.js} +19 -1
- package/dist/reward-hacking-DKI9T52l.js.map +1 -0
- package/dist/rl.d.ts +2 -2
- package/dist/rl.js +2 -2
- package/dist/{semantic-concept-judge-B7KPD57W.js → semantic-concept-judge-laMCnTLn.js} +2 -2
- package/dist/{semantic-concept-judge-B7KPD57W.js.map → semantic-concept-judge-laMCnTLn.js.map} +1 -1
- package/dist/{series-convergence-VhTad5TM.d.ts → series-convergence-BxKEgBwA.d.ts} +2 -2
- package/dist/{series-convergence-VhTad5TM.d.ts.map → series-convergence-BxKEgBwA.d.ts.map} +1 -1
- package/dist/{server-DNiUVirV.js → server-dIWwF3j_.js} +2 -2
- package/dist/{server-DNiUVirV.js.map → server-dIWwF3j_.js.map} +1 -1
- package/dist/{skillopt-optimization-method-BtgL6PMH.d.ts → skillopt-optimization-method-BDD_o1xE.d.ts} +5 -5
- package/dist/{skillopt-optimization-method-BtgL6PMH.d.ts.map → skillopt-optimization-method-BDD_o1xE.d.ts.map} +1 -1
- package/dist/{skillopt-optimization-method-L8vi4h4m.js → skillopt-optimization-method-DLeUcK-K.js} +5 -5
- package/dist/{skillopt-optimization-method-L8vi4h4m.js.map → skillopt-optimization-method-DLeUcK-K.js.map} +1 -1
- package/dist/{statistical-heldout-Caxd30ZF.d.ts → statistical-heldout-_woZ9q9j.d.ts} +2 -2
- package/dist/{statistical-heldout-Caxd30ZF.d.ts.map → statistical-heldout-_woZ9q9j.d.ts.map} +1 -1
- package/dist/{store-tool-spans-BGg2QFk1.d.ts → store-tool-spans-Br2_IUhm.d.ts} +4 -4
- package/dist/{store-tool-spans-BGg2QFk1.d.ts.map → store-tool-spans-Br2_IUhm.d.ts.map} +1 -1
- package/dist/{tool-groups-BuhFx38z.d.ts → tool-groups-B4tqh8jB.d.ts} +3 -3
- package/dist/tool-groups-B4tqh8jB.d.ts.map +1 -0
- package/dist/trace-repair/index.d.ts +1 -1
- package/dist/traces.d.ts +7 -7
- package/dist/{types-BNbhD_Vj.d.ts → types-B2NsbrNy.d.ts} +2 -2
- package/dist/{types-BNbhD_Vj.d.ts.map → types-B2NsbrNy.d.ts.map} +1 -1
- package/dist/{types-B61iI2ZG.d.ts → types-CLAwnY-L.d.ts} +2 -2
- package/dist/{types-B61iI2ZG.d.ts.map → types-CLAwnY-L.d.ts.map} +1 -1
- package/dist/{types-COEjZMLY.d.ts → types-DdFNuyxQ.d.ts} +3 -3
- package/dist/{types-COEjZMLY.d.ts.map → types-DdFNuyxQ.d.ts.map} +1 -1
- package/dist/{types-COAJy8Ku.d.ts → types-jUBXJ7Iz.d.ts} +36 -3
- package/dist/types-jUBXJ7Iz.d.ts.map +1 -0
- package/dist/wire/index.d.ts +2 -2
- package/dist/wire/index.js +1 -1
- package/docs/campaign-proposers.md +8 -1
- package/package.json +1 -1
- package/dist/campaign-CR6Yt5ZS.js.map +0 -1
- package/dist/external-optimizer-subprocess-BepyNwtj.js.map +0 -1
- package/dist/index-DjpxROrT.d.ts +0 -1
- package/dist/index-dh0kp2GM.d.ts.map +0 -1
- package/dist/llm-client-CxyJ8Oft.js.map +0 -1
- package/dist/reward-hacking-DFo2FU5J.js.map +0 -1
- package/dist/tool-groups-BuhFx38z.d.ts.map +0 -1
- package/dist/types-COAJy8Ku.d.ts.map +0 -1
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { R as Scenario, b as GenerationRecord, p as Gate, w as JudgeScore } from "./types-
|
|
1
|
+
import { R as Scenario, b as GenerationRecord, p as Gate, w as JudgeScore } from "./types-DdFNuyxQ.js";
|
|
2
2
|
import { r as SignedManifest, w as EProcessState } from "./pre-registration-zFSLEiFU.js";
|
|
3
3
|
import { a as PairedPromotionDecision, d as PairedBootstrapResult, i as PairedMcNemarEvidence, r as PairedDecisionStatistic, t as PairedDecisionMethod } from "./paired-promotion-decision-CGzg0cI_.js";
|
|
4
4
|
//#region src/campaign/gates/sequential.d.ts
|
|
@@ -279,4 +279,4 @@ declare function dimensionRegressions(candidate: Map<string, Record<string, Judg
|
|
|
279
279
|
}): DimensionRegression[];
|
|
280
280
|
//#endregion
|
|
281
281
|
export { detectScale as a, pairHoldout as c, SequentialDecision as d, SequentialObservation as f, sequentialPairedGate as g, sequentialDecide as h, PairedHoldout as i, SequentialDecideFn as l, SequentialPairedGateOptions as m, HeldoutSignificance as n, dimensionRegressions as o, SequentialPairedGate as p, HeldoutSignificanceOptions as r, heldoutSignificance as s, DimensionRegression as t, SequentialDecideOptions as u };
|
|
282
|
-
//# sourceMappingURL=statistical-heldout-
|
|
282
|
+
//# sourceMappingURL=statistical-heldout-_woZ9q9j.d.ts.map
|
package/dist/{statistical-heldout-Caxd30ZF.d.ts.map → statistical-heldout-_woZ9q9j.d.ts.map}
RENAMED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"statistical-heldout-
|
|
1
|
+
{"version":3,"file":"statistical-heldout-_woZ9q9j.d.ts","names":[],"sources":["../src/campaign/gates/sequential.ts","../src/campaign/gates/statistical-heldout.ts"],"mappings":";;;;KAuCY;UAEK;EACf,UAAU;;EAEV;;EAEA;;;EAGA;;UAGe;;;EAGf;;;;EAIA;;;EAGA;;EAEA;;;;EAIA;;;EAGA;;;EAGA,kBAAkB;;EAElB;;UAGe,qBAAqB,qBAAqB,kBAAkB,WAAW,kBAC9E,KAAK,WAAW;;;;;;;;EAQxB,QAAQ,gBAAgB;;EAExB,SAAS;IAAkB,UAAU;;;;;;;;;;;;;;;;;iBA0MvB,qBAAqB,qBAAqB,kBAAkB,WAAW,UACrF,SAAS,8BACR,qBAAqB,WAAW;UAsFlB;;EAEf;;EAEA;;EAEA;;EAEA;;UAGe;GACd;IAAQ,SAAS;;IAAyB;IAAe;;;EAE1D,SAAS;;;;;;;;;;;;;;;;;;;;;;;;;iBA0BK,iBAAiB,UAAS,0BAA+B;;;UCtXxD;;EAEf;;EAEA;;EAEA;;;;;;;;;;;;;iBAcc,YACd,WAAW,YAAY,eAAe,cACtC,UAAU,YAAY,eAAe,cACrC,aAAa,aACb,SAAS,GAAG,oCACX;UAsDc;EACf,QAAQ;;;;;;;;;;;;EAYR,WAAW;;;;EAIX,iBAAiB;;;;;;;EAOjB,UAAU;;EAEV,mBAAmB;;EAEnB,SAAS;;;;EAIT;;EAEA;;EAEA;;EAEA,gBAAgB;;EAEhB;;;;EAIA;;EAEA;;UAGe;EACf;EACA;EACA;EACA;;EAEA;EACA;;;;;;;;;;;;;;;;;;;;;;;;;;;;iBA6Bc,oBACd,QAAQ,eACR,OAAM,6BACL;UAkEc;EACf;;;EAGA,WAAW;;;;EAIX;;EAEA;IAAM;IAAa;;;EAEnB,mBAAmB;;EAEnB,SAAS;;EAET;;;;;EAKA;EACA;EACA;;;;;iBAMc,YAAY;;;;;;;;;;;;;;;;;;;iBAsBZ,qBACd,WAAW,YAAY,eAAe,cACtC,UAAU,YAAY,eAAe,cACrC,aAAa,aACb,8BACA;EACE;EACA;EACA;EACA;;;EAGA;IAED"}
|
|
@@ -2,9 +2,9 @@ import { r as CaptureIntegrityError } from "./errors-DEE6u6ot.js";
|
|
|
2
2
|
import { c as CostLedgerHandle } from "./cost-ledger-DbQdN3nO.js";
|
|
3
3
|
import { S as ToolSpan } from "./schema-BtVldJ3T.js";
|
|
4
4
|
import { s as TraceStore } from "./store-CT9YIIve.js";
|
|
5
|
-
import { A as SearchSpanResult, B as ViewSpansResult, D as DatasetOverview, F as TraceAnalystFilters, H as ViewTraceResult, S as BoundedTraceAnalysisStoreOptions, T as TraceAnalysisStoreContext, f as AnalystUsageReceipt, i as AnalystFinding, j as SearchTraceResult, k as QueryTracesPage, w as TraceAnalysisStore } from "./types-
|
|
6
|
-
import { a as TraceAnalystLimits, i as TraceAnalysisEngineResult, n as TraceAnalysisEngine } from "./engine-
|
|
7
|
-
import { t as TraceToolGroupName } from "./tool-groups-
|
|
5
|
+
import { A as SearchSpanResult, B as ViewSpansResult, D as DatasetOverview, F as TraceAnalystFilters, H as ViewTraceResult, S as BoundedTraceAnalysisStoreOptions, T as TraceAnalysisStoreContext, f as AnalystUsageReceipt, i as AnalystFinding, j as SearchTraceResult, k as QueryTracesPage, w as TraceAnalysisStore } from "./types-B2NsbrNy.js";
|
|
6
|
+
import { a as TraceAnalystLimits, i as TraceAnalysisEngineResult, n as TraceAnalysisEngine } from "./engine-BLzhNzoY.js";
|
|
7
|
+
import { t as TraceToolGroupName } from "./tool-groups-B4tqh8jB.js";
|
|
8
8
|
//#region src/trace/otel.d.ts
|
|
9
9
|
declare const OTEL_AGENT_EVAL_SCOPE: {
|
|
10
10
|
name: string;
|
|
@@ -339,4 +339,4 @@ declare class ToolTraceMissingError extends CaptureIntegrityError {
|
|
|
339
339
|
declare function toolSpansToTraceAnalysisStore(spans: readonly ToolSpan[] | null | undefined, options?: ToolSpansToTraceAnalysisStoreOptions): TraceAnalysisStore;
|
|
340
340
|
//#endregion
|
|
341
341
|
export { createBoundedTraceAnalysisStore as A, OtlpSpan as B, scoreTraceInsightReadiness as C, AnalyzeTracesResult as D, AnalyzeTracesOptions as E, redactString as F, redactValue as I, OTEL_AGENT_EVAL_SCOPE as L, REDACTION_VERSION as M, RedactionReport as N, analyzeTraces as O, RedactionRule as P, OtlpExport as R, planTraceInsightQuestions as S, AnalyzeTracesInput as T, exportRunAsOtlp as V, buildTraceInsightPrompt as _, ToolSpansToTraceAnalysisStoreOptions as a, domainEvidencePattern as b, TraceInsightFinding as c, TraceInsightQualityGate as d, TraceInsightQuestion as f, buildTraceInsightContext as g, TraceInsightTask as h, OtlpFileTraceStoreOptions as i, DEFAULT_REDACTION_RULES as j, OtlpFlatLine as k, TraceInsightPanelRole as l, TraceInsightSuite as m, toolSpansToTraceAnalysisStore as n, otlpTextToTraceAnalysisStore as o, TraceInsightReadiness as p, OtlpFileTraceStore as r, TraceInsightContext as s, ToolTraceMissingError as t, TraceInsightPromptInput as u, defaultTraceInsightPanel as v, tokenizeDomainWords as w, inferDomainKeywords as x, describeTraceInsightScope as y, OtlpResourceSpans as z };
|
|
342
|
-
//# sourceMappingURL=store-tool-spans-
|
|
342
|
+
//# sourceMappingURL=store-tool-spans-Br2_IUhm.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"store-tool-spans-
|
|
1
|
+
{"version":3,"file":"store-tool-spans-Br2_IUhm.d.ts","names":[],"sources":["../src/trace/otel.ts","../src/trace/redact.ts","../src/trace-analyst/store-boundary.ts","../src/trace/otlp-flat.ts","../src/trace-analyst/analyst.ts","../src/trace-analyst/insights.ts","../src/trace-analyst/store-otlp.ts","../src/trace-analyst/store-tool-spans.ts"],"mappings":";;;;;;;;cAqBa;EAA0B;EAAoC;;UAE1D;EACf;EACA;EACA;EACA;EACA;EACA;EACA;EACA,YAAY;IACV;IACA;MAAS;MAAsB;MAAmB;MAAsB;;;EAE1E,SAAS;IAAQ;IAAsB;IAAc,aAAa;;EAClE;IAAW;IAAc;;;UAGV;EACf;IAAY,YAAY;;EACxB,YAAY;IAAQ,cAAc;IAAuB,OAAO;;;UAGjD;EACf,eAAe;;;iBAIK,gBACpB,OAAO,YACP,eACA,gBAAe,4CACd,QAAQ;;;;;;;;;;;;;;;;UCvCM;EACf;EACA,SAAS;;EAET;;UAGe;EACf;EACA,QAAQ;;;cAIG,yBAAyB;cAezB;;;;;iBAMG,aACd,eACA,QAAO;EACJ;EAAgB,QAAQ;;;;;;;iBAuBb,YACd,gBACA,QAAO,iBACP,SAAQ;EACL;EAAgB,QAAQ;;;;;iBCjDb,gCACd,QAAQ,oBACR,UAAS,mCACR;;;KC5BS;UAEK;EACf;EACA;EACA;EACA;EACA;EACA;EACA;EACA;IACE,MAAM;IACN;;EAEF;IAAY,YAAY;;EACxB,YAAY;EACZ,SAAS;IAAQ;IAAc;IAAuB,aAAa;;;;;UCNpD;EACf;EACA;EACA;EACA;;KAGU,sBAAsB;UAEjB;;EAEf,iBAAiB;;EAEjB,QAAQ;;EAER;;EAEA,YAAY;EACZ,SAAS,QAAQ;EACjB;EACA;EACA,aAAa;EACb;EACA,yBAAyB;EACzB,4BAA4B;EAC5B,eAAe,SAAS;EACxB,OAAO;EACP,OAAO,iBAAiB,SAAS;EACjC,SAAS;;;;;;;;iBASW,cACpB,OAAO,oBACP,SAAS,uBACR,QAAQ;;;UCtDM;EACf;EACA;EACA;EACA;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA,OAAO;;UAGQ;EACf;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA;;UAGe;EACf,OAAO;EACP,WAAW;EACX,QAAQ;EACR,SAAS;EACT;;UAGe;EACf,OAAO;EACP;EACA;EACA,WAAW;EACX,OAAO;EACP,UAAU;EACV,OAAO;EACP,QAAQ;;UAGO;EACf;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA,OAAO;;iBA0BO,oBAAoB;iBAMpB,oBAAoB,OAAO;iBAuB3B,sBAAsB,qBAAqB;iBAS3C,0BAA0B,OAAO;iBAsBjC,0BAA0B,OAAO,0BAA0B;iBAsD3D,yBAAyB,OAAO,0BAA0B;iBAa1D,2BAA2B,SAAS,sBAAsB;iBA2D1D,4BAA4B;iBA6B5B,wBAAwB,OAAO;;;UC1K9B;;EAEf;;EAEA;;EAEA;;EAEA;;KAGG,gCAAgC;UAEpB,kCAAkC;;EAEjD;;;;;;;;EAQA;;uBAMa,kCAAkC;mBAC9B;mBACA;mBACA;mBACA;UACT;UACA;UACA;UACA;EAER,YAAY,MAAM;EAyBZ,SAAS,kBAAkB,UAAU,4BAA4B;EAOjE,SACJ;IAAQ;IAAkB;KAC1B,UAAU,4BACT;EAeG,YACJ,UAAU,qBACV,UAAU,4BACT,QAAQ;EAoDL,YACJ;IAAQ,UAAU;IAAqB;IAAe;KACtD,UAAU,4BACT,QAAQ;EAmBL,YACJ,UAAU,qBACV,UAAU,4BACT;EAMG,UACJ;IAAQ;IAAkB;KAC1B,UAAU,4BACT,QAAQ;EAuCL,UACJ;IAAQ;IAAkB;IAA6B;KACvD,UAAU,4BACT,QAAQ;EAsFL,YACJ;IAAQ;IAAkB;IAAuB;KACjD,UAAU,4BACT,QAAQ;EA4CL,WACJ;IAAQ;IAAkB;IAAiB;IAAuB;KAClE,UAAU,4BACT,QAAQ;;;EA6CL,cAAc,UAAU,4BAA4B;UAI5C;qBAmBK,WAAW,SAAS,cAAc,QAAQ;qBAE1C;UAEL;UAmBA;UAsHA;UA+DN;UAkBA;UAwDA;UAmBM;;cA+CH,2BAA2B;mBACrB;mBACA;EAEjB,YAAY,MAAM;;;YAYF,WAAW,SAAS,cAAc,QAAQ;YAkBhD;;;iBAwBI,6BACd,cACA,OAAM,uCACL;;;;cC/7BU,8BAA8B;EACzC;;;;;;iBASc,8BACd,gBAAgB,+BAChB,UAAS,uCACR"}
|
|
@@ -1,5 +1,5 @@
|
|
|
1
|
-
import { w as TraceAnalysisStore } from "./types-
|
|
2
|
-
import { _ as TraceAnalysisToolDescriptor } from "./engine-
|
|
1
|
+
import { w as TraceAnalysisStore } from "./types-B2NsbrNy.js";
|
|
2
|
+
import { _ as TraceAnalysisToolDescriptor } from "./engine-BLzhNzoY.js";
|
|
3
3
|
//#region src/analyst/tool-groups.d.ts
|
|
4
4
|
/** Named tool sets. Kinds pass `tools: TRACE_TOOL_GROUPS.failureForensics` etc. */
|
|
5
5
|
type TraceToolGroupName =
|
|
@@ -25,4 +25,4 @@ type TraceToolGroupName =
|
|
|
25
25
|
declare function buildTraceToolsForGroup(group: TraceToolGroupName, store: TraceAnalysisStore): TraceAnalysisToolDescriptor[];
|
|
26
26
|
//#endregion
|
|
27
27
|
export { buildTraceToolsForGroup as n, TraceToolGroupName as t };
|
|
28
|
-
//# sourceMappingURL=tool-groups-
|
|
28
|
+
//# sourceMappingURL=tool-groups-B4tqh8jB.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"tool-groups-B4tqh8jB.d.ts","names":[],"sources":["../src/analyst/tool-groups.ts"],"mappings":";;;;KAmBY;;;;;;;;;;;;;;;;;;;;iBAgDI,wBACd,OAAO,oBACP,OAAO,qBACN"}
|
|
@@ -2,7 +2,7 @@ import { r as CaptureIntegrityError } from "../errors-DEE6u6ot.js";
|
|
|
2
2
|
import { t as DefaultVerdict } from "../verdict-E4eRNf7-.js";
|
|
3
3
|
import { b as CustomTokenPricing, c as CostLedgerHandle, p as CostProvenance } from "../cost-ledger-DbQdN3nO.js";
|
|
4
4
|
import { u as RunTokenUsage } from "../run-record-DVV82Gwh.js";
|
|
5
|
-
import { a as TraceAnalystLimits, n as TraceAnalysisEngine } from "../engine-
|
|
5
|
+
import { a as TraceAnalystLimits, n as TraceAnalysisEngine } from "../engine-BLzhNzoY.js";
|
|
6
6
|
import { t as PrimeBridgeTransport } from "../prime-bridge-transport-6feEglLf.js";
|
|
7
7
|
import { g as isRecordedTimeout, o as RecordedTrajectoryStep } from "../steps-AmkT-GIM.js";
|
|
8
8
|
//#region src/trace-repair/mini-swe-scaffold.d.ts
|
package/dist/traces.d.ts
CHANGED
|
@@ -1,17 +1,17 @@
|
|
|
1
1
|
import { c as ValidationError, o as LimitExceededError, r as CaptureIntegrityError, s as NotFoundError, t as AgentEvalError } from "./errors-DEE6u6ot.js";
|
|
2
2
|
import { C as TraceEvent, D as isSandboxSpan, E as isRetrievalSpan, O as isToolSpan, S as ToolSpan, T as isLlmSpan, _ as Span, a as FAILURE_CLASSES, b as SpanStatus, c as JudgeSpan, d as RetrievalSpan, f as Run, g as SandboxSpan, h as RunStatus, i as EventKind, l as LlmSpan, m as RunOutcome, n as BudgetLedgerEntry, o as FailureClass, p as RunLayer, r as BudgetSpec, s as GenericSpan, t as Artifact, u as Message, v as SpanBase, w as isJudgeSpan, x as TRACE_SCHEMA_VERSION, y as SpanKind } from "./schema-BtVldJ3T.js";
|
|
3
3
|
import { a as RunRecord, l as RunTerminalOutcome, s as RunSplitTag, u as RunTokenUsage } from "./run-record-DVV82Gwh.js";
|
|
4
|
-
import { a as ExtractedUsage, c as extractUsageFromResponse, i as ExtractUsageFromSseOptions, l as extractUsageFromSse, n as CaptureFetchOptions, o as SseUsageMode, r as captureFetchToRawSink, s as extractUsage, t as CaptureFetchContext } from "./capture-fetch-
|
|
5
|
-
import { _t as defaultProviderRedactor, ct as FileSystemRawProviderSinkOptions, dt as NoopRawProviderSink, ft as ProviderRedactor, gt as RawProviderSinkFilter, ht as RawProviderSink, lt as InMemoryRawProviderSink, mt as RawProviderEvent, pt as RawProviderDirection, st as FileSystemRawProviderSink, ut as InMemoryRawProviderSinkOptions, vt as providerFromBaseUrl } from "./types-
|
|
4
|
+
import { a as ExtractedUsage, c as extractUsageFromResponse, i as ExtractUsageFromSseOptions, l as extractUsageFromSse, n as CaptureFetchOptions, o as SseUsageMode, r as captureFetchToRawSink, s as extractUsage, t as CaptureFetchContext } from "./capture-fetch-BBVFzhkk.js";
|
|
5
|
+
import { _t as defaultProviderRedactor, ct as FileSystemRawProviderSinkOptions, dt as NoopRawProviderSink, ft as ProviderRedactor, gt as RawProviderSinkFilter, ht as RawProviderSink, lt as InMemoryRawProviderSink, mt as RawProviderEvent, pt as RawProviderDirection, st as FileSystemRawProviderSink, ut as InMemoryRawProviderSinkOptions, vt as providerFromBaseUrl } from "./types-jUBXJ7Iz.js";
|
|
6
6
|
import { a as RunFilter, i as InMemoryTraceStore, n as FileSystemTraceStore, o as SpanFilter, r as FileSystemTraceStoreOptions, s as TraceStore, t as EventFilter } from "./store-CT9YIIve.js";
|
|
7
7
|
import { a as TraceEmitterOptions, i as TraceEmitter, n as RunCompleteHookContext, o as llmSpanFromProvider, r as SpanHandle, t as RunCompleteHook } from "./emitter-DGQGoLyj.js";
|
|
8
|
-
import { a as RunIntegrityReport, i as RunIntegrityIssueCode, n as RunIntegrityExpectations, o as assertRunCaptured, r as RunIntegrityIssue, s as throwIfRunIncomplete, t as RunIntegrityError } from "./integrity-
|
|
9
|
-
import { A as createBoundedTraceAnalysisStore, B as OtlpSpan, C as scoreTraceInsightReadiness, D as AnalyzeTracesResult, E as AnalyzeTracesOptions, F as redactString, I as redactValue, L as OTEL_AGENT_EVAL_SCOPE, M as REDACTION_VERSION, N as RedactionReport, O as analyzeTraces, P as RedactionRule, R as OtlpExport, S as planTraceInsightQuestions, T as AnalyzeTracesInput, V as exportRunAsOtlp, _ as buildTraceInsightPrompt, a as ToolSpansToTraceAnalysisStoreOptions, b as domainEvidencePattern, c as TraceInsightFinding, d as TraceInsightQualityGate, f as TraceInsightQuestion, g as buildTraceInsightContext, h as TraceInsightTask, i as OtlpFileTraceStoreOptions, j as DEFAULT_REDACTION_RULES, k as OtlpFlatLine, l as TraceInsightPanelRole, m as TraceInsightSuite, n as toolSpansToTraceAnalysisStore, o as otlpTextToTraceAnalysisStore, p as TraceInsightReadiness, r as OtlpFileTraceStore, s as TraceInsightContext, t as ToolTraceMissingError, u as TraceInsightPromptInput, v as defaultTraceInsightPanel, w as tokenizeDomainWords, x as inferDomainKeywords, y as describeTraceInsightScope, z as OtlpResourceSpans } from "./store-tool-spans-
|
|
8
|
+
import { a as RunIntegrityReport, i as RunIntegrityIssueCode, n as RunIntegrityExpectations, o as assertRunCaptured, r as RunIntegrityIssue, s as throwIfRunIncomplete, t as RunIntegrityError } from "./integrity-B0dZ96EO.js";
|
|
9
|
+
import { A as createBoundedTraceAnalysisStore, B as OtlpSpan, C as scoreTraceInsightReadiness, D as AnalyzeTracesResult, E as AnalyzeTracesOptions, F as redactString, I as redactValue, L as OTEL_AGENT_EVAL_SCOPE, M as REDACTION_VERSION, N as RedactionReport, O as analyzeTraces, P as RedactionRule, R as OtlpExport, S as planTraceInsightQuestions, T as AnalyzeTracesInput, V as exportRunAsOtlp, _ as buildTraceInsightPrompt, a as ToolSpansToTraceAnalysisStoreOptions, b as domainEvidencePattern, c as TraceInsightFinding, d as TraceInsightQualityGate, f as TraceInsightQuestion, g as buildTraceInsightContext, h as TraceInsightTask, i as OtlpFileTraceStoreOptions, j as DEFAULT_REDACTION_RULES, k as OtlpFlatLine, l as TraceInsightPanelRole, m as TraceInsightSuite, n as toolSpansToTraceAnalysisStore, o as otlpTextToTraceAnalysisStore, p as TraceInsightReadiness, r as OtlpFileTraceStore, s as TraceInsightContext, t as ToolTraceMissingError, u as TraceInsightPromptInput, v as defaultTraceInsightPanel, w as tokenizeDomainWords, x as inferDomainKeywords, y as describeTraceInsightScope, z as OtlpResourceSpans } from "./store-tool-spans-Br2_IUhm.js";
|
|
10
10
|
import { C as TOOL_LATENCY_MS, D as asNumber, E as applyLlmSpanOtlpAttributes, O as contextInputTokens, S as TOOL_ARGS_CAPTURED, T as TOOL_NAME_ATTR_KEYS, _ as LlmSpanOtlpInput, a as LLM_CACHE_WRITE_TOKEN_ATTR_KEYS, b as RUN_COST_ATTR_KEYS, c as LLM_COST_USD, d as LLM_MODEL_ATTR_KEYS, f as LLM_MODEL_NAME, g as LLM_REASONING_TOKEN_ATTR_KEYS, h as LLM_REASONING_TOKENS, i as LLM_CACHE_WRITE_TOKENS, k as firstNumberAttr, l as LLM_INPUT_TOKENS, m as LLM_OUTPUT_TOKEN_ATTR_KEYS, n as LLM_CACHED_TOKENS, o as LLM_CONTEXT_TOKENS, p as LLM_OUTPUT_TOKENS, r as LLM_CACHED_TOKEN_ATTR_KEYS, s as LLM_COST_ATTR_KEYS, t as INPUT_VALUE, u as LLM_INPUT_TOKEN_ATTR_KEYS, v as OPENINFERENCE_SPAN_KIND, w as TOOL_NAME, x as SPAN_KIND_ATTR_KEYS, y as OUTPUT_VALUE } from "./attribute-vocabulary-DLJ6303h.js";
|
|
11
11
|
import { a as judgeSpans, c as runsForScenario, i as hasCapturedToolArgs, l as toolSpans, n as argHash, o as llmSpans, r as groupBy, s as runFailureClass, t as aggregateLlm } from "./query-CJ_DX8vl.js";
|
|
12
|
-
import "./index-
|
|
13
|
-
import { A as SearchSpanResult, B as ViewSpansResult, C as TRACE_ANALYSIS_LIMITS, D as DatasetOverview, E as DEFAULT_TRACE_ANALYST_BUDGETS, F as TraceAnalystFilters, H as ViewTraceResult, I as TraceAnalystSpan, L as TraceAnalystSpanKind, M as SpanMatchRecord, N as TRACE_ANALYST_TRUNCATION_MARKER_PREFIX, O as ErrorCluster, P as TraceAnalystByteBudgets, R as TraceAnalystSpanStatus, S as BoundedTraceAnalysisStoreOptions, T as TraceAnalysisStoreContext, V as ViewTraceOversized, j as SearchTraceResult, k as QueryTracesPage, w as TraceAnalysisStore, z as TraceAnalystTraceSummary } from "./types-
|
|
14
|
-
import { _ as TraceAnalysisToolDescriptor, g as TRACE_ANALYST_TOOL_NAMESPACE, h as BuildTraceAnalysisToolsOptions, v as buildTraceAnalysisToolDescriptors, y as traceAnalystFunctionGroup } from "./engine-
|
|
12
|
+
import "./index-C1ravkGA.js";
|
|
13
|
+
import { A as SearchSpanResult, B as ViewSpansResult, C as TRACE_ANALYSIS_LIMITS, D as DatasetOverview, E as DEFAULT_TRACE_ANALYST_BUDGETS, F as TraceAnalystFilters, H as ViewTraceResult, I as TraceAnalystSpan, L as TraceAnalystSpanKind, M as SpanMatchRecord, N as TRACE_ANALYST_TRUNCATION_MARKER_PREFIX, O as ErrorCluster, P as TraceAnalystByteBudgets, R as TraceAnalystSpanStatus, S as BoundedTraceAnalysisStoreOptions, T as TraceAnalysisStoreContext, V as ViewTraceOversized, j as SearchTraceResult, k as QueryTracesPage, w as TraceAnalysisStore, z as TraceAnalystTraceSummary } from "./types-B2NsbrNy.js";
|
|
14
|
+
import { _ as TraceAnalysisToolDescriptor, g as TRACE_ANALYST_TOOL_NAMESPACE, h as BuildTraceAnalysisToolsOptions, v as buildTraceAnalysisToolDescriptors, y as traceAnalystFunctionGroup } from "./engine-BLzhNzoY.js";
|
|
15
15
|
//#region src/trace/otlp-attributes.d.ts
|
|
16
16
|
type ToolSpanOtlpInput = Pick<ToolSpan, 'toolName' | 'args' | 'argsCaptured' | 'result' | 'latencyMs'>;
|
|
17
17
|
type OtlpSpanRole = 'AGENT' | 'CHAIN' | 'EVALUATOR' | 'GUARDRAIL' | 'LLM' | 'SPAN' | 'TOOL' | 'UNKNOWN';
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { c as CostLedgerHandle } from "./cost-ledger-DbQdN3nO.js";
|
|
2
2
|
import { a as RunRecord, n as RunCostProvenance, u as RunTokenUsage } from "./run-record-DVV82Gwh.js";
|
|
3
|
-
import { a as JudgeInput, p as ChatClient } from "./types-
|
|
3
|
+
import { a as JudgeInput, p as ChatClient } from "./types-jUBXJ7Iz.js";
|
|
4
4
|
import { RE2JS } from "re2js";
|
|
5
5
|
//#region src/trace-analyst/types.d.ts
|
|
6
6
|
/**
|
|
@@ -594,4 +594,4 @@ type AnalystRunEvent = {
|
|
|
594
594
|
};
|
|
595
595
|
//#endregion
|
|
596
596
|
export { SearchSpanResult as A, ViewSpansResult as B, TRACE_ANALYSIS_LIMITS as C, DatasetOverview as D, DEFAULT_TRACE_ANALYST_BUDGETS as E, TraceAnalystFilters as F, ViewTraceResult as H, TraceAnalystSpan as I, TraceAnalystSpanKind as L, SpanMatchRecord as M, TRACE_ANALYST_TRUNCATION_MARKER_PREFIX as N, ErrorCluster as O, TraceAnalystByteBudgets as P, TraceAnalystSpanStatus as R, BoundedTraceAnalysisStoreOptions as S, TraceAnalysisStoreContext as T, ViewTraceOversized as V, ProposalFinding as _, AnalystInputKind as a, makeFinding as b, AnalystRunInputs as c, AnalystSeverity as d, AnalystUsageReceipt as f, ExecutionProbeRequest as g, ExecutionProbeOutcome as h, AnalystFinding as i, SearchTraceResult as j, QueryTracesPage as k, AnalystRunResult as l, ExecutionProbe as m, AnalystContext as n, AnalystRequirements as o, EvidenceRef as p, AnalystCost as r, AnalystRunEvent as s, Analyst as t, AnalystRunSummary as u, ProposalFindingOrigin as v, TraceAnalysisStore as w, makeProposalFinding as x, computeFindingId as y, TraceAnalystTraceSummary as z };
|
|
597
|
-
//# sourceMappingURL=types-
|
|
597
|
+
//# sourceMappingURL=types-B2NsbrNy.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types-
|
|
1
|
+
{"version":3,"file":"types-B2NsbrNy.d.ts","names":[],"sources":["../src/trace-analyst/types.ts","../src/trace-analyst/store-contract.ts","../src/analyst/types.ts"],"mappings":";;;;;;;;;;;;;;;;;;;;KAgBY;KAUA;;;;UAKK;EACf;EACA;EACA;EACA;EACA,MAAM;EACN;EACA;EACA;EACA,QAAQ;EACR;EACA;EACA;EACA;EACA;;;EAGA,YAAY;;UAGG;EACf;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;;UAGe;;EAEf;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA;;;EAGA;;;;;;;UAQe;;EAEf;;EAEA;;EAEA;;EAEA;EACA;EACA;;EAEA;;EAEA;;EAEA;;UAGe;EACf;EACA;EACA;EACA;EACA;EACA;;EAEA;EACA;IAAU;IAAqB;;;;;;EAK/B,gBAAgB;EAChB;IAAc;IAAkB;;;UAGjB;EACf,QAAQ;EACR;EACA;;;;;UAMe;EACf;EACA,QAAQ;EACR,YAAY;;UAGG;EACf;;EAEA,gBAAgB;;EAEhB;EACA;;UAGe;EACf;EACA,OAAO;;EAEP;;EAEA;;EAEA;;EAEA;;UAGe;EACf;EACA;EACA;EACA,WAAW;;;EAGX;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA,MAAM;EACN;;UAGe;EACf;EACA;EACA,MAAM;EACN;;;UAIe;;;EAGf;;;EAGA;;;EAGA;;;EAGA;;cAGW,+BAA+B;;;cAS/B;;;cC7MA;WACX;WACA;WACA;WACA;WACA;WACA;WACA;WACA;;UAGe;EACf,SAAS;;;;;;;;;UAUM;EACf,SAAS,kBAAkB,UAAU,4BAA4B;EAEjE,SACE;IAAS;IAAkB;KAC3B,UAAU,4BACT;EAEH,YACE,UAAU,qBACV,UAAU,4BACT,QAAQ;EAEX,YACE;IAAS,UAAU;IAAqB;IAAe;KACvD,UAAU,4BACT,QAAQ;EAEX,YAAY,UAAU,qBAAqB,UAAU,4BAA4B;EAEjF,UACE;IACE;IACA;KAEF,UAAU,4BACT,QAAQ;EAEX,UACE;IACE;IACA;IACA;KAEF,UAAU,4BACT,QAAQ;EAEX,YACE;IACE;IACA;IACA;KAEF,UAAU,4BACT,QAAQ;EAEX,WACE;IACE;IACA;IACA;IACA;KAEF,UAAU,4BACT,QAAQ;;UAGI;EACf,UAAU,QAAQ;;;;;;;;UC/DH;EACf;;;;;;;EAOA;EACA;EACA;EACA,UAAU;;;;;;;EAOV;EACA;EACA;EACA,eAAe;EACf;EACA;;EAEA;;;;;;EAMA;;;;EAIA;;EAEA,WAAW;;KAGD;;KAGA;;KAGA,kBAAkB;WACnB,iBAAiB;;UAGX;;;;;;;EAOf;EACA;EACA;;;;;;;;KAWU;UAOK;;EAEf;;EAEA;;EAEA;;EAEA;;UAGe;;EAEf;;EAEA;;;;;;UAOe;EACf,aAAa;EACb;EACA,YAAY;EACZ,aAAa;;EAEb,SAAS;;UAGM;EACf;;EAEA;;EAEA;;EAEA;;EAEA,aAAa;;EAEb;;;;;;;EAOA,OAAO;;;;;;;;;;EAUP,gBAAgB,cAAc;;;;;;;EAO9B,mBAAmB,cAAc;;;;;EAKjC,eAAe,SAAS;;EAExB,OAAO;;EAEP,OAAO,aAAa,SAAS;;EAE7B,SAAS;;;;;;;;EAQT,QAAQ;;;UAMO;EACf;;EAEA;;EAEA;;EAEA;EACA,SAAS;;;;;;;KAQC;EAEN;EACA;EACA;EACA;EACA;;EAEA;;EAEA;EAAkB;IAAS;IAAe;;;;;;;;UAO/B;;WAEN;EACT,QAAQ,SAAS,wBAAwB,QAAQ;;;;;;;;UASlC,QAAQ;;WAEd;;WAEA;WACA,WAAW;WACX,MAAM;WACN,WAAW;;WAEX;EACT,QAAQ,OAAO,QAAQ,KAAK,iBAAiB,QAAQ;;;UAItC;;EAEf;;EAEA,QAAQ;;EAER,MAAM;;EAEN;;;;;;;;EAQA;IAAkB;IAAsB;;;;;;;;EAOxC;;;;;;;;;;iBAac,iBAAiB;EAC/B;EACA;EACA;EACA;;EAEA;;;;;;iBAyBc,YACd,MAAM,KAAK;EACT;EACA;IAED;;iBAiBa,oBACd,MAAM,KAAK;EACT;EACA;IAED;UAOc;EACf;EACA;;EAEA;EACA;EACA;;EAEA,OAAO;;EAEP;IAAU;IAAe;;;UAGV;EACf;EACA;EACA;EACA;EACA,UAAU;EACV,aAAa;;EAEb;;;;;EAKA,wBAAwB;;;;;;;;;;;;;;;KAkBd;EAEN;EACA;EACA;EACA;;EAEA,aAAa;;EAGb;EACA,SAAS;;EAGT;EACA;EACA;;EAGA;;EAEA,SAAS;EACT,UAAU,cAAc;;EAGxB;EACA,QAAQ"}
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { s as RunSplitTag } from "./run-record-DVV82Gwh.js";
|
|
2
|
-
import { R as Scenario, d as DispatchContext } from "./types-
|
|
2
|
+
import { R as Scenario, d as DispatchContext } from "./types-DdFNuyxQ.js";
|
|
3
3
|
//#region src/benchmarks/types.d.ts
|
|
4
4
|
type BenchmarkTaskKind = 'retrieval' | 'rag-answer' | 'hallucination' | 'kb-improvement' | 'routing' | 'custom';
|
|
5
5
|
type BenchmarkFamily = 'beir' | 'mteb-retrieval' | 'msmarco' | 'trec-dl' | 'miracl' | 'lotte' | 'bright' | 'crag' | 'hotpotqa' | 'kilt' | 'ragtruth' | 'faithbench' | 'first-party' | 'custom';
|
|
@@ -90,4 +90,4 @@ declare const BENCHMARK_SPLIT_SEED = "agent-eval-v1";
|
|
|
90
90
|
declare function deterministicSplit(itemId: string, seed?: string): RunSplitTag;
|
|
91
91
|
//#endregion
|
|
92
92
|
export { BenchmarkFamily as a, BenchmarkSource as c, BenchmarkEvaluation as i, BenchmarkTaskKind as l, BenchmarkAdapter as n, BenchmarkResponder as o, BenchmarkDatasetItem as r, BenchmarkScenario as s, BENCHMARK_SPLIT_SEED as t, deterministicSplit as u };
|
|
93
|
-
//# sourceMappingURL=types-
|
|
93
|
+
//# sourceMappingURL=types-CLAwnY-L.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types-
|
|
1
|
+
{"version":3,"file":"types-CLAwnY-L.d.ts","names":[],"sources":["../src/benchmarks/types.ts"],"mappings":";;;KASY;KAQA;UAgBK,qBAAqB;;;EAGpC;;EAEA,SAAS;;EAET,QAAQ;;EAER,SAAS;;EAET,WAAW;;EAEX;;EAEA,SAAS;EACT,WAAW;;UAGI;;;;EAIf;;EAEA;;EAEA,aAAa;;;EAGb,KAAK;EACL;;UAGe;EACf;EACA;EACA;EACA;EACA;;;UAQe,iBAAiB,kBAAkB,oBAAoB;;EAEtE;EACA,SAAS;EACT,WAAW;EACX;EACA,SAAS;EACT;;;;;EAKA,YAAY,OAAO,cAAc,QAAQ,qBAAqB;;EAE9D,SAAS,MAAM,qBAAqB,WAAW,UAAU,YAAY,QAAQ;;;;EAI7E,YAAY,iBAAiB;;UAGd,kBAAkB,4BAA4B;EAC7D;EACA;EACA,QAAQ;EACR,UAAU;EACV,UAAU;EACV,MAAM,qBAAqB;;KAGjB,mBAAmB,oBAAoB,uBAAuB;EACxE,UAAU,kBAAkB;EAC5B,MAAM,qBAAqB;EAC3B,SAAS;MACL,QAAQ,aAAa;;;cAoBd;;;;;;;;;iBAUG,mBACd,gBACA,gBACC"}
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { S as PaidCallResult, T as RunPaidCallInput, a as CostChannel, c as CostLedgerHandle, f as CostLedgerSummary, p as CostProvenance } from "./cost-ledger-DbQdN3nO.js";
|
|
2
2
|
import { u as RunTokenUsage } from "./run-record-DVV82Gwh.js";
|
|
3
|
-
import { E as LlmCallMetadata } from "./types-
|
|
4
|
-
import { _ as ProposalFinding } from "./types-
|
|
3
|
+
import { E as LlmCallMetadata } from "./types-jUBXJ7Iz.js";
|
|
4
|
+
import { _ as ProposalFinding } from "./types-B2NsbrNy.js";
|
|
5
5
|
//#region src/campaign/types.d.ts
|
|
6
6
|
/** Stable identifier + kind tag for any scenario. Consumers
|
|
7
7
|
* extend with their per-domain payload (persona, task, requirement, ...). */
|
|
@@ -626,4 +626,4 @@ interface CampaignResult<TArtifact = unknown, TScenario extends Scenario = Scena
|
|
|
626
626
|
}
|
|
627
627
|
//#endregion
|
|
628
628
|
export { LabeledScenarioWrite as A, ScoredSurfaceOutcome as B, JudgeDimension as C, LabeledScenarioSampleArgs as D, LabeledScenarioRecord as E, ProposeContext as F, labelTrustRank as G, SurfaceProposer as H, ProposedCandidate as I, RedactionStatus as L, OptimizerConfig as M, ParetoParent as N, LabeledScenarioSource as O, ProposalTrackContext as P, Scenario as R, JudgeConfig as S, LabelTrust as T, TraceSpan as U, SessionScript as V, isProposedCandidate as W, GateDecision as _, CampaignResult as a, GenerationRecord as b, CampaignTraceWriter as c, DispatchContext as d, DispatchFn as f, GateContribution as g, GateContext as h, CampaignCostMeter as i, MutableSurface as j, LabeledScenarioStore as k, CodeSurface as l, GateCheckStatus as m, CampaignArtifactWriter as n, CampaignScenarioIdentity as o, Gate as p, CampaignCellResult as r, CampaignTokenUsage as s, CampaignAggregates as t, ComponentSurface as u, GateResult as v, JudgeScore as w, JudgeAggregate as x, GenerationCandidate as y, ScenarioAggregate as z };
|
|
629
|
-
//# sourceMappingURL=types-
|
|
629
|
+
//# sourceMappingURL=types-DdFNuyxQ.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types-
|
|
1
|
+
{"version":3,"file":"types-DdFNuyxQ.d.ts","names":[],"sources":["../src/campaign/types.ts"],"mappings":";;;;;;;UAgCiB;EACf;EACA;EACA;;;;;EAKA;;;UAIe,iCAAiC,KAAK;EACrD;;;;;UAMe;EACf;;EAEA;EACA;EACA;EACA;EACA,QAAQ;EACR,OAAO;EACP,WAAW;EACX,MAAM;;EAEN;;EAEA;;;;;;;;EAQA;;;;KAKU,WAAW,kBAAkB,UAAU,cACjD,UAAU,WACV,KAAK,oBACF,QAAQ;;;;UAOI,cAAc,WAAW;EACxC;EACA;EACA;;EAEA;;;EAGA,sBAAsB,UAAU,WAAW,sBAAsB,UAAU,cAAc;;UAK1E;;EAEf;;EAEA;;;;;;;;;UAUe,YAAY,WAAW,kBAAkB,WAAW;EACnE;EACA,YAAY;;;;EAIZ;;;EAGA,MAAM;IACJ,UAAU;IACV,UAAU;IACV,QAAQ;;IAER,aAAa;IACb;IACA,WAAW;MACT,aAAa,QAAQ;EACzB,aAAa,UAAU;;;;;;;;;;;UAYR;EACf,YAAY;EACZ;EACA;;EAEA,UAAU;;;;EAIV;;;EAGA;;EAEA;;EAEA,WAAW,eAAe;;;;;;;UAUX;WACN;;;WAGA;;WAEA;;WAEA;;WAEA;;WAEA;;WAEA;;;WAGA;aACE;aACA;aACA;;;WAGF;;;UAIM;WACN;WACA,YAAY,SAAS;;;;;;;;;;;KAYpB,0BAA0B,mBAAmB;;;;;;;UAQxC;EACf,SAAS;;EAET;;;;EAIA;;;;iBAKc,oBACd,OAAO,iBAAiB,oBACvB,SAAS;;;;;;;;;UAkBK;EACf,SAAS;EACT;;;EAGA,YAAY;;;EAGZ;;EAEA;EACA;EACA;;;UAIe;EACf;EACA;EACA;EACA;EACA;;;;;UAMe;;;EAGf;;EAEA;EACA;EACA;EACA,YAAY;;;;;EAKZ,WAAW;IAAQ;IAAoB;IAAmB;IAAgB;;EAC1E;IACE;IACA;;;;;UAMa,eAAe,YAAY;WACjC,gBAAgB;WAChB,SAAS,cAAc;WACvB,UAAU,cAAc;;WAExB;WACA;WACA,QAAQ;;WAER,QAAQ;;;WAGR,kBAAkB;;;WAGlB,mBAAmB;;;;WAInB;;;;;;;WAOA,gBAAgB,cAAc;;WAE9B,aAAa;WACb;;;;;;;;;;;;;;UAeM,gBAAgB,YAAY;EAC3C;;;;;EAKA,QAAQ,KAAK,eAAe,aAAa,QAAQ,MAAM,iBAAiB;;;EAGxE,QAAQ;IAAQ,SAAS,cAAc;;IAAwB;IAAe;;;UAG/D;EACf;EACA;EACA,mBAAmB,qBAAqB;;UAGzB,wBAAwB;EACvC,UAAU;;;KAMA;;KAGA;UAEK;EACf;EACA,QAAQ;EACR;;UAGe,YAAY,WAAW,kBAAkB;EACxD,oBAAoB,YAAY;EAChC,oBAAoB,YAAY;;EAEhC,aAAa,YAAY,eAAe;;;;;EAKxC,sBAAsB,YAAY,eAAe;;;;;;;EAOjD,yBAAyB,YAAY,eAAe;;;EAGpD,uBAAuB,YAAY;EACnC,WAAW;EACX;IAAQ;IAAmB;;;EAE3B,aAAa;EACb;EACA,QAAQ;;UAGO;EACf,UAAU;EACV;EACA,mBAAmB;EACnB;;;UAIe,KAAK,qBAAqB,kBAAkB,WAAW;EACtE;EACA,OAAO,KAAK,YAAY,WAAW,aAAa,QAAQ;;;;UAOzC;EACf,KAAK,cAAc,aAAa,0BAA0B;EAC1D,SAAS;;UAGM;EACf,IAAI,aAAa;EACjB,aAAa,aAAa;;;;UAKX;EACf,MAAM,cAAc,kBAAkB,aAAa;EACnD,UAAU,cAAc,iBAAiB;;;;;;KAO/B,qBAAqB;;;;;UAMhB;;EAEf,YAAY,GACV,OAAO,KAAK,iBAAiB;IAC3B,UAAU;MAEX,QAAQ,eAAe;;;;;KAQhB;KAOA;;;;;;;;;;;;;;;KAgBA;;iBASI,eAAe,OAAO;;;;UAOrB,qBAAqB,kBAAkB,WAAW,UAAU;EAC3E,UAAU;EACV,UAAU;EACV,aAAa,eAAe;EAC5B,QAAQ;EACR;EACA;EACA,iBAAiB;;;;;EAKjB,aAAa;;EAEb;;UAGe,sBAAsB,kBAAkB,WAAW,UAAU,6BACpE,qBAAqB,WAAW;;EAExC;;;EAGA;;UAGe;EACf;;EAEA;;;;EAIA;EACA;IACE;IACA,SAAS,wBAAwB;IACjC;IACA;;;;;IAKA,WAAW;;;UAIE;EACf,QAAQ,OAAO,uBAAuB;EACtC,OAAO,MAAM,4BAA4B,QAAQ;EACjD,QAAQ;IACN;IACA;IACA,UAAU;;;IAGV,SAAS,OAAO;;;UAMH,mBAAmB;;;EAGlC;EACA;EACA;EACA;EACA;EACA,UAAU;EACV,aAAa,eAAe;;;EAG5B;;EAEA,gBAAgB;;EAEhB;;;EAGA,YAAY;;;EAGZ;;;EAGA;EACA;EACA;EACA;;EAEA;;EAEA;EACA;;UAGe;EACf;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA;;UAGe;EACf;EACA,YAAY;EACZ;;;;;;UAOe;EACf;;EAEA;;EAEA;;EAEA;;EAEA;;;EAGA;;;;EAIA;;;;EAIA;IACE;IACA;IACA,iBAAiB;MAAQ;MAAgB;;;;;EAI3C,YAAY;;;;;;;;;;EAUZ,WAAW;IAAQ;IAAoB;IAAmB;IAAgB;;;;EAG1E;;;;EAIA;;UAGe;EACf,SAAS,eAAe;EACxB,YAAY,eAAe;;EAE3B,MAAM;;EAEN;EACA;EACA;;EAEA;;EAEA;;EAEA;;EAEA;;UAGe,eAAe,qBAAqB,kBAAkB,WAAW;;EAEhF;;EAEA;EACA;;EAEA;EACA;EACA;EACA;EACA,OAAO,MAAM,mBAAmB;EAChC,YAAY;EACZ;IACE,aAAa;IACb;;EAEF,OAAO;EACP;EACA;EACA,iBAAiB;;;EAGjB,WAAW,MAAM,2BAA2B,KAAK"}
|
|
@@ -289,7 +289,7 @@ declare function assertCrossFamilyServed(pairs: ReadonlyArray<{
|
|
|
289
289
|
//#endregion
|
|
290
290
|
//#region src/llm-client.d.ts
|
|
291
291
|
interface LlmMessage {
|
|
292
|
-
role: 'system' | 'user' | 'assistant';
|
|
292
|
+
role: 'system' | 'user' | 'assistant' | 'tool';
|
|
293
293
|
/**
|
|
294
294
|
* Either a plain text content string OR a multimodal content array
|
|
295
295
|
* (text + image_url parts) for vision-capable models.
|
|
@@ -304,8 +304,35 @@ interface LlmMessage {
|
|
|
304
304
|
detail?: 'auto' | 'low' | 'high';
|
|
305
305
|
};
|
|
306
306
|
}>;
|
|
307
|
+
/** Tool invocations made by an `assistant` message earlier in the turn. */
|
|
308
|
+
toolCalls?: LlmToolCall[];
|
|
309
|
+
/** The invocation a `tool` message answers. Required when role is `tool`. */
|
|
310
|
+
toolCallId?: string;
|
|
307
311
|
}
|
|
308
312
|
type LlmThinkingMode = 'enabled' | 'disabled';
|
|
313
|
+
/** Canonical function-tool definition offered to the model. */
|
|
314
|
+
interface LlmToolDefinition {
|
|
315
|
+
type: 'function';
|
|
316
|
+
function: {
|
|
317
|
+
name: string;
|
|
318
|
+
description?: string;
|
|
319
|
+
parameters: Record<string, unknown>;
|
|
320
|
+
};
|
|
321
|
+
}
|
|
322
|
+
/** Canonical tool-choice policy; meaningful only when `tools` is present. */
|
|
323
|
+
type LlmToolChoice = 'auto' | 'none' | 'required' | {
|
|
324
|
+
type: 'function';
|
|
325
|
+
function: {
|
|
326
|
+
name: string;
|
|
327
|
+
};
|
|
328
|
+
};
|
|
329
|
+
/** One tool invocation on a response or an assistant history message. */
|
|
330
|
+
interface LlmToolCall {
|
|
331
|
+
id: string;
|
|
332
|
+
name: string;
|
|
333
|
+
/** JSON-encoded arguments, exactly as the provider produced them. */
|
|
334
|
+
argumentsJson: string;
|
|
335
|
+
}
|
|
309
336
|
interface LlmCallRequest {
|
|
310
337
|
model: string;
|
|
311
338
|
messages: LlmMessage[];
|
|
@@ -316,6 +343,10 @@ interface LlmCallRequest {
|
|
|
316
343
|
name: string;
|
|
317
344
|
schema: Record<string, unknown>;
|
|
318
345
|
};
|
|
346
|
+
/** Function tools offered to the model for this call. */
|
|
347
|
+
tools?: LlmToolDefinition[];
|
|
348
|
+
/** Tool-choice policy for `tools`. */
|
|
349
|
+
toolChoice?: LlmToolChoice;
|
|
319
350
|
temperature?: number;
|
|
320
351
|
maxTokens?: number;
|
|
321
352
|
/** OpenAI-compatible reasoning mode. Omitted when the provider default should apply. */
|
|
@@ -327,7 +358,7 @@ interface LlmCallRequest {
|
|
|
327
358
|
* Returns undefined when output or multimodal input is not bounded, causing a
|
|
328
359
|
* capped CostLedger to reject the call before execution. Pass
|
|
329
360
|
* `customTokenPricing` when package pricing does not cover the model or endpoint. */
|
|
330
|
-
declare function maximumChargeForLlmRequest(request: Pick<LlmCallRequest, 'model' | 'messages' | 'jsonSchema' | 'maxTokens' | 'thinking'>, options?: LlmClientOptions): MaximumCharge | undefined;
|
|
361
|
+
declare function maximumChargeForLlmRequest(request: Pick<LlmCallRequest, 'model' | 'messages' | 'jsonSchema' | 'tools' | 'toolChoice' | 'maxTokens' | 'thinking'>, options?: LlmClientOptions): MaximumCharge | undefined;
|
|
331
362
|
interface LlmUsage {
|
|
332
363
|
promptTokens: number;
|
|
333
364
|
completionTokens: number;
|
|
@@ -342,6 +373,8 @@ interface LlmUsage {
|
|
|
342
373
|
interface LlmCallResult {
|
|
343
374
|
/** The text content of the first choice. Empty string if none. */
|
|
344
375
|
content: string;
|
|
376
|
+
/** Tool invocations from the first choice, when the model called tools. */
|
|
377
|
+
toolCalls?: LlmToolCall[];
|
|
345
378
|
usage: LlmUsage;
|
|
346
379
|
/**
|
|
347
380
|
* Cost in USD. Uses the provider's reported cost when present, otherwise
|
|
@@ -848,4 +881,4 @@ interface CheckResult {
|
|
|
848
881
|
}
|
|
849
882
|
//#endregion
|
|
850
883
|
export { assertServedModels as $, LlmClientOptions as A, isTransientLlmError as B, SandboxSdkTransportOpts as C, LlmCallRequest as D, LlmCallMetadata as E, assertLlmRoute as F, AssertServedModelOptions as G, probeLlm as H, callLlm as I, ServedModelCheck as J, ModelSubstitutionError as K, callLlmJson as L, LlmResponseError as M, LlmRouteRequirements as N, LlmCallResult as O, LlmUsage as P, assertServedModel as Q, costReceiptFromLlm as R, RouterTransportOpts as S, LlmCallError as T, stripFencedJson as U, maximumChargeForLlmRequest as V, AssertCrossFamilyServedOptions as W, ServedModelVerdict as X, ServedModelPolicy as Y, assertCrossFamilyServed as Z, CliBridgeTransportOpts as _, defaultProviderRedactor as _t, JudgeInput as a, assertCrossFamily as at, DirectProviderTransportOpts as b, PersonaConfig as c, FileSystemRawProviderSinkOptions as ct, Scenario as d, NoopRawProviderSink as dt, checkServedModel as et, ChatCallOpts as f, ProviderRedactor as ft, ChatTransport as g, RawProviderSinkFilter as gt, ChatResponse as h, RawProviderSink as ht, JudgeFn as i, JudgeFamily as it, LlmMessage as j, LlmClient as k, ProductClientConfig as l, InMemoryRawProviderSink as lt, ChatRequest as m, RawProviderEvent as mt, CompletionCriterion as n, AssertCrossFamilyOptions as nt, JudgeRubric as o, judgeFamily as ot, ChatClient as p, RawProviderDirection as pt, ServedCrossFamilyError as q, DriverState as r, CrossFamilyError as rt, JudgeScore as s, FileSystemRawProviderSink as st, CheckResult as t, servedModelAcceptable as tt, RouteMap as u, InMemoryRawProviderSinkOptions as ut, CreateChatClientOpts as v, providerFromBaseUrl as vt, createChatClient as w, MockTransportOpts as x, CustomTransportOpts as y, costReceiptFromLlmError as z };
|
|
851
|
-
//# sourceMappingURL=types-
|
|
884
|
+
//# sourceMappingURL=types-jUBXJ7Iz.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"types-jUBXJ7Iz.d.ts","names":[],"sources":["../src/trace/raw-provider-sink.ts","../src/judge-families.ts","../src/integrity/served-model.ts","../src/llm-client.ts","../src/analyst/chat-client.ts","../src/types.ts"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;KA8BY;UAEK;;EAEf;;EAEA;EACA;;;;;;EAMA;EACA;;EAEA;;EAEA;;EAEA;EACA,WAAW;;EAEX;;EAEA;EACA;EACA,iBAAiB;EACjB;EACA,kBAAkB;EAClB;;EAEA;;EAEA;;UAGe;EACf;EACA;EACA,YAAY;EACZ;;UAGe;EACf,OAAO,OAAO,mBAAmB;;EAEjC,MAAM,SAAS,wBAAwB,QAAQ;;EAE/C,UAAU;;KAGA,oBAAoB,OAAO,qBAAqB;;;;;;iBAmB5C,wBAAwB,OAAO,mBAAmB;UA8CjD;EACf,WAAW;;cAGA,mCAAmC;UACtC;UACA;EAER,YAAY,OAAM;EAIZ,OAAO,OAAO,mBAAmB;EAIjC,KAAK,SAAQ,wBAA6B,QAAQ;EAUxD;;cAKW,+BAA+B;EACpC,UAAU;;;;;;;EASV,QAAQ,QAAQ;;UAOP;;EAEf;;EAEA;;EAEA;EACA,WAAW;;cAGA,qCAAqC;UACxC;UACA;UACA;UACA;UACA;UACA;UACA;EAER,YAAY,MAAM;UAOJ;UAON;EAKF,OAAO,OAAO,mBAAmB;EAYjC,KAAK,SAAQ,wBAA6B,QAAQ;;;;;;iBAkC1C,oBAAoB;;;;;;;;;;;;;KC5QxB;;;;;;;iBAkEI,YAAY,kBAAkB;UAc7B;;EAEf;;;;EAIA;;cAGW,yBAAyB;WAGlB,UAAU;WACV;EAHlB,YACE,iBACgB,UAAU,eACV;;;;;;;;;;;;;;;iBAoBJ,kBACd,kBACA,OAAM,2BACL;;;;KCnFS;;;;;;;;;;;UAYK;;EAEf;;EAEA;EACA,iBAAiB;;EAEjB,cAAc;EACd,SAAS;;EAET;;;;;;;;;;;;iBA4Cc,iBACd,mBACA,oCACC;cA4CU,+BAA+B;WAGxB,QAAQ,cAAc;EAFxC,YACE,iBACgB,QAAQ,cAAc;;;;;;;;;KAc9B;UAWK;;;;;;EAMf;;;;;EAKA;;EAEA;;;;;;;iBAQc,sBACd,OAAO,kBACP,OAAM;;;;;;iBA6BQ,kBACd,mBACA,mCACA,OAAM,2BACL;;;;;iBAea,mBACd,OAAO;EAAgB;EAAmB;IAC1C,OAAM,2BACL;UAec,uCAAuC;;EAEtD;;EAEA;;cAGW,+BAA+B;WAGxB,UAAU;WACV,QAAQ,cAAc;EAHxC,YACE,iBACgB,UAAU,eACV,QAAQ,cAAc;;;;;;;;;;;iBAgB1B,wBACd,OAAO;EAAgB;EAAmB;IAC1C,OAAM,iCACL;;;UCnQc;EACf;;;;;EAKA,kBAEI;IACM;IAAc;;IACd;IAAmB;MAAa;MAAa;;;;EAGvD,YAAY;;EAEZ;;KAGU;;UAGK;EACf;EACA;IACE;IACA;IACA,YAAY;;;;KAKJ;EAIN;EAAkB;IAAY;;;;UAGnB;EACf;EACA;;EAEA;;UAGe;EACf;EACA,UAAU;;EAEV;;EAEA;IAAe;IAAc,QAAQ;;;EAErC,QAAQ;;EAER,aAAa;EACb;EACA;;EAEA,WAAW;;EAEX;;;;;;iBAOc,2BACd,SAAS,KACP,0GAGF,UAAS,mBACR;UAgCc;EACf;EACA;EACA;;EAEA;;EAEA;;EAEA;;UAGe;;EAEf;;EAEA,YAAY;EACZ,OAAO;;;;;EAKP;;;;;;;;;EASA;;;;;;;;;;;;EAYA;;EAEA;;;;;;;;;EASA;;;;;;;EAOA;;EAEA,KAAK;;KAGK,kBAAkB,KAAK;;;;;;iBAOnB,mBACd,QAAQ,eACR,qBAAqB,qBACpB;;iBA6Ba,wBACd,OAAO,OACP,qBAAqB,qBACpB;cAMU,qBAAqB;WAGd;WACA;WACA;EAJlB,YACE,iBACgB,gBACA,cACA;;;;;cASP,yBAAyB;WAGlB,QAAQ;EAF1B,YACE,iBACgB,QAAQ,eACxB;IAAY;;;UAMC;;EAEf;;EAEA;EACA;;EAEA;IAAe;IAAc;;;EAE7B;;EAEA;;;;;;;EAOA,SAAS;;;;;;;;EAQT;;EAEA;;EAEA,qBAAqB;;;;;;;EAOrB;;;;;;EAMA;;EAEA,WAAW;;EAEX,eAAe;;;;;;;;EAQf,UAAU;;;;;EAKV;;EAEA;IAAiB;IAAgB;;;EAEjC,WAAW;;;;;;;;;;EAUX,8BAA8B;;;;;;;;;;;;;iBAyEhB,oBAAoB;;;;;;iBAqMpB,gBAAgB;;;;;;iBA6EV,QACpB,KAAK,gBACL,OAAM,mBACL,QAAQ;;;;;;;iBAyVW,YAAY,aAChC,KAAK,gBACL,OAAM,mBACL;EAAU,OAAO;EAAG,QAAQ;;UA6Ed;;;;;;;EAOf;;;;;EAKA,kBAAkB,eAAe;;EAEjC,kBAAkB,eAAe;;EAEjC;;;;;EAKA;;;;;;;;;;;iBAYc,eAAe,MAAM,kBAAkB,MAAK;;;;;;;;;;;;;;;;;;iBA6EtC,SACpB,eACA,OAAM;EAAqB;IAC1B;EACD;EACA;EACA;;EAEA;;EAEA;;;;;;;cAoCW;WACF;mBACQ;EAEjB,YAAY,OAAM;EAKlB,KAAK,KAAK,gBAAgB,MAAM,mBAAmB,QAAQ;EAK3D,SAAS,aACP,KAAK,gBACL,MAAM,mBACL;IAAU,OAAO;IAAG,QAAQ;;;;;;;;UC7wChB;;WAEN,WAAW;;WAEX;;WAEA;;EAGT,KAAK,KAAK,aAAa,OAAO,eAAe,QAAQ;;KAG3C;UAQK,oBAAoB,KAAK;;EAExC;;KAGU,eAAe;UAEV;;EAEf,SAAS;;EAET;;EAEA;;EAEA;;KAKU,uBACR,sBACA,yBACA,8BACA,0BACA,sBACA;UAEM;EACR;;EAEA;;UAGe,4BAA4B;EAC3C;EACA;EACA;;UAGe,+BAA+B;EAC9C;EACA;EACA;;UAGe,oCAAoC;EACnD;EACA;EACA;;;;;;UAOe,gCAAgC;EAC/C;EACA,OAAO,KAAK,aAAa,OAAO,iBAAiB,QAAQ;;;UAI1C,4BAA4B;EAC3C;EACA,OAAO,KAAK,aAAa,OAAO,iBAAiB,QAAQ;;;;;;UAO1C,0BAA0B;EACzC;EACA,UAAU,KAAK,aAAa,OAAO,iBAAiB,QAAQ;;;;;;iBAO9C,iBAAiB,MAAM,uBAAuB;;;UCjH7C;EACf;EACA;EACA;EACA;EACA;EACA,OAAO;EACP,gBAAgB;EAChB;;UAGe;EACf;EACA;EACA;EACA;;UAKe;EACf;EAQA;EACA;EACA;EACA;;UAKe;EACf;EACA;EACA,YAAY;;UAGG;EACf;EACA;EACA;EACA;EACA;;UAmBe;EACf;EACA;EACA;EACA;EACA;IAAmB;IAAc;;EACjC;EACA;;UASe;EACf;EACA;EACA;EACA;EACA;;UAGe;EACf;IAAc;IAAc;;EAC5B;IAAmB;IAAc,QAAQ;;EACzC;IAAc;IAAkB;;EAChC;;UAKe;EACf;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;GACC;;UAGc;EACf;EACA,QAAQ;;EAER;;UAyBe;EACf;EACA,QAAQ,OAAO;EACf,YAAY,OAAO;;;;;;;;;;KAWT;UAEK;EACf;EACA;EACA;EACA,oBAAoB;EACpB;;EAEA,QAAQ;;;;;;;EAOR;;;;;;;EAOA;;;;;;EAMA;;UAGe;EACf;EACA;EACA;IAAa;IAAiB;IAAkB;;EAChD;EACA;EACA;;UAGe;EACf,UAAU;EACV,OAAO;EACP,WAAW;;EAEX,aAAa;EACb;EACA,WAAW;EACX,SAAS;;KAGC,WAAW,MAAM,YAAY,OAAO,eAAe,QAAQ;UAYtD;EACf;EACA;EACA;EACA"}
|
package/dist/wire/index.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { c as CostLedgerHandle } from "../cost-ledger-DbQdN3nO.js";
|
|
2
|
-
import { A as LlmClientOptions, N as LlmRouteRequirements } from "../types-
|
|
2
|
+
import { A as LlmClientOptions, N as LlmRouteRequirements } from "../types-jUBXJ7Iz.js";
|
|
3
3
|
import { s as TraceStore } from "../store-CT9YIIve.js";
|
|
4
|
-
import { m as FeedbackTrajectoryStore } from "../feedback-trajectory-
|
|
4
|
+
import { m as FeedbackTrajectoryStore } from "../feedback-trajectory-DpTTjo0q.js";
|
|
5
5
|
import { z } from "zod";
|
|
6
6
|
import { ServerType } from "@hono/node-server";
|
|
7
7
|
import { Hono } from "hono";
|
package/dist/wire/index.js
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
import { A as TraceEventSchema, C as HealthResponseSchema, D as RubricDimensionSchema, E as ListRubricsResponseSchema, F as hashRubric, M as TracesIngestResponseSchema, N as VersionResponseSchema, O as RubricInfoSchema, P as WIRE_VERSION, S as FeedbackTrajectorySchema, T as JudgeResultSchema, _ as ErrorResponseSchema, a as runRpcBatch, b as FeedbackIngestResponseSchema, c as WireError, d as handleListRubrics, f as handleTracesIngest, g as listBuiltinRubrics, h as getBuiltinRubric, i as dispatchRpc, j as TracesIngestRequestSchema, k as RubricSchema, l as handleFeedbackIngest, m as BUILTIN_RUBRICS, n as startServer, o as runRpcOnce, p as handleVersion, r as startServerAsync, s as buildOpenApi, t as createApp, u as handleJudge, v as FailureModeSchema, w as JudgeRequestSchema, x as FeedbackLabelSchema, y as FeedbackAttemptSchema } from "../server-
|
|
1
|
+
import { A as TraceEventSchema, C as HealthResponseSchema, D as RubricDimensionSchema, E as ListRubricsResponseSchema, F as hashRubric, M as TracesIngestResponseSchema, N as VersionResponseSchema, O as RubricInfoSchema, P as WIRE_VERSION, S as FeedbackTrajectorySchema, T as JudgeResultSchema, _ as ErrorResponseSchema, a as runRpcBatch, b as FeedbackIngestResponseSchema, c as WireError, d as handleListRubrics, f as handleTracesIngest, g as listBuiltinRubrics, h as getBuiltinRubric, i as dispatchRpc, j as TracesIngestRequestSchema, k as RubricSchema, l as handleFeedbackIngest, m as BUILTIN_RUBRICS, n as startServer, o as runRpcOnce, p as handleVersion, r as startServerAsync, s as buildOpenApi, t as createApp, u as handleJudge, v as FailureModeSchema, w as JudgeRequestSchema, x as FeedbackLabelSchema, y as FeedbackAttemptSchema } from "../server-dIWwF3j_.js";
|
|
2
2
|
export { BUILTIN_RUBRICS, ErrorResponseSchema, FailureModeSchema, FeedbackAttemptSchema, FeedbackIngestResponseSchema, FeedbackLabelSchema, FeedbackTrajectorySchema, HealthResponseSchema, JudgeRequestSchema, JudgeResultSchema, ListRubricsResponseSchema, RubricDimensionSchema, RubricInfoSchema, RubricSchema, TraceEventSchema, TracesIngestRequestSchema, TracesIngestResponseSchema, VersionResponseSchema, WIRE_VERSION, WireError, buildOpenApi, createApp, dispatchRpc, getBuiltinRubric, handleFeedbackIngest, handleJudge, handleListRubrics, handleTracesIngest, handleVersion, hashRubric, listBuiltinRubrics, runRpcBatch, runRpcOnce, startServer, startServerAsync };
|
|
@@ -336,11 +336,18 @@ Set `optimizer.anthropicEndpoint: true` to admit them in proxied mode.
|
|
|
336
336
|
The loopback proxy then also serves `POST /v1/messages` (Anthropic Messages API) and the bridge child receives `ANTHROPIC_BASE_URL`, an ephemeral `ANTHROPIC_AUTH_TOKEN`, and `ANTHROPIC_MODEL` in its environment.
|
|
337
337
|
Every CLI call becomes one canonical execution-owner call with the same reservation, receipt, and budget pipeline as reflection traffic; the run fails if the receipt count differs from the admitted call count.
|
|
338
338
|
Each agent engine run must set `engineConfig.model` to `optimizer.model`, because the engines pass `--model` and that flag beats the injected environment.
|
|
339
|
-
The endpoint translates text
|
|
339
|
+
The endpoint translates text and tool-use conversations.
|
|
340
|
+
Anthropic `tools`, `tool_choice`, `tool_use`, and `tool_result` map onto the canonical execution-owner contract, and a tool-calling response is synthesized back as the Anthropic stream shape the CLI expects.
|
|
341
|
+
Claude-specific control fields (`thinking`, `context_management`, `output_config`) carry no token-billing semantics on the owner wire; the shim strips them and records the names in the ledger tag `strippedFields`.
|
|
342
|
+
It still refuses images, server tools, `top_p`, `top_k`, and `stop_sequences` with a loud Anthropic error envelope.
|
|
340
343
|
A budget refusal surfaces to the CLI as HTTP 402, which the CLI treats as terminal instead of retrying.
|
|
341
344
|
Agent sessions are chatty: size `budget.maxRequests` for tens of calls per engine run.
|
|
342
345
|
Without the flag, agent engines stay rejected in proxied mode.
|
|
343
346
|
|
|
347
|
+
One measured pothole: `selfImprove` defaults `expectUsage` to `'assert'`.
|
|
348
|
+
A deterministic evaluator that makes no LLM calls records zero usage, so the assertion fails the run.
|
|
349
|
+
Set `expectUsage: 'off'` when the evaluator is deterministic and makes no LLM calls.
|
|
350
|
+
|
|
344
351
|
Other official engines can still receive their own settings:
|
|
345
352
|
|
|
346
353
|
```ts
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tangle-network/agent-eval",
|
|
3
|
-
"version": "0.150.
|
|
3
|
+
"version": "0.150.1",
|
|
4
4
|
"description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
|
|
5
5
|
"homepage": "https://github.com/tangle-network/agent-eval#readme",
|
|
6
6
|
"repository": {
|