@tangle-network/agent-eval 0.142.1 → 0.143.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -0
- package/README.md +21 -0
- package/dist/analyst/index.d.ts +5 -5
- package/dist/analyst/index.js +2 -2
- package/dist/{benchmark-command-DTyVFMUV.js → benchmark-command-CSddvlq7.js} +5 -5
- package/dist/{benchmark-command-DTyVFMUV.js.map → benchmark-command-CSddvlq7.js.map} +1 -1
- package/dist/benchmarks/index.d.ts +1 -1
- package/dist/benchmarks/index.js +1 -1
- package/dist/{benchmarks-Ce8GdJbc.js → benchmarks-Dg0B56tK.js} +4 -4
- package/dist/{benchmarks-Ce8GdJbc.js.map → benchmarks-Dg0B56tK.js.map} +1 -1
- package/dist/campaign/index.d.ts +3 -3
- package/dist/campaign/index.js +3 -3
- package/dist/{campaign-COFTVy3Q.js → campaign-C7ZGiZYG.js} +7 -5
- package/dist/campaign-C7ZGiZYG.js.map +1 -0
- package/dist/cli.js +1 -1
- package/dist/{client-BoqGxEqx.d.ts → client-Cgl6KasJ.d.ts} +2 -2
- package/dist/{client-BoqGxEqx.d.ts.map → client-Cgl6KasJ.d.ts.map} +1 -1
- package/dist/contract/index.d.ts +5 -5
- package/dist/contract/index.js +3 -3
- package/dist/{default-registry-Ci7wAAR8.d.ts → default-registry-D3uqKbo6.d.ts} +2 -2
- package/dist/{default-registry-Ci7wAAR8.d.ts.map → default-registry-D3uqKbo6.d.ts.map} +1 -1
- package/dist/{dspy-rlm-engine-Bw8Zz9CX.js → dspy-rlm-engine-CJ4sN8Wx.js} +2 -2
- package/dist/{dspy-rlm-engine-Bw8Zz9CX.js.map → dspy-rlm-engine-CJ4sN8Wx.js.map} +1 -1
- package/dist/hosted/index.d.ts +2 -2
- package/dist/{index-CKI1CXTL.d.ts → index-CaOjqCli.d.ts} +4 -4
- package/dist/{index-CKI1CXTL.d.ts.map → index-CaOjqCli.d.ts.map} +1 -1
- package/dist/{index-DCP4I2Qx.d.ts → index-DY83M1WL.d.ts} +7 -5
- package/dist/{index-DCP4I2Qx.d.ts.map → index-DY83M1WL.d.ts.map} +1 -1
- package/dist/index.d.ts +8 -8
- package/dist/index.js +4 -4
- package/dist/multishot/index.d.ts +1 -1
- package/dist/openapi.json +1 -1
- package/dist/{replay-BqTgoioO.d.ts → replay-BuJM6kLh.d.ts} +2 -2
- package/dist/{replay-BqTgoioO.d.ts.map → replay-BuJM6kLh.d.ts.map} +1 -1
- package/dist/{reward-hacking-GyN0kMd8.js → reward-hacking-DjTi9HLb.js} +34 -14
- package/dist/reward-hacking-DjTi9HLb.js.map +1 -0
- package/dist/rl.d.ts +263 -2
- package/dist/rl.d.ts.map +1 -1
- package/dist/rl.js +332 -3
- package/dist/rl.js.map +1 -1
- package/dist/{single-run-lock-BcvPQY57.js → single-run-lock-Do-8vBIJ.js} +2 -2
- package/dist/{single-run-lock-BcvPQY57.js.map → single-run-lock-Do-8vBIJ.js.map} +1 -1
- package/dist/{skill-usage-CNZTk_LT.d.ts → skill-usage-CRqaHi0R.d.ts} +3 -3
- package/dist/{skill-usage-CNZTk_LT.d.ts.map → skill-usage-CRqaHi0R.d.ts.map} +1 -1
- package/dist/{skillopt-optimization-method-D1dqGzzH.d.ts → skillopt-optimization-method-CwRvi-rL.d.ts} +21 -5
- package/dist/skillopt-optimization-method-CwRvi-rL.d.ts.map +1 -0
- package/dist/{skillopt-optimization-method-DV6c4ooy.js → skillopt-optimization-method-Di8Xkalt.js} +153 -50
- package/dist/skillopt-optimization-method-Di8Xkalt.js.map +1 -0
- package/dist/{tool-groups-DVQTy9lq.d.ts → tool-groups-Bqy4A3QB.d.ts} +2 -2
- package/dist/{tool-groups-DVQTy9lq.d.ts.map → tool-groups-Bqy4A3QB.d.ts.map} +1 -1
- package/dist/traces.d.ts +2 -2
- package/dist/{types-Dk7PB7vh.d.ts → types-KEqL1pZc.d.ts} +6 -4
- package/dist/{types-Dk7PB7vh.d.ts.map → types-KEqL1pZc.d.ts.map} +1 -1
- package/docs/verified-labels-flywheel.md +46 -0
- package/package.json +3 -3
- package/dist/campaign-COFTVy3Q.js.map +0 -1
- package/dist/reward-hacking-GyN0kMd8.js.map +0 -1
- package/dist/skillopt-optimization-method-D1dqGzzH.d.ts.map +0 -1
- package/dist/skillopt-optimization-method-DV6c4ooy.js.map +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,30 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
|
|
|
4
4
|
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
+
## [0.143.0] - 2026-08-03 - exact campaign cost provenance
|
|
8
|
+
|
|
9
|
+
### Changed
|
|
10
|
+
|
|
11
|
+
- **Breaking:** `CampaignCellResult.costEstimated` is replaced by the required `costProvenance` union, and `ProfileSummary.totalCostUsd` becomes `number | null` with an adjacent `costProvenance` value.
|
|
12
|
+
- Campaign measurement digests now include the complete cost-provenance value, so digest equality is intentionally not preserved across the `0.142.x` to `0.143.0` data boundary.
|
|
13
|
+
- Run-record raw cost fields now describe the total as observed, estimated, or uncaptured, while `cost_known_subtotal_usd` retains any measured subtotal when the total remains unknown.
|
|
14
|
+
- `planCampaignRun` reports `cellsBlocked`, and each planned cell may be `blocked` when saved data cannot prove its cost or receipt identity.
|
|
15
|
+
- Caches created before `0.143.0` have no complete cost provenance and therefore stop a resumable campaign by default.
|
|
16
|
+
Inspect the plan and set `rerunInvalidCachedCells: true` for a one-time selective rerun, or set `resumable: false` for an intentional full rerun.
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
|
|
20
|
+
- Failed provider calls retain input, cached-input, cache-write, output, and reasoning tokens without presenting an unknown USD total as zero.
|
|
21
|
+
- Campaign execution inspects every saved cell and its exact billing receipts before any concurrent dispatch begins, so a late stale cell cannot waste spend in earlier cells.
|
|
22
|
+
- Malformed, unreadable, mismatched, or receipt-incomplete saved cells require explicit rerun approval instead of being silently reused or re-executed.
|
|
23
|
+
- `planCampaignRun` no longer creates the run directory while performing read-only inspection.
|
|
24
|
+
|
|
25
|
+
## [0.142.2] - 2026-08-02 - current shared type cohort
|
|
26
|
+
|
|
27
|
+
### Changed
|
|
28
|
+
|
|
29
|
+
- Align the exact `@tangle-network/agent-core` and `@tangle-network/agent-interface` dependencies with `0.4.33` and `0.43.0`, so Eval, Runtime, Knowledge, and Sandbox share the current canonical profile contract.
|
|
30
|
+
|
|
7
31
|
## [0.142.1] - 2026-08-02 - current shared type patch
|
|
8
32
|
|
|
9
33
|
### Changed
|
package/README.md
CHANGED
|
@@ -113,6 +113,27 @@ console.log(
|
|
|
113
113
|
Each call runs every case, records the artifact, applies the same judge, and returns score distributions.
|
|
114
114
|
The surface is the value being changed, such as a prompt, skill, or serialized configuration.
|
|
115
115
|
|
|
116
|
+
### Inspect cached cells before rerunning
|
|
117
|
+
|
|
118
|
+
`runCampaign()` refuses to start when an existing cache file is unreadable or lacks trustworthy cost data.
|
|
119
|
+
This check covers the full schedule before concurrent work begins, so one bad cache cannot waste paid calls from earlier cells.
|
|
120
|
+
Use `planCampaignRun()` to see which cells are reusable, runnable, or blocked:
|
|
121
|
+
|
|
122
|
+
```ts
|
|
123
|
+
const plan = planCampaignRun({
|
|
124
|
+
scenarios,
|
|
125
|
+
dispatch,
|
|
126
|
+
judges: [judge],
|
|
127
|
+
runDir: 'release-candidate',
|
|
128
|
+
})
|
|
129
|
+
|
|
130
|
+
console.table(plan.cells)
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
After inspecting the plan, set `rerunInvalidCachedCells: true` to rerun only blocked cells while retaining valid cached cells.
|
|
134
|
+
Set `resumable: false` only when you intend to rerun every cell.
|
|
135
|
+
When execution uses a shared `costLedger` or custom `costTags`, pass those same values to `planCampaignRun()` so both calls inspect the same receipts.
|
|
136
|
+
|
|
116
137
|
### Stop after the first failed cell
|
|
117
138
|
|
|
118
139
|
`runCampaign()` normally records a dispatch or judge error on that cell and continues the remaining cases.
|
package/dist/analyst/index.d.ts
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
import { a as MultiLayerVerifier, l as VerifyOptions, o as Severity } from "../multi-layer-verifier-BHY1gWAc.js";
|
|
2
|
-
import { A as parseFindingSubject, C as FINDING_SUBJECT_KINDS, D as FindingSubjectStringSchema, E as FindingSubjectKind, F as DefineExactCustomAnalystOptions, G as SemanticConceptJudgeOptions, I as defineCustomAnalyst, L as defineTraceAnalyst, M as DspyRlmTraceEngineOptions, N as createDspyRlmTraceEngine, O as KIND_EXPECTED_SUBJECTS, P as DefineCustomAnalystOptions, S as FINDING_SUBJECT_GRAMMAR_PROMPT, T as FindingSubject, W as SemanticConceptJudgeInput, Y as RunCritic, Z as RunTrace, _ as FindingsDiff, a as SkillUsageScanConfig, b as defaultIsMaterial, c as DEFAULT_TRACE_ANALYST_KINDS, d as IMPROVEMENT_KIND_SPEC, f as FAILURE_MODE_KIND_SPEC, g as DiffPolicy, h as emitControlIntegrityFindings, i as SkillUsageReport, j as renderFindingSubject, k as findingSubjectGrammarPromptFor, l as KNOWLEDGE_POISONING_KIND_SPEC, m as ControlIntegrityAnalyst, n as SkillUsageAnalyst, o as buildSkillUsageReport, p as CONTROL_INTEGRITY_ANALYST, r as SkillUsageRecord, s as emitSkillUsageFindings, t as SKILL_USAGE_ANALYST, u as KNOWLEDGE_GAP_KIND_SPEC, v as FindingsStore, w as FINDING_SUBJECT_SYNTAX, x as diffFindings, y as PersistedFinding } from "../skill-usage-
|
|
2
|
+
import { A as parseFindingSubject, C as FINDING_SUBJECT_KINDS, D as FindingSubjectStringSchema, E as FindingSubjectKind, F as DefineExactCustomAnalystOptions, G as SemanticConceptJudgeOptions, I as defineCustomAnalyst, L as defineTraceAnalyst, M as DspyRlmTraceEngineOptions, N as createDspyRlmTraceEngine, O as KIND_EXPECTED_SUBJECTS, P as DefineCustomAnalystOptions, S as FINDING_SUBJECT_GRAMMAR_PROMPT, T as FindingSubject, W as SemanticConceptJudgeInput, Y as RunCritic, Z as RunTrace, _ as FindingsDiff, a as SkillUsageScanConfig, b as defaultIsMaterial, c as DEFAULT_TRACE_ANALYST_KINDS, d as IMPROVEMENT_KIND_SPEC, f as FAILURE_MODE_KIND_SPEC, g as DiffPolicy, h as emitControlIntegrityFindings, i as SkillUsageReport, j as renderFindingSubject, k as findingSubjectGrammarPromptFor, l as KNOWLEDGE_POISONING_KIND_SPEC, m as ControlIntegrityAnalyst, n as SkillUsageAnalyst, o as buildSkillUsageReport, p as CONTROL_INTEGRITY_ANALYST, r as SkillUsageRecord, s as emitSkillUsageFindings, t as SKILL_USAGE_ANALYST, u as KNOWLEDGE_GAP_KIND_SPEC, v as FindingsStore, w as FINDING_SUBJECT_SYNTAX, x as diffFindings, y as PersistedFinding } from "../skill-usage-CRqaHi0R.js";
|
|
3
3
|
import { b as CustomTokenPricing, c as CostLedgerHandle } from "../cost-ledger-FuQvHxPm.js";
|
|
4
4
|
import { A as ChatClient, B as SandboxSdkTransportOpts, F as CreateChatClientOpts, I as CustomTransportOpts, L as DirectProviderTransportOpts, M as ChatResponse, N as ChatTransport, P as CliBridgeTransportOpts, R as MockTransportOpts, V as createChatClient, j as ChatRequest, k as ChatCallOpts, m as JudgeInput, p as JudgeFn, z as RouterTransportOpts } from "../types-BjMFz88h.js";
|
|
5
5
|
import { _ as makeFinding, a as AnalystInputKind, c as AnalystRunInputs, d as AnalystSeverity, f as AnalystUsageReceipt, g as computeFindingId, h as ProposalFindingOrigin, i as AnalystFinding, l as AnalystRunResult, m as ProposalFinding, n as AnalystContext, o as AnalystRequirements, p as EvidenceRef, r as AnalystCost, s as AnalystRunEvent, t as Analyst, u as AnalystRunSummary, v as makeProposalFinding, x as TraceAnalysisStore } from "../types-D3jh6F98.js";
|
|
6
|
-
import { a as createTraceAnalyst, c as runTraceAnalyst, d as deriveEfficiencyFindings, i as TraceAnalystDefinition, l as BehavioralAnalystOptions, n as buildDefaultAnalystRegistry, o as renderPriorFindings, r as CreateTraceAnalystOptions, s as renderUpstreamFindings, t as DefaultAnalystRegistryOptions, u as behavioralAnalyst } from "../default-registry-
|
|
6
|
+
import { a as createTraceAnalyst, c as runTraceAnalyst, d as deriveEfficiencyFindings, i as TraceAnalystDefinition, l as BehavioralAnalystOptions, n as buildDefaultAnalystRegistry, o as renderPriorFindings, r as CreateTraceAnalystOptions, s as renderUpstreamFindings, t as DefaultAnalystRegistryOptions, u as behavioralAnalyst } from "../default-registry-D3uqKbo6.js";
|
|
7
7
|
import { a as ExactAnalystRunPolicySnapshot, c as ExactAnalystSnapshot, d as ExactExecutionComponentSnapshot, i as ExactAnalystRunEvent, l as ExactCapableAnalyst, n as ExactAnalystExecutionPlanSnapshot, o as ExactAnalystRunResult, r as ExactAnalystRunCompletion, s as ExactAnalystRunSummary, t as ExactAnalystBudgetSnapshot, u as ExactExecutionComponentIdentity } from "../exact-types-B0lJV3tu.js";
|
|
8
8
|
import { A as ExactAnalystRunExecutionError, D as AnalystRegistryOptions, E as AnalystRegistry, M as RegistryRunOpts, N as assertExactRegistryRunOpts, O as BudgetPolicy, T as AnalystHooks, j as ExactRegistryRunOpts, k as ExactAnalystBudgetPolicy } from "../completion-verifier-D15NHYSk.js";
|
|
9
9
|
import { C as scoreAnalystFindings, S as traceStoreEvidenceResolver, _ as AnalystIssueExpectation, a as AnalystBenchmarkLabelState, b as registryBenchmarkRunner, c as AnalystBenchmarkProvenance, d as AnalystBenchmarkSummary, f as AnalystEvidenceExpectation, g as AnalystFindingScore, h as AnalystEvidenceResolver, i as AnalystBenchmarkError, l as AnalystBenchmarkResult, m as AnalystEvidenceResolutionError, n as AnalystBenchmarkDatasetRef, o as AnalystBenchmarkObservation, p as AnalystEvidenceResolution, r as AnalystBenchmarkDescriptor, s as AnalystBenchmarkOutput, t as AnalystBenchmarkCase, u as AnalystBenchmarkRunner, v as AnalystLatencyDistribution, x as runAnalystBenchmark, y as RunAnalystBenchmarkOptions } from "../benchmark-DxaZfy0w.js";
|
|
10
10
|
import { r as ExternalOptimizerRunnerCommand } from "../external-optimizer-contracts-nb7c_WAR.js";
|
|
11
|
-
import { a as TraceAnalysisEngineRequest, c as resolveTraceAnalystLimits, d as RawAnalystEvidence, f as RawAnalystEvidenceSchema, g as parseRawFinding, h as evidenceRefsFromRawFinding, i as TraceAnalysisEngine, l as ANALYST_SEVERITIES, m as RawAnalystFindingSchema, n as buildTraceToolsForGroup, o as TraceAnalysisEngineResult, p as RawAnalystFinding, r as DEFAULT_TRACE_ANALYST_LIMITS, s as TraceAnalystLimits, t as TraceToolGroupName, u as RAW_FINDING_SCHEMA_PROMPT } from "../tool-groups-
|
|
11
|
+
import { a as TraceAnalysisEngineRequest, c as resolveTraceAnalystLimits, d as RawAnalystEvidence, f as RawAnalystEvidenceSchema, g as parseRawFinding, h as evidenceRefsFromRawFinding, i as TraceAnalysisEngine, l as ANALYST_SEVERITIES, m as RawAnalystFindingSchema, n as buildTraceToolsForGroup, o as TraceAnalysisEngineResult, p as RawAnalystFinding, r as DEFAULT_TRACE_ANALYST_LIMITS, s as TraceAnalystLimits, t as TraceToolGroupName, u as RAW_FINDING_SCHEMA_PROMPT } from "../tool-groups-Bqy4A3QB.js";
|
|
12
12
|
//#region src/analyst/adapters.d.ts
|
|
13
13
|
declare function liftSeverity(s: Severity): AnalystSeverity;
|
|
14
14
|
interface VerifierAdapterOpts<Env> {
|
|
@@ -741,7 +741,7 @@ declare const ANALYST_BENCHMARK_HELP = "agent-eval analyst-benchmark\n\nRun the
|
|
|
741
741
|
declare const ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM = "sha256-canonical-source-manifest";
|
|
742
742
|
declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM = "sha256-canonical-file-manifest";
|
|
743
743
|
declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES: readonly string[];
|
|
744
|
-
declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "
|
|
744
|
+
declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "2867725e505a600f4f1f9d934fb797f5fb94a71c906577d05009aecc254f85f3";
|
|
745
745
|
/** The published benchmark evidence was produced at this package version, by
|
|
746
746
|
* the retired one-shot direct runner, before trace analysts moved to the
|
|
747
747
|
* recursive DSPy RLM engine. Both evidence digests below are historical facts
|
|
@@ -753,7 +753,7 @@ declare const ANALYST_BENCHMARK_EVIDENCE_PACKAGE_VERSION = "0.137.0";
|
|
|
753
753
|
declare const ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256 = "1e03f2daed356d60316aabefb407ec1e437ac94d408d61eea4ae096e9c6fbb5b";
|
|
754
754
|
declare const ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256 = "4dba263b6256a30d56c7fdb2d992d3a953c0035d731f359b704db806f68f75ac";
|
|
755
755
|
declare const ANALYST_BENCHMARK_IMPLEMENTATION_FILES: readonly string[];
|
|
756
|
-
declare const ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 = "
|
|
756
|
+
declare const ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 = "b8b6d4f7a01f8341a0565159806f93ff9ca71c88322e63e4e52a86eed2c1d9c7";
|
|
757
757
|
declare function analystBenchmarkImplementationDigest(): string;
|
|
758
758
|
declare function analystBenchmarkDependencyLockDigest(): string;
|
|
759
759
|
//#endregion
|
package/dist/analyst/index.js
CHANGED
|
@@ -4,9 +4,9 @@ import { A as coerceJson, B as renderFindingSubject, D as RawAnalystFindingSchem
|
|
|
4
4
|
import { a as computeFindingId, i as validateUsageSettlementTimeout, n as settleUsageReceiptFromCostLedger, o as makeFinding, s as makeProposalFinding } from "../usage-receipt-CgxMEBZq.js";
|
|
5
5
|
import { n as isProposalFinding, t as assertProposalFindings } from "../proposal-findings-2GIUo1et.js";
|
|
6
6
|
import { a as RunCritic, c as buildSkillUsageReport, d as defaultIsMaterial, f as diffFindings, h as defineTraceAnalyst, i as runSemanticConceptJudge, l as emitSkillUsageFindings, m as defineCustomAnalyst, n as SEMANTIC_CONCEPT_JUDGE_VERSION, o as SKILL_USAGE_ANALYST, s as SkillUsageAnalyst, u as FindingsStore } from "../semantic-concept-judge-DJQtFr95.js";
|
|
7
|
-
import { t as createDspyRlmTraceEngine } from "../dspy-rlm-engine-
|
|
7
|
+
import { t as createDspyRlmTraceEngine } from "../dspy-rlm-engine-CJ4sN8Wx.js";
|
|
8
8
|
import { a as scoreAnalystFindings, i as summarizeAnalystBenchmarkRunner, n as runAnalystBenchmark, r as traceStoreEvidenceResolver, t as registryBenchmarkRunner } from "../benchmark-CYtcIF2V.js";
|
|
9
|
-
import { $ as normalizeAgentRxCategory, A as parseVerificationOutcome, B as analystBenchmarkDependencyLockDigest, C as MAX_INCORRECT_BLOCK_STEPS, D as DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES, E as publicBenchmarkSystemPrompt, F as ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256, G as ANALYST_BENCHMARK_OBSERVATIONS_FILE, H as ANALYST_BENCHMARK_COST_LEDGER_FILE, I as ANALYST_BENCHMARK_EVIDENCE_PACKAGE_VERSION, J as summarizeAgentRxCalibration, K as AGENT_RX_UPSTREAM_REVISION, L as ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM, M as ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES, N as ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256, O as appendVerificationArtifactsToOtlp, P as ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256, Q as agentRxPredictionsToFindings, R as ANALYST_BENCHMARK_IMPLEMENTATION_FILES, S as MAX_INCORRECT_BLOCKS, T as publicBenchmarkRlmInstructions, U as ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE, V as analystBenchmarkImplementationDigest, W as ANALYST_BENCHMARK_MANIFEST_FILE, X as codeTracerPredictionsToFindings, Y as codeTraceBenchCase, Z as agentRxBenchmarkCase, _ as compareAnalystRunners, a as preparePublicAnalystBenchmark, b as readAnalystInstructionsOverride, c as selectPublicBenchmarkRows, d as adaptPublicBenchmarkFindings, et as roundAgentRxStep, f as emptyPublicBenchmarkRunner, g as summarizeCodeTraceCalibration, h as renderCodeTraceCalibrationMarkdown, i as loadPublicBenchmarkRows, j as ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM, k as loadCodeTraceVerificationArtifacts, l as createPublicBenchmarkRlmRunner, m as readAnalystBenchmarkArtifact, n as runAnalystBenchmarkCommand, o as publicBenchmarkDistributions, p as expandCodeTraceFailureBlocks, q as renderAgentRxCalibrationMarkdown, r as renderAnalystBenchmarkMarkdown, s as publicBenchmarkSelectionReport, t as ANALYST_BENCHMARK_HELP, tt as normalizeBenchmarkLabel, u as createPublicBenchmarkDirectRunner, v as analystInstructionsOverrideFromText, w as publicBenchmarkProtocolSha256, x as CODE_TRACE_BENCH_ANALYST_PROMPT, y as effectiveAnalystProtocolSha256, z as ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 } from "../benchmark-command-
|
|
9
|
+
import { $ as normalizeAgentRxCategory, A as parseVerificationOutcome, B as analystBenchmarkDependencyLockDigest, C as MAX_INCORRECT_BLOCK_STEPS, D as DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES, E as publicBenchmarkSystemPrompt, F as ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256, G as ANALYST_BENCHMARK_OBSERVATIONS_FILE, H as ANALYST_BENCHMARK_COST_LEDGER_FILE, I as ANALYST_BENCHMARK_EVIDENCE_PACKAGE_VERSION, J as summarizeAgentRxCalibration, K as AGENT_RX_UPSTREAM_REVISION, L as ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM, M as ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES, N as ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256, O as appendVerificationArtifactsToOtlp, P as ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256, Q as agentRxPredictionsToFindings, R as ANALYST_BENCHMARK_IMPLEMENTATION_FILES, S as MAX_INCORRECT_BLOCKS, T as publicBenchmarkRlmInstructions, U as ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE, V as analystBenchmarkImplementationDigest, W as ANALYST_BENCHMARK_MANIFEST_FILE, X as codeTracerPredictionsToFindings, Y as codeTraceBenchCase, Z as agentRxBenchmarkCase, _ as compareAnalystRunners, a as preparePublicAnalystBenchmark, b as readAnalystInstructionsOverride, c as selectPublicBenchmarkRows, d as adaptPublicBenchmarkFindings, et as roundAgentRxStep, f as emptyPublicBenchmarkRunner, g as summarizeCodeTraceCalibration, h as renderCodeTraceCalibrationMarkdown, i as loadPublicBenchmarkRows, j as ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM, k as loadCodeTraceVerificationArtifacts, l as createPublicBenchmarkRlmRunner, m as readAnalystBenchmarkArtifact, n as runAnalystBenchmarkCommand, o as publicBenchmarkDistributions, p as expandCodeTraceFailureBlocks, q as renderAgentRxCalibrationMarkdown, r as renderAnalystBenchmarkMarkdown, s as publicBenchmarkSelectionReport, t as ANALYST_BENCHMARK_HELP, tt as normalizeBenchmarkLabel, u as createPublicBenchmarkDirectRunner, v as analystInstructionsOverrideFromText, w as publicBenchmarkProtocolSha256, x as CODE_TRACE_BENCH_ANALYST_PROMPT, y as effectiveAnalystProtocolSha256, z as ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 } from "../benchmark-command-CSddvlq7.js";
|
|
10
10
|
//#region src/analyst/adapters.ts
|
|
11
11
|
/**
|
|
12
12
|
* Adapter factories — lift each existing agent-eval primitive into the
|
|
@@ -5,8 +5,8 @@ import { c as callLlmJson, f as maximumChargeForLlmRequest, l as costReceiptFrom
|
|
|
5
5
|
import { O as evidenceRefsFromRawFinding, h as TRACE_ANALYSIS_LIMITS, i as runTraceAnalyst } from "./kind-factory-Bvwe3pup.js";
|
|
6
6
|
import { o as makeFinding, r as usageReceiptFromCostLedger } from "./usage-receipt-CgxMEBZq.js";
|
|
7
7
|
import { i as hashCanonical, r as canonicalString } from "./canonical-D011XM8r.js";
|
|
8
|
-
import { n as createRunCostLedger, r as fsCampaignStorage, t as acquireSingleRunLock } from "./single-run-lock-
|
|
9
|
-
import { t as createDspyRlmTraceEngine } from "./dspy-rlm-engine-
|
|
8
|
+
import { n as createRunCostLedger, r as fsCampaignStorage, t as acquireSingleRunLock } from "./single-run-lock-Do-8vBIJ.js";
|
|
9
|
+
import { t as createDspyRlmTraceEngine } from "./dspy-rlm-engine-CJ4sN8Wx.js";
|
|
10
10
|
import { c as writeLedgerFileAtomically, s as withLedgerFileLock } from "./ledger-core-Dxz0Rkwa.js";
|
|
11
11
|
import { E as pairedBootstrap } from "./statistics-ByxzSiOM.js";
|
|
12
12
|
import { i as otlpTextToTraceAnalysisStore, r as createOtlpBufferTraceStore, t as DEFAULT_MAX_TRACE_FILE_BYTES } from "./store-otlp-D4I90_vR.js";
|
|
@@ -1173,7 +1173,7 @@ const ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES = Object.freeze([
|
|
|
1173
1173
|
"package.json",
|
|
1174
1174
|
"pnpm-lock.yaml"
|
|
1175
1175
|
]);
|
|
1176
|
-
const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "
|
|
1176
|
+
const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "2867725e505a600f4f1f9d934fb797f5fb94a71c906577d05009aecc254f85f3";
|
|
1177
1177
|
/** The published benchmark evidence was produced at this package version, by
|
|
1178
1178
|
* the retired one-shot direct runner, before trace analysts moved to the
|
|
1179
1179
|
* recursive DSPy RLM engine. Both evidence digests below are historical facts
|
|
@@ -1269,7 +1269,7 @@ const ANALYST_BENCHMARK_IMPLEMENTATION_FILES = Object.freeze([
|
|
|
1269
1269
|
"src/trace/otlp-attributes.ts",
|
|
1270
1270
|
"src/trace/raw-provider-sink.ts"
|
|
1271
1271
|
]);
|
|
1272
|
-
const ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 = "
|
|
1272
|
+
const ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 = "b8b6d4f7a01f8341a0565159806f93ff9ca71c88322e63e4e52a86eed2c1d9c7";
|
|
1273
1273
|
function analystBenchmarkImplementationDigest() {
|
|
1274
1274
|
return ANALYST_BENCHMARK_IMPLEMENTATION_SHA256;
|
|
1275
1275
|
}
|
|
@@ -5247,4 +5247,4 @@ function shellQuote(value) {
|
|
|
5247
5247
|
//#endregion
|
|
5248
5248
|
export { normalizeAgentRxCategory as $, parseVerificationOutcome as A, analystBenchmarkDependencyLockDigest as B, MAX_INCORRECT_BLOCK_STEPS as C, DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES as D, publicBenchmarkSystemPrompt as E, ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256 as F, ANALYST_BENCHMARK_OBSERVATIONS_FILE as G, ANALYST_BENCHMARK_COST_LEDGER_FILE as H, ANALYST_BENCHMARK_EVIDENCE_PACKAGE_VERSION as I, summarizeAgentRxCalibration as J, AGENT_RX_UPSTREAM_REVISION as K, ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM as L, ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES as M, ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 as N, appendVerificationArtifactsToOtlp as O, ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256 as P, agentRxPredictionsToFindings as Q, ANALYST_BENCHMARK_IMPLEMENTATION_FILES as R, MAX_INCORRECT_BLOCKS as S, publicBenchmarkRlmInstructions as T, ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE as U, analystBenchmarkImplementationDigest as V, ANALYST_BENCHMARK_MANIFEST_FILE as W, codeTracerPredictionsToFindings as X, codeTraceBenchCase as Y, agentRxBenchmarkCase as Z, compareAnalystRunners as _, preparePublicAnalystBenchmark as a, readAnalystInstructionsOverride as b, selectPublicBenchmarkRows as c, adaptPublicBenchmarkFindings as d, roundAgentRxStep as et, emptyPublicBenchmarkRunner as f, summarizeCodeTraceCalibration as g, renderCodeTraceCalibrationMarkdown as h, loadPublicBenchmarkRows as i, ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM as j, loadCodeTraceVerificationArtifacts as k, createPublicBenchmarkRlmRunner as l, readAnalystBenchmarkArtifact as m, runAnalystBenchmarkCommand as n, publicBenchmarkDistributions as o, expandCodeTraceFailureBlocks as p, renderAgentRxCalibrationMarkdown as q, renderAnalystBenchmarkMarkdown as r, publicBenchmarkSelectionReport as s, ANALYST_BENCHMARK_HELP as t, normalizeBenchmarkLabel as tt, createPublicBenchmarkDirectRunner as u, analystInstructionsOverrideFromText as v, publicBenchmarkProtocolSha256 as w, CODE_TRACE_BENCH_ANALYST_PROMPT as x, effectiveAnalystProtocolSha256 as y, ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 as z };
|
|
5249
5249
|
|
|
5250
|
-
//# sourceMappingURL=benchmark-command-
|
|
5250
|
+
//# sourceMappingURL=benchmark-command-CSddvlq7.js.map
|