@tangle-network/agent-eval 0.194.0 → 0.195.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/dist/adapters/http.d.ts +1 -1
- package/dist/analyst/index.d.ts +1 -1
- package/dist/analyst/index.js +1 -1
- package/dist/{benchmark-command-Boqtfx4Z.js → benchmark-command-D2kaOHat.js} +2 -2
- package/dist/{benchmark-command-Boqtfx4Z.js.map → benchmark-command-D2kaOHat.js.map} +1 -1
- package/dist/benchmarks/index.d.ts +1 -1
- package/dist/benchmarks/index.js +2 -2
- package/dist/campaign/index.d.ts +2 -2
- package/dist/campaign/index.js +4 -4
- package/dist/{campaign-Cnx0Duzs.js → campaign-DkMRR1Co.js} +4 -4
- package/dist/{campaign-Cnx0Duzs.js.map → campaign-DkMRR1Co.js.map} +1 -1
- package/dist/cli.js +1 -1
- package/dist/contract/index.d.ts +3 -3
- package/dist/contract/index.js +3 -3
- package/dist/{define-agent-eval-Dq0MgLnw.js → define-agent-eval-Ddut4mkS.js} +2 -2
- package/dist/{define-agent-eval-Dq0MgLnw.js.map → define-agent-eval-Ddut4mkS.js.map} +1 -1
- package/dist/{define-agent-eval-CKTbbeyT.d.ts → define-agent-eval-Dl0tmtJU.d.ts} +2 -2
- package/dist/{define-agent-eval-CKTbbeyT.d.ts.map → define-agent-eval-Dl0tmtJU.d.ts.map} +1 -1
- package/dist/{index-DKUIMOst.d.ts → index-CZYB36IQ.d.ts} +3 -3
- package/dist/{index-DKUIMOst.d.ts.map → index-CZYB36IQ.d.ts.map} +1 -1
- package/dist/{index-Dvc9zaWD.d.ts → index-CdHX0_c3.d.ts} +81 -13
- package/dist/index-CdHX0_c3.d.ts.map +1 -0
- package/dist/index.d.ts +2 -2
- package/dist/index.js +3 -3
- package/dist/{llm-judge-D6NIvyrM.js → llm-judge-CvKpaRKY.js} +370 -38
- package/dist/llm-judge-CvKpaRKY.js.map +1 -0
- package/dist/openapi.json +1 -1
- package/dist/{produced-state-BjkNRWAe.js → produced-state-C8rk-GCK.js} +2 -2
- package/dist/{produced-state-BjkNRWAe.js.map → produced-state-C8rk-GCK.js.map} +1 -1
- package/dist/{skillopt-optimization-method-C4GiaBzp.js → skillopt-optimization-method-CIJqP8oS.js} +2 -2
- package/dist/{skillopt-optimization-method-C4GiaBzp.js.map → skillopt-optimization-method-CIJqP8oS.js.map} +1 -1
- package/docs/search-ledger.md +27 -5
- package/package.json +1 -1
- package/dist/index-Dvc9zaWD.d.ts.map +0 -1
- package/dist/llm-judge-D6NIvyrM.js.map +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,28 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
|
|
|
6
6
|
|
|
7
7
|
## Unreleased
|
|
8
8
|
|
|
9
|
+
## [0.195.0] — 2026-09-25
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `asha({ units, eta, trainUnits, reps })` (`/campaign`) is a `SearchAllocator` for asynchronous successive halving over one permutation of the selection units, seeded by the search's seed ([search ledger](./docs/search-ledger.md#run-a-search-the-kernel)).
|
|
14
|
+
Rung k is the first `units × 2^k` units (default 6, 12, 24, ...); the root runs every unit first, and a new node screens on rung 0 plus 2 train units that only the proposer reads.
|
|
15
|
+
Every node at a rung runs the units its parent, the root and its siblings ran, so every contrast pairs.
|
|
16
|
+
A node that finished rung k advances once it ranks in the top floor(n / 3) of the n nodes that finished rung k (the root included, invalid nodes excluded); there is no barrier, and a node left waiting is decided `pruned` with its rank at close.
|
|
17
|
+
In 200 seeded searches over a pool of 50 candidates with one planted 0.10 above the rest (24 selection units), `asha` kept the same node as `uniform` in 200 of 200, the planted one every time, with 45.8 % of the cells; every measured edge paired on at least 6 units.
|
|
18
|
+
- The kernel records the allocator's rank decisions: an `advanced {rung}` decision is appended only once the cap admits the rung's cells, which the kernel then allocates, and rungs keep opening after expansion stops, until the deadline or the claim.
|
|
19
|
+
A resumed `asha` search holds only rank decisions its own ledger re-derives; because a restart changes the order cells finish in, it may promote differently from an uninterrupted run.
|
|
20
|
+
- `scripts/search-sim.ts compare` runs one search per seed under `uniform` and `asha` and reports cells, the kept node, node statuses and the units each edge pairs on; `--pool-gap` swaps the hill climb for a fixed pool with one planted best candidate.
|
|
21
|
+
Every simulated run re-derives each `advanced` and `pruned` decision, with its rule, rank reason and estimate, from the ledger before it.
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
|
|
25
|
+
- **Breaking:** `SearchAllocator.plan(state, nodeId, rung)` takes the node's rung, and an allocator implements `advance(view)` and `prune(view, keep)`; `uniform` returns no decisions.
|
|
26
|
+
- `incumbent` and `crowdedFrontierParent`: a node leads only when it scored every unit the leader scored and beats the leader's mean on them, so a node an allocator has only screened cannot take the lead on less evidence than the leader holds.
|
|
27
|
+
A node dodged a unit only when a cell ran and ended unscored; a cell still to run no longer counts against it.
|
|
28
|
+
- The divergence check judges a node's screen once; a node an allocator advanced is not judged again on its rung cells, after a restart either.
|
|
29
|
+
- Migration: a custom allocator adds a `rung` parameter to `plan` and returns `[]` from `advance` and `prune` to keep its behavior.
|
|
30
|
+
|
|
9
31
|
## [0.194.0] — 2026-09-25
|
|
10
32
|
|
|
11
33
|
### Fixed
|
package/dist/adapters/http.d.ts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { m as CostReceipt } from "../cost-ledger-DbQdN3nO.js";
|
|
2
2
|
import { R as Scenario, d as DispatchContext, f as DispatchFn } from "../types-DLtLe21j.js";
|
|
3
|
-
import "../index-
|
|
3
|
+
import "../index-CZYB36IQ.js";
|
|
4
4
|
//#region src/adapters/http.d.ts
|
|
5
5
|
interface HttpDispatchOptions<TScenario extends Scenario, _TArtifact> {
|
|
6
6
|
/** Static endpoint URL. Mutually exclusive with `resolveUrl`. */
|
package/dist/analyst/index.d.ts
CHANGED
|
@@ -1181,7 +1181,7 @@ declare function runAnalystBenchmarkCommand(argv: readonly string[], env?: NodeJ
|
|
|
1181
1181
|
declare const ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM = "sha256-canonical-source-manifest";
|
|
1182
1182
|
declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM = "sha256-canonical-file-manifest";
|
|
1183
1183
|
declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES: readonly string[];
|
|
1184
|
-
declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "
|
|
1184
|
+
declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "85a2c8e125ae61eb13d329f837957a5f530a480131db44644884206513214961";
|
|
1185
1185
|
declare const ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256 = "1e03f2daed356d60316aabefb407ec1e437ac94d408d61eea4ae096e9c6fbb5b";
|
|
1186
1186
|
declare const ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256 = "4dba263b6256a30d56c7fdb2d992d3a953c0035d731f359b704db806f68f75ac";
|
|
1187
1187
|
declare const ANALYST_BENCHMARK_IMPLEMENTATION_FILES: readonly string[];
|
package/dist/analyst/index.js
CHANGED
|
@@ -8,7 +8,7 @@ import { t as createDspyRlmTraceEngine } from "../dspy-rlm-engine-k_QdEBZo.js";
|
|
|
8
8
|
import { a as diffFindings, i as defaultIsMaterial, n as runSemanticConceptJudge, o as createChatTraceEngine, r as FindingsStore, t as SEMANTIC_CONCEPT_JUDGE_VERSION } from "../semantic-concept-judge-Bh5lvKGg.js";
|
|
9
9
|
import { t as createChatClient } from "../chat-client-CJORSG_J.js";
|
|
10
10
|
import { a as scoreAnalystFindings, i as summarizeAnalystBenchmarkRunner, n as runAnalystBenchmark, r as traceStoreEvidenceResolver, t as registryBenchmarkRunner } from "../benchmark-BeQJyrRC.js";
|
|
11
|
-
import { $ as ANALYST_BENCHMARK_MANIFEST_FILE, A as readAnalystInstructionsOverride, B as parseVerificationOutcome, C as emptyPublicBenchmarkRunner, D as summarizeCodeTraceCalibration, E as renderCodeTraceCalibrationMarkdown, F as publicBenchmarkRlmInstructions, G as ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256, H as ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES, I as publicBenchmarkSystemPrompt, J as ANALYST_BENCHMARK_IMPLEMENTATION_SHA256, K as ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM, L as DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES, M as MAX_INCORRECT_BLOCKS, N as MAX_INCORRECT_BLOCK_STEPS, O as analystInstructionsOverrideFromText, P as publicBenchmarkProtocolSha256, Q as ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE, R as appendVerificationArtifactsToOtlp, S as adaptPublicBenchmarkFindings, T as readAnalystBenchmarkArtifact, U as ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256, V as ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM, W as ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256, X as analystBenchmarkImplementationDigest, Y as analystBenchmarkDependencyLockDigest, Z as ANALYST_BENCHMARK_COST_LEDGER_FILE, _ as publicDirectAnalystDefinition, a as primeCodeTraceAnalystDefinition, at as codeTraceBenchCase, b as analystDefinitionAsymmetries, c as preparePublicAnalystBenchmark, ct as agentRxPredictionsToFindings, d as selectPublicBenchmarkRows, dt as normalizeBenchmarkLabel, et as ANALYST_BENCHMARK_OBSERVATIONS_FILE, f as createPublicBenchmarkRlmRunner, g as createPublicBenchmarkDirectRunner, h as runReplVariableAnalystDefinition, i as primeAnalystProtocolSha256, it as summarizeAgentRxCalibration, j as CODE_TRACE_BENCH_ANALYST_PROMPT, k as effectiveAnalystProtocolSha256, l as publicBenchmarkDistributions, lt as normalizeAgentRxCategory, m as rlmEngineLimits, n as renderAnalystBenchmarkMarkdown, nt as AGENT_RX_UPSTREAM_REVISION, o as runInlineAnalystDefinition, ot as codeTracerPredictionsToFindings, p as publicRlmAnalystDefinition, q as ANALYST_BENCHMARK_IMPLEMENTATION_FILES, r as createPrimeBenchmarkRunner, rt as renderAgentRxCalibrationMarkdown, s as loadPublicBenchmarkRows, st as agentRxBenchmarkCase, t as runAnalystBenchmarkCommand, tt as compareAnalystRunners, u as publicBenchmarkSelectionReport, ut as roundAgentRxStep, v as runChunkedAnalystDefinition, w as expandCodeTraceFailureBlocks, x as analystDefinitionProtocolSha256, y as AnalystExpressivenessError, z as loadCodeTraceVerificationArtifacts } from "../benchmark-command-
|
|
11
|
+
import { $ as ANALYST_BENCHMARK_MANIFEST_FILE, A as readAnalystInstructionsOverride, B as parseVerificationOutcome, C as emptyPublicBenchmarkRunner, D as summarizeCodeTraceCalibration, E as renderCodeTraceCalibrationMarkdown, F as publicBenchmarkRlmInstructions, G as ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256, H as ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES, I as publicBenchmarkSystemPrompt, J as ANALYST_BENCHMARK_IMPLEMENTATION_SHA256, K as ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM, L as DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES, M as MAX_INCORRECT_BLOCKS, N as MAX_INCORRECT_BLOCK_STEPS, O as analystInstructionsOverrideFromText, P as publicBenchmarkProtocolSha256, Q as ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE, R as appendVerificationArtifactsToOtlp, S as adaptPublicBenchmarkFindings, T as readAnalystBenchmarkArtifact, U as ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256, V as ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM, W as ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256, X as analystBenchmarkImplementationDigest, Y as analystBenchmarkDependencyLockDigest, Z as ANALYST_BENCHMARK_COST_LEDGER_FILE, _ as publicDirectAnalystDefinition, a as primeCodeTraceAnalystDefinition, at as codeTraceBenchCase, b as analystDefinitionAsymmetries, c as preparePublicAnalystBenchmark, ct as agentRxPredictionsToFindings, d as selectPublicBenchmarkRows, dt as normalizeBenchmarkLabel, et as ANALYST_BENCHMARK_OBSERVATIONS_FILE, f as createPublicBenchmarkRlmRunner, g as createPublicBenchmarkDirectRunner, h as runReplVariableAnalystDefinition, i as primeAnalystProtocolSha256, it as summarizeAgentRxCalibration, j as CODE_TRACE_BENCH_ANALYST_PROMPT, k as effectiveAnalystProtocolSha256, l as publicBenchmarkDistributions, lt as normalizeAgentRxCategory, m as rlmEngineLimits, n as renderAnalystBenchmarkMarkdown, nt as AGENT_RX_UPSTREAM_REVISION, o as runInlineAnalystDefinition, ot as codeTracerPredictionsToFindings, p as publicRlmAnalystDefinition, q as ANALYST_BENCHMARK_IMPLEMENTATION_FILES, r as createPrimeBenchmarkRunner, rt as renderAgentRxCalibrationMarkdown, s as loadPublicBenchmarkRows, st as agentRxBenchmarkCase, t as runAnalystBenchmarkCommand, tt as compareAnalystRunners, u as publicBenchmarkSelectionReport, ut as roundAgentRxStep, v as runChunkedAnalystDefinition, w as expandCodeTraceFailureBlocks, x as analystDefinitionProtocolSha256, y as AnalystExpressivenessError, z as loadCodeTraceVerificationArtifacts } from "../benchmark-command-D2kaOHat.js";
|
|
12
12
|
import { a as extractPrimeJsonObject, c as primeProtocolSha256, d as runPrimeExchange, f as decodeReplyRows, i as emptyPrimeRawUsage, l as primeReplyDefect, n as buildPrimePrompt, o as mergePrimeRawUsage, r as buildPrimeRepairPrompt, s as normalizePrimeUsage, t as analystUsageReceiptFromPrimeUsage, u as projectPrimeTrajectory } from "../prime-protocol-DcAK9pC8.js";
|
|
13
13
|
import { t as nodeHttpPrimeBridgeTransport } from "../prime-bridge-transport-Z3cpKGoV.js";
|
|
14
14
|
import { existsSync, readFileSync, readdirSync, statSync } from "node:fs";
|
|
@@ -1415,7 +1415,7 @@ const ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES = Object.freeze([
|
|
|
1415
1415
|
"package.json",
|
|
1416
1416
|
"pnpm-lock.yaml"
|
|
1417
1417
|
]);
|
|
1418
|
-
const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "
|
|
1418
|
+
const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "85a2c8e125ae61eb13d329f837957a5f530a480131db44644884206513214961";
|
|
1419
1419
|
const ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256 = "1e03f2daed356d60316aabefb407ec1e437ac94d408d61eea4ae096e9c6fbb5b";
|
|
1420
1420
|
const ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256 = "4dba263b6256a30d56c7fdb2d992d3a953c0035d731f359b704db806f68f75ac";
|
|
1421
1421
|
const ANALYST_BENCHMARK_IMPLEMENTATION_FILES = Object.freeze([
|
|
@@ -6316,4 +6316,4 @@ function shellQuote(value) {
|
|
|
6316
6316
|
//#endregion
|
|
6317
6317
|
export { ANALYST_BENCHMARK_MANIFEST_FILE as $, readAnalystInstructionsOverride as A, parseVerificationOutcome as B, emptyPublicBenchmarkRunner as C, summarizeCodeTraceCalibration as D, renderCodeTraceCalibrationMarkdown as E, publicBenchmarkRlmInstructions as F, ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256 as G, ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES as H, publicBenchmarkSystemPrompt as I, ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 as J, ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM as K, DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES as L, MAX_INCORRECT_BLOCKS as M, MAX_INCORRECT_BLOCK_STEPS as N, analystInstructionsOverrideFromText as O, publicBenchmarkProtocolSha256 as P, ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE as Q, appendVerificationArtifactsToOtlp as R, adaptPublicBenchmarkFindings as S, readAnalystBenchmarkArtifact as T, ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 as U, ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM as V, ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256 as W, analystBenchmarkImplementationDigest as X, analystBenchmarkDependencyLockDigest as Y, ANALYST_BENCHMARK_COST_LEDGER_FILE as Z, publicDirectAnalystDefinition as _, primeCodeTraceAnalystDefinition as a, codeTraceBenchCase as at, analystDefinitionAsymmetries as b, preparePublicAnalystBenchmark as c, agentRxPredictionsToFindings as ct, selectPublicBenchmarkRows as d, normalizeBenchmarkLabel as dt, ANALYST_BENCHMARK_OBSERVATIONS_FILE as et, createPublicBenchmarkRlmRunner as f, createPublicBenchmarkDirectRunner as g, runReplVariableAnalystDefinition as h, primeAnalystProtocolSha256 as i, summarizeAgentRxCalibration as it, CODE_TRACE_BENCH_ANALYST_PROMPT as j, effectiveAnalystProtocolSha256 as k, publicBenchmarkDistributions as l, normalizeAgentRxCategory as lt, rlmEngineLimits as m, renderAnalystBenchmarkMarkdown as n, AGENT_RX_UPSTREAM_REVISION as nt, runInlineAnalystDefinition as o, codeTracerPredictionsToFindings as ot, publicRlmAnalystDefinition as p, ANALYST_BENCHMARK_IMPLEMENTATION_FILES as q, createPrimeBenchmarkRunner as r, renderAgentRxCalibrationMarkdown as rt, loadPublicBenchmarkRows as s, agentRxBenchmarkCase as st, runAnalystBenchmarkCommand as t, compareAnalystRunners as tt, publicBenchmarkSelectionReport as u, roundAgentRxStep as ut, runChunkedAnalystDefinition as v, expandCodeTraceFailureBlocks as w, analystDefinitionProtocolSha256 as x, AnalystExpressivenessError as y, loadCodeTraceVerificationArtifacts as z };
|
|
6318
6318
|
|
|
6319
|
-
//# sourceMappingURL=benchmark-command-
|
|
6319
|
+
//# sourceMappingURL=benchmark-command-D2kaOHat.js.map
|