@tangle-network/agent-eval 0.20.2 → 0.20.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +124 -305
- package/dist/{chunk-OZPRSK4A.js → chunk-CJJSB6ZQ.js} +2 -2
- package/dist/{chunk-ITN4YOZY.js → chunk-JAOLXRIA.js} +52 -2
- package/dist/chunk-JAOLXRIA.js.map +1 -0
- package/dist/cli.js +2 -2
- package/dist/index.d.ts +442 -1
- package/dist/index.js +1024 -112
- package/dist/index.js.map +1 -1
- package/dist/wire/index.js +2 -2
- package/examples/benchmarks/README.md +44 -0
- package/examples/benchmarks/gsm8k/index.ts +126 -0
- package/examples/benchmarks/swebench-lite/index.ts +178 -0
- package/examples/multi-shot-optimization/index.ts +114 -0
- package/examples/same-sandbox-harness/index.ts +63 -0
- package/package.json +15 -12
- package/dist/chunk-ITN4YOZY.js.map +0 -1
- /package/dist/{chunk-OZPRSK4A.js.map → chunk-CJJSB6ZQ.js.map} +0 -0
package/README.md
CHANGED
|
@@ -1,343 +1,162 @@
|
|
|
1
1
|
# @tangle-network/agent-eval
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
3
|
+
Trace-first evaluation infrastructure for agent systems.
|
|
4
|
+
|
|
5
|
+
`agent-eval` provides the contracts and runtime primitives for measuring agent
|
|
6
|
+
behavior: traces, harnesses, verifier pipelines, judges, datasets, holdout
|
|
7
|
+
gates, failure classification, optimization loops, and release reports.
|
|
8
|
+
|
|
9
|
+
It does not own your product state, credentials, UI, or model routing. Product
|
|
10
|
+
teams keep those boundaries; this package standardizes how runs are recorded,
|
|
11
|
+
checked, compared, and promoted.
|
|
12
|
+
|
|
13
|
+
## Contents
|
|
14
|
+
|
|
15
|
+
- [When To Use It](#when-to-use-it)
|
|
16
|
+
- [Architecture](#architecture)
|
|
17
|
+
- [Install](#install)
|
|
18
|
+
- [Core Primitives](#core-primitives)
|
|
19
|
+
- [Examples](#examples)
|
|
20
|
+
- [Documentation](#documentation)
|
|
21
|
+
- [Development](#development)
|
|
22
|
+
- [Related Packages](#related-packages)
|
|
23
|
+
|
|
24
|
+
## When To Use It
|
|
25
|
+
|
|
26
|
+
Use `agent-eval` when you need one or more of these:
|
|
27
|
+
|
|
28
|
+
- A reproducible eval harness for coding agents, builder agents, or multi-tool
|
|
29
|
+
workflows.
|
|
30
|
+
- Structured traces for agent runs: spans, artifacts, events, budgets, tool
|
|
31
|
+
calls, retrieval, judge output, and sandbox execution.
|
|
32
|
+
- Deterministic gates around build/test/deploy checks.
|
|
33
|
+
- LLM-as-judge or deterministic judge fleets with calibration and canaries.
|
|
34
|
+
- Dataset splits, holdouts, paired statistics, and release confidence gates.
|
|
35
|
+
- Failure taxonomy that distinguishes prompt, tool, sandbox, retrieval,
|
|
36
|
+
evaluator, and knowledge-readiness failures.
|
|
37
|
+
- Optimization loops over prompts, steering, code mutations, or full multi-shot
|
|
38
|
+
trajectories.
|
|
39
|
+
- Report data for internal launch reviews, CI gates, and research analysis.
|
|
40
|
+
|
|
41
|
+
## Architecture
|
|
42
|
+
|
|
43
|
+
```txt
|
|
44
|
+
agent/product run
|
|
45
|
+
-> TraceEmitter / TraceStore
|
|
46
|
+
-> SandboxHarness / MultiLayerVerifier / JudgeRunner
|
|
47
|
+
-> failure taxonomy + metrics
|
|
48
|
+
-> paired stats + held-out gates
|
|
49
|
+
-> optimization + release confidence + reports
|
|
16
50
|
```
|
|
17
51
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
- You ship a code generator (scaffolder, patcher, refactor agent) and need to gate on whether its output actually works.
|
|
21
|
-
- You ship a content generator and need quality signal beyond "the LLM said it's good".
|
|
22
|
-
- You want a release gate that fails on regressions you can name, not vibes.
|
|
52
|
+
Package responsibilities:
|
|
23
53
|
|
|
24
|
-
|
|
54
|
+
- `agent-eval`: run evidence, eval contracts, verification, statistics,
|
|
55
|
+
optimization, reporting.
|
|
56
|
+
- Product app: domain state, tools, credentials, UI, storage, deployment, model
|
|
57
|
+
gateway.
|
|
58
|
+
- `agent-runtime`: production agent-loop/session runtime.
|
|
59
|
+
- `agent-knowledge`: evidence stores, claim/page synthesis, retrieval, knowledge
|
|
60
|
+
readiness implementation.
|
|
25
61
|
|
|
26
|
-
##
|
|
62
|
+
## Install
|
|
27
63
|
|
|
28
|
-
|
|
64
|
+
```sh
|
|
65
|
+
pnpm add @tangle-network/agent-eval
|
|
66
|
+
```
|
|
29
67
|
|
|
30
|
-
|
|
68
|
+
Wire protocol / CLI:
|
|
31
69
|
|
|
32
70
|
```sh
|
|
33
71
|
npm i -g @tangle-network/agent-eval
|
|
34
|
-
|
|
35
|
-
# HTTP — long-running
|
|
36
72
|
agent-eval serve --port 5005
|
|
37
|
-
|
|
38
|
-
# stdio RPC — one-shot, batch
|
|
39
|
-
echo '{"rubricName":"anti-slop","content":"…"}' | agent-eval rpc judge
|
|
40
73
|
```
|
|
41
74
|
|
|
42
|
-
Python:
|
|
75
|
+
Python client:
|
|
76
|
+
|
|
43
77
|
```sh
|
|
44
78
|
pip install tangle-agent-eval
|
|
45
79
|
```
|
|
46
|
-
```python
|
|
47
|
-
from tangle_agent_eval import Client
|
|
48
|
-
c = Client()
|
|
49
|
-
r = c.judge(content="our scaffold ships zero-copy IO", rubric_name="anti-slop")
|
|
50
|
-
print(r.composite, r.failure_modes)
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
See [`docs/wire-protocol.md`](./docs/wire-protocol.md) for the full surface.
|
|
54
80
|
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
81
|
+
## Core Primitives
|
|
82
|
+
|
|
83
|
+
| Primitive | Purpose |
|
|
84
|
+
|---|---|
|
|
85
|
+
| `TraceEmitter`, `TraceStore` | Append-only run/span/event/artifact/budget records. |
|
|
86
|
+
| `SandboxHarness` | Build/test/runtime checks with captured stdout, stderr, exit codes, wall time, and parsed test counts. |
|
|
87
|
+
| `MultiLayerVerifier` | Ordered verification stages with dependencies, skip-on-fail, findings, scores, and time caps. |
|
|
88
|
+
| `JudgeRunner` | Parallel deterministic or LLM-backed judges over the same artifact/run. |
|
|
89
|
+
| `runAgentControlLoop` | Observe/validate/decide/act loop with budgets, stop policies, and structured eval results. |
|
|
90
|
+
| `Dataset`, `RunRecord`, `HeldOutGate` | Versioned corpora, reproducible run metadata, and held-out promotion decisions. |
|
|
91
|
+
| `pairedBootstrap`, `pairedWilcoxon`, `bhAdjust` | Paired experiment statistics and multiple-comparison correction. |
|
|
92
|
+
| `classifyFailure` | Rule-based failure classification for agent, tool, sandbox, retrieval, and knowledge failures. |
|
|
93
|
+
| `runMultiShotOptimization` | Optimization over full agent trajectories with actionable side information. |
|
|
94
|
+
| `runPromptEvolution` | Prompt/steering/code evolution over scenario scores. |
|
|
95
|
+
| `evaluateReleaseConfidence` | Release scorecard across evidence volume, pass rate, score, overfit, cost, latency, and gates. |
|
|
96
|
+
| `summaryTable`, `paretoChart`, `gainHistogram` | Report-ready structured outputs. |
|
|
97
|
+
| `KnowledgeRequirement`, `KnowledgeBundle` | Shared contracts for knowledge readiness. |
|
|
98
|
+
|
|
99
|
+
## Examples
|
|
100
|
+
|
|
101
|
+
Runnable examples live in [`examples/`](./examples):
|
|
102
|
+
|
|
103
|
+
- [`examples/same-sandbox-harness`](./examples/same-sandbox-harness) - run
|
|
104
|
+
multiple eval passes against the same workspace.
|
|
105
|
+
- [`examples/multi-shot-optimization`](./examples/multi-shot-optimization) -
|
|
106
|
+
optimize full agent trajectories with held-out promotion.
|
|
107
|
+
- [`examples/benchmarks`](./examples/benchmarks) - benchmark adapter shape and
|
|
108
|
+
reference benchmark wrappers.
|
|
109
|
+
|
|
110
|
+
The examples are intentionally kept outside the README so they can be expanded,
|
|
111
|
+
tested, and copied without turning this page into a tutorial.
|
|
112
|
+
|
|
113
|
+
## Documentation
|
|
114
|
+
|
|
115
|
+
- [Concepts](./docs/concepts.md)
|
|
116
|
+
- [Feature Guide](./docs/feature-guide.md)
|
|
117
|
+
- [Control Runtime](./docs/control-runtime.md)
|
|
118
|
+
- [Knowledge Readiness](./docs/knowledge-readiness.md)
|
|
119
|
+
- [Multi-Shot Optimization](./docs/multi-shot-optimization.md)
|
|
120
|
+
- [Feedback Trajectories](./docs/feedback-trajectories.md)
|
|
121
|
+
- [Wire Protocol](./docs/wire-protocol.md)
|
|
122
|
+
|
|
123
|
+
## Development
|
|
58
124
|
|
|
59
125
|
```sh
|
|
60
|
-
pnpm
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
## Two ways to read this repo
|
|
66
|
-
|
|
67
|
-
- **You're a human onboarding** — read [`docs/concepts.md`](./docs/concepts.md) for the mental model, then [`docs/wire-protocol.md`](./docs/wire-protocol.md) if you'll call from another language, or `SKILL.md` if you'll embed in TS.
|
|
68
|
-
- **You're deciding what to integrate** — read [`docs/feature-guide.md`](./docs/feature-guide.md) for the layman explanation, use cases, feature map, and guardrails.
|
|
69
|
-
- **You're an LLM agent writing integration code** — read `SKILL.md`. Every directive there encodes a shipped bug; skipping one reintroduces the bug class.
|
|
70
|
-
|
|
71
|
-
## What's in the box
|
|
72
|
-
|
|
73
|
-
| Module | What it does | Doc |
|
|
74
|
-
|---|---|---|
|
|
75
|
-
| `BuilderSession` | Three-layer eval orchestrator (builder → app-build → app-runtime) for code generators. | concepts.md §three-layer eval |
|
|
76
|
-
| `MultiLayerVerifier` | Pipeline of layers (install → typecheck → build → semantic). Skip-on-fail, weighted aggregate. | concepts.md §verifiers |
|
|
77
|
-
| `judges`, `createCustomJudge`, `createAntiSlopJudge` | LLM and deterministic judges. | SKILL.md |
|
|
78
|
-
| Wire protocol (`agent-eval serve` / `rpc`) | HTTP and stdio RPC interface for cross-language clients. | wire-protocol.md |
|
|
79
|
-
| `clients/python/` | First-party Python client (`tangle-agent-eval` on PyPI). Version-locked to npm. | clients/python/README.md |
|
|
80
|
-
| `BenchmarkRunner`, `executeScenario`, `ConvergenceTracker` | Multi-turn scenario execution + cross-run tracking. | SKILL.md |
|
|
81
|
-
| `runAgentControlLoop` | Policy-based runtime for agentic tasks: observe typed state, validate, decide, act, repeat with budgets, tracing, and stuck-loop guards. | [control-runtime.md](./docs/control-runtime.md) |
|
|
82
|
-
| `FeedbackTrajectory`, `InMemoryFeedbackTrajectoryStore`, `FileSystemFeedbackTrajectoryStore` | Human/environment feedback loops: capture approvals, rejections, choices, revisions, metrics, and policy blocks as train/dev/test/holdout examples. | [feedback-trajectories.md](./docs/feedback-trajectories.md) |
|
|
83
|
-
| `evaluateActionPolicy` | Generic action preflight for approval, budget, expected-outcome, and kill-criteria checks. | [feature-guide.md](./docs/feature-guide.md) |
|
|
84
|
-
| `ExperimentTracker`, steering optimizers, `bisector` | A/B prompts, optimize steering, bisect regressions. | SKILL.md |
|
|
85
|
-
| `runMultiShotOptimization`, `trialTraceFromMultiShotTrial` | GEPA-style optimization for variable-length agent trajectories with ASI, paired seeds, and optional held-out promotion gating. | [multi-shot-optimization.md](./docs/multi-shot-optimization.md) |
|
|
86
|
-
| `evaluateReleaseConfidence`, `assertReleaseConfidence` | Release scorecard that composes corpus coverage, search/holdout run evidence, ASI diagnostics, overfit checks, and cost/latency budgets. | §Release confidence |
|
|
87
|
-
| `runPromptEvolution`, `createCompositeMutator`, `createSandboxPool`, `createSandboxCodeMutator`, `MutationTelemetry`, `LineageRecorder`, `CostLedger`, `JsonlTrialCache` | Prompt + code evolution loops with bounded sandbox pools, durable JSONL telemetry, plateau-detecting composite mutators, crash-resumable trial cache. | §Evolution loop |
|
|
88
|
-
| `reflective-mutation` (`buildReflectionPrompt`, `parseReflectionResponse`, `DEFAULT_MUTATION_PRIMITIVES`) | Trace-conditioned LLM mutator that reasons over top/bottom trials instead of blind rewrites. | inline JSDoc |
|
|
89
|
-
| `correlationStudy`, `OutcomeStore`, `ProductRegistry` | Meta-eval: do our scores predict deployment outcomes (revenue, retention)? | inline JSDoc |
|
|
90
|
-
| Telemetry (`telemetry/`, `telemetry/file`) | OTLP export, trace replay, file sinks. | inline JSDoc |
|
|
91
|
-
|
|
92
|
-
## Release confidence
|
|
93
|
-
|
|
94
|
-
Use `evaluateReleaseConfidence` at the release boundary for every consuming
|
|
95
|
-
agent surface. It fails closed unless the release has a versioned corpus,
|
|
96
|
-
search and holdout run evidence, score/pass-rate evidence, ASI for failures,
|
|
97
|
-
and budget/overfit checks. Single-shot and multi-shot apps use the same path:
|
|
98
|
-
single-shot traces are just trace evidence with `turnCount: 1`.
|
|
99
|
-
|
|
100
|
-
```ts
|
|
101
|
-
import {
|
|
102
|
-
evaluateReleaseConfidence,
|
|
103
|
-
releaseTraceEvidenceFromMultiShotTrials,
|
|
104
|
-
} from '@tangle-network/agent-eval'
|
|
105
|
-
|
|
106
|
-
const scorecard = evaluateReleaseConfidence({
|
|
107
|
-
target: 'blueprint-agent/autoresearch',
|
|
108
|
-
candidateId: 'candidate-v3',
|
|
109
|
-
baselineId: 'baseline',
|
|
110
|
-
dataset: await dataset.manifest(),
|
|
111
|
-
runs: [...candidateRuns, ...baselineRuns],
|
|
112
|
-
traces: releaseTraceEvidenceFromMultiShotTrials(result.evolution.generations.flatMap((g) => g.trials)),
|
|
113
|
-
gateDecision: result.gate?.decision,
|
|
114
|
-
thresholds: {
|
|
115
|
-
minScenarioCount: 50,
|
|
116
|
-
minSearchRuns: 50,
|
|
117
|
-
minHoldoutRuns: 20,
|
|
118
|
-
minPassRate: 0.9,
|
|
119
|
-
minMeanScore: 0.8,
|
|
120
|
-
maxOverfitGap: 0.1,
|
|
121
|
-
maxMeanCostUsd: 0.05,
|
|
122
|
-
maxP95WallMs: 120_000,
|
|
123
|
-
},
|
|
124
|
-
})
|
|
125
|
-
|
|
126
|
-
if (!scorecard.promote) throw new Error(scorecard.summary)
|
|
126
|
+
pnpm install
|
|
127
|
+
pnpm typecheck
|
|
128
|
+
pnpm test
|
|
129
|
+
pnpm build
|
|
130
|
+
pnpm openapi
|
|
127
131
|
```
|
|
128
132
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
For agent tasks that run across many chat turns or tool calls, start with
|
|
132
|
-
[`runMultiShotOptimization`](./docs/multi-shot-optimization.md). It runs the
|
|
133
|
-
same prompt-evolution core over full trajectories, carries actionable side
|
|
134
|
-
information into reflection, and separates the search winner from the variant
|
|
135
|
-
that actually passes held-out promotion.
|
|
136
|
-
|
|
137
|
-
Closing the loop on a prompt or codebase is **two adapters + a config**. Compose `runPromptEvolution` with `createCompositeMutator` (plateau policy) and you get prompt-only optimization until improvement stalls, then automatic switch to code-channel mutations from a coding agent inside a `SandboxPool`.
|
|
138
|
-
|
|
139
|
-
```ts
|
|
140
|
-
import {
|
|
141
|
-
createSandboxPool,
|
|
142
|
-
createSandboxCodeMutator,
|
|
143
|
-
createCompositeMutator,
|
|
144
|
-
buildReflectionPrompt,
|
|
145
|
-
parseReflectionResponse,
|
|
146
|
-
runPromptEvolution,
|
|
147
|
-
MutationTelemetry,
|
|
148
|
-
LineageRecorder,
|
|
149
|
-
CostLedger,
|
|
150
|
-
JsonlTrialCache,
|
|
151
|
-
} from '@tangle-network/agent-eval'
|
|
152
|
-
|
|
153
|
-
// 1. Prompt mutator — reflective-mutation reasons over top/bottom trials
|
|
154
|
-
const promptMutator = {
|
|
155
|
-
async mutate({ parent, topTrials, bottomTrials, childCount }) {
|
|
156
|
-
const ctx = { target: 'forge-prompt', parentPayload: parent.payload, topTrials, bottomTrials, childCount }
|
|
157
|
-
const reflection = buildReflectionPrompt(ctx)
|
|
158
|
-
const raw = await yourLlm(reflection)
|
|
159
|
-
return parseReflectionResponse(raw, childCount).map((p, i) => ({
|
|
160
|
-
id: `${parent.id}.g${parent.generation + 1}.prompt.${i}`,
|
|
161
|
-
payload: p.payload,
|
|
162
|
-
generation: parent.generation + 1,
|
|
163
|
-
parentId: parent.id,
|
|
164
|
-
label: p.label,
|
|
165
|
-
rationale: p.rationale,
|
|
166
|
-
}))
|
|
167
|
-
},
|
|
168
|
-
}
|
|
169
|
-
|
|
170
|
-
// 2. Code mutator — runs a coding agent in a sandbox slot, captures the diff
|
|
171
|
-
const pool = createSandboxPool({
|
|
172
|
-
size: 4,
|
|
173
|
-
factory: {
|
|
174
|
-
async create(id) { return await yourSandboxClient.create({ name: id }) },
|
|
175
|
-
async reset(slot) { await slot.resource.exec('git reset --hard origin/main && git clean -fd') },
|
|
176
|
-
async destroy(slot) { await slot.resource.delete() },
|
|
177
|
-
},
|
|
178
|
-
})
|
|
179
|
-
const codeMutator = createSandboxCodeMutator({
|
|
180
|
-
pool,
|
|
181
|
-
runner: async ({ slot, parent, topTrials, bottomTrials }) => {
|
|
182
|
-
const result = await slot.resource.task(`Improve the prompt at /repo/forge-prompt.ts...`)
|
|
183
|
-
return [{ ok: true, latencyMs: result.durationMs, costUsd: result.costUsd, artifact: { diff: result.diff } }]
|
|
184
|
-
},
|
|
185
|
-
toVariantPayload: (outcome, parent) => ({ ...parent.payload, codeMutation: outcome.artifact }),
|
|
186
|
-
})
|
|
187
|
-
|
|
188
|
-
// 3. Compose — plateau policy auto-switches when prompt evolution stalls
|
|
189
|
-
const composite = createCompositeMutator({
|
|
190
|
-
primary: promptMutator,
|
|
191
|
-
secondary: codeMutator,
|
|
192
|
-
policy: 'plateau',
|
|
193
|
-
plateauThreshold: 0.02,
|
|
194
|
-
plateauPatience: 2,
|
|
195
|
-
})
|
|
196
|
-
|
|
197
|
-
// 4. Run — durable telemetry to disk, crash-resumable
|
|
198
|
-
const result = await runPromptEvolution({
|
|
199
|
-
runId: `forge_${Date.now()}`,
|
|
200
|
-
target: 'forge-prompt',
|
|
201
|
-
seedVariants: [{ id: 'v0', payload: { text: currentPrompt }, generation: 0, label: 'baseline' }],
|
|
202
|
-
scenarioIds: referenceCorpus.map(s => s.id),
|
|
203
|
-
reps: 3,
|
|
204
|
-
generations: 5,
|
|
205
|
-
populationSize: 4,
|
|
206
|
-
scoreAdapter: { /* runs your eval against (variant, scenario, rep) */ },
|
|
207
|
-
mutateAdapter: composite,
|
|
208
|
-
cache: new JsonlTrialCache('.evolve/cache.jsonl'),
|
|
209
|
-
objectives: [
|
|
210
|
-
{ name: 'score', direction: 'maximize', value: a => a.meanScore },
|
|
211
|
-
{ name: 'cost', direction: 'minimize', value: a => a.meanCost },
|
|
212
|
-
],
|
|
213
|
-
})
|
|
214
|
-
```
|
|
133
|
+
Run the local server:
|
|
215
134
|
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
## Feedback trajectory loop
|
|
221
|
-
|
|
222
|
-
When normal agent usage should generate training/eval signal, use feedback
|
|
223
|
-
trajectories. They turn approvals, rejections, option choices, edits, metrics,
|
|
224
|
-
and policy blocks into reusable examples.
|
|
225
|
-
|
|
226
|
-
```ts
|
|
227
|
-
import {
|
|
228
|
-
createFeedbackTrajectory,
|
|
229
|
-
summarizePreferenceMemory,
|
|
230
|
-
feedbackTrajectoriesToDatasetScenarios,
|
|
231
|
-
feedbackTrajectoriesToOptimizerRows,
|
|
232
|
-
} from '@tangle-network/agent-eval'
|
|
233
|
-
|
|
234
|
-
const trajectory = createFeedbackTrajectory({
|
|
235
|
-
projectId: 'research-agent',
|
|
236
|
-
scenarioId: 'brief-review',
|
|
237
|
-
task: { intent: 'Revise a research brief until it is specific and sourced.' },
|
|
238
|
-
attempts: [{
|
|
239
|
-
id: 'draft-1',
|
|
240
|
-
stepIndex: 0,
|
|
241
|
-
artifactType: 'research',
|
|
242
|
-
artifact: { summary: 'Initial brief with weak sourcing.' },
|
|
243
|
-
createdAt: new Date().toISOString(),
|
|
244
|
-
}],
|
|
245
|
-
labels: [{
|
|
246
|
-
source: 'user',
|
|
247
|
-
kind: 'revision_request',
|
|
248
|
-
value: 'needs stronger evidence',
|
|
249
|
-
reason: 'add primary sources and remove unsupported claims',
|
|
250
|
-
severity: 'error',
|
|
251
|
-
createdAt: new Date().toISOString(),
|
|
252
|
-
}],
|
|
253
|
-
})
|
|
254
|
-
|
|
255
|
-
const memory = summarizePreferenceMemory([trajectory])
|
|
256
|
-
const scenarios = feedbackTrajectoriesToDatasetScenarios([trajectory])
|
|
257
|
-
const optimizerRows = feedbackTrajectoriesToOptimizerRows([trajectory])
|
|
135
|
+
```sh
|
|
136
|
+
pnpm build
|
|
137
|
+
node dist/cli.js serve --port 5005
|
|
258
138
|
```
|
|
259
139
|
|
|
260
|
-
|
|
261
|
-
immediate memory, replayable eval scenarios, and prompt/signature/code optimizer
|
|
262
|
-
input. See [`docs/feedback-trajectories.md`](./docs/feedback-trajectories.md).
|
|
263
|
-
|
|
264
|
-
## v0.16 highlights — production-rigor primitives
|
|
265
|
-
|
|
266
|
-
These are the primitives any team running prompt-optimization in production needs, regardless of whether they're writing a paper. v0.15 shipped them under "paper-grade" naming; v0.16 corrects that — they're production-first, paper-grade as a side effect.
|
|
267
|
-
|
|
268
|
-
- `HeldOutGate` — held-out paired-delta gate with `few_runs` /
|
|
269
|
-
`negative_delta` / `overfit_gap` rejection codes and a full evidence
|
|
270
|
-
block on every decision. Sits alongside the existing bootstrap-CI
|
|
271
|
-
`promotion-gate.ts`: that one asks "is this real or noise?", this one
|
|
272
|
-
asks "is this a real win on held-out and not overfit?". Use both.
|
|
273
|
-
- `RunRecord` — typed run schema with mandatory snapshot-pinned `model`,
|
|
274
|
-
`promptHash`, `configHash`, `commitSha`, `costUsd`, `splitTag`.
|
|
275
|
-
Runtime validator throws on missing fields. Reproducibility falls
|
|
276
|
-
out for free.
|
|
277
|
-
- `pairedBootstrap`, `pairedWilcoxon`, `bhAdjust` — statistical
|
|
278
|
-
primitives every rigorous A/B test needs. Already-existing primitives
|
|
279
|
-
are re-exported for paper-style aliases.
|
|
280
|
-
- `runCanaries` — silent judge-fallback, calibration drift (KS test),
|
|
281
|
-
distribution shift (chi-square). Catches the failure mode where your
|
|
282
|
-
judge silently degrades to a constant-0.30 confidence and you ship
|
|
283
|
-
configs graded by a stub.
|
|
284
|
-
- `summaryTable`, `paretoChart`, `gainHistogram` — A/B reporting
|
|
285
|
-
helpers. `summaryTable` emits markdown with means + 95% bootstrap
|
|
286
|
-
CIs + paired Wilcoxon p (BH-adjusted) + Cohen's d. Useful for both
|
|
287
|
-
internal status reports and paper Table 1s.
|
|
288
|
-
- `Researcher` — stable interface for an external agent that drives the
|
|
289
|
-
meta-loop (`inspectFailures` → `proposeChange` → `applyChange` →
|
|
290
|
-
`evaluateChange`). Ship a `NoopResearcher` as a placeholder; real
|
|
291
|
-
implementations live downstream.
|
|
292
|
-
- `benchmarks/routing` — synthetic 16-task router benchmark we own.
|
|
293
|
-
Ships in the package. Reference wrappers for GSM8K and SWE-Bench
|
|
294
|
-
Lite live under `examples/benchmarks/` — read, copy, adapt. All
|
|
295
|
-
three implement one `BenchmarkAdapter` shape with deterministic
|
|
296
|
-
splits and fail-loud env-var configuration.
|
|
297
|
-
|
|
298
|
-
### v0.16 changes from v0.15
|
|
299
|
-
|
|
300
|
-
- Renamed `paperTable` → `summaryTable`, `paretoFigure` → `paretoChart`,
|
|
301
|
-
`gainDistributionFigure` → `gainHistogram`. Underlying semantics
|
|
302
|
-
unchanged. Type names follow (`SummaryTable`, `SummaryTableOptions`,
|
|
303
|
-
`SummaryTableRow`).
|
|
304
|
-
- File: `src/paper-report.ts` → `src/summary-report.ts`.
|
|
305
|
-
- Drop the "paper-grade" framing — the primitives are production-first.
|
|
306
|
-
|
|
307
|
-
See `CHANGELOG.md` for the full list. `.claude/skills/agent-eval/SKILL.md`
|
|
308
|
-
covers usage directives and pitfalls.
|
|
309
|
-
|
|
310
|
-
## Tech stack
|
|
311
|
-
|
|
312
|
-
- TypeScript strict, no semicolons, single quotes, 2-space indent
|
|
313
|
-
- `tsup` for bundling, `vitest` for tests
|
|
314
|
-
- `@tangle-network/tcloud` for LLM calls (judges, driver)
|
|
315
|
-
- `hono` + `@asteasolutions/zod-to-openapi` for the wire protocol
|
|
316
|
-
|
|
317
|
-
## Develop
|
|
140
|
+
Python client tests:
|
|
318
141
|
|
|
319
142
|
```sh
|
|
320
|
-
pnpm install
|
|
321
|
-
pnpm typecheck
|
|
322
|
-
pnpm test
|
|
323
143
|
pnpm build
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
node dist/cli.js serve --port 5005
|
|
328
|
-
|
|
329
|
-
# Python client tests (require pnpm build first)
|
|
330
|
-
cd clients/python && pip install -e ".[dev]" && pytest
|
|
144
|
+
cd clients/python
|
|
145
|
+
pip install -e ".[dev]"
|
|
146
|
+
pytest
|
|
331
147
|
```
|
|
332
148
|
|
|
333
149
|
## Release
|
|
334
150
|
|
|
335
|
-
`@tangle-network/agent-eval`
|
|
151
|
+
`@tangle-network/agent-eval` publishes to npm. The Python client lives under
|
|
152
|
+
`clients/python` and is versioned from this repository.
|
|
336
153
|
|
|
337
|
-
## Related
|
|
154
|
+
## Related Packages
|
|
338
155
|
|
|
156
|
+
- [`@tangle-network/agent-runtime`](https://github.com/tangle-network/agent-runtime)
|
|
157
|
+
- [`@tangle-network/agent-knowledge`](https://github.com/tangle-network/agent-knowledge)
|
|
158
|
+
- [`@tangle-network/agent-integrations`](https://github.com/tangle-network/agent-integrations)
|
|
339
159
|
- [`@tangle-network/agent-gateway`](https://github.com/tangle-network/agent-gateway)
|
|
340
|
-
- [`@tangle-network/agent-client`](https://github.com/tangle-network/agent-client)
|
|
341
160
|
- [`@tangle-network/tcloud`](https://github.com/tangle-network/tcloud)
|
|
342
161
|
|
|
343
162
|
## License
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import {
|
|
2
2
|
callLlmJson
|
|
3
|
-
} from "./chunk-
|
|
3
|
+
} from "./chunk-JAOLXRIA.js";
|
|
4
4
|
|
|
5
5
|
// src/wire/schemas.ts
|
|
6
6
|
import { extendZodWithOpenApi } from "@asteasolutions/zod-to-openapi";
|
|
@@ -591,4 +591,4 @@ export {
|
|
|
591
591
|
runRpcOnce,
|
|
592
592
|
runRpcBatch
|
|
593
593
|
};
|
|
594
|
-
//# sourceMappingURL=chunk-
|
|
594
|
+
//# sourceMappingURL=chunk-CJJSB6ZQ.js.map
|
|
@@ -76,6 +76,56 @@ function stripFencedJson(raw) {
|
|
|
76
76
|
const m = trimmed.match(/^```(?:json)?\s*\n?([\s\S]*?)\n?```\s*$/);
|
|
77
77
|
return m ? m[1].trim() : trimmed;
|
|
78
78
|
}
|
|
79
|
+
function extractJsonPayload(raw) {
|
|
80
|
+
const stripped = stripFencedJson(raw);
|
|
81
|
+
try {
|
|
82
|
+
JSON.parse(stripped);
|
|
83
|
+
return stripped;
|
|
84
|
+
} catch {
|
|
85
|
+
}
|
|
86
|
+
const starts = [...stripped.matchAll(/[\[{]/g)].map((match) => match.index).filter((index) => index != null);
|
|
87
|
+
for (const start of starts) {
|
|
88
|
+
const candidate = extractBalancedJson(stripped, start);
|
|
89
|
+
if (!candidate) continue;
|
|
90
|
+
try {
|
|
91
|
+
JSON.parse(candidate);
|
|
92
|
+
return candidate;
|
|
93
|
+
} catch {
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
return stripped;
|
|
97
|
+
}
|
|
98
|
+
function extractBalancedJson(input, start) {
|
|
99
|
+
const opener = input[start];
|
|
100
|
+
const closer = opener === "{" ? "}" : opener === "[" ? "]" : null;
|
|
101
|
+
if (!closer) return null;
|
|
102
|
+
const stack = [closer];
|
|
103
|
+
let isInString = false;
|
|
104
|
+
let isEscaped = false;
|
|
105
|
+
for (let i = start + 1; i < input.length; i++) {
|
|
106
|
+
const char = input[i];
|
|
107
|
+
if (isEscaped) {
|
|
108
|
+
isEscaped = false;
|
|
109
|
+
continue;
|
|
110
|
+
}
|
|
111
|
+
if (char === "\\") {
|
|
112
|
+
isEscaped = isInString;
|
|
113
|
+
continue;
|
|
114
|
+
}
|
|
115
|
+
if (char === '"') {
|
|
116
|
+
isInString = !isInString;
|
|
117
|
+
continue;
|
|
118
|
+
}
|
|
119
|
+
if (isInString) continue;
|
|
120
|
+
if (char === "{") stack.push("}");
|
|
121
|
+
else if (char === "[") stack.push("]");
|
|
122
|
+
else if (char === stack[stack.length - 1]) {
|
|
123
|
+
stack.pop();
|
|
124
|
+
if (stack.length === 0) return input.slice(start, i + 1);
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
return null;
|
|
128
|
+
}
|
|
79
129
|
async function callLlm(req, opts = {}) {
|
|
80
130
|
const baseUrl = (opts.baseUrl ?? DEFAULT_BASE_URL).replace(/\/+$/, "");
|
|
81
131
|
const url = `${baseUrl}/chat/completions`;
|
|
@@ -159,7 +209,7 @@ async function callLlmJson(req, opts = {}) {
|
|
|
159
209
|
}
|
|
160
210
|
}
|
|
161
211
|
function parseJsonSafely(content, model) {
|
|
162
|
-
const stripped =
|
|
212
|
+
const stripped = extractJsonPayload(content);
|
|
163
213
|
try {
|
|
164
214
|
return JSON.parse(stripped);
|
|
165
215
|
} catch (err) {
|
|
@@ -212,4 +262,4 @@ export {
|
|
|
212
262
|
probeLlm,
|
|
213
263
|
LlmClient
|
|
214
264
|
};
|
|
215
|
-
//# sourceMappingURL=chunk-
|
|
265
|
+
//# sourceMappingURL=chunk-JAOLXRIA.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":["../src/llm-client.ts"],"sourcesContent":["/**\n * LLM client with graceful degrade.\n *\n * OpenAI-compatible `/v1/chat/completions` client with:\n * - Exponential-backoff retry on 429 + 5xx gateway errors (502/503/504).\n * - Retry on transient network errors (fetch failed, AbortError, ECONNRESET).\n * - Graceful json_schema → json_object degrade on 400 with schema-reject body.\n * - Fenced-JSON stripping (```json ... ```) for models that wrap structured output.\n * - Configurable base URL + api key / bearer, works with LiteLLM proxies, OpenAI\n * directly, cli-bridge subscriptions, and any router that speaks the spec.\n *\n * Usage:\n * const { value, result } = await callLlmJson<MyType>(\n * { model: 'gpt-4o', messages: [...], jsonSchema: { name: 'x', schema: {...} } },\n * { baseUrl: 'https://router.tangle.tools/v1', apiKey: process.env.KEY },\n * )\n *\n * This is THE llm-calling seam for agent-eval primitives that need structured\n * output (semantic concept judge, reviewer directives, critic scores). Primitives\n * that need free-form text use `callLlm` and parse output themselves.\n */\n\n// ─── Types ──────────────────────────────────────────────────────────────\n\nexport interface LlmMessage {\n role: 'system' | 'user' | 'assistant'\n /**\n * Either a plain text content string OR a multimodal content array\n * (text + image_url parts) for vision-capable models.\n */\n content:\n | string\n | Array<\n | { type: 'text'; text: string }\n | { type: 'image_url'; image_url: { url: string; detail?: 'auto' | 'low' | 'high' } }\n >\n}\n\nexport interface LlmCallRequest {\n model: string\n messages: LlmMessage[]\n /** Optional JSON-mode response format (response_format: json_object). */\n jsonMode?: boolean\n /** Optional structured output via JSON Schema. Falls back to json_object on 400. */\n jsonSchema?: { name: string; schema: Record<string, unknown> }\n temperature?: number\n maxTokens?: number\n /** Per-call timeout, default 60s. */\n timeoutMs?: number\n}\n\nexport interface LlmUsage {\n promptTokens: number\n completionTokens: number\n totalTokens: number\n /** Proxies populate this when prompt caching is on. */\n cachedPromptTokens?: number\n}\n\nexport interface LlmCallResult {\n /** The text content of the first choice. Empty string if none. */\n content: string\n usage: LlmUsage\n /**\n * Cost in USD. Pulled from proxy's `_response_cost` field when present;\n * `null` when neither the proxy nor the caller can derive it.\n */\n costUsd: number | null\n /** Model name actually used (echoed from response). */\n model: string\n /** Wall-clock duration of the HTTP call (last attempt, if retried). */\n durationMs: number\n /** Raw response body. */\n raw: Record<string, unknown>\n}\n\nexport class LlmCallError extends Error {\n constructor(\n message: string,\n public readonly status: number,\n public readonly body: string,\n public readonly model: string,\n ) {\n super(message)\n this.name = 'LlmCallError'\n }\n}\n\nexport interface LlmClientOptions {\n /** Base URL (without trailing slash). Must end at the `/v1` prefix. */\n baseUrl?: string\n /** Bearer token — either `apiKey` or `bearer` populates `Authorization: Bearer ...`. */\n apiKey?: string\n bearer?: string\n /** Override for the `Authorization` header (e.g. `X-Auth: ...`). Takes precedence over apiKey/bearer. */\n authHeader?: { name: string; value: string }\n /** Default timeout in ms. Per-call can override. */\n defaultTimeoutMs?: number\n /** Max retry attempts on retriable errors. Default 3 (1 initial + 2 retries). */\n maxRetries?: number\n /** Fetch implementation — defaults to global `fetch`. Override for custom transport (e.g. tests). */\n fetch?: typeof fetch\n}\n\n// ─── Internals ──────────────────────────────────────────────────────────\n\nconst DEFAULT_BASE_URL = 'https://router.tangle.tools/v1'\nconst DEFAULT_TIMEOUT_MS = 60_000\nconst DEFAULT_MAX_RETRIES = 3\n\nconst RETRYABLE_STATUS = new Set([429, 502, 503, 504])\n\nfunction isRetryableError(err: unknown): boolean {\n if (err instanceof LlmCallError) return RETRYABLE_STATUS.has(err.status)\n if (err instanceof Error) {\n return (\n err.name === 'AbortError' ||\n err.name === 'TimeoutError' ||\n /fetch failed|ECONNRESET|ETIMEDOUT|EAI_AGAIN/i.test(err.message)\n )\n }\n return false\n}\n\nfunction parseRetryAfter(headers: Headers): number | null {\n const h = headers.get('retry-after')\n if (!h) return null\n const asNumber = Number(h)\n if (Number.isFinite(asNumber) && asNumber > 0) return asNumber * 1000\n const asDate = Date.parse(h)\n if (Number.isFinite(asDate)) return Math.max(0, asDate - Date.now())\n return null\n}\n\nfunction backoffMs(attempt: number): number {\n // 500ms, 1s, 2s, 4s, ...\n return Math.min(500 * Math.pow(2, attempt), 16_000)\n}\n\nfunction buildHeaders(opts: LlmClientOptions): Record<string, string> {\n const headers: Record<string, string> = {\n 'Content-Type': 'application/json',\n Accept: 'application/json',\n }\n if (opts.authHeader) {\n headers[opts.authHeader.name] = opts.authHeader.value\n } else if (opts.bearer || opts.apiKey) {\n headers.Authorization = `Bearer ${opts.bearer ?? opts.apiKey}`\n }\n return headers\n}\n\nfunction isSchemaRejection(status: number, body: string): boolean {\n if (status !== 400) return false\n const lower = body.toLowerCase()\n return (\n lower.includes('response_format') ||\n lower.includes('json_schema') ||\n lower.includes('is unavailable') ||\n lower.includes('not supported')\n )\n}\n\nfunction buildBody(req: LlmCallRequest, forceJsonObject: boolean): Record<string, unknown> {\n const body: Record<string, unknown> = {\n model: req.model,\n messages: req.messages,\n temperature: req.temperature ?? 0,\n }\n if (req.maxTokens != null) body.max_tokens = req.maxTokens\n\n if (req.jsonSchema && !forceJsonObject) {\n body.response_format = {\n type: 'json_schema',\n json_schema: { name: req.jsonSchema.name, schema: req.jsonSchema.schema, strict: true },\n }\n } else if (req.jsonMode || req.jsonSchema) {\n body.response_format = { type: 'json_object' }\n }\n\n return body\n}\n\nasync function sleep(ms: number): Promise<void> {\n return new Promise((resolve) => setTimeout(resolve, ms))\n}\n\n// ─── Public API ─────────────────────────────────────────────────────────\n\n/**\n * Strip a ```json / ``` code fence if the model emitted one.\n * Idempotent for naked JSON. Some models (claude-code via router, certain\n * deepseek models) wrap output even under json_object.\n */\nexport function stripFencedJson(raw: string): string {\n const trimmed = raw.trim()\n const m = trimmed.match(/^```(?:json)?\\s*\\n?([\\s\\S]*?)\\n?```\\s*$/)\n return m ? m[1]!.trim() : trimmed\n}\n\nexport function extractJsonPayload(raw: string): string {\n const stripped = stripFencedJson(raw)\n try {\n JSON.parse(stripped)\n return stripped\n } catch {\n // Continue with balanced extraction below.\n }\n\n const starts = [...stripped.matchAll(/[\\[{]/g)].map((match) => match.index).filter((index) => index != null)\n for (const start of starts) {\n const candidate = extractBalancedJson(stripped, start)\n if (!candidate) continue\n try {\n JSON.parse(candidate)\n return candidate\n } catch {\n // Keep scanning; earlier braces may belong to prose.\n }\n }\n\n return stripped\n}\n\nfunction extractBalancedJson(input: string, start: number): string | null {\n const opener = input[start]\n const closer = opener === '{' ? '}' : opener === '[' ? ']' : null\n if (!closer) return null\n\n const stack: string[] = [closer]\n let isInString = false\n let isEscaped = false\n\n for (let i = start + 1; i < input.length; i++) {\n const char = input[i]!\n if (isEscaped) {\n isEscaped = false\n continue\n }\n if (char === '\\\\') {\n isEscaped = isInString\n continue\n }\n if (char === '\"') {\n isInString = !isInString\n continue\n }\n if (isInString) continue\n\n if (char === '{') stack.push('}')\n else if (char === '[') stack.push(']')\n else if (char === stack[stack.length - 1]) {\n stack.pop()\n if (stack.length === 0) return input.slice(start, i + 1)\n }\n }\n\n return null\n}\n\n/**\n * Low-level call. Returns raw content + usage + cost. Retries on transient\n * failures; does NOT degrade schema here — callers that want graceful\n * degrade use `callLlmJson`.\n */\nexport async function callLlm(\n req: LlmCallRequest,\n opts: LlmClientOptions = {},\n): Promise<LlmCallResult> {\n const baseUrl = (opts.baseUrl ?? DEFAULT_BASE_URL).replace(/\\/+$/, '')\n const url = `${baseUrl}/chat/completions`\n const timeoutMs = req.timeoutMs ?? opts.defaultTimeoutMs ?? DEFAULT_TIMEOUT_MS\n const maxRetries = opts.maxRetries ?? DEFAULT_MAX_RETRIES\n const fetchFn = opts.fetch ?? globalThis.fetch\n const headers = buildHeaders(opts)\n\n let lastErr: unknown\n for (let attempt = 0; attempt < maxRetries; attempt++) {\n const controller = new AbortController()\n const timeoutHandle = setTimeout(() => controller.abort(), timeoutMs)\n const started = Date.now()\n\n try {\n const res = await fetchFn(url, {\n method: 'POST',\n headers,\n body: JSON.stringify(buildBody(req, false)),\n signal: controller.signal,\n })\n clearTimeout(timeoutHandle)\n\n if (!res.ok) {\n const body = await res.text()\n const err = new LlmCallError(\n `LLM call ${res.status}: ${body.slice(0, 300)}`,\n res.status,\n body,\n req.model,\n )\n if (RETRYABLE_STATUS.has(res.status) && attempt < maxRetries - 1) {\n lastErr = err\n const retryAfter = parseRetryAfter(res.headers)\n await sleep(retryAfter ?? backoffMs(attempt))\n continue\n }\n throw err\n }\n\n const json = (await res.json()) as Record<string, unknown>\n const choice = (json.choices as Array<{ message?: { content?: string } }> | undefined)?.[0]\n const usageRaw = (json.usage as Record<string, unknown> | undefined) ?? {}\n const costFromProxy = (json._response_cost ?? json.cost_usd) as number | undefined\n\n return {\n content: choice?.message?.content ?? '',\n usage: {\n promptTokens: Number(usageRaw.prompt_tokens ?? 0),\n completionTokens: Number(usageRaw.completion_tokens ?? 0),\n totalTokens: Number(usageRaw.total_tokens ?? 0),\n cachedPromptTokens:\n usageRaw.prompt_tokens_details &&\n typeof usageRaw.prompt_tokens_details === 'object'\n ? Number(\n (usageRaw.prompt_tokens_details as Record<string, unknown>).cached_tokens ?? 0,\n )\n : undefined,\n },\n costUsd: typeof costFromProxy === 'number' ? costFromProxy : null,\n model: (json.model as string) ?? req.model,\n durationMs: Date.now() - started,\n raw: json,\n }\n } catch (err) {\n clearTimeout(timeoutHandle)\n lastErr = err\n if (attempt < maxRetries - 1 && isRetryableError(err)) {\n await sleep(backoffMs(attempt))\n continue\n }\n throw err\n }\n }\n throw lastErr instanceof Error ? lastErr : new Error(String(lastErr))\n}\n\n/**\n * Structured-output call. Returns parsed JSON plus the raw result envelope.\n * Degrades `jsonSchema` → `jsonMode` on a 400 that names the schema param —\n * critical for deepseek-v3/v4, kimi-k2.6, and other models that don't accept\n * the `response_format.json_schema` shape but DO accept `json_object`.\n */\nexport async function callLlmJson<T = unknown>(\n req: LlmCallRequest,\n opts: LlmClientOptions = {},\n): Promise<{ value: T; result: LlmCallResult }> {\n try {\n const result = await callLlm({ ...req, jsonMode: req.jsonMode ?? !req.jsonSchema }, opts)\n const value = parseJsonSafely<T>(result.content, result.model)\n return { value, result }\n } catch (err) {\n if (err instanceof LlmCallError && isSchemaRejection(err.status, err.body) && req.jsonSchema) {\n // Degrade to json_object + retry.\n const degradedReq: LlmCallRequest = { ...req, jsonMode: true, jsonSchema: undefined }\n const result = await callLlm(degradedReq, opts)\n const value = parseJsonSafely<T>(result.content, result.model)\n return { value, result }\n }\n throw err\n }\n}\n\nfunction parseJsonSafely<T>(content: string, model: string): T {\n const stripped = extractJsonPayload(content)\n try {\n return JSON.parse(stripped) as T\n } catch (err) {\n throw new Error(\n `LLM returned non-JSON content (model=${model}): ${\n err instanceof Error ? err.message : String(err)\n }\\n--- raw content ---\\n${content.slice(0, 800)}`,\n )\n }\n}\n\n/**\n * Probe whether a model is reachable. Returns latency + null error on\n * success; `ok=false` + error message on any failure (HTTP, timeout,\n * network, parse). Designed for sweep preflights — fail loud at the\n * boundary before burning a 30-leaf run on a misconfigured router.\n *\n * Sends a tiny `ping` message with `maxTokens=64`. Reasoning models\n * (glm-5.1, deepseek-v4) can burn the entire budget on internal reasoning\n * for short prompts, so don't tighten this further. We don't validate\n * content; HTTP 200 means reachable.\n */\nexport async function probeLlm(\n model: string,\n opts: LlmClientOptions & { timeoutMs?: number } = {},\n): Promise<{ ok: boolean; latencyMs: number; error: string | null }> {\n const start = Date.now()\n try {\n await callLlm(\n {\n model,\n messages: [{ role: 'user', content: 'ping' }],\n maxTokens: 64,\n timeoutMs: opts.timeoutMs ?? 30_000,\n },\n opts,\n )\n return { ok: true, latencyMs: Date.now() - start, error: null }\n } catch (err) {\n return {\n ok: false,\n latencyMs: Date.now() - start,\n error: err instanceof Error ? err.message : String(err),\n }\n }\n}\n\n/**\n * Stateful client — construct once with defaults, call many times.\n * Thin wrapper around the free functions; exists for callers that want\n * to inject a single configured instance into multiple primitives.\n */\nexport class LlmClient {\n constructor(private readonly opts: LlmClientOptions = {}) {}\n\n call(req: LlmCallRequest, per?: LlmClientOptions): Promise<LlmCallResult> {\n return callLlm(req, { ...this.opts, ...per })\n }\n\n callJson<T = unknown>(\n req: LlmCallRequest,\n per?: LlmClientOptions,\n ): Promise<{ value: T; result: LlmCallResult }> {\n return callLlmJson<T>(req, { ...this.opts, ...per })\n }\n}\n"],"mappings":";AA4EO,IAAM,eAAN,cAA2B,MAAM;AAAA,EACtC,YACE,SACgB,QACA,MACA,OAChB;AACA,UAAM,OAAO;AAJG;AACA;AACA;AAGhB,SAAK,OAAO;AAAA,EACd;AAAA,EANkB;AAAA,EACA;AAAA,EACA;AAKpB;AAoBA,IAAM,mBAAmB;AACzB,IAAM,qBAAqB;AAC3B,IAAM,sBAAsB;AAE5B,IAAM,mBAAmB,oBAAI,IAAI,CAAC,KAAK,KAAK,KAAK,GAAG,CAAC;AAErD,SAAS,iBAAiB,KAAuB;AAC/C,MAAI,eAAe,aAAc,QAAO,iBAAiB,IAAI,IAAI,MAAM;AACvE,MAAI,eAAe,OAAO;AACxB,WACE,IAAI,SAAS,gBACb,IAAI,SAAS,kBACb,+CAA+C,KAAK,IAAI,OAAO;AAAA,EAEnE;AACA,SAAO;AACT;AAEA,SAAS,gBAAgB,SAAiC;AACxD,QAAM,IAAI,QAAQ,IAAI,aAAa;AACnC,MAAI,CAAC,EAAG,QAAO;AACf,QAAM,WAAW,OAAO,CAAC;AACzB,MAAI,OAAO,SAAS,QAAQ,KAAK,WAAW,EAAG,QAAO,WAAW;AACjE,QAAM,SAAS,KAAK,MAAM,CAAC;AAC3B,MAAI,OAAO,SAAS,MAAM,EAAG,QAAO,KAAK,IAAI,GAAG,SAAS,KAAK,IAAI,CAAC;AACnE,SAAO;AACT;AAEA,SAAS,UAAU,SAAyB;AAE1C,SAAO,KAAK,IAAI,MAAM,KAAK,IAAI,GAAG,OAAO,GAAG,IAAM;AACpD;AAEA,SAAS,aAAa,MAAgD;AACpE,QAAM,UAAkC;AAAA,IACtC,gBAAgB;AAAA,IAChB,QAAQ;AAAA,EACV;AACA,MAAI,KAAK,YAAY;AACnB,YAAQ,KAAK,WAAW,IAAI,IAAI,KAAK,WAAW;AAAA,EAClD,WAAW,KAAK,UAAU,KAAK,QAAQ;AACrC,YAAQ,gBAAgB,UAAU,KAAK,UAAU,KAAK,MAAM;AAAA,EAC9D;AACA,SAAO;AACT;AAEA,SAAS,kBAAkB,QAAgB,MAAuB;AAChE,MAAI,WAAW,IAAK,QAAO;AAC3B,QAAM,QAAQ,KAAK,YAAY;AAC/B,SACE,MAAM,SAAS,iBAAiB,KAChC,MAAM,SAAS,aAAa,KAC5B,MAAM,SAAS,gBAAgB,KAC/B,MAAM,SAAS,eAAe;AAElC;AAEA,SAAS,UAAU,KAAqB,iBAAmD;AACzF,QAAM,OAAgC;AAAA,IACpC,OAAO,IAAI;AAAA,IACX,UAAU,IAAI;AAAA,IACd,aAAa,IAAI,eAAe;AAAA,EAClC;AACA,MAAI,IAAI,aAAa,KAAM,MAAK,aAAa,IAAI;AAEjD,MAAI,IAAI,cAAc,CAAC,iBAAiB;AACtC,SAAK,kBAAkB;AAAA,MACrB,MAAM;AAAA,MACN,aAAa,EAAE,MAAM,IAAI,WAAW,MAAM,QAAQ,IAAI,WAAW,QAAQ,QAAQ,KAAK;AAAA,IACxF;AAAA,EACF,WAAW,IAAI,YAAY,IAAI,YAAY;AACzC,SAAK,kBAAkB,EAAE,MAAM,cAAc;AAAA,EAC/C;AAEA,SAAO;AACT;AAEA,eAAe,MAAM,IAA2B;AAC9C,SAAO,IAAI,QAAQ,CAAC,YAAY,WAAW,SAAS,EAAE,CAAC;AACzD;AASO,SAAS,gBAAgB,KAAqB;AACnD,QAAM,UAAU,IAAI,KAAK;AACzB,QAAM,IAAI,QAAQ,MAAM,yCAAyC;AACjE,SAAO,IAAI,EAAE,CAAC,EAAG,KAAK,IAAI;AAC5B;AAEO,SAAS,mBAAmB,KAAqB;AACtD,QAAM,WAAW,gBAAgB,GAAG;AACpC,MAAI;AACF,SAAK,MAAM,QAAQ;AACnB,WAAO;AAAA,EACT,QAAQ;AAAA,EAER;AAEA,QAAM,SAAS,CAAC,GAAG,SAAS,SAAS,QAAQ,CAAC,EAAE,IAAI,CAAC,UAAU,MAAM,KAAK,EAAE,OAAO,CAAC,UAAU,SAAS,IAAI;AAC3G,aAAW,SAAS,QAAQ;AAC1B,UAAM,YAAY,oBAAoB,UAAU,KAAK;AACrD,QAAI,CAAC,UAAW;AAChB,QAAI;AACF,WAAK,MAAM,SAAS;AACpB,aAAO;AAAA,IACT,QAAQ;AAAA,IAER;AAAA,EACF;AAEA,SAAO;AACT;AAEA,SAAS,oBAAoB,OAAe,OAA8B;AACxE,QAAM,SAAS,MAAM,KAAK;AAC1B,QAAM,SAAS,WAAW,MAAM,MAAM,WAAW,MAAM,MAAM;AAC7D,MAAI,CAAC,OAAQ,QAAO;AAEpB,QAAM,QAAkB,CAAC,MAAM;AAC/B,MAAI,aAAa;AACjB,MAAI,YAAY;AAEhB,WAAS,IAAI,QAAQ,GAAG,IAAI,MAAM,QAAQ,KAAK;AAC7C,UAAM,OAAO,MAAM,CAAC;AACpB,QAAI,WAAW;AACb,kBAAY;AACZ;AAAA,IACF;AACA,QAAI,SAAS,MAAM;AACjB,kBAAY;AACZ;AAAA,IACF;AACA,QAAI,SAAS,KAAK;AAChB,mBAAa,CAAC;AACd;AAAA,IACF;AACA,QAAI,WAAY;AAEhB,QAAI,SAAS,IAAK,OAAM,KAAK,GAAG;AAAA,aACvB,SAAS,IAAK,OAAM,KAAK,GAAG;AAAA,aAC5B,SAAS,MAAM,MAAM,SAAS,CAAC,GAAG;AACzC,YAAM,IAAI;AACV,UAAI,MAAM,WAAW,EAAG,QAAO,MAAM,MAAM,OAAO,IAAI,CAAC;AAAA,IACzD;AAAA,EACF;AAEA,SAAO;AACT;AAOA,eAAsB,QACpB,KACA,OAAyB,CAAC,GACF;AACxB,QAAM,WAAW,KAAK,WAAW,kBAAkB,QAAQ,QAAQ,EAAE;AACrE,QAAM,MAAM,GAAG,OAAO;AACtB,QAAM,YAAY,IAAI,aAAa,KAAK,oBAAoB;AAC5D,QAAM,aAAa,KAAK,cAAc;AACtC,QAAM,UAAU,KAAK,SAAS,WAAW;AACzC,QAAM,UAAU,aAAa,IAAI;AAEjC,MAAI;AACJ,WAAS,UAAU,GAAG,UAAU,YAAY,WAAW;AACrD,UAAM,aAAa,IAAI,gBAAgB;AACvC,UAAM,gBAAgB,WAAW,MAAM,WAAW,MAAM,GAAG,SAAS;AACpE,UAAM,UAAU,KAAK,IAAI;AAEzB,QAAI;AACF,YAAM,MAAM,MAAM,QAAQ,KAAK;AAAA,QAC7B,QAAQ;AAAA,QACR;AAAA,QACA,MAAM,KAAK,UAAU,UAAU,KAAK,KAAK,CAAC;AAAA,QAC1C,QAAQ,WAAW;AAAA,MACrB,CAAC;AACD,mBAAa,aAAa;AAE1B,UAAI,CAAC,IAAI,IAAI;AACX,cAAM,OAAO,MAAM,IAAI,KAAK;AAC5B,cAAM,MAAM,IAAI;AAAA,UACd,YAAY,IAAI,MAAM,KAAK,KAAK,MAAM,GAAG,GAAG,CAAC;AAAA,UAC7C,IAAI;AAAA,UACJ;AAAA,UACA,IAAI;AAAA,QACN;AACA,YAAI,iBAAiB,IAAI,IAAI,MAAM,KAAK,UAAU,aAAa,GAAG;AAChE,oBAAU;AACV,gBAAM,aAAa,gBAAgB,IAAI,OAAO;AAC9C,gBAAM,MAAM,cAAc,UAAU,OAAO,CAAC;AAC5C;AAAA,QACF;AACA,cAAM;AAAA,MACR;AAEA,YAAM,OAAQ,MAAM,IAAI,KAAK;AAC7B,YAAM,SAAU,KAAK,UAAoE,CAAC;AAC1F,YAAM,WAAY,KAAK,SAAiD,CAAC;AACzE,YAAM,gBAAiB,KAAK,kBAAkB,KAAK;AAEnD,aAAO;AAAA,QACL,SAAS,QAAQ,SAAS,WAAW;AAAA,QACrC,OAAO;AAAA,UACL,cAAc,OAAO,SAAS,iBAAiB,CAAC;AAAA,UAChD,kBAAkB,OAAO,SAAS,qBAAqB,CAAC;AAAA,UACxD,aAAa,OAAO,SAAS,gBAAgB,CAAC;AAAA,UAC9C,oBACE,SAAS,yBACT,OAAO,SAAS,0BAA0B,WACtC;AAAA,YACG,SAAS,sBAAkD,iBAAiB;AAAA,UAC/E,IACA;AAAA,QACR;AAAA,QACA,SAAS,OAAO,kBAAkB,WAAW,gBAAgB;AAAA,QAC7D,OAAQ,KAAK,SAAoB,IAAI;AAAA,QACrC,YAAY,KAAK,IAAI,IAAI;AAAA,QACzB,KAAK;AAAA,MACP;AAAA,IACF,SAAS,KAAK;AACZ,mBAAa,aAAa;AAC1B,gBAAU;AACV,UAAI,UAAU,aAAa,KAAK,iBAAiB,GAAG,GAAG;AACrD,cAAM,MAAM,UAAU,OAAO,CAAC;AAC9B;AAAA,MACF;AACA,YAAM;AAAA,IACR;AAAA,EACF;AACA,QAAM,mBAAmB,QAAQ,UAAU,IAAI,MAAM,OAAO,OAAO,CAAC;AACtE;AAQA,eAAsB,YACpB,KACA,OAAyB,CAAC,GACoB;AAC9C,MAAI;AACF,UAAM,SAAS,MAAM,QAAQ,EAAE,GAAG,KAAK,UAAU,IAAI,YAAY,CAAC,IAAI,WAAW,GAAG,IAAI;AACxF,UAAM,QAAQ,gBAAmB,OAAO,SAAS,OAAO,KAAK;AAC7D,WAAO,EAAE,OAAO,OAAO;AAAA,EACzB,SAAS,KAAK;AACZ,QAAI,eAAe,gBAAgB,kBAAkB,IAAI,QAAQ,IAAI,IAAI,KAAK,IAAI,YAAY;AAE5F,YAAM,cAA8B,EAAE,GAAG,KAAK,UAAU,MAAM,YAAY,OAAU;AACpF,YAAM,SAAS,MAAM,QAAQ,aAAa,IAAI;AAC9C,YAAM,QAAQ,gBAAmB,OAAO,SAAS,OAAO,KAAK;AAC7D,aAAO,EAAE,OAAO,OAAO;AAAA,IACzB;AACA,UAAM;AAAA,EACR;AACF;AAEA,SAAS,gBAAmB,SAAiB,OAAkB;AAC7D,QAAM,WAAW,mBAAmB,OAAO;AAC3C,MAAI;AACF,WAAO,KAAK,MAAM,QAAQ;AAAA,EAC5B,SAAS,KAAK;AACZ,UAAM,IAAI;AAAA,MACR,wCAAwC,KAAK,MAC3C,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,CACjD;AAAA;AAAA,EAA0B,QAAQ,MAAM,GAAG,GAAG,CAAC;AAAA,IACjD;AAAA,EACF;AACF;AAaA,eAAsB,SACpB,OACA,OAAkD,CAAC,GACgB;AACnE,QAAM,QAAQ,KAAK,IAAI;AACvB,MAAI;AACF,UAAM;AAAA,MACJ;AAAA,QACE;AAAA,QACA,UAAU,CAAC,EAAE,MAAM,QAAQ,SAAS,OAAO,CAAC;AAAA,QAC5C,WAAW;AAAA,QACX,WAAW,KAAK,aAAa;AAAA,MAC/B;AAAA,MACA;AAAA,IACF;AACA,WAAO,EAAE,IAAI,MAAM,WAAW,KAAK,IAAI,IAAI,OAAO,OAAO,KAAK;AAAA,EAChE,SAAS,KAAK;AACZ,WAAO;AAAA,MACL,IAAI;AAAA,MACJ,WAAW,KAAK,IAAI,IAAI;AAAA,MACxB,OAAO,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG;AAAA,IACxD;AAAA,EACF;AACF;AAOO,IAAM,YAAN,MAAgB;AAAA,EACrB,YAA6B,OAAyB,CAAC,GAAG;AAA7B;AAAA,EAA8B;AAAA,EAA9B;AAAA,EAE7B,KAAK,KAAqB,KAAgD;AACxE,WAAO,QAAQ,KAAK,EAAE,GAAG,KAAK,MAAM,GAAG,IAAI,CAAC;AAAA,EAC9C;AAAA,EAEA,SACE,KACA,KAC8C;AAC9C,WAAO,YAAe,KAAK,EAAE,GAAG,KAAK,MAAM,GAAG,IAAI,CAAC;AAAA,EACrD;AACF;","names":[]}
|