@tangle-network/agent-eval 0.20.10 → 0.20.12

package/README.md

@@ -1,63 +1,24 @@
 # @tangle-network/agent-eval
 
-Trace-first evaluation infrastructure for agent systems.
-
-`agent-eval` provides the contracts and runtime primitives for measuring agent
-behavior: traces, harnesses, verifier pipelines, judges, datasets, holdout
-gates, failure classification, optimization loops, and release reports.
-
-It does not own your product state, credentials, UI, or model routing. Product
-teams keep those boundaries; this package standardizes how runs are recorded,
-checked, compared, and promoted.
-
-## Contents
-
-- [When To Use It](#when-to-use-it)
-- [Architecture](#architecture)
-- [Install](#install)
-- [Core Primitives](#core-primitives)
-- [Examples](#examples)
-- [Documentation](#documentation)
-- [Development](#development)
-- [Related Packages](#related-packages)
-
-## When To Use It
-
-Use `agent-eval` when you need one or more of these:
-
-- A reproducible eval harness for coding agents, builder agents, or multi-tool
-  workflows.
-- Structured traces for agent runs: spans, artifacts, events, budgets, tool
-  calls, retrieval, judge output, and sandbox execution.
-- Deterministic gates around build/test/deploy checks.
-- LLM-as-judge or deterministic judge fleets with calibration and canaries.
-- Dataset splits, holdouts, paired statistics, and release confidence gates.
-- Failure taxonomy that distinguishes prompt, tool, sandbox, retrieval,
-  evaluator, and knowledge-readiness failures.
-- Optimization loops over prompts, steering, code mutations, or full multi-shot
-  trajectories.
-- Report data for internal launch reviews, CI gates, and research analysis.
-
-## Architecture
+Evaluation infrastructure for agent products.
+
+Use it to wrap the real workflow your users run, record what happened, verify
+the result, turn feedback into replay data, compare variants, and ship only
+when the evidence improves.
 
 ```txt
-agent/product run
-  -> TraceEmitter / TraceStore
-  -> SandboxHarness / MultiLayerVerifier / JudgeRunner
-  -> failure taxonomy + metrics
-  -> paired stats + held-out gates
-  -> optimization + release confidence + reports
+product task
+  -> observe state
+  -> validate with deterministic gates first
+  -> act through the real product adapter
+  -> trace + feedback trajectory
+  -> replay / optimize / release gate
 ```
 
-Package responsibilities:
-
-- `agent-eval`: run evidence, eval contracts, verification, statistics,
-  optimization, reporting.
-- Product app: domain state, tools, credentials, UI, storage, deployment, model
-  gateway.
-- `@tangle-network/agent-runtime`: production agent-loop/session runtime.
-- `@tangle-network/agent-knowledge`: evidence stores, claim/page synthesis,
-  retrieval, knowledge readiness implementation.
+`agent-eval` does not own product state, credentials, UI, storage, model
+routing, browser drivers, sandbox policy, or deployment. Products own those.
+This package owns eval contracts, loop mechanics, traces, statistics,
+optimization inputs, and release evidence.
 
 ## Install
 
@@ -65,106 +26,148 @@ Package responsibilities:
 pnpm add @tangle-network/agent-eval
 ```
 
-Wire protocol / CLI:
-
-```sh
-npm i -g @tangle-network/agent-eval
-agent-eval serve --port 5005
+## Quick Start
+
+```ts
+import {
+  objectiveEval,
+  runAgentControlLoop,
+} from '@tangle-network/agent-eval/control'
+
+const result = await runAgentControlLoop({
+  intent: task.prompt,
+  budget: { maxSteps: 8, maxWallMs: 180_000, maxCostUsd: 2 },
+
+  observe() {
+    return product.readState(task.id)
+  },
+
+  validate({ state }) {
+    return [
+      objectiveEval({
+        id: 'build-passes',
+        passed: state.build.exitCode === 0,
+        severity: 'critical',
+        metadata: state.build,
+      }),
+      objectiveEval({
+        id: 'preview-serves',
+        passed: state.preview.httpStatus === 200,
+        severity: 'critical',
+      }),
+    ]
+  },
+
+  decide({ evals }) {
+    const failed = evals.filter((e) => !e.passed)
+    if (failed.length === 0) {
+      return { type: 'stop', pass: true, reason: 'all gates passed' }
+    }
+    return {
+      type: 'continue',
+      action: { type: 'repair', failed: failed.map((e) => e.id) },
+      reason: 'repair failed gates',
+    }
+  },
+
+  act(action) {
+    return product.runAgentStep(task.id, action)
+  },
+})
+
+await product.storeEvalResult(task.id, result)
 ```
 
-Python client source lives in `clients/python`. Until the PyPI package is
-published, install it from the repo:
+That loop should be the same shape in production, replay, benchmark, and
+optimization. Swap dependencies behind `observe()` and `act()`, not the eval
+contract itself.
 
-```sh
-cd clients/python
-pip install -e .
+## Import Paths
+
+The root export remains available, but new code should prefer focused subpaths:
+
+```ts
+import { runAgentControlLoop } from '@tangle-network/agent-eval/control'
+import { runMultiShotOptimization } from '@tangle-network/agent-eval/optimization'
+import { TraceEmitter } from '@tangle-network/agent-eval/traces'
+import { renderReleaseReport } from '@tangle-network/agent-eval/reporting'
 ```
 
-## Core Primitives
-
-| Primitive | Purpose |
-|---|---|
-| `TraceEmitter`, `TraceStore` | Append-only run/span/event/artifact/budget records. |
-| `SandboxHarness` | Build/test/runtime checks with captured stdout, stderr, exit codes, wall time, and parsed test counts. |
-| `MultiLayerVerifier` | Ordered verification stages with dependencies, skip-on-fail, findings, scores, and time caps. |
-| `JudgeRunner` | Parallel deterministic or LLM-backed judges over the same artifact/run. |
-| `runAgentControlLoop` | Observe/validate/decide/act loop with budgets, stop policies, and structured eval results. |
-| `Dataset`, `RunRecord`, `HeldOutGate` | Versioned corpora, reproducible run metadata, and held-out promotion decisions. |
-| `pairedBootstrap`, `pairedWilcoxon`, `bhAdjust` | Paired experiment statistics and multiple-comparison correction. |
-| `classifyFailure` | Rule-based failure classification for agent, tool, sandbox, retrieval, and knowledge failures. |
-| `runMultiShotOptimization` | Optimization over full agent trajectories with actionable side information. |
-| `runPromptEvolution` | Prompt/steering/code evolution over scenario scores. |
-| `evaluateReleaseConfidence` | Release scorecard across evidence volume, pass rate, score, overfit, cost, latency, and gates. |
-| `summaryTable`, `paretoChart`, `gainHistogram` | Report-ready structured outputs. |
-| `KnowledgeRequirement`, `KnowledgeBundle` | Shared contracts for knowledge readiness. |
-
-`NoopResearcher` is a fail-loud sentinel for wiring tests. Production systems
-should implement `Researcher` directly or use `CallbackResearcher`.
+| Subpath | Use for |
+| --- | --- |
+| `@tangle-network/agent-eval/control` | `observe -> validate -> decide -> act`, action policy, propose/review loops |
+| `@tangle-network/agent-eval/traces` | trace stores, emitters, TraceAnalyst |
+| `@tangle-network/agent-eval/optimization` | feedback trajectories, multi-shot optimization, prompt evolution |
+| `@tangle-network/agent-eval/reporting` | release confidence, paired stats, report/table/chart specs |
+| `@tangle-network/agent-eval/wire` | HTTP/RPC judge server and schemas |
+| `@tangle-network/agent-eval/benchmarks` | benchmark adapter contracts and reference wrappers |
+
+## Core Pieces
+
+| Need | Use |
+| --- | --- |
+| Keep an agent working until objective state passes | `runAgentControlLoop` |
+| Turn user/reviewer feedback into replay data | `FeedbackTrajectory` |
+| Compare prompt/tool/retrieval policies over full trajectories | `runMultiShotOptimization` |
+| Gate releases with paired evidence and holdouts | `evaluateReleaseConfidence`, `HeldOutGate` |
+| Explain regressions across trace corpora | `TraceAnalyst` / `analyzeTraces` |
+| Report a launch decision | `renderReleaseReport`, `summaryTable`, `paretoChart`, `gainHistogram` |
+| Model missing context separately from bad reasoning | `KnowledgeRequirement`, `KnowledgeBundle` |
 
 ## Examples
 
-Runnable examples live in the repository's
-[`examples/`](https://github.com/tangle-network/agent-eval/tree/main/examples)
-directory. They are not part of the published npm package.
+Runnable examples live in
+[`examples/`](https://github.com/tangle-network/agent-eval/tree/main/examples).
 
-- [`examples/same-sandbox-harness`](https://github.com/tangle-network/agent-eval/tree/main/examples/same-sandbox-harness) - run
-  multiple eval passes against the same workspace.
-- [`examples/multi-shot-optimization`](https://github.com/tangle-network/agent-eval/tree/main/examples/multi-shot-optimization) -
-  optimize full agent trajectories with held-out promotion.
-- [`examples/benchmarks`](https://github.com/tangle-network/agent-eval/tree/main/examples/benchmarks) - benchmark adapter shape and
-  reference benchmark wrappers.
+- [`examples/multi-shot-optimization`](https://github.com/tangle-network/agent-eval/tree/main/examples/multi-shot-optimization):
+  optimize full trajectories with held-out promotion.
+- [`examples/same-sandbox-harness`](https://github.com/tangle-network/agent-eval/tree/main/examples/same-sandbox-harness):
+  run setup/build/test and evidence checks in one workspace.
+- [`examples/benchmarks`](https://github.com/tangle-network/agent-eval/tree/main/examples/benchmarks):
+  benchmark adapter shape and reference wrappers.
 
-The examples are intentionally kept outside the README so they can be expanded,
-tested, and copied without turning this page into a tutorial.
+## Docs
 
-## Documentation
+Read in this order:
 
-- [Concepts](./docs/concepts.md)
-- [Feature Guide](./docs/feature-guide.md)
-- [Control Runtime](./docs/control-runtime.md)
-- [Knowledge Readiness](./docs/knowledge-readiness.md)
-- [Multi-Shot Optimization](./docs/multi-shot-optimization.md)
-- [Feedback Trajectories](./docs/feedback-trajectories.md)
-- [Wire Protocol](./docs/wire-protocol.md)
+1. [Product Eval Adoption](./docs/product-eval-adoption.md)
+2. [Control Runtime](./docs/control-runtime.md)
+3. [Feedback Trajectories](./docs/feedback-trajectories.md)
+4. [Multi-Shot Optimization](./docs/multi-shot-optimization.md)
+5. [Trace Analysis](./docs/trace-analysis.md)
+6. [Knowledge Readiness](./docs/knowledge-readiness.md)
+7. [Integration Launch Gates](./docs/integration-launch-gates.md)
+8. [Wire Protocol](./docs/wire-protocol.md)
 
-## Development
+## CLI / Wire Protocol
 
 ```sh
-pnpm install
-pnpm typecheck
-pnpm test
-pnpm build
-pnpm openapi
+npm i -g @tangle-network/agent-eval
+agent-eval serve --port 5005
 ```
 
-Run the local server:
+The Python client lives in `clients/python`:
 
 ```sh
-pnpm build
-node dist/cli.js serve --port 5005
+cd clients/python
+pip install -e .
 ```
 
-Python client tests:
+## Development
 
 ```sh
+pnpm install
+pnpm typecheck
+pnpm test
 pnpm build
-cd clients/python
-pip install -e ".[dev]"
-pytest
+pnpm openapi
 ```
 
-## Release
-
-`@tangle-network/agent-eval` publishes to npm. The Python client lives under
-`clients/python` and is versioned from this repository.
-
 ## Related Packages
 
-- [`@tangle-network/agent-runtime`](https://github.com/tangle-network/agent-runtime)
-- [`@tangle-network/agent-knowledge`](https://github.com/tangle-network/agent-knowledge)
-- [`@tangle-network/agent-integrations`](https://github.com/tangle-network/agent-integrations)
-- [`@tangle-network/agent-gateway`](https://github.com/tangle-network/agent-gateway)
-- [`@tangle-network/tcloud`](https://github.com/tangle-network/tcloud)
+- `@tangle-network/agent-runtime`: production session/runtime layer.
+- `@tangle-network/agent-knowledge`: source-grounded knowledge bases and readiness.
+- `@tangle-network/agent-integrations`: connection, grant, capability, and integration invocation contracts.
 
 ## License
 

package/dist/benchmarks/index.d.ts

@@ -1 +1,2 @@
-export { B as BENCHMARK_SPLIT_SEED, b as BenchmarkAdapter, c as BenchmarkDatasetItem, d as BenchmarkEvaluation, i as deterministicSplit, l as routing } from '../index-1PZOtZFr.js';
+export { B as BENCHMARK_SPLIT_SEED, a as BenchmarkAdapter, b as BenchmarkDatasetItem, c as BenchmarkEvaluation, d as deterministicSplit, e as routing } from '../index-c5saLbKD.js';
+import '../run-record-CX_jcAyr.js';

package/dist/{chunk-JAOLXRIA.js → chunk-75MCTH7P.js}

@@ -57,7 +57,10 @@ function buildBody(req, forceJsonObject) {
     messages: req.messages,
     temperature: req.temperature ?? 0
   };
-  if (req.maxTokens != null) body.max_tokens = req.maxTokens;
+  if (req.maxTokens != null) {
+    if (usesMaxCompletionTokens(req.model)) body.max_completion_tokens = req.maxTokens;
+    else body.max_tokens = req.maxTokens;
+  }
   if (req.jsonSchema && !forceJsonObject) {
     body.response_format = {
       type: "json_schema",
@@ -68,6 +71,9 @@ function buildBody(req, forceJsonObject) {
   }
   return body;
 }
+function usesMaxCompletionTokens(model) {
+  return /^gpt-5(?:[.\-]|$)/i.test(model);
+}
 async function sleep(ms) {
   return new Promise((resolve) => setTimeout(resolve, ms));
 }
@@ -262,4 +268,4 @@ export {
   probeLlm,
   LlmClient
 };
-//# sourceMappingURL=chunk-JAOLXRIA.js.map
+//# sourceMappingURL=chunk-75MCTH7P.js.map

package/dist/chunk-75MCTH7P.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../src/llm-client.ts"],"sourcesContent":["/**\n * LLM client with graceful degrade.\n *\n * OpenAI-compatible `/v1/chat/completions` client with:\n *   - Exponential-backoff retry on 429 + 5xx gateway errors (502/503/504).\n *   - Retry on transient network errors (fetch failed, AbortError, ECONNRESET).\n *   - Graceful json_schema → json_object degrade on 400 with schema-reject body.\n *   - Fenced-JSON stripping (```json ... ```) for models that wrap structured output.\n *   - Configurable base URL + api key / bearer, works with LiteLLM proxies, OpenAI\n *     directly, cli-bridge subscriptions, and any router that speaks the spec.\n *\n * Usage:\n *   const { value, result } = await callLlmJson<MyType>(\n *     { model: 'gpt-4o', messages: [...], jsonSchema: { name: 'x', schema: {...} } },\n *     { baseUrl: 'https://router.tangle.tools/v1', apiKey: process.env.KEY },\n *   )\n *\n * This is THE llm-calling seam for agent-eval primitives that need structured\n * output (semantic concept judge, reviewer directives, critic scores). Primitives\n * that need free-form text use `callLlm` and parse output themselves.\n */\n\n// ─── Types ──────────────────────────────────────────────────────────────\n\nexport interface LlmMessage {\n  role: 'system' | 'user' | 'assistant'\n  /**\n   * Either a plain text content string OR a multimodal content array\n   * (text + image_url parts) for vision-capable models.\n   */\n  content:\n    | string\n    | Array<\n        | { type: 'text'; text: string }\n        | { type: 'image_url'; image_url: { url: string; detail?: 'auto' | 'low' | 'high' } }\n      >\n}\n\nexport interface LlmCallRequest {\n  model: string\n  messages: LlmMessage[]\n  /** Optional JSON-mode response format (response_format: json_object). */\n  jsonMode?: boolean\n  /** Optional structured output via JSON Schema. Falls back to json_object on 400. */\n  jsonSchema?: { name: string; schema: Record<string, unknown> }\n  temperature?: number\n  maxTokens?: number\n  /** Per-call timeout, default 60s. */\n  timeoutMs?: number\n}\n\nexport interface LlmUsage {\n  promptTokens: number\n  completionTokens: number\n  totalTokens: number\n  /** Proxies populate this when prompt caching is on. */\n  cachedPromptTokens?: number\n}\n\nexport interface LlmCallResult {\n  /** The text content of the first choice. Empty string if none. */\n  content: string\n  usage: LlmUsage\n  /**\n   * Cost in USD. Pulled from proxy's `_response_cost` field when present;\n   * `null` when neither the proxy nor the caller can derive it.\n   */\n  costUsd: number | null\n  /** Model name actually used (echoed from response). */\n  model: string\n  /** Wall-clock duration of the HTTP call (last attempt, if retried). */\n  durationMs: number\n  /** Raw response body. */\n  raw: Record<string, unknown>\n}\n\nexport class LlmCallError extends Error {\n  constructor(\n    message: string,\n    public readonly status: number,\n    public readonly body: string,\n    public readonly model: string,\n  ) {\n    super(message)\n    this.name = 'LlmCallError'\n  }\n}\n\nexport interface LlmClientOptions {\n  /** Base URL (without trailing slash). Must end at the `/v1` prefix. */\n  baseUrl?: string\n  /** Bearer token — either `apiKey` or `bearer` populates `Authorization: Bearer ...`. */\n  apiKey?: string\n  bearer?: string\n  /** Override for the `Authorization` header (e.g. `X-Auth: ...`). Takes precedence over apiKey/bearer. */\n  authHeader?: { name: string; value: string }\n  /** Default timeout in ms. Per-call can override. */\n  defaultTimeoutMs?: number\n  /** Max retry attempts on retriable errors. Default 3 (1 initial + 2 retries). */\n  maxRetries?: number\n  /** Fetch implementation — defaults to global `fetch`. Override for custom transport (e.g. tests). */\n  fetch?: typeof fetch\n}\n\n// ─── Internals ──────────────────────────────────────────────────────────\n\nconst DEFAULT_BASE_URL = 'https://router.tangle.tools/v1'\nconst DEFAULT_TIMEOUT_MS = 60_000\nconst DEFAULT_MAX_RETRIES = 3\n\nconst RETRYABLE_STATUS = new Set([429, 502, 503, 504])\n\nfunction isRetryableError(err: unknown): boolean {\n  if (err instanceof LlmCallError) return RETRYABLE_STATUS.has(err.status)\n  if (err instanceof Error) {\n    return (\n      err.name === 'AbortError' ||\n      err.name === 'TimeoutError' ||\n      /fetch failed|ECONNRESET|ETIMEDOUT|EAI_AGAIN/i.test(err.message)\n    )\n  }\n  return false\n}\n\nfunction parseRetryAfter(headers: Headers): number | null {\n  const h = headers.get('retry-after')\n  if (!h) return null\n  const asNumber = Number(h)\n  if (Number.isFinite(asNumber) && asNumber > 0) return asNumber * 1000\n  const asDate = Date.parse(h)\n  if (Number.isFinite(asDate)) return Math.max(0, asDate - Date.now())\n  return null\n}\n\nfunction backoffMs(attempt: number): number {\n  // 500ms, 1s, 2s, 4s, ...\n  return Math.min(500 * Math.pow(2, attempt), 16_000)\n}\n\nfunction buildHeaders(opts: LlmClientOptions): Record<string, string> {\n  const headers: Record<string, string> = {\n    'Content-Type': 'application/json',\n    Accept: 'application/json',\n  }\n  if (opts.authHeader) {\n    headers[opts.authHeader.name] = opts.authHeader.value\n  } else if (opts.bearer || opts.apiKey) {\n    headers.Authorization = `Bearer ${opts.bearer ?? opts.apiKey}`\n  }\n  return headers\n}\n\nfunction isSchemaRejection(status: number, body: string): boolean {\n  if (status !== 400) return false\n  const lower = body.toLowerCase()\n  return (\n    lower.includes('response_format') ||\n    lower.includes('json_schema') ||\n    lower.includes('is unavailable') ||\n    lower.includes('not supported')\n  )\n}\n\nfunction buildBody(req: LlmCallRequest, forceJsonObject: boolean): Record<string, unknown> {\n  const body: Record<string, unknown> = {\n    model: req.model,\n    messages: req.messages,\n    temperature: req.temperature ?? 0,\n  }\n  if (req.maxTokens != null) {\n    if (usesMaxCompletionTokens(req.model)) body.max_completion_tokens = req.maxTokens\n    else body.max_tokens = req.maxTokens\n  }\n\n  if (req.jsonSchema && !forceJsonObject) {\n    body.response_format = {\n      type: 'json_schema',\n      json_schema: { name: req.jsonSchema.name, schema: req.jsonSchema.schema, strict: true },\n    }\n  } else if (req.jsonMode || req.jsonSchema) {\n    body.response_format = { type: 'json_object' }\n  }\n\n  return body\n}\n\nfunction usesMaxCompletionTokens(model: string): boolean {\n  return /^gpt-5(?:[.\\-]|$)/i.test(model)\n}\n\nasync function sleep(ms: number): Promise<void> {\n  return new Promise((resolve) => setTimeout(resolve, ms))\n}\n\n// ─── Public API ─────────────────────────────────────────────────────────\n\n/**\n * Strip a ```json / ``` code fence if the model emitted one.\n * Idempotent for naked JSON. Some models (claude-code via router, certain\n * deepseek models) wrap output even under json_object.\n */\nexport function stripFencedJson(raw: string): string {\n  const trimmed = raw.trim()\n  const m = trimmed.match(/^```(?:json)?\\s*\\n?([\\s\\S]*?)\\n?```\\s*$/)\n  return m ? m[1]!.trim() : trimmed\n}\n\nexport function extractJsonPayload(raw: string): string {\n  const stripped = stripFencedJson(raw)\n  try {\n    JSON.parse(stripped)\n    return stripped\n  } catch {\n    // Continue with balanced extraction below.\n  }\n\n  const starts = [...stripped.matchAll(/[\\[{]/g)].map((match) => match.index).filter((index) => index != null)\n  for (const start of starts) {\n    const candidate = extractBalancedJson(stripped, start)\n    if (!candidate) continue\n    try {\n      JSON.parse(candidate)\n      return candidate\n    } catch {\n      // Keep scanning; earlier braces may belong to prose.\n    }\n  }\n\n  return stripped\n}\n\nfunction extractBalancedJson(input: string, start: number): string | null {\n  const opener = input[start]\n  const closer = opener === '{' ? '}' : opener === '[' ? ']' : null\n  if (!closer) return null\n\n  const stack: string[] = [closer]\n  let isInString = false\n  let isEscaped = false\n\n  for (let i = start + 1; i < input.length; i++) {\n    const char = input[i]!\n    if (isEscaped) {\n      isEscaped = false\n      continue\n    }\n    if (char === '\\\\') {\n      isEscaped = isInString\n      continue\n    }\n    if (char === '\"') {\n      isInString = !isInString\n      continue\n    }\n    if (isInString) continue\n\n    if (char === '{') stack.push('}')\n    else if (char === '[') stack.push(']')\n    else if (char === stack[stack.length - 1]) {\n      stack.pop()\n      if (stack.length === 0) return input.slice(start, i + 1)\n    }\n  }\n\n  return null\n}\n\n/**\n * Low-level call. Returns raw content + usage + cost. Retries on transient\n * failures; does NOT degrade schema here — callers that want graceful\n * degrade use `callLlmJson`.\n */\nexport async function callLlm(\n  req: LlmCallRequest,\n  opts: LlmClientOptions = {},\n): Promise<LlmCallResult> {\n  const baseUrl = (opts.baseUrl ?? DEFAULT_BASE_URL).replace(/\\/+$/, '')\n  const url = `${baseUrl}/chat/completions`\n  const timeoutMs = req.timeoutMs ?? opts.defaultTimeoutMs ?? DEFAULT_TIMEOUT_MS\n  const maxRetries = opts.maxRetries ?? DEFAULT_MAX_RETRIES\n  const fetchFn = opts.fetch ?? globalThis.fetch\n  const headers = buildHeaders(opts)\n\n  let lastErr: unknown\n  for (let attempt = 0; attempt < maxRetries; attempt++) {\n    const controller = new AbortController()\n    const timeoutHandle = setTimeout(() => controller.abort(), timeoutMs)\n    const started = Date.now()\n\n    try {\n      const res = await fetchFn(url, {\n        method: 'POST',\n        headers,\n        body: JSON.stringify(buildBody(req, false)),\n        signal: controller.signal,\n      })\n      clearTimeout(timeoutHandle)\n\n      if (!res.ok) {\n        const body = await res.text()\n        const err = new LlmCallError(\n          `LLM call ${res.status}: ${body.slice(0, 300)}`,\n          res.status,\n          body,\n          req.model,\n        )\n        if (RETRYABLE_STATUS.has(res.status) && attempt < maxRetries - 1) {\n          lastErr = err\n          const retryAfter = parseRetryAfter(res.headers)\n          await sleep(retryAfter ?? backoffMs(attempt))\n          continue\n        }\n        throw err\n      }\n\n      const json = (await res.json()) as Record<string, unknown>\n      const choice = (json.choices as Array<{ message?: { content?: string } }> | undefined)?.[0]\n      const usageRaw = (json.usage as Record<string, unknown> | undefined) ?? {}\n      const costFromProxy = (json._response_cost ?? json.cost_usd) as number | undefined\n\n      return {\n        content: choice?.message?.content ?? '',\n        usage: {\n          promptTokens: Number(usageRaw.prompt_tokens ?? 0),\n          completionTokens: Number(usageRaw.completion_tokens ?? 0),\n          totalTokens: Number(usageRaw.total_tokens ?? 0),\n          cachedPromptTokens:\n            usageRaw.prompt_tokens_details &&\n            typeof usageRaw.prompt_tokens_details === 'object'\n              ? Number(\n                  (usageRaw.prompt_tokens_details as Record<string, unknown>).cached_tokens ?? 0,\n                )\n              : undefined,\n        },\n        costUsd: typeof costFromProxy === 'number' ? costFromProxy : null,\n        model: (json.model as string) ?? req.model,\n        durationMs: Date.now() - started,\n        raw: json,\n      }\n    } catch (err) {\n      clearTimeout(timeoutHandle)\n      lastErr = err\n      if (attempt < maxRetries - 1 && isRetryableError(err)) {\n        await sleep(backoffMs(attempt))\n        continue\n      }\n      throw err\n    }\n  }\n  throw lastErr instanceof Error ? lastErr : new Error(String(lastErr))\n}\n\n/**\n * Structured-output call. Returns parsed JSON plus the raw result envelope.\n * Degrades `jsonSchema` → `jsonMode` on a 400 that names the schema param —\n * critical for deepseek-v3/v4, kimi-k2.6, and other models that don't accept\n * the `response_format.json_schema` shape but DO accept `json_object`.\n */\nexport async function callLlmJson<T = unknown>(\n  req: LlmCallRequest,\n  opts: LlmClientOptions = {},\n): Promise<{ value: T; result: LlmCallResult }> {\n  try {\n    const result = await callLlm({ ...req, jsonMode: req.jsonMode ?? !req.jsonSchema }, opts)\n    const value = parseJsonSafely<T>(result.content, result.model)\n    return { value, result }\n  } catch (err) {\n    if (err instanceof LlmCallError && isSchemaRejection(err.status, err.body) && req.jsonSchema) {\n      // Degrade to json_object + retry.\n      const degradedReq: LlmCallRequest = { ...req, jsonMode: true, jsonSchema: undefined }\n      const result = await callLlm(degradedReq, opts)\n      const value = parseJsonSafely<T>(result.content, result.model)\n      return { value, result }\n    }\n    throw err\n  }\n}\n\nfunction parseJsonSafely<T>(content: string, model: string): T {\n  const stripped = extractJsonPayload(content)\n  try {\n    return JSON.parse(stripped) as T\n  } catch (err) {\n    throw new Error(\n      `LLM returned non-JSON content (model=${model}): ${\n        err instanceof Error ? err.message : String(err)\n      }\\n--- raw content ---\\n${content.slice(0, 800)}`,\n    )\n  }\n}\n\n/**\n * Probe whether a model is reachable. Returns latency + null error on\n * success; `ok=false` + error message on any failure (HTTP, timeout,\n * network, parse). Designed for sweep preflights — fail loud at the\n * boundary before burning a 30-leaf run on a misconfigured router.\n *\n * Sends a tiny `ping` message with `maxTokens=64`. Reasoning models\n * (glm-5.1, deepseek-v4) can burn the entire budget on internal reasoning\n * for short prompts, so don't tighten this further. We don't validate\n * content; HTTP 200 means reachable.\n */\nexport async function probeLlm(\n  model: string,\n  opts: LlmClientOptions & { timeoutMs?: number } = {},\n): Promise<{ ok: boolean; latencyMs: number; error: string | null }> {\n  const start = Date.now()\n  try {\n    await callLlm(\n      {\n        model,\n        messages: [{ role: 'user', content: 'ping' }],\n        maxTokens: 64,\n        timeoutMs: opts.timeoutMs ?? 30_000,\n      },\n      opts,\n    )\n    return { ok: true, latencyMs: Date.now() - start, error: null }\n  } catch (err) {\n    return {\n      ok: false,\n      latencyMs: Date.now() - start,\n      error: err instanceof Error ? err.message : String(err),\n    }\n  }\n}\n\n/**\n * Stateful client — construct once with defaults, call many times.\n * Thin wrapper around the free functions; exists for callers that want\n * to inject a single configured instance into multiple primitives.\n */\nexport class LlmClient {\n  constructor(private readonly opts: LlmClientOptions = {}) {}\n\n  call(req: LlmCallRequest, per?: LlmClientOptions): Promise<LlmCallResult> {\n    return callLlm(req, { ...this.opts, ...per })\n  }\n\n  callJson<T = unknown>(\n    req: LlmCallRequest,\n    per?: LlmClientOptions,\n  ): Promise<{ value: T; result: LlmCallResult }> {\n    return callLlmJson<T>(req, { ...this.opts, ...per })\n  }\n}\n"],"mappings":";AA4EO,IAAM,eAAN,cAA2B,MAAM;AAAA,EACtC,YACE,SACgB,QACA,MACA,OAChB;AACA,UAAM,OAAO;AAJG;AACA;AACA;AAGhB,SAAK,OAAO;AAAA,EACd;AAAA,EANkB;AAAA,EACA;AAAA,EACA;AAKpB;AAoBA,IAAM,mBAAmB;AACzB,IAAM,qBAAqB;AAC3B,IAAM,sBAAsB;AAE5B,IAAM,mBAAmB,oBAAI,IAAI,CAAC,KAAK,KAAK,KAAK,GAAG,CAAC;AAErD,SAAS,iBAAiB,KAAuB;AAC/C,MAAI,eAAe,aAAc,QAAO,iBAAiB,IAAI,IAAI,MAAM;AACvE,MAAI,eAAe,OAAO;AACxB,WACE,IAAI,SAAS,gBACb,IAAI,SAAS,kBACb,+CAA+C,KAAK,IAAI,OAAO;AAAA,EAEnE;AACA,SAAO;AACT;AAEA,SAAS,gBAAgB,SAAiC;AACxD,QAAM,IAAI,QAAQ,IAAI,aAAa;AACnC,MAAI,CAAC,EAAG,QAAO;AACf,QAAM,WAAW,OAAO,CAAC;AACzB,MAAI,OAAO,SAAS,QAAQ,KAAK,WAAW,EAAG,QAAO,WAAW;AACjE,QAAM,SAAS,KAAK,MAAM,CAAC;AAC3B,MAAI,OAAO,SAAS,MAAM,EAAG,QAAO,KAAK,IAAI,GAAG,SAAS,KAAK,IAAI,CAAC;AACnE,SAAO;AACT;AAEA,SAAS,UAAU,SAAyB;AAE1C,SAAO,KAAK,IAAI,MAAM,KAAK,IAAI,GAAG,OAAO,GAAG,IAAM;AACpD;AAEA,SAAS,aAAa,MAAgD;AACpE,QAAM,UAAkC;AAAA,IACtC,gBAAgB;AAAA,IAChB,QAAQ;AAAA,EACV;AACA,MAAI,KAAK,YAAY;AACnB,YAAQ,KAAK,WAAW,IAAI,IAAI,KAAK,WAAW;AAAA,EAClD,WAAW,KAAK,UAAU,KAAK,QAAQ;AACrC,YAAQ,gBAAgB,UAAU,KAAK,UAAU,KAAK,MAAM;AAAA,EAC9D;AACA,SAAO;AACT;AAEA,SAAS,kBAAkB,QAAgB,MAAuB;AAChE,MAAI,WAAW,IAAK,QAAO;AAC3B,QAAM,QAAQ,KAAK,YAAY;AAC/B,SACE,MAAM,SAAS,iBAAiB,KAChC,MAAM,SAAS,aAAa,KAC5B,MAAM,SAAS,gBAAgB,KAC/B,MAAM,SAAS,eAAe;AAElC;AAEA,SAAS,UAAU,KAAqB,iBAAmD;AACzF,QAAM,OAAgC;AAAA,IACpC,OAAO,IAAI;AAAA,IACX,UAAU,IAAI;AAAA,IACd,aAAa,IAAI,eAAe;AAAA,EAClC;AACA,MAAI,IAAI,aAAa,MAAM;AACzB,QAAI,wBAAwB,IAAI,KAAK,EAAG,MAAK,wBAAwB,IAAI;AAAA,QACpE,MAAK,aAAa,IAAI;AAAA,EAC7B;AAEA,MAAI,IAAI,cAAc,CAAC,iBAAiB;AACtC,SAAK,kBAAkB;AAAA,MACrB,MAAM;AAAA,MACN,aAAa,EAAE,MAAM,IAAI,WAAW,MAAM,QAAQ,IAAI,WAAW,QAAQ,QAAQ,KAAK;AAAA,IACxF;AAAA,EACF,WAAW,IAAI,YAAY,IAAI,YAAY;AACzC,SAAK,kBAAkB,EAAE,MAAM,cAAc;AAAA,EAC/C;AAEA,SAAO;AACT;AAEA,SAAS,wBAAwB,OAAwB;AACvD,SAAO,qBAAqB,KAAK,KAAK;AACxC;AAEA,eAAe,MAAM,IAA2B;AAC9C,SAAO,IAAI,QAAQ,CAAC,YAAY,WAAW,SAAS,EAAE,CAAC;AACzD;AASO,SAAS,gBAAgB,KAAqB;AACnD,QAAM,UAAU,IAAI,KAAK;AACzB,QAAM,IAAI,QAAQ,MAAM,yCAAyC;AACjE,SAAO,IAAI,EAAE,CAAC,EAAG,KAAK,IAAI;AAC5B;AAEO,SAAS,mBAAmB,KAAqB;AACtD,QAAM,WAAW,gBAAgB,GAAG;AACpC,MAAI;AACF,SAAK,MAAM,QAAQ;AACnB,WAAO;AAAA,EACT,QAAQ;AAAA,EAER;AAEA,QAAM,SAAS,CAAC,GAAG,SAAS,SAAS,QAAQ,CAAC,EAAE,IAAI,CAAC,UAAU,MAAM,KAAK,EAAE,OAAO,CAAC,UAAU,SAAS,IAAI;AAC3G,aAAW,SAAS,QAAQ;AAC1B,UAAM,YAAY,oBAAoB,UAAU,KAAK;AACrD,QAAI,CAAC,UAAW;AAChB,QAAI;AACF,WAAK,MAAM,SAAS;AACpB,aAAO;AAAA,IACT,QAAQ;AAAA,IAER;AAAA,EACF;AAEA,SAAO;AACT;AAEA,SAAS,oBAAoB,OAAe,OAA8B;AACxE,QAAM,SAAS,MAAM,KAAK;AAC1B,QAAM,SAAS,WAAW,MAAM,MAAM,WAAW,MAAM,MAAM;AAC7D,MAAI,CAAC,OAAQ,QAAO;AAEpB,QAAM,QAAkB,CAAC,MAAM;AAC/B,MAAI,aAAa;AACjB,MAAI,YAAY;AAEhB,WAAS,IAAI,QAAQ,GAAG,IAAI,MAAM,QAAQ,KAAK;AAC7C,UAAM,OAAO,MAAM,CAAC;AACpB,QAAI,WAAW;AACb,kBAAY;AACZ;AAAA,IACF;AACA,QAAI,SAAS,MAAM;AACjB,kBAAY;AACZ;AAAA,IACF;AACA,QAAI,SAAS,KAAK;AAChB,mBAAa,CAAC;AACd;AAAA,IACF;AACA,QAAI,WAAY;AAEhB,QAAI,SAAS,IAAK,OAAM,KAAK,GAAG;AAAA,aACvB,SAAS,IAAK,OAAM,KAAK,GAAG;AAAA,aAC5B,SAAS,MAAM,MAAM,SAAS,CAAC,GAAG;AACzC,YAAM,IAAI;AACV,UAAI,MAAM,WAAW,EAAG,QAAO,MAAM,MAAM,OAAO,IAAI,CAAC;AAAA,IACzD;AAAA,EACF;AAEA,SAAO;AACT;AAOA,eAAsB,QACpB,KACA,OAAyB,CAAC,GACF;AACxB,QAAM,WAAW,KAAK,WAAW,kBAAkB,QAAQ,QAAQ,EAAE;AACrE,QAAM,MAAM,GAAG,OAAO;AACtB,QAAM,YAAY,IAAI,aAAa,KAAK,oBAAoB;AAC5D,QAAM,aAAa,KAAK,cAAc;AACtC,QAAM,UAAU,KAAK,SAAS,WAAW;AACzC,QAAM,UAAU,aAAa,IAAI;AAEjC,MAAI;AACJ,WAAS,UAAU,GAAG,UAAU,YAAY,WAAW;AACrD,UAAM,aAAa,IAAI,gBAAgB;AACvC,UAAM,gBAAgB,WAAW,MAAM,WAAW,MAAM,GAAG,SAAS;AACpE,UAAM,UAAU,KAAK,IAAI;AAEzB,QAAI;AACF,YAAM,MAAM,MAAM,QAAQ,KAAK;AAAA,QAC7B,QAAQ;AAAA,QACR;AAAA,QACA,MAAM,KAAK,UAAU,UAAU,KAAK,KAAK,CAAC;AAAA,QAC1C,QAAQ,WAAW;AAAA,MACrB,CAAC;AACD,mBAAa,aAAa;AAE1B,UAAI,CAAC,IAAI,IAAI;AACX,cAAM,OAAO,MAAM,IAAI,KAAK;AAC5B,cAAM,MAAM,IAAI;AAAA,UACd,YAAY,IAAI,MAAM,KAAK,KAAK,MAAM,GAAG,GAAG,CAAC;AAAA,UAC7C,IAAI;AAAA,UACJ;AAAA,UACA,IAAI;AAAA,QACN;AACA,YAAI,iBAAiB,IAAI,IAAI,MAAM,KAAK,UAAU,aAAa,GAAG;AAChE,oBAAU;AACV,gBAAM,aAAa,gBAAgB,IAAI,OAAO;AAC9C,gBAAM,MAAM,cAAc,UAAU,OAAO,CAAC;AAC5C;AAAA,QACF;AACA,cAAM;AAAA,MACR;AAEA,YAAM,OAAQ,MAAM,IAAI,KAAK;AAC7B,YAAM,SAAU,KAAK,UAAoE,CAAC;AAC1F,YAAM,WAAY,KAAK,SAAiD,CAAC;AACzE,YAAM,gBAAiB,KAAK,kBAAkB,KAAK;AAEnD,aAAO;AAAA,QACL,SAAS,QAAQ,SAAS,WAAW;AAAA,QACrC,OAAO;AAAA,UACL,cAAc,OAAO,SAAS,iBAAiB,CAAC;AAAA,UAChD,kBAAkB,OAAO,SAAS,qBAAqB,CAAC;AAAA,UACxD,aAAa,OAAO,SAAS,gBAAgB,CAAC;AAAA,UAC9C,oBACE,SAAS,yBACT,OAAO,SAAS,0BAA0B,WACtC;AAAA,YACG,SAAS,sBAAkD,iBAAiB;AAAA,UAC/E,IACA;AAAA,QACR;AAAA,QACA,SAAS,OAAO,kBAAkB,WAAW,gBAAgB;AAAA,QAC7D,OAAQ,KAAK,SAAoB,IAAI;AAAA,QACrC,YAAY,KAAK,IAAI,IAAI;AAAA,QACzB,KAAK;AAAA,MACP;AAAA,IACF,SAAS,KAAK;AACZ,mBAAa,aAAa;AAC1B,gBAAU;AACV,UAAI,UAAU,aAAa,KAAK,iBAAiB,GAAG,GAAG;AACrD,cAAM,MAAM,UAAU,OAAO,CAAC;AAC9B;AAAA,MACF;AACA,YAAM;AAAA,IACR;AAAA,EACF;AACA,QAAM,mBAAmB,QAAQ,UAAU,IAAI,MAAM,OAAO,OAAO,CAAC;AACtE;AAQA,eAAsB,YACpB,KACA,OAAyB,CAAC,GACoB;AAC9C,MAAI;AACF,UAAM,SAAS,MAAM,QAAQ,EAAE,GAAG,KAAK,UAAU,IAAI,YAAY,CAAC,IAAI,WAAW,GAAG,IAAI;AACxF,UAAM,QAAQ,gBAAmB,OAAO,SAAS,OAAO,KAAK;AAC7D,WAAO,EAAE,OAAO,OAAO;AAAA,EACzB,SAAS,KAAK;AACZ,QAAI,eAAe,gBAAgB,kBAAkB,IAAI,QAAQ,IAAI,IAAI,KAAK,IAAI,YAAY;AAE5F,YAAM,cAA8B,EAAE,GAAG,KAAK,UAAU,MAAM,YAAY,OAAU;AACpF,YAAM,SAAS,MAAM,QAAQ,aAAa,IAAI;AAC9C,YAAM,QAAQ,gBAAmB,OAAO,SAAS,OAAO,KAAK;AAC7D,aAAO,EAAE,OAAO,OAAO;AAAA,IACzB;AACA,UAAM;AAAA,EACR;AACF;AAEA,SAAS,gBAAmB,SAAiB,OAAkB;AAC7D,QAAM,WAAW,mBAAmB,OAAO;AAC3C,MAAI;AACF,WAAO,KAAK,MAAM,QAAQ;AAAA,EAC5B,SAAS,KAAK;AACZ,UAAM,IAAI;AAAA,MACR,wCAAwC,KAAK,MAC3C,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,CACjD;AAAA;AAAA,EAA0B,QAAQ,MAAM,GAAG,GAAG,CAAC;AAAA,IACjD;AAAA,EACF;AACF;AAaA,eAAsB,SACpB,OACA,OAAkD,CAAC,GACgB;AACnE,QAAM,QAAQ,KAAK,IAAI;AACvB,MAAI;AACF,UAAM;AAAA,MACJ;AAAA,QACE;AAAA,QACA,UAAU,CAAC,EAAE,MAAM,QAAQ,SAAS,OAAO,CAAC;AAAA,QAC5C,WAAW;AAAA,QACX,WAAW,KAAK,aAAa;AAAA,MAC/B;AAAA,MACA;AAAA,IACF;AACA,WAAO,EAAE,IAAI,MAAM,WAAW,KAAK,IAAI,IAAI,OAAO,OAAO,KAAK;AAAA,EAChE,SAAS,KAAK;AACZ,WAAO;AAAA,MACL,IAAI;AAAA,MACJ,WAAW,KAAK,IAAI,IAAI;AAAA,MACxB,OAAO,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG;AAAA,IACxD;AAAA,EACF;AACF;AAOO,IAAM,YAAN,MAAgB;AAAA,EACrB,YAA6B,OAAyB,CAAC,GAAG;AAA7B;AAAA,EAA8B;AAAA,EAA9B;AAAA,EAE7B,KAAK,KAAqB,KAAgD;AACxE,WAAO,QAAQ,KAAK,EAAE,GAAG,KAAK,MAAM,GAAG,IAAI,CAAC;AAAA,EAC9C;AAAA,EAEA,SACE,KACA,KAC8C;AAC9C,WAAO,YAAe,KAAK,EAAE,GAAG,KAAK,MAAM,GAAG,IAAI,CAAC;AAAA,EACrD;AACF;","names":[]}