@tangle-network/agent-eval 0.123.1 → 0.123.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +29 -0
- package/README.md +191 -157
- package/dist/analyst/index.d.ts +27 -4
- package/dist/analyst/index.js +6 -6
- package/dist/authenticity/index.js +3 -2
- package/dist/authenticity/index.js.map +1 -1
- package/dist/benchmarks/index.d.ts +12 -4
- package/dist/benchmarks/index.js +9 -9
- package/dist/campaign/index.d.ts +270 -254
- package/dist/campaign/index.js +22 -18
- package/dist/{chunk-GS3FJGUF.js → chunk-22VO7T2I.js} +805 -366
- package/dist/chunk-22VO7T2I.js.map +1 -0
- package/dist/{chunk-LBAHQOBI.js → chunk-4KO6D2BO.js} +38 -12
- package/dist/chunk-4KO6D2BO.js.map +1 -0
- package/dist/{chunk-3FCG7FBV.js → chunk-D5JZ7UDZ.js} +2 -2
- package/dist/{chunk-XJYR7XFV.js → chunk-GC4ATIKK.js} +1 -1
- package/dist/chunk-GC4ATIKK.js.map +1 -0
- package/dist/{chunk-BGVTIE2C.js → chunk-GRCDRKII.js} +13 -2
- package/dist/chunk-GRCDRKII.js.map +1 -0
- package/dist/{chunk-FHFTYX2Q.js → chunk-JKDNAOF5.js} +2 -2
- package/dist/{chunk-S3UZOQ5Y.js → chunk-LOW3U7JZ.js} +2 -2
- package/dist/{chunk-FC5NDO3E.js → chunk-MHPEGJHC.js} +4 -4
- package/dist/{chunk-NJC7U437.js → chunk-MK3Q6GCL.js} +33 -18
- package/dist/chunk-MK3Q6GCL.js.map +1 -0
- package/dist/{chunk-U5CHZ5M3.js → chunk-N7ZS6PEF.js} +4 -4
- package/dist/{chunk-A5S77LSE.js → chunk-QFQZ3U3X.js} +2 -2
- package/dist/chunk-QFQZ3U3X.js.map +1 -0
- package/dist/{chunk-G2GPNLSX.js → chunk-QNVBALEX.js} +4 -4
- package/dist/{chunk-OYZAPX5G.js → chunk-R226UZOI.js} +2 -2
- package/dist/{chunk-6WX7CBAR.js → chunk-SJT4OBVL.js} +20 -9
- package/dist/chunk-SJT4OBVL.js.map +1 -0
- package/dist/{chunk-HZJF4IUO.js → chunk-SUN7QLPB.js} +6 -6
- package/dist/{chunk-HZJF4IUO.js.map → chunk-SUN7QLPB.js.map} +1 -1
- package/dist/{chunk-VJ7T5WIO.js → chunk-WP3GEXKI.js} +3 -3
- package/dist/cli.js +101 -11
- package/dist/cli.js.map +1 -1
- package/dist/contract/index.d.ts +115 -8
- package/dist/contract/index.js +11 -9
- package/dist/contract/index.js.map +1 -1
- package/dist/control.js +3 -3
- package/dist/fuzz.d.ts +8 -1
- package/dist/fuzz.js +7 -6
- package/dist/fuzz.js.map +1 -1
- package/dist/hosted/index.d.ts +8 -2
- package/dist/index.d.ts +36 -10
- package/dist/index.js +18 -16
- package/dist/index.js.map +1 -1
- package/dist/multishot/index.d.ts +2 -2
- package/dist/openapi.json +1 -1
- package/dist/rl.d.ts +24 -3
- package/dist/rl.js +6 -4
- package/dist/rl.js.map +1 -1
- package/dist/{run-campaign-ZKR5MQMQ.js → run-campaign-I3JXKVAK.js} +3 -3
- package/dist/storyboard/index.js +1 -1
- package/dist/storyboard/index.js.map +1 -1
- package/dist/traces.d.ts +2 -2
- package/dist/traces.js +3 -3
- package/dist/wire/index.d.ts +77 -5
- package/dist/wire/index.js +3 -3
- package/docs/adapters-observability.md +6 -6
- package/docs/building-doctrine.md +5 -5
- package/docs/campaign-proposers.md +142 -131
- package/docs/concepts.md +49 -53
- package/docs/customer-journeys.md +80 -155
- package/docs/design/loop-taxonomy.md +27 -29
- package/docs/design.md +69 -0
- package/docs/distributed-driver.md +14 -14
- package/docs/eval-surface-map.md +11 -11
- package/docs/hosted-ingest-spec.md +4 -4
- package/docs/insight-report.md +32 -27
- package/docs/multi-shot-optimization.md +41 -49
- package/docs/research-report-methodology.md +9 -9
- package/docs/trace-analysis.md +2 -2
- package/docs/wire-protocol.md +16 -16
- package/package.json +2 -1
- package/dist/chunk-6WX7CBAR.js.map +0 -1
- package/dist/chunk-A5S77LSE.js.map +0 -1
- package/dist/chunk-BGVTIE2C.js.map +0 -1
- package/dist/chunk-GS3FJGUF.js.map +0 -1
- package/dist/chunk-LBAHQOBI.js.map +0 -1
- package/dist/chunk-NJC7U437.js.map +0 -1
- package/dist/chunk-XJYR7XFV.js.map +0 -1
- package/docs/auto-research-loop-end-to-end.md +0 -186
- package/docs/improvement-glossary.md +0 -204
- package/docs/self-improvement-map.md +0 -115
- /package/dist/{chunk-3FCG7FBV.js.map → chunk-D5JZ7UDZ.js.map} +0 -0
- /package/dist/{chunk-FHFTYX2Q.js.map → chunk-JKDNAOF5.js.map} +0 -0
- /package/dist/{chunk-S3UZOQ5Y.js.map → chunk-LOW3U7JZ.js.map} +0 -0
- /package/dist/{chunk-FC5NDO3E.js.map → chunk-MHPEGJHC.js.map} +0 -0
- /package/dist/{chunk-U5CHZ5M3.js.map → chunk-N7ZS6PEF.js.map} +0 -0
- /package/dist/{chunk-G2GPNLSX.js.map → chunk-QNVBALEX.js.map} +0 -0
- /package/dist/{chunk-OYZAPX5G.js.map → chunk-R226UZOI.js.map} +0 -0
- /package/dist/{chunk-VJ7T5WIO.js.map → chunk-WP3GEXKI.js.map} +0 -0
- /package/dist/{run-campaign-ZKR5MQMQ.js.map → run-campaign-I3JXKVAK.js.map} +0 -0
|
@@ -1 +0,0 @@
|
|
|
1
|
-
{"version":3,"sources":["../src/wire/schemas.ts","../src/wire/rubrics.ts","../src/wire/handlers.ts","../src/wire/openapi.ts","../src/wire/rpc.ts","../src/wire/server.ts"],"sourcesContent":["/**\n * Wire-protocol schemas.\n *\n * These Zod schemas are the contract between the agent-eval runtime and\n * any non-TypeScript client (Python, Rust, Go, …). They get rendered to\n * OpenAPI by `wire/openapi.ts` and code-generators consume that spec to\n * produce typed clients in other languages.\n *\n * Rule: if it's not in this file, it isn't on the wire. Keep names and\n * shapes self-explanatory — every field has a `.describe()` so the\n * generated docs are useful without reading the source.\n */\nimport { extendZodWithOpenApi } from '@asteasolutions/zod-to-openapi'\nimport { z } from 'zod'\n\nextendZodWithOpenApi(z)\n\n// ── Building blocks ─────────────────────────────────────────────────\n\nexport const RubricDimensionSchema = z\n .object({\n id: z\n .string()\n .min(1)\n .describe('Short stable id like \"buyer_quality\" — used as the key in scoring output.'),\n description: z\n .string()\n .min(1)\n .describe('One-line plain-English meaning. Read by humans reviewing low scores.'),\n weight: z\n .number()\n .min(0)\n .default(1)\n .describe('Relative weight in the composite score. Default 1; 0 disables.'),\n min: z.number().default(0).describe('Lower bound of valid score for this dimension.'),\n max: z.number().default(1).describe('Upper bound of valid score for this dimension.'),\n })\n .openapi('RubricDimension')\n\nexport const FailureModeSchema = z\n .object({\n id: z.string().min(1).describe('Short stable id like \"ai-cadence\" — used in detection lists.'),\n description: z.string().min(1).describe('Plain-English description of the failure pattern.'),\n })\n .openapi('FailureMode')\n\n// ── Rubric ──────────────────────────────────────────────────────────\n\nexport const RubricSchema = z\n .object({\n name: z\n .string()\n .min(1)\n .describe('Stable name like \"anti-slop\" — used by clients to invoke this rubric.'),\n description: z\n .string()\n .min(1)\n .describe('What this rubric measures. Shown in /v1/rubrics listing.'),\n systemPrompt: z\n .string()\n .min(1)\n .describe(\n 'Instructs the judging LLM. Should explain the persona (e.g. \"senior engineer reviewing voice\"), what to score on, and what to return.',\n ),\n dimensions: z\n .array(RubricDimensionSchema)\n .min(1)\n .describe('Scoring axes. The composite score is a weighted sum of these.'),\n failureModes: z\n .array(FailureModeSchema)\n .default([])\n .describe('Patterns to detect; each detected mode appears in the result.failureModes list.'),\n wins: z\n .array(FailureModeSchema)\n .default([])\n .describe('Positive patterns; each detected one appears in the result.wins list.'),\n })\n .openapi('Rubric')\n\n// ── Judge call ──────────────────────────────────────────────────────\n\nexport const JudgeRequestSchema = z\n .object({\n rubricName: z\n .string()\n .optional()\n .describe('Use a built-in rubric by name. Mutually exclusive with `rubric`.'),\n rubric: RubricSchema.optional().describe(\n 'Inline rubric definition. Mutually exclusive with `rubricName`.',\n ),\n content: z\n .string()\n .min(1)\n .describe('The text being judged — a tweet, a blog post, a code snippet, anything stringly.'),\n context: z\n .record(z.string(), z.unknown())\n .optional()\n .describe(\n 'Free-form metadata for the rubric to use — analytics, source URL, author, etc. Surfaced to the LLM.',\n ),\n model: z\n .string()\n .optional()\n .describe('Override the judge model (default routes via tcloud). e.g. \"claude-opus-4-7\".'),\n })\n .refine((v) => Boolean(v.rubricName) !== Boolean(v.rubric), {\n message: 'Provide exactly one of `rubricName` or `rubric`.',\n })\n .openapi('JudgeRequest')\n\nexport const JudgeResultSchema = z\n .object({\n composite: z\n .number()\n .min(0)\n .max(1)\n .describe('Weighted combination of dimension scores in 0..1. The single number to gate on.'),\n dimensions: z\n .record(z.string(), z.number())\n .describe('Per-dimension score, keyed by RubricDimension.id.'),\n failureModes: z\n .array(z.string())\n .default([])\n .describe('Failure-mode ids detected in the content (subset of rubric.failureModes ids).'),\n wins: z\n .array(z.string())\n .default([])\n .describe('Win ids detected in the content (subset of rubric.wins ids).'),\n rationale: z\n .string()\n .describe('Plain-English explanation of the score. Surfaced to the human reviewer.'),\n rubricVersion: z\n .string()\n .describe(\n 'Stable hash of the rubric used. Scores are only comparable across runs when this matches.',\n ),\n model: z.string().describe('Model that produced the judgement, for reproducibility.'),\n durationMs: z.number().int().nonnegative().describe('End-to-end wall time for this call.'),\n })\n .openapi('JudgeResult')\n\n// ── Rubric listing ──────────────────────────────────────────────────\n\nexport const RubricInfoSchema = z\n .object({\n name: z.string().describe('Pass this to /v1/judge as `rubricName`.'),\n description: z.string().describe('What this rubric measures.'),\n dimensions: z\n .array(z.object({ id: z.string(), description: z.string(), weight: z.number() }))\n .describe('The scoring axes this rubric uses, with weights.'),\n failureModes: z.array(z.string()).default([]).describe('Failure-mode ids this rubric detects.'),\n rubricVersion: z.string().describe('Stable hash — match this to compare scores across runs.'),\n })\n .openapi('RubricInfo')\n\nexport const ListRubricsResponseSchema = z\n .object({\n rubrics: z.array(RubricInfoSchema),\n })\n .openapi('ListRubricsResponse')\n\n// ── Version / health ────────────────────────────────────────────────\n\nexport const VersionResponseSchema = z\n .object({\n package: z.string().describe('Package name (always \"@tangle-network/agent-eval\").'),\n version: z.string().describe('Semver of the running server. Match your client to this.'),\n wireVersion: z\n .string()\n .describe(\n 'Wire-protocol semver. Bumps separately from package version when the schema changes.',\n ),\n apiSurface: z.array(z.string()).describe('List of supported method names.'),\n })\n .openapi('VersionResponse')\n\nexport const HealthResponseSchema = z\n .object({\n status: z.literal('ok'),\n uptimeSec: z.number(),\n })\n .openapi('HealthResponse')\n\n// ── Ingestion: production traces + user feedback ────────────────────\n\n/**\n * Minimal `TraceEvent` shape that the production runtime emits.\n * Matches `trace/schema.ts` `TraceEvent` but is duplicated here as a\n * wire schema so non-TypeScript clients can validate without depending\n * on internal types.\n */\nexport const TraceEventSchema = z\n .object({\n eventId: z.string().min(1).describe('Stable id for the event. Use ULID or UUID.'),\n runId: z.string().min(1).describe('Run this event belongs to.'),\n spanId: z.string().optional().describe('Span that emitted the event, if any.'),\n kind: z\n .enum([\n 'log',\n 'error',\n 'budget_decrement',\n 'budget_breach',\n 'state_mutation',\n 'policy_violation',\n 'redaction_applied',\n 'custom',\n ])\n .describe('Coarse event category — matches the TraceSchema v1 EventKind enum.'),\n timestamp: z\n .number()\n .int()\n .nonnegative()\n .describe('Unix millis. Must be monotonically non-decreasing within a span.'),\n payload: z\n .record(z.string(), z.unknown())\n .describe('Free-form payload — the runtime owns the shape.'),\n })\n .openapi('TraceEvent')\n\nexport const TracesIngestRequestSchema = z\n .object({\n events: z\n .array(TraceEventSchema)\n .min(1)\n .max(10_000)\n .describe('Batch of events. Max 10k per call — bigger streams should be chunked.'),\n })\n .openapi('TracesIngestRequest')\n\nexport const TracesIngestResponseSchema = z\n .object({\n accepted: z.number().int().nonnegative().describe('Number of events persisted.'),\n rejected: z\n .number()\n .int()\n .nonnegative()\n .describe('Number of events the store refused — see `errors[]` for reasons.'),\n errors: z\n .array(\n z.object({\n eventId: z.string().describe('Event id this error applies to.'),\n message: z.string().describe('Why the event was rejected.'),\n }),\n )\n .default([]),\n })\n .openapi('TracesIngestResponse')\n\nexport const FeedbackLabelSchema = z\n .object({\n id: z.string().optional(),\n source: z.enum(['user', 'judge', 'environment', 'metric', 'policy', 'system']),\n kind: z.enum([\n 'approve',\n 'reject',\n 'select',\n 'edit',\n 'rank',\n 'rate',\n 'comment',\n 'metric_outcome',\n 'policy_block',\n 'revision_request',\n ]),\n value: z.unknown(),\n reason: z.string().optional(),\n severity: z.enum(['info', 'warning', 'error', 'critical']).optional(),\n createdAt: z.string().describe('ISO-8601 UTC.'),\n metadata: z.record(z.string(), z.unknown()).optional(),\n })\n .openapi('FeedbackLabel')\n\nexport const FeedbackAttemptSchema = z\n .object({\n id: z.string().min(1),\n stepIndex: z.number().int().nonnegative(),\n artifactType: z.enum([\n 'text',\n 'code',\n 'plan',\n 'research',\n 'action',\n 'ui',\n 'decision',\n 'data',\n 'other',\n ]),\n artifact: z.unknown(),\n options: z.array(z.unknown()).optional(),\n proposedAction: z\n .object({\n type: z.string(),\n risk: z.enum(['low', 'medium', 'high']).optional(),\n costUsd: z.number().optional(),\n externalSideEffect: z.boolean().optional(),\n requiresApproval: z.boolean().optional(),\n metadata: z.record(z.string(), z.unknown()).optional(),\n })\n .optional(),\n feedback: z.array(FeedbackLabelSchema).optional(),\n createdAt: z.string(),\n metadata: z.record(z.string(), z.unknown()).optional(),\n })\n .openapi('FeedbackAttempt')\n\nexport const FeedbackTrajectorySchema = z\n .object({\n id: z.string().min(1).describe('Stable id; idempotency key for the trajectory.'),\n projectId: z.string().optional(),\n scenarioId: z.string().optional(),\n task: z.object({\n intent: z.string().min(1),\n context: z.unknown().optional(),\n }),\n attempts: z.array(FeedbackAttemptSchema).default([]),\n labels: z.array(FeedbackLabelSchema).default([]),\n outcome: z\n .object({\n success: z.boolean().optional(),\n score: z.number().optional(),\n metrics: z.record(z.string(), z.number()).optional(),\n costUsd: z.number().optional(),\n detail: z.string().optional(),\n observedAt: z.string().optional(),\n metadata: z.record(z.string(), z.unknown()).optional(),\n })\n .optional(),\n split: z.enum(['train', 'dev', 'test', 'holdout']).optional(),\n tags: z.record(z.string(), z.string()).optional(),\n createdAt: z.string().describe('ISO-8601 UTC.'),\n updatedAt: z.string().optional(),\n metadata: z.record(z.string(), z.unknown()).optional(),\n })\n .openapi('FeedbackTrajectory')\n\nexport const FeedbackIngestResponseSchema = z\n .object({\n id: z.string().describe('Trajectory id that was persisted.'),\n persisted: z.boolean().describe('True when the trajectory was saved (idempotent on id).'),\n })\n .openapi('FeedbackIngestResponse')\n\nexport type TraceEvent = z.infer<typeof TraceEventSchema>\nexport type TracesIngestRequest = z.infer<typeof TracesIngestRequestSchema>\nexport type TracesIngestResponse = z.infer<typeof TracesIngestResponseSchema>\nexport type FeedbackTrajectory = z.infer<typeof FeedbackTrajectorySchema>\nexport type FeedbackIngestResponse = z.infer<typeof FeedbackIngestResponseSchema>\n\n// ── Errors ──────────────────────────────────────────────────────────\n\nexport const ErrorResponseSchema = z\n .object({\n error: z\n .object({\n code: z\n .string()\n .describe(\n 'Machine-readable code: \"validation_error\", \"rubric_not_found\", \"judge_error\".',\n ),\n message: z.string().describe('Human-readable message.'),\n details: z.unknown().optional().describe('Optional structured detail.'),\n })\n .describe('Errors are always wrapped in this shape across all endpoints.'),\n })\n .openapi('ErrorResponse')\n\n// ── Type exports for callers in the same package ────────────────────\n\nexport type RubricDimension = z.infer<typeof RubricDimensionSchema>\nexport type FailureMode = z.infer<typeof FailureModeSchema>\nexport type Rubric = z.infer<typeof RubricSchema>\nexport type JudgeRequest = z.infer<typeof JudgeRequestSchema>\nexport type JudgeResult = z.infer<typeof JudgeResultSchema>\nexport type RubricInfo = z.infer<typeof RubricInfoSchema>\nexport type ListRubricsResponse = z.infer<typeof ListRubricsResponseSchema>\nexport type VersionResponse = z.infer<typeof VersionResponseSchema>\nexport type ErrorResponse = z.infer<typeof ErrorResponseSchema>\n\n// ── Wire-protocol version ───────────────────────────────────────────\n\n/**\n * Bump on any breaking change to a request/response schema.\n * Non-breaking (additive) changes don't require a bump.\n */\nexport const WIRE_VERSION = '1.0.0'\n\n/**\n * Stable hash of a rubric. Used to make scores comparable across runs:\n * if the rubricVersion matches, the rubric was identical.\n */\nexport function hashRubric(rubric: Rubric): string {\n const stable = stableStringify(rubric)\n let h = 5381\n for (let i = 0; i < stable.length; i++) {\n h = (h * 33) ^ stable.charCodeAt(i)\n }\n // Unsigned 32-bit hex, prefixed with rubric name + version slot\n return `${rubric.name}@${(h >>> 0).toString(16).padStart(8, '0')}`\n}\n\nfunction stableStringify(value: unknown): string {\n if (Array.isArray(value)) return `[${value.map((item) => stableStringify(item)).join(',')}]`\n if (value && typeof value === 'object') {\n const entries = Object.entries(value as Record<string, unknown>)\n .sort(([a], [b]) => a.localeCompare(b))\n .map(([key, item]) => `${JSON.stringify(key)}:${stableStringify(item)}`)\n return `{${entries.join(',')}}`\n }\n return JSON.stringify(value)\n}\n","/**\n * Built-in rubrics shipped with agent-eval.\n *\n * A rubric is a set of scoring axes plus a system prompt that tells the\n * judging LLM how to grade against those axes. Built-in rubrics are\n * curated for use cases that recur across Tangle projects — call them\n * by name from any client.\n *\n * Adding a rubric:\n * 1. Define the Rubric object below with a clear `description` and\n * named `dimensions`.\n * 2. Register it in `BUILTIN_RUBRICS` at the bottom.\n * 3. Add a test in `tests/wire/rubrics.test.ts`.\n *\n * Custom rubrics: callers pass `rubric` inline to /v1/judge instead of\n * `rubricName` — see schemas.ts.\n */\nimport type { Rubric } from './schemas'\nimport { hashRubric } from './schemas'\n\n// ── anti-slop ───────────────────────────────────────────────────────\n// Voice/style judge tuned for technical-buyer audiences. Used by the\n// Postiz autoresearch loop and any content-quality gate.\n\nconst ANTI_SLOP: Rubric = {\n name: 'anti-slop',\n description:\n 'Voice and signal quality for content aimed at senior engineers. Catches AI cadence, marketing tone, and engagement-bait shapes.',\n systemPrompt: `You are evaluating a piece of content written for senior engineers and technical founders.\n\nYou score three things:\n- buyer_quality (0..1): would a senior engineer in the target ICP find this worth their attention? High = specific, earned, technically interesting. Low = generic, hyped, off-target.\n- voice (0..1): does it read like a person who built the thing, or like AI/marketing copy?\n- signal (0..1): does it contain a non-obvious detail, constraint, or claim a reader couldn't get from the public docs?\n\nDetect failure modes (return ids matching):\n- ai-cadence: rule-of-three openings, em-dash flourish, \"Let me explain\", \"Here's the thing\", AI rhythm\n- marketing-tone: \"We're excited to announce\", \"thrilled\", \"delighted\", \"game-changer\", buzzword stack\n- vague-claim: technical claim without a specific component, file, or measurement\n- no-hook: opening doesn't earn attention from the target reader\n- engagement-bait: \"agree?\", \"thoughts?\", listicles, controversy-fishing, hook-detail-pitch\n- off-icp: content shape would attract motivational/grift/hype audiences instead of buyers\n- stale-claim: repeats a positioning line we've used many times this month\n\nDetect wins (return ids matching):\n- specific-component: names a real file, component, or measurement\n- earned-detail: shares a non-obvious detail not derivable from public docs\n- constraint-articulated: names a real tradeoff and the side chosen\n- honest-failure: describes a real failure mode and what was done about it\n\nReturn ONLY JSON matching the response schema. Be conservative — most content has 0-1 wins and 1-2 failure modes, not many of each.`,\n dimensions: [\n {\n id: 'buyer_quality',\n description: 'Would the target buyer find this worth attention?',\n weight: 0.5,\n min: 0,\n max: 1,\n },\n {\n id: 'voice',\n description: 'Does it sound like a builder, not AI or marketing?',\n weight: 0.3,\n min: 0,\n max: 1,\n },\n {\n id: 'signal',\n description: 'Non-obvious detail, constraint, or claim?',\n weight: 0.2,\n min: 0,\n max: 1,\n },\n ],\n failureModes: [\n { id: 'ai-cadence', description: 'AI-rhythm openings and transitions' },\n { id: 'marketing-tone', description: 'Buzzwords, hype, corporate-PR voice' },\n { id: 'vague-claim', description: 'Technical claim without specifics' },\n { id: 'no-hook', description: 'Opening fails to earn attention' },\n { id: 'engagement-bait', description: 'Listicle/controversy/agree-pattern' },\n { id: 'off-icp', description: 'Voice attracts the wrong audience' },\n { id: 'stale-claim', description: 'Reuses an over-used positioning line' },\n ],\n wins: [\n { id: 'specific-component', description: 'Names a real file/component/number' },\n { id: 'earned-detail', description: 'Detail not in public docs' },\n { id: 'constraint-articulated', description: 'Names a real tradeoff' },\n { id: 'honest-failure', description: 'Describes a real failure honestly' },\n ],\n}\n\n// ── Registry ────────────────────────────────────────────────────────\n\nexport const BUILTIN_RUBRICS: Record<string, Rubric> = {\n 'anti-slop': ANTI_SLOP,\n}\n\n/** Get a built-in rubric by name, or undefined. */\nexport function getBuiltinRubric(name: string): Rubric | undefined {\n return BUILTIN_RUBRICS[name]\n}\n\n/** List built-in rubrics with their stable versions. */\nexport function listBuiltinRubrics() {\n return Object.values(BUILTIN_RUBRICS).map((r) => ({\n name: r.name,\n description: r.description,\n dimensions: r.dimensions.map((d) => ({\n id: d.id,\n description: d.description,\n weight: d.weight,\n })),\n failureModes: r.failureModes.map((f) => f.id),\n rubricVersion: hashRubric(r),\n }))\n}\n","/**\n * Pure handler functions — the \"business logic\" behind every wire-protocol\n * method. The HTTP server (`server.ts`) and the stdio RPC (`rpc.ts`) both\n * call these. Tests call these directly without spinning a server.\n *\n * Each handler:\n * - Takes a parsed request (already Zod-validated by the transport).\n * - Returns a result that matches the response schema.\n * - Throws `WireError` for caller-fixable errors (404, 400, 422).\n * - Lets unexpected errors bubble — the transport maps them to 500.\n */\n\nimport { CostLedger, type CostLedgerHandle } from '../cost-ledger'\nimport type { FeedbackTrajectoryStore } from '../feedback-trajectory'\nimport {\n callLlmJson,\n costReceiptFromLlm,\n costReceiptFromLlmError,\n type LlmCallRequest,\n type LlmClientOptions,\n maximumChargeForLlmRequest,\n} from '../llm-client'\nimport type { TraceEvent as InternalTraceEvent } from '../trace/schema'\nimport type { TraceStore } from '../trace/store'\nimport { getBuiltinRubric, listBuiltinRubrics } from './rubrics'\nimport {\n type FeedbackIngestResponse,\n hashRubric,\n type JudgeRequest,\n type JudgeResult,\n type ListRubricsResponse,\n type Rubric,\n type TracesIngestRequest,\n type TracesIngestResponse,\n type VersionResponse,\n WIRE_VERSION,\n type FeedbackTrajectory as WireFeedbackTrajectory,\n} from './schemas'\n\n/** Caller-fixable error. The transport renders this to 4xx + ErrorResponse. */\nexport class WireError extends Error {\n constructor(\n public readonly code: string,\n message: string,\n public readonly status: number = 400,\n public readonly details?: unknown,\n ) {\n super(message)\n this.name = 'WireError'\n }\n}\n\n// ── judge ───────────────────────────────────────────────────────────\n\n/** The JSON schema we ask the judging LLM to fill in. */\nfunction judgeOutputSchema(rubric: Rubric) {\n return {\n name: 'JudgeOutput',\n schema: {\n type: 'object',\n additionalProperties: false,\n properties: {\n dimensions: {\n type: 'object',\n additionalProperties: false,\n properties: Object.fromEntries(\n rubric.dimensions.map((d) => [\n d.id,\n { type: 'number', minimum: d.min, maximum: d.max },\n ]),\n ),\n required: rubric.dimensions.map((d) => d.id),\n },\n failureModes: {\n type: 'array',\n items: { type: 'string', enum: rubric.failureModes.map((f) => f.id) },\n },\n wins: {\n type: 'array',\n items: { type: 'string', enum: rubric.wins.map((w) => w.id) },\n },\n rationale: { type: 'string' },\n },\n required: ['dimensions', 'rationale'],\n } as Record<string, unknown>,\n }\n}\n\ninterface JudgeOutput {\n dimensions: Record<string, number>\n failureModes?: string[]\n wins?: string[]\n rationale: string\n}\n\nfunction validateJudgeOutput(value: unknown, rubric: Rubric): JudgeOutput {\n if (!value || typeof value !== 'object') {\n throw new WireError('judge_error', 'Judge returned malformed output.', 500, value)\n }\n const raw = value as Record<string, unknown>\n const rawDimensions = raw.dimensions\n if (!rawDimensions || typeof rawDimensions !== 'object' || Array.isArray(rawDimensions)) {\n throw new WireError('judge_error', 'Judge returned malformed dimensions.', 500, value)\n }\n\n const dimensions: Record<string, number> = {}\n const dimensionRecord = rawDimensions as Record<string, unknown>\n for (const dim of rubric.dimensions) {\n const score = dimensionRecord[dim.id]\n if (\n typeof score !== 'number' ||\n !Number.isFinite(score) ||\n score < dim.min ||\n score > dim.max\n ) {\n throw new WireError(\n 'judge_error',\n `Judge returned invalid score for dimension \"${dim.id}\".`,\n 500,\n value,\n )\n }\n dimensions[dim.id] = score\n }\n\n const allowedFailures = new Set(rubric.failureModes.map((mode) => mode.id))\n const allowedWins = new Set(rubric.wins.map((win) => win.id))\n const failureModes = validateIdArray(raw.failureModes, allowedFailures, 'failureModes', value)\n const wins = validateIdArray(raw.wins, allowedWins, 'wins', value)\n if (typeof raw.rationale !== 'string' || raw.rationale.trim().length === 0) {\n throw new WireError('judge_error', 'Judge returned missing rationale.', 500, value)\n }\n\n return { dimensions, failureModes, wins, rationale: raw.rationale }\n}\n\nfunction validateIdArray(\n raw: unknown,\n allowed: Set<string>,\n field: 'failureModes' | 'wins',\n original: unknown,\n): string[] {\n if (raw === undefined) return []\n if (!Array.isArray(raw)) {\n throw new WireError('judge_error', `Judge returned non-array ${field}.`, 500, original)\n }\n const out: string[] = []\n for (const item of raw) {\n if (typeof item !== 'string' || !allowed.has(item)) {\n throw new WireError(\n 'judge_error',\n `Judge returned unknown ${field} id \"${String(item)}\".`,\n 500,\n original,\n )\n }\n out.push(item)\n }\n return out\n}\n\nfunction compositeScore(dimensions: Record<string, number>, rubric: Rubric): number {\n let weighted = 0\n let totalWeight = 0\n for (const dim of rubric.dimensions) {\n const raw = dimensions[dim.id] ?? 0\n const range = dim.max - dim.min || 1\n const normalized = Math.max(0, Math.min(1, (raw - dim.min) / range))\n weighted += normalized * dim.weight\n totalWeight += dim.weight\n }\n return totalWeight > 0 ? weighted / totalWeight : 0\n}\n\nfunction buildJudgePrompt(content: string, context: unknown): string {\n const ctx = context && Object.keys(context as object).length ? JSON.stringify(context) : ''\n return [\n `CONTENT TO JUDGE:`,\n content,\n '',\n ctx ? `CONTEXT (metadata, analytics, etc.):` : '',\n ctx ? ctx : '',\n ]\n .filter(Boolean)\n .join('\\n')\n}\n\nconst DEFAULT_JUDGE_MODEL = 'claude-sonnet-4-6'\n\nexport interface HandleJudgeOptions {\n costLedger?: CostLedgerHandle\n costPhase?: string\n llm?: LlmClientOptions\n signal?: AbortSignal\n}\n\nexport async function handleJudge(\n req: JudgeRequest,\n options: HandleJudgeOptions = {},\n): Promise<JudgeResult> {\n // Resolve rubric\n let rubric: Rubric\n if (req.rubricName) {\n const found = getBuiltinRubric(req.rubricName)\n if (!found) {\n throw new WireError('rubric_not_found', `No built-in rubric named \"${req.rubricName}\".`, 404)\n }\n rubric = found\n } else if (req.rubric) {\n rubric = req.rubric\n } else {\n // refine() in the schema should already have caught this — defense in depth\n throw new WireError('validation_error', 'Provide either `rubricName` or `rubric`.', 422)\n }\n\n const startedAt = Date.now()\n const model = req.model ?? DEFAULT_JUDGE_MODEL\n\n const request = {\n model,\n messages: [\n { role: 'system', content: rubric.systemPrompt },\n { role: 'user', content: buildJudgePrompt(req.content, req.context) },\n ],\n jsonSchema: judgeOutputSchema(rubric),\n temperature: 0.0,\n maxTokens: 4_000,\n timeoutMs: 60_000,\n } satisfies LlmCallRequest\n const ledger = options.costLedger ?? new CostLedger()\n const paid = await ledger.runPaidCall({\n channel: 'judge',\n phase: options.costPhase ?? 'wire.judge',\n actor: `wire.${req.rubricName ?? 'inline'}`,\n model,\n maximumCharge: maximumChargeForLlmRequest(request, options.llm),\n signal: options.signal,\n execute: (signal, callId) =>\n callLlmJson<JudgeOutput>(request, {\n ...options.llm,\n signal,\n idempotencyKey: callId,\n }),\n receipt: ({ result }) => costReceiptFromLlm(result),\n receiptFromError: costReceiptFromLlmError,\n })\n if (!paid.succeeded) throw paid.error\n const { value, result } = paid.value\n\n const output = validateJudgeOutput(value, rubric)\n\n const composite = compositeScore(output.dimensions, rubric)\n const durationMs = Date.now() - startedAt\n\n return {\n composite,\n dimensions: output.dimensions,\n failureModes: output.failureModes ?? [],\n wins: output.wins ?? [],\n rationale: output.rationale,\n rubricVersion: hashRubric(rubric),\n model: result.model,\n durationMs,\n }\n}\n\n// ── listRubrics ─────────────────────────────────────────────────────\n\nexport function handleListRubrics(): ListRubricsResponse {\n return { rubrics: listBuiltinRubrics() }\n}\n\n// ── version ─────────────────────────────────────────────────────────\n\nimport { readFileSync } from 'node:fs'\nimport { dirname, resolve } from 'node:path'\nimport { fileURLToPath } from 'node:url'\n\nlet CACHED_VERSION: string | undefined\n\nfunction readPackageVersion(): string {\n if (CACHED_VERSION) return CACHED_VERSION\n // Walk up from this file looking for the nearest package.json.\n // In dist/ this is dist/.., in src/wire/ this is ../../package.json.\n const here = dirname(fileURLToPath(import.meta.url))\n const candidates = [\n resolve(here, '..', '..', 'package.json'), // src/wire → repo root\n resolve(here, '..', 'package.json'), // dist → repo root\n ]\n for (const path of candidates) {\n try {\n const pkg = JSON.parse(readFileSync(path, 'utf-8')) as { version?: string }\n if (pkg.version) {\n CACHED_VERSION = pkg.version\n return pkg.version\n }\n } catch {\n // try next\n }\n }\n return '0.0.0-unknown'\n}\n\nexport function handleVersion(): VersionResponse {\n return {\n package: '@tangle-network/agent-eval',\n version: readPackageVersion(),\n wireVersion: WIRE_VERSION,\n apiSurface: ['judge', 'listRubrics', 'version', 'feedback.ingest', 'traces.ingest'],\n }\n}\n\n// ── Ingestion handlers ───────────────────────────────────────────────\n\n/**\n * Pluggable stores the wire layer routes ingestion writes into. Both\n * are optional — when omitted, the corresponding endpoint returns 503.\n *\n * Production deployments wire a `FileSystemTraceStore` and\n * `FileSystemFeedbackTrajectoryStore` here. Tests substitute in-memory\n * stores.\n */\nexport interface IngestionStores {\n traceStore?: TraceStore\n feedbackStore?: FeedbackTrajectoryStore\n}\n\n/**\n * `POST /v1/traces/ingest` — accept a batch of `TraceEvent`s from the\n * production runtime. Best-effort: each event is appended independently;\n * one bad event does not poison the batch.\n *\n * Idempotency: the underlying store is append-only; consumers retrying\n * the same payload will get duplicate events. Consumers should\n * de-duplicate by `eventId` downstream — production traces frequently\n * land via at-least-once buses (Kafka, SQS) where dedup is unavoidable.\n */\nexport async function handleTracesIngest(\n req: TracesIngestRequest,\n stores: IngestionStores,\n): Promise<TracesIngestResponse> {\n if (!stores.traceStore) {\n throw new WireError(\n 'service_unavailable',\n 'No trace store configured on this server. Pass `traceStore` to `createApp`.',\n 503,\n )\n }\n const errors: Array<{ eventId: string; message: string }> = []\n let accepted = 0\n for (const event of req.events) {\n try {\n // The wire `TraceEvent` is structurally identical to the internal one.\n await stores.traceStore.appendEvent(event as InternalTraceEvent)\n accepted++\n } catch (err) {\n errors.push({\n eventId: event.eventId,\n message: err instanceof Error ? err.message : String(err),\n })\n }\n }\n return { accepted, rejected: errors.length, errors }\n}\n\n/**\n * `POST /v1/feedback` — accept a single `FeedbackTrajectory` from the\n * production runtime. Idempotent on `id`: re-posting the same trajectory\n * replaces the prior record.\n */\nexport async function handleFeedbackIngest(\n req: WireFeedbackTrajectory,\n stores: IngestionStores,\n): Promise<FeedbackIngestResponse> {\n if (!stores.feedbackStore) {\n throw new WireError(\n 'service_unavailable',\n 'No feedback store configured on this server. Pass `feedbackStore` to `createApp`.',\n 503,\n )\n }\n // The wire `FeedbackTrajectory` aligns 1:1 with the internal type;\n // cast through `unknown` since the wire schema is a Zod-inferred\n // structural type with optional fields the internal store consumes.\n await stores.feedbackStore.save(req as unknown as Parameters<FeedbackTrajectoryStore['save']>[0])\n return { id: req.id, persisted: true }\n}\n","/**\n * Build an OpenAPI spec from the wire schemas.\n *\n * The spec is the contract that other-language clients (Python, Rust,\n * Go, …) generate from. There is no hand-written client — clients are\n * derived artifacts of this file plus `schemas.ts`.\n *\n * Run `pnpm openapi` (defined in package.json) to write the spec to\n * `dist/openapi.json`. CI uses that file to regenerate the Python\n * client and gate the dual-publish workflow.\n */\nimport { OpenAPIRegistry, OpenApiGeneratorV31 } from '@asteasolutions/zod-to-openapi'\nimport type { OpenAPIObject } from 'openapi3-ts/oas31'\n\nimport {\n ErrorResponseSchema,\n FeedbackIngestResponseSchema,\n FeedbackTrajectorySchema,\n HealthResponseSchema,\n JudgeRequestSchema,\n JudgeResultSchema,\n ListRubricsResponseSchema,\n TracesIngestRequestSchema,\n TracesIngestResponseSchema,\n VersionResponseSchema,\n WIRE_VERSION,\n} from './schemas'\n\nexport function buildOpenApi(packageVersion: string): OpenAPIObject {\n const registry = new OpenAPIRegistry()\n\n // Components — each schema becomes a $ref-able component\n registry.register('JudgeRequest', JudgeRequestSchema)\n registry.register('JudgeResult', JudgeResultSchema)\n registry.register('ListRubricsResponse', ListRubricsResponseSchema)\n registry.register('VersionResponse', VersionResponseSchema)\n registry.register('HealthResponse', HealthResponseSchema)\n registry.register('ErrorResponse', ErrorResponseSchema)\n registry.register('TracesIngestRequest', TracesIngestRequestSchema)\n registry.register('TracesIngestResponse', TracesIngestResponseSchema)\n registry.register('FeedbackTrajectory', FeedbackTrajectorySchema)\n registry.register('FeedbackIngestResponse', FeedbackIngestResponseSchema)\n\n // Routes\n registry.registerPath({\n method: 'post',\n path: '/v1/judge',\n summary: 'Score a piece of content against a rubric',\n description:\n 'Runs the judging LLM with the named (or inline) rubric and returns dimension scores, detected failure modes, wins, and a composite score in 0..1.',\n request: {\n body: {\n content: {\n 'application/json': { schema: JudgeRequestSchema },\n },\n },\n },\n responses: {\n 200: {\n description: 'Successful judgement',\n content: { 'application/json': { schema: JudgeResultSchema } },\n },\n 400: {\n description: 'Validation error',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n 404: {\n description: 'Rubric not found',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n 500: {\n description: 'Judge error',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n },\n })\n\n registry.registerPath({\n method: 'get',\n path: '/v1/rubrics',\n summary: 'List built-in rubrics',\n description:\n 'Returns every rubric registered server-side, with their dimensions and stable rubricVersion hash.',\n responses: {\n 200: {\n description: 'Listing',\n content: { 'application/json': { schema: ListRubricsResponseSchema } },\n },\n },\n })\n\n registry.registerPath({\n method: 'get',\n path: '/v1/version',\n summary: 'Server and wire-protocol version',\n description: 'Match your client version to `version`; check `wireVersion` for compatibility.',\n responses: {\n 200: {\n description: 'Version info',\n content: { 'application/json': { schema: VersionResponseSchema } },\n },\n },\n })\n\n registry.registerPath({\n method: 'get',\n path: '/healthz',\n summary: 'Liveness check',\n responses: {\n 200: {\n description: 'OK',\n content: { 'application/json': { schema: HealthResponseSchema } },\n },\n },\n })\n\n registry.registerPath({\n method: 'post',\n path: '/v1/traces/ingest',\n summary: 'Ingest a batch of production TraceEvents',\n description:\n 'Append a batch of TraceEvents to the configured TraceStore. Accepts application/json ({events:[...]}) or application/x-ndjson (one event per line). Returns counts of accepted + rejected events.',\n request: {\n body: {\n content: {\n 'application/json': { schema: TracesIngestRequestSchema },\n 'application/x-ndjson': { schema: TracesIngestRequestSchema },\n },\n },\n },\n responses: {\n 200: {\n description: 'Ingestion summary',\n content: { 'application/json': { schema: TracesIngestResponseSchema } },\n },\n 400: {\n description: 'Validation error',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n 401: {\n description: 'Unauthorized (when bearer auth is configured)',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n 503: {\n description: 'No trace store configured',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n },\n })\n\n registry.registerPath({\n method: 'post',\n path: '/v1/feedback',\n summary: 'Ingest a FeedbackTrajectory from production',\n description:\n 'Persist a single FeedbackTrajectory. Idempotent on trajectory.id — re-posting replaces the prior record. Used by production runtimes to forward user 👍/👎/edits into the eval substrate.',\n request: {\n body: {\n content: {\n 'application/json': { schema: FeedbackTrajectorySchema },\n },\n },\n },\n responses: {\n 200: {\n description: 'Persisted',\n content: { 'application/json': { schema: FeedbackIngestResponseSchema } },\n },\n 400: {\n description: 'Validation error',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n 401: {\n description: 'Unauthorized (when bearer auth is configured)',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n 503: {\n description: 'No feedback store configured',\n content: { 'application/json': { schema: ErrorResponseSchema } },\n },\n },\n })\n\n const generator = new OpenApiGeneratorV31(registry.definitions)\n const doc = generator.generateDocument({\n openapi: '3.1.0',\n info: {\n title: '@tangle-network/agent-eval — wire protocol',\n version: packageVersion,\n description: `HTTP and stdio RPC interface to agent-eval. The TypeScript runtime is the source of truth; this spec is the contract that cross-language clients (Python, Rust, Go) generate from.\n\nWire-protocol version: ${WIRE_VERSION}. Bumps on breaking changes to request/response schemas.`,\n contact: { name: 'Tangle Network', url: 'https://github.com/tangle-network/agent-eval' },\n license: { name: 'MIT' },\n },\n servers: [{ url: 'http://localhost:5005', description: 'Local agent-eval serve' }],\n })\n const rubricRef = { $ref: '#/components/schemas/Rubric' } as const\n const commonJudgeFields = {\n content: { type: 'string', minLength: 1 },\n context: { type: 'object', additionalProperties: true },\n model: { type: 'string' },\n } as const\n doc.components ??= {}\n doc.components.schemas ??= {}\n doc.components.schemas.JudgeRequest = {\n oneOf: [\n {\n type: 'object',\n additionalProperties: false,\n required: ['rubricName', 'content'],\n properties: {\n rubricName: { type: 'string', minLength: 1 },\n ...commonJudgeFields,\n },\n },\n {\n type: 'object',\n additionalProperties: false,\n required: ['rubric', 'content'],\n properties: {\n rubric: rubricRef,\n ...commonJudgeFields,\n },\n },\n ],\n description: 'Judge request. Provide exactly one of rubricName or rubric.',\n }\n return doc\n}\n","/**\n * stdio RPC transport.\n *\n * For batch / cron use without a running server. The Python client falls\n * back to this when no server is reachable.\n *\n * Protocol (line-delimited JSON over stdin/stdout):\n * IN: one JSON object on stdin: {\"method\":\"judge\",\"params\":{...}}\n * OUT: one JSON object on stdout: {\"result\":{...}} or {\"error\":{...}}\n *\n * One request per process invocation. To pipeline many calls, the client\n * writes JSONL to stdin and reads JSONL from stdout — see batch mode below.\n */\nimport { handleJudge, handleListRubrics, handleVersion, WireError } from './handlers'\nimport { JudgeRequestSchema } from './schemas'\n\ninterface RpcRequest {\n method: 'judge' | 'listRubrics' | 'version'\n params?: unknown\n}\n\ninterface RpcSuccess {\n result: unknown\n}\n\ninterface RpcError {\n error: { code: string; message: string; details?: unknown }\n}\n\nexport async function dispatchRpc(req: RpcRequest): Promise<RpcSuccess | RpcError> {\n try {\n switch (req.method) {\n case 'judge': {\n const parsed = JudgeRequestSchema.safeParse(req.params)\n if (!parsed.success) {\n return {\n error: {\n code: 'validation_error',\n message: 'params did not match JudgeRequest schema.',\n details: parsed.error.issues,\n },\n }\n }\n return { result: await handleJudge(parsed.data) }\n }\n case 'listRubrics':\n return { result: handleListRubrics() }\n case 'version':\n return { result: handleVersion() }\n default:\n return {\n error: {\n code: 'unknown_method',\n message: `No such method: ${(req as { method: string }).method}`,\n },\n }\n }\n } catch (err) {\n if (err instanceof WireError) {\n return { error: { code: err.code, message: err.message, details: err.details } }\n }\n const message = err instanceof Error ? err.message : String(err)\n return { error: { code: 'internal_error', message } }\n }\n}\n\n// ── stdin/stdout driver ─────────────────────────────────────────────\n\nasync function readAll(stream: NodeJS.ReadableStream): Promise<string> {\n const chunks: Buffer[] = []\n for await (const chunk of stream) {\n chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk as string))\n }\n return Buffer.concat(chunks).toString('utf-8')\n}\n\n/** Read one JSON request from stdin, write one JSON response to stdout. */\nexport async function runRpcOnce(method?: string): Promise<number> {\n const raw = await readAll(process.stdin)\n let req: RpcRequest\n try {\n const body = JSON.parse(raw)\n req = method ? { method: method as RpcRequest['method'], params: body } : (body as RpcRequest)\n } catch (err) {\n process.stdout.write(\n `${JSON.stringify({\n error: {\n code: 'parse_error',\n message: `stdin was not valid JSON: ${err instanceof Error ? err.message : String(err)}`,\n },\n })}\\n`,\n )\n return 1\n }\n const out = await dispatchRpc(req)\n process.stdout.write(`${JSON.stringify(out)}\\n`)\n return 'error' in out ? 1 : 0\n}\n\n/** Read JSONL requests from stdin, write JSONL responses to stdout. */\nexport async function runRpcBatch(method?: string): Promise<number> {\n const raw = await readAll(process.stdin)\n const lines = raw.split('\\n').filter((l) => l.trim().length > 0)\n let exitCode = 0\n for (const line of lines) {\n let req: RpcRequest\n try {\n const body = JSON.parse(line)\n req = method ? { method: method as RpcRequest['method'], params: body } : (body as RpcRequest)\n } catch (err) {\n process.stdout.write(\n `${JSON.stringify({\n error: {\n code: 'parse_error',\n message: `line was not valid JSON: ${err instanceof Error ? err.message : String(err)}`,\n },\n })}\\n`,\n )\n exitCode = 1\n continue\n }\n const out = await dispatchRpc(req)\n process.stdout.write(`${JSON.stringify(out)}\\n`)\n if ('error' in out) exitCode = 1\n }\n return exitCode\n}\n","/**\n * HTTP transport for the wire protocol.\n *\n * Hono + @hono/node-server. Every endpoint:\n * 1. Validates the request against its Zod schema.\n * 2. Calls the matching handler in `handlers.ts`.\n * 3. Renders 4xx for `WireError` with structured body, 500 for unexpected.\n *\n * The server holds optional `IngestionStores` (passed to `createApp`)\n * to receive production traces and user feedback. With no stores wired,\n * the ingestion endpoints return 503 — read endpoints (`/v1/judge`,\n * `/v1/rubrics`, `/v1/version`) remain fully functional.\n *\n * Run via `agent-eval serve --port 5005`.\n */\nimport { type ServerType, serve } from '@hono/node-server'\nimport { Hono } from 'hono'\nimport { cors } from 'hono/cors'\n\nimport {\n handleFeedbackIngest,\n handleJudge,\n handleListRubrics,\n handleTracesIngest,\n handleVersion,\n type IngestionStores,\n WireError,\n} from './handlers'\nimport { buildOpenApi } from './openapi'\nimport { FeedbackTrajectorySchema, JudgeRequestSchema, TracesIngestRequestSchema } from './schemas'\n\nconst STARTED_AT = Date.now()\n\nexport interface CreateAppOptions {\n /** Stores wired to the ingestion endpoints. */\n stores?: IngestionStores\n /**\n * Bearer-token auth. When provided, every endpoint EXCEPT `/healthz`\n * and `/v1/version` requires `Authorization: Bearer <token>`. The\n * token may be a static string OR a function for time-bounded /\n * rotating tokens.\n *\n * Recommended for any server that accepts ingestion writes from the\n * public internet. Read-only deployments may omit it.\n */\n auth?: {\n bearer: string | ((token: string) => boolean | Promise<boolean>)\n }\n}\n\nconst AUTH_EXEMPT_PATHS = new Set(['/healthz', '/v1/version', '/openapi.json'])\n\nexport function createApp(opts: CreateAppOptions = {}) {\n const app = new Hono()\n\n app.use('*', cors())\n\n // Bearer-token middleware (only attached when configured).\n if (opts.auth) {\n const verify = opts.auth.bearer\n app.use('*', async (c, next) => {\n const path = new URL(c.req.url).pathname\n if (AUTH_EXEMPT_PATHS.has(path)) return next()\n const raw = c.req.header('authorization') ?? ''\n const match = raw.match(/^Bearer\\s+(.+)$/i)\n if (!match) {\n throw new WireError('unauthorized', 'Missing or malformed Authorization header.', 401)\n }\n const token = match[1] as string\n const ok = typeof verify === 'string' ? token === verify : await verify(token)\n if (!ok) {\n throw new WireError('unauthorized', 'Invalid bearer token.', 401)\n }\n return next()\n })\n }\n\n app.onError((err, c) => {\n if (err instanceof WireError) {\n const status = err.status as 400 | 401 | 404 | 422 | 500 | 503\n return c.json(\n { error: { code: err.code, message: err.message, details: err.details } },\n status,\n )\n }\n // Unexpected — log and return generic 500 without leaking internals.\n console.error('[agent-eval] unhandled error:', err)\n return c.json({ error: { code: 'internal_error', message: 'Internal server error.' } }, 500)\n })\n\n // ── Health ──\n app.get('/healthz', (c) =>\n c.json({ status: 'ok' as const, uptimeSec: (Date.now() - STARTED_AT) / 1000 }),\n )\n\n // ── Version ──\n app.get('/v1/version', (c) => c.json(handleVersion()))\n\n // ── Rubrics ──\n app.get('/v1/rubrics', (c) => c.json(handleListRubrics()))\n\n // ── Judge ──\n app.post('/v1/judge', async (c) => {\n const raw = await c.req.json().catch(() => null)\n if (raw == null) {\n throw new WireError('validation_error', 'Request body must be JSON.', 400)\n }\n const parsed = JudgeRequestSchema.safeParse(raw)\n if (!parsed.success) {\n throw new WireError(\n 'validation_error',\n 'Request did not match JudgeRequest schema.',\n 400,\n parsed.error.issues,\n )\n }\n const result = await handleJudge(parsed.data)\n return c.json(result)\n })\n\n // ── Traces ingest (NDJSON-friendly: accepts either {events:[...]} or NDJSON) ──\n app.post('/v1/traces/ingest', async (c) => {\n const contentType = c.req.header('content-type') ?? ''\n let payload: unknown\n if (contentType.includes('application/x-ndjson')) {\n const text = await c.req.text()\n const events = text\n .split('\\n')\n .map((line) => line.trim())\n .filter((line) => line.length > 0)\n .map((line) => {\n try {\n return JSON.parse(line)\n } catch {\n throw new WireError(\n 'validation_error',\n 'NDJSON line did not parse as JSON.',\n 400,\n line.slice(0, 200),\n )\n }\n })\n payload = { events }\n } else {\n payload = await c.req.json().catch(() => null)\n }\n if (payload == null) {\n throw new WireError('validation_error', 'Request body must be JSON or NDJSON.', 400)\n }\n const parsed = TracesIngestRequestSchema.safeParse(payload)\n if (!parsed.success) {\n throw new WireError(\n 'validation_error',\n 'Request did not match TracesIngestRequest schema.',\n 400,\n parsed.error.issues,\n )\n }\n const result = await handleTracesIngest(parsed.data, opts.stores ?? {})\n return c.json(result)\n })\n\n // ── Feedback ingest ──\n app.post('/v1/feedback', async (c) => {\n const raw = await c.req.json().catch(() => null)\n if (raw == null) {\n throw new WireError('validation_error', 'Request body must be JSON.', 400)\n }\n const parsed = FeedbackTrajectorySchema.safeParse(raw)\n if (!parsed.success) {\n throw new WireError(\n 'validation_error',\n 'Request did not match FeedbackTrajectory schema.',\n 400,\n parsed.error.issues,\n )\n }\n const result = await handleFeedbackIngest(parsed.data, opts.stores ?? {})\n return c.json(result)\n })\n\n // ── OpenAPI spec ──\n app.get('/openapi.json', (c) => c.json(buildOpenApi(handleVersion().version)))\n\n return app\n}\n\nexport interface ServeOptions extends CreateAppOptions {\n /** Default 5005. */\n port?: number\n /** Default '127.0.0.1'. Set to '0.0.0.0' to listen on all interfaces. */\n host?: string\n}\n\nexport function startServer(opts: ServeOptions = {}): ServerType {\n const app = createApp(opts)\n const port = opts.port ?? 5005\n const host = opts.host ?? '127.0.0.1'\n return serve({ fetch: app.fetch, port, hostname: host }, ({ address, port: actualPort }) => {\n // eslint-disable-next-line no-console\n console.log(`[agent-eval] serving on http://${address}:${actualPort}`)\n })\n}\n\nexport interface StartedServer {\n server: ServerType\n /** The OS-assigned port. When opts.port was 0, this is the actual port the\n * kernel bound — callers that need to dial back (smoke tests, sidecars\n * registering with a parent) read this rather than guessing a free port. */\n port: number\n /** Resolved host the server bound to (defaults to 127.0.0.1). */\n host: string\n /** Close the server. Resolves once active connections have drained. */\n close(): Promise<void>\n}\n\n/**\n * Promise-returning variant of `startServer` that resolves once the server is\n * listening and surfaces the resolved bound port. Use this from smoke tests\n * (`startServerAsync({ port: 0 })`) and any caller that needs to dial back.\n */\nexport function startServerAsync(opts: ServeOptions = {}): Promise<StartedServer> {\n const app = createApp(opts)\n const port = opts.port ?? 5005\n const host = opts.host ?? '127.0.0.1'\n return new Promise((resolve, reject) => {\n let settled = false\n let server: ServerType | undefined\n server = serve({ fetch: app.fetch, port, hostname: host }, ({ address, port: actualPort }) => {\n if (settled) return\n settled = true\n // eslint-disable-next-line no-console\n console.log(`[agent-eval] serving on http://${address}:${actualPort}`)\n resolve({\n server: server!,\n port: actualPort,\n host: address,\n close: () =>\n new Promise<void>((res, rej) => {\n server!.close((err) => (err ? rej(err) : res()))\n }),\n })\n })\n server.on('error', (err) => {\n if (settled) return\n settled = true\n reject(err)\n })\n })\n}\n"],"mappings":";;;;;;;;;;;AAYA,SAAS,4BAA4B;AACrC,SAAS,SAAS;AAElB,qBAAqB,CAAC;AAIf,IAAM,wBAAwB,EAClC,OAAO;AAAA,EACN,IAAI,EACD,OAAO,EACP,IAAI,CAAC,EACL,SAAS,gFAA2E;AAAA,EACvF,aAAa,EACV,OAAO,EACP,IAAI,CAAC,EACL,SAAS,sEAAsE;AAAA,EAClF,QAAQ,EACL,OAAO,EACP,IAAI,CAAC,EACL,QAAQ,CAAC,EACT,SAAS,gEAAgE;AAAA,EAC5E,KAAK,EAAE,OAAO,EAAE,QAAQ,CAAC,EAAE,SAAS,gDAAgD;AAAA,EACpF,KAAK,EAAE,OAAO,EAAE,QAAQ,CAAC,EAAE,SAAS,gDAAgD;AACtF,CAAC,EACA,QAAQ,iBAAiB;AAErB,IAAM,oBAAoB,EAC9B,OAAO;AAAA,EACN,IAAI,EAAE,OAAO,EAAE,IAAI,CAAC,EAAE,SAAS,mEAA8D;AAAA,EAC7F,aAAa,EAAE,OAAO,EAAE,IAAI,CAAC,EAAE,SAAS,mDAAmD;AAC7F,CAAC,EACA,QAAQ,aAAa;AAIjB,IAAM,eAAe,EACzB,OAAO;AAAA,EACN,MAAM,EACH,OAAO,EACP,IAAI,CAAC,EACL,SAAS,4EAAuE;AAAA,EACnF,aAAa,EACV,OAAO,EACP,IAAI,CAAC,EACL,SAAS,0DAA0D;AAAA,EACtE,cAAc,EACX,OAAO,EACP,IAAI,CAAC,EACL;AAAA,IACC;AAAA,EACF;AAAA,EACF,YAAY,EACT,MAAM,qBAAqB,EAC3B,IAAI,CAAC,EACL,SAAS,+DAA+D;AAAA,EAC3E,cAAc,EACX,MAAM,iBAAiB,EACvB,QAAQ,CAAC,CAAC,EACV,SAAS,iFAAiF;AAAA,EAC7F,MAAM,EACH,MAAM,iBAAiB,EACvB,QAAQ,CAAC,CAAC,EACV,SAAS,uEAAuE;AACrF,CAAC,EACA,QAAQ,QAAQ;AAIZ,IAAM,qBAAqB,EAC/B,OAAO;AAAA,EACN,YAAY,EACT,OAAO,EACP,SAAS,EACT,SAAS,kEAAkE;AAAA,EAC9E,QAAQ,aAAa,SAAS,EAAE;AAAA,IAC9B;AAAA,EACF;AAAA,EACA,SAAS,EACN,OAAO,EACP,IAAI,CAAC,EACL,SAAS,uFAAkF;AAAA,EAC9F,SAAS,EACN,OAAO,EAAE,OAAO,GAAG,EAAE,QAAQ,CAAC,EAC9B,SAAS,EACT;AAAA,IACC;AAAA,EACF;AAAA,EACF,OAAO,EACJ,OAAO,EACP,SAAS,EACT,SAAS,+EAA+E;AAC7F,CAAC,EACA,OAAO,CAAC,MAAM,QAAQ,EAAE,UAAU,MAAM,QAAQ,EAAE,MAAM,GAAG;AAAA,EAC1D,SAAS;AACX,CAAC,EACA,QAAQ,cAAc;AAElB,IAAM,oBAAoB,EAC9B,OAAO;AAAA,EACN,WAAW,EACR,OAAO,EACP,IAAI,CAAC,EACL,IAAI,CAAC,EACL,SAAS,iFAAiF;AAAA,EAC7F,YAAY,EACT,OAAO,EAAE,OAAO,GAAG,EAAE,OAAO,CAAC,EAC7B,SAAS,mDAAmD;AAAA,EAC/D,cAAc,EACX,MAAM,EAAE,OAAO,CAAC,EAChB,QAAQ,CAAC,CAAC,EACV,SAAS,+EAA+E;AAAA,EAC3F,MAAM,EACH,MAAM,EAAE,OAAO,CAAC,EAChB,QAAQ,CAAC,CAAC,EACV,SAAS,8DAA8D;AAAA,EAC1E,WAAW,EACR,OAAO,EACP,SAAS,yEAAyE;AAAA,EACrF,eAAe,EACZ,OAAO,EACP;AAAA,IACC;AAAA,EACF;AAAA,EACF,OAAO,EAAE,OAAO,EAAE,SAAS,yDAAyD;AAAA,EACpF,YAAY,EAAE,OAAO,EAAE,IAAI,EAAE,YAAY,EAAE,SAAS,qCAAqC;AAC3F,CAAC,EACA,QAAQ,aAAa;AAIjB,IAAM,mBAAmB,EAC7B,OAAO;AAAA,EACN,MAAM,EAAE,OAAO,EAAE,SAAS,yCAAyC;AAAA,EACnE,aAAa,EAAE,OAAO,EAAE,SAAS,4BAA4B;AAAA,EAC7D,YAAY,EACT,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,GAAG,aAAa,EAAE,OAAO,GAAG,QAAQ,EAAE,OAAO,EAAE,CAAC,CAAC,EAC/E,SAAS,kDAAkD;AAAA,EAC9D,cAAc,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,QAAQ,CAAC,CAAC,EAAE,SAAS,uCAAuC;AAAA,EAC9F,eAAe,EAAE,OAAO,EAAE,SAAS,8DAAyD;AAC9F,CAAC,EACA,QAAQ,YAAY;AAEhB,IAAM,4BAA4B,EACtC,OAAO;AAAA,EACN,SAAS,EAAE,MAAM,gBAAgB;AACnC,CAAC,EACA,QAAQ,qBAAqB;AAIzB,IAAM,wBAAwB,EAClC,OAAO;AAAA,EACN,SAAS,EAAE,OAAO,EAAE,SAAS,qDAAqD;AAAA,EAClF,SAAS,EAAE,OAAO,EAAE,SAAS,0DAA0D;AAAA,EACvF,aAAa,EACV,OAAO,EACP;AAAA,IACC;AAAA,EACF;AAAA,EACF,YAAY,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,SAAS,iCAAiC;AAC5E,CAAC,EACA,QAAQ,iBAAiB;AAErB,IAAM,uBAAuB,EACjC,OAAO;AAAA,EACN,QAAQ,EAAE,QAAQ,IAAI;AAAA,EACtB,WAAW,EAAE,OAAO;AACtB,CAAC,EACA,QAAQ,gBAAgB;AAUpB,IAAM,mBAAmB,EAC7B,OAAO;AAAA,EACN,SAAS,EAAE,OAAO,EAAE,IAAI,CAAC,EAAE,SAAS,4CAA4C;AAAA,EAChF,OAAO,EAAE,OAAO,EAAE,IAAI,CAAC,EAAE,SAAS,4BAA4B;AAAA,EAC9D,QAAQ,EAAE,OAAO,EAAE,SAAS,EAAE,SAAS,sCAAsC;AAAA,EAC7E,MAAM,EACH,KAAK;AAAA,IACJ;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,EACF,CAAC,EACA,SAAS,yEAAoE;AAAA,EAChF,WAAW,EACR,OAAO,EACP,IAAI,EACJ,YAAY,EACZ,SAAS,kEAAkE;AAAA,EAC9E,SAAS,EACN,OAAO,EAAE,OAAO,GAAG,EAAE,QAAQ,CAAC,EAC9B,SAAS,sDAAiD;AAC/D,CAAC,EACA,QAAQ,YAAY;AAEhB,IAAM,4BAA4B,EACtC,OAAO;AAAA,EACN,QAAQ,EACL,MAAM,gBAAgB,EACtB,IAAI,CAAC,EACL,IAAI,GAAM,EACV,SAAS,4EAAuE;AACrF,CAAC,EACA,QAAQ,qBAAqB;AAEzB,IAAM,6BAA6B,EACvC,OAAO;AAAA,EACN,UAAU,EAAE,OAAO,EAAE,IAAI,EAAE,YAAY,EAAE,SAAS,6BAA6B;AAAA,EAC/E,UAAU,EACP,OAAO,EACP,IAAI,EACJ,YAAY,EACZ,SAAS,uEAAkE;AAAA,EAC9E,QAAQ,EACL;AAAA,IACC,EAAE,OAAO;AAAA,MACP,SAAS,EAAE,OAAO,EAAE,SAAS,iCAAiC;AAAA,MAC9D,SAAS,EAAE,OAAO,EAAE,SAAS,6BAA6B;AAAA,IAC5D,CAAC;AAAA,EACH,EACC,QAAQ,CAAC,CAAC;AACf,CAAC,EACA,QAAQ,sBAAsB;AAE1B,IAAM,sBAAsB,EAChC,OAAO;AAAA,EACN,IAAI,EAAE,OAAO,EAAE,SAAS;AAAA,EACxB,QAAQ,EAAE,KAAK,CAAC,QAAQ,SAAS,eAAe,UAAU,UAAU,QAAQ,CAAC;AAAA,EAC7E,MAAM,EAAE,KAAK;AAAA,IACX;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,EACF,CAAC;AAAA,EACD,OAAO,EAAE,QAAQ;AAAA,EACjB,QAAQ,EAAE,OAAO,EAAE,SAAS;AAAA,EAC5B,UAAU,EAAE,KAAK,CAAC,QAAQ,WAAW,SAAS,UAAU,CAAC,EAAE,SAAS;AAAA,EACpE,WAAW,EAAE,OAAO,EAAE,SAAS,eAAe;AAAA,EAC9C,UAAU,EAAE,OAAO,EAAE,OAAO,GAAG,EAAE,QAAQ,CAAC,EAAE,SAAS;AACvD,CAAC,EACA,QAAQ,eAAe;AAEnB,IAAM,wBAAwB,EAClC,OAAO;AAAA,EACN,IAAI,EAAE,OAAO,EAAE,IAAI,CAAC;AAAA,EACpB,WAAW,EAAE,OAAO,EAAE,IAAI,EAAE,YAAY;AAAA,EACxC,cAAc,EAAE,KAAK;AAAA,IACnB;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,IACA;AAAA,EACF,CAAC;AAAA,EACD,UAAU,EAAE,QAAQ;AAAA,EACpB,SAAS,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,SAAS;AAAA,EACvC,gBAAgB,EACb,OAAO;AAAA,IACN,MAAM,EAAE,OAAO;AAAA,IACf,MAAM,EAAE,KAAK,CAAC,OAAO,UAAU,MAAM,CAAC,EAAE,SAAS;AAAA,IACjD,SAAS,EAAE,OAAO,EAAE,SAAS;AAAA,IAC7B,oBAAoB,EAAE,QAAQ,EAAE,SAAS;AAAA,IACzC,kBAAkB,EAAE,QAAQ,EAAE,SAAS;AAAA,IACvC,UAAU,EAAE,OAAO,EAAE,OAAO,GAAG,EAAE,QAAQ,CAAC,EAAE,SAAS;AAAA,EACvD,CAAC,EACA,SAAS;AAAA,EACZ,UAAU,EAAE,MAAM,mBAAmB,EAAE,SAAS;AAAA,EAChD,WAAW,EAAE,OAAO;AAAA,EACpB,UAAU,EAAE,OAAO,EAAE,OAAO,GAAG,EAAE,QAAQ,CAAC,EAAE,SAAS;AACvD,CAAC,EACA,QAAQ,iBAAiB;AAErB,IAAM,2BAA2B,EACrC,OAAO;AAAA,EACN,IAAI,EAAE,OAAO,EAAE,IAAI,CAAC,EAAE,SAAS,gDAAgD;AAAA,EAC/E,WAAW,EAAE,OAAO,EAAE,SAAS;AAAA,EAC/B,YAAY,EAAE,OAAO,EAAE,SAAS;AAAA,EAChC,MAAM,EAAE,OAAO;AAAA,IACb,QAAQ,EAAE,OAAO,EAAE,IAAI,CAAC;AAAA,IACxB,SAAS,EAAE,QAAQ,EAAE,SAAS;AAAA,EAChC,CAAC;AAAA,EACD,UAAU,EAAE,MAAM,qBAAqB,EAAE,QAAQ,CAAC,CAAC;AAAA,EACnD,QAAQ,EAAE,MAAM,mBAAmB,EAAE,QAAQ,CAAC,CAAC;AAAA,EAC/C,SAAS,EACN,OAAO;AAAA,IACN,SAAS,EAAE,QAAQ,EAAE,SAAS;AAAA,IAC9B,OAAO,EAAE,OAAO,EAAE,SAAS;AAAA,IAC3B,SAAS,EAAE,OAAO,EAAE,OAAO,GAAG,EAAE,OAAO,CAAC,EAAE,SAAS;AAAA,IACnD,SAAS,EAAE,OAAO,EAAE,SAAS;AAAA,IAC7B,QAAQ,EAAE,OAAO,EAAE,SAAS;AAAA,IAC5B,YAAY,EAAE,OAAO,EAAE,SAAS;AAAA,IAChC,UAAU,EAAE,OAAO,EAAE,OAAO,GAAG,EAAE,QAAQ,CAAC,EAAE,SAAS;AAAA,EACvD,CAAC,EACA,SAAS;AAAA,EACZ,OAAO,EAAE,KAAK,CAAC,SAAS,OAAO,QAAQ,SAAS,CAAC,EAAE,SAAS;AAAA,EAC5D,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,EAAE,OAAO,CAAC,EAAE,SAAS;AAAA,EAChD,WAAW,EAAE,OAAO,EAAE,SAAS,eAAe;AAAA,EAC9C,WAAW,EAAE,OAAO,EAAE,SAAS;AAAA,EAC/B,UAAU,EAAE,OAAO,EAAE,OAAO,GAAG,EAAE,QAAQ,CAAC,EAAE,SAAS;AACvD,CAAC,EACA,QAAQ,oBAAoB;AAExB,IAAM,+BAA+B,EACzC,OAAO;AAAA,EACN,IAAI,EAAE,OAAO,EAAE,SAAS,mCAAmC;AAAA,EAC3D,WAAW,EAAE,QAAQ,EAAE,SAAS,wDAAwD;AAC1F,CAAC,EACA,QAAQ,wBAAwB;AAU5B,IAAM,sBAAsB,EAChC,OAAO;AAAA,EACN,OAAO,EACJ,OAAO;AAAA,IACN,MAAM,EACH,OAAO,EACP;AAAA,MACC;AAAA,IACF;AAAA,IACF,SAAS,EAAE,OAAO,EAAE,SAAS,yBAAyB;AAAA,IACtD,SAAS,EAAE,QAAQ,EAAE,SAAS,EAAE,SAAS,6BAA6B;AAAA,EACxE,CAAC,EACA,SAAS,+DAA+D;AAC7E,CAAC,EACA,QAAQ,eAAe;AAoBnB,IAAM,eAAe;AAMrB,SAAS,WAAW,QAAwB;AACjD,QAAM,SAAS,gBAAgB,MAAM;AACrC,MAAI,IAAI;AACR,WAAS,IAAI,GAAG,IAAI,OAAO,QAAQ,KAAK;AACtC,QAAK,IAAI,KAAM,OAAO,WAAW,CAAC;AAAA,EACpC;AAEA,SAAO,GAAG,OAAO,IAAI,KAAK,MAAM,GAAG,SAAS,EAAE,EAAE,SAAS,GAAG,GAAG,CAAC;AAClE;AAEA,SAAS,gBAAgB,OAAwB;AAC/C,MAAI,MAAM,QAAQ,KAAK,EAAG,QAAO,IAAI,MAAM,IAAI,CAAC,SAAS,gBAAgB,IAAI,CAAC,EAAE,KAAK,GAAG,CAAC;AACzF,MAAI,SAAS,OAAO,UAAU,UAAU;AACtC,UAAM,UAAU,OAAO,QAAQ,KAAgC,EAC5D,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,cAAc,CAAC,CAAC,EACrC,IAAI,CAAC,CAAC,KAAK,IAAI,MAAM,GAAG,KAAK,UAAU,GAAG,CAAC,IAAI,gBAAgB,IAAI,CAAC,EAAE;AACzE,WAAO,IAAI,QAAQ,KAAK,GAAG,CAAC;AAAA,EAC9B;AACA,SAAO,KAAK,UAAU,KAAK;AAC7B;;;ACjYA,IAAM,YAAoB;AAAA,EACxB,MAAM;AAAA,EACN,aACE;AAAA,EACF,cAAc;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAuBd,YAAY;AAAA,IACV;AAAA,MACE,IAAI;AAAA,MACJ,aAAa;AAAA,MACb,QAAQ;AAAA,MACR,KAAK;AAAA,MACL,KAAK;AAAA,IACP;AAAA,IACA;AAAA,MACE,IAAI;AAAA,MACJ,aAAa;AAAA,MACb,QAAQ;AAAA,MACR,KAAK;AAAA,MACL,KAAK;AAAA,IACP;AAAA,IACA;AAAA,MACE,IAAI;AAAA,MACJ,aAAa;AAAA,MACb,QAAQ;AAAA,MACR,KAAK;AAAA,MACL,KAAK;AAAA,IACP;AAAA,EACF;AAAA,EACA,cAAc;AAAA,IACZ,EAAE,IAAI,cAAc,aAAa,qCAAqC;AAAA,IACtE,EAAE,IAAI,kBAAkB,aAAa,sCAAsC;AAAA,IAC3E,EAAE,IAAI,eAAe,aAAa,oCAAoC;AAAA,IACtE,EAAE,IAAI,WAAW,aAAa,kCAAkC;AAAA,IAChE,EAAE,IAAI,mBAAmB,aAAa,qCAAqC;AAAA,IAC3E,EAAE,IAAI,WAAW,aAAa,oCAAoC;AAAA,IAClE,EAAE,IAAI,eAAe,aAAa,uCAAuC;AAAA,EAC3E;AAAA,EACA,MAAM;AAAA,IACJ,EAAE,IAAI,sBAAsB,aAAa,qCAAqC;AAAA,IAC9E,EAAE,IAAI,iBAAiB,aAAa,4BAA4B;AAAA,IAChE,EAAE,IAAI,0BAA0B,aAAa,wBAAwB;AAAA,IACrE,EAAE,IAAI,kBAAkB,aAAa,oCAAoC;AAAA,EAC3E;AACF;AAIO,IAAM,kBAA0C;AAAA,EACrD,aAAa;AACf;AAGO,SAAS,iBAAiB,MAAkC;AACjE,SAAO,gBAAgB,IAAI;AAC7B;AAGO,SAAS,qBAAqB;AACnC,SAAO,OAAO,OAAO,eAAe,EAAE,IAAI,CAAC,OAAO;AAAA,IAChD,MAAM,EAAE;AAAA,IACR,aAAa,EAAE;AAAA,IACf,YAAY,EAAE,WAAW,IAAI,CAAC,OAAO;AAAA,MACnC,IAAI,EAAE;AAAA,MACN,aAAa,EAAE;AAAA,MACf,QAAQ,EAAE;AAAA,IACZ,EAAE;AAAA,IACF,cAAc,EAAE,aAAa,IAAI,CAAC,MAAM,EAAE,EAAE;AAAA,IAC5C,eAAe,WAAW,CAAC;AAAA,EAC7B,EAAE;AACJ;;;AC+JA,SAAS,oBAAoB;AAC7B,SAAS,SAAS,eAAe;AACjC,SAAS,qBAAqB;AA5OvB,IAAM,YAAN,cAAwB,MAAM;AAAA,EACnC,YACkB,MAChB,SACgB,SAAiB,KACjB,SAChB;AACA,UAAM,OAAO;AALG;AAEA;AACA;AAGhB,SAAK,OAAO;AAAA,EACd;AAAA,EAPkB;AAAA,EAEA;AAAA,EACA;AAKpB;AAKA,SAAS,kBAAkB,QAAgB;AACzC,SAAO;AAAA,IACL,MAAM;AAAA,IACN,QAAQ;AAAA,MACN,MAAM;AAAA,MACN,sBAAsB;AAAA,MACtB,YAAY;AAAA,QACV,YAAY;AAAA,UACV,MAAM;AAAA,UACN,sBAAsB;AAAA,UACtB,YAAY,OAAO;AAAA,YACjB,OAAO,WAAW,IAAI,CAAC,MAAM;AAAA,cAC3B,EAAE;AAAA,cACF,EAAE,MAAM,UAAU,SAAS,EAAE,KAAK,SAAS,EAAE,IAAI;AAAA,YACnD,CAAC;AAAA,UACH;AAAA,UACA,UAAU,OAAO,WAAW,IAAI,CAAC,MAAM,EAAE,EAAE;AAAA,QAC7C;AAAA,QACA,cAAc;AAAA,UACZ,MAAM;AAAA,UACN,OAAO,EAAE,MAAM,UAAU,MAAM,OAAO,aAAa,IAAI,CAAC,MAAM,EAAE,EAAE,EAAE;AAAA,QACtE;AAAA,QACA,MAAM;AAAA,UACJ,MAAM;AAAA,UACN,OAAO,EAAE,MAAM,UAAU,MAAM,OAAO,KAAK,IAAI,CAAC,MAAM,EAAE,EAAE,EAAE;AAAA,QAC9D;AAAA,QACA,WAAW,EAAE,MAAM,SAAS;AAAA,MAC9B;AAAA,MACA,UAAU,CAAC,cAAc,WAAW;AAAA,IACtC;AAAA,EACF;AACF;AASA,SAAS,oBAAoB,OAAgB,QAA6B;AACxE,MAAI,CAAC,SAAS,OAAO,UAAU,UAAU;AACvC,UAAM,IAAI,UAAU,eAAe,oCAAoC,KAAK,KAAK;AAAA,EACnF;AACA,QAAM,MAAM;AACZ,QAAM,gBAAgB,IAAI;AAC1B,MAAI,CAAC,iBAAiB,OAAO,kBAAkB,YAAY,MAAM,QAAQ,aAAa,GAAG;AACvF,UAAM,IAAI,UAAU,eAAe,wCAAwC,KAAK,KAAK;AAAA,EACvF;AAEA,QAAM,aAAqC,CAAC;AAC5C,QAAM,kBAAkB;AACxB,aAAW,OAAO,OAAO,YAAY;AACnC,UAAM,QAAQ,gBAAgB,IAAI,EAAE;AACpC,QACE,OAAO,UAAU,YACjB,CAAC,OAAO,SAAS,KAAK,KACtB,QAAQ,IAAI,OACZ,QAAQ,IAAI,KACZ;AACA,YAAM,IAAI;AAAA,QACR;AAAA,QACA,+CAA+C,IAAI,EAAE;AAAA,QACrD;AAAA,QACA;AAAA,MACF;AAAA,IACF;AACA,eAAW,IAAI,EAAE,IAAI;AAAA,EACvB;AAEA,QAAM,kBAAkB,IAAI,IAAI,OAAO,aAAa,IAAI,CAAC,SAAS,KAAK,EAAE,CAAC;AAC1E,QAAM,cAAc,IAAI,IAAI,OAAO,KAAK,IAAI,CAAC,QAAQ,IAAI,EAAE,CAAC;AAC5D,QAAM,eAAe,gBAAgB,IAAI,cAAc,iBAAiB,gBAAgB,KAAK;AAC7F,QAAM,OAAO,gBAAgB,IAAI,MAAM,aAAa,QAAQ,KAAK;AACjE,MAAI,OAAO,IAAI,cAAc,YAAY,IAAI,UAAU,KAAK,EAAE,WAAW,GAAG;AAC1E,UAAM,IAAI,UAAU,eAAe,qCAAqC,KAAK,KAAK;AAAA,EACpF;AAEA,SAAO,EAAE,YAAY,cAAc,MAAM,WAAW,IAAI,UAAU;AACpE;AAEA,SAAS,gBACP,KACA,SACA,OACA,UACU;AACV,MAAI,QAAQ,OAAW,QAAO,CAAC;AAC/B,MAAI,CAAC,MAAM,QAAQ,GAAG,GAAG;AACvB,UAAM,IAAI,UAAU,eAAe,4BAA4B,KAAK,KAAK,KAAK,QAAQ;AAAA,EACxF;AACA,QAAM,MAAgB,CAAC;AACvB,aAAW,QAAQ,KAAK;AACtB,QAAI,OAAO,SAAS,YAAY,CAAC,QAAQ,IAAI,IAAI,GAAG;AAClD,YAAM,IAAI;AAAA,QACR;AAAA,QACA,0BAA0B,KAAK,QAAQ,OAAO,IAAI,CAAC;AAAA,QACnD;AAAA,QACA;AAAA,MACF;AAAA,IACF;AACA,QAAI,KAAK,IAAI;AAAA,EACf;AACA,SAAO;AACT;AAEA,SAAS,eAAe,YAAoC,QAAwB;AAClF,MAAI,WAAW;AACf,MAAI,cAAc;AAClB,aAAW,OAAO,OAAO,YAAY;AACnC,UAAM,MAAM,WAAW,IAAI,EAAE,KAAK;AAClC,UAAM,QAAQ,IAAI,MAAM,IAAI,OAAO;AACnC,UAAM,aAAa,KAAK,IAAI,GAAG,KAAK,IAAI,IAAI,MAAM,IAAI,OAAO,KAAK,CAAC;AACnE,gBAAY,aAAa,IAAI;AAC7B,mBAAe,IAAI;AAAA,EACrB;AACA,SAAO,cAAc,IAAI,WAAW,cAAc;AACpD;AAEA,SAAS,iBAAiB,SAAiB,SAA0B;AACnE,QAAM,MAAM,WAAW,OAAO,KAAK,OAAiB,EAAE,SAAS,KAAK,UAAU,OAAO,IAAI;AACzF,SAAO;AAAA,IACL;AAAA,IACA;AAAA,IACA;AAAA,IACA,MAAM,yCAAyC;AAAA,IAC/C,MAAM,MAAM;AAAA,EACd,EACG,OAAO,OAAO,EACd,KAAK,IAAI;AACd;AAEA,IAAM,sBAAsB;AAS5B,eAAsB,YACpB,KACA,UAA8B,CAAC,GACT;AAEtB,MAAI;AACJ,MAAI,IAAI,YAAY;AAClB,UAAM,QAAQ,iBAAiB,IAAI,UAAU;AAC7C,QAAI,CAAC,OAAO;AACV,YAAM,IAAI,UAAU,oBAAoB,6BAA6B,IAAI,UAAU,MAAM,GAAG;AAAA,IAC9F;AACA,aAAS;AAAA,EACX,WAAW,IAAI,QAAQ;AACrB,aAAS,IAAI;AAAA,EACf,OAAO;AAEL,UAAM,IAAI,UAAU,oBAAoB,4CAA4C,GAAG;AAAA,EACzF;AAEA,QAAM,YAAY,KAAK,IAAI;AAC3B,QAAM,QAAQ,IAAI,SAAS;AAE3B,QAAM,UAAU;AAAA,IACd;AAAA,IACA,UAAU;AAAA,MACR,EAAE,MAAM,UAAU,SAAS,OAAO,aAAa;AAAA,MAC/C,EAAE,MAAM,QAAQ,SAAS,iBAAiB,IAAI,SAAS,IAAI,OAAO,EAAE;AAAA,IACtE;AAAA,IACA,YAAY,kBAAkB,MAAM;AAAA,IACpC,aAAa;AAAA,IACb,WAAW;AAAA,IACX,WAAW;AAAA,EACb;AACA,QAAM,SAAS,QAAQ,cAAc,IAAI,WAAW;AACpD,QAAM,OAAO,MAAM,OAAO,YAAY;AAAA,IACpC,SAAS;AAAA,IACT,OAAO,QAAQ,aAAa;AAAA,IAC5B,OAAO,QAAQ,IAAI,cAAc,QAAQ;AAAA,IACzC;AAAA,IACA,eAAe,2BAA2B,SAAS,QAAQ,GAAG;AAAA,IAC9D,QAAQ,QAAQ;AAAA,IAChB,SAAS,CAAC,QAAQ,WAChB,YAAyB,SAAS;AAAA,MAChC,GAAG,QAAQ;AAAA,MACX;AAAA,MACA,gBAAgB;AAAA,IAClB,CAAC;AAAA,IACH,SAAS,CAAC,EAAE,QAAAA,QAAO,MAAM,mBAAmBA,OAAM;AAAA,IAClD,kBAAkB;AAAA,EACpB,CAAC;AACD,MAAI,CAAC,KAAK,UAAW,OAAM,KAAK;AAChC,QAAM,EAAE,OAAO,OAAO,IAAI,KAAK;AAE/B,QAAM,SAAS,oBAAoB,OAAO,MAAM;AAEhD,QAAM,YAAY,eAAe,OAAO,YAAY,MAAM;AAC1D,QAAM,aAAa,KAAK,IAAI,IAAI;AAEhC,SAAO;AAAA,IACL;AAAA,IACA,YAAY,OAAO;AAAA,IACnB,cAAc,OAAO,gBAAgB,CAAC;AAAA,IACtC,MAAM,OAAO,QAAQ,CAAC;AAAA,IACtB,WAAW,OAAO;AAAA,IAClB,eAAe,WAAW,MAAM;AAAA,IAChC,OAAO,OAAO;AAAA,IACd;AAAA,EACF;AACF;AAIO,SAAS,oBAAyC;AACvD,SAAO,EAAE,SAAS,mBAAmB,EAAE;AACzC;AAQA,IAAI;AAEJ,SAAS,qBAA6B;AACpC,MAAI,eAAgB,QAAO;AAG3B,QAAM,OAAO,QAAQ,cAAc,YAAY,GAAG,CAAC;AACnD,QAAM,aAAa;AAAA,IACjB,QAAQ,MAAM,MAAM,MAAM,cAAc;AAAA;AAAA,IACxC,QAAQ,MAAM,MAAM,cAAc;AAAA;AAAA,EACpC;AACA,aAAW,QAAQ,YAAY;AAC7B,QAAI;AACF,YAAM,MAAM,KAAK,MAAM,aAAa,MAAM,OAAO,CAAC;AAClD,UAAI,IAAI,SAAS;AACf,yBAAiB,IAAI;AACrB,eAAO,IAAI;AAAA,MACb;AAAA,IACF,QAAQ;AAAA,IAER;AAAA,EACF;AACA,SAAO;AACT;AAEO,SAAS,gBAAiC;AAC/C,SAAO;AAAA,IACL,SAAS;AAAA,IACT,SAAS,mBAAmB;AAAA,IAC5B,aAAa;AAAA,IACb,YAAY,CAAC,SAAS,eAAe,WAAW,mBAAmB,eAAe;AAAA,EACpF;AACF;AA2BA,eAAsB,mBACpB,KACA,QAC+B;AAC/B,MAAI,CAAC,OAAO,YAAY;AACtB,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACA,QAAM,SAAsD,CAAC;AAC7D,MAAI,WAAW;AACf,aAAW,SAAS,IAAI,QAAQ;AAC9B,QAAI;AAEF,YAAM,OAAO,WAAW,YAAY,KAA2B;AAC/D;AAAA,IACF,SAAS,KAAK;AACZ,aAAO,KAAK;AAAA,QACV,SAAS,MAAM;AAAA,QACf,SAAS,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG;AAAA,MAC1D,CAAC;AAAA,IACH;AAAA,EACF;AACA,SAAO,EAAE,UAAU,UAAU,OAAO,QAAQ,OAAO;AACrD;AAOA,eAAsB,qBACpB,KACA,QACiC;AACjC,MAAI,CAAC,OAAO,eAAe;AACzB,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,MACA;AAAA,IACF;AAAA,EACF;AAIA,QAAM,OAAO,cAAc,KAAK,GAAgE;AAChG,SAAO,EAAE,IAAI,IAAI,IAAI,WAAW,KAAK;AACvC;;;ACvXA,SAAS,iBAAiB,2BAA2B;AAiB9C,SAAS,aAAa,gBAAuC;AAClE,QAAM,WAAW,IAAI,gBAAgB;AAGrC,WAAS,SAAS,gBAAgB,kBAAkB;AACpD,WAAS,SAAS,eAAe,iBAAiB;AAClD,WAAS,SAAS,uBAAuB,yBAAyB;AAClE,WAAS,SAAS,mBAAmB,qBAAqB;AAC1D,WAAS,SAAS,kBAAkB,oBAAoB;AACxD,WAAS,SAAS,iBAAiB,mBAAmB;AACtD,WAAS,SAAS,uBAAuB,yBAAyB;AAClE,WAAS,SAAS,wBAAwB,0BAA0B;AACpE,WAAS,SAAS,sBAAsB,wBAAwB;AAChE,WAAS,SAAS,0BAA0B,4BAA4B;AAGxE,WAAS,aAAa;AAAA,IACpB,QAAQ;AAAA,IACR,MAAM;AAAA,IACN,SAAS;AAAA,IACT,aACE;AAAA,IACF,SAAS;AAAA,MACP,MAAM;AAAA,QACJ,SAAS;AAAA,UACP,oBAAoB,EAAE,QAAQ,mBAAmB;AAAA,QACnD;AAAA,MACF;AAAA,IACF;AAAA,IACA,WAAW;AAAA,MACT,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,kBAAkB,EAAE;AAAA,MAC/D;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,IACF;AAAA,EACF,CAAC;AAED,WAAS,aAAa;AAAA,IACpB,QAAQ;AAAA,IACR,MAAM;AAAA,IACN,SAAS;AAAA,IACT,aACE;AAAA,IACF,WAAW;AAAA,MACT,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,0BAA0B,EAAE;AAAA,MACvE;AAAA,IACF;AAAA,EACF,CAAC;AAED,WAAS,aAAa;AAAA,IACpB,QAAQ;AAAA,IACR,MAAM;AAAA,IACN,SAAS;AAAA,IACT,aAAa;AAAA,IACb,WAAW;AAAA,MACT,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,sBAAsB,EAAE;AAAA,MACnE;AAAA,IACF;AAAA,EACF,CAAC;AAED,WAAS,aAAa;AAAA,IACpB,QAAQ;AAAA,IACR,MAAM;AAAA,IACN,SAAS;AAAA,IACT,WAAW;AAAA,MACT,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,qBAAqB,EAAE;AAAA,MAClE;AAAA,IACF;AAAA,EACF,CAAC;AAED,WAAS,aAAa;AAAA,IACpB,QAAQ;AAAA,IACR,MAAM;AAAA,IACN,SAAS;AAAA,IACT,aACE;AAAA,IACF,SAAS;AAAA,MACP,MAAM;AAAA,QACJ,SAAS;AAAA,UACP,oBAAoB,EAAE,QAAQ,0BAA0B;AAAA,UACxD,wBAAwB,EAAE,QAAQ,0BAA0B;AAAA,QAC9D;AAAA,MACF;AAAA,IACF;AAAA,IACA,WAAW;AAAA,MACT,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,2BAA2B,EAAE;AAAA,MACxE;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,IACF;AAAA,EACF,CAAC;AAED,WAAS,aAAa;AAAA,IACpB,QAAQ;AAAA,IACR,MAAM;AAAA,IACN,SAAS;AAAA,IACT,aACE;AAAA,IACF,SAAS;AAAA,MACP,MAAM;AAAA,QACJ,SAAS;AAAA,UACP,oBAAoB,EAAE,QAAQ,yBAAyB;AAAA,QACzD;AAAA,MACF;AAAA,IACF;AAAA,IACA,WAAW;AAAA,MACT,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,6BAA6B,EAAE;AAAA,MAC1E;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,MACA,KAAK;AAAA,QACH,aAAa;AAAA,QACb,SAAS,EAAE,oBAAoB,EAAE,QAAQ,oBAAoB,EAAE;AAAA,MACjE;AAAA,IACF;AAAA,EACF,CAAC;AAED,QAAM,YAAY,IAAI,oBAAoB,SAAS,WAAW;AAC9D,QAAM,MAAM,UAAU,iBAAiB;AAAA,IACrC,SAAS;AAAA,IACT,MAAM;AAAA,MACJ,OAAO;AAAA,MACP,SAAS;AAAA,MACT,aAAa;AAAA;AAAA,yBAEM,YAAY;AAAA,MAC/B,SAAS,EAAE,MAAM,kBAAkB,KAAK,+CAA+C;AAAA,MACvF,SAAS,EAAE,MAAM,MAAM;AAAA,IACzB;AAAA,IACA,SAAS,CAAC,EAAE,KAAK,yBAAyB,aAAa,yBAAyB,CAAC;AAAA,EACnF,CAAC;AACD,QAAM,YAAY,EAAE,MAAM,8BAA8B;AACxD,QAAM,oBAAoB;AAAA,IACxB,SAAS,EAAE,MAAM,UAAU,WAAW,EAAE;AAAA,IACxC,SAAS,EAAE,MAAM,UAAU,sBAAsB,KAAK;AAAA,IACtD,OAAO,EAAE,MAAM,SAAS;AAAA,EAC1B;AACA,MAAI,eAAe,CAAC;AACpB,MAAI,WAAW,YAAY,CAAC;AAC5B,MAAI,WAAW,QAAQ,eAAe;AAAA,IACpC,OAAO;AAAA,MACL;AAAA,QACE,MAAM;AAAA,QACN,sBAAsB;AAAA,QACtB,UAAU,CAAC,cAAc,SAAS;AAAA,QAClC,YAAY;AAAA,UACV,YAAY,EAAE,MAAM,UAAU,WAAW,EAAE;AAAA,UAC3C,GAAG;AAAA,QACL;AAAA,MACF;AAAA,MACA;AAAA,QACE,MAAM;AAAA,QACN,sBAAsB;AAAA,QACtB,UAAU,CAAC,UAAU,SAAS;AAAA,QAC9B,YAAY;AAAA,UACV,QAAQ;AAAA,UACR,GAAG;AAAA,QACL;AAAA,MACF;AAAA,IACF;AAAA,IACA,aAAa;AAAA,EACf;AACA,SAAO;AACT;;;ACxMA,eAAsB,YAAY,KAAiD;AACjF,MAAI;AACF,YAAQ,IAAI,QAAQ;AAAA,MAClB,KAAK,SAAS;AACZ,cAAM,SAAS,mBAAmB,UAAU,IAAI,MAAM;AACtD,YAAI,CAAC,OAAO,SAAS;AACnB,iBAAO;AAAA,YACL,OAAO;AAAA,cACL,MAAM;AAAA,cACN,SAAS;AAAA,cACT,SAAS,OAAO,MAAM;AAAA,YACxB;AAAA,UACF;AAAA,QACF;AACA,eAAO,EAAE,QAAQ,MAAM,YAAY,OAAO,IAAI,EAAE;AAAA,MAClD;AAAA,MACA,KAAK;AACH,eAAO,EAAE,QAAQ,kBAAkB,EAAE;AAAA,MACvC,KAAK;AACH,eAAO,EAAE,QAAQ,cAAc,EAAE;AAAA,MACnC;AACE,eAAO;AAAA,UACL,OAAO;AAAA,YACL,MAAM;AAAA,YACN,SAAS,mBAAoB,IAA2B,MAAM;AAAA,UAChE;AAAA,QACF;AAAA,IACJ;AAAA,EACF,SAAS,KAAK;AACZ,QAAI,eAAe,WAAW;AAC5B,aAAO,EAAE,OAAO,EAAE,MAAM,IAAI,MAAM,SAAS,IAAI,SAAS,SAAS,IAAI,QAAQ,EAAE;AAAA,IACjF;AACA,UAAM,UAAU,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG;AAC/D,WAAO,EAAE,OAAO,EAAE,MAAM,kBAAkB,QAAQ,EAAE;AAAA,EACtD;AACF;AAIA,eAAe,QAAQ,QAAgD;AACrE,QAAM,SAAmB,CAAC;AAC1B,mBAAiB,SAAS,QAAQ;AAChC,WAAO,KAAK,OAAO,SAAS,KAAK,IAAI,QAAQ,OAAO,KAAK,KAAe,CAAC;AAAA,EAC3E;AACA,SAAO,OAAO,OAAO,MAAM,EAAE,SAAS,OAAO;AAC/C;AAGA,eAAsB,WAAW,QAAkC;AACjE,QAAM,MAAM,MAAM,QAAQ,QAAQ,KAAK;AACvC,MAAI;AACJ,MAAI;AACF,UAAM,OAAO,KAAK,MAAM,GAAG;AAC3B,UAAM,SAAS,EAAE,QAAwC,QAAQ,KAAK,IAAK;AAAA,EAC7E,SAAS,KAAK;AACZ,YAAQ,OAAO;AAAA,MACb,GAAG,KAAK,UAAU;AAAA,QAChB,OAAO;AAAA,UACL,MAAM;AAAA,UACN,SAAS,6BAA6B,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,CAAC;AAAA,QACxF;AAAA,MACF,CAAC,CAAC;AAAA;AAAA,IACJ;AACA,WAAO;AAAA,EACT;AACA,QAAM,MAAM,MAAM,YAAY,GAAG;AACjC,UAAQ,OAAO,MAAM,GAAG,KAAK,UAAU,GAAG,CAAC;AAAA,CAAI;AAC/C,SAAO,WAAW,MAAM,IAAI;AAC9B;AAGA,eAAsB,YAAY,QAAkC;AAClE,QAAM,MAAM,MAAM,QAAQ,QAAQ,KAAK;AACvC,QAAM,QAAQ,IAAI,MAAM,IAAI,EAAE,OAAO,CAAC,MAAM,EAAE,KAAK,EAAE,SAAS,CAAC;AAC/D,MAAI,WAAW;AACf,aAAW,QAAQ,OAAO;AACxB,QAAI;AACJ,QAAI;AACF,YAAM,OAAO,KAAK,MAAM,IAAI;AAC5B,YAAM,SAAS,EAAE,QAAwC,QAAQ,KAAK,IAAK;AAAA,IAC7E,SAAS,KAAK;AACZ,cAAQ,OAAO;AAAA,QACb,GAAG,KAAK,UAAU;AAAA,UAChB,OAAO;AAAA,YACL,MAAM;AAAA,YACN,SAAS,4BAA4B,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,CAAC;AAAA,UACvF;AAAA,QACF,CAAC,CAAC;AAAA;AAAA,MACJ;AACA,iBAAW;AACX;AAAA,IACF;AACA,UAAM,MAAM,MAAM,YAAY,GAAG;AACjC,YAAQ,OAAO,MAAM,GAAG,KAAK,UAAU,GAAG,CAAC;AAAA,CAAI;AAC/C,QAAI,WAAW,IAAK,YAAW;AAAA,EACjC;AACA,SAAO;AACT;;;AC/GA,SAA0B,aAAa;AACvC,SAAS,YAAY;AACrB,SAAS,YAAY;AAcrB,IAAM,aAAa,KAAK,IAAI;AAmB5B,IAAM,oBAAoB,oBAAI,IAAI,CAAC,YAAY,eAAe,eAAe,CAAC;AAEvE,SAAS,UAAU,OAAyB,CAAC,GAAG;AACrD,QAAM,MAAM,IAAI,KAAK;AAErB,MAAI,IAAI,KAAK,KAAK,CAAC;AAGnB,MAAI,KAAK,MAAM;AACb,UAAM,SAAS,KAAK,KAAK;AACzB,QAAI,IAAI,KAAK,OAAO,GAAG,SAAS;AAC9B,YAAM,OAAO,IAAI,IAAI,EAAE,IAAI,GAAG,EAAE;AAChC,UAAI,kBAAkB,IAAI,IAAI,EAAG,QAAO,KAAK;AAC7C,YAAM,MAAM,EAAE,IAAI,OAAO,eAAe,KAAK;AAC7C,YAAM,QAAQ,IAAI,MAAM,kBAAkB;AAC1C,UAAI,CAAC,OAAO;AACV,cAAM,IAAI,UAAU,gBAAgB,8CAA8C,GAAG;AAAA,MACvF;AACA,YAAM,QAAQ,MAAM,CAAC;AACrB,YAAM,KAAK,OAAO,WAAW,WAAW,UAAU,SAAS,MAAM,OAAO,KAAK;AAC7E,UAAI,CAAC,IAAI;AACP,cAAM,IAAI,UAAU,gBAAgB,yBAAyB,GAAG;AAAA,MAClE;AACA,aAAO,KAAK;AAAA,IACd,CAAC;AAAA,EACH;AAEA,MAAI,QAAQ,CAAC,KAAK,MAAM;AACtB,QAAI,eAAe,WAAW;AAC5B,YAAM,SAAS,IAAI;AACnB,aAAO,EAAE;AAAA,QACP,EAAE,OAAO,EAAE,MAAM,IAAI,MAAM,SAAS,IAAI,SAAS,SAAS,IAAI,QAAQ,EAAE;AAAA,QACxE;AAAA,MACF;AAAA,IACF;AAEA,YAAQ,MAAM,iCAAiC,GAAG;AAClD,WAAO,EAAE,KAAK,EAAE,OAAO,EAAE,MAAM,kBAAkB,SAAS,yBAAyB,EAAE,GAAG,GAAG;AAAA,EAC7F,CAAC;AAGD,MAAI;AAAA,IAAI;AAAA,IAAY,CAAC,MACnB,EAAE,KAAK,EAAE,QAAQ,MAAe,YAAY,KAAK,IAAI,IAAI,cAAc,IAAK,CAAC;AAAA,EAC/E;AAGA,MAAI,IAAI,eAAe,CAAC,MAAM,EAAE,KAAK,cAAc,CAAC,CAAC;AAGrD,MAAI,IAAI,eAAe,CAAC,MAAM,EAAE,KAAK,kBAAkB,CAAC,CAAC;AAGzD,MAAI,KAAK,aAAa,OAAO,MAAM;AACjC,UAAM,MAAM,MAAM,EAAE,IAAI,KAAK,EAAE,MAAM,MAAM,IAAI;AAC/C,QAAI,OAAO,MAAM;AACf,YAAM,IAAI,UAAU,oBAAoB,8BAA8B,GAAG;AAAA,IAC3E;AACA,UAAM,SAAS,mBAAmB,UAAU,GAAG;AAC/C,QAAI,CAAC,OAAO,SAAS;AACnB,YAAM,IAAI;AAAA,QACR;AAAA,QACA;AAAA,QACA;AAAA,QACA,OAAO,MAAM;AAAA,MACf;AAAA,IACF;AACA,UAAM,SAAS,MAAM,YAAY,OAAO,IAAI;AAC5C,WAAO,EAAE,KAAK,MAAM;AAAA,EACtB,CAAC;AAGD,MAAI,KAAK,qBAAqB,OAAO,MAAM;AACzC,UAAM,cAAc,EAAE,IAAI,OAAO,cAAc,KAAK;AACpD,QAAI;AACJ,QAAI,YAAY,SAAS,sBAAsB,GAAG;AAChD,YAAM,OAAO,MAAM,EAAE,IAAI,KAAK;AAC9B,YAAM,SAAS,KACZ,MAAM,IAAI,EACV,IAAI,CAAC,SAAS,KAAK,KAAK,CAAC,EACzB,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,EAChC,IAAI,CAAC,SAAS;AACb,YAAI;AACF,iBAAO,KAAK,MAAM,IAAI;AAAA,QACxB,QAAQ;AACN,gBAAM,IAAI;AAAA,YACR;AAAA,YACA;AAAA,YACA;AAAA,YACA,KAAK,MAAM,GAAG,GAAG;AAAA,UACnB;AAAA,QACF;AAAA,MACF,CAAC;AACH,gBAAU,EAAE,OAAO;AAAA,IACrB,OAAO;AACL,gBAAU,MAAM,EAAE,IAAI,KAAK,EAAE,MAAM,MAAM,IAAI;AAAA,IAC/C;AACA,QAAI,WAAW,MAAM;AACnB,YAAM,IAAI,UAAU,oBAAoB,wCAAwC,GAAG;AAAA,IACrF;AACA,UAAM,SAAS,0BAA0B,UAAU,OAAO;AAC1D,QAAI,CAAC,OAAO,SAAS;AACnB,YAAM,IAAI;AAAA,QACR;AAAA,QACA;AAAA,QACA;AAAA,QACA,OAAO,MAAM;AAAA,MACf;AAAA,IACF;AACA,UAAM,SAAS,MAAM,mBAAmB,OAAO,MAAM,KAAK,UAAU,CAAC,CAAC;AACtE,WAAO,EAAE,KAAK,MAAM;AAAA,EACtB,CAAC;AAGD,MAAI,KAAK,gBAAgB,OAAO,MAAM;AACpC,UAAM,MAAM,MAAM,EAAE,IAAI,KAAK,EAAE,MAAM,MAAM,IAAI;AAC/C,QAAI,OAAO,MAAM;AACf,YAAM,IAAI,UAAU,oBAAoB,8BAA8B,GAAG;AAAA,IAC3E;AACA,UAAM,SAAS,yBAAyB,UAAU,GAAG;AACrD,QAAI,CAAC,OAAO,SAAS;AACnB,YAAM,IAAI;AAAA,QACR;AAAA,QACA;AAAA,QACA;AAAA,QACA,OAAO,MAAM;AAAA,MACf;AAAA,IACF;AACA,UAAM,SAAS,MAAM,qBAAqB,OAAO,MAAM,KAAK,UAAU,CAAC,CAAC;AACxE,WAAO,EAAE,KAAK,MAAM;AAAA,EACtB,CAAC;AAGD,MAAI,IAAI,iBAAiB,CAAC,MAAM,EAAE,KAAK,aAAa,cAAc,EAAE,OAAO,CAAC,CAAC;AAE7E,SAAO;AACT;AASO,SAAS,YAAY,OAAqB,CAAC,GAAe;AAC/D,QAAM,MAAM,UAAU,IAAI;AAC1B,QAAM,OAAO,KAAK,QAAQ;AAC1B,QAAM,OAAO,KAAK,QAAQ;AAC1B,SAAO,MAAM,EAAE,OAAO,IAAI,OAAO,MAAM,UAAU,KAAK,GAAG,CAAC,EAAE,SAAS,MAAM,WAAW,MAAM;AAE1F,YAAQ,IAAI,kCAAkC,OAAO,IAAI,UAAU,EAAE;AAAA,EACvE,CAAC;AACH;AAmBO,SAAS,iBAAiB,OAAqB,CAAC,GAA2B;AAChF,QAAM,MAAM,UAAU,IAAI;AAC1B,QAAM,OAAO,KAAK,QAAQ;AAC1B,QAAM,OAAO,KAAK,QAAQ;AAC1B,SAAO,IAAI,QAAQ,CAACC,UAAS,WAAW;AACtC,QAAI,UAAU;AACd,QAAI;AACJ,aAAS,MAAM,EAAE,OAAO,IAAI,OAAO,MAAM,UAAU,KAAK,GAAG,CAAC,EAAE,SAAS,MAAM,WAAW,MAAM;AAC5F,UAAI,QAAS;AACb,gBAAU;AAEV,cAAQ,IAAI,kCAAkC,OAAO,IAAI,UAAU,EAAE;AACrE,MAAAA,SAAQ;AAAA,QACN;AAAA,QACA,MAAM;AAAA,QACN,MAAM;AAAA,QACN,OAAO,MACL,IAAI,QAAc,CAAC,KAAK,QAAQ;AAC9B,iBAAQ,MAAM,CAAC,QAAS,MAAM,IAAI,GAAG,IAAI,IAAI,CAAE;AAAA,QACjD,CAAC;AAAA,MACL,CAAC;AAAA,IACH,CAAC;AACD,WAAO,GAAG,SAAS,CAAC,QAAQ;AAC1B,UAAI,QAAS;AACb,gBAAU;AACV,aAAO,GAAG;AAAA,IACZ,CAAC;AAAA,EACH,CAAC;AACH;","names":["result","resolve"]}
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
{"version":3,"sources":["../src/llm-client.ts"],"sourcesContent":["/**\n * LLM client with graceful degrade.\n *\n * OpenAI-compatible `/v1/chat/completions` client with:\n * - Exponential-backoff retry on 429 + 5xx gateway errors (502/503/504).\n * - Retry on transient network errors (fetch failed, AbortError, ECONNRESET).\n * - Graceful json_schema → json_object degrade on 400 with schema-reject body.\n * - Fenced-JSON stripping (```json ... ```) for models that wrap structured output.\n * - Configurable base URL + api key / bearer, works with LiteLLM proxies, OpenAI\n * directly, cli-bridge subscriptions, and any router that speaks the spec.\n *\n * Usage:\n * const { value, result } = await callLlmJson<MyType>(\n * { model: 'gpt-4o', messages: [...], jsonSchema: { name: 'x', schema: {...} } },\n * { baseUrl: 'https://router.tangle.tools/v1', apiKey: process.env.KEY },\n * )\n *\n * This is THE llm-calling seam for agent-eval primitives that need structured\n * output (semantic concept judge, reviewer directives, critic scores). Primitives\n * that need free-form text use `callLlm` and parse output themselves.\n */\n\nimport type { CostReceiptInput, MaximumCharge } from './cost-ledger'\nimport { AgentEvalError, CaptureIntegrityError } from './errors'\nimport {\n defaultProviderRedactor,\n type ProviderRedactor,\n providerFromBaseUrl,\n type RawProviderEvent,\n type RawProviderSink,\n} from './trace/raw-provider-sink'\n\n// ─── Types ──────────────────────────────────────────────────────────────\n\nexport interface LlmMessage {\n role: 'system' | 'user' | 'assistant'\n /**\n * Either a plain text content string OR a multimodal content array\n * (text + image_url parts) for vision-capable models.\n */\n content:\n | string\n | Array<\n | { type: 'text'; text: string }\n | { type: 'image_url'; image_url: { url: string; detail?: 'auto' | 'low' | 'high' } }\n >\n}\n\nexport interface LlmCallRequest {\n model: string\n messages: LlmMessage[]\n /** Optional JSON-mode response format (response_format: json_object). */\n jsonMode?: boolean\n /** Optional structured output via JSON Schema. Falls back to json_object on 400. */\n jsonSchema?: { name: string; schema: Record<string, unknown> }\n temperature?: number\n maxTokens?: number\n /** Per-call timeout, default 300s. */\n timeoutMs?: number\n}\n\n/** Conservative priced bound for the exact text request sent to a provider.\n * Returns undefined when output or multimodal input is not bounded, causing a\n * capped CostLedger to reject the call before execution. */\nexport function maximumChargeForLlmRequest(\n request: Pick<LlmCallRequest, 'model' | 'messages' | 'jsonSchema' | 'maxTokens'>,\n options: LlmClientOptions = {},\n): MaximumCharge | undefined {\n if (request.maxTokens === undefined) return undefined\n if (!Number.isInteger(request.maxTokens) || request.maxTokens <= 0) {\n throw new RangeError(`maximumChargeForLlmRequest: maxTokens must be a positive integer`)\n }\n if (\n request.messages.some(\n (message) =>\n Array.isArray(message.content) && message.content.some((part) => part.type === 'image_url'),\n )\n ) {\n return undefined\n }\n\n const attempts = resolveMaximumAttempts(options.maxRetries)\n // A byte-level tokenizer cannot emit more input tokens than request bytes.\n // Pricing the complete body also covers role/schema framing omitted from content-only estimates.\n const requestBytes = new TextEncoder().encode(\n JSON.stringify(buildBody(request, false)),\n ).byteLength\n // A rejected response schema can trigger one JSON-mode batch with the same output limit.\n const batches = request.jsonSchema ? 2 : 1\n return {\n model: request.model,\n inputTokens: requestBytes * attempts * batches,\n outputTokens: request.maxTokens * attempts * batches,\n }\n}\n\nexport interface LlmUsage {\n promptTokens: number\n completionTokens: number\n totalTokens: number\n /** False when the provider omitted or malformed prompt/completion usage. */\n captured?: boolean\n /** Proxies populate this when prompt caching is on. */\n cachedPromptTokens?: number\n}\n\nexport interface LlmCallResult {\n /** The text content of the first choice. Empty string if none. */\n content: string\n usage: LlmUsage\n /**\n * Cost in USD. Pulled from proxy's `_response_cost` field when present;\n * `null` when neither the proxy nor the caller can derive it.\n */\n costUsd: number | null\n /** Model name actually used (echoed from response). */\n model: string\n /** Wall-clock duration of the HTTP call (last attempt, if retried). */\n durationMs: number\n /**\n * `finish_reason` echoed from the first choice (`stop`, `length`,\n * `content_filter`, `tool_calls`, ...). `null` when the provider omits it.\n * Exposed so a free-form `callLlm` caller CAN detect a truncated answer\n * (`length`) instead of treating a cut-off completion as complete. Note:\n * `callLlm` does not itself reject on it — acting on this signal is the\n * caller's responsibility (in-repo free-form drivers do not yet enforce it).\n */\n finishReason?: string | null\n /**\n * True when `content.trim()` is empty. An empty completion is a silent zero\n * for free-form `callLlm` callers; this flag is the signal a caller can\n * inspect to fail loud rather than proceed on an empty string. `callLlm`\n * surfaces it but does not throw on it.\n */\n contentEmpty?: boolean\n /** Raw response body. */\n raw: Record<string, unknown>\n}\n\nexport type LlmCallMetadata = Pick<LlmCallResult, 'usage' | 'costUsd' | 'model' | 'durationMs'>\n\n/** Convert a provider result into the canonical paid-call receipt input. */\nexport function costReceiptFromLlm(result: LlmCallResult): CostReceiptInput {\n const cachedTokens = result.usage.cachedPromptTokens ?? 0\n return {\n model: result.model,\n inputTokens: Math.max(0, result.usage.promptTokens - cachedTokens),\n outputTokens: result.usage.completionTokens,\n cachedTokens: cachedTokens > 0 ? cachedTokens : undefined,\n actualCostUsd: result.costUsd ?? undefined,\n usageUnknown: result.usage.captured === false,\n }\n}\n\n/** Structured-response failures retain their completed provider receipt. */\nexport function costReceiptFromLlmError(error: Error): CostReceiptInput | undefined {\n return error instanceof LlmResponseError ? costReceiptFromLlm(error.result) : undefined\n}\n\nexport class LlmCallError extends AgentEvalError {\n constructor(\n message: string,\n public readonly status: number,\n public readonly body: string,\n public readonly model: string,\n ) {\n super('judge', message)\n }\n}\n\n/** A provider response completed and incurred measurable usage, but its content\n * could not satisfy the caller's response contract. The response envelope is\n * retained so accounting can commit the receipt before the error propagates. */\nexport class LlmResponseError extends AgentEvalError {\n constructor(\n message: string,\n public readonly result: LlmCallResult,\n options?: { cause?: unknown },\n ) {\n super('judge', message, options)\n }\n}\n\nexport interface LlmClientOptions {\n /** Base URL (without trailing slash). Must end at the `/v1` prefix. */\n baseUrl?: string\n /** Bearer token — either `apiKey` or `bearer` populates `Authorization: Bearer ...`. */\n apiKey?: string\n bearer?: string\n /** Override for the `Authorization` header (e.g. `X-Auth: ...`). Takes precedence over apiKey/bearer. */\n authHeader?: { name: string; value: string }\n /** Stable provider idempotency key, reused across retries of this logical call. */\n idempotencyKey?: string\n /** Default timeout in ms. Per-call can override. */\n defaultTimeoutMs?: number\n /**\n * Caller-supplied abort signal — e.g. a campaign-wide cancel. Linked to\n * each attempt's per-attempt timeout controller, so aborting it cancels\n * the in-flight fetch. A caller abort is FATAL: it is not retried even\n * though an AbortError otherwise matches the transient patterns.\n */\n signal?: AbortSignal\n /**\n * Cross-attempt wall-clock budget in ms, measured from the first attempt.\n * Before launching each attempt the loop checks the remaining budget and\n * stops retrying once it is exhausted, rather than waiting the full\n * per-attempt timeout on every retry. Bounds total time independent of\n * total attempts × `timeoutMs`.\n */\n deadlineMs?: number\n /** Total provider attempts. Legacy option name; default 3 (1 initial + 2 retries). */\n maxRetries?: number\n /** Fetch implementation — defaults to global `fetch`. Override for custom transport (e.g. tests). */\n fetch?: typeof fetch\n /**\n * Optional raw HTTP capture sink. When provided, every request, response,\n * and error (across all retry attempts) is recorded to the sink, with auth\n * headers and credential-shaped body fields redacted by default. This is\n * the layer-1 forensics primitive: structured `LlmSpan`s record intent,\n * raw events record what actually crossed the wire.\n */\n rawSink?: RawProviderSink\n /**\n * Logical provider id attached to raw events. When omitted, derived from\n * `baseUrl` via `providerFromBaseUrl`.\n */\n provider?: string\n /** Trace context attached to raw events; populated by emitter-aware callers. */\n traceContext?: { runId?: string; spanId?: string }\n /** Override the redaction strategy for this call. Defaults to `defaultProviderRedactor`. */\n redactor?: ProviderRedactor\n}\n\n// ─── Internals ──────────────────────────────────────────────────────────\n\nconst DEFAULT_BASE_URL = 'https://router.tangle.tools/v1'\n// Flagship / reasoning models routinely take several minutes on large prompts (a\n// reflection over many failures, a long tool transcript). A tight cap aborts a\n// legitimately-slow but healthy call — and because every retry attempt re-uses\n// the same window, such a model aborts on ALL attempts and the loop throws. The\n// default is generous enough to let those complete, bounded enough that a truly\n// hung call still fails over after retries, and tunable per deployment via\n// TANGLE_LLM_TIMEOUT_MS. Per-call `req.timeoutMs` / `opts.defaultTimeoutMs`\n// still win for callers that know their model's latency.\nconst DEFAULT_TIMEOUT_MS = Number(process.env.TANGLE_LLM_TIMEOUT_MS) || 300_000\nconst DEFAULT_MAX_RETRIES = Number(process.env.TANGLE_LLM_MAX_RETRIES) || 3\n\nfunction resolveMaximumAttempts(configured: number | undefined): number {\n const attempts = configured ?? DEFAULT_MAX_RETRIES\n if (!Number.isInteger(attempts) || attempts <= 0) {\n throw new RangeError('LLM maximum attempts must be a positive integer')\n }\n return attempts\n}\n\nfunction providerTokenCount(value: unknown): number | undefined {\n return typeof value === 'number' && Number.isSafeInteger(value) && value >= 0 ? value : undefined\n}\n\nconst RETRYABLE_STATUS = new Set([429, 502, 503, 504])\n\n/**\n * Transient transport/network error signatures, matched against an error's\n * name, message, and `code`. Covers fetch/undici network failures, aborts\n * and timeouts, and — critically — HTTP/2 transport faults a keep-alive\n * connection raises mid-response: `terminated`, `NGHTTP2_INTERNAL_ERROR`,\n * `UND_ERR_*`, `other side closed`. Those last ones carry no clean HTTP\n * status; unrecognised, they escape the retry loop and surface as an\n * uncaught rejection.\n */\nconst TRANSIENT_ERROR_PATTERNS: readonly RegExp[] = [\n /AbortError/i,\n /TimeoutError/i,\n /this operation was aborted/i,\n /fetch failed/i,\n /ECONNRESET/i,\n /ETIMEDOUT/i,\n /EAI_AGAIN/i,\n /socket hang up/i,\n /stream.*ended.*unexpectedly/i,\n /terminated/i,\n /other side closed/i,\n /NGHTTP2/i,\n /UND_ERR/i,\n]\n\n/**\n * True when an error is a transient transport/network fault worth retrying,\n * as opposed to a deterministic failure (4xx schema reject, JSON parse) that\n * a retry cannot fix. Inspects `LlmCallError.status`, then the error's\n * name/message/code, then recurses into `error.cause` — undici nests the\n * real socket fault one or more levels under `.cause`.\n *\n * This is THE retry classifier for the package: `callLlm` and\n * `withJudgeRetry` both route through it, so a connection-class error is\n * treated identically whether it surfaces in the HTTP client or a\n * TCloud-backed judge.\n */\nexport function isTransientLlmError(err: unknown): boolean {\n return classifyTransient(err, 0)\n}\n\nfunction classifyTransient(err: unknown, depth: number): boolean {\n if (err instanceof LlmCallError) return RETRYABLE_STATUS.has(err.status)\n if (!(err instanceof Error)) return false\n // Foreign errors (e.g. a TCloud judge SDK error) can carry a numeric HTTP\n // status without being an LlmCallError — a retryable status is decisive.\n const status = (err as { status?: unknown }).status\n if (typeof status === 'number' && RETRYABLE_STATUS.has(status)) return true\n const code = (err as { code?: unknown }).code\n const haystack = `${err.name}\\n${err.message}\\n${typeof code === 'string' ? code : ''}`\n if (TRANSIENT_ERROR_PATTERNS.some((p) => p.test(haystack))) return true\n const cause = (err as { cause?: unknown }).cause\n if (depth < 4 && cause instanceof Error && cause !== err) {\n return classifyTransient(cause, depth + 1)\n }\n return false\n}\n\nfunction parseRetryAfter(headers: Headers): number | null {\n const h = headers.get('retry-after')\n if (!h) return null\n const asNumber = Number(h)\n if (Number.isFinite(asNumber) && asNumber > 0) return asNumber * 1000\n const asDate = Date.parse(h)\n if (Number.isFinite(asDate)) return Math.max(0, asDate - Date.now())\n return null\n}\n\n/** Exponential backoff: 500ms, 1s, 2s, 4s, ... capped at 16s. Attempt is 0-indexed. */\nexport function backoffMs(attempt: number): number {\n return Math.min(500 * 2 ** attempt, 16_000)\n}\n\nfunction buildHeaders(opts: LlmClientOptions): Record<string, string> {\n const headers: Record<string, string> = {\n 'Content-Type': 'application/json',\n Accept: 'application/json',\n }\n if (opts.authHeader) {\n headers[opts.authHeader.name] = opts.authHeader.value\n } else if (opts.bearer || opts.apiKey) {\n headers.Authorization = `Bearer ${opts.bearer ?? opts.apiKey}`\n }\n if (opts.idempotencyKey) headers['Idempotency-Key'] = opts.idempotencyKey\n return headers\n}\n\nfunction isSchemaRejection(status: number, body: string): boolean {\n if (status !== 400) return false\n const lower = body.toLowerCase()\n return (\n lower.includes('response_format') ||\n lower.includes('json_schema') ||\n lower.includes('is unavailable') ||\n lower.includes('not supported')\n )\n}\n\nfunction buildBody(req: LlmCallRequest, forceJsonObject: boolean): Record<string, unknown> {\n const body: Record<string, unknown> = {\n model: req.model,\n messages: req.messages,\n temperature: req.temperature ?? 0,\n }\n if (req.maxTokens != null) {\n if (usesMaxCompletionTokens(req.model)) body.max_completion_tokens = req.maxTokens\n else body.max_tokens = req.maxTokens\n }\n\n if (req.jsonSchema && !forceJsonObject) {\n body.response_format = {\n type: 'json_schema',\n json_schema: { name: req.jsonSchema.name, schema: req.jsonSchema.schema, strict: true },\n }\n } else if (req.jsonMode || req.jsonSchema) {\n body.response_format = { type: 'json_object' }\n }\n\n return body\n}\n\nfunction usesMaxCompletionTokens(model: string): boolean {\n return /^gpt-5(?:[.-]|$)/i.test(model)\n}\n\nasync function sleep(ms: number): Promise<void> {\n return new Promise((resolve) => setTimeout(resolve, ms))\n}\n\n/**\n * Combine the per-attempt timeout signal with an optional caller signal into\n * one signal the fetch listens on. Prefers the native `AbortSignal.any`; falls\n * back to manual wiring on runtimes that predate it. The caller signal is also\n * propagated to the timeout controller so aborting it cancels the in-flight\n * fetch immediately.\n */\nfunction linkSignals(timeoutController: AbortController, caller?: AbortSignal): AbortSignal {\n if (!caller) return timeoutController.signal\n if (typeof (AbortSignal as { any?: unknown }).any === 'function') {\n return AbortSignal.any([timeoutController.signal, caller])\n }\n if (caller.aborted) {\n timeoutController.abort()\n } else {\n caller.addEventListener('abort', () => timeoutController.abort(), { once: true })\n }\n return timeoutController.signal\n}\n\n/** True once the cross-attempt wall-clock budget (if any) is exhausted. */\nfunction deadlineExceeded(start: number, deadlineMs: number | undefined): boolean {\n return deadlineMs != null && Date.now() - start >= deadlineMs\n}\n\n// ─── Public API ─────────────────────────────────────────────────────────\n\n/**\n * Strip a ```json / ``` code fence if the model emitted one.\n * Idempotent for naked JSON. Some models (claude-code via router, certain\n * deepseek models) wrap output even under json_object.\n */\nexport function stripFencedJson(raw: string): string {\n const trimmed = raw.trim()\n const m = trimmed.match(/^```(?:json)?\\s*\\n?([\\s\\S]*?)\\n?```\\s*$/)\n return m ? m[1]!.trim() : trimmed\n}\n\nexport function extractJsonPayload(raw: string): string {\n const stripped = stripFencedJson(raw)\n try {\n JSON.parse(stripped)\n return stripped\n } catch {\n // A response that declares a JSON root must parse as that complete root.\n // Scanning onward could turn a truncated object into one of its valid nested\n // arrays or objects and silently change the response schema.\n if (stripped.startsWith('{') || stripped.startsWith('[')) return stripped\n }\n\n // Only prose-leading responses may contain a recoverable JSON payload.\n const starts = [...stripped.matchAll(/[[{]/g)]\n .map((match) => match.index)\n .filter((index) => index != null)\n for (const start of starts) {\n const candidate = extractBalancedJson(stripped, start)\n if (!candidate) continue\n try {\n JSON.parse(candidate)\n return candidate\n } catch {\n // Keep scanning; earlier braces may belong to prose.\n }\n }\n\n return stripped\n}\n\nfunction extractBalancedJson(input: string, start: number): string | null {\n const opener = input[start]\n const closer = opener === '{' ? '}' : opener === '[' ? ']' : null\n if (!closer) return null\n\n const stack: string[] = [closer]\n let isInString = false\n let isEscaped = false\n\n for (let i = start + 1; i < input.length; i++) {\n const char = input[i]!\n if (isEscaped) {\n isEscaped = false\n continue\n }\n if (char === '\\\\') {\n isEscaped = isInString\n continue\n }\n if (char === '\"') {\n isInString = !isInString\n continue\n }\n if (isInString) continue\n\n if (char === '{') stack.push('}')\n else if (char === '[') stack.push(']')\n else if (char === stack[stack.length - 1]) {\n stack.pop()\n if (stack.length === 0) return input.slice(start, i + 1)\n }\n }\n\n return null\n}\n\n/**\n * Low-level call. Returns raw content + usage + cost. Retries on transient\n * failures; does NOT degrade schema here — callers that want graceful\n * degrade use `callLlmJson`.\n */\nexport async function callLlm(\n req: LlmCallRequest,\n opts: LlmClientOptions = {},\n): Promise<LlmCallResult> {\n const baseUrl = (opts.baseUrl ?? DEFAULT_BASE_URL).replace(/\\/+$/, '')\n const url = `${baseUrl}/chat/completions`\n const endpoint = '/chat/completions'\n const timeoutMs = req.timeoutMs ?? opts.defaultTimeoutMs ?? DEFAULT_TIMEOUT_MS\n const maximumAttempts = resolveMaximumAttempts(opts.maxRetries)\n const fetchFn = opts.fetch ?? globalThis.fetch\n const headers = buildHeaders(opts)\n const provider = opts.provider ?? providerFromBaseUrl(baseUrl)\n const sink = opts.rawSink\n const redactor = opts.redactor ?? defaultProviderRedactor\n const traceContext = opts.traceContext\n const callerSignal = opts.signal\n const deadlineMs = opts.deadlineMs\n const deadlineStart = Date.now()\n\n let lastErr: unknown\n for (let attempt = 0; attempt < maximumAttempts; attempt++) {\n // A caller cancel is fatal — never retried. Checking before each attempt\n // means an already-aborted signal short-circuits without firing fetch.\n if (callerSignal?.aborted) {\n throw new DOMException('callLlm aborted by caller signal', 'AbortError')\n }\n // Stop retrying once the cross-attempt budget is spent rather than burning\n // a full per-attempt timeout on each remaining retry.\n if (attempt > 0 && deadlineExceeded(deadlineStart, deadlineMs)) {\n throw lastErr instanceof Error ? lastErr : new Error(String(lastErr))\n }\n const controller = new AbortController()\n const attemptSignal = linkSignals(controller, callerSignal)\n const timeoutHandle = setTimeout(() => controller.abort(), timeoutMs)\n const started = Date.now()\n const requestBody = buildBody(req, false)\n let attemptErrorRecorded = false\n if (sink) {\n await recordRaw(sink, redactor, {\n eventId: cryptoEventId(),\n runId: traceContext?.runId,\n spanId: traceContext?.spanId,\n provider,\n model: req.model,\n endpoint,\n baseUrl,\n attemptIndex: attempt,\n direction: 'request',\n timestamp: started,\n requestHeaders: headers,\n requestBody,\n redactedFields: [],\n })\n }\n\n try {\n const res = await fetchFn(url, {\n method: 'POST',\n headers,\n body: JSON.stringify(requestBody),\n signal: attemptSignal,\n })\n clearTimeout(timeoutHandle)\n const responseHeaders = sink ? headersToObject(res.headers) : undefined\n\n if (!res.ok) {\n const body = await res.text()\n if (sink) {\n await recordRaw(sink, redactor, {\n eventId: cryptoEventId(),\n runId: traceContext?.runId,\n spanId: traceContext?.spanId,\n provider,\n model: req.model,\n endpoint,\n baseUrl,\n attemptIndex: attempt,\n direction: 'error',\n timestamp: Date.now(),\n durationMs: Date.now() - started,\n statusCode: res.status,\n responseHeaders,\n responseBody: body,\n errorMessage: `HTTP ${res.status}`,\n redactedFields: [],\n })\n attemptErrorRecorded = true\n }\n const err = new LlmCallError(\n `LLM call ${res.status}: ${body.slice(0, 300)}`,\n res.status,\n body,\n req.model,\n )\n if (\n RETRYABLE_STATUS.has(res.status) &&\n attempt < maximumAttempts - 1 &&\n !deadlineExceeded(deadlineStart, deadlineMs)\n ) {\n lastErr = err\n const retryAfter = parseRetryAfter(res.headers)\n await sleep(retryAfter ?? backoffMs(attempt))\n continue\n }\n throw err\n }\n\n const text = await res.text()\n let json: Record<string, unknown>\n try {\n json = JSON.parse(text) as Record<string, unknown>\n } catch (parseErr) {\n if (sink) {\n await recordRaw(sink, redactor, {\n eventId: cryptoEventId(),\n runId: traceContext?.runId,\n spanId: traceContext?.spanId,\n provider,\n model: req.model,\n endpoint,\n baseUrl,\n attemptIndex: attempt,\n direction: 'error',\n timestamp: Date.now(),\n durationMs: Date.now() - started,\n statusCode: res.status,\n responseHeaders,\n responseBody: text,\n errorMessage: `non-JSON response: ${parseErr instanceof Error ? parseErr.message : String(parseErr)}`,\n redactedFields: [],\n })\n attemptErrorRecorded = true\n }\n throw parseErr\n }\n if (sink) {\n await recordRaw(sink, redactor, {\n eventId: cryptoEventId(),\n runId: traceContext?.runId,\n spanId: traceContext?.spanId,\n provider,\n model: req.model,\n endpoint,\n baseUrl,\n attemptIndex: attempt,\n direction: 'response',\n timestamp: Date.now(),\n durationMs: Date.now() - started,\n statusCode: res.status,\n responseHeaders,\n responseBody: json,\n redactedFields: [],\n })\n }\n const choice = (\n json.choices as\n | Array<{ message?: { content?: string }; finish_reason?: string | null }>\n | undefined\n )?.[0]\n const usageRaw =\n json.usage && typeof json.usage === 'object' && !Array.isArray(json.usage)\n ? (json.usage as Record<string, unknown>)\n : undefined\n const promptTokens = providerTokenCount(usageRaw?.prompt_tokens)\n const completionTokens = providerTokenCount(usageRaw?.completion_tokens)\n const totalTokens = providerTokenCount(usageRaw?.total_tokens)\n const cachedRaw =\n usageRaw?.prompt_tokens_details &&\n typeof usageRaw.prompt_tokens_details === 'object' &&\n !Array.isArray(usageRaw.prompt_tokens_details)\n ? (usageRaw.prompt_tokens_details as Record<string, unknown>).cached_tokens\n : undefined\n const cachedPromptTokens = cachedRaw === undefined ? undefined : providerTokenCount(cachedRaw)\n const usageCaptured =\n promptTokens !== undefined &&\n completionTokens !== undefined &&\n (cachedRaw === undefined ||\n (cachedPromptTokens !== undefined && cachedPromptTokens <= promptTokens)) &&\n (totalTokens === undefined || totalTokens === promptTokens + completionTokens)\n const costFromProxy = (json._response_cost ?? json.cost_usd) as number | undefined\n const content = choice?.message?.content ?? ''\n\n return {\n content,\n finishReason: choice?.finish_reason ?? null,\n contentEmpty: content.trim().length === 0,\n usage: {\n promptTokens: promptTokens ?? 0,\n completionTokens: completionTokens ?? 0,\n totalTokens: totalTokens ?? (promptTokens ?? 0) + (completionTokens ?? 0),\n captured: usageCaptured,\n cachedPromptTokens,\n },\n costUsd: typeof costFromProxy === 'number' ? costFromProxy : null,\n model: (json.model as string) ?? req.model,\n durationMs: Date.now() - started,\n raw: json,\n }\n } catch (err) {\n clearTimeout(timeoutHandle)\n lastErr = err\n // A caller cancel is fatal even though an AbortError matches the\n // transient patterns — a cancelled call must surface immediately, not\n // be retried against the same dead intent.\n if (callerSignal?.aborted) {\n if (sink && !attemptErrorRecorded) {\n await recordRaw(sink, redactor, {\n eventId: cryptoEventId(),\n runId: traceContext?.runId,\n spanId: traceContext?.spanId,\n provider,\n model: req.model,\n endpoint,\n baseUrl,\n attemptIndex: attempt,\n direction: 'error',\n timestamp: Date.now(),\n durationMs: Date.now() - started,\n errorMessage: err instanceof Error ? err.message : String(err),\n redactedFields: [],\n })\n }\n throw err\n }\n if (sink && !attemptErrorRecorded) {\n // Record only if neither the !res.ok branch nor the JSON.parse catch\n // already produced an error event for this attempt. Covers network\n // failures, timeouts, and aborts.\n await recordRaw(sink, redactor, {\n eventId: cryptoEventId(),\n runId: traceContext?.runId,\n spanId: traceContext?.spanId,\n provider,\n model: req.model,\n endpoint,\n baseUrl,\n attemptIndex: attempt,\n direction: 'error',\n timestamp: Date.now(),\n durationMs: Date.now() - started,\n errorMessage: err instanceof Error ? err.message : String(err),\n redactedFields: [],\n })\n }\n if (\n attempt < maximumAttempts - 1 &&\n isTransientLlmError(err) &&\n !deadlineExceeded(deadlineStart, deadlineMs)\n ) {\n await sleep(backoffMs(attempt))\n continue\n }\n throw err\n }\n }\n throw lastErr instanceof Error ? lastErr : new Error(String(lastErr))\n}\n\nasync function recordRaw(\n sink: RawProviderSink,\n redactor: ProviderRedactor,\n event: RawProviderEvent,\n): Promise<void> {\n // Errors from sinks must not crash the LLM call. Forensic capture is\n // best-effort; the structured trace is the system of record.\n try {\n await sink.record(redactor(event))\n } catch {\n // Intentionally swallowed.\n }\n}\n\nfunction headersToObject(h: Headers): Record<string, string> {\n const out: Record<string, string> = {}\n h.forEach((value, key) => {\n out[key] = value\n })\n return out\n}\n\nfunction cryptoEventId(): string {\n if (typeof globalThis.crypto?.randomUUID === 'function') return globalThis.crypto.randomUUID()\n return `${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 10)}`\n}\n\n/**\n * Structured-output call. Returns parsed JSON plus the raw result envelope.\n * Degrades `jsonSchema` → `jsonMode` on a 400 that names the schema param —\n * critical for deepseek-v3/v4, kimi-k2.6, and other models that don't accept\n * the `response_format.json_schema` shape but DO accept `json_object`.\n */\nexport async function callLlmJson<T = unknown>(\n req: LlmCallRequest,\n opts: LlmClientOptions = {},\n): Promise<{ value: T; result: LlmCallResult }> {\n const result = await callLlmStructured(req, opts)\n const value = parseJsonResult<T>(result)\n return { value, result }\n}\n\n/** Shared schema-to-JSON-mode fallback that preserves the raw result. */\nasync function callLlmStructured(\n req: LlmCallRequest,\n opts: LlmClientOptions = {},\n): Promise<LlmCallResult> {\n try {\n return await callLlm({ ...req, jsonMode: req.jsonMode ?? !req.jsonSchema }, opts)\n } catch (err) {\n if (err instanceof LlmCallError && isSchemaRejection(err.status, err.body) && req.jsonSchema) {\n const degradedReq: LlmCallRequest = { ...req, jsonMode: true, jsonSchema: undefined }\n return await callLlm(degradedReq, opts)\n }\n throw err\n }\n}\n\nfunction parseJsonResult<T>(result: LlmCallResult): T {\n try {\n if (result.finishReason === 'length') {\n throw new Error(\n `LLM returned truncated JSON content (model=${result.model}, finishReason=length)`,\n )\n }\n return parseJsonSafely<T>(result.content, result.model)\n } catch (error) {\n if (error instanceof LlmResponseError) throw error\n const cause = error instanceof Error ? error : new Error(String(error))\n throw new LlmResponseError(cause.message, result, { cause })\n }\n}\n\nfunction parseJsonSafely<T>(content: string, model: string): T {\n const stripped = extractJsonPayload(content)\n try {\n return JSON.parse(stripped) as T\n } catch (err) {\n throw new Error(\n `LLM returned non-JSON content (model=${model}): ${\n err instanceof Error ? err.message : String(err)\n }\\n--- raw content ---\\n${content.slice(0, 800)}`,\n )\n }\n}\n\n// ─── Route assertion ────────────────────────────────────────────────────\n\nexport type LlmRouteAssertionReason =\n | 'no_explicit_base_url'\n | 'base_url_blocked'\n | 'base_url_not_allowed'\n | 'no_auth'\n | 'wrong_provider'\n\nexport class LlmRouteAssertionError extends CaptureIntegrityError {\n constructor(\n message: string,\n public readonly reason: LlmRouteAssertionReason,\n public readonly baseUrl: string,\n ) {\n super(message)\n }\n}\n\nexport interface LlmRouteRequirements {\n /**\n * Throw if `opts.baseUrl` is undefined, i.e. the call would fall back to\n * `DEFAULT_BASE_URL`. Set this for evaluation runs where silently using\n * the public/free-tier router is a defect — the launch reviewer needs to\n * know exactly which provider answered.\n */\n requireExplicitBaseUrl?: boolean\n /**\n * Allowlist of acceptable base URLs. Strings match by prefix\n * (case-insensitive); RegExps test against the full base URL.\n */\n allowedBaseUrls?: Array<string | RegExp>\n /** Blocklist that takes precedence over `allowedBaseUrls`. */\n blockedBaseUrls?: Array<string | RegExp>\n /** Throw if no auth header / api key is configured. */\n requireAuth?: boolean\n /**\n * Logical provider id the configured `baseUrl` is expected to match (via\n * `providerFromBaseUrl`). Mainly useful when paired with `requireExplicitBaseUrl`.\n */\n expectedProvider?: string\n}\n\n/**\n * Fail-loud assertion that the configured LLM client points at the route\n * the caller intends. Designed for the matrix-runner preflight: invoke\n * once before any LLM call to catch misconfiguration before a sweep burns\n * dollars on the wrong provider.\n *\n * Throws `LlmRouteAssertionError`. Pure — no I/O — so it's safe to call\n * from constructors and CI gates.\n */\nexport function assertLlmRoute(opts: LlmClientOptions, req: LlmRouteRequirements = {}): void {\n const baseUrlExplicit = opts.baseUrl !== undefined\n const baseUrl = (opts.baseUrl ?? DEFAULT_BASE_URL).replace(/\\/+$/, '')\n\n if (req.requireExplicitBaseUrl && !baseUrlExplicit) {\n throw new LlmRouteAssertionError(\n `assertLlmRoute: requireExplicitBaseUrl set but opts.baseUrl is undefined; would fall back to ${DEFAULT_BASE_URL}.`,\n 'no_explicit_base_url',\n baseUrl,\n )\n }\n\n if (req.blockedBaseUrls?.some((p) => matchUrl(baseUrl, p))) {\n throw new LlmRouteAssertionError(\n `assertLlmRoute: baseUrl ${baseUrl} matches a blocked pattern.`,\n 'base_url_blocked',\n baseUrl,\n )\n }\n\n if (req.allowedBaseUrls && req.allowedBaseUrls.length > 0) {\n const ok = req.allowedBaseUrls.some((p) => matchUrl(baseUrl, p))\n if (!ok) {\n throw new LlmRouteAssertionError(\n `assertLlmRoute: baseUrl ${baseUrl} is not in the allowed list (${req.allowedBaseUrls.map(describePattern).join(', ')}).`,\n 'base_url_not_allowed',\n baseUrl,\n )\n }\n }\n\n if (req.requireAuth && !opts.apiKey && !opts.bearer && !opts.authHeader) {\n throw new LlmRouteAssertionError(\n `assertLlmRoute: requireAuth set but no apiKey, bearer, or authHeader was supplied.`,\n 'no_auth',\n baseUrl,\n )\n }\n\n if (req.expectedProvider) {\n const actual = opts.provider ?? providerFromBaseUrl(baseUrl)\n if (actual !== req.expectedProvider) {\n throw new LlmRouteAssertionError(\n `assertLlmRoute: expected provider ${req.expectedProvider} but baseUrl ${baseUrl} resolves to ${actual}.`,\n 'wrong_provider',\n baseUrl,\n )\n }\n }\n}\n\nfunction matchUrl(url: string, pattern: string | RegExp): boolean {\n if (pattern instanceof RegExp) return pattern.test(url)\n return url.toLowerCase().startsWith(pattern.toLowerCase())\n}\n\nfunction describePattern(p: string | RegExp): string {\n return p instanceof RegExp ? p.source : p\n}\n\n/**\n * Probe whether a model is reachable. Returns latency + null error on\n * success; `ok=false` + error message on any failure (HTTP, timeout,\n * network, parse). Designed for sweep preflights — fail loud at the\n * boundary before burning a 30-leaf run on a misconfigured router.\n *\n * Sends a tiny `ping` message with `maxTokens=64`. Reasoning models\n * (glm-5.1, deepseek-v4) can burn the entire budget on internal reasoning\n * for short prompts, so don't tighten this further. We don't validate\n * content; HTTP 200 means reachable.\n */\nexport async function probeLlm(\n model: string,\n opts: LlmClientOptions & { timeoutMs?: number } = {},\n): Promise<{ ok: boolean; latencyMs: number; error: string | null }> {\n const start = Date.now()\n try {\n await callLlm(\n {\n model,\n messages: [{ role: 'user', content: 'ping' }],\n maxTokens: 64,\n timeoutMs: opts.timeoutMs ?? 30_000,\n },\n opts,\n )\n return { ok: true, latencyMs: Date.now() - start, error: null }\n } catch (err) {\n return {\n ok: false,\n latencyMs: Date.now() - start,\n error: err instanceof Error ? err.message : String(err),\n }\n }\n}\n\n/**\n * Stateful client — construct once with defaults, call many times.\n * Thin wrapper around the free functions; exists for callers that want\n * to inject a single configured instance into multiple primitives.\n */\nexport class LlmClient {\n readonly maximumAttempts: number\n private readonly opts: LlmClientOptions\n\n constructor(opts: LlmClientOptions = {}) {\n this.opts = opts\n this.maximumAttempts = resolveMaximumAttempts(opts.maxRetries)\n }\n\n call(req: LlmCallRequest, per?: LlmClientOptions): Promise<LlmCallResult> {\n const options = { ...this.opts, ...per }\n return req.jsonSchema ? callLlmStructured(req, options) : callLlm(req, options)\n }\n\n callJson<T = unknown>(\n req: LlmCallRequest,\n per?: LlmClientOptions,\n ): Promise<{ value: T; result: LlmCallResult }> {\n return callLlmJson<T>(req, { ...this.opts, ...per })\n }\n}\n"],"mappings":";;;;;;;;;;AAgEO,SAAS,2BACd,SACA,UAA4B,CAAC,GACF;AAC3B,MAAI,QAAQ,cAAc,OAAW,QAAO;AAC5C,MAAI,CAAC,OAAO,UAAU,QAAQ,SAAS,KAAK,QAAQ,aAAa,GAAG;AAClE,UAAM,IAAI,WAAW,kEAAkE;AAAA,EACzF;AACA,MACE,QAAQ,SAAS;AAAA,IACf,CAAC,YACC,MAAM,QAAQ,QAAQ,OAAO,KAAK,QAAQ,QAAQ,KAAK,CAAC,SAAS,KAAK,SAAS,WAAW;AAAA,EAC9F,GACA;AACA,WAAO;AAAA,EACT;AAEA,QAAM,WAAW,uBAAuB,QAAQ,UAAU;AAG1D,QAAM,eAAe,IAAI,YAAY,EAAE;AAAA,IACrC,KAAK,UAAU,UAAU,SAAS,KAAK,CAAC;AAAA,EAC1C,EAAE;AAEF,QAAM,UAAU,QAAQ,aAAa,IAAI;AACzC,SAAO;AAAA,IACL,OAAO,QAAQ;AAAA,IACf,aAAa,eAAe,WAAW;AAAA,IACvC,cAAc,QAAQ,YAAY,WAAW;AAAA,EAC/C;AACF;AAgDO,SAAS,mBAAmB,QAAyC;AAC1E,QAAM,eAAe,OAAO,MAAM,sBAAsB;AACxD,SAAO;AAAA,IACL,OAAO,OAAO;AAAA,IACd,aAAa,KAAK,IAAI,GAAG,OAAO,MAAM,eAAe,YAAY;AAAA,IACjE,cAAc,OAAO,MAAM;AAAA,IAC3B,cAAc,eAAe,IAAI,eAAe;AAAA,IAChD,eAAe,OAAO,WAAW;AAAA,IACjC,cAAc,OAAO,MAAM,aAAa;AAAA,EAC1C;AACF;AAGO,SAAS,wBAAwB,OAA4C;AAClF,SAAO,iBAAiB,mBAAmB,mBAAmB,MAAM,MAAM,IAAI;AAChF;AAEO,IAAM,eAAN,cAA2B,eAAe;AAAA,EAC/C,YACE,SACgB,QACA,MACA,OAChB;AACA,UAAM,SAAS,OAAO;AAJN;AACA;AACA;AAAA,EAGlB;AAAA,EALkB;AAAA,EACA;AAAA,EACA;AAIpB;AAKO,IAAM,mBAAN,cAA+B,eAAe;AAAA,EACnD,YACE,SACgB,QAChB,SACA;AACA,UAAM,SAAS,SAAS,OAAO;AAHf;AAAA,EAIlB;AAAA,EAJkB;AAKpB;AAsDA,IAAM,mBAAmB;AASzB,IAAM,qBAAqB,OAAO,QAAQ,IAAI,qBAAqB,KAAK;AACxE,IAAM,sBAAsB,OAAO,QAAQ,IAAI,sBAAsB,KAAK;AAE1E,SAAS,uBAAuB,YAAwC;AACtE,QAAM,WAAW,cAAc;AAC/B,MAAI,CAAC,OAAO,UAAU,QAAQ,KAAK,YAAY,GAAG;AAChD,UAAM,IAAI,WAAW,iDAAiD;AAAA,EACxE;AACA,SAAO;AACT;AAEA,SAAS,mBAAmB,OAAoC;AAC9D,SAAO,OAAO,UAAU,YAAY,OAAO,cAAc,KAAK,KAAK,SAAS,IAAI,QAAQ;AAC1F;AAEA,IAAM,mBAAmB,oBAAI,IAAI,CAAC,KAAK,KAAK,KAAK,GAAG,CAAC;AAWrD,IAAM,2BAA8C;AAAA,EAClD;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AACF;AAcO,SAAS,oBAAoB,KAAuB;AACzD,SAAO,kBAAkB,KAAK,CAAC;AACjC;AAEA,SAAS,kBAAkB,KAAc,OAAwB;AAC/D,MAAI,eAAe,aAAc,QAAO,iBAAiB,IAAI,IAAI,MAAM;AACvE,MAAI,EAAE,eAAe,OAAQ,QAAO;AAGpC,QAAM,SAAU,IAA6B;AAC7C,MAAI,OAAO,WAAW,YAAY,iBAAiB,IAAI,MAAM,EAAG,QAAO;AACvE,QAAM,OAAQ,IAA2B;AACzC,QAAM,WAAW,GAAG,IAAI,IAAI;AAAA,EAAK,IAAI,OAAO;AAAA,EAAK,OAAO,SAAS,WAAW,OAAO,EAAE;AACrF,MAAI,yBAAyB,KAAK,CAAC,MAAM,EAAE,KAAK,QAAQ,CAAC,EAAG,QAAO;AACnE,QAAM,QAAS,IAA4B;AAC3C,MAAI,QAAQ,KAAK,iBAAiB,SAAS,UAAU,KAAK;AACxD,WAAO,kBAAkB,OAAO,QAAQ,CAAC;AAAA,EAC3C;AACA,SAAO;AACT;AAEA,SAAS,gBAAgB,SAAiC;AACxD,QAAM,IAAI,QAAQ,IAAI,aAAa;AACnC,MAAI,CAAC,EAAG,QAAO;AACf,QAAM,WAAW,OAAO,CAAC;AACzB,MAAI,OAAO,SAAS,QAAQ,KAAK,WAAW,EAAG,QAAO,WAAW;AACjE,QAAM,SAAS,KAAK,MAAM,CAAC;AAC3B,MAAI,OAAO,SAAS,MAAM,EAAG,QAAO,KAAK,IAAI,GAAG,SAAS,KAAK,IAAI,CAAC;AACnE,SAAO;AACT;AAGO,SAAS,UAAU,SAAyB;AACjD,SAAO,KAAK,IAAI,MAAM,KAAK,SAAS,IAAM;AAC5C;AAEA,SAAS,aAAa,MAAgD;AACpE,QAAM,UAAkC;AAAA,IACtC,gBAAgB;AAAA,IAChB,QAAQ;AAAA,EACV;AACA,MAAI,KAAK,YAAY;AACnB,YAAQ,KAAK,WAAW,IAAI,IAAI,KAAK,WAAW;AAAA,EAClD,WAAW,KAAK,UAAU,KAAK,QAAQ;AACrC,YAAQ,gBAAgB,UAAU,KAAK,UAAU,KAAK,MAAM;AAAA,EAC9D;AACA,MAAI,KAAK,eAAgB,SAAQ,iBAAiB,IAAI,KAAK;AAC3D,SAAO;AACT;AAEA,SAAS,kBAAkB,QAAgB,MAAuB;AAChE,MAAI,WAAW,IAAK,QAAO;AAC3B,QAAM,QAAQ,KAAK,YAAY;AAC/B,SACE,MAAM,SAAS,iBAAiB,KAChC,MAAM,SAAS,aAAa,KAC5B,MAAM,SAAS,gBAAgB,KAC/B,MAAM,SAAS,eAAe;AAElC;AAEA,SAAS,UAAU,KAAqB,iBAAmD;AACzF,QAAM,OAAgC;AAAA,IACpC,OAAO,IAAI;AAAA,IACX,UAAU,IAAI;AAAA,IACd,aAAa,IAAI,eAAe;AAAA,EAClC;AACA,MAAI,IAAI,aAAa,MAAM;AACzB,QAAI,wBAAwB,IAAI,KAAK,EAAG,MAAK,wBAAwB,IAAI;AAAA,QACpE,MAAK,aAAa,IAAI;AAAA,EAC7B;AAEA,MAAI,IAAI,cAAc,CAAC,iBAAiB;AACtC,SAAK,kBAAkB;AAAA,MACrB,MAAM;AAAA,MACN,aAAa,EAAE,MAAM,IAAI,WAAW,MAAM,QAAQ,IAAI,WAAW,QAAQ,QAAQ,KAAK;AAAA,IACxF;AAAA,EACF,WAAW,IAAI,YAAY,IAAI,YAAY;AACzC,SAAK,kBAAkB,EAAE,MAAM,cAAc;AAAA,EAC/C;AAEA,SAAO;AACT;AAEA,SAAS,wBAAwB,OAAwB;AACvD,SAAO,oBAAoB,KAAK,KAAK;AACvC;AAEA,eAAe,MAAM,IAA2B;AAC9C,SAAO,IAAI,QAAQ,CAAC,YAAY,WAAW,SAAS,EAAE,CAAC;AACzD;AASA,SAAS,YAAY,mBAAoC,QAAmC;AAC1F,MAAI,CAAC,OAAQ,QAAO,kBAAkB;AACtC,MAAI,OAAQ,YAAkC,QAAQ,YAAY;AAChE,WAAO,YAAY,IAAI,CAAC,kBAAkB,QAAQ,MAAM,CAAC;AAAA,EAC3D;AACA,MAAI,OAAO,SAAS;AAClB,sBAAkB,MAAM;AAAA,EAC1B,OAAO;AACL,WAAO,iBAAiB,SAAS,MAAM,kBAAkB,MAAM,GAAG,EAAE,MAAM,KAAK,CAAC;AAAA,EAClF;AACA,SAAO,kBAAkB;AAC3B;AAGA,SAAS,iBAAiB,OAAe,YAAyC;AAChF,SAAO,cAAc,QAAQ,KAAK,IAAI,IAAI,SAAS;AACrD;AASO,SAAS,gBAAgB,KAAqB;AACnD,QAAM,UAAU,IAAI,KAAK;AACzB,QAAM,IAAI,QAAQ,MAAM,yCAAyC;AACjE,SAAO,IAAI,EAAE,CAAC,EAAG,KAAK,IAAI;AAC5B;AAEO,SAAS,mBAAmB,KAAqB;AACtD,QAAM,WAAW,gBAAgB,GAAG;AACpC,MAAI;AACF,SAAK,MAAM,QAAQ;AACnB,WAAO;AAAA,EACT,QAAQ;AAIN,QAAI,SAAS,WAAW,GAAG,KAAK,SAAS,WAAW,GAAG,EAAG,QAAO;AAAA,EACnE;AAGA,QAAM,SAAS,CAAC,GAAG,SAAS,SAAS,OAAO,CAAC,EAC1C,IAAI,CAAC,UAAU,MAAM,KAAK,EAC1B,OAAO,CAAC,UAAU,SAAS,IAAI;AAClC,aAAW,SAAS,QAAQ;AAC1B,UAAM,YAAY,oBAAoB,UAAU,KAAK;AACrD,QAAI,CAAC,UAAW;AAChB,QAAI;AACF,WAAK,MAAM,SAAS;AACpB,aAAO;AAAA,IACT,QAAQ;AAAA,IAER;AAAA,EACF;AAEA,SAAO;AACT;AAEA,SAAS,oBAAoB,OAAe,OAA8B;AACxE,QAAM,SAAS,MAAM,KAAK;AAC1B,QAAM,SAAS,WAAW,MAAM,MAAM,WAAW,MAAM,MAAM;AAC7D,MAAI,CAAC,OAAQ,QAAO;AAEpB,QAAM,QAAkB,CAAC,MAAM;AAC/B,MAAI,aAAa;AACjB,MAAI,YAAY;AAEhB,WAAS,IAAI,QAAQ,GAAG,IAAI,MAAM,QAAQ,KAAK;AAC7C,UAAM,OAAO,MAAM,CAAC;AACpB,QAAI,WAAW;AACb,kBAAY;AACZ;AAAA,IACF;AACA,QAAI,SAAS,MAAM;AACjB,kBAAY;AACZ;AAAA,IACF;AACA,QAAI,SAAS,KAAK;AAChB,mBAAa,CAAC;AACd;AAAA,IACF;AACA,QAAI,WAAY;AAEhB,QAAI,SAAS,IAAK,OAAM,KAAK,GAAG;AAAA,aACvB,SAAS,IAAK,OAAM,KAAK,GAAG;AAAA,aAC5B,SAAS,MAAM,MAAM,SAAS,CAAC,GAAG;AACzC,YAAM,IAAI;AACV,UAAI,MAAM,WAAW,EAAG,QAAO,MAAM,MAAM,OAAO,IAAI,CAAC;AAAA,IACzD;AAAA,EACF;AAEA,SAAO;AACT;AAOA,eAAsB,QACpB,KACA,OAAyB,CAAC,GACF;AACxB,QAAM,WAAW,KAAK,WAAW,kBAAkB,QAAQ,QAAQ,EAAE;AACrE,QAAM,MAAM,GAAG,OAAO;AACtB,QAAM,WAAW;AACjB,QAAM,YAAY,IAAI,aAAa,KAAK,oBAAoB;AAC5D,QAAM,kBAAkB,uBAAuB,KAAK,UAAU;AAC9D,QAAM,UAAU,KAAK,SAAS,WAAW;AACzC,QAAM,UAAU,aAAa,IAAI;AACjC,QAAM,WAAW,KAAK,YAAY,oBAAoB,OAAO;AAC7D,QAAM,OAAO,KAAK;AAClB,QAAM,WAAW,KAAK,YAAY;AAClC,QAAM,eAAe,KAAK;AAC1B,QAAM,eAAe,KAAK;AAC1B,QAAM,aAAa,KAAK;AACxB,QAAM,gBAAgB,KAAK,IAAI;AAE/B,MAAI;AACJ,WAAS,UAAU,GAAG,UAAU,iBAAiB,WAAW;AAG1D,QAAI,cAAc,SAAS;AACzB,YAAM,IAAI,aAAa,oCAAoC,YAAY;AAAA,IACzE;AAGA,QAAI,UAAU,KAAK,iBAAiB,eAAe,UAAU,GAAG;AAC9D,YAAM,mBAAmB,QAAQ,UAAU,IAAI,MAAM,OAAO,OAAO,CAAC;AAAA,IACtE;AACA,UAAM,aAAa,IAAI,gBAAgB;AACvC,UAAM,gBAAgB,YAAY,YAAY,YAAY;AAC1D,UAAM,gBAAgB,WAAW,MAAM,WAAW,MAAM,GAAG,SAAS;AACpE,UAAM,UAAU,KAAK,IAAI;AACzB,UAAM,cAAc,UAAU,KAAK,KAAK;AACxC,QAAI,uBAAuB;AAC3B,QAAI,MAAM;AACR,YAAM,UAAU,MAAM,UAAU;AAAA,QAC9B,SAAS,cAAc;AAAA,QACvB,OAAO,cAAc;AAAA,QACrB,QAAQ,cAAc;AAAA,QACtB;AAAA,QACA,OAAO,IAAI;AAAA,QACX;AAAA,QACA;AAAA,QACA,cAAc;AAAA,QACd,WAAW;AAAA,QACX,WAAW;AAAA,QACX,gBAAgB;AAAA,QAChB;AAAA,QACA,gBAAgB,CAAC;AAAA,MACnB,CAAC;AAAA,IACH;AAEA,QAAI;AACF,YAAM,MAAM,MAAM,QAAQ,KAAK;AAAA,QAC7B,QAAQ;AAAA,QACR;AAAA,QACA,MAAM,KAAK,UAAU,WAAW;AAAA,QAChC,QAAQ;AAAA,MACV,CAAC;AACD,mBAAa,aAAa;AAC1B,YAAM,kBAAkB,OAAO,gBAAgB,IAAI,OAAO,IAAI;AAE9D,UAAI,CAAC,IAAI,IAAI;AACX,cAAM,OAAO,MAAM,IAAI,KAAK;AAC5B,YAAI,MAAM;AACR,gBAAM,UAAU,MAAM,UAAU;AAAA,YAC9B,SAAS,cAAc;AAAA,YACvB,OAAO,cAAc;AAAA,YACrB,QAAQ,cAAc;AAAA,YACtB;AAAA,YACA,OAAO,IAAI;AAAA,YACX;AAAA,YACA;AAAA,YACA,cAAc;AAAA,YACd,WAAW;AAAA,YACX,WAAW,KAAK,IAAI;AAAA,YACpB,YAAY,KAAK,IAAI,IAAI;AAAA,YACzB,YAAY,IAAI;AAAA,YAChB;AAAA,YACA,cAAc;AAAA,YACd,cAAc,QAAQ,IAAI,MAAM;AAAA,YAChC,gBAAgB,CAAC;AAAA,UACnB,CAAC;AACD,iCAAuB;AAAA,QACzB;AACA,cAAM,MAAM,IAAI;AAAA,UACd,YAAY,IAAI,MAAM,KAAK,KAAK,MAAM,GAAG,GAAG,CAAC;AAAA,UAC7C,IAAI;AAAA,UACJ;AAAA,UACA,IAAI;AAAA,QACN;AACA,YACE,iBAAiB,IAAI,IAAI,MAAM,KAC/B,UAAU,kBAAkB,KAC5B,CAAC,iBAAiB,eAAe,UAAU,GAC3C;AACA,oBAAU;AACV,gBAAM,aAAa,gBAAgB,IAAI,OAAO;AAC9C,gBAAM,MAAM,cAAc,UAAU,OAAO,CAAC;AAC5C;AAAA,QACF;AACA,cAAM;AAAA,MACR;AAEA,YAAM,OAAO,MAAM,IAAI,KAAK;AAC5B,UAAI;AACJ,UAAI;AACF,eAAO,KAAK,MAAM,IAAI;AAAA,MACxB,SAAS,UAAU;AACjB,YAAI,MAAM;AACR,gBAAM,UAAU,MAAM,UAAU;AAAA,YAC9B,SAAS,cAAc;AAAA,YACvB,OAAO,cAAc;AAAA,YACrB,QAAQ,cAAc;AAAA,YACtB;AAAA,YACA,OAAO,IAAI;AAAA,YACX;AAAA,YACA;AAAA,YACA,cAAc;AAAA,YACd,WAAW;AAAA,YACX,WAAW,KAAK,IAAI;AAAA,YACpB,YAAY,KAAK,IAAI,IAAI;AAAA,YACzB,YAAY,IAAI;AAAA,YAChB;AAAA,YACA,cAAc;AAAA,YACd,cAAc,sBAAsB,oBAAoB,QAAQ,SAAS,UAAU,OAAO,QAAQ,CAAC;AAAA,YACnG,gBAAgB,CAAC;AAAA,UACnB,CAAC;AACD,iCAAuB;AAAA,QACzB;AACA,cAAM;AAAA,MACR;AACA,UAAI,MAAM;AACR,cAAM,UAAU,MAAM,UAAU;AAAA,UAC9B,SAAS,cAAc;AAAA,UACvB,OAAO,cAAc;AAAA,UACrB,QAAQ,cAAc;AAAA,UACtB;AAAA,UACA,OAAO,IAAI;AAAA,UACX;AAAA,UACA;AAAA,UACA,cAAc;AAAA,UACd,WAAW;AAAA,UACX,WAAW,KAAK,IAAI;AAAA,UACpB,YAAY,KAAK,IAAI,IAAI;AAAA,UACzB,YAAY,IAAI;AAAA,UAChB;AAAA,UACA,cAAc;AAAA,UACd,gBAAgB,CAAC;AAAA,QACnB,CAAC;AAAA,MACH;AACA,YAAM,SACJ,KAAK,UAGH,CAAC;AACL,YAAM,WACJ,KAAK,SAAS,OAAO,KAAK,UAAU,YAAY,CAAC,MAAM,QAAQ,KAAK,KAAK,IACpE,KAAK,QACN;AACN,YAAM,eAAe,mBAAmB,UAAU,aAAa;AAC/D,YAAM,mBAAmB,mBAAmB,UAAU,iBAAiB;AACvE,YAAM,cAAc,mBAAmB,UAAU,YAAY;AAC7D,YAAM,YACJ,UAAU,yBACV,OAAO,SAAS,0BAA0B,YAC1C,CAAC,MAAM,QAAQ,SAAS,qBAAqB,IACxC,SAAS,sBAAkD,gBAC5D;AACN,YAAM,qBAAqB,cAAc,SAAY,SAAY,mBAAmB,SAAS;AAC7F,YAAM,gBACJ,iBAAiB,UACjB,qBAAqB,WACpB,cAAc,UACZ,uBAAuB,UAAa,sBAAsB,kBAC5D,gBAAgB,UAAa,gBAAgB,eAAe;AAC/D,YAAM,gBAAiB,KAAK,kBAAkB,KAAK;AACnD,YAAM,UAAU,QAAQ,SAAS,WAAW;AAE5C,aAAO;AAAA,QACL;AAAA,QACA,cAAc,QAAQ,iBAAiB;AAAA,QACvC,cAAc,QAAQ,KAAK,EAAE,WAAW;AAAA,QACxC,OAAO;AAAA,UACL,cAAc,gBAAgB;AAAA,UAC9B,kBAAkB,oBAAoB;AAAA,UACtC,aAAa,gBAAgB,gBAAgB,MAAM,oBAAoB;AAAA,UACvE,UAAU;AAAA,UACV;AAAA,QACF;AAAA,QACA,SAAS,OAAO,kBAAkB,WAAW,gBAAgB;AAAA,QAC7D,OAAQ,KAAK,SAAoB,IAAI;AAAA,QACrC,YAAY,KAAK,IAAI,IAAI;AAAA,QACzB,KAAK;AAAA,MACP;AAAA,IACF,SAAS,KAAK;AACZ,mBAAa,aAAa;AAC1B,gBAAU;AAIV,UAAI,cAAc,SAAS;AACzB,YAAI,QAAQ,CAAC,sBAAsB;AACjC,gBAAM,UAAU,MAAM,UAAU;AAAA,YAC9B,SAAS,cAAc;AAAA,YACvB,OAAO,cAAc;AAAA,YACrB,QAAQ,cAAc;AAAA,YACtB;AAAA,YACA,OAAO,IAAI;AAAA,YACX;AAAA,YACA;AAAA,YACA,cAAc;AAAA,YACd,WAAW;AAAA,YACX,WAAW,KAAK,IAAI;AAAA,YACpB,YAAY,KAAK,IAAI,IAAI;AAAA,YACzB,cAAc,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG;AAAA,YAC7D,gBAAgB,CAAC;AAAA,UACnB,CAAC;AAAA,QACH;AACA,cAAM;AAAA,MACR;AACA,UAAI,QAAQ,CAAC,sBAAsB;AAIjC,cAAM,UAAU,MAAM,UAAU;AAAA,UAC9B,SAAS,cAAc;AAAA,UACvB,OAAO,cAAc;AAAA,UACrB,QAAQ,cAAc;AAAA,UACtB;AAAA,UACA,OAAO,IAAI;AAAA,UACX;AAAA,UACA;AAAA,UACA,cAAc;AAAA,UACd,WAAW;AAAA,UACX,WAAW,KAAK,IAAI;AAAA,UACpB,YAAY,KAAK,IAAI,IAAI;AAAA,UACzB,cAAc,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG;AAAA,UAC7D,gBAAgB,CAAC;AAAA,QACnB,CAAC;AAAA,MACH;AACA,UACE,UAAU,kBAAkB,KAC5B,oBAAoB,GAAG,KACvB,CAAC,iBAAiB,eAAe,UAAU,GAC3C;AACA,cAAM,MAAM,UAAU,OAAO,CAAC;AAC9B;AAAA,MACF;AACA,YAAM;AAAA,IACR;AAAA,EACF;AACA,QAAM,mBAAmB,QAAQ,UAAU,IAAI,MAAM,OAAO,OAAO,CAAC;AACtE;AAEA,eAAe,UACb,MACA,UACA,OACe;AAGf,MAAI;AACF,UAAM,KAAK,OAAO,SAAS,KAAK,CAAC;AAAA,EACnC,QAAQ;AAAA,EAER;AACF;AAEA,SAAS,gBAAgB,GAAoC;AAC3D,QAAM,MAA8B,CAAC;AACrC,IAAE,QAAQ,CAAC,OAAO,QAAQ;AACxB,QAAI,GAAG,IAAI;AAAA,EACb,CAAC;AACD,SAAO;AACT;AAEA,SAAS,gBAAwB;AAC/B,MAAI,OAAO,WAAW,QAAQ,eAAe,WAAY,QAAO,WAAW,OAAO,WAAW;AAC7F,SAAO,GAAG,KAAK,IAAI,EAAE,SAAS,EAAE,CAAC,IAAI,KAAK,OAAO,EAAE,SAAS,EAAE,EAAE,MAAM,GAAG,EAAE,CAAC;AAC9E;AAQA,eAAsB,YACpB,KACA,OAAyB,CAAC,GACoB;AAC9C,QAAM,SAAS,MAAM,kBAAkB,KAAK,IAAI;AAChD,QAAM,QAAQ,gBAAmB,MAAM;AACvC,SAAO,EAAE,OAAO,OAAO;AACzB;AAGA,eAAe,kBACb,KACA,OAAyB,CAAC,GACF;AACxB,MAAI;AACF,WAAO,MAAM,QAAQ,EAAE,GAAG,KAAK,UAAU,IAAI,YAAY,CAAC,IAAI,WAAW,GAAG,IAAI;AAAA,EAClF,SAAS,KAAK;AACZ,QAAI,eAAe,gBAAgB,kBAAkB,IAAI,QAAQ,IAAI,IAAI,KAAK,IAAI,YAAY;AAC5F,YAAM,cAA8B,EAAE,GAAG,KAAK,UAAU,MAAM,YAAY,OAAU;AACpF,aAAO,MAAM,QAAQ,aAAa,IAAI;AAAA,IACxC;AACA,UAAM;AAAA,EACR;AACF;AAEA,SAAS,gBAAmB,QAA0B;AACpD,MAAI;AACF,QAAI,OAAO,iBAAiB,UAAU;AACpC,YAAM,IAAI;AAAA,QACR,8CAA8C,OAAO,KAAK;AAAA,MAC5D;AAAA,IACF;AACA,WAAO,gBAAmB,OAAO,SAAS,OAAO,KAAK;AAAA,EACxD,SAAS,OAAO;AACd,QAAI,iBAAiB,iBAAkB,OAAM;AAC7C,UAAM,QAAQ,iBAAiB,QAAQ,QAAQ,IAAI,MAAM,OAAO,KAAK,CAAC;AACtE,UAAM,IAAI,iBAAiB,MAAM,SAAS,QAAQ,EAAE,MAAM,CAAC;AAAA,EAC7D;AACF;AAEA,SAAS,gBAAmB,SAAiB,OAAkB;AAC7D,QAAM,WAAW,mBAAmB,OAAO;AAC3C,MAAI;AACF,WAAO,KAAK,MAAM,QAAQ;AAAA,EAC5B,SAAS,KAAK;AACZ,UAAM,IAAI;AAAA,MACR,wCAAwC,KAAK,MAC3C,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,CACjD;AAAA;AAAA,EAA0B,QAAQ,MAAM,GAAG,GAAG,CAAC;AAAA,IACjD;AAAA,EACF;AACF;AAWO,IAAM,yBAAN,cAAqC,sBAAsB;AAAA,EAChE,YACE,SACgB,QACA,SAChB;AACA,UAAM,OAAO;AAHG;AACA;AAAA,EAGlB;AAAA,EAJkB;AAAA,EACA;AAIpB;AAmCO,SAAS,eAAe,MAAwB,MAA4B,CAAC,GAAS;AAC3F,QAAM,kBAAkB,KAAK,YAAY;AACzC,QAAM,WAAW,KAAK,WAAW,kBAAkB,QAAQ,QAAQ,EAAE;AAErE,MAAI,IAAI,0BAA0B,CAAC,iBAAiB;AAClD,UAAM,IAAI;AAAA,MACR,gGAAgG,gBAAgB;AAAA,MAChH;AAAA,MACA;AAAA,IACF;AAAA,EACF;AAEA,MAAI,IAAI,iBAAiB,KAAK,CAAC,MAAM,SAAS,SAAS,CAAC,CAAC,GAAG;AAC1D,UAAM,IAAI;AAAA,MACR,2BAA2B,OAAO;AAAA,MAClC;AAAA,MACA;AAAA,IACF;AAAA,EACF;AAEA,MAAI,IAAI,mBAAmB,IAAI,gBAAgB,SAAS,GAAG;AACzD,UAAM,KAAK,IAAI,gBAAgB,KAAK,CAAC,MAAM,SAAS,SAAS,CAAC,CAAC;AAC/D,QAAI,CAAC,IAAI;AACP,YAAM,IAAI;AAAA,QACR,2BAA2B,OAAO,gCAAgC,IAAI,gBAAgB,IAAI,eAAe,EAAE,KAAK,IAAI,CAAC;AAAA,QACrH;AAAA,QACA;AAAA,MACF;AAAA,IACF;AAAA,EACF;AAEA,MAAI,IAAI,eAAe,CAAC,KAAK,UAAU,CAAC,KAAK,UAAU,CAAC,KAAK,YAAY;AACvE,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,MACA;AAAA,IACF;AAAA,EACF;AAEA,MAAI,IAAI,kBAAkB;AACxB,UAAM,SAAS,KAAK,YAAY,oBAAoB,OAAO;AAC3D,QAAI,WAAW,IAAI,kBAAkB;AACnC,YAAM,IAAI;AAAA,QACR,qCAAqC,IAAI,gBAAgB,gBAAgB,OAAO,gBAAgB,MAAM;AAAA,QACtG;AAAA,QACA;AAAA,MACF;AAAA,IACF;AAAA,EACF;AACF;AAEA,SAAS,SAAS,KAAa,SAAmC;AAChE,MAAI,mBAAmB,OAAQ,QAAO,QAAQ,KAAK,GAAG;AACtD,SAAO,IAAI,YAAY,EAAE,WAAW,QAAQ,YAAY,CAAC;AAC3D;AAEA,SAAS,gBAAgB,GAA4B;AACnD,SAAO,aAAa,SAAS,EAAE,SAAS;AAC1C;AAaA,eAAsB,SACpB,OACA,OAAkD,CAAC,GACgB;AACnE,QAAM,QAAQ,KAAK,IAAI;AACvB,MAAI;AACF,UAAM;AAAA,MACJ;AAAA,QACE;AAAA,QACA,UAAU,CAAC,EAAE,MAAM,QAAQ,SAAS,OAAO,CAAC;AAAA,QAC5C,WAAW;AAAA,QACX,WAAW,KAAK,aAAa;AAAA,MAC/B;AAAA,MACA;AAAA,IACF;AACA,WAAO,EAAE,IAAI,MAAM,WAAW,KAAK,IAAI,IAAI,OAAO,OAAO,KAAK;AAAA,EAChE,SAAS,KAAK;AACZ,WAAO;AAAA,MACL,IAAI;AAAA,MACJ,WAAW,KAAK,IAAI,IAAI;AAAA,MACxB,OAAO,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG;AAAA,IACxD;AAAA,EACF;AACF;AAOO,IAAM,YAAN,MAAgB;AAAA,EACZ;AAAA,EACQ;AAAA,EAEjB,YAAY,OAAyB,CAAC,GAAG;AACvC,SAAK,OAAO;AACZ,SAAK,kBAAkB,uBAAuB,KAAK,UAAU;AAAA,EAC/D;AAAA,EAEA,KAAK,KAAqB,KAAgD;AACxE,UAAM,UAAU,EAAE,GAAG,KAAK,MAAM,GAAG,IAAI;AACvC,WAAO,IAAI,aAAa,kBAAkB,KAAK,OAAO,IAAI,QAAQ,KAAK,OAAO;AAAA,EAChF;AAAA,EAEA,SACE,KACA,KAC8C;AAC9C,WAAO,YAAe,KAAK,EAAE,GAAG,KAAK,MAAM,GAAG,IAAI,CAAC;AAAA,EACrD;AACF;","names":[]}
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
{"version":3,"sources":["../src/agent-profile-cell.ts"],"sourcesContent":["import type { AgentProfile } from '@tangle-network/agent-interface'\nimport { ValidationError } from './errors'\nimport { hashJson } from './pre-registration'\n\nexport type AgentProfileCellSchemaVersion = 'agent-profile-cell/v1'\n\nexport type AgentProfileJson =\n | string\n | number\n | boolean\n | null\n | AgentProfileJson[]\n | { [key: string]: AgentProfileJson }\n\nexport type AgentProfileDimensionValue = string | number | boolean | null\n\nexport interface AgentProfileSource {\n /** Runtime/profile contract being fingerprinted, e.g. `agent-interface-profile`. */\n kind: string\n /** sha256 over the canonical source profile object. */\n hash: string\n}\n\nexport interface AgentProfileSourceInput {\n kind: string\n /** Precomputed sha256 for callers that already sign their profile artifact. */\n hash?: string\n /** Full canonical runtime profile; hashed and then discarded from the cell. */\n profile?: AgentProfileJson\n}\n\nexport interface AgentProfileHarness {\n id: string\n version?: string\n hash?: string\n}\n\nexport interface AgentProfileCellInput {\n profileId: string\n sourceProfile: AgentProfileSourceInput\n harness?: AgentProfileHarness\n model?: string\n promptHash?: string\n dimensions?: Record<string, AgentProfileDimensionValue>\n}\n\nexport interface AgentProfileCell {\n schemaVersion: AgentProfileCellSchemaVersion\n cellId: string\n profileId: string\n sourceProfile: AgentProfileSource\n harness?: AgentProfileHarness\n model?: string\n promptHash?: string\n dimensions?: Record<string, AgentProfileDimensionValue>\n}\n\nexport class AgentProfileCellValidationError extends ValidationError {\n readonly path: string\n constructor(message: string, path = '') {\n super(path ? `${message} (at ${path})` : message)\n this.path = path\n }\n}\n\nconst SHA256_HEX = /^[0-9a-f]{64}$/\nconst CELL_ID = /^agent-profile-cell:sha256:[0-9a-f]{64}$/\n\nexport async function buildAgentProfileCell(\n input: AgentProfileCellInput,\n): Promise<AgentProfileCell> {\n const material = await normalizeAgentProfileCellInput(input)\n const cellId = `agent-profile-cell:sha256:${await hashJson(material)}`\n return { ...material, cellId }\n}\n\nexport function agentProfileCellHashMaterial(\n cell: AgentProfileCell,\n): Omit<AgentProfileCell, 'cellId'> {\n const { cellId: _cellId, ...material } = cell\n void _cellId\n return normalizeAgentProfileCell(material)\n}\n\n/**\n * Verify an `AgentProfileCell`'s `cellId` matches the sha256 of its hash-material fields, confirming the record has not been tampered with.\n */\nexport async function verifyAgentProfileCell(cell: AgentProfileCell): Promise<boolean> {\n validateAgentProfileCell(cell)\n return (\n cell.cellId ===\n `agent-profile-cell:sha256:${await hashJson(agentProfileCellHashMaterial(cell))}`\n )\n}\n\nexport function validateAgentProfileCell(input: unknown): AgentProfileCell {\n if (input === null || typeof input !== 'object') {\n throw new AgentProfileCellValidationError('expected object')\n }\n const obj = input as Record<string, unknown>\n expectLiteral(obj.schemaVersion, 'agent-profile-cell/v1', 'schemaVersion')\n if (typeof obj.cellId !== 'string' || !CELL_ID.test(obj.cellId)) {\n throw new AgentProfileCellValidationError(\n 'cellId must match agent-profile-cell:sha256:<64 lowercase hex chars>',\n 'cellId',\n )\n }\n expectString(obj.profileId, 'profileId')\n validateSource(obj.sourceProfile, 'sourceProfile')\n if (obj.harness !== undefined) validateHarness(obj.harness, 'harness')\n if (obj.model !== undefined) expectString(obj.model, 'model')\n if (obj.promptHash !== undefined) expectString(obj.promptHash, 'promptHash')\n if (obj.dimensions !== undefined) validateDimensions(obj.dimensions, 'dimensions')\n return input as AgentProfileCell\n}\n\nexport function requireAgentProfileCell(record: {\n runId: string\n agentProfile?: AgentProfileCell\n}): AgentProfileCell {\n if (!record.agentProfile) {\n throw new AgentProfileCellValidationError(\n `run \"${record.runId}\" is missing agentProfile; profile-cell grouping requires explicit profile identity`,\n 'agentProfile',\n )\n }\n return validateAgentProfileCell(record.agentProfile)\n}\n\nexport function agentProfileCellKey(record: {\n runId: string\n agentProfile?: AgentProfileCell\n}): string {\n return requireAgentProfileCell(record).cellId\n}\n\nexport async function assertRunAgentProfileCell(record: {\n runId: string\n model: string\n promptHash: string\n agentProfile?: AgentProfileCell\n}): Promise<AgentProfileCell> {\n const profile = requireAgentProfileCell(record)\n if (!(await verifyAgentProfileCell(profile))) {\n throw new AgentProfileCellValidationError(\n `run \"${record.runId}\" has an agentProfile.cellId that does not match its content`,\n 'agentProfile.cellId',\n )\n }\n if (profile.model !== undefined && profile.model !== record.model) {\n throw new AgentProfileCellValidationError(\n `run \"${record.runId}\" agentProfile.model \"${profile.model}\" does not match model \"${record.model}\"`,\n 'agentProfile.model',\n )\n }\n if (profile.promptHash !== undefined && profile.promptHash !== record.promptHash) {\n throw new AgentProfileCellValidationError(\n `run \"${record.runId}\" agentProfile.promptHash \"${profile.promptHash}\" does not match promptHash \"${record.promptHash}\"`,\n 'agentProfile.promptHash',\n )\n }\n return profile\n}\n\nexport function groupRunsByAgentProfileCell<\n T extends { runId: string; agentProfile?: AgentProfileCell },\n>(records: readonly T[]): Map<string, T[]> {\n const groups = new Map<string, T[]>()\n for (const record of records) {\n const key = agentProfileCellKey(record)\n const bucket = groups.get(key)\n if (bucket) bucket.push(record)\n else groups.set(key, [record])\n }\n return groups\n}\n\nasync function normalizeAgentProfileCellInput(\n input: AgentProfileCellInput,\n): Promise<Omit<AgentProfileCell, 'cellId'>> {\n return normalizeAgentProfileCell({\n schemaVersion: 'agent-profile-cell/v1',\n profileId: input.profileId,\n sourceProfile: await normalizeSourceInput(input.sourceProfile),\n harness: input.harness,\n model: input.model,\n promptHash: input.promptHash,\n dimensions: input.dimensions,\n })\n}\n\nfunction normalizeAgentProfileCell(\n input: Omit<AgentProfileCell, 'cellId'>,\n): Omit<AgentProfileCell, 'cellId'> {\n return compactObject({\n schemaVersion: 'agent-profile-cell/v1' as const,\n profileId: requireNonEmpty(input.profileId, 'profileId'),\n sourceProfile: normalizeSource(input.sourceProfile),\n harness: input.harness ? normalizeHarness(input.harness, 'harness') : undefined,\n model: optionalNonEmpty(input.model, 'model'),\n promptHash: optionalNonEmpty(input.promptHash, 'promptHash'),\n dimensions: input.dimensions\n ? nonEmptyRecord(normalizeDimensions(input.dimensions))\n : undefined,\n })\n}\n\nasync function normalizeSourceInput(input: AgentProfileSourceInput): Promise<AgentProfileSource> {\n const kind = requireNonEmpty(input.kind, 'sourceProfile.kind')\n if (input.hash !== undefined && input.profile !== undefined) {\n throw new AgentProfileCellValidationError(\n 'sourceProfile must provide either hash or profile, not both',\n 'sourceProfile',\n )\n }\n if (input.hash !== undefined) {\n return { kind, hash: requireSha256Hex(input.hash, 'sourceProfile.hash') }\n }\n if (input.profile === undefined) {\n throw new AgentProfileCellValidationError(\n 'sourceProfile must provide hash or profile',\n 'sourceProfile',\n )\n }\n assertJson(input.profile, 'sourceProfile.profile')\n return { kind, hash: await hashJson(input.profile) }\n}\n\nfunction normalizeSource(input: AgentProfileSource): AgentProfileSource {\n return {\n kind: requireNonEmpty(input.kind, 'sourceProfile.kind'),\n hash: requireSha256Hex(input.hash, 'sourceProfile.hash'),\n }\n}\n\nfunction normalizeHarness(input: AgentProfileHarness, path: string): AgentProfileHarness {\n return compactObject({\n id: requireNonEmpty(input.id, `${path}.id`),\n version: optionalNonEmpty(input.version, `${path}.version`),\n hash: optionalNonEmpty(input.hash, `${path}.hash`),\n })\n}\n\nfunction normalizeDimensions(\n input: Record<string, AgentProfileDimensionValue>,\n): Record<string, AgentProfileDimensionValue> {\n const out: Record<string, AgentProfileDimensionValue> = {}\n for (const key of Object.keys(input).sort()) {\n const value = input[key]\n requireNonEmpty(key, 'dimensions.<key>')\n if (\n value !== null &&\n typeof value !== 'string' &&\n typeof value !== 'number' &&\n typeof value !== 'boolean'\n ) {\n throw new AgentProfileCellValidationError(\n 'expected primitive dimension value',\n `dimensions.${key}`,\n )\n }\n if (typeof value === 'number' && !Number.isFinite(value)) {\n throw new AgentProfileCellValidationError('expected finite number', `dimensions.${key}`)\n }\n out[key] = value\n }\n return out\n}\n\nfunction compactObject<T extends Record<string, unknown>>(input: T): T {\n const out: Record<string, unknown> = {}\n for (const [key, value] of Object.entries(input)) {\n if (value !== undefined) out[key] = value\n }\n return out as T\n}\n\nfunction nonEmptyRecord<T extends Record<string, unknown>>(input: T): T | undefined {\n return Object.keys(input).length > 0 ? input : undefined\n}\n\nfunction validateSource(value: unknown, path: string): void {\n if (value === null || typeof value !== 'object' || Array.isArray(value)) {\n throw new AgentProfileCellValidationError('expected object', path)\n }\n const rec = value as Record<string, unknown>\n expectString(rec.kind, `${path}.kind`)\n requireSha256Hex(rec.hash, `${path}.hash`)\n}\n\nfunction validateHarness(value: unknown, path: string): void {\n if (value === null || typeof value !== 'object' || Array.isArray(value)) {\n throw new AgentProfileCellValidationError('expected object', path)\n }\n const rec = value as Record<string, unknown>\n expectString(rec.id, `${path}.id`)\n if (rec.version !== undefined) expectString(rec.version, `${path}.version`)\n if (rec.hash !== undefined) expectString(rec.hash, `${path}.hash`)\n}\n\nfunction validateDimensions(value: unknown, path: string): void {\n if (value === null || typeof value !== 'object' || Array.isArray(value)) {\n throw new AgentProfileCellValidationError('expected object', path)\n }\n normalizeDimensions(value as Record<string, AgentProfileDimensionValue>)\n}\n\nfunction assertJson(value: AgentProfileJson, path: string): void {\n if (value === null) return\n const type = typeof value\n if (type === 'string' || type === 'boolean') return\n if (type === 'number') {\n if (!Number.isFinite(value)) {\n throw new AgentProfileCellValidationError('expected finite number', path)\n }\n return\n }\n if (Array.isArray(value)) {\n value.forEach((item, index) => {\n assertJson(item, `${path}[${index}]`)\n })\n return\n }\n if (type === 'object') {\n for (const [key, nested] of Object.entries(value)) {\n requireNonEmpty(key, `${path}.<key>`)\n assertJson(nested, `${path}.${key}`)\n }\n return\n }\n throw new AgentProfileCellValidationError('expected JSON-compatible value', path)\n}\n\nfunction expectLiteral(value: unknown, expected: string, path: string): void {\n if (value !== expected) {\n throw new AgentProfileCellValidationError(`expected ${expected}`, path)\n }\n}\n\nfunction expectString(value: unknown, path: string): void {\n if (typeof value !== 'string' || value.length === 0) {\n throw new AgentProfileCellValidationError('expected non-empty string', path)\n }\n}\n\nfunction requireNonEmpty(value: string, path: string): string {\n if (typeof value !== 'string' || value.length === 0) {\n throw new AgentProfileCellValidationError('expected non-empty string', path)\n }\n return value\n}\n\nfunction optionalNonEmpty(value: string | undefined, path: string): string | undefined {\n if (value === undefined) return undefined\n return requireNonEmpty(value, path)\n}\n\nfunction requireSha256Hex(value: unknown, path: string): string {\n if (typeof value !== 'string' || !SHA256_HEX.test(value)) {\n throw new AgentProfileCellValidationError('expected 64 lowercase sha256 hex chars', path)\n }\n return value\n}\n\n// ── Consumer helpers ─────────────────────────────────────────────────\n//\n// Boilerplate every product consuming `buildAgentProfileCell` used to duplicate:\n//\n// 1. A `JSON.parse(JSON.stringify(value))` helper that canonicalizes an\n// arbitrary `@tangle-network/agent-interface` `AgentProfile` into the recursive\n// `AgentProfileJson` shape, with a fail-loud error when the profile\n// is not JSON-serializable.\n//\n// 2. The magic string `'agent-interface-profile'` for `sourceProfile.kind`.\n//\n// Both belong here so the cross-product cell join (same canonical profile\n// hashes to the same `sourceProfile.hash` across products) is enforced by\n// the type system, not by every consumer remembering to do it right.\n// See blueprint-agent issue tangle-network/agent-eval#82.\n\n/** Canonical `sourceProfile.kind` values. Two products fingerprinting the\n * same canonical profile MUST use the same kind for their cells to share\n * `sourceProfile.hash`. Extend rather than create new strings — adding a\n * new kind is a deliberate cross-product schema change. */\nexport const AGENT_PROFILE_KINDS = {\n /** A profile declared via `defineAgentProfile(...)` from\n * `@tangle-network/agent-interface`. The default kind for router-backed\n * and sandbox-backed products. */\n AGENT_INTERFACE_PROFILE: 'agent-interface-profile',\n} as const\n\nexport type AgentProfileKind = (typeof AGENT_PROFILE_KINDS)[keyof typeof AGENT_PROFILE_KINDS]\n\n/** Canonicalize an arbitrary value into `AgentProfileJson` by JSON\n * round-trip. Throws when the value contains anything not representable\n * as JSON (functions, BigInt, cycles) — non-portable profiles fail loud\n * rather than silently dropping fields. */\nexport function toAgentProfileJson(value: unknown): AgentProfileJson {\n let serialized: string | undefined\n try {\n serialized = JSON.stringify(value)\n } catch (err) {\n throw new AgentProfileCellValidationError(\n `agent profile must be JSON-serializable: ${err instanceof Error ? err.message : String(err)}`,\n 'sourceProfile.profile',\n )\n }\n if (serialized === undefined) {\n throw new AgentProfileCellValidationError(\n 'agent profile must be JSON-serializable (got undefined after JSON.stringify)',\n 'sourceProfile.profile',\n )\n }\n return JSON.parse(serialized) as AgentProfileJson\n}\n\n/** Canonical AgentProfile shape required when deriving a stable cell id. */\nexport type AgentInterfaceProfileLike = AgentProfile & { name: string; version: string }\n\n/** Higher-level helper that hard-codes the canonical\n * `agent-interface-profile` kind plus the JSON canonicalization. Equivalent\n * to calling `buildAgentProfileCell` with `profileId = \\`${name}@${version}\\``\n * and `sourceProfile = { kind: AGENT_INTERFACE_PROFILE, profile: <round-tripped> }`.\n *\n * Use this from any product consuming an agent-interface `AgentProfile`; the\n * manual `buildAgentProfileCell` call is reserved for advanced cases\n * (custom kinds, pre-computed source hashes, alternate profileId\n * conventions). */\nexport async function buildAgentInterfaceProfileCell(\n profile: AgentInterfaceProfileLike,\n input: Omit<AgentProfileCellInput, 'profileId' | 'sourceProfile'>,\n): Promise<AgentProfileCell> {\n if (!profile || typeof profile !== 'object') {\n throw new AgentProfileCellValidationError('AgentProfile must be an object', 'profile')\n }\n if (typeof profile.name !== 'string' || profile.name.length === 0) {\n throw new AgentProfileCellValidationError(\n 'AgentProfile must have a non-empty `name`',\n 'profile.name',\n )\n }\n if (typeof profile.version !== 'string' || profile.version.length === 0) {\n throw new AgentProfileCellValidationError(\n 'AgentProfile must have a non-empty `version`',\n 'profile.version',\n )\n }\n return buildAgentProfileCell({\n ...input,\n profileId: `${profile.name}@${profile.version}`,\n sourceProfile: {\n kind: AGENT_PROFILE_KINDS.AGENT_INTERFACE_PROFILE,\n profile: toAgentProfileJson(profile),\n },\n })\n}\n"],"mappings":";;;;;;;;AAyDO,IAAM,kCAAN,cAA8C,gBAAgB;AAAA,EAC1D;AAAA,EACT,YAAY,SAAiB,OAAO,IAAI;AACtC,UAAM,OAAO,GAAG,OAAO,QAAQ,IAAI,MAAM,OAAO;AAChD,SAAK,OAAO;AAAA,EACd;AACF;AAEA,IAAM,aAAa;AACnB,IAAM,UAAU;AAEhB,eAAsB,sBACpB,OAC2B;AAC3B,QAAM,WAAW,MAAM,+BAA+B,KAAK;AAC3D,QAAM,SAAS,6BAA6B,MAAM,SAAS,QAAQ,CAAC;AACpE,SAAO,EAAE,GAAG,UAAU,OAAO;AAC/B;AAEO,SAAS,6BACd,MACkC;AAClC,QAAM,EAAE,QAAQ,SAAS,GAAG,SAAS,IAAI;AACzC,OAAK;AACL,SAAO,0BAA0B,QAAQ;AAC3C;AAKA,eAAsB,uBAAuB,MAA0C;AACrF,2BAAyB,IAAI;AAC7B,SACE,KAAK,WACL,6BAA6B,MAAM,SAAS,6BAA6B,IAAI,CAAC,CAAC;AAEnF;AAEO,SAAS,yBAAyB,OAAkC;AACzE,MAAI,UAAU,QAAQ,OAAO,UAAU,UAAU;AAC/C,UAAM,IAAI,gCAAgC,iBAAiB;AAAA,EAC7D;AACA,QAAM,MAAM;AACZ,gBAAc,IAAI,eAAe,yBAAyB,eAAe;AACzE,MAAI,OAAO,IAAI,WAAW,YAAY,CAAC,QAAQ,KAAK,IAAI,MAAM,GAAG;AAC/D,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACA,eAAa,IAAI,WAAW,WAAW;AACvC,iBAAe,IAAI,eAAe,eAAe;AACjD,MAAI,IAAI,YAAY,OAAW,iBAAgB,IAAI,SAAS,SAAS;AACrE,MAAI,IAAI,UAAU,OAAW,cAAa,IAAI,OAAO,OAAO;AAC5D,MAAI,IAAI,eAAe,OAAW,cAAa,IAAI,YAAY,YAAY;AAC3E,MAAI,IAAI,eAAe,OAAW,oBAAmB,IAAI,YAAY,YAAY;AACjF,SAAO;AACT;AAEO,SAAS,wBAAwB,QAGnB;AACnB,MAAI,CAAC,OAAO,cAAc;AACxB,UAAM,IAAI;AAAA,MACR,QAAQ,OAAO,KAAK;AAAA,MACpB;AAAA,IACF;AAAA,EACF;AACA,SAAO,yBAAyB,OAAO,YAAY;AACrD;AAEO,SAAS,oBAAoB,QAGzB;AACT,SAAO,wBAAwB,MAAM,EAAE;AACzC;AAEA,eAAsB,0BAA0B,QAKlB;AAC5B,QAAM,UAAU,wBAAwB,MAAM;AAC9C,MAAI,CAAE,MAAM,uBAAuB,OAAO,GAAI;AAC5C,UAAM,IAAI;AAAA,MACR,QAAQ,OAAO,KAAK;AAAA,MACpB;AAAA,IACF;AAAA,EACF;AACA,MAAI,QAAQ,UAAU,UAAa,QAAQ,UAAU,OAAO,OAAO;AACjE,UAAM,IAAI;AAAA,MACR,QAAQ,OAAO,KAAK,yBAAyB,QAAQ,KAAK,2BAA2B,OAAO,KAAK;AAAA,MACjG;AAAA,IACF;AAAA,EACF;AACA,MAAI,QAAQ,eAAe,UAAa,QAAQ,eAAe,OAAO,YAAY;AAChF,UAAM,IAAI;AAAA,MACR,QAAQ,OAAO,KAAK,8BAA8B,QAAQ,UAAU,gCAAgC,OAAO,UAAU;AAAA,MACrH;AAAA,IACF;AAAA,EACF;AACA,SAAO;AACT;AAEO,SAAS,4BAEd,SAAyC;AACzC,QAAM,SAAS,oBAAI,IAAiB;AACpC,aAAW,UAAU,SAAS;AAC5B,UAAM,MAAM,oBAAoB,MAAM;AACtC,UAAM,SAAS,OAAO,IAAI,GAAG;AAC7B,QAAI,OAAQ,QAAO,KAAK,MAAM;AAAA,QACzB,QAAO,IAAI,KAAK,CAAC,MAAM,CAAC;AAAA,EAC/B;AACA,SAAO;AACT;AAEA,eAAe,+BACb,OAC2C;AAC3C,SAAO,0BAA0B;AAAA,IAC/B,eAAe;AAAA,IACf,WAAW,MAAM;AAAA,IACjB,eAAe,MAAM,qBAAqB,MAAM,aAAa;AAAA,IAC7D,SAAS,MAAM;AAAA,IACf,OAAO,MAAM;AAAA,IACb,YAAY,MAAM;AAAA,IAClB,YAAY,MAAM;AAAA,EACpB,CAAC;AACH;AAEA,SAAS,0BACP,OACkC;AAClC,SAAO,cAAc;AAAA,IACnB,eAAe;AAAA,IACf,WAAW,gBAAgB,MAAM,WAAW,WAAW;AAAA,IACvD,eAAe,gBAAgB,MAAM,aAAa;AAAA,IAClD,SAAS,MAAM,UAAU,iBAAiB,MAAM,SAAS,SAAS,IAAI;AAAA,IACtE,OAAO,iBAAiB,MAAM,OAAO,OAAO;AAAA,IAC5C,YAAY,iBAAiB,MAAM,YAAY,YAAY;AAAA,IAC3D,YAAY,MAAM,aACd,eAAe,oBAAoB,MAAM,UAAU,CAAC,IACpD;AAAA,EACN,CAAC;AACH;AAEA,eAAe,qBAAqB,OAA6D;AAC/F,QAAM,OAAO,gBAAgB,MAAM,MAAM,oBAAoB;AAC7D,MAAI,MAAM,SAAS,UAAa,MAAM,YAAY,QAAW;AAC3D,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACA,MAAI,MAAM,SAAS,QAAW;AAC5B,WAAO,EAAE,MAAM,MAAM,iBAAiB,MAAM,MAAM,oBAAoB,EAAE;AAAA,EAC1E;AACA,MAAI,MAAM,YAAY,QAAW;AAC/B,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACA,aAAW,MAAM,SAAS,uBAAuB;AACjD,SAAO,EAAE,MAAM,MAAM,MAAM,SAAS,MAAM,OAAO,EAAE;AACrD;AAEA,SAAS,gBAAgB,OAA+C;AACtE,SAAO;AAAA,IACL,MAAM,gBAAgB,MAAM,MAAM,oBAAoB;AAAA,IACtD,MAAM,iBAAiB,MAAM,MAAM,oBAAoB;AAAA,EACzD;AACF;AAEA,SAAS,iBAAiB,OAA4B,MAAmC;AACvF,SAAO,cAAc;AAAA,IACnB,IAAI,gBAAgB,MAAM,IAAI,GAAG,IAAI,KAAK;AAAA,IAC1C,SAAS,iBAAiB,MAAM,SAAS,GAAG,IAAI,UAAU;AAAA,IAC1D,MAAM,iBAAiB,MAAM,MAAM,GAAG,IAAI,OAAO;AAAA,EACnD,CAAC;AACH;AAEA,SAAS,oBACP,OAC4C;AAC5C,QAAM,MAAkD,CAAC;AACzD,aAAW,OAAO,OAAO,KAAK,KAAK,EAAE,KAAK,GAAG;AAC3C,UAAM,QAAQ,MAAM,GAAG;AACvB,oBAAgB,KAAK,kBAAkB;AACvC,QACE,UAAU,QACV,OAAO,UAAU,YACjB,OAAO,UAAU,YACjB,OAAO,UAAU,WACjB;AACA,YAAM,IAAI;AAAA,QACR;AAAA,QACA,cAAc,GAAG;AAAA,MACnB;AAAA,IACF;AACA,QAAI,OAAO,UAAU,YAAY,CAAC,OAAO,SAAS,KAAK,GAAG;AACxD,YAAM,IAAI,gCAAgC,0BAA0B,cAAc,GAAG,EAAE;AAAA,IACzF;AACA,QAAI,GAAG,IAAI;AAAA,EACb;AACA,SAAO;AACT;AAEA,SAAS,cAAiD,OAAa;AACrE,QAAM,MAA+B,CAAC;AACtC,aAAW,CAAC,KAAK,KAAK,KAAK,OAAO,QAAQ,KAAK,GAAG;AAChD,QAAI,UAAU,OAAW,KAAI,GAAG,IAAI;AAAA,EACtC;AACA,SAAO;AACT;AAEA,SAAS,eAAkD,OAAyB;AAClF,SAAO,OAAO,KAAK,KAAK,EAAE,SAAS,IAAI,QAAQ;AACjD;AAEA,SAAS,eAAe,OAAgB,MAAoB;AAC1D,MAAI,UAAU,QAAQ,OAAO,UAAU,YAAY,MAAM,QAAQ,KAAK,GAAG;AACvE,UAAM,IAAI,gCAAgC,mBAAmB,IAAI;AAAA,EACnE;AACA,QAAM,MAAM;AACZ,eAAa,IAAI,MAAM,GAAG,IAAI,OAAO;AACrC,mBAAiB,IAAI,MAAM,GAAG,IAAI,OAAO;AAC3C;AAEA,SAAS,gBAAgB,OAAgB,MAAoB;AAC3D,MAAI,UAAU,QAAQ,OAAO,UAAU,YAAY,MAAM,QAAQ,KAAK,GAAG;AACvE,UAAM,IAAI,gCAAgC,mBAAmB,IAAI;AAAA,EACnE;AACA,QAAM,MAAM;AACZ,eAAa,IAAI,IAAI,GAAG,IAAI,KAAK;AACjC,MAAI,IAAI,YAAY,OAAW,cAAa,IAAI,SAAS,GAAG,IAAI,UAAU;AAC1E,MAAI,IAAI,SAAS,OAAW,cAAa,IAAI,MAAM,GAAG,IAAI,OAAO;AACnE;AAEA,SAAS,mBAAmB,OAAgB,MAAoB;AAC9D,MAAI,UAAU,QAAQ,OAAO,UAAU,YAAY,MAAM,QAAQ,KAAK,GAAG;AACvE,UAAM,IAAI,gCAAgC,mBAAmB,IAAI;AAAA,EACnE;AACA,sBAAoB,KAAmD;AACzE;AAEA,SAAS,WAAW,OAAyB,MAAoB;AAC/D,MAAI,UAAU,KAAM;AACpB,QAAM,OAAO,OAAO;AACpB,MAAI,SAAS,YAAY,SAAS,UAAW;AAC7C,MAAI,SAAS,UAAU;AACrB,QAAI,CAAC,OAAO,SAAS,KAAK,GAAG;AAC3B,YAAM,IAAI,gCAAgC,0BAA0B,IAAI;AAAA,IAC1E;AACA;AAAA,EACF;AACA,MAAI,MAAM,QAAQ,KAAK,GAAG;AACxB,UAAM,QAAQ,CAAC,MAAM,UAAU;AAC7B,iBAAW,MAAM,GAAG,IAAI,IAAI,KAAK,GAAG;AAAA,IACtC,CAAC;AACD;AAAA,EACF;AACA,MAAI,SAAS,UAAU;AACrB,eAAW,CAAC,KAAK,MAAM,KAAK,OAAO,QAAQ,KAAK,GAAG;AACjD,sBAAgB,KAAK,GAAG,IAAI,QAAQ;AACpC,iBAAW,QAAQ,GAAG,IAAI,IAAI,GAAG,EAAE;AAAA,IACrC;AACA;AAAA,EACF;AACA,QAAM,IAAI,gCAAgC,kCAAkC,IAAI;AAClF;AAEA,SAAS,cAAc,OAAgB,UAAkB,MAAoB;AAC3E,MAAI,UAAU,UAAU;AACtB,UAAM,IAAI,gCAAgC,YAAY,QAAQ,IAAI,IAAI;AAAA,EACxE;AACF;AAEA,SAAS,aAAa,OAAgB,MAAoB;AACxD,MAAI,OAAO,UAAU,YAAY,MAAM,WAAW,GAAG;AACnD,UAAM,IAAI,gCAAgC,6BAA6B,IAAI;AAAA,EAC7E;AACF;AAEA,SAAS,gBAAgB,OAAe,MAAsB;AAC5D,MAAI,OAAO,UAAU,YAAY,MAAM,WAAW,GAAG;AACnD,UAAM,IAAI,gCAAgC,6BAA6B,IAAI;AAAA,EAC7E;AACA,SAAO;AACT;AAEA,SAAS,iBAAiB,OAA2B,MAAkC;AACrF,MAAI,UAAU,OAAW,QAAO;AAChC,SAAO,gBAAgB,OAAO,IAAI;AACpC;AAEA,SAAS,iBAAiB,OAAgB,MAAsB;AAC9D,MAAI,OAAO,UAAU,YAAY,CAAC,WAAW,KAAK,KAAK,GAAG;AACxD,UAAM,IAAI,gCAAgC,0CAA0C,IAAI;AAAA,EAC1F;AACA,SAAO;AACT;AAsBO,IAAM,sBAAsB;AAAA;AAAA;AAAA;AAAA,EAIjC,yBAAyB;AAC3B;AAQO,SAAS,mBAAmB,OAAkC;AACnE,MAAI;AACJ,MAAI;AACF,iBAAa,KAAK,UAAU,KAAK;AAAA,EACnC,SAAS,KAAK;AACZ,UAAM,IAAI;AAAA,MACR,4CAA4C,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,CAAC;AAAA,MAC5F;AAAA,IACF;AAAA,EACF;AACA,MAAI,eAAe,QAAW;AAC5B,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACA,SAAO,KAAK,MAAM,UAAU;AAC9B;AAcA,eAAsB,+BACpB,SACA,OAC2B;AAC3B,MAAI,CAAC,WAAW,OAAO,YAAY,UAAU;AAC3C,UAAM,IAAI,gCAAgC,kCAAkC,SAAS;AAAA,EACvF;AACA,MAAI,OAAO,QAAQ,SAAS,YAAY,QAAQ,KAAK,WAAW,GAAG;AACjE,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACA,MAAI,OAAO,QAAQ,YAAY,YAAY,QAAQ,QAAQ,WAAW,GAAG;AACvE,UAAM,IAAI;AAAA,MACR;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACA,SAAO,sBAAsB;AAAA,IAC3B,GAAG;AAAA,IACH,WAAW,GAAG,QAAQ,IAAI,IAAI,QAAQ,OAAO;AAAA,IAC7C,eAAe;AAAA,MACb,MAAM,oBAAoB;AAAA,MAC1B,SAAS,mBAAmB,OAAO;AAAA,IACrC;AAAA,EACF,CAAC;AACH;","names":[]}
|
|
@@ -1,186 +0,0 @@
|
|
|
1
|
-
# Auto-research loop end-to-end
|
|
2
|
-
|
|
3
|
-
This is the runnable composition pattern that closes the loop the package
|
|
4
|
-
was originally designed for: capture-integrity → eval → preferences →
|
|
5
|
-
mutation → improved candidate → repeat.
|
|
6
|
-
|
|
7
|
-
There's no new orchestrator primitive that runs this for you (and we
|
|
8
|
-
deliberately resisted shipping one — every consumer's loop has different
|
|
9
|
-
invariants). What this doc gives you is **the integration recipe**: the
|
|
10
|
-
imports, the wiring, and the explicit invariants every iteration must
|
|
11
|
-
preserve.
|
|
12
|
-
|
|
13
|
-
A working version of this recipe lives at
|
|
14
|
-
[`examples/auto-research-with-agent-builder/`](../examples/auto-research-with-agent-builder/) —
|
|
15
|
-
runnable, ~250 lines, demonstrates the score climbing across iterations.
|
|
16
|
-
|
|
17
|
-
## The pattern
|
|
18
|
-
|
|
19
|
-
```ts
|
|
20
|
-
import {
|
|
21
|
-
runEvalCampaign,
|
|
22
|
-
analyzeOptimizationResult,
|
|
23
|
-
trialsToRunRecords,
|
|
24
|
-
PredictiveValidityResearcher,
|
|
25
|
-
} from '@tangle-network/agent-eval'
|
|
26
|
-
import { traceAnalystOnRunComplete } from '@tangle-network/agent-eval/traces'
|
|
27
|
-
|
|
28
|
-
async function runAutoResearchLoop(opts: {
|
|
29
|
-
task: string
|
|
30
|
-
initialVariants: Variant[]
|
|
31
|
-
scenarios: Scenario[]
|
|
32
|
-
iterations: number
|
|
33
|
-
// The thing that turns a Variant into a scoreable artifact.
|
|
34
|
-
// For agent-builder this is `runForgeBuilderSim`; for tax-agent it's
|
|
35
|
-
// their domain runner; for the multi-shot prompt evolution case it's
|
|
36
|
-
// already wired inside `runPromptEvolution`.
|
|
37
|
-
candidateRunner: CandidateRunner<Variant>
|
|
38
|
-
// The thing that proposes the next variants given the analysis output.
|
|
39
|
-
// For prompt-only optimization, this is `reflective-mutation` against
|
|
40
|
-
// the top/bottom trials. For code+prompt, this is `createCompositeMutator`.
|
|
41
|
-
// For agent-builder, this can be a hand-rolled "edit the system prompt"
|
|
42
|
-
// function — the example shows one.
|
|
43
|
-
mutator: (champion: Variant, analysis: AnalysisReport) => Promise<Variant[]>
|
|
44
|
-
// Optional: outcome store for predictive validity. When present, the
|
|
45
|
-
// loop learns which scoring rubrics actually predict deployment outcomes
|
|
46
|
-
// and reweights the composite score accordingly.
|
|
47
|
-
outcomes?: { store: OutcomeStore; metrics: string[] }
|
|
48
|
-
}): Promise<IterationReport[]> {
|
|
49
|
-
const reports: IterationReport[] = []
|
|
50
|
-
let variants = opts.initialVariants
|
|
51
|
-
|
|
52
|
-
// (Optional) standing researcher that drives rubric reweighting.
|
|
53
|
-
const researcher = opts.outcomes
|
|
54
|
-
? new PredictiveValidityResearcher({
|
|
55
|
-
outcomes: opts.outcomes.store,
|
|
56
|
-
outcomeMetrics: opts.outcomes.metrics,
|
|
57
|
-
})
|
|
58
|
-
: null
|
|
59
|
-
|
|
60
|
-
for (let iter = 0; iter < opts.iterations; iter++) {
|
|
61
|
-
// 1. Capture-integrity-by-construction matrix run.
|
|
62
|
-
const campaign = await runEvalCampaign({
|
|
63
|
-
campaignId: `auto-research-iter-${iter}`,
|
|
64
|
-
commitSha: opts.task,
|
|
65
|
-
variants: variants.map((v) => ({ id: v.id, payload: v })),
|
|
66
|
-
scenarios: opts.scenarios,
|
|
67
|
-
seeds: [0, 1, 2],
|
|
68
|
-
llmOpts: { ... },
|
|
69
|
-
storeFactory: () => new InMemoryTraceStore(),
|
|
70
|
-
rawSinkFactory: () => new InMemoryRawProviderSink(),
|
|
71
|
-
runner: makeCampaignRunner(opts.candidateRunner),
|
|
72
|
-
onRunComplete: opts.outcomes
|
|
73
|
-
? [traceAnalystOnRunComplete({ analyze: ..., save: ... })]
|
|
74
|
-
: [],
|
|
75
|
-
report: { comparator: variants[0]!.id },
|
|
76
|
-
})
|
|
77
|
-
|
|
78
|
-
// 2. RL-bridge analysis: preferences, verifiable rewards, sequential
|
|
79
|
-
// interim verdict, reward-hacking diagnosis.
|
|
80
|
-
const analysis = await analyzeOptimizationResult({
|
|
81
|
-
result: pretendItsAPromptEvolution(campaign),
|
|
82
|
-
ctx: { experimentId: 'task', model: '...', commitSha: '...', promptHash: '...', configHash: '...' },
|
|
83
|
-
comparator: variants[0]!.id,
|
|
84
|
-
outcomes: opts.outcomes,
|
|
85
|
-
})
|
|
86
|
-
|
|
87
|
-
// 3. Periodic rubric recalibration via predictive validity.
|
|
88
|
-
if (researcher && iter > 0 && iter % 5 === 0) {
|
|
89
|
-
await researcher.runValidityCheck(campaign.runs)
|
|
90
|
-
// The researcher's `proposeChange` output can be folded into the
|
|
91
|
-
// mutator as a steering signal in the next iteration.
|
|
92
|
-
}
|
|
93
|
-
|
|
94
|
-
// 4. Pick champion + record this iteration.
|
|
95
|
-
const champion = pickChampion(campaign.runs)
|
|
96
|
-
reports.push({ iter, champion, score: champion.score, analysis })
|
|
97
|
-
|
|
98
|
-
// 5. Sequential stop: the anytime-valid e-value can decisively call
|
|
99
|
-
// 'promote_now' or 'reject_now' before iterations exhausted.
|
|
100
|
-
if (analysis.interimConfidence?.recommendation.decision === 'promote_now') {
|
|
101
|
-
break
|
|
102
|
-
}
|
|
103
|
-
|
|
104
|
-
// 6. Propose next variants via the mutator.
|
|
105
|
-
if (iter < opts.iterations - 1) {
|
|
106
|
-
variants = await opts.mutator(champion.variant, analysis)
|
|
107
|
-
}
|
|
108
|
-
}
|
|
109
|
-
|
|
110
|
-
return reports
|
|
111
|
-
}
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
## Invariants every iteration must preserve
|
|
115
|
-
|
|
116
|
-
1. **The campaign produces RunRecord[] with `scenarioId` populated.** Every
|
|
117
|
-
downstream primitive (preferences, sequential, predictive validity,
|
|
118
|
-
tournament) keys on this. `runEvalCampaign` populates it canonically;
|
|
119
|
-
if you adapt from `runPromptEvolution` use `trialsToRunRecords`.
|
|
120
|
-
|
|
121
|
-
2. **Capture is wired by construction.** Don't pass `NoopRawProviderSink`
|
|
122
|
-
to `rawSinkFactory` unless the iteration is exploratory. Every
|
|
123
|
-
captured run is replayable, every replayable run is free judge-iteration
|
|
124
|
-
data for the next loop.
|
|
125
|
-
|
|
126
|
-
3. **`commitSha` is real.** It's how downstream tooling (predictive
|
|
127
|
-
validity, contamination probe, tournament) ties iterations together.
|
|
128
|
-
|
|
129
|
-
4. **The comparator is stable across iterations.** Either the original
|
|
130
|
-
`baseline` or whichever champion you froze. Shifting the comparator
|
|
131
|
-
between iterations corrupts the paired-delta semantics.
|
|
132
|
-
|
|
133
|
-
5. **The mutator is deterministic given the analysis output.** Otherwise
|
|
134
|
-
the iteration isn't reproducible and the auto-research artifacts
|
|
135
|
-
become unfalsifiable. If you need stochastic mutation, seed the
|
|
136
|
-
mutator and emit the seed onto the run record.
|
|
137
|
-
|
|
138
|
-
## When to run each primitive
|
|
139
|
-
|
|
140
|
-
| Frequency | Primitive | Why |
|
|
141
|
-
|---|---|---|
|
|
142
|
-
| Every iteration | `runEvalCampaign` | core measurement |
|
|
143
|
-
| Every iteration | `analyzeOptimizationResult` | preferences + verifiable rewards + reward-hacking |
|
|
144
|
-
| Every iteration | `evaluateInterimReleaseConfidence` (via `analyzeOptimizationResult`) | anytime-valid stop signal |
|
|
145
|
-
| Every 5–10 iterations | `rubricPredictiveValidity` | rubric weights drift; recalibrate |
|
|
146
|
-
| Every release | `runContaminationProbe` | scenario set freshness |
|
|
147
|
-
| Once per task | `runComputeCurve` | cost-quality frontier |
|
|
148
|
-
| As-needed | `adversarialScenarioSearch` | discover failure modes the curated set missed |
|
|
149
|
-
|
|
150
|
-
## When to drop into the smaller primitives
|
|
151
|
-
|
|
152
|
-
Two cases:
|
|
153
|
-
|
|
154
|
-
1. **Trajectory-shaped optimization with steering.** Use
|
|
155
|
-
`runImprovementLoop` directly — it already runs the inner
|
|
156
|
-
search-vs-holdout loop. Wrap with `analyzeOptimizationResult` after
|
|
157
|
-
for the RL bridge.
|
|
158
|
-
|
|
159
|
-
2. **Prompt + code evolution with sandboxed code mutation.** Use
|
|
160
|
-
`runPromptEvolution` + `createCompositeMutator` directly. Same wrap
|
|
161
|
-
pattern.
|
|
162
|
-
|
|
163
|
-
The auto-research loop above wraps these primitives in a higher-level
|
|
164
|
-
loop that runs them across multiple campaigns. They're each one tick of
|
|
165
|
-
the bigger loop.
|
|
166
|
-
|
|
167
|
-
## What this does NOT do
|
|
168
|
-
|
|
169
|
-
- It doesn't fine-tune model weights. That's the
|
|
170
|
-
[`fine-tune-with-prime-rl`](../examples/fine-tune-with-prime-rl/) example
|
|
171
|
-
— separate concern, separate trainer.
|
|
172
|
-
- It doesn't drive a production deployment decision on its own. The
|
|
173
|
-
artifacts feed a launch-review process (humans, the `researchReport`
|
|
174
|
-
output, the `assertReleaseConfidence` gate). Loop ≠ promotion gate.
|
|
175
|
-
- It doesn't substitute for a real preregistration trail. The
|
|
176
|
-
`preregistrationHash` field on the report exists so iterations can be
|
|
177
|
-
audited, but the auto-research loop *is* iterative and post-hoc by
|
|
178
|
-
definition. Use the standing `assertReleaseConfidence` gate at the
|
|
179
|
-
release boundary; use the auto-research loop everywhere upstream of it.
|
|
180
|
-
|
|
181
|
-
## Reading order for the example
|
|
182
|
-
|
|
183
|
-
1. [`examples/auto-research-with-agent-builder/README.md`](../examples/auto-research-with-agent-builder/README.md) — architectural picture.
|
|
184
|
-
2. [`examples/auto-research-with-agent-builder/auto-research-with-agent-builder.ts`](../examples/auto-research-with-agent-builder/auto-research-with-agent-builder.ts) — runnable demo.
|
|
185
|
-
3. Run it: `npx tsx examples/auto-research-with-agent-builder/auto-research-with-agent-builder.ts`.
|
|
186
|
-
It prints the iteration progression and the score climbing.
|