@tangle-network/agent-eval 0.20.8 → 0.20.10

package/CHANGELOG.md

@@ -0,0 +1,302 @@
+# Changelog
+
+## 0.20.10 — hardening audit follow-up
+
+### Fixed
+
+- `hashRubric` now recursively sorts nested rubric fields before hashing, so
+  dimension, failure-mode, and win changes alter `rubricVersion`.
+- Wire judge handling now validates LLM output before returning it: finite
+  dimension scores, rationale, and known failure/win ids are enforced.
+- Control-runtime budgets reject invalid numeric config, and invalid action
+  costs are omitted from step telemetry instead of leaking `NaN`/`Infinity`.
+- Knowledge readiness now treats invalid `validUntil` timestamps as stale.
+- Trace-analyst regex search supports leading `(?i)` and stops scanning once
+  bounded match output is reached.
+- SWE-Bench Lite example wording now reflects the implemented external-grader
+  adapter, with quoted command parsing and timeout coverage.
+
+### Changed
+
+- Published package contents now include `CHANGELOG.md`.
+- Public docs now use GitHub URLs for repository-only examples and Python
+  client source.
+- Publish CI now checks npm, Python package, runtime fallback version, and tag
+  version agree before publishing.
+
+## 0.20.9 — release hygiene and runtime failure fixes
+
+### Fixed
+
+- Initial `runAgentControlLoop` observe/validate failures now report the
+  actual observe/validate error even when trace start/end emission also fails.
+- Knowledge readiness recommended actions now honor non-blocking gap
+  acquisition modes such as `ask_user`, `search_web`, `query_connector`, and
+  `inspect_repo`.
+- Npm builds now generate `dist/openapi.json`, and the package exports
+  `@tangle-network/agent-eval/openapi.json`.
+- Npm and Python client versions are locked at `0.20.9`.
+
+### Added
+
+- `CallbackResearcher`, a concrete callback-backed implementation of the
+  stable `Researcher` interface for scripts, tests, and small integrations.
+- Public `@tangle-network/agent-eval/benchmarks` subpath for the supported
+  routing benchmark surface.
+- Root MIT `LICENSE`.
+
+### Changed
+
+- Raw TypeScript examples are no longer included in the npm package; they remain
+  repository examples to read, copy, and adapt.
+
+## 0.20.2 — freshness-aware knowledge readiness
+
+### Added
+
+- `KnowledgeRequirement.validUntil` and `lastVerifiedAt` for explicit freshness
+  contracts.
+- `scoreKnowledgeReadiness({ now })` support for deterministic freshness gates.
+
+### Changed
+
+- Expired knowledge requirements now score as missing even when confidence and
+  evidence are otherwise high.
+
+## 0.20.0 — knowledge readiness contracts
+
+### Added
+
+- First-class knowledge-readiness contracts: `KnowledgeRequirement`,
+  `KnowledgeBundle`, `KnowledgeReadinessReport`, `UserQuestion`, and
+  `DataAcquisitionPlan`.
+- `scoreKnowledgeReadiness`, `blockingKnowledgeEval`,
+  `userQuestionsForKnowledgeGaps`, and `acquisitionPlansForKnowledgeGaps`.
+- Knowledge/data failure classes including `knowledge_readiness_blocked`,
+  `missing_credentials`, `bad_retrieval`, `insufficient_evidence`, and
+  `contradictory_evidence`.
+- `docs/knowledge-readiness.md`, plus documented knowledge-related ASI
+  responsible surfaces for multi-shot optimization.
+
+## 0.19.1 — release confidence gate
+
+### Added
+
+- `evaluateReleaseConfidence`, a conservative release scorecard over corpus
+  coverage, search/holdout run evidence, ASI diagnostics, overfit checks, and
+  cost/latency budgets.
+- `assertReleaseConfidence`, a throwing variant for CI/release scripts.
+- `releaseTraceEvidenceFromMultiShotTrials`, a helper that projects
+  `MultiShotTrialResult` rows into release trace evidence so single-shot and
+  variable multi-shot apps use the same release gate.
+
+## 0.19.0 — legacy optimizer removal
+
+### Removed
+
+- Removed the legacy pairwise prompt optimizer surface:
+  `PromptOptimizer`, `OptimizationLoop`, and their associated root-exported
+  types are gone. The blessed optimization path is now
+  `runMultiShotOptimization` for task trajectories and the steering-specific
+  optimizers for explicit steering tables.
+- Removed the old `PromptVariant` root export. Public callers should use
+  `MultiShotVariant` for multi-shot trajectory optimization or
+  `EvolvableVariant` for the lower-level prompt/code evolution core.
+
+### Changed
+
+- Documentation now points optimization users at `runMultiShotOptimization`
+  instead of the removed pairwise prompt optimizer.
+
+## 0.18.0 — multi-shot optimization
+
+### Added
+
+- `runMultiShotOptimization`, the canonical GEPA-style adapter for
+  variable-length agent trajectories. It wraps `runPromptEvolution` while
+  preserving full multi-shot traces, actionable side information, stable paired
+  seeds, score/cost objectives, and optional held-out promotion gating.
+- `trialTraceFromMultiShotTrial`, a bridge from multi-shot trial results into
+  reflective mutation prompts.
+- `ActionableSideInfo`, `MultiShotVariant`, `MultiShotTrace`, `MultiShotRun`,
+  `MultiShotScore`, `MultiShotTrialResult`, `MultiShotMutateAdapter`, and
+  related public types.
+- `docs/multi-shot-optimization.md` and
+  `examples/multi-shot-optimization/index.ts`.
+
+### Changed
+
+- The multi-shot result shape explicitly separates `searchBestVariant` from
+  `promotedVariant`. If a holdout gate rejects the search winner, the promoted
+  variant is the baseline.
+- `runMultiShotOptimization` validates release-critical configuration up front:
+  unique variant/scenario ids, positive integer run counts, population size,
+  disjoint search/holdout ids, and a gate baseline key matching the first seed
+  variant.
+
+## 0.17.2 — agent control runtime
+
+### Added
+
+- `runAgentControlLoop`, a generic `observe -> validate -> decide -> act`
+  runtime for agentic tasks with step, wall-clock, and recorded-cost budgets;
+  no-progress and repeated-action stop policies; structured runtime failures;
+  objective/subjective eval helpers; and `TraceStore` emission.
+- `runProposeReviewAsControlLoop`, a bridge preset that expresses
+  propose/verify/review as a specialization of the generic control runtime.
+- feedback trajectory helpers for turning control-loop runs and user/judge
+  labels into reusable dataset scenarios, optimizer rows, and preference
+  memory.
+- `docs/control-runtime.md`, with integration patterns for tax, legal,
+  agent-builder, and film-agent products.
+
+### Changed
+
+- control runtime trace sink and `onStep` callback failures are now recorded
+  as structured runtime errors without aborting an otherwise valid run.
+- `runProposeReviewAsControlLoop` accepts a caller-provided verifier failure
+  mapper for domain-specific failure classes.
+
+## 0.17.0 — surface cleanup + usage-guidance pitfalls
+
+This release tightens the public benchmark surface and lands internal usage guidance that the v0.15 dispatch couldn't write.
+
+### Moved
+
+- `src/benchmarks/gsm8k/` → `examples/benchmarks/gsm8k/`
+- `src/benchmarks/swebench-lite/` → `examples/benchmarks/swebench-lite/`
+
+These are reference implementations of `BenchmarkAdapter`, not core surface. Consumers read them, copy them, adapt them. The novel `routing` benchmark stays in `src/benchmarks/` because it's our own and broadly useful.
+
+`src/benchmarks/index.ts` now exports the shared types + the `routing` benchmark only. The previous `gsm8k` and `swebenchLite` namespace exports are gone — import directly from `examples/benchmarks/<name>/index.ts` (or copy the wrapper into your own project).
+
+### Added
+
+- `examples/benchmarks/README.md` documents how to use, copy, and extend the example wrappers.
+- Internal agent-eval usage guidance gains production-rigor and pitfalls sections covering the v0.16 primitives.
+
+### Migration
+
+If you imported `gsm8k` or `swebenchLite` from `@tangle-network/agent-eval/benchmarks`:
+
+```ts
+// before
+import { gsm8k, swebenchLite } from '@tangle-network/agent-eval/benchmarks'
+
+// after — copy the file from examples/benchmarks/<name>/index.ts into your project,
+// or import via relative path from the cloned repo.
+```
+
+The `routing` benchmark and the shared `BenchmarkAdapter` types are unchanged.
+
+## 0.16.0 — naming cleanup
+
+The v0.15 primitives were framed as "paper-grade" but most are production-rigor utilities any team needs. This release renames the three reporting helpers and drops the "paper" framing from the public API. Behavior unchanged.
+
+### Renamed
+
+- `paperTable` → `summaryTable`
+- `paretoFigure` → `paretoChart`
+- `gainDistributionFigure` → `gainHistogram`
+- `PaperTable` / `PaperTableOptions` / `PaperTableRow` types → `SummaryTable` / `SummaryTableOptions` / `SummaryTableRow`
+- File: `src/paper-report.ts` → `src/summary-report.ts`
+
+### Migration
+
+Drop-in: search-and-replace the three function names and the file path. Type names follow the same pattern. No behavior change.
+
+```ts
+// before
+import { paperTable, paretoFigure, gainDistributionFigure } from '@tangle-network/agent-eval'
+// after
+import { summaryTable, paretoChart, gainHistogram } from '@tangle-network/agent-eval'
+```
+
+## 0.15.0 — paper-grade primitives
+
+Substrate for the "Two Loops, Three Roles" paper on multi-level prompt
+optimization with held-out promotion gates.
+
+### Added
+
+- **`HeldOutGate`** (`src/promotion-gate.ts`) — first-class held-out
+  paired-delta promotion gate. Three checks: minimum productive runs,
+  positive lower bound on bootstrap CI of paired holdout median delta,
+  bounded overfit-gap relative to baseline. Decisions carry a
+  machine-readable `rejectionCode` (`few_runs` | `negative_delta` |
+  `overfit_gap`) plus an `evidence` block with every number the gate
+  read. Generalizes the inline pattern that lived in
+  `redteam/scripts/agent-eval-autoresearch.ts:138–171`.
+- **`RunRecord`** (`src/run-record.ts`) — paper-grade JSON-friendly run
+  schema with mandatory fields: `runId`, `experimentId`, `candidateId`,
+  `seed`, snapshot-versioned `model`, `promptHash`, `configHash`,
+  `commitSha`, `wallMs`, `costUsd`, `tokenUsage`, `outcome`, `splitTag`.
+  Runtime validator (`validateRunRecord`, `isRunRecord`,
+  `parseRunRecordSafe`, `roundTripRunRecord`) throws on missing fields
+  and on bare model aliases without snapshot suffix.
+- **`Researcher`** (`src/researcher.ts`) — stable hook for an
+  autonomous-research agent: `inspectFailures` → `proposeChange` →
+  `applyChange` → `evaluateChange`. `NoopResearcher` is the
+  fail-loud placeholder. Implementations live downstream.
+- **Reference benchmarks** (`src/benchmarks/`) — three adapters that
+  share the `BenchmarkAdapter<TItem, TPayload>` shape:
+  - `gsm8k`: HF-mirror loader (JSONL via `AGENT_EVAL_GSM8K_PATH`),
+    exact-match grading via `parseGsm8kAnswer`.
+  - `swebench-lite`: 30-instance subset stub. Loader reads
+    `AGENT_EVAL_SWEBENCH_PATH`; grader shells out to
+    `AGENT_EVAL_SWEBENCH_GRADER_CMD`. Both fail loud when unset.
+  - `routing`: synthetic 16-task router benchmark, ships in the
+    package, dependency-free. Format documented in
+    `src/benchmarks/routing/README.md`.
+  - `deterministicSplit(itemId, seed?)`: stable 60/20/20 split via
+    FNV-1a hash. Default seed `agent-eval-v1`.
+- **`summaryTable`, `paretoChart`, `gainHistogram`**
+  (`sr./summary-report.ts`) — Table 1 + Pareto + gain-distribution specs.
+  Returns data structures (markdown table, point lists, histogram bins);
+  caller picks the plotting library.
+- **`runCanaries`** (`src/canary.ts`) — three liveness canaries:
+  silent judge fallback (consecutive constant-confidence streak),
+  judge calibration drift (KS test on confidence distribution), eval-set
+  distribution shift (chi-square on category bucket counts).
+- **`pairedBootstrap`, `pairedWilcoxon`, `bhAdjust`**
+  (`src/paired-stats.ts`) — paper-style aliases + the missing paired
+  bootstrap CI primitive. Deterministic with optional seed.
+
+### Notes
+
+- No breaking changes. Every existing module is untouched; new types
+  are additive.
+- All new public symbols carry JSDoc.
+- 87 new tests across 7 new test files. 571 total tests pass.
+- See the package docs for usage directives and pitfalls.
+
+## 0.11.0
+
+intent-match + flow-layer + deploy-gate + concept complexity
+weighting.
+
+## 0.10.0
+
+`LayerResult.diagnostics` + `buildReviewerPrompt` +
+`createDefaultReviewer` + `mergeLayerResults` options.
+
+## 0.9.0
+
+`CommandRunner` contract + `multiToolchainLayer` + `Finding.detail`.
+
+## 0.8.x
+
+`probeLlm` + `keyword-coverage-judge`. Honestly-absent primitives
+backfilled — `llm-client`, multi-layer verifier, semantic concept judge,
+extractor utilities.
+
+## 0.7.x
+
+Extracted muffled-gate scanner; `CostTracker.recordVerdict`. Footgun
+fix: `cwd` belongs in `HarnessConfig`, not the driver constructor.
+
+## 0.6.x
+
+Tier 1 (meta-eval correlation, PRM, bisector), Tier 2 (counterfactual,
+cross-trace diff, pre-registration), Tier 3 (self-play, causal
+attribution, active learning, RM export), governance templates.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tangle Network
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -55,9 +55,9 @@ Package responsibilities:
   optimization, reporting.
 - Product app: domain state, tools, credentials, UI, storage, deployment, model
   gateway.
-- `agent-runtime`: production agent-loop/session runtime.
-- `agent-knowledge`: evidence stores, claim/page synthesis, retrieval, knowledge
-  readiness implementation.
+- `@tangle-network/agent-runtime`: production agent-loop/session runtime.
+- `@tangle-network/agent-knowledge`: evidence stores, claim/page synthesis,
+  retrieval, knowledge readiness implementation.
 
 ## Install
 
@@ -72,10 +72,12 @@ npm i -g @tangle-network/agent-eval
 agent-eval serve --port 5005
 ```
 
-Python client:
+Python client source lives in `clients/python`. Until the PyPI package is
+published, install it from the repo:
 
 ```sh
-pip install tangle-agent-eval
+cd clients/python
+pip install -e .
 ```
 
 ## Core Primitives
@@ -96,15 +98,20 @@ pip install tangle-agent-eval
 | `summaryTable`, `paretoChart`, `gainHistogram` | Report-ready structured outputs. |
 | `KnowledgeRequirement`, `KnowledgeBundle` | Shared contracts for knowledge readiness. |
 
+`NoopResearcher` is a fail-loud sentinel for wiring tests. Production systems
+should implement `Researcher` directly or use `CallbackResearcher`.
+
 ## Examples
 
-Runnable examples live in [`examples/`](./examples):
+Runnable examples live in the repository's
+[`examples/`](https://github.com/tangle-network/agent-eval/tree/main/examples)
+directory. They are not part of the published npm package.
 
-- [`examples/same-sandbox-harness`](./examples/same-sandbox-harness) - run
+- [`examples/same-sandbox-harness`](https://github.com/tangle-network/agent-eval/tree/main/examples/same-sandbox-harness) - run
   multiple eval passes against the same workspace.
-- [`examples/multi-shot-optimization`](./examples/multi-shot-optimization) -
+- [`examples/multi-shot-optimization`](https://github.com/tangle-network/agent-eval/tree/main/examples/multi-shot-optimization) -
   optimize full agent trajectories with held-out promotion.
-- [`examples/benchmarks`](./examples/benchmarks) - benchmark adapter shape and
+- [`examples/benchmarks`](https://github.com/tangle-network/agent-eval/tree/main/examples/benchmarks) - benchmark adapter shape and
   reference benchmark wrappers.
 
 The examples are intentionally kept outside the README so they can be expanded,

package/dist/benchmarks/index.d.ts

@@ -0,0 +1 @@
+export { B as BENCHMARK_SPLIT_SEED, b as BenchmarkAdapter, c as BenchmarkDatasetItem, d as BenchmarkEvaluation, i as deterministicSplit, l as routing } from '../index-1PZOtZFr.js';

package/dist/benchmarks/index.js

@@ -0,0 +1,12 @@
+import {
+  BENCHMARK_SPLIT_SEED,
+  deterministicSplit,
+  routing_exports
+} from "../chunk-42I2QC2L.js";
+import "../chunk-PZ5AY32C.js";
+export {
+  BENCHMARK_SPLIT_SEED,
+  deterministicSplit,
+  routing_exports as routing
+};
+//# sourceMappingURL=index.js.map

package/dist/benchmarks/index.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":[],"sourcesContent":[],"mappings":"","names":[]}

package/dist/chunk-42I2QC2L.js

@@ -0,0 +1,219 @@
+import {
+  __export
+} from "./chunk-PZ5AY32C.js";
+
+// src/benchmarks/index.ts
+var benchmarks_exports = {};
+__export(benchmarks_exports, {
+  BENCHMARK_SPLIT_SEED: () => BENCHMARK_SPLIT_SEED,
+  deterministicSplit: () => deterministicSplit,
+  routing: () => routing_exports
+});
+
+// src/benchmarks/types.ts
+function fnv1a32(input) {
+  let h = 2166136261;
+  for (let i = 0; i < input.length; i++) {
+    h ^= input.charCodeAt(i) & 255;
+    h = h + ((h << 1) + (h << 4) + (h << 7) + (h << 8) + (h << 24)) >>> 0;
+  }
+  return h >>> 0;
+}
+var BENCHMARK_SPLIT_SEED = "agent-eval-v1";
+function deterministicSplit(itemId, seed = BENCHMARK_SPLIT_SEED) {
+  const h = fnv1a32(`${seed}::${itemId}`);
+  const pos = h / 4294967296;
+  if (pos < 0.6) return "search";
+  if (pos < 0.8) return "dev";
+  return "holdout";
+}
+
+// src/benchmarks/routing/index.ts
+var routing_exports = {};
+__export(routing_exports, {
+  ROUTING_DATASET: () => ROUTING_DATASET,
+  RoutingAdapter: () => RoutingAdapter,
+  assignSplit: () => assignSplit,
+  evaluate: () => evaluate,
+  extractRouteTokens: () => extractRouteTokens,
+  loadDataset: () => loadDataset
+});
+
+// src/benchmarks/routing/dataset.ts
+var ROUTING_DATASET = [
+  {
+    id: "file_001",
+    category: "file",
+    prompt: "Save the meeting notes to /tmp/notes-2025-04.md as markdown.",
+    route: "fs.write",
+    synonyms: ["filesystem.write", "write_file"],
+    hardNegatives: ["fs.read", "chat.reply"]
+  },
+  {
+    id: "file_002",
+    category: "file",
+    prompt: "Read the contents of /etc/hosts and summarize the entries.",
+    route: "fs.read",
+    synonyms: ["filesystem.read", "read_file"],
+    hardNegatives: ["fs.write", "search.web"]
+  },
+  {
+    id: "file_003",
+    category: "file",
+    prompt: "List every Python file under src/ recursively.",
+    route: "fs.list",
+    synonyms: ["filesystem.list", "list_files"],
+    hardNegatives: ["fs.read", "search.code"]
+  },
+  {
+    id: "file_004",
+    category: "file",
+    prompt: "Delete the cached build at .turbo/cache.",
+    route: "fs.delete",
+    synonyms: ["filesystem.delete", "remove_file"],
+    hardNegatives: ["fs.write", "fs.list"]
+  },
+  {
+    id: "math_001",
+    category: "math",
+    prompt: "What is the integral of 3x^2 + 2x from 0 to 5?",
+    route: "math.integral",
+    synonyms: ["calculator.integral", "math.solve"],
+    hardNegatives: ["math.derivative", "chat.reply"]
+  },
+  {
+    id: "math_002",
+    category: "math",
+    prompt: "Compute the derivative of sin(x) * cos(x).",
+    route: "math.derivative",
+    synonyms: ["calculator.derivative", "math.solve"],
+    hardNegatives: ["math.integral", "math.algebra"]
+  },
+  {
+    id: "math_003",
+    category: "math",
+    prompt: "Solve 2x + 7 = 19 for x.",
+    route: "math.algebra",
+    synonyms: ["calculator.algebra", "math.solve"],
+    hardNegatives: ["math.derivative", "math.integral"]
+  },
+  {
+    id: "math_004",
+    category: "math",
+    prompt: "What is the prime factorization of 360?",
+    route: "math.numbertheory",
+    synonyms: ["calculator.factor", "math.solve"],
+    hardNegatives: ["math.algebra", "search.web"]
+  },
+  {
+    id: "search_001",
+    category: "search",
+    prompt: "Find recent papers on agent prompt optimization with held-out promotion gates.",
+    route: "search.web",
+    synonyms: ["web.search", "search.papers"],
+    hardNegatives: ["search.code", "chat.reply"]
+  },
+  {
+    id: "search_002",
+    category: "search",
+    prompt: "Search the codebase for every call site of `runProposeReview`.",
+    route: "search.code",
+    synonyms: ["code.search", "grep"],
+    hardNegatives: ["search.web", "fs.read"]
+  },
+  {
+    id: "search_003",
+    category: "search",
+    prompt: "What is the latest release of the Tangle network on GitHub?",
+    route: "search.web",
+    synonyms: ["web.search", "github.releases"],
+    hardNegatives: ["search.code", "chat.reply"]
+  },
+  {
+    id: "search_004",
+    category: "search",
+    prompt: "Find all TODO comments in the agent-eval src tree.",
+    route: "search.code",
+    synonyms: ["code.search", "grep"],
+    hardNegatives: ["search.web", "fs.list"]
+  },
+  {
+    id: "chat_001",
+    category: "chat",
+    prompt: "Hi there, how are you doing today?",
+    route: "chat.reply",
+    synonyms: ["conversation.reply"],
+    hardNegatives: ["search.web", "fs.read"]
+  },
+  {
+    id: "chat_002",
+    category: "chat",
+    prompt: "Please explain the difference between an LLM and a foundation model.",
+    route: "chat.reply",
+    synonyms: ["conversation.reply", "qa.answer"],
+    hardNegatives: ["search.web", "math.algebra"]
+  },
+  {
+    id: "chat_003",
+    category: "chat",
+    prompt: "Tell me a short joke about distributed systems.",
+    route: "chat.reply",
+    synonyms: ["conversation.reply"],
+    hardNegatives: ["search.web", "fs.read"]
+  },
+  {
+    id: "chat_004",
+    category: "chat",
+    prompt: "Acknowledge my last message with a thumbs up.",
+    route: "chat.reply",
+    synonyms: ["conversation.reply", "react"],
+    hardNegatives: ["fs.write", "search.web"]
+  }
+];
+
+// src/benchmarks/routing/index.ts
+var RoutingAdapter = class {
+  async loadDataset(split) {
+    return ROUTING_DATASET.map((item) => ({ id: item.id, payload: item })).filter((it) => assignSplitImpl(it.id) === split);
+  }
+  async evaluate(item, response) {
+    const tokens = extractRouteTokens(response);
+    const correct = new Set([item.payload.route, ...item.payload.synonyms].map((s) => s.toLowerCase()));
+    const hardNeg = new Set(item.payload.hardNegatives.map((s) => s.toLowerCase()));
+    const firstMatch = tokens.find((t) => correct.has(t.toLowerCase())) ?? null;
+    const firstHardNeg = tokens.find((t) => hardNeg.has(t.toLowerCase())) ?? null;
+    const score = firstMatch ? 1 : 0;
+    return {
+      score,
+      raw: {
+        firstToken: tokens[0] ?? null,
+        matchedRoute: firstMatch,
+        hitHardNegative: Boolean(firstHardNeg),
+        hardNegativeRoute: firstHardNeg,
+        category: item.payload.category
+      }
+    };
+  }
+  assignSplit(itemId) {
+    return assignSplitImpl(itemId);
+  }
+};
+function assignSplitImpl(itemId) {
+  return deterministicSplit(`routing::${itemId}`);
+}
+function extractRouteTokens(response) {
+  const matches = response.match(/[a-z][a-z0-9_]*\.[a-z][a-z0-9_]*/gi);
+  return matches ?? [];
+}
+var adapter = new RoutingAdapter();
+var loadDataset = adapter.loadDataset.bind(adapter);
+var evaluate = adapter.evaluate.bind(adapter);
+var assignSplit = adapter.assignSplit.bind(adapter);
+
+export {
+  BENCHMARK_SPLIT_SEED,
+  deterministicSplit,
+  routing_exports,
+  benchmarks_exports
+};
+//# sourceMappingURL=chunk-42I2QC2L.js.map

package/dist/chunk-42I2QC2L.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../src/benchmarks/index.ts","../src/benchmarks/types.ts","../src/benchmarks/routing/index.ts","../src/benchmarks/routing/dataset.ts"],"sourcesContent":["/**\n * Reference benchmark wrappers — entry point.\n *\n * Core surface (exported here):\n *   - The `BenchmarkAdapter` contract.\n *   - `deterministicSplit` + `BENCHMARK_SPLIT_SEED` for split assignment.\n *   - `routing` — synthetic 16-task router benchmark. The only novel\n *     benchmark we built; ships in the package.\n *\n * Example wrappers (under `examples/benchmarks/`, NOT in the bundle):\n *   - `gsm8k`         — exact-match math reasoning (HF mirror, dataset\n *                       not bundled).\n *   - `swebench-lite` — 30-instance SWE-Bench subset via an external\n *                       grader command.\n *\n * The example wrappers are reference implementations of `BenchmarkAdapter`.\n * Read them, copy them, adapt them. They're intentionally not in the main\n * entry — every team will configure them differently.\n */\n\nexport type {\n  BenchmarkAdapter,\n  BenchmarkDatasetItem,\n  BenchmarkEvaluation,\n} from './types'\nexport { deterministicSplit, BENCHMARK_SPLIT_SEED } from './types'\n\nexport * as routing from './routing/index'\n","/**\n * Shared types for the reference benchmark wrappers under\n * `src/benchmarks/`. Each wrapper exports the three functions in\n * `BenchmarkAdapter` plus its own typed `DatasetItem` shape.\n */\n\nimport type { RunSplitTag } from '../run-record'\n\nexport interface BenchmarkDatasetItem<TPayload = unknown> {\n  /** Stable dataset-local item id (used for split assignment + paper\n   *  references). Unique within a benchmark. */\n  id: string\n  /** Free-form payload. Each benchmark defines its own shape. */\n  payload: TPayload\n}\n\nexport interface BenchmarkEvaluation {\n  /** [0, 1] score for the response on this item. Exact-match\n   *  benchmarks use 0/1; partial-credit benchmarks may return\n   *  fractional values. */\n  score: number\n  /** Optional bag of raw scoring signals — e.g. parsed numeric\n   *  answer, regex match, judge sub-scores. */\n  raw: Record<string, unknown>\n}\n\n/** Common signature implemented by every adapter under `src/benchmarks/*`. */\n// `TPayload` is the per-item payload type; `_TItem` is preserved for\n// downstream type-narrowing extensions (a richer `BenchmarkDatasetItem`\n// subclass that adds e.g. provenance metadata) but is intentionally\n// unused here. `noUnusedLocals` requires the leading underscore.\nexport interface BenchmarkAdapter<_TItem = unknown, TPayload = unknown> {\n  /** Load the dataset for the given split. May hit the network on\n   *  first call but should be cache-friendly. Adapters that don't\n   *  ship the dataset itself MUST throw a clearly-marked error\n   *  pointing the caller at the loader script. */\n  loadDataset(split: RunSplitTag): Promise<BenchmarkDatasetItem<TPayload>[]>\n  /** Score a single response. Pure with respect to the inputs. */\n  evaluate(item: BenchmarkDatasetItem<TPayload>, response: string): Promise<BenchmarkEvaluation>\n  /** Deterministic split assignment via item id hashing. The\n   *  fraction of items in each split is implementation-defined but\n   *  MUST be stable across processes and platforms. */\n  assignSplit(itemId: string): RunSplitTag\n}\n\n// ── Deterministic split assignment ───────────────────────────────────\n\n/**\n * 32-bit FNV-1a hash. Stable, allocation-free, deterministic across\n * runtimes. We use it to assign items to splits rather than depending\n * on a polyfilled crypto.subtle path.\n */\nfunction fnv1a32(input: string): number {\n  let h = 0x811c9dc5\n  for (let i = 0; i < input.length; i++) {\n    h ^= input.charCodeAt(i) & 0xff\n    h = (h + ((h << 1) + (h << 4) + (h << 7) + (h << 8) + (h << 24))) >>> 0\n  }\n  return h >>> 0\n}\n\n/** Split-assignment seed shared across all benchmarks. Bumping this\n *  value reshuffles every split — do NOT do that lightly. */\nexport const BENCHMARK_SPLIT_SEED = 'agent-eval-v1'\n\n/**\n * Assign an item id to one of `'search' | 'dev' | 'holdout'` using a\n * stable 32-bit hash of `${seed}::${id}`. Default proportions:\n *\n *   search: 60%   (optimization-readable)\n *   dev:    20%   (held-out for tuning, leak-on-purpose during dev)\n *   holdout:20%   (paper-grade held-out, gated reads)\n */\nexport function deterministicSplit(\n  itemId: string,\n  seed: string = BENCHMARK_SPLIT_SEED,\n): RunSplitTag {\n  const h = fnv1a32(`${seed}::${itemId}`)\n  const pos = h / 0x100000000\n  if (pos < 0.6) return 'search'\n  if (pos < 0.8) return 'dev'\n  return 'holdout'\n}\n","/**\n * Routing benchmark — synthetic, dependency-free, ships in the\n * package. 16 cross-category items in `dataset.ts`. See\n * `routing/README.md` for the format.\n *\n * `evaluate` does case-insensitive exact match against the canonical\n * route plus declared synonyms. The first valid route token in the\n * response wins; everything else is ignored. Wrong answers also\n * report whether they hit a hard negative — useful when triaging\n * \"always picks the popular route\" failure modes.\n */\n\nimport type {\n  BenchmarkAdapter,\n  BenchmarkDatasetItem,\n  BenchmarkEvaluation,\n} from '../types'\nimport { deterministicSplit } from '../types'\nimport type { RunSplitTag } from '../../run-record'\nimport { ROUTING_DATASET, type RoutingItem } from './dataset'\n\nexport type { RoutingItem }\nexport type RoutingPayload = RoutingItem\nexport type RoutingDatasetItem = BenchmarkDatasetItem<RoutingPayload>\n\nclass RoutingAdapter\n  implements BenchmarkAdapter<RoutingDatasetItem, RoutingPayload>\n{\n  async loadDataset(split: RunSplitTag): Promise<RoutingDatasetItem[]> {\n    return ROUTING_DATASET\n      .map((item) => ({ id: item.id, payload: item }))\n      .filter((it) => assignSplitImpl(it.id) === split)\n  }\n\n  async evaluate(\n    item: RoutingDatasetItem,\n    response: string,\n  ): Promise<BenchmarkEvaluation> {\n    const tokens = extractRouteTokens(response)\n    const correct = new Set<string>([item.payload.route, ...item.payload.synonyms].map((s) => s.toLowerCase()))\n    const hardNeg = new Set<string>(item.payload.hardNegatives.map((s) => s.toLowerCase()))\n    const firstMatch = tokens.find((t) => correct.has(t.toLowerCase())) ?? null\n    const firstHardNeg = tokens.find((t) => hardNeg.has(t.toLowerCase())) ?? null\n    const score = firstMatch ? 1 : 0\n    return {\n      score,\n      raw: {\n        firstToken: tokens[0] ?? null,\n        matchedRoute: firstMatch,\n        hitHardNegative: Boolean(firstHardNeg),\n        hardNegativeRoute: firstHardNeg,\n        category: item.payload.category,\n      },\n    }\n  }\n\n  assignSplit(itemId: string): RunSplitTag {\n    return assignSplitImpl(itemId)\n  }\n}\n\nfunction assignSplitImpl(itemId: string): RunSplitTag {\n  return deterministicSplit(`routing::${itemId}`)\n}\n\n/**\n * Pull route-shaped tokens out of a model response. Routes look like\n * `category.action` (`fs.write`, `chat.reply`). Bare alphanumerics\n * are not routes, but `category.action` patterns are robust to most\n * model wrappers (JSON output, prose explanations, code fences).\n */\nexport function extractRouteTokens(response: string): string[] {\n  const matches = response.match(/[a-z][a-z0-9_]*\\.[a-z][a-z0-9_]*/gi)\n  return matches ?? []\n}\n\nconst adapter = new RoutingAdapter()\n\nexport const loadDataset = adapter.loadDataset.bind(adapter)\nexport const evaluate = adapter.evaluate.bind(adapter)\nexport const assignSplit = adapter.assignSplit.bind(adapter)\nexport { RoutingAdapter, ROUTING_DATASET }\n","/**\n * Synthetic routing dataset. 16 tasks across 4 categories. Used as a\n * deterministic, dependency-free benchmark for any router that maps a\n * natural-language request to one of a fixed set of route labels.\n *\n * Format (see `routing/README.md` for prose):\n *\n *   {\n *     id:           stable per-task ID (matches across processes).\n *     category:     one of the four route labels.\n *     prompt:       the user-facing request the router must classify.\n *     route:        the ground-truth route the router should pick.\n *     synonyms:     other strings that count as a correct answer.\n *     hardNegatives:close-but-wrong route labels — used to detect the\n *                   \"always picks the popular route\" failure mode.\n *   }\n *\n * The four categories are intentionally cross-domain (file ops,\n * math, search, conversation) so a router that collapses to one\n * category is easy to spot.\n */\n\nexport interface RoutingItem {\n  id: string\n  category: 'file' | 'math' | 'search' | 'chat'\n  prompt: string\n  /** Canonical correct route label. */\n  route: string\n  /** Alternate route labels that also count as correct. */\n  synonyms: string[]\n  /** Wrong-but-tempting route labels (for analysis, not grading). */\n  hardNegatives: string[]\n}\n\nexport const ROUTING_DATASET: RoutingItem[] = [\n  {\n    id: 'file_001',\n    category: 'file',\n    prompt: 'Save the meeting notes to /tmp/notes-2025-04.md as markdown.',\n    route: 'fs.write',\n    synonyms: ['filesystem.write', 'write_file'],\n    hardNegatives: ['fs.read', 'chat.reply'],\n  },\n  {\n    id: 'file_002',\n    category: 'file',\n    prompt: 'Read the contents of /etc/hosts and summarize the entries.',\n    route: 'fs.read',\n    synonyms: ['filesystem.read', 'read_file'],\n    hardNegatives: ['fs.write', 'search.web'],\n  },\n  {\n    id: 'file_003',\n    category: 'file',\n    prompt: 'List every Python file under src/ recursively.',\n    route: 'fs.list',\n    synonyms: ['filesystem.list', 'list_files'],\n    hardNegatives: ['fs.read', 'search.code'],\n  },\n  {\n    id: 'file_004',\n    category: 'file',\n    prompt: 'Delete the cached build at .turbo/cache.',\n    route: 'fs.delete',\n    synonyms: ['filesystem.delete', 'remove_file'],\n    hardNegatives: ['fs.write', 'fs.list'],\n  },\n  {\n    id: 'math_001',\n    category: 'math',\n    prompt: 'What is the integral of 3x^2 + 2x from 0 to 5?',\n    route: 'math.integral',\n    synonyms: ['calculator.integral', 'math.solve'],\n    hardNegatives: ['math.derivative', 'chat.reply'],\n  },\n  {\n    id: 'math_002',\n    category: 'math',\n    prompt: 'Compute the derivative of sin(x) * cos(x).',\n    route: 'math.derivative',\n    synonyms: ['calculator.derivative', 'math.solve'],\n    hardNegatives: ['math.integral', 'math.algebra'],\n  },\n  {\n    id: 'math_003',\n    category: 'math',\n    prompt: 'Solve 2x + 7 = 19 for x.',\n    route: 'math.algebra',\n    synonyms: ['calculator.algebra', 'math.solve'],\n    hardNegatives: ['math.derivative', 'math.integral'],\n  },\n  {\n    id: 'math_004',\n    category: 'math',\n    prompt: 'What is the prime factorization of 360?',\n    route: 'math.numbertheory',\n    synonyms: ['calculator.factor', 'math.solve'],\n    hardNegatives: ['math.algebra', 'search.web'],\n  },\n  {\n    id: 'search_001',\n    category: 'search',\n    prompt: 'Find recent papers on agent prompt optimization with held-out promotion gates.',\n    route: 'search.web',\n    synonyms: ['web.search', 'search.papers'],\n    hardNegatives: ['search.code', 'chat.reply'],\n  },\n  {\n    id: 'search_002',\n    category: 'search',\n    prompt: 'Search the codebase for every call site of `runProposeReview`.',\n    route: 'search.code',\n    synonyms: ['code.search', 'grep'],\n    hardNegatives: ['search.web', 'fs.read'],\n  },\n  {\n    id: 'search_003',\n    category: 'search',\n    prompt: 'What is the latest release of the Tangle network on GitHub?',\n    route: 'search.web',\n    synonyms: ['web.search', 'github.releases'],\n    hardNegatives: ['search.code', 'chat.reply'],\n  },\n  {\n    id: 'search_004',\n    category: 'search',\n    prompt: 'Find all TODO comments in the agent-eval src tree.',\n    route: 'search.code',\n    synonyms: ['code.search', 'grep'],\n    hardNegatives: ['search.web', 'fs.list'],\n  },\n  {\n    id: 'chat_001',\n    category: 'chat',\n    prompt: 'Hi there, how are you doing today?',\n    route: 'chat.reply',\n    synonyms: ['conversation.reply'],\n    hardNegatives: ['search.web', 'fs.read'],\n  },\n  {\n    id: 'chat_002',\n    category: 'chat',\n    prompt: 'Please explain the difference between an LLM and a foundation model.',\n    route: 'chat.reply',\n    synonyms: ['conversation.reply', 'qa.answer'],\n    hardNegatives: ['search.web', 'math.algebra'],\n  },\n  {\n    id: 'chat_003',\n    category: 'chat',\n    prompt: 'Tell me a short joke about distributed systems.',\n    route: 'chat.reply',\n    synonyms: ['conversation.reply'],\n    hardNegatives: ['search.web', 'fs.read'],\n  },\n  {\n    id: 'chat_004',\n    category: 'chat',\n    prompt: 'Acknowledge my last message with a thumbs up.',\n    route: 'chat.reply',\n    synonyms: ['conversation.reply', 'react'],\n    hardNegatives: ['fs.write', 'search.web'],\n  },\n]\n"],"mappings":";;;;;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;;;ACoDA,SAAS,QAAQ,OAAuB;AACtC,MAAI,IAAI;AACR,WAAS,IAAI,GAAG,IAAI,MAAM,QAAQ,KAAK;AACrC,SAAK,MAAM,WAAW,CAAC,IAAI;AAC3B,QAAK,MAAM,KAAK,MAAM,KAAK,MAAM,KAAK,MAAM,KAAK,MAAM,KAAK,SAAU;AAAA,EACxE;AACA,SAAO,MAAM;AACf;AAIO,IAAM,uBAAuB;AAU7B,SAAS,mBACd,QACA,OAAe,sBACF;AACb,QAAM,IAAI,QAAQ,GAAG,IAAI,KAAK,MAAM,EAAE;AACtC,QAAM,MAAM,IAAI;AAChB,MAAI,MAAM,IAAK,QAAO;AACtB,MAAI,MAAM,IAAK,QAAO;AACtB,SAAO;AACT;;;AClFA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;;;ACkCO,IAAM,kBAAiC;AAAA,EAC5C;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,oBAAoB,YAAY;AAAA,IAC3C,eAAe,CAAC,WAAW,YAAY;AAAA,EACzC;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,mBAAmB,WAAW;AAAA,IACzC,eAAe,CAAC,YAAY,YAAY;AAAA,EAC1C;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,mBAAmB,YAAY;AAAA,IAC1C,eAAe,CAAC,WAAW,aAAa;AAAA,EAC1C;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,qBAAqB,aAAa;AAAA,IAC7C,eAAe,CAAC,YAAY,SAAS;AAAA,EACvC;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,uBAAuB,YAAY;AAAA,IAC9C,eAAe,CAAC,mBAAmB,YAAY;AAAA,EACjD;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,yBAAyB,YAAY;AAAA,IAChD,eAAe,CAAC,iBAAiB,cAAc;AAAA,EACjD;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,sBAAsB,YAAY;AAAA,IAC7C,eAAe,CAAC,mBAAmB,eAAe;AAAA,EACpD;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,qBAAqB,YAAY;AAAA,IAC5C,eAAe,CAAC,gBAAgB,YAAY;AAAA,EAC9C;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,cAAc,eAAe;AAAA,IACxC,eAAe,CAAC,eAAe,YAAY;AAAA,EAC7C;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,eAAe,MAAM;AAAA,IAChC,eAAe,CAAC,cAAc,SAAS;AAAA,EACzC;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,cAAc,iBAAiB;AAAA,IAC1C,eAAe,CAAC,eAAe,YAAY;AAAA,EAC7C;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,eAAe,MAAM;AAAA,IAChC,eAAe,CAAC,cAAc,SAAS;AAAA,EACzC;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,oBAAoB;AAAA,IAC/B,eAAe,CAAC,cAAc,SAAS;AAAA,EACzC;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,sBAAsB,WAAW;AAAA,IAC5C,eAAe,CAAC,cAAc,cAAc;AAAA,EAC9C;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,oBAAoB;AAAA,IAC/B,eAAe,CAAC,cAAc,SAAS;AAAA,EACzC;AAAA,EACA;AAAA,IACE,IAAI;AAAA,IACJ,UAAU;AAAA,IACV,QAAQ;AAAA,IACR,OAAO;AAAA,IACP,UAAU,CAAC,sBAAsB,OAAO;AAAA,IACxC,eAAe,CAAC,YAAY,YAAY;AAAA,EAC1C;AACF;;;AD1IA,IAAM,iBAAN,MAEA;AAAA,EACE,MAAM,YAAY,OAAmD;AACnE,WAAO,gBACJ,IAAI,CAAC,UAAU,EAAE,IAAI,KAAK,IAAI,SAAS,KAAK,EAAE,EAC9C,OAAO,CAAC,OAAO,gBAAgB,GAAG,EAAE,MAAM,KAAK;AAAA,EACpD;AAAA,EAEA,MAAM,SACJ,MACA,UAC8B;AAC9B,UAAM,SAAS,mBAAmB,QAAQ;AAC1C,UAAM,UAAU,IAAI,IAAY,CAAC,KAAK,QAAQ,OAAO,GAAG,KAAK,QAAQ,QAAQ,EAAE,IAAI,CAAC,MAAM,EAAE,YAAY,CAAC,CAAC;AAC1G,UAAM,UAAU,IAAI,IAAY,KAAK,QAAQ,cAAc,IAAI,CAAC,MAAM,EAAE,YAAY,CAAC,CAAC;AACtF,UAAM,aAAa,OAAO,KAAK,CAAC,MAAM,QAAQ,IAAI,EAAE,YAAY,CAAC,CAAC,KAAK;AACvE,UAAM,eAAe,OAAO,KAAK,CAAC,MAAM,QAAQ,IAAI,EAAE,YAAY,CAAC,CAAC,KAAK;AACzE,UAAM,QAAQ,aAAa,IAAI;AAC/B,WAAO;AAAA,MACL;AAAA,MACA,KAAK;AAAA,QACH,YAAY,OAAO,CAAC,KAAK;AAAA,QACzB,cAAc;AAAA,QACd,iBAAiB,QAAQ,YAAY;AAAA,QACrC,mBAAmB;AAAA,QACnB,UAAU,KAAK,QAAQ;AAAA,MACzB;AAAA,IACF;AAAA,EACF;AAAA,EAEA,YAAY,QAA6B;AACvC,WAAO,gBAAgB,MAAM;AAAA,EAC/B;AACF;AAEA,SAAS,gBAAgB,QAA6B;AACpD,SAAO,mBAAmB,YAAY,MAAM,EAAE;AAChD;AAQO,SAAS,mBAAmB,UAA4B;AAC7D,QAAM,UAAU,SAAS,MAAM,oCAAoC;AACnE,SAAO,WAAW,CAAC;AACrB;AAEA,IAAM,UAAU,IAAI,eAAe;AAE5B,IAAM,cAAc,QAAQ,YAAY,KAAK,OAAO;AACpD,IAAM,WAAW,QAAQ,SAAS,KAAK,OAAO;AAC9C,IAAM,cAAc,QAAQ,YAAY,KAAK,OAAO;","names":[]}