benchmark-latency-budget 0.0.0-stage → 0.1.0

package/AGENTS.md

@@ -0,0 +1,17 @@
+# Working on benchmark-latency-budget
+
+## Commands
+
+- Install with `npm ci`.
+- Run the full gate with `npm run ci:local`.
+- Generate tasks with `npm run generate:tasks`.
+- Generate provenance after building with `npm run provenance`.
+- Add no unit tests. Extend `scripts/e2e.mjs` through a public surface.
+
+## Contracts
+
+Keep tasks, deterministic grading, and provenance in sync. Episode environments must be seeded. Grade episode traces by replaying their tool log against a fresh simulator. Time is virtual: scheduled charge per call plus model turn time (measured wall-clock from the environment, or declared thinkMs per turn when the clock is declared). Write the failure modes in docs/grading.md before changing grader code.
+
+Never read provider keys from repository files. OpenRouter uses `OPENROUTER_API_KEY`. OpenAI-compatible providers use `OPENAI_BASE_URL` and `OPENAI_API_KEY`.
+
+Open an issue before a branch. Use `blbd-<issue>-<slug>` and `[blbd-<issue>] Imperative subject`. Keep pull requests below 500 counted lines. Do not merge your own pull request or push `release`.

package/CHANGELOG.md

@@ -0,0 +1,24 @@
+# Changelog
+
+## 0.1.0 - 2026-10-16
+
+Initial alpha release.
+
+- 36 seeded episode tasks across voice, UI, and background deadline
+  classes; each deadline is set at 1.5-2x the task's shortest correct path
+  (voice 447-641 ms, UI 875-1494 ms, background 2025-3714 ms).
+- Deterministic `qualityWithinBudget` grader: the answer must name exactly
+  one candidate value, be grounded in replayed tool output, and arrive
+  inside the deadline.
+- Virtual clock: scheduled per-tool latencies plus model turn time,
+  measured wall-clock by default or declared `thinkMs` per turn
+  (`--clock declared`, `LATENCY_BUDGET_CLOCK=declared`).
+- SDK, CLI (`--tasks`, `--clock`), MCP stdio server, and fetch-style HTTP
+  API from one `defineBenchmark` package.
+- Run reports carry a `latency` block: p50/p95 billed time,
+  `lateButCorrect`, clock counts, and the 0.5x-4x deadline curve. MCP
+  `run_benchmark` returns the same block; `POST /v1/score` and MCP
+  `score_answer` return a per-submission `latency` block and bill
+  submitted traces at the declared rate.
+- Hidden split generation (`--split hidden`) derives prompts and answers
+  from a transformed seed and prefixes task ids with `h-`.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Pooria Arab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,302 @@
-# Temporary Holding Version
+# benchmark-latency-budget
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[![npm](https://img.shields.io/npm/v/benchmark-latency-budget)](https://www.npmjs.com/package/benchmark-latency-budget) [![downloads](https://img.shields.io/npm/dm/benchmark-latency-budget)](https://www.npmjs.com/package/benchmark-latency-budget) [![license](https://img.shields.io/npm/l/benchmark-latency-budget)](./LICENSE) [![types](https://img.shields.io/badge/TypeScript-types-3178c6)](#sdk)
+
+[AI Benchmark Index](https://aibenchmarkindex.com/benchmarks/benchmark-latency-budget) · [npm](https://www.npmjs.com/package/benchmark-latency-budget) · [API](#http-api) · [MCP](#model-context-protocol)
+
+A right answer that arrives too late is a wrong answer in a voice call or a
+busy UI. benchmark-latency-budget checks whether an agent gives the correct
+answer before its deadline. Tasks come in voice, UI, and background classes,
+and each deadline is set at 1.5-2x the shortest correct path, so chatty
+agents run out of time. The agent sees its deadline in the system text and
+can audit tool costs with a `describe_tools` call.
+
+```bash
+npx benchmark-latency-budget run --model offline --provider mock
+```
+
+```text
+# Latency Budget
+
+- Model: `mock:1`
+- Score: **47.2%**
+- 95% CI: 30.6%–63.9%
+- Tasks: 36
+
+| Task | Score | Reason |
+| --- | ---: | --- |
+| voice-01 | 0.0% | No candidate value found in the answer; 0ms against a 641ms deadline. |
+| voice-02 | 100.0% | Correct answer in 121ms, within the 484ms deadline. |
+```
+
+The tasks are seeded synthetic directory, calendar, and doc lookups. Seed
+20261015 reproduces the public set; a hidden split derives from a
+transformed seed, so its prompts and answers never repeat the public one
+and its task ids carry an `h-` prefix.
+
+## Install
+
+benchmark-latency-budget requires Node.js 20 or newer.
+
+```bash
+npm install benchmark-latency-budget
+```
+
+## Run your first benchmark
+
+The mock provider needs no key or network access.
+
+```bash
+npx benchmark-latency-budget run --model offline --provider mock
+```
+
+About half the mock answers are right and on time; the rest are wrong, so
+expect a mean near 50%. A successful run prints 36 scored rows and a mean
+with a bootstrap 95% confidence interval.
+
+## Run on a real model
+
+Point the runner at OpenRouter with any chat model:
+
+```bash
+OPENROUTER_API_KEY=sk-or-... npx benchmark-latency-budget run --model openai/gpt-4o-mini --provider openrouter
+```
+
+Any OpenAI-compatible endpoint works too:
+
+```bash
+OPENAI_BASE_URL=https://my-proxy/v1 OPENAI_API_KEY=... npx benchmark-latency-budget run --model my-model --provider openai
+```
+
+Never put keys in repo files; the runner reads them from the environment.
+
+## How to read the score
+
+The headline is `qualityWithinBudget`: 1 only for a correct answer whose
+total time fits the deadline, else 0. Totals add model turn time to every
+scheduled tool charge. Turn time is measured wall-clock by default, or the
+declared `thinkMs` (80 ms) per turn when the clock is declared
+(`--clock declared` or `LATENCY_BUDGET_CLOCK=declared`) for repeatable
+runs. A correct answer past the deadline scores 0 on the headline and is
+counted as `lateButCorrect`, so you can tell slowness apart from
+wrongness. Every run report also carries a `latency` block with p50/p95
+total time, the `lateButCorrect` count, and a quality-vs-deadline curve
+at 0.5x, 1x, 2x, and 4x each deadline.
+
+Measured mode bills real wall-clock turn time, and the deadlines assume
+about 80 ms of model time per turn (the declared `thinkMs`). A hosted chat
+model that needs a second or more per turn will therefore score near 0 in
+measured mode on voice and UI tasks — that is what the benchmark is
+measuring. For cross-model comparisons where raw speed should not
+dominate, run `--clock declared` so every model is billed the same 80 ms
+per turn.
+
+## How it works
+
+```text
+seeded tasks -> agent tools (lookup, calendar_get, search_docs, describe_tools)
+  -> virtual clock (model turn time + scheduled ms per call)
+  -> deterministic grader (one candidate + grounded + on time)
+  -> CLI / SDK / MCP / HTTP API
+```
+
+The runner keeps task order and records score, reason, latency, retries,
+usage, and cost when available. Model turn time is billed from the
+environment's wall-clock record; run with a declared clock when results
+must repeat bit for bit across machines.
+
+## Why it exists
+
+General model leaderboards do not isolate agents behavior under time
+pressure. This package keeps the task data, grading rules, and provenance
+together for inspection.
+
+## Compare
+
+Checked on 2026-10-16.
+
+| Property | benchmark-latency-budget | General chat leaderboard | Ad hoc prompt test |
+| --- | --- | --- | --- |
+| Target | Correct answers inside voice, UI, and background deadlines. | Broad model quality | One local example |
+| Grading | Deterministic code | Varies | Manual |
+| Offline run | Mock provider | Varies | Varies |
+| Provenance | Package record | Varies | Usually absent |
+
+## Coverage
+
+| Surface | Status |
+| --- | --- |
+| episode tasks | Included (36: 12 voice, 12 UI, 12 background) |
+| SDK | Included |
+| CLI | Included |
+| MCP server | Included |
+| HTTP API | Included |
+
+The E2E gate checks every listed surface against the package source.
+
+## Deadlines
+
+Each task sets `deadlineMs` at 1.5-2x its shortest correct path (declared
+turn time plus scheduled call charges), so slower schedules get wider
+budgets and nothing is loose enough to hide wasted calls. The shipped data
+spans voice 447-641 ms, UI 875-1494 ms, background 2025-3714 ms.
+
+| Class | Minimal calls | Deadline range | What burns the budget |
+| --- | --- | --- | --- |
+| voice | 1 | 447-641 ms | A repeat lookup or a wrong first guess |
+| UI | 2 | 875-1494 ms | Redundant searches before the follow-up call |
+| background | 3 | 2025-3714 ms | Long chains of speculative calls |
+
+Per-call charges range from 100 ms (voice lookup) to 700 ms (background
+search). Retrying a failed call costs the full charge again, and the
+read-only `describe_tools` audit costs a flat 25 ms.
+
+## Common use cases
+
+1. **Model developers:** compare adapters with the same seeded tasks.
+2. **Agent developers:** inspect task-level failures before changing prompts.
+3. **Release teams:** run the mock gate before publishing a data update.
+4. **Researchers:** export per-task scores for a reproducible analysis.
+
+## Features
+
+- **Deterministic grading:** Each row includes a score and reason.
+- **Reproducible data:** A committed seeded script creates the public tasks.
+- **Virtual clock:** Tool time is scheduled and model turns are measured or declared, so runs are fair and can repeat exactly.
+- **Share cards:** The `card` command writes a 1200×630 SVG with the benchmark name, the model, and the score.
+- **Provenance:** Code and Markdown carry the same source record.
+
+## SDK
+
+```ts
+import { benchmark, run, score } from "benchmark-latency-budget";
+import { createLatencyBenchmark, latencyReport } from "benchmark-latency-budget";
+```
+
+`run(adapter, options)` creates a report with a `latency` summary block.
+`score(taskId, submission)` grades one trace. `totalTimeMs(task, submission)`
+returns the billed total and `sweepDeadlines(task, submission)` returns the
+0.5x–4x curve. `createLatencyBenchmark({ tasksPath, clock })` builds a
+benchmark over a different tasks file or clock mode, and
+`latencyReport(report)` adds the summary to any stored run report.
+
+## CLI
+
+Commands are `list`, `show`, `run`, `score`, `card`, `provenance`,
+`index-entry`, `serve`, and `mcp`. Pass `--json` for machine-readable output.
+Two flags work on every command: `--tasks <path>` swaps the packaged task set
+for another JSONL file (for example a generated hidden split), and
+`--clock measured|declared` picks measured wall-clock turns or the repeatable
+declared `thinkMs` per turn.
+
+## Model Context Protocol
+
+Start the MCP stdio server:
+
+```bash
+npx benchmark-latency-budget mcp
+```
+
+Tools are `list_tasks`, `get_task`, `score_answer`, `run_benchmark`, and
+`get_provenance`. `run_benchmark` returns the report with the `latency`
+summary and accepts `provider: "mock"` for offline runs; `score_answer`
+replies with a per-submission `latency` timing block.
+
+## HTTP API
+
+Import `handler` from `benchmark-latency-budget/api`, or run
+`npx benchmark-latency-budget serve`. Routes are `GET /health`,
+`GET /v1/tasks`, `GET /v1/tasks/:id`, `POST /v1/score`,
+`GET /v1/provenance`, and `GET /v1/index-entry`. `POST /v1/score` returns
+the score plus a `latency` timing block (deadline, billed total, clock
+mode, on-time flag). Server surfaces ignore self-reported measured turn
+times — a caller cannot prove them — so submitted traces are billed at the
+declared rate.
+
+## Data provenance
+
+`data/tasks.jsonl` contains the 36 public tasks. `data/PROVENANCE.md` names
+the generator, seed, licence, cutoff, and refresh steps. Run
+`npm run generate:tasks` to rebuild the public set and
+`node scripts/gen-tasks.mjs --seed <n> --split hidden` for a hidden split
+(written to gitignored `artifacts/`, never published).
+
+## Limits and known weaknesses
+
+No public leaderboard yet: only the mock provider has been run here, so the
+sample table above shows mock behavior, not model rankings. Declared mode
+cannot see where turns broke in a flat tool log, so each recorded call is
+billed as its own turn; measured mode fixes that but depends on the machine.
+Synthetic fixtures are small, so agents can look faster than on messy real
+tools. Public tasks can enter training data; keep a separately seeded hidden
+split for final comparisons. Remote providers can add cost and retain
+prompts under their own policies.
+
+## Prior art
+
+There is no single prior benchmark this copies. The episode simulator
+pattern follows `benchmark-core`'s seeded `OrderLookupEnvironment`, and the
+deadline framing follows the conversational-systems practice of treating a
+late reply as a failure. The difference here is the virtual clock: tool time
+is scheduled per call and model turn time is recorded by the environment or
+declared, so nothing depends on per-call wall-clock jitter.
+
+## Integration guides
+
+- [TypeScript SDK integration](#sdk)
+- [Model Context Protocol integration](#model-context-protocol)
+- [HTTP API integration](#http-api)
+
+## Status
+
+Version 0.1.0 is alpha, saturation unknown. Task schemas and grading may
+change before 1.0. Report defects through the GitHub issue tracker. Public
+case studies are not yet published.
+
+## FAQ
+
+### What does benchmark-latency-budget measure?
+
+Whether an agent answers correctly before a voice, UI, or background
+deadline. The committed grader defines the exact pass conditions.
+
+### Can benchmark-latency-budget run offline?
+
+It runs end to end with the mock provider and no key. Real model runs need
+a configured provider.
+
+### Which providers work?
+
+OpenRouter, an OpenAI-compatible base URL, and the offline mock adapter
+through benchmark-core.
+
+### Does benchmark-latency-budget provide a hosted service?
+
+It provides a server you run. It does not provide a managed benchmark
+service.
+
+### Where do the tasks come from?
+
+Seeded synthetic directory, calendar, and doc fixtures. The provenance file
+explains how to reproduce and refresh them.
+
+### How are answers graded?
+
+The answer must name exactly one of the task's candidate values
+(case-insensitive, word-bounded) and a replayed tool result must ground it,
+then the total time must fit the deadline. Model judging is off.
+
+### How is this different from a general leaderboard?
+
+It measures one named behavior with inspectable tasks and grading. General
+leaderboards combine broader capabilities.
+
+### Can I use a hidden split?
+
+Yes. Generate one with `--split hidden` into `artifacts/` and run it with
+`--tasks artifacts/tasks.hidden.jsonl`. Keep it out of the npm package and
+the public repository.
+
+## License
+
+[MIT](LICENSE) © Pooria Arab

package/data/PROVENANCE.md

@@ -0,0 +1,13 @@
+# Latency Budget provenance
+
+This file is generated from the benchmark definition.
+
+## Seeded latency-budget task generator
+
+- Kind: `synthetic`
+- URL: https://github.com/pooriaarab/benchmark-latency-budget/blob/main/scripts/gen-tasks.mjs
+- Licence: `MIT`
+- Retrieved: 2026-10-15T00:00:00.000Z
+- How to get it: Run `node scripts/gen-tasks.mjs --seed 20261015` for the public tasks, or `node scripts/gen-tasks.mjs --seed <n> --split hidden` for a hidden split (written to gitignored artifacts/ with h- prefixed task ids).
+- Contamination notes: Public tasks generated with seed 20261015, benchmark cutoff 2026-10-15. Public tasks may enter training data; use a separately seeded hidden split for final comparisons.
+- Refresh: Pick a new seed, run the generator, review the JSONL diff. Regenerate quarterly or when saturation is shown.

package/data/tasks.jsonl

@@ -0,0 +1,36 @@
+{"id":"voice-01","kind":"episode","prompt":"What is the phone extension for Lina Haddad?","expected":"8929","metadata":{"budgetClass":"voice","deadlineMs":641,"thinkMs":80,"latencyMs":{"lookup":220,"calendar_get":176,"search_docs":202},"fixtures":{"directory":{"Lina Haddad":{"extension":"8929","room":"Atrium-1"},"Sofia Marchetti":{"extension":"1196","room":"Atrium-1"},"David Kim":{"extension":"9549","room":"West-3"},"Omar Farouk":{"extension":"7198","room":"Atrium-1"}}},"minimalCalls":[{"name":"lookup","args":{"key":"Lina Haddad"}}],"candidates":["8929","1196","9549","7198"],"seed":20261395}}
+{"id":"voice-02","kind":"episode","prompt":"Which room is Demo Friday on 2026-11-09?","expected":"Garden-A","metadata":{"budgetClass":"voice","deadlineMs":484,"thinkMs":80,"latencyMs":{"lookup":106,"calendar_get":121,"search_docs":207},"fixtures":{"directory":{"Grace Mwangi":{"extension":"7778","room":"Focus-4"},"Sofia Marchetti":{"extension":"2475","room":"North-2"},"Amara Okafor":{"extension":"2688","room":"West-3"},"David Kim":{"extension":"6041","room":"Garden-A"}},"calendar":{"2026-11-09":[{"event":"Demo Friday","room":"Garden-A","organizer":"Grace Mwangi"},{"event":"Release train","room":"1D","organizer":"Sofia Marchetti"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-09"}}],"candidates":["Garden-A","1D"],"seed":20261296}}
+{"id":"voice-03","kind":"episode","prompt":"What internal doc covers parental leave?","expected":"Parental leave, step by step","metadata":{"budgetClass":"voice","deadlineMs":447,"thinkMs":80,"latencyMs":{"lookup":102,"calendar_get":151,"search_docs":109},"fixtures":{"directory":{"Amara Okafor":{"extension":"7208","room":"Focus-4"},"Lina Haddad":{"extension":"9106","room":"Focus-4"},"Ana Petrova":{"extension":"4400","room":"Atrium-1"},"Tomas Ribeiro":{"extension":"1443","room":"West-3"}},"docs":[{"title":"Parental leave, step by step","topic":"parental leave","author":"Amara Okafor"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Lina Haddad"},{"title":"Where each team sits","topic":"seating plan","author":"Lina Haddad"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"parental leave"}}],"candidates":["Parental leave, step by step","Ten shortcuts for the editor","Where each team sits"],"seed":20261284}}
+{"id":"voice-04","kind":"episode","prompt":"Give me Yuki Tanaka's phone extension.","expected":"5747","metadata":{"budgetClass":"voice","deadlineMs":476,"thinkMs":80,"latencyMs":{"lookup":156,"calendar_get":118,"search_docs":210},"fixtures":{"directory":{"Yuki Tanaka":{"extension":"5747","room":"1D"},"Ana Petrova":{"extension":"9989","room":"Garden-A"},"Grace Mwangi":{"extension":"3594","room":"Atrium-1"},"Ella Novak":{"extension":"8494","room":"3B"}}},"minimalCalls":[{"name":"lookup","args":{"key":"Yuki Tanaka"}}],"candidates":["5747","9989","3594","8494"],"seed":20261331}}
+{"id":"voice-05","kind":"episode","prompt":"Which room is Roadmap sync on 2026-11-10?","expected":"West-3","metadata":{"budgetClass":"voice","deadlineMs":517,"thinkMs":80,"latencyMs":{"lookup":122,"calendar_get":102,"search_docs":188},"fixtures":{"directory":{"David Kim":{"extension":"1247","room":"Atrium-1"},"Grace Mwangi":{"extension":"4988","room":"3B"},"Omar Farouk":{"extension":"6395","room":"Atrium-1"},"Marco Silva":{"extension":"3831","room":"Garden-A"}},"calendar":{"2026-11-10":[{"event":"Roadmap sync","room":"West-3","organizer":"David Kim"},{"event":"Release train","room":"3B","organizer":"Grace Mwangi"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-10"}}],"candidates":["West-3","3B"],"seed":20261277}}
+{"id":"voice-06","kind":"episode","prompt":"What internal doc covers parental leave?","expected":"Parental leave, step by step","metadata":{"budgetClass":"voice","deadlineMs":579,"thinkMs":80,"latencyMs":{"lookup":114,"calendar_get":139,"search_docs":214},"fixtures":{"directory":{"Yuki Tanaka":{"extension":"3094","room":"2C"},"Amara Okafor":{"extension":"9840","room":"Garden-A"},"Omar Farouk":{"extension":"3245","room":"North-2"},"David Kim":{"extension":"1512","room":"3B"}},"docs":[{"title":"Parental leave, step by step","topic":"parental leave","author":"Yuki Tanaka"},{"title":"Refunds within 14 days","topic":"refund window","author":"Amara Okafor"},{"title":"Where each team sits","topic":"seating plan","author":"Amara Okafor"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"parental leave"}}],"candidates":["Parental leave, step by step","Refunds within 14 days","Where each team sits"],"seed":20261389}}
+{"id":"voice-07","kind":"episode","prompt":"Give me Ravi Menon's phone extension.","expected":"4292","metadata":{"budgetClass":"voice","deadlineMs":619,"thinkMs":80,"latencyMs":{"lookup":197,"calendar_get":146,"search_docs":163},"fixtures":{"directory":{"Ravi Menon":{"extension":"4292","room":"North-2"},"Marco Silva":{"extension":"5312","room":"North-2"},"David Kim":{"extension":"1945","room":"Atrium-1"},"Grace Mwangi":{"extension":"2394","room":"West-3"}}},"minimalCalls":[{"name":"lookup","args":{"key":"Ravi Menon"}}],"candidates":["4292","5312","1945","2394"],"seed":20261372}}
+{"id":"voice-08","kind":"episode","prompt":"Where does Demo Friday take place on 2026-11-03?","expected":"Focus-4","metadata":{"budgetClass":"voice","deadlineMs":488,"thinkMs":80,"latencyMs":{"lookup":164,"calendar_get":123,"search_docs":172},"fixtures":{"directory":{"Priya Nair":{"extension":"7888","room":"West-3"},"Sofia Marchetti":{"extension":"2382","room":"Focus-4"},"Jonas Lindqvist":{"extension":"1333","room":"1D"},"Fatima Diallo":{"extension":"9373","room":"West-3"}},"calendar":{"2026-11-03":[{"event":"Demo Friday","room":"Focus-4","organizer":"Priya Nair"},{"event":"Sprint review","room":"West-3","organizer":"Sofia Marchetti"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-03"}}],"candidates":["Focus-4","West-3"],"seed":20261298}}
+{"id":"voice-09","kind":"episode","prompt":"What internal doc covers expense policy?","expected":"Expensing travel the simple way","metadata":{"budgetClass":"voice","deadlineMs":488,"thinkMs":80,"latencyMs":{"lookup":115,"calendar_get":187,"search_docs":131},"fixtures":{"directory":{"Amara Okafor":{"extension":"9110","room":"West-3"},"David Kim":{"extension":"2811","room":"3B"},"Yuki Tanaka":{"extension":"7127","room":"2C"},"Tomas Ribeiro":{"extension":"6577","room":"2C"}},"docs":[{"title":"Expensing travel the simple way","topic":"expense policy","author":"Amara Okafor"},{"title":"Parental leave, step by step","topic":"parental leave","author":"David Kim"},{"title":"Refunds within 14 days","topic":"refund window","author":"David Kim"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"expense policy"}}],"candidates":["Expensing travel the simple way","Parental leave, step by step","Refunds within 14 days"],"seed":20261306}}
+{"id":"voice-10","kind":"episode","prompt":"Give me Sofia Marchetti's phone extension.","expected":"5040","metadata":{"budgetClass":"voice","deadlineMs":536,"thinkMs":80,"latencyMs":{"lookup":188,"calendar_get":104,"search_docs":180},"fixtures":{"directory":{"Sofia Marchetti":{"extension":"5040","room":"West-3"},"Fatima Diallo":{"extension":"5537","room":"2C"},"Amara Okafor":{"extension":"5363","room":"1D"},"Marco Silva":{"extension":"3288","room":"Atrium-1"}}},"minimalCalls":[{"name":"lookup","args":{"key":"Sofia Marchetti"}}],"candidates":["5040","5537","5363","3288"],"seed":20261363}}
+{"id":"voice-11","kind":"episode","prompt":"Where does Onboarding take place on 2026-11-09?","expected":"West-3","metadata":{"budgetClass":"voice","deadlineMs":544,"thinkMs":80,"latencyMs":{"lookup":124,"calendar_get":198,"search_docs":206},"fixtures":{"directory":{"Marco Silva":{"extension":"2401","room":"Atrium-1"},"Yuki Tanaka":{"extension":"6430","room":"West-3"},"Sofia Marchetti":{"extension":"1424","room":"West-3"},"Grace Mwangi":{"extension":"1225","room":"Garden-A"}},"calendar":{"2026-11-09":[{"event":"Onboarding","room":"West-3","organizer":"Marco Silva"},{"event":"Demo Friday","room":"Focus-4","organizer":"Yuki Tanaka"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-09"}}],"candidates":["West-3","Focus-4"],"seed":20261373}}
+{"id":"voice-12","kind":"episode","prompt":"Which internal doc explains data retention?","expected":"Keeping customer data for 30 days","metadata":{"budgetClass":"voice","deadlineMs":625,"thinkMs":80,"latencyMs":{"lookup":105,"calendar_get":176,"search_docs":219},"fixtures":{"directory":{"Ravi Menon":{"extension":"3802","room":"1D"},"Fatima Diallo":{"extension":"2164","room":"3B"},"Sofia Marchetti":{"extension":"7996","room":"1D"},"Ella Novak":{"extension":"2209","room":"1D"}},"docs":[{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Ravi Menon"},{"title":"Parental leave, step by step","topic":"parental leave","author":"Fatima Diallo"},{"title":"Who to page when prod is down","topic":"incident response","author":"Fatima Diallo"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"data retention"}}],"candidates":["Keeping customer data for 30 days","Parental leave, step by step","Who to page when prod is down"],"seed":20261394}}
+{"id":"ui-01","kind":"episode","prompt":"Who organizes Onboarding on 2026-11-06, and what is their extension?","expected":"1266","metadata":{"budgetClass":"ui","deadlineMs":1316,"thinkMs":80,"latencyMs":{"lookup":211,"calendar_get":249,"search_docs":285},"fixtures":{"directory":{"David Kim":{"extension":"1266","room":"West-3"},"Marco Silva":{"extension":"3310","room":"North-2"},"Priya Nair":{"extension":"9542","room":"Focus-4"},"Tomas Ribeiro":{"extension":"4706","room":"Garden-A"},"Ella Novak":{"extension":"9239","room":"Garden-A"}},"calendar":{"2026-11-06":[{"event":"Onboarding","room":"2C","organizer":"David Kim"},{"event":"Support handoff","room":"3B","organizer":"Marco Silva"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-06"}},{"name":"lookup","args":{"key":"David Kim"}}],"candidates":["1266","3310","9542","4706","9239"],"seed":20261715}}
+{"id":"ui-02","kind":"episode","prompt":"What is the phone extension of the author of the doc about vpn setup?","expected":"2662","metadata":{"budgetClass":"ui","deadlineMs":1342,"thinkMs":80,"latencyMs":{"lookup":243,"calendar_get":261,"search_docs":255},"fixtures":{"directory":{"Tomas Ribeiro":{"extension":"2662","room":"West-3"},"Lina Haddad":{"extension":"2518","room":"3B"},"Grace Mwangi":{"extension":"4649","room":"1D"},"Ravi Menon":{"extension":"6593","room":"3B"},"Marco Silva":{"extension":"9973","room":"Atrium-1"}},"docs":[{"title":"Connect to the office VPN","topic":"vpn setup","author":"Tomas Ribeiro"},{"title":"Who to page when prod is down","topic":"incident response","author":"Lina Haddad"},{"title":"Expensing travel the simple way","topic":"expense policy","author":"Grace Mwangi"},{"title":"Parental leave, step by step","topic":"parental leave","author":"Ravi Menon"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"vpn setup"}},{"name":"lookup","args":{"key":"Tomas Ribeiro"}}],"candidates":["2662","2518","4649","6593","9973"],"seed":20261753}}
+{"id":"ui-03","kind":"episode","prompt":"What is the phone extension of the organizer of Sprint review on 2026-11-11?","expected":"5181","metadata":{"budgetClass":"ui","deadlineMs":1494,"thinkMs":80,"latencyMs":{"lookup":246,"calendar_get":308,"search_docs":194},"fixtures":{"directory":{"Amara Okafor":{"extension":"5181","room":"Garden-A"},"Jonas Lindqvist":{"extension":"2517","room":"2C"},"Ella Novak":{"extension":"1853","room":"North-2"},"Kenji Sato":{"extension":"1622","room":"3B"},"Omar Farouk":{"extension":"5971","room":"1D"}},"calendar":{"2026-11-11":[{"event":"Sprint review","room":"3B","organizer":"Amara Okafor"},{"event":"Support handoff","room":"Garden-A","organizer":"Jonas Lindqvist"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-11"}},{"name":"lookup","args":{"key":"Amara Okafor"}}],"candidates":["5181","2517","1853","1622","5971"],"seed":20261809}}
+{"id":"ui-04","kind":"episode","prompt":"What extension does the author of the doc on editor shortcuts have?","expected":"8030","metadata":{"budgetClass":"ui","deadlineMs":1341,"thinkMs":80,"latencyMs":{"lookup":245,"calendar_get":257,"search_docs":314},"fixtures":{"directory":{"Omar Farouk":{"extension":"8030","room":"Atrium-1"},"Ana Petrova":{"extension":"9942","room":"3B"},"Yuki Tanaka":{"extension":"2814","room":"Garden-A"},"Ravi Menon":{"extension":"3104","room":"2C"},"Priya Nair":{"extension":"4677","room":"West-3"}},"docs":[{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Omar Farouk"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Ana Petrova"},{"title":"Parental leave, step by step","topic":"parental leave","author":"Yuki Tanaka"},{"title":"Who to page when prod is down","topic":"incident response","author":"Ravi Menon"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"editor shortcuts"}},{"name":"lookup","args":{"key":"Omar Farouk"}}],"candidates":["8030","9942","2814","3104","4677"],"seed":20261814}}
+{"id":"ui-05","kind":"episode","prompt":"Extension of the organizer for Design crit on 2026-11-05?","expected":"7295","metadata":{"budgetClass":"ui","deadlineMs":938,"thinkMs":80,"latencyMs":{"lookup":140,"calendar_get":159,"search_docs":325},"fixtures":{"directory":{"Omar Farouk":{"extension":"7295","room":"North-2"},"Lina Haddad":{"extension":"9044","room":"3B"},"Tomas Ribeiro":{"extension":"8311","room":"Focus-4"},"Ana Petrova":{"extension":"5214","room":"North-2"},"Kenji Sato":{"extension":"5035","room":"1D"}},"calendar":{"2026-11-05":[{"event":"Design crit","room":"North-2","organizer":"Omar Farouk"},{"event":"Support handoff","room":"Focus-4","organizer":"Lina Haddad"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-05"}},{"name":"lookup","args":{"key":"Omar Farouk"}}],"candidates":["7295","9044","8311","5214","5035"],"seed":20261554}}
+{"id":"ui-06","kind":"episode","prompt":"What extension does the author of the doc on vpn setup have?","expected":"9013","metadata":{"budgetClass":"ui","deadlineMs":1475,"thinkMs":80,"latencyMs":{"lookup":269,"calendar_get":314,"search_docs":341},"fixtures":{"directory":{"Omar Farouk":{"extension":"9013","room":"Focus-4"},"Ana Petrova":{"extension":"5344","room":"West-3"},"Lina Haddad":{"extension":"3785","room":"Atrium-1"},"Yuki Tanaka":{"extension":"3368","room":"3B"},"Amara Okafor":{"extension":"4626","room":"Garden-A"}},"docs":[{"title":"Connect to the office VPN","topic":"vpn setup","author":"Omar Farouk"},{"title":"Who to page when prod is down","topic":"incident response","author":"Ana Petrova"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Lina Haddad"},{"title":"Where each team sits","topic":"seating plan","author":"Yuki Tanaka"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"vpn setup"}},{"name":"lookup","args":{"key":"Omar Farouk"}}],"candidates":["9013","5344","3785","3368","4626"],"seed":20261865}}
+{"id":"ui-07","kind":"episode","prompt":"What is the phone extension of the organizer of Roadmap sync on 2026-11-02?","expected":"1660","metadata":{"budgetClass":"ui","deadlineMs":1396,"thinkMs":80,"latencyMs":{"lookup":239,"calendar_get":275,"search_docs":240},"fixtures":{"directory":{"Marco Silva":{"extension":"1660","room":"1D"},"Priya Nair":{"extension":"8499","room":"1D"},"Kenji Sato":{"extension":"2413","room":"3B"},"Sofia Marchetti":{"extension":"7259","room":"2C"},"Ana Petrova":{"extension":"1763","room":"Focus-4"}},"calendar":{"2026-11-02":[{"event":"Roadmap sync","room":"Atrium-1","organizer":"Marco Silva"},{"event":"Retro","room":"3B","organizer":"Priya Nair"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-02"}},{"name":"lookup","args":{"key":"Marco Silva"}}],"candidates":["1660","8499","2413","7259","1763"],"seed":20261769}}
+{"id":"ui-08","kind":"episode","prompt":"Who wrote the doc about parental leave, and what is their extension?","expected":"9594","metadata":{"budgetClass":"ui","deadlineMs":1439,"thinkMs":80,"latencyMs":{"lookup":292,"calendar_get":246,"search_docs":359},"fixtures":{"directory":{"Priya Nair":{"extension":"9594","room":"Atrium-1"},"Grace Mwangi":{"extension":"7922","room":"Atrium-1"},"Jonas Lindqvist":{"extension":"4334","room":"Focus-4"},"Lina Haddad":{"extension":"7324","room":"1D"},"Ella Novak":{"extension":"7817","room":"2C"}},"docs":[{"title":"Parental leave, step by step","topic":"parental leave","author":"Priya Nair"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Grace Mwangi"},{"title":"Refunds within 14 days","topic":"refund window","author":"Jonas Lindqvist"},{"title":"Connect to the office VPN","topic":"vpn setup","author":"Lina Haddad"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"parental leave"}},{"name":"lookup","args":{"key":"Priya Nair"}}],"candidates":["9594","7922","4334","7324","7817"],"seed":20261906}}
+{"id":"ui-09","kind":"episode","prompt":"Who organizes Sprint review on 2026-11-12, and what is their extension?","expected":"6017","metadata":{"budgetClass":"ui","deadlineMs":1244,"thinkMs":80,"latencyMs":{"lookup":286,"calendar_get":187,"search_docs":319},"fixtures":{"directory":{"Fatima Diallo":{"extension":"6017","room":"3B"},"David Kim":{"extension":"5238","room":"North-2"},"Ella Novak":{"extension":"9856","room":"3B"},"Tomas Ribeiro":{"extension":"7970","room":"2C"},"Ana Petrova":{"extension":"3179","room":"2C"}},"calendar":{"2026-11-12":[{"event":"Sprint review","room":"Garden-A","organizer":"Fatima Diallo"},{"event":"Sprint review","room":"Garden-A","organizer":"David Kim"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-12"}},{"name":"lookup","args":{"key":"Fatima Diallo"}}],"candidates":["6017","5238","9856","7970","3179"],"seed":20261728}}
+{"id":"ui-10","kind":"episode","prompt":"What extension does the author of the doc on seating plan have?","expected":"4532","metadata":{"budgetClass":"ui","deadlineMs":875,"thinkMs":80,"latencyMs":{"lookup":159,"calendar_get":215,"search_docs":181},"fixtures":{"directory":{"Marco Silva":{"extension":"4532","room":"West-3"},"Sofia Marchetti":{"extension":"6640","room":"3B"},"Priya Nair":{"extension":"1403","room":"3B"},"Ravi Menon":{"extension":"6378","room":"3B"},"Ella Novak":{"extension":"7344","room":"1D"}},"docs":[{"title":"Where each team sits","topic":"seating plan","author":"Marco Silva"},{"title":"Expensing travel the simple way","topic":"expense policy","author":"Sofia Marchetti"},{"title":"Parental leave, step by step","topic":"parental leave","author":"Priya Nair"},{"title":"Who to page when prod is down","topic":"incident response","author":"Ravi Menon"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"seating plan"}},{"name":"lookup","args":{"key":"Marco Silva"}}],"candidates":["4532","6640","1403","6378","7344"],"seed":20261595}}
+{"id":"ui-11","kind":"episode","prompt":"Who organizes Onboarding on 2026-11-04, and what is their extension?","expected":"7666","metadata":{"budgetClass":"ui","deadlineMs":1129,"thinkMs":80,"latencyMs":{"lookup":182,"calendar_get":167,"search_docs":265},"fixtures":{"directory":{"Amara Okafor":{"extension":"7666","room":"Focus-4"},"David Kim":{"extension":"5885","room":"Atrium-1"},"Sofia Marchetti":{"extension":"9879","room":"2C"},"Marco Silva":{"extension":"4579","room":"Atrium-1"},"Ana Petrova":{"extension":"8225","room":"Atrium-1"}},"calendar":{"2026-11-04":[{"event":"Onboarding","room":"Atrium-1","organizer":"Amara Okafor"},{"event":"Onboarding","room":"2C","organizer":"David Kim"}]}},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-04"}},{"name":"lookup","args":{"key":"Amara Okafor"}}],"candidates":["7666","5885","9879","4579","8225"],"seed":20261604}}
+{"id":"ui-12","kind":"episode","prompt":"What extension does the author of the doc on data retention have?","expected":"4055","metadata":{"budgetClass":"ui","deadlineMs":1036,"thinkMs":80,"latencyMs":{"lookup":214,"calendar_get":161,"search_docs":214},"fixtures":{"directory":{"Priya Nair":{"extension":"4055","room":"West-3"},"Yuki Tanaka":{"extension":"3883","room":"1D"},"Kenji Sato":{"extension":"1212","room":"Garden-A"},"Sofia Marchetti":{"extension":"5327","room":"West-3"},"Ravi Menon":{"extension":"5357","room":"1D"}},"docs":[{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Priya Nair"},{"title":"Where each team sits","topic":"seating plan","author":"Yuki Tanaka"},{"title":"Expensing travel the simple way","topic":"expense policy","author":"Kenji Sato"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Sofia Marchetti"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"data retention"}},{"name":"lookup","args":{"key":"Priya Nair"}}],"candidates":["4055","3883","1212","5327","5357"],"seed":20261683}}
+{"id":"bg-01","kind":"episode","prompt":"Find the internal doc covering Roadmap sync on 2026-11-04; what extension does its author have?","expected":"8751","metadata":{"budgetClass":"background","deadlineMs":3446,"thinkMs":80,"latencyMs":{"lookup":660,"calendar_get":493,"search_docs":615},"fixtures":{"directory":{"Ella Novak":{"extension":"8751","room":"Atrium-1"},"Ravi Menon":{"extension":"2293","room":"1D"},"Priya Nair":{"extension":"5114","room":"North-2"},"Ana Petrova":{"extension":"2495","room":"2C"},"Grace Mwangi":{"extension":"9915","room":"3B"},"Yuki Tanaka":{"extension":"4887","room":"Garden-A"}},"calendar":{"2026-11-04":[{"event":"Roadmap sync","room":"Atrium-1","organizer":"Ravi Menon","topic":"parental leave"}]},"docs":[{"title":"Parental leave, step by step","topic":"parental leave","author":"Ella Novak"},{"title":"Refunds within 14 days","topic":"refund window","author":"Ravi Menon"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Priya Nair"},{"title":"Expensing travel the simple way","topic":"expense policy","author":"Ana Petrova"},{"title":"Who to page when prod is down","topic":"incident response","author":"Grace Mwangi"}]},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-04"}},{"name":"search_docs","args":{"query":"parental leave"}},{"name":"lookup","args":{"key":"Ella Novak"}}],"candidates":["8751","2293","5114","2495","9915","4887"],"seed":20263103}}
+{"id":"bg-02","kind":"episode","prompt":"A doc about vpn setup mentions an event. What is the phone extension of that event's organizer?","expected":"9366","metadata":{"budgetClass":"background","deadlineMs":2644,"thinkMs":80,"latencyMs":{"lookup":466,"calendar_get":275,"search_docs":599},"fixtures":{"directory":{"David Kim":{"extension":"9366","room":"1D"},"Kenji Sato":{"extension":"1742","room":"West-3"},"Ana Petrova":{"extension":"6007","room":"3B"},"Yuki Tanaka":{"extension":"7712","room":"Garden-A"},"Amara Okafor":{"extension":"7969","room":"Focus-4"},"Marco Silva":{"extension":"4085","room":"Atrium-1"}},"calendar":{"2026-11-12":[{"event":"Release train","room":"2C","organizer":"David Kim"}]},"docs":[{"title":"Connect to the office VPN","topic":"vpn setup","author":"Kenji Sato","event":"Release train","date":"2026-11-12"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Ana Petrova"},{"title":"Refunds within 14 days","topic":"refund window","author":"Yuki Tanaka"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Amara Okafor"},{"title":"Who to page when prod is down","topic":"incident response","author":"Marco Silva"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"vpn setup"}},{"name":"calendar_get","args":{"date":"2026-11-12"}},{"name":"lookup","args":{"key":"David Kim"}}],"candidates":["9366","1742","6007","7712","7969","4085"],"seed":20262675}}
+{"id":"bg-03","kind":"episode","prompt":"The internal doc on Sprint review for 2026-11-02 has an author. What is their extension?","expected":"7214","metadata":{"budgetClass":"background","deadlineMs":3194,"thinkMs":80,"latencyMs":{"lookup":604,"calendar_get":352,"search_docs":660},"fixtures":{"directory":{"Priya Nair":{"extension":"7214","room":"2C"},"Ella Novak":{"extension":"2303","room":"West-3"},"Fatima Diallo":{"extension":"9946","room":"West-3"},"Kenji Sato":{"extension":"6705","room":"3B"},"David Kim":{"extension":"3324","room":"West-3"},"Tomas Ribeiro":{"extension":"3774","room":"Garden-A"}},"calendar":{"2026-11-02":[{"event":"Sprint review","room":"3B","organizer":"Ella Novak","topic":"incident response"}]},"docs":[{"title":"Who to page when prod is down","topic":"incident response","author":"Priya Nair"},{"title":"Where each team sits","topic":"seating plan","author":"Ella Novak"},{"title":"Refunds within 14 days","topic":"refund window","author":"Fatima Diallo"},{"title":"Expensing travel the simple way","topic":"expense policy","author":"Kenji Sato"},{"title":"Parental leave, step by step","topic":"parental leave","author":"David Kim"}]},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-02"}},{"name":"search_docs","args":{"query":"incident response"}},{"name":"lookup","args":{"key":"Priya Nair"}}],"candidates":["7214","2303","9946","6705","3324","3774"],"seed":20262951}}
+{"id":"bg-04","kind":"episode","prompt":"A doc about vpn setup mentions an event. What is the phone extension of that event's organizer?","expected":"3734","metadata":{"budgetClass":"background","deadlineMs":2991,"thinkMs":80,"latencyMs":{"lookup":493,"calendar_get":608,"search_docs":295},"fixtures":{"directory":{"Jonas Lindqvist":{"extension":"3734","room":"2C"},"Ella Novak":{"extension":"8869","room":"3B"},"Ravi Menon":{"extension":"4808","room":"2C"},"Yuki Tanaka":{"extension":"4756","room":"1D"},"Omar Farouk":{"extension":"5119","room":"2C"},"Tomas Ribeiro":{"extension":"4530","room":"North-2"}},"calendar":{"2026-11-04":[{"event":"Roadmap sync","room":"Garden-A","organizer":"Jonas Lindqvist"}]},"docs":[{"title":"Connect to the office VPN","topic":"vpn setup","author":"Ella Novak","event":"Roadmap sync","date":"2026-11-04"},{"title":"Expensing travel the simple way","topic":"expense policy","author":"Ravi Menon"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Yuki Tanaka"},{"title":"Parental leave, step by step","topic":"parental leave","author":"Omar Farouk"},{"title":"Where each team sits","topic":"seating plan","author":"Tomas Ribeiro"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"vpn setup"}},{"name":"calendar_get","args":{"date":"2026-11-04"}},{"name":"lookup","args":{"key":"Jonas Lindqvist"}}],"candidates":["3734","8869","4808","4756","5119","4530"],"seed":20262731}}
+{"id":"bg-05","kind":"episode","prompt":"The internal doc on Design crit for 2026-11-06 has an author. What is their extension?","expected":"4320","metadata":{"budgetClass":"background","deadlineMs":3589,"thinkMs":80,"latencyMs":{"lookup":638,"calendar_get":513,"search_docs":658},"fixtures":{"directory":{"Kenji Sato":{"extension":"4320","room":"2C"},"Grace Mwangi":{"extension":"4496","room":"Focus-4"},"David Kim":{"extension":"7927","room":"Garden-A"},"Marco Silva":{"extension":"7650","room":"1D"},"Tomas Ribeiro":{"extension":"5400","room":"Garden-A"},"Jonas Lindqvist":{"extension":"3954","room":"Atrium-1"}},"calendar":{"2026-11-06":[{"event":"Design crit","room":"Atrium-1","organizer":"Grace Mwangi","topic":"data retention"}]},"docs":[{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Kenji Sato"},{"title":"Where each team sits","topic":"seating plan","author":"Grace Mwangi"},{"title":"Connect to the office VPN","topic":"vpn setup","author":"David Kim"},{"title":"Refunds within 14 days","topic":"refund window","author":"Marco Silva"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Tomas Ribeiro"}]},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-06"}},{"name":"search_docs","args":{"query":"data retention"}},{"name":"lookup","args":{"key":"Kenji Sato"}}],"candidates":["4320","4496","7927","7650","5400","3954"],"seed":20263144}}
+{"id":"bg-06","kind":"episode","prompt":"The doc on vpn setup names an event. What extension does its organizer have?","expected":"3488","metadata":{"budgetClass":"background","deadlineMs":3283,"thinkMs":80,"latencyMs":{"lookup":600,"calendar_get":508,"search_docs":602},"fixtures":{"directory":{"Marco Silva":{"extension":"3488","room":"Atrium-1"},"David Kim":{"extension":"1277","room":"West-3"},"Ella Novak":{"extension":"7560","room":"North-2"},"Amara Okafor":{"extension":"8306","room":"1D"},"Grace Mwangi":{"extension":"2593","room":"Focus-4"},"Omar Farouk":{"extension":"4143","room":"Focus-4"}},"calendar":{"2026-11-13":[{"event":"Support handoff","room":"Atrium-1","organizer":"Marco Silva"}]},"docs":[{"title":"Connect to the office VPN","topic":"vpn setup","author":"David Kim","event":"Support handoff","date":"2026-11-13"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Ella Novak"},{"title":"Refunds within 14 days","topic":"refund window","author":"Amara Okafor"},{"title":"Parental leave, step by step","topic":"parental leave","author":"Grace Mwangi"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Omar Farouk"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"vpn setup"}},{"name":"calendar_get","args":{"date":"2026-11-13"}},{"name":"lookup","args":{"key":"Marco Silva"}}],"candidates":["3488","1277","7560","8306","2593","4143"],"seed":20263045}}
+{"id":"bg-07","kind":"episode","prompt":"Find the internal doc covering Roadmap sync on 2026-11-09; what extension does its author have?","expected":"2253","metadata":{"budgetClass":"background","deadlineMs":2553,"thinkMs":80,"latencyMs":{"lookup":366,"calendar_get":641,"search_docs":293},"fixtures":{"directory":{"Grace Mwangi":{"extension":"2253","room":"North-2"},"Sofia Marchetti":{"extension":"5927","room":"North-2"},"Priya Nair":{"extension":"2524","room":"North-2"},"David Kim":{"extension":"2314","room":"2C"},"Lina Haddad":{"extension":"3363","room":"2C"},"Jonas Lindqvist":{"extension":"6368","room":"Atrium-1"}},"calendar":{"2026-11-09":[{"event":"Roadmap sync","room":"3B","organizer":"Sofia Marchetti","topic":"incident response"}]},"docs":[{"title":"Who to page when prod is down","topic":"incident response","author":"Grace Mwangi"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Sofia Marchetti"},{"title":"Where each team sits","topic":"seating plan","author":"Priya Nair"},{"title":"Connect to the office VPN","topic":"vpn setup","author":"David Kim"},{"title":"Refunds within 14 days","topic":"refund window","author":"Lina Haddad"}]},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-09"}},{"name":"search_docs","args":{"query":"incident response"}},{"name":"lookup","args":{"key":"Grace Mwangi"}}],"candidates":["2253","5927","2524","2314","3363","6368"],"seed":20262635}}
+{"id":"bg-08","kind":"episode","prompt":"Read the doc about parental leave and give me the extension of the organizer of the event it mentions.","expected":"5055","metadata":{"budgetClass":"background","deadlineMs":2606,"thinkMs":80,"latencyMs":{"lookup":415,"calendar_get":531,"search_docs":263},"fixtures":{"directory":{"Grace Mwangi":{"extension":"5055","room":"North-2"},"Jonas Lindqvist":{"extension":"7862","room":"1D"},"Kenji Sato":{"extension":"1601","room":"Focus-4"},"Lina Haddad":{"extension":"9378","room":"2C"},"Amara Okafor":{"extension":"3632","room":"Garden-A"},"Ana Petrova":{"extension":"1714","room":"1D"}},"calendar":{"2026-11-09":[{"event":"Sprint review","room":"West-3","organizer":"Grace Mwangi"}]},"docs":[{"title":"Parental leave, step by step","topic":"parental leave","author":"Jonas Lindqvist","event":"Sprint review","date":"2026-11-09"},{"title":"Refunds within 14 days","topic":"refund window","author":"Kenji Sato"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Lina Haddad"},{"title":"Who to page when prod is down","topic":"incident response","author":"Amara Okafor"},{"title":"Connect to the office VPN","topic":"vpn setup","author":"Ana Petrova"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"parental leave"}},{"name":"calendar_get","args":{"date":"2026-11-09"}},{"name":"lookup","args":{"key":"Grace Mwangi"}}],"candidates":["5055","7862","1601","9378","3632","1714"],"seed":20262544}}
+{"id":"bg-09","kind":"episode","prompt":"Find the internal doc covering Demo Friday on 2026-11-02; what extension does its author have?","expected":"1111","metadata":{"budgetClass":"background","deadlineMs":3714,"thinkMs":80,"latencyMs":{"lookup":459,"calendar_get":694,"search_docs":676},"fixtures":{"directory":{"Grace Mwangi":{"extension":"1111","room":"Garden-A"},"Omar Farouk":{"extension":"5932","room":"Atrium-1"},"Sofia Marchetti":{"extension":"8536","room":"Focus-4"},"Yuki Tanaka":{"extension":"6542","room":"Atrium-1"},"Ella Novak":{"extension":"3877","room":"West-3"},"Fatima Diallo":{"extension":"2253","room":"Garden-A"}},"calendar":{"2026-11-02":[{"event":"Demo Friday","room":"3B","organizer":"Omar Farouk","topic":"refund window"}]},"docs":[{"title":"Refunds within 14 days","topic":"refund window","author":"Grace Mwangi"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Omar Farouk"},{"title":"Connect to the office VPN","topic":"vpn setup","author":"Sofia Marchetti"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Yuki Tanaka"},{"title":"Where each team sits","topic":"seating plan","author":"Ella Novak"}]},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-02"}},{"name":"search_docs","args":{"query":"refund window"}},{"name":"lookup","args":{"key":"Grace Mwangi"}}],"candidates":["1111","5932","8536","6542","3877","2253"],"seed":20263164}}
+{"id":"bg-10","kind":"episode","prompt":"Read the doc about incident response and give me the extension of the organizer of the event it mentions.","expected":"3903","metadata":{"budgetClass":"background","deadlineMs":2025,"thinkMs":80,"latencyMs":{"lookup":270,"calendar_get":458,"search_docs":279},"fixtures":{"directory":{"Jonas Lindqvist":{"extension":"3903","room":"Atrium-1"},"Omar Farouk":{"extension":"2713","room":"2C"},"Yuki Tanaka":{"extension":"5153","room":"1D"},"Priya Nair":{"extension":"7887","room":"2C"},"Kenji Sato":{"extension":"7610","room":"3B"},"Tomas Ribeiro":{"extension":"3129","room":"1D"}},"calendar":{"2026-11-03":[{"event":"Sprint review","room":"2C","organizer":"Jonas Lindqvist"}]},"docs":[{"title":"Who to page when prod is down","topic":"incident response","author":"Omar Farouk","event":"Sprint review","date":"2026-11-03"},{"title":"Parental leave, step by step","topic":"parental leave","author":"Yuki Tanaka"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Priya Nair"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Kenji Sato"},{"title":"Connect to the office VPN","topic":"vpn setup","author":"Tomas Ribeiro"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"incident response"}},{"name":"calendar_get","args":{"date":"2026-11-03"}},{"name":"lookup","args":{"key":"Jonas Lindqvist"}}],"candidates":["3903","2713","5153","7887","7610","3129"],"seed":20262342}}
+{"id":"bg-11","kind":"episode","prompt":"What is the phone extension of the author of the internal doc covering the topic of Sprint review on 2026-11-03?","expected":"9967","metadata":{"budgetClass":"background","deadlineMs":3070,"thinkMs":80,"latencyMs":{"lookup":518,"calendar_get":335,"search_docs":637},"fixtures":{"directory":{"Marco Silva":{"extension":"9967","room":"2C"},"Ravi Menon":{"extension":"5377","room":"2C"},"Fatima Diallo":{"extension":"3773","room":"Garden-A"},"Ella Novak":{"extension":"5883","room":"West-3"},"Priya Nair":{"extension":"6786","room":"1D"},"Grace Mwangi":{"extension":"3887","room":"Atrium-1"}},"calendar":{"2026-11-03":[{"event":"Sprint review","room":"1D","organizer":"Ravi Menon","topic":"vpn setup"}]},"docs":[{"title":"Connect to the office VPN","topic":"vpn setup","author":"Marco Silva"},{"title":"Refunds within 14 days","topic":"refund window","author":"Ravi Menon"},{"title":"Ten shortcuts for the editor","topic":"editor shortcuts","author":"Fatima Diallo"},{"title":"Where each team sits","topic":"seating plan","author":"Ella Novak"},{"title":"Expensing travel the simple way","topic":"expense policy","author":"Priya Nair"}]},"minimalCalls":[{"name":"calendar_get","args":{"date":"2026-11-03"}},{"name":"search_docs","args":{"query":"vpn setup"}},{"name":"lookup","args":{"key":"Marco Silva"}}],"candidates":["9967","5377","3773","5883","6786","3887"],"seed":20262825}}
+{"id":"bg-12","kind":"episode","prompt":"A doc about parental leave mentions an event. What is the phone extension of that event's organizer?","expected":"8843","metadata":{"budgetClass":"background","deadlineMs":3476,"thinkMs":80,"latencyMs":{"lookup":546,"calendar_get":588,"search_docs":635},"fixtures":{"directory":{"David Kim":{"extension":"8843","room":"West-3"},"Ella Novak":{"extension":"1641","room":"Atrium-1"},"Omar Farouk":{"extension":"7356","room":"Atrium-1"},"Lina Haddad":{"extension":"6821","room":"Garden-A"},"Ana Petrova":{"extension":"8391","room":"3B"},"Ravi Menon":{"extension":"9405","room":"Focus-4"}},"calendar":{"2026-11-11":[{"event":"Design crit","room":"2C","organizer":"David Kim"}]},"docs":[{"title":"Parental leave, step by step","topic":"parental leave","author":"Ella Novak","event":"Design crit","date":"2026-11-11"},{"title":"Keeping customer data for 30 days","topic":"data retention","author":"Omar Farouk"},{"title":"Where each team sits","topic":"seating plan","author":"Lina Haddad"},{"title":"Refunds within 14 days","topic":"refund window","author":"Ana Petrova"},{"title":"Expensing travel the simple way","topic":"expense policy","author":"Ravi Menon"}]},"minimalCalls":[{"name":"search_docs","args":{"query":"parental leave"}},{"name":"calendar_get","args":{"date":"2026-11-11"}},{"name":"lookup","args":{"key":"David Kim"}}],"candidates":["8843","1641","7356","6821","8391","9405"],"seed":20263104}}

package/dist/api.d.ts

@@ -0,0 +1,13 @@
+import { Benchmark, Task } from 'benchmark-core';
+
+/**
+ * Fetch-style handler over the core routes. For `POST /v1/score` the
+ * submission's self-reported measured clock is stripped before grading (a
+ * caller cannot prove its wall-clock turn times) and the response gains a
+ * `latency` timing block: deadlineMs, totalTimeMs, modelMs, toolMs,
+ * modelTurns, clock, onTime, lateButCorrect.
+ */
+declare function createApi(bench?: Benchmark<Task>): (request: Request) => Promise<Response>;
+declare const handler: (request: Request) => Promise<Response>;
+
+export { createApi, handler };

package/dist/api.js

@@ -0,0 +1,10 @@
+import {
+  createApi,
+  handler
+} from "./chunk-65SHHDK7.js";
+import "./chunk-MRKAHJN3.js";
+export {
+  createApi,
+  handler
+};
+//# sourceMappingURL=api.js.map

package/dist/api.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":[],"sourcesContent":[],"mappings":"","names":[]}