@layr-labs/benchmaxx-arena-mcp 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +155 -0
- package/dist/apiTypes.d.ts +379 -0
- package/dist/apiTypes.js +24 -0
- package/dist/apiTypes.js.map +1 -0
- package/dist/browser.d.ts +33 -0
- package/dist/browser.js +65 -0
- package/dist/browser.js.map +1 -0
- package/dist/client.d.ts +210 -0
- package/dist/client.js +407 -0
- package/dist/client.js.map +1 -0
- package/dist/config.d.ts +25 -0
- package/dist/config.js +47 -0
- package/dist/config.js.map +1 -0
- package/dist/credentialStore.d.ts +103 -0
- package/dist/credentialStore.js +168 -0
- package/dist/credentialStore.js.map +1 -0
- package/dist/device.d.ts +87 -0
- package/dist/device.js +92 -0
- package/dist/device.js.map +1 -0
- package/dist/errors.d.ts +50 -0
- package/dist/errors.js +167 -0
- package/dist/errors.js.map +1 -0
- package/dist/index.d.ts +11 -0
- package/dist/index.js +49 -0
- package/dist/index.js.map +1 -0
- package/dist/loopback.d.ts +29 -0
- package/dist/loopback.js +120 -0
- package/dist/loopback.js.map +1 -0
- package/dist/pkce.d.ts +24 -0
- package/dist/pkce.js +27 -0
- package/dist/pkce.js.map +1 -0
- package/dist/tools.d.ts +28 -0
- package/dist/tools.js +350 -0
- package/dist/tools.js.map +1 -0
- package/package.json +44 -0
package/README.md
ADDED
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
# @layr-labs/benchmaxx-arena-mcp
|
|
2
|
+
|
|
3
|
+
An MCP server that lets an agent submit to benchmaxx-arena. It wraps the public
|
|
4
|
+
benchmaxx-arena HTTP API as discoverable MCP tools, runs the browser login,
|
|
5
|
+
stores the API key, and returns structured results with actionable errors.
|
|
6
|
+
|
|
7
|
+
The server speaks the MCP protocol over stdio. It needs no configuration to run
|
|
8
|
+
against production.
|
|
9
|
+
|
|
10
|
+
## Install and run
|
|
11
|
+
|
|
12
|
+
Once the package is published, an MCP host can spawn it with `npx` directly:
|
|
13
|
+
|
|
14
|
+
```jsonc
|
|
15
|
+
{
|
|
16
|
+
"mcpServers": {
|
|
17
|
+
"benchmaxx-arena": {
|
|
18
|
+
"command": "npx",
|
|
19
|
+
"args": ["-y", "@layr-labs/benchmaxx-arena-mcp"],
|
|
20
|
+
},
|
|
21
|
+
},
|
|
22
|
+
}
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
> **Not published yet.** Until the first release lands on npm the `npx` form
|
|
26
|
+
> above cannot resolve. Publishing is a manual, reviewer-gated workflow — see the
|
|
27
|
+
> [publish runbook](../docs/runbooks/publish-mcp-npm.md).
|
|
28
|
+
|
|
29
|
+
### Running from a local build
|
|
30
|
+
|
|
31
|
+
Until then, build it and point your host at the built entry point:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
cd mcp
|
|
35
|
+
npm ci
|
|
36
|
+
npm run build
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
```jsonc
|
|
40
|
+
{
|
|
41
|
+
"mcpServers": {
|
|
42
|
+
"benchmaxx-arena": {
|
|
43
|
+
"command": "node",
|
|
44
|
+
"args": ["/absolute/path/to/mcp/dist/index.js"],
|
|
45
|
+
},
|
|
46
|
+
},
|
|
47
|
+
}
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Configuration
|
|
51
|
+
|
|
52
|
+
- `BENCHMAXX_ARENA_URL` overrides the base URL. Only the origin (scheme, host,
|
|
53
|
+
port) is used; any path is dropped. The default is production
|
|
54
|
+
(`https://harnessarena.xyz`). The scheme must be `https`, except that
|
|
55
|
+
`http://localhost` and `http://127.0.0.1` are allowed for local development.
|
|
56
|
+
|
|
57
|
+
There is no config file and there are no command-line flags.
|
|
58
|
+
|
|
59
|
+
## Log in
|
|
60
|
+
|
|
61
|
+
Run the `login` tool once. It opens the arena in your browser on an approval
|
|
62
|
+
page that names the local port the server is listening on. Approve that one
|
|
63
|
+
request — check the port matches — and the arena hands the key back to that
|
|
64
|
+
local port. The tool then stores the minted API key under
|
|
65
|
+
`~/.benchmaxx/credentials.json` (directory mode `0700`, file mode `0600`), keyed
|
|
66
|
+
by the base-URL origin. The key expires after one year.
|
|
67
|
+
|
|
68
|
+
Approval is human-paced and usually takes longer than a host's default tool
|
|
69
|
+
timeout, so `login` emits progress while it waits, which keeps the host from
|
|
70
|
+
timing it out. Progress and any operator text go to **stderr** (stdout carries
|
|
71
|
+
the MCP protocol only). Cancelling the tool stops the wait and writes no
|
|
72
|
+
credential.
|
|
73
|
+
|
|
74
|
+
### No browser on this machine
|
|
75
|
+
|
|
76
|
+
If no browser can be opened, `login` returns an authorize URL instead. Open it
|
|
77
|
+
on any machine, approve the request, and pass the code the page displays to the
|
|
78
|
+
`finish_login` tool. The code is single-use and short-lived; the secret half of
|
|
79
|
+
the exchange never leaves this process and is held in memory only, so restarting
|
|
80
|
+
the server voids a pending login and you run `login` again.
|
|
81
|
+
|
|
82
|
+
The API key is returned once, at login, and cannot be recovered later. The
|
|
83
|
+
server never sees or stores your GitHub token.
|
|
84
|
+
|
|
85
|
+
## Tools
|
|
86
|
+
|
|
87
|
+
Auth:
|
|
88
|
+
|
|
89
|
+
- `login` — open the arena in your browser, wait for you to approve one login
|
|
90
|
+
request, store the key, and return your identity.
|
|
91
|
+
- `finish_login` — complete a login on a machine with no browser, with the
|
|
92
|
+
single-use code the approval page displayed.
|
|
93
|
+
- `whoami` — show the stored identity (username, key prefix) and base URL.
|
|
94
|
+
- `logout` — delete the stored credential locally (does not revoke it
|
|
95
|
+
server-side).
|
|
96
|
+
|
|
97
|
+
Catalog and leaderboard:
|
|
98
|
+
|
|
99
|
+
- `get_catalog` — benchmarks, variants, allowed models, harnesses, and latest
|
|
100
|
+
harness versions.
|
|
101
|
+
- `get_leaderboard` — ranked and below-the-bar rows with score, pass rate, cost,
|
|
102
|
+
and cost-per-pass, plus the baseline row a submission must beat. Filterable by
|
|
103
|
+
benchmark, variant, class, submitter, model, harness, and harness version, and
|
|
104
|
+
sortable via `order_by`/`asc`. Explains an empty board.
|
|
105
|
+
- `list_benchmarks` — benchmarks with allowed models and concurrency limits.
|
|
106
|
+
- `get_benchmark` — benchmark detail with variants: task names, trials,
|
|
107
|
+
concurrency, and timeouts.
|
|
108
|
+
- `get_insights` — per-task competition insights: which tasks are solved, which
|
|
109
|
+
are hard, and how submissions compare across them.
|
|
110
|
+
|
|
111
|
+
Submit and inspect:
|
|
112
|
+
|
|
113
|
+
- `submit_prompt` — create a `prompt_only` submission and return the submission
|
|
114
|
+
and run id. The model must be on the variant's allowed list. Always sends an
|
|
115
|
+
idempotency key; pass the same `idempotency_key` when retrying a submit whose
|
|
116
|
+
response you never saw, so the retry does not bill a second run.
|
|
117
|
+
- `list_my_submissions` — your submissions, newest first, with class and model.
|
|
118
|
+
- `get_submission` — public submission detail: state, per-task results, model
|
|
119
|
+
costs, and fingerprints.
|
|
120
|
+
- `get_submission_config` — the frozen harness config.
|
|
121
|
+
- `list_runs` — your runs with status, passes, cost, and integrity verdict.
|
|
122
|
+
- `get_run` — full run detail plus per-task, per-trial results.
|
|
123
|
+
- `get_run_timeline` — persisted run progress events after a sequence number,
|
|
124
|
+
with trial progress and a completion flag. Pass `since_seq` as the `next_seq`
|
|
125
|
+
from your last call. This is the way to follow a live run.
|
|
126
|
+
- `get_run_log` — the run's stdout/stderr chunks after a sequence number, with
|
|
127
|
+
gap and delivery-loss flags and an archive URL once one exists.
|
|
128
|
+
- `cancel_run` — request cancellation of one of your runs. Asynchronous, and
|
|
129
|
+
work already performed stays billed.
|
|
130
|
+
- `get_accounting_status` — whether run cost accounting is healthy, stale, or
|
|
131
|
+
blocked, and how far it has materialized. Check this before treating a run's
|
|
132
|
+
cost as final.
|
|
133
|
+
|
|
134
|
+
## Important
|
|
135
|
+
|
|
136
|
+
- **System prompts are PUBLIC.** A `prompt_only` submission's system prompt is
|
|
137
|
+
returned by public submission reads. Never put a secret in a prompt.
|
|
138
|
+
- **One run per submission.** To iterate, create a new submission.
|
|
139
|
+
- **Integrity is strict.** Forged reward files void and bill the run. Do not
|
|
140
|
+
target verifier paths (`/logs/verifier/`) or reward files.
|
|
141
|
+
- **Caps apply at submit time.** Per-user and benchmark-wide concurrency and a
|
|
142
|
+
monthly spend cap can reject a submission. Passing the caps still queues
|
|
143
|
+
behind run availability; a run can take one to two hours.
|
|
144
|
+
- **A variant's `capUsd` is deprecated.** It remains on the wire for
|
|
145
|
+
compatibility and now reports only a zero, so it is not the cap that governs a
|
|
146
|
+
run. Spend is governed by the platform and per-user spend policy.
|
|
147
|
+
|
|
148
|
+
## Not in this version
|
|
149
|
+
|
|
150
|
+
- `submit_harness` (full-harness image submission) is deferred to BEN-102.
|
|
151
|
+
- Training-consent management stays in the web UI.
|
|
152
|
+
- No remote/hosted transport; stdio only.
|
|
153
|
+
- No streaming event subscription. The server-streaming `GetRunEvents` RPC is
|
|
154
|
+
Unimplemented, because the in-process grpc-gateway cannot serve a stream, so
|
|
155
|
+
there is no tool for it — poll `get_run_timeline` instead.
|
|
@@ -0,0 +1,379 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Wire types for the benchmaxx-arena HTTP API.
|
|
3
|
+
*
|
|
4
|
+
* These mirror the committed OpenAPI spec
|
|
5
|
+
* (gen/openapi/benchmaxx-arena.swagger.json). grpc-gateway emits camelCase
|
|
6
|
+
* JSON, so these field names are camelCase, not the proto's snake_case. Only
|
|
7
|
+
* the fields the MCP tools read are typed; extra fields on the wire are
|
|
8
|
+
* ignored.
|
|
9
|
+
*
|
|
10
|
+
* Two gateway behaviors shape these declarations:
|
|
11
|
+
*
|
|
12
|
+
* - `EmitUnpopulated: true` is a repository-wide gateway policy
|
|
13
|
+
* (sdk/rpcServer/httpBodyMarshaler.go). An unset MESSAGE field therefore
|
|
14
|
+
* arrives as an explicit `null`, not as an absent key, so every
|
|
15
|
+
* message-typed field below is `| null`. Reading one with `=== undefined`
|
|
16
|
+
* is always false and is a bug. Unset SCALARS arrive as their protobuf zero
|
|
17
|
+
* (`0`, `""`, `false`), not as absent keys, so a zero is not evidence that
|
|
18
|
+
* the server omitted the field.
|
|
19
|
+
* - proto3 `int64` serializes as a JSON string, `int32` as a number. Fields
|
|
20
|
+
* that can exceed 2^53 (`seq`, `nextSeq`, `durationMs`) are therefore
|
|
21
|
+
* `string` and must be compared as BigInt, not parsed as Number.
|
|
22
|
+
*/
|
|
23
|
+
export interface CreateApiKeyResponse {
|
|
24
|
+
token: string;
|
|
25
|
+
id: string;
|
|
26
|
+
prefix: string;
|
|
27
|
+
}
|
|
28
|
+
/** POST /v1/auth/cli/authorize — minted by the signed-in browser, not by this client. */
|
|
29
|
+
export interface CreateCliAuthorizationResponse {
|
|
30
|
+
code?: string;
|
|
31
|
+
expiresAt?: string;
|
|
32
|
+
}
|
|
33
|
+
/** POST /v1/auth/cli/redeem — the one-time code plus the PKCE verifier. */
|
|
34
|
+
export interface RedeemCliAuthorizationResponse {
|
|
35
|
+
token?: string;
|
|
36
|
+
id?: string;
|
|
37
|
+
prefix?: string;
|
|
38
|
+
userId?: string;
|
|
39
|
+
username?: string;
|
|
40
|
+
expiresAt?: string;
|
|
41
|
+
}
|
|
42
|
+
export interface Benchmark {
|
|
43
|
+
id: string;
|
|
44
|
+
name: string;
|
|
45
|
+
datasetRef?: string;
|
|
46
|
+
active?: boolean;
|
|
47
|
+
registryUrl?: string;
|
|
48
|
+
maxConcurrentRuns?: number;
|
|
49
|
+
maxConcurrentRunsPerUser?: number;
|
|
50
|
+
allowedModels?: string[];
|
|
51
|
+
}
|
|
52
|
+
export interface BenchmarkVariant {
|
|
53
|
+
id: string;
|
|
54
|
+
benchmarkId: string;
|
|
55
|
+
name: string;
|
|
56
|
+
trials?: number;
|
|
57
|
+
concurrency?: number;
|
|
58
|
+
agentTimeoutMultiplier?: string;
|
|
59
|
+
wallTimeoutSeconds?: number;
|
|
60
|
+
/**
|
|
61
|
+
* @deprecated The per-variant spend cap was retired. The field remains on the
|
|
62
|
+
* wire for compatibility and now emits only its protobuf zero, so it is not a
|
|
63
|
+
* catalog assertion — do not present it as the cap that governs a run. Spend
|
|
64
|
+
* is governed by the admin-set platform and per-user spend policy.
|
|
65
|
+
*/
|
|
66
|
+
capUsd?: string;
|
|
67
|
+
taskNames?: string[];
|
|
68
|
+
}
|
|
69
|
+
export interface CatalogResponse {
|
|
70
|
+
benchmarks?: string[];
|
|
71
|
+
variants?: Record<string, {
|
|
72
|
+
values?: string[];
|
|
73
|
+
}>;
|
|
74
|
+
variantSummaries?: Record<string, {
|
|
75
|
+
values?: Record<string, unknown>;
|
|
76
|
+
}>;
|
|
77
|
+
benchmarkMeta?: Record<string, {
|
|
78
|
+
active?: boolean;
|
|
79
|
+
dataset?: string;
|
|
80
|
+
}>;
|
|
81
|
+
models?: string[];
|
|
82
|
+
harnesses?: string[];
|
|
83
|
+
latestHarnessVersion?: Record<string, string>;
|
|
84
|
+
templateRevision?: string;
|
|
85
|
+
}
|
|
86
|
+
export interface LeaderboardRow {
|
|
87
|
+
submissionId?: string;
|
|
88
|
+
submitters?: string[];
|
|
89
|
+
username?: string;
|
|
90
|
+
model?: string;
|
|
91
|
+
harness?: string;
|
|
92
|
+
harnessVersion?: string;
|
|
93
|
+
variant?: string;
|
|
94
|
+
benchmark?: string;
|
|
95
|
+
/** The primary ranking metric. */
|
|
96
|
+
score?: number;
|
|
97
|
+
scoreStddev?: number;
|
|
98
|
+
scoreMin?: number;
|
|
99
|
+
scoreMax?: number;
|
|
100
|
+
durationMs?: number;
|
|
101
|
+
costUsd?: number;
|
|
102
|
+
costMin?: number;
|
|
103
|
+
costMax?: number;
|
|
104
|
+
costPerPass?: number;
|
|
105
|
+
passRate?: number;
|
|
106
|
+
rank?: number;
|
|
107
|
+
runCount?: number;
|
|
108
|
+
excludedRuns?: number;
|
|
109
|
+
runtimeEpochs?: number;
|
|
110
|
+
unknownRuntimeRuns?: number;
|
|
111
|
+
taskCount?: number;
|
|
112
|
+
highFailureRateRuns?: number;
|
|
113
|
+
}
|
|
114
|
+
export interface LeaderboardResponse {
|
|
115
|
+
rows?: LeaderboardRow[];
|
|
116
|
+
ranked?: LeaderboardRow[];
|
|
117
|
+
belowBar?: LeaderboardRow[];
|
|
118
|
+
/**
|
|
119
|
+
* The baseline entry a submission must beat to rank. A message field, so an
|
|
120
|
+
* unset baseline is `null` — see the EmitUnpopulated note above. This
|
|
121
|
+
* replaced the scalar `baselinePass`, which no longer exists on the wire.
|
|
122
|
+
*/
|
|
123
|
+
baseline?: LeaderboardRow | null;
|
|
124
|
+
limit?: number;
|
|
125
|
+
offset?: number;
|
|
126
|
+
epochPendingRuns?: number;
|
|
127
|
+
runsExcludedByFailure?: number;
|
|
128
|
+
rankedRuntimeFingerprint?: string;
|
|
129
|
+
}
|
|
130
|
+
export interface ListBenchmarksResponse {
|
|
131
|
+
benchmarks?: Benchmark[];
|
|
132
|
+
nextPageToken?: string;
|
|
133
|
+
}
|
|
134
|
+
export interface GetBenchmarkResponse {
|
|
135
|
+
benchmark?: Benchmark | null;
|
|
136
|
+
variants?: BenchmarkVariant[];
|
|
137
|
+
}
|
|
138
|
+
export interface Submission {
|
|
139
|
+
id: string;
|
|
140
|
+
userId?: string;
|
|
141
|
+
imageId?: string;
|
|
142
|
+
submissionClass?: string;
|
|
143
|
+
benchmarkVariantId?: string;
|
|
144
|
+
status?: string;
|
|
145
|
+
createdAt?: string;
|
|
146
|
+
}
|
|
147
|
+
export interface CreateSubmissionResponse {
|
|
148
|
+
submission?: Submission | null;
|
|
149
|
+
runId?: string;
|
|
150
|
+
}
|
|
151
|
+
/** Reconciliation status for a run's model-call accounting. */
|
|
152
|
+
export interface RunReconciliationSummary {
|
|
153
|
+
status?: string;
|
|
154
|
+
affectedCallCount?: number;
|
|
155
|
+
recoveredCallCount?: number;
|
|
156
|
+
unresolvedCallCount?: number;
|
|
157
|
+
firstPendingAt?: string;
|
|
158
|
+
deadlineAt?: string;
|
|
159
|
+
resolvedAt?: string;
|
|
160
|
+
expiredAt?: string;
|
|
161
|
+
source?: string;
|
|
162
|
+
reason?: string;
|
|
163
|
+
routeAttestationStatus?: string;
|
|
164
|
+
}
|
|
165
|
+
export interface MySubmissionSummary {
|
|
166
|
+
id: string;
|
|
167
|
+
benchmarkId?: string;
|
|
168
|
+
variantId?: string;
|
|
169
|
+
submissionClass?: string;
|
|
170
|
+
model?: string;
|
|
171
|
+
createdAt?: string;
|
|
172
|
+
benchmarkName?: string;
|
|
173
|
+
variantName?: string;
|
|
174
|
+
runStatus?: string;
|
|
175
|
+
runStartedAt?: string;
|
|
176
|
+
runFinishedAt?: string;
|
|
177
|
+
passes?: number;
|
|
178
|
+
completedTasks?: number;
|
|
179
|
+
totalTasks?: number;
|
|
180
|
+
reconciliation?: RunReconciliationSummary | null;
|
|
181
|
+
duplicateOfSubmissionId?: string;
|
|
182
|
+
/** Set when the submission exists but is held off the leaderboard. */
|
|
183
|
+
leaderboardExclusionReason?: string;
|
|
184
|
+
}
|
|
185
|
+
export interface ListMySubmissionsResponse {
|
|
186
|
+
submissions?: MySubmissionSummary[];
|
|
187
|
+
}
|
|
188
|
+
export interface SubmissionResponse {
|
|
189
|
+
id: string;
|
|
190
|
+
state?: string;
|
|
191
|
+
tasksDone?: number;
|
|
192
|
+
tasksTotal?: number;
|
|
193
|
+
primaryModelName?: string;
|
|
194
|
+
benchmark?: string;
|
|
195
|
+
variant?: string;
|
|
196
|
+
createdAt?: string;
|
|
197
|
+
submissionClass?: string;
|
|
198
|
+
[key: string]: unknown;
|
|
199
|
+
}
|
|
200
|
+
export interface SubmissionConfigResponse {
|
|
201
|
+
instruction?: string;
|
|
202
|
+
skills?: string;
|
|
203
|
+
agents?: string;
|
|
204
|
+
harnessVersion?: string;
|
|
205
|
+
}
|
|
206
|
+
/** Exact decimal money, carried as text so no float rounding occurs. */
|
|
207
|
+
export interface ExactDecimal {
|
|
208
|
+
[key: string]: unknown;
|
|
209
|
+
}
|
|
210
|
+
/** Cost accounting for a run, including unresolved exposure. */
|
|
211
|
+
export interface RunAccountingSummary {
|
|
212
|
+
runExactCostUsd?: ExactDecimal | null;
|
|
213
|
+
runMaximumUnknownExposureUsd?: ExactDecimal | null;
|
|
214
|
+
runSafetyCostUsd?: ExactDecimal | null;
|
|
215
|
+
runEffectiveCostUsd?: ExactDecimal | null;
|
|
216
|
+
runTotalFinal?: boolean;
|
|
217
|
+
costUnresolved?: boolean;
|
|
218
|
+
accountingHealth?: string;
|
|
219
|
+
unsettledCallCount?: number;
|
|
220
|
+
unresolvedCallCount?: number;
|
|
221
|
+
materializedThrough?: string;
|
|
222
|
+
failureReason?: string;
|
|
223
|
+
}
|
|
224
|
+
export interface RunQueueWait {
|
|
225
|
+
/** int64 on the wire. */
|
|
226
|
+
runsAhead?: string;
|
|
227
|
+
waitingSince?: string;
|
|
228
|
+
}
|
|
229
|
+
export interface RunDiagnostic {
|
|
230
|
+
[key: string]: unknown;
|
|
231
|
+
}
|
|
232
|
+
export interface Run {
|
|
233
|
+
id: string;
|
|
234
|
+
submissionId?: string;
|
|
235
|
+
benchmarkVariantId?: string;
|
|
236
|
+
imageId?: string;
|
|
237
|
+
status?: string;
|
|
238
|
+
protocol?: string;
|
|
239
|
+
protocolHash?: string;
|
|
240
|
+
attempt?: number;
|
|
241
|
+
integrityVerdict?: string;
|
|
242
|
+
passes?: number;
|
|
243
|
+
costUsd?: string;
|
|
244
|
+
artifactsMissing?: boolean;
|
|
245
|
+
startedAt?: string;
|
|
246
|
+
completedAt?: string;
|
|
247
|
+
createdAt?: string;
|
|
248
|
+
infrastructureFailedAt?: string;
|
|
249
|
+
infrastructureFailureReason?: string;
|
|
250
|
+
replacesRunId?: string;
|
|
251
|
+
replacementRunId?: string;
|
|
252
|
+
modelCallBilledCount?: number;
|
|
253
|
+
modelCallFailureCount?: number;
|
|
254
|
+
failedCallRate?: number;
|
|
255
|
+
accounting?: RunAccountingSummary | null;
|
|
256
|
+
executionContractVersion?: string;
|
|
257
|
+
diagnostic?: RunDiagnostic | null;
|
|
258
|
+
unrankedAccountingUnresolved?: boolean;
|
|
259
|
+
/** Set once a cancel has been requested; the run may still be winding down. */
|
|
260
|
+
cancelRequestedAt?: string;
|
|
261
|
+
reconciliation?: RunReconciliationSummary | null;
|
|
262
|
+
}
|
|
263
|
+
export interface RunTaskResult {
|
|
264
|
+
id?: string;
|
|
265
|
+
runId?: string;
|
|
266
|
+
taskName?: string;
|
|
267
|
+
trialIndex?: number;
|
|
268
|
+
reward?: string;
|
|
269
|
+
passed?: boolean;
|
|
270
|
+
durationMs?: string;
|
|
271
|
+
forgeryDetected?: boolean;
|
|
272
|
+
score?: number;
|
|
273
|
+
error?: string;
|
|
274
|
+
costUsd?: number;
|
|
275
|
+
}
|
|
276
|
+
export interface ListRunsResponse {
|
|
277
|
+
runs?: Run[];
|
|
278
|
+
nextPageToken?: string;
|
|
279
|
+
}
|
|
280
|
+
export interface GetRunResponse {
|
|
281
|
+
run?: Run | null;
|
|
282
|
+
taskResults?: RunTaskResult[];
|
|
283
|
+
queueWait?: RunQueueWait | null;
|
|
284
|
+
}
|
|
285
|
+
/** One persisted run-progress event from the unary timeline. */
|
|
286
|
+
export interface RunTimelineEvent {
|
|
287
|
+
/** int64 on the wire; compare as BigInt. */
|
|
288
|
+
seq?: string;
|
|
289
|
+
eventType?: string;
|
|
290
|
+
occurredAt?: string;
|
|
291
|
+
taskName?: string;
|
|
292
|
+
trialIndex?: number;
|
|
293
|
+
/** int64 on the wire. */
|
|
294
|
+
durationMs?: string;
|
|
295
|
+
phase?: string;
|
|
296
|
+
passed?: boolean;
|
|
297
|
+
}
|
|
298
|
+
export interface RunTimelineProgress {
|
|
299
|
+
completedTrials?: number;
|
|
300
|
+
totalTrials?: number;
|
|
301
|
+
flaggedTrials?: number;
|
|
302
|
+
}
|
|
303
|
+
/**
|
|
304
|
+
* GET /v1/runs/{id}/timeline. The unary reader for persisted run events.
|
|
305
|
+
*
|
|
306
|
+
* This replaced the server-streaming GetRunEvents RPC, which stays
|
|
307
|
+
* Unimplemented because the in-process grpc-gateway cannot serve a stream (see
|
|
308
|
+
* sdk/arenarpc/registrar.go). `nextSeq` is the resume cursor for the next poll
|
|
309
|
+
* and `complete` reports whether the run has finished producing events.
|
|
310
|
+
*/
|
|
311
|
+
export interface GetRunTimelineResponse {
|
|
312
|
+
events?: RunTimelineEvent[];
|
|
313
|
+
/** int64 on the wire. Pass back as `sinceSeq` to resume. */
|
|
314
|
+
nextSeq?: string;
|
|
315
|
+
complete?: boolean;
|
|
316
|
+
progress?: RunTimelineProgress | null;
|
|
317
|
+
}
|
|
318
|
+
/** One chunk of run stdout/stderr. */
|
|
319
|
+
export interface RunLogChunk {
|
|
320
|
+
/** int64 on the wire. */
|
|
321
|
+
seq?: string;
|
|
322
|
+
trialName?: string;
|
|
323
|
+
text?: string;
|
|
324
|
+
ts?: string;
|
|
325
|
+
/** True when this chunk continues the previous chunk's line. */
|
|
326
|
+
continuesLine?: boolean;
|
|
327
|
+
}
|
|
328
|
+
/** GET /v1/runs/{id}/log. */
|
|
329
|
+
export interface GetRunLogResponse {
|
|
330
|
+
chunks?: RunLogChunk[];
|
|
331
|
+
/** int64 on the wire. Pass back as `sinceSeq` to resume. */
|
|
332
|
+
nextSeq?: string;
|
|
333
|
+
complete?: boolean;
|
|
334
|
+
attempt?: number;
|
|
335
|
+
archiveAvailable?: boolean;
|
|
336
|
+
archiveUrl?: string;
|
|
337
|
+
/** True when chunks are missing from the middle of the log. */
|
|
338
|
+
hasGap?: boolean;
|
|
339
|
+
/** True when chunks were lost in delivery and cannot be recovered. */
|
|
340
|
+
hasDeliveryLoss?: boolean;
|
|
341
|
+
}
|
|
342
|
+
/** GET /v1/insights. */
|
|
343
|
+
export interface CompetitionInsightsResponse {
|
|
344
|
+
benchmark?: string;
|
|
345
|
+
variant?: string;
|
|
346
|
+
taskCount?: number;
|
|
347
|
+
rows?: Record<string, unknown>[];
|
|
348
|
+
tasks?: Record<string, unknown>[];
|
|
349
|
+
}
|
|
350
|
+
/** GET /v1/accounting/status. */
|
|
351
|
+
export interface GetAccountingStatusResponse {
|
|
352
|
+
/** One of ACCOUNTING_HEALTH_{UNSPECIFIED,HEALTHY,STALE,DLQ_BLOCKED}. */
|
|
353
|
+
health?: string;
|
|
354
|
+
materializedThrough?: string;
|
|
355
|
+
reason?: string;
|
|
356
|
+
retryGuidance?: string;
|
|
357
|
+
supportReference?: string;
|
|
358
|
+
}
|
|
359
|
+
/** POST /v1/runs/{runId}/cancel. */
|
|
360
|
+
export interface CancelRunResponse {
|
|
361
|
+
cancelRequestedAt?: string;
|
|
362
|
+
runStatus?: string;
|
|
363
|
+
}
|
|
364
|
+
export interface ApiKeyMetadata {
|
|
365
|
+
id: string;
|
|
366
|
+
name?: string;
|
|
367
|
+
prefix?: string;
|
|
368
|
+
createdAt?: string;
|
|
369
|
+
lastUsedAt?: string;
|
|
370
|
+
revoked?: boolean;
|
|
371
|
+
}
|
|
372
|
+
export interface ListApiKeysResponse {
|
|
373
|
+
keys?: ApiKeyMetadata[];
|
|
374
|
+
}
|
|
375
|
+
/** grpc-gateway error body. */
|
|
376
|
+
export interface RpcStatus {
|
|
377
|
+
code?: number;
|
|
378
|
+
message?: string;
|
|
379
|
+
}
|
package/dist/apiTypes.js
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Wire types for the benchmaxx-arena HTTP API.
|
|
3
|
+
*
|
|
4
|
+
* These mirror the committed OpenAPI spec
|
|
5
|
+
* (gen/openapi/benchmaxx-arena.swagger.json). grpc-gateway emits camelCase
|
|
6
|
+
* JSON, so these field names are camelCase, not the proto's snake_case. Only
|
|
7
|
+
* the fields the MCP tools read are typed; extra fields on the wire are
|
|
8
|
+
* ignored.
|
|
9
|
+
*
|
|
10
|
+
* Two gateway behaviors shape these declarations:
|
|
11
|
+
*
|
|
12
|
+
* - `EmitUnpopulated: true` is a repository-wide gateway policy
|
|
13
|
+
* (sdk/rpcServer/httpBodyMarshaler.go). An unset MESSAGE field therefore
|
|
14
|
+
* arrives as an explicit `null`, not as an absent key, so every
|
|
15
|
+
* message-typed field below is `| null`. Reading one with `=== undefined`
|
|
16
|
+
* is always false and is a bug. Unset SCALARS arrive as their protobuf zero
|
|
17
|
+
* (`0`, `""`, `false`), not as absent keys, so a zero is not evidence that
|
|
18
|
+
* the server omitted the field.
|
|
19
|
+
* - proto3 `int64` serializes as a JSON string, `int32` as a number. Fields
|
|
20
|
+
* that can exceed 2^53 (`seq`, `nextSeq`, `durationMs`) are therefore
|
|
21
|
+
* `string` and must be compared as BigInt, not parsed as Number.
|
|
22
|
+
*/
|
|
23
|
+
export {};
|
|
24
|
+
//# sourceMappingURL=apiTypes.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"apiTypes.js","sourceRoot":"","sources":["../src/apiTypes.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;GAqBG"}
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Open a URL in the user's default browser.
|
|
3
|
+
*
|
|
4
|
+
* The opener is invoked with an argv ARRAY and never through a shell, so no part
|
|
5
|
+
* of the URL can be interpreted as a shell metacharacter. A failure is reported
|
|
6
|
+
* as `false` rather than thrown: a machine with no browser is an ordinary case
|
|
7
|
+
* that falls back to printing the URL, not an error.
|
|
8
|
+
*/
|
|
9
|
+
import type { EventEmitter } from 'node:events';
|
|
10
|
+
/** The spawn shape this module depends on, injected for testing. */
|
|
11
|
+
export type SpawnLike = (command: string, args: string[], options: {
|
|
12
|
+
stdio: 'ignore';
|
|
13
|
+
detached: boolean;
|
|
14
|
+
}) => EventEmitter & {
|
|
15
|
+
unref: () => void;
|
|
16
|
+
};
|
|
17
|
+
export interface BrowserDeps {
|
|
18
|
+
spawn?: SpawnLike;
|
|
19
|
+
platform?: NodeJS.Platform | string;
|
|
20
|
+
/** How long to wait for the opener to report. */
|
|
21
|
+
timeoutMs?: number;
|
|
22
|
+
}
|
|
23
|
+
/** The platform opener and its arguments. */
|
|
24
|
+
export declare function browserCommand(platform: NodeJS.Platform | string, url: string): {
|
|
25
|
+
command: string;
|
|
26
|
+
args: string[];
|
|
27
|
+
};
|
|
28
|
+
/**
|
|
29
|
+
* Try to open `url`. Resolves true when the opener reported success. Resolves
|
|
30
|
+
* false when no browser could be opened, which is the signal to fall back to
|
|
31
|
+
* printing the URL.
|
|
32
|
+
*/
|
|
33
|
+
export declare function openInBrowser(url: string, deps?: BrowserDeps): Promise<boolean>;
|
package/dist/browser.js
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Open a URL in the user's default browser.
|
|
3
|
+
*
|
|
4
|
+
* The opener is invoked with an argv ARRAY and never through a shell, so no part
|
|
5
|
+
* of the URL can be interpreted as a shell metacharacter. A failure is reported
|
|
6
|
+
* as `false` rather than thrown: a machine with no browser is an ordinary case
|
|
7
|
+
* that falls back to printing the URL, not an error.
|
|
8
|
+
*/
|
|
9
|
+
import { spawn as nodeSpawn } from 'node:child_process';
|
|
10
|
+
/** The platform opener and its arguments. */
|
|
11
|
+
export function browserCommand(platform, url) {
|
|
12
|
+
if (platform === 'darwin') {
|
|
13
|
+
return { command: 'open', args: [url] };
|
|
14
|
+
}
|
|
15
|
+
if (platform === 'win32') {
|
|
16
|
+
// The empty string is `start`'s title argument; without it a URL containing
|
|
17
|
+
// '&' is mis-parsed.
|
|
18
|
+
return { command: 'cmd', args: ['/c', 'start', '', url] };
|
|
19
|
+
}
|
|
20
|
+
return { command: 'xdg-open', args: [url] };
|
|
21
|
+
}
|
|
22
|
+
/**
|
|
23
|
+
* Try to open `url`. Resolves true when the opener reported success. Resolves
|
|
24
|
+
* false when no browser could be opened, which is the signal to fall back to
|
|
25
|
+
* printing the URL.
|
|
26
|
+
*/
|
|
27
|
+
export async function openInBrowser(url, deps = {}) {
|
|
28
|
+
// Only ever hand a web URL to the platform opener.
|
|
29
|
+
let parsed;
|
|
30
|
+
try {
|
|
31
|
+
parsed = new URL(url);
|
|
32
|
+
}
|
|
33
|
+
catch {
|
|
34
|
+
return false;
|
|
35
|
+
}
|
|
36
|
+
if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
|
|
37
|
+
return false;
|
|
38
|
+
}
|
|
39
|
+
const spawn = deps.spawn ?? nodeSpawn;
|
|
40
|
+
const platform = deps.platform ?? process.platform;
|
|
41
|
+
const timeoutMs = deps.timeoutMs ?? 5000;
|
|
42
|
+
const { command, args } = browserCommand(platform, url);
|
|
43
|
+
return new Promise((resolve) => {
|
|
44
|
+
let settled = false;
|
|
45
|
+
const finish = (value) => {
|
|
46
|
+
if (settled)
|
|
47
|
+
return;
|
|
48
|
+
settled = true;
|
|
49
|
+
clearTimeout(timer);
|
|
50
|
+
resolve(value);
|
|
51
|
+
};
|
|
52
|
+
const timer = setTimeout(() => finish(false), timeoutMs);
|
|
53
|
+
let child;
|
|
54
|
+
try {
|
|
55
|
+
child = spawn(command, args, { stdio: 'ignore', detached: false });
|
|
56
|
+
}
|
|
57
|
+
catch {
|
|
58
|
+
finish(false);
|
|
59
|
+
return;
|
|
60
|
+
}
|
|
61
|
+
child.on('error', () => finish(false));
|
|
62
|
+
child.on('close', (code) => finish(code === 0));
|
|
63
|
+
});
|
|
64
|
+
}
|
|
65
|
+
//# sourceMappingURL=browser.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"browser.js","sourceRoot":"","sources":["../src/browser.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,EAAE,KAAK,IAAI,SAAS,EAAE,MAAM,oBAAoB,CAAA;AAiBvD,6CAA6C;AAC7C,MAAM,UAAU,cAAc,CAC5B,QAAkC,EAClC,GAAW;IAEX,IAAI,QAAQ,KAAK,QAAQ,EAAE,CAAC;QAC1B,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,GAAG,CAAC,EAAE,CAAA;IACzC,CAAC;IACD,IAAI,QAAQ,KAAK,OAAO,EAAE,CAAC;QACzB,4EAA4E;QAC5E,qBAAqB;QACrB,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,IAAI,EAAE,OAAO,EAAE,EAAE,EAAE,GAAG,CAAC,EAAE,CAAA;IAC3D,CAAC;IACD,OAAO,EAAE,OAAO,EAAE,UAAU,EAAE,IAAI,EAAE,CAAC,GAAG,CAAC,EAAE,CAAA;AAC7C,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,aAAa,CAAC,GAAW,EAAE,OAAoB,EAAE;IACrE,mDAAmD;IACnD,IAAI,MAAW,CAAA;IACf,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAA;IACvB,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,KAAK,CAAA;IACd,CAAC;IACD,IAAI,MAAM,CAAC,QAAQ,KAAK,OAAO,IAAI,MAAM,CAAC,QAAQ,KAAK,QAAQ,EAAE,CAAC;QAChE,OAAO,KAAK,CAAA;IACd,CAAC;IAED,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,IAAK,SAAkC,CAAA;IAC/D,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,OAAO,CAAC,QAAQ,CAAA;IAClD,MAAM,SAAS,GAAG,IAAI,CAAC,SAAS,IAAI,IAAI,CAAA;IACxC,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,GAAG,cAAc,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAA;IAEvD,OAAO,IAAI,OAAO,CAAU,CAAC,OAAO,EAAE,EAAE;QACtC,IAAI,OAAO,GAAG,KAAK,CAAA;QACnB,MAAM,MAAM,GAAG,CAAC,KAAc,EAAQ,EAAE;YACtC,IAAI,OAAO;gBAAE,OAAM;YACnB,OAAO,GAAG,IAAI,CAAA;YACd,YAAY,CAAC,KAAK,CAAC,CAAA;YACnB,OAAO,CAAC,KAAK,CAAC,CAAA;QAChB,CAAC,CAAA;QACD,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,SAAS,CAAC,CAAA;QAExD,IAAI,KAA2C,CAAA;QAC/C,IAAI,CAAC;YACH,KAAK,GAAG,KAAK,CAAC,OAAO,EAAE,IAAI,EAAE,EAAE,KAAK,EAAE,QAAQ,EAAE,QAAQ,EAAE,KAAK,EAAE,CAAC,CAAA;QACpE,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,CAAC,KAAK,CAAC,CAAA;YACb,OAAM;QACR,CAAC;QACD,KAAK,CAAC,EAAE,CAAC,OAAO,EAAE,GAAG,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAA;QACtC,KAAK,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,IAAY,EAAE,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,CAAC,CAAC,CAAA;IACzD,CAAC,CAAC,CAAA;AACJ,CAAC"}
|