@sandboxrouter/sdk 0.3.2 → 0.3.3

package/README.md

@@ -66,8 +66,8 @@ How to read it:
 |---|---|---|---|---|---|
 | Agent37 | $0.007 | none | not run yet | Whole-minute billing: monthly shape total / 730 / 60; running estimate includes default disk (2 vCPU / 4 GB / 4 GB disk = $4.76/mo); sleeping/stopped disk $0.09/GB-mo; gVisor; $5 minimum top-up, wallet must hold one running day | [pricing](https://www.agent37.com/docs/agents-api/billing) |
 | Boat | $0.018 | $20/mo (credited) | not run yet | Fixed sizes billed per second ($0.018/h for 2 vCPU / 4 GB up to $0.20/h for 16 / 32); needs a plan from $20/mo, paid back as VM time | [pricing](https://docs.boat.dev/pricing.md) |
-| Maritime | $0.022 | $20/mo (credited) | passed (2026-11-06) | Per second, no minimum runtime; running disk included; Starter $20/mo incl. $20 usage; paused disk $0.08/GB-month (default idle action deletes sandbox) | [pricing](https://maritime.sh/pricing) |
-| Sailboxes (Sail Research) | up to $0.062 | none | passed (2026-11-06) | Per second on what the box uses (CPU time, RAM it holds, disk $0.0007/GiB-h), billed per minute; RAM counts the VM's file cache, up to the 16 GiB minimum limit, so file-heavy work can hold more than you asked for; paused/sleeping free; creation $0.01 at 2-4 vCPU on the free plan ($5/mo credit with a card), waived on Pro $250/mo | [pricing](https://docs.sailresearch.com/sailboxes-pricing.md) |
+| Maritime | $0.022 | $20/mo (credited) | passed (2026-11-07) | Per second, no minimum runtime; running disk included; Starter $20/mo incl. $20 usage; paused disk $0.08/GB-month (default idle action deletes sandbox) | [pricing](https://maritime.sh/pricing) |
+| Sailboxes (Sail Research) | up to $0.062 | none | passed (2026-11-07) | Per second on what the box uses (CPU time, RAM it holds, disk $0.0007/GiB-h), billed per minute; RAM counts the VM's file cache, up to the 16 GiB minimum limit, so file-heavy work can hold more than you asked for; paused/sleeping free; creation $0.01 at 2-4 vCPU on the free plan ($5/mo credit with a card), waived on Pro $250/mo | [pricing](https://docs.sailresearch.com/sailboxes-pricing.md) |
 | Northflank | $0.067 | none | not run yet | Per second on Northflank Cloud, ephemeral disk included; sandboxes run as deployment services (microVMs) | [pricing](https://northflank.com/pricing) |
 | Morph Cloud | $0.100 | none | not run yet | $0.05/MCU; MCU-hours = max(vCPU, RAM/4 GB, disk/16 GB); snapshots extra | [pricing](https://cloud.morph.so/web/pricing) |
 | exe.dev | $0.105 | $15/mo | not run yet | Standalone reserved size, per second: $0.105/h per 2 vCPU / 4 GiB unit; stopped CPU/RAM free, retained disk separate; Personal from $15/mo (up to 16 vCPU / 32 GiB), Work $75/mo (32 / 64 without approval) | [pricing](https://exe.dev/docs/standalone-vms.md) |
@@ -76,10 +76,9 @@ How to read it:
 | Beam | $0.140 | $20/mo (credited) | not run yet | Per second, Performance tier (sandbox default); Developer plan $20/mo incl. $20 credits; GPUs billed separately | [pricing](https://www.beam.cloud/pricing) |
 | CodeSandbox SDK (Together) | $0.149 | none | not run yet | Fixed tiers: Pico 2 vCPU / 1 GB $0.0743/h, Nano 2 / 4 $0.1486/h, up to 64 / 128 $4.7552/h; credits $0.01486, billed per started minute; Build free, 10 concurrent; Scale $170/mo incl. 1,100 credits | [pricing](https://docs.together.ai/docs/together-code-sandbox-legacy) |
 | Fly.io Sprites | up to $0.164 | none | not run yet | Bills actual CPU and memory use; this is the full-use price; storage extra | [pricing](https://fly.io/sprites) |
-| Blaxel | $0.166 | none | passed (2026-11-06) | Billed on memory only ($0.0000115/GB-s), CPU scales with memory (~2 GB per core) | [pricing](https://blaxel.ai/pricing) |
+| Blaxel | $0.166 | none | passed (2026-11-07) | Billed on memory only ($0.0000115/GB-s), CPU scales with memory (~2 GB per core) | [pricing](https://blaxel.ai/pricing) |
 | HOPX | $0.166 | none | not run yet | Per second, plus storage; size comes from the template, not the request | [pricing](https://hopx.ai/pricing) |
 | Superserve | $0.166 | maybe $20/mo (credited) | not run yet | Per second active; superserve/base fixed at 2 vCPU / 4 GiB / 8 GiB disk; retained storage $0.000108/GiB-h (shared template plus changed blocks); Developer $20/mo incl. $20, max 4/16 with team templates; recurring Free allowance not published on pricing page | [pricing](https://superserve.ai/pricing) |
-| LangSmith Sandboxes | $0.189 | none | not run yet | Per second; memory must be 2 to 6 GiB per vCPU; storage $0.000123/GiB-h; 8 LSU ($8) of sandbox usage a month included; Developer plan max 10 sandboxes | [pricing](https://www.langchain.com/pricing) |
 | Tensorlake | up to $0.200 | none | not run yet | Bills active CPU (floor 20% of allocation); this is the full-use price; Pro plan is lower | [pricing](https://www.tensorlake.ai/pricing) |
 | Upstash Box | up to $0.200 | none | not run yet | Billed per active CPU core-hour only (memory and idle time free); estimate assumes every core of the box busy; fixed sizes 2/4, 4/8, 8/16 | [pricing](https://upstash.com/pricing/box) |
 | Cloudflare Sandbox | up to $0.216 | $5/mo | not run yet | 10 ms billing on active CPU and allocated instance memory; full-CPU estimate excludes allocated disk ($0.000252/GB-h), Worker and Durable Objects charges; needs Workers Paid ($5/mo) and bridges/cloudflare | [pricing](https://developers.cloudflare.com/containers/platform/pricing/) |
@@ -92,12 +91,13 @@ How to read it:
 | InstaBox (sign-up waitlisted) | $0.017 | none | not run yet | Per second while running on fixed sizes (2/4, 4/8, 8/16); paused or stopped boxes bill only /data beyond the included disk ($0.00011/GiB-h); Pro $20/mo includes $20 credits; free plan: 1 box at 4/8 | [pricing](https://instabox.sh/docs/pricing.md) |
 | Koyeb Sandboxes (not taking new accounts) | $0.058 | $29/mo | not run yet | Per second; fixed Standard instance sizes (1 vCPU + 1 GB = $0.0144/h); sandboxes need a paid plan (Pro $29/mo); public preview | [pricing](https://www.koyeb.com/docs/reference/instances) |
 | Dedalus Machines (sign-up waitlisted) | $0.149 | none | not run yet | Per second while awake; sleeping machines bill storage only (10 GiB included on Hobby, 20 on Pro); Hobby plan max 4 vCPU / 16 GiB; private beta | [pricing](https://docs.dedaluslabs.ai/dcs/dm/pricing) |
-| OpenScale Sandboxes | skipped (default image is 2 vCPU / 1 GB; a bigger size needs image.openscale) | none | passed (2026-11-06) | Per second on allocated size; no platform fee or minimum; paused = free; size comes from the template (E2B-compatible API); Singapore only; needs a pay-as-you-go key | [pricing](https://openscale.so/sandboxes) |
+| LangSmith Sandboxes (sign-up waitlisted) | $0.189 | none | not run yet | Per second; memory must be 2 to 6 GiB per vCPU; storage $0.000123/GiB-h; 8 LSU ($8) of sandbox usage a month included; Developer plan max 10 sandboxes | [pricing](https://www.langchain.com/pricing) |
+| OpenScale Sandboxes | skipped (default image is 2 vCPU / 1 GB; a bigger size needs image.openscale) | none | passed (2026-11-07) | Per second on allocated size; no platform fee or minimum; paused = free; size comes from the template (E2B-compatible API); Singapore only; needs a pay-as-you-go key | [pricing](https://openscale.so/sandboxes) |
 | Azure Container Apps dynamic sessions | skipped (max 1 vCPU) | none | not run yet | Flat $0.03 per session-hour (built-in Shell pools), billed per allocated session in 1-hour increments, East US; needs a session pool you create once | [pricing](https://learn.microsoft.com/en-us/azure/container-apps/billing#dynamic-sessions) |
-| Daytona | skipped (default image is 1 vCPU / 1 GB; a bigger size needs image.daytona) | none | failed 1 step (2026-11-06) | Per second, plus disk above 5 GiB | [pricing](https://www.daytona.io/pricing) |
+| Daytona | skipped (default image is 1 vCPU / 1 GB; a bigger size needs image.daytona) | none | passed (2026-11-07) | Per second, plus disk above 5 GiB | [pricing](https://www.daytona.io/pricing) |
 | Alibaba Cloud Agent Sandbox | skipped (max 2 GB memory) | none | not run yet | Per second; Default QoS, Hong Kong and regions outside mainland China; disk separately $0.00025308/GiB-h; code-interpreter-v1 template is 2 vCPU / 2 GiB with 10 GiB disk (E2B-compatible API) | [pricing](https://www.alibabacloud.com/help/en/agent-sandbox/product-overview/billing-overview) |
-| E2B | skipped (default image is 2 vCPU / 0.5 GB; a bigger size needs image.e2b) | none | passed (2026-11-06) | Per second; free Hobby tier ($100 one-time credit, 1-hour sessions, 20 at once); Pro $150/mo for longer sessions | [pricing](https://e2b.dev/pricing) |
-| Docker (local) | free | none | passed (2026-11-06) | Runs on your own machine; free, but shares your kernel, so weaker isolation than a cloud micro-VM | [pricing](https://docs.docker.com/engine/) |
+| E2B | skipped (default image is 2 vCPU / 0.5 GB; a bigger size needs image.e2b) | none | passed (2026-11-07) | Per second; free Hobby tier ($100 one-time credit, 1-hour sessions, 20 at once); Pro $150/mo for longer sessions | [pricing](https://e2b.dev/pricing) |
+| Docker (local) | free | none | passed (2026-11-07) | Runs on your own machine; free, but shares your kernel, so weaker isolation than a cloud micro-VM | [pricing](https://docs.docker.com/engine/) |
 
 Checked 2026-11-04 to 2026-11-06.
 
@@ -185,8 +185,9 @@ const c = await router.create({}, { provider: ["e2b", "daytona"], fallback: fals
 
 // Fallback moves on when a provider is down: a timeout, a network error, 429 or 5xx, and a create
 // that takes longer than createTimeoutSeconds (default 120; a sandbox that turns up later is
-// killed, or left in the ledger for `gc`). A setup error (400, 401, 402, 403, 404, 422: a bad key,
-// no billing, a bad image) stops instead, so a mistake isn't hidden by a pricier provider.
+// killed, or left in the ledger for `gc`), and when an account can't pay (402: no balance, plan
+// needed). A setup error (400, 401, 403, 404, 422: a bad key, a bad image or request) stops instead,
+// so a mistake isn't hidden by a pricier provider. Every failed attempt is on sbx.attempts.
 // fallback: "any" falls back on every error.
 const e = await router.create({}, { fallback: "any", createTimeoutSeconds: 300 });
 
@@ -306,8 +307,8 @@ It prints one line per step and exits 1 if any fails. `checkProvider(router, id)
 |---|---|---|
 | Agent37 | not run yet | |
 | Boat | not run yet | |
-| Maritime | 14/14 passed | 2026-11-06, v0.2.0 |
-| Sailboxes (Sail Research) | 14/14 passed | 2026-11-06, v0.3.0 |
+| Maritime | 14/14 passed | 2026-11-07, v0.3.2 |
+| Sailboxes (Sail Research) | 14/14 passed | 2026-11-07, v0.3.2 |
 | Northflank | not run yet | |
 | Morph Cloud | not run yet | |
 | exe.dev | not run yet | |
@@ -316,10 +317,9 @@ It prints one line per step and exits 1 if any fails. `checkProvider(router, id)
 | Beam | not run yet | |
 | CodeSandbox SDK (Together) | not run yet | |
 | Fly.io Sprites | not run yet | |
-| Blaxel | 13/13 passed; n/a: pause and resume | 2026-11-06, v0.2.0 |
+| Blaxel | 13/13 passed; n/a: pause and resume | 2026-11-07, v0.3.2 |
 | HOPX | not run yet | |
 | Superserve | not run yet | |
-| LangSmith Sandboxes | not run yet | |
 | Tensorlake | not run yet | |
 | Upstash Box | not run yet | |
 | Cloudflare Sandbox | not run yet | |
@@ -332,12 +332,13 @@ It prints one line per step and exits 1 if any fails. `checkProvider(router, id)
 | InstaBox | not run yet | |
 | Koyeb Sandboxes | not run yet | |
 | Dedalus Machines | not run yet | |
-| OpenScale Sandboxes | 14/14 passed | 2026-11-06, v0.2.0 |
+| LangSmith Sandboxes | not run yet | |
+| OpenScale Sandboxes | 14/14 passed | 2026-11-07, v0.3.2 |
 | Azure Container Apps dynamic sessions | not run yet | |
-| Daytona | 0/14 passed; failed: create | 2026-11-06, v0.2.0 |
+| Daytona | 14/14 passed | 2026-11-07, v0.3.2 |
 | Alibaba Cloud Agent Sandbox | not run yet | |
-| E2B | 14/14 passed | 2026-11-06, v0.2.0 |
-| Docker (local) | 14/14 passed | 2026-11-06, v0.2.0 |
+| E2B | 14/14 passed | 2026-11-07, v0.3.2 |
+| Docker (local) | 14/14 passed | 2026-11-07, v0.3.2 |
 
 ## HTTP API
 

package/dist/catalog.d.ts

@@ -58,7 +58,20 @@ export interface ProviderInfo {
         note: string;
         checked: string;
     };
+    /**
+     * Set when the provider's terms forbid publishing benchmarks or performance information about it.
+     * Its cold starts, success rate, measured size and real bills are then left out of shared reports,
+     * results/ and the website, and saved checks keep pass/fail without timings. List prices (public
+     * pricing pages) and the adapter itself are unaffected.
+     */
+    noBenchmarks?: {
+        clause: string;
+        source: string;
+        checked: string;
+    };
 }
+/** False when the provider's terms forbid publishing its benchmarks (see `noBenchmarks`). */
+export declare function benchmarksPublishable(id: string): boolean;
 /** Short label for a gated provider's sign-up status, or null when anyone can sign up. */
 export declare function accessLabel(id: ProviderId): string | null;
 /** Short label for the plan a provider needs before you can use it: "none", "$20/mo (credited)", "$5/mo". */

package/dist/catalog.js

@@ -3,6 +3,10 @@ import { AGENT37_SHAPES, agent37Shape } from "./providers/agent37.js";
 import { INSTABOX_SIZES, instaboxSize } from "./providers/instabox.js";
 import { FREESTYLE_SIZES, freestyleSize } from "./providers/freestyle.js";
 import { langsmithSize } from "./providers/langsmith.js";
+/** False when the provider's terms forbid publishing its benchmarks (see `noBenchmarks`). */
+export function benchmarksPublishable(id) {
+    return !CATALOG.find((c) => c.id === id)?.noBenchmarks;
+}
 /** Short label for a gated provider's sign-up status, or null when anyone can sign up. */
 export function accessLabel(id) {
     const a = CATALOG.find((c) => c.id === id)?.access;
@@ -172,6 +176,11 @@ const ENTRIES = [
         limits: { maxVcpu: 8, maxMemoryGb: 16 },
         credentials: [["BL_API_KEY", "BL_WORKSPACE"]],
         docs: "https://docs.blaxel.ai/Sandboxes/Overview",
+        noBenchmarks: {
+            clause: "Blaxel's terms (§2.6(f)) don't allow publishing benchmarks or performance information about it",
+            source: "https://blaxel.ai/legal/terms/2025-12-10-blaxel-terms-and-conditions.pdf",
+            checked: "2026-11-07",
+        },
     },
     {
         id: "tensorlake",
@@ -524,6 +533,11 @@ ENTRIES.push({
     limits: { maxVcpu: 16, maxMemoryGb: 96 },
     credentials: [["LANGSMITH_API_KEY"]],
     docs: "https://docs.langchain.com/langsmith/sandboxes",
+    access: {
+        status: "waitlist",
+        note: "Sandboxes are a private preview: until LangChain enables them for your organization, the API answers \"Sandbox feature is not enabled for this organization\"",
+        checked: "2026-11-07",
+    },
 }, {
     id: "novita",
     name: "Novita AI Agent Sandbox",

package/dist/cli.js

@@ -6,9 +6,9 @@ import { bench, benchMarkdown } from "./bench.js";
 import { costMarkdown, costRun, settle } from "./cost.js";
 import { checkProvider } from "./check.js";
 import { gc, listOpen } from "./gc.js";
-import { contributor } from "./results.js";
+import { contributor, publishableCheck } from "./results.js";
 import { loadReports, makeReport, renderLeaderboard } from "./leaderboard.js";
-import { DEFAULT_MEMORY_GB, DEFAULT_VCPU, accessLabel, hourlyCost, listingOrder, planLabel, unfit } from "./catalog.js";
+import { DEFAULT_MEMORY_GB, DEFAULT_VCPU, accessLabel, benchmarksPublishable, hourlyCost, listingOrder, planLabel, unfit } from "./catalog.js";
 import { Router } from "./router.js";
 import { createApp } from "./server.js";
 import { RouterError } from "./types.js";
@@ -215,7 +215,7 @@ async function main() {
             reports.push(report);
             if (saveDir) {
                 mkdirSync(saveDir, { recursive: true });
-                const saved = { tool: "sandboxrouter-check", version, date: new Date().toISOString(), platform: `${platform()}-${arch()}`, ...(by ? { by } : {}), ...report };
+                const saved = publishableCheck({ tool: "sandboxrouter-check", version, date: new Date().toISOString(), platform: `${platform()}-${arch()}`, ...(by ? { by } : {}), ...report });
                 writeFileSync(`${saveDir}/${p}.json`, JSON.stringify(saved, null, 2) + "\n");
             }
         }
@@ -277,6 +277,9 @@ async function main() {
         if (values.share) {
             const version = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8")).version;
             const report = makeReport(rows, { version, runs: rows[0].runs, spec, label: values.label, by });
+            const left = rows.filter((r) => !benchmarksPublishable(r.provider)).map((r) => r.provider);
+            if (left.length)
+                console.log(`\nleft out of the shared report: ${left.join(", ")} (their terms don't allow publishing benchmarks)`);
             const file = values.out ?? `sandboxrouter-bench-${report.date.slice(0, 10)}-${Math.random().toString(36).slice(2, 8)}.json`;
             writeFileSync(file, JSON.stringify(report, null, 2) + "\n");
             console.log(`\nwrote ${file}. Add it to results/ in a PR to github.com/gigialc/sandboxrouter to put it on the leaderboard.`);
@@ -301,7 +304,11 @@ async function main() {
             });
             const file = values.out ?? `results/cost/cost-${run.date.slice(0, 16).replace(/[:T]/g, "-")}.json`;
             mkdirSync(file.replace(/\/[^/]*$/, "") || ".", { recursive: true });
-            writeFileSync(file, JSON.stringify(await settle(run), null, 2) + "\n");
+            const settled = await settle(run);
+            const left = settled.entries.filter((e) => !benchmarksPublishable(e.provider)).map((e) => e.provider);
+            writeFileSync(file, JSON.stringify({ ...settled, entries: settled.entries.filter((e) => benchmarksPublishable(e.provider)) }, null, 2) + "\n");
+            if (left.length)
+                console.log(`left out of ${file}: ${left.join(", ")} (their terms don't allow publishing benchmarks; the table above is the only copy)`);
             console.log(costMarkdown(run));
             console.log(`\nwrote ${file}. Bills post late on some providers: run "sandboxrouter cost settle ${file}" again later.`);
             return;

package/dist/cost.d.ts

@@ -52,6 +52,13 @@ export interface CostRun {
     by?: string;
     entries: CostEntry[];
 }
+/**
+ * The size the sandbox really has, as "<vcpus> <memoryMb>". nproc and MemTotal can show the whole host
+ * (Daytona's containers report 64 CPUs and 755 GB while limited to 1 vCPU and 1 GB), so the cgroup limits
+ * win when they're set: cgroup v2 cpu.max / memory.max, or v1 cfs quota / memory limit. vCPUs can be
+ * fractional (a 50% quota is 0.5).
+ */
+export declare const PROBE: string;
 export declare function costRun(opts: {
     providers: ProviderId[];
     minutes: number;

package/dist/cost.js

@@ -1,6 +1,18 @@
 import { fittingSpec, hourlyCost } from "./catalog.js";
 import { Router } from "./router.js";
-const PROBE = `echo "$(nproc) $(awk '/MemTotal/{printf "%d", $2/1024}' /proc/meminfo)"`;
+/**
+ * The size the sandbox really has, as "<vcpus> <memoryMb>". nproc and MemTotal can show the whole host
+ * (Daytona's containers report 64 CPUs and 755 GB while limited to 1 vCPU and 1 GB), so the cgroup limits
+ * win when they're set: cgroup v2 cpu.max / memory.max, or v1 cfs quota / memory limit. vCPUs can be
+ * fractional (a 50% quota is 0.5).
+ */
+export const PROBE = [
+    `n=$(nproc); m=$(awk '/MemTotal/{printf "%d", $2/1024}' /proc/meminfo)`,
+    `q=; p=; l=`,
+    `if [ -r /sys/fs/cgroup/cpu.max ]; then read q p < /sys/fs/cgroup/cpu.max; elif [ -r /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]; then q=$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us); p=$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us); fi`,
+    `if [ -r /sys/fs/cgroup/memory.max ]; then l=$(cat /sys/fs/cgroup/memory.max); elif [ -r /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then l=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes); fi`,
+    `awk -v n="$n" -v m="$m" -v q="$q" -v p="$p" -v l="$l" 'BEGIN { c = n; if (q ~ /^[0-9]+$/ && p ~ /^[0-9]+$/ && p > 0 && q / p < c) c = q / p; if (l ~ /^[0-9]+$/ && l / 1048576 < m) m = int(l / 1048576); printf "%s %d\\n", (c == int(c) ? c : sprintf("%.2f", c)), m }'`,
+].join("; ");
 // One `yes` per core, detached so the exec returns; they die with the sandbox.
 const LOAD = `nohup sh -c 'for i in $(seq $(nproc)); do yes > /dev/null & done; wait' > /dev/null 2>&1 &`;
 const round = (n, d = 6) => Math.round(n * 10 ** d) / 10 ** d;

package/dist/leaderboard.js

@@ -1,6 +1,6 @@
 import { readdirSync, readFileSync } from "node:fs";
 import { join } from "node:path";
-import { CATALOG, fittingSpec, hourlyCost } from "./catalog.js";
+import { benchmarksPublishable, CATALOG, fittingSpec, hourlyCost } from "./catalog.js";
 import { median } from "./bench-store.js";
 export function makeReport(rows, opts) {
     return {
@@ -14,7 +14,8 @@ export function makeReport(rows, opts) {
         spec: { vcpu: opts.spec.vcpu, memoryGb: opts.spec.memoryGb },
         node: process.version,
         platform: `${process.platform}-${process.arch}`,
-        rows: rows.map(({ lastError: _drop, ...r }) => r),
+        // Providers whose terms forbid publishing benchmarks stay out of anything meant to be shared.
+        rows: rows.filter((r) => benchmarksPublishable(r.provider)).map(({ lastError: _drop, ...r }) => r),
     };
 }
 export function loadReports(dir) {

package/dist/results.d.ts

@@ -14,6 +14,17 @@ export interface Issue {
 }
 /** A GitHub handle, without the "@": what `--by` accepts and what reports store. */
 export declare function contributor(by?: string): string | undefined;
+/**
+ * A saved check as it may be published: for a provider whose terms forbid publishing benchmarks,
+ * pass/fail stays and every timing goes (the per-step `ms`, and notes that quote a duration).
+ */
+export declare function publishableCheck<T extends {
+    provider: string;
+    results: {
+        ms?: number;
+        detail?: string;
+    }[];
+}>(saved: T): T;
 /** Problems with one parsed result file. `now` is injectable for tests. */
 export declare function validateResult(j: any, now?: number): string[];
 /** Problems found anywhere in a file's raw text: secrets, emails, IP addresses. */
@@ -29,5 +40,6 @@ export declare function validateResults(dir?: string, opts?: {
         status: string;
         path: string;
     }[];
+    previous?: (path: string) => string | undefined;
     now?: number;
 }): Issue[];

package/dist/results.js

@@ -1,6 +1,6 @@
 import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
 import { join, relative } from "node:path";
-import { CATALOG } from "./catalog.js";
+import { benchmarksPublishable, CATALOG } from "./catalog.js";
 import { CHECK_STEPS } from "./check.js";
 /** A GitHub handle, without the "@": what `--by` accepts and what reports store. */
 export function contributor(by) {
@@ -13,6 +13,22 @@ export function contributor(by) {
     return h;
 }
 const IDS = new Set(CATALOG.map((c) => c.id));
+/** A timing in a check step's note, e.g. "first output after 100 ms" or "threw after 3.0 s". */
+const TIMING = /\d[\d,.]*\s?(?:ms|s|sec|seconds?)\b/i;
+/**
+ * A saved check as it may be published: for a provider whose terms forbid publishing benchmarks,
+ * pass/fail stays and every timing goes (the per-step `ms`, and notes that quote a duration).
+ */
+export function publishableCheck(saved) {
+    if (benchmarksPublishable(saved.provider))
+        return saved;
+    return {
+        ...saved,
+        results: saved.results.map(({ ms: _ms, detail, ...r }) => ({ ...r, ...(detail && !TIMING.test(detail) ? { detail } : {}) })),
+    };
+}
+const noBench = (id) => typeof id === "string" && IDS.has(id) && !benchmarksPublishable(id);
+const NO_BENCH = "its terms don't allow publishing benchmarks (noBenchmarks in src/catalog.ts); leave it out of shared results";
 const STEPS = new Set(CHECK_STEPS);
 const MAX_MS = 15 * 60_000;
 /** Key and token shapes from the providers here and the usual suspects. Matches print as a prefix only. */
@@ -88,6 +104,8 @@ export function validateResult(j, now = Date.now()) {
             out.push("rows is empty");
         for (const r of Array.isArray(j.rows) ? j.rows : []) {
             provider(r?.provider, "row provider");
+            if (noBench(r?.provider))
+                out.push(`${r.provider}: ${NO_BENCH}`);
             if (!isNum(r?.runs, 1, 1000))
                 out.push(`${r?.provider}: runs is not a positive number`);
             if (!isNum(r?.failures, 0, r?.runs ?? 0))
@@ -113,8 +131,11 @@ export function validateResult(j, now = Date.now()) {
                 out.push(`unknown check step "${r?.step}"`);
             if (!["pass", "fail", "skip", "info"].includes(r?.status))
                 out.push(`${r?.step}: unknown status "${r?.status}"`);
-            if (!isNum(r?.ms, 0, MAX_MS))
+            if (r?.ms !== undefined && !isNum(r.ms, 0, MAX_MS))
                 out.push(`${r?.step}: ms must be between 0 and 15 min`);
+            if (noBench(j.provider) && (r?.ms !== undefined || (typeof r?.detail === "string" && TIMING.test(r.detail)))) {
+                out.push(`${r?.step}: has timings, but ${j.provider}'s terms don't allow publishing performance information; save with check --save`);
+            }
         }
     }
     else if (j.tool === "sandboxrouter-cost") {
@@ -126,6 +147,8 @@ export function validateResult(j, now = Date.now()) {
             out.push("entries is empty");
         for (const e of Array.isArray(j.entries) ? j.entries : []) {
             provider(e?.provider, "entry provider");
+            if (noBench(e?.provider))
+                out.push(`${e.provider}: ${NO_BENCH}`);
             if (e?.seconds !== undefined && !isNum(e.seconds, 0, 2 * 24 * 3600))
                 out.push(`${e?.provider}: seconds is out of range`);
             if (e?.predictedUsd !== undefined && !isNum(e.predictedUsd, 0, 1000))
@@ -206,10 +229,32 @@ export function validateResults(dir = "results", opts = {}) {
             continue;
         if (p.startsWith(`${dir}/conformance/`) || p.startsWith(`${dir}/cost/`))
             continue;
-        if (/^[MR]/.test(c.status))
+        if (/^[MR]/.test(c.status) && !onlyDropsUnpublishable(opts.previous?.(p), p)) {
             issues.push({ file: p, problem: "edits an existing bench report; add your run as a new file instead" });
+        }
         if (c.status.startsWith("D"))
             issues.push({ file: p, problem: "deletes a bench report; reports are only removed by a maintainer" });
     }
     return issues;
 }
+/**
+ * True when a bench report's only change is removing rows for providers whose terms forbid publishing
+ * benchmarks: the one edit to someone's report that's allowed, so those numbers can be taken down.
+ */
+function onlyDropsUnpublishable(before, path) {
+    if (before === undefined || !existsSync(path))
+        return false;
+    try {
+        const a = JSON.parse(before);
+        const b = JSON.parse(readFileSync(path, "utf8"));
+        const { rows: ra, ...restA } = a;
+        const { rows: rb, ...restB } = b;
+        if (JSON.stringify(restA) !== JSON.stringify(restB))
+            return false;
+        const kept = ra.filter((r) => benchmarksPublishable(r.provider));
+        return JSON.stringify(kept) === JSON.stringify(rb);
+    }
+    catch {
+        return false;
+    }
+}

package/dist/router.js

@@ -211,7 +211,7 @@ export class Router {
                 const setup = err instanceof SetupError || (status !== undefined && SETUP_ERRORS.has(status));
                 if (route.fallback !== "any" && setup) {
                     const msg = err.message.replace(/[.\s]+$/, "");
-                    throw new RouterError(`${msg.startsWith(c.provider) ? msg : `${c.provider}: ${msg}`}. This looks like a setup problem with ${c.provider} (key, billing, image, request or a missing package), ` +
+                    throw new RouterError(`${msg.startsWith(c.provider) ? msg : `${c.provider}: ${msg}`}. This looks like a setup problem with ${c.provider} (key, image, request or a missing package), ` +
                         `so the router stopped instead of moving the work to another provider; pass fallback: "any" to fall back on every error`, attempts);
                 }
             }
@@ -336,8 +336,12 @@ function round(n) {
 }
 /** Seconds to wait for one provider's create before trying the next (RouteOptions.createTimeoutSeconds). */
 export const DEFAULT_CREATE_TIMEOUT_S = 120;
-/** HTTP statuses that mean the request or the account is wrong, not that the provider is down. */
-const SETUP_ERRORS = new Set([400, 401, 402, 403, 404, 422]);
+/**
+ * HTTP statuses that mean the request or the key is wrong, not that the provider is down. 402 (out of
+ * balance, plan needed) isn't one: an unfunded account is exactly when the next provider should take the
+ * sandbox, and the failed attempt stays on `attempts` (the CLI prints it), so it isn't hidden.
+ */
+const SETUP_ERRORS = new Set([400, 401, 403, 404, 422]);
 /** Seconds past a command's timeout before the router stops waiting for the adapter. */
 export const EXEC_GRACE_S = 30;
 /** Resolve with `work`, or reject with `onExpire()` after `ms`. A late `work` rejection is swallowed. */

package/dist/types.d.ts

@@ -33,9 +33,10 @@ export interface RouteOptions {
     strategy?: Strategy;
     /**
      * What to do when a create fails. `true` (default): try the next candidate when the provider
-     * looks unavailable (timeout, network error, 429, 5xx, or an error without a status), but stop on
-     * a setup error (400, 401, 402, 403, 404, 422: a bad key, no billing, a bad image or request), so
-     * a mistake isn't hidden by quietly moving the work to another, maybe pricier, provider.
+     * looks unavailable (timeout, network error, 429, 5xx, or an error without a status) or can't bill
+     * (402: no balance, plan needed), but stop on a setup error (400, 401, 403, 404, 422: a bad key, a
+     * bad image or request), so a mistake isn't hidden by quietly moving the work to another, maybe
+     * pricier, provider. Every failed attempt is on `sandbox.attempts`.
      * `"any"`: try the next candidate on every error. `false`: only the first candidate.
      */
     fallback?: boolean | "any";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sandboxrouter/sdk",
-  "version": "0.3.2",
+  "version": "0.3.3",
   "description": "One API for every sandbox provider: route AI agents' code by price, speed or capability, with automatic fallback. Free and open source.",
   "type": "module",
   "main": "dist/index.js",