@genex-ai/cli-demo 1.36.11 → 1.36.12-dev.995
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{blender-mcp-QVEGHE5L.js → blender-mcp-WSURPZET.js} +2 -2
- package/dist/{blender-serve-66OHC4ML.js → blender-serve-EH2MOW3U.js} +1 -1
- package/dist/{chunk-4BBTPRJW.js → chunk-EABIVAGT.js} +2 -2
- package/dist/{chunk-QI7FIYBY.js → chunk-K4EHXOZK.js} +1 -1
- package/dist/index.js +490 -229
- package/package.json +1 -1
- package/templates/skills/genex-game-director/SKILL.md +1 -1
- package/templates/skills/genex-getting-started/SKILL.md +2 -2
- package/templates/skills/genex-llm-in-games/SKILL.md +209 -72
- package/templates/skills/genex-llm-in-games/references/pricing.md +126 -85
- package/templates/skills/genex-monetization/SKILL.md +1 -9
- package/templates/skills/genex-tool-llm/SKILL.md +11 -9
- package/templates/skills/genex-updates/SKILL.md +1 -1
package/dist/index.js
CHANGED
|
@@ -57,7 +57,7 @@ import {
|
|
|
57
57
|
writeSecretFile,
|
|
58
58
|
writeUserToken,
|
|
59
59
|
writeWorkspace
|
|
60
|
-
} from "./chunk-
|
|
60
|
+
} from "./chunk-EABIVAGT.js";
|
|
61
61
|
import {
|
|
62
62
|
CLI_CHANNEL,
|
|
63
63
|
DEFAULT_API_URL,
|
|
@@ -78,7 +78,7 @@ import {
|
|
|
78
78
|
normalizeApiOrigin,
|
|
79
79
|
originKey,
|
|
80
80
|
resolveAgentTargets
|
|
81
|
-
} from "./chunk-
|
|
81
|
+
} from "./chunk-K4EHXOZK.js";
|
|
82
82
|
|
|
83
83
|
// src/instrument.ts
|
|
84
84
|
import { setTimeout as sleep } from "timers/promises";
|
|
@@ -1107,7 +1107,7 @@ Important note: put soul into your creations, with many details and love. Aim to
|
|
|
1107
1107
|
21. A turn ends in exactly one of two ways: on a question the player must answer before the next step can be chosen, or on a handoff \u2014 the draft link, what changed, what to try, and what is still open. Never close a turn on a promise. "I'll keep building", "while I keep working", "adding that now" are things you say and then DO before you stop; if you are stopping, say that you have stopped and what remains. While the Build plan's \`Now:\` line has open work and no answer is needed from the player, the turn is not over: preview, hand off, and start the next milestone in the same turn, until the requested outcome is reached or the platform's own budget ends the session. If a stop was forced on you anyway, the first line of your next turn is the promise you left and where it stands.
|
|
1108
1108
|
22. **What the player sees must read as the thing it is**, in this game's own style: a building as that building, a person as a person, a prop as that prop, a surface as its material \u2014 and a coloured box, a capsule, or a flat grey block standing in for one of them is never its finished version. Generation is the default route to that bar wherever code will not honestly reach it: the player's character, whatever the request names, whatever the player walks up to, enters, or interacts with, and the music and sound underneath \u2014 reach for the belt on your own judgment, without stopping to ask; a status line saying what you queued is enough, and \`--no-wait\` keeps you building while it lands. Procedural code is a first-class engine for what is structural, repeated, distant, or parametric \u2014 terrain, sky, fences, paving, modular kits, filler \u2014 and for anything else only when the result meets the same bar and you have looked at it in a capture before its row says \`landed\`. Start with one reusable implementation per distinct required gameplay or visual role. Background populations default to reusable procedural bodies and motion or compatible existing rigged assets. Catalog clips need a compatible skeleton and may be paid. Reserve paid custom bodies for the player and characters examined or interacted with closely, while honoring explicit user requests. Add variants only for an unmet requirement; unspent allowance is not unfinished work. Allowance questions use timeoutPolicy no-consent: silence does not raise the allowance. Record role, reuse and completion criteria in the existing Assets table. Decide per object, not per category, and write the route on each Assets row in \`DESIGN.md\` \u2014 the paid flow for generated pieces, the procedural flow with its local path for code-built ones. An Assets table with no generation in it is a decision, not a default: record it as \`Generation: none \u2014 <why code alone reaches the bar here>\` or generate.
|
|
1109
1109
|
23. **The screenshots you take to look at your own work go in \`.genex/scratch/\`.** Captures, render comparisons, before/after strips, traces and metrics dumps are how you SEE what you built \u2014 they are not part of what you built, and \`genex preview\` pushes the whole folder to the game's source, every time, forever. \`.genex/scratch/\` already exists for this and is already ignored, so there is nothing to set up: write the capture as \`.genex/scratch/arena-before.png\` and read it back from there. Never put them in a folder of your own at the top level: \`progress/\`, \`reports/\`, \`shots/\` and their kind are pushed like source and become permanent weight in the game and in every remix of it \u2014 one real game reached 11 GB and 10,332 committed screenshots exactly this way. If you find such a folder already there, add it to \`.gitignore\`; never delete the player's files (law 18). One file in \`.genex/scratch/\` IS read, by the platform: \`.genex/scratch/cover.png\` is the game's cover \u2014 the frame you would put on its poster: real gameplay at its signature moment, well lit, no menus, popups or busy HUD, landscape 16:9, and never any text (the page sets the title beside it). The next \`genex preview\` makes it the cover; save a better moment over it whenever one comes along, and if your browser tool can set the cover itself, use that instead. Before the first publish, and whenever the game's look changes, load \`$genex-cover\`: it chooses, stages, lights and judges that frame.
|
|
1110
|
-
24. **Benchmark before declaring an in-game LLM
|
|
1110
|
+
24. **Benchmark before declaring an in-game LLM ceiling.** If the game calls a language model while the player plays \u2014 an NPC answering in its own words, a quest written for this save, a judge reading what the player typed \u2014 the PLAYER pays for it from their Genex credits, billed as used, and the game declares no price, only a per-call CEILING: \`maxCredits\` on \`generate()\`, \`perCallMaxCredits\` on a standing budget. That ceiling bounds what one call may cost AND funds how long its answer may be, so never pick it by judgement or from a vendor's rate card. Run the real prompt on your own credits \u2014 \`npx genex llm bench "<the prompt>" --schema <file> --samples 3 --max-credits <n> --user-approved\` \u2014 and declare the figures it prints, recorded in \`DESIGN.md\` with its model and date; re-benchmark whenever the prompt, the schema or the model changes, because a prompt edit changes the ceiling. Check the lane first with \`npx genex llm models\`: where it is off the routes 404, so build the feature behind a graceful unavailable state and say so in the handoff. A repeated call is a standing budget the player approves once (\`requestSpendGrant()\`, then \`generate({ grantId })\`), never a popup per turn, and its disclosed call rate and credits per period are computed from the game's own loop and the bench's measured average. The coin spellings (\`estimateCoins\`, \`perCallMaxCoins\`) are deprecated \u2014 write the credit names. Model output is data the game validates against its own expectation \u2014 never executed, and never authority over coin, credits, items, entitlements or rewards. Load \`$genex-llm-in-games\` before writing any of it.
|
|
1111
1111
|
${CONTRACT_END}
|
|
1112
1112
|
`;
|
|
1113
1113
|
var TOOLS_CONTRACT_HEAD = `# Genex Tools (always in effect in this folder)
|
|
@@ -1121,7 +1121,7 @@ var TOOLS_CONTRACT_OFFER = `5. **When the game is built and playable \u2014 neve
|
|
|
1121
1121
|
var TOOLS_CONTRACT_HOSTED_LAWS = `5. **This game is hosted on Genex now, and publishing is glue \u2014 never a rebuild.** The game ships exactly as the user built it: \`initEmbed()\` first in the boot code (the \`genex-threejs-embed-auth\` card has the call; the renderer draws before any \`await waitForPlayer()\`), a static build into \`dist/\` with relative asset paths, then \`npx genex preview\`. Never restructure, reformat or "improve" the game to ship it, never scaffold a new app around it, and never start a design document or a build plan for it \u2014 the user's own process is theirs. Load the \`genex-tool-publish\` card before the first preview; it owns the vocabulary, the links and the limits.
|
|
1122
1122
|
6. **Ship first, then report.** \`preview\` prints a preflight \u2014 phone memory, a missing volume slider, an asset nobody wired, a viewport line. It never blocks a deploy, so push the build as it is, hand over the link, THEN relay each preflight line to the user in one plain sentence with an offer to fix it, and fix one only on their yes. The preflight is a report for the user, not a to-do list for you.
|
|
1123
1123
|
7. **The link is the game's page, and there are two versions.** After every preview give the user \`<dashboard>/draft/<slug>\` \u2014 \`dashboardOrigins[0]\` and \`slug\` from \`.genex/project.json\` \u2014 never localhost, a file path or the bare play origin. \`preview\` updates the draft and never touches what players are on; the first release is \`npx genex publish\` (it lists the game); after that "publish it", "update it" and "yes" all mean \`npx genex promote\` \u2014 the exact draft build, no rebuild. Ask once per round of work, in one line, and keep working while you wait.`;
|
|
1124
|
-
var TOOLS_CONTRACT_LLM_OFFER = `A model running while people play is built into the Genex platform \u2014 the player pays, with Genex
|
|
1124
|
+
var TOOLS_CONTRACT_LLM_OFFER = `A model running while people play is built into the Genex platform \u2014 the player pays, with their Genex credits or their own Claude/ChatGPT subscription, and approves it on a Genex sheet; your game just calls \`generate()\`. Want it that way?`;
|
|
1125
1125
|
function toolsContractLlmLaw(n, hosted) {
|
|
1126
1126
|
const onYes = hosted ? "On a yes, load `$genex-tool-llm` and then `$genex-llm-in-games`, which owns the build." : "On a yes, load `$genex-tool-llm`, which owns the rest: it checks the lane with `npx genex llm models` first, then runs `npx genex init --convert` (a hosted game is a static browser build \u2014 a local server holding a key can never ship).";
|
|
1127
1127
|
return `${n}. **A model running while people PLAY is a platform feature \u2014 never something you wire onto the user's own meter.** When a request implies one (NPCs that talk or decide in their own words, content written from what the player types, a prompt box in the game, "let the player pick a model"), recognise it, say in ONE line: "${TOOLS_CONTRACT_LLM_OFFER}" \u2014 then ASK and wait. ${onYes} On a no, build the authored version instead. Either way, never ship a game that calls a model on a key or an account of the user's own: every visitor would spend their money with nobody approving it, and the credential is readable in the bundle. Player-funded generation is text and JSON only \u2014 3D, images, video and audio stay the asset lanes above, on the user's meter.`;
|
|
@@ -2671,8 +2671,11 @@ function tierFor(estVramMb) {
|
|
|
2671
2671
|
}
|
|
2672
2672
|
|
|
2673
2673
|
// src/lib/download-assets.ts
|
|
2674
|
+
import { createWriteStream } from "fs";
|
|
2674
2675
|
import fs12 from "fs/promises";
|
|
2675
2676
|
import path13 from "path";
|
|
2677
|
+
import { Readable } from "stream";
|
|
2678
|
+
import { pipeline } from "stream/promises";
|
|
2676
2679
|
var RUNG_ROLE = /@\d+$/;
|
|
2677
2680
|
function isPrimaryRole(role) {
|
|
2678
2681
|
return !RUNG_ROLE.test(role);
|
|
@@ -2704,21 +2707,37 @@ async function downloadAssets(files, opts) {
|
|
|
2704
2707
|
return result;
|
|
2705
2708
|
}
|
|
2706
2709
|
const multi = wanted.length > 1;
|
|
2707
|
-
|
|
2708
|
-
|
|
2710
|
+
const outcomes = await Promise.all(
|
|
2711
|
+
wanted.map(
|
|
2712
|
+
(file) => saveOne(file, path13.join(opts.outDir, localAssetName(opts.kind, opts.prompt, opts.id, file, multi)))
|
|
2713
|
+
)
|
|
2714
|
+
);
|
|
2715
|
+
for (const outcome2 of outcomes) {
|
|
2716
|
+
if ("failure" in outcome2) result.failures.push(outcome2.failure);
|
|
2717
|
+
else result.saved.push(outcome2);
|
|
2718
|
+
}
|
|
2719
|
+
return result;
|
|
2720
|
+
}
|
|
2721
|
+
async function saveOne(file, dest) {
|
|
2722
|
+
try {
|
|
2723
|
+
const res = await fetch(file.url);
|
|
2724
|
+
if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);
|
|
2725
|
+
let bytes = 0;
|
|
2726
|
+
const counted = Readable.fromWeb(res.body).on("data", (chunk) => {
|
|
2727
|
+
bytes += chunk.byteLength;
|
|
2728
|
+
});
|
|
2709
2729
|
try {
|
|
2710
|
-
|
|
2711
|
-
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
|
2712
|
-
const buf = Buffer.from(await res.arrayBuffer());
|
|
2713
|
-
await fs12.writeFile(dest, buf);
|
|
2714
|
-
result.saved.push({ role: file.role, path: dest, url: file.url, bytes: buf.byteLength });
|
|
2730
|
+
await pipeline(counted, createWriteStream(dest));
|
|
2715
2731
|
} catch (err) {
|
|
2716
|
-
|
|
2717
|
-
|
|
2718
|
-
);
|
|
2732
|
+
await fs12.rm(dest, { force: true });
|
|
2733
|
+
throw err;
|
|
2719
2734
|
}
|
|
2735
|
+
return { role: file.role, path: dest, url: file.url, bytes };
|
|
2736
|
+
} catch (err) {
|
|
2737
|
+
return {
|
|
2738
|
+
failure: `${file.role}: ${err instanceof Error ? err.message : String(err)} \u2014 the asset is still live at ${file.url}`
|
|
2739
|
+
};
|
|
2720
2740
|
}
|
|
2721
|
-
return result;
|
|
2722
2741
|
}
|
|
2723
2742
|
function formatBytes(bytes) {
|
|
2724
2743
|
if (bytes >= 1024 * 1024) return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
|
|
@@ -6245,6 +6264,16 @@ async function pushSource(cwd, ctx, log, branch = "main", expectStagingCommit, o
|
|
|
6245
6264
|
) === true;
|
|
6246
6265
|
}
|
|
6247
6266
|
var WORKSPACE_SOURCE_REF = "workspace";
|
|
6267
|
+
function gitLfsInstallCommand(platform) {
|
|
6268
|
+
if (platform === "darwin") {
|
|
6269
|
+
return {
|
|
6270
|
+
bin: "brew",
|
|
6271
|
+
args: ["install", "git-lfs"],
|
|
6272
|
+
env: { HOMEBREW_NO_AUTO_UPDATE: "1", HOMEBREW_NO_INSTALL_CLEANUP: "1" }
|
|
6273
|
+
};
|
|
6274
|
+
}
|
|
6275
|
+
return { bin: "sudo", args: ["-n", "apt-get", "install", "-y", "git-lfs"], env: {} };
|
|
6276
|
+
}
|
|
6248
6277
|
async function pushWorktree(cwd, pushUrl, managed, log, branch = "main", onCommit, skipSizeCheck = false, limits) {
|
|
6249
6278
|
const EMPTY_TREE = "4b825dc642cb6eb9a060e54bf8d69288fbee4904";
|
|
6250
6279
|
const failed = () => {
|
|
@@ -6271,8 +6300,8 @@ async function pushWorktree(cwd, pushUrl, managed, log, branch = "main", onCommi
|
|
|
6271
6300
|
let lfs = (await run("git", ["lfs", "version"], base)).code !== 0 ? false : true;
|
|
6272
6301
|
if (!lfs) {
|
|
6273
6302
|
log.step("Installing git-lfs (keeps large binary assets out of the source push)\u2026");
|
|
6274
|
-
const
|
|
6275
|
-
await run(bin, args, base);
|
|
6303
|
+
const install2 = gitLfsInstallCommand(process.platform);
|
|
6304
|
+
await run(install2.bin, install2.args, { ...base, ...install2.env });
|
|
6276
6305
|
lfs = (await run("git", ["lfs", "version"], base)).code === 0;
|
|
6277
6306
|
if (!lfs) {
|
|
6278
6307
|
log.warn(
|
|
@@ -23778,7 +23807,7 @@ async function runBlender(opts) {
|
|
|
23778
23807
|
return 1;
|
|
23779
23808
|
}
|
|
23780
23809
|
if (sub === "serve") {
|
|
23781
|
-
const { serveLocalBlender } = await import("./blender-serve-
|
|
23810
|
+
const { serveLocalBlender } = await import("./blender-serve-EH2MOW3U.js");
|
|
23782
23811
|
const port = Number(process.env.GENEX_BLENDER_PORT ?? 8088);
|
|
23783
23812
|
log.step(`Starting a local Blender service on port ${port}`);
|
|
23784
23813
|
log.plain(
|
|
@@ -23787,7 +23816,7 @@ async function runBlender(opts) {
|
|
|
23787
23816
|
return serveLocalBlender({ port, log });
|
|
23788
23817
|
}
|
|
23789
23818
|
if (sub === "mcp") {
|
|
23790
|
-
const { runBlenderMcp } = await import("./blender-mcp-
|
|
23819
|
+
const { runBlenderMcp } = await import("./blender-mcp-WSURPZET.js");
|
|
23791
23820
|
return runBlenderMcp();
|
|
23792
23821
|
}
|
|
23793
23822
|
if (sub === "seat") {
|
|
@@ -24295,7 +24324,12 @@ import fs35 from "fs/promises";
|
|
|
24295
24324
|
import path36 from "path";
|
|
24296
24325
|
import { randomUUID as randomUUID2 } from "crypto";
|
|
24297
24326
|
var SUBS3 = ["models", "bench", "price", "status", "cancel"];
|
|
24298
|
-
var LLM_BENCH_APPROVAL_REQUIRED = "STOP: `genex llm bench` runs the real model and spends YOUR OWN
|
|
24327
|
+
var LLM_BENCH_APPROVAL_REQUIRED = "STOP: `genex llm bench` runs the real model and spends YOUR OWN credits \u2014 your balance, not a player's, and a spend this build's asset allowance does not count. Re-run with --max-credits <n> --user-approved only after the person at the keyboard agreed to the number.";
|
|
24328
|
+
var LLM_MAX_COINS_DEPRECATED = "--max-coins is deprecated: it is read as --max-credits, the same number \u2014 a bench spends credits now.";
|
|
24329
|
+
function benchBalanceShortLine(spendable, maxCredits) {
|
|
24330
|
+
return `Your spendable credits (${spendable}) cannot cover one attempt at --max-credits ${maxCredits} \u2014 nothing was sent, nothing was spent. Add credits, or re-run with a lower --max-credits once the person at the keyboard agrees to it.`;
|
|
24331
|
+
}
|
|
24332
|
+
var LLM_BENCH_UNVERIFIED_LINE = "Credits pay only for a verified email, and this account's is not verified yet (credits_unverified) \u2014 nothing was sent, nothing was spent. Verify it, then re-run.";
|
|
24299
24333
|
var LLM_LANE_OFF_LINE = "The runtime LLM lane is off on this stand \u2014 in-game generate() answers 404 here, and nothing can be benchmarked.";
|
|
24300
24334
|
var LLM_TOOLS_CONVERT_HINT = [
|
|
24301
24335
|
"In-game model calls need a hosted game. This folder is a Genex Tools workspace, so nothing here",
|
|
@@ -24308,7 +24342,7 @@ function serverSlotCounts(body) {
|
|
|
24308
24342
|
return { held: body.slotsHeld, limit: body.slotLimit };
|
|
24309
24343
|
}
|
|
24310
24344
|
var PENDING_SLOT_RELEASE_MINUTES = 10;
|
|
24311
|
-
var LLM_STATUS_LEGEND = `"awaiting bill" = the call has stopped but the provider's bill is not final, so
|
|
24345
|
+
var LLM_STATUS_LEGEND = `"awaiting bill" = the call has stopped but the provider's bill is not final, so what it reserved stays held until the bill resolves; such a row stops holding a slot ${PENDING_SLOT_RELEASE_MINUTES} minutes after dispatch.`;
|
|
24312
24346
|
var LLM_STATUS_NO_COUNT_LINE = "This stand does not report how many slots are held across your projects, so no count is shown.";
|
|
24313
24347
|
var LLM_CANCEL_AWAITING_BILL = "this call already stopped; its bill is awaiting the provider";
|
|
24314
24348
|
var LLM_CANCEL_ALREADY_FINAL = "this call is already final";
|
|
@@ -24322,11 +24356,33 @@ var DEFAULT_SAMPLE_TIMEOUT_SEC = 180;
|
|
|
24322
24356
|
function parseProviderKey(value) {
|
|
24323
24357
|
return value === "ok" || value === "rejected" || value === "unknown" ? value : null;
|
|
24324
24358
|
}
|
|
24359
|
+
function bpsOrNull(value) {
|
|
24360
|
+
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : null;
|
|
24361
|
+
}
|
|
24362
|
+
function standingCreditGrantOrNull(value) {
|
|
24363
|
+
if (!value || typeof value !== "object") return null;
|
|
24364
|
+
const v = value;
|
|
24365
|
+
const whole = (x) => typeof x === "number" && Number.isSafeInteger(x) && x >= 0;
|
|
24366
|
+
if (!whole(v.minCredits) || !whole(v.maxCredits) || !whole(v.stepCredits) || !whole(v.defaultCredits)) return null;
|
|
24367
|
+
return {
|
|
24368
|
+
minCredits: v.minCredits,
|
|
24369
|
+
maxCredits: v.maxCredits,
|
|
24370
|
+
stepCredits: v.stepCredits,
|
|
24371
|
+
defaultCredits: v.defaultCredits,
|
|
24372
|
+
...whole(v.minUsdCents) ? { minUsdCents: v.minUsdCents } : {},
|
|
24373
|
+
...whole(v.maxUsdCents) ? { maxUsdCents: v.maxUsdCents } : {},
|
|
24374
|
+
...whole(v.stepUsdCents) ? { stepUsdCents: v.stepUsdCents } : {}
|
|
24375
|
+
};
|
|
24376
|
+
}
|
|
24325
24377
|
async function readRuntimeLane(apiUrl, token) {
|
|
24326
24378
|
const empty = (state) => ({
|
|
24327
24379
|
state,
|
|
24328
24380
|
models: null,
|
|
24329
24381
|
recommendedDeclaredHeadroomBps: null,
|
|
24382
|
+
currency: null,
|
|
24383
|
+
feeBps: null,
|
|
24384
|
+
usdCentsPerCredit: null,
|
|
24385
|
+
standingCreditGrant: null,
|
|
24330
24386
|
externalProviders: null,
|
|
24331
24387
|
total: null,
|
|
24332
24388
|
featuredCount: null,
|
|
@@ -24345,7 +24401,11 @@ async function readRuntimeLane(apiUrl, token) {
|
|
|
24345
24401
|
return {
|
|
24346
24402
|
state: "live",
|
|
24347
24403
|
models: body.models,
|
|
24348
|
-
recommendedDeclaredHeadroomBps:
|
|
24404
|
+
recommendedDeclaredHeadroomBps: bpsOrNull(body.recommendedDeclaredHeadroomBps),
|
|
24405
|
+
currency: body.currency === "credits" || body.currency === "coins" ? body.currency : null,
|
|
24406
|
+
feeBps: bpsOrNull(body.feeBps),
|
|
24407
|
+
usdCentsPerCredit: typeof body.usdCentsPerCredit === "number" && Number.isSafeInteger(body.usdCentsPerCredit) && body.usdCentsPerCredit >= 1 ? body.usdCentsPerCredit : null,
|
|
24408
|
+
standingCreditGrant: standingCreditGrantOrNull(body.standingCreditGrant),
|
|
24349
24409
|
externalProviders: Array.isArray(body.externalProviders) ? body.externalProviders : null,
|
|
24350
24410
|
total: typeof body.total === "number" && Number.isFinite(body.total) ? body.total : null,
|
|
24351
24411
|
featuredCount: typeof body.featuredCount === "number" && Number.isFinite(body.featuredCount) ? body.featuredCount : null,
|
|
@@ -24361,18 +24421,45 @@ function percentile2(values, p) {
|
|
|
24361
24421
|
const rank2 = Math.ceil(p / 100 * sorted.length);
|
|
24362
24422
|
return sorted[Math.min(sorted.length - 1, Math.max(0, rank2 - 1))];
|
|
24363
24423
|
}
|
|
24364
|
-
function
|
|
24365
|
-
|
|
24424
|
+
function percentilePicos(values, p) {
|
|
24425
|
+
if (values.length === 0) return null;
|
|
24426
|
+
const sorted = [...values].sort((a, b) => a < b ? -1 : a > b ? 1 : 0);
|
|
24427
|
+
const rank2 = Math.ceil(p / 100 * sorted.length);
|
|
24428
|
+
return sorted[Math.min(sorted.length - 1, Math.max(0, rank2 - 1))];
|
|
24366
24429
|
}
|
|
24367
|
-
|
|
24368
|
-
|
|
24430
|
+
var PICOS_PER_CENT = 10000000000n;
|
|
24431
|
+
function ceilingCreditsFor(costPicos, terms) {
|
|
24432
|
+
if (costPicos <= 0n) return 1;
|
|
24433
|
+
const num = costPicos * BigInt(1e4 + terms.feeBps) * BigInt(1e4 + terms.headroomBps);
|
|
24434
|
+
const den = 100000000n * BigInt(terms.usdCentsPerCredit) * PICOS_PER_CENT;
|
|
24435
|
+
return Math.max(1, Number((num + den - 1n) / den));
|
|
24436
|
+
}
|
|
24437
|
+
function averageCreditsPerCall(costsPicos, terms) {
|
|
24438
|
+
if (costsPicos.length === 0) return null;
|
|
24439
|
+
const sum = costsPicos.reduce((a, b) => a + b, 0n);
|
|
24440
|
+
return Number(sum * BigInt(1e4 + terms.feeBps)) / Number(BigInt(costsPicos.length) * 10000n * BigInt(terms.usdCentsPerCredit) * PICOS_PER_CENT);
|
|
24441
|
+
}
|
|
24442
|
+
function perCallEstimateCreditsFor(costsPicos, terms) {
|
|
24443
|
+
if (costsPicos.length === 0) return null;
|
|
24444
|
+
const sum = costsPicos.reduce((a, b) => a + b, 0n);
|
|
24445
|
+
const num = sum * BigInt(1e4 + terms.feeBps);
|
|
24446
|
+
const den = BigInt(costsPicos.length) * 10000n * BigInt(terms.usdCentsPerCredit) * PICOS_PER_CENT;
|
|
24447
|
+
return Math.max(1, Number((num + den - 1n) / den));
|
|
24448
|
+
}
|
|
24449
|
+
function formatCredits(value) {
|
|
24450
|
+
if (!Number.isFinite(value)) return "\u2014";
|
|
24451
|
+
if (value >= 100) return String(Math.round(value));
|
|
24452
|
+
return String(Number(value.toPrecision(value < 1 ? 2 : 3)));
|
|
24453
|
+
}
|
|
24454
|
+
function usdFromPicos(picos) {
|
|
24455
|
+
if (picos === null || picos === void 0) return "unknown";
|
|
24456
|
+
const value = typeof picos === "bigint" ? picos : /^\d+$/.test(picos) ? BigInt(picos) : null;
|
|
24457
|
+
if (value === null) return "unknown";
|
|
24458
|
+
return `$${(Number(value) / 1e12).toFixed(6)}`;
|
|
24369
24459
|
}
|
|
24370
24460
|
function usdPerMillion(value) {
|
|
24371
24461
|
return `$${value.toFixed(value < 1 ? 4 : 2)}`;
|
|
24372
24462
|
}
|
|
24373
|
-
function usd(value) {
|
|
24374
|
-
return value === null ? "unknown" : `$${value.toFixed(6)}`;
|
|
24375
|
-
}
|
|
24376
24463
|
async function runLlm(opts = {}) {
|
|
24377
24464
|
const log = createLogger({ quiet: opts.quiet || opts.json });
|
|
24378
24465
|
const cwd = opts.cwd ?? process.cwd();
|
|
@@ -24398,16 +24485,23 @@ async function runLlm(opts = {}) {
|
|
|
24398
24485
|
return;
|
|
24399
24486
|
}
|
|
24400
24487
|
if (sub === "bench") {
|
|
24401
|
-
if (opts.maxCoins
|
|
24488
|
+
if (opts.maxCoins !== void 0) process.stderr.write(`${c.yellow("!")} ${LLM_MAX_COINS_DEPRECATED}
|
|
24489
|
+
`);
|
|
24490
|
+
if (opts.maxCoins !== void 0 && opts.maxCredits !== void 0 && opts.maxCoins !== opts.maxCredits) {
|
|
24491
|
+
fail5("llm bench", `Pass --max-credits alone \u2014 --max-coins is its deprecated name, and the two disagree (${opts.maxCredits} vs ${opts.maxCoins}).`);
|
|
24492
|
+
return;
|
|
24493
|
+
}
|
|
24494
|
+
const approvedCredits = opts.maxCredits ?? opts.maxCoins;
|
|
24495
|
+
if (approvedCredits === void 0 || opts.userApproved !== true) {
|
|
24402
24496
|
fail5("llm bench", LLM_BENCH_APPROVAL_REQUIRED);
|
|
24403
24497
|
return;
|
|
24404
24498
|
}
|
|
24405
|
-
if (!Number.isInteger(
|
|
24406
|
-
fail5("llm bench", `--max-
|
|
24499
|
+
if (!Number.isInteger(approvedCredits) || approvedCredits < 1) {
|
|
24500
|
+
fail5("llm bench", `--max-credits takes a whole number of credits, 1 or more (got ${String(approvedCredits)}).`);
|
|
24407
24501
|
return;
|
|
24408
24502
|
}
|
|
24409
24503
|
if (!opts.benchPrompt?.trim()) {
|
|
24410
|
-
fail5("llm bench", '`genex llm bench` needs the prompt your game would send, e.g. genex llm bench "<prompt>" --max-
|
|
24504
|
+
fail5("llm bench", '`genex llm bench` needs the prompt your game would send, e.g. genex llm bench "<prompt>" --max-credits <n> --user-approved.');
|
|
24411
24505
|
return;
|
|
24412
24506
|
}
|
|
24413
24507
|
if (opts.jsonOutput && opts.textOutput) {
|
|
@@ -24499,6 +24593,10 @@ async function reportModels(apiUrl, token, opts, log, toolsOnly = false) {
|
|
|
24499
24593
|
models: null,
|
|
24500
24594
|
externalProviders: null,
|
|
24501
24595
|
recommendedDeclaredHeadroomBps: null,
|
|
24596
|
+
currency: null,
|
|
24597
|
+
feeBps: null,
|
|
24598
|
+
usdCentsPerCredit: null,
|
|
24599
|
+
standingCreditGrant: null,
|
|
24502
24600
|
total: null,
|
|
24503
24601
|
featuredCount: null,
|
|
24504
24602
|
// The stand was never asked, so the verdict is unknown — but the key is
|
|
@@ -24524,6 +24622,12 @@ async function reportModels(apiUrl, token, opts, log, toolsOnly = false) {
|
|
|
24524
24622
|
models: lane.models,
|
|
24525
24623
|
externalProviders: lane.externalProviders,
|
|
24526
24624
|
recommendedDeclaredHeadroomBps: lane.recommendedDeclaredHeadroomBps,
|
|
24625
|
+
// The credit terms a call is billed on, the server's — null on a stand
|
|
24626
|
+
// that predates credit billing, never a number the CLI filled in.
|
|
24627
|
+
currency: lane.currency,
|
|
24628
|
+
feeBps: lane.feeBps,
|
|
24629
|
+
usdCentsPerCredit: lane.usdCentsPerCredit,
|
|
24630
|
+
standingCreditGrant: lane.standingCreditGrant,
|
|
24527
24631
|
total: lane.total ?? lane.models?.length ?? null,
|
|
24528
24632
|
featuredCount: lane.featuredCount ?? (lane.models ? lane.models.filter((m) => m.featured).length : null),
|
|
24529
24633
|
// `null` when the lane is not live or the server predates the probe.
|
|
@@ -24570,8 +24674,9 @@ async function reportModels(apiUrl, token, opts, log, toolsOnly = false) {
|
|
|
24570
24674
|
);
|
|
24571
24675
|
for (const m of shown) {
|
|
24572
24676
|
const plan = m.personalPlan ? ` \xB7 personal plan: ${m.personalPlan}` : "";
|
|
24677
|
+
const kind = m.kind === "typesafe" ? " \xB7 judge (classifier): judge() under a budget, not generate() or bench" : "";
|
|
24573
24678
|
log.plain(
|
|
24574
|
-
` ${c.cyan(m.id)} ${m.label} \u2014 ${usdPerMillion(m.inputUsdPerMillion)} in / ${usdPerMillion(m.outputUsdPerMillion)} out per million tokens${plan}`
|
|
24679
|
+
` ${c.cyan(m.id)} ${m.label} \u2014 ${usdPerMillion(m.inputUsdPerMillion)} in / ${usdPerMillion(m.outputUsdPerMillion)} out per million tokens${plan}${kind}`
|
|
24575
24680
|
);
|
|
24576
24681
|
}
|
|
24577
24682
|
if (hiddenCount > 0) {
|
|
@@ -24587,74 +24692,104 @@ async function reportModels(apiUrl, token, opts, log, toolsOnly = false) {
|
|
|
24587
24692
|
}
|
|
24588
24693
|
log.plain("");
|
|
24589
24694
|
if (lane.recommendedDeclaredHeadroomBps === null) {
|
|
24590
|
-
log.dim(" This stand serves no recommended headroom, so no
|
|
24695
|
+
log.dim(" This stand serves no recommended headroom, so no ceiling can be recommended from a bench.");
|
|
24591
24696
|
} else {
|
|
24592
|
-
log.dim(` Recommended headroom over benchmarked
|
|
24697
|
+
log.dim(` Recommended headroom over a benchmarked call's cost on this stand: ${lane.recommendedDeclaredHeadroomBps} bps.`);
|
|
24698
|
+
}
|
|
24699
|
+
if (lane.feeBps === null) {
|
|
24700
|
+
log.dim(" This stand serves no platform fee \u2014 it predates credit billing, so no ceiling can be recommended from a bench.");
|
|
24701
|
+
} else {
|
|
24702
|
+
log.dim(` In-game calls are billed in the player's credits, as used: the provider's cost plus this stand's platform fee (${lane.feeBps} bps).`);
|
|
24593
24703
|
}
|
|
24594
24704
|
if (toolsOnly) {
|
|
24595
24705
|
printConvertHint();
|
|
24596
24706
|
return;
|
|
24597
24707
|
}
|
|
24598
|
-
log.dim(" Those per-million rates are the PROVIDER's; what
|
|
24599
|
-
log.dim(`
|
|
24708
|
+
log.dim(" Those per-million rates are the PROVIDER's list; what one call of YOUR prompt really costs,");
|
|
24709
|
+
log.dim(` and the per-call ceiling to declare, only a real call measures: ${c.cyan('genex llm bench "<prompt>" --max-credits <n> --user-approved')}`);
|
|
24600
24710
|
}
|
|
24601
24711
|
function wholeOrNull(value) {
|
|
24602
24712
|
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : null;
|
|
24603
24713
|
}
|
|
24714
|
+
function costPicosOf(usage) {
|
|
24715
|
+
if (typeof usage?.costUsdPicos === "string" && /^\d+$/.test(usage.costUsdPicos)) return usage.costUsdPicos;
|
|
24716
|
+
if (typeof usage?.costUsd === "number" && Number.isFinite(usage.costUsd) && usage.costUsd >= 0) {
|
|
24717
|
+
return String(BigInt(Math.round(usage.costUsd * 1e12)));
|
|
24718
|
+
}
|
|
24719
|
+
return null;
|
|
24720
|
+
}
|
|
24604
24721
|
function sampleFrom(id, settled) {
|
|
24605
24722
|
const usage = settled?.usage;
|
|
24723
|
+
const picos = costPicosOf(usage);
|
|
24606
24724
|
return {
|
|
24607
24725
|
id,
|
|
24608
24726
|
status: settled?.status ?? null,
|
|
24609
24727
|
billingStatus: settled?.billingStatus ?? null,
|
|
24610
|
-
|
|
24611
|
-
costUsd: typeof usage?.costUsd === "number" ? usage.costUsd : null,
|
|
24728
|
+
chargedCredits: wholeOrNull(settled?.chargedCredits),
|
|
24729
|
+
costUsd: typeof usage?.costUsd === "number" ? usage.costUsd : picos !== null ? Number(picos) / 1e12 : null,
|
|
24730
|
+
costUsdPicos: picos,
|
|
24612
24731
|
error: settled?.error ?? (settled ? null : "timed_out"),
|
|
24613
24732
|
providerMessage: typeof settled?.providerMessage === "string" && settled.providerMessage.trim() ? settled.providerMessage.trim() : null,
|
|
24614
24733
|
inputTokens: wholeOrNull(usage?.inputTokens),
|
|
24615
24734
|
outputTokens: wholeOrNull(usage?.outputTokens),
|
|
24616
24735
|
maxOutputTokens: wholeOrNull(usage?.maxOutputTokens),
|
|
24617
|
-
|
|
24736
|
+
fitCredits: typeof usage?.fitCredits === "number" && Number.isSafeInteger(usage.fitCredits) && usage.fitCredits >= 1 ? usage.fitCredits : null,
|
|
24618
24737
|
fitExceedsCap: typeof usage?.fitExceedsCap === "boolean" ? usage.fitExceedsCap : null,
|
|
24619
24738
|
truncated: typeof usage?.truncated === "boolean" ? usage.truncated : settled?.error === "provider_token_limit" ? true : null
|
|
24620
24739
|
};
|
|
24621
24740
|
}
|
|
24622
24741
|
var PROVIDER_TOKEN_LIMIT = "provider_token_limit";
|
|
24623
|
-
function benchRecommendation(results,
|
|
24624
|
-
const settled = results.filter(
|
|
24625
|
-
|
|
24626
|
-
|
|
24627
|
-
const
|
|
24628
|
-
const
|
|
24742
|
+
function benchRecommendation(results, terms) {
|
|
24743
|
+
const settled = results.filter(
|
|
24744
|
+
(r) => r.status === "succeeded" && r.billingStatus === "final" && r.chargedCredits !== null && r.costUsdPicos !== null
|
|
24745
|
+
);
|
|
24746
|
+
const costs = settled.map((r) => BigInt(r.costUsdPicos));
|
|
24747
|
+
const charged = settled.map((r) => r.chargedCredits);
|
|
24748
|
+
const fits = settled.map((r) => r.fitCredits).filter((v) => typeof v === "number");
|
|
24749
|
+
const costP95 = percentilePicos(costs, 95);
|
|
24750
|
+
const costMax = costs.length ? costs.reduce((a, b) => b > a ? b : a) : null;
|
|
24629
24751
|
const fitP95 = percentile2(fits, 95);
|
|
24630
24752
|
const fitMax = fits.length ? Math.max(...fits) : null;
|
|
24631
24753
|
const exceedsStandCap = results.some((r) => r.fitExceedsCap === true);
|
|
24632
24754
|
const cutOffSamples = results.filter((r) => r.error === PROVIDER_TOKEN_LIMIT && r.fitExceedsCap !== true).length;
|
|
24633
24755
|
const noRecommendationReason = exceedsStandCap ? "answer_exceeds_stand_cap" : cutOffSamples > 0 ? "answer_cut_off" : null;
|
|
24634
|
-
const
|
|
24635
|
-
const
|
|
24636
|
-
const
|
|
24756
|
+
const { headroomBps, feeBps, usdCentsPerCredit } = terms;
|
|
24757
|
+
const missingTerms = headroomBps === null ? "headroom" : feeBps === null || usdCentsPerCredit === null ? "credit_terms" : null;
|
|
24758
|
+
const full = headroomBps !== null && feeBps !== null && usdCentsPerCredit !== null ? { headroomBps, feeBps, usdCentsPerCredit } : null;
|
|
24759
|
+
const billing = feeBps !== null && usdCentsPerCredit !== null ? { feeBps, usdCentsPerCredit } : null;
|
|
24760
|
+
const costBased = full !== null && costP95 !== null ? ceilingCreditsFor(costP95, full) : null;
|
|
24761
|
+
const recommended = costBased === null || noRecommendationReason !== null ? null : Math.max(costBased, fitP95 ?? 0);
|
|
24762
|
+
const ceiling = recommended === null || costMax === null || full === null ? null : Math.max(ceilingCreditsFor(costMax, full), recommended, fitMax ?? 0);
|
|
24763
|
+
const average = billing !== null ? averageCreditsPerCall(costs, billing) : null;
|
|
24764
|
+
const estimate = ceiling === null || billing === null ? null : Math.min(ceiling, perCallEstimateCreditsFor(costs, billing) ?? 1);
|
|
24637
24765
|
return {
|
|
24766
|
+
costs,
|
|
24638
24767
|
charged,
|
|
24639
|
-
|
|
24640
|
-
|
|
24641
|
-
|
|
24768
|
+
costP50: percentilePicos(costs, 50),
|
|
24769
|
+
costP95,
|
|
24770
|
+
costMax,
|
|
24771
|
+
chargedP50: percentile2(charged, 50),
|
|
24772
|
+
chargedP95: percentile2(charged, 95),
|
|
24773
|
+
chargedMax: charged.length ? Math.max(...charged) : null,
|
|
24642
24774
|
fits,
|
|
24643
24775
|
fitP95,
|
|
24644
24776
|
fitMax,
|
|
24645
|
-
|
|
24646
|
-
|
|
24647
|
-
|
|
24648
|
-
|
|
24777
|
+
costBasedMaxCredits: costBased,
|
|
24778
|
+
recommendedMaxCredits: recommended,
|
|
24779
|
+
recommendedPerCallMaxCredits: ceiling,
|
|
24780
|
+
averageCreditsPerCall: average,
|
|
24781
|
+
recommendedPerCallEstimateCredits: estimate,
|
|
24782
|
+
lengthRaisedCeiling: recommended !== null && costBased !== null && recommended > costBased,
|
|
24649
24783
|
cutOffSamples,
|
|
24650
24784
|
exceedsStandCap,
|
|
24651
|
-
noRecommendationReason
|
|
24785
|
+
noRecommendationReason,
|
|
24786
|
+
missingTerms
|
|
24652
24787
|
};
|
|
24653
24788
|
}
|
|
24654
|
-
var LLM_LENGTH_RAISED_LINE = "The
|
|
24655
|
-
var LLM_EXCEEDS_STAND_CAP_LINE = "No
|
|
24656
|
-
function benchCutOffLine(cutOff,
|
|
24657
|
-
return `No
|
|
24789
|
+
var LLM_LENGTH_RAISED_LINE = "The ceiling also decides how long the answer may be: at the cost-based ceiling the answer would be cut off, so declare this one.";
|
|
24790
|
+
var LLM_EXCEEDS_STAND_CAP_LINE = "No ceiling recommended: this answer is longer than one call on this stand may produce. Ask for a shorter answer \u2014 fewer fields, shorter strings, a length the prompt states \u2014 and benchmark again.";
|
|
24791
|
+
function benchCutOffLine(cutOff, maxCredits) {
|
|
24792
|
+
return `No ceiling recommended: ${cutOff} sample${cutOff === 1 ? " was" : "s were"} cut off at this ceiling (--max-credits ${maxCredits}) before the answer was finished, so its real length is unknown. Re-run with a higher --max-credits.`;
|
|
24658
24793
|
}
|
|
24659
24794
|
var ACTIVE_STATUSES = /* @__PURE__ */ new Set(["requires_confirmation", "queued", "dispatching", "awaiting_external"]);
|
|
24660
24795
|
function rowState(row) {
|
|
@@ -24665,7 +24800,22 @@ function rowState(row) {
|
|
|
24665
24800
|
function rowHoldsSlot(row) {
|
|
24666
24801
|
if (typeof row.slotHeld === "boolean") return row.slotHeld;
|
|
24667
24802
|
const state = rowState(row);
|
|
24668
|
-
return state === "active" || state === "awaiting bill" && (row.reservedCoins ?? 0) > 0;
|
|
24803
|
+
return state === "active" || state === "awaiting bill" && ((row.reservedCoins ?? 0) > 0 || (row.reservedCredits ?? 0) > 0);
|
|
24804
|
+
}
|
|
24805
|
+
function rowMoney(row) {
|
|
24806
|
+
const credits = row.currency === "credits" || row.currency === void 0 && typeof row.reservedCredits === "number" && typeof row.reservedCoins !== "number";
|
|
24807
|
+
if (credits) {
|
|
24808
|
+
return {
|
|
24809
|
+
unit: "credits",
|
|
24810
|
+
reserved: typeof row.reservedCredits === "number" ? row.reservedCredits : null,
|
|
24811
|
+
charged: typeof row.chargedCredits === "number" ? row.chargedCredits : null
|
|
24812
|
+
};
|
|
24813
|
+
}
|
|
24814
|
+
return {
|
|
24815
|
+
unit: "coin",
|
|
24816
|
+
reserved: typeof row.reservedCoins === "number" ? row.reservedCoins : null,
|
|
24817
|
+
charged: typeof row.chargedCoins === "number" ? row.chargedCoins : null
|
|
24818
|
+
};
|
|
24669
24819
|
}
|
|
24670
24820
|
function providerRefusalStatus(error) {
|
|
24671
24821
|
const m = /^provider_http_(\d{3})$/.exec(error ?? "");
|
|
@@ -24684,7 +24834,7 @@ function formatAge(iso, now = Date.now()) {
|
|
|
24684
24834
|
}
|
|
24685
24835
|
async function runBench(args) {
|
|
24686
24836
|
const { apiUrl, token, projectId, cwd, opts, log } = args;
|
|
24687
|
-
const
|
|
24837
|
+
const maxCredits = opts.maxCredits ?? opts.maxCoins;
|
|
24688
24838
|
const samples = opts.samples ?? DEFAULT_BENCH_SAMPLES;
|
|
24689
24839
|
const prompt = opts.benchPrompt.trim();
|
|
24690
24840
|
const auth = { Authorization: `Bearer ${token}`, "Content-Type": "application/json" };
|
|
@@ -24706,7 +24856,7 @@ async function runBench(args) {
|
|
|
24706
24856
|
const lane = await readRuntimeLane(apiUrl, token);
|
|
24707
24857
|
if (lane.state !== "live" || !lane.models || lane.models.length === 0) {
|
|
24708
24858
|
if (lane.state === "off") {
|
|
24709
|
-
if (opts.json) writeJsonLine({ ...emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples,
|
|
24859
|
+
if (opts.json) writeJsonLine({ ...emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples, maxCredits, opts), status: "off", error: null });
|
|
24710
24860
|
else {
|
|
24711
24861
|
log.plain(LLM_LANE_OFF_LINE);
|
|
24712
24862
|
log.dim(" Nothing was spent.");
|
|
@@ -24733,17 +24883,27 @@ async function runBench(args) {
|
|
|
24733
24883
|
);
|
|
24734
24884
|
return;
|
|
24735
24885
|
}
|
|
24736
|
-
const
|
|
24886
|
+
const walletBefore = await fetchCreditsSnapshot(apiUrl, token);
|
|
24887
|
+
if (walletBefore?.emailVerified === false) {
|
|
24888
|
+
benchFailed(opts, log, LLM_BENCH_UNVERIFIED_LINE);
|
|
24889
|
+
return;
|
|
24890
|
+
}
|
|
24891
|
+
if (walletBefore !== null && walletBefore.spendable < maxCredits) {
|
|
24892
|
+
benchFailed(opts, log, benchBalanceShortLine(walletBefore.spendable, maxCredits));
|
|
24893
|
+
return;
|
|
24894
|
+
}
|
|
24895
|
+
const balanceBefore = walletBefore?.spendable ?? null;
|
|
24737
24896
|
if (!opts.json) {
|
|
24738
24897
|
log.plain(c.bold("genex llm bench"));
|
|
24739
24898
|
log.dim(` ${apiUrl}`);
|
|
24740
24899
|
log.plain("");
|
|
24741
24900
|
log.plain(` Model ${c.cyan(modelId)}`);
|
|
24742
24901
|
log.plain(` Samples ${samples} real attempt${samples === 1 ? "" : "s"}, ${outputFormat} output`);
|
|
24743
|
-
log.plain(` Approved ${
|
|
24902
|
+
log.plain(` Approved ${maxCredits} credit${maxCredits === 1 ? "" : "s"} per attempt \u2014 at worst ${maxCredits * samples} credits for this run, from your own balance`);
|
|
24744
24903
|
log.plain(
|
|
24745
|
-
` Balance ${balanceBefore === null ? "couldn't be read" : `${balanceBefore}
|
|
24904
|
+
` Balance ${balanceBefore === null ? "couldn't be read" : `${balanceBefore} credits spendable`}`
|
|
24746
24905
|
);
|
|
24906
|
+
log.dim(" Billed as used: each attempt is charged its real cost plus the platform fee, rounded up to a whole credit.");
|
|
24747
24907
|
log.plain("");
|
|
24748
24908
|
}
|
|
24749
24909
|
const base = `${apiUrl}/api/runtime/development/projects/${encodeURIComponent(projectId)}/generations`;
|
|
@@ -24769,7 +24929,7 @@ async function runBench(args) {
|
|
|
24769
24929
|
const res = await call2(base, {
|
|
24770
24930
|
method: "POST",
|
|
24771
24931
|
body: JSON.stringify({
|
|
24772
|
-
|
|
24932
|
+
maxCredits,
|
|
24773
24933
|
request: {
|
|
24774
24934
|
idempotencyKey: `bench-${randomUUID2()}`,
|
|
24775
24935
|
modelId,
|
|
@@ -24799,13 +24959,14 @@ async function runBench(args) {
|
|
|
24799
24959
|
if (refusedAt !== null) providerRefused++;
|
|
24800
24960
|
if (!opts.json) {
|
|
24801
24961
|
const length = row.outputTokens !== null ? ` \xB7 ${row.outputTokens}${row.maxOutputTokens !== null ? ` of ${row.maxOutputTokens}` : ""} tokens out` : "";
|
|
24962
|
+
const charged = row.chargedCredits === null ? "\u2014" : String(row.chargedCredits);
|
|
24802
24963
|
log.plain(
|
|
24803
|
-
` ${row.status === "succeeded" ? c.green("\u2713") : c.yellow("!")} sample ${i + 1} ${
|
|
24964
|
+
` ${row.status === "succeeded" ? c.green("\u2713") : c.yellow("!")} sample ${i + 1} ${charged.padStart(4)} credit${row.chargedCredits === 1 ? "" : "s"} charged \xB7 cost ${usdFromPicos(row.costUsdPicos)}${length}${row.error ? ` \xB7 ${row.error}` : ""}`
|
|
24804
24965
|
);
|
|
24805
24966
|
if (refusedAt !== null) printProviderRefusal(log, refusedAt, row.providerMessage, row.status === "unknown");
|
|
24806
24967
|
if (row.error === PROVIDER_TOKEN_LIMIT) {
|
|
24807
24968
|
log.dim(
|
|
24808
|
-
row.fitExceedsCap === true ? " The answer was cut off at this stand's own output limit \u2014 no
|
|
24969
|
+
row.fitExceedsCap === true ? " The answer was cut off at this stand's own output limit \u2014 no ceiling makes room for it; it is not a sample." : ` The answer was cut off at this ceiling (--max-credits ${maxCredits}) before it was finished \u2014 it was charged, and it is not a sample.`
|
|
24809
24970
|
);
|
|
24810
24971
|
}
|
|
24811
24972
|
}
|
|
@@ -24813,16 +24974,19 @@ async function runBench(args) {
|
|
|
24813
24974
|
} finally {
|
|
24814
24975
|
process.removeListener("SIGINT", onSigint);
|
|
24815
24976
|
}
|
|
24816
|
-
const
|
|
24817
|
-
|
|
24818
|
-
|
|
24819
|
-
|
|
24820
|
-
|
|
24821
|
-
const
|
|
24977
|
+
const terms = {
|
|
24978
|
+
headroomBps: lane.recommendedDeclaredHeadroomBps,
|
|
24979
|
+
feeBps: lane.feeBps,
|
|
24980
|
+
usdCentsPerCredit: lane.usdCentsPerCredit
|
|
24981
|
+
};
|
|
24982
|
+
const rec = benchRecommendation(results, terms);
|
|
24983
|
+
const settledCount = rec.charged.length;
|
|
24822
24984
|
const lengthBlocked = rec.noRecommendationReason !== null;
|
|
24823
|
-
const balanceAfter = await
|
|
24985
|
+
const balanceAfter = (await fetchCreditsSnapshot(apiUrl, token))?.spendable ?? null;
|
|
24986
|
+
const picosOrNull = (v) => v === null ? null : String(v);
|
|
24824
24987
|
const record = {
|
|
24825
|
-
v:
|
|
24988
|
+
v: 2,
|
|
24989
|
+
currency: "credits",
|
|
24826
24990
|
ranAt: (/* @__PURE__ */ new Date()).toISOString(),
|
|
24827
24991
|
apiUrl,
|
|
24828
24992
|
projectId,
|
|
@@ -24831,28 +24995,34 @@ async function runBench(args) {
|
|
|
24831
24995
|
outputFormat,
|
|
24832
24996
|
samples,
|
|
24833
24997
|
completed: settledCount,
|
|
24834
|
-
|
|
24835
|
-
|
|
24836
|
-
|
|
24837
|
-
max,
|
|
24838
|
-
|
|
24839
|
-
|
|
24840
|
-
recommendedPerCallMaxCoins: ceiling,
|
|
24841
|
-
maxCoinsPerSample: maxCoins,
|
|
24842
|
-
fitCoins: rec.fits,
|
|
24998
|
+
maxCreditsPerSample: maxCredits,
|
|
24999
|
+
costUsdPicos: rec.costs.map(String),
|
|
25000
|
+
chargedCredits: rec.charged,
|
|
25001
|
+
cost: { p50: picosOrNull(rec.costP50), p95: picosOrNull(rec.costP95), max: picosOrNull(rec.costMax) },
|
|
25002
|
+
charged: { p50: rec.chargedP50, p95: rec.chargedP95, max: rec.chargedMax },
|
|
25003
|
+
fitCredits: rec.fits,
|
|
24843
25004
|
fitP95: rec.fitP95,
|
|
24844
|
-
|
|
24845
|
-
|
|
25005
|
+
fitMax: rec.fitMax,
|
|
25006
|
+
recommendedDeclaredHeadroomBps: terms.headroomBps,
|
|
25007
|
+
feeBps: terms.feeBps,
|
|
25008
|
+
usdCentsPerCredit: terms.usdCentsPerCredit,
|
|
25009
|
+
costBasedMaxCredits: rec.costBasedMaxCredits,
|
|
25010
|
+
recommendedMaxCredits: rec.recommendedMaxCredits,
|
|
25011
|
+
recommendedPerCallMaxCredits: rec.recommendedPerCallMaxCredits,
|
|
25012
|
+
averageCreditsPerCall: rec.averageCreditsPerCall,
|
|
25013
|
+
recommendedPerCallEstimateCredits: rec.recommendedPerCallEstimateCredits,
|
|
25014
|
+
lengthRaisedCeiling: rec.lengthRaisedCeiling,
|
|
24846
25015
|
cutOffSamples: rec.cutOffSamples,
|
|
24847
25016
|
noRecommendationReason: rec.noRecommendationReason
|
|
24848
25017
|
};
|
|
24849
25018
|
const savedTo = await saveBench(cwd, record);
|
|
24850
25019
|
if (opts.json) {
|
|
24851
|
-
const ok =
|
|
25020
|
+
const ok = settledCount > 0 && !lengthBlocked;
|
|
24852
25021
|
writeJsonLine({
|
|
24853
25022
|
command: "llm bench",
|
|
24854
25023
|
status: ok ? "ok" : "failed",
|
|
24855
25024
|
error: ok ? null : rec.noRecommendationReason ?? "no_sample_settled",
|
|
25025
|
+
currency: "credits",
|
|
24856
25026
|
apiUrl,
|
|
24857
25027
|
projectId,
|
|
24858
25028
|
modelId,
|
|
@@ -24861,35 +25031,54 @@ async function runBench(args) {
|
|
|
24861
25031
|
schemaPath: opts.schemaPath ?? null,
|
|
24862
25032
|
samples,
|
|
24863
25033
|
completed: settledCount,
|
|
24864
|
-
|
|
25034
|
+
maxCreditsPerSample: maxCredits,
|
|
25035
|
+
// Spendable credits before and after the run; null when unreadable.
|
|
24865
25036
|
balanceBefore,
|
|
24866
25037
|
balanceAfter,
|
|
24867
25038
|
results,
|
|
24868
25039
|
// The create-time refusal that ended the run, or null. Beside the
|
|
24869
25040
|
// samples, never among them: no attempt existed and nothing was spent.
|
|
24870
25041
|
refusal: refusal2,
|
|
24871
|
-
|
|
24872
|
-
//
|
|
25042
|
+
// Over the settled samples: the real provider cost (pico-dollar strings
|
|
25043
|
+
// and USD) and the whole credits charged.
|
|
25044
|
+
costUsdPicos: record.cost,
|
|
25045
|
+
costUsd: {
|
|
25046
|
+
p50: rec.costP50 === null ? null : Number(rec.costP50) / 1e12,
|
|
25047
|
+
p95: rec.costP95 === null ? null : Number(rec.costP95) / 1e12,
|
|
25048
|
+
max: rec.costMax === null ? null : Number(rec.costMax) / 1e12
|
|
25049
|
+
},
|
|
25050
|
+
chargedCredits: record.charged,
|
|
25051
|
+
// The answer's LENGTH, priced by the server per sample (`fitCredits`,
|
|
24873
25052
|
// headroom already on the tokens), over the settled samples.
|
|
24874
|
-
|
|
24875
|
-
|
|
24876
|
-
|
|
25053
|
+
fitCredits: { p95: rec.fitP95, max: rec.fitMax },
|
|
25054
|
+
recommendedDeclaredHeadroomBps: terms.headroomBps,
|
|
25055
|
+
feeBps: terms.feeBps,
|
|
25056
|
+
usdCentsPerCredit: terms.usdCentsPerCredit,
|
|
25057
|
+
missingTerms: rec.missingTerms,
|
|
25058
|
+
costBasedMaxCredits: rec.costBasedMaxCredits,
|
|
25059
|
+
lengthRaisedCeiling: rec.lengthRaisedCeiling,
|
|
24877
25060
|
cutOffSamples: rec.cutOffSamples,
|
|
24878
25061
|
exceedsStandCap: rec.exceedsStandCap,
|
|
24879
25062
|
noRecommendationReason: rec.noRecommendationReason,
|
|
24880
|
-
|
|
24881
|
-
|
|
24882
|
-
|
|
25063
|
+
recommendedMaxCredits: rec.recommendedMaxCredits,
|
|
25064
|
+
recommendedPerCallMaxCredits: rec.recommendedPerCallMaxCredits,
|
|
25065
|
+
averageCreditsPerCall: rec.averageCreditsPerCall,
|
|
25066
|
+
recommendedPerCallEstimateCredits: rec.recommendedPerCallEstimateCredits,
|
|
24883
25067
|
savedTo
|
|
24884
25068
|
});
|
|
24885
25069
|
if (!ok) process.exitCode = 1;
|
|
24886
25070
|
return;
|
|
24887
25071
|
}
|
|
24888
25072
|
log.plain("");
|
|
25073
|
+
const printMeasured = () => {
|
|
25074
|
+
log.plain(c.bold(" Real cost per call"));
|
|
25075
|
+
log.plain(` p50 ${usdFromPicos(rec.costP50)} p95 ${usdFromPicos(rec.costP95)} max ${usdFromPicos(rec.costMax)} (${settledCount} of ${samples} settled)`);
|
|
25076
|
+
log.plain(c.bold(" Charged credits"));
|
|
25077
|
+
log.plain(` p50 ${rec.chargedP50} p95 ${rec.chargedP95} max ${rec.chargedMax} (whole credits, rounded up per call)`);
|
|
25078
|
+
};
|
|
24889
25079
|
if (lengthBlocked) {
|
|
24890
|
-
if (
|
|
24891
|
-
|
|
24892
|
-
log.plain(` p50 ${p50} p95 ${p95} max ${max} (${charged.length} of ${samples} settled)`);
|
|
25080
|
+
if (settledCount > 0) {
|
|
25081
|
+
printMeasured();
|
|
24893
25082
|
log.plain("");
|
|
24894
25083
|
}
|
|
24895
25084
|
printRecommendation(log, record);
|
|
@@ -24897,17 +25086,16 @@ async function runBench(args) {
|
|
|
24897
25086
|
process.exitCode = 1;
|
|
24898
25087
|
return;
|
|
24899
25088
|
}
|
|
24900
|
-
if (
|
|
25089
|
+
if (settledCount === 0) {
|
|
24901
25090
|
log.error(
|
|
24902
|
-
providerRefused > 0 && providerRefused === results.length ? " No sample ran \u2014 the provider refused every attempt at its door \u2014 so there is nothing to
|
|
25091
|
+
providerRefused > 0 && providerRefused === results.length ? " No sample ran \u2014 the provider refused every attempt at its door \u2014 so there is nothing to measure from." : " No sample settled, so there is nothing to measure from."
|
|
24903
25092
|
);
|
|
24904
25093
|
process.exitCode = 1;
|
|
24905
25094
|
return;
|
|
24906
25095
|
}
|
|
24907
|
-
|
|
24908
|
-
log.plain(` p50 ${p50} p95 ${p95} max ${max} (${charged.length} of ${samples} settled)`);
|
|
25096
|
+
printMeasured();
|
|
24909
25097
|
if (balanceBefore !== null && balanceAfter !== null) {
|
|
24910
|
-
log.plain(` Balance ${balanceBefore} \u2192 ${balanceAfter}
|
|
25098
|
+
log.plain(` Balance ${balanceBefore} \u2192 ${balanceAfter} credits spendable`);
|
|
24911
25099
|
}
|
|
24912
25100
|
log.plain("");
|
|
24913
25101
|
printRecommendation(log, record);
|
|
@@ -24919,43 +25107,58 @@ function printRecommendation(log, record) {
|
|
|
24919
25107
|
return;
|
|
24920
25108
|
}
|
|
24921
25109
|
if (record.noRecommendationReason === "answer_cut_off") {
|
|
24922
|
-
log.warn(` ${benchCutOffLine(record.cutOffSamples
|
|
25110
|
+
log.warn(` ${benchCutOffLine(record.cutOffSamples || 1, record.maxCreditsPerSample)}`);
|
|
24923
25111
|
return;
|
|
24924
25112
|
}
|
|
24925
|
-
|
|
24926
|
-
|
|
25113
|
+
const attempts = `${record.completed} settled attempt${record.completed === 1 ? "" : "s"} on ${record.modelId}`;
|
|
25114
|
+
if (record.recommendedMaxCredits === null || record.cost.p95 === null) {
|
|
25115
|
+
log.warn(" No ceiling recommended.");
|
|
24927
25116
|
log.dim(
|
|
24928
|
-
record.recommendedDeclaredHeadroomBps === null ? " This stand served no recommended headroom, and the multiplier is the server's to set \u2014" : "
|
|
25117
|
+
record.cost.p95 === null ? " No sample settled, so there is no p95 to build a ceiling on \u2014" : record.recommendedDeclaredHeadroomBps === null ? " This stand served no recommended headroom, and the multiplier is the server's to set \u2014" : " This stand served no platform fee or credit value, and those are the server's to set \u2014"
|
|
24929
25118
|
);
|
|
24930
25119
|
log.dim(" declaring a number from this run would be a guess dressed as a measurement.");
|
|
25120
|
+
printAverage(log, record, attempts);
|
|
24931
25121
|
return;
|
|
24932
25122
|
}
|
|
24933
|
-
log.plain(c.bold(` Declare
|
|
24934
|
-
if (record.
|
|
25123
|
+
log.plain(c.bold(` Declare maxCredits: ${record.recommendedMaxCredits}`));
|
|
25124
|
+
if (record.lengthRaisedCeiling) {
|
|
24935
25125
|
log.plain(` ${LLM_LENGTH_RAISED_LINE}`);
|
|
24936
25126
|
log.dim(
|
|
24937
|
-
` = the smallest
|
|
25127
|
+
` = the smallest ceiling whose answer allowance holds the answer plus this stand's headroom (p95 ${record.fitP95 ?? "\u2014"}), above the`
|
|
24938
25128
|
);
|
|
24939
25129
|
log.dim(
|
|
24940
|
-
`
|
|
25130
|
+
` cost-based ${record.costBasedMaxCredits ?? "\u2014"} (p95 real cost ${usdFromPicos(record.cost.p95)} with the platform fee, ${record.feeBps} bps, and ${record.recommendedDeclaredHeadroomBps} bps of headroom), over ${attempts}.`
|
|
24941
25131
|
);
|
|
24942
25132
|
} else {
|
|
24943
25133
|
log.dim(
|
|
24944
|
-
` = p95
|
|
25134
|
+
` = p95 real cost (${usdFromPicos(record.cost.p95)}) with this stand's platform fee (${record.feeBps} bps) and recommended headroom (${record.recommendedDeclaredHeadroomBps} bps), in whole credits,`
|
|
24945
25135
|
);
|
|
24946
|
-
log.dim(` measured over ${
|
|
25136
|
+
log.dim(` measured over ${attempts}.`);
|
|
24947
25137
|
}
|
|
24948
|
-
log.dim(" That number is a
|
|
24949
|
-
|
|
24950
|
-
|
|
25138
|
+
log.dim(" That number is a CEILING, not a price: a call is billed its real cost as used, never more than it \u2014");
|
|
25139
|
+
log.dim(" and it also sizes how long the answer may be.");
|
|
25140
|
+
if (record.recommendedPerCallMaxCredits !== null) {
|
|
25141
|
+
log.plain(c.bold(` Grant perCallMaxCredits: ${record.recommendedPerCallMaxCredits}`));
|
|
24951
25142
|
log.dim(
|
|
24952
|
-
record.
|
|
25143
|
+
record.lengthRaisedCeiling ? ` = the same over the worst sample (${usdFromPicos(record.cost.max)}) or the longest answer's ceiling, whichever is larger, and never below the ceiling above it.` : ` = the same over the worst sample (${usdFromPicos(record.cost.max)}), and never below the ceiling above it.`
|
|
24953
25144
|
);
|
|
24954
|
-
|
|
25145
|
+
}
|
|
25146
|
+
printAverage(log, record, attempts);
|
|
25147
|
+
}
|
|
25148
|
+
function printAverage(log, record, attempts) {
|
|
25149
|
+
if (record.averageCreditsPerCall === null) return;
|
|
25150
|
+
log.plain(` Per-call estimate: about ${formatCredits(record.averageCreditsPerCall)} credits per call`);
|
|
25151
|
+
log.dim(` = the average real cost with the platform fee, no headroom, over ${attempts}.`);
|
|
25152
|
+
if (record.recommendedPerCallEstimateCredits !== null) {
|
|
25153
|
+
log.plain(c.bold(` Grant perCallEstimateCredits: ${record.recommendedPerCallEstimateCredits}`));
|
|
25154
|
+
log.dim(` = that average rounded up to a whole credit. disclosure.estimatedCreditsPerPeriod = your calls per period \xD7 ${formatCredits(record.averageCreditsPerCall)}, rounded up.`);
|
|
24955
25155
|
}
|
|
24956
25156
|
}
|
|
24957
25157
|
async function explainBenchRefusal(res, call2, base, log, opts, index, settledSoFar) {
|
|
24958
|
-
if (printedStructuredError(res))
|
|
25158
|
+
if (printedStructuredError(res)) {
|
|
25159
|
+
const printed = await res.json().catch(() => ({}));
|
|
25160
|
+
return { stop: true, refusal: { status: res.status, error: typeof printed.error === "string" ? printed.error : null, slotsHeld: null, slotLimit: null } };
|
|
25161
|
+
}
|
|
24959
25162
|
const body = await res.json().catch(() => ({}));
|
|
24960
25163
|
const refusal2 = { status: res.status, error: body.error ?? null, slotsHeld: null, slotLimit: null };
|
|
24961
25164
|
if (res.status === 429 && body.error === "generation_limit") {
|
|
@@ -24965,6 +25168,10 @@ async function explainBenchRefusal(res, call2, base, log, opts, index, settledSo
|
|
|
24965
25168
|
if (!opts.json) log.error(` Sample ${index + 1} ${benchSlotsHeldSentence(counts)}`);
|
|
24966
25169
|
return { stop: true, refusal: refusal2 };
|
|
24967
25170
|
}
|
|
25171
|
+
if (res.status === 403 && body.error === "credits_unverified") {
|
|
25172
|
+
if (!opts.json) log.error(` ${LLM_BENCH_UNVERIFIED_LINE}`);
|
|
25173
|
+
return { stop: true, refusal: refusal2 };
|
|
25174
|
+
}
|
|
24968
25175
|
if (opts.json) return { stop: res.status !== 429, refusal: refusal2 };
|
|
24969
25176
|
if (res.status === 404 && body.error === "not_found") {
|
|
24970
25177
|
log.error(` ${LLM_LANE_OFF_LINE}`);
|
|
@@ -24980,7 +25187,7 @@ async function explainBenchRefusal(res, call2, base, log, opts, index, settledSo
|
|
|
24980
25187
|
return { stop: true, refusal: refusal2 };
|
|
24981
25188
|
}
|
|
24982
25189
|
if (res.status === 402) {
|
|
24983
|
-
log.error(" Not enough
|
|
25190
|
+
log.error(" Not enough credits to start the attempt \u2014 a bench is paid from your own balance.");
|
|
24984
25191
|
return { stop: true, refusal: refusal2 };
|
|
24985
25192
|
}
|
|
24986
25193
|
log.error(
|
|
@@ -24989,7 +25196,7 @@ async function explainBenchRefusal(res, call2, base, log, opts, index, settledSo
|
|
|
24989
25196
|
return { stop: res.status >= 500 || res.status === 401 || res.status === 403, refusal: refusal2 };
|
|
24990
25197
|
}
|
|
24991
25198
|
function printProviderRefusal(log, status2, message, awaitingBill = false) {
|
|
24992
|
-
if (awaitingBill) log.dim(` The provider answered ${status2} after routing \u2014 its bill is not final, so this call
|
|
25199
|
+
if (awaitingBill) log.dim(` The provider answered ${status2} after routing \u2014 its bill is not final, so what this call reserved stays held until it resolves (see \`genex llm status\`); it is not a sample.`);
|
|
24993
25200
|
else log.dim(" The provider refused this call at its door \u2014 no inference ran, it cost nothing, and it is not a sample.");
|
|
24994
25201
|
if (message) log.dim(` Provider said: ${message}`);
|
|
24995
25202
|
if (status2 === 401 || status2 === 403) {
|
|
@@ -25015,19 +25222,6 @@ async function pollSettled(call2, url, timeoutSec) {
|
|
|
25015
25222
|
}
|
|
25016
25223
|
return last;
|
|
25017
25224
|
}
|
|
25018
|
-
async function readCoinBalance(apiUrl, token) {
|
|
25019
|
-
try {
|
|
25020
|
-
const res = await apiFetch(`${apiUrl}/api/coin/balance`, {
|
|
25021
|
-
headers: { Authorization: `Bearer ${token}` },
|
|
25022
|
-
signal: AbortSignal.timeout(6e3)
|
|
25023
|
-
});
|
|
25024
|
-
if (!res.ok) return null;
|
|
25025
|
-
const body = await res.json().catch(() => null);
|
|
25026
|
-
return typeof body?.spendable === "number" ? body.spendable : null;
|
|
25027
|
-
} catch {
|
|
25028
|
-
return null;
|
|
25029
|
-
}
|
|
25030
|
-
}
|
|
25031
25225
|
async function saveBench(cwd, record) {
|
|
25032
25226
|
try {
|
|
25033
25227
|
const file = path36.join(cwd, LLM_BENCH_FILE);
|
|
@@ -25043,9 +25237,10 @@ function benchFailed(opts, log, message) {
|
|
|
25043
25237
|
else log.error(message);
|
|
25044
25238
|
process.exitCode = 1;
|
|
25045
25239
|
}
|
|
25046
|
-
function emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples,
|
|
25240
|
+
function emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples, maxCredits, opts) {
|
|
25047
25241
|
return {
|
|
25048
25242
|
command: "llm bench",
|
|
25243
|
+
currency: "credits",
|
|
25049
25244
|
apiUrl,
|
|
25050
25245
|
projectId,
|
|
25051
25246
|
modelId: opts.modelId ?? null,
|
|
@@ -25054,21 +25249,28 @@ function emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples, maxCoi
|
|
|
25054
25249
|
schemaPath: opts.schemaPath ?? null,
|
|
25055
25250
|
samples,
|
|
25056
25251
|
completed: 0,
|
|
25057
|
-
|
|
25252
|
+
maxCreditsPerSample: maxCredits,
|
|
25058
25253
|
balanceBefore: null,
|
|
25059
25254
|
balanceAfter: null,
|
|
25060
25255
|
results: [],
|
|
25061
25256
|
refusal: null,
|
|
25062
|
-
|
|
25063
|
-
|
|
25064
|
-
|
|
25065
|
-
|
|
25257
|
+
costUsdPicos: { p50: null, p95: null, max: null },
|
|
25258
|
+
costUsd: { p50: null, p95: null, max: null },
|
|
25259
|
+
chargedCredits: { p50: null, p95: null, max: null },
|
|
25260
|
+
fitCredits: { p95: null, max: null },
|
|
25261
|
+
recommendedDeclaredHeadroomBps: null,
|
|
25262
|
+
feeBps: null,
|
|
25263
|
+
usdCentsPerCredit: null,
|
|
25264
|
+
missingTerms: null,
|
|
25265
|
+
costBasedMaxCredits: null,
|
|
25266
|
+
lengthRaisedCeiling: false,
|
|
25066
25267
|
cutOffSamples: 0,
|
|
25067
25268
|
exceedsStandCap: false,
|
|
25068
25269
|
noRecommendationReason: null,
|
|
25069
|
-
|
|
25070
|
-
|
|
25071
|
-
|
|
25270
|
+
recommendedMaxCredits: null,
|
|
25271
|
+
recommendedPerCallMaxCredits: null,
|
|
25272
|
+
averageCreditsPerCall: null,
|
|
25273
|
+
recommendedPerCallEstimateCredits: null,
|
|
25072
25274
|
savedTo: null
|
|
25073
25275
|
};
|
|
25074
25276
|
}
|
|
@@ -25131,7 +25333,8 @@ async function reportStatus(args) {
|
|
|
25131
25333
|
for (const row of rows) {
|
|
25132
25334
|
const state = rowState(row);
|
|
25133
25335
|
const mark = state === "active" ? c.cyan("\u25CF") : state === "awaiting bill" ? c.yellow("\u25D0") : c.green("\u25CB");
|
|
25134
|
-
const
|
|
25336
|
+
const money2 = rowMoney(row);
|
|
25337
|
+
const reserved = money2.reserved === null ? "reserved \u2014" : `${money2.reserved} ${money2.unit} reserved`;
|
|
25135
25338
|
log.plain(
|
|
25136
25339
|
` ${mark} ${row.id ?? "?"} ${row.modelId ?? "?"} ${state.padEnd(13)} ${formatAge(row.createdAt, now).padStart(4)} old ${reserved}${row.error ? ` ${row.error}` : ""}${rowHoldsSlot(row) ? "" : c.dim(" (no slot)")}`
|
|
25137
25340
|
);
|
|
@@ -25203,10 +25406,11 @@ async function cancelCall(args) {
|
|
|
25203
25406
|
if (!opts.json) {
|
|
25204
25407
|
log.warn(` ${row.id} (${row.status ?? "?"}) \u2014 ${sentence}.`);
|
|
25205
25408
|
if (state === "awaiting bill") {
|
|
25206
|
-
log.dim(` Nothing to cancel:
|
|
25409
|
+
log.dim(` Nothing to cancel: what it reserved releases when the bill resolves, and it stops holding a slot ${PENDING_SLOT_RELEASE_MINUTES} minutes after dispatch.`);
|
|
25207
25410
|
if (row.providerMessage?.trim()) log.dim(` Provider said: ${row.providerMessage.trim()}`);
|
|
25208
25411
|
} else {
|
|
25209
|
-
|
|
25412
|
+
const money2 = rowMoney(row);
|
|
25413
|
+
log.dim(` ${money2.charged === null ? "charge unknown" : `${money2.charged} ${money2.unit} charged`}${row.error ? ` \xB7 ${row.error}` : ""}.`);
|
|
25210
25414
|
}
|
|
25211
25415
|
}
|
|
25212
25416
|
return done("ok", null, state === "awaiting bill" ? "awaiting_bill" : "already_final", row);
|
|
@@ -25228,78 +25432,126 @@ async function cancelCall(args) {
|
|
|
25228
25432
|
const after = await res.json().catch(() => null) ?? row;
|
|
25229
25433
|
if (!opts.json) {
|
|
25230
25434
|
log.success(` Cancelled ${row.id} (was ${row.status ?? "active"}).`);
|
|
25231
|
-
log.dim(`
|
|
25435
|
+
log.dim(` What it reserved releases once the bill settles; ${c.cyan("genex llm status")} shows it until then.`);
|
|
25232
25436
|
}
|
|
25233
25437
|
return done("ok", null, "cancelled", after);
|
|
25234
25438
|
}
|
|
25235
|
-
async function
|
|
25236
|
-
let
|
|
25439
|
+
async function readSavedBench(cwd) {
|
|
25440
|
+
let parsed;
|
|
25237
25441
|
try {
|
|
25238
|
-
|
|
25442
|
+
parsed = JSON.parse(await fs35.readFile(path36.join(cwd, LLM_BENCH_FILE), "utf8"));
|
|
25239
25443
|
} catch {
|
|
25240
|
-
|
|
25444
|
+
return null;
|
|
25241
25445
|
}
|
|
25242
|
-
if (!
|
|
25446
|
+
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return null;
|
|
25447
|
+
const v = parsed;
|
|
25448
|
+
if (v.v === 2 && v.currency === "credits" && v.cost && v.charged) return { kind: "credits", record: parsed };
|
|
25449
|
+
return { kind: "coins", record: parsed };
|
|
25450
|
+
}
|
|
25451
|
+
var LLM_PRICE_COIN_TERMS_LINE = "This bench was measured under the old COIN terms, when a game declared a fixed price per call. In-game calls are now billed in the player's credits, as used, and a game declares a per-call CEILING (maxCredits) instead \u2014 a coin price is not that number. Re-run the bench for the figures to declare.";
|
|
25452
|
+
async function reportSavedPrice(cwd, opts, log) {
|
|
25453
|
+
const saved = await readSavedBench(cwd);
|
|
25454
|
+
const rerun = 'genex llm bench "<prompt>" --max-credits <n> --user-approved';
|
|
25455
|
+
const creditKeys = (record2) => ({
|
|
25456
|
+
costUsdPicos: record2?.cost ?? { p50: null, p95: null, max: null },
|
|
25457
|
+
chargedCredits: record2?.charged ?? { p50: null, p95: null, max: null },
|
|
25458
|
+
fitP95: record2?.fitP95 ?? null,
|
|
25459
|
+
fitMax: record2?.fitMax ?? null,
|
|
25460
|
+
recommendedDeclaredHeadroomBps: record2?.recommendedDeclaredHeadroomBps ?? null,
|
|
25461
|
+
feeBps: record2?.feeBps ?? null,
|
|
25462
|
+
usdCentsPerCredit: record2?.usdCentsPerCredit ?? null,
|
|
25463
|
+
costBasedMaxCredits: record2?.costBasedMaxCredits ?? null,
|
|
25464
|
+
lengthRaisedCeiling: record2?.lengthRaisedCeiling ?? false,
|
|
25465
|
+
cutOffSamples: record2?.cutOffSamples ?? 0,
|
|
25466
|
+
noRecommendationReason: record2?.noRecommendationReason ?? null,
|
|
25467
|
+
recommendedMaxCredits: record2?.recommendedMaxCredits ?? null,
|
|
25468
|
+
recommendedPerCallMaxCredits: record2?.recommendedPerCallMaxCredits ?? null,
|
|
25469
|
+
averageCreditsPerCall: record2?.averageCreditsPerCall ?? null,
|
|
25470
|
+
recommendedPerCallEstimateCredits: record2?.recommendedPerCallEstimateCredits ?? null
|
|
25471
|
+
});
|
|
25472
|
+
if (!saved) {
|
|
25243
25473
|
if (opts.json) {
|
|
25244
25474
|
writeJsonLine({
|
|
25245
25475
|
command: "llm price",
|
|
25246
25476
|
status: "failed",
|
|
25247
25477
|
error: "no_bench",
|
|
25478
|
+
currency: null,
|
|
25248
25479
|
ranAt: null,
|
|
25249
25480
|
modelId: null,
|
|
25250
25481
|
samples: null,
|
|
25251
25482
|
completed: null,
|
|
25252
|
-
|
|
25253
|
-
|
|
25254
|
-
max: null,
|
|
25255
|
-
fitP95: null,
|
|
25256
|
-
chargedBasedEstimateCoins: null,
|
|
25257
|
-
lengthRaisedPrice: false,
|
|
25258
|
-
cutOffSamples: 0,
|
|
25259
|
-
noRecommendationReason: null,
|
|
25260
|
-
recommendedDeclaredHeadroomBps: null,
|
|
25261
|
-
recommendedEstimateCoins: null,
|
|
25262
|
-
recommendedPerCallMaxCoins: null,
|
|
25483
|
+
...creditKeys(null),
|
|
25484
|
+
legacyCoinTerms: null,
|
|
25263
25485
|
savedTo: null
|
|
25264
25486
|
});
|
|
25265
25487
|
} else {
|
|
25266
25488
|
log.error("Nothing has been benchmarked in this folder yet.");
|
|
25267
|
-
log.dim(` Run ${c.cyan(
|
|
25489
|
+
log.dim(` Run ${c.cyan(rerun)} first.`);
|
|
25490
|
+
}
|
|
25491
|
+
process.exitCode = 1;
|
|
25492
|
+
return;
|
|
25493
|
+
}
|
|
25494
|
+
if (saved.kind === "coins") {
|
|
25495
|
+
const old = saved.record;
|
|
25496
|
+
const legacy = {
|
|
25497
|
+
p50: old.p50 ?? null,
|
|
25498
|
+
p95: old.p95 ?? null,
|
|
25499
|
+
max: old.max ?? null,
|
|
25500
|
+
recommendedEstimateCoins: old.recommendedEstimateCoins ?? null,
|
|
25501
|
+
recommendedPerCallMaxCoins: old.recommendedPerCallMaxCoins ?? null,
|
|
25502
|
+
maxCoinsPerSample: old.maxCoinsPerSample ?? null,
|
|
25503
|
+
noRecommendationReason: old.noRecommendationReason ?? null
|
|
25504
|
+
};
|
|
25505
|
+
if (opts.json) {
|
|
25506
|
+
writeJsonLine({
|
|
25507
|
+
command: "llm price",
|
|
25508
|
+
status: "failed",
|
|
25509
|
+
error: "coin_terms",
|
|
25510
|
+
currency: "coins",
|
|
25511
|
+
ranAt: old.ranAt ?? null,
|
|
25512
|
+
modelId: old.modelId ?? null,
|
|
25513
|
+
samples: old.samples ?? null,
|
|
25514
|
+
completed: old.completed ?? null,
|
|
25515
|
+
...creditKeys(null),
|
|
25516
|
+
legacyCoinTerms: legacy,
|
|
25517
|
+
savedTo: LLM_BENCH_FILE
|
|
25518
|
+
});
|
|
25519
|
+
} else {
|
|
25520
|
+
log.plain(c.bold("genex llm price"));
|
|
25521
|
+
log.dim(` from ${LLM_BENCH_FILE}, benchmarked ${old.ranAt ?? "at an unknown time"}${old.modelId ? ` on ${old.modelId}` : ""}`);
|
|
25522
|
+
log.plain("");
|
|
25523
|
+
log.warn(` ${LLM_PRICE_COIN_TERMS_LINE}`);
|
|
25524
|
+
const measured = legacy.p95 === null ? "no settled sample" : `charged coins p50 ${legacy.p50} \xB7 p95 ${legacy.p95} \xB7 max ${legacy.max}`;
|
|
25525
|
+
const recommended = legacy.recommendedEstimateCoins === null ? "no price recommended" : `it recommended estimateCoins ${legacy.recommendedEstimateCoins}${legacy.recommendedPerCallMaxCoins != null ? ` and perCallMaxCoins ${legacy.recommendedPerCallMaxCoins}` : ""}`;
|
|
25526
|
+
log.dim(` What it measured then: ${measured}; ${recommended}.`);
|
|
25527
|
+
log.dim(` Re-run: ${c.cyan(rerun)}`);
|
|
25268
25528
|
}
|
|
25269
25529
|
process.exitCode = 1;
|
|
25270
25530
|
return;
|
|
25271
25531
|
}
|
|
25532
|
+
const record = saved.record;
|
|
25272
25533
|
if (opts.json) {
|
|
25273
25534
|
writeJsonLine({
|
|
25274
25535
|
command: "llm price",
|
|
25275
|
-
status: record.
|
|
25276
|
-
error: record.
|
|
25536
|
+
status: record.recommendedMaxCredits === null ? "failed" : "ok",
|
|
25537
|
+
error: record.recommendedMaxCredits === null ? record.noRecommendationReason ?? "no_recommendation" : null,
|
|
25538
|
+
currency: "credits",
|
|
25277
25539
|
ranAt: record.ranAt ?? null,
|
|
25278
25540
|
modelId: record.modelId ?? null,
|
|
25279
25541
|
samples: record.samples ?? null,
|
|
25280
25542
|
completed: record.completed ?? null,
|
|
25281
|
-
|
|
25282
|
-
|
|
25283
|
-
max: record.max ?? null,
|
|
25284
|
-
// Absent from a file an older CLI wrote; null / false / 0 then, never missing.
|
|
25285
|
-
fitP95: record.fitP95 ?? null,
|
|
25286
|
-
chargedBasedEstimateCoins: record.chargedBasedEstimateCoins ?? null,
|
|
25287
|
-
lengthRaisedPrice: record.lengthRaisedPrice ?? false,
|
|
25288
|
-
cutOffSamples: record.cutOffSamples ?? 0,
|
|
25289
|
-
noRecommendationReason: record.noRecommendationReason ?? null,
|
|
25290
|
-
recommendedDeclaredHeadroomBps: record.recommendedDeclaredHeadroomBps ?? null,
|
|
25291
|
-
recommendedEstimateCoins: record.recommendedEstimateCoins ?? null,
|
|
25292
|
-
recommendedPerCallMaxCoins: record.recommendedPerCallMaxCoins ?? null,
|
|
25543
|
+
...creditKeys(record),
|
|
25544
|
+
legacyCoinTerms: null,
|
|
25293
25545
|
savedTo: LLM_BENCH_FILE
|
|
25294
25546
|
});
|
|
25295
|
-
if (record.
|
|
25547
|
+
if (record.recommendedMaxCredits === null) process.exitCode = 1;
|
|
25296
25548
|
return;
|
|
25297
25549
|
}
|
|
25298
25550
|
log.plain(c.bold("genex llm price"));
|
|
25299
25551
|
log.dim(` from ${LLM_BENCH_FILE}, benchmarked ${record.ranAt}`);
|
|
25300
25552
|
log.plain("");
|
|
25301
25553
|
printRecommendation(log, record);
|
|
25302
|
-
if (record.
|
|
25554
|
+
if (record.recommendedMaxCredits === null) process.exitCode = 1;
|
|
25303
25555
|
}
|
|
25304
25556
|
function writeJsonLine(value) {
|
|
25305
25557
|
process.stdout.write(`${JSON.stringify(value)}
|
|
@@ -25559,7 +25811,7 @@ function runtimeRow(state, lane, toolsOnly = false) {
|
|
|
25559
25811
|
return {
|
|
25560
25812
|
label: "Runtime",
|
|
25561
25813
|
value: `in-game LLM lane live \xB7 ${n} model${n === 1 ? "" : "s"}${keyRefused ? ` \xB7 ${c.yellow("provider key refused")}` : ""}`,
|
|
25562
|
-
fix: keyRefused ? LLM_PROVIDER_KEY_REJECTED_LINE : toolsOnly ? convertFix : '
|
|
25814
|
+
fix: keyRefused ? LLM_PROVIDER_KEY_REJECTED_LINE : toolsOnly ? convertFix : 'Measure a call before a game declares its per-call ceiling: `npx genex llm bench "<prompt>" --max-credits <n> --user-approved`.'
|
|
25563
25815
|
};
|
|
25564
25816
|
}
|
|
25565
25817
|
case "off":
|
|
@@ -28236,16 +28488,16 @@ ${c.bold("Usage")}
|
|
|
28236
28488
|
status | cancel. "models" says whether this
|
|
28237
28489
|
stand serves the runtime LLM lane at all, and
|
|
28238
28490
|
at what provider rates. "bench" runs the real
|
|
28239
|
-
model on YOUR OWN
|
|
28240
|
-
attempt
|
|
28491
|
+
model on YOUR OWN credits and reports what each
|
|
28492
|
+
attempt really cost; it needs --max-credits <n>
|
|
28241
28493
|
--user-approved, like any spend the player has
|
|
28242
28494
|
to agree to. "price" reprints the last run's
|
|
28243
|
-
recommended
|
|
28495
|
+
recommended maxCredits. "status" lists this
|
|
28244
28496
|
project's open calls and the slots they hold;
|
|
28245
28497
|
"cancel <id>" stops an active one. Declare a
|
|
28246
|
-
game's
|
|
28247
|
-
a
|
|
28248
|
-
|
|
28498
|
+
game's per-call ceiling from a bench, never
|
|
28499
|
+
from a guess: the player is billed as used, and
|
|
28500
|
+
the ceiling also sizes how long an answer may be.
|
|
28249
28501
|
genex player <sub> [options] Answer the in-game model requests you approve, on your
|
|
28250
28502
|
OWN Claude or ChatGPT subscription: install | run |
|
|
28251
28503
|
status | stop | uninstall. "install" signs this machine
|
|
@@ -28712,7 +28964,7 @@ ${c.bold("Examples")}
|
|
|
28712
28964
|
genex shop remove sku_123
|
|
28713
28965
|
genex llm models
|
|
28714
28966
|
genex llm models --all
|
|
28715
|
-
genex llm bench "Reply with one short taunt." --samples 5 --max-
|
|
28967
|
+
genex llm bench "Reply with one short taunt." --samples 5 --max-credits 20 --user-approved
|
|
28716
28968
|
genex llm price
|
|
28717
28969
|
genex llm status
|
|
28718
28970
|
genex llm cancel <id>
|
|
@@ -28949,22 +29201,24 @@ ${c.bold("How the allowance works")}
|
|
|
28949
29201
|
|
|
28950
29202
|
Every number printed is live from your account; prices can change without a deploy.
|
|
28951
29203
|
`,
|
|
28952
|
-
llm: `${c.bold("genex llm")} \u2014 in-game model calls: is the lane live, what does one call cost, what should the game declare?
|
|
29204
|
+
llm: `${c.bold("genex llm")} \u2014 in-game model calls: is the lane live, what does one call cost, what ceiling should the game declare?
|
|
28953
29205
|
|
|
28954
29206
|
${c.bold("Usage")}
|
|
28955
29207
|
genex llm models [--all] [--json] Which models this stand serves, their provider rates per
|
|
28956
|
-
million tokens,
|
|
28957
|
-
|
|
28958
|
-
|
|
28959
|
-
|
|
28960
|
-
on this stand: it says so
|
|
28961
|
-
|
|
28962
|
-
|
|
28963
|
-
|
|
28964
|
-
|
|
28965
|
-
|
|
28966
|
-
|
|
28967
|
-
|
|
29208
|
+
million tokens, the platform fee a call is billed at, and
|
|
29209
|
+
the headroom it recommends over a benchmark. The catalog
|
|
29210
|
+
is synced from OpenRouter, so the FEATURED short list is
|
|
29211
|
+
printed by default and --all prints every row (--json
|
|
29212
|
+
always carries every row). Off on this stand: it says so
|
|
29213
|
+
and exits 0 \u2014 build the feature without a model rather
|
|
29214
|
+
than promising a 404.
|
|
29215
|
+
genex llm bench "<prompt>" --max-credits <n> --user-approved [options]
|
|
29216
|
+
Run the real model through the development lane and
|
|
29217
|
+
report what each attempt really COST and was charged.
|
|
29218
|
+
Refused without both flags, before any network call:
|
|
29219
|
+
this spends YOUR OWN CREDITS, a spend the build's asset
|
|
29220
|
+
allowance does not count. Needs a folder linked to a
|
|
29221
|
+
game you own, and a balance that covers one attempt.
|
|
28968
29222
|
genex llm price [--json] The recommendation from the last bench in this folder.
|
|
28969
29223
|
Reads a file; spends nothing.
|
|
28970
29224
|
genex llm status [--json] This project's open development calls: active, or
|
|
@@ -28981,29 +29235,32 @@ ${c.bold("Bench options")}
|
|
|
28981
29235
|
--samples <n> How many real attempts (default ${DEFAULT_BENCH_SAMPLES}). More samples, better p95.
|
|
28982
29236
|
--schema <file> A JSON file holding the output schema; implies JSON output.
|
|
28983
29237
|
--json-output Ask for JSON without a schema. --text Ask for text (the default).
|
|
28984
|
-
--max-
|
|
29238
|
+
--max-credits <n> The per-attempt ceiling YOU approved \u2014 a bench runs on your own credits,
|
|
28985
29239
|
never a player's. The run's worst case is that number times --samples,
|
|
28986
|
-
and it is printed before anything starts.
|
|
29240
|
+
and it is printed before anything starts. (--max-coins is its deprecated
|
|
29241
|
+
name, read as the same number.)
|
|
28987
29242
|
--timeout <seconds> Bounds the wait for each attempt.
|
|
28988
29243
|
--json One machine-readable object; every key is always present, null when
|
|
28989
29244
|
the CLI could not learn it.
|
|
28990
29245
|
|
|
28991
29246
|
${c.bold("Why a benchmark and not an estimate")}
|
|
28992
|
-
|
|
28993
|
-
|
|
28994
|
-
|
|
28995
|
-
|
|
28996
|
-
|
|
28997
|
-
|
|
28998
|
-
|
|
28999
|
-
|
|
29000
|
-
|
|
29247
|
+
An in-game call is paid from the PLAYER's credits and billed as used: its real provider cost
|
|
29248
|
+
plus the platform fee, never more than the per-call CEILING the game declares (\`maxCredits\`,
|
|
29249
|
+
or \`perCallMaxCredits\` on a standing budget). The ceiling is not a price, but it is still a
|
|
29250
|
+
number with two jobs: it bounds what one call may cost, and it funds how LONG the answer may
|
|
29251
|
+
be. Three layers sit between the provider's rate card and it \u2014 the provider's cost for YOUR
|
|
29252
|
+
prompt, the platform fee, and your headroom \u2014 and only a real call measures the first. The
|
|
29253
|
+
bench is billed to YOU, on the development lane; the player-funded lane is what the
|
|
29254
|
+
published game uses. The recommendation is the p95 real cost with the server's fee and
|
|
29255
|
+
headroom applied, in whole credits; a stand that serves no headroom or no fee gets no
|
|
29256
|
+
recommendation, because those multipliers are not the CLI's to invent.
|
|
29001
29257
|
|
|
29002
|
-
|
|
29003
|
-
|
|
29004
|
-
|
|
29005
|
-
|
|
29006
|
-
|
|
29258
|
+
Because the ceiling sizes each call's output allowance, the recommendation is never below
|
|
29259
|
+
the smallest ceiling that leaves room for the benchmarked answer, as the server computes it.
|
|
29260
|
+
A sample cut off at your --max-credits (provider_token_limit) is not a sample \u2014 re-run with a
|
|
29261
|
+
higher --max-credits; an answer longer than one call on this stand may produce gets no
|
|
29262
|
+
ceiling at all \u2014 ask for a shorter one. The run also prints the measured average per call,
|
|
29263
|
+
fee included, which is the honest estimate a budget's disclosure is built on.
|
|
29007
29264
|
|
|
29008
29265
|
Results land in ${LLM_BENCH_FILE} \u2014 gitignored, machine-local, re-read by \`genex llm price\`.
|
|
29009
29266
|
`,
|
|
@@ -29251,11 +29508,13 @@ function parseArgs(argv) {
|
|
|
29251
29508
|
"--limit",
|
|
29252
29509
|
// `genex budget --assets <credits>` — the allowance the player approved.
|
|
29253
29510
|
"--assets",
|
|
29254
|
-
// `genex llm bench` value flags. `--max-
|
|
29255
|
-
//
|
|
29511
|
+
// `genex llm bench` value flags. `--max-credits` is the same approval
|
|
29512
|
+
// shape as `--assets`, but its own gate: the asset allowance does not
|
|
29513
|
+
// count a bench. `--max-coins` is its deprecated name (same number).
|
|
29256
29514
|
"--model",
|
|
29257
29515
|
"--samples",
|
|
29258
29516
|
"--schema",
|
|
29517
|
+
"--max-credits",
|
|
29259
29518
|
"--max-coins",
|
|
29260
29519
|
// `genex player install --label <name>` — what this machine is called in
|
|
29261
29520
|
// the dashboard. Its own key rather than the generic 2nd positional,
|
|
@@ -29809,12 +30068,14 @@ function applyValueFlag(options, flag, value) {
|
|
|
29809
30068
|
options.samples = n;
|
|
29810
30069
|
break;
|
|
29811
30070
|
}
|
|
30071
|
+
case "--max-credits":
|
|
29812
30072
|
case "--max-coins": {
|
|
29813
30073
|
const n = Number(value);
|
|
29814
30074
|
if (!Number.isInteger(n) || n < 1) {
|
|
29815
|
-
throw new Error(`Invalid
|
|
30075
|
+
throw new Error(`Invalid ${flag} value: ${value} (whole credits, 1 or more)`);
|
|
29816
30076
|
}
|
|
29817
|
-
options.
|
|
30077
|
+
if (flag === "--max-credits") options.maxCredits = n;
|
|
30078
|
+
else options.maxCoins = n;
|
|
29818
30079
|
break;
|
|
29819
30080
|
}
|
|
29820
30081
|
case "--timeout": {
|
|
@@ -30112,9 +30373,9 @@ async function main() {
|
|
|
30112
30373
|
await runBudget(parsed.options);
|
|
30113
30374
|
break;
|
|
30114
30375
|
// The in-game model lane: is it live here, what does one call really
|
|
30115
|
-
// cost, what should the game declare. `bench`
|
|
30116
|
-
//
|
|
30117
|
-
//
|
|
30376
|
+
// cost, what ceiling should the game declare. `bench` spends the
|
|
30377
|
+
// builder's own credits OUTSIDE the asset allowance, so it carries its
|
|
30378
|
+
// own --max-credits/--user-approved gate.
|
|
30118
30379
|
case "llm":
|
|
30119
30380
|
await runLlm(parsed.options);
|
|
30120
30381
|
break;
|