@genex-ai/cli-demo 1.36.6 → 1.36.7-dev.889
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{blender-mcp-QVEGHE5L.js → blender-mcp-WSURPZET.js} +2 -2
- package/dist/{blender-serve-66OHC4ML.js → blender-serve-EH2MOW3U.js} +1 -1
- package/dist/{chunk-4BBTPRJW.js → chunk-EABIVAGT.js} +2 -2
- package/dist/{chunk-QI7FIYBY.js → chunk-K4EHXOZK.js} +1 -1
- package/dist/index.js +448 -216
- package/package.json +1 -1
- package/templates/skills/genex-game-director/SKILL.md +1 -1
- package/templates/skills/genex-getting-started/SKILL.md +2 -2
- package/templates/skills/genex-llm-in-games/SKILL.md +209 -72
- package/templates/skills/genex-llm-in-games/references/pricing.md +126 -85
- package/templates/skills/genex-monetization/SKILL.md +160 -222
- package/templates/skills/genex-tool-llm/SKILL.md +11 -9
- package/templates/skills/genex-updates/SKILL.md +1 -1
package/dist/index.js
CHANGED
|
@@ -57,7 +57,7 @@ import {
|
|
|
57
57
|
writeSecretFile,
|
|
58
58
|
writeUserToken,
|
|
59
59
|
writeWorkspace
|
|
60
|
-
} from "./chunk-
|
|
60
|
+
} from "./chunk-EABIVAGT.js";
|
|
61
61
|
import {
|
|
62
62
|
CLI_CHANNEL,
|
|
63
63
|
DEFAULT_API_URL,
|
|
@@ -78,7 +78,7 @@ import {
|
|
|
78
78
|
normalizeApiOrigin,
|
|
79
79
|
originKey,
|
|
80
80
|
resolveAgentTargets
|
|
81
|
-
} from "./chunk-
|
|
81
|
+
} from "./chunk-K4EHXOZK.js";
|
|
82
82
|
|
|
83
83
|
// src/instrument.ts
|
|
84
84
|
import * as Sentry from "@sentry/node";
|
|
@@ -981,7 +981,7 @@ Important note: put soul into your creations, with many details and love. Aim to
|
|
|
981
981
|
21. A turn ends in exactly one of two ways: on a question the player must answer before the next step can be chosen, or on a handoff \u2014 the draft link, what changed, what to try, and what is still open. Never close a turn on a promise. "I'll keep building", "while I keep working", "adding that now" are things you say and then DO before you stop; if you are stopping, say that you have stopped and what remains. While the Build plan's \`Now:\` line has open work and no answer is needed from the player, the turn is not over: preview, hand off, and start the next milestone in the same turn, until the requested outcome is reached or the platform's own budget ends the session. If a stop was forced on you anyway, the first line of your next turn is the promise you left and where it stands.
|
|
982
982
|
22. **What the player sees must read as the thing it is**, in this game's own style: a building as that building, a person as a person, a prop as that prop, a surface as its material \u2014 and a coloured box, a capsule, or a flat grey block standing in for one of them is never its finished version. Generation is the default route to that bar wherever code will not honestly reach it: the player's character, whatever the request names, whatever the player walks up to, enters, or interacts with, and the music and sound underneath \u2014 reach for the belt on your own judgment, without stopping to ask; a status line saying what you queued is enough, and \`--no-wait\` keeps you building while it lands. Procedural code is a first-class engine for what is structural, repeated, distant, or parametric \u2014 terrain, sky, fences, paving, modular kits, filler \u2014 and for anything else only when the result meets the same bar and you have looked at it in a capture before its row says \`landed\`. Start with one reusable implementation per distinct required gameplay or visual role. Background populations default to reusable procedural bodies and motion or compatible existing rigged assets. Catalog clips need a compatible skeleton and may be paid. Reserve paid custom bodies for the player and characters examined or interacted with closely, while honoring explicit user requests. Add variants only for an unmet requirement; unspent allowance is not unfinished work. Allowance questions use timeoutPolicy no-consent: silence does not raise the allowance. Record role, reuse and completion criteria in the existing Assets table. Decide per object, not per category, and write the route on each Assets row in \`DESIGN.md\` \u2014 the paid flow for generated pieces, the procedural flow with its local path for code-built ones. An Assets table with no generation in it is a decision, not a default: record it as \`Generation: none \u2014 <why code alone reaches the bar here>\` or generate.
|
|
983
983
|
23. **The screenshots you take to look at your own work go in \`.genex/scratch/\`.** Captures, render comparisons, before/after strips, traces and metrics dumps are how you SEE what you built \u2014 they are not part of what you built, and \`genex preview\` pushes the whole folder to the game's source, every time, forever. \`.genex/scratch/\` already exists for this and is already ignored, so there is nothing to set up: write the capture as \`.genex/scratch/arena-before.png\` and read it back from there. Never put them in a folder of your own at the top level: \`progress/\`, \`reports/\`, \`shots/\` and their kind are pushed like source and become permanent weight in the game and in every remix of it \u2014 one real game reached 11 GB and 10,332 committed screenshots exactly this way. If you find such a folder already there, add it to \`.gitignore\`; never delete the player's files (law 18). One file in \`.genex/scratch/\` IS read, by the platform: \`.genex/scratch/cover.png\` is the game's cover \u2014 the frame you would put on its poster: real gameplay at its signature moment, well lit, no menus, popups or busy HUD, landscape 16:9, and never any text (the page sets the title beside it). The next \`genex preview\` makes it the cover; save a better moment over it whenever one comes along, and if your browser tool can set the cover itself, use that instead. Before the first publish, and whenever the game's look changes, load \`$genex-cover\`: it chooses, stages, lights and judges that frame.
|
|
984
|
-
24. **Benchmark before declaring an in-game LLM
|
|
984
|
+
24. **Benchmark before declaring an in-game LLM ceiling.** If the game calls a language model while the player plays \u2014 an NPC answering in its own words, a quest written for this save, a judge reading what the player typed \u2014 the PLAYER pays for it from their Genex credits, billed as used, and the game declares no price, only a per-call CEILING: \`maxCredits\` on \`generate()\`, \`perCallMaxCredits\` on a standing budget. That ceiling bounds what one call may cost AND funds how long its answer may be, so never pick it by judgement or from a vendor's rate card. Run the real prompt on your own credits \u2014 \`npx genex llm bench "<the prompt>" --schema <file> --samples 3 --max-credits <n> --user-approved\` \u2014 and declare the figures it prints, recorded in \`DESIGN.md\` with its model and date; re-benchmark whenever the prompt, the schema or the model changes, because a prompt edit changes the ceiling. Check the lane first with \`npx genex llm models\`: where it is off the routes 404, so build the feature behind a graceful unavailable state and say so in the handoff. A repeated call is a standing budget the player approves once (\`requestSpendGrant()\`, then \`generate({ grantId })\`), never a popup per turn, and its disclosed call rate and credits per period are computed from the game's own loop and the bench's measured average. The coin spellings (\`estimateCoins\`, \`perCallMaxCoins\`) are deprecated \u2014 write the credit names. Model output is data the game validates against its own expectation \u2014 never executed, and never authority over coin, credits, items, entitlements or rewards. Load \`$genex-llm-in-games\` before writing any of it.
|
|
985
985
|
${CONTRACT_END}
|
|
986
986
|
`;
|
|
987
987
|
var TOOLS_CONTRACT_HEAD = `# Genex Tools (always in effect in this folder)
|
|
@@ -995,7 +995,7 @@ var TOOLS_CONTRACT_OFFER = `5. **When the game is built and playable \u2014 neve
|
|
|
995
995
|
var TOOLS_CONTRACT_HOSTED_LAWS = `5. **This game is hosted on Genex now, and publishing is glue \u2014 never a rebuild.** The game ships exactly as the user built it: \`initEmbed()\` first in the boot code (the \`genex-threejs-embed-auth\` card has the call; the renderer draws before any \`await waitForPlayer()\`), a static build into \`dist/\` with relative asset paths, then \`npx genex preview\`. Never restructure, reformat or "improve" the game to ship it, never scaffold a new app around it, and never start a design document or a build plan for it \u2014 the user's own process is theirs. Load the \`genex-tool-publish\` card before the first preview; it owns the vocabulary, the links and the limits.
|
|
996
996
|
6. **Ship first, then report.** \`preview\` prints a preflight \u2014 phone memory, a missing volume slider, an asset nobody wired, a viewport line. It never blocks a deploy, so push the build as it is, hand over the link, THEN relay each preflight line to the user in one plain sentence with an offer to fix it, and fix one only on their yes. The preflight is a report for the user, not a to-do list for you.
|
|
997
997
|
7. **The link is the game's page, and there are two versions.** After every preview give the user \`<dashboard>/draft/<slug>\` \u2014 \`dashboardOrigins[0]\` and \`slug\` from \`.genex/project.json\` \u2014 never localhost, a file path or the bare play origin. \`preview\` updates the draft and never touches what players are on; the first release is \`npx genex publish\` (it lists the game); after that "publish it", "update it" and "yes" all mean \`npx genex promote\` \u2014 the exact draft build, no rebuild. Ask once per round of work, in one line, and keep working while you wait.`;
|
|
998
|
-
var TOOLS_CONTRACT_LLM_OFFER = `A model running while people play is built into the Genex platform \u2014 the player pays, with Genex
|
|
998
|
+
var TOOLS_CONTRACT_LLM_OFFER = `A model running while people play is built into the Genex platform \u2014 the player pays, with their Genex credits or their own Claude/ChatGPT subscription, and approves it on a Genex sheet; your game just calls \`generate()\`. Want it that way?`;
|
|
999
999
|
function toolsContractLlmLaw(n, hosted) {
|
|
1000
1000
|
const onYes = hosted ? "On a yes, load `$genex-tool-llm` and then `$genex-llm-in-games`, which owns the build." : "On a yes, load `$genex-tool-llm`, which owns the rest: it checks the lane with `npx genex llm models` first, then runs `npx genex init --convert` (a hosted game is a static browser build \u2014 a local server holding a key can never ship).";
|
|
1001
1001
|
return `${n}. **A model running while people PLAY is a platform feature \u2014 never something you wire onto the user's own meter.** When a request implies one (NPCs that talk or decide in their own words, content written from what the player types, a prompt box in the game, "let the player pick a model"), recognise it, say in ONE line: "${TOOLS_CONTRACT_LLM_OFFER}" \u2014 then ASK and wait. ${onYes} On a no, build the authored version instead. Either way, never ship a game that calls a model on a key or an account of the user's own: every visitor would spend their money with nobody approving it, and the credential is readable in the bundle. Player-funded generation is text and JSON only \u2014 3D, images, video and audio stay the asset lanes above, on the user's meter.`;
|
|
@@ -23441,7 +23441,7 @@ async function runBlender(opts) {
|
|
|
23441
23441
|
return 1;
|
|
23442
23442
|
}
|
|
23443
23443
|
if (sub === "serve") {
|
|
23444
|
-
const { serveLocalBlender } = await import("./blender-serve-
|
|
23444
|
+
const { serveLocalBlender } = await import("./blender-serve-EH2MOW3U.js");
|
|
23445
23445
|
const port = Number(process.env.GENEX_BLENDER_PORT ?? 8088);
|
|
23446
23446
|
log.step(`Starting a local Blender service on port ${port}`);
|
|
23447
23447
|
log.plain(
|
|
@@ -23450,7 +23450,7 @@ async function runBlender(opts) {
|
|
|
23450
23450
|
return serveLocalBlender({ port, log });
|
|
23451
23451
|
}
|
|
23452
23452
|
if (sub === "mcp") {
|
|
23453
|
-
const { runBlenderMcp } = await import("./blender-mcp-
|
|
23453
|
+
const { runBlenderMcp } = await import("./blender-mcp-WSURPZET.js");
|
|
23454
23454
|
return runBlenderMcp();
|
|
23455
23455
|
}
|
|
23456
23456
|
if (sub === "seat") {
|
|
@@ -23958,7 +23958,12 @@ import fs35 from "fs/promises";
|
|
|
23958
23958
|
import path36 from "path";
|
|
23959
23959
|
import { randomUUID as randomUUID2 } from "crypto";
|
|
23960
23960
|
var SUBS3 = ["models", "bench", "price", "status", "cancel"];
|
|
23961
|
-
var LLM_BENCH_APPROVAL_REQUIRED = "STOP: `genex llm bench` runs the real model and spends YOUR OWN
|
|
23961
|
+
var LLM_BENCH_APPROVAL_REQUIRED = "STOP: `genex llm bench` runs the real model and spends YOUR OWN credits \u2014 your balance, not a player's, and a spend this build's asset allowance does not count. Re-run with --max-credits <n> --user-approved only after the person at the keyboard agreed to the number.";
|
|
23962
|
+
var LLM_MAX_COINS_DEPRECATED = "--max-coins is deprecated: it is read as --max-credits, the same number \u2014 a bench spends credits now.";
|
|
23963
|
+
function benchBalanceShortLine(spendable, maxCredits) {
|
|
23964
|
+
return `Your spendable credits (${spendable}) cannot cover one attempt at --max-credits ${maxCredits} \u2014 nothing was sent, nothing was spent. Add credits, or re-run with a lower --max-credits once the person at the keyboard agrees to it.`;
|
|
23965
|
+
}
|
|
23966
|
+
var LLM_BENCH_UNVERIFIED_LINE = "Credits pay only for a verified email, and this account's is not verified yet (credits_unverified) \u2014 nothing was sent, nothing was spent. Verify it, then re-run.";
|
|
23962
23967
|
var LLM_LANE_OFF_LINE = "The runtime LLM lane is off on this stand \u2014 in-game generate() answers 404 here, and nothing can be benchmarked.";
|
|
23963
23968
|
var LLM_TOOLS_CONVERT_HINT = [
|
|
23964
23969
|
"In-game model calls need a hosted game. This folder is a Genex Tools workspace, so nothing here",
|
|
@@ -23971,7 +23976,7 @@ function serverSlotCounts(body) {
|
|
|
23971
23976
|
return { held: body.slotsHeld, limit: body.slotLimit };
|
|
23972
23977
|
}
|
|
23973
23978
|
var PENDING_SLOT_RELEASE_MINUTES = 10;
|
|
23974
|
-
var LLM_STATUS_LEGEND = `"awaiting bill" = the call has stopped but the provider's bill is not final, so
|
|
23979
|
+
var LLM_STATUS_LEGEND = `"awaiting bill" = the call has stopped but the provider's bill is not final, so what it reserved stays held until the bill resolves; such a row stops holding a slot ${PENDING_SLOT_RELEASE_MINUTES} minutes after dispatch.`;
|
|
23975
23980
|
var LLM_STATUS_NO_COUNT_LINE = "This stand does not report how many slots are held across your projects, so no count is shown.";
|
|
23976
23981
|
var LLM_CANCEL_AWAITING_BILL = "this call already stopped; its bill is awaiting the provider";
|
|
23977
23982
|
var LLM_CANCEL_ALREADY_FINAL = "this call is already final";
|
|
@@ -23985,11 +23990,33 @@ var DEFAULT_SAMPLE_TIMEOUT_SEC = 180;
|
|
|
23985
23990
|
function parseProviderKey(value) {
|
|
23986
23991
|
return value === "ok" || value === "rejected" || value === "unknown" ? value : null;
|
|
23987
23992
|
}
|
|
23993
|
+
function bpsOrNull(value) {
|
|
23994
|
+
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : null;
|
|
23995
|
+
}
|
|
23996
|
+
function standingCreditGrantOrNull(value) {
|
|
23997
|
+
if (!value || typeof value !== "object") return null;
|
|
23998
|
+
const v = value;
|
|
23999
|
+
const whole = (x) => typeof x === "number" && Number.isSafeInteger(x) && x >= 0;
|
|
24000
|
+
if (!whole(v.minCredits) || !whole(v.maxCredits) || !whole(v.stepCredits) || !whole(v.defaultCredits)) return null;
|
|
24001
|
+
return {
|
|
24002
|
+
minCredits: v.minCredits,
|
|
24003
|
+
maxCredits: v.maxCredits,
|
|
24004
|
+
stepCredits: v.stepCredits,
|
|
24005
|
+
defaultCredits: v.defaultCredits,
|
|
24006
|
+
...whole(v.minUsdCents) ? { minUsdCents: v.minUsdCents } : {},
|
|
24007
|
+
...whole(v.maxUsdCents) ? { maxUsdCents: v.maxUsdCents } : {},
|
|
24008
|
+
...whole(v.stepUsdCents) ? { stepUsdCents: v.stepUsdCents } : {}
|
|
24009
|
+
};
|
|
24010
|
+
}
|
|
23988
24011
|
async function readRuntimeLane(apiUrl, token) {
|
|
23989
24012
|
const empty = (state) => ({
|
|
23990
24013
|
state,
|
|
23991
24014
|
models: null,
|
|
23992
24015
|
recommendedDeclaredHeadroomBps: null,
|
|
24016
|
+
currency: null,
|
|
24017
|
+
feeBps: null,
|
|
24018
|
+
usdCentsPerCredit: null,
|
|
24019
|
+
standingCreditGrant: null,
|
|
23993
24020
|
externalProviders: null,
|
|
23994
24021
|
total: null,
|
|
23995
24022
|
featuredCount: null,
|
|
@@ -24008,7 +24035,11 @@ async function readRuntimeLane(apiUrl, token) {
|
|
|
24008
24035
|
return {
|
|
24009
24036
|
state: "live",
|
|
24010
24037
|
models: body.models,
|
|
24011
|
-
recommendedDeclaredHeadroomBps:
|
|
24038
|
+
recommendedDeclaredHeadroomBps: bpsOrNull(body.recommendedDeclaredHeadroomBps),
|
|
24039
|
+
currency: body.currency === "credits" || body.currency === "coins" ? body.currency : null,
|
|
24040
|
+
feeBps: bpsOrNull(body.feeBps),
|
|
24041
|
+
usdCentsPerCredit: typeof body.usdCentsPerCredit === "number" && Number.isSafeInteger(body.usdCentsPerCredit) && body.usdCentsPerCredit >= 1 ? body.usdCentsPerCredit : null,
|
|
24042
|
+
standingCreditGrant: standingCreditGrantOrNull(body.standingCreditGrant),
|
|
24012
24043
|
externalProviders: Array.isArray(body.externalProviders) ? body.externalProviders : null,
|
|
24013
24044
|
total: typeof body.total === "number" && Number.isFinite(body.total) ? body.total : null,
|
|
24014
24045
|
featuredCount: typeof body.featuredCount === "number" && Number.isFinite(body.featuredCount) ? body.featuredCount : null,
|
|
@@ -24024,18 +24055,45 @@ function percentile2(values, p) {
|
|
|
24024
24055
|
const rank2 = Math.ceil(p / 100 * sorted.length);
|
|
24025
24056
|
return sorted[Math.min(sorted.length - 1, Math.max(0, rank2 - 1))];
|
|
24026
24057
|
}
|
|
24027
|
-
function
|
|
24028
|
-
|
|
24058
|
+
function percentilePicos(values, p) {
|
|
24059
|
+
if (values.length === 0) return null;
|
|
24060
|
+
const sorted = [...values].sort((a, b) => a < b ? -1 : a > b ? 1 : 0);
|
|
24061
|
+
const rank2 = Math.ceil(p / 100 * sorted.length);
|
|
24062
|
+
return sorted[Math.min(sorted.length - 1, Math.max(0, rank2 - 1))];
|
|
24029
24063
|
}
|
|
24030
|
-
|
|
24031
|
-
|
|
24064
|
+
var PICOS_PER_CENT = 10000000000n;
|
|
24065
|
+
function ceilingCreditsFor(costPicos, terms) {
|
|
24066
|
+
if (costPicos <= 0n) return 1;
|
|
24067
|
+
const num = costPicos * BigInt(1e4 + terms.feeBps) * BigInt(1e4 + terms.headroomBps);
|
|
24068
|
+
const den = 100000000n * BigInt(terms.usdCentsPerCredit) * PICOS_PER_CENT;
|
|
24069
|
+
return Math.max(1, Number((num + den - 1n) / den));
|
|
24070
|
+
}
|
|
24071
|
+
function averageCreditsPerCall(costsPicos, terms) {
|
|
24072
|
+
if (costsPicos.length === 0) return null;
|
|
24073
|
+
const sum = costsPicos.reduce((a, b) => a + b, 0n);
|
|
24074
|
+
return Number(sum * BigInt(1e4 + terms.feeBps)) / Number(BigInt(costsPicos.length) * 10000n * BigInt(terms.usdCentsPerCredit) * PICOS_PER_CENT);
|
|
24075
|
+
}
|
|
24076
|
+
function perCallEstimateCreditsFor(costsPicos, terms) {
|
|
24077
|
+
if (costsPicos.length === 0) return null;
|
|
24078
|
+
const sum = costsPicos.reduce((a, b) => a + b, 0n);
|
|
24079
|
+
const num = sum * BigInt(1e4 + terms.feeBps);
|
|
24080
|
+
const den = BigInt(costsPicos.length) * 10000n * BigInt(terms.usdCentsPerCredit) * PICOS_PER_CENT;
|
|
24081
|
+
return Math.max(1, Number((num + den - 1n) / den));
|
|
24082
|
+
}
|
|
24083
|
+
function formatCredits(value) {
|
|
24084
|
+
if (!Number.isFinite(value)) return "\u2014";
|
|
24085
|
+
if (value >= 100) return String(Math.round(value));
|
|
24086
|
+
return String(Number(value.toPrecision(value < 1 ? 2 : 3)));
|
|
24087
|
+
}
|
|
24088
|
+
function usdFromPicos(picos) {
|
|
24089
|
+
if (picos === null || picos === void 0) return "unknown";
|
|
24090
|
+
const value = typeof picos === "bigint" ? picos : /^\d+$/.test(picos) ? BigInt(picos) : null;
|
|
24091
|
+
if (value === null) return "unknown";
|
|
24092
|
+
return `$${(Number(value) / 1e12).toFixed(6)}`;
|
|
24032
24093
|
}
|
|
24033
24094
|
function usdPerMillion(value) {
|
|
24034
24095
|
return `$${value.toFixed(value < 1 ? 4 : 2)}`;
|
|
24035
24096
|
}
|
|
24036
|
-
function usd(value) {
|
|
24037
|
-
return value === null ? "unknown" : `$${value.toFixed(6)}`;
|
|
24038
|
-
}
|
|
24039
24097
|
async function runLlm(opts = {}) {
|
|
24040
24098
|
const log = createLogger({ quiet: opts.quiet || opts.json });
|
|
24041
24099
|
const cwd = opts.cwd ?? process.cwd();
|
|
@@ -24061,16 +24119,23 @@ async function runLlm(opts = {}) {
|
|
|
24061
24119
|
return;
|
|
24062
24120
|
}
|
|
24063
24121
|
if (sub === "bench") {
|
|
24064
|
-
if (opts.maxCoins
|
|
24122
|
+
if (opts.maxCoins !== void 0) process.stderr.write(`${c.yellow("!")} ${LLM_MAX_COINS_DEPRECATED}
|
|
24123
|
+
`);
|
|
24124
|
+
if (opts.maxCoins !== void 0 && opts.maxCredits !== void 0 && opts.maxCoins !== opts.maxCredits) {
|
|
24125
|
+
fail5("llm bench", `Pass --max-credits alone \u2014 --max-coins is its deprecated name, and the two disagree (${opts.maxCredits} vs ${opts.maxCoins}).`);
|
|
24126
|
+
return;
|
|
24127
|
+
}
|
|
24128
|
+
const approvedCredits = opts.maxCredits ?? opts.maxCoins;
|
|
24129
|
+
if (approvedCredits === void 0 || opts.userApproved !== true) {
|
|
24065
24130
|
fail5("llm bench", LLM_BENCH_APPROVAL_REQUIRED);
|
|
24066
24131
|
return;
|
|
24067
24132
|
}
|
|
24068
|
-
if (!Number.isInteger(
|
|
24069
|
-
fail5("llm bench", `--max-
|
|
24133
|
+
if (!Number.isInteger(approvedCredits) || approvedCredits < 1) {
|
|
24134
|
+
fail5("llm bench", `--max-credits takes a whole number of credits, 1 or more (got ${String(approvedCredits)}).`);
|
|
24070
24135
|
return;
|
|
24071
24136
|
}
|
|
24072
24137
|
if (!opts.benchPrompt?.trim()) {
|
|
24073
|
-
fail5("llm bench", '`genex llm bench` needs the prompt your game would send, e.g. genex llm bench "<prompt>" --max-
|
|
24138
|
+
fail5("llm bench", '`genex llm bench` needs the prompt your game would send, e.g. genex llm bench "<prompt>" --max-credits <n> --user-approved.');
|
|
24074
24139
|
return;
|
|
24075
24140
|
}
|
|
24076
24141
|
if (opts.jsonOutput && opts.textOutput) {
|
|
@@ -24162,6 +24227,10 @@ async function reportModels(apiUrl, token, opts, log, toolsOnly = false) {
|
|
|
24162
24227
|
models: null,
|
|
24163
24228
|
externalProviders: null,
|
|
24164
24229
|
recommendedDeclaredHeadroomBps: null,
|
|
24230
|
+
currency: null,
|
|
24231
|
+
feeBps: null,
|
|
24232
|
+
usdCentsPerCredit: null,
|
|
24233
|
+
standingCreditGrant: null,
|
|
24165
24234
|
total: null,
|
|
24166
24235
|
featuredCount: null,
|
|
24167
24236
|
// The stand was never asked, so the verdict is unknown — but the key is
|
|
@@ -24187,6 +24256,12 @@ async function reportModels(apiUrl, token, opts, log, toolsOnly = false) {
|
|
|
24187
24256
|
models: lane.models,
|
|
24188
24257
|
externalProviders: lane.externalProviders,
|
|
24189
24258
|
recommendedDeclaredHeadroomBps: lane.recommendedDeclaredHeadroomBps,
|
|
24259
|
+
// The credit terms a call is billed on, the server's — null on a stand
|
|
24260
|
+
// that predates credit billing, never a number the CLI filled in.
|
|
24261
|
+
currency: lane.currency,
|
|
24262
|
+
feeBps: lane.feeBps,
|
|
24263
|
+
usdCentsPerCredit: lane.usdCentsPerCredit,
|
|
24264
|
+
standingCreditGrant: lane.standingCreditGrant,
|
|
24190
24265
|
total: lane.total ?? lane.models?.length ?? null,
|
|
24191
24266
|
featuredCount: lane.featuredCount ?? (lane.models ? lane.models.filter((m) => m.featured).length : null),
|
|
24192
24267
|
// `null` when the lane is not live or the server predates the probe.
|
|
@@ -24233,8 +24308,9 @@ async function reportModels(apiUrl, token, opts, log, toolsOnly = false) {
|
|
|
24233
24308
|
);
|
|
24234
24309
|
for (const m of shown) {
|
|
24235
24310
|
const plan = m.personalPlan ? ` \xB7 personal plan: ${m.personalPlan}` : "";
|
|
24311
|
+
const kind = m.kind === "typesafe" ? " \xB7 judge (classifier): judge() under a budget, not generate() or bench" : "";
|
|
24236
24312
|
log.plain(
|
|
24237
|
-
` ${c.cyan(m.id)} ${m.label} \u2014 ${usdPerMillion(m.inputUsdPerMillion)} in / ${usdPerMillion(m.outputUsdPerMillion)} out per million tokens${plan}`
|
|
24313
|
+
` ${c.cyan(m.id)} ${m.label} \u2014 ${usdPerMillion(m.inputUsdPerMillion)} in / ${usdPerMillion(m.outputUsdPerMillion)} out per million tokens${plan}${kind}`
|
|
24238
24314
|
);
|
|
24239
24315
|
}
|
|
24240
24316
|
if (hiddenCount > 0) {
|
|
@@ -24250,74 +24326,104 @@ async function reportModels(apiUrl, token, opts, log, toolsOnly = false) {
|
|
|
24250
24326
|
}
|
|
24251
24327
|
log.plain("");
|
|
24252
24328
|
if (lane.recommendedDeclaredHeadroomBps === null) {
|
|
24253
|
-
log.dim(" This stand serves no recommended headroom, so no
|
|
24329
|
+
log.dim(" This stand serves no recommended headroom, so no ceiling can be recommended from a bench.");
|
|
24254
24330
|
} else {
|
|
24255
|
-
log.dim(` Recommended headroom over benchmarked
|
|
24331
|
+
log.dim(` Recommended headroom over a benchmarked call's cost on this stand: ${lane.recommendedDeclaredHeadroomBps} bps.`);
|
|
24332
|
+
}
|
|
24333
|
+
if (lane.feeBps === null) {
|
|
24334
|
+
log.dim(" This stand serves no platform fee \u2014 it predates credit billing, so no ceiling can be recommended from a bench.");
|
|
24335
|
+
} else {
|
|
24336
|
+
log.dim(` In-game calls are billed in the player's credits, as used: the provider's cost plus this stand's platform fee (${lane.feeBps} bps).`);
|
|
24256
24337
|
}
|
|
24257
24338
|
if (toolsOnly) {
|
|
24258
24339
|
printConvertHint();
|
|
24259
24340
|
return;
|
|
24260
24341
|
}
|
|
24261
|
-
log.dim(" Those per-million rates are the PROVIDER's; what
|
|
24262
|
-
log.dim(`
|
|
24342
|
+
log.dim(" Those per-million rates are the PROVIDER's list; what one call of YOUR prompt really costs,");
|
|
24343
|
+
log.dim(` and the per-call ceiling to declare, only a real call measures: ${c.cyan('genex llm bench "<prompt>" --max-credits <n> --user-approved')}`);
|
|
24263
24344
|
}
|
|
24264
24345
|
function wholeOrNull(value) {
|
|
24265
24346
|
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : null;
|
|
24266
24347
|
}
|
|
24348
|
+
function costPicosOf(usage) {
|
|
24349
|
+
if (typeof usage?.costUsdPicos === "string" && /^\d+$/.test(usage.costUsdPicos)) return usage.costUsdPicos;
|
|
24350
|
+
if (typeof usage?.costUsd === "number" && Number.isFinite(usage.costUsd) && usage.costUsd >= 0) {
|
|
24351
|
+
return String(BigInt(Math.round(usage.costUsd * 1e12)));
|
|
24352
|
+
}
|
|
24353
|
+
return null;
|
|
24354
|
+
}
|
|
24267
24355
|
function sampleFrom(id, settled) {
|
|
24268
24356
|
const usage = settled?.usage;
|
|
24357
|
+
const picos = costPicosOf(usage);
|
|
24269
24358
|
return {
|
|
24270
24359
|
id,
|
|
24271
24360
|
status: settled?.status ?? null,
|
|
24272
24361
|
billingStatus: settled?.billingStatus ?? null,
|
|
24273
|
-
|
|
24274
|
-
costUsd: typeof usage?.costUsd === "number" ? usage.costUsd : null,
|
|
24362
|
+
chargedCredits: wholeOrNull(settled?.chargedCredits),
|
|
24363
|
+
costUsd: typeof usage?.costUsd === "number" ? usage.costUsd : picos !== null ? Number(picos) / 1e12 : null,
|
|
24364
|
+
costUsdPicos: picos,
|
|
24275
24365
|
error: settled?.error ?? (settled ? null : "timed_out"),
|
|
24276
24366
|
providerMessage: typeof settled?.providerMessage === "string" && settled.providerMessage.trim() ? settled.providerMessage.trim() : null,
|
|
24277
24367
|
inputTokens: wholeOrNull(usage?.inputTokens),
|
|
24278
24368
|
outputTokens: wholeOrNull(usage?.outputTokens),
|
|
24279
24369
|
maxOutputTokens: wholeOrNull(usage?.maxOutputTokens),
|
|
24280
|
-
|
|
24370
|
+
fitCredits: typeof usage?.fitCredits === "number" && Number.isSafeInteger(usage.fitCredits) && usage.fitCredits >= 1 ? usage.fitCredits : null,
|
|
24281
24371
|
fitExceedsCap: typeof usage?.fitExceedsCap === "boolean" ? usage.fitExceedsCap : null,
|
|
24282
24372
|
truncated: typeof usage?.truncated === "boolean" ? usage.truncated : settled?.error === "provider_token_limit" ? true : null
|
|
24283
24373
|
};
|
|
24284
24374
|
}
|
|
24285
24375
|
var PROVIDER_TOKEN_LIMIT = "provider_token_limit";
|
|
24286
|
-
function benchRecommendation(results,
|
|
24287
|
-
const settled = results.filter(
|
|
24288
|
-
|
|
24289
|
-
|
|
24290
|
-
const
|
|
24291
|
-
const
|
|
24376
|
+
function benchRecommendation(results, terms) {
|
|
24377
|
+
const settled = results.filter(
|
|
24378
|
+
(r) => r.status === "succeeded" && r.billingStatus === "final" && r.chargedCredits !== null && r.costUsdPicos !== null
|
|
24379
|
+
);
|
|
24380
|
+
const costs = settled.map((r) => BigInt(r.costUsdPicos));
|
|
24381
|
+
const charged = settled.map((r) => r.chargedCredits);
|
|
24382
|
+
const fits = settled.map((r) => r.fitCredits).filter((v) => typeof v === "number");
|
|
24383
|
+
const costP95 = percentilePicos(costs, 95);
|
|
24384
|
+
const costMax = costs.length ? costs.reduce((a, b) => b > a ? b : a) : null;
|
|
24292
24385
|
const fitP95 = percentile2(fits, 95);
|
|
24293
24386
|
const fitMax = fits.length ? Math.max(...fits) : null;
|
|
24294
24387
|
const exceedsStandCap = results.some((r) => r.fitExceedsCap === true);
|
|
24295
24388
|
const cutOffSamples = results.filter((r) => r.error === PROVIDER_TOKEN_LIMIT && r.fitExceedsCap !== true).length;
|
|
24296
24389
|
const noRecommendationReason = exceedsStandCap ? "answer_exceeds_stand_cap" : cutOffSamples > 0 ? "answer_cut_off" : null;
|
|
24297
|
-
const
|
|
24298
|
-
const
|
|
24299
|
-
const
|
|
24390
|
+
const { headroomBps, feeBps, usdCentsPerCredit } = terms;
|
|
24391
|
+
const missingTerms = headroomBps === null ? "headroom" : feeBps === null || usdCentsPerCredit === null ? "credit_terms" : null;
|
|
24392
|
+
const full = headroomBps !== null && feeBps !== null && usdCentsPerCredit !== null ? { headroomBps, feeBps, usdCentsPerCredit } : null;
|
|
24393
|
+
const billing = feeBps !== null && usdCentsPerCredit !== null ? { feeBps, usdCentsPerCredit } : null;
|
|
24394
|
+
const costBased = full !== null && costP95 !== null ? ceilingCreditsFor(costP95, full) : null;
|
|
24395
|
+
const recommended = costBased === null || noRecommendationReason !== null ? null : Math.max(costBased, fitP95 ?? 0);
|
|
24396
|
+
const ceiling = recommended === null || costMax === null || full === null ? null : Math.max(ceilingCreditsFor(costMax, full), recommended, fitMax ?? 0);
|
|
24397
|
+
const average = billing !== null ? averageCreditsPerCall(costs, billing) : null;
|
|
24398
|
+
const estimate = ceiling === null || billing === null ? null : Math.min(ceiling, perCallEstimateCreditsFor(costs, billing) ?? 1);
|
|
24300
24399
|
return {
|
|
24400
|
+
costs,
|
|
24301
24401
|
charged,
|
|
24302
|
-
|
|
24303
|
-
|
|
24304
|
-
|
|
24402
|
+
costP50: percentilePicos(costs, 50),
|
|
24403
|
+
costP95,
|
|
24404
|
+
costMax,
|
|
24405
|
+
chargedP50: percentile2(charged, 50),
|
|
24406
|
+
chargedP95: percentile2(charged, 95),
|
|
24407
|
+
chargedMax: charged.length ? Math.max(...charged) : null,
|
|
24305
24408
|
fits,
|
|
24306
24409
|
fitP95,
|
|
24307
24410
|
fitMax,
|
|
24308
|
-
|
|
24309
|
-
|
|
24310
|
-
|
|
24311
|
-
|
|
24411
|
+
costBasedMaxCredits: costBased,
|
|
24412
|
+
recommendedMaxCredits: recommended,
|
|
24413
|
+
recommendedPerCallMaxCredits: ceiling,
|
|
24414
|
+
averageCreditsPerCall: average,
|
|
24415
|
+
recommendedPerCallEstimateCredits: estimate,
|
|
24416
|
+
lengthRaisedCeiling: recommended !== null && costBased !== null && recommended > costBased,
|
|
24312
24417
|
cutOffSamples,
|
|
24313
24418
|
exceedsStandCap,
|
|
24314
|
-
noRecommendationReason
|
|
24419
|
+
noRecommendationReason,
|
|
24420
|
+
missingTerms
|
|
24315
24421
|
};
|
|
24316
24422
|
}
|
|
24317
|
-
var LLM_LENGTH_RAISED_LINE = "The
|
|
24318
|
-
var LLM_EXCEEDS_STAND_CAP_LINE = "No
|
|
24319
|
-
function benchCutOffLine(cutOff,
|
|
24320
|
-
return `No
|
|
24423
|
+
var LLM_LENGTH_RAISED_LINE = "The ceiling also decides how long the answer may be: at the cost-based ceiling the answer would be cut off, so declare this one.";
|
|
24424
|
+
var LLM_EXCEEDS_STAND_CAP_LINE = "No ceiling recommended: this answer is longer than one call on this stand may produce. Ask for a shorter answer \u2014 fewer fields, shorter strings, a length the prompt states \u2014 and benchmark again.";
|
|
24425
|
+
function benchCutOffLine(cutOff, maxCredits) {
|
|
24426
|
+
return `No ceiling recommended: ${cutOff} sample${cutOff === 1 ? " was" : "s were"} cut off at this ceiling (--max-credits ${maxCredits}) before the answer was finished, so its real length is unknown. Re-run with a higher --max-credits.`;
|
|
24321
24427
|
}
|
|
24322
24428
|
var ACTIVE_STATUSES = /* @__PURE__ */ new Set(["requires_confirmation", "queued", "dispatching", "awaiting_external"]);
|
|
24323
24429
|
function rowState(row) {
|
|
@@ -24328,7 +24434,22 @@ function rowState(row) {
|
|
|
24328
24434
|
function rowHoldsSlot(row) {
|
|
24329
24435
|
if (typeof row.slotHeld === "boolean") return row.slotHeld;
|
|
24330
24436
|
const state = rowState(row);
|
|
24331
|
-
return state === "active" || state === "awaiting bill" && (row.reservedCoins ?? 0) > 0;
|
|
24437
|
+
return state === "active" || state === "awaiting bill" && ((row.reservedCoins ?? 0) > 0 || (row.reservedCredits ?? 0) > 0);
|
|
24438
|
+
}
|
|
24439
|
+
function rowMoney(row) {
|
|
24440
|
+
const credits = row.currency === "credits" || row.currency === void 0 && typeof row.reservedCredits === "number" && typeof row.reservedCoins !== "number";
|
|
24441
|
+
if (credits) {
|
|
24442
|
+
return {
|
|
24443
|
+
unit: "credits",
|
|
24444
|
+
reserved: typeof row.reservedCredits === "number" ? row.reservedCredits : null,
|
|
24445
|
+
charged: typeof row.chargedCredits === "number" ? row.chargedCredits : null
|
|
24446
|
+
};
|
|
24447
|
+
}
|
|
24448
|
+
return {
|
|
24449
|
+
unit: "coin",
|
|
24450
|
+
reserved: typeof row.reservedCoins === "number" ? row.reservedCoins : null,
|
|
24451
|
+
charged: typeof row.chargedCoins === "number" ? row.chargedCoins : null
|
|
24452
|
+
};
|
|
24332
24453
|
}
|
|
24333
24454
|
function providerRefusalStatus(error) {
|
|
24334
24455
|
const m = /^provider_http_(\d{3})$/.exec(error ?? "");
|
|
@@ -24347,7 +24468,7 @@ function formatAge(iso, now = Date.now()) {
|
|
|
24347
24468
|
}
|
|
24348
24469
|
async function runBench(args) {
|
|
24349
24470
|
const { apiUrl, token, projectId, cwd, opts, log } = args;
|
|
24350
|
-
const
|
|
24471
|
+
const maxCredits = opts.maxCredits ?? opts.maxCoins;
|
|
24351
24472
|
const samples = opts.samples ?? DEFAULT_BENCH_SAMPLES;
|
|
24352
24473
|
const prompt = opts.benchPrompt.trim();
|
|
24353
24474
|
const auth = { Authorization: `Bearer ${token}`, "Content-Type": "application/json" };
|
|
@@ -24369,7 +24490,7 @@ async function runBench(args) {
|
|
|
24369
24490
|
const lane = await readRuntimeLane(apiUrl, token);
|
|
24370
24491
|
if (lane.state !== "live" || !lane.models || lane.models.length === 0) {
|
|
24371
24492
|
if (lane.state === "off") {
|
|
24372
|
-
if (opts.json) writeJsonLine({ ...emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples,
|
|
24493
|
+
if (opts.json) writeJsonLine({ ...emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples, maxCredits, opts), status: "off", error: null });
|
|
24373
24494
|
else {
|
|
24374
24495
|
log.plain(LLM_LANE_OFF_LINE);
|
|
24375
24496
|
log.dim(" Nothing was spent.");
|
|
@@ -24396,17 +24517,27 @@ async function runBench(args) {
|
|
|
24396
24517
|
);
|
|
24397
24518
|
return;
|
|
24398
24519
|
}
|
|
24399
|
-
const
|
|
24520
|
+
const walletBefore = await fetchCreditsSnapshot(apiUrl, token);
|
|
24521
|
+
if (walletBefore?.emailVerified === false) {
|
|
24522
|
+
benchFailed(opts, log, LLM_BENCH_UNVERIFIED_LINE);
|
|
24523
|
+
return;
|
|
24524
|
+
}
|
|
24525
|
+
if (walletBefore !== null && walletBefore.spendable < maxCredits) {
|
|
24526
|
+
benchFailed(opts, log, benchBalanceShortLine(walletBefore.spendable, maxCredits));
|
|
24527
|
+
return;
|
|
24528
|
+
}
|
|
24529
|
+
const balanceBefore = walletBefore?.spendable ?? null;
|
|
24400
24530
|
if (!opts.json) {
|
|
24401
24531
|
log.plain(c.bold("genex llm bench"));
|
|
24402
24532
|
log.dim(` ${apiUrl}`);
|
|
24403
24533
|
log.plain("");
|
|
24404
24534
|
log.plain(` Model ${c.cyan(modelId)}`);
|
|
24405
24535
|
log.plain(` Samples ${samples} real attempt${samples === 1 ? "" : "s"}, ${outputFormat} output`);
|
|
24406
|
-
log.plain(` Approved ${
|
|
24536
|
+
log.plain(` Approved ${maxCredits} credit${maxCredits === 1 ? "" : "s"} per attempt \u2014 at worst ${maxCredits * samples} credits for this run, from your own balance`);
|
|
24407
24537
|
log.plain(
|
|
24408
|
-
` Balance ${balanceBefore === null ? "couldn't be read" : `${balanceBefore}
|
|
24538
|
+
` Balance ${balanceBefore === null ? "couldn't be read" : `${balanceBefore} credits spendable`}`
|
|
24409
24539
|
);
|
|
24540
|
+
log.dim(" Billed as used: each attempt is charged its real cost plus the platform fee, rounded up to a whole credit.");
|
|
24410
24541
|
log.plain("");
|
|
24411
24542
|
}
|
|
24412
24543
|
const base = `${apiUrl}/api/runtime/development/projects/${encodeURIComponent(projectId)}/generations`;
|
|
@@ -24432,7 +24563,7 @@ async function runBench(args) {
|
|
|
24432
24563
|
const res = await call(base, {
|
|
24433
24564
|
method: "POST",
|
|
24434
24565
|
body: JSON.stringify({
|
|
24435
|
-
|
|
24566
|
+
maxCredits,
|
|
24436
24567
|
request: {
|
|
24437
24568
|
idempotencyKey: `bench-${randomUUID2()}`,
|
|
24438
24569
|
modelId,
|
|
@@ -24462,13 +24593,14 @@ async function runBench(args) {
|
|
|
24462
24593
|
if (refusedAt !== null) providerRefused++;
|
|
24463
24594
|
if (!opts.json) {
|
|
24464
24595
|
const length = row.outputTokens !== null ? ` \xB7 ${row.outputTokens}${row.maxOutputTokens !== null ? ` of ${row.maxOutputTokens}` : ""} tokens out` : "";
|
|
24596
|
+
const charged = row.chargedCredits === null ? "\u2014" : String(row.chargedCredits);
|
|
24465
24597
|
log.plain(
|
|
24466
|
-
` ${row.status === "succeeded" ? c.green("\u2713") : c.yellow("!")} sample ${i + 1} ${
|
|
24598
|
+
` ${row.status === "succeeded" ? c.green("\u2713") : c.yellow("!")} sample ${i + 1} ${charged.padStart(4)} credit${row.chargedCredits === 1 ? "" : "s"} charged \xB7 cost ${usdFromPicos(row.costUsdPicos)}${length}${row.error ? ` \xB7 ${row.error}` : ""}`
|
|
24467
24599
|
);
|
|
24468
24600
|
if (refusedAt !== null) printProviderRefusal(log, refusedAt, row.providerMessage, row.status === "unknown");
|
|
24469
24601
|
if (row.error === PROVIDER_TOKEN_LIMIT) {
|
|
24470
24602
|
log.dim(
|
|
24471
|
-
row.fitExceedsCap === true ? " The answer was cut off at this stand's own output limit \u2014 no
|
|
24603
|
+
row.fitExceedsCap === true ? " The answer was cut off at this stand's own output limit \u2014 no ceiling makes room for it; it is not a sample." : ` The answer was cut off at this ceiling (--max-credits ${maxCredits}) before it was finished \u2014 it was charged, and it is not a sample.`
|
|
24472
24604
|
);
|
|
24473
24605
|
}
|
|
24474
24606
|
}
|
|
@@ -24476,16 +24608,19 @@ async function runBench(args) {
|
|
|
24476
24608
|
} finally {
|
|
24477
24609
|
process.removeListener("SIGINT", onSigint);
|
|
24478
24610
|
}
|
|
24479
|
-
const
|
|
24480
|
-
|
|
24481
|
-
|
|
24482
|
-
|
|
24483
|
-
|
|
24484
|
-
const
|
|
24611
|
+
const terms = {
|
|
24612
|
+
headroomBps: lane.recommendedDeclaredHeadroomBps,
|
|
24613
|
+
feeBps: lane.feeBps,
|
|
24614
|
+
usdCentsPerCredit: lane.usdCentsPerCredit
|
|
24615
|
+
};
|
|
24616
|
+
const rec = benchRecommendation(results, terms);
|
|
24617
|
+
const settledCount = rec.charged.length;
|
|
24485
24618
|
const lengthBlocked = rec.noRecommendationReason !== null;
|
|
24486
|
-
const balanceAfter = await
|
|
24619
|
+
const balanceAfter = (await fetchCreditsSnapshot(apiUrl, token))?.spendable ?? null;
|
|
24620
|
+
const picosOrNull = (v) => v === null ? null : String(v);
|
|
24487
24621
|
const record = {
|
|
24488
|
-
v:
|
|
24622
|
+
v: 2,
|
|
24623
|
+
currency: "credits",
|
|
24489
24624
|
ranAt: (/* @__PURE__ */ new Date()).toISOString(),
|
|
24490
24625
|
apiUrl,
|
|
24491
24626
|
projectId,
|
|
@@ -24494,28 +24629,34 @@ async function runBench(args) {
|
|
|
24494
24629
|
outputFormat,
|
|
24495
24630
|
samples,
|
|
24496
24631
|
completed: settledCount,
|
|
24497
|
-
|
|
24498
|
-
|
|
24499
|
-
|
|
24500
|
-
max,
|
|
24501
|
-
|
|
24502
|
-
|
|
24503
|
-
recommendedPerCallMaxCoins: ceiling,
|
|
24504
|
-
maxCoinsPerSample: maxCoins,
|
|
24505
|
-
fitCoins: rec.fits,
|
|
24632
|
+
maxCreditsPerSample: maxCredits,
|
|
24633
|
+
costUsdPicos: rec.costs.map(String),
|
|
24634
|
+
chargedCredits: rec.charged,
|
|
24635
|
+
cost: { p50: picosOrNull(rec.costP50), p95: picosOrNull(rec.costP95), max: picosOrNull(rec.costMax) },
|
|
24636
|
+
charged: { p50: rec.chargedP50, p95: rec.chargedP95, max: rec.chargedMax },
|
|
24637
|
+
fitCredits: rec.fits,
|
|
24506
24638
|
fitP95: rec.fitP95,
|
|
24507
|
-
|
|
24508
|
-
|
|
24639
|
+
fitMax: rec.fitMax,
|
|
24640
|
+
recommendedDeclaredHeadroomBps: terms.headroomBps,
|
|
24641
|
+
feeBps: terms.feeBps,
|
|
24642
|
+
usdCentsPerCredit: terms.usdCentsPerCredit,
|
|
24643
|
+
costBasedMaxCredits: rec.costBasedMaxCredits,
|
|
24644
|
+
recommendedMaxCredits: rec.recommendedMaxCredits,
|
|
24645
|
+
recommendedPerCallMaxCredits: rec.recommendedPerCallMaxCredits,
|
|
24646
|
+
averageCreditsPerCall: rec.averageCreditsPerCall,
|
|
24647
|
+
recommendedPerCallEstimateCredits: rec.recommendedPerCallEstimateCredits,
|
|
24648
|
+
lengthRaisedCeiling: rec.lengthRaisedCeiling,
|
|
24509
24649
|
cutOffSamples: rec.cutOffSamples,
|
|
24510
24650
|
noRecommendationReason: rec.noRecommendationReason
|
|
24511
24651
|
};
|
|
24512
24652
|
const savedTo = await saveBench(cwd, record);
|
|
24513
24653
|
if (opts.json) {
|
|
24514
|
-
const ok =
|
|
24654
|
+
const ok = settledCount > 0 && !lengthBlocked;
|
|
24515
24655
|
writeJsonLine({
|
|
24516
24656
|
command: "llm bench",
|
|
24517
24657
|
status: ok ? "ok" : "failed",
|
|
24518
24658
|
error: ok ? null : rec.noRecommendationReason ?? "no_sample_settled",
|
|
24659
|
+
currency: "credits",
|
|
24519
24660
|
apiUrl,
|
|
24520
24661
|
projectId,
|
|
24521
24662
|
modelId,
|
|
@@ -24524,35 +24665,54 @@ async function runBench(args) {
|
|
|
24524
24665
|
schemaPath: opts.schemaPath ?? null,
|
|
24525
24666
|
samples,
|
|
24526
24667
|
completed: settledCount,
|
|
24527
|
-
|
|
24668
|
+
maxCreditsPerSample: maxCredits,
|
|
24669
|
+
// Spendable credits before and after the run; null when unreadable.
|
|
24528
24670
|
balanceBefore,
|
|
24529
24671
|
balanceAfter,
|
|
24530
24672
|
results,
|
|
24531
24673
|
// The create-time refusal that ended the run, or null. Beside the
|
|
24532
24674
|
// samples, never among them: no attempt existed and nothing was spent.
|
|
24533
24675
|
refusal: refusal2,
|
|
24534
|
-
|
|
24535
|
-
//
|
|
24676
|
+
// Over the settled samples: the real provider cost (pico-dollar strings
|
|
24677
|
+
// and USD) and the whole credits charged.
|
|
24678
|
+
costUsdPicos: record.cost,
|
|
24679
|
+
costUsd: {
|
|
24680
|
+
p50: rec.costP50 === null ? null : Number(rec.costP50) / 1e12,
|
|
24681
|
+
p95: rec.costP95 === null ? null : Number(rec.costP95) / 1e12,
|
|
24682
|
+
max: rec.costMax === null ? null : Number(rec.costMax) / 1e12
|
|
24683
|
+
},
|
|
24684
|
+
chargedCredits: record.charged,
|
|
24685
|
+
// The answer's LENGTH, priced by the server per sample (`fitCredits`,
|
|
24536
24686
|
// headroom already on the tokens), over the settled samples.
|
|
24537
|
-
|
|
24538
|
-
|
|
24539
|
-
|
|
24687
|
+
fitCredits: { p95: rec.fitP95, max: rec.fitMax },
|
|
24688
|
+
recommendedDeclaredHeadroomBps: terms.headroomBps,
|
|
24689
|
+
feeBps: terms.feeBps,
|
|
24690
|
+
usdCentsPerCredit: terms.usdCentsPerCredit,
|
|
24691
|
+
missingTerms: rec.missingTerms,
|
|
24692
|
+
costBasedMaxCredits: rec.costBasedMaxCredits,
|
|
24693
|
+
lengthRaisedCeiling: rec.lengthRaisedCeiling,
|
|
24540
24694
|
cutOffSamples: rec.cutOffSamples,
|
|
24541
24695
|
exceedsStandCap: rec.exceedsStandCap,
|
|
24542
24696
|
noRecommendationReason: rec.noRecommendationReason,
|
|
24543
|
-
|
|
24544
|
-
|
|
24545
|
-
|
|
24697
|
+
recommendedMaxCredits: rec.recommendedMaxCredits,
|
|
24698
|
+
recommendedPerCallMaxCredits: rec.recommendedPerCallMaxCredits,
|
|
24699
|
+
averageCreditsPerCall: rec.averageCreditsPerCall,
|
|
24700
|
+
recommendedPerCallEstimateCredits: rec.recommendedPerCallEstimateCredits,
|
|
24546
24701
|
savedTo
|
|
24547
24702
|
});
|
|
24548
24703
|
if (!ok) process.exitCode = 1;
|
|
24549
24704
|
return;
|
|
24550
24705
|
}
|
|
24551
24706
|
log.plain("");
|
|
24707
|
+
const printMeasured = () => {
|
|
24708
|
+
log.plain(c.bold(" Real cost per call"));
|
|
24709
|
+
log.plain(` p50 ${usdFromPicos(rec.costP50)} p95 ${usdFromPicos(rec.costP95)} max ${usdFromPicos(rec.costMax)} (${settledCount} of ${samples} settled)`);
|
|
24710
|
+
log.plain(c.bold(" Charged credits"));
|
|
24711
|
+
log.plain(` p50 ${rec.chargedP50} p95 ${rec.chargedP95} max ${rec.chargedMax} (whole credits, rounded up per call)`);
|
|
24712
|
+
};
|
|
24552
24713
|
if (lengthBlocked) {
|
|
24553
|
-
if (
|
|
24554
|
-
|
|
24555
|
-
log.plain(` p50 ${p50} p95 ${p95} max ${max} (${charged.length} of ${samples} settled)`);
|
|
24714
|
+
if (settledCount > 0) {
|
|
24715
|
+
printMeasured();
|
|
24556
24716
|
log.plain("");
|
|
24557
24717
|
}
|
|
24558
24718
|
printRecommendation(log, record);
|
|
@@ -24560,17 +24720,16 @@ async function runBench(args) {
|
|
|
24560
24720
|
process.exitCode = 1;
|
|
24561
24721
|
return;
|
|
24562
24722
|
}
|
|
24563
|
-
if (
|
|
24723
|
+
if (settledCount === 0) {
|
|
24564
24724
|
log.error(
|
|
24565
|
-
providerRefused > 0 && providerRefused === results.length ? " No sample ran \u2014 the provider refused every attempt at its door \u2014 so there is nothing to
|
|
24725
|
+
providerRefused > 0 && providerRefused === results.length ? " No sample ran \u2014 the provider refused every attempt at its door \u2014 so there is nothing to measure from." : " No sample settled, so there is nothing to measure from."
|
|
24566
24726
|
);
|
|
24567
24727
|
process.exitCode = 1;
|
|
24568
24728
|
return;
|
|
24569
24729
|
}
|
|
24570
|
-
|
|
24571
|
-
log.plain(` p50 ${p50} p95 ${p95} max ${max} (${charged.length} of ${samples} settled)`);
|
|
24730
|
+
printMeasured();
|
|
24572
24731
|
if (balanceBefore !== null && balanceAfter !== null) {
|
|
24573
|
-
log.plain(` Balance ${balanceBefore} \u2192 ${balanceAfter}
|
|
24732
|
+
log.plain(` Balance ${balanceBefore} \u2192 ${balanceAfter} credits spendable`);
|
|
24574
24733
|
}
|
|
24575
24734
|
log.plain("");
|
|
24576
24735
|
printRecommendation(log, record);
|
|
@@ -24582,43 +24741,58 @@ function printRecommendation(log, record) {
|
|
|
24582
24741
|
return;
|
|
24583
24742
|
}
|
|
24584
24743
|
if (record.noRecommendationReason === "answer_cut_off") {
|
|
24585
|
-
log.warn(` ${benchCutOffLine(record.cutOffSamples
|
|
24744
|
+
log.warn(` ${benchCutOffLine(record.cutOffSamples || 1, record.maxCreditsPerSample)}`);
|
|
24586
24745
|
return;
|
|
24587
24746
|
}
|
|
24588
|
-
|
|
24589
|
-
|
|
24747
|
+
const attempts = `${record.completed} settled attempt${record.completed === 1 ? "" : "s"} on ${record.modelId}`;
|
|
24748
|
+
if (record.recommendedMaxCredits === null || record.cost.p95 === null) {
|
|
24749
|
+
log.warn(" No ceiling recommended.");
|
|
24590
24750
|
log.dim(
|
|
24591
|
-
record.recommendedDeclaredHeadroomBps === null ? " This stand served no recommended headroom, and the multiplier is the server's to set \u2014" : "
|
|
24751
|
+
record.cost.p95 === null ? " No sample settled, so there is no p95 to build a ceiling on \u2014" : record.recommendedDeclaredHeadroomBps === null ? " This stand served no recommended headroom, and the multiplier is the server's to set \u2014" : " This stand served no platform fee or credit value, and those are the server's to set \u2014"
|
|
24592
24752
|
);
|
|
24593
24753
|
log.dim(" declaring a number from this run would be a guess dressed as a measurement.");
|
|
24754
|
+
printAverage(log, record, attempts);
|
|
24594
24755
|
return;
|
|
24595
24756
|
}
|
|
24596
|
-
log.plain(c.bold(` Declare
|
|
24597
|
-
if (record.
|
|
24757
|
+
log.plain(c.bold(` Declare maxCredits: ${record.recommendedMaxCredits}`));
|
|
24758
|
+
if (record.lengthRaisedCeiling) {
|
|
24598
24759
|
log.plain(` ${LLM_LENGTH_RAISED_LINE}`);
|
|
24599
24760
|
log.dim(
|
|
24600
|
-
` = the smallest
|
|
24761
|
+
` = the smallest ceiling whose answer allowance holds the answer plus this stand's headroom (p95 ${record.fitP95 ?? "\u2014"}), above the`
|
|
24601
24762
|
);
|
|
24602
24763
|
log.dim(
|
|
24603
|
-
`
|
|
24764
|
+
` cost-based ${record.costBasedMaxCredits ?? "\u2014"} (p95 real cost ${usdFromPicos(record.cost.p95)} with the platform fee, ${record.feeBps} bps, and ${record.recommendedDeclaredHeadroomBps} bps of headroom), over ${attempts}.`
|
|
24604
24765
|
);
|
|
24605
24766
|
} else {
|
|
24606
24767
|
log.dim(
|
|
24607
|
-
` = p95
|
|
24768
|
+
` = p95 real cost (${usdFromPicos(record.cost.p95)}) with this stand's platform fee (${record.feeBps} bps) and recommended headroom (${record.recommendedDeclaredHeadroomBps} bps), in whole credits,`
|
|
24608
24769
|
);
|
|
24609
|
-
log.dim(` measured over ${
|
|
24770
|
+
log.dim(` measured over ${attempts}.`);
|
|
24610
24771
|
}
|
|
24611
|
-
log.dim(" That number is a
|
|
24612
|
-
|
|
24613
|
-
|
|
24772
|
+
log.dim(" That number is a CEILING, not a price: a call is billed its real cost as used, never more than it \u2014");
|
|
24773
|
+
log.dim(" and it also sizes how long the answer may be.");
|
|
24774
|
+
if (record.recommendedPerCallMaxCredits !== null) {
|
|
24775
|
+
log.plain(c.bold(` Grant perCallMaxCredits: ${record.recommendedPerCallMaxCredits}`));
|
|
24614
24776
|
log.dim(
|
|
24615
|
-
record.
|
|
24777
|
+
record.lengthRaisedCeiling ? ` = the same over the worst sample (${usdFromPicos(record.cost.max)}) or the longest answer's ceiling, whichever is larger, and never below the ceiling above it.` : ` = the same over the worst sample (${usdFromPicos(record.cost.max)}), and never below the ceiling above it.`
|
|
24616
24778
|
);
|
|
24617
|
-
|
|
24779
|
+
}
|
|
24780
|
+
printAverage(log, record, attempts);
|
|
24781
|
+
}
|
|
24782
|
+
function printAverage(log, record, attempts) {
|
|
24783
|
+
if (record.averageCreditsPerCall === null) return;
|
|
24784
|
+
log.plain(` Per-call estimate: about ${formatCredits(record.averageCreditsPerCall)} credits per call`);
|
|
24785
|
+
log.dim(` = the average real cost with the platform fee, no headroom, over ${attempts}.`);
|
|
24786
|
+
if (record.recommendedPerCallEstimateCredits !== null) {
|
|
24787
|
+
log.plain(c.bold(` Grant perCallEstimateCredits: ${record.recommendedPerCallEstimateCredits}`));
|
|
24788
|
+
log.dim(` = that average rounded up to a whole credit. disclosure.estimatedCreditsPerPeriod = your calls per period \xD7 ${formatCredits(record.averageCreditsPerCall)}, rounded up.`);
|
|
24618
24789
|
}
|
|
24619
24790
|
}
|
|
24620
24791
|
async function explainBenchRefusal(res, call, base, log, opts, index, settledSoFar) {
|
|
24621
|
-
if (printedStructuredError(res))
|
|
24792
|
+
if (printedStructuredError(res)) {
|
|
24793
|
+
const printed = await res.json().catch(() => ({}));
|
|
24794
|
+
return { stop: true, refusal: { status: res.status, error: typeof printed.error === "string" ? printed.error : null, slotsHeld: null, slotLimit: null } };
|
|
24795
|
+
}
|
|
24622
24796
|
const body = await res.json().catch(() => ({}));
|
|
24623
24797
|
const refusal2 = { status: res.status, error: body.error ?? null, slotsHeld: null, slotLimit: null };
|
|
24624
24798
|
if (res.status === 429 && body.error === "generation_limit") {
|
|
@@ -24628,6 +24802,10 @@ async function explainBenchRefusal(res, call, base, log, opts, index, settledSoF
|
|
|
24628
24802
|
if (!opts.json) log.error(` Sample ${index + 1} ${benchSlotsHeldSentence(counts)}`);
|
|
24629
24803
|
return { stop: true, refusal: refusal2 };
|
|
24630
24804
|
}
|
|
24805
|
+
if (res.status === 403 && body.error === "credits_unverified") {
|
|
24806
|
+
if (!opts.json) log.error(` ${LLM_BENCH_UNVERIFIED_LINE}`);
|
|
24807
|
+
return { stop: true, refusal: refusal2 };
|
|
24808
|
+
}
|
|
24631
24809
|
if (opts.json) return { stop: res.status !== 429, refusal: refusal2 };
|
|
24632
24810
|
if (res.status === 404 && body.error === "not_found") {
|
|
24633
24811
|
log.error(` ${LLM_LANE_OFF_LINE}`);
|
|
@@ -24643,7 +24821,7 @@ async function explainBenchRefusal(res, call, base, log, opts, index, settledSoF
|
|
|
24643
24821
|
return { stop: true, refusal: refusal2 };
|
|
24644
24822
|
}
|
|
24645
24823
|
if (res.status === 402) {
|
|
24646
|
-
log.error(" Not enough
|
|
24824
|
+
log.error(" Not enough credits to start the attempt \u2014 a bench is paid from your own balance.");
|
|
24647
24825
|
return { stop: true, refusal: refusal2 };
|
|
24648
24826
|
}
|
|
24649
24827
|
log.error(
|
|
@@ -24652,7 +24830,7 @@ async function explainBenchRefusal(res, call, base, log, opts, index, settledSoF
|
|
|
24652
24830
|
return { stop: res.status >= 500 || res.status === 401 || res.status === 403, refusal: refusal2 };
|
|
24653
24831
|
}
|
|
24654
24832
|
function printProviderRefusal(log, status2, message, awaitingBill = false) {
|
|
24655
|
-
if (awaitingBill) log.dim(` The provider answered ${status2} after routing \u2014 its bill is not final, so this call
|
|
24833
|
+
if (awaitingBill) log.dim(` The provider answered ${status2} after routing \u2014 its bill is not final, so what this call reserved stays held until it resolves (see \`genex llm status\`); it is not a sample.`);
|
|
24656
24834
|
else log.dim(" The provider refused this call at its door \u2014 no inference ran, it cost nothing, and it is not a sample.");
|
|
24657
24835
|
if (message) log.dim(` Provider said: ${message}`);
|
|
24658
24836
|
if (status2 === 401 || status2 === 403) {
|
|
@@ -24678,19 +24856,6 @@ async function pollSettled(call, url, timeoutSec) {
|
|
|
24678
24856
|
}
|
|
24679
24857
|
return last;
|
|
24680
24858
|
}
|
|
24681
|
-
async function readCoinBalance(apiUrl, token) {
|
|
24682
|
-
try {
|
|
24683
|
-
const res = await apiFetch(`${apiUrl}/api/coin/balance`, {
|
|
24684
|
-
headers: { Authorization: `Bearer ${token}` },
|
|
24685
|
-
signal: AbortSignal.timeout(6e3)
|
|
24686
|
-
});
|
|
24687
|
-
if (!res.ok) return null;
|
|
24688
|
-
const body = await res.json().catch(() => null);
|
|
24689
|
-
return typeof body?.spendable === "number" ? body.spendable : null;
|
|
24690
|
-
} catch {
|
|
24691
|
-
return null;
|
|
24692
|
-
}
|
|
24693
|
-
}
|
|
24694
24859
|
async function saveBench(cwd, record) {
|
|
24695
24860
|
try {
|
|
24696
24861
|
const file = path36.join(cwd, LLM_BENCH_FILE);
|
|
@@ -24706,9 +24871,10 @@ function benchFailed(opts, log, message) {
|
|
|
24706
24871
|
else log.error(message);
|
|
24707
24872
|
process.exitCode = 1;
|
|
24708
24873
|
}
|
|
24709
|
-
function emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples,
|
|
24874
|
+
function emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples, maxCredits, opts) {
|
|
24710
24875
|
return {
|
|
24711
24876
|
command: "llm bench",
|
|
24877
|
+
currency: "credits",
|
|
24712
24878
|
apiUrl,
|
|
24713
24879
|
projectId,
|
|
24714
24880
|
modelId: opts.modelId ?? null,
|
|
@@ -24717,21 +24883,28 @@ function emptyBenchJson(apiUrl, projectId, prompt, outputFormat, samples, maxCoi
|
|
|
24717
24883
|
schemaPath: opts.schemaPath ?? null,
|
|
24718
24884
|
samples,
|
|
24719
24885
|
completed: 0,
|
|
24720
|
-
|
|
24886
|
+
maxCreditsPerSample: maxCredits,
|
|
24721
24887
|
balanceBefore: null,
|
|
24722
24888
|
balanceAfter: null,
|
|
24723
24889
|
results: [],
|
|
24724
24890
|
refusal: null,
|
|
24725
|
-
|
|
24726
|
-
|
|
24727
|
-
|
|
24728
|
-
|
|
24891
|
+
costUsdPicos: { p50: null, p95: null, max: null },
|
|
24892
|
+
costUsd: { p50: null, p95: null, max: null },
|
|
24893
|
+
chargedCredits: { p50: null, p95: null, max: null },
|
|
24894
|
+
fitCredits: { p95: null, max: null },
|
|
24895
|
+
recommendedDeclaredHeadroomBps: null,
|
|
24896
|
+
feeBps: null,
|
|
24897
|
+
usdCentsPerCredit: null,
|
|
24898
|
+
missingTerms: null,
|
|
24899
|
+
costBasedMaxCredits: null,
|
|
24900
|
+
lengthRaisedCeiling: false,
|
|
24729
24901
|
cutOffSamples: 0,
|
|
24730
24902
|
exceedsStandCap: false,
|
|
24731
24903
|
noRecommendationReason: null,
|
|
24732
|
-
|
|
24733
|
-
|
|
24734
|
-
|
|
24904
|
+
recommendedMaxCredits: null,
|
|
24905
|
+
recommendedPerCallMaxCredits: null,
|
|
24906
|
+
averageCreditsPerCall: null,
|
|
24907
|
+
recommendedPerCallEstimateCredits: null,
|
|
24735
24908
|
savedTo: null
|
|
24736
24909
|
};
|
|
24737
24910
|
}
|
|
@@ -24794,7 +24967,8 @@ async function reportStatus(args) {
|
|
|
24794
24967
|
for (const row of rows) {
|
|
24795
24968
|
const state = rowState(row);
|
|
24796
24969
|
const mark = state === "active" ? c.cyan("\u25CF") : state === "awaiting bill" ? c.yellow("\u25D0") : c.green("\u25CB");
|
|
24797
|
-
const
|
|
24970
|
+
const money2 = rowMoney(row);
|
|
24971
|
+
const reserved = money2.reserved === null ? "reserved \u2014" : `${money2.reserved} ${money2.unit} reserved`;
|
|
24798
24972
|
log.plain(
|
|
24799
24973
|
` ${mark} ${row.id ?? "?"} ${row.modelId ?? "?"} ${state.padEnd(13)} ${formatAge(row.createdAt, now).padStart(4)} old ${reserved}${row.error ? ` ${row.error}` : ""}${rowHoldsSlot(row) ? "" : c.dim(" (no slot)")}`
|
|
24800
24974
|
);
|
|
@@ -24866,10 +25040,11 @@ async function cancelCall(args) {
|
|
|
24866
25040
|
if (!opts.json) {
|
|
24867
25041
|
log.warn(` ${row.id} (${row.status ?? "?"}) \u2014 ${sentence}.`);
|
|
24868
25042
|
if (state === "awaiting bill") {
|
|
24869
|
-
log.dim(` Nothing to cancel:
|
|
25043
|
+
log.dim(` Nothing to cancel: what it reserved releases when the bill resolves, and it stops holding a slot ${PENDING_SLOT_RELEASE_MINUTES} minutes after dispatch.`);
|
|
24870
25044
|
if (row.providerMessage?.trim()) log.dim(` Provider said: ${row.providerMessage.trim()}`);
|
|
24871
25045
|
} else {
|
|
24872
|
-
|
|
25046
|
+
const money2 = rowMoney(row);
|
|
25047
|
+
log.dim(` ${money2.charged === null ? "charge unknown" : `${money2.charged} ${money2.unit} charged`}${row.error ? ` \xB7 ${row.error}` : ""}.`);
|
|
24873
25048
|
}
|
|
24874
25049
|
}
|
|
24875
25050
|
return done("ok", null, state === "awaiting bill" ? "awaiting_bill" : "already_final", row);
|
|
@@ -24891,78 +25066,126 @@ async function cancelCall(args) {
|
|
|
24891
25066
|
const after = await res.json().catch(() => null) ?? row;
|
|
24892
25067
|
if (!opts.json) {
|
|
24893
25068
|
log.success(` Cancelled ${row.id} (was ${row.status ?? "active"}).`);
|
|
24894
|
-
log.dim(`
|
|
25069
|
+
log.dim(` What it reserved releases once the bill settles; ${c.cyan("genex llm status")} shows it until then.`);
|
|
24895
25070
|
}
|
|
24896
25071
|
return done("ok", null, "cancelled", after);
|
|
24897
25072
|
}
|
|
24898
|
-
async function
|
|
24899
|
-
let
|
|
25073
|
+
async function readSavedBench(cwd) {
|
|
25074
|
+
let parsed;
|
|
24900
25075
|
try {
|
|
24901
|
-
|
|
25076
|
+
parsed = JSON.parse(await fs35.readFile(path36.join(cwd, LLM_BENCH_FILE), "utf8"));
|
|
24902
25077
|
} catch {
|
|
24903
|
-
|
|
25078
|
+
return null;
|
|
24904
25079
|
}
|
|
24905
|
-
if (!
|
|
25080
|
+
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return null;
|
|
25081
|
+
const v = parsed;
|
|
25082
|
+
if (v.v === 2 && v.currency === "credits" && v.cost && v.charged) return { kind: "credits", record: parsed };
|
|
25083
|
+
return { kind: "coins", record: parsed };
|
|
25084
|
+
}
|
|
25085
|
+
var LLM_PRICE_COIN_TERMS_LINE = "This bench was measured under the old COIN terms, when a game declared a fixed price per call. In-game calls are now billed in the player's credits, as used, and a game declares a per-call CEILING (maxCredits) instead \u2014 a coin price is not that number. Re-run the bench for the figures to declare.";
|
|
25086
|
+
async function reportSavedPrice(cwd, opts, log) {
|
|
25087
|
+
const saved = await readSavedBench(cwd);
|
|
25088
|
+
const rerun = 'genex llm bench "<prompt>" --max-credits <n> --user-approved';
|
|
25089
|
+
const creditKeys = (record2) => ({
|
|
25090
|
+
costUsdPicos: record2?.cost ?? { p50: null, p95: null, max: null },
|
|
25091
|
+
chargedCredits: record2?.charged ?? { p50: null, p95: null, max: null },
|
|
25092
|
+
fitP95: record2?.fitP95 ?? null,
|
|
25093
|
+
fitMax: record2?.fitMax ?? null,
|
|
25094
|
+
recommendedDeclaredHeadroomBps: record2?.recommendedDeclaredHeadroomBps ?? null,
|
|
25095
|
+
feeBps: record2?.feeBps ?? null,
|
|
25096
|
+
usdCentsPerCredit: record2?.usdCentsPerCredit ?? null,
|
|
25097
|
+
costBasedMaxCredits: record2?.costBasedMaxCredits ?? null,
|
|
25098
|
+
lengthRaisedCeiling: record2?.lengthRaisedCeiling ?? false,
|
|
25099
|
+
cutOffSamples: record2?.cutOffSamples ?? 0,
|
|
25100
|
+
noRecommendationReason: record2?.noRecommendationReason ?? null,
|
|
25101
|
+
recommendedMaxCredits: record2?.recommendedMaxCredits ?? null,
|
|
25102
|
+
recommendedPerCallMaxCredits: record2?.recommendedPerCallMaxCredits ?? null,
|
|
25103
|
+
averageCreditsPerCall: record2?.averageCreditsPerCall ?? null,
|
|
25104
|
+
recommendedPerCallEstimateCredits: record2?.recommendedPerCallEstimateCredits ?? null
|
|
25105
|
+
});
|
|
25106
|
+
if (!saved) {
|
|
24906
25107
|
if (opts.json) {
|
|
24907
25108
|
writeJsonLine({
|
|
24908
25109
|
command: "llm price",
|
|
24909
25110
|
status: "failed",
|
|
24910
25111
|
error: "no_bench",
|
|
25112
|
+
currency: null,
|
|
24911
25113
|
ranAt: null,
|
|
24912
25114
|
modelId: null,
|
|
24913
25115
|
samples: null,
|
|
24914
25116
|
completed: null,
|
|
24915
|
-
|
|
24916
|
-
|
|
24917
|
-
max: null,
|
|
24918
|
-
fitP95: null,
|
|
24919
|
-
chargedBasedEstimateCoins: null,
|
|
24920
|
-
lengthRaisedPrice: false,
|
|
24921
|
-
cutOffSamples: 0,
|
|
24922
|
-
noRecommendationReason: null,
|
|
24923
|
-
recommendedDeclaredHeadroomBps: null,
|
|
24924
|
-
recommendedEstimateCoins: null,
|
|
24925
|
-
recommendedPerCallMaxCoins: null,
|
|
25117
|
+
...creditKeys(null),
|
|
25118
|
+
legacyCoinTerms: null,
|
|
24926
25119
|
savedTo: null
|
|
24927
25120
|
});
|
|
24928
25121
|
} else {
|
|
24929
25122
|
log.error("Nothing has been benchmarked in this folder yet.");
|
|
24930
|
-
log.dim(` Run ${c.cyan(
|
|
25123
|
+
log.dim(` Run ${c.cyan(rerun)} first.`);
|
|
25124
|
+
}
|
|
25125
|
+
process.exitCode = 1;
|
|
25126
|
+
return;
|
|
25127
|
+
}
|
|
25128
|
+
if (saved.kind === "coins") {
|
|
25129
|
+
const old = saved.record;
|
|
25130
|
+
const legacy = {
|
|
25131
|
+
p50: old.p50 ?? null,
|
|
25132
|
+
p95: old.p95 ?? null,
|
|
25133
|
+
max: old.max ?? null,
|
|
25134
|
+
recommendedEstimateCoins: old.recommendedEstimateCoins ?? null,
|
|
25135
|
+
recommendedPerCallMaxCoins: old.recommendedPerCallMaxCoins ?? null,
|
|
25136
|
+
maxCoinsPerSample: old.maxCoinsPerSample ?? null,
|
|
25137
|
+
noRecommendationReason: old.noRecommendationReason ?? null
|
|
25138
|
+
};
|
|
25139
|
+
if (opts.json) {
|
|
25140
|
+
writeJsonLine({
|
|
25141
|
+
command: "llm price",
|
|
25142
|
+
status: "failed",
|
|
25143
|
+
error: "coin_terms",
|
|
25144
|
+
currency: "coins",
|
|
25145
|
+
ranAt: old.ranAt ?? null,
|
|
25146
|
+
modelId: old.modelId ?? null,
|
|
25147
|
+
samples: old.samples ?? null,
|
|
25148
|
+
completed: old.completed ?? null,
|
|
25149
|
+
...creditKeys(null),
|
|
25150
|
+
legacyCoinTerms: legacy,
|
|
25151
|
+
savedTo: LLM_BENCH_FILE
|
|
25152
|
+
});
|
|
25153
|
+
} else {
|
|
25154
|
+
log.plain(c.bold("genex llm price"));
|
|
25155
|
+
log.dim(` from ${LLM_BENCH_FILE}, benchmarked ${old.ranAt ?? "at an unknown time"}${old.modelId ? ` on ${old.modelId}` : ""}`);
|
|
25156
|
+
log.plain("");
|
|
25157
|
+
log.warn(` ${LLM_PRICE_COIN_TERMS_LINE}`);
|
|
25158
|
+
const measured = legacy.p95 === null ? "no settled sample" : `charged coins p50 ${legacy.p50} \xB7 p95 ${legacy.p95} \xB7 max ${legacy.max}`;
|
|
25159
|
+
const recommended = legacy.recommendedEstimateCoins === null ? "no price recommended" : `it recommended estimateCoins ${legacy.recommendedEstimateCoins}${legacy.recommendedPerCallMaxCoins != null ? ` and perCallMaxCoins ${legacy.recommendedPerCallMaxCoins}` : ""}`;
|
|
25160
|
+
log.dim(` What it measured then: ${measured}; ${recommended}.`);
|
|
25161
|
+
log.dim(` Re-run: ${c.cyan(rerun)}`);
|
|
24931
25162
|
}
|
|
24932
25163
|
process.exitCode = 1;
|
|
24933
25164
|
return;
|
|
24934
25165
|
}
|
|
25166
|
+
const record = saved.record;
|
|
24935
25167
|
if (opts.json) {
|
|
24936
25168
|
writeJsonLine({
|
|
24937
25169
|
command: "llm price",
|
|
24938
|
-
status: record.
|
|
24939
|
-
error: record.
|
|
25170
|
+
status: record.recommendedMaxCredits === null ? "failed" : "ok",
|
|
25171
|
+
error: record.recommendedMaxCredits === null ? record.noRecommendationReason ?? "no_recommendation" : null,
|
|
25172
|
+
currency: "credits",
|
|
24940
25173
|
ranAt: record.ranAt ?? null,
|
|
24941
25174
|
modelId: record.modelId ?? null,
|
|
24942
25175
|
samples: record.samples ?? null,
|
|
24943
25176
|
completed: record.completed ?? null,
|
|
24944
|
-
|
|
24945
|
-
|
|
24946
|
-
max: record.max ?? null,
|
|
24947
|
-
// Absent from a file an older CLI wrote; null / false / 0 then, never missing.
|
|
24948
|
-
fitP95: record.fitP95 ?? null,
|
|
24949
|
-
chargedBasedEstimateCoins: record.chargedBasedEstimateCoins ?? null,
|
|
24950
|
-
lengthRaisedPrice: record.lengthRaisedPrice ?? false,
|
|
24951
|
-
cutOffSamples: record.cutOffSamples ?? 0,
|
|
24952
|
-
noRecommendationReason: record.noRecommendationReason ?? null,
|
|
24953
|
-
recommendedDeclaredHeadroomBps: record.recommendedDeclaredHeadroomBps ?? null,
|
|
24954
|
-
recommendedEstimateCoins: record.recommendedEstimateCoins ?? null,
|
|
24955
|
-
recommendedPerCallMaxCoins: record.recommendedPerCallMaxCoins ?? null,
|
|
25177
|
+
...creditKeys(record),
|
|
25178
|
+
legacyCoinTerms: null,
|
|
24956
25179
|
savedTo: LLM_BENCH_FILE
|
|
24957
25180
|
});
|
|
24958
|
-
if (record.
|
|
25181
|
+
if (record.recommendedMaxCredits === null) process.exitCode = 1;
|
|
24959
25182
|
return;
|
|
24960
25183
|
}
|
|
24961
25184
|
log.plain(c.bold("genex llm price"));
|
|
24962
25185
|
log.dim(` from ${LLM_BENCH_FILE}, benchmarked ${record.ranAt}`);
|
|
24963
25186
|
log.plain("");
|
|
24964
25187
|
printRecommendation(log, record);
|
|
24965
|
-
if (record.
|
|
25188
|
+
if (record.recommendedMaxCredits === null) process.exitCode = 1;
|
|
24966
25189
|
}
|
|
24967
25190
|
function writeJsonLine(value) {
|
|
24968
25191
|
process.stdout.write(`${JSON.stringify(value)}
|
|
@@ -25222,7 +25445,7 @@ function runtimeRow(state, lane, toolsOnly = false) {
|
|
|
25222
25445
|
return {
|
|
25223
25446
|
label: "Runtime",
|
|
25224
25447
|
value: `in-game LLM lane live \xB7 ${n} model${n === 1 ? "" : "s"}${keyRefused ? ` \xB7 ${c.yellow("provider key refused")}` : ""}`,
|
|
25225
|
-
fix: keyRefused ? LLM_PROVIDER_KEY_REJECTED_LINE : toolsOnly ? convertFix : '
|
|
25448
|
+
fix: keyRefused ? LLM_PROVIDER_KEY_REJECTED_LINE : toolsOnly ? convertFix : 'Measure a call before a game declares its per-call ceiling: `npx genex llm bench "<prompt>" --max-credits <n> --user-approved`.'
|
|
25226
25449
|
};
|
|
25227
25450
|
}
|
|
25228
25451
|
case "off":
|
|
@@ -27178,16 +27401,16 @@ ${c.bold("Usage")}
|
|
|
27178
27401
|
status | cancel. "models" says whether this
|
|
27179
27402
|
stand serves the runtime LLM lane at all, and
|
|
27180
27403
|
at what provider rates. "bench" runs the real
|
|
27181
|
-
model on YOUR OWN
|
|
27182
|
-
attempt
|
|
27404
|
+
model on YOUR OWN credits and reports what each
|
|
27405
|
+
attempt really cost; it needs --max-credits <n>
|
|
27183
27406
|
--user-approved, like any spend the player has
|
|
27184
27407
|
to agree to. "price" reprints the last run's
|
|
27185
|
-
recommended
|
|
27408
|
+
recommended maxCredits. "status" lists this
|
|
27186
27409
|
project's open calls and the slots they hold;
|
|
27187
27410
|
"cancel <id>" stops an active one. Declare a
|
|
27188
|
-
game's
|
|
27189
|
-
a
|
|
27190
|
-
|
|
27411
|
+
game's per-call ceiling from a bench, never
|
|
27412
|
+
from a guess: the player is billed as used, and
|
|
27413
|
+
the ceiling also sizes how long an answer may be.
|
|
27191
27414
|
genex player <sub> [options] Answer the in-game model requests you approve, on your
|
|
27192
27415
|
OWN Claude or ChatGPT subscription: install | run |
|
|
27193
27416
|
status | stop | uninstall. "install" signs this machine
|
|
@@ -27642,7 +27865,7 @@ ${c.bold("Examples")}
|
|
|
27642
27865
|
genex shop remove sku_123
|
|
27643
27866
|
genex llm models
|
|
27644
27867
|
genex llm models --all
|
|
27645
|
-
genex llm bench "Reply with one short taunt." --samples 5 --max-
|
|
27868
|
+
genex llm bench "Reply with one short taunt." --samples 5 --max-credits 20 --user-approved
|
|
27646
27869
|
genex llm price
|
|
27647
27870
|
genex llm status
|
|
27648
27871
|
genex llm cancel <id>
|
|
@@ -27879,22 +28102,24 @@ ${c.bold("How the allowance works")}
|
|
|
27879
28102
|
|
|
27880
28103
|
Every number printed is live from your account; prices can change without a deploy.
|
|
27881
28104
|
`,
|
|
27882
|
-
llm: `${c.bold("genex llm")} \u2014 in-game model calls: is the lane live, what does one call cost, what should the game declare?
|
|
28105
|
+
llm: `${c.bold("genex llm")} \u2014 in-game model calls: is the lane live, what does one call cost, what ceiling should the game declare?
|
|
27883
28106
|
|
|
27884
28107
|
${c.bold("Usage")}
|
|
27885
28108
|
genex llm models [--all] [--json] Which models this stand serves, their provider rates per
|
|
27886
|
-
million tokens,
|
|
27887
|
-
|
|
27888
|
-
|
|
27889
|
-
|
|
27890
|
-
on this stand: it says so
|
|
27891
|
-
|
|
27892
|
-
|
|
27893
|
-
|
|
27894
|
-
|
|
27895
|
-
|
|
27896
|
-
|
|
27897
|
-
|
|
28109
|
+
million tokens, the platform fee a call is billed at, and
|
|
28110
|
+
the headroom it recommends over a benchmark. The catalog
|
|
28111
|
+
is synced from OpenRouter, so the FEATURED short list is
|
|
28112
|
+
printed by default and --all prints every row (--json
|
|
28113
|
+
always carries every row). Off on this stand: it says so
|
|
28114
|
+
and exits 0 \u2014 build the feature without a model rather
|
|
28115
|
+
than promising a 404.
|
|
28116
|
+
genex llm bench "<prompt>" --max-credits <n> --user-approved [options]
|
|
28117
|
+
Run the real model through the development lane and
|
|
28118
|
+
report what each attempt really COST and was charged.
|
|
28119
|
+
Refused without both flags, before any network call:
|
|
28120
|
+
this spends YOUR OWN CREDITS, a spend the build's asset
|
|
28121
|
+
allowance does not count. Needs a folder linked to a
|
|
28122
|
+
game you own, and a balance that covers one attempt.
|
|
27898
28123
|
genex llm price [--json] The recommendation from the last bench in this folder.
|
|
27899
28124
|
Reads a file; spends nothing.
|
|
27900
28125
|
genex llm status [--json] This project's open development calls: active, or
|
|
@@ -27911,29 +28136,32 @@ ${c.bold("Bench options")}
|
|
|
27911
28136
|
--samples <n> How many real attempts (default ${DEFAULT_BENCH_SAMPLES}). More samples, better p95.
|
|
27912
28137
|
--schema <file> A JSON file holding the output schema; implies JSON output.
|
|
27913
28138
|
--json-output Ask for JSON without a schema. --text Ask for text (the default).
|
|
27914
|
-
--max-
|
|
28139
|
+
--max-credits <n> The per-attempt ceiling YOU approved \u2014 a bench runs on your own credits,
|
|
27915
28140
|
never a player's. The run's worst case is that number times --samples,
|
|
27916
|
-
and it is printed before anything starts.
|
|
28141
|
+
and it is printed before anything starts. (--max-coins is its deprecated
|
|
28142
|
+
name, read as the same number.)
|
|
27917
28143
|
--timeout <seconds> Bounds the wait for each attempt.
|
|
27918
28144
|
--json One machine-readable object; every key is always present, null when
|
|
27919
28145
|
the CLI could not learn it.
|
|
27920
28146
|
|
|
27921
28147
|
${c.bold("Why a benchmark and not an estimate")}
|
|
27922
|
-
|
|
27923
|
-
|
|
27924
|
-
|
|
27925
|
-
|
|
27926
|
-
|
|
27927
|
-
|
|
27928
|
-
|
|
27929
|
-
|
|
27930
|
-
|
|
28148
|
+
An in-game call is paid from the PLAYER's credits and billed as used: its real provider cost
|
|
28149
|
+
plus the platform fee, never more than the per-call CEILING the game declares (\`maxCredits\`,
|
|
28150
|
+
or \`perCallMaxCredits\` on a standing budget). The ceiling is not a price, but it is still a
|
|
28151
|
+
number with two jobs: it bounds what one call may cost, and it funds how LONG the answer may
|
|
28152
|
+
be. Three layers sit between the provider's rate card and it \u2014 the provider's cost for YOUR
|
|
28153
|
+
prompt, the platform fee, and your headroom \u2014 and only a real call measures the first. The
|
|
28154
|
+
bench is billed to YOU, on the development lane; the player-funded lane is what the
|
|
28155
|
+
published game uses. The recommendation is the p95 real cost with the server's fee and
|
|
28156
|
+
headroom applied, in whole credits; a stand that serves no headroom or no fee gets no
|
|
28157
|
+
recommendation, because those multipliers are not the CLI's to invent.
|
|
27931
28158
|
|
|
27932
|
-
|
|
27933
|
-
|
|
27934
|
-
|
|
27935
|
-
|
|
27936
|
-
|
|
28159
|
+
Because the ceiling sizes each call's output allowance, the recommendation is never below
|
|
28160
|
+
the smallest ceiling that leaves room for the benchmarked answer, as the server computes it.
|
|
28161
|
+
A sample cut off at your --max-credits (provider_token_limit) is not a sample \u2014 re-run with a
|
|
28162
|
+
higher --max-credits; an answer longer than one call on this stand may produce gets no
|
|
28163
|
+
ceiling at all \u2014 ask for a shorter one. The run also prints the measured average per call,
|
|
28164
|
+
fee included, which is the honest estimate a budget's disclosure is built on.
|
|
27937
28165
|
|
|
27938
28166
|
Results land in ${LLM_BENCH_FILE} \u2014 gitignored, machine-local, re-read by \`genex llm price\`.
|
|
27939
28167
|
`,
|
|
@@ -28148,11 +28376,13 @@ function parseArgs(argv) {
|
|
|
28148
28376
|
"--limit",
|
|
28149
28377
|
// `genex budget --assets <credits>` — the allowance the player approved.
|
|
28150
28378
|
"--assets",
|
|
28151
|
-
// `genex llm bench` value flags. `--max-
|
|
28152
|
-
//
|
|
28379
|
+
// `genex llm bench` value flags. `--max-credits` is the same approval
|
|
28380
|
+
// shape as `--assets`, but its own gate: the asset allowance does not
|
|
28381
|
+
// count a bench. `--max-coins` is its deprecated name (same number).
|
|
28153
28382
|
"--model",
|
|
28154
28383
|
"--samples",
|
|
28155
28384
|
"--schema",
|
|
28385
|
+
"--max-credits",
|
|
28156
28386
|
"--max-coins",
|
|
28157
28387
|
// `genex player install --label <name>` — what this machine is called in
|
|
28158
28388
|
// the dashboard. Its own key rather than the generic 2nd positional,
|
|
@@ -28667,12 +28897,14 @@ function applyValueFlag(options, flag, value) {
|
|
|
28667
28897
|
options.samples = n;
|
|
28668
28898
|
break;
|
|
28669
28899
|
}
|
|
28900
|
+
case "--max-credits":
|
|
28670
28901
|
case "--max-coins": {
|
|
28671
28902
|
const n = Number(value);
|
|
28672
28903
|
if (!Number.isInteger(n) || n < 1) {
|
|
28673
|
-
throw new Error(`Invalid
|
|
28904
|
+
throw new Error(`Invalid ${flag} value: ${value} (whole credits, 1 or more)`);
|
|
28674
28905
|
}
|
|
28675
|
-
options.
|
|
28906
|
+
if (flag === "--max-credits") options.maxCredits = n;
|
|
28907
|
+
else options.maxCoins = n;
|
|
28676
28908
|
break;
|
|
28677
28909
|
}
|
|
28678
28910
|
case "--timeout": {
|
|
@@ -28964,9 +29196,9 @@ async function main() {
|
|
|
28964
29196
|
await runBudget(parsed.options);
|
|
28965
29197
|
break;
|
|
28966
29198
|
// The in-game model lane: is it live here, what does one call really
|
|
28967
|
-
// cost, what should the game declare. `bench`
|
|
28968
|
-
//
|
|
28969
|
-
//
|
|
29199
|
+
// cost, what ceiling should the game declare. `bench` spends the
|
|
29200
|
+
// builder's own credits OUTSIDE the asset allowance, so it carries its
|
|
29201
|
+
// own --max-credits/--user-approved gate.
|
|
28970
29202
|
case "llm":
|
|
28971
29203
|
await runLlm(parsed.options);
|
|
28972
29204
|
break;
|