badgr-cli 1.0.42 → 1.0.44
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +81 -2
- package/package.json +2 -1
- package/src/badgr.js +3 -0
- package/src/catalog.js +15 -0
- package/src/commands/comfyui.js +35 -8
- package/src/commands/run.js +87 -99
- package/src/commands/serve.js +186 -123
- package/src/commands/train.js +85 -11
- package/src/progress.js +42 -0
- package/tests/productized-dry-run.test.js +141 -0
- package/tests/serve-lifecycle.test.js +196 -137
- package/tests/template.test.js +11 -13
- package/tests/workload-templates.test.js +37 -2
package/README.md
CHANGED
|
@@ -66,10 +66,26 @@ badgr down <deployment-id>
|
|
|
66
66
|
**Shortcuts** — wrappers around `run` / `serve` for common workloads:
|
|
67
67
|
|
|
68
68
|
| `badgr comfyui run <workflow.json>` | Launch ComfyUI, queue workflow, return endpoint URL |
|
|
69
|
+
| `badgr comfyui batch --workflow ...` | Productized batch image generation — no ComfyUI setup, blessed workflow only |
|
|
69
70
|
| `badgr train <config.yaml>` | LoRA / fine-tuning job, stream logs |
|
|
71
|
+
| `badgr train lora --base-model ...` | Productized LoRA training — preset + dataset, no config file needed |
|
|
70
72
|
| `badgr transcribe <audio>` | Whisper transcription, print transcript |
|
|
71
73
|
| `badgr embed <model> <input>` | Text embeddings, output JSONL |
|
|
72
74
|
|
|
75
|
+
`badgr serve --list-aliases` lists the tested vLLM model routes (`qwen-7b`, `llama-8b`, `qwen-coder-7b`) that can be used in place of a full model ID.
|
|
76
|
+
|
|
77
|
+
### Model support levels
|
|
78
|
+
|
|
79
|
+
`badgr serve qwen-7b` is the happy path — a tested route with no extra setup. `badgr serve` also accepts any other model ID or a custom container, with the CLI honest (but brief) about what that means:
|
|
80
|
+
|
|
81
|
+
| Level | What it means |
|
|
82
|
+
|-------|---------------|
|
|
83
|
+
| **Tested route** (`badgr serve qwen-7b`) | One of the aliases above — tested and officially supported. No extra caveats printed. |
|
|
84
|
+
| **Best-effort Hugging Face model** (`badgr serve <org>/<model>`) | Any other Hugging Face model ID. Badgr will try a compatible route — not a guarantee every model works. |
|
|
85
|
+
| **Custom container** (`badgr serve --image ...`) | You own the server behavior; Badgr manages runtime, logs, spend caps, teardown, and the receipt. |
|
|
86
|
+
|
|
87
|
+
Gated Hugging Face models (e.g. Llama, Gemma) may need `--env HF_TOKEN=$HF_TOKEN`. Badgr doesn't warn about this up front — it only prints the hint if the deployment actually fails to start, so tested and working launches stay short.
|
|
88
|
+
|
|
73
89
|
---
|
|
74
90
|
|
|
75
91
|
## `badgr serve` options
|
|
@@ -91,6 +107,9 @@ badgr serve meta-llama/Llama-3.1-8B-Instruct --gpu L40S --region EU
|
|
|
91
107
|
| `--max-cost <$>` | — | Auto-stop when total spend reaches this amount |
|
|
92
108
|
| `--health-path <path>` | auto | Readiness path to poll (auto-detected for ComfyUI → `/system_stats`) |
|
|
93
109
|
| `--no-wait` | — | Skip endpoint health check and return immediately |
|
|
110
|
+
| `--list-aliases` | — | List blessed vLLM model aliases (`qwen-7b`, `llama-8b`, `qwen-coder-7b`) and exit — no provisioning, no API key required |
|
|
111
|
+
|
|
112
|
+
Blessed aliases expand to a full model ID + preset GPU, e.g. `badgr serve qwen-7b` → `Qwen/Qwen2.5-7B-Instruct` on an RTX 4090. Run `badgr serve --list-aliases` to see the current list.
|
|
94
113
|
|
|
95
114
|
---
|
|
96
115
|
|
|
@@ -153,13 +172,39 @@ Requires either `--max-cost` or `--persistent` to prevent runaway billing.
|
|
|
153
172
|
|
|
154
173
|
---
|
|
155
174
|
|
|
175
|
+
## `badgr comfyui batch` options
|
|
176
|
+
|
|
177
|
+
Productized batch image generation — runs a list of prompts through a **blessed** ComfyUI workflow and returns image URLs. No ComfyUI setup, no workflow file, no manual teardown.
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
badgr comfyui batch --workflow sdxl-basic --prompts prompts.txt --max-cost 10
|
|
181
|
+
badgr comfyui batch --workflow sdxl-basic --prompt "a cat on a beach" --prompt "a dog in the park" --max-cost 5
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Blessed workflows: `sdxl-basic` (SDXL text-to-image, default sampler settings). Max 20 prompts per batch.
|
|
185
|
+
|
|
186
|
+
| Flag | Default | Description |
|
|
187
|
+
|------|---------|-------------|
|
|
188
|
+
| `--workflow <name>` | — | Blessed workflow ID (required) — currently `sdxl-basic` |
|
|
189
|
+
| `--prompts <file>` | — | Text file, one prompt per line |
|
|
190
|
+
| `--prompt <text>` | — | Inline prompt (repeatable) — combine with `--prompts` if needed |
|
|
191
|
+
| `--max-cost <$>` | — | Auto-stop when total spend reaches this amount (required unless `--dry-run`) |
|
|
192
|
+
| `--max-runtime <min>` | 60 | Auto-stop after N minutes |
|
|
193
|
+
| `--gpu-type <type>` | workflow default | GPU type override |
|
|
194
|
+
| `--tier 1\|2` | 1 | Provider tier |
|
|
195
|
+
| `--dry-run` | — | Preview the batch (workflow, GPU, prompt count, cost) without provisioning |
|
|
196
|
+
|
|
197
|
+
Polls until complete and prints image URLs, or detaches with `badgr status` guidance if it outlives `--max-runtime`.
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
156
201
|
## `badgr train` options
|
|
157
202
|
|
|
158
203
|
```bash
|
|
159
204
|
badgr train config.yaml --gpu A100 --max-runtime 240 --env HF_TOKEN=$HF_TOKEN
|
|
160
205
|
```
|
|
161
206
|
|
|
162
|
-
|
|
207
|
+
Detects framework (axolotl, unsloth, trl) from config content, but **only Axolotl configs run today** — the container command for `unsloth`/`trl`/unrecognized configs isn't wired up yet, so `badgr train` blocks before provisioning rather than billing a GPU that's guaranteed to fail. Use `--framework axolotl` to force it, or use `badgr train lora` for a config-free productized path. Default max-runtime is 120 min.
|
|
163
208
|
|
|
164
209
|
| Flag | Default | Description |
|
|
165
210
|
|------|---------|-------------|
|
|
@@ -169,12 +214,46 @@ Auto-detects framework (axolotl, unsloth, trl) from config content. Default max-
|
|
|
169
214
|
| `--max-price <$/hr>` | — | Hard spend cap per GPU-hour |
|
|
170
215
|
| `--tier 1\|2` | 1 | Provider tier |
|
|
171
216
|
| `--region US\|EU\|AU` | — | Region preference |
|
|
172
|
-
| `--framework <name>` | auto-detect | Force framework: `axolotl`, `unsloth`, `trl` |
|
|
217
|
+
| `--framework <name>` | auto-detect | Force framework: `axolotl`, `unsloth`, `trl` (only `axolotl` currently runs) |
|
|
173
218
|
| `--env KEY=VALUE` | — | Environment variable (repeatable) |
|
|
174
219
|
| `--detach` | — | Launch and return immediately, don't stream logs |
|
|
175
220
|
|
|
176
221
|
---
|
|
177
222
|
|
|
223
|
+
## `badgr train lora` options
|
|
224
|
+
|
|
225
|
+
Productized LoRA training — pass a base model and dataset, no Axolotl config file needed. Badgr generates the config from a preset and returns a downloadable adapter.
|
|
226
|
+
|
|
227
|
+
```bash
|
|
228
|
+
badgr train lora --base-model mistralai/Mistral-7B-v0.1 --dataset ./train.jsonl --preset small --max-cost 20
|
|
229
|
+
badgr train lora --base-model meta-llama/Llama-3.1-8B-Instruct --dataset https://example.com/data.jsonl --preset medium --max-cost 40
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
Dataset sources: local file (uploaded first), direct URL (`https://`, `s3://`), or `--file-id` from a prior `badgr` upload.
|
|
233
|
+
|
|
234
|
+
| Flag | Default | Description |
|
|
235
|
+
|------|---------|-------------|
|
|
236
|
+
| `--base-model <id>` | — | HuggingFace model ID (required) — validated to exist before provisioning |
|
|
237
|
+
| `--dataset <path\|url>` | — | Local file, direct URL, or `s3://` URI |
|
|
238
|
+
| `--file-id <id>` | — | Badgr upload ID instead of `--dataset` |
|
|
239
|
+
| `--preset small\|medium` | `small` | Training profile — see below |
|
|
240
|
+
| `--max-cost <$>` | — | Auto-stop when total spend reaches this amount (required unless `--dry-run`) |
|
|
241
|
+
| `--max-runtime <min>` | 240 | Auto-stop after N minutes |
|
|
242
|
+
| `--gpu-type <type>` | preset default | GPU type override |
|
|
243
|
+
| `--tier 1\|2` | 1 | Provider tier |
|
|
244
|
+
| `--dry-run` | — | Preview the job (preset, GPU, rank, epochs, cost) without provisioning |
|
|
245
|
+
|
|
246
|
+
**Presets:**
|
|
247
|
+
|
|
248
|
+
| Preset | GPU | LoRA rank | Epochs | Best for |
|
|
249
|
+
|--------|-----|-----------|--------|----------|
|
|
250
|
+
| `small` (default) | RTX 4090 | 16 | 3 | Fast, low-cost — good default for most datasets |
|
|
251
|
+
| `medium` | A100 | 32 | 5 | Larger rank/more epochs — bigger datasets or higher quality |
|
|
252
|
+
|
|
253
|
+
On completion, prints an `adapter_url` — download with `GET /v1/jobs/{job_id}/adapter`, or via `badgr workload info` if saved.
|
|
254
|
+
|
|
255
|
+
---
|
|
256
|
+
|
|
178
257
|
## `badgr transcribe` options
|
|
179
258
|
|
|
180
259
|
```bash
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "badgr-cli",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.44",
|
|
4
4
|
"description": "Badgr — run or serve GPU workloads from one command",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -13,6 +13,7 @@
|
|
|
13
13
|
},
|
|
14
14
|
"dependencies": {
|
|
15
15
|
"@inquirer/prompts": "^8.5.2",
|
|
16
|
+
"archiver": "^7.0.1",
|
|
16
17
|
"chalk": "^5.3.0"
|
|
17
18
|
},
|
|
18
19
|
"devDependencies": {
|
package/src/badgr.js
CHANGED
|
@@ -42,12 +42,15 @@ ${chalk.bold('COMMANDS')}
|
|
|
42
42
|
|
|
43
43
|
${chalk.bold('SHORTCUTS')} ${chalk.dim('(wrappers around run / serve for common workloads)')}
|
|
44
44
|
${chalk.cyan('badgr comfyui run <workflow.json>')} Launch ComfyUI, return endpoint URL
|
|
45
|
+
${chalk.cyan('badgr comfyui batch --workflow ...')} Blessed-workflow batch image gen (no ComfyUI setup needed)
|
|
45
46
|
${chalk.cyan('badgr train <config.yaml>')} LoRA / fine-tuning job, stream logs
|
|
47
|
+
${chalk.cyan('badgr train lora --base-model ...')} Productized LoRA training (preset + dataset, no config file)
|
|
46
48
|
${chalk.cyan('badgr transcribe <audio>')} Whisper transcription, print transcript
|
|
47
49
|
${chalk.cyan('badgr embed <model> <input>')} Text embeddings, output JSONL
|
|
48
50
|
${chalk.cyan('badgr serve template <name>')} Launch an endpoint template (vllm, invokeai, comfyui, …)
|
|
49
51
|
${chalk.cyan('badgr run template <name>')} Launch a job template (axolotl, unsloth)
|
|
50
52
|
${chalk.cyan('badgr template list')} Browse all pre-built templates
|
|
53
|
+
${chalk.cyan('badgr serve --list-aliases')} List blessed vLLM model shortcuts (qwen-7b, llama-8b, …)
|
|
51
54
|
|
|
52
55
|
${chalk.bold('EXAMPLES')}
|
|
53
56
|
${chalk.dim('# Verify the stack works end-to-end:')}
|
package/src/catalog.js
CHANGED
|
@@ -471,6 +471,21 @@ export const BLESSED_VLLM_MODELS = {
|
|
|
471
471
|
},
|
|
472
472
|
};
|
|
473
473
|
|
|
474
|
+
// Hugging Face org/model prefixes that are known to gate access behind a license
|
|
475
|
+
// click-through. Used only to print a helpful HF_TOKEN hint — not exhaustive.
|
|
476
|
+
const GATED_MODEL_PREFIXES = [
|
|
477
|
+
'meta-llama/',
|
|
478
|
+
'google/gemma',
|
|
479
|
+
'mistralai/Mistral-Large',
|
|
480
|
+
'mistralai/Mixtral-8x22B',
|
|
481
|
+
];
|
|
482
|
+
|
|
483
|
+
/** Best-effort heuristic: is this HF model ID likely to require HF_TOKEN? */
|
|
484
|
+
export function isLikelyGatedModel(modelId) {
|
|
485
|
+
if (!modelId) return false;
|
|
486
|
+
return GATED_MODEL_PREFIXES.some(prefix => modelId.startsWith(prefix));
|
|
487
|
+
}
|
|
488
|
+
|
|
474
489
|
/** Blessed ComfyUI workflows accepted by `POST /v1/jobs` comfy.batch. */
|
|
475
490
|
export const BLESSED_COMFY_WORKFLOWS = {
|
|
476
491
|
'sdxl-basic': {
|
package/src/commands/comfyui.js
CHANGED
|
@@ -11,6 +11,7 @@ import { callApi, listDeployments } from '../api.js';
|
|
|
11
11
|
import { addDeployment, addReceipt, updateReceipt, generateReceiptId } from '../store.js';
|
|
12
12
|
import { normalizeTier, callWithFallback, HIGH_RATE_THRESHOLD } from '../fallback.js';
|
|
13
13
|
import { formatCliError } from '../errors.js';
|
|
14
|
+
import { BLESSED_COMFY_WORKFLOWS } from '../catalog.js';
|
|
14
15
|
|
|
15
16
|
const COMFYUI_IMAGE = process.env.COMFYUI_IMAGE || 'yanwk/comfyui-boot:cu126-megapak';
|
|
16
17
|
const HEALTH_PATH = '/system_stats';
|
|
@@ -126,6 +127,7 @@ export function parseComfyBatchArgs(args) {
|
|
|
126
127
|
if (a === '--max-runtime') { flags.maxRuntime = parseFloat(args[++i]); i++; continue; }
|
|
127
128
|
if (a === '--tier') { flags.tier = args[++i]; i++; continue; }
|
|
128
129
|
if (a === '--gpu-type') { flags.gpuType = args[++i]; i++; continue; }
|
|
130
|
+
if (a === '--dry-run') { flags.dryRun = true; i++; continue; }
|
|
129
131
|
i++;
|
|
130
132
|
}
|
|
131
133
|
return flags;
|
|
@@ -145,14 +147,12 @@ export async function comfyBatchCommand(config, args, chalk) {
|
|
|
145
147
|
return;
|
|
146
148
|
}
|
|
147
149
|
|
|
148
|
-
if (!flags.maxCost) {
|
|
150
|
+
if (!flags.maxCost && !flags.dryRun) {
|
|
149
151
|
console.error(chalk.red('\n ✗ --max-cost is required.\n'));
|
|
150
152
|
process.exitCode = 1;
|
|
151
153
|
return;
|
|
152
154
|
}
|
|
153
155
|
|
|
154
|
-
requireApiKey(config);
|
|
155
|
-
|
|
156
156
|
// Collect prompts from file or --prompt flags
|
|
157
157
|
let prompts = flags.inlinePrompts || [];
|
|
158
158
|
if (flags.prompts) {
|
|
@@ -168,6 +168,23 @@ export async function comfyBatchCommand(config, args, chalk) {
|
|
|
168
168
|
prompts = prompts.concat(lines);
|
|
169
169
|
}
|
|
170
170
|
|
|
171
|
+
if (flags.dryRun) {
|
|
172
|
+
const workflowSpec = BLESSED_COMFY_WORKFLOWS[flags.workflow];
|
|
173
|
+
console.log(chalk.bold('\n⚡ Dry run — no GPU will be provisioned\n'));
|
|
174
|
+
console.log(` ${chalk.bold('Workflow:')} ${flags.workflow}${workflowSpec ? '' : chalk.yellow(' (unknown — server will reject this)')}`);
|
|
175
|
+
if (workflowSpec) {
|
|
176
|
+
console.log(` ${chalk.bold('GPU:')} ${flags.gpuType || workflowSpec.gpu_type}`);
|
|
177
|
+
console.log(` ${chalk.dim(workflowSpec.description)}`);
|
|
178
|
+
}
|
|
179
|
+
console.log(` ${chalk.bold('Prompts:')} ${prompts.length}${prompts.length > 20 ? chalk.yellow(' (exceeds the 20-prompt limit — server will reject this)') : ''}`);
|
|
180
|
+
if (flags.maxCost) console.log(` ${chalk.bold('Max cost:')} $${flags.maxCost}`);
|
|
181
|
+
console.log(` ${chalk.bold('Max runtime:')} ${flags.maxRuntime ?? 60}min`);
|
|
182
|
+
console.log(chalk.dim('\n Remove --dry-run to submit.\n'));
|
|
183
|
+
return;
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
requireApiKey(config);
|
|
187
|
+
|
|
171
188
|
if (prompts.length === 0) {
|
|
172
189
|
console.error(chalk.red('\n ✗ No prompts provided. Use --prompts file.txt or --prompt "text"\n'));
|
|
173
190
|
process.exitCode = 1;
|
|
@@ -194,10 +211,15 @@ export async function comfyBatchCommand(config, args, chalk) {
|
|
|
194
211
|
|
|
195
212
|
let job;
|
|
196
213
|
try {
|
|
197
|
-
job = await callApi(
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
214
|
+
job = await callApi('/jobs', {
|
|
215
|
+
method: 'POST',
|
|
216
|
+
apiKey: config.apiKey,
|
|
217
|
+
baseUrl: config.baseUrl,
|
|
218
|
+
body: {
|
|
219
|
+
type: 'comfy.batch',
|
|
220
|
+
input,
|
|
221
|
+
policy: { max_cost: flags.maxCost, max_runtime_minutes: maxRuntime, tier: flags.tier },
|
|
222
|
+
},
|
|
201
223
|
});
|
|
202
224
|
} catch (err) {
|
|
203
225
|
console.error(chalk.red(`\n ✗ Failed to submit job: ${err.message}\n`));
|
|
@@ -215,7 +237,12 @@ export async function comfyBatchCommand(config, args, chalk) {
|
|
|
215
237
|
while (Date.now() - startMs < maxMs) {
|
|
216
238
|
await new Promise(r => setTimeout(r, 15_000));
|
|
217
239
|
let detail;
|
|
218
|
-
try {
|
|
240
|
+
try {
|
|
241
|
+
detail = await callApi(`/jobs/${job.job_id}`, {
|
|
242
|
+
apiKey: config.apiKey,
|
|
243
|
+
baseUrl: config.baseUrl,
|
|
244
|
+
});
|
|
245
|
+
} catch { continue; }
|
|
219
246
|
process.stdout.write(`\r Status: ${detail.status} elapsed: ${Math.floor((Date.now() - startMs) / 1000)}s `);
|
|
220
247
|
if (detail.status === 'completed') {
|
|
221
248
|
const out = detail.output || {};
|
package/src/commands/run.js
CHANGED
|
@@ -8,6 +8,7 @@ import { addReceipt, updateReceipt, generateReceiptId } from '../store.js';
|
|
|
8
8
|
import { normalizeTier, callWithFallback, HIGH_RATE_THRESHOLD } from '../fallback.js';
|
|
9
9
|
import { formatCliError } from '../errors.js';
|
|
10
10
|
import { TEMPLATE_MAP, buildTemplateFlags, parseTemplateOverrides } from '../catalog.js';
|
|
11
|
+
import { stage as _stage, stageDone as _stageDone, writeBlock as _writeBlock, clearBlock as _clearBlock, renderLiveBlock as _renderLiveBlock } from '../progress.js';
|
|
11
12
|
|
|
12
13
|
/**
|
|
13
14
|
* Flow 1 — local project (primary):
|
|
@@ -105,6 +106,20 @@ const HEARTBEAT_WARN_POLLS = 3;
|
|
|
105
106
|
const HEARTBEAT_KILL_POLLS = 15;
|
|
106
107
|
const STARTUP_STATES = new Set(['queued', 'provisioning', 'starting']);
|
|
107
108
|
|
|
109
|
+
// Shared closing block for every terminal path (success, failure, cap, heartbeat loss) —
|
|
110
|
+
// always states exit code (when known), whether teardown/billing succeeded, and how to
|
|
111
|
+
// pull the receipt, so the user is never left guessing what happened.
|
|
112
|
+
function _printFinalInfo(chalk, { exitCode = undefined, teardownOk, jobId, rcptId, logsAvailable = true }) {
|
|
113
|
+
if (exitCode !== null && exitCode !== undefined) {
|
|
114
|
+
console.log(` ${chalk.bold('Exit code:')} ${exitCode !== 0 ? chalk.red(exitCode) : chalk.green(exitCode)}`);
|
|
115
|
+
}
|
|
116
|
+
console.log(` ${chalk.bold('Teardown:')} ${teardownOk ? chalk.green('succeeded') : chalk.red(`failed — run: badgr down ${jobId}`)}`);
|
|
117
|
+
console.log(` ${chalk.bold('Billing:')} ${teardownOk ? 'stopped' : 'unconfirmed — check receipt'}`);
|
|
118
|
+
if (logsAvailable) console.log(` ${chalk.bold('Logs:')} badgr logs ${jobId}`);
|
|
119
|
+
console.log(` ${chalk.bold('Job ID:')} ${jobId}`);
|
|
120
|
+
console.log(` ${chalk.bold('Receipt:')} badgr receipts ${rcptId}`);
|
|
121
|
+
}
|
|
122
|
+
|
|
108
123
|
export function classifyFailure(finalStatus, exitCode) {
|
|
109
124
|
if (finalStatus === 'failed' && (exitCode === null || exitCode === undefined)) return 'infrastructure';
|
|
110
125
|
if (exitCode !== null && exitCode !== undefined && exitCode !== 0) return 'customer_code';
|
|
@@ -128,24 +143,6 @@ function parseProviderLine(line) {
|
|
|
128
143
|
};
|
|
129
144
|
}
|
|
130
145
|
|
|
131
|
-
function renderStatusBar(chalk, { elapsedMs, ratePerHour, gpuUtil, cpuUtil, maxRuntimeMs, maxCost }) {
|
|
132
|
-
const spent = ratePerHour * (elapsedMs / 3_600_000);
|
|
133
|
-
const parts = [`⏱ ${fmtRuntime(elapsedMs)}`];
|
|
134
|
-
if (ratePerHour > 0) parts.push(`$${spent.toFixed(4)} spent`);
|
|
135
|
-
if (gpuUtil !== null) parts.push(`GPU ${gpuUtil.toFixed(0)}%`);
|
|
136
|
-
if (cpuUtil !== null) parts.push(`CPU ${cpuUtil.toFixed(0)}%`);
|
|
137
|
-
if (maxRuntimeMs) {
|
|
138
|
-
const left = Math.max(0, maxRuntimeMs - elapsedMs);
|
|
139
|
-
parts.push(`${fmtRuntime(left)} left`);
|
|
140
|
-
}
|
|
141
|
-
if (maxCost && ratePerHour > 0) {
|
|
142
|
-
const budgetLeft = Math.max(0, maxCost - spent);
|
|
143
|
-
parts.push(`$${budgetLeft.toFixed(4)} budget left`);
|
|
144
|
-
}
|
|
145
|
-
parts.push('Ctrl+C to stop');
|
|
146
|
-
return chalk.dim(' ' + parts.join(' • '));
|
|
147
|
-
}
|
|
148
|
-
|
|
149
146
|
// Wait for status to leave 'starting'/'queued'/'provisioning'.
|
|
150
147
|
// Returns the dep once it leaves startup states (or the last known state on timeout).
|
|
151
148
|
async function waitForRunning(config, depId, chalk) {
|
|
@@ -200,7 +197,7 @@ async function waitForRunning(config, depId, chalk) {
|
|
|
200
197
|
});
|
|
201
198
|
}
|
|
202
199
|
|
|
203
|
-
async function attachToJob(config, depId, { chalk, maxRuntimeMs = null, maxCost = null, ratePerHour = 0, onTeardown, isShuttingDown }) {
|
|
200
|
+
async function attachToJob(config, depId, { chalk, maxRuntimeMs = null, maxCost = null, ratePerHour = 0, onTeardown, isShuttingDown, stageLine }) {
|
|
204
201
|
const TERMINAL = new Set(['stopped', 'failed', 'completed', 'succeeded']);
|
|
205
202
|
const POLL_MS = 4000;
|
|
206
203
|
let seenContent = new Set();
|
|
@@ -209,7 +206,8 @@ async function attachToJob(config, depId, { chalk, maxRuntimeMs = null, maxCost
|
|
|
209
206
|
let gpuUtil = null;
|
|
210
207
|
let cpuUtil = null;
|
|
211
208
|
let sshShown = false;
|
|
212
|
-
let
|
|
209
|
+
let statusWord = 'running';
|
|
210
|
+
let blockLines = 0;
|
|
213
211
|
const startMs = Date.now();
|
|
214
212
|
|
|
215
213
|
// tearing: guards against double-teardown for cap/heartbeat paths within this function.
|
|
@@ -218,17 +216,17 @@ async function attachToJob(config, depId, { chalk, maxRuntimeMs = null, maxCost
|
|
|
218
216
|
let tickerInterval = null;
|
|
219
217
|
const startTicker = () => {
|
|
220
218
|
if (tickerInterval) return;
|
|
221
|
-
statusBarActive = true;
|
|
222
219
|
tickerInterval = setInterval(() => {
|
|
223
|
-
const
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
220
|
+
const elapsedSec = Math.round((Date.now() - startMs) / 1000);
|
|
221
|
+
const spend = ratePerHour * (elapsedSec / 3600);
|
|
222
|
+
blockLines = _writeBlock(blockLines, _renderLiveBlock(chalk, {
|
|
223
|
+
stageLine, elapsedSec, statusWord, spend, id: depId,
|
|
224
|
+
}));
|
|
227
225
|
}, 1000);
|
|
228
226
|
};
|
|
229
227
|
const stopTicker = () => {
|
|
230
228
|
if (tickerInterval) { clearInterval(tickerInterval); tickerInterval = null; }
|
|
231
|
-
if (
|
|
229
|
+
if (blockLines > 0) { _clearBlock(blockLines); blockLines = 0; }
|
|
232
230
|
};
|
|
233
231
|
|
|
234
232
|
try {
|
|
@@ -271,6 +269,7 @@ async function attachToJob(config, depId, { chalk, maxRuntimeMs = null, maxCost
|
|
|
271
269
|
consecutiveErrs++;
|
|
272
270
|
if (lastStatus === 'running') {
|
|
273
271
|
const lostSec = Math.round((consecutiveErrs * POLL_MS) / 1000);
|
|
272
|
+
if (consecutiveErrs >= HEARTBEAT_WARN_POLLS) statusWord = 'stuck';
|
|
274
273
|
if (consecutiveErrs === HEARTBEAT_WARN_POLLS) {
|
|
275
274
|
stopTicker();
|
|
276
275
|
console.log(chalk.yellow(`\n ⚠ No heartbeat for ${lostSec}s — cloud machine may be unresponsive`));
|
|
@@ -293,6 +292,7 @@ async function attachToJob(config, depId, { chalk, maxRuntimeMs = null, maxCost
|
|
|
293
292
|
}
|
|
294
293
|
lastStatus = status;
|
|
295
294
|
}
|
|
295
|
+
if (status === 'running') statusWord = seenContent.size === 0 ? 'no logs yet' : 'running';
|
|
296
296
|
|
|
297
297
|
try {
|
|
298
298
|
const logData = await callApi(`/deployments/${depId}/logs`, {
|
|
@@ -383,11 +383,10 @@ function _isGitHubUrl(arg) {
|
|
|
383
383
|
|
|
384
384
|
/**
|
|
385
385
|
* Zip a local directory into a temp file, returning the temp file path.
|
|
386
|
-
* Uses
|
|
386
|
+
* Uses the `archiver` package so the CLI never depends on a system `zip`
|
|
387
|
+
* binary being present (Windows, minimal containers, etc).
|
|
387
388
|
*/
|
|
388
389
|
async function _zipDirectory(dirPath, chalk) {
|
|
389
|
-
// Use archiver if available, otherwise fall back to a manual approach via
|
|
390
|
-
// a child process calling `zip` (available on Linux/macOS) or PowerShell on Windows.
|
|
391
390
|
const absDir = path.resolve(dirPath);
|
|
392
391
|
if (!fs.existsSync(absDir)) {
|
|
393
392
|
throw new Error(`Directory not found: ${absDir}`);
|
|
@@ -395,37 +394,19 @@ async function _zipDirectory(dirPath, chalk) {
|
|
|
395
394
|
|
|
396
395
|
const tmpFile = path.join(os.tmpdir(), `badgr-upload-${Date.now()}.zip`);
|
|
397
396
|
|
|
398
|
-
|
|
399
|
-
try {
|
|
400
|
-
const { default: archiver } = await import('archiver');
|
|
401
|
-
await new Promise((resolve, reject) => {
|
|
402
|
-
const output = createWriteStream(tmpFile);
|
|
403
|
-
const archive = archiver('zip', { zlib: { level: 6 } });
|
|
404
|
-
output.on('close', resolve);
|
|
405
|
-
archive.on('error', reject);
|
|
406
|
-
archive.pipe(output);
|
|
407
|
-
archive.glob('**/*', {
|
|
408
|
-
cwd: absDir,
|
|
409
|
-
dot: false,
|
|
410
|
-
ignore: [..._ZIP_EXCLUDES].map(e => `**/${e}/**`).concat([..._ZIP_EXCLUDES].map(e => e)),
|
|
411
|
-
});
|
|
412
|
-
archive.finalize();
|
|
413
|
-
});
|
|
414
|
-
return tmpFile;
|
|
415
|
-
} catch {
|
|
416
|
-
// archiver not installed — fall back to system zip command
|
|
417
|
-
}
|
|
418
|
-
|
|
419
|
-
const { spawn } = await import('child_process');
|
|
397
|
+
const { default: archiver } = await import('archiver');
|
|
420
398
|
await new Promise((resolve, reject) => {
|
|
421
|
-
|
|
422
|
-
const
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
399
|
+
const output = createWriteStream(tmpFile);
|
|
400
|
+
const archive = archiver('zip', { zlib: { level: 6 } });
|
|
401
|
+
output.on('close', resolve);
|
|
402
|
+
archive.on('error', reject);
|
|
403
|
+
archive.pipe(output);
|
|
404
|
+
archive.glob('**/*', {
|
|
405
|
+
cwd: absDir,
|
|
406
|
+
dot: false,
|
|
407
|
+
ignore: [..._ZIP_EXCLUDES].map(e => `**/${e}/**`).concat([..._ZIP_EXCLUDES].map(e => e)),
|
|
408
|
+
});
|
|
409
|
+
archive.finalize();
|
|
429
410
|
});
|
|
430
411
|
|
|
431
412
|
return tmpFile;
|
|
@@ -624,14 +605,14 @@ export async function runCommand(config, args, chalk) {
|
|
|
624
605
|
if (flags.cmd) console.log(` ${chalk.bold('Command:')} ${flags.cmd}`);
|
|
625
606
|
if (command) console.log(` ${chalk.bold('Command:')} ${command.join(' ')}`);
|
|
626
607
|
if (image) console.log(` ${chalk.bold('Image:')} ${image}`);
|
|
627
|
-
if (gpu)
|
|
628
|
-
else console.log(` ${chalk.bold('GPU:')} ${chalk.dim('auto')}`);
|
|
608
|
+
if (flags.gpu) console.log(` ${chalk.bold('GPU:')} ${gpu}`);
|
|
629
609
|
if (flags.minVram) console.log(` ${chalk.bold('Min VRAM:')} ${flags.minVram} GB`);
|
|
630
610
|
const runtimeLabel = isDefaultRuntime
|
|
631
|
-
? `${effectiveMaxRuntime}min ${chalk.dim('(default — use --max-runtime N to override)')}`
|
|
632
|
-
: `${effectiveMaxRuntime}min`;
|
|
611
|
+
? `${effectiveMaxRuntime} min ${chalk.dim('(default — use --max-runtime N to override)')}`
|
|
612
|
+
: `${effectiveMaxRuntime} min`;
|
|
613
|
+
console.log(` ${chalk.bold('Max cost:')} ${maxCost ? `$${maxCost.toFixed(2)}` : chalk.dim('none')}`);
|
|
633
614
|
console.log(` ${chalk.bold('Max runtime:')} ${runtimeLabel}`);
|
|
634
|
-
|
|
615
|
+
console.log(` ${chalk.bold('Auto-stop:')} ${maxCost ? 'enabled' : chalk.yellow('disabled — stop manually with badgr down')}`);
|
|
635
616
|
if (flags.maxPrice) console.log(` ${chalk.bold('Max price:')} $${flags.maxPrice.toFixed(2)}/hr`);
|
|
636
617
|
if (detach) console.log(` ${chalk.dim('(detached — returns immediately)')}`);
|
|
637
618
|
if (flags.env?.length) console.log(` ${chalk.bold('Env:')} ${flags.env.join(', ')}`);
|
|
@@ -655,6 +636,10 @@ export async function runCommand(config, args, chalk) {
|
|
|
655
636
|
}
|
|
656
637
|
}
|
|
657
638
|
|
|
639
|
+
// Total stages: local-path runs get a "Preparing upload" stage the others don't.
|
|
640
|
+
const STAGE_TOTAL = isLocalPath ? 5 : 4;
|
|
641
|
+
let stageN = 1;
|
|
642
|
+
|
|
658
643
|
// ── Upload local project zip (Flow 1) ─────────────────────────────────────
|
|
659
644
|
let codeUri = null;
|
|
660
645
|
if (isLocalPath) {
|
|
@@ -665,9 +650,10 @@ export async function runCommand(config, args, chalk) {
|
|
|
665
650
|
process.exitCode = 1;
|
|
666
651
|
return;
|
|
667
652
|
}
|
|
653
|
+
console.log(chalk.dim(_stageDone(stageN, STAGE_TOTAL, 'Preparing upload...')));
|
|
654
|
+
stageN++;
|
|
668
655
|
}
|
|
669
656
|
|
|
670
|
-
console.log(chalk.dim(' Finding suitable capacity...'));
|
|
671
657
|
if (process.env.BADGR_DEBUG === '1' || process.env.BADGR_DEBUG === 'true') {
|
|
672
658
|
console.log(chalk.dim(` API: ${config.baseUrl}`));
|
|
673
659
|
}
|
|
@@ -736,12 +722,10 @@ export async function runCommand(config, args, chalk) {
|
|
|
736
722
|
|
|
737
723
|
const rate = dep.cost_per_hour || 0;
|
|
738
724
|
|
|
739
|
-
console.log(chalk.dim('
|
|
740
|
-
|
|
741
|
-
console.log(
|
|
742
|
-
|
|
743
|
-
if (maxCost) console.log(` ${chalk.bold('Max cost:')} $${maxCost.toFixed(2)}`);
|
|
744
|
-
console.log(` ${chalk.bold('Receipt:')} ${chalk.dim(rcptId)}`);
|
|
725
|
+
console.log(chalk.dim(_stageDone(stageN, STAGE_TOTAL, 'Finding a working route...')));
|
|
726
|
+
stageN++;
|
|
727
|
+
console.log(chalk.dim(_stageDone(stageN, STAGE_TOTAL, 'Starting runtime...')));
|
|
728
|
+
stageN++;
|
|
745
729
|
|
|
746
730
|
if (rate > HIGH_RATE_THRESHOLD && !maxCost) {
|
|
747
731
|
console.log(chalk.yellow(`\n Selected capacity rate: $${rate.toFixed(2)}/hr`));
|
|
@@ -764,30 +748,34 @@ export async function runCommand(config, args, chalk) {
|
|
|
764
748
|
if (teardownCalled) return;
|
|
765
749
|
teardownCalled = true;
|
|
766
750
|
|
|
767
|
-
const
|
|
768
|
-
'max-runtime':
|
|
769
|
-
'max-cost':
|
|
770
|
-
'heartbeat-lost':
|
|
771
|
-
'interrupted':
|
|
751
|
+
const stageLabels = {
|
|
752
|
+
'max-runtime': 'Stopped — runtime cap reached',
|
|
753
|
+
'max-cost': 'Stopped — spend cap reached',
|
|
754
|
+
'heartbeat-lost': 'Failed — lost connection to machine',
|
|
755
|
+
'interrupted': 'Stopped',
|
|
772
756
|
};
|
|
773
|
-
console.log(labels[reason] ?? chalk.yellow('\n Stopping job...'));
|
|
774
757
|
|
|
758
|
+
let teardownOk = true;
|
|
775
759
|
try {
|
|
776
760
|
await terminateDeployment(config, dep.deployment_id);
|
|
777
761
|
} catch {
|
|
778
762
|
// terminateDeployment retries 3×; best-effort if all fail
|
|
763
|
+
teardownOk = false;
|
|
779
764
|
}
|
|
780
765
|
|
|
781
766
|
const runtimeMs = Date.now() - attachStart;
|
|
782
767
|
const finalCost = ratePerHour * (runtimeMs / 3_600_000);
|
|
783
768
|
updateReceipt(rcptId, {
|
|
784
769
|
status: reason,
|
|
785
|
-
teardownStatus: 'terminated',
|
|
770
|
+
teardownStatus: teardownOk ? 'terminated' : 'failed',
|
|
786
771
|
runtimeSeconds: Math.round(runtimeMs / 1000),
|
|
787
772
|
finalCost,
|
|
788
773
|
});
|
|
789
|
-
console.log(
|
|
790
|
-
console.log(chalk.dim(
|
|
774
|
+
console.log();
|
|
775
|
+
console.log(chalk.dim(_stage(STAGE_TOTAL, STAGE_TOTAL, stageLabels[reason] ?? 'Stopped')));
|
|
776
|
+
console.log(chalk.dim(` Runtime: ${fmtRuntime(runtimeMs)} • Estimated cost: ~$${finalCost.toFixed(2)}`));
|
|
777
|
+
_printFinalInfo(chalk, { exitCode: null, teardownOk, jobId: dep.deployment_id, rcptId });
|
|
778
|
+
console.log();
|
|
791
779
|
}
|
|
792
780
|
|
|
793
781
|
// ── SIGINT handler — installed immediately after we have a deployment ID ───
|
|
@@ -821,7 +809,8 @@ export async function runCommand(config, args, chalk) {
|
|
|
821
809
|
return;
|
|
822
810
|
}
|
|
823
811
|
|
|
824
|
-
|
|
812
|
+
const runStageLine = _stage(stageN, STAGE_TOTAL, 'Running command...');
|
|
813
|
+
console.log(chalk.dim(`\n [${stageN}/${STAGE_TOTAL}] Running command (Ctrl+C to stop)`));
|
|
825
814
|
|
|
826
815
|
attachStart = Date.now();
|
|
827
816
|
const { status: finalStatus, exitCode, runtimeMs, failureType } = await attachToJob(config, dep.deployment_id, {
|
|
@@ -831,6 +820,7 @@ export async function runCommand(config, args, chalk) {
|
|
|
831
820
|
ratePerHour,
|
|
832
821
|
onTeardown: teardown,
|
|
833
822
|
isShuttingDown: () => shuttingDown,
|
|
823
|
+
stageLine: runStageLine,
|
|
834
824
|
});
|
|
835
825
|
|
|
836
826
|
// Remove SIGINT handler — job is done (or SIGINT was handled)
|
|
@@ -839,7 +829,11 @@ export async function runCommand(config, args, chalk) {
|
|
|
839
829
|
// 'interrupted' = SIGINT handler is managing teardown + exit — don't duplicate
|
|
840
830
|
if (finalStatus === 'interrupted') return;
|
|
841
831
|
|
|
842
|
-
|
|
832
|
+
// 'capped' = max-runtime or max-cost path; teardown() already printed the full summary
|
|
833
|
+
if (finalStatus === 'capped') {
|
|
834
|
+
process.exitCode = 1;
|
|
835
|
+
return;
|
|
836
|
+
}
|
|
843
837
|
|
|
844
838
|
const finalCost = ratePerHour * (runtimeMs / 3_600_000);
|
|
845
839
|
updateReceipt(rcptId, {
|
|
@@ -851,20 +845,7 @@ export async function runCommand(config, args, chalk) {
|
|
|
851
845
|
teardownStatus: (finalStatus === 'completed' || finalStatus === 'succeeded') ? 'terminated' : 'failed',
|
|
852
846
|
});
|
|
853
847
|
|
|
854
|
-
console.log(
|
|
855
|
-
if (ratePerHour > 0) {
|
|
856
|
-
console.log(` ${chalk.bold('Cost:')} $${finalCost.toFixed(4)} (${chalk.dim(`$${ratePerHour.toFixed(2)}/hr`)})`);
|
|
857
|
-
}
|
|
858
|
-
if (exitCode !== null && exitCode !== undefined) {
|
|
859
|
-
console.log(` ${chalk.bold('Exit code:')} ${exitCode !== 0 ? chalk.red(exitCode) : chalk.green(exitCode)}`);
|
|
860
|
-
}
|
|
861
|
-
console.log(` ${chalk.bold('Receipt:')} ${chalk.dim(rcptId)}`);
|
|
862
|
-
|
|
863
|
-
// 'capped' = max-runtime or max-cost path; teardown message already printed
|
|
864
|
-
if (finalStatus === 'capped') {
|
|
865
|
-
process.exitCode = 1;
|
|
866
|
-
return;
|
|
867
|
-
}
|
|
848
|
+
console.log();
|
|
868
849
|
|
|
869
850
|
if (finalStatus === 'failed' || (exitCode !== null && exitCode !== 0)) {
|
|
870
851
|
if (failureType === 'infrastructure') {
|
|
@@ -872,16 +853,23 @@ export async function runCommand(config, args, chalk) {
|
|
|
872
853
|
} else {
|
|
873
854
|
console.error(formatCliError('JOB_FAILED', { exitCode, deploymentId: dep.deployment_id }, chalk));
|
|
874
855
|
}
|
|
856
|
+
console.log(chalk.dim(_stage(STAGE_TOTAL, STAGE_TOTAL, 'Failed')));
|
|
857
|
+
console.log(chalk.dim(` Runtime: ${fmtRuntime(runtimeMs)} • Estimated cost: ~$${finalCost.toFixed(2)}`));
|
|
858
|
+
// The container already reached a terminal state on the provider side by the
|
|
859
|
+
// time we observe it here, so billing is already stopped — no extra teardown call needed.
|
|
860
|
+
_printFinalInfo(chalk, { exitCode, teardownOk: true, jobId: dep.deployment_id, rcptId });
|
|
875
861
|
process.exitCode = exitCode ?? 1;
|
|
876
862
|
return;
|
|
877
863
|
}
|
|
878
864
|
|
|
879
865
|
if ((finalStatus === 'completed' || finalStatus === 'succeeded') && (exitCode === 0 || exitCode === null)) {
|
|
866
|
+
let teardownOk = true;
|
|
880
867
|
try {
|
|
881
868
|
await terminateDeployment(config, dep.deployment_id);
|
|
882
|
-
} catch { /* already stopped */ }
|
|
883
|
-
console.log(chalk.green(
|
|
884
|
-
console.log(chalk.dim(`
|
|
869
|
+
} catch { teardownOk = false; /* already stopped, or best-effort */ }
|
|
870
|
+
console.log(chalk.green(_stage(STAGE_TOTAL, STAGE_TOTAL, 'Complete')));
|
|
871
|
+
console.log(chalk.dim(` Runtime: ${fmtRuntime(runtimeMs)} • Estimated cost: ~$${finalCost.toFixed(2)}`));
|
|
872
|
+
_printFinalInfo(chalk, { exitCode, teardownOk, jobId: dep.deployment_id, rcptId });
|
|
885
873
|
|
|
886
874
|
if (flags.save && config.apiKey) {
|
|
887
875
|
try {
|