@thermai/flightdeck-cli 0.1.0-pilot.10 → 0.1.0-pilot.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +43 -4
- package/dist/index.js +254 -55
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -88,6 +88,20 @@ the supervisor release that Flightdeck 26.11.3 runs. Both new commands need Flig
|
|
|
88
88
|
or later, and `dataset download` also needs storage enabled for your
|
|
89
89
|
organisation. There are no new runtime dependencies.
|
|
90
90
|
|
|
91
|
+
**`0.1.0-pilot.11`** adds `--secret ENV=name` to `flightdeck submit` and
|
|
92
|
+
`flightdeck sbatch`, which gives the workload one of your organisation's secrets
|
|
93
|
+
(see "Keeping secrets"). `flightdeck logs --follow` and `slurm-test` keep going
|
|
94
|
+
through the few seconds a release restarts the backend. A refused submit now
|
|
95
|
+
says what was wrong with `--base-image` or `--provider` in plain words instead
|
|
96
|
+
of printing the API's JSON. `flightdeck outputs` and `flightdeck download` say
|
|
97
|
+
when collection after a cancel was cut off, so some outputs or logs may be
|
|
98
|
+
missing, and
|
|
99
|
+
`flightdeck logs` says when a flight's logs were deleted under the 30-day limit.
|
|
100
|
+
`flightdeck image check` checks images against the supervisor release that
|
|
101
|
+
Flightdeck 26.11.4 runs. `--secret` needs Flightdeck 26.11.4 or later and
|
|
102
|
+
secrets enabled for your organisation; an older backend ignores it. There are no
|
|
103
|
+
new runtime dependencies.
|
|
104
|
+
|
|
91
105
|
Members and organization administrators receive whole-credit balances,
|
|
92
106
|
estimates and actual customer charges, without provider-cost information.
|
|
93
107
|
Platform administrators retain their authorized monetary API fields. Credits
|
|
@@ -228,6 +242,19 @@ Environment variables (`--env`, `#SBATCH --export`) are checked before upload:
|
|
|
228
242
|
names are at most 64 characters, and names starting with `FLIGHTDECK_` belong
|
|
229
243
|
to the platform and are refused.
|
|
230
244
|
|
|
245
|
+
`--secret ENV=name` (on `submit` and `sbatch`, repeatable, up to 16) gives the
|
|
246
|
+
workload your organisation's secret `name` as the environment variable `ENV`
|
|
247
|
+
and as the read-only file `/run/flightdeck/secrets/ENV`. The value is read when
|
|
248
|
+
the flight is dispatched and never passes through the CLI. Secrets run on
|
|
249
|
+
Lambda only, so add `--provider lambda` if other providers are active. `ENV`
|
|
250
|
+
follows the `--env` rules and cannot also be given with `--env`. Logs are
|
|
251
|
+
redacted on a best-effort basis only, so never print a secret. See
|
|
252
|
+
`docs/customer-secrets.md` "Delivery to workloads".
|
|
253
|
+
|
|
254
|
+
```sh
|
|
255
|
+
flightdeck submit --provider lambda --secret HF_TOKEN=hf-token "python train.py"
|
|
256
|
+
```
|
|
257
|
+
|
|
231
258
|
An array prints a separate Flightdeck ID for each accepted task. Record them
|
|
232
259
|
all: inspect, download and cancel each flight individually (`flightdeck atc
|
|
233
260
|
<job-id>` shows them together). Partial submission
|
|
@@ -425,6 +452,7 @@ something was not saved. The flight's events say why:
|
|
|
425
452
|
| `finalization_abandoned` | Saving failed five times and was given up. | Run again. A flight whose workload exited 0 is not charged; one whose workload failed is charged like any failed flight. |
|
|
426
453
|
| `deadline_collection_abandoned` | The flight hit its time limit and saving did not finish in the time left. | Raise `--time`, or write results earlier, and run again. |
|
|
427
454
|
| `cancel_stop_failed` | The flight was cancelled but its workload could not be confirmed stopped, so its outputs may not have been saved. | Check `outputs`. Run again if they are missing. |
|
|
455
|
+
| `cancel_collection_cutoff` | The flight was cancelled and saving had not finished an hour later, so it was stopped and the instance destroyed. | Check `outputs`. Run again if they are missing. |
|
|
428
456
|
|
|
429
457
|
**Known limits.**
|
|
430
458
|
|
|
@@ -433,10 +461,10 @@ something was not saved. The flight's events say why:
|
|
|
433
461
|
run the flight again.
|
|
434
462
|
- **Outputs that were not saved are gone.** The instance is destroyed when
|
|
435
463
|
saving is given up, and anything that was only on it cannot be recovered.
|
|
436
|
-
- **A cancelled flight
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
464
|
+
- **A cancelled flight saves after it is cancelled.** Its events show
|
|
465
|
+
`cancel_collection_started` while saving and `cancel_collection_finished`
|
|
466
|
+
once it is done. Outputs appear in `outputs` when they are stored, usually
|
|
467
|
+
within a minute.
|
|
440
468
|
- **Results do not expire yet.** A saved output stays until you delete it. A
|
|
441
469
|
30-day purge after a pilot ends is planned.
|
|
442
470
|
- **One archive per flight on Lambda.** You cannot download a single file from
|
|
@@ -920,6 +948,13 @@ until the job is terminal and its stored backlog is exhausted. Every new
|
|
|
920
948
|
invocation starts at the beginning; stdout/stderr retain their respective
|
|
921
949
|
output streams.
|
|
922
950
|
|
|
951
|
+
If the API stops answering while `--follow` is running (a connection failure, or
|
|
952
|
+
HTTP 502, 503 or 504, as happens for a few seconds when a release restarts the
|
|
953
|
+
backend), the CLI says so on stderr and retries for up to a minute before it
|
|
954
|
+
gives up. The flight is not affected. A read without `--follow` does not retry.
|
|
955
|
+
`flightdeck slurm-test` retries the same way while it waits for its flights and
|
|
956
|
+
when it reads their logs and outputs. It never sends the submission twice.
|
|
957
|
+
|
|
923
958
|
Each log stream is capped at 5 MiB. On Lambda, a finished flight's stream that
|
|
924
959
|
passes the cap keeps its first 5 MiB and its last 64 KiB (each cut between
|
|
925
960
|
characters), with a `flightdeck: log truncated` marker line between them when
|
|
@@ -928,6 +963,10 @@ anything was left out, so the end of the run and any
|
|
|
928
963
|
`flightdeck logs` prints a one-line warning on stderr after the text; stdout
|
|
929
964
|
carries only the flight's own output.
|
|
930
965
|
|
|
966
|
+
Flight logs are kept for 30 days after they're written. For an older flight,
|
|
967
|
+
`flightdeck logs` prints nothing on stdout and says on stderr when the logs were
|
|
968
|
+
deleted.
|
|
969
|
+
|
|
931
970
|
**Rollout order:** deploy and verify the pagination-capable backend before
|
|
932
971
|
releasing/upgrading this CLI. A backend without the `has_more` response field
|
|
933
972
|
produces an explicit compatibility error and exit status 1, not a silently
|
package/dist/index.js
CHANGED
|
@@ -249,6 +249,8 @@ function suspensionMessage(status2, code) {
|
|
|
249
249
|
return "Your membership in this organization is suspended. Contact your organization administrator.";
|
|
250
250
|
case "organization_suspended":
|
|
251
251
|
return "This organization is suspended. Contact your organization administrator or ThermAI support.";
|
|
252
|
+
case "organization_service_ended":
|
|
253
|
+
return "This organization's service has ended, so it can't start flights. You can still list and download results and datasets during the export window. Contact your organization administrator or ThermAI support.";
|
|
252
254
|
default:
|
|
253
255
|
return void 0;
|
|
254
256
|
}
|
|
@@ -594,7 +596,7 @@ async function request(url, options) {
|
|
|
594
596
|
}
|
|
595
597
|
|
|
596
598
|
// src/image-check/runtime-facts.ts
|
|
597
|
-
var COMPILED_SUPERVISOR_IMAGE = "ghcr.io/thermai-group/flightdeck-supervisor@sha256:
|
|
599
|
+
var COMPILED_SUPERVISOR_IMAGE = "ghcr.io/thermai-group/flightdeck-supervisor@sha256:b8faf8866abd65667b5bf30b503362e8ccfb49b943ce709cdb72514a4363dfc6";
|
|
598
600
|
var COMPILED_DEFAULT_IMAGE = "ghcr.io/thermai-group/flightdeck-base:2026.10-a47b161@sha256:d2a3aa966805e24b64b8227e3dce85c22b22a8ba54b9ed158932611a7e07570c";
|
|
599
601
|
var SUPERVISOR_IMAGE_PATTERN = /^ghcr\.io\/thermai-group\/flightdeck-supervisor@sha256:[0-9a-f]{64}$/;
|
|
600
602
|
var COMPILED_RUNTIME_FACTS = {
|
|
@@ -2345,6 +2347,50 @@ function parseVersion(body) {
|
|
|
2345
2347
|
};
|
|
2346
2348
|
}
|
|
2347
2349
|
|
|
2350
|
+
// src/commands/submit-refusals.ts
|
|
2351
|
+
var MANAGED_REFERENCE_HINT = "use managed://name:alias";
|
|
2352
|
+
var CONFIGURED_PROVIDERS = /configured: (.+)$/;
|
|
2353
|
+
function serverMessage(body) {
|
|
2354
|
+
if (typeof body.message !== "string") return void 0;
|
|
2355
|
+
const message = cleanText(body.message).trim().replace(/\.$/, "");
|
|
2356
|
+
return message === "" ? void 0 : message;
|
|
2357
|
+
}
|
|
2358
|
+
function invalidImage(body) {
|
|
2359
|
+
const message = serverMessage(body);
|
|
2360
|
+
if (message === MANAGED_REFERENCE_HINT) {
|
|
2361
|
+
return "The managed image in --base-image is not a valid reference. Use managed://<name>:<alias>, for example managed://solver:latest.";
|
|
2362
|
+
}
|
|
2363
|
+
if (message !== void 0) return `The image in --base-image was refused: ${message}.`;
|
|
2364
|
+
return "The image in --base-image is not a valid reference. Use a public docker.io, ghcr.io or nvcr.io image, or managed://<name>:<alias>.";
|
|
2365
|
+
}
|
|
2366
|
+
function unknownProvider(body) {
|
|
2367
|
+
const configured = CONFIGURED_PROVIDERS.exec(serverMessage(body) ?? "")?.[1].trim();
|
|
2368
|
+
const next = configured === void 0 || configured === "" ? "Check the provider ID with your ThermAI operator" : `Use one of: ${configured}`;
|
|
2369
|
+
return `The provider in --provider is not one Flightdeck runs on. ${next}, or leave out --provider.`;
|
|
2370
|
+
}
|
|
2371
|
+
var DESCRIBERS = /* @__PURE__ */ new Map([
|
|
2372
|
+
[
|
|
2373
|
+
"managed_image_not_found",
|
|
2374
|
+
() => "The managed image or alias in --base-image was not found in your organization. Check the name with flightdeck image list and the alias with flightdeck image alias ls <name>."
|
|
2375
|
+
],
|
|
2376
|
+
[
|
|
2377
|
+
"managed_image_provider_required",
|
|
2378
|
+
() => "A managed image in --base-image runs on Lambda only. Add --provider lambda."
|
|
2379
|
+
],
|
|
2380
|
+
["invalid_image", invalidImage],
|
|
2381
|
+
["unknown_provider", unknownProvider],
|
|
2382
|
+
[
|
|
2383
|
+
"alias_target_disabled",
|
|
2384
|
+
() => "The alias in --base-image names an image version that is disabled. Ask an organization admin to point it at an enabled version with flightdeck image alias set, or use another alias."
|
|
2385
|
+
]
|
|
2386
|
+
]);
|
|
2387
|
+
function describeSubmitRefusal(body) {
|
|
2388
|
+
const dataset2 = describeSubmitDatasetError(body);
|
|
2389
|
+
if (dataset2 !== void 0) return dataset2;
|
|
2390
|
+
if (!isRecord2(body) || typeof body.error !== "string") return void 0;
|
|
2391
|
+
return DESCRIBERS.get(body.error)?.(body);
|
|
2392
|
+
}
|
|
2393
|
+
|
|
2348
2394
|
// src/commands/dataset-ref.ts
|
|
2349
2395
|
var MAX_DATASET_MOUNTS = 4;
|
|
2350
2396
|
var MAX_NAME_LENGTH = 64;
|
|
@@ -2425,6 +2471,50 @@ function datasetPlanFields(values) {
|
|
|
2425
2471
|
return datasets.length > 0 ? { datasets } : {};
|
|
2426
2472
|
}
|
|
2427
2473
|
|
|
2474
|
+
// src/commands/flight-secrets.ts
|
|
2475
|
+
var MAX_FLIGHT_SECRETS = 16;
|
|
2476
|
+
var MAX_NAME_LENGTH2 = 64;
|
|
2477
|
+
var SECRET_NAME_PATTERN = /^[a-z0-9]+(?:[._-][a-z0-9]+)*$/;
|
|
2478
|
+
var USAGE = "Give --secret as ENV=name, for example --secret HF_TOKEN=hf-token.";
|
|
2479
|
+
function collectSecret(value, previous = []) {
|
|
2480
|
+
return [...previous, value];
|
|
2481
|
+
}
|
|
2482
|
+
function parseSecretArgument(value) {
|
|
2483
|
+
const equals = value.indexOf("=");
|
|
2484
|
+
if (equals < 1 || equals === value.length - 1) throw new CliError(USAGE);
|
|
2485
|
+
const env = value.slice(0, equals);
|
|
2486
|
+
const name = value.slice(equals + 1);
|
|
2487
|
+
const issues = environmentContractIssues(Object.fromEntries([[env, ""]]), "customer");
|
|
2488
|
+
if (issues.length > 0) throw new CliError(`Invalid --secret ${JSON.stringify(env)}: ${issues.join("; ")}`);
|
|
2489
|
+
if (name.length > MAX_NAME_LENGTH2 || !SECRET_NAME_PATTERN.test(name)) {
|
|
2490
|
+
throw new CliError(
|
|
2491
|
+
`Secret names use lowercase letters and digits, joined by '.', '_' or '-', at most ${String(MAX_NAME_LENGTH2)} characters. ${USAGE}`
|
|
2492
|
+
);
|
|
2493
|
+
}
|
|
2494
|
+
return { name, env };
|
|
2495
|
+
}
|
|
2496
|
+
function parseSecretArguments(values, envVars = {}) {
|
|
2497
|
+
if (values.length > MAX_FLIGHT_SECRETS) {
|
|
2498
|
+
throw new CliError(`A flight uses at most ${String(MAX_FLIGHT_SECRETS)} secrets.`);
|
|
2499
|
+
}
|
|
2500
|
+
const seen = /* @__PURE__ */ new Set();
|
|
2501
|
+
return values.map((value) => {
|
|
2502
|
+
const secret2 = parseSecretArgument(value);
|
|
2503
|
+
if (seen.has(secret2.env)) throw new CliError(`--secret ${secret2.env} is given more than once.`);
|
|
2504
|
+
if (Object.hasOwn(envVars, secret2.env)) {
|
|
2505
|
+
throw new CliError(
|
|
2506
|
+
`${secret2.env} is already set as an environment variable (by --env, or by sbatch as SLURM_*). Give the secret another name.`
|
|
2507
|
+
);
|
|
2508
|
+
}
|
|
2509
|
+
seen.add(secret2.env);
|
|
2510
|
+
return secret2;
|
|
2511
|
+
});
|
|
2512
|
+
}
|
|
2513
|
+
function secretPlanFields(values, envVars) {
|
|
2514
|
+
const secrets = parseSecretArguments(values ?? [], envVars);
|
|
2515
|
+
return secrets.length > 0 ? { secrets } : {};
|
|
2516
|
+
}
|
|
2517
|
+
|
|
2428
2518
|
// src/commands/submit.ts
|
|
2429
2519
|
function parseDuration(input) {
|
|
2430
2520
|
const match = input.match(/^(\d+)(h|m|s)?$/);
|
|
@@ -2448,6 +2538,8 @@ function collectDataset(value, previous = []) {
|
|
|
2448
2538
|
}
|
|
2449
2539
|
async function submit(entrypoint, options) {
|
|
2450
2540
|
const datasetFields = datasetPlanFields(options.dataset);
|
|
2541
|
+
const envVars = parseEnvironmentArguments(options.env);
|
|
2542
|
+
const secretFields = secretPlanFields(options.secret, envVars);
|
|
2451
2543
|
console.log(`Packaging files...`);
|
|
2452
2544
|
let bundle;
|
|
2453
2545
|
try {
|
|
@@ -2465,10 +2557,11 @@ async function submit(entrypoint, options) {
|
|
|
2465
2557
|
gpu_type: options.gpu,
|
|
2466
2558
|
gpu_count: parseInt(options.gpuCount, 10),
|
|
2467
2559
|
max_duration_s: parseDuration(options.time),
|
|
2468
|
-
env_vars:
|
|
2560
|
+
env_vars: envVars,
|
|
2469
2561
|
...options.baseImage ? { base_image: options.baseImage } : {},
|
|
2470
2562
|
...options.provider ? { requested_provider: options.provider } : {},
|
|
2471
|
-
...datasetFields
|
|
2563
|
+
...datasetFields,
|
|
2564
|
+
...secretFields
|
|
2472
2565
|
};
|
|
2473
2566
|
console.log(`Submitting to FlightDeck...`);
|
|
2474
2567
|
try {
|
|
@@ -2488,7 +2581,7 @@ Track with: flightdeck status ${body.flight_id}`);
|
|
|
2488
2581
|
if (err instanceof SubmitError) {
|
|
2489
2582
|
console.error(
|
|
2490
2583
|
`Submit failed (${err.status}):`,
|
|
2491
|
-
|
|
2584
|
+
describeSubmitRefusal(err.body) ?? JSON.stringify(err.body, null, 2)
|
|
2492
2585
|
);
|
|
2493
2586
|
process.exit(1);
|
|
2494
2587
|
}
|
|
@@ -3030,34 +3123,33 @@ function buildSbatchPlans(args) {
|
|
|
3030
3123
|
const extraEnv = parseEnvVars(args.options.env);
|
|
3031
3124
|
let maxDurationS = args.directives.maxDurationS;
|
|
3032
3125
|
if (maxDurationS !== void 0 && maxDurationS > MAX_DURATION_S) maxDurationS = MAX_DURATION_S;
|
|
3033
|
-
return taskIds.map((taskId) =>
|
|
3034
|
-
taskId,
|
|
3035
|
-
|
|
3036
|
-
|
|
3037
|
-
|
|
3038
|
-
|
|
3039
|
-
|
|
3040
|
-
|
|
3041
|
-
|
|
3042
|
-
|
|
3043
|
-
|
|
3044
|
-
|
|
3045
|
-
|
|
3046
|
-
|
|
3047
|
-
|
|
3048
|
-
|
|
3049
|
-
|
|
3050
|
-
|
|
3051
|
-
|
|
3052
|
-
|
|
3053
|
-
|
|
3054
|
-
|
|
3055
|
-
|
|
3056
|
-
|
|
3057
|
-
|
|
3058
|
-
|
|
3059
|
-
|
|
3060
|
-
}));
|
|
3126
|
+
return taskIds.map((taskId) => {
|
|
3127
|
+
const envVars = slurmEnvironment({ directives: args.directives, taskId, jobId: args.jobId, jobName, extraEnv });
|
|
3128
|
+
return {
|
|
3129
|
+
taskId,
|
|
3130
|
+
plan: {
|
|
3131
|
+
name: taskId === null ? jobName : `${jobName}[${taskId}]`,
|
|
3132
|
+
entrypoint: buildEntrypoint({
|
|
3133
|
+
relativeScript: args.relativeScript,
|
|
3134
|
+
workingDirectory: args.directives.workingDirectory,
|
|
3135
|
+
outputPath: args.directives.outputPath,
|
|
3136
|
+
errorPath: args.directives.errorPath,
|
|
3137
|
+
jobId: args.jobId,
|
|
3138
|
+
jobName,
|
|
3139
|
+
taskId
|
|
3140
|
+
}),
|
|
3141
|
+
requirements_file: args.options.requirements,
|
|
3142
|
+
gpu_type: args.directives.gpuType,
|
|
3143
|
+
gpu_count: args.directives.gpuCount,
|
|
3144
|
+
max_duration_s: maxDurationS,
|
|
3145
|
+
env_vars: envVars,
|
|
3146
|
+
base_image: args.options.baseImage,
|
|
3147
|
+
requested_provider: args.options.provider,
|
|
3148
|
+
// Checked against the SLURM_* variables too, which the API would see as a clash.
|
|
3149
|
+
...secretPlanFields(args.options.secret, envVars)
|
|
3150
|
+
}
|
|
3151
|
+
};
|
|
3152
|
+
});
|
|
3061
3153
|
}
|
|
3062
3154
|
function trackWithLine(submitted) {
|
|
3063
3155
|
const hint = `Track with: flightdeck status ${submitted[0].flightId}`;
|
|
@@ -3203,7 +3295,9 @@ Packaging files...`);
|
|
|
3203
3295
|
} catch (err) {
|
|
3204
3296
|
failures += 1;
|
|
3205
3297
|
if (err instanceof SubmitError) {
|
|
3206
|
-
console.error(
|
|
3298
|
+
console.error(
|
|
3299
|
+
` ${task.plan.name}: failed (${err.status}) ${describeSubmitRefusal(err.body) ?? JSON.stringify(err.body)}`
|
|
3300
|
+
);
|
|
3207
3301
|
} else {
|
|
3208
3302
|
console.error(` ${task.plan.name}: ${err.message}`);
|
|
3209
3303
|
}
|
|
@@ -3294,6 +3388,9 @@ async function status(flightId) {
|
|
|
3294
3388
|
|
|
3295
3389
|
// src/log-reader.ts
|
|
3296
3390
|
var TRUNCATED_LOG_NOTICE = "Warning: this log is truncated: a stream passed the 5 MiB per-stream log cap, or the log could not be read in full.";
|
|
3391
|
+
function expiredLogNotice(expiredAt) {
|
|
3392
|
+
return `This flight's logs were deleted on ${expiredAt.slice(0, "YYYY-MM-DD".length)}. Flight logs are kept for 30 days.`;
|
|
3393
|
+
}
|
|
3297
3394
|
async function drainLogs(follow, deps) {
|
|
3298
3395
|
let cursor;
|
|
3299
3396
|
const seen = /* @__PURE__ */ new Set();
|
|
@@ -3306,7 +3403,7 @@ async function drainLogs(follow, deps) {
|
|
|
3306
3403
|
"This backend does not support complete log pagination. Deploy the updated backend before using this CLI log command."
|
|
3307
3404
|
);
|
|
3308
3405
|
}
|
|
3309
|
-
if (!Array.isArray(body.logs) || !body.logs.every((entry) => entry && typeof entry.stream === "string" && typeof entry.content === "string") || typeof body.complete !== "boolean" || typeof body.next_cursor !== "string" || !body.next_cursor || body.has_more && body.complete || body.truncated !== void 0 && typeof body.truncated !== "boolean") {
|
|
3406
|
+
if (!Array.isArray(body.logs) || !body.logs.every((entry) => entry && typeof entry.stream === "string" && typeof entry.content === "string") || typeof body.complete !== "boolean" || typeof body.next_cursor !== "string" || !body.next_cursor || body.has_more && body.complete || body.truncated !== void 0 && typeof body.truncated !== "boolean" || body.expired_at !== void 0 && body.expired_at !== null && typeof body.expired_at !== "string") {
|
|
3310
3407
|
throw new CliError("Invalid log pagination response from backend.");
|
|
3311
3408
|
}
|
|
3312
3409
|
if (body.has_more && (body.next_cursor === cursor || seen.has(body.next_cursor))) {
|
|
@@ -3319,6 +3416,10 @@ async function drainLogs(follow, deps) {
|
|
|
3319
3416
|
seen.add(cursor);
|
|
3320
3417
|
continue;
|
|
3321
3418
|
}
|
|
3419
|
+
if (typeof body.expired_at === "string") {
|
|
3420
|
+
deps.notice(expiredLogNotice(body.expired_at));
|
|
3421
|
+
return;
|
|
3422
|
+
}
|
|
3322
3423
|
if (truncated && !truncationNoticed) {
|
|
3323
3424
|
deps.notice(TRUNCATED_LOG_NOTICE);
|
|
3324
3425
|
truncationNoticed = true;
|
|
@@ -3329,7 +3430,61 @@ async function drainLogs(follow, deps) {
|
|
|
3329
3430
|
}
|
|
3330
3431
|
}
|
|
3331
3432
|
|
|
3433
|
+
// src/restart-retry.ts
|
|
3434
|
+
var BAD_GATEWAY = 502;
|
|
3435
|
+
var SERVICE_UNAVAILABLE = 503;
|
|
3436
|
+
var GATEWAY_TIMEOUT = 504;
|
|
3437
|
+
var RESTART_STATUS = /* @__PURE__ */ new Set([BAD_GATEWAY, SERVICE_UNAVAILABLE, GATEWAY_TIMEOUT]);
|
|
3438
|
+
var GIVE_UP_AFTER_MS = 6e4;
|
|
3439
|
+
var FIRST_DELAY_MS = 1e3;
|
|
3440
|
+
var MAX_DELAY_MS = 8e3;
|
|
3441
|
+
var RESTART_NOTICE = "The Flightdeck API is not answering. Retrying for up to a minute.";
|
|
3442
|
+
var terminalRestartRetry = {
|
|
3443
|
+
now: () => Date.now(),
|
|
3444
|
+
sleep: (ms) => new Promise((resolve6) => setTimeout(resolve6, ms)),
|
|
3445
|
+
notice(message) {
|
|
3446
|
+
process.stderr.write(`${message}
|
|
3447
|
+
`);
|
|
3448
|
+
}
|
|
3449
|
+
};
|
|
3450
|
+
async function attempt(send) {
|
|
3451
|
+
try {
|
|
3452
|
+
return { response: await send() };
|
|
3453
|
+
} catch (error) {
|
|
3454
|
+
if (error instanceof NetworkError) return { failure: error };
|
|
3455
|
+
throw error;
|
|
3456
|
+
}
|
|
3457
|
+
}
|
|
3458
|
+
function settle(result) {
|
|
3459
|
+
if ("failure" in result) throw result.failure;
|
|
3460
|
+
return result.response;
|
|
3461
|
+
}
|
|
3462
|
+
async function readThroughRestart(send, deps = terminalRestartRetry) {
|
|
3463
|
+
let deadline;
|
|
3464
|
+
let delayMs = FIRST_DELAY_MS;
|
|
3465
|
+
for (; ; ) {
|
|
3466
|
+
const result = await attempt(send);
|
|
3467
|
+
if ("response" in result && !RESTART_STATUS.has(result.response.status)) return result.response;
|
|
3468
|
+
const now = deps.now();
|
|
3469
|
+
if (deadline === void 0) {
|
|
3470
|
+
deadline = now + GIVE_UP_AFTER_MS;
|
|
3471
|
+
deps.notice(RESTART_NOTICE);
|
|
3472
|
+
}
|
|
3473
|
+
if (now >= deadline) return settle(result);
|
|
3474
|
+
if ("response" in result) await result.response.body?.cancel();
|
|
3475
|
+
await deps.sleep(Math.min(delayMs, deadline - now));
|
|
3476
|
+
delayMs = Math.min(MAX_DELAY_MS, delayMs * 2);
|
|
3477
|
+
}
|
|
3478
|
+
}
|
|
3479
|
+
|
|
3332
3480
|
// src/commands/logs.ts
|
|
3481
|
+
async function readLogPage(flightId, cursor, follow, deps = { request: apiRequest }) {
|
|
3482
|
+
const query = cursor ? `?cursor=${encodeURIComponent(cursor)}` : "";
|
|
3483
|
+
const send = () => deps.request("GET", `/flights/${flightId}/logs${query}`);
|
|
3484
|
+
const response = follow ? await readThroughRestart(send, deps.retry) : await send();
|
|
3485
|
+
if (!response.ok) throw httpError("Log request failed", response.status);
|
|
3486
|
+
return await response.json();
|
|
3487
|
+
}
|
|
3333
3488
|
var terminalLogOutput = {
|
|
3334
3489
|
write(stream, content) {
|
|
3335
3490
|
(stream === "stderr" ? process.stderr : process.stdout).write(content);
|
|
@@ -3340,15 +3495,10 @@ var terminalLogOutput = {
|
|
|
3340
3495
|
}
|
|
3341
3496
|
};
|
|
3342
3497
|
async function logs(flightId, options) {
|
|
3498
|
+
const follow = options.follow ?? false;
|
|
3343
3499
|
try {
|
|
3344
|
-
await drainLogs(
|
|
3345
|
-
|
|
3346
|
-
const query = cursor ? `?cursor=${encodeURIComponent(cursor)}` : "";
|
|
3347
|
-
const response = await apiRequest("GET", `/flights/${flightId}/logs${query}`);
|
|
3348
|
-
if (!response.ok) throw httpError("Log request failed", response.status);
|
|
3349
|
-
const body = await response.json();
|
|
3350
|
-
return body;
|
|
3351
|
-
},
|
|
3500
|
+
await drainLogs(follow, {
|
|
3501
|
+
readPage: (cursor) => readLogPage(flightId, cursor, follow),
|
|
3352
3502
|
...terminalLogOutput,
|
|
3353
3503
|
wait: () => new Promise((resolve6) => setTimeout(resolve6, 2e3))
|
|
3354
3504
|
});
|
|
@@ -5093,6 +5243,9 @@ async function atc(target, options) {
|
|
|
5093
5243
|
|
|
5094
5244
|
// src/output-notices.ts
|
|
5095
5245
|
var SIZE_LIMIT_EVENT = "outputs_too_large";
|
|
5246
|
+
var CUTOFF_EVENT = "cancel_collection_cutoff";
|
|
5247
|
+
var FINISHED_EVENT = "cancel_collection_finished";
|
|
5248
|
+
var SECONDS_PER_MINUTE2 = 60;
|
|
5096
5249
|
var MAY_BE_MISSING = "so some outputs or logs may be missing.";
|
|
5097
5250
|
var BYTES_PER_KIB = 1024;
|
|
5098
5251
|
var BYTES_PER_MB = BYTES_PER_KIB * BYTES_PER_KIB;
|
|
@@ -5111,17 +5264,29 @@ var COLLECTION_ENDED = /* @__PURE__ */ new Map([
|
|
|
5111
5264
|
[
|
|
5112
5265
|
"cancel_stop_failed",
|
|
5113
5266
|
() => "The flight was cancelled but its workload could not be confirmed stopped, so its outputs may not have been collected."
|
|
5114
|
-
]
|
|
5267
|
+
],
|
|
5268
|
+
[CUTOFF_EVENT, (metadata) => cutoffNotice(metadata.backstop_s)]
|
|
5115
5269
|
]);
|
|
5270
|
+
function cutoffNotice(backstopS) {
|
|
5271
|
+
if (typeof backstopS !== "number" || !Number.isSafeInteger(backstopS) || backstopS < SECONDS_PER_MINUTE2) {
|
|
5272
|
+
return `Collection was cut off after the flight was cancelled, ${MAY_BE_MISSING}`;
|
|
5273
|
+
}
|
|
5274
|
+
const minutes = String(Math.floor(backstopS / SECONDS_PER_MINUTE2));
|
|
5275
|
+
return `Collection stopped ${minutes} minutes after the flight was cancelled, ${MAY_BE_MISSING}`;
|
|
5276
|
+
}
|
|
5116
5277
|
function sizeLimitNotice(metadata) {
|
|
5117
5278
|
const { filename, max_bytes: maxBytes } = metadata;
|
|
5118
5279
|
const name = typeof filename === "string" ? sanitize(filename).trim() : "";
|
|
5119
5280
|
const limit = typeof maxBytes === "number" && Number.isSafeInteger(maxBytes) && maxBytes > 0 && maxBytes % BYTES_PER_MB === 0 ? `the ${String(maxBytes / BYTES_PER_MB)} MB result limit` : "the result size limit";
|
|
5120
5281
|
return `${name ? `Output ${name}` : "An output"} exceeds ${limit} and was not stored.`;
|
|
5121
5282
|
}
|
|
5283
|
+
function withoutOvertakenCutoff(events) {
|
|
5284
|
+
if (!events.some((event) => event.type === FINISHED_EVENT)) return events;
|
|
5285
|
+
return events.filter((event) => event.type !== CUTOFF_EVENT);
|
|
5286
|
+
}
|
|
5122
5287
|
function outputNotices(events, selection = {}) {
|
|
5123
5288
|
const notices = [];
|
|
5124
|
-
for (const event of events) {
|
|
5289
|
+
for (const event of withoutOvertakenCutoff(events)) {
|
|
5125
5290
|
const metadata = event.metadata ?? {};
|
|
5126
5291
|
if (event.type === SIZE_LIMIT_EVENT) {
|
|
5127
5292
|
if (selection.attemptSeq !== void 0 && metadata.attempt_seq !== selection.attemptSeq) continue;
|
|
@@ -5205,7 +5370,7 @@ var LOCK_ATTEMPTS = 3;
|
|
|
5205
5370
|
async function acquireLock(lock, target) {
|
|
5206
5371
|
const token = randomUUID();
|
|
5207
5372
|
const owner = JSON.stringify({ pid: process.pid, host: hostname(), token });
|
|
5208
|
-
for (let
|
|
5373
|
+
for (let attempt2 = 0; attempt2 < LOCK_ATTEMPTS; attempt2++) {
|
|
5209
5374
|
try {
|
|
5210
5375
|
await writeFile(lock, owner, { flag: "wx", mode: 384 });
|
|
5211
5376
|
return token;
|
|
@@ -5619,16 +5784,26 @@ function positiveSeconds(value, option) {
|
|
|
5619
5784
|
}
|
|
5620
5785
|
return Number(value);
|
|
5621
5786
|
}
|
|
5622
|
-
async function readFlight(flightId) {
|
|
5623
|
-
const response = await
|
|
5787
|
+
async function readFlight(flightId, deps = { request: apiRequest }) {
|
|
5788
|
+
const response = await readThroughRestart(() => deps.request("GET", `/flights/${flightId}`), deps.retry);
|
|
5624
5789
|
if (!response.ok) throw httpError("Unable to read conformance flight", response.status);
|
|
5625
5790
|
const body = await response.json();
|
|
5626
5791
|
return body;
|
|
5627
5792
|
}
|
|
5628
|
-
async function verifyResults(flightId, taskId) {
|
|
5793
|
+
async function verifyResults(flightId, taskId, deps = { request: apiRequest }) {
|
|
5794
|
+
const retry = deps.retry ?? terminalRestartRetry;
|
|
5795
|
+
let noticed = false;
|
|
5796
|
+
const once = {
|
|
5797
|
+
...retry,
|
|
5798
|
+
notice(message) {
|
|
5799
|
+
if (!noticed) retry.notice(message);
|
|
5800
|
+
noticed = true;
|
|
5801
|
+
}
|
|
5802
|
+
};
|
|
5803
|
+
const read = (path) => readThroughRestart(() => deps.request("GET", path), once);
|
|
5629
5804
|
const [logsResponse, outputsResponse] = await Promise.all([
|
|
5630
|
-
|
|
5631
|
-
|
|
5805
|
+
read(`/flights/${flightId}/logs`),
|
|
5806
|
+
read(`/flights/${flightId}/outputs`)
|
|
5632
5807
|
]);
|
|
5633
5808
|
if (!logsResponse.ok) throw httpError("Unable to read conformance logs", logsResponse.status);
|
|
5634
5809
|
if (!outputsResponse.ok) throw httpError("Unable to read conformance outputs", outputsResponse.status);
|
|
@@ -8280,7 +8455,7 @@ async function sendPart(context, job, signal) {
|
|
|
8280
8455
|
const { state, number, checksum } = job.task;
|
|
8281
8456
|
const range = partRange(state.file.size, state.file.partSize, number);
|
|
8282
8457
|
if (job.url.expiresAt - context.now() < MIN_URL_LIFE_MS) return { kind: "stale", number };
|
|
8283
|
-
for (let
|
|
8458
|
+
for (let attempt2 = 1; ; attempt2 += 1) {
|
|
8284
8459
|
try {
|
|
8285
8460
|
const sent = await context.putPart({
|
|
8286
8461
|
url: job.url.url,
|
|
@@ -8294,12 +8469,12 @@ async function sendPart(context, job, signal) {
|
|
|
8294
8469
|
signal.throwIfAborted();
|
|
8295
8470
|
if (error instanceof PartRejectedError && error.status === HTTP_FORBIDDEN) return { kind: "stale", number };
|
|
8296
8471
|
if (error instanceof PartRejectedError && error.status === HTTP_BAD_REQUEST) throw changedWhileUploading();
|
|
8297
|
-
if (
|
|
8472
|
+
if (attempt2 >= MAX_PUT_ATTEMPTS) {
|
|
8298
8473
|
throw new CliError(
|
|
8299
8474
|
`Part ${String(number)} of ${state.entry.path} failed ${String(MAX_PUT_ATTEMPTS)} times. Check the connection and run the same push again to resume.`
|
|
8300
8475
|
);
|
|
8301
8476
|
}
|
|
8302
|
-
await context.client.sleep(RETRY_BACKOFF_MS *
|
|
8477
|
+
await context.client.sleep(RETRY_BACKOFF_MS * attempt2, signal);
|
|
8303
8478
|
}
|
|
8304
8479
|
}
|
|
8305
8480
|
}
|
|
@@ -9155,9 +9330,25 @@ async function secretRevoke(name, options, command) {
|
|
|
9155
9330
|
});
|
|
9156
9331
|
}
|
|
9157
9332
|
|
|
9333
|
+
// src/banner.ts
|
|
9334
|
+
var BANNER_LINES = [
|
|
9335
|
+
"___________________ ______ _____________ ______",
|
|
9336
|
+
"___ ____/__ /__(_)______ ___ /___ /___ __ \\_____________ /__",
|
|
9337
|
+
"__ /_ __ /__ /__ __ `/_ __ \\ __/_ / / / _ \\ ___/_ //_/",
|
|
9338
|
+
"_ __/ _ / _ / _ /_/ /_ / / / /_ _ /_/ // __/ /__ _ ,<",
|
|
9339
|
+
"/_/ /_/ /_/ _\\__, / /_/ /_/\\__/ /_____/ \\___/\\___/ /_/|_|",
|
|
9340
|
+
" /____/"
|
|
9341
|
+
];
|
|
9342
|
+
var INDENT = " ";
|
|
9343
|
+
function helpBanner(toStderr) {
|
|
9344
|
+
const stream = toStderr ? process.stderr : process.stdout;
|
|
9345
|
+
return stream.isTTY ? `${BANNER_LINES.map((line) => `${INDENT}${line}`).join("\n")}
|
|
9346
|
+
` : "";
|
|
9347
|
+
}
|
|
9348
|
+
|
|
9158
9349
|
// src/index.ts
|
|
9159
9350
|
var program = new Command();
|
|
9160
|
-
program.name("flightdeck").description("FlightDeck CLI \u2014 submit GPU compute jobs").option("--debug", "include safe error type/code and CLI frame diagnostics (no raw stack or credentials)").addHelpText("after", "\nRun flightdeck --version, or flightdeck -V, on its own to print the CLI version.");
|
|
9351
|
+
program.name("flightdeck").description("FlightDeck CLI \u2014 submit GPU compute jobs").option("--debug", "include safe error type/code and CLI frame diagnostics (no raw stack or credentials)").addHelpText("before", ({ error }) => helpBanner(error)).addHelpText("after", "\nRun flightdeck --version, or flightdeck -V, on its own to print the CLI version.");
|
|
9161
9352
|
var cliVersion = JSON.parse(readFileSync6(new URL("../package.json", import.meta.url), "utf8")).version;
|
|
9162
9353
|
program.command("login").description("Log in through the FlightDeck portal").option("--api-url <url>", "FlightDeck API URL", DEFAULT_API_URL).option("--name <name>", "name for this CLI credential (default: flightdeck-cli@<hostname>)").option("--admin", "authorise this workstation to list flights as an administrator").option("--images", "authorise this workstation to publish and list managed container images").option("--storage", "authorise this workstation to upload, list and mount managed datasets").option("--secrets", "authorise this workstation to list, create, rotate and revoke secrets").action(login);
|
|
9163
9354
|
program.command("logout").description("Remove the local login and attempt server credential revocation").option("--local", "remove the local login without revoking the credential").action(logout);
|
|
@@ -9254,11 +9445,19 @@ program.command("submit").description("Submit a job from local files").argument(
|
|
|
9254
9445
|
"--dataset <ref>",
|
|
9255
9446
|
"mount a dataset version read-only under /data/<mount-name>: <name>:<label>[:<mount-name>] or a version ID; repeat for up to 4",
|
|
9256
9447
|
collectDataset
|
|
9448
|
+
).option(
|
|
9449
|
+
"--secret <ENV=name>",
|
|
9450
|
+
"give the workload your organisation's secret <name> as environment variable ENV and as /run/flightdeck/secrets/ENV (Lambda only); repeat for more",
|
|
9451
|
+
collectSecret
|
|
9257
9452
|
).action(submit);
|
|
9258
9453
|
program.command("sbatch").description("Submit a Slurm batch script (#SBATCH directives, incl. job arrays)").argument("<script>", "path to the .slurm/.sh batch script").option("-f, --files <glob...>", "files to include (default: current directory; see .flightdeckignore)").option("-r, --requirements <file>", "pip requirements file to install before the script runs").option(
|
|
9259
9454
|
"--base-image <image>",
|
|
9260
9455
|
"image to run in (default: the official Flightdeck base); any linux/amd64 image meeting the workload runtime contract (see /docs/images)"
|
|
9261
|
-
).option("--provider <id>", "pin every task to one compute provider (default: operator's order)").option("-n, --name <name>", "override job name (default: #SBATCH --job-name)").option("-e, --env <vars...>", "extra environment variables (KEY=VALUE)").option(
|
|
9456
|
+
).option("--provider <id>", "pin every task to one compute provider (default: operator's order)").option("-n, --name <name>", "override job name (default: #SBATCH --job-name)").option("-e, --env <vars...>", "extra environment variables (KEY=VALUE)").option(
|
|
9457
|
+
"--secret <ENV=name>",
|
|
9458
|
+
"give every task your organisation's secret <name> as environment variable ENV (Lambda only); repeat for more",
|
|
9459
|
+
collectSecret
|
|
9460
|
+
).option("--dry-run", "parse and show what would be submitted, without submitting").option("--ignore-array-limit", "submit every array task even if %N asks for fewer at a time").action(async (script, options) => {
|
|
9262
9461
|
await sbatch(script, options);
|
|
9263
9462
|
});
|
|
9264
9463
|
program.command("slurm-test").description("Run the provider-neutral Slurm conformance workload").option("--timeout <seconds>", "maximum time to wait for all tasks", "1800").option("--poll-interval <seconds>", "status polling interval", "5").action(slurmTest);
|