@thermai/flightdeck-cli 0.1.0-pilot.10 → 0.1.0-pilot.11

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +43 -4
  2. package/dist/index.js +254 -55
  3. package/package.json +1 -1
package/README.md CHANGED
@@ -88,6 +88,20 @@ the supervisor release that Flightdeck 26.11.3 runs. Both new commands need Flig
88
88
  or later, and `dataset download` also needs storage enabled for your
89
89
  organisation. There are no new runtime dependencies.
90
90
 
91
+ **`0.1.0-pilot.11`** adds `--secret ENV=name` to `flightdeck submit` and
92
+ `flightdeck sbatch`, which gives the workload one of your organisation's secrets
93
+ (see "Keeping secrets"). `flightdeck logs --follow` and `slurm-test` keep going
94
+ through the few seconds a release restarts the backend. A refused submit now
95
+ says what was wrong with `--base-image` or `--provider` in plain words instead
96
+ of printing the API's JSON. `flightdeck outputs` and `flightdeck download` say
97
+ when collection after a cancel was cut off, so some outputs or logs may be
98
+ missing, and
99
+ `flightdeck logs` says when a flight's logs were deleted under the 30-day limit.
100
+ `flightdeck image check` checks images against the supervisor release that
101
+ Flightdeck 26.11.4 runs. `--secret` needs Flightdeck 26.11.4 or later and
102
+ secrets enabled for your organisation; an older backend ignores it. There are no
103
+ new runtime dependencies.
104
+
91
105
  Members and organization administrators receive whole-credit balances,
92
106
  estimates and actual customer charges, without provider-cost information.
93
107
  Platform administrators retain their authorized monetary API fields. Credits
@@ -228,6 +242,19 @@ Environment variables (`--env`, `#SBATCH --export`) are checked before upload:
228
242
  names are at most 64 characters, and names starting with `FLIGHTDECK_` belong
229
243
  to the platform and are refused.
230
244
 
245
+ `--secret ENV=name` (on `submit` and `sbatch`, repeatable, up to 16) gives the
246
+ workload your organisation's secret `name` as the environment variable `ENV`
247
+ and as the read-only file `/run/flightdeck/secrets/ENV`. The value is read when
248
+ the flight is dispatched and never passes through the CLI. Secrets run on
249
+ Lambda only, so add `--provider lambda` if other providers are active. `ENV`
250
+ follows the `--env` rules and cannot also be given with `--env`. Logs are
251
+ redacted on a best-effort basis only, so never print a secret. See
252
+ `docs/customer-secrets.md` "Delivery to workloads".
253
+
254
+ ```sh
255
+ flightdeck submit --provider lambda --secret HF_TOKEN=hf-token "python train.py"
256
+ ```
257
+
231
258
  An array prints a separate Flightdeck ID for each accepted task. Record them
232
259
  all: inspect, download and cancel each flight individually (`flightdeck atc
233
260
  <job-id>` shows them together). Partial submission
@@ -425,6 +452,7 @@ something was not saved. The flight's events say why:
425
452
  | `finalization_abandoned` | Saving failed five times and was given up. | Run again. A flight whose workload exited 0 is not charged; one whose workload failed is charged like any failed flight. |
426
453
  | `deadline_collection_abandoned` | The flight hit its time limit and saving did not finish in the time left. | Raise `--time`, or write results earlier, and run again. |
427
454
  | `cancel_stop_failed` | The flight was cancelled but its workload could not be confirmed stopped, so its outputs may not have been saved. | Check `outputs`. Run again if they are missing. |
455
+ | `cancel_collection_cutoff` | The flight was cancelled and saving had not finished an hour later, so it was stopped and the instance destroyed. | Check `outputs`. Run again if they are missing. |
428
456
 
429
457
  **Known limits.**
430
458
 
@@ -433,10 +461,10 @@ something was not saved. The flight's events say why:
433
461
  run the flight again.
434
462
  - **Outputs that were not saved are gone.** The instance is destroyed when
435
463
  saving is given up, and anything that was only on it cannot be recovered.
436
- - **A cancelled flight gives no sign while it saves.** Its events stop at
437
- "Flight cancelled". Outputs appear in `outputs` when they are stored, usually
438
- within a minute. If saving is still going after an hour the instance is
439
- destroyed, with no outputs and no event to say so.
464
+ - **A cancelled flight saves after it is cancelled.** Its events show
465
+ `cancel_collection_started` while saving and `cancel_collection_finished`
466
+ once it is done. Outputs appear in `outputs` when they are stored, usually
467
+ within a minute.
440
468
  - **Results do not expire yet.** A saved output stays until you delete it. A
441
469
  30-day purge after a pilot ends is planned.
442
470
  - **One archive per flight on Lambda.** You cannot download a single file from
@@ -920,6 +948,13 @@ until the job is terminal and its stored backlog is exhausted. Every new
920
948
  invocation starts at the beginning; stdout/stderr retain their respective
921
949
  output streams.
922
950
 
951
+ If the API stops answering while `--follow` is running (a connection failure, or
952
+ HTTP 502, 503 or 504, as happens for a few seconds when a release restarts the
953
+ backend), the CLI says so on stderr and retries for up to a minute before it
954
+ gives up. The flight is not affected. A read without `--follow` does not retry.
955
+ `flightdeck slurm-test` retries the same way while it waits for its flights and
956
+ when it reads their logs and outputs. It never sends the submission twice.
957
+
923
958
  Each log stream is capped at 5 MiB. On Lambda, a finished flight's stream that
924
959
  passes the cap keeps its first 5 MiB and its last 64 KiB (each cut between
925
960
  characters), with a `flightdeck: log truncated` marker line between them when
@@ -928,6 +963,10 @@ anything was left out, so the end of the run and any
928
963
  `flightdeck logs` prints a one-line warning on stderr after the text; stdout
929
964
  carries only the flight's own output.
930
965
 
966
+ Flight logs are kept for 30 days after they're written. For an older flight,
967
+ `flightdeck logs` prints nothing on stdout and says on stderr when the logs were
968
+ deleted.
969
+
931
970
  **Rollout order:** deploy and verify the pagination-capable backend before
932
971
  releasing/upgrading this CLI. A backend without the `has_more` response field
933
972
  produces an explicit compatibility error and exit status 1, not a silently
package/dist/index.js CHANGED
@@ -249,6 +249,8 @@ function suspensionMessage(status2, code) {
249
249
  return "Your membership in this organization is suspended. Contact your organization administrator.";
250
250
  case "organization_suspended":
251
251
  return "This organization is suspended. Contact your organization administrator or ThermAI support.";
252
+ case "organization_service_ended":
253
+ return "This organization's service has ended, so it can't start flights. You can still list and download results and datasets during the export window. Contact your organization administrator or ThermAI support.";
252
254
  default:
253
255
  return void 0;
254
256
  }
@@ -594,7 +596,7 @@ async function request(url, options) {
594
596
  }
595
597
 
596
598
  // src/image-check/runtime-facts.ts
597
- var COMPILED_SUPERVISOR_IMAGE = "ghcr.io/thermai-group/flightdeck-supervisor@sha256:0269d0e25d4b511f75e2cf4427a1adc4ef74f2bcfee038e55f2cd8b81c0e8e62";
599
+ var COMPILED_SUPERVISOR_IMAGE = "ghcr.io/thermai-group/flightdeck-supervisor@sha256:b8faf8866abd65667b5bf30b503362e8ccfb49b943ce709cdb72514a4363dfc6";
598
600
  var COMPILED_DEFAULT_IMAGE = "ghcr.io/thermai-group/flightdeck-base:2026.10-a47b161@sha256:d2a3aa966805e24b64b8227e3dce85c22b22a8ba54b9ed158932611a7e07570c";
599
601
  var SUPERVISOR_IMAGE_PATTERN = /^ghcr\.io\/thermai-group\/flightdeck-supervisor@sha256:[0-9a-f]{64}$/;
600
602
  var COMPILED_RUNTIME_FACTS = {
@@ -2345,6 +2347,50 @@ function parseVersion(body) {
2345
2347
  };
2346
2348
  }
2347
2349
 
2350
+ // src/commands/submit-refusals.ts
2351
+ var MANAGED_REFERENCE_HINT = "use managed://name:alias";
2352
+ var CONFIGURED_PROVIDERS = /configured: (.+)$/;
2353
+ function serverMessage(body) {
2354
+ if (typeof body.message !== "string") return void 0;
2355
+ const message = cleanText(body.message).trim().replace(/\.$/, "");
2356
+ return message === "" ? void 0 : message;
2357
+ }
2358
+ function invalidImage(body) {
2359
+ const message = serverMessage(body);
2360
+ if (message === MANAGED_REFERENCE_HINT) {
2361
+ return "The managed image in --base-image is not a valid reference. Use managed://<name>:<alias>, for example managed://solver:latest.";
2362
+ }
2363
+ if (message !== void 0) return `The image in --base-image was refused: ${message}.`;
2364
+ return "The image in --base-image is not a valid reference. Use a public docker.io, ghcr.io or nvcr.io image, or managed://<name>:<alias>.";
2365
+ }
2366
+ function unknownProvider(body) {
2367
+ const configured = CONFIGURED_PROVIDERS.exec(serverMessage(body) ?? "")?.[1].trim();
2368
+ const next = configured === void 0 || configured === "" ? "Check the provider ID with your ThermAI operator" : `Use one of: ${configured}`;
2369
+ return `The provider in --provider is not one Flightdeck runs on. ${next}, or leave out --provider.`;
2370
+ }
2371
+ var DESCRIBERS = /* @__PURE__ */ new Map([
2372
+ [
2373
+ "managed_image_not_found",
2374
+ () => "The managed image or alias in --base-image was not found in your organization. Check the name with flightdeck image list and the alias with flightdeck image alias ls <name>."
2375
+ ],
2376
+ [
2377
+ "managed_image_provider_required",
2378
+ () => "A managed image in --base-image runs on Lambda only. Add --provider lambda."
2379
+ ],
2380
+ ["invalid_image", invalidImage],
2381
+ ["unknown_provider", unknownProvider],
2382
+ [
2383
+ "alias_target_disabled",
2384
+ () => "The alias in --base-image names an image version that is disabled. Ask an organization admin to point it at an enabled version with flightdeck image alias set, or use another alias."
2385
+ ]
2386
+ ]);
2387
+ function describeSubmitRefusal(body) {
2388
+ const dataset2 = describeSubmitDatasetError(body);
2389
+ if (dataset2 !== void 0) return dataset2;
2390
+ if (!isRecord2(body) || typeof body.error !== "string") return void 0;
2391
+ return DESCRIBERS.get(body.error)?.(body);
2392
+ }
2393
+
2348
2394
  // src/commands/dataset-ref.ts
2349
2395
  var MAX_DATASET_MOUNTS = 4;
2350
2396
  var MAX_NAME_LENGTH = 64;
@@ -2425,6 +2471,50 @@ function datasetPlanFields(values) {
2425
2471
  return datasets.length > 0 ? { datasets } : {};
2426
2472
  }
2427
2473
 
2474
+ // src/commands/flight-secrets.ts
2475
+ var MAX_FLIGHT_SECRETS = 16;
2476
+ var MAX_NAME_LENGTH2 = 64;
2477
+ var SECRET_NAME_PATTERN = /^[a-z0-9]+(?:[._-][a-z0-9]+)*$/;
2478
+ var USAGE = "Give --secret as ENV=name, for example --secret HF_TOKEN=hf-token.";
2479
+ function collectSecret(value, previous = []) {
2480
+ return [...previous, value];
2481
+ }
2482
+ function parseSecretArgument(value) {
2483
+ const equals = value.indexOf("=");
2484
+ if (equals < 1 || equals === value.length - 1) throw new CliError(USAGE);
2485
+ const env = value.slice(0, equals);
2486
+ const name = value.slice(equals + 1);
2487
+ const issues = environmentContractIssues(Object.fromEntries([[env, ""]]), "customer");
2488
+ if (issues.length > 0) throw new CliError(`Invalid --secret ${JSON.stringify(env)}: ${issues.join("; ")}`);
2489
+ if (name.length > MAX_NAME_LENGTH2 || !SECRET_NAME_PATTERN.test(name)) {
2490
+ throw new CliError(
2491
+ `Secret names use lowercase letters and digits, joined by '.', '_' or '-', at most ${String(MAX_NAME_LENGTH2)} characters. ${USAGE}`
2492
+ );
2493
+ }
2494
+ return { name, env };
2495
+ }
2496
+ function parseSecretArguments(values, envVars = {}) {
2497
+ if (values.length > MAX_FLIGHT_SECRETS) {
2498
+ throw new CliError(`A flight uses at most ${String(MAX_FLIGHT_SECRETS)} secrets.`);
2499
+ }
2500
+ const seen = /* @__PURE__ */ new Set();
2501
+ return values.map((value) => {
2502
+ const secret2 = parseSecretArgument(value);
2503
+ if (seen.has(secret2.env)) throw new CliError(`--secret ${secret2.env} is given more than once.`);
2504
+ if (Object.hasOwn(envVars, secret2.env)) {
2505
+ throw new CliError(
2506
+ `${secret2.env} is already set as an environment variable (by --env, or by sbatch as SLURM_*). Give the secret another name.`
2507
+ );
2508
+ }
2509
+ seen.add(secret2.env);
2510
+ return secret2;
2511
+ });
2512
+ }
2513
+ function secretPlanFields(values, envVars) {
2514
+ const secrets = parseSecretArguments(values ?? [], envVars);
2515
+ return secrets.length > 0 ? { secrets } : {};
2516
+ }
2517
+
2428
2518
  // src/commands/submit.ts
2429
2519
  function parseDuration(input) {
2430
2520
  const match = input.match(/^(\d+)(h|m|s)?$/);
@@ -2448,6 +2538,8 @@ function collectDataset(value, previous = []) {
2448
2538
  }
2449
2539
  async function submit(entrypoint, options) {
2450
2540
  const datasetFields = datasetPlanFields(options.dataset);
2541
+ const envVars = parseEnvironmentArguments(options.env);
2542
+ const secretFields = secretPlanFields(options.secret, envVars);
2451
2543
  console.log(`Packaging files...`);
2452
2544
  let bundle;
2453
2545
  try {
@@ -2465,10 +2557,11 @@ async function submit(entrypoint, options) {
2465
2557
  gpu_type: options.gpu,
2466
2558
  gpu_count: parseInt(options.gpuCount, 10),
2467
2559
  max_duration_s: parseDuration(options.time),
2468
- env_vars: parseEnvironmentArguments(options.env),
2560
+ env_vars: envVars,
2469
2561
  ...options.baseImage ? { base_image: options.baseImage } : {},
2470
2562
  ...options.provider ? { requested_provider: options.provider } : {},
2471
- ...datasetFields
2563
+ ...datasetFields,
2564
+ ...secretFields
2472
2565
  };
2473
2566
  console.log(`Submitting to FlightDeck...`);
2474
2567
  try {
@@ -2488,7 +2581,7 @@ Track with: flightdeck status ${body.flight_id}`);
2488
2581
  if (err instanceof SubmitError) {
2489
2582
  console.error(
2490
2583
  `Submit failed (${err.status}):`,
2491
- describeSubmitDatasetError(err.body) ?? JSON.stringify(err.body, null, 2)
2584
+ describeSubmitRefusal(err.body) ?? JSON.stringify(err.body, null, 2)
2492
2585
  );
2493
2586
  process.exit(1);
2494
2587
  }
@@ -3030,34 +3123,33 @@ function buildSbatchPlans(args) {
3030
3123
  const extraEnv = parseEnvVars(args.options.env);
3031
3124
  let maxDurationS = args.directives.maxDurationS;
3032
3125
  if (maxDurationS !== void 0 && maxDurationS > MAX_DURATION_S) maxDurationS = MAX_DURATION_S;
3033
- return taskIds.map((taskId) => ({
3034
- taskId,
3035
- plan: {
3036
- name: taskId === null ? jobName : `${jobName}[${taskId}]`,
3037
- entrypoint: buildEntrypoint({
3038
- relativeScript: args.relativeScript,
3039
- workingDirectory: args.directives.workingDirectory,
3040
- outputPath: args.directives.outputPath,
3041
- errorPath: args.directives.errorPath,
3042
- jobId: args.jobId,
3043
- jobName,
3044
- taskId
3045
- }),
3046
- requirements_file: args.options.requirements,
3047
- gpu_type: args.directives.gpuType,
3048
- gpu_count: args.directives.gpuCount,
3049
- max_duration_s: maxDurationS,
3050
- env_vars: slurmEnvironment({
3051
- directives: args.directives,
3052
- taskId,
3053
- jobId: args.jobId,
3054
- jobName,
3055
- extraEnv
3056
- }),
3057
- base_image: args.options.baseImage,
3058
- requested_provider: args.options.provider
3059
- }
3060
- }));
3126
+ return taskIds.map((taskId) => {
3127
+ const envVars = slurmEnvironment({ directives: args.directives, taskId, jobId: args.jobId, jobName, extraEnv });
3128
+ return {
3129
+ taskId,
3130
+ plan: {
3131
+ name: taskId === null ? jobName : `${jobName}[${taskId}]`,
3132
+ entrypoint: buildEntrypoint({
3133
+ relativeScript: args.relativeScript,
3134
+ workingDirectory: args.directives.workingDirectory,
3135
+ outputPath: args.directives.outputPath,
3136
+ errorPath: args.directives.errorPath,
3137
+ jobId: args.jobId,
3138
+ jobName,
3139
+ taskId
3140
+ }),
3141
+ requirements_file: args.options.requirements,
3142
+ gpu_type: args.directives.gpuType,
3143
+ gpu_count: args.directives.gpuCount,
3144
+ max_duration_s: maxDurationS,
3145
+ env_vars: envVars,
3146
+ base_image: args.options.baseImage,
3147
+ requested_provider: args.options.provider,
3148
+ // Checked against the SLURM_* variables too, which the API would see as a clash.
3149
+ ...secretPlanFields(args.options.secret, envVars)
3150
+ }
3151
+ };
3152
+ });
3061
3153
  }
3062
3154
  function trackWithLine(submitted) {
3063
3155
  const hint = `Track with: flightdeck status ${submitted[0].flightId}`;
@@ -3203,7 +3295,9 @@ Packaging files...`);
3203
3295
  } catch (err) {
3204
3296
  failures += 1;
3205
3297
  if (err instanceof SubmitError) {
3206
- console.error(` ${task.plan.name}: failed (${err.status}) ${JSON.stringify(err.body)}`);
3298
+ console.error(
3299
+ ` ${task.plan.name}: failed (${err.status}) ${describeSubmitRefusal(err.body) ?? JSON.stringify(err.body)}`
3300
+ );
3207
3301
  } else {
3208
3302
  console.error(` ${task.plan.name}: ${err.message}`);
3209
3303
  }
@@ -3294,6 +3388,9 @@ async function status(flightId) {
3294
3388
 
3295
3389
  // src/log-reader.ts
3296
3390
  var TRUNCATED_LOG_NOTICE = "Warning: this log is truncated: a stream passed the 5 MiB per-stream log cap, or the log could not be read in full.";
3391
+ function expiredLogNotice(expiredAt) {
3392
+ return `This flight's logs were deleted on ${expiredAt.slice(0, "YYYY-MM-DD".length)}. Flight logs are kept for 30 days.`;
3393
+ }
3297
3394
  async function drainLogs(follow, deps) {
3298
3395
  let cursor;
3299
3396
  const seen = /* @__PURE__ */ new Set();
@@ -3306,7 +3403,7 @@ async function drainLogs(follow, deps) {
3306
3403
  "This backend does not support complete log pagination. Deploy the updated backend before using this CLI log command."
3307
3404
  );
3308
3405
  }
3309
- if (!Array.isArray(body.logs) || !body.logs.every((entry) => entry && typeof entry.stream === "string" && typeof entry.content === "string") || typeof body.complete !== "boolean" || typeof body.next_cursor !== "string" || !body.next_cursor || body.has_more && body.complete || body.truncated !== void 0 && typeof body.truncated !== "boolean") {
3406
+ if (!Array.isArray(body.logs) || !body.logs.every((entry) => entry && typeof entry.stream === "string" && typeof entry.content === "string") || typeof body.complete !== "boolean" || typeof body.next_cursor !== "string" || !body.next_cursor || body.has_more && body.complete || body.truncated !== void 0 && typeof body.truncated !== "boolean" || body.expired_at !== void 0 && body.expired_at !== null && typeof body.expired_at !== "string") {
3310
3407
  throw new CliError("Invalid log pagination response from backend.");
3311
3408
  }
3312
3409
  if (body.has_more && (body.next_cursor === cursor || seen.has(body.next_cursor))) {
@@ -3319,6 +3416,10 @@ async function drainLogs(follow, deps) {
3319
3416
  seen.add(cursor);
3320
3417
  continue;
3321
3418
  }
3419
+ if (typeof body.expired_at === "string") {
3420
+ deps.notice(expiredLogNotice(body.expired_at));
3421
+ return;
3422
+ }
3322
3423
  if (truncated && !truncationNoticed) {
3323
3424
  deps.notice(TRUNCATED_LOG_NOTICE);
3324
3425
  truncationNoticed = true;
@@ -3329,7 +3430,61 @@ async function drainLogs(follow, deps) {
3329
3430
  }
3330
3431
  }
3331
3432
 
3433
+ // src/restart-retry.ts
3434
+ var BAD_GATEWAY = 502;
3435
+ var SERVICE_UNAVAILABLE = 503;
3436
+ var GATEWAY_TIMEOUT = 504;
3437
+ var RESTART_STATUS = /* @__PURE__ */ new Set([BAD_GATEWAY, SERVICE_UNAVAILABLE, GATEWAY_TIMEOUT]);
3438
+ var GIVE_UP_AFTER_MS = 6e4;
3439
+ var FIRST_DELAY_MS = 1e3;
3440
+ var MAX_DELAY_MS = 8e3;
3441
+ var RESTART_NOTICE = "The Flightdeck API is not answering. Retrying for up to a minute.";
3442
+ var terminalRestartRetry = {
3443
+ now: () => Date.now(),
3444
+ sleep: (ms) => new Promise((resolve6) => setTimeout(resolve6, ms)),
3445
+ notice(message) {
3446
+ process.stderr.write(`${message}
3447
+ `);
3448
+ }
3449
+ };
3450
+ async function attempt(send) {
3451
+ try {
3452
+ return { response: await send() };
3453
+ } catch (error) {
3454
+ if (error instanceof NetworkError) return { failure: error };
3455
+ throw error;
3456
+ }
3457
+ }
3458
+ function settle(result) {
3459
+ if ("failure" in result) throw result.failure;
3460
+ return result.response;
3461
+ }
3462
+ async function readThroughRestart(send, deps = terminalRestartRetry) {
3463
+ let deadline;
3464
+ let delayMs = FIRST_DELAY_MS;
3465
+ for (; ; ) {
3466
+ const result = await attempt(send);
3467
+ if ("response" in result && !RESTART_STATUS.has(result.response.status)) return result.response;
3468
+ const now = deps.now();
3469
+ if (deadline === void 0) {
3470
+ deadline = now + GIVE_UP_AFTER_MS;
3471
+ deps.notice(RESTART_NOTICE);
3472
+ }
3473
+ if (now >= deadline) return settle(result);
3474
+ if ("response" in result) await result.response.body?.cancel();
3475
+ await deps.sleep(Math.min(delayMs, deadline - now));
3476
+ delayMs = Math.min(MAX_DELAY_MS, delayMs * 2);
3477
+ }
3478
+ }
3479
+
3332
3480
  // src/commands/logs.ts
3481
+ async function readLogPage(flightId, cursor, follow, deps = { request: apiRequest }) {
3482
+ const query = cursor ? `?cursor=${encodeURIComponent(cursor)}` : "";
3483
+ const send = () => deps.request("GET", `/flights/${flightId}/logs${query}`);
3484
+ const response = follow ? await readThroughRestart(send, deps.retry) : await send();
3485
+ if (!response.ok) throw httpError("Log request failed", response.status);
3486
+ return await response.json();
3487
+ }
3333
3488
  var terminalLogOutput = {
3334
3489
  write(stream, content) {
3335
3490
  (stream === "stderr" ? process.stderr : process.stdout).write(content);
@@ -3340,15 +3495,10 @@ var terminalLogOutput = {
3340
3495
  }
3341
3496
  };
3342
3497
  async function logs(flightId, options) {
3498
+ const follow = options.follow ?? false;
3343
3499
  try {
3344
- await drainLogs(options.follow ?? false, {
3345
- async readPage(cursor) {
3346
- const query = cursor ? `?cursor=${encodeURIComponent(cursor)}` : "";
3347
- const response = await apiRequest("GET", `/flights/${flightId}/logs${query}`);
3348
- if (!response.ok) throw httpError("Log request failed", response.status);
3349
- const body = await response.json();
3350
- return body;
3351
- },
3500
+ await drainLogs(follow, {
3501
+ readPage: (cursor) => readLogPage(flightId, cursor, follow),
3352
3502
  ...terminalLogOutput,
3353
3503
  wait: () => new Promise((resolve6) => setTimeout(resolve6, 2e3))
3354
3504
  });
@@ -5093,6 +5243,9 @@ async function atc(target, options) {
5093
5243
 
5094
5244
  // src/output-notices.ts
5095
5245
  var SIZE_LIMIT_EVENT = "outputs_too_large";
5246
+ var CUTOFF_EVENT = "cancel_collection_cutoff";
5247
+ var FINISHED_EVENT = "cancel_collection_finished";
5248
+ var SECONDS_PER_MINUTE2 = 60;
5096
5249
  var MAY_BE_MISSING = "so some outputs or logs may be missing.";
5097
5250
  var BYTES_PER_KIB = 1024;
5098
5251
  var BYTES_PER_MB = BYTES_PER_KIB * BYTES_PER_KIB;
@@ -5111,17 +5264,29 @@ var COLLECTION_ENDED = /* @__PURE__ */ new Map([
5111
5264
  [
5112
5265
  "cancel_stop_failed",
5113
5266
  () => "The flight was cancelled but its workload could not be confirmed stopped, so its outputs may not have been collected."
5114
- ]
5267
+ ],
5268
+ [CUTOFF_EVENT, (metadata) => cutoffNotice(metadata.backstop_s)]
5115
5269
  ]);
5270
+ function cutoffNotice(backstopS) {
5271
+ if (typeof backstopS !== "number" || !Number.isSafeInteger(backstopS) || backstopS < SECONDS_PER_MINUTE2) {
5272
+ return `Collection was cut off after the flight was cancelled, ${MAY_BE_MISSING}`;
5273
+ }
5274
+ const minutes = String(Math.floor(backstopS / SECONDS_PER_MINUTE2));
5275
+ return `Collection stopped ${minutes} minutes after the flight was cancelled, ${MAY_BE_MISSING}`;
5276
+ }
5116
5277
  function sizeLimitNotice(metadata) {
5117
5278
  const { filename, max_bytes: maxBytes } = metadata;
5118
5279
  const name = typeof filename === "string" ? sanitize(filename).trim() : "";
5119
5280
  const limit = typeof maxBytes === "number" && Number.isSafeInteger(maxBytes) && maxBytes > 0 && maxBytes % BYTES_PER_MB === 0 ? `the ${String(maxBytes / BYTES_PER_MB)} MB result limit` : "the result size limit";
5120
5281
  return `${name ? `Output ${name}` : "An output"} exceeds ${limit} and was not stored.`;
5121
5282
  }
5283
+ function withoutOvertakenCutoff(events) {
5284
+ if (!events.some((event) => event.type === FINISHED_EVENT)) return events;
5285
+ return events.filter((event) => event.type !== CUTOFF_EVENT);
5286
+ }
5122
5287
  function outputNotices(events, selection = {}) {
5123
5288
  const notices = [];
5124
- for (const event of events) {
5289
+ for (const event of withoutOvertakenCutoff(events)) {
5125
5290
  const metadata = event.metadata ?? {};
5126
5291
  if (event.type === SIZE_LIMIT_EVENT) {
5127
5292
  if (selection.attemptSeq !== void 0 && metadata.attempt_seq !== selection.attemptSeq) continue;
@@ -5205,7 +5370,7 @@ var LOCK_ATTEMPTS = 3;
5205
5370
  async function acquireLock(lock, target) {
5206
5371
  const token = randomUUID();
5207
5372
  const owner = JSON.stringify({ pid: process.pid, host: hostname(), token });
5208
- for (let attempt = 0; attempt < LOCK_ATTEMPTS; attempt++) {
5373
+ for (let attempt2 = 0; attempt2 < LOCK_ATTEMPTS; attempt2++) {
5209
5374
  try {
5210
5375
  await writeFile(lock, owner, { flag: "wx", mode: 384 });
5211
5376
  return token;
@@ -5619,16 +5784,26 @@ function positiveSeconds(value, option) {
5619
5784
  }
5620
5785
  return Number(value);
5621
5786
  }
5622
- async function readFlight(flightId) {
5623
- const response = await apiRequest("GET", `/flights/${flightId}`);
5787
+ async function readFlight(flightId, deps = { request: apiRequest }) {
5788
+ const response = await readThroughRestart(() => deps.request("GET", `/flights/${flightId}`), deps.retry);
5624
5789
  if (!response.ok) throw httpError("Unable to read conformance flight", response.status);
5625
5790
  const body = await response.json();
5626
5791
  return body;
5627
5792
  }
5628
- async function verifyResults(flightId, taskId) {
5793
+ async function verifyResults(flightId, taskId, deps = { request: apiRequest }) {
5794
+ const retry = deps.retry ?? terminalRestartRetry;
5795
+ let noticed = false;
5796
+ const once = {
5797
+ ...retry,
5798
+ notice(message) {
5799
+ if (!noticed) retry.notice(message);
5800
+ noticed = true;
5801
+ }
5802
+ };
5803
+ const read = (path) => readThroughRestart(() => deps.request("GET", path), once);
5629
5804
  const [logsResponse, outputsResponse] = await Promise.all([
5630
- apiRequest("GET", `/flights/${flightId}/logs`),
5631
- apiRequest("GET", `/flights/${flightId}/outputs`)
5805
+ read(`/flights/${flightId}/logs`),
5806
+ read(`/flights/${flightId}/outputs`)
5632
5807
  ]);
5633
5808
  if (!logsResponse.ok) throw httpError("Unable to read conformance logs", logsResponse.status);
5634
5809
  if (!outputsResponse.ok) throw httpError("Unable to read conformance outputs", outputsResponse.status);
@@ -8280,7 +8455,7 @@ async function sendPart(context, job, signal) {
8280
8455
  const { state, number, checksum } = job.task;
8281
8456
  const range = partRange(state.file.size, state.file.partSize, number);
8282
8457
  if (job.url.expiresAt - context.now() < MIN_URL_LIFE_MS) return { kind: "stale", number };
8283
- for (let attempt = 1; ; attempt += 1) {
8458
+ for (let attempt2 = 1; ; attempt2 += 1) {
8284
8459
  try {
8285
8460
  const sent = await context.putPart({
8286
8461
  url: job.url.url,
@@ -8294,12 +8469,12 @@ async function sendPart(context, job, signal) {
8294
8469
  signal.throwIfAborted();
8295
8470
  if (error instanceof PartRejectedError && error.status === HTTP_FORBIDDEN) return { kind: "stale", number };
8296
8471
  if (error instanceof PartRejectedError && error.status === HTTP_BAD_REQUEST) throw changedWhileUploading();
8297
- if (attempt >= MAX_PUT_ATTEMPTS) {
8472
+ if (attempt2 >= MAX_PUT_ATTEMPTS) {
8298
8473
  throw new CliError(
8299
8474
  `Part ${String(number)} of ${state.entry.path} failed ${String(MAX_PUT_ATTEMPTS)} times. Check the connection and run the same push again to resume.`
8300
8475
  );
8301
8476
  }
8302
- await context.client.sleep(RETRY_BACKOFF_MS * attempt, signal);
8477
+ await context.client.sleep(RETRY_BACKOFF_MS * attempt2, signal);
8303
8478
  }
8304
8479
  }
8305
8480
  }
@@ -9155,9 +9330,25 @@ async function secretRevoke(name, options, command) {
9155
9330
  });
9156
9331
  }
9157
9332
 
9333
+ // src/banner.ts
9334
+ var BANNER_LINES = [
9335
+ "___________________ ______ _____________ ______",
9336
+ "___ ____/__ /__(_)______ ___ /___ /___ __ \\_____________ /__",
9337
+ "__ /_ __ /__ /__ __ `/_ __ \\ __/_ / / / _ \\ ___/_ //_/",
9338
+ "_ __/ _ / _ / _ /_/ /_ / / / /_ _ /_/ // __/ /__ _ ,<",
9339
+ "/_/ /_/ /_/ _\\__, / /_/ /_/\\__/ /_____/ \\___/\\___/ /_/|_|",
9340
+ " /____/"
9341
+ ];
9342
+ var INDENT = " ";
9343
+ function helpBanner(toStderr) {
9344
+ const stream = toStderr ? process.stderr : process.stdout;
9345
+ return stream.isTTY ? `${BANNER_LINES.map((line) => `${INDENT}${line}`).join("\n")}
9346
+ ` : "";
9347
+ }
9348
+
9158
9349
  // src/index.ts
9159
9350
  var program = new Command();
9160
- program.name("flightdeck").description("FlightDeck CLI \u2014 submit GPU compute jobs").option("--debug", "include safe error type/code and CLI frame diagnostics (no raw stack or credentials)").addHelpText("after", "\nRun flightdeck --version, or flightdeck -V, on its own to print the CLI version.");
9351
+ program.name("flightdeck").description("FlightDeck CLI \u2014 submit GPU compute jobs").option("--debug", "include safe error type/code and CLI frame diagnostics (no raw stack or credentials)").addHelpText("before", ({ error }) => helpBanner(error)).addHelpText("after", "\nRun flightdeck --version, or flightdeck -V, on its own to print the CLI version.");
9161
9352
  var cliVersion = JSON.parse(readFileSync6(new URL("../package.json", import.meta.url), "utf8")).version;
9162
9353
  program.command("login").description("Log in through the FlightDeck portal").option("--api-url <url>", "FlightDeck API URL", DEFAULT_API_URL).option("--name <name>", "name for this CLI credential (default: flightdeck-cli@<hostname>)").option("--admin", "authorise this workstation to list flights as an administrator").option("--images", "authorise this workstation to publish and list managed container images").option("--storage", "authorise this workstation to upload, list and mount managed datasets").option("--secrets", "authorise this workstation to list, create, rotate and revoke secrets").action(login);
9163
9354
  program.command("logout").description("Remove the local login and attempt server credential revocation").option("--local", "remove the local login without revoking the credential").action(logout);
@@ -9254,11 +9445,19 @@ program.command("submit").description("Submit a job from local files").argument(
9254
9445
  "--dataset <ref>",
9255
9446
  "mount a dataset version read-only under /data/<mount-name>: <name>:<label>[:<mount-name>] or a version ID; repeat for up to 4",
9256
9447
  collectDataset
9448
+ ).option(
9449
+ "--secret <ENV=name>",
9450
+ "give the workload your organisation's secret <name> as environment variable ENV and as /run/flightdeck/secrets/ENV (Lambda only); repeat for more",
9451
+ collectSecret
9257
9452
  ).action(submit);
9258
9453
  program.command("sbatch").description("Submit a Slurm batch script (#SBATCH directives, incl. job arrays)").argument("<script>", "path to the .slurm/.sh batch script").option("-f, --files <glob...>", "files to include (default: current directory; see .flightdeckignore)").option("-r, --requirements <file>", "pip requirements file to install before the script runs").option(
9259
9454
  "--base-image <image>",
9260
9455
  "image to run in (default: the official Flightdeck base); any linux/amd64 image meeting the workload runtime contract (see /docs/images)"
9261
- ).option("--provider <id>", "pin every task to one compute provider (default: operator's order)").option("-n, --name <name>", "override job name (default: #SBATCH --job-name)").option("-e, --env <vars...>", "extra environment variables (KEY=VALUE)").option("--dry-run", "parse and show what would be submitted, without submitting").option("--ignore-array-limit", "submit every array task even if %N asks for fewer at a time").action(async (script, options) => {
9456
+ ).option("--provider <id>", "pin every task to one compute provider (default: operator's order)").option("-n, --name <name>", "override job name (default: #SBATCH --job-name)").option("-e, --env <vars...>", "extra environment variables (KEY=VALUE)").option(
9457
+ "--secret <ENV=name>",
9458
+ "give every task your organisation's secret <name> as environment variable ENV (Lambda only); repeat for more",
9459
+ collectSecret
9460
+ ).option("--dry-run", "parse and show what would be submitted, without submitting").option("--ignore-array-limit", "submit every array task even if %N asks for fewer at a time").action(async (script, options) => {
9262
9461
  await sbatch(script, options);
9263
9462
  });
9264
9463
  program.command("slurm-test").description("Run the provider-neutral Slurm conformance workload").option("--timeout <seconds>", "maximum time to wait for all tasks", "1800").option("--poll-interval <seconds>", "status polling interval", "5").action(slurmTest);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@thermai/flightdeck-cli",
3
- "version": "0.1.0-pilot.10",
3
+ "version": "0.1.0-pilot.11",
4
4
  "description": "FlightDeck CLI — submit GPU compute jobs",
5
5
  "license": "SEE LICENSE IN LICENSE",
6
6
  "publishConfig": {