@thermai/flightdeck-cli 0.1.0-pilot.1 → 0.1.0-pilot.11

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +984 -17
  2. package/dist/index.js +9226 -921
  3. package/package.json +8 -3
package/README.md CHANGED
@@ -2,31 +2,126 @@
2
2
 
3
3
  Install the command-line client, authorize it through the Flightdeck portal,
4
4
  then submit and inspect GPU jobs. No npm account is required to install the
5
- planned public package. A provisioned Flightdeck account is required to use the
5
+ public package. A provisioned Flightdeck account is required to use the
6
6
  service; downloading the CLI does not create an account or grant compute credit.
7
7
 
8
8
  ## Release status
9
9
 
10
- The npm publishing pathway is being prepared. The operator has created the
11
- `@thermai` npm organization; the package name is `@thermai/flightdeck-cli`.
12
- The prepared candidate is `0.1.0-pilot.1`, with operator-supplied proprietary
13
- terms in `LICENSE`, amended at the operator's direction to allow redistribution
14
- of unchanged official releases. The operator has approved proceeding with the
15
- internal proving release; publication awaits technical release checks and
16
- completion of publishing setup. The commands
17
- below describe the intended released experience, not an announcement that this
18
- package is already on npm.
10
+ The package is `@thermai/flightdeck-cli`. The first pilot, `0.1.0-pilot.1`, was
11
+ published manually on 5 September 2026 and its anonymous install was verified.
12
+ `0.1.0-pilot.2` subsequently passed GitHub Actions trusted publishing and an
13
+ anonymous install check on the same date. Releases publish through the
14
+ versioned GitHub Release workflow, not merely by merging a version change. The
15
+ installation command below follows the published `next` channel without
16
+ tracking an unpublished candidate version.
17
+ Proprietary terms in `LICENSE` permit redistribution of unchanged official
18
+ releases. Service access still requires an invitation, configured credits and
19
+ the agreed pilot limits; installing the package alone does not grant access.
20
+
21
+ **`0.1.0-pilot.3`** was published through GitHub Actions on 8 September 2026,
22
+ including complete log pagination and concise authentication errors. Its
23
+ anonymous Node 22 install and packaged dry run passed. Use the operator-approved
24
+ exact version for a recorded rehearsal.
25
+
26
+ **`0.1.0-pilot.4`**, published on 20 September 2026, adds customer credits and
27
+ requires a backend advertising `X-Flightdeck-Accounting: credits-v1`. Upgrade
28
+ with the install command below before accessing account/job data. An incompatible
29
+ backend is rejected before job submission; login and credential revocation do not require
30
+ the accounting contract. See the
31
+ operator rollout checklist (`flightdeck-ops/runbooks/customer-credits-rollout.md`).
32
+
33
+ **`0.1.0-pilot.5`**, published on 21 September 2026, fixes result downloads redirected to signed object
34
+ storage ([#157](https://github.com/ThermAI-Group/thermai_flightdeck/issues/157))
35
+ and includes the merged GPU-recovery status display. Downloads validate the
36
+ accounting contract on Flightdeck's response, then fetch the authorized storage
37
+ object without forwarding the CLI credential. Anonymous Node 22 installation
38
+ and a live result download passed. Upgrade from
39
+ pilot.4 before downloading results. Future releases still require an approved
40
+ versioned GitHub Release; merging alone does not publish.
41
+
42
+ **`0.1.0-pilot.6`**, published on 21 September 2026, adds
43
+ offline-first logout and explicit suspension messages. Release CI, registry
44
+ integrity and anonymous Node 22 installation passed; grouped live acceptance
45
+ was confirmed by the operator on 24 September. These are historical pilot.6
46
+ results, not pilot.7 publication evidence. This update does not modify the
47
+ README inside the already-published immutable tarball.
48
+
49
+ **`0.1.0-pilot.7`**, published on 1 October 2026, adds the read-only
50
+ `flightdeck atc` terminal view (alias `tui`), flight details/events/logs, name
51
+ filtering and local sbatch-array tracking from
52
+ [PR #206](https://github.com/ThermAI-Group/thermai_flightdeck/pull/206).
53
+ Downloads/cancellation inside ATC remain deferred to #205; stale
54
+ fallback-status indication is tracked in #207.
55
+
56
+ **`0.1.0-pilot.8`** adds `flightdeck admin flights` for organisation and
57
+ platform administrators ([PR #281](https://github.com/ThermAI-Group/thermai_flightdeck/pull/281)).
58
+ It needs a credential approved through `flightdeck login --admin`; existing
59
+ credentials keep working for every other command, so administrators run
60
+ `flightdeck logout` then `flightdeck login --admin` only when they want the
61
+ admin view. `flightdeck login` now defaults the credential name to
62
+ `flightdeck-cli@<hostname>` so a second machine no longer collides (#254), and
63
+ `flightdeck sbatch` prints a runnable status hint with the submitted flight ID
64
+ and labels the job ID on the ATC hint (#257, #259). The required backend is
65
+ already deployed; there are no new runtime dependencies. It was published to
66
+ npm's `next` channel on 8 October 2026. Use the generated `@next` command below, or pin
67
+ `@thermai/flightdeck-cli@0.1.0-pilot.8`, and confirm `flightdeck --version`
68
+ reports `0.1.0-pilot.8`.
69
+
70
+ **`0.1.0-pilot.9`** adds managed images and datasets. `flightdeck image check`,
71
+ `push`, `list` and `inspect` check and publish workload images to your
72
+ organization's managed registry. `flightdeck image alias ls|set|rm` and
73
+ `image push --alias` manage channels, and `managed://<name>:<alias>` (including
74
+ the computed `latest`) can be submitted as an image. `flightdeck dataset push`,
75
+ `list` and `rm` manage datasets, and `submit --dataset` mounts them. `sbatch`
76
+ refuses an array `%N` limit Flightdeck cannot enforce unless
77
+ `--ignore-array-limit` is passed. Alias commands need a backend with image
78
+ aliases; the other commands work against the current backend. There are no new
79
+ runtime dependencies.
80
+
81
+ **`0.1.0-pilot.10`** adds `flightdeck delete-output <flight-id> <output-id>`,
82
+ which deletes one output of a finished flight, and
83
+ `flightdeck dataset download <version> <directory>`, which downloads a dataset
84
+ version and checks every file against its recorded checksum; running it again
85
+ resumes. While a flight copies its datasets, `flightdeck status` shows their
86
+ progress on a `Data:` line. `flightdeck image check` now checks images against
87
+ the supervisor release that Flightdeck 26.11.3 runs. Both new commands need Flightdeck 26.11.3
88
+ or later, and `dataset download` also needs storage enabled for your
89
+ organisation. There are no new runtime dependencies.
90
+
91
+ **`0.1.0-pilot.11`** adds `--secret ENV=name` to `flightdeck submit` and
92
+ `flightdeck sbatch`, which gives the workload one of your organisation's secrets
93
+ (see "Keeping secrets"). `flightdeck logs --follow` and `slurm-test` keep going
94
+ through the few seconds a release restarts the backend. A refused submit now
95
+ says what was wrong with `--base-image` or `--provider` in plain words instead
96
+ of printing the API's JSON. `flightdeck outputs` and `flightdeck download` say
97
+ when collection after a cancel was cut off, so some outputs or logs may be
98
+ missing, and
99
+ `flightdeck logs` says when a flight's logs were deleted under the 30-day limit.
100
+ `flightdeck image check` checks images against the supervisor release that
101
+ Flightdeck 26.11.4 runs. `--secret` needs Flightdeck 26.11.4 or later and
102
+ secrets enabled for your organisation; an older backend ignores it. There are no
103
+ new runtime dependencies.
104
+
105
+ Members and organization administrators receive whole-credit balances,
106
+ estimates and actual customer charges, without provider-cost information.
107
+ Platform administrators retain their authorized monetary API fields. Credits
108
+ are not GPU-hours; consumption depends on the selected hardware/provider and
109
+ the accepted job pricing. Existing limits and customer charge caps still apply.
19
110
 
20
111
  ## Installation and login
21
112
 
22
113
  Requires Node.js 22 and npm. Use a user-owned Node installation or npm prefix;
23
114
  the CLI does not require administrator/root privileges.
24
115
 
116
+ <!-- flightdeck-cli-install:start -->
117
+
25
118
  ```sh
26
- npm install -g @thermai/flightdeck-cli@0.1.0-pilot.1
119
+ npm install -g @thermai/flightdeck-cli@next
27
120
  flightdeck login
28
121
  ```
29
122
 
123
+ <!-- flightdeck-cli-install:end -->
124
+
30
125
  Login prints a URL and verification code. Open that URL, sign into
31
126
  `https://portal.thermai.uk`, and approve the named CLI credential and requested
32
127
  scopes. The CLI waits for approval; no API key needs to be copied manually.
@@ -42,26 +137,896 @@ test environment can be selected with `flightdeck login --api-url <HTTPS origin>
42
137
 
43
138
  Login saves a scoped credential in `~/.flightdeck/config.json` with restrictive
44
139
  permissions. Never share that file or include it in a workload bundle. Use
45
- `flightdeck logout` to revoke the credential and remove the local login. Portal
46
- credential management can revoke an individual workstation as well.
140
+ `flightdeck logout` to remove the local login and attempt server revocation.
141
+ Portal credential management can revoke an individual workstation as well.
142
+
143
+ `--name` labels a credential, not a local account profile. It defaults to
144
+ `flightdeck-cli@<short hostname>`, which is shown on the approval page, in
145
+ credential lists, and in the audit history; pass `--name` to choose another
146
+ label. Active credential names must be unique per account, so machines that
147
+ share a short hostname need distinct `--name` values. There is one saved
148
+ login per OS user. If already logged in, use `whoami` to inspect it; `login`
149
+ refuses to replace it. Use a separate OS user to preserve an operator's working
150
+ credential, or explicitly choose logout/re-login on the same workstation.
151
+ Logging out removes the saved credential; it does not delete the account or its
152
+ roles. A later login creates a new credential. Check the separate revocation
153
+ result: other copies may remain valid if server revocation was not confirmed.
154
+
155
+ Pilot.6 explains `account_suspended`, `membership_suspended`
156
+ and `organization_suspended` denials during device login and authenticated
157
+ commands. Contact your organization administrator; organization suspension may
158
+ need ThermAI support. These are access restrictions, not a missing local login
159
+ or a request for additional credential scopes. Other 403 responses retain
160
+ generic permissions guidance. Commands still exit nonzero and do not print
161
+ server-supplied messages or stack traces. Older pilot.5 installations need an
162
+ explicit upgrade to receive these messages.
163
+
164
+ ## Pilot quick-start
165
+
166
+ Before live use, your ThermAI operator must provision your organization/account,
167
+ provide a single-use portal invitation, and approve your policy and credit.
168
+ Activate through the portal, then follow the installation/login commands above.
169
+ Confirm the expected account with `whoami` and the available credit with
170
+ `balance`. ThermAI will guide the first partner submission under the agreed limits.
171
+
172
+ For a **local-only** readiness check, save the following as `readiness.sbatch`
173
+ in a new empty directory. Run the dry run from that directory; no account,
174
+ upload, API request, or paid resource is needed:
175
+
176
+ <!-- flightdeck-cli-readiness-script:start -->
177
+
178
+ ```bash
179
+ #!/bin/bash
180
+ #SBATCH --array=1-2
181
+ #SBATCH --gres=gpu:A10:1
182
+ #SBATCH --time=00:01:00
183
+ #SBATCH --output=%A_%a.out
184
+ echo smoke
185
+ ```
186
+
187
+ <!-- flightdeck-cli-readiness-script:end -->
188
+
189
+ ```sh
190
+ flightdeck --version
191
+ flightdeck sbatch readiness.sbatch --dry-run
192
+ ```
193
+
194
+ Expect A10 x1, 60 seconds and **two flights would be submitted**, with no
195
+ warnings. Keep `--dry-run`: this is a translation exercise, not the approved
196
+ pilot workload or permission to launch it. A dry run does not check provider
197
+ capacity, account policy/credit, bundle contents, image access or GPU execution.
198
+
199
+ For the real workload, obtain the reviewed private bundle, exact image
200
+ digest, provider selection and runtime from the operator. Review its dry run,
201
+ then use the supplied submission command only after execution approval. Do not
202
+ copy internal test data into public packages or partner bundles. Use explicit
203
+ `--files` selections from a clean directory; the submission default includes
204
+ the current directory, less the default excludes listed under
205
+ [What goes into a bundle](#what-goes-into-a-bundle). Never include credentials
206
+ or private workstation files.
47
207
 
48
208
  ## Workload commands
49
209
 
50
- After internal qualification and explicit policy/credit approval:
210
+ After your account's policy/credit setup and agreed first-run approval:
51
211
 
52
212
  ```sh
213
+ flightdeck image check <image> # before using --base-image
53
214
  flightdeck sbatch workload.sbatch
54
215
  flightdeck status <flight-id>
55
216
  flightdeck logs <flight-id> --follow
56
217
  flightdeck outputs <flight-id>
57
218
  flightdeck download <flight-id> --output-dir ./results
219
+ flightdeck delete-output <flight-id> <output-id>
58
220
  flightdeck cancel <flight-id>
59
221
  ```
60
222
 
61
223
  Submission, including `flightdeck slurm-test`, can provision paid resources.
224
+ `submit` and `sbatch` print the deployment's output size limit first. Everything
225
+ in `/outputs` is stored as one compressed archive, and an archive over the limit
226
+ is not stored at all. `sbatch --dry-run` stays offline and states the limit
227
+ without a figure.
62
228
  `sbatch --dry-run` only translates locally and does not submit. Flightdeck
63
229
  supports a subset of single-node Slurm directives, not an arbitrary HPC cluster.
64
- Workload images must satisfy the Flightdeck runner contract.
230
+ Without `--base-image`, a flight runs on the official Flightdeck base
231
+ (`ghcr.io/thermai-group/flightdeck-base`): Ubuntu 24.04 with Python 3 and pip,
232
+ and no PyTorch or CUDA, so supply Python packages with `--requirements` or use
233
+ your own image. A `--base-image` can be any linux/amd64 image that meets the
234
+ workload runtime contract (see `/docs/images`); it does not need the
235
+ Flightdeck runner. Before charging, submission refuses an image whose
236
+ `ENTRYPOINT` is shell form
237
+ (`shell_form_entrypoint`) or whose `uk.thermai.flightdeck.contract` label names
238
+ a contract version Flightdeck does not run (`image_unsupported_contract`).
239
+ Check an image locally first with `flightdeck image check`; see
240
+ [Checking an image before you submit](#checking-an-image-before-you-submit).
241
+ Environment variables (`--env`, `#SBATCH --export`) are checked before upload:
242
+ names are at most 64 characters, and names starting with `FLIGHTDECK_` belong
243
+ to the platform and are refused.
244
+
245
+ `--secret ENV=name` (on `submit` and `sbatch`, repeatable, up to 16) gives the
246
+ workload your organisation's secret `name` as the environment variable `ENV`
247
+ and as the read-only file `/run/flightdeck/secrets/ENV`. The value is read when
248
+ the flight is dispatched and never passes through the CLI. Secrets run on
249
+ Lambda only, so add `--provider lambda` if other providers are active. `ENV`
250
+ follows the `--env` rules and cannot also be given with `--env`. Logs are
251
+ redacted on a best-effort basis only, so never print a secret. See
252
+ `docs/customer-secrets.md` "Delivery to workloads".
253
+
254
+ ```sh
255
+ flightdeck submit --provider lambda --secret HF_TOKEN=hf-token "python train.py"
256
+ ```
257
+
258
+ An array prints a separate Flightdeck ID for each accepted task. Record them
259
+ all: inspect, download and cancel each flight individually (`flightdeck atc
260
+ <job-id>` shows them together). Partial submission
261
+ can leave accepted flights running even if the command exits unsuccessfully;
262
+ do not blindly resubmit the whole array. Downloads go to
263
+ `<output-dir>/<flight-id>/attempt-<n>/`, or `legacy/` for results from before
264
+ attempts were recorded, keeping each output's relative path. `--attempt <n>`
265
+ selects an earlier attempt. An interrupted download resumes when you run the same
266
+ command again, and a file only appears once its size and SHA-256 match the stored
267
+ output. Partial data is kept in `<file>.part` until then. A `.part` file that
268
+ the CLI did not leave for that output is refused, never deleted. A matching file is
269
+ skipped and a different one is never overwritten. A second download of the same
270
+ file stops with an error while the first is running. Stored results have no expiry
271
+ today. `delete-output` removes one output, by the ID `outputs` prints, from the
272
+ outputs list and from storage; the stored copy is kept for 30 days before it is
273
+ removed. It needs the `jobs:cancel` scope and only works once the flight has
274
+ finished and, for a cancelled flight, its outputs have been collected. If storage
275
+ does not confirm the delete, the output stays listed and may or may not have been
276
+ removed from storage; run the same command again to finish the delete. `outputs` and `download` print a warning on stderr when an output was
277
+ over the size limit or collection ended early, including a cancelled flight
278
+ whose outputs may not have been collected, so a short list isn't mistaken for
279
+ everything. With `--attempt`, only that attempt's size refusals are reported. Check terminal status and exit code as well as the expected artifacts;
280
+ successful submission or download alone does not mean the workload succeeded.
281
+
282
+ For an incident, stop submitting, cancel the known active flights, and contact
283
+ your named ThermAI operator. Logging out does **not** stop jobs. The operator
284
+ must confirm provider teardown and reconcile any outstanding charges before a
285
+ retry. Support contact, supported hours and credit increases come from your
286
+ agreed pilot handoff; this client does not promise unattended support.
287
+
288
+ ### Saving, downloading and deleting results
289
+
290
+ Anything a flight writes under `/outputs` (also `$OUTPUTS_DIR`) is saved when the
291
+ flight ends and can be downloaded after the GPU instance has gone. Files anywhere
292
+ else are not kept. This walk-through saves one file, lists it, downloads it,
293
+ checks it and deletes it. The output shown is from a run on staging on
294
+ 14 November 2026. `delete-output` needs CLI `0.1.0-pilot.10` or later. On macOS
295
+ use `shasum -a 256` wherever this walk-through says `sha256sum`.
296
+
297
+ **1. Write to `/outputs`.** Save this as `results-demo.sbatch` in a new empty
298
+ directory:
299
+
300
+ ```bash
301
+ #!/bin/bash
302
+ #SBATCH --gres=gpu:A10:1
303
+ #SBATCH --time=00:05:00
304
+ #SBATCH --output=%j.out
305
+ echo "report $(date -u +%FT%TZ)" > "$OUTPUTS_DIR/report.txt"
306
+ sha256sum "$OUTPUTS_DIR/report.txt"
307
+ ```
308
+
309
+ ```sh
310
+ flightdeck sbatch results-demo.sbatch
311
+ flightdeck status <flight-id> # wait for completed
312
+ flightdeck logs <flight-id> # shows the checksum the workload printed
313
+ ```
314
+
315
+ `sbatch` prints the flight ID, and `logs` prints the workload's line:
316
+
317
+ ```text
318
+ sbatch-job: fl-00e67a3d3ed2ae2229952634 (queue #1)
319
+
320
+ 1 submitted, 0 failed.
321
+ Track with: flightdeck status fl-00e67a3d3ed2ae2229952634
322
+ ```
323
+
324
+ ```text
325
+ 6bc1d843833bc55a3f66ec342533ed7586b757d0f4f7c87c211ddcde597f3bba /outputs/report.txt
326
+ ```
327
+
328
+ **2. List what was saved.** On Lambda, everything in `/outputs` is saved as one
329
+ archive, `outputs.tar.gz`. Each line is the output ID, its size and its name:
330
+
331
+ ```sh
332
+ flightdeck outputs <flight-id>
333
+ ```
334
+
335
+ ```text
336
+ 59925a74-7a23-46e8-8bb0-26c7be1eddb0 274 bytes outputs.tar.gz
337
+ ```
338
+
339
+ A flight that is still running, or a cancelled flight whose outputs are still
340
+ being collected, prints `No outputs available.` An output is listed only once
341
+ it is stored.
342
+
343
+ **3. Download and check it.** The file lands in
344
+ `<output-dir>/<flight-id>/attempt-<n>/`. The CLI checks its size and SHA-256
345
+ against the stored output before the file appears, so a file that is there is
346
+ complete.
347
+
348
+ ```sh
349
+ flightdeck download <flight-id> --output-dir ./results
350
+ cd ./results/<flight-id>/attempt-1
351
+ tar -tzf outputs.tar.gz # look before you extract
352
+ tar -xzf outputs.tar.gz
353
+ sha256sum report.txt # compare with the line in the logs
354
+ ```
355
+
356
+ ```text
357
+ Downloaded /home/you/results/fl-00e67a3d3ed2ae2229952634/attempt-1/outputs.tar.gz
358
+ ./
359
+ ./1794696621749957.out
360
+ ./report.txt
361
+ 6bc1d843833bc55a3f66ec342533ed7586b757d0f4f7c87c211ddcde597f3bba report.txt
362
+ ```
363
+
364
+ The archive also holds the `#SBATCH --output` file, here `1794696621749957.out`.
365
+
366
+ Running the same `download` again prints `Already downloaded …` and changes
367
+ nothing.
368
+
369
+ **4. Delete it.** Pass the output ID from step 2. This needs a login with the
370
+ `jobs:cancel` scope.
371
+
372
+ ```sh
373
+ flightdeck delete-output <flight-id> <output-id>
374
+ ```
375
+
376
+ ```text
377
+ Deleted output 59925a74-7a23-46e8-8bb0-26c7be1eddb0 from fl-00e67a3d3ed2ae2229952634
378
+ ```
379
+
380
+ The output leaves the list at once, and `flightdeck outputs` then prints
381
+ `No outputs available.` The stored copy is removed 30 days later.
382
+
383
+ **The same steps through the API.** Every request needs your credential and the
384
+ accounting header. The credential is `api_key` in `~/.flightdeck/config.json`.
385
+
386
+ ```sh
387
+ API=https://api.flightdeck.thermai.uk/api/v1
388
+ AUTH=(-H "Authorization: Bearer $FLIGHTDECK_API_KEY" -H "X-Flightdeck-Accounting: credits-v1")
389
+
390
+ # List: artifact_id, filename, size_bytes, checksum_sha256, media_type, created_at
391
+ curl -s "${AUTH[@]}" "$API/flights/<flight-id>/outputs"
392
+
393
+ # Download: the API answers with a redirect to a signed URL that works for 5 minutes
394
+ curl -sL "${AUTH[@]}" -o outputs.tar.gz "$API/flights/<flight-id>/outputs/<artifact-id>"
395
+ sha256sum outputs.tar.gz # must equal checksum_sha256 from the list
396
+
397
+ # Delete: 204 on success
398
+ curl -s -o /dev/null -w '%{http_code}\n' -X DELETE "${AUTH[@]}" \
399
+ "$API/flights/<flight-id>/outputs/<artifact-id>"
400
+
401
+ # What happened to the flight, including collection
402
+ curl -s "${AUTH[@]}" "$API/flights/<flight-id>/events"
403
+ ```
404
+
405
+ The list answers like this, and the downloaded file's SHA-256 equalled
406
+ `checksum_sha256`:
407
+
408
+ ```json
409
+ {
410
+ "outputs": [
411
+ {
412
+ "artifact_id": "59925a74-7a23-46e8-8bb0-26c7be1eddb0",
413
+ "filename": "outputs.tar.gz",
414
+ "size_bytes": 274,
415
+ "checksum_sha256": "0f88381b16b970004118af20db4877aa891d0b028638f336a1ffcba98e4f8b11",
416
+ "media_type": "application/gzip",
417
+ "created_at": "2026-11-14T22:56:09.464Z"
418
+ }
419
+ ],
420
+ "attempt_seq": 1,
421
+ "available_attempts": [1]
422
+ }
423
+ ```
424
+
425
+ The events for the same flight, with the delete recorded last:
426
+
427
+ ```text
428
+ 22:50:22.613Z state_changed Flight queued
429
+ 22:55:47.958Z state_changed Flight dispatched
430
+ 22:55:54.311Z state_changed Collecting logs and outputs
431
+ 22:56:09.484Z state_changed Flight completed
432
+ 22:57:34.672Z output_deleted An output was deleted at the customer's request.
433
+ ```
434
+
435
+ The list also returns `attempt_seq` and `available_attempts`; add `?attempt=<n>`
436
+ for an earlier attempt. `curl -L` does not send your credential to the signed
437
+ URL, and nor should your own client.
438
+
439
+ | Delete answer | Meaning |
440
+ | --- | --- |
441
+ | `204` | Deleted. |
442
+ | `404 not_found` | No such output, or the flight is not yours. |
443
+ | `409 flight_not_finished` | The flight is still running. Nothing was deleted. |
444
+ | `409 outputs_being_collected` | A cancelled flight is still saving its outputs. Try again in a few minutes. |
445
+
446
+ **When outputs are missing.** `outputs` and `download` print a warning when
447
+ something was not saved. The flight's events say why:
448
+
449
+ | Event | What happened | What to do |
450
+ | --- | --- | --- |
451
+ | `outputs_too_large` | The archive was over the size limit (5 GiB unless your operator says otherwise) and was not stored. | Write less to `/outputs` and run again. |
452
+ | `finalization_abandoned` | Saving failed five times and was given up. | Run again. A flight whose workload exited 0 is not charged; one whose workload failed is charged like any failed flight. |
453
+ | `deadline_collection_abandoned` | The flight hit its time limit and saving did not finish in the time left. | Raise `--time`, or write results earlier, and run again. |
454
+ | `cancel_stop_failed` | The flight was cancelled but its workload could not be confirmed stopped, so its outputs may not have been saved. | Check `outputs`. Run again if they are missing. |
455
+ | `cancel_collection_cutoff` | The flight was cancelled and saving had not finished an hour later, so it was stopped and the instance destroyed. | Check `outputs`. Run again if they are missing. |
456
+
457
+ **Known limits.**
458
+
459
+ - **No retry on request.** Saving is retried automatically, up to five times,
460
+ while the instance exists. After that the only way to get the results is to
461
+ run the flight again.
462
+ - **Outputs that were not saved are gone.** The instance is destroyed when
463
+ saving is given up, and anything that was only on it cannot be recovered.
464
+ - **A cancelled flight saves after it is cancelled.** Its events show
465
+ `cancel_collection_started` while saving and `cancel_collection_finished`
466
+ once it is done. Outputs appear in `outputs` when they are stored, usually
467
+ within a minute.
468
+ - **Results do not expire yet.** A saved output stays until you delete it. A
469
+ 30-day purge after a pilot ends is planned.
470
+ - **One archive per flight on Lambda.** You cannot download a single file from
471
+ inside `outputs.tar.gz`, or delete one.
472
+ - **Only your own flights.** Another account, in your organisation or outside
473
+ it, gets `404 not_found` for your flight's outputs.
474
+
475
+ ### What goes into a bundle
476
+
477
+ `submit` and `sbatch` pack the current directory, or the `--files` paths, into
478
+ the bundle that is unpacked on the node. These are left out by default:
479
+
480
+ - version control: `.git`, `.hg`, `.svn`, `.bzr` (directories or files)
481
+ - environment and credential files: `.env*`, `.netrc`, `.pypirc`, `.npmrc`,
482
+ `.git-credentials`, `.ssh/`, `.aws/`, `.gnupg/`, `.docker/`, `.kube/`,
483
+ `.flightdeck/`
484
+ - key files: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.ppk`, `id_rsa`, `id_dsa`,
485
+ `id_ecdsa`, `id_ed25519`, `id_ecdsa_sk`, `id_ed25519_sk`
486
+ - virtualenvs: `venv/`, `.venv/` and any directory containing `pyvenv.cfg`
487
+
488
+ These match regardless of case, so `.ENV` and `ID_RSA` are left out too.
489
+
490
+ A `.flightdeckignore` in the current directory adds rules in `.gitignore`
491
+ syntax, and `!<path>` includes something a default would leave out. Only a `!`
492
+ rule without wildcards lifts a default; one such as `!*/` re-includes only what
493
+ your own rules excluded:
494
+
495
+ ```gitignore
496
+ checkpoints/*
497
+ !checkpoints/final.pt
498
+ !.env.example
499
+ ```
500
+
501
+ As with `.gitignore`, a path under an excluded directory cannot be re-included
502
+ on its own; re-include the directory, or exclude its contents with `dir/*`
503
+ instead. Naming an excluded path with `--files` does not include it (a shell
504
+ glob such as `--files *` expands to every name); use `!<path>` in
505
+ `.flightdeckignore`. A `--files` path that goes through a symlinked directory
506
+ (`alias/credentials` with `alias -> .aws`) is checked under the path it resolves
507
+ to as well, so the `!<path>` must name the resolved path; one that resolves
508
+ outside the current directory is refused. Symlinks that dangle, whose target is not in the bundle
509
+ (such as `data -> /scratch/...`), or whose target is an absolute path are
510
+ skipped with a warning because they cannot resolve on the node; use relative
511
+ links within the directory. To use a dataset that lives outside it, copy it
512
+ in or stage it on the node another way. The file count and size print before upload, and
513
+ `sbatch --dry-run` shows the same summary.
514
+
515
+ ### Checking an image before you submit
516
+
517
+ `flightdeck image check <image>` checks an image with your local Docker, so you
518
+ learn before you submit whether Flightdeck is likely to admit it and how it
519
+ behaves. It
520
+ needs no login. When you are logged in, it reads the platform's current runtime
521
+ facts (the supervisor release and the host drivers) from the API; otherwise it
522
+ uses the ones built into the CLI.
523
+
524
+ ```sh
525
+ flightdeck image check ghcr.io/your-org/your-image:1.0
526
+ flightdeck image check ghcr.io/your-org/your-image:1.0 --json
527
+ ```
528
+
529
+ It reports each check as PASS, WARN, FAIL, INFO, SKIP or ERROR (the check
530
+ could not run), with a one-line reason, and ends with a verdict:
531
+
532
+ - **Submission rules**, the same code the API uses: `linux/amd64`, a docker.io,
533
+ ghcr.io or nvcr.io reference within 255 characters once pinned to its digest,
534
+ and the `uk.thermai.flightdeck.contract` label. If Docker's copy is for
535
+ another platform (Docker on Apple silicon pulls `arm64`), the check asks the
536
+ registry whether it has the `linux/amd64` image Flightdeck runs: if it does,
537
+ that is a warning and the dry run is skipped; if not, it fails. The check
538
+ never re-pulls or retags your copy. A shell-form `ENTRYPOINT` is a
539
+ warning: Flightdeck refuses it for flights with a command, and runs it as it
540
+ is for image-only flights. `tini` or `dumb-init` as the `ENTRYPOINT` is a
541
+ warning too, because the supervisor already does their job.
542
+ - **Registry**: Flightdeck reads the image from its registry, without
543
+ credentials. A local image that is not in its registry yet is a warning, and
544
+ so is one the registry will not serve without credentials (the check asks it
545
+ anonymously, with `docker manifest inspect` and an empty Docker
546
+ configuration). New GitHub packages are private until you make them public.
547
+ - **Dry run**: the image starts the way a flight starts, under the Flightdeck
548
+ supervisor with `/workspace` and `/outputs` mounted, CPU only unless you pass
549
+ `--gpu`. `--gpu` attaches GPUs with Docker's `--gpus`; Flightdeck's hosts use
550
+ CDI, so the dry run does not test a provider's GPUs or driver. It runs a command as the image's `USER`; an `ENTRYPOINT` that ignores
551
+ the command fails. It installs a requirements file with no packages in it; a
552
+ missing or unusable `pip` is a warning, since only a bundle with a
553
+ requirements file needs one. It stops a running command with `SIGTERM` and
554
+ expects it to exit within the 30-second grace. A missing `bash` is a warning,
555
+ because commands then run under `sh` and sbatch scripts need `bash`.
556
+ - **CUDA**, from `CUDA_VERSION`, `libcudart`, pip's CUDA runtime wheel and
557
+ `torch.version.cuda`: a CUDA newer than every published host driver runs is a
558
+ warning. An image without CUDA is fine. `flightdeck submit` and
559
+ `flightdeck sbatch` also warn, after the flight is accepted, when the image
560
+ sets a `CUDA_VERSION` newer than the lowest published host driver runs; the
561
+ other sources are read only here.
562
+ - **Secrets**: the image's file names are listed through `docker export`, without
563
+ extracting anything, and matched against the credential and key patterns in
564
+ [What goes into a bundle](#what-goes-into-a-bundle). Empty files, CA
565
+ certificates and the test keys that packages ship are skipped.
566
+
567
+ The exit status is 0 when nothing failed (warnings allowed), 1 when a check
568
+ failed, and 2 when the check itself could not run, for example without Docker
569
+ or when the pull failed. `--static-only` runs nothing from the image, and
570
+ `--supervisor <image>` runs the dry run under another supervisor image.
571
+
572
+ The verdict says what the result means for submission:
573
+
574
+ - **PASS**, with every check run: Flightdeck will admit the image, and it ran
575
+ commands as the workload runtime contract describes.
576
+ - **PASS for the checks that ran**: nothing failed, but the verdict lists what
577
+ was not checked, such as the dry run under `--static-only`.
578
+ - **WARN, "Flightdeck may refuse this image"**: a warning that admission acts
579
+ on, named in the verdict. The image is not in its registry yet, the registry
580
+ will not serve it without credentials, or its `ENTRYPOINT` is shell form,
581
+ which is refused for every flight with a command.
582
+ - **WARN** otherwise: Flightdeck will admit the image; read the warnings.
583
+ - **FAIL**: Flightdeck would refuse the image, or its flights would not run.
584
+
585
+ If Docker does not have the image, the check pulls it for `linux/amd64` and
586
+ leaves it for you; it never removes an image you already had. Everything else
587
+ it creates is removed when it finishes, fails or is interrupted with Ctrl-C:
588
+ its containers (named `flightdeck-image-check-<id>-…` and labelled
589
+ `uk.thermai.flightdeck.image-check`), its temporary directory, and the
590
+ supervisor image if the check pulled it (unless another image check is
591
+ running, which may still need it). After Ctrl-C the check waits for Docker to
592
+ remove its containers, up to a minute each, and a second Ctrl-C does not cut
593
+ that short. If the check is killed outright, list what it left with
594
+ `docker ps -a --filter label=uk.thermai.flightdeck.image-check`, remove each with
595
+ `docker rm -f <name>`, and delete any `flightdeck-image-check-*` directory in
596
+ your temporary directory. On Apple silicon and other machines that are not
597
+ amd64, the image runs under emulation, which is slow.
598
+
599
+ The check does not test your code or the provider's GPUs, so run a short flight
600
+ before real work.
601
+
602
+ ### Publishing a private image
603
+
604
+ `flightdeck image push <local-image> --name <name> --version <label>` publishes
605
+ a local image to your organization's private registry. `image list` and
606
+ `image inspect <version-id> --format reference` show it and print the reference
607
+ to pass to `--base-image`. Log in with `flightdeck login --images` first (run
608
+ `logout` before it if you are already logged in). Rerun the same push to resume.
609
+ Managed images run on Lambda only, so add `--provider lambda` to `submit`.
610
+ `--base-image managed://<name>:latest` runs the newest ready version, and
611
+ `image push --alias ci` or `image alias set <name> ci --version <label>` moves a channel
612
+ you can run as `managed://<name>:ci`. The full guide, including errors and limits, is in
613
+ [`docs/managed-images.md`](../docs/managed-images.md).
614
+
615
+ 1. `flightdeck logout`, then `flightdeck login --images`.
616
+ 2. `flightdeck image push solver-local:v1 --name solver --version v1`.
617
+ 3. `flightdeck image inspect <version-id>` shows the reference and the scan state.
618
+
619
+ Messages you may see:
620
+
621
+ - A 401 prints "Your FlightDeck login is no longer valid. The credential expired
622
+ or was revoked. Run flightdeck logout, then flightdeck login --images."
623
+ - A login made without `--images` prints "This login cannot use managed images.
624
+ Run flightdeck logout, then flightdeck login --images."
625
+ - An image above the registry's size limit, when one is set, is rejected after
626
+ the push and is not added to your list. Push a smaller image with a new
627
+ `--version`.
628
+ - A push that is not a single linux/amd64 image is rejected and names the
629
+ platform it found.
630
+
631
+ `image inspect` and `image list` show the registry's scan of the image, and
632
+ `image push` points to `image inspect` for it. Findings are advisory. They never block a push or a flight.
633
+
634
+ ### Datasets: upload once, use in many flights
635
+
636
+ Managed storage keeps a directory of files in your organisation's space. You
637
+ upload it once as a version that never changes, then mount it read-only in any
638
+ Lambda flight. ThermAI must turn storage on for your organisation first. An
639
+ organisation admin uploads and deletes; any active member can list, download
640
+ and mount. This walk-through uploads a small dataset, uses it in a flight,
641
+ downloads it and deletes it. The output shown is from a run on staging on
642
+ 14 November 2026. `dataset download` needs CLI `0.1.0-pilot.10` or later.
643
+
644
+ **1. Log in with storage access.** `login` refuses a second sign-in, so log out
645
+ first.
646
+
647
+ ```sh
648
+ flightdeck logout
649
+ flightdeck login --storage
650
+ ```
651
+
652
+ **2. Upload a directory.** Note the hash, so you can check it later. On macOS
653
+ use `shasum -a 256` wherever this walk-through says `sha256sum`.
654
+
655
+ ```sh
656
+ mkdir corpus && head -c 1048576 /dev/urandom > corpus/sample.bin
657
+ sha256sum corpus/sample.bin
658
+ flightdeck dataset push ./corpus --name corpus --version 2026-11
659
+ ```
660
+
661
+ ```text
662
+ f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 corpus/sample.bin
663
+ Uploading corpus:2026-11: 1 files, 1.0 MiB
664
+ [1/1] sample.bin (1.0 MiB)
665
+ Verifying the upload
666
+ Published corpus:2026-11
667
+ Version ID: 33d54875-8fdd-4a25-a4d0-17e4a7449229
668
+ Files: 1, size: 1.0 MiB
669
+ Use it with: flightdeck submit "<command>" --dataset corpus:2026-11
670
+ ```
671
+
672
+ If the upload is interrupted (Ctrl-C, a dropped connection, a closed laptop),
673
+ run the same command again. It sends only the parts that are missing. A 400 MiB
674
+ push stopped with Ctrl-C after ten seconds, then run again, printed:
675
+
676
+ ```text
677
+ Interrupted. Stopping the upload.
678
+ The push was interrupted and did not finish. Run the same command again to resume.
679
+ ```
680
+
681
+ ```text
682
+ Uploading resume-check:v1: 1 files, 400.0 MiB
683
+ [1/1] big.bin (400.0 MiB)
684
+ resumed: 1 parts were already uploaded
685
+ Verifying the upload
686
+ Published resume-check:v1
687
+ ```
688
+
689
+ Between the two, `dataset list` shows the version as `reserved`, and a flight
690
+ cannot use it yet. A push
691
+ of a version that is already complete prints `Already published corpus:2026-11`
692
+ in place of the upload lines and uploads nothing. A version never changes: to upload different files, choose a new
693
+ `--version`.
694
+
695
+ **3. List it.**
696
+
697
+ ```sh
698
+ flightdeck dataset list
699
+ ```
700
+
701
+ ```text
702
+ NAME VERSION STATE SIZE FILES CREATED VERSION ID
703
+ corpus 2026-11 ready 1.0 MiB 1 2026-11-14 33d54875-8fdd-4a25-a4d0-17e4a7449229
704
+ ```
705
+
706
+ Only a `ready` version can be mounted. `--json` gives the same list for
707
+ scripts.
708
+
709
+ **4. Use it in a flight.** Give `--dataset` as `<name>:<version>`, or as the
710
+ version ID. Add `:<mount>` to choose the directory name. Repeat the flag for up
711
+ to four datasets.
712
+
713
+ `submit` packs the current directory into the flight's bundle, so run it from a
714
+ directory that holds only the job's files. Do not run it from the directory
715
+ that holds `corpus/`, or the dataset is packed into the bundle as well. An
716
+ empty directory is refused with `Nothing to bundle`.
717
+
718
+ ```sh
719
+ mkdir job && cd job
720
+ echo 'sha256sum /data/corpus/sample.bin' > check.sh
721
+ flightdeck submit "bash check.sh" --provider lambda \
722
+ --dataset corpus:2026-11 --time 5m
723
+ flightdeck status <flight-id> # a Data: line shows the copy's progress
724
+ flightdeck logs <flight-id> # the hash must equal the one from step 2
725
+ cd ..
726
+ ```
727
+
728
+ ```text
729
+ Flight submitted successfully!
730
+ Flight ID: fl-27aec32986c7f001885230f2
731
+ Status: queued
732
+ Est. credits: 14
733
+ Queue position: 1
734
+ ```
735
+
736
+ While the flight is `dispatching`, `status` shows the copy, and `logs` shows
737
+ the same hash as step 2 once it has run:
738
+
739
+ ```text
740
+ Data: Datasets fetched: 1.0 MiB in 1 file
741
+ ```
742
+
743
+ ```text
744
+ f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 /data/corpus/sample.bin
745
+ ```
746
+
747
+ A second flight submitted with the version ID in place of the name
748
+ (`--dataset 33d54875-8fdd-4a25-a4d0-17e4a7449229`) printed the same line.
749
+
750
+ The flight reads the files at `/data/<mount>/`, read-only. `FLIGHTDECK_DATA_DIR`
751
+ is `/data`. Flightdeck copies the files to the machine before your container
752
+ starts, so reads are at local disk speed, and the flight stays `dispatching`
753
+ while it copies. The version is fixed when you submit: a flight always runs the
754
+ data it named, whatever is uploaded later. A second flight with the same
755
+ `--dataset` reuses the same upload.
756
+
757
+ **5. Download it, then delete it.**
758
+
759
+ ```sh
760
+ flightdeck dataset download corpus:2026-11 ./corpus-copy
761
+ sha256sum ./corpus-copy/sample.bin
762
+ flightdeck dataset rm corpus:2026-11
763
+ ```
764
+
765
+ ```text
766
+ Downloaded 1 file(s), 1.0 MiB, of corpus:2026-11 to /home/you/corpus-copy.
767
+ f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 ./corpus-copy/sample.bin
768
+ Deleted corpus:2026-11. New flights cannot use it. Its files are removed after the retention period.
769
+ ```
770
+
771
+ After the delete, the version is gone from `dataset list`, and a submit that
772
+ names it is refused before anything launches:
773
+
774
+ ```text
775
+ Submit failed (404): A dataset in --dataset was not found in your organization.
776
+ ```
777
+
778
+ `download` checks every file against the checksum recorded at upload. Run it
779
+ again to resume; it reports files it already has as `already present and
780
+ verified` and never overwrites a file that differs. `rm` is refused while a
781
+ queued, dispatching or running flight uses the version. Deleted files are
782
+ removed from storage 7 days later.
783
+
784
+ **The same steps through the API.** The routes are under `/api/v1/storage`.
785
+ Reading needs the `storage:read` scope and changing needs `storage:write`,
786
+ which a `login --storage` credential has. The credential is `api_key` in
787
+ `~/.flightdeck/config.json`.
788
+
789
+ ```sh
790
+ API=https://api.flightdeck.thermai.uk/api/v1
791
+ AUTH=(-H "Authorization: Bearer $FLIGHTDECK_API_KEY" -H "X-Flightdeck-Accounting: credits-v1")
792
+ JSON=(-H "Content-Type: application/json")
793
+
794
+ curl -s "${AUTH[@]}" "$API/storage" # enabled, entitled, usage, quota
795
+ curl -s "${AUTH[@]}" "$API/storage/datasets" # datasets
796
+ curl -s "${AUTH[@]}" "$API/storage/datasets/<dataset-id>/versions" # versions and their IDs
797
+ curl -s "${AUTH[@]}" "$API/storage/versions/<version-id>/files" # files, sizes and checksums
798
+
799
+ # Use it in a flight: add datasets to the body of POST /flights
800
+ # "datasets": [{"ref": "corpus:2026-11"}, {"ref": "<version-id>", "mount": "weights"}]
801
+
802
+ # Download: signed links for up to 100 files, each valid for one hour
803
+ curl -s "${AUTH[@]}" "${JSON[@]}" -X POST -d '{"paths":["sample.bin"]}' \
804
+ "$API/storage/versions/<version-id>/download-urls"
805
+
806
+ # Delete
807
+ curl -s "${AUTH[@]}" "${JSON[@]}" -X POST -d '{}' "$API/storage/versions/<version-id>/delete"
808
+ ```
809
+
810
+ The files call returns each file's path, size and checksum, and the delete
811
+ call answers with the version's new state:
812
+
813
+ ```json
814
+ {"files":[{"path":"sample.bin","size":1048576,"checksum":"SWvdRr7il3MHAC6btaOHPbHcf+lGzGNqWNpClfFNWfk=-1"}],"next_cursor":null}
815
+ {"id":"8e935b05-ac09-43a4-8ce9-1626ad587c45","state":"deleted","changed":true}
816
+ ```
817
+
818
+ The download call returns one entry per file with the same fields plus
819
+ `part_size`, a signed `url` and `expires_at`.
820
+
821
+ Uploading through the API takes several calls per file (open a session, sign
822
+ each part, complete each file, then verify). Use `dataset push`, which also
823
+ resumes; the calls are described in `docs/managed-storage.md` if you need them.
824
+
825
+ **When something is refused.** Each refusal has a code, and nothing is
826
+ charged. The CLI prints a sentence for most of them. At submit,
827
+ `credential_scope_required` (a login without `--storage`) and
828
+ `dataset_mount_invalid` are printed as raw JSON.
829
+
830
+ | Code | When | What to do |
831
+ | --- | --- | --- |
832
+ | `storage_not_entitled` | Storage is not turned on for your organisation. | Ask ThermAI. |
833
+ | `credential_scope_required` | Your login has no storage access. | `flightdeck logout`, then `flightdeck login --storage`. |
834
+ | `org_admin_required` | A member who is not an admin tried to upload or delete. | Ask an organisation admin. |
835
+ | `storage_quota_exceeded` | The upload does not fit in your organisation's space. | Delete a version with `dataset rm`, or ask ThermAI. Space is freed when the deleted files are removed. |
836
+ | `dataset_limit_reached`, `version_limit_reached` | Too many datasets, or too many versions of one. | Delete one, or ask ThermAI. |
837
+ | `file_too_large` | One file is over the largest file size. | Split the file. |
838
+ | `version_exists` | That `--version` already exists for the dataset, from any upload. A deleted version's label stays reserved for good. | Choose a new `--version`. |
839
+ | `idempotency_conflict` | The same `--version` was pushed again with different files. | Choose a new `--version`. A version never changes. |
840
+ | `storage_fetch_quota_exceeded` | Your organisation reached its daily download limit, on a download or on a submit. | Wait. The limit counts the last 24 hours. Files already downloaded are kept. |
841
+ | `dataset_not_found` | The dataset is not in your organisation, or the version was deleted. | Check `dataset list`. |
842
+ | `dataset_not_ready` | The version is still uploading, or was disabled. | Check its state in `dataset list`. |
843
+ | `dataset_limit_exceeded` | The datasets are over the per-flight size or file limit, or do not fit the machine's disk. | Mount less. |
844
+ | `version_in_use` | `rm` on a version a queued, dispatching or running flight uses. | Wait for those flights to end. |
845
+
846
+ Four of these were produced on staging on 14 November 2026:
847
+ `dataset_not_found`, `dataset_not_ready`, `version_in_use` and
848
+ `idempotency_conflict`. The CLI printed:
849
+
850
+ ```text
851
+ A dataset in --dataset was not found in your organization.
852
+ A dataset in --dataset is not ready. Check flightdeck dataset list.
853
+ corpus:2026-11b is in use by a queued, dispatching or running flight. Try again when those flights end.
854
+ A different upload is already recorded for corpus:2026-11b. If the directory changed since the first push, choose a new --version.
855
+ ```
856
+
857
+ **When a flight cannot get its data.** The copy happens before your container
858
+ starts, so a flight that fails here is not charged. `flightdeck status` shows
859
+ the code.
860
+
861
+ | Code | What happened | What to do |
862
+ | --- | --- | --- |
863
+ | `data_fetch_failed` | The copy failed or was too slow. Flightdeck tries again on a new machine, up to three attempts. | Nothing. Watch `status`. |
864
+ | `data_fetch_exhausted` | The copy failed the same way twice, or the daily download limit ran out during it. | Submit again later. If it repeats, send ThermAI the flight ID. |
865
+ | `data_unavailable` | The version was disabled or deleted after you submitted, or storage was turned off for your organisation. | Check `dataset list`. |
866
+ | `data_too_large` | The data does not fit the machine's disk. | Mount less. |
867
+
868
+ **Known limits.**
869
+
870
+ - **Pilot limits:** 1 TiB and 50 datasets per organisation, 50 GiB for the
871
+ largest file, 50 GiB mounted per flight, and 150 GiB of downloads per
872
+ organisation in any 24 hours.
873
+ - **Downloads share one allowance.** `dataset download` and every flight that
874
+ mounts data count toward the same 150 GiB. Each attempt of a flight counts
875
+ again, so a flight that retries uses its size more than once. A large
876
+ download can leave too little for that day's flights.
877
+ - **Up to 4 datasets per flight**, 5000 files per version and 20 versions per
878
+ dataset.
879
+ - **Lambda only.** A flight on another provider is refused with
880
+ `dataset_provider_unsupported`.
881
+ - **Read-only, and copied every time.** A flight cannot write to `/data`, and
882
+ nothing is cached between flights: each flight copies its data again, which
883
+ adds to its start-up time.
884
+ - **File names.** Up to 8 path segments of letters, digits, `_`, `-` and `.`,
885
+ each up to 128 characters with no leading dot, and 255 characters in all.
886
+ Names with spaces, symbolic links and special files are refused at upload.
887
+ - **No `inspect`, `disable` or `enable` commands yet.** Use the API for those.
888
+
889
+ ### Keeping secrets
890
+
891
+ `flightdeck secret` keeps private credentials, such as a Hugging Face token,
892
+ for your organization's flights. Secrets must be enabled for your organization
893
+ by your ThermAI operator. Any member can list names and details; only an
894
+ organization admin can create, rotate or revoke. Nothing ever prints a value
895
+ back.
896
+
897
+ ```sh
898
+ flightdeck logout
899
+ flightdeck login --secrets
900
+ printf '%s' "$HF_TOKEN" | flightdeck secret create hf-token --value-stdin --description "Hub token"
901
+ flightdeck secret list
902
+ flightdeck secret rotate hf-token --from-file ./new-token.txt --expires-at 2027-06-30
903
+ flightdeck secret revoke hf-token --yes
904
+ ```
905
+
906
+ - A value is never taken from the command line, where it would stay in shell
907
+ history. Pipe it in with `--value-stdin` (a terminal is refused, since the
908
+ value would be shown) or name a regular file with `--from-file`. One trailing
909
+ line break is removed. A value mistyped as an argument or an unknown option
910
+ is refused without being repeated.
911
+ - Names use lowercase letters and digits joined by `.`, `_` or `-`, up to 64
912
+ characters. Values are text up to 32 KiB. An organization holds up to 100.
913
+ - `--expires-at` takes a date or a date and time with a zone. A date alone means
914
+ midnight UTC at the start of that day. On `rotate`, leaving it out keeps the
915
+ current expiry and `--no-expiry` removes it. An expired secret must be rotated
916
+ with one of the two.
917
+ - `revoke` needs `--yes`. A running flight keeps the value it started with, and
918
+ revoking here does not revoke the token where it was issued: revoke it there
919
+ too.
920
+ - `--json` prints the metadata as JSON on every command.
921
+
922
+ The full guide is in [`docs/customer-secrets.md`](../docs/customer-secrets.md).
923
+
924
+ ### Slurm differences
925
+
926
+ - Single-node jobs only; this is not a full Slurm scheduler or an HPC login node.
927
+ - Array `%N` concurrency throttles are not enforced, so `sbatch` refuses one
928
+ smaller than the array. `--ignore-array-limit` submits every task
929
+ immediately. Admission ceilings may reject excess tasks rather than queue
930
+ them for later admission. Review every warning before real submission.
931
+ - In an array task, `%j` is the array job id, not a per-task id; use `%a` in
932
+ `--output`/`--error` names so tasks don't share a file.
933
+ - Array tasks default to `slurm-%A_%a.out`, as in Slurm; write results under
934
+ `$OUTPUTS_DIR`. There is no shared university filesystem or implicit `module`,
935
+ Singularity, nested Docker, or cluster software environment.
936
+ - Hardware, runtime, image and credit must fit the operator-approved policy.
937
+ The image must meet the workload runtime contract: linux/amd64, and an
938
+ exec-form `ENTRYPOINT` if it sets one. Submission refuses an image that does
939
+ not.
940
+
941
+ ### Reading logs completely
942
+
943
+ From `0.1.0-pilot.3` (containing the #99 pagination fix), `flightdeck logs`
944
+ drains all pages in the current stored-log snapshot, then returns. For a running
945
+ job it then reads the live tail, which the backend serves a bounded page at a
946
+ time. `--follow` drains those pages immediately, then polls every two seconds
947
+ until the job is terminal and its stored backlog is exhausted. Every new
948
+ invocation starts at the beginning; stdout/stderr retain their respective
949
+ output streams.
950
+
951
+ If the API stops answering while `--follow` is running (a connection failure, or
952
+ HTTP 502, 503 or 504, as happens for a few seconds when a release restarts the
953
+ backend), the CLI says so on stderr and retries for up to a minute before it
954
+ gives up. The flight is not affected. A read without `--follow` does not retry.
955
+ `flightdeck slurm-test` retries the same way while it waits for its flights and
956
+ when it reads their logs and outputs. It never sends the submission twice.
957
+
958
+ Each log stream is capped at 5 MiB. On Lambda, a finished flight's stream that
959
+ passes the cap keeps its first 5 MiB and its last 64 KiB (each cut between
960
+ characters), with a `flightdeck: log truncated` marker line between them when
961
+ anything was left out, so the end of the run and any
962
+ `flightdeck: failure reason` line survive. When a log is truncated,
963
+ `flightdeck logs` prints a one-line warning on stderr after the text; stdout
964
+ carries only the flight's own output.
965
+
966
+ Flight logs are kept for 30 days after they're written. For an older flight,
967
+ `flightdeck logs` prints nothing on stdout and says on stderr when the logs were
968
+ deleted.
969
+
970
+ **Rollout order:** deploy and verify the pagination-capable backend before
971
+ releasing/upgrading this CLI. A backend without the `has_more` response field
972
+ produces an explicit compatibility error and exit status 1, not a silently
973
+ partial read. Older `0.1.0-pilot.2` installations do not contain this fix;
974
+ merging source alone neither replaces that immutable package nor publishes a
975
+ new version. Older non-following clients still need upgrading for full paging.
976
+
977
+ ### Watching flights live
978
+
979
+ `flightdeck atc` (alias `flightdeck tui`) opens a read-only, full-screen view in
980
+ the current terminal. Arrow keys select, Enter shows a flight's details and
981
+ events, `l` opens its logs (scroll, `f` to follow), `r` refreshes, Esc goes back
982
+ and `q` quits. It never cancels, submits or downloads anything.
983
+
984
+ ```sh
985
+ flightdeck atc # your recent flights (last 50)
986
+ flightdeck atc <flight-id> # one flight
987
+ flightdeck atc <job-id> # an sbatch array submitted from this machine
988
+ flightdeck atc --name 'molsim*' # flights whose name matches, among your last 200
989
+ flightdeck atc --demo # a simulated array; no account or credits needed
990
+ ```
991
+
992
+ `flightdeck sbatch` records each array's flight IDs under
993
+ `~/.flightdeck/batches/` and ends with `Watch live: flightdeck atc <job-id>`;
994
+ that job ID is the one to pass. It is a local sbatch job ID, not a flight ID:
995
+ `status`, `logs`, `outputs`, `download`, `delete-output` and `cancel` take the `fl-…` flight
996
+ IDs printed for each task. The record exists only on the machine that
997
+ submitted it (if it could not be written, sbatch says so; use `--name`). Name
998
+ patterns are case-sensitive. Leaving the view does not stop any job. In
999
+ scripts, keep using `status` and `logs`.
1000
+
1001
+ ### Error handling and support
1002
+
1003
+ The CLI exits with status 1 on command failure. Expected authentication,
1004
+ network and HTTP failures have concise diagnostics instead of a Node stack.
1005
+ An already-configured login leaves the saved credential unchanged.
1006
+
1007
+ **Pilot.6 logout behavior (#143):** remove the local file first, then make
1008
+ one best-effort server revocation request with a three-second deadline, including
1009
+ the response body. Offline, suspended, failed or malformed API responses cannot
1010
+ retain or restore the local login. Local success exits 0; a warning separately
1011
+ states when remote revocation was not confirmed and other copies may remain
1012
+ valid. Use portal credential management or contact your organization admin.
1013
+ Failed local deletion exits 1, reports the problem and makes no API request.
1014
+ Missing local login is a successful no-op; malformed config is removed without
1015
+ trusting its contents for a request. No credential recovery/retry file is saved.
1016
+ `logout --local` removes only the local copy and makes no network request.
1017
+
1018
+ Older pilot.5 does not include this change; use `flightdeck logout --local`
1019
+ as a local-removal workaround before upgrading. It does not revoke remote
1020
+ access. The current production backend permits narrowly authenticated
1021
+ self-revocation under suspension; ordinary access remains blocked.
1022
+
1023
+ For unexpected faults, `flightdeck --debug <command>` adds allowlisted error
1024
+ types/codes and bundled CLI line locations, not raw stacks or error bodies.
1025
+ It is not a dry-run flag: only repeat an operation that is safe to repeat.
1026
+ Provide support with the CLI/Node versions, exit status, redacted command and
1027
+ flight ID if available. Review output before sharing: account details and
1028
+ workload logs/results can contain private information. Never send passwords,
1029
+ tokens, activation links or `~/.flightdeck/config.json`.
65
1030
 
66
1031
  ## Versions
67
1032
 
@@ -70,8 +1035,10 @@ flightdeck --version
70
1035
  npm install -g @thermai/flightdeck-cli@<approved-version>
71
1036
  ```
72
1037
 
73
- Stable releases use npm's `latest` tag. Explicit pilot/prerelease versions use
74
- `next`; they do not automatically replace stable installations. Updating is an
1038
+ Stable releases will use npm's `latest` tag. Explicit pilot/prerelease versions use
1039
+ `next`; they do not automatically replace stable installations. Currently
1040
+ `latest` still points at the initial pilot.1 bootstrap, not a qualified stable
1041
+ release; use `next` or the approved exact pilot version. Updating is an
75
1042
  explicit install, not an automatic mutation performed when the CLI runs.
76
1043
  Once a stable release exists, `npm install -g @thermai/flightdeck-cli` selects it.
77
1044