@thermai/flightdeck-cli 0.1.0-pilot.1 → 0.1.0-pilot.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +945 -17
  2. package/dist/index.js +9033 -927
  3. package/package.json +8 -3
package/README.md CHANGED
@@ -2,31 +2,112 @@
2
2
 
3
3
  Install the command-line client, authorize it through the Flightdeck portal,
4
4
  then submit and inspect GPU jobs. No npm account is required to install the
5
- planned public package. A provisioned Flightdeck account is required to use the
5
+ public package. A provisioned Flightdeck account is required to use the
6
6
  service; downloading the CLI does not create an account or grant compute credit.
7
7
 
8
8
  ## Release status
9
9
 
10
- The npm publishing pathway is being prepared. The operator has created the
11
- `@thermai` npm organization; the package name is `@thermai/flightdeck-cli`.
12
- The prepared candidate is `0.1.0-pilot.1`, with operator-supplied proprietary
13
- terms in `LICENSE`, amended at the operator's direction to allow redistribution
14
- of unchanged official releases. The operator has approved proceeding with the
15
- internal proving release; publication awaits technical release checks and
16
- completion of publishing setup. The commands
17
- below describe the intended released experience, not an announcement that this
18
- package is already on npm.
10
+ The package is `@thermai/flightdeck-cli`. The first pilot, `0.1.0-pilot.1`, was
11
+ published manually on 5 September 2026 and its anonymous install was verified.
12
+ `0.1.0-pilot.2` subsequently passed GitHub Actions trusted publishing and an
13
+ anonymous install check on the same date. Releases publish through the
14
+ versioned GitHub Release workflow, not merely by merging a version change. The
15
+ installation command below follows the published `next` channel without
16
+ tracking an unpublished candidate version.
17
+ Proprietary terms in `LICENSE` permit redistribution of unchanged official
18
+ releases. Service access still requires an invitation, configured credits and
19
+ the agreed pilot limits; installing the package alone does not grant access.
20
+
21
+ **`0.1.0-pilot.3`** was published through GitHub Actions on 8 September 2026,
22
+ including complete log pagination and concise authentication errors. Its
23
+ anonymous Node 22 install and packaged dry run passed. Use the operator-approved
24
+ exact version for a recorded rehearsal.
25
+
26
+ **`0.1.0-pilot.4`**, published on 20 September 2026, adds customer credits and
27
+ requires a backend advertising `X-Flightdeck-Accounting: credits-v1`. Upgrade
28
+ with the install command below before accessing account/job data. An incompatible
29
+ backend is rejected before job submission; login and credential revocation do not require
30
+ the accounting contract. See the
31
+ operator rollout checklist (`flightdeck-ops/runbooks/customer-credits-rollout.md`).
32
+
33
+ **`0.1.0-pilot.5`**, published on 21 September 2026, fixes result downloads redirected to signed object
34
+ storage ([#157](https://github.com/ThermAI-Group/thermai_flightdeck/issues/157))
35
+ and includes the merged GPU-recovery status display. Downloads validate the
36
+ accounting contract on Flightdeck's response, then fetch the authorized storage
37
+ object without forwarding the CLI credential. Anonymous Node 22 installation
38
+ and a live result download passed. Upgrade from
39
+ pilot.4 before downloading results. Future releases still require an approved
40
+ versioned GitHub Release; merging alone does not publish.
41
+
42
+ **`0.1.0-pilot.6`**, published on 21 September 2026, adds
43
+ offline-first logout and explicit suspension messages. Release CI, registry
44
+ integrity and anonymous Node 22 installation passed; grouped live acceptance
45
+ was confirmed by the operator on 24 September. These are historical pilot.6
46
+ results, not pilot.7 publication evidence. This update does not modify the
47
+ README inside the already-published immutable tarball.
48
+
49
+ **`0.1.0-pilot.7`**, published on 1 October 2026, adds the read-only
50
+ `flightdeck atc` terminal view (alias `tui`), flight details/events/logs, name
51
+ filtering and local sbatch-array tracking from
52
+ [PR #206](https://github.com/ThermAI-Group/thermai_flightdeck/pull/206).
53
+ Downloads/cancellation inside ATC remain deferred to #205; stale
54
+ fallback-status indication is tracked in #207.
55
+
56
+ **`0.1.0-pilot.8`** adds `flightdeck admin flights` for organisation and
57
+ platform administrators ([PR #281](https://github.com/ThermAI-Group/thermai_flightdeck/pull/281)).
58
+ It needs a credential approved through `flightdeck login --admin`; existing
59
+ credentials keep working for every other command, so administrators run
60
+ `flightdeck logout` then `flightdeck login --admin` only when they want the
61
+ admin view. `flightdeck login` now defaults the credential name to
62
+ `flightdeck-cli@<hostname>` so a second machine no longer collides (#254), and
63
+ `flightdeck sbatch` prints a runnable status hint with the submitted flight ID
64
+ and labels the job ID on the ATC hint (#257, #259). The required backend is
65
+ already deployed; there are no new runtime dependencies. It was published to
66
+ npm's `next` channel on 8 October 2026. Use the generated `@next` command below, or pin
67
+ `@thermai/flightdeck-cli@0.1.0-pilot.8`, and confirm `flightdeck --version`
68
+ reports `0.1.0-pilot.8`.
69
+
70
+ **`0.1.0-pilot.9`** adds managed images and datasets. `flightdeck image check`,
71
+ `push`, `list` and `inspect` check and publish workload images to your
72
+ organization's managed registry. `flightdeck image alias ls|set|rm` and
73
+ `image push --alias` manage channels, and `managed://<name>:<alias>` (including
74
+ the computed `latest`) can be submitted as an image. `flightdeck dataset push`,
75
+ `list` and `rm` manage datasets, and `submit --dataset` mounts them. `sbatch`
76
+ refuses an array `%N` limit Flightdeck cannot enforce unless
77
+ `--ignore-array-limit` is passed. Alias commands need a backend with image
78
+ aliases; the other commands work against the current backend. There are no new
79
+ runtime dependencies.
80
+
81
+ **`0.1.0-pilot.10`** adds `flightdeck delete-output <flight-id> <output-id>`,
82
+ which deletes one output of a finished flight, and
83
+ `flightdeck dataset download <version> <directory>`, which downloads a dataset
84
+ version and checks every file against its recorded checksum; running it again
85
+ resumes. While a flight copies its datasets, `flightdeck status` shows their
86
+ progress on a `Data:` line. `flightdeck image check` now checks images against
87
+ the supervisor release that Flightdeck 26.11.3 runs. Both new commands need Flightdeck 26.11.3
88
+ or later, and `dataset download` also needs storage enabled for your
89
+ organisation. There are no new runtime dependencies.
90
+
91
+ Members and organization administrators receive whole-credit balances,
92
+ estimates and actual customer charges, without provider-cost information.
93
+ Platform administrators retain their authorized monetary API fields. Credits
94
+ are not GPU-hours; consumption depends on the selected hardware/provider and
95
+ the accepted job pricing. Existing limits and customer charge caps still apply.
19
96
 
20
97
  ## Installation and login
21
98
 
22
99
  Requires Node.js 22 and npm. Use a user-owned Node installation or npm prefix;
23
100
  the CLI does not require administrator/root privileges.
24
101
 
102
+ <!-- flightdeck-cli-install:start -->
103
+
25
104
  ```sh
26
- npm install -g @thermai/flightdeck-cli@0.1.0-pilot.1
105
+ npm install -g @thermai/flightdeck-cli@next
27
106
  flightdeck login
28
107
  ```
29
108
 
109
+ <!-- flightdeck-cli-install:end -->
110
+
30
111
  Login prints a URL and verification code. Open that URL, sign into
31
112
  `https://portal.thermai.uk`, and approve the named CLI credential and requested
32
113
  scopes. The CLI waits for approval; no API key needs to be copied manually.
@@ -42,26 +123,871 @@ test environment can be selected with `flightdeck login --api-url <HTTPS origin>
42
123
 
43
124
  Login saves a scoped credential in `~/.flightdeck/config.json` with restrictive
44
125
  permissions. Never share that file or include it in a workload bundle. Use
45
- `flightdeck logout` to revoke the credential and remove the local login. Portal
46
- credential management can revoke an individual workstation as well.
126
+ `flightdeck logout` to remove the local login and attempt server revocation.
127
+ Portal credential management can revoke an individual workstation as well.
128
+
129
+ `--name` labels a credential, not a local account profile. It defaults to
130
+ `flightdeck-cli@<short hostname>`, which is shown on the approval page, in
131
+ credential lists, and in the audit history; pass `--name` to choose another
132
+ label. Active credential names must be unique per account, so machines that
133
+ share a short hostname need distinct `--name` values. There is one saved
134
+ login per OS user. If already logged in, use `whoami` to inspect it; `login`
135
+ refuses to replace it. Use a separate OS user to preserve an operator's working
136
+ credential, or explicitly choose logout/re-login on the same workstation.
137
+ Logging out removes the saved credential; it does not delete the account or its
138
+ roles. A later login creates a new credential. Check the separate revocation
139
+ result: other copies may remain valid if server revocation was not confirmed.
140
+
141
+ Pilot.6 explains `account_suspended`, `membership_suspended`
142
+ and `organization_suspended` denials during device login and authenticated
143
+ commands. Contact your organization administrator; organization suspension may
144
+ need ThermAI support. These are access restrictions, not a missing local login
145
+ or a request for additional credential scopes. Other 403 responses retain
146
+ generic permissions guidance. Commands still exit nonzero and do not print
147
+ server-supplied messages or stack traces. Older pilot.5 installations need an
148
+ explicit upgrade to receive these messages.
149
+
150
+ ## Pilot quick-start
151
+
152
+ Before live use, your ThermAI operator must provision your organization/account,
153
+ provide a single-use portal invitation, and approve your policy and credit.
154
+ Activate through the portal, then follow the installation/login commands above.
155
+ Confirm the expected account with `whoami` and the available credit with
156
+ `balance`. ThermAI will guide the first partner submission under the agreed limits.
157
+
158
+ For a **local-only** readiness check, save the following as `readiness.sbatch`
159
+ in a new empty directory. Run the dry run from that directory; no account,
160
+ upload, API request, or paid resource is needed:
161
+
162
+ <!-- flightdeck-cli-readiness-script:start -->
163
+
164
+ ```bash
165
+ #!/bin/bash
166
+ #SBATCH --array=1-2
167
+ #SBATCH --gres=gpu:A10:1
168
+ #SBATCH --time=00:01:00
169
+ #SBATCH --output=%A_%a.out
170
+ echo smoke
171
+ ```
172
+
173
+ <!-- flightdeck-cli-readiness-script:end -->
174
+
175
+ ```sh
176
+ flightdeck --version
177
+ flightdeck sbatch readiness.sbatch --dry-run
178
+ ```
179
+
180
+ Expect A10 x1, 60 seconds and **two flights would be submitted**, with no
181
+ warnings. Keep `--dry-run`: this is a translation exercise, not the approved
182
+ pilot workload or permission to launch it. A dry run does not check provider
183
+ capacity, account policy/credit, bundle contents, image access or GPU execution.
184
+
185
+ For the real workload, obtain the reviewed private bundle, exact image
186
+ digest, provider selection and runtime from the operator. Review its dry run,
187
+ then use the supplied submission command only after execution approval. Do not
188
+ copy internal test data into public packages or partner bundles. Use explicit
189
+ `--files` selections from a clean directory; the submission default includes
190
+ the current directory, less the default excludes listed under
191
+ [What goes into a bundle](#what-goes-into-a-bundle). Never include credentials
192
+ or private workstation files.
47
193
 
48
194
  ## Workload commands
49
195
 
50
- After internal qualification and explicit policy/credit approval:
196
+ After your account's policy/credit setup and agreed first-run approval:
51
197
 
52
198
  ```sh
199
+ flightdeck image check <image> # before using --base-image
53
200
  flightdeck sbatch workload.sbatch
54
201
  flightdeck status <flight-id>
55
202
  flightdeck logs <flight-id> --follow
56
203
  flightdeck outputs <flight-id>
57
204
  flightdeck download <flight-id> --output-dir ./results
205
+ flightdeck delete-output <flight-id> <output-id>
58
206
  flightdeck cancel <flight-id>
59
207
  ```
60
208
 
61
209
  Submission, including `flightdeck slurm-test`, can provision paid resources.
210
+ `submit` and `sbatch` print the deployment's output size limit first. Everything
211
+ in `/outputs` is stored as one compressed archive, and an archive over the limit
212
+ is not stored at all. `sbatch --dry-run` stays offline and states the limit
213
+ without a figure.
62
214
  `sbatch --dry-run` only translates locally and does not submit. Flightdeck
63
215
  supports a subset of single-node Slurm directives, not an arbitrary HPC cluster.
64
- Workload images must satisfy the Flightdeck runner contract.
216
+ Without `--base-image`, a flight runs on the official Flightdeck base
217
+ (`ghcr.io/thermai-group/flightdeck-base`): Ubuntu 24.04 with Python 3 and pip,
218
+ and no PyTorch or CUDA, so supply Python packages with `--requirements` or use
219
+ your own image. A `--base-image` can be any linux/amd64 image that meets the
220
+ workload runtime contract (see `/docs/images`); it does not need the
221
+ Flightdeck runner. Before charging, submission refuses an image whose
222
+ `ENTRYPOINT` is shell form
223
+ (`shell_form_entrypoint`) or whose `uk.thermai.flightdeck.contract` label names
224
+ a contract version Flightdeck does not run (`image_unsupported_contract`).
225
+ Check an image locally first with `flightdeck image check`; see
226
+ [Checking an image before you submit](#checking-an-image-before-you-submit).
227
+ Environment variables (`--env`, `#SBATCH --export`) are checked before upload:
228
+ names are at most 64 characters, and names starting with `FLIGHTDECK_` belong
229
+ to the platform and are refused.
230
+
231
+ An array prints a separate Flightdeck ID for each accepted task. Record them
232
+ all: inspect, download and cancel each flight individually (`flightdeck atc
233
+ <job-id>` shows them together). Partial submission
234
+ can leave accepted flights running even if the command exits unsuccessfully;
235
+ do not blindly resubmit the whole array. Downloads go to
236
+ `<output-dir>/<flight-id>/attempt-<n>/`, or `legacy/` for results from before
237
+ attempts were recorded, keeping each output's relative path. `--attempt <n>`
238
+ selects an earlier attempt. An interrupted download resumes when you run the same
239
+ command again, and a file only appears once its size and SHA-256 match the stored
240
+ output. Partial data is kept in `<file>.part` until then. A `.part` file that
241
+ the CLI did not leave for that output is refused, never deleted. A matching file is
242
+ skipped and a different one is never overwritten. A second download of the same
243
+ file stops with an error while the first is running. Stored results have no expiry
244
+ today. `delete-output` removes one output, by the ID `outputs` prints, from the
245
+ outputs list and from storage; the stored copy is kept for 30 days before it is
246
+ removed. It needs the `jobs:cancel` scope and only works once the flight has
247
+ finished and, for a cancelled flight, its outputs have been collected. If storage
248
+ does not confirm the delete, the output stays listed and may or may not have been
249
+ removed from storage; run the same command again to finish the delete. `outputs` and `download` print a warning on stderr when an output was
250
+ over the size limit or collection ended early, including a cancelled flight
251
+ whose outputs may not have been collected, so a short list isn't mistaken for
252
+ everything. With `--attempt`, only that attempt's size refusals are reported. Check terminal status and exit code as well as the expected artifacts;
253
+ successful submission or download alone does not mean the workload succeeded.
254
+
255
+ For an incident, stop submitting, cancel the known active flights, and contact
256
+ your named ThermAI operator. Logging out does **not** stop jobs. The operator
257
+ must confirm provider teardown and reconcile any outstanding charges before a
258
+ retry. Support contact, supported hours and credit increases come from your
259
+ agreed pilot handoff; this client does not promise unattended support.
260
+
261
+ ### Saving, downloading and deleting results
262
+
263
+ Anything a flight writes under `/outputs` (also `$OUTPUTS_DIR`) is saved when the
264
+ flight ends and can be downloaded after the GPU instance has gone. Files anywhere
265
+ else are not kept. This walk-through saves one file, lists it, downloads it,
266
+ checks it and deletes it. The output shown is from a run on staging on
267
+ 14 November 2026. `delete-output` needs CLI `0.1.0-pilot.10` or later. On macOS
268
+ use `shasum -a 256` wherever this walk-through says `sha256sum`.
269
+
270
+ **1. Write to `/outputs`.** Save this as `results-demo.sbatch` in a new empty
271
+ directory:
272
+
273
+ ```bash
274
+ #!/bin/bash
275
+ #SBATCH --gres=gpu:A10:1
276
+ #SBATCH --time=00:05:00
277
+ #SBATCH --output=%j.out
278
+ echo "report $(date -u +%FT%TZ)" > "$OUTPUTS_DIR/report.txt"
279
+ sha256sum "$OUTPUTS_DIR/report.txt"
280
+ ```
281
+
282
+ ```sh
283
+ flightdeck sbatch results-demo.sbatch
284
+ flightdeck status <flight-id> # wait for completed
285
+ flightdeck logs <flight-id> # shows the checksum the workload printed
286
+ ```
287
+
288
+ `sbatch` prints the flight ID, and `logs` prints the workload's line:
289
+
290
+ ```text
291
+ sbatch-job: fl-00e67a3d3ed2ae2229952634 (queue #1)
292
+
293
+ 1 submitted, 0 failed.
294
+ Track with: flightdeck status fl-00e67a3d3ed2ae2229952634
295
+ ```
296
+
297
+ ```text
298
+ 6bc1d843833bc55a3f66ec342533ed7586b757d0f4f7c87c211ddcde597f3bba /outputs/report.txt
299
+ ```
300
+
301
+ **2. List what was saved.** On Lambda, everything in `/outputs` is saved as one
302
+ archive, `outputs.tar.gz`. Each line is the output ID, its size and its name:
303
+
304
+ ```sh
305
+ flightdeck outputs <flight-id>
306
+ ```
307
+
308
+ ```text
309
+ 59925a74-7a23-46e8-8bb0-26c7be1eddb0 274 bytes outputs.tar.gz
310
+ ```
311
+
312
+ A flight that is still running, or a cancelled flight whose outputs are still
313
+ being collected, prints `No outputs available.` An output is listed only once
314
+ it is stored.
315
+
316
+ **3. Download and check it.** The file lands in
317
+ `<output-dir>/<flight-id>/attempt-<n>/`. The CLI checks its size and SHA-256
318
+ against the stored output before the file appears, so a file that is there is
319
+ complete.
320
+
321
+ ```sh
322
+ flightdeck download <flight-id> --output-dir ./results
323
+ cd ./results/<flight-id>/attempt-1
324
+ tar -tzf outputs.tar.gz # look before you extract
325
+ tar -xzf outputs.tar.gz
326
+ sha256sum report.txt # compare with the line in the logs
327
+ ```
328
+
329
+ ```text
330
+ Downloaded /home/you/results/fl-00e67a3d3ed2ae2229952634/attempt-1/outputs.tar.gz
331
+ ./
332
+ ./1794696621749957.out
333
+ ./report.txt
334
+ 6bc1d843833bc55a3f66ec342533ed7586b757d0f4f7c87c211ddcde597f3bba report.txt
335
+ ```
336
+
337
+ The archive also holds the `#SBATCH --output` file, here `1794696621749957.out`.
338
+
339
+ Running the same `download` again prints `Already downloaded …` and changes
340
+ nothing.
341
+
342
+ **4. Delete it.** Pass the output ID from step 2. This needs a login with the
343
+ `jobs:cancel` scope.
344
+
345
+ ```sh
346
+ flightdeck delete-output <flight-id> <output-id>
347
+ ```
348
+
349
+ ```text
350
+ Deleted output 59925a74-7a23-46e8-8bb0-26c7be1eddb0 from fl-00e67a3d3ed2ae2229952634
351
+ ```
352
+
353
+ The output leaves the list at once, and `flightdeck outputs` then prints
354
+ `No outputs available.` The stored copy is removed 30 days later.
355
+
356
+ **The same steps through the API.** Every request needs your credential and the
357
+ accounting header. The credential is `api_key` in `~/.flightdeck/config.json`.
358
+
359
+ ```sh
360
+ API=https://api.flightdeck.thermai.uk/api/v1
361
+ AUTH=(-H "Authorization: Bearer $FLIGHTDECK_API_KEY" -H "X-Flightdeck-Accounting: credits-v1")
362
+
363
+ # List: artifact_id, filename, size_bytes, checksum_sha256, media_type, created_at
364
+ curl -s "${AUTH[@]}" "$API/flights/<flight-id>/outputs"
365
+
366
+ # Download: the API answers with a redirect to a signed URL that works for 5 minutes
367
+ curl -sL "${AUTH[@]}" -o outputs.tar.gz "$API/flights/<flight-id>/outputs/<artifact-id>"
368
+ sha256sum outputs.tar.gz # must equal checksum_sha256 from the list
369
+
370
+ # Delete: 204 on success
371
+ curl -s -o /dev/null -w '%{http_code}\n' -X DELETE "${AUTH[@]}" \
372
+ "$API/flights/<flight-id>/outputs/<artifact-id>"
373
+
374
+ # What happened to the flight, including collection
375
+ curl -s "${AUTH[@]}" "$API/flights/<flight-id>/events"
376
+ ```
377
+
378
+ The list answers like this, and the downloaded file's SHA-256 equalled
379
+ `checksum_sha256`:
380
+
381
+ ```json
382
+ {
383
+ "outputs": [
384
+ {
385
+ "artifact_id": "59925a74-7a23-46e8-8bb0-26c7be1eddb0",
386
+ "filename": "outputs.tar.gz",
387
+ "size_bytes": 274,
388
+ "checksum_sha256": "0f88381b16b970004118af20db4877aa891d0b028638f336a1ffcba98e4f8b11",
389
+ "media_type": "application/gzip",
390
+ "created_at": "2026-11-14T22:56:09.464Z"
391
+ }
392
+ ],
393
+ "attempt_seq": 1,
394
+ "available_attempts": [1]
395
+ }
396
+ ```
397
+
398
+ The events for the same flight, with the delete recorded last:
399
+
400
+ ```text
401
+ 22:50:22.613Z state_changed Flight queued
402
+ 22:55:47.958Z state_changed Flight dispatched
403
+ 22:55:54.311Z state_changed Collecting logs and outputs
404
+ 22:56:09.484Z state_changed Flight completed
405
+ 22:57:34.672Z output_deleted An output was deleted at the customer's request.
406
+ ```
407
+
408
+ The list also returns `attempt_seq` and `available_attempts`; add `?attempt=<n>`
409
+ for an earlier attempt. `curl -L` does not send your credential to the signed
410
+ URL, and nor should your own client.
411
+
412
+ | Delete answer | Meaning |
413
+ | --- | --- |
414
+ | `204` | Deleted. |
415
+ | `404 not_found` | No such output, or the flight is not yours. |
416
+ | `409 flight_not_finished` | The flight is still running. Nothing was deleted. |
417
+ | `409 outputs_being_collected` | A cancelled flight is still saving its outputs. Try again in a few minutes. |
418
+
419
+ **When outputs are missing.** `outputs` and `download` print a warning when
420
+ something was not saved. The flight's events say why:
421
+
422
+ | Event | What happened | What to do |
423
+ | --- | --- | --- |
424
+ | `outputs_too_large` | The archive was over the size limit (5 GiB unless your operator says otherwise) and was not stored. | Write less to `/outputs` and run again. |
425
+ | `finalization_abandoned` | Saving failed five times and was given up. | Run again. A flight whose workload exited 0 is not charged; one whose workload failed is charged like any failed flight. |
426
+ | `deadline_collection_abandoned` | The flight hit its time limit and saving did not finish in the time left. | Raise `--time`, or write results earlier, and run again. |
427
+ | `cancel_stop_failed` | The flight was cancelled but its workload could not be confirmed stopped, so its outputs may not have been saved. | Check `outputs`. Run again if they are missing. |
428
+
429
+ **Known limits.**
430
+
431
+ - **No retry on request.** Saving is retried automatically, up to five times,
432
+ while the instance exists. After that the only way to get the results is to
433
+ run the flight again.
434
+ - **Outputs that were not saved are gone.** The instance is destroyed when
435
+ saving is given up, and anything that was only on it cannot be recovered.
436
+ - **A cancelled flight gives no sign while it saves.** Its events stop at
437
+ "Flight cancelled". Outputs appear in `outputs` when they are stored, usually
438
+ within a minute. If saving is still going after an hour the instance is
439
+ destroyed, with no outputs and no event to say so.
440
+ - **Results do not expire yet.** A saved output stays until you delete it. A
441
+ 30-day purge after a pilot ends is planned.
442
+ - **One archive per flight on Lambda.** You cannot download a single file from
443
+ inside `outputs.tar.gz`, or delete one.
444
+ - **Only your own flights.** Another account, in your organisation or outside
445
+ it, gets `404 not_found` for your flight's outputs.
446
+
447
+ ### What goes into a bundle
448
+
449
+ `submit` and `sbatch` pack the current directory, or the `--files` paths, into
450
+ the bundle that is unpacked on the node. These are left out by default:
451
+
452
+ - version control: `.git`, `.hg`, `.svn`, `.bzr` (directories or files)
453
+ - environment and credential files: `.env*`, `.netrc`, `.pypirc`, `.npmrc`,
454
+ `.git-credentials`, `.ssh/`, `.aws/`, `.gnupg/`, `.docker/`, `.kube/`,
455
+ `.flightdeck/`
456
+ - key files: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.ppk`, `id_rsa`, `id_dsa`,
457
+ `id_ecdsa`, `id_ed25519`, `id_ecdsa_sk`, `id_ed25519_sk`
458
+ - virtualenvs: `venv/`, `.venv/` and any directory containing `pyvenv.cfg`
459
+
460
+ These match regardless of case, so `.ENV` and `ID_RSA` are left out too.
461
+
462
+ A `.flightdeckignore` in the current directory adds rules in `.gitignore`
463
+ syntax, and `!<path>` includes something a default would leave out. Only a `!`
464
+ rule without wildcards lifts a default; one such as `!*/` re-includes only what
465
+ your own rules excluded:
466
+
467
+ ```gitignore
468
+ checkpoints/*
469
+ !checkpoints/final.pt
470
+ !.env.example
471
+ ```
472
+
473
+ As with `.gitignore`, a path under an excluded directory cannot be re-included
474
+ on its own; re-include the directory, or exclude its contents with `dir/*`
475
+ instead. Naming an excluded path with `--files` does not include it (a shell
476
+ glob such as `--files *` expands to every name); use `!<path>` in
477
+ `.flightdeckignore`. A `--files` path that goes through a symlinked directory
478
+ (`alias/credentials` with `alias -> .aws`) is checked under the path it resolves
479
+ to as well, so the `!<path>` must name the resolved path; one that resolves
480
+ outside the current directory is refused. Symlinks that dangle, whose target is not in the bundle
481
+ (such as `data -> /scratch/...`), or whose target is an absolute path are
482
+ skipped with a warning because they cannot resolve on the node; use relative
483
+ links within the directory. To use a dataset that lives outside it, copy it
484
+ in or stage it on the node another way. The file count and size print before upload, and
485
+ `sbatch --dry-run` shows the same summary.
486
+
487
+ ### Checking an image before you submit
488
+
489
+ `flightdeck image check <image>` checks an image with your local Docker, so you
490
+ learn before you submit whether Flightdeck is likely to admit it and how it
491
+ behaves. It
492
+ needs no login. When you are logged in, it reads the platform's current runtime
493
+ facts (the supervisor release and the host drivers) from the API; otherwise it
494
+ uses the ones built into the CLI.
495
+
496
+ ```sh
497
+ flightdeck image check ghcr.io/your-org/your-image:1.0
498
+ flightdeck image check ghcr.io/your-org/your-image:1.0 --json
499
+ ```
500
+
501
+ It reports each check as PASS, WARN, FAIL, INFO, SKIP or ERROR (the check
502
+ could not run), with a one-line reason, and ends with a verdict:
503
+
504
+ - **Submission rules**, the same code the API uses: `linux/amd64`, a docker.io,
505
+ ghcr.io or nvcr.io reference within 255 characters once pinned to its digest,
506
+ and the `uk.thermai.flightdeck.contract` label. If Docker's copy is for
507
+ another platform (Docker on Apple silicon pulls `arm64`), the check asks the
508
+ registry whether it has the `linux/amd64` image Flightdeck runs: if it does,
509
+ that is a warning and the dry run is skipped; if not, it fails. The check
510
+ never re-pulls or retags your copy. A shell-form `ENTRYPOINT` is a
511
+ warning: Flightdeck refuses it for flights with a command, and runs it as it
512
+ is for image-only flights. `tini` or `dumb-init` as the `ENTRYPOINT` is a
513
+ warning too, because the supervisor already does their job.
514
+ - **Registry**: Flightdeck reads the image from its registry, without
515
+ credentials. A local image that is not in its registry yet is a warning, and
516
+ so is one the registry will not serve without credentials (the check asks it
517
+ anonymously, with `docker manifest inspect` and an empty Docker
518
+ configuration). New GitHub packages are private until you make them public.
519
+ - **Dry run**: the image starts the way a flight starts, under the Flightdeck
520
+ supervisor with `/workspace` and `/outputs` mounted, CPU only unless you pass
521
+ `--gpu`. `--gpu` attaches GPUs with Docker's `--gpus`; Flightdeck's hosts use
522
+ CDI, so the dry run does not test a provider's GPUs or driver. It runs a command as the image's `USER`; an `ENTRYPOINT` that ignores
523
+ the command fails. It installs a requirements file with no packages in it; a
524
+ missing or unusable `pip` is a warning, since only a bundle with a
525
+ requirements file needs one. It stops a running command with `SIGTERM` and
526
+ expects it to exit within the 30-second grace. A missing `bash` is a warning,
527
+ because commands then run under `sh` and sbatch scripts need `bash`.
528
+ - **CUDA**, from `CUDA_VERSION`, `libcudart`, pip's CUDA runtime wheel and
529
+ `torch.version.cuda`: a CUDA newer than every published host driver runs is a
530
+ warning. An image without CUDA is fine. `flightdeck submit` and
531
+ `flightdeck sbatch` also warn, after the flight is accepted, when the image
532
+ sets a `CUDA_VERSION` newer than the lowest published host driver runs; the
533
+ other sources are read only here.
534
+ - **Secrets**: the image's file names are listed through `docker export`, without
535
+ extracting anything, and matched against the credential and key patterns in
536
+ [What goes into a bundle](#what-goes-into-a-bundle). Empty files, CA
537
+ certificates and the test keys that packages ship are skipped.
538
+
539
+ The exit status is 0 when nothing failed (warnings allowed), 1 when a check
540
+ failed, and 2 when the check itself could not run, for example without Docker
541
+ or when the pull failed. `--static-only` runs nothing from the image, and
542
+ `--supervisor <image>` runs the dry run under another supervisor image.
543
+
544
+ The verdict says what the result means for submission:
545
+
546
+ - **PASS**, with every check run: Flightdeck will admit the image, and it ran
547
+ commands as the workload runtime contract describes.
548
+ - **PASS for the checks that ran**: nothing failed, but the verdict lists what
549
+ was not checked, such as the dry run under `--static-only`.
550
+ - **WARN, "Flightdeck may refuse this image"**: a warning that admission acts
551
+ on, named in the verdict. The image is not in its registry yet, the registry
552
+ will not serve it without credentials, or its `ENTRYPOINT` is shell form,
553
+ which is refused for every flight with a command.
554
+ - **WARN** otherwise: Flightdeck will admit the image; read the warnings.
555
+ - **FAIL**: Flightdeck would refuse the image, or its flights would not run.
556
+
557
+ If Docker does not have the image, the check pulls it for `linux/amd64` and
558
+ leaves it for you; it never removes an image you already had. Everything else
559
+ it creates is removed when it finishes, fails or is interrupted with Ctrl-C:
560
+ its containers (named `flightdeck-image-check-<id>-…` and labelled
561
+ `uk.thermai.flightdeck.image-check`), its temporary directory, and the
562
+ supervisor image if the check pulled it (unless another image check is
563
+ running, which may still need it). After Ctrl-C the check waits for Docker to
564
+ remove its containers, up to a minute each, and a second Ctrl-C does not cut
565
+ that short. If the check is killed outright, list what it left with
566
+ `docker ps -a --filter label=uk.thermai.flightdeck.image-check`, remove each with
567
+ `docker rm -f <name>`, and delete any `flightdeck-image-check-*` directory in
568
+ your temporary directory. On Apple silicon and other machines that are not
569
+ amd64, the image runs under emulation, which is slow.
570
+
571
+ The check does not test your code or the provider's GPUs, so run a short flight
572
+ before real work.
573
+
574
+ ### Publishing a private image
575
+
576
+ `flightdeck image push <local-image> --name <name> --version <label>` publishes
577
+ a local image to your organization's private registry. `image list` and
578
+ `image inspect <version-id> --format reference` show it and print the reference
579
+ to pass to `--base-image`. Log in with `flightdeck login --images` first (run
580
+ `logout` before it if you are already logged in). Rerun the same push to resume.
581
+ Managed images run on Lambda only, so add `--provider lambda` to `submit`.
582
+ `--base-image managed://<name>:latest` runs the newest ready version, and
583
+ `image push --alias ci` or `image alias set <name> ci --version <label>` moves a channel
584
+ you can run as `managed://<name>:ci`. The full guide, including errors and limits, is in
585
+ [`docs/managed-images.md`](../docs/managed-images.md).
586
+
587
+ 1. `flightdeck logout`, then `flightdeck login --images`.
588
+ 2. `flightdeck image push solver-local:v1 --name solver --version v1`.
589
+ 3. `flightdeck image inspect <version-id>` shows the reference and the scan state.
590
+
591
+ Messages you may see:
592
+
593
+ - A 401 prints "Your FlightDeck login is no longer valid. The credential expired
594
+ or was revoked. Run flightdeck logout, then flightdeck login --images."
595
+ - A login made without `--images` prints "This login cannot use managed images.
596
+ Run flightdeck logout, then flightdeck login --images."
597
+ - An image above the registry's size limit, when one is set, is rejected after
598
+ the push and is not added to your list. Push a smaller image with a new
599
+ `--version`.
600
+ - A push that is not a single linux/amd64 image is rejected and names the
601
+ platform it found.
602
+
603
+ `image inspect` and `image list` show the registry's scan of the image, and
604
+ `image push` points to `image inspect` for it. Findings are advisory. They never block a push or a flight.
605
+
606
+ ### Datasets: upload once, use in many flights
607
+
608
+ Managed storage keeps a directory of files in your organisation's space. You
609
+ upload it once as a version that never changes, then mount it read-only in any
610
+ Lambda flight. ThermAI must turn storage on for your organisation first. An
611
+ organisation admin uploads and deletes; any active member can list, download
612
+ and mount. This walk-through uploads a small dataset, uses it in a flight,
613
+ downloads it and deletes it. The output shown is from a run on staging on
614
+ 14 November 2026. `dataset download` needs CLI `0.1.0-pilot.10` or later.
615
+
616
+ **1. Log in with storage access.** `login` refuses a second sign-in, so log out
617
+ first.
618
+
619
+ ```sh
620
+ flightdeck logout
621
+ flightdeck login --storage
622
+ ```
623
+
624
+ **2. Upload a directory.** Note the hash, so you can check it later. On macOS
625
+ use `shasum -a 256` wherever this walk-through says `sha256sum`.
626
+
627
+ ```sh
628
+ mkdir corpus && head -c 1048576 /dev/urandom > corpus/sample.bin
629
+ sha256sum corpus/sample.bin
630
+ flightdeck dataset push ./corpus --name corpus --version 2026-11
631
+ ```
632
+
633
+ ```text
634
+ f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 corpus/sample.bin
635
+ Uploading corpus:2026-11: 1 files, 1.0 MiB
636
+ [1/1] sample.bin (1.0 MiB)
637
+ Verifying the upload
638
+ Published corpus:2026-11
639
+ Version ID: 33d54875-8fdd-4a25-a4d0-17e4a7449229
640
+ Files: 1, size: 1.0 MiB
641
+ Use it with: flightdeck submit "<command>" --dataset corpus:2026-11
642
+ ```
643
+
644
+ If the upload is interrupted (Ctrl-C, a dropped connection, a closed laptop),
645
+ run the same command again. It sends only the parts that are missing. A 400 MiB
646
+ push stopped with Ctrl-C after ten seconds, then run again, printed:
647
+
648
+ ```text
649
+ Interrupted. Stopping the upload.
650
+ The push was interrupted and did not finish. Run the same command again to resume.
651
+ ```
652
+
653
+ ```text
654
+ Uploading resume-check:v1: 1 files, 400.0 MiB
655
+ [1/1] big.bin (400.0 MiB)
656
+ resumed: 1 parts were already uploaded
657
+ Verifying the upload
658
+ Published resume-check:v1
659
+ ```
660
+
661
+ Between the two, `dataset list` shows the version as `reserved`, and a flight
662
+ cannot use it yet. A push
663
+ of a version that is already complete prints `Already published corpus:2026-11`
664
+ in place of the upload lines and uploads nothing. A version never changes: to upload different files, choose a new
665
+ `--version`.
666
+
667
+ **3. List it.**
668
+
669
+ ```sh
670
+ flightdeck dataset list
671
+ ```
672
+
673
+ ```text
674
+ NAME VERSION STATE SIZE FILES CREATED VERSION ID
675
+ corpus 2026-11 ready 1.0 MiB 1 2026-11-14 33d54875-8fdd-4a25-a4d0-17e4a7449229
676
+ ```
677
+
678
+ Only a `ready` version can be mounted. `--json` gives the same list for
679
+ scripts.
680
+
681
+ **4. Use it in a flight.** Give `--dataset` as `<name>:<version>`, or as the
682
+ version ID. Add `:<mount>` to choose the directory name. Repeat the flag for up
683
+ to four datasets.
684
+
685
+ `submit` packs the current directory into the flight's bundle, so run it from a
686
+ directory that holds only the job's files. Do not run it from the directory
687
+ that holds `corpus/`, or the dataset is packed into the bundle as well. An
688
+ empty directory is refused with `Nothing to bundle`.
689
+
690
+ ```sh
691
+ mkdir job && cd job
692
+ echo 'sha256sum /data/corpus/sample.bin' > check.sh
693
+ flightdeck submit "bash check.sh" --provider lambda \
694
+ --dataset corpus:2026-11 --time 5m
695
+ flightdeck status <flight-id> # a Data: line shows the copy's progress
696
+ flightdeck logs <flight-id> # the hash must equal the one from step 2
697
+ cd ..
698
+ ```
699
+
700
+ ```text
701
+ Flight submitted successfully!
702
+ Flight ID: fl-27aec32986c7f001885230f2
703
+ Status: queued
704
+ Est. credits: 14
705
+ Queue position: 1
706
+ ```
707
+
708
+ While the flight is `dispatching`, `status` shows the copy, and `logs` shows
709
+ the same hash as step 2 once it has run:
710
+
711
+ ```text
712
+ Data: Datasets fetched: 1.0 MiB in 1 file
713
+ ```
714
+
715
+ ```text
716
+ f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 /data/corpus/sample.bin
717
+ ```
718
+
719
+ A second flight submitted with the version ID in place of the name
720
+ (`--dataset 33d54875-8fdd-4a25-a4d0-17e4a7449229`) printed the same line.
721
+
722
+ The flight reads the files at `/data/<mount>/`, read-only. `FLIGHTDECK_DATA_DIR`
723
+ is `/data`. Flightdeck copies the files to the machine before your container
724
+ starts, so reads are at local disk speed, and the flight stays `dispatching`
725
+ while it copies. The version is fixed when you submit: a flight always runs the
726
+ data it named, whatever is uploaded later. A second flight with the same
727
+ `--dataset` reuses the same upload.
728
+
729
+ **5. Download it, then delete it.**
730
+
731
+ ```sh
732
+ flightdeck dataset download corpus:2026-11 ./corpus-copy
733
+ sha256sum ./corpus-copy/sample.bin
734
+ flightdeck dataset rm corpus:2026-11
735
+ ```
736
+
737
+ ```text
738
+ Downloaded 1 file(s), 1.0 MiB, of corpus:2026-11 to /home/you/corpus-copy.
739
+ f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 ./corpus-copy/sample.bin
740
+ Deleted corpus:2026-11. New flights cannot use it. Its files are removed after the retention period.
741
+ ```
742
+
743
+ After the delete, the version is gone from `dataset list`, and a submit that
744
+ names it is refused before anything launches:
745
+
746
+ ```text
747
+ Submit failed (404): A dataset in --dataset was not found in your organization.
748
+ ```
749
+
750
+ `download` checks every file against the checksum recorded at upload. Run it
751
+ again to resume; it reports files it already has as `already present and
752
+ verified` and never overwrites a file that differs. `rm` is refused while a
753
+ queued, dispatching or running flight uses the version. Deleted files are
754
+ removed from storage 7 days later.
755
+
756
+ **The same steps through the API.** The routes are under `/api/v1/storage`.
757
+ Reading needs the `storage:read` scope and changing needs `storage:write`,
758
+ which a `login --storage` credential has. The credential is `api_key` in
759
+ `~/.flightdeck/config.json`.
760
+
761
+ ```sh
762
+ API=https://api.flightdeck.thermai.uk/api/v1
763
+ AUTH=(-H "Authorization: Bearer $FLIGHTDECK_API_KEY" -H "X-Flightdeck-Accounting: credits-v1")
764
+ JSON=(-H "Content-Type: application/json")
765
+
766
+ curl -s "${AUTH[@]}" "$API/storage" # enabled, entitled, usage, quota
767
+ curl -s "${AUTH[@]}" "$API/storage/datasets" # datasets
768
+ curl -s "${AUTH[@]}" "$API/storage/datasets/<dataset-id>/versions" # versions and their IDs
769
+ curl -s "${AUTH[@]}" "$API/storage/versions/<version-id>/files" # files, sizes and checksums
770
+
771
+ # Use it in a flight: add datasets to the body of POST /flights
772
+ # "datasets": [{"ref": "corpus:2026-11"}, {"ref": "<version-id>", "mount": "weights"}]
773
+
774
+ # Download: signed links for up to 100 files, each valid for one hour
775
+ curl -s "${AUTH[@]}" "${JSON[@]}" -X POST -d '{"paths":["sample.bin"]}' \
776
+ "$API/storage/versions/<version-id>/download-urls"
777
+
778
+ # Delete
779
+ curl -s "${AUTH[@]}" "${JSON[@]}" -X POST -d '{}' "$API/storage/versions/<version-id>/delete"
780
+ ```
781
+
782
+ The files call returns each file's path, size and checksum, and the delete
783
+ call answers with the version's new state:
784
+
785
+ ```json
786
+ {"files":[{"path":"sample.bin","size":1048576,"checksum":"SWvdRr7il3MHAC6btaOHPbHcf+lGzGNqWNpClfFNWfk=-1"}],"next_cursor":null}
787
+ {"id":"8e935b05-ac09-43a4-8ce9-1626ad587c45","state":"deleted","changed":true}
788
+ ```
789
+
790
+ The download call returns one entry per file with the same fields plus
791
+ `part_size`, a signed `url` and `expires_at`.
792
+
793
+ Uploading through the API takes several calls per file (open a session, sign
794
+ each part, complete each file, then verify). Use `dataset push`, which also
795
+ resumes; the calls are described in `docs/managed-storage.md` if you need them.
796
+
797
+ **When something is refused.** Each refusal has a code, and nothing is
798
+ charged. The CLI prints a sentence for most of them. At submit,
799
+ `credential_scope_required` (a login without `--storage`) and
800
+ `dataset_mount_invalid` are printed as raw JSON.
801
+
802
+ | Code | When | What to do |
803
+ | --- | --- | --- |
804
+ | `storage_not_entitled` | Storage is not turned on for your organisation. | Ask ThermAI. |
805
+ | `credential_scope_required` | Your login has no storage access. | `flightdeck logout`, then `flightdeck login --storage`. |
806
+ | `org_admin_required` | A member who is not an admin tried to upload or delete. | Ask an organisation admin. |
807
+ | `storage_quota_exceeded` | The upload does not fit in your organisation's space. | Delete a version with `dataset rm`, or ask ThermAI. Space is freed when the deleted files are removed. |
808
+ | `dataset_limit_reached`, `version_limit_reached` | Too many datasets, or too many versions of one. | Delete one, or ask ThermAI. |
809
+ | `file_too_large` | One file is over the largest file size. | Split the file. |
810
+ | `version_exists` | That `--version` already exists for the dataset, from any upload. A deleted version's label stays reserved for good. | Choose a new `--version`. |
811
+ | `idempotency_conflict` | The same `--version` was pushed again with different files. | Choose a new `--version`. A version never changes. |
812
+ | `storage_fetch_quota_exceeded` | Your organisation reached its daily download limit, on a download or on a submit. | Wait. The limit counts the last 24 hours. Files already downloaded are kept. |
813
+ | `dataset_not_found` | The dataset is not in your organisation, or the version was deleted. | Check `dataset list`. |
814
+ | `dataset_not_ready` | The version is still uploading, or was disabled. | Check its state in `dataset list`. |
815
+ | `dataset_limit_exceeded` | The datasets are over the per-flight size or file limit, or do not fit the machine's disk. | Mount less. |
816
+ | `version_in_use` | `rm` on a version a queued, dispatching or running flight uses. | Wait for those flights to end. |
817
+
818
+ Four of these were produced on staging on 14 November 2026:
819
+ `dataset_not_found`, `dataset_not_ready`, `version_in_use` and
820
+ `idempotency_conflict`. The CLI printed:
821
+
822
+ ```text
823
+ A dataset in --dataset was not found in your organization.
824
+ A dataset in --dataset is not ready. Check flightdeck dataset list.
825
+ corpus:2026-11b is in use by a queued, dispatching or running flight. Try again when those flights end.
826
+ A different upload is already recorded for corpus:2026-11b. If the directory changed since the first push, choose a new --version.
827
+ ```
828
+
829
+ **When a flight cannot get its data.** The copy happens before your container
830
+ starts, so a flight that fails here is not charged. `flightdeck status` shows
831
+ the code.
832
+
833
+ | Code | What happened | What to do |
834
+ | --- | --- | --- |
835
+ | `data_fetch_failed` | The copy failed or was too slow. Flightdeck tries again on a new machine, up to three attempts. | Nothing. Watch `status`. |
836
+ | `data_fetch_exhausted` | The copy failed the same way twice, or the daily download limit ran out during it. | Submit again later. If it repeats, send ThermAI the flight ID. |
837
+ | `data_unavailable` | The version was disabled or deleted after you submitted, or storage was turned off for your organisation. | Check `dataset list`. |
838
+ | `data_too_large` | The data does not fit the machine's disk. | Mount less. |
839
+
840
+ **Known limits.**
841
+
842
+ - **Pilot limits:** 1 TiB and 50 datasets per organisation, 50 GiB for the
843
+ largest file, 50 GiB mounted per flight, and 150 GiB of downloads per
844
+ organisation in any 24 hours.
845
+ - **Downloads share one allowance.** `dataset download` and every flight that
846
+ mounts data count toward the same 150 GiB. Each attempt of a flight counts
847
+ again, so a flight that retries uses its size more than once. A large
848
+ download can leave too little for that day's flights.
849
+ - **Up to 4 datasets per flight**, 5000 files per version and 20 versions per
850
+ dataset.
851
+ - **Lambda only.** A flight on another provider is refused with
852
+ `dataset_provider_unsupported`.
853
+ - **Read-only, and copied every time.** A flight cannot write to `/data`, and
854
+ nothing is cached between flights: each flight copies its data again, which
855
+ adds to its start-up time.
856
+ - **File names.** Up to 8 path segments of letters, digits, `_`, `-` and `.`,
857
+ each up to 128 characters with no leading dot, and 255 characters in all.
858
+ Names with spaces, symbolic links and special files are refused at upload.
859
+ - **No `inspect`, `disable` or `enable` commands yet.** Use the API for those.
860
+
861
+ ### Keeping secrets
862
+
863
+ `flightdeck secret` keeps private credentials, such as a Hugging Face token,
864
+ for your organization's flights. Secrets must be enabled for your organization
865
+ by your ThermAI operator. Any member can list names and details; only an
866
+ organization admin can create, rotate or revoke. Nothing ever prints a value
867
+ back.
868
+
869
+ ```sh
870
+ flightdeck logout
871
+ flightdeck login --secrets
872
+ printf '%s' "$HF_TOKEN" | flightdeck secret create hf-token --value-stdin --description "Hub token"
873
+ flightdeck secret list
874
+ flightdeck secret rotate hf-token --from-file ./new-token.txt --expires-at 2027-06-30
875
+ flightdeck secret revoke hf-token --yes
876
+ ```
877
+
878
+ - A value is never taken from the command line, where it would stay in shell
879
+ history. Pipe it in with `--value-stdin` (a terminal is refused, since the
880
+ value would be shown) or name a regular file with `--from-file`. One trailing
881
+ line break is removed. A value mistyped as an argument or an unknown option
882
+ is refused without being repeated.
883
+ - Names use lowercase letters and digits joined by `.`, `_` or `-`, up to 64
884
+ characters. Values are text up to 32 KiB. An organization holds up to 100.
885
+ - `--expires-at` takes a date or a date and time with a zone. A date alone means
886
+ midnight UTC at the start of that day. On `rotate`, leaving it out keeps the
887
+ current expiry and `--no-expiry` removes it. An expired secret must be rotated
888
+ with one of the two.
889
+ - `revoke` needs `--yes`. A running flight keeps the value it started with, and
890
+ revoking here does not revoke the token where it was issued: revoke it there
891
+ too.
892
+ - `--json` prints the metadata as JSON on every command.
893
+
894
+ The full guide is in [`docs/customer-secrets.md`](../docs/customer-secrets.md).
895
+
896
+ ### Slurm differences
897
+
898
+ - Single-node jobs only; this is not a full Slurm scheduler or an HPC login node.
899
+ - Array `%N` concurrency throttles are not enforced, so `sbatch` refuses one
900
+ smaller than the array. `--ignore-array-limit` submits every task
901
+ immediately. Admission ceilings may reject excess tasks rather than queue
902
+ them for later admission. Review every warning before real submission.
903
+ - In an array task, `%j` is the array job id, not a per-task id; use `%a` in
904
+ `--output`/`--error` names so tasks don't share a file.
905
+ - Array tasks default to `slurm-%A_%a.out`, as in Slurm; write results under
906
+ `$OUTPUTS_DIR`. There is no shared university filesystem or implicit `module`,
907
+ Singularity, nested Docker, or cluster software environment.
908
+ - Hardware, runtime, image and credit must fit the operator-approved policy.
909
+ The image must meet the workload runtime contract: linux/amd64, and an
910
+ exec-form `ENTRYPOINT` if it sets one. Submission refuses an image that does
911
+ not.
912
+
913
+ ### Reading logs completely
914
+
915
+ From `0.1.0-pilot.3` (containing the #99 pagination fix), `flightdeck logs`
916
+ drains all pages in the current stored-log snapshot, then returns. For a running
917
+ job it then reads the live tail, which the backend serves a bounded page at a
918
+ time. `--follow` drains those pages immediately, then polls every two seconds
919
+ until the job is terminal and its stored backlog is exhausted. Every new
920
+ invocation starts at the beginning; stdout/stderr retain their respective
921
+ output streams.
922
+
923
+ Each log stream is capped at 5 MiB. On Lambda, a finished flight's stream that
924
+ passes the cap keeps its first 5 MiB and its last 64 KiB (each cut between
925
+ characters), with a `flightdeck: log truncated` marker line between them when
926
+ anything was left out, so the end of the run and any
927
+ `flightdeck: failure reason` line survive. When a log is truncated,
928
+ `flightdeck logs` prints a one-line warning on stderr after the text; stdout
929
+ carries only the flight's own output.
930
+
931
+ **Rollout order:** deploy and verify the pagination-capable backend before
932
+ releasing/upgrading this CLI. A backend without the `has_more` response field
933
+ produces an explicit compatibility error and exit status 1, not a silently
934
+ partial read. Older `0.1.0-pilot.2` installations do not contain this fix;
935
+ merging source alone neither replaces that immutable package nor publishes a
936
+ new version. Older non-following clients still need upgrading for full paging.
937
+
938
+ ### Watching flights live
939
+
940
+ `flightdeck atc` (alias `flightdeck tui`) opens a read-only, full-screen view in
941
+ the current terminal. Arrow keys select, Enter shows a flight's details and
942
+ events, `l` opens its logs (scroll, `f` to follow), `r` refreshes, Esc goes back
943
+ and `q` quits. It never cancels, submits or downloads anything.
944
+
945
+ ```sh
946
+ flightdeck atc # your recent flights (last 50)
947
+ flightdeck atc <flight-id> # one flight
948
+ flightdeck atc <job-id> # an sbatch array submitted from this machine
949
+ flightdeck atc --name 'molsim*' # flights whose name matches, among your last 200
950
+ flightdeck atc --demo # a simulated array; no account or credits needed
951
+ ```
952
+
953
+ `flightdeck sbatch` records each array's flight IDs under
954
+ `~/.flightdeck/batches/` and ends with `Watch live: flightdeck atc <job-id>`;
955
+ that job ID is the one to pass. It is a local sbatch job ID, not a flight ID:
956
+ `status`, `logs`, `outputs`, `download`, `delete-output` and `cancel` take the `fl-…` flight
957
+ IDs printed for each task. The record exists only on the machine that
958
+ submitted it (if it could not be written, sbatch says so; use `--name`). Name
959
+ patterns are case-sensitive. Leaving the view does not stop any job. In
960
+ scripts, keep using `status` and `logs`.
961
+
962
+ ### Error handling and support
963
+
964
+ The CLI exits with status 1 on command failure. Expected authentication,
965
+ network and HTTP failures have concise diagnostics instead of a Node stack.
966
+ An already-configured login leaves the saved credential unchanged.
967
+
968
+ **Pilot.6 logout behavior (#143):** remove the local file first, then make
969
+ one best-effort server revocation request with a three-second deadline, including
970
+ the response body. Offline, suspended, failed or malformed API responses cannot
971
+ retain or restore the local login. Local success exits 0; a warning separately
972
+ states when remote revocation was not confirmed and other copies may remain
973
+ valid. Use portal credential management or contact your organization admin.
974
+ Failed local deletion exits 1, reports the problem and makes no API request.
975
+ Missing local login is a successful no-op; malformed config is removed without
976
+ trusting its contents for a request. No credential recovery/retry file is saved.
977
+ `logout --local` removes only the local copy and makes no network request.
978
+
979
+ Older pilot.5 does not include this change; use `flightdeck logout --local`
980
+ as a local-removal workaround before upgrading. It does not revoke remote
981
+ access. The current production backend permits narrowly authenticated
982
+ self-revocation under suspension; ordinary access remains blocked.
983
+
984
+ For unexpected faults, `flightdeck --debug <command>` adds allowlisted error
985
+ types/codes and bundled CLI line locations, not raw stacks or error bodies.
986
+ It is not a dry-run flag: only repeat an operation that is safe to repeat.
987
+ Provide support with the CLI/Node versions, exit status, redacted command and
988
+ flight ID if available. Review output before sharing: account details and
989
+ workload logs/results can contain private information. Never send passwords,
990
+ tokens, activation links or `~/.flightdeck/config.json`.
65
991
 
66
992
  ## Versions
67
993
 
@@ -70,8 +996,10 @@ flightdeck --version
70
996
  npm install -g @thermai/flightdeck-cli@<approved-version>
71
997
  ```
72
998
 
73
- Stable releases use npm's `latest` tag. Explicit pilot/prerelease versions use
74
- `next`; they do not automatically replace stable installations. Updating is an
999
+ Stable releases will use npm's `latest` tag. Explicit pilot/prerelease versions use
1000
+ `next`; they do not automatically replace stable installations. Currently
1001
+ `latest` still points at the initial pilot.1 bootstrap, not a qualified stable
1002
+ release; use `next` or the approved exact pilot version. Updating is an
75
1003
  explicit install, not an automatic mutation performed when the CLI runs.
76
1004
  Once a stable release exists, `npm install -g @thermai/flightdeck-cli` selects it.
77
1005