@thermai/flightdeck-cli 0.1.0-pilot.1 → 0.1.0-pilot.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +984 -17
- package/dist/index.js +9226 -921
- package/package.json +8 -3
package/README.md
CHANGED
|
@@ -2,31 +2,126 @@
|
|
|
2
2
|
|
|
3
3
|
Install the command-line client, authorize it through the Flightdeck portal,
|
|
4
4
|
then submit and inspect GPU jobs. No npm account is required to install the
|
|
5
|
-
|
|
5
|
+
public package. A provisioned Flightdeck account is required to use the
|
|
6
6
|
service; downloading the CLI does not create an account or grant compute credit.
|
|
7
7
|
|
|
8
8
|
## Release status
|
|
9
9
|
|
|
10
|
-
The
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
10
|
+
The package is `@thermai/flightdeck-cli`. The first pilot, `0.1.0-pilot.1`, was
|
|
11
|
+
published manually on 5 September 2026 and its anonymous install was verified.
|
|
12
|
+
`0.1.0-pilot.2` subsequently passed GitHub Actions trusted publishing and an
|
|
13
|
+
anonymous install check on the same date. Releases publish through the
|
|
14
|
+
versioned GitHub Release workflow, not merely by merging a version change. The
|
|
15
|
+
installation command below follows the published `next` channel without
|
|
16
|
+
tracking an unpublished candidate version.
|
|
17
|
+
Proprietary terms in `LICENSE` permit redistribution of unchanged official
|
|
18
|
+
releases. Service access still requires an invitation, configured credits and
|
|
19
|
+
the agreed pilot limits; installing the package alone does not grant access.
|
|
20
|
+
|
|
21
|
+
**`0.1.0-pilot.3`** was published through GitHub Actions on 8 September 2026,
|
|
22
|
+
including complete log pagination and concise authentication errors. Its
|
|
23
|
+
anonymous Node 22 install and packaged dry run passed. Use the operator-approved
|
|
24
|
+
exact version for a recorded rehearsal.
|
|
25
|
+
|
|
26
|
+
**`0.1.0-pilot.4`**, published on 20 September 2026, adds customer credits and
|
|
27
|
+
requires a backend advertising `X-Flightdeck-Accounting: credits-v1`. Upgrade
|
|
28
|
+
with the install command below before accessing account/job data. An incompatible
|
|
29
|
+
backend is rejected before job submission; login and credential revocation do not require
|
|
30
|
+
the accounting contract. See the
|
|
31
|
+
operator rollout checklist (`flightdeck-ops/runbooks/customer-credits-rollout.md`).
|
|
32
|
+
|
|
33
|
+
**`0.1.0-pilot.5`**, published on 21 September 2026, fixes result downloads redirected to signed object
|
|
34
|
+
storage ([#157](https://github.com/ThermAI-Group/thermai_flightdeck/issues/157))
|
|
35
|
+
and includes the merged GPU-recovery status display. Downloads validate the
|
|
36
|
+
accounting contract on Flightdeck's response, then fetch the authorized storage
|
|
37
|
+
object without forwarding the CLI credential. Anonymous Node 22 installation
|
|
38
|
+
and a live result download passed. Upgrade from
|
|
39
|
+
pilot.4 before downloading results. Future releases still require an approved
|
|
40
|
+
versioned GitHub Release; merging alone does not publish.
|
|
41
|
+
|
|
42
|
+
**`0.1.0-pilot.6`**, published on 21 September 2026, adds
|
|
43
|
+
offline-first logout and explicit suspension messages. Release CI, registry
|
|
44
|
+
integrity and anonymous Node 22 installation passed; grouped live acceptance
|
|
45
|
+
was confirmed by the operator on 24 September. These are historical pilot.6
|
|
46
|
+
results, not pilot.7 publication evidence. This update does not modify the
|
|
47
|
+
README inside the already-published immutable tarball.
|
|
48
|
+
|
|
49
|
+
**`0.1.0-pilot.7`**, published on 1 October 2026, adds the read-only
|
|
50
|
+
`flightdeck atc` terminal view (alias `tui`), flight details/events/logs, name
|
|
51
|
+
filtering and local sbatch-array tracking from
|
|
52
|
+
[PR #206](https://github.com/ThermAI-Group/thermai_flightdeck/pull/206).
|
|
53
|
+
Downloads/cancellation inside ATC remain deferred to #205; stale
|
|
54
|
+
fallback-status indication is tracked in #207.
|
|
55
|
+
|
|
56
|
+
**`0.1.0-pilot.8`** adds `flightdeck admin flights` for organisation and
|
|
57
|
+
platform administrators ([PR #281](https://github.com/ThermAI-Group/thermai_flightdeck/pull/281)).
|
|
58
|
+
It needs a credential approved through `flightdeck login --admin`; existing
|
|
59
|
+
credentials keep working for every other command, so administrators run
|
|
60
|
+
`flightdeck logout` then `flightdeck login --admin` only when they want the
|
|
61
|
+
admin view. `flightdeck login` now defaults the credential name to
|
|
62
|
+
`flightdeck-cli@<hostname>` so a second machine no longer collides (#254), and
|
|
63
|
+
`flightdeck sbatch` prints a runnable status hint with the submitted flight ID
|
|
64
|
+
and labels the job ID on the ATC hint (#257, #259). The required backend is
|
|
65
|
+
already deployed; there are no new runtime dependencies. It was published to
|
|
66
|
+
npm's `next` channel on 8 October 2026. Use the generated `@next` command below, or pin
|
|
67
|
+
`@thermai/flightdeck-cli@0.1.0-pilot.8`, and confirm `flightdeck --version`
|
|
68
|
+
reports `0.1.0-pilot.8`.
|
|
69
|
+
|
|
70
|
+
**`0.1.0-pilot.9`** adds managed images and datasets. `flightdeck image check`,
|
|
71
|
+
`push`, `list` and `inspect` check and publish workload images to your
|
|
72
|
+
organization's managed registry. `flightdeck image alias ls|set|rm` and
|
|
73
|
+
`image push --alias` manage channels, and `managed://<name>:<alias>` (including
|
|
74
|
+
the computed `latest`) can be submitted as an image. `flightdeck dataset push`,
|
|
75
|
+
`list` and `rm` manage datasets, and `submit --dataset` mounts them. `sbatch`
|
|
76
|
+
refuses an array `%N` limit Flightdeck cannot enforce unless
|
|
77
|
+
`--ignore-array-limit` is passed. Alias commands need a backend with image
|
|
78
|
+
aliases; the other commands work against the current backend. There are no new
|
|
79
|
+
runtime dependencies.
|
|
80
|
+
|
|
81
|
+
**`0.1.0-pilot.10`** adds `flightdeck delete-output <flight-id> <output-id>`,
|
|
82
|
+
which deletes one output of a finished flight, and
|
|
83
|
+
`flightdeck dataset download <version> <directory>`, which downloads a dataset
|
|
84
|
+
version and checks every file against its recorded checksum; running it again
|
|
85
|
+
resumes. While a flight copies its datasets, `flightdeck status` shows their
|
|
86
|
+
progress on a `Data:` line. `flightdeck image check` now checks images against
|
|
87
|
+
the supervisor release that Flightdeck 26.11.3 runs. Both new commands need Flightdeck 26.11.3
|
|
88
|
+
or later, and `dataset download` also needs storage enabled for your
|
|
89
|
+
organisation. There are no new runtime dependencies.
|
|
90
|
+
|
|
91
|
+
**`0.1.0-pilot.11`** adds `--secret ENV=name` to `flightdeck submit` and
|
|
92
|
+
`flightdeck sbatch`, which gives the workload one of your organisation's secrets
|
|
93
|
+
(see "Keeping secrets"). `flightdeck logs --follow` and `slurm-test` keep going
|
|
94
|
+
through the few seconds a release restarts the backend. A refused submit now
|
|
95
|
+
says what was wrong with `--base-image` or `--provider` in plain words instead
|
|
96
|
+
of printing the API's JSON. `flightdeck outputs` and `flightdeck download` say
|
|
97
|
+
when collection after a cancel was cut off, so some outputs or logs may be
|
|
98
|
+
missing, and
|
|
99
|
+
`flightdeck logs` says when a flight's logs were deleted under the 30-day limit.
|
|
100
|
+
`flightdeck image check` checks images against the supervisor release that
|
|
101
|
+
Flightdeck 26.11.4 runs. `--secret` needs Flightdeck 26.11.4 or later and
|
|
102
|
+
secrets enabled for your organisation; an older backend ignores it. There are no
|
|
103
|
+
new runtime dependencies.
|
|
104
|
+
|
|
105
|
+
Members and organization administrators receive whole-credit balances,
|
|
106
|
+
estimates and actual customer charges, without provider-cost information.
|
|
107
|
+
Platform administrators retain their authorized monetary API fields. Credits
|
|
108
|
+
are not GPU-hours; consumption depends on the selected hardware/provider and
|
|
109
|
+
the accepted job pricing. Existing limits and customer charge caps still apply.
|
|
19
110
|
|
|
20
111
|
## Installation and login
|
|
21
112
|
|
|
22
113
|
Requires Node.js 22 and npm. Use a user-owned Node installation or npm prefix;
|
|
23
114
|
the CLI does not require administrator/root privileges.
|
|
24
115
|
|
|
116
|
+
<!-- flightdeck-cli-install:start -->
|
|
117
|
+
|
|
25
118
|
```sh
|
|
26
|
-
npm install -g @thermai/flightdeck-cli@
|
|
119
|
+
npm install -g @thermai/flightdeck-cli@next
|
|
27
120
|
flightdeck login
|
|
28
121
|
```
|
|
29
122
|
|
|
123
|
+
<!-- flightdeck-cli-install:end -->
|
|
124
|
+
|
|
30
125
|
Login prints a URL and verification code. Open that URL, sign into
|
|
31
126
|
`https://portal.thermai.uk`, and approve the named CLI credential and requested
|
|
32
127
|
scopes. The CLI waits for approval; no API key needs to be copied manually.
|
|
@@ -42,26 +137,896 @@ test environment can be selected with `flightdeck login --api-url <HTTPS origin>
|
|
|
42
137
|
|
|
43
138
|
Login saves a scoped credential in `~/.flightdeck/config.json` with restrictive
|
|
44
139
|
permissions. Never share that file or include it in a workload bundle. Use
|
|
45
|
-
`flightdeck logout` to
|
|
46
|
-
credential management can revoke an individual workstation as well.
|
|
140
|
+
`flightdeck logout` to remove the local login and attempt server revocation.
|
|
141
|
+
Portal credential management can revoke an individual workstation as well.
|
|
142
|
+
|
|
143
|
+
`--name` labels a credential, not a local account profile. It defaults to
|
|
144
|
+
`flightdeck-cli@<short hostname>`, which is shown on the approval page, in
|
|
145
|
+
credential lists, and in the audit history; pass `--name` to choose another
|
|
146
|
+
label. Active credential names must be unique per account, so machines that
|
|
147
|
+
share a short hostname need distinct `--name` values. There is one saved
|
|
148
|
+
login per OS user. If already logged in, use `whoami` to inspect it; `login`
|
|
149
|
+
refuses to replace it. Use a separate OS user to preserve an operator's working
|
|
150
|
+
credential, or explicitly choose logout/re-login on the same workstation.
|
|
151
|
+
Logging out removes the saved credential; it does not delete the account or its
|
|
152
|
+
roles. A later login creates a new credential. Check the separate revocation
|
|
153
|
+
result: other copies may remain valid if server revocation was not confirmed.
|
|
154
|
+
|
|
155
|
+
Pilot.6 explains `account_suspended`, `membership_suspended`
|
|
156
|
+
and `organization_suspended` denials during device login and authenticated
|
|
157
|
+
commands. Contact your organization administrator; organization suspension may
|
|
158
|
+
need ThermAI support. These are access restrictions, not a missing local login
|
|
159
|
+
or a request for additional credential scopes. Other 403 responses retain
|
|
160
|
+
generic permissions guidance. Commands still exit nonzero and do not print
|
|
161
|
+
server-supplied messages or stack traces. Older pilot.5 installations need an
|
|
162
|
+
explicit upgrade to receive these messages.
|
|
163
|
+
|
|
164
|
+
## Pilot quick-start
|
|
165
|
+
|
|
166
|
+
Before live use, your ThermAI operator must provision your organization/account,
|
|
167
|
+
provide a single-use portal invitation, and approve your policy and credit.
|
|
168
|
+
Activate through the portal, then follow the installation/login commands above.
|
|
169
|
+
Confirm the expected account with `whoami` and the available credit with
|
|
170
|
+
`balance`. ThermAI will guide the first partner submission under the agreed limits.
|
|
171
|
+
|
|
172
|
+
For a **local-only** readiness check, save the following as `readiness.sbatch`
|
|
173
|
+
in a new empty directory. Run the dry run from that directory; no account,
|
|
174
|
+
upload, API request, or paid resource is needed:
|
|
175
|
+
|
|
176
|
+
<!-- flightdeck-cli-readiness-script:start -->
|
|
177
|
+
|
|
178
|
+
```bash
|
|
179
|
+
#!/bin/bash
|
|
180
|
+
#SBATCH --array=1-2
|
|
181
|
+
#SBATCH --gres=gpu:A10:1
|
|
182
|
+
#SBATCH --time=00:01:00
|
|
183
|
+
#SBATCH --output=%A_%a.out
|
|
184
|
+
echo smoke
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
<!-- flightdeck-cli-readiness-script:end -->
|
|
188
|
+
|
|
189
|
+
```sh
|
|
190
|
+
flightdeck --version
|
|
191
|
+
flightdeck sbatch readiness.sbatch --dry-run
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
Expect A10 x1, 60 seconds and **two flights would be submitted**, with no
|
|
195
|
+
warnings. Keep `--dry-run`: this is a translation exercise, not the approved
|
|
196
|
+
pilot workload or permission to launch it. A dry run does not check provider
|
|
197
|
+
capacity, account policy/credit, bundle contents, image access or GPU execution.
|
|
198
|
+
|
|
199
|
+
For the real workload, obtain the reviewed private bundle, exact image
|
|
200
|
+
digest, provider selection and runtime from the operator. Review its dry run,
|
|
201
|
+
then use the supplied submission command only after execution approval. Do not
|
|
202
|
+
copy internal test data into public packages or partner bundles. Use explicit
|
|
203
|
+
`--files` selections from a clean directory; the submission default includes
|
|
204
|
+
the current directory, less the default excludes listed under
|
|
205
|
+
[What goes into a bundle](#what-goes-into-a-bundle). Never include credentials
|
|
206
|
+
or private workstation files.
|
|
47
207
|
|
|
48
208
|
## Workload commands
|
|
49
209
|
|
|
50
|
-
After
|
|
210
|
+
After your account's policy/credit setup and agreed first-run approval:
|
|
51
211
|
|
|
52
212
|
```sh
|
|
213
|
+
flightdeck image check <image> # before using --base-image
|
|
53
214
|
flightdeck sbatch workload.sbatch
|
|
54
215
|
flightdeck status <flight-id>
|
|
55
216
|
flightdeck logs <flight-id> --follow
|
|
56
217
|
flightdeck outputs <flight-id>
|
|
57
218
|
flightdeck download <flight-id> --output-dir ./results
|
|
219
|
+
flightdeck delete-output <flight-id> <output-id>
|
|
58
220
|
flightdeck cancel <flight-id>
|
|
59
221
|
```
|
|
60
222
|
|
|
61
223
|
Submission, including `flightdeck slurm-test`, can provision paid resources.
|
|
224
|
+
`submit` and `sbatch` print the deployment's output size limit first. Everything
|
|
225
|
+
in `/outputs` is stored as one compressed archive, and an archive over the limit
|
|
226
|
+
is not stored at all. `sbatch --dry-run` stays offline and states the limit
|
|
227
|
+
without a figure.
|
|
62
228
|
`sbatch --dry-run` only translates locally and does not submit. Flightdeck
|
|
63
229
|
supports a subset of single-node Slurm directives, not an arbitrary HPC cluster.
|
|
64
|
-
|
|
230
|
+
Without `--base-image`, a flight runs on the official Flightdeck base
|
|
231
|
+
(`ghcr.io/thermai-group/flightdeck-base`): Ubuntu 24.04 with Python 3 and pip,
|
|
232
|
+
and no PyTorch or CUDA, so supply Python packages with `--requirements` or use
|
|
233
|
+
your own image. A `--base-image` can be any linux/amd64 image that meets the
|
|
234
|
+
workload runtime contract (see `/docs/images`); it does not need the
|
|
235
|
+
Flightdeck runner. Before charging, submission refuses an image whose
|
|
236
|
+
`ENTRYPOINT` is shell form
|
|
237
|
+
(`shell_form_entrypoint`) or whose `uk.thermai.flightdeck.contract` label names
|
|
238
|
+
a contract version Flightdeck does not run (`image_unsupported_contract`).
|
|
239
|
+
Check an image locally first with `flightdeck image check`; see
|
|
240
|
+
[Checking an image before you submit](#checking-an-image-before-you-submit).
|
|
241
|
+
Environment variables (`--env`, `#SBATCH --export`) are checked before upload:
|
|
242
|
+
names are at most 64 characters, and names starting with `FLIGHTDECK_` belong
|
|
243
|
+
to the platform and are refused.
|
|
244
|
+
|
|
245
|
+
`--secret ENV=name` (on `submit` and `sbatch`, repeatable, up to 16) gives the
|
|
246
|
+
workload your organisation's secret `name` as the environment variable `ENV`
|
|
247
|
+
and as the read-only file `/run/flightdeck/secrets/ENV`. The value is read when
|
|
248
|
+
the flight is dispatched and never passes through the CLI. Secrets run on
|
|
249
|
+
Lambda only, so add `--provider lambda` if other providers are active. `ENV`
|
|
250
|
+
follows the `--env` rules and cannot also be given with `--env`. Logs are
|
|
251
|
+
redacted on a best-effort basis only, so never print a secret. See
|
|
252
|
+
`docs/customer-secrets.md` "Delivery to workloads".
|
|
253
|
+
|
|
254
|
+
```sh
|
|
255
|
+
flightdeck submit --provider lambda --secret HF_TOKEN=hf-token "python train.py"
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
An array prints a separate Flightdeck ID for each accepted task. Record them
|
|
259
|
+
all: inspect, download and cancel each flight individually (`flightdeck atc
|
|
260
|
+
<job-id>` shows them together). Partial submission
|
|
261
|
+
can leave accepted flights running even if the command exits unsuccessfully;
|
|
262
|
+
do not blindly resubmit the whole array. Downloads go to
|
|
263
|
+
`<output-dir>/<flight-id>/attempt-<n>/`, or `legacy/` for results from before
|
|
264
|
+
attempts were recorded, keeping each output's relative path. `--attempt <n>`
|
|
265
|
+
selects an earlier attempt. An interrupted download resumes when you run the same
|
|
266
|
+
command again, and a file only appears once its size and SHA-256 match the stored
|
|
267
|
+
output. Partial data is kept in `<file>.part` until then. A `.part` file that
|
|
268
|
+
the CLI did not leave for that output is refused, never deleted. A matching file is
|
|
269
|
+
skipped and a different one is never overwritten. A second download of the same
|
|
270
|
+
file stops with an error while the first is running. Stored results have no expiry
|
|
271
|
+
today. `delete-output` removes one output, by the ID `outputs` prints, from the
|
|
272
|
+
outputs list and from storage; the stored copy is kept for 30 days before it is
|
|
273
|
+
removed. It needs the `jobs:cancel` scope and only works once the flight has
|
|
274
|
+
finished and, for a cancelled flight, its outputs have been collected. If storage
|
|
275
|
+
does not confirm the delete, the output stays listed and may or may not have been
|
|
276
|
+
removed from storage; run the same command again to finish the delete. `outputs` and `download` print a warning on stderr when an output was
|
|
277
|
+
over the size limit or collection ended early, including a cancelled flight
|
|
278
|
+
whose outputs may not have been collected, so a short list isn't mistaken for
|
|
279
|
+
everything. With `--attempt`, only that attempt's size refusals are reported. Check terminal status and exit code as well as the expected artifacts;
|
|
280
|
+
successful submission or download alone does not mean the workload succeeded.
|
|
281
|
+
|
|
282
|
+
For an incident, stop submitting, cancel the known active flights, and contact
|
|
283
|
+
your named ThermAI operator. Logging out does **not** stop jobs. The operator
|
|
284
|
+
must confirm provider teardown and reconcile any outstanding charges before a
|
|
285
|
+
retry. Support contact, supported hours and credit increases come from your
|
|
286
|
+
agreed pilot handoff; this client does not promise unattended support.
|
|
287
|
+
|
|
288
|
+
### Saving, downloading and deleting results
|
|
289
|
+
|
|
290
|
+
Anything a flight writes under `/outputs` (also `$OUTPUTS_DIR`) is saved when the
|
|
291
|
+
flight ends and can be downloaded after the GPU instance has gone. Files anywhere
|
|
292
|
+
else are not kept. This walk-through saves one file, lists it, downloads it,
|
|
293
|
+
checks it and deletes it. The output shown is from a run on staging on
|
|
294
|
+
14 November 2026. `delete-output` needs CLI `0.1.0-pilot.10` or later. On macOS
|
|
295
|
+
use `shasum -a 256` wherever this walk-through says `sha256sum`.
|
|
296
|
+
|
|
297
|
+
**1. Write to `/outputs`.** Save this as `results-demo.sbatch` in a new empty
|
|
298
|
+
directory:
|
|
299
|
+
|
|
300
|
+
```bash
|
|
301
|
+
#!/bin/bash
|
|
302
|
+
#SBATCH --gres=gpu:A10:1
|
|
303
|
+
#SBATCH --time=00:05:00
|
|
304
|
+
#SBATCH --output=%j.out
|
|
305
|
+
echo "report $(date -u +%FT%TZ)" > "$OUTPUTS_DIR/report.txt"
|
|
306
|
+
sha256sum "$OUTPUTS_DIR/report.txt"
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
```sh
|
|
310
|
+
flightdeck sbatch results-demo.sbatch
|
|
311
|
+
flightdeck status <flight-id> # wait for completed
|
|
312
|
+
flightdeck logs <flight-id> # shows the checksum the workload printed
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
`sbatch` prints the flight ID, and `logs` prints the workload's line:
|
|
316
|
+
|
|
317
|
+
```text
|
|
318
|
+
sbatch-job: fl-00e67a3d3ed2ae2229952634 (queue #1)
|
|
319
|
+
|
|
320
|
+
1 submitted, 0 failed.
|
|
321
|
+
Track with: flightdeck status fl-00e67a3d3ed2ae2229952634
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
```text
|
|
325
|
+
6bc1d843833bc55a3f66ec342533ed7586b757d0f4f7c87c211ddcde597f3bba /outputs/report.txt
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
**2. List what was saved.** On Lambda, everything in `/outputs` is saved as one
|
|
329
|
+
archive, `outputs.tar.gz`. Each line is the output ID, its size and its name:
|
|
330
|
+
|
|
331
|
+
```sh
|
|
332
|
+
flightdeck outputs <flight-id>
|
|
333
|
+
```
|
|
334
|
+
|
|
335
|
+
```text
|
|
336
|
+
59925a74-7a23-46e8-8bb0-26c7be1eddb0 274 bytes outputs.tar.gz
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
A flight that is still running, or a cancelled flight whose outputs are still
|
|
340
|
+
being collected, prints `No outputs available.` An output is listed only once
|
|
341
|
+
it is stored.
|
|
342
|
+
|
|
343
|
+
**3. Download and check it.** The file lands in
|
|
344
|
+
`<output-dir>/<flight-id>/attempt-<n>/`. The CLI checks its size and SHA-256
|
|
345
|
+
against the stored output before the file appears, so a file that is there is
|
|
346
|
+
complete.
|
|
347
|
+
|
|
348
|
+
```sh
|
|
349
|
+
flightdeck download <flight-id> --output-dir ./results
|
|
350
|
+
cd ./results/<flight-id>/attempt-1
|
|
351
|
+
tar -tzf outputs.tar.gz # look before you extract
|
|
352
|
+
tar -xzf outputs.tar.gz
|
|
353
|
+
sha256sum report.txt # compare with the line in the logs
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
```text
|
|
357
|
+
Downloaded /home/you/results/fl-00e67a3d3ed2ae2229952634/attempt-1/outputs.tar.gz
|
|
358
|
+
./
|
|
359
|
+
./1794696621749957.out
|
|
360
|
+
./report.txt
|
|
361
|
+
6bc1d843833bc55a3f66ec342533ed7586b757d0f4f7c87c211ddcde597f3bba report.txt
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
The archive also holds the `#SBATCH --output` file, here `1794696621749957.out`.
|
|
365
|
+
|
|
366
|
+
Running the same `download` again prints `Already downloaded …` and changes
|
|
367
|
+
nothing.
|
|
368
|
+
|
|
369
|
+
**4. Delete it.** Pass the output ID from step 2. This needs a login with the
|
|
370
|
+
`jobs:cancel` scope.
|
|
371
|
+
|
|
372
|
+
```sh
|
|
373
|
+
flightdeck delete-output <flight-id> <output-id>
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
```text
|
|
377
|
+
Deleted output 59925a74-7a23-46e8-8bb0-26c7be1eddb0 from fl-00e67a3d3ed2ae2229952634
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
The output leaves the list at once, and `flightdeck outputs` then prints
|
|
381
|
+
`No outputs available.` The stored copy is removed 30 days later.
|
|
382
|
+
|
|
383
|
+
**The same steps through the API.** Every request needs your credential and the
|
|
384
|
+
accounting header. The credential is `api_key` in `~/.flightdeck/config.json`.
|
|
385
|
+
|
|
386
|
+
```sh
|
|
387
|
+
API=https://api.flightdeck.thermai.uk/api/v1
|
|
388
|
+
AUTH=(-H "Authorization: Bearer $FLIGHTDECK_API_KEY" -H "X-Flightdeck-Accounting: credits-v1")
|
|
389
|
+
|
|
390
|
+
# List: artifact_id, filename, size_bytes, checksum_sha256, media_type, created_at
|
|
391
|
+
curl -s "${AUTH[@]}" "$API/flights/<flight-id>/outputs"
|
|
392
|
+
|
|
393
|
+
# Download: the API answers with a redirect to a signed URL that works for 5 minutes
|
|
394
|
+
curl -sL "${AUTH[@]}" -o outputs.tar.gz "$API/flights/<flight-id>/outputs/<artifact-id>"
|
|
395
|
+
sha256sum outputs.tar.gz # must equal checksum_sha256 from the list
|
|
396
|
+
|
|
397
|
+
# Delete: 204 on success
|
|
398
|
+
curl -s -o /dev/null -w '%{http_code}\n' -X DELETE "${AUTH[@]}" \
|
|
399
|
+
"$API/flights/<flight-id>/outputs/<artifact-id>"
|
|
400
|
+
|
|
401
|
+
# What happened to the flight, including collection
|
|
402
|
+
curl -s "${AUTH[@]}" "$API/flights/<flight-id>/events"
|
|
403
|
+
```
|
|
404
|
+
|
|
405
|
+
The list answers like this, and the downloaded file's SHA-256 equalled
|
|
406
|
+
`checksum_sha256`:
|
|
407
|
+
|
|
408
|
+
```json
|
|
409
|
+
{
|
|
410
|
+
"outputs": [
|
|
411
|
+
{
|
|
412
|
+
"artifact_id": "59925a74-7a23-46e8-8bb0-26c7be1eddb0",
|
|
413
|
+
"filename": "outputs.tar.gz",
|
|
414
|
+
"size_bytes": 274,
|
|
415
|
+
"checksum_sha256": "0f88381b16b970004118af20db4877aa891d0b028638f336a1ffcba98e4f8b11",
|
|
416
|
+
"media_type": "application/gzip",
|
|
417
|
+
"created_at": "2026-11-14T22:56:09.464Z"
|
|
418
|
+
}
|
|
419
|
+
],
|
|
420
|
+
"attempt_seq": 1,
|
|
421
|
+
"available_attempts": [1]
|
|
422
|
+
}
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
The events for the same flight, with the delete recorded last:
|
|
426
|
+
|
|
427
|
+
```text
|
|
428
|
+
22:50:22.613Z state_changed Flight queued
|
|
429
|
+
22:55:47.958Z state_changed Flight dispatched
|
|
430
|
+
22:55:54.311Z state_changed Collecting logs and outputs
|
|
431
|
+
22:56:09.484Z state_changed Flight completed
|
|
432
|
+
22:57:34.672Z output_deleted An output was deleted at the customer's request.
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
The list also returns `attempt_seq` and `available_attempts`; add `?attempt=<n>`
|
|
436
|
+
for an earlier attempt. `curl -L` does not send your credential to the signed
|
|
437
|
+
URL, and nor should your own client.
|
|
438
|
+
|
|
439
|
+
| Delete answer | Meaning |
|
|
440
|
+
| --- | --- |
|
|
441
|
+
| `204` | Deleted. |
|
|
442
|
+
| `404 not_found` | No such output, or the flight is not yours. |
|
|
443
|
+
| `409 flight_not_finished` | The flight is still running. Nothing was deleted. |
|
|
444
|
+
| `409 outputs_being_collected` | A cancelled flight is still saving its outputs. Try again in a few minutes. |
|
|
445
|
+
|
|
446
|
+
**When outputs are missing.** `outputs` and `download` print a warning when
|
|
447
|
+
something was not saved. The flight's events say why:
|
|
448
|
+
|
|
449
|
+
| Event | What happened | What to do |
|
|
450
|
+
| --- | --- | --- |
|
|
451
|
+
| `outputs_too_large` | The archive was over the size limit (5 GiB unless your operator says otherwise) and was not stored. | Write less to `/outputs` and run again. |
|
|
452
|
+
| `finalization_abandoned` | Saving failed five times and was given up. | Run again. A flight whose workload exited 0 is not charged; one whose workload failed is charged like any failed flight. |
|
|
453
|
+
| `deadline_collection_abandoned` | The flight hit its time limit and saving did not finish in the time left. | Raise `--time`, or write results earlier, and run again. |
|
|
454
|
+
| `cancel_stop_failed` | The flight was cancelled but its workload could not be confirmed stopped, so its outputs may not have been saved. | Check `outputs`. Run again if they are missing. |
|
|
455
|
+
| `cancel_collection_cutoff` | The flight was cancelled and saving had not finished an hour later, so it was stopped and the instance destroyed. | Check `outputs`. Run again if they are missing. |
|
|
456
|
+
|
|
457
|
+
**Known limits.**
|
|
458
|
+
|
|
459
|
+
- **No retry on request.** Saving is retried automatically, up to five times,
|
|
460
|
+
while the instance exists. After that the only way to get the results is to
|
|
461
|
+
run the flight again.
|
|
462
|
+
- **Outputs that were not saved are gone.** The instance is destroyed when
|
|
463
|
+
saving is given up, and anything that was only on it cannot be recovered.
|
|
464
|
+
- **A cancelled flight saves after it is cancelled.** Its events show
|
|
465
|
+
`cancel_collection_started` while saving and `cancel_collection_finished`
|
|
466
|
+
once it is done. Outputs appear in `outputs` when they are stored, usually
|
|
467
|
+
within a minute.
|
|
468
|
+
- **Results do not expire yet.** A saved output stays until you delete it. A
|
|
469
|
+
30-day purge after a pilot ends is planned.
|
|
470
|
+
- **One archive per flight on Lambda.** You cannot download a single file from
|
|
471
|
+
inside `outputs.tar.gz`, or delete one.
|
|
472
|
+
- **Only your own flights.** Another account, in your organisation or outside
|
|
473
|
+
it, gets `404 not_found` for your flight's outputs.
|
|
474
|
+
|
|
475
|
+
### What goes into a bundle
|
|
476
|
+
|
|
477
|
+
`submit` and `sbatch` pack the current directory, or the `--files` paths, into
|
|
478
|
+
the bundle that is unpacked on the node. These are left out by default:
|
|
479
|
+
|
|
480
|
+
- version control: `.git`, `.hg`, `.svn`, `.bzr` (directories or files)
|
|
481
|
+
- environment and credential files: `.env*`, `.netrc`, `.pypirc`, `.npmrc`,
|
|
482
|
+
`.git-credentials`, `.ssh/`, `.aws/`, `.gnupg/`, `.docker/`, `.kube/`,
|
|
483
|
+
`.flightdeck/`
|
|
484
|
+
- key files: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.ppk`, `id_rsa`, `id_dsa`,
|
|
485
|
+
`id_ecdsa`, `id_ed25519`, `id_ecdsa_sk`, `id_ed25519_sk`
|
|
486
|
+
- virtualenvs: `venv/`, `.venv/` and any directory containing `pyvenv.cfg`
|
|
487
|
+
|
|
488
|
+
These match regardless of case, so `.ENV` and `ID_RSA` are left out too.
|
|
489
|
+
|
|
490
|
+
A `.flightdeckignore` in the current directory adds rules in `.gitignore`
|
|
491
|
+
syntax, and `!<path>` includes something a default would leave out. Only a `!`
|
|
492
|
+
rule without wildcards lifts a default; one such as `!*/` re-includes only what
|
|
493
|
+
your own rules excluded:
|
|
494
|
+
|
|
495
|
+
```gitignore
|
|
496
|
+
checkpoints/*
|
|
497
|
+
!checkpoints/final.pt
|
|
498
|
+
!.env.example
|
|
499
|
+
```
|
|
500
|
+
|
|
501
|
+
As with `.gitignore`, a path under an excluded directory cannot be re-included
|
|
502
|
+
on its own; re-include the directory, or exclude its contents with `dir/*`
|
|
503
|
+
instead. Naming an excluded path with `--files` does not include it (a shell
|
|
504
|
+
glob such as `--files *` expands to every name); use `!<path>` in
|
|
505
|
+
`.flightdeckignore`. A `--files` path that goes through a symlinked directory
|
|
506
|
+
(`alias/credentials` with `alias -> .aws`) is checked under the path it resolves
|
|
507
|
+
to as well, so the `!<path>` must name the resolved path; one that resolves
|
|
508
|
+
outside the current directory is refused. Symlinks that dangle, whose target is not in the bundle
|
|
509
|
+
(such as `data -> /scratch/...`), or whose target is an absolute path are
|
|
510
|
+
skipped with a warning because they cannot resolve on the node; use relative
|
|
511
|
+
links within the directory. To use a dataset that lives outside it, copy it
|
|
512
|
+
in or stage it on the node another way. The file count and size print before upload, and
|
|
513
|
+
`sbatch --dry-run` shows the same summary.
|
|
514
|
+
|
|
515
|
+
### Checking an image before you submit
|
|
516
|
+
|
|
517
|
+
`flightdeck image check <image>` checks an image with your local Docker, so you
|
|
518
|
+
learn before you submit whether Flightdeck is likely to admit it and how it
|
|
519
|
+
behaves. It
|
|
520
|
+
needs no login. When you are logged in, it reads the platform's current runtime
|
|
521
|
+
facts (the supervisor release and the host drivers) from the API; otherwise it
|
|
522
|
+
uses the ones built into the CLI.
|
|
523
|
+
|
|
524
|
+
```sh
|
|
525
|
+
flightdeck image check ghcr.io/your-org/your-image:1.0
|
|
526
|
+
flightdeck image check ghcr.io/your-org/your-image:1.0 --json
|
|
527
|
+
```
|
|
528
|
+
|
|
529
|
+
It reports each check as PASS, WARN, FAIL, INFO, SKIP or ERROR (the check
|
|
530
|
+
could not run), with a one-line reason, and ends with a verdict:
|
|
531
|
+
|
|
532
|
+
- **Submission rules**, the same code the API uses: `linux/amd64`, a docker.io,
|
|
533
|
+
ghcr.io or nvcr.io reference within 255 characters once pinned to its digest,
|
|
534
|
+
and the `uk.thermai.flightdeck.contract` label. If Docker's copy is for
|
|
535
|
+
another platform (Docker on Apple silicon pulls `arm64`), the check asks the
|
|
536
|
+
registry whether it has the `linux/amd64` image Flightdeck runs: if it does,
|
|
537
|
+
that is a warning and the dry run is skipped; if not, it fails. The check
|
|
538
|
+
never re-pulls or retags your copy. A shell-form `ENTRYPOINT` is a
|
|
539
|
+
warning: Flightdeck refuses it for flights with a command, and runs it as it
|
|
540
|
+
is for image-only flights. `tini` or `dumb-init` as the `ENTRYPOINT` is a
|
|
541
|
+
warning too, because the supervisor already does their job.
|
|
542
|
+
- **Registry**: Flightdeck reads the image from its registry, without
|
|
543
|
+
credentials. A local image that is not in its registry yet is a warning, and
|
|
544
|
+
so is one the registry will not serve without credentials (the check asks it
|
|
545
|
+
anonymously, with `docker manifest inspect` and an empty Docker
|
|
546
|
+
configuration). New GitHub packages are private until you make them public.
|
|
547
|
+
- **Dry run**: the image starts the way a flight starts, under the Flightdeck
|
|
548
|
+
supervisor with `/workspace` and `/outputs` mounted, CPU only unless you pass
|
|
549
|
+
`--gpu`. `--gpu` attaches GPUs with Docker's `--gpus`; Flightdeck's hosts use
|
|
550
|
+
CDI, so the dry run does not test a provider's GPUs or driver. It runs a command as the image's `USER`; an `ENTRYPOINT` that ignores
|
|
551
|
+
the command fails. It installs a requirements file with no packages in it; a
|
|
552
|
+
missing or unusable `pip` is a warning, since only a bundle with a
|
|
553
|
+
requirements file needs one. It stops a running command with `SIGTERM` and
|
|
554
|
+
expects it to exit within the 30-second grace. A missing `bash` is a warning,
|
|
555
|
+
because commands then run under `sh` and sbatch scripts need `bash`.
|
|
556
|
+
- **CUDA**, from `CUDA_VERSION`, `libcudart`, pip's CUDA runtime wheel and
|
|
557
|
+
`torch.version.cuda`: a CUDA newer than every published host driver runs is a
|
|
558
|
+
warning. An image without CUDA is fine. `flightdeck submit` and
|
|
559
|
+
`flightdeck sbatch` also warn, after the flight is accepted, when the image
|
|
560
|
+
sets a `CUDA_VERSION` newer than the lowest published host driver runs; the
|
|
561
|
+
other sources are read only here.
|
|
562
|
+
- **Secrets**: the image's file names are listed through `docker export`, without
|
|
563
|
+
extracting anything, and matched against the credential and key patterns in
|
|
564
|
+
[What goes into a bundle](#what-goes-into-a-bundle). Empty files, CA
|
|
565
|
+
certificates and the test keys that packages ship are skipped.
|
|
566
|
+
|
|
567
|
+
The exit status is 0 when nothing failed (warnings allowed), 1 when a check
|
|
568
|
+
failed, and 2 when the check itself could not run, for example without Docker
|
|
569
|
+
or when the pull failed. `--static-only` runs nothing from the image, and
|
|
570
|
+
`--supervisor <image>` runs the dry run under another supervisor image.
|
|
571
|
+
|
|
572
|
+
The verdict says what the result means for submission:
|
|
573
|
+
|
|
574
|
+
- **PASS**, with every check run: Flightdeck will admit the image, and it ran
|
|
575
|
+
commands as the workload runtime contract describes.
|
|
576
|
+
- **PASS for the checks that ran**: nothing failed, but the verdict lists what
|
|
577
|
+
was not checked, such as the dry run under `--static-only`.
|
|
578
|
+
- **WARN, "Flightdeck may refuse this image"**: a warning that admission acts
|
|
579
|
+
on, named in the verdict. The image is not in its registry yet, the registry
|
|
580
|
+
will not serve it without credentials, or its `ENTRYPOINT` is shell form,
|
|
581
|
+
which is refused for every flight with a command.
|
|
582
|
+
- **WARN** otherwise: Flightdeck will admit the image; read the warnings.
|
|
583
|
+
- **FAIL**: Flightdeck would refuse the image, or its flights would not run.
|
|
584
|
+
|
|
585
|
+
If Docker does not have the image, the check pulls it for `linux/amd64` and
|
|
586
|
+
leaves it for you; it never removes an image you already had. Everything else
|
|
587
|
+
it creates is removed when it finishes, fails or is interrupted with Ctrl-C:
|
|
588
|
+
its containers (named `flightdeck-image-check-<id>-…` and labelled
|
|
589
|
+
`uk.thermai.flightdeck.image-check`), its temporary directory, and the
|
|
590
|
+
supervisor image if the check pulled it (unless another image check is
|
|
591
|
+
running, which may still need it). After Ctrl-C the check waits for Docker to
|
|
592
|
+
remove its containers, up to a minute each, and a second Ctrl-C does not cut
|
|
593
|
+
that short. If the check is killed outright, list what it left with
|
|
594
|
+
`docker ps -a --filter label=uk.thermai.flightdeck.image-check`, remove each with
|
|
595
|
+
`docker rm -f <name>`, and delete any `flightdeck-image-check-*` directory in
|
|
596
|
+
your temporary directory. On Apple silicon and other machines that are not
|
|
597
|
+
amd64, the image runs under emulation, which is slow.
|
|
598
|
+
|
|
599
|
+
The check does not test your code or the provider's GPUs, so run a short flight
|
|
600
|
+
before real work.
|
|
601
|
+
|
|
602
|
+
### Publishing a private image
|
|
603
|
+
|
|
604
|
+
`flightdeck image push <local-image> --name <name> --version <label>` publishes
|
|
605
|
+
a local image to your organization's private registry. `image list` and
|
|
606
|
+
`image inspect <version-id> --format reference` show it and print the reference
|
|
607
|
+
to pass to `--base-image`. Log in with `flightdeck login --images` first (run
|
|
608
|
+
`logout` before it if you are already logged in). Rerun the same push to resume.
|
|
609
|
+
Managed images run on Lambda only, so add `--provider lambda` to `submit`.
|
|
610
|
+
`--base-image managed://<name>:latest` runs the newest ready version, and
|
|
611
|
+
`image push --alias ci` or `image alias set <name> ci --version <label>` moves a channel
|
|
612
|
+
you can run as `managed://<name>:ci`. The full guide, including errors and limits, is in
|
|
613
|
+
[`docs/managed-images.md`](../docs/managed-images.md).
|
|
614
|
+
|
|
615
|
+
1. `flightdeck logout`, then `flightdeck login --images`.
|
|
616
|
+
2. `flightdeck image push solver-local:v1 --name solver --version v1`.
|
|
617
|
+
3. `flightdeck image inspect <version-id>` shows the reference and the scan state.
|
|
618
|
+
|
|
619
|
+
Messages you may see:
|
|
620
|
+
|
|
621
|
+
- A 401 prints "Your FlightDeck login is no longer valid. The credential expired
|
|
622
|
+
or was revoked. Run flightdeck logout, then flightdeck login --images."
|
|
623
|
+
- A login made without `--images` prints "This login cannot use managed images.
|
|
624
|
+
Run flightdeck logout, then flightdeck login --images."
|
|
625
|
+
- An image above the registry's size limit, when one is set, is rejected after
|
|
626
|
+
the push and is not added to your list. Push a smaller image with a new
|
|
627
|
+
`--version`.
|
|
628
|
+
- A push that is not a single linux/amd64 image is rejected and names the
|
|
629
|
+
platform it found.
|
|
630
|
+
|
|
631
|
+
`image inspect` and `image list` show the registry's scan of the image, and
|
|
632
|
+
`image push` points to `image inspect` for it. Findings are advisory. They never block a push or a flight.
|
|
633
|
+
|
|
634
|
+
### Datasets: upload once, use in many flights
|
|
635
|
+
|
|
636
|
+
Managed storage keeps a directory of files in your organisation's space. You
|
|
637
|
+
upload it once as a version that never changes, then mount it read-only in any
|
|
638
|
+
Lambda flight. ThermAI must turn storage on for your organisation first. An
|
|
639
|
+
organisation admin uploads and deletes; any active member can list, download
|
|
640
|
+
and mount. This walk-through uploads a small dataset, uses it in a flight,
|
|
641
|
+
downloads it and deletes it. The output shown is from a run on staging on
|
|
642
|
+
14 November 2026. `dataset download` needs CLI `0.1.0-pilot.10` or later.
|
|
643
|
+
|
|
644
|
+
**1. Log in with storage access.** `login` refuses a second sign-in, so log out
|
|
645
|
+
first.
|
|
646
|
+
|
|
647
|
+
```sh
|
|
648
|
+
flightdeck logout
|
|
649
|
+
flightdeck login --storage
|
|
650
|
+
```
|
|
651
|
+
|
|
652
|
+
**2. Upload a directory.** Note the hash, so you can check it later. On macOS
|
|
653
|
+
use `shasum -a 256` wherever this walk-through says `sha256sum`.
|
|
654
|
+
|
|
655
|
+
```sh
|
|
656
|
+
mkdir corpus && head -c 1048576 /dev/urandom > corpus/sample.bin
|
|
657
|
+
sha256sum corpus/sample.bin
|
|
658
|
+
flightdeck dataset push ./corpus --name corpus --version 2026-11
|
|
659
|
+
```
|
|
660
|
+
|
|
661
|
+
```text
|
|
662
|
+
f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 corpus/sample.bin
|
|
663
|
+
Uploading corpus:2026-11: 1 files, 1.0 MiB
|
|
664
|
+
[1/1] sample.bin (1.0 MiB)
|
|
665
|
+
Verifying the upload
|
|
666
|
+
Published corpus:2026-11
|
|
667
|
+
Version ID: 33d54875-8fdd-4a25-a4d0-17e4a7449229
|
|
668
|
+
Files: 1, size: 1.0 MiB
|
|
669
|
+
Use it with: flightdeck submit "<command>" --dataset corpus:2026-11
|
|
670
|
+
```
|
|
671
|
+
|
|
672
|
+
If the upload is interrupted (Ctrl-C, a dropped connection, a closed laptop),
|
|
673
|
+
run the same command again. It sends only the parts that are missing. A 400 MiB
|
|
674
|
+
push stopped with Ctrl-C after ten seconds, then run again, printed:
|
|
675
|
+
|
|
676
|
+
```text
|
|
677
|
+
Interrupted. Stopping the upload.
|
|
678
|
+
The push was interrupted and did not finish. Run the same command again to resume.
|
|
679
|
+
```
|
|
680
|
+
|
|
681
|
+
```text
|
|
682
|
+
Uploading resume-check:v1: 1 files, 400.0 MiB
|
|
683
|
+
[1/1] big.bin (400.0 MiB)
|
|
684
|
+
resumed: 1 parts were already uploaded
|
|
685
|
+
Verifying the upload
|
|
686
|
+
Published resume-check:v1
|
|
687
|
+
```
|
|
688
|
+
|
|
689
|
+
Between the two, `dataset list` shows the version as `reserved`, and a flight
|
|
690
|
+
cannot use it yet. A push
|
|
691
|
+
of a version that is already complete prints `Already published corpus:2026-11`
|
|
692
|
+
in place of the upload lines and uploads nothing. A version never changes: to upload different files, choose a new
|
|
693
|
+
`--version`.
|
|
694
|
+
|
|
695
|
+
**3. List it.**
|
|
696
|
+
|
|
697
|
+
```sh
|
|
698
|
+
flightdeck dataset list
|
|
699
|
+
```
|
|
700
|
+
|
|
701
|
+
```text
|
|
702
|
+
NAME VERSION STATE SIZE FILES CREATED VERSION ID
|
|
703
|
+
corpus 2026-11 ready 1.0 MiB 1 2026-11-14 33d54875-8fdd-4a25-a4d0-17e4a7449229
|
|
704
|
+
```
|
|
705
|
+
|
|
706
|
+
Only a `ready` version can be mounted. `--json` gives the same list for
|
|
707
|
+
scripts.
|
|
708
|
+
|
|
709
|
+
**4. Use it in a flight.** Give `--dataset` as `<name>:<version>`, or as the
|
|
710
|
+
version ID. Add `:<mount>` to choose the directory name. Repeat the flag for up
|
|
711
|
+
to four datasets.
|
|
712
|
+
|
|
713
|
+
`submit` packs the current directory into the flight's bundle, so run it from a
|
|
714
|
+
directory that holds only the job's files. Do not run it from the directory
|
|
715
|
+
that holds `corpus/`, or the dataset is packed into the bundle as well. An
|
|
716
|
+
empty directory is refused with `Nothing to bundle`.
|
|
717
|
+
|
|
718
|
+
```sh
|
|
719
|
+
mkdir job && cd job
|
|
720
|
+
echo 'sha256sum /data/corpus/sample.bin' > check.sh
|
|
721
|
+
flightdeck submit "bash check.sh" --provider lambda \
|
|
722
|
+
--dataset corpus:2026-11 --time 5m
|
|
723
|
+
flightdeck status <flight-id> # a Data: line shows the copy's progress
|
|
724
|
+
flightdeck logs <flight-id> # the hash must equal the one from step 2
|
|
725
|
+
cd ..
|
|
726
|
+
```
|
|
727
|
+
|
|
728
|
+
```text
|
|
729
|
+
Flight submitted successfully!
|
|
730
|
+
Flight ID: fl-27aec32986c7f001885230f2
|
|
731
|
+
Status: queued
|
|
732
|
+
Est. credits: 14
|
|
733
|
+
Queue position: 1
|
|
734
|
+
```
|
|
735
|
+
|
|
736
|
+
While the flight is `dispatching`, `status` shows the copy, and `logs` shows
|
|
737
|
+
the same hash as step 2 once it has run:
|
|
738
|
+
|
|
739
|
+
```text
|
|
740
|
+
Data: Datasets fetched: 1.0 MiB in 1 file
|
|
741
|
+
```
|
|
742
|
+
|
|
743
|
+
```text
|
|
744
|
+
f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 /data/corpus/sample.bin
|
|
745
|
+
```
|
|
746
|
+
|
|
747
|
+
A second flight submitted with the version ID in place of the name
|
|
748
|
+
(`--dataset 33d54875-8fdd-4a25-a4d0-17e4a7449229`) printed the same line.
|
|
749
|
+
|
|
750
|
+
The flight reads the files at `/data/<mount>/`, read-only. `FLIGHTDECK_DATA_DIR`
|
|
751
|
+
is `/data`. Flightdeck copies the files to the machine before your container
|
|
752
|
+
starts, so reads are at local disk speed, and the flight stays `dispatching`
|
|
753
|
+
while it copies. The version is fixed when you submit: a flight always runs the
|
|
754
|
+
data it named, whatever is uploaded later. A second flight with the same
|
|
755
|
+
`--dataset` reuses the same upload.
|
|
756
|
+
|
|
757
|
+
**5. Download it, then delete it.**
|
|
758
|
+
|
|
759
|
+
```sh
|
|
760
|
+
flightdeck dataset download corpus:2026-11 ./corpus-copy
|
|
761
|
+
sha256sum ./corpus-copy/sample.bin
|
|
762
|
+
flightdeck dataset rm corpus:2026-11
|
|
763
|
+
```
|
|
764
|
+
|
|
765
|
+
```text
|
|
766
|
+
Downloaded 1 file(s), 1.0 MiB, of corpus:2026-11 to /home/you/corpus-copy.
|
|
767
|
+
f043ef26308ca8bc01a8d15b60ae5385fccbe4074f5a7ce6c85c01f5fd140034 ./corpus-copy/sample.bin
|
|
768
|
+
Deleted corpus:2026-11. New flights cannot use it. Its files are removed after the retention period.
|
|
769
|
+
```
|
|
770
|
+
|
|
771
|
+
After the delete, the version is gone from `dataset list`, and a submit that
|
|
772
|
+
names it is refused before anything launches:
|
|
773
|
+
|
|
774
|
+
```text
|
|
775
|
+
Submit failed (404): A dataset in --dataset was not found in your organization.
|
|
776
|
+
```
|
|
777
|
+
|
|
778
|
+
`download` checks every file against the checksum recorded at upload. Run it
|
|
779
|
+
again to resume; it reports files it already has as `already present and
|
|
780
|
+
verified` and never overwrites a file that differs. `rm` is refused while a
|
|
781
|
+
queued, dispatching or running flight uses the version. Deleted files are
|
|
782
|
+
removed from storage 7 days later.
|
|
783
|
+
|
|
784
|
+
**The same steps through the API.** The routes are under `/api/v1/storage`.
|
|
785
|
+
Reading needs the `storage:read` scope and changing needs `storage:write`,
|
|
786
|
+
which a `login --storage` credential has. The credential is `api_key` in
|
|
787
|
+
`~/.flightdeck/config.json`.
|
|
788
|
+
|
|
789
|
+
```sh
|
|
790
|
+
API=https://api.flightdeck.thermai.uk/api/v1
|
|
791
|
+
AUTH=(-H "Authorization: Bearer $FLIGHTDECK_API_KEY" -H "X-Flightdeck-Accounting: credits-v1")
|
|
792
|
+
JSON=(-H "Content-Type: application/json")
|
|
793
|
+
|
|
794
|
+
curl -s "${AUTH[@]}" "$API/storage" # enabled, entitled, usage, quota
|
|
795
|
+
curl -s "${AUTH[@]}" "$API/storage/datasets" # datasets
|
|
796
|
+
curl -s "${AUTH[@]}" "$API/storage/datasets/<dataset-id>/versions" # versions and their IDs
|
|
797
|
+
curl -s "${AUTH[@]}" "$API/storage/versions/<version-id>/files" # files, sizes and checksums
|
|
798
|
+
|
|
799
|
+
# Use it in a flight: add datasets to the body of POST /flights
|
|
800
|
+
# "datasets": [{"ref": "corpus:2026-11"}, {"ref": "<version-id>", "mount": "weights"}]
|
|
801
|
+
|
|
802
|
+
# Download: signed links for up to 100 files, each valid for one hour
|
|
803
|
+
curl -s "${AUTH[@]}" "${JSON[@]}" -X POST -d '{"paths":["sample.bin"]}' \
|
|
804
|
+
"$API/storage/versions/<version-id>/download-urls"
|
|
805
|
+
|
|
806
|
+
# Delete
|
|
807
|
+
curl -s "${AUTH[@]}" "${JSON[@]}" -X POST -d '{}' "$API/storage/versions/<version-id>/delete"
|
|
808
|
+
```
|
|
809
|
+
|
|
810
|
+
The files call returns each file's path, size and checksum, and the delete
|
|
811
|
+
call answers with the version's new state:
|
|
812
|
+
|
|
813
|
+
```json
|
|
814
|
+
{"files":[{"path":"sample.bin","size":1048576,"checksum":"SWvdRr7il3MHAC6btaOHPbHcf+lGzGNqWNpClfFNWfk=-1"}],"next_cursor":null}
|
|
815
|
+
{"id":"8e935b05-ac09-43a4-8ce9-1626ad587c45","state":"deleted","changed":true}
|
|
816
|
+
```
|
|
817
|
+
|
|
818
|
+
The download call returns one entry per file with the same fields plus
|
|
819
|
+
`part_size`, a signed `url` and `expires_at`.
|
|
820
|
+
|
|
821
|
+
Uploading through the API takes several calls per file (open a session, sign
|
|
822
|
+
each part, complete each file, then verify). Use `dataset push`, which also
|
|
823
|
+
resumes; the calls are described in `docs/managed-storage.md` if you need them.
|
|
824
|
+
|
|
825
|
+
**When something is refused.** Each refusal has a code, and nothing is
|
|
826
|
+
charged. The CLI prints a sentence for most of them. At submit,
|
|
827
|
+
`credential_scope_required` (a login without `--storage`) and
|
|
828
|
+
`dataset_mount_invalid` are printed as raw JSON.
|
|
829
|
+
|
|
830
|
+
| Code | When | What to do |
|
|
831
|
+
| --- | --- | --- |
|
|
832
|
+
| `storage_not_entitled` | Storage is not turned on for your organisation. | Ask ThermAI. |
|
|
833
|
+
| `credential_scope_required` | Your login has no storage access. | `flightdeck logout`, then `flightdeck login --storage`. |
|
|
834
|
+
| `org_admin_required` | A member who is not an admin tried to upload or delete. | Ask an organisation admin. |
|
|
835
|
+
| `storage_quota_exceeded` | The upload does not fit in your organisation's space. | Delete a version with `dataset rm`, or ask ThermAI. Space is freed when the deleted files are removed. |
|
|
836
|
+
| `dataset_limit_reached`, `version_limit_reached` | Too many datasets, or too many versions of one. | Delete one, or ask ThermAI. |
|
|
837
|
+
| `file_too_large` | One file is over the largest file size. | Split the file. |
|
|
838
|
+
| `version_exists` | That `--version` already exists for the dataset, from any upload. A deleted version's label stays reserved for good. | Choose a new `--version`. |
|
|
839
|
+
| `idempotency_conflict` | The same `--version` was pushed again with different files. | Choose a new `--version`. A version never changes. |
|
|
840
|
+
| `storage_fetch_quota_exceeded` | Your organisation reached its daily download limit, on a download or on a submit. | Wait. The limit counts the last 24 hours. Files already downloaded are kept. |
|
|
841
|
+
| `dataset_not_found` | The dataset is not in your organisation, or the version was deleted. | Check `dataset list`. |
|
|
842
|
+
| `dataset_not_ready` | The version is still uploading, or was disabled. | Check its state in `dataset list`. |
|
|
843
|
+
| `dataset_limit_exceeded` | The datasets are over the per-flight size or file limit, or do not fit the machine's disk. | Mount less. |
|
|
844
|
+
| `version_in_use` | `rm` on a version a queued, dispatching or running flight uses. | Wait for those flights to end. |
|
|
845
|
+
|
|
846
|
+
Four of these were produced on staging on 14 November 2026:
|
|
847
|
+
`dataset_not_found`, `dataset_not_ready`, `version_in_use` and
|
|
848
|
+
`idempotency_conflict`. The CLI printed:
|
|
849
|
+
|
|
850
|
+
```text
|
|
851
|
+
A dataset in --dataset was not found in your organization.
|
|
852
|
+
A dataset in --dataset is not ready. Check flightdeck dataset list.
|
|
853
|
+
corpus:2026-11b is in use by a queued, dispatching or running flight. Try again when those flights end.
|
|
854
|
+
A different upload is already recorded for corpus:2026-11b. If the directory changed since the first push, choose a new --version.
|
|
855
|
+
```
|
|
856
|
+
|
|
857
|
+
**When a flight cannot get its data.** The copy happens before your container
|
|
858
|
+
starts, so a flight that fails here is not charged. `flightdeck status` shows
|
|
859
|
+
the code.
|
|
860
|
+
|
|
861
|
+
| Code | What happened | What to do |
|
|
862
|
+
| --- | --- | --- |
|
|
863
|
+
| `data_fetch_failed` | The copy failed or was too slow. Flightdeck tries again on a new machine, up to three attempts. | Nothing. Watch `status`. |
|
|
864
|
+
| `data_fetch_exhausted` | The copy failed the same way twice, or the daily download limit ran out during it. | Submit again later. If it repeats, send ThermAI the flight ID. |
|
|
865
|
+
| `data_unavailable` | The version was disabled or deleted after you submitted, or storage was turned off for your organisation. | Check `dataset list`. |
|
|
866
|
+
| `data_too_large` | The data does not fit the machine's disk. | Mount less. |
|
|
867
|
+
|
|
868
|
+
**Known limits.**
|
|
869
|
+
|
|
870
|
+
- **Pilot limits:** 1 TiB and 50 datasets per organisation, 50 GiB for the
|
|
871
|
+
largest file, 50 GiB mounted per flight, and 150 GiB of downloads per
|
|
872
|
+
organisation in any 24 hours.
|
|
873
|
+
- **Downloads share one allowance.** `dataset download` and every flight that
|
|
874
|
+
mounts data count toward the same 150 GiB. Each attempt of a flight counts
|
|
875
|
+
again, so a flight that retries uses its size more than once. A large
|
|
876
|
+
download can leave too little for that day's flights.
|
|
877
|
+
- **Up to 4 datasets per flight**, 5000 files per version and 20 versions per
|
|
878
|
+
dataset.
|
|
879
|
+
- **Lambda only.** A flight on another provider is refused with
|
|
880
|
+
`dataset_provider_unsupported`.
|
|
881
|
+
- **Read-only, and copied every time.** A flight cannot write to `/data`, and
|
|
882
|
+
nothing is cached between flights: each flight copies its data again, which
|
|
883
|
+
adds to its start-up time.
|
|
884
|
+
- **File names.** Up to 8 path segments of letters, digits, `_`, `-` and `.`,
|
|
885
|
+
each up to 128 characters with no leading dot, and 255 characters in all.
|
|
886
|
+
Names with spaces, symbolic links and special files are refused at upload.
|
|
887
|
+
- **No `inspect`, `disable` or `enable` commands yet.** Use the API for those.
|
|
888
|
+
|
|
889
|
+
### Keeping secrets
|
|
890
|
+
|
|
891
|
+
`flightdeck secret` keeps private credentials, such as a Hugging Face token,
|
|
892
|
+
for your organization's flights. Secrets must be enabled for your organization
|
|
893
|
+
by your ThermAI operator. Any member can list names and details; only an
|
|
894
|
+
organization admin can create, rotate or revoke. Nothing ever prints a value
|
|
895
|
+
back.
|
|
896
|
+
|
|
897
|
+
```sh
|
|
898
|
+
flightdeck logout
|
|
899
|
+
flightdeck login --secrets
|
|
900
|
+
printf '%s' "$HF_TOKEN" | flightdeck secret create hf-token --value-stdin --description "Hub token"
|
|
901
|
+
flightdeck secret list
|
|
902
|
+
flightdeck secret rotate hf-token --from-file ./new-token.txt --expires-at 2027-06-30
|
|
903
|
+
flightdeck secret revoke hf-token --yes
|
|
904
|
+
```
|
|
905
|
+
|
|
906
|
+
- A value is never taken from the command line, where it would stay in shell
|
|
907
|
+
history. Pipe it in with `--value-stdin` (a terminal is refused, since the
|
|
908
|
+
value would be shown) or name a regular file with `--from-file`. One trailing
|
|
909
|
+
line break is removed. A value mistyped as an argument or an unknown option
|
|
910
|
+
is refused without being repeated.
|
|
911
|
+
- Names use lowercase letters and digits joined by `.`, `_` or `-`, up to 64
|
|
912
|
+
characters. Values are text up to 32 KiB. An organization holds up to 100.
|
|
913
|
+
- `--expires-at` takes a date or a date and time with a zone. A date alone means
|
|
914
|
+
midnight UTC at the start of that day. On `rotate`, leaving it out keeps the
|
|
915
|
+
current expiry and `--no-expiry` removes it. An expired secret must be rotated
|
|
916
|
+
with one of the two.
|
|
917
|
+
- `revoke` needs `--yes`. A running flight keeps the value it started with, and
|
|
918
|
+
revoking here does not revoke the token where it was issued: revoke it there
|
|
919
|
+
too.
|
|
920
|
+
- `--json` prints the metadata as JSON on every command.
|
|
921
|
+
|
|
922
|
+
The full guide is in [`docs/customer-secrets.md`](../docs/customer-secrets.md).
|
|
923
|
+
|
|
924
|
+
### Slurm differences
|
|
925
|
+
|
|
926
|
+
- Single-node jobs only; this is not a full Slurm scheduler or an HPC login node.
|
|
927
|
+
- Array `%N` concurrency throttles are not enforced, so `sbatch` refuses one
|
|
928
|
+
smaller than the array. `--ignore-array-limit` submits every task
|
|
929
|
+
immediately. Admission ceilings may reject excess tasks rather than queue
|
|
930
|
+
them for later admission. Review every warning before real submission.
|
|
931
|
+
- In an array task, `%j` is the array job id, not a per-task id; use `%a` in
|
|
932
|
+
`--output`/`--error` names so tasks don't share a file.
|
|
933
|
+
- Array tasks default to `slurm-%A_%a.out`, as in Slurm; write results under
|
|
934
|
+
`$OUTPUTS_DIR`. There is no shared university filesystem or implicit `module`,
|
|
935
|
+
Singularity, nested Docker, or cluster software environment.
|
|
936
|
+
- Hardware, runtime, image and credit must fit the operator-approved policy.
|
|
937
|
+
The image must meet the workload runtime contract: linux/amd64, and an
|
|
938
|
+
exec-form `ENTRYPOINT` if it sets one. Submission refuses an image that does
|
|
939
|
+
not.
|
|
940
|
+
|
|
941
|
+
### Reading logs completely
|
|
942
|
+
|
|
943
|
+
From `0.1.0-pilot.3` (containing the #99 pagination fix), `flightdeck logs`
|
|
944
|
+
drains all pages in the current stored-log snapshot, then returns. For a running
|
|
945
|
+
job it then reads the live tail, which the backend serves a bounded page at a
|
|
946
|
+
time. `--follow` drains those pages immediately, then polls every two seconds
|
|
947
|
+
until the job is terminal and its stored backlog is exhausted. Every new
|
|
948
|
+
invocation starts at the beginning; stdout/stderr retain their respective
|
|
949
|
+
output streams.
|
|
950
|
+
|
|
951
|
+
If the API stops answering while `--follow` is running (a connection failure, or
|
|
952
|
+
HTTP 502, 503 or 504, as happens for a few seconds when a release restarts the
|
|
953
|
+
backend), the CLI says so on stderr and retries for up to a minute before it
|
|
954
|
+
gives up. The flight is not affected. A read without `--follow` does not retry.
|
|
955
|
+
`flightdeck slurm-test` retries the same way while it waits for its flights and
|
|
956
|
+
when it reads their logs and outputs. It never sends the submission twice.
|
|
957
|
+
|
|
958
|
+
Each log stream is capped at 5 MiB. On Lambda, a finished flight's stream that
|
|
959
|
+
passes the cap keeps its first 5 MiB and its last 64 KiB (each cut between
|
|
960
|
+
characters), with a `flightdeck: log truncated` marker line between them when
|
|
961
|
+
anything was left out, so the end of the run and any
|
|
962
|
+
`flightdeck: failure reason` line survive. When a log is truncated,
|
|
963
|
+
`flightdeck logs` prints a one-line warning on stderr after the text; stdout
|
|
964
|
+
carries only the flight's own output.
|
|
965
|
+
|
|
966
|
+
Flight logs are kept for 30 days after they're written. For an older flight,
|
|
967
|
+
`flightdeck logs` prints nothing on stdout and says on stderr when the logs were
|
|
968
|
+
deleted.
|
|
969
|
+
|
|
970
|
+
**Rollout order:** deploy and verify the pagination-capable backend before
|
|
971
|
+
releasing/upgrading this CLI. A backend without the `has_more` response field
|
|
972
|
+
produces an explicit compatibility error and exit status 1, not a silently
|
|
973
|
+
partial read. Older `0.1.0-pilot.2` installations do not contain this fix;
|
|
974
|
+
merging source alone neither replaces that immutable package nor publishes a
|
|
975
|
+
new version. Older non-following clients still need upgrading for full paging.
|
|
976
|
+
|
|
977
|
+
### Watching flights live
|
|
978
|
+
|
|
979
|
+
`flightdeck atc` (alias `flightdeck tui`) opens a read-only, full-screen view in
|
|
980
|
+
the current terminal. Arrow keys select, Enter shows a flight's details and
|
|
981
|
+
events, `l` opens its logs (scroll, `f` to follow), `r` refreshes, Esc goes back
|
|
982
|
+
and `q` quits. It never cancels, submits or downloads anything.
|
|
983
|
+
|
|
984
|
+
```sh
|
|
985
|
+
flightdeck atc # your recent flights (last 50)
|
|
986
|
+
flightdeck atc <flight-id> # one flight
|
|
987
|
+
flightdeck atc <job-id> # an sbatch array submitted from this machine
|
|
988
|
+
flightdeck atc --name 'molsim*' # flights whose name matches, among your last 200
|
|
989
|
+
flightdeck atc --demo # a simulated array; no account or credits needed
|
|
990
|
+
```
|
|
991
|
+
|
|
992
|
+
`flightdeck sbatch` records each array's flight IDs under
|
|
993
|
+
`~/.flightdeck/batches/` and ends with `Watch live: flightdeck atc <job-id>`;
|
|
994
|
+
that job ID is the one to pass. It is a local sbatch job ID, not a flight ID:
|
|
995
|
+
`status`, `logs`, `outputs`, `download`, `delete-output` and `cancel` take the `fl-…` flight
|
|
996
|
+
IDs printed for each task. The record exists only on the machine that
|
|
997
|
+
submitted it (if it could not be written, sbatch says so; use `--name`). Name
|
|
998
|
+
patterns are case-sensitive. Leaving the view does not stop any job. In
|
|
999
|
+
scripts, keep using `status` and `logs`.
|
|
1000
|
+
|
|
1001
|
+
### Error handling and support
|
|
1002
|
+
|
|
1003
|
+
The CLI exits with status 1 on command failure. Expected authentication,
|
|
1004
|
+
network and HTTP failures have concise diagnostics instead of a Node stack.
|
|
1005
|
+
An already-configured login leaves the saved credential unchanged.
|
|
1006
|
+
|
|
1007
|
+
**Pilot.6 logout behavior (#143):** remove the local file first, then make
|
|
1008
|
+
one best-effort server revocation request with a three-second deadline, including
|
|
1009
|
+
the response body. Offline, suspended, failed or malformed API responses cannot
|
|
1010
|
+
retain or restore the local login. Local success exits 0; a warning separately
|
|
1011
|
+
states when remote revocation was not confirmed and other copies may remain
|
|
1012
|
+
valid. Use portal credential management or contact your organization admin.
|
|
1013
|
+
Failed local deletion exits 1, reports the problem and makes no API request.
|
|
1014
|
+
Missing local login is a successful no-op; malformed config is removed without
|
|
1015
|
+
trusting its contents for a request. No credential recovery/retry file is saved.
|
|
1016
|
+
`logout --local` removes only the local copy and makes no network request.
|
|
1017
|
+
|
|
1018
|
+
Older pilot.5 does not include this change; use `flightdeck logout --local`
|
|
1019
|
+
as a local-removal workaround before upgrading. It does not revoke remote
|
|
1020
|
+
access. The current production backend permits narrowly authenticated
|
|
1021
|
+
self-revocation under suspension; ordinary access remains blocked.
|
|
1022
|
+
|
|
1023
|
+
For unexpected faults, `flightdeck --debug <command>` adds allowlisted error
|
|
1024
|
+
types/codes and bundled CLI line locations, not raw stacks or error bodies.
|
|
1025
|
+
It is not a dry-run flag: only repeat an operation that is safe to repeat.
|
|
1026
|
+
Provide support with the CLI/Node versions, exit status, redacted command and
|
|
1027
|
+
flight ID if available. Review output before sharing: account details and
|
|
1028
|
+
workload logs/results can contain private information. Never send passwords,
|
|
1029
|
+
tokens, activation links or `~/.flightdeck/config.json`.
|
|
65
1030
|
|
|
66
1031
|
## Versions
|
|
67
1032
|
|
|
@@ -70,8 +1035,10 @@ flightdeck --version
|
|
|
70
1035
|
npm install -g @thermai/flightdeck-cli@<approved-version>
|
|
71
1036
|
```
|
|
72
1037
|
|
|
73
|
-
Stable releases use npm's `latest` tag. Explicit pilot/prerelease versions use
|
|
74
|
-
`next`; they do not automatically replace stable installations.
|
|
1038
|
+
Stable releases will use npm's `latest` tag. Explicit pilot/prerelease versions use
|
|
1039
|
+
`next`; they do not automatically replace stable installations. Currently
|
|
1040
|
+
`latest` still points at the initial pilot.1 bootstrap, not a qualified stable
|
|
1041
|
+
release; use `next` or the approved exact pilot version. Updating is an
|
|
75
1042
|
explicit install, not an automatic mutation performed when the CLI runs.
|
|
76
1043
|
Once a stable release exists, `npm install -g @thermai/flightdeck-cli` selects it.
|
|
77
1044
|
|