@thermai/flightdeck-cli 0.1.0-pilot.8 → 0.1.0-pilot.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +206 -16
- package/dist/index.js +7901 -2608
- package/package.json +6 -2
package/README.md
CHANGED
|
@@ -62,12 +62,22 @@ admin view. `flightdeck login` now defaults the credential name to
|
|
|
62
62
|
`flightdeck-cli@<hostname>` so a second machine no longer collides (#254), and
|
|
63
63
|
`flightdeck sbatch` prints a runnable status hint with the submitted flight ID
|
|
64
64
|
and labels the job ID on the ATC hint (#257, #259). The required backend is
|
|
65
|
-
already deployed; there are no new runtime dependencies.
|
|
66
|
-
|
|
67
|
-
After publication, use the generated `@next` command below, or pin
|
|
65
|
+
already deployed; there are no new runtime dependencies. It was published to
|
|
66
|
+
npm's `next` channel on 8 October 2026. Use the generated `@next` command below, or pin
|
|
68
67
|
`@thermai/flightdeck-cli@0.1.0-pilot.8`, and confirm `flightdeck --version`
|
|
69
68
|
reports `0.1.0-pilot.8`.
|
|
70
69
|
|
|
70
|
+
**`0.1.0-pilot.9`** adds managed images and datasets. `flightdeck image check`,
|
|
71
|
+
`push`, `list` and `inspect` check and publish workload images to your
|
|
72
|
+
organization's managed registry. `flightdeck image alias ls|set|rm` and
|
|
73
|
+
`image push --alias` manage channels, and `managed://<name>:<alias>` (including
|
|
74
|
+
the computed `latest`) can be submitted as an image. `flightdeck dataset push`,
|
|
75
|
+
`list` and `rm` manage datasets, and `submit --dataset` mounts them. `sbatch`
|
|
76
|
+
refuses an array `%N` limit Flightdeck cannot enforce unless
|
|
77
|
+
`--ignore-array-limit` is passed. Alias commands need a backend with image
|
|
78
|
+
aliases; the other commands work against the current backend. There are no new
|
|
79
|
+
runtime dependencies.
|
|
80
|
+
|
|
71
81
|
Members and organization administrators receive whole-credit balances,
|
|
72
82
|
estimates and actual customer charges, without provider-cost information.
|
|
73
83
|
Platform administrators retain their authorized monetary API fields. Credits
|
|
@@ -162,18 +172,21 @@ warnings. Keep `--dry-run`: this is a translation exercise, not the approved
|
|
|
162
172
|
pilot workload or permission to launch it. A dry run does not check provider
|
|
163
173
|
capacity, account policy/credit, bundle contents, image access or GPU execution.
|
|
164
174
|
|
|
165
|
-
For the real workload, obtain the reviewed private bundle, exact
|
|
175
|
+
For the real workload, obtain the reviewed private bundle, exact image
|
|
166
176
|
digest, provider selection and runtime from the operator. Review its dry run,
|
|
167
177
|
then use the supplied submission command only after execution approval. Do not
|
|
168
178
|
copy internal test data into public packages or partner bundles. Use explicit
|
|
169
179
|
`--files` selections from a clean directory; the submission default includes
|
|
170
|
-
the current directory
|
|
180
|
+
the current directory, less the default excludes listed under
|
|
181
|
+
[What goes into a bundle](#what-goes-into-a-bundle). Never include credentials
|
|
182
|
+
or private workstation files.
|
|
171
183
|
|
|
172
184
|
## Workload commands
|
|
173
185
|
|
|
174
186
|
After your account's policy/credit setup and agreed first-run approval:
|
|
175
187
|
|
|
176
188
|
```sh
|
|
189
|
+
flightdeck image check <image> # before using --base-image
|
|
177
190
|
flightdeck sbatch workload.sbatch
|
|
178
191
|
flightdeck status <flight-id>
|
|
179
192
|
flightdeck logs <flight-id> --follow
|
|
@@ -183,17 +196,44 @@ flightdeck cancel <flight-id>
|
|
|
183
196
|
```
|
|
184
197
|
|
|
185
198
|
Submission, including `flightdeck slurm-test`, can provision paid resources.
|
|
199
|
+
`submit` and `sbatch` print the deployment's output size limit first. Everything
|
|
200
|
+
in `/outputs` is stored as one compressed archive, and an archive over the limit
|
|
201
|
+
is not stored at all. `sbatch --dry-run` stays offline and states the limit
|
|
202
|
+
without a figure.
|
|
186
203
|
`sbatch --dry-run` only translates locally and does not submit. Flightdeck
|
|
187
204
|
supports a subset of single-node Slurm directives, not an arbitrary HPC cluster.
|
|
188
|
-
|
|
205
|
+
Without `--base-image`, a flight runs on the official Flightdeck base
|
|
206
|
+
(`ghcr.io/thermai-group/flightdeck-base`): Ubuntu 24.04 with Python 3 and pip,
|
|
207
|
+
and no PyTorch or CUDA, so supply Python packages with `--requirements` or use
|
|
208
|
+
your own image. A `--base-image` can be any linux/amd64 image that meets the
|
|
209
|
+
workload runtime contract (see `/docs/images`); it does not need the
|
|
210
|
+
Flightdeck runner. Before charging, submission refuses an image whose
|
|
211
|
+
`ENTRYPOINT` is shell form
|
|
212
|
+
(`shell_form_entrypoint`) or whose `uk.thermai.flightdeck.contract` label names
|
|
213
|
+
a contract version Flightdeck does not run (`image_unsupported_contract`).
|
|
214
|
+
Check an image locally first with `flightdeck image check`; see
|
|
215
|
+
[Checking an image before you submit](#checking-an-image-before-you-submit).
|
|
216
|
+
Environment variables (`--env`, `#SBATCH --export`) are checked before upload:
|
|
217
|
+
names are at most 64 characters, and names starting with `FLIGHTDECK_` belong
|
|
218
|
+
to the platform and are refused.
|
|
189
219
|
|
|
190
220
|
An array prints a separate Flightdeck ID for each accepted task. Record them
|
|
191
221
|
all: inspect, download and cancel each flight individually (`flightdeck atc
|
|
192
222
|
<job-id>` shows them together). Partial submission
|
|
193
223
|
can leave accepted flights running even if the command exits unsuccessfully;
|
|
194
|
-
do not blindly resubmit the whole array.
|
|
195
|
-
|
|
196
|
-
|
|
224
|
+
do not blindly resubmit the whole array. Downloads go to
|
|
225
|
+
`<output-dir>/<flight-id>/attempt-<n>/`, or `legacy/` for results from before
|
|
226
|
+
attempts were recorded, keeping each output's relative path. `--attempt <n>`
|
|
227
|
+
selects an earlier attempt. An interrupted download resumes when you run the same
|
|
228
|
+
command again, and a file only appears once its size and SHA-256 match the stored
|
|
229
|
+
output. Partial data is kept in `<file>.part` until then. A `.part` file that
|
|
230
|
+
the CLI did not leave for that output is refused, never deleted. A matching file is
|
|
231
|
+
skipped and a different one is never overwritten. A second download of the same
|
|
232
|
+
file stops with an error while the first is running. Stored results have no expiry
|
|
233
|
+
today. `outputs` and `download` print a warning on stderr when an output was
|
|
234
|
+
over the size limit or collection ended early, including a cancelled flight
|
|
235
|
+
whose outputs may not have been collected, so a short list isn't mistaken for
|
|
236
|
+
everything. With `--attempt`, only that attempt's size refusals are reported. Check terminal status and exit code as well as the expected artifacts;
|
|
197
237
|
successful submission or download alone does not mean the workload succeeded.
|
|
198
238
|
|
|
199
239
|
For an incident, stop submitting, cancel the known active flights, and contact
|
|
@@ -202,26 +242,176 @@ must confirm provider teardown and reconcile any outstanding charges before a
|
|
|
202
242
|
retry. Support contact, supported hours and credit increases come from your
|
|
203
243
|
agreed pilot handoff; this client does not promise unattended support.
|
|
204
244
|
|
|
245
|
+
### What goes into a bundle
|
|
246
|
+
|
|
247
|
+
`submit` and `sbatch` pack the current directory, or the `--files` paths, into
|
|
248
|
+
the bundle that is unpacked on the node. These are left out by default:
|
|
249
|
+
|
|
250
|
+
- version control: `.git`, `.hg`, `.svn`, `.bzr` (directories or files)
|
|
251
|
+
- environment and credential files: `.env*`, `.netrc`, `.pypirc`, `.npmrc`,
|
|
252
|
+
`.git-credentials`, `.ssh/`, `.aws/`, `.gnupg/`, `.docker/`, `.kube/`,
|
|
253
|
+
`.flightdeck/`
|
|
254
|
+
- key files: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.ppk`, `id_rsa`, `id_dsa`,
|
|
255
|
+
`id_ecdsa`, `id_ed25519`, `id_ecdsa_sk`, `id_ed25519_sk`
|
|
256
|
+
- virtualenvs: `venv/`, `.venv/` and any directory containing `pyvenv.cfg`
|
|
257
|
+
|
|
258
|
+
These match regardless of case, so `.ENV` and `ID_RSA` are left out too.
|
|
259
|
+
|
|
260
|
+
A `.flightdeckignore` in the current directory adds rules in `.gitignore`
|
|
261
|
+
syntax, and `!<path>` includes something a default would leave out. Only a `!`
|
|
262
|
+
rule without wildcards lifts a default; one such as `!*/` re-includes only what
|
|
263
|
+
your own rules excluded:
|
|
264
|
+
|
|
265
|
+
```gitignore
|
|
266
|
+
checkpoints/*
|
|
267
|
+
!checkpoints/final.pt
|
|
268
|
+
!.env.example
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
As with `.gitignore`, a path under an excluded directory cannot be re-included
|
|
272
|
+
on its own; re-include the directory, or exclude its contents with `dir/*`
|
|
273
|
+
instead. Naming an excluded path with `--files` does not include it (a shell
|
|
274
|
+
glob such as `--files *` expands to every name); use `!<path>` in
|
|
275
|
+
`.flightdeckignore`. A `--files` path that goes through a symlinked directory
|
|
276
|
+
(`alias/credentials` with `alias -> .aws`) is checked under the path it resolves
|
|
277
|
+
to as well, so the `!<path>` must name the resolved path; one that resolves
|
|
278
|
+
outside the current directory is refused. Symlinks that dangle, whose target is not in the bundle
|
|
279
|
+
(such as `data -> /scratch/...`), or whose target is an absolute path are
|
|
280
|
+
skipped with a warning because they cannot resolve on the node; use relative
|
|
281
|
+
links within the directory. To use a dataset that lives outside it, copy it
|
|
282
|
+
in or stage it on the node another way. The file count and size print before upload, and
|
|
283
|
+
`sbatch --dry-run` shows the same summary.
|
|
284
|
+
|
|
285
|
+
### Checking an image before you submit
|
|
286
|
+
|
|
287
|
+
`flightdeck image check <image>` checks an image with your local Docker, so you
|
|
288
|
+
learn before you submit whether Flightdeck is likely to admit it and how it
|
|
289
|
+
behaves. It
|
|
290
|
+
needs no login. When you are logged in, it reads the platform's current runtime
|
|
291
|
+
facts (the supervisor release and the host drivers) from the API; otherwise it
|
|
292
|
+
uses the ones built into the CLI.
|
|
293
|
+
|
|
294
|
+
```sh
|
|
295
|
+
flightdeck image check ghcr.io/your-org/your-image:1.0
|
|
296
|
+
flightdeck image check ghcr.io/your-org/your-image:1.0 --json
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
It reports each check as PASS, WARN, FAIL, INFO, SKIP or ERROR (the check
|
|
300
|
+
could not run), with a one-line reason, and ends with a verdict:
|
|
301
|
+
|
|
302
|
+
- **Submission rules**, the same code the API uses: `linux/amd64`, a docker.io,
|
|
303
|
+
ghcr.io or nvcr.io reference within 255 characters once pinned to its digest,
|
|
304
|
+
and the `uk.thermai.flightdeck.contract` label. If Docker's copy is for
|
|
305
|
+
another platform (Docker on Apple silicon pulls `arm64`), the check asks the
|
|
306
|
+
registry whether it has the `linux/amd64` image Flightdeck runs: if it does,
|
|
307
|
+
that is a warning and the dry run is skipped; if not, it fails. The check
|
|
308
|
+
never re-pulls or retags your copy. A shell-form `ENTRYPOINT` is a
|
|
309
|
+
warning: Flightdeck refuses it for flights with a command, and runs it as it
|
|
310
|
+
is for image-only flights. `tini` or `dumb-init` as the `ENTRYPOINT` is a
|
|
311
|
+
warning too, because the supervisor already does their job.
|
|
312
|
+
- **Registry**: Flightdeck reads the image from its registry, without
|
|
313
|
+
credentials. A local image that is not in its registry yet is a warning, and
|
|
314
|
+
so is one the registry will not serve without credentials (the check asks it
|
|
315
|
+
anonymously, with `docker manifest inspect` and an empty Docker
|
|
316
|
+
configuration). New GitHub packages are private until you make them public.
|
|
317
|
+
- **Dry run**: the image starts the way a flight starts, under the Flightdeck
|
|
318
|
+
supervisor with `/workspace` and `/outputs` mounted, CPU only unless you pass
|
|
319
|
+
`--gpu`. `--gpu` attaches GPUs with Docker's `--gpus`; Flightdeck's hosts use
|
|
320
|
+
CDI, so the dry run does not test a provider's GPUs or driver. It runs a command as the image's `USER`; an `ENTRYPOINT` that ignores
|
|
321
|
+
the command fails. It installs a requirements file with no packages in it; a
|
|
322
|
+
missing or unusable `pip` is a warning, since only a bundle with a
|
|
323
|
+
requirements file needs one. It stops a running command with `SIGTERM` and
|
|
324
|
+
expects it to exit within the 30-second grace. A missing `bash` is a warning,
|
|
325
|
+
because commands then run under `sh` and sbatch scripts need `bash`.
|
|
326
|
+
- **CUDA**, from `CUDA_VERSION`, `libcudart`, pip's CUDA runtime wheel and
|
|
327
|
+
`torch.version.cuda`: a CUDA newer than every published host driver runs is a
|
|
328
|
+
warning. An image without CUDA is fine.
|
|
329
|
+
- **Secrets**: the image's file names are listed through `docker export`, without
|
|
330
|
+
extracting anything, and matched against the credential and key patterns in
|
|
331
|
+
[What goes into a bundle](#what-goes-into-a-bundle). Empty files, CA
|
|
332
|
+
certificates and the test keys that packages ship are skipped.
|
|
333
|
+
|
|
334
|
+
The exit status is 0 when nothing failed (warnings allowed), 1 when a check
|
|
335
|
+
failed, and 2 when the check itself could not run, for example without Docker
|
|
336
|
+
or when the pull failed. `--static-only` runs nothing from the image, and
|
|
337
|
+
`--supervisor <image>` runs the dry run under another supervisor image.
|
|
338
|
+
|
|
339
|
+
The verdict says what the result means for submission:
|
|
340
|
+
|
|
341
|
+
- **PASS**, with every check run: Flightdeck will admit the image, and it ran
|
|
342
|
+
commands as the workload runtime contract describes.
|
|
343
|
+
- **PASS for the checks that ran**: nothing failed, but the verdict lists what
|
|
344
|
+
was not checked, such as the dry run under `--static-only`.
|
|
345
|
+
- **WARN, "Flightdeck may refuse this image"**: a warning that admission acts
|
|
346
|
+
on, named in the verdict. The image is not in its registry yet, the registry
|
|
347
|
+
will not serve it without credentials, or its `ENTRYPOINT` is shell form,
|
|
348
|
+
which is refused for every flight with a command.
|
|
349
|
+
- **WARN** otherwise: Flightdeck will admit the image; read the warnings.
|
|
350
|
+
- **FAIL**: Flightdeck would refuse the image, or its flights would not run.
|
|
351
|
+
|
|
352
|
+
If Docker does not have the image, the check pulls it for `linux/amd64` and
|
|
353
|
+
leaves it for you; it never removes an image you already had. Everything else
|
|
354
|
+
it creates is removed when it finishes, fails or is interrupted with Ctrl-C:
|
|
355
|
+
its containers (named `flightdeck-image-check-<id>-…` and labelled
|
|
356
|
+
`uk.thermai.flightdeck.image-check`), its temporary directory, and the
|
|
357
|
+
supervisor image if the check pulled it (unless another image check is
|
|
358
|
+
running, which may still need it). After Ctrl-C the check waits for Docker to
|
|
359
|
+
remove its containers, up to a minute each, and a second Ctrl-C does not cut
|
|
360
|
+
that short. If the check is killed outright, list what it left with
|
|
361
|
+
`docker ps -a --filter label=uk.thermai.flightdeck.image-check`, remove each with
|
|
362
|
+
`docker rm -f <name>`, and delete any `flightdeck-image-check-*` directory in
|
|
363
|
+
your temporary directory. On Apple silicon and other machines that are not
|
|
364
|
+
amd64, the image runs under emulation, which is slow.
|
|
365
|
+
|
|
366
|
+
The check does not test your code or the provider's GPUs, so run a short flight
|
|
367
|
+
before real work.
|
|
368
|
+
|
|
369
|
+
### Publishing a private image
|
|
370
|
+
|
|
371
|
+
`flightdeck image push <local-image> --name <name> --version <label>` publishes
|
|
372
|
+
a local image to your organization's private registry. `image list` and
|
|
373
|
+
`image inspect <version-id> --format reference` show it and print the reference
|
|
374
|
+
to pass to `--base-image`. Log in with `flightdeck login --images` first (run
|
|
375
|
+
`logout` before it if you are already logged in). Rerun the same push to resume.
|
|
376
|
+
`--base-image managed://<name>:latest` runs the newest ready version, and
|
|
377
|
+
`image push --alias ci` or `image alias set <name> ci --version <label>` moves a channel
|
|
378
|
+
you can run as `managed://<name>:ci`. The full guide, including errors and limits, is in
|
|
379
|
+
[`docs/managed-images.md`](../docs/managed-images.md).
|
|
380
|
+
|
|
205
381
|
### Slurm differences
|
|
206
382
|
|
|
207
383
|
- Single-node jobs only; this is not a full Slurm scheduler or an HPC login node.
|
|
208
|
-
- Array `%N` concurrency throttles are not enforced
|
|
384
|
+
- Array `%N` concurrency throttles are not enforced, so `sbatch` refuses one
|
|
385
|
+
smaller than the array. `--ignore-array-limit` submits every task
|
|
209
386
|
immediately. Admission ceilings may reject excess tasks rather than queue
|
|
210
387
|
them for later admission. Review every warning before real submission.
|
|
211
|
-
-
|
|
388
|
+
- In an array task, `%j` is the array job id, not a per-task id; use `%a` in
|
|
389
|
+
`--output`/`--error` names so tasks don't share a file.
|
|
390
|
+
- Array tasks default to `slurm-%A_%a.out`, as in Slurm; write results under
|
|
212
391
|
`$OUTPUTS_DIR`. There is no shared university filesystem or implicit `module`,
|
|
213
392
|
Singularity, nested Docker, or cluster software environment.
|
|
214
393
|
- Hardware, runtime, image and credit must fit the operator-approved policy.
|
|
215
|
-
|
|
394
|
+
The image must meet the workload runtime contract: linux/amd64, and an
|
|
395
|
+
exec-form `ENTRYPOINT` if it sets one. Submission refuses an image that does
|
|
396
|
+
not.
|
|
216
397
|
|
|
217
398
|
### Reading logs completely
|
|
218
399
|
|
|
219
400
|
From `0.1.0-pilot.3` (containing the #99 pagination fix), `flightdeck logs`
|
|
220
401
|
drains all pages in the current stored-log snapshot, then returns. For a running
|
|
221
|
-
job it
|
|
222
|
-
drains those pages immediately, then polls every two seconds
|
|
223
|
-
terminal and its stored backlog is exhausted. Every new
|
|
224
|
-
the beginning; stdout/stderr retain their respective
|
|
402
|
+
job it then reads the live tail, which the backend serves a bounded page at a
|
|
403
|
+
time. `--follow` drains those pages immediately, then polls every two seconds
|
|
404
|
+
until the job is terminal and its stored backlog is exhausted. Every new
|
|
405
|
+
invocation starts at the beginning; stdout/stderr retain their respective
|
|
406
|
+
output streams.
|
|
407
|
+
|
|
408
|
+
Each log stream is capped at 5 MiB. On Lambda, a finished flight's stream that
|
|
409
|
+
passes the cap keeps its first 5 MiB and its last 64 KiB (each cut between
|
|
410
|
+
characters), with a `flightdeck: log truncated` marker line between them when
|
|
411
|
+
anything was left out, so the end of the run and any
|
|
412
|
+
`flightdeck: failure reason` line survive. When a log is truncated,
|
|
413
|
+
`flightdeck logs` prints a one-line warning on stderr after the text; stdout
|
|
414
|
+
carries only the flight's own output.
|
|
225
415
|
|
|
226
416
|
**Rollout order:** deploy and verify the pagination-capable backend before
|
|
227
417
|
releasing/upgrading this CLI. A backend without the `has_more` response field
|