fair-cli 0.10.2__tar.gz → 0.11.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {fair_cli-0.10.2 → fair_cli-0.11.0}/CHANGELOG.md +99 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/CITATION.cff +2 -2
- {fair_cli-0.10.2 → fair_cli-0.11.0}/PKG-INFO +56 -10
- {fair_cli-0.10.2 → fair_cli-0.11.0}/README.md +54 -9
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/cli.py +132 -40
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/configuration/__init__.py +34 -16
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/exceptions.py +13 -5
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/identifiers.py +53 -6
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/register.py +130 -1
- fair_cli-0.11.0/fair/registry/lifecycle.py +821 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/registry/server.py +194 -51
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/registry/storage.py +150 -24
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/registry/sync.py +41 -8
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/session.py +350 -146
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/user_config/__init__.py +135 -9
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/user_config/globbing.py +4 -3
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/user_config/validation.py +20 -2
- {fair_cli-0.10.2 → fair_cli-0.11.0}/pyproject.toml +2 -1
- {fair_cli-0.10.2 → fair_cli-0.11.0}/LICENSE +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/__init__.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/common.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/configuration/validation.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/files.txt +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/history.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/logging.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/registry/__init__.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/registry/file_formats.json +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/registry/file_types.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/registry/requests.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/registry/versioning.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/run.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/staging.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/templates/__init__.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/templates/config.jinja +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/templates/hist.jinja +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/testing.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/utilities.py +0 -0
- {fair_cli-0.10.2 → fair_cli-0.11.0}/fair/virtualenv.py +0 -0
|
@@ -1,3 +1,102 @@
|
|
|
1
|
+
# 2026-10-19 [v0.11.0](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.11.0)
|
|
2
|
+
|
|
3
|
+
## Changed behaviour
|
|
4
|
+
- A command that starts the local registry stops it again when the last command using it finishes; it used to leave
|
|
5
|
+
it running. Each command holds a session on the registry while it runs. `fair registry start` keeps it running
|
|
6
|
+
until `fair registry stop`, and run while a command has it running keeps that registry rather than refusing.
|
|
7
|
+
- `fair registry start` with no `--port`, and a command that starts the registry, use the configured port, where
|
|
8
|
+
they used 8000 whatever the configuration said.
|
|
9
|
+
- An install's registry runs on one host at a time: where its directory is shared between machines, starting,
|
|
10
|
+
stopping or using it from another machine is refused, naming the machine running it and the file that records it.
|
|
11
|
+
- `fair purge --all`, `fair registry uninstall` and `fair registry install --force` stop the registry first, and
|
|
12
|
+
remove nothing while a command is using it or it runs on another machine.
|
|
13
|
+
- Two versions of the CLI sharing one registry install are not supported: one of an earlier version neither records
|
|
14
|
+
its sessions nor respects this one's.
|
|
15
|
+
- Records left behind when the registry or a session ends outside `fair` - a restart, a crash, a killed process - are
|
|
16
|
+
removed with a line saying what was removed and why, naming `fair registry start` where that had started the
|
|
17
|
+
registry. A record left half-written is removed the same way. One that cannot be removed - on Windows, a file
|
|
18
|
+
another program has open - is waited for briefly, then named and left, and the command carries on; one that
|
|
19
|
+
cannot be written is waited for the same way, then is an error naming it. A session whose process cannot be
|
|
20
|
+
checked is taken to be running, and says so where it stops the registry being stopped.
|
|
21
|
+
- `fair init` asks for the user's identifier before their email, and offers as the email the address the
|
|
22
|
+
identifier's record makes public - an ORCID's or a GitHub account's. Anything answering its questions in order
|
|
23
|
+
must put the email after the identifier.
|
|
24
|
+
- A `read:` wildcard's own entry is kept in the working config beside the entries it expands to, as a `write:`
|
|
25
|
+
wildcard's is, so that an API can tell a pattern that matched nothing from one never given. `fair pull` fetches
|
|
26
|
+
nothing for it. Its version - its matches' highest, the version it names, or 0.0.0 - means nothing.
|
|
27
|
+
- A `*` in a wildcard no longer matches `[`, `]`, `?` or `*`, which the registry refuses in a name, so a pattern
|
|
28
|
+
never matches itself.
|
|
29
|
+
|
|
30
|
+
## Added
|
|
31
|
+
- `fair registry session open --pid <pid> --json` and `fair registry session close <id>`, for a program using the
|
|
32
|
+
registry outside a `fair` command, such as a language API. A script run by `fair run` is told the run's session
|
|
33
|
+
in `FDP_SESSION`.
|
|
34
|
+
- `fair registry status` says who holds the registry and which processes are using it.
|
|
35
|
+
- `fair run --rehearse` prepares a job as a run would without running it - no git check, nothing staged - and prints
|
|
36
|
+
its files as one JSON object, `{"job_dir", "config", "script", "data_store", "token_file"}`, for a language API's
|
|
37
|
+
rehearsal of a run.
|
|
38
|
+
- The README says which parts of the CLI's configuration a program such as a language API may read.
|
|
39
|
+
- A `register:` entry for an external object may have `no_file: true` in place of `root`, `path` and `file_type`,
|
|
40
|
+
for a source with no file of its own, such as a data store a run queries: `fair pull` registers it as a data
|
|
41
|
+
product with no file, which a run may read. Its `version` must be given.
|
|
42
|
+
- `run_metadata.shell` may give flags for the shell's program after its name: `shell: julia --project=@.` runs a
|
|
43
|
+
Julia script in the repository's project. A shell the CLI does not know - an empty `shell:` included - is
|
|
44
|
+
refused, naming those it does, where it stopped with a `KeyError` or a `TypeError`.
|
|
45
|
+
|
|
46
|
+
## Fixed
|
|
47
|
+
- `fair purge --all` answered "no" removed the project's `.fair` folder, and with it a data store kept there; it now
|
|
48
|
+
removes nothing. `fair purge --yes` asked its question anyway, and now answers every question as the README says.
|
|
49
|
+
- `fair registry install --force` removed the old registry before it had the new one, so a `--version` that does not
|
|
50
|
+
exist left no working registry. The old one is now kept until the new one is built, and put back if the install
|
|
51
|
+
fails; a first install that fails leaves nothing behind.
|
|
52
|
+
- `fair push` and `fair pull` left out a code repository's release (`CodeRepoRelease`), so the remote's record of a
|
|
53
|
+
run or a data product lost it. The release now goes with its repository's object.
|
|
54
|
+
- A data product on a code repository's object, such as a release of the code registered with its DOI, was pushed
|
|
55
|
+
with the repository's web page as its file, and could not then be pulled. Its location is the repository at a
|
|
56
|
+
commit, and it is now pushed and pulled as that, with nothing downloaded.
|
|
57
|
+
- A `register:` entry with no version was registered again at the next patch version on every `fair pull`, whether
|
|
58
|
+
or not its file had changed, and `fair run` then read a version that did not exist (#317). The highest version is
|
|
59
|
+
now kept while the file is the same, and `fair run` reads it. `${{LATEST}}` on such an entry keeps the highest
|
|
60
|
+
version and refuses a changed file.
|
|
61
|
+
- `fair run` did not check that `fair pull` had registered what a `register:` entry names, so the model found out;
|
|
62
|
+
it now refuses to start.
|
|
63
|
+
- A ROR identifier (`https://ror.org/...`) of an author or a source was checked by fetching ROR's website, which
|
|
64
|
+
refuses automated requests that come too often (429), and was then rejected. It is now checked through ROR's API.
|
|
65
|
+
- On Windows, `fair registry start` could fail - no response, or no token - when the registry's start script gave up
|
|
66
|
+
waiting for the server, leaving the server running. The CLI now waits for the server itself, up to 30 seconds, and
|
|
67
|
+
writes the registry's token if the script did not.
|
|
68
|
+
- A command that crashed, or a reboot after `fair registry start`, left a session file behind that made every later
|
|
69
|
+
command skip starting the registry and `fair registry stop` refuse. A session now names its process, and one whose
|
|
70
|
+
process has gone is removed. This needs a new dependency, `psutil`.
|
|
71
|
+
- A failed `fair registry start` left a session file behind, and a server on the port that never answered was waited
|
|
72
|
+
on for ever; neither now happens. Another install's registry on the port is named as such.
|
|
73
|
+
- An internal error, and a registry's refusal of a request (HTTP 403), printed their message and exited 0, so a
|
|
74
|
+
script or a CI step saw success. Both now exit non-zero, the refusal shown as an error.
|
|
75
|
+
- `fair pull` of a `register:` entry at a version given outright, registered already with another file, copied the
|
|
76
|
+
new file over the registered one before the registry refused it, so the registered file no longer matched its
|
|
77
|
+
record. It is now refused before anything is copied; the same file is passed over, as before.
|
|
78
|
+
- `fair init --ci` named the remote user `FAIRDataPipeline`, which a freshly built remote registry does not have, so
|
|
79
|
+
`fair push` stopped. It now takes the user the remote's token belongs to, where the remote answers.
|
|
80
|
+
- `fair run` refused a detached checkout - a tag, a commit, a pull request on CI - unless given `--dirty`. It now runs
|
|
81
|
+
where the commit is one the remote has, on a branch or a tag, and refuses one the remote does not have.
|
|
82
|
+
`${{GIT_BRANCH}}` there is an error saying there is no branch.
|
|
83
|
+
- A `fair init` that failed - a token the remote refused, a registry that would not start, Ctrl-C other than at a
|
|
84
|
+
question, a `--using` file it refused - left `.fair` and an empty global configuration behind, so `fair init` again
|
|
85
|
+
failed too (#272). It now puts back what it changed, and `fair init` starts afresh where an earlier one left `.fair`
|
|
86
|
+
unfinished, or the global configuration empty.
|
|
87
|
+
- `fair init` failing in a subdirectory of a project could write the configuration it had refused over the project's;
|
|
88
|
+
the project's is now left as it was.
|
|
89
|
+
- `fair init` in a subdirectory of a project made the project's staging file and not its own, so a later command
|
|
90
|
+
there that staged something failed; it now makes its own.
|
|
91
|
+
- A local registry that answered every request with an internal error (HTTP 500) was taken for one that works, and a
|
|
92
|
+
command failed at its first request. Starting it, or a command that finds it running, now says so and names its
|
|
93
|
+
log; `fair registry stop` still stops it.
|
|
94
|
+
- `fair init` with an ORCID whose record keeps its holder's name private, or gives no family name, stopped with a
|
|
95
|
+
`TypeError`. It now asks for a name the record does not give, and takes a missing family name as empty. An author
|
|
96
|
+
in a `register:` entry whose ORCID keeps its name private is registered by its identifier. A one-word name - given
|
|
97
|
+
with no identifier, or to `fair config user.name` - failed the configuration's validation and broke the job logs;
|
|
98
|
+
its family name is now empty too.
|
|
99
|
+
|
|
1
100
|
# 2026-10-05 [v0.10.2](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.10.2)
|
|
2
101
|
|
|
3
102
|
## Changed behaviour
|
|
@@ -26,7 +26,7 @@ authors:
|
|
|
26
26
|
affiliation: University of Glasgow
|
|
27
27
|
orcid: https://orcid.org/0000-0002-4424-9890
|
|
28
28
|
|
|
29
|
-
date-released: 2026-10-
|
|
29
|
+
date-released: 2026-10-19
|
|
30
30
|
keywords:
|
|
31
31
|
- FAIR Data Pipeline
|
|
32
32
|
- FAIR
|
|
@@ -36,4 +36,4 @@ license: BSD-2-Clause
|
|
|
36
36
|
message: If you use this software, please cite it using these metadata.
|
|
37
37
|
repository-code: https://github.com/FAIRDataPipeline/FAIR-CLI/
|
|
38
38
|
title: "The FAIR Data Pipeline command line tool"
|
|
39
|
-
version: 0.
|
|
39
|
+
version: 0.11.0
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: fair-cli
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.11.0
|
|
4
4
|
Summary: Synchronization interface for the SCRC FAIR Data Pipeline registry
|
|
5
5
|
License: BSD-2-Clause
|
|
6
6
|
License-File: LICENSE
|
|
@@ -31,6 +31,7 @@ Requires-Dist: click (>=8.0.0,<9.0.0)
|
|
|
31
31
|
Requires-Dist: email-validator (>=1.1.3,<2.0.0)
|
|
32
32
|
Requires-Dist: fake-useragent (>=1,<2)
|
|
33
33
|
Requires-Dist: pre-commit (>=3,<4)
|
|
34
|
+
Requires-Dist: psutil (>=5.9.0)
|
|
34
35
|
Requires-Dist: pydantic[email] (>=1.9.0,<2.0.0)
|
|
35
36
|
Requires-Dist: requests (>=2.23.0,<3.0.0)
|
|
36
37
|
Requires-Dist: rich (>=10.2.3,<12.0.0)
|
|
@@ -135,7 +136,7 @@ fair registry install --force --version <branch>
|
|
|
135
136
|
|
|
136
137
|
Without `--force` an existing registry is left as it is and the command stops. `fair registry uninstall`, which asks first, followed by `fair registry install` does the same in two steps, and is the way with `fair-cli` 0.10.1 or earlier, where `--force` is refused.
|
|
137
138
|
|
|
138
|
-
|
|
139
|
+
With `--force` the old registry is kept until the new one is built. If the install fails - a `--version` that does not exist, no network, a build that fails - the old registry is put back as it was, and the configuration still names it. A running registry is stopped first, and nothing is replaced while a command is using it.
|
|
139
140
|
|
|
140
141
|
The registry is installed in `~/.fair/registry`, or in the directory given with `--directory`, which has to be given again to replace a registry that was installed elsewhere. A reinstall replaces that directory:
|
|
141
142
|
|
|
@@ -177,6 +178,8 @@ By default the shell used to execute a process is `sh` or `batch` for UNIX and W
|
|
|
177
178
|
| `sh` | `sh -e {0}` |
|
|
178
179
|
| `batch` | `{0}` |
|
|
179
180
|
|
|
181
|
+
Flags for a shell's program follow its name, separated by spaces, and go straight after the program: `shell: julia --project=@.` runs `julia --project=@. {0}`, starting Julia in the repository's project, and `shell: bash -x` runs `bash -x -eo pipefail {0}`. A flag cannot contain a space, and `batch`, which has no program, takes none.
|
|
182
|
+
|
|
180
183
|
A full description of `config.yaml` files can be found [here](https://www.fairdatapipeline.org/docs/interface/config/).
|
|
181
184
|
|
|
182
185
|
## Available Commands
|
|
@@ -185,8 +188,12 @@ A full description of `config.yaml` files can be found [here](https://www.fairda
|
|
|
185
188
|
|
|
186
189
|
Initialises a new FAIR repository within the given directory. This should ideally be the same location as the `.git` folder for the current project, however during setup an option is given to specify an alternative. The command will ask the user a series of questions which will provide metadata for tracking run authors, and also allow for the creation of a starter `config.yaml` file. Initialisation will also configure the CLI itself.
|
|
187
190
|
|
|
191
|
+
The user's identifier is asked for before their email: where the identifier's record makes an address public - an ORCID's, or a GitHub account's - that address is offered as the email, and with none given a GitHub no-reply address is used.
|
|
192
|
+
|
|
188
193
|
In a repository that is already initialised the command asks nothing and leaves the repository as it is. It registers the repository's user in the local registry, which a registry [reinstalled](#upgrading-and-reinstalling) since the repository was initialised no longer holds.
|
|
189
194
|
|
|
195
|
+
An initialisation that fails, or is interrupted, before the repository's configuration is written leaves things as they were: no `.fair` folder, and the global configuration unchanged. One that fails after it - registering the user in a local registry that is not running, say - leaves the repository initialised, and running `fair init` again completes it. A `.fair` folder with no configuration in it, as a process killed during `fair init` leaves, is removed and the repository initialised afresh, unless it holds more than that, when it is named and left for you to look through.
|
|
196
|
+
|
|
190
197
|
#### Custom CLI Configuration
|
|
191
198
|
After setup is complete, the current CLI configuration can also be saved using the command:
|
|
192
199
|
```
|
|
@@ -230,7 +237,7 @@ For integration into a CI workflow, the setup can be skipped by running:
|
|
|
230
237
|
fair init --ci
|
|
231
238
|
```
|
|
232
239
|
|
|
233
|
-
which will create temporary directories for some of the required location paths.
|
|
240
|
+
which will create temporary directories for some of the required location paths. The remote registry is expected at `http://127.0.0.1:8001/api/`, with its token in the file `token` in the directory the command is run from; the user it records for the remote is the one that token belongs to, or `FAIRDataPipeline` where the remote cannot be asked.
|
|
234
241
|
|
|
235
242
|
|
|
236
243
|
### `run`
|
|
@@ -257,9 +264,25 @@ note the command itself must be quoted as it is a single argument.
|
|
|
257
264
|
|
|
258
265
|
By default the CLI will not allow the user to perform a run if the state of the analysis repository is such that it is behind the git remote, or contains uncommitted changes. To override this behaviour use the `--dirty` flag.
|
|
259
266
|
|
|
267
|
+
A detached checkout - a tag or a commit checked out, or a pull request on a CI system - has no branch to compare, so there the commit has to be one the remote has, on a branch or a tag; `--dirty` lets one the remote does not have through.
|
|
268
|
+
|
|
269
|
+
To prepare a job without running it, as a language API's rehearsal of a run does, use `--rehearse`:
|
|
270
|
+
|
|
271
|
+
```sh
|
|
272
|
+
fair run --rehearse
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
It writes the job's folder, its working config and its script exactly as a run would, and runs nothing. It makes no git check - the commit is recorded as it is, marked `-dirty` where there are uncommitted changes - and stages nothing. Its only output on stdout is one JSON object naming the job's files, its log going to stderr:
|
|
276
|
+
|
|
277
|
+
```json
|
|
278
|
+
{"job_dir": "...", "config": "...", "script": "...", "data_store": "...", "token_file": "..."}
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
`token_file` is the file holding the local registry's token; the token itself is never printed. Like every command, a rehearsal holds the local registry only while it runs, so a program that goes on to run the job's script holds a [session](#registry) of its own around both. `fair run --ci` also prepares a job without running it, but stages it, warns about the repository's state, and prints the job's folder as its last line.
|
|
282
|
+
|
|
260
283
|
### `pull`
|
|
261
284
|
|
|
262
|
-
The command `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. Any data required for a run is downloaded and stored within the local registry. In addition any data products requested that are available on the remote registry are pulled locally.
|
|
285
|
+
The command `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. An `external_object` entry with `no_file: true`, and no `root`, `path` or `file_type`, registers a source that has no file of its own, such as a data store that a run queries to extract data, as a data product with no file; it must give its `version`. Any data required for a run is downloaded and stored within the local registry. In addition any data products requested that are available on the remote registry are pulled locally.
|
|
263
286
|
|
|
264
287
|
```sh
|
|
265
288
|
fair pull /path/to/config.yaml
|
|
@@ -323,23 +346,34 @@ fair purge --yes
|
|
|
323
346
|
|
|
324
347
|
### `registry`
|
|
325
348
|
|
|
326
|
-
|
|
349
|
+
A command that uses the local registry - `run`, `pull`, `push`, `status`, `list`, `find`, `identify`, `reset` - starts it, on the port the configuration names, if it is not running, and a registry started that way is stopped when the last command using it finishes. Each command holds a *session* on the registry while it runs, and the registry is not stopped while any session is open.
|
|
327
350
|
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
The commands:
|
|
351
|
+
To keep the registry running - to look at it in the browser, or to save each command the few seconds a start takes - start it yourself:
|
|
331
352
|
|
|
332
353
|
```sh
|
|
333
354
|
fair registry start
|
|
334
355
|
```
|
|
335
356
|
|
|
336
|
-
|
|
357
|
+
It then runs until
|
|
337
358
|
|
|
338
359
|
```sh
|
|
339
360
|
fair registry stop
|
|
340
361
|
```
|
|
341
362
|
|
|
342
|
-
|
|
363
|
+
`fair registry start` takes the port in the configuration (`registries.local.uri` in `~/.fair/cli/cli-config.yaml`), or the one given with `--port`, which becomes the configured one when the registry starts there. Run while a command has the registry running, it keeps that registry running from then on rather than starting another. `fair registry stop` refuses while a command is using the registry; `--force` stops it anyway. `fair registry status` says where the registry is running, whether it was started by `fair registry start` or by a command, and which processes are using it.
|
|
364
|
+
|
|
365
|
+
An install's registry runs on one host at a time, since its database is a file in its directory: where that directory is shared between machines, as a home directory on a cluster may be, the registry cannot be started, stopped or used from a machine other than the one running it. The refusal names that machine, and the file in the registry's directory that records it. If that machine no longer runs the registry - it has shut down or restarted - delete the file and try again.
|
|
366
|
+
|
|
367
|
+
`fair purge --all`, `fair registry uninstall` and `fair registry install --force` stop the registry before removing it, and remove nothing while a command is using it or while it is running on another machine.
|
|
368
|
+
|
|
369
|
+
A program that uses the registry outside a `fair` command, such as a language API, can hold a session of its own:
|
|
370
|
+
|
|
371
|
+
```sh
|
|
372
|
+
fair registry session open --pid <process id> --json
|
|
373
|
+
fair registry session close <session id>
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
`session open` starts the registry if it is not running, and prints the session's id, the registry's address and the file holding its token, as one JSON object (`{"session": ..., "registry_url": ..., "token_file": ...}`). The session ends when it is closed or when the process it names ends. A script run by `fair run` is told the run's session in `FDP_SESSION`, and needs none of its own.
|
|
343
377
|
|
|
344
378
|
The registry can be installed using the CLI as well by running:
|
|
345
379
|
```sh
|
|
@@ -400,3 +434,15 @@ Within the `config.yaml` file, template variables can be specified by using the
|
|
|
400
434
|
| `GIT_REMOTE` | The URI of the git repository specified during setup |
|
|
401
435
|
| `GIT_TAG` | The latest tag on `git` |
|
|
402
436
|
|
|
437
|
+
## What a program may read of the CLI's configuration
|
|
438
|
+
|
|
439
|
+
A language API, or any other program working with the pipeline, may rely on these parts of the CLI's configuration files:
|
|
440
|
+
|
|
441
|
+
- `registries.local.uri` and `registries.local.token` in `~/.fair/cli/cli-config.yaml`: the local registry's API address, and the file holding its token.
|
|
442
|
+
- `registries.<name>.uri` and `registries.<name>.token` in a project's `.fair/cli-config.yaml`: the same for each remote registry, by its name - `origin`, or one added with `fair remote add`.
|
|
443
|
+
- A token file holds that registry's token alone, read as it is with surrounding white space stripped. Use it only for the registry whose entry names it.
|
|
444
|
+
- `git.remote` in a project's `.fair/cli-config.yaml`: the name of the project's git remote. Its URL is the git repository's to give (`git remote get-url <name>`); `git.remote_repo` beside it is a copy taken by `fair init`, and goes out of date when the remote moves.
|
|
445
|
+
- A directory with a `.fair` folder is a FAIR project, found by searching upwards from the working directory, as every `fair` command does.
|
|
446
|
+
|
|
447
|
+
Within a run the working config holds what a script needs - among it `run_metadata.local_data_registry_url` and `run_metadata.remote_repo`, the git remote's URL - and the script's environment holds `FDP_CONFIG_DIR`, `FDP_LOCAL_TOKEN` and `FDP_SESSION`.
|
|
448
|
+
|
|
@@ -89,7 +89,7 @@ fair registry install --force --version <branch>
|
|
|
89
89
|
|
|
90
90
|
Without `--force` an existing registry is left as it is and the command stops. `fair registry uninstall`, which asks first, followed by `fair registry install` does the same in two steps, and is the way with `fair-cli` 0.10.1 or earlier, where `--force` is refused.
|
|
91
91
|
|
|
92
|
-
|
|
92
|
+
With `--force` the old registry is kept until the new one is built. If the install fails - a `--version` that does not exist, no network, a build that fails - the old registry is put back as it was, and the configuration still names it. A running registry is stopped first, and nothing is replaced while a command is using it.
|
|
93
93
|
|
|
94
94
|
The registry is installed in `~/.fair/registry`, or in the directory given with `--directory`, which has to be given again to replace a registry that was installed elsewhere. A reinstall replaces that directory:
|
|
95
95
|
|
|
@@ -131,6 +131,8 @@ By default the shell used to execute a process is `sh` or `batch` for UNIX and W
|
|
|
131
131
|
| `sh` | `sh -e {0}` |
|
|
132
132
|
| `batch` | `{0}` |
|
|
133
133
|
|
|
134
|
+
Flags for a shell's program follow its name, separated by spaces, and go straight after the program: `shell: julia --project=@.` runs `julia --project=@. {0}`, starting Julia in the repository's project, and `shell: bash -x` runs `bash -x -eo pipefail {0}`. A flag cannot contain a space, and `batch`, which has no program, takes none.
|
|
135
|
+
|
|
134
136
|
A full description of `config.yaml` files can be found [here](https://www.fairdatapipeline.org/docs/interface/config/).
|
|
135
137
|
|
|
136
138
|
## Available Commands
|
|
@@ -139,8 +141,12 @@ A full description of `config.yaml` files can be found [here](https://www.fairda
|
|
|
139
141
|
|
|
140
142
|
Initialises a new FAIR repository within the given directory. This should ideally be the same location as the `.git` folder for the current project, however during setup an option is given to specify an alternative. The command will ask the user a series of questions which will provide metadata for tracking run authors, and also allow for the creation of a starter `config.yaml` file. Initialisation will also configure the CLI itself.
|
|
141
143
|
|
|
144
|
+
The user's identifier is asked for before their email: where the identifier's record makes an address public - an ORCID's, or a GitHub account's - that address is offered as the email, and with none given a GitHub no-reply address is used.
|
|
145
|
+
|
|
142
146
|
In a repository that is already initialised the command asks nothing and leaves the repository as it is. It registers the repository's user in the local registry, which a registry [reinstalled](#upgrading-and-reinstalling) since the repository was initialised no longer holds.
|
|
143
147
|
|
|
148
|
+
An initialisation that fails, or is interrupted, before the repository's configuration is written leaves things as they were: no `.fair` folder, and the global configuration unchanged. One that fails after it - registering the user in a local registry that is not running, say - leaves the repository initialised, and running `fair init` again completes it. A `.fair` folder with no configuration in it, as a process killed during `fair init` leaves, is removed and the repository initialised afresh, unless it holds more than that, when it is named and left for you to look through.
|
|
149
|
+
|
|
144
150
|
#### Custom CLI Configuration
|
|
145
151
|
After setup is complete, the current CLI configuration can also be saved using the command:
|
|
146
152
|
```
|
|
@@ -184,7 +190,7 @@ For integration into a CI workflow, the setup can be skipped by running:
|
|
|
184
190
|
fair init --ci
|
|
185
191
|
```
|
|
186
192
|
|
|
187
|
-
which will create temporary directories for some of the required location paths.
|
|
193
|
+
which will create temporary directories for some of the required location paths. The remote registry is expected at `http://127.0.0.1:8001/api/`, with its token in the file `token` in the directory the command is run from; the user it records for the remote is the one that token belongs to, or `FAIRDataPipeline` where the remote cannot be asked.
|
|
188
194
|
|
|
189
195
|
|
|
190
196
|
### `run`
|
|
@@ -211,9 +217,25 @@ note the command itself must be quoted as it is a single argument.
|
|
|
211
217
|
|
|
212
218
|
By default the CLI will not allow the user to perform a run if the state of the analysis repository is such that it is behind the git remote, or contains uncommitted changes. To override this behaviour use the `--dirty` flag.
|
|
213
219
|
|
|
220
|
+
A detached checkout - a tag or a commit checked out, or a pull request on a CI system - has no branch to compare, so there the commit has to be one the remote has, on a branch or a tag; `--dirty` lets one the remote does not have through.
|
|
221
|
+
|
|
222
|
+
To prepare a job without running it, as a language API's rehearsal of a run does, use `--rehearse`:
|
|
223
|
+
|
|
224
|
+
```sh
|
|
225
|
+
fair run --rehearse
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
It writes the job's folder, its working config and its script exactly as a run would, and runs nothing. It makes no git check - the commit is recorded as it is, marked `-dirty` where there are uncommitted changes - and stages nothing. Its only output on stdout is one JSON object naming the job's files, its log going to stderr:
|
|
229
|
+
|
|
230
|
+
```json
|
|
231
|
+
{"job_dir": "...", "config": "...", "script": "...", "data_store": "...", "token_file": "..."}
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
`token_file` is the file holding the local registry's token; the token itself is never printed. Like every command, a rehearsal holds the local registry only while it runs, so a program that goes on to run the job's script holds a [session](#registry) of its own around both. `fair run --ci` also prepares a job without running it, but stages it, warns about the repository's state, and prints the job's folder as its last line.
|
|
235
|
+
|
|
214
236
|
### `pull`
|
|
215
237
|
|
|
216
|
-
The command `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. Any data required for a run is downloaded and stored within the local registry. In addition any data products requested that are available on the remote registry are pulled locally.
|
|
238
|
+
The command `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. An `external_object` entry with `no_file: true`, and no `root`, `path` or `file_type`, registers a source that has no file of its own, such as a data store that a run queries to extract data, as a data product with no file; it must give its `version`. Any data required for a run is downloaded and stored within the local registry. In addition any data products requested that are available on the remote registry are pulled locally.
|
|
217
239
|
|
|
218
240
|
```sh
|
|
219
241
|
fair pull /path/to/config.yaml
|
|
@@ -277,23 +299,34 @@ fair purge --yes
|
|
|
277
299
|
|
|
278
300
|
### `registry`
|
|
279
301
|
|
|
280
|
-
|
|
302
|
+
A command that uses the local registry - `run`, `pull`, `push`, `status`, `list`, `find`, `identify`, `reset` - starts it, on the port the configuration names, if it is not running, and a registry started that way is stopped when the last command using it finishes. Each command holds a *session* on the registry while it runs, and the registry is not stopped while any session is open.
|
|
281
303
|
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
The commands:
|
|
304
|
+
To keep the registry running - to look at it in the browser, or to save each command the few seconds a start takes - start it yourself:
|
|
285
305
|
|
|
286
306
|
```sh
|
|
287
307
|
fair registry start
|
|
288
308
|
```
|
|
289
309
|
|
|
290
|
-
|
|
310
|
+
It then runs until
|
|
291
311
|
|
|
292
312
|
```sh
|
|
293
313
|
fair registry stop
|
|
294
314
|
```
|
|
295
315
|
|
|
296
|
-
|
|
316
|
+
`fair registry start` takes the port in the configuration (`registries.local.uri` in `~/.fair/cli/cli-config.yaml`), or the one given with `--port`, which becomes the configured one when the registry starts there. Run while a command has the registry running, it keeps that registry running from then on rather than starting another. `fair registry stop` refuses while a command is using the registry; `--force` stops it anyway. `fair registry status` says where the registry is running, whether it was started by `fair registry start` or by a command, and which processes are using it.
|
|
317
|
+
|
|
318
|
+
An install's registry runs on one host at a time, since its database is a file in its directory: where that directory is shared between machines, as a home directory on a cluster may be, the registry cannot be started, stopped or used from a machine other than the one running it. The refusal names that machine, and the file in the registry's directory that records it. If that machine no longer runs the registry - it has shut down or restarted - delete the file and try again.
|
|
319
|
+
|
|
320
|
+
`fair purge --all`, `fair registry uninstall` and `fair registry install --force` stop the registry before removing it, and remove nothing while a command is using it or while it is running on another machine.
|
|
321
|
+
|
|
322
|
+
A program that uses the registry outside a `fair` command, such as a language API, can hold a session of its own:
|
|
323
|
+
|
|
324
|
+
```sh
|
|
325
|
+
fair registry session open --pid <process id> --json
|
|
326
|
+
fair registry session close <session id>
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
`session open` starts the registry if it is not running, and prints the session's id, the registry's address and the file holding its token, as one JSON object (`{"session": ..., "registry_url": ..., "token_file": ...}`). The session ends when it is closed or when the process it names ends. A script run by `fair run` is told the run's session in `FDP_SESSION`, and needs none of its own.
|
|
297
330
|
|
|
298
331
|
The registry can be installed using the CLI as well by running:
|
|
299
332
|
```sh
|
|
@@ -353,3 +386,15 @@ Within the `config.yaml` file, template variables can be specified by using the
|
|
|
353
386
|
| `GIT_BRANCH` | Current branch of the `git` repository |
|
|
354
387
|
| `GIT_REMOTE` | The URI of the git repository specified during setup |
|
|
355
388
|
| `GIT_TAG` | The latest tag on `git` |
|
|
389
|
+
|
|
390
|
+
## What a program may read of the CLI's configuration
|
|
391
|
+
|
|
392
|
+
A language API, or any other program working with the pipeline, may rely on these parts of the CLI's configuration files:
|
|
393
|
+
|
|
394
|
+
- `registries.local.uri` and `registries.local.token` in `~/.fair/cli/cli-config.yaml`: the local registry's API address, and the file holding its token.
|
|
395
|
+
- `registries.<name>.uri` and `registries.<name>.token` in a project's `.fair/cli-config.yaml`: the same for each remote registry, by its name - `origin`, or one added with `fair remote add`.
|
|
396
|
+
- A token file holds that registry's token alone, read as it is with surrounding white space stripped. Use it only for the registry whose entry names it.
|
|
397
|
+
- `git.remote` in a project's `.fair/cli-config.yaml`: the name of the project's git remote. Its URL is the git repository's to give (`git remote get-url <name>`); `git.remote_repo` beside it is a copy taken by `fair init`, and goes out of date when the remote moves.
|
|
398
|
+
- A directory with a `.fair` folder is a FAIR project, found by searching upwards from the working directory, as every `fair` command does.
|
|
399
|
+
|
|
400
|
+
Within a run the working config holds what a script needs - among it `run_metadata.local_data_registry_url` and `run_metadata.remote_repo`, the git remote's URL - and the script's environment holds `FDP_CONFIG_DIR`, `FDP_LOCAL_TOKEN` and `FDP_SESSION`.
|