fair-cli 0.10.1__tar.gz → 0.11.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. fair_cli-0.11.0/CHANGELOG.md +291 -0
  2. {fair_cli-0.10.1 → fair_cli-0.11.0}/CITATION.cff +2 -2
  3. {fair_cli-0.10.1 → fair_cli-0.11.0}/PKG-INFO +137 -15
  4. fair_cli-0.11.0/README.md +400 -0
  5. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/cli.py +132 -40
  6. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/configuration/__init__.py +34 -16
  7. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/exceptions.py +13 -5
  8. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/identifiers.py +106 -9
  9. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/register.py +180 -33
  10. fair_cli-0.11.0/fair/registry/lifecycle.py +821 -0
  11. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/registry/server.py +221 -54
  12. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/registry/storage.py +248 -14
  13. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/registry/sync.py +259 -41
  14. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/session.py +356 -146
  15. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/user_config/__init__.py +135 -9
  16. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/user_config/globbing.py +4 -3
  17. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/user_config/validation.py +25 -2
  18. fair_cli-0.11.0/fair/virtualenv.py +141 -0
  19. {fair_cli-0.10.1 → fair_cli-0.11.0}/pyproject.toml +3 -5
  20. fair_cli-0.10.1/CHANGELOG.md +0 -132
  21. fair_cli-0.10.1/README.md +0 -281
  22. fair_cli-0.10.1/fair/virtualenv.py +0 -45
  23. {fair_cli-0.10.1 → fair_cli-0.11.0}/LICENSE +0 -0
  24. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/__init__.py +0 -0
  25. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/common.py +0 -0
  26. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/configuration/validation.py +0 -0
  27. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/files.txt +0 -0
  28. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/history.py +0 -0
  29. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/logging.py +0 -0
  30. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/registry/__init__.py +0 -0
  31. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/registry/file_formats.json +0 -0
  32. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/registry/file_types.py +0 -0
  33. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/registry/requests.py +0 -0
  34. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/registry/versioning.py +0 -0
  35. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/run.py +0 -0
  36. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/staging.py +0 -0
  37. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/templates/__init__.py +0 -0
  38. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/templates/config.jinja +0 -0
  39. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/templates/hist.jinja +0 -0
  40. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/testing.py +0 -0
  41. {fair_cli-0.10.1 → fair_cli-0.11.0}/fair/utilities.py +0 -0
@@ -0,0 +1,291 @@
1
+ # 2026-10-19 [v0.11.0](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.11.0)
2
+
3
+ ## Changed behaviour
4
+ - A command that starts the local registry stops it again when the last command using it finishes; it used to leave
5
+ it running. Each command holds a session on the registry while it runs. `fair registry start` keeps it running
6
+ until `fair registry stop`, and run while a command has it running keeps that registry rather than refusing.
7
+ - `fair registry start` with no `--port`, and a command that starts the registry, use the configured port, where
8
+ they used 8000 whatever the configuration said.
9
+ - An install's registry runs on one host at a time: where its directory is shared between machines, starting,
10
+ stopping or using it from another machine is refused, naming the machine running it and the file that records it.
11
+ - `fair purge --all`, `fair registry uninstall` and `fair registry install --force` stop the registry first, and
12
+ remove nothing while a command is using it or it runs on another machine.
13
+ - Two versions of the CLI sharing one registry install are not supported: one of an earlier version neither records
14
+ its sessions nor respects this one's.
15
+ - Records left behind when the registry or a session ends outside `fair` - a restart, a crash, a killed process - are
16
+ removed with a line saying what was removed and why, naming `fair registry start` where that had started the
17
+ registry. A record left half-written is removed the same way. One that cannot be removed - on Windows, a file
18
+ another program has open - is waited for briefly, then named and left, and the command carries on; one that
19
+ cannot be written is waited for the same way, then is an error naming it. A session whose process cannot be
20
+ checked is taken to be running, and says so where it stops the registry being stopped.
21
+ - `fair init` asks for the user's identifier before their email, and offers as the email the address the
22
+ identifier's record makes public - an ORCID's or a GitHub account's. Anything answering its questions in order
23
+ must put the email after the identifier.
24
+ - A `read:` wildcard's own entry is kept in the working config beside the entries it expands to, as a `write:`
25
+ wildcard's is, so that an API can tell a pattern that matched nothing from one never given. `fair pull` fetches
26
+ nothing for it. Its version - its matches' highest, the version it names, or 0.0.0 - means nothing.
27
+ - A `*` in a wildcard no longer matches `[`, `]`, `?` or `*`, which the registry refuses in a name, so a pattern
28
+ never matches itself.
29
+
30
+ ## Added
31
+ - `fair registry session open --pid <pid> --json` and `fair registry session close <id>`, for a program using the
32
+ registry outside a `fair` command, such as a language API. A script run by `fair run` is told the run's session
33
+ in `FDP_SESSION`.
34
+ - `fair registry status` says who holds the registry and which processes are using it.
35
+ - `fair run --rehearse` prepares a job as a run would without running it - no git check, nothing staged - and prints
36
+ its files as one JSON object, `{"job_dir", "config", "script", "data_store", "token_file"}`, for a language API's
37
+ rehearsal of a run.
38
+ - The README says which parts of the CLI's configuration a program such as a language API may read.
39
+ - A `register:` entry for an external object may have `no_file: true` in place of `root`, `path` and `file_type`,
40
+ for a source with no file of its own, such as a data store a run queries: `fair pull` registers it as a data
41
+ product with no file, which a run may read. Its `version` must be given.
42
+ - `run_metadata.shell` may give flags for the shell's program after its name: `shell: julia --project=@.` runs a
43
+ Julia script in the repository's project. A shell the CLI does not know - an empty `shell:` included - is
44
+ refused, naming those it does, where it stopped with a `KeyError` or a `TypeError`.
45
+
46
+ ## Fixed
47
+ - `fair purge --all` answered "no" removed the project's `.fair` folder, and with it a data store kept there; it now
48
+ removes nothing. `fair purge --yes` asked its question anyway, and now answers every question as the README says.
49
+ - `fair registry install --force` removed the old registry before it had the new one, so a `--version` that does not
50
+ exist left no working registry. The old one is now kept until the new one is built, and put back if the install
51
+ fails; a first install that fails leaves nothing behind.
52
+ - `fair push` and `fair pull` left out a code repository's release (`CodeRepoRelease`), so the remote's record of a
53
+ run or a data product lost it. The release now goes with its repository's object.
54
+ - A data product on a code repository's object, such as a release of the code registered with its DOI, was pushed
55
+ with the repository's web page as its file, and could not then be pulled. Its location is the repository at a
56
+ commit, and it is now pushed and pulled as that, with nothing downloaded.
57
+ - A `register:` entry with no version was registered again at the next patch version on every `fair pull`, whether
58
+ or not its file had changed, and `fair run` then read a version that did not exist (#317). The highest version is
59
+ now kept while the file is the same, and `fair run` reads it. `${{LATEST}}` on such an entry keeps the highest
60
+ version and refuses a changed file.
61
+ - `fair run` did not check that `fair pull` had registered what a `register:` entry names, so the model found out;
62
+ it now refuses to start.
63
+ - A ROR identifier (`https://ror.org/...`) of an author or a source was checked by fetching ROR's website, which
64
+ refuses automated requests that come too often (429), and was then rejected. It is now checked through ROR's API.
65
+ - On Windows, `fair registry start` could fail - no response, or no token - when the registry's start script gave up
66
+ waiting for the server, leaving the server running. The CLI now waits for the server itself, up to 30 seconds, and
67
+ writes the registry's token if the script did not.
68
+ - A command that crashed, or a reboot after `fair registry start`, left a session file behind that made every later
69
+ command skip starting the registry and `fair registry stop` refuse. A session now names its process, and one whose
70
+ process has gone is removed. This needs a new dependency, `psutil`.
71
+ - A failed `fair registry start` left a session file behind, and a server on the port that never answered was waited
72
+ on for ever; neither now happens. Another install's registry on the port is named as such.
73
+ - An internal error, and a registry's refusal of a request (HTTP 403), printed their message and exited 0, so a
74
+ script or a CI step saw success. Both now exit non-zero, the refusal shown as an error.
75
+ - `fair pull` of a `register:` entry at a version given outright, registered already with another file, copied the
76
+ new file over the registered one before the registry refused it, so the registered file no longer matched its
77
+ record. It is now refused before anything is copied; the same file is passed over, as before.
78
+ - `fair init --ci` named the remote user `FAIRDataPipeline`, which a freshly built remote registry does not have, so
79
+ `fair push` stopped. It now takes the user the remote's token belongs to, where the remote answers.
80
+ - `fair run` refused a detached checkout - a tag, a commit, a pull request on CI - unless given `--dirty`. It now runs
81
+ where the commit is one the remote has, on a branch or a tag, and refuses one the remote does not have.
82
+ `${{GIT_BRANCH}}` there is an error saying there is no branch.
83
+ - A `fair init` that failed - a token the remote refused, a registry that would not start, Ctrl-C other than at a
84
+ question, a `--using` file it refused - left `.fair` and an empty global configuration behind, so `fair init` again
85
+ failed too (#272). It now puts back what it changed, and `fair init` starts afresh where an earlier one left `.fair`
86
+ unfinished, or the global configuration empty.
87
+ - `fair init` failing in a subdirectory of a project could write the configuration it had refused over the project's;
88
+ the project's is now left as it was.
89
+ - `fair init` in a subdirectory of a project made the project's staging file and not its own, so a later command
90
+ there that staged something failed; it now makes its own.
91
+ - A local registry that answered every request with an internal error (HTTP 500) was taken for one that works, and a
92
+ command failed at its first request. Starting it, or a command that finds it running, now says so and names its
93
+ log; `fair registry stop` still stops it.
94
+ - `fair init` with an ORCID whose record keeps its holder's name private, or gives no family name, stopped with a
95
+ `TypeError`. It now asks for a name the record does not give, and takes a missing family name as empty. An author
96
+ in a `register:` entry whose ORCID keeps its name private is registered by its identifier. A one-word name - given
97
+ with no identifier, or to `fair config user.name` - failed the configuration's validation and broke the job logs;
98
+ its family name is now empty too.
99
+
100
+ # 2026-10-05 [v0.10.2](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.10.2)
101
+
102
+ ## Changed behaviour
103
+ - `fair pull` refuses a `register:` entry for an external object whose identifier or unique name, title and
104
+ release version are those of a source already registered from a different file: a source is one file. Give
105
+ the entry a `title` of its own, or its `release_version`; the same file under another name still registers.
106
+ With a local registry before data-registry v1.4.0, where each data product has its own source, nothing is
107
+ checked.
108
+ - `fair init` in a repository that is already initialised registers its user in the local registry, where it
109
+ did nothing. A registry reinstalled since the repository was initialised has no record of the user, and a
110
+ run failed until the repository was purged and initialised afresh. The repository itself is left as it is.
111
+
112
+ ## Added
113
+ - A `register:` entry for an external object may give `release_version`, the version of the source it was taken
114
+ from, beside `release_date`. A local registry at data-registry v1.4.0 or later records it as the external
115
+ object's version, 1.0.0 without it; an earlier one records the data product's version there whatever is
116
+ given. It is not the entry's `version`, which is the data product's.
117
+
118
+ ## Fixed
119
+ - A GitHub author or user was refused as "not a recognised github" once the GitHub API's limit for
120
+ anonymous requests (60 an hour per address) was spent. GitHub lookups now send the token in `GITHUB_TOKEN`
121
+ or `GITHUB_PAT` if set, retry without it if GitHub refuses it (expired or revoked), and say when the rate
122
+ limit is the problem.
123
+ - The FAIR-CLI binaries could not install a local registry, so `fair init` and `fair registry install` failed:
124
+ the registry's virtual environment was built from the binary in place of a Python. They now use a Python
125
+ >= 3.10 from `FAIR_PYTHON` or the `PATH`, or have `uv` fetch one, and say so if there is none. A missing
126
+ Python is found before anything is installed.
127
+ - `fair push` failed, with "Failed to access [] on remote registry", for a data product registered from a web
128
+ address once a code run had used it, when the local registry was data-registry v1.4.0. The remote records
129
+ such a file in two places with one hash - where it is stored and where it came from - and the wrong one
130
+ could be taken.
131
+ - `fair registry install --force` refused an existing install, as it does without `--force`, instead of
132
+ replacing it.
133
+ - A file that was already registered was not registered again under a second data product name, or in a
134
+ second namespace: `fair pull` passed over the entry without a word, and a run that read it then failed. An
135
+ entry is now passed over only when that data product, in that namespace and at that version, already holds
136
+ the file.
137
+ - `fair push` and `fair pull` left out a data product that shares its external object with another, as
138
+ data-registry v1.4.0 allows, while reporting it synchronised: its file was moved and its record was not made.
139
+ - Starting the local registry reported success when another registry already held its port, and the commands
140
+ that followed went to that one. The start now fails, naming the address: the server that answers there
141
+ must accept the installed registry's token.
142
+ - `fair push` and `fair pull` stopped with a Python error at a data product that has no file - one whose
143
+ object has no storage location, as an entry for a deposit as a whole may be - and so at a code run that
144
+ read one. Its records are now synchronised, and a `register:` entry of the same name is told it exists.
145
+ - `fair push` gave a code run an input it did not read when two data products held the same file, as one
146
+ file registered under two names does: a run that read one of them and wrote something new arrived on the
147
+ remote as having read both. A run pushed before this keeps the extra input.
148
+ - A file registered under a second data product name, or in a second namespace, was copied into the data store
149
+ again, though the registry records a file once there and nothing referred to the copy. The copy is no
150
+ longer kept: the data product's file is the one already in the store.
151
+ - A data product pulled from a remote could not be read through an API. `fair pull` copied the remote's record
152
+ of where the file is - the remote's own store - and fetched the file to a place no record named, so a model
153
+ was handed an address and no file. A pulled file is now recorded at its place in the local data store, and
154
+ a file that two data products share is fetched once. Data products pulled with an earlier version keep
155
+ the old record.
156
+
157
+ ## Development
158
+ - The test of a data product's dependencies allows for any a registry adds; data-registry v1.4.0 adds one.
159
+
160
+ # 2026-10-05 [v0.10.1](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.10.1)
161
+
162
+ ## Changed behaviour
163
+ - A wildcard entry keeps to one namespace: the one its `use:` names, else the default for its block. It
164
+ used to match names in every namespace, so a `write:` pattern could write a name into a namespace
165
+ other than its own.
166
+ - A wildcard entry gives one entry for each name it matches: for a `read:`, the version its `use:` names,
167
+ else the highest; for a `write:`, the version to be written. It used to give one for every version, and
168
+ to ignore a version on a `read:`. A `read:` pattern that matches nothing is not an error.
169
+ - `fair run` refuses a `read:` of a data product, or of a version of one, that is not in the registry,
170
+ naming the namespace and registry it looked in. Such a read used to pass unchecked, or as version 0.0.0.
171
+ - `fair push` fails if a file cannot be uploaded. It used to warn and carry on, leaving the remote with
172
+ the file's records and no file. A file is now uploaded before its records are written, so a push that
173
+ failed can be run again.
174
+ - `fair pull` checks each file it fetches against the hash the remote registry holds for it, and refuses
175
+ one that differs.
176
+
177
+ ## Added
178
+ - Registering an external object records where its file was fetched from: the `root` and `path` of its
179
+ `register:` entry, as the object's `original_store`, with the hash of the file found there. `fair push`
180
+ takes that record to the remote. A file registered from the machine itself has no such record.
181
+ - `fair push` takes a refusal of an upload address with HTTP 409 to mean that the remote's store already
182
+ holds the file, sends nothing, and carries on. No released registry answers so yet.
183
+
184
+ ## Fixed
185
+ - `fair push` sent no value that was false, leaving the remote to apply its default: an external object
186
+ registered with `primary: false` arrived as primary, and a location with `public: false` as public.
187
+ - A list from a registry stopped at its first 100 rows, so wildcards, staging and version lookups went
188
+ wrong, with no message, in a registry holding more.
189
+ - `fair pull` looked for the version of a `read:` entry in the local registry only, so an entry at the
190
+ default version, for something not yet local, asked the remote for version 0.0.0 and failed.
191
+ - `fair push` held each file whole in memory, `fair pull` more than twice over, and both left a copy of
192
+ it in the temporary directory. Files are now sent and received a block at a time, whatever their
193
+ size, and nothing is left behind.
194
+ - An upload of 2 GiB or more over plain http failed on macOS.
195
+ - An external object could not be registered with a `unique_name` and `alternate_identifier_type` in
196
+ place of an `identifier`: `fair pull` stopped, reporting that the object already existed.
197
+ - `fair identify` looked at only the newest of a file's storage locations, so it could miss the data
198
+ products of a file the registry records in more than one place.
199
+ - `fair run` could not start the script when the path of the data store, which holds the job
200
+ directory, had a space in it.
201
+
202
+ ## Development
203
+ - The Python job of the implementations workflow starts an object store for its remote registry, so
204
+ that its `fair push` uploads the files. It had none, and passed while a failed upload was a warning.
205
+
206
+ # 2026-10-03 [v0.10.0](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.10.0)
207
+
208
+ ## Changed behaviour
209
+ - `fair remote add` takes the remote's token file with `--token FILE`, and prompts for it if omitted.
210
+ The command never worked before: the label was stored as the token path.
211
+ - `fair run --ci` prepares the job directory and prints it without running the script, as documented.
212
+ - `fair init --ci` leaves an existing `.fair/` alone, as `fair init` does; it used to delete it, data
213
+ store included. Use `fair purge` to start afresh.
214
+ - Wildcards: each `*` matches one segment of a name (not `/`), anchored at both ends, in `read:` and
215
+ `write:` alike, matching DataPipeline.jl. `data_product/*` no longer matches `data_product/1/2`.
216
+ - `${{GIT_TAG}}` is the nearest tag in the history of the current commit (`git describe`), not the
217
+ last tag by name; with no such tag it is an error rather than a crash.
218
+ - `run_metadata.shell` now chooses the interpreter the script runs with.
219
+
220
+ ## Fixed
221
+ - `write:` entries with a wildcard were dropped from the working config (#140). The pattern is now kept
222
+ for new names, and existing matches are written as the pattern entry describes them (its `use: version`,
223
+ `file_type`, `description`).
224
+ - `fair push` from a project whose data store was not the local registry's first storage root recorded
225
+ its files on the remote under the pusher's `file://` path (data-registry #229).
226
+ - `fair pull` of a `read:` entry from a remote registry failed.
227
+ - A DOI whose publisher refuses automated requests (e.g. Wiley, HTTP 403) was rejected as an invalid
228
+ identifier. An identifier that resolves is now accepted; one that does not (404) is still refused.
229
+ - `${{DATETIME-<format>}}` always failed with "Failed to parse formatted datetime variable".
230
+ - Substituting a path with backslashes, such as `${{CONFIG_DIR}}` on Windows, failed `fair pull` and `fair run`
231
+ (#267). Variables are now substituted into the configuration's values, so any text is inserted as it is.
232
+ - `fair remote remove` crashed.
233
+ - A ROR or GRID ID the ROR API could not parse (e.g. one containing `!`) crashed with `KeyError`
234
+ instead of being reported as not found.
235
+ - Run from a subdirectory of a project, commands lost configuration changes and left a stale session
236
+ file, so the next command did not start the registry and `fair registry stop` needed `--force`.
237
+ - Staging ignored the configured local registry and always used port 8000.
238
+ - A download that failed with an HTTP error registered the error page as the data.
239
+ - `fair run` could crash on Windows when the model printed a character the console cannot encode.
240
+ - `fair run` in a git repository without the configured remote gave a bare `IndexError`; it now says
241
+ which remote is missing, or reads `run_metadata: remote_repo:` if given.
242
+ - `fair init` in a subdirectory of a FAIR repository named it as `'True'`.
243
+
244
+ ## Development
245
+ - CI ran no tests: an unused dev dependency (`pylama`) crashed `pytest --markers`, leaving the test loop
246
+ empty. It is removed, and a failure to list the tests now fails the job; all 8 jobs run the full suite.
247
+ - numpy is chosen per Python version (>= 2.3 from 3.11), so Python 3.14 gets a wheel, not a source build.
248
+ - Download tests use a local HTTP server rather than github.com.
249
+ - `test_configuration.py::test_local_config_query` is marked `xfail`: it has never passed, and its intent
250
+ is unknown.
251
+
252
+ # 2026-09-22 [v0.9.9](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.9.9)
253
+ - Requires Python >= 3.10, in step with data-registry v1.2.2.
254
+ - `${{RUN_ID}}` in a `write:` name is left for the language API to fill in at `finalise`.
255
+ - ROR/GRID organisation lookups failed after the ROR API moved to its v2 schema.
256
+
257
+ # Between v0.2.3 and v0.9.8
258
+ Not recorded release by release; see the [releases](https://github.com/FAIRDataPipeline/FAIR-CLI/releases)
259
+ and the git log.
260
+ - Fix virtual environment bug in `fair` binaries where `venv` assumes the main executable is `python`.
261
+ - Registration of `author` and `namespace` objects.
262
+ - Allow specification of tag to install registry from during `fair registry install`.
263
+ - Wildcard '*' parsing introduced for data products.
264
+ - Ability to `push` to a registry added.
265
+ - Added `--dirty` option to `fair run` to allow running with uncommitted changes.
266
+ - Added `config.yaml` file validation.
267
+ - Added initialisation from existing registry.
268
+ - Switch to setting port not local URI during initialisation.
269
+ - Added option to specify port on `fair registry start`.
270
+ - Added `cli-config.yaml` file validation during initialisation.
271
+ - Added `fair reset` to list of commands.
272
+ - Add tab completion for staging data products.
273
+
274
+ # 2021-11-17 [v0.2.3](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.3)
275
+ - Move handling of the user `config.yaml` file to a separate class `JobConfiguration`.
276
+ - Added various fixes to improve functionality within Windows.
277
+ - Move registry installation from script execution to internal function which sets up virtual environment etc.
278
+ - Added a test suite to the project.
279
+ - Added additional recognised identifiers for author setup from an organisation GRID and ROR.
280
+
281
+ # 2021-10-06 [v0.2.2](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.2)
282
+ - Update to package metadata for PyPi
283
+
284
+ # 2021-10-06 [v0.2.1](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.1)
285
+ - Automatic starter `config.yaml` generation.
286
+ - Local and Global CLI configurations.
287
+ - Start/stop local registry either explicitly or during synchronisations.
288
+ - Run logs available via git-like interface.
289
+ - Added ability to add/remove files.
290
+ - Repository style handling, acts like another git-like tool per project.
291
+ - Creation of an interface for `fair` using `click`.
@@ -26,7 +26,7 @@ authors:
26
26
  affiliation: University of Glasgow
27
27
  orcid: https://orcid.org/0000-0002-4424-9890
28
28
 
29
- date-released: 2026-10-05
29
+ date-released: 2026-10-19
30
30
  keywords:
31
31
  - FAIR Data Pipeline
32
32
  - FAIR
@@ -36,4 +36,4 @@ license: BSD-2-Clause
36
36
  message: If you use this software, please cite it using these metadata.
37
37
  repository-code: https://github.com/FAIRDataPipeline/FAIR-CLI/
38
38
  title: "The FAIR Data Pipeline command line tool"
39
- version: 0.10.1
39
+ version: 0.11.0
@@ -1,9 +1,9 @@
1
- Metadata-Version: 2.1
1
+ Metadata-Version: 2.4
2
2
  Name: fair-cli
3
- Version: 0.10.1
3
+ Version: 0.11.0
4
4
  Summary: Synchronization interface for the SCRC FAIR Data Pipeline registry
5
- Home-page: https://www.fairdatapipeline.org/
6
5
  License: BSD-2-Clause
6
+ License-File: LICENSE
7
7
  Keywords: FAIR Data Pipeline,FAIR,Data Management,Provenance
8
8
  Author: Richard Reeve
9
9
  Author-email: richard.reeve@glasgow.ac.uk
@@ -20,9 +20,10 @@ Classifier: Programming Language :: Python :: 3.10
20
20
  Classifier: Programming Language :: Python :: 3.11
21
21
  Classifier: Programming Language :: Python :: 3.12
22
22
  Classifier: Programming Language :: Python :: 3.13
23
+ Classifier: Programming Language :: Python :: 3.14
24
+ Classifier: Programming Language :: Python :: 3.15
23
25
  Classifier: Topic :: Database :: Front-Ends
24
26
  Classifier: Topic :: Scientific/Engineering
25
- Provides-Extra: all
26
27
  Requires-Dist: GitPython (>=3.1.18,<4.0.0)
27
28
  Requires-Dist: Jinja2 (>=3.0.1,<4.0.0)
28
29
  Requires-Dist: PyYAML (>=5.4.1,<7.0.0)
@@ -30,6 +31,7 @@ Requires-Dist: click (>=8.0.0,<9.0.0)
30
31
  Requires-Dist: email-validator (>=1.1.3,<2.0.0)
31
32
  Requires-Dist: fake-useragent (>=1,<2)
32
33
  Requires-Dist: pre-commit (>=3,<4)
34
+ Requires-Dist: psutil (>=5.9.0)
33
35
  Requires-Dist: pydantic[email] (>=1.9.0,<2.0.0)
34
36
  Requires-Dist: requests (>=2.23.0,<3.0.0)
35
37
  Requires-Dist: rich (>=10.2.3,<12.0.0)
@@ -38,6 +40,7 @@ Requires-Dist: simplejson (>=3.17.5,<4.0.0)
38
40
  Requires-Dist: toml (>=0.10.2,<0.11.0)
39
41
  Requires-Dist: validators (>=0.18.2,<0.19.0)
40
42
  Project-URL: Documentation, https://www.fairdatapipeline.org/docs/interface/fdp/
43
+ Project-URL: Homepage, https://www.fairdatapipeline.org/
41
44
  Project-URL: Issue Tracker, https://github.com/FAIRDataPipeline/FAIR-CLI/issues
42
45
  Project-URL: Repository, https://github.com/FAIRDataPipeline/FAIR-CLI
43
46
  Description-Content-Type: text/markdown
@@ -58,6 +61,8 @@ The package is installed using Pip:
58
61
  pip install fair-cli
59
62
  ```
60
63
 
64
+ It needs Python 3.10 or later. A virtual environment is recommended: the local registry is installed with the same Python as the CLI, in an environment of its own.
65
+
61
66
  To enable tab completion you need to modify your shell:
62
67
 
63
68
  ### Bash
@@ -78,12 +83,81 @@ _FAIR_COMPLETE=bash_source fair > ~/.config/fish/.fair-complete.fish
78
83
  echo '. ~/.config/fish/.fair-complete.fish' >> ~/.bashrc
79
84
  ```
80
85
 
86
+ ## Upgrading and reinstalling
87
+
88
+ The CLI and the local registry are installed separately - the CLI by `pip`, the registry by the CLI - so upgrading one does not upgrade the other.
89
+
90
+ ### The CLI
91
+
92
+ To move to the newest release, or to a particular one:
93
+
94
+ ```sh
95
+ pip install --upgrade fair-cli
96
+ pip install fair-cli==0.10.1
97
+ ```
98
+
99
+ To install a branch of this repository, give `pip` its name after an `@`. A tag or a commit works the same way:
100
+
101
+ ```sh
102
+ pip install git+https://github.com/FAIRDataPipeline/FAIR-CLI@<branch>
103
+ ```
104
+
105
+ or clone the repository, check the branch out and install from the clone:
106
+
107
+ ```sh
108
+ git clone -b <branch> https://github.com/FAIRDataPipeline/FAIR-CLI
109
+ pip install ./FAIR-CLI
110
+ ```
111
+
112
+ A branch keeps its version number from one commit to the next, and `pip` does not replace an installed package with another of the same version. To bring an installed branch up to date, or to install one branch over another of the same version, add `--force-reinstall --no-deps`:
113
+
114
+ ```sh
115
+ pip install --force-reinstall --no-deps git+https://github.com/FAIRDataPipeline/FAIR-CLI@<branch>
116
+ ```
117
+
118
+ `fair --version` gives the version number alone, which a branch may share with a release. `pip freeze` says what is installed: `fair-cli==0.10.1` for a release, and `fair-cli @ git+https://github.com/FAIRDataPipeline/FAIR-CLI@<commit>` for a branch.
119
+
120
+ ### The local registry
121
+
122
+ To replace the local registry with the newest release of it, stop it if it is running, install with `--force`, and start it again:
123
+
124
+ ```sh
125
+ fair registry stop
126
+ fair registry install --force
127
+ fair registry start
128
+ ```
129
+
130
+ `--version` installs a tag or a branch of the [registry's repository](https://github.com/FAIRDataPipeline/data-registry) in place of the newest release:
131
+
132
+ ```sh
133
+ fair registry install --force --version v1.3.0
134
+ fair registry install --force --version <branch>
135
+ ```
136
+
137
+ Without `--force` an existing registry is left as it is and the command stops. `fair registry uninstall`, which asks first, followed by `fair registry install` does the same in two steps, and is the way with `fair-cli` 0.10.1 or earlier, where `--force` is refused.
138
+
139
+ With `--force` the old registry is kept until the new one is built. If the install fails - a `--version` that does not exist, no network, a build that fails - the old registry is put back as it was, and the configuration still names it. A running registry is stopped first, and nothing is replaced while a command is using it.
140
+
141
+ The registry is installed in `~/.fair/registry`, or in the directory given with `--directory`, which has to be given again to replace a registry that was installed elsewhere. A reinstall replaces that directory:
142
+
143
+ - every record in the local registry goes with its database: the data products, the code runs, and what `fair init` put there;
144
+ - the local registry has a new token, written to `token` in its directory when it is next started, and anything else that was kept in that directory is gone;
145
+ - the data store, the projects and `~/.fair/cli` are not touched.
146
+
147
+ The new registry does not know the user of a project that was initialised before the reinstall, and a run in that project fails until it does. There are two ways to put that right:
148
+
149
+ - run `fair init` in the project again, which leaves the project as it is - its configuration and any data store kept inside it - and registers its user in the new registry. This needs `fair-cli` 0.10.2 or later: before that, `fair init` does nothing in a project that is already initialised;
150
+ - or start the project afresh: run `fair purge` in it, which removes its `.fair` folder and any data store kept inside it, and then `fair init`.
151
+
152
+ After either, `fair pull` registers again what the project registers or reads. What had been pushed to a remote registry can be pulled back from it. The records of anything that had not been pushed cannot be recovered, though its files are still in the data store unless that was removed with the project's `.fair` folder.
153
+
81
154
  ## Uninstallation
82
155
  To uninstall the CLI run:
83
156
  ```
84
157
  fair purge --all
85
- pip uninstall fair
158
+ pip uninstall fair-cli
86
159
  ```
160
+ `fair purge --all` removes the `.fair` folder of the current project and the whole of `~/.fair`: the local registry, the default data store, the CLI's configuration and any token kept there. Leave it out to keep them.
87
161
 
88
162
  ## The User Configuration File
89
163
  Job runs are configured via `config.yaml` files. Upon initialisation of a project, FAIR-CLI automatically generates a starter configuration file with all requirements in place. To execute a process (e.g. perform a model run from a compiled binary/script) an additional key of either `script` or `script_path` must be provided. Alternatively the command `fair run bash` can be used to append the key and run a command directly.
@@ -104,6 +178,8 @@ By default the shell used to execute a process is `sh` or `batch` for UNIX and W
104
178
  | `sh` | `sh -e {0}` |
105
179
  | `batch` | `{0}` |
106
180
 
181
+ Flags for a shell's program follow its name, separated by spaces, and go straight after the program: `shell: julia --project=@.` runs `julia --project=@. {0}`, starting Julia in the repository's project, and `shell: bash -x` runs `bash -x -eo pipefail {0}`. A flag cannot contain a space, and `batch`, which has no program, takes none.
182
+
107
183
  A full description of `config.yaml` files can be found [here](https://www.fairdatapipeline.org/docs/interface/config/).
108
184
 
109
185
  ## Available Commands
@@ -112,6 +188,12 @@ A full description of `config.yaml` files can be found [here](https://www.fairda
112
188
 
113
189
  Initialises a new FAIR repository within the given directory. This should ideally be the same location as the `.git` folder for the current project, however during setup an option is given to specify an alternative. The command will ask the user a series of questions which will provide metadata for tracking run authors, and also allow for the creation of a starter `config.yaml` file. Initialisation will also configure the CLI itself.
114
190
 
191
+ The user's identifier is asked for before their email: where the identifier's record makes an address public - an ORCID's, or a GitHub account's - that address is offered as the email, and with none given a GitHub no-reply address is used.
192
+
193
+ In a repository that is already initialised the command asks nothing and leaves the repository as it is. It registers the repository's user in the local registry, which a registry [reinstalled](#upgrading-and-reinstalling) since the repository was initialised no longer holds.
194
+
195
+ An initialisation that fails, or is interrupted, before the repository's configuration is written leaves things as they were: no `.fair` folder, and the global configuration unchanged. One that fails after it - registering the user in a local registry that is not running, say - leaves the repository initialised, and running `fair init` again completes it. A `.fair` folder with no configuration in it, as a process killed during `fair init` leaves, is removed and the repository initialised afresh, unless it holds more than that, when it is named and left for you to look through.
196
+
115
197
  #### Custom CLI Configuration
116
198
  After setup is complete, the current CLI configuration can also be saved using the command:
117
199
  ```
@@ -155,7 +237,7 @@ For integration into a CI workflow, the setup can be skipped by running:
155
237
  fair init --ci
156
238
  ```
157
239
 
158
- which will create temporary directories for some of the required location paths.
240
+ which will create temporary directories for some of the required location paths. The remote registry is expected at `http://127.0.0.1:8001/api/`, with its token in the file `token` in the directory the command is run from; the user it records for the remote is the one that token belongs to, or `FAIRDataPipeline` where the remote cannot be asked.
159
241
 
160
242
 
161
243
  ### `run`
@@ -182,9 +264,25 @@ note the command itself must be quoted as it is a single argument.
182
264
 
183
265
  By default the CLI will not allow the user to perform a run if the state of the analysis repository is such that it is behind the git remote, or contains uncommitted changes. To override this behaviour use the `--dirty` flag.
184
266
 
267
+ A detached checkout - a tag or a commit checked out, or a pull request on a CI system - has no branch to compare, so there the commit has to be one the remote has, on a branch or a tag; `--dirty` lets one the remote does not have through.
268
+
269
+ To prepare a job without running it, as a language API's rehearsal of a run does, use `--rehearse`:
270
+
271
+ ```sh
272
+ fair run --rehearse
273
+ ```
274
+
275
+ It writes the job's folder, its working config and its script exactly as a run would, and runs nothing. It makes no git check - the commit is recorded as it is, marked `-dirty` where there are uncommitted changes - and stages nothing. Its only output on stdout is one JSON object naming the job's files, its log going to stderr:
276
+
277
+ ```json
278
+ {"job_dir": "...", "config": "...", "script": "...", "data_store": "...", "token_file": "..."}
279
+ ```
280
+
281
+ `token_file` is the file holding the local registry's token; the token itself is never printed. Like every command, a rehearsal holds the local registry only while it runs, so a program that goes on to run the job's script holds a [session](#registry) of its own around both. `fair run --ci` also prepares a job without running it, but stages it, warns about the repository's state, and prints the job's folder as its last line.
282
+
185
283
  ### `pull`
186
284
 
187
- The command `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. Any data required for a run is downloaded and stored within the local registry. In addition any data products requested that are available on the remote registry are pulled locally.
285
+ The command `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. An `external_object` entry with `no_file: true`, and no `root`, `path` or `file_type`, registers a source that has no file of its own, such as a data store that a run queries to extract data, as a data product with no file; it must give its `version`. Any data required for a run is downloaded and stored within the local registry. In addition any data products requested that are available on the remote registry are pulled locally.
188
286
 
189
287
  ```sh
190
288
  fair pull /path/to/config.yaml
@@ -248,32 +346,44 @@ fair purge --yes
248
346
 
249
347
  ### `registry`
250
348
 
251
- By default the CLI will launch the registry whenever a synchronisation or run is called. The server will only be halted once all ongoing CLI processes (in the case of multiple parallel calls) have been completed.
349
+ A command that uses the local registry - `run`, `pull`, `push`, `status`, `list`, `find`, `identify`, `reset` - starts it, on the port the configuration names, if it is not running, and a registry started that way is stopped when the last command using it finishes. Each command holds a *session* on the registry while it runs, and the registry is not stopped while any session is open.
252
350
 
253
- However the user may also specify a manual launch that will override this behaviour, instead leaving the server running constantly allowing them to view the registry in the browser.
254
-
255
- The commands:
351
+ To keep the registry running - to look at it in the browser, or to save each command the few seconds a start takes - start it yourself:
256
352
 
257
353
  ```sh
258
354
  fair registry start
259
355
  ```
260
356
 
261
- and
357
+ It then runs until
262
358
 
263
359
  ```sh
264
360
  fair registry stop
265
361
  ```
266
362
 
267
- will launch and halt the server respectively.
363
+ `fair registry start` takes the port in the configuration (`registries.local.uri` in `~/.fair/cli/cli-config.yaml`), or the one given with `--port`, which becomes the configured one when the registry starts there. Run while a command has the registry running, it keeps that registry running from then on rather than starting another. `fair registry stop` refuses while a command is using the registry; `--force` stops it anyway. `fair registry status` says where the registry is running, whether it was started by `fair registry start` or by a command, and which processes are using it.
364
+
365
+ An install's registry runs on one host at a time, since its database is a file in its directory: where that directory is shared between machines, as a home directory on a cluster may be, the registry cannot be started, stopped or used from a machine other than the one running it. The refusal names that machine, and the file in the registry's directory that records it. If that machine no longer runs the registry - it has shut down or restarted - delete the file and try again.
366
+
367
+ `fair purge --all`, `fair registry uninstall` and `fair registry install --force` stop the registry before removing it, and remove nothing while a command is using it or while it is running on another machine.
368
+
369
+ A program that uses the registry outside a `fair` command, such as a language API, can hold a session of its own:
370
+
371
+ ```sh
372
+ fair registry session open --pid <process id> --json
373
+ fair registry session close <session id>
374
+ ```
375
+
376
+ `session open` starts the registry if it is not running, and prints the session's id, the registry's address and the file holding its token, as one JSON object (`{"session": ..., "registry_url": ..., "token_file": ...}`). The session ends when it is closed or when the process it names ends. A script run by `fair run` is told the run's session in `FDP_SESSION`, and needs none of its own.
268
377
 
269
378
  The registry can be installed using the CLI as well by running:
270
379
  ```sh
271
380
  fair registry install
272
381
  ```
273
- with the additional options to specify the installation location, and the data registry repository tag to install from:
382
+ with the additional options to specify the installation location, and the data registry repository tag or branch to install from:
274
383
  ```sh
275
- fair registry install --directory ~/.fair/my_registry --version v1.0-rc5
384
+ fair registry install --directory ~/.fair/my_registry --version v1.4.0
276
385
  ```
386
+ To replace a registry that is already installed, see [Upgrading and reinstalling](#upgrading-and-reinstalling).
277
387
 
278
388
  ### `log`
279
389
 
@@ -324,3 +434,15 @@ Within the `config.yaml` file, template variables can be specified by using the
324
434
  | `GIT_REMOTE` | The URI of the git repository specified during setup |
325
435
  | `GIT_TAG` | The latest tag on `git` |
326
436
 
437
+ ## What a program may read of the CLI's configuration
438
+
439
+ A language API, or any other program working with the pipeline, may rely on these parts of the CLI's configuration files:
440
+
441
+ - `registries.local.uri` and `registries.local.token` in `~/.fair/cli/cli-config.yaml`: the local registry's API address, and the file holding its token.
442
+ - `registries.<name>.uri` and `registries.<name>.token` in a project's `.fair/cli-config.yaml`: the same for each remote registry, by its name - `origin`, or one added with `fair remote add`.
443
+ - A token file holds that registry's token alone, read as it is with surrounding white space stripped. Use it only for the registry whose entry names it.
444
+ - `git.remote` in a project's `.fair/cli-config.yaml`: the name of the project's git remote. Its URL is the git repository's to give (`git remote get-url <name>`); `git.remote_repo` beside it is a copy taken by `fair init`, and goes out of date when the remote moves.
445
+ - A directory with a `.fair` folder is a FAIR project, found by searching upwards from the working directory, as every `fair` command does.
446
+
447
+ Within a run the working config holds what a script needs - among it `run_metadata.local_data_registry_url` and `run_metadata.remote_repo`, the git remote's URL - and the script's environment holds `FDP_CONFIG_DIR`, `FDP_LOCAL_TOKEN` and `FDP_SESSION`.
448
+