fair-cli 0.9.8__tar.gz → 0.10.0__tar.gz

fair_cli-0.10.0/CHANGELOG.md

@@ -0,0 +1,86 @@
+# 2026-10-03 [v0.10.0](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.10.0)
+
+## Changed behaviour
+- `fair remote add` takes the remote's token file with `--token FILE`, and prompts for it if omitted.
+  The command never worked before: the label was stored as the token path.
+- `fair run --ci` prepares the job directory and prints it without running the script, as documented.
+- `fair init --ci` leaves an existing `.fair/` alone, as `fair init` does; it used to delete it, data
+  store included. Use `fair purge` to start afresh.
+- Wildcards: each `*` matches one segment of a name (not `/`), anchored at both ends, in `read:` and
+  `write:` alike, matching DataPipeline.jl. `data_product/*` no longer matches `data_product/1/2`.
+- `${{GIT_TAG}}` is the nearest tag in the history of the current commit (`git describe`), not the
+  last tag by name; with no such tag it is an error rather than a crash.
+- `run_metadata.shell` now chooses the interpreter the script runs with.
+
+## Fixed
+- `write:` entries with a wildcard were dropped from the working config (#140). The pattern is now kept
+  for new names, and existing matches are written as the pattern entry describes them (its `use: version`,
+  `file_type`, `description`).
+- `fair push` from a project whose data store was not the local registry's first storage root recorded
+  its files on the remote under the pusher's `file://` path (data-registry #229).
+- `fair pull` of a `read:` entry from a remote registry failed.
+- A DOI whose publisher refuses automated requests (e.g. Wiley, HTTP 403) was rejected as an invalid
+  identifier. An identifier that resolves is now accepted; one that does not (404) is still refused.
+- `${{DATETIME-<format>}}` always failed with "Failed to parse formatted datetime variable".
+- Substituting a path with backslashes, such as `${{CONFIG_DIR}}` on Windows, failed `fair pull` and `fair run`
+  (#267). Variables are now substituted into the configuration's values, so any text is inserted as it is.
+- `fair remote remove` crashed.
+- A ROR or GRID ID the ROR API could not parse (e.g. one containing `!`) crashed with `KeyError`
+  instead of being reported as not found.
+- Run from a subdirectory of a project, commands lost configuration changes and left a stale session
+  file, so the next command did not start the registry and `fair registry stop` needed `--force`.
+- Staging ignored the configured local registry and always used port 8000.
+- A download that failed with an HTTP error registered the error page as the data.
+- `fair run` could crash on Windows when the model printed a character the console cannot encode.
+- `fair run` in a git repository without the configured remote gave a bare `IndexError`; it now says
+  which remote is missing, or reads `run_metadata: remote_repo:` if given.
+- `fair init` in a subdirectory of a FAIR repository named it as `'True'`.
+
+## Development
+- CI ran no tests: an unused dev dependency (`pylama`) crashed `pytest --markers`, leaving the test loop
+  empty. It is removed, and a failure to list the tests now fails the job; all 8 jobs run the full suite.
+- numpy is chosen per Python version (>= 2.3 from 3.11), so Python 3.14 gets a wheel, not a source build.
+- Download tests use a local HTTP server rather than github.com.
+- `test_configuration.py::test_local_config_query` is marked `xfail`: it has never passed, and its intent
+  is unknown.
+
+# 2026-09-22 [v0.9.9](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.9.9)
+- Requires Python >= 3.10, in step with data-registry v1.2.2.
+- `${{RUN_ID}}` in a `write:` name is left for the language API to fill in at `finalise`.
+- ROR/GRID organisation lookups failed after the ROR API moved to its v2 schema.
+
+# Between v0.2.3 and v0.9.8
+Not recorded release by release; see the [releases](https://github.com/FAIRDataPipeline/FAIR-CLI/releases)
+and the git log.
+- Fix virtual environment bug in `fair` binaries where `venv` assumes the main executable is `python`.
+- Registration of `author` and `namespace` objects.
+- Allow specification of tag to install registry from during `fair registry install`.
+- Wildcard '*' parsing introduced for data products.
+- Ability to `push` to a registry added.
+- Added `--dirty` option to `fair run` to allow running with uncommitted changes.
+- Added `config.yaml` file validation.
+- Added initialisation from existing registry.
+- Switch to setting port not local URI during initialisation.
+- Added option to specify port on `fair registry start`.
+- Added `cli-config.yaml` file validation during initialisation.
+- Added `fair reset` to list of commands.
+- Add tab completion for staging data products.
+
+# 2021-11-17 [v0.2.3](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.3)
+- Move handling of the user `config.yaml` file to a separate class `JobConfiguration`.
+- Added various fixes to improve functionality within Windows.
+- Move registry installation from script execution to internal function which sets up virtual environment etc.
+- Added a test suite to the project.
+- Added additional recognised identifiers for author setup from an organisation GRID and ROR.
+
+# 2021-10-06 [v0.2.2](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.2)
+- Update to package metadata for PyPi
+
+# 2021-10-06 [v0.2.1](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.1)
+- Automatic starter `config.yaml` generation.
+- Local and Global CLI configurations.
+- Start/stop local registry either explicitly or during synchronisations.
+- Run logs available via git-like interface.
+- Added ability to add/remove files.
+- Repository style handling, acts like another git-like tool per project.
+- Creation of an interface for `fair` using `click`.

{fair_cli-0.9.8 → fair_cli-0.10.0}/CITATION.cff

@@ -26,7 +26,7 @@ authors:
   affiliation: University of Glasgow
   orcid: https://orcid.org/0000-0002-4424-9890
 
-date-released: 2025-04-08
+date-released: 2026-10-03
 keywords:
 - FAIR Data Pipeline
 - FAIR
@@ -36,4 +36,4 @@ license: BSD-2-Clause
 message: If you use this software, please cite it using these metadata.
 repository-code: https://github.com/FAIRDataPipeline/FAIR-CLI/
 title: "The FAIR Data Pipeline command line tool"
-version: 0.9.8
+version: 0.10.0

{fair_cli-0.9.8 → fair_cli-0.10.0}/PKG-INFO

@@ -1,12 +1,13 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.1
 Name: fair-cli
-Version: 0.9.8
+Version: 0.10.0
 Summary: Synchronization interface for the SCRC FAIR Data Pipeline registry
+Home-page: https://www.fairdatapipeline.org/
 License: BSD-2-Clause
 Keywords: FAIR Data Pipeline,FAIR,Data Management,Provenance
 Author: Richard Reeve
 Author-email: richard.reeve@glasgow.ac.uk
-Requires-Python: >=3.9.0,<4.0.0
+Requires-Python: >=3.10.0,<4.0.0
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Science/Research
@@ -15,7 +16,6 @@ Classifier: Operating System :: MacOS
 Classifier: Operating System :: OS Independent
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
@@ -38,7 +38,6 @@ Requires-Dist: simplejson (>=3.17.5,<4.0.0)
 Requires-Dist: toml (>=0.10.2,<0.11.0)
 Requires-Dist: validators (>=0.18.2,<0.19.0)
 Project-URL: Documentation, https://www.fairdatapipeline.org/docs/interface/fdp/
-Project-URL: Homepage, https://www.fairdatapipeline.org/
 Project-URL: Issue Tracker, https://github.com/FAIRDataPipeline/FAIR-CLI/issues
 Project-URL: Repository, https://github.com/FAIRDataPipeline/FAIR-CLI
 Description-Content-Type: text/markdown

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/cli.py

@@ -616,9 +616,16 @@ def remote(ctx, verbose: bool = False, debug: bool = False):
 
 
 @remote.command()
-@click.argument("options", nargs=-1)
+@click.argument("options", nargs=-1, required=True)
+@click.option(
+    "--token",
+    "token_file",
+    help="File containing the API token for the remote registry",
+    prompt="Remote API Token File",
+    type=click.Path(exists=True, dir_okay=False, resolve_path=True),
+)
 @click.option("--debug/--no-debug", help="Run in debug mode", default=False)
-def add(options: typing.List[str], debug: bool) -> None:
+def add(options: typing.List[str], token_file: str, debug: bool) -> None:
     """Add a remote registry URL with option to give it a label if multiple
     remotes may be used.
 
@@ -628,13 +635,15 @@ def add(options: typing.List[str], debug: bool) -> None:
         size 1 or 2 list containing either:
             - label, url
             - url
+    token_file : str
+        file containing the API token for the remote registry
     """
     _url = options[1] if len(options) > 1 else options[0]
     _label = options[0] if len(options) > 1 else "origin"
 
     try:
         with fdp_session.FAIR(os.getcwd(), debug=debug) as fair_session:
-            fair_session.add_remote(_url, _label)
+            fair_session.add_remote(_url, token_file, _label)
     except fdp_exc.FAIRCLIException as e:
         e.err_print()
         if e.level.lower() == "error":
@@ -654,7 +663,7 @@ def remove(label: str, debug: bool) -> None:
     """
     try:
         with fdp_session.FAIR(os.getcwd(), debug=debug) as fair_session:
-            fair_session.remove_remove(label)
+            fair_session.remove_remote(label)
     except fdp_exc.FAIRCLIException as e:
         e.err_print()
         if e.level.lower() == "error":

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/common.py

@@ -19,7 +19,7 @@ Members
     FAIR_FOLDER     - name for FAIR local repository directory
 
 Functions
--------
+---------
     registry_home       - returns the location of the local data registry
     find_fair_root      - returns the closest '.fair' directory in the upper hierarchy
     find_git_root       - returns the closest '.git' directory

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/identifiers.py

@@ -192,10 +192,12 @@ def check_ror(ror: str) -> typing.Dict:
 
 def check_grid(grid_id: str) -> typing.Dict:
     """Checks if valid GRID ID using ROR (https://ror.org/) public api
+
     Parameters
     ----------
     grid_id : str
         GRID ID to be checked
+
     Returns
     -------
     typing.Dict
@@ -230,11 +232,20 @@ def _check_generic_ror(id: str) -> typing.Dict:
         logger.debug(f"{_url} Responded with {_response.status_code}")
         return _result_dict
 
-    if _response.json()["number_of_results"] == 0:
+    # A query ROR cannot parse (e.g. one containing "!") still returns 200, with
+    # "errors" in place of results
+    _items = _response.json().get("items")
+    if not _items:
+        logger.debug(f"{_url} returned no results: {_response.json().get('errors')}")
         return _result_dict
 
-    _id = _response.json()["items"][0]["id"]
-    _name = _response.json()["items"][0]["name"]
+    _id = _items[0]["id"]
+    # The ROR API lists every name of an organisation, marking the one to show as "ror_display"
+    _names = _items[0]["names"]
+    _name = next(
+        (n["value"] for n in _names if "ror_display" in n["types"]),
+        _names[0]["value"],
+    )
     _result_dict["name"] = _name
     _result_dict["family_name"] = _name
     _result_dict["given_names"] = None
@@ -246,7 +257,9 @@ def _check_generic_ror(id: str) -> typing.Dict:
 def check_id_permitted(identifier: str, retries: int = 5) -> bool:
     """Check a user provided identifier is permitted
 
-    This ID is expected to be a valid URL
+    This ID is expected to be a valid URL. Permitted means that it
+    resolves: the URL, after any redirects, is served, or it redirects to a
+    page that refuses access (401/403), as a publisher behind a DOI may.
 
     Parameters
     ----------
@@ -269,9 +282,16 @@ def check_id_permitted(identifier: str, retries: int = 5) -> bool:
             headers = {}
             if fake_agent:
                 headers = {"User-Agent": str(UserAgent().chrome)}
-            requests.get(
+            _response = requests.get(
                 identifier, verify=False, allow_redirects=True, headers=headers
-            ).raise_for_status()
+            )
+            # Some publishers answer 403 to any automated request whatever
+            # the user agent, so a refusal reached through a redirect (e.g.
+            # from doi.org) still means the identifier resolved. A 404
+            # anywhere is still rejected.
+            if _response.history and _response.status_code in (401, 403):
+                return True
+            _response.raise_for_status()
             return True
         except (
             requests.exceptions.MissingSchema,

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/register.py

@@ -13,7 +13,7 @@ Contents
 ========
 
 Functions
--------
+---------
 
 """
 

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/registry/requests.py

@@ -506,6 +506,11 @@ def download_file(url: str, chunk_size: int = 8192) -> str:
     -------
     str
         path of downloaded temporary file
+
+    Raises
+    ------
+    requests.HTTPError
+        if the server answers with an error status
     """
     # Save the data to a temporary file so we can calculate the hash
     _file = tempfile.NamedTemporaryFile(delete=False)
@@ -528,12 +533,15 @@ def download_file(url: str, chunk_size: int = 8192) -> str:
             response = requests.get(
                 url, allow_redirects=True, verify=False, headers=headers
             )
-            open(_fname, mode="wb").write(response.content)
         except Exception as e:
             raise fdp_exc.FAIRCLIException(
                 f"Failed to download file '{url}'"
                 f" due to connection error: {traceback.format_exc()}"
             ) from e
+        # Outside the try, so that callers can tell an HTTP error from a
+        # failure to connect, and an error page is never saved as the data
+        response.raise_for_status()
+        open(_fname, mode="wb").write(response.content)
 
     return _fname
 

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/registry/sync.py

@@ -226,6 +226,35 @@ def sync_dependency_chain(
     return _new_urls
 
 
+# Whether a storage_root record is on the pushing machine's file system
+def _is_local_root(storage_root: typing.Dict) -> bool:
+    return storage_root.get("root", "").startswith("file://")
+
+
+# URL of a registry's own data store: the storage_root whose root is the
+# registry's data/ path, beside its api/, from which it serves stored files.
+# A registry whose site URL differs from its API's address records that root
+# under another name; its first storage_root is then taken, as set_site_info
+# creates the data store first.
+def _remote_data_store_url(registry_uri: str, token: str) -> str:
+    _registry_uri = fdp_util.check_trailing_slash(registry_uri)
+    _root = urllib.parse.urljoin(_registry_uri, "../data/")
+    _data_store = fdp_req.get(
+        registry_uri, "storage_root", token, params={"root": _root}
+    )
+    if _data_store:
+        return _data_store[0]["url"]
+    _fallback = urllib.parse.urljoin(_registry_uri, "storage_root/1/")
+    logger.warning(
+        "Registry '%s' has no storage root '%s'; assuming its data store is "
+        "'%s'. Its site URL may not match its address.",
+        registry_uri,
+        _root,
+        _fallback,
+    )
+    return _fallback
+
+
 def _get_new_url(
     origin_uri: str,
     origin_token: str,
@@ -281,8 +310,9 @@ def _get_new_url(
         # Make sure that a URL for the component does exist
         if value not in new_urls:
             raise fdp_exc.RegistryError(
-                f"Expected URL from remote '{dest_uri}' for component "
-                f"'{key}' of local object '{value}' during push."
+                f"Expected a URL on '{dest_uri}' for component '{key}', "
+                f"whose value '{value}' should have been synchronised "
+                "before it."
             )
 
         # Retrieve from the new URLs the correct value and substitute
@@ -293,7 +323,6 @@ def _get_new_url(
     # (as remote URL will never match local)
 
     _obj_type = fdp_req.get_obj_type_from_url(object_url, token=origin_token)
-    _obj_id = fdp_req.get_obj_id_from_url(object_url)
 
     _filters = {
         k: v
@@ -311,20 +340,21 @@ def _get_new_url(
     # If Public is true then any files will be uploaded to the remote registry
     # If local_data_store is set we're pulling to a local registry
     if public and not local_data_store:
-        # The remote data_store storage_root URL should
-        # always be the 1st storage_root
-        _remote_storage_root_url = urllib.parse.urljoin(dest_uri, "storage_root/1/")
-        # If the current objects a storage_root and the root
-        # is the first (data_store) then simply return the remote
-        if _obj_type == "storage_root" and _obj_id == "1":
-            return _remote_storage_root_url
+        # A local data store is any root on the pusher's own file system,
+        # however many projects share the local registry; it is replaced by
+        # the remote data store. Other roots (e.g. https://github.com/) are
+        # pushed as they are.
+        if _obj_type == "storage_root" and _is_local_root(object_data):
+            return _remote_data_store_url(dest_uri, dest_token)
         # If the current object is a storage_location
         elif _obj_type == "storage_location":
-            # Again if the storage_root is the data_store (storage_root 1)
-            if (
-                fdp_req.get_obj_id_from_url(object_data.get("storage_root", "/2"))
-                == "1"
+            # Again if the storage_root is a local data store
+            if _is_local_root(
+                fdp_req.url_get(object_data["storage_root"], origin_token)
             ):
+                _remote_storage_root_url = _remote_data_store_url(
+                    dest_uri, dest_token
+                )
                 # Update the new object path and filter path
                 _new_obj_data["path"] = _filters["path"] = _new_obj_data["hash"]
                 # Update the new object storage_root
@@ -522,6 +552,11 @@ def sync_data_products(
             )
         result = result[0]
 
+        # Held separately because `result` becomes the external object below,
+        # and the file is fetched by the data product's own namespace, name
+        # and version
+        _data_product = result
+
         result_object = fdp_req.url_get(result["object"], token=origin_token)
         result_storage_location = fdp_req.url_get(
             result_object["storage_location"], token=origin_token
@@ -544,7 +579,7 @@ def sync_data_products(
         # If local_data_store assume we're syncing from remote to local
         if local_data_store:
             logger.debug("Retrieving files from remote registry data storage")
-            fetch_data_product(origin_token, local_data_store, result[0])
+            fetch_data_product(origin_token, local_data_store, _data_product)
         # Else going from local to remote
         else:
             # If the storage location is public upload the files to object storage
@@ -1142,7 +1177,7 @@ def fetch_data_product(
         _file_type = ""
 
     _local_dir = os.path.join(
-        local_data_store, _namespace["name"], data_product["data_product"]
+        local_data_store, _namespace["name"], data_product["name"]
     )
 
     os.makedirs(_local_dir, exist_ok=True)
@@ -1201,7 +1236,8 @@ def download_from_registry(registry_url: str, root: str, path: str) -> str:
         logger.debug("Downloaded file from '%s' to temporary file", _download_url)
     except requests.HTTPError as r_in:
         raise fdp_exc.UserConfigError(
-            f"Failed to fetch item '{_download_url}' with exit code {r_in.response}"
+            f"Failed to fetch item '{_download_url}' with status code "
+            f"{r_in.response.status_code}"
         ) from r_in
 
     return _temp_data_file

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/session.py

@@ -542,7 +542,10 @@ class FAIR:
 
         self._session_config.write()
 
-        self._session_config.execute()
+        # A passive run only prepares the job directory, for the model to be
+        # run from it separately
+        if not passive:
+            self._session_config.execute()
 
         self._post_job_breakdown(add_run=True)
 
@@ -754,11 +757,14 @@ class FAIR:
     def remove_remote(self, label: str) -> None:
         """Remove a remote URL from the list of remotes by label"""
         self.check_is_repo()
-        if "registries" not in self._local_config or label not in self._local_config:
+        if (
+            "registries" not in self._local_config
+            or label not in self._local_config["registries"]
+        ):
             raise fdp_exc.CLIConfigurationError(
                 f"No such entry '{label}' in available remotes"
             )
-        del self._local_config[label]
+        del self._local_config["registries"][label]
 
     def modify_remote(self, label: str, url: str) -> None:
         """Update a remote URL for a given remote"""
@@ -1089,7 +1095,8 @@ class FAIR:
             raise fdp_exc.CLIConfigurationError(
                 "Cannot generate user 'config.yaml'",
                 hint="You need to set the remote URL"
-                " by running: \n\n\tfair remote add <url>\n",
+                " by running: \n\n\tfair remote add <url>"
+                " --token <token-file>\n",
             )
 
         with open(output_file_name, encoding="utf-8", mode="w") as f:
@@ -1132,11 +1139,17 @@ class FAIR:
 
         _first_time = not os.path.exists(fdp_com.global_fdpconfig())
 
+        # Also with --ci: the data store lives in .fair/, and the registry
+        # still holds records pointing into it. 'fair purge' starts afresh.
+        if os.path.exists(_fair_dir):
+            if export_as:
+                self._export_cli_configuration(export_as)
+                return
+            else:
+                click.echo("FAIR repository is already initialised.")
+                return
+
         if self._testing:
-            if os.path.exists(_fair_dir):
-                if platform.system() == "Windows":
-                    fdp_com.set_file_permissions(_fair_dir)
-                shutil.rmtree(_fair_dir, onerror=fdp_com.remove_readonly)
             using = fdp_test.create_configurations(
                 registry,
                 fdp_com.find_git_root(os.getcwd()),
@@ -1144,15 +1157,9 @@ class FAIR:
                 os.path.join(os.getcwd(), ".fair"),
             )
 
-        if os.path.exists(_fair_dir) and not self._testing:
-            if export_as:
-                self._export_cli_configuration(export_as)
-                return
-            else:
-                click.echo("FAIR repository is already initialised.")
-                return
-
-        if _existing := fdp_com.find_fair_root(self._session_loc) and not self._testing:
+        if (
+            _existing := fdp_com.find_fair_root(self._session_loc)
+        ) and not self._testing:
             click.echo(
                 "A FAIR repository was initialised for this location at"
                 f" '{_existing}'"
@@ -1256,7 +1263,7 @@ class FAIR:
 
     def close_session(self) -> None:
         """Upon exiting, dump all configurations to file"""
-        if not os.path.exists(os.path.join(self._session_loc, fdp_com.FAIR_FOLDER)):
+        if not fdp_com.find_fair_root(self._session_loc):
             return
 
         if self._session_id:
@@ -1266,10 +1273,12 @@ class FAIR:
             )
             os.remove(_cache_addr)
 
-        if os.path.exists(fdp_com.global_config_dir()):
+        if os.path.exists(os.path.dirname(fdp_com.global_fdpconfig())):
             with open(fdp_com.global_fdpconfig(), encoding="utf-8", mode="w") as f:
                 yaml.dump(self._global_config, f)
-        if os.path.exists(os.path.dirname(fdp_com.local_fdpconfig())):
+        if os.path.exists(
+            os.path.dirname(fdp_com.local_fdpconfig(self._session_loc))
+        ):
             with open(
                 fdp_com.local_fdpconfig(self._session_loc), encoding="utf-8", mode="w"
             ) as f:

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/staging.py

@@ -418,9 +418,8 @@ class Stager:
         with open(self._staging_file, encoding="utf-8") as f:
             _staging_dict = yaml.safe_load(f)
 
-        result = fdp_req.url_get(
-            f"{fdp_com.DEFAULT_LOCAL_REGISTRY_URL}data_product",
-            fdp_req.local_token(),
+        result = fdp_req.get(
+            fdp_conf.get_local_uri(), "data_product", fdp_req.local_token()
         )
 
         for data_product in result:
@@ -441,9 +440,8 @@ class Stager:
         with open(self._staging_file, encoding="utf-8") as f:
             _staging_dict = yaml.safe_load(f)
 
-        result = fdp_req.url_get(
-            f"{fdp_com.DEFAULT_LOCAL_REGISTRY_URL}code_run",
-            fdp_req.local_token(),
+        result = fdp_req.get(
+            fdp_conf.get_local_uri(), "code_run", fdp_req.local_token()
         )
 
         for code_run in result:

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/user_config/__init__.py

@@ -93,6 +93,28 @@ SHELLS: typing.Dict[str, str] = {
 }
 
 
+# Apply a function to every string in a configuration's nested dicts and lists,
+# keys included, returning the new structure
+def _map_strings(item: typing.Any, func: typing.Callable) -> typing.Any:
+    if isinstance(item, str):
+        return func(item)
+    if isinstance(item, dict):
+        return {
+            _map_strings(k, func): _map_strings(v, func)
+            for k, v in item.items()
+        }
+    if isinstance(item, list):
+        return [_map_strings(v, func) for v in item]
+    return item
+
+
+# Every string in a configuration's nested dicts and lists, keys included
+def _strings_in(item: typing.Any) -> typing.List[str]:
+    _strings: typing.List[str] = []
+    _map_strings(item, lambda s: _strings.append(s) or s)
+    return _strings
+
+
 class JobConfiguration(MutableMapping):
     _logger = logging.getLogger("FAIRDataPipeline.ConfigYAML")
     _block_types = ("register", "write", "read")
@@ -408,6 +430,15 @@ class JobConfiguration(MutableMapping):
                 f" wildcard '{block_entry[_obj_type]}'"
             ) from e
 
+        # The registry's own filter lets '*' match across '/'
+        _results_local = [
+            result
+            for result in _results_local
+            if fdp_glob.matches_wildcard(
+                block_entry[_obj_type], result[_search_key]
+            )
+        ]
+
         if _obj_type in ("namespace", "author"):
             # If the object is a namespace or an author then there is no
             # additional info in the registry so we can just add the entries
@@ -425,12 +456,26 @@ class JobConfiguration(MutableMapping):
             )
 
         elif _obj_type == "data_product":
-            _version = block_entry.get("version", None)
+            _version = block_entry.get("use", {}).get(
+                "version", block_entry.get("version", None)
+            )
 
             _new_entries = fdp_glob.get_data_product_objects(
                 registry_token, _results_local, block_type, _version
             )
 
+            # A match is written again as the pattern entry describes it
+            # (file_type, description, public), not as its registry row
+            if block_type == "write":
+                for _new_entry in _new_entries:
+                    _new_entry.update(
+                        {
+                            k: v
+                            for k, v in block_entry.items()
+                            if k not in ("data_product", "use")
+                        }
+                    )
+
         if block_type == "write":
             _new_entries.append(block_entry)
 
@@ -637,7 +682,16 @@ class JobConfiguration(MutableMapping):
                     "not a valid git repository."
                 ) from e
 
-            _url = _git_repo.remotes[_remote].url
+            try:
+                _url = _git_repo.remotes[_remote].url
+            except IndexError as e:
+                raise fdp_exc.FDPRepositoryError(
+                    f"Failed to update job configuration from location '{fair_repo_dir}', "
+                    f"git repository '{_local_repo}' has no remote '{_remote}'.",
+                    hint="Add the remote to the repository, or give "
+                    "'run_metadata: remote_repo:' in the configuration.",
+                ) from e
+
             self["run_metadata.remote_repo"] = _url
 
     def pop(self, key: str, default: typing.Optional[typing.Any] = None) -> typing.Any:
@@ -808,7 +862,13 @@ class JobConfiguration(MutableMapping):
         # Additional parser for formatted datetime
         _regex_fmt = re.compile(r"\$\{\{\s*([^}${\s]+)\s*\}\}")
 
-        if _unparsed := _regex_fmt.findall(_conf_str):
+        # RUN_ID is the code run's UUID, which does not exist until the API
+        # creates the code run, so the API substitutes it at finalise
+        _unparsed = [
+            var for var in _regex_fmt.findall(_conf_str) if var != "RUN_ID"
+        ]
+
+        if _unparsed:
             raise fdp_exc.InternalError(f"Failed to parse variables '{_unparsed}'")
 
     def _subst_cli_vars(self, job_time: datetime.datetime) -> str:
@@ -822,11 +882,14 @@ class JobConfiguration(MutableMapping):
 
         def _tag_check():
             _repo = git.Repo(fdp_conf.local_git_repo(self.local_repository))
-            if len(_repo.tags) < 1:
-                fdp_exc.UserConfigError(
-                    "Cannot use GIT_TAG variable, no git tags found."
-                )
-            return _repo.tags[-1].name
+            # The nearest tag in the history of HEAD, not the last by name
+            try:
+                return _repo.git.describe("--tags", "--abbrev=0")
+            except git.GitCommandError as e:
+                raise fdp_exc.UserConfigError(
+                    "Cannot use GIT_TAG variable, no git tags found "
+                    "in the history of the current commit."
+                ) from e
 
         _substitutes: typing.Dict[str, typing.Callable] = {
             "DATE": lambda: job_time.strftime("%Y%m%d"),
@@ -842,32 +905,20 @@ class JobConfiguration(MutableMapping):
             "GIT_TAG": _tag_check,
         }
 
+        # Substitutions are made in the parsed configuration, never in its
+        # YAML text, so that a value is inserted as it is: a Windows path's
+        # backslashes are neither regular-expression nor YAML escapes
+
         # Additional parser for formatted datetime
-        _regex_dt_fmt = re.compile(r"\$\{\{\s*DATETIME\-[^}${\s]]+\s*\}\}")
         _regex_fmt = re.compile(r"\$\{\{\s*DATETIME\-([^}${\s]+)\s*\}\}")
 
-        _config_str: str = yaml.dump(self._config)
-
-        _dt_fmt_res: typing.Optional[typing.List[str]] = _regex_dt_fmt.findall(
-            _config_str
-        )
-        _fmt_res: typing.Optional[typing.List[str]] = _regex_fmt.findall(_config_str)
-
-        self._logger.debug(
-            "Found datetime substitutions: %s %s",
-            _dt_fmt_res or "",
-            _fmt_res or "",
+        self._config = _map_strings(
+            self._config,
+            lambda s: _regex_fmt.sub(
+                lambda m: job_time.strftime(m.group(1).strip()), s
+            ),
         )
 
-        # The two regex searches should match lengths
-        if len(_dt_fmt_res) != len(_fmt_res):
-            raise fdp_exc.UserConfigError("Failed to parse formatted datetime variable")
-
-        if _dt_fmt_res:
-            for i, _ in enumerate(_dt_fmt_res):
-                _time_str = job_time.strftime(_fmt_res[i].strip())
-                _config_str = _config_str.replace(_dt_fmt_res[i], _time_str)
-
         _regex_dict = {
             var: r"\$\{\{\s*" + f"{var}" + r"\s*\}\}" for var in _substitutes
         }
@@ -875,16 +926,16 @@ class JobConfiguration(MutableMapping):
         # Perform string substitutions
         for var, subst in _regex_dict.items():
             # Only execute functions in var substitutions that are required
-            if re.findall(subst, _config_str):
-                _value = _substitutes[var]()
+            if any(re.search(subst, s) for s in _strings_in(self._config)):
+                _value = str(_substitutes[var]() or "")
                 if not _value:
                     raise fdp_exc.InternalError(
                         f"Expected value for substitution of '{var}' but returned None",
                     )
-                _config_str = re.sub(subst, str(_value), _config_str)
-                self._logger.debug("Substituting %s: %s", var, str(_value))
-
-        self._config = yaml.safe_load(_config_str)
+                self._config = _map_strings(
+                    self._config, lambda s: re.sub(subst, lambda _: _value, s)
+                )
+                self._logger.debug("Substituting %s: %s", var, _value)
 
     def _register_to_read(
         self, register_block: typing.List[typing.Dict]
@@ -1010,16 +1061,18 @@ class JobConfiguration(MutableMapping):
 
             _version = item["use"]["version"]
 
-            if "data_product" not in item["use"]:
-                if "external_object" in item and "*" in item["external_object"]:
-                    _name = item["external_object"]
-                elif "data_product" in item and "*" in item["data_product"]:
-                    _name = item["data_product"]
-                else:
-                    self._logger.warning(f"Missing use:data_product in {item}")
+            if "data_product" in item["use"]:
+                _name = item["use"]["data_product"]
+            elif "external_object" in item and "*" in item["external_object"]:
+                _name = item["external_object"]
+            elif "data_product" in item and "*" in item["data_product"]:
+                _name = item["data_product"]
+            else:
+                self._logger.warning(f"Missing use:data_product in {item}")
                 continue
-
-            _name = item["use"]["data_product"]
+            # Only a wildcard entry lacks this: kept in the write block as the
+            # template for names the model writes that match it
+            _new_item["use"]["data_product"] = _name
             _namespace = item["use"]["namespace"]
 
             # If no ID exists for the namespace then this object has not yet
@@ -1055,6 +1108,13 @@ class JobConfiguration(MutableMapping):
             except fdp_exc.RegistryError:
                 _results = []
 
+            if "*" in _name:
+                _results = [
+                    result
+                    for result in _results
+                    if fdp_glob.matches_wildcard(_name, result["name"])
+                ]
+
             try:
                 _version = fdp_ver.get_correct_version(
                     version=_version,
@@ -1180,7 +1240,7 @@ class JobConfiguration(MutableMapping):
     def shell(self) -> str:
         """Retrieve the shell choice"""
         _shell_default = "batch" if platform.system() == "Windows" else "sh"
-        return self.get("shell", _shell_default)
+        return self.get("run_metadata.shell", _shell_default)
 
     @property
     def local_repository(self) -> str:
@@ -1312,11 +1372,18 @@ class JobConfiguration(MutableMapping):
             encoding="utf-8",
         )
 
+        # The job log is UTF-8, but the console may not be (cp1252 on a
+        # Windows runner), and one unencodable character must not stop the run
+        _encoding = sys.stdout.encoding or "utf-8"
+
         # Write any stdout to the job log
         for line in iter(_process.stdout.readline, ""):
             self._log_file.writelines([line])
             _log_tail.append(line)
-            click.echo(line, nl=False)
+            click.echo(
+                line.encode(_encoding, errors="replace").decode(_encoding),
+                nl=False,
+            )
             sys.stdout.flush()
 
         _process.wait()

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/user_config/globbing.py

@@ -14,6 +14,7 @@ Constants
     - DISPOSABLES: tuple of keys to be removed before adding to config.yaml
 
 """
+import re
 import typing
 
 import fair.exceptions as fdp_exc
@@ -38,6 +39,29 @@ DISPOSABLES = (
 )
 
 
+def matches_wildcard(pattern: str, name: str) -> bool:
+    """Return whether a name matches a wildcard pattern
+
+    Each '*' matches one segment of the name: one or more characters other
+    than '/'. The pattern is anchored at both ends, and every other character
+    matches itself.
+
+    Parameters
+    ----------
+    pattern : str
+        name containing one or more '*' wildcards
+    name : str
+        name to test against the pattern
+
+    Returns
+    -------
+    bool
+        whether the name matches
+    """
+    _regex = "[^/]+".join(re.escape(part) for part in pattern.split("*"))
+    return re.fullmatch(_regex, name) is not None
+
+
 def get_single_layer_objects(
     results_list: typing.List[typing.Dict], object_type: str
 ) -> typing.List[typing.Dict]:

{fair_cli-0.9.8 → fair_cli-0.10.0}/fair/utilities.py

@@ -202,9 +202,19 @@ def is_api_url(uri: str, string: str) -> bool:
     _uri = urllib.parse.urlparse(uri)
 
     logger.debug(
-        "Checking if '%s' is a valid API URL against net location '%s'",
+        "Checking if '%s' is a valid API URL against endpoint '%s'",
         string,
-        _uri.netloc,
+        uri,
     )
 
-    return _url.netloc == _uri.netloc
+    if _url.netloc != _uri.netloc:
+        return False
+
+    # The API is served under a path, and a registry serves other things from
+    # the same host - a remote registry's own data store is <domain>/data/ -
+    # so the path has to match too, or a StorageRoot's root is taken for a
+    # reference to another object. An endpoint given as a bare host has no
+    # path to compare, and then the net location is all there is to go on.
+    _api_path = check_trailing_slash(_uri.path) if _uri.path else "/"
+
+    return _url.path.startswith(_api_path)

{fair_cli-0.9.8 → fair_cli-0.10.0}/pyproject.toml

@@ -10,7 +10,6 @@ authors = [
 classifiers = [
   "License :: OSI Approved :: BSD License",
   "Intended Audience :: Science/Research",
-  "Programming Language :: Python :: 3.9",
   "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
   "Programming Language :: Python :: 3.12",
@@ -24,7 +23,7 @@ classifiers = [
 ]
 description = "Synchronization interface for the SCRC FAIR Data Pipeline registry"
 name = "fair-cli"
-version = "0.9.8"
+version = "0.10.0"
 
 homepage = "https://www.fairdatapipeline.org/"
 
@@ -56,7 +55,7 @@ click = "^8.0.0"
 email-validator = "^1.1.3"
 pre-commit = "^3"
 pydantic = {extras = ["email"], version = "^1.9.0"}
-python = "^3.9.0,<4.0"
+python = "^3.10.0,<4.0"
 requests = "^2.23.0"
 rich = ">=10.2.3,<12.0.0"
 semver = "^2.13.0"
@@ -65,7 +64,7 @@ toml = "^0.10.2"
 validators = "^0.18.2"
 fake-useragent = "^1"
 
-[tool.poetry.dev-dependencies]
+[tool.poetry.group.dev.dependencies]
 bandit = "*"
 black = "*"
 coverage = "*"
@@ -77,7 +76,6 @@ poetry = "^1.1.12"
 pre-commit = "*"
 pycodestyle = "^2.7.0"
 pydocstyle = "^6.0.0"
-pylama = "^8"
 pytest = "*"
 pytest-shutil = "*"
 pytest-cov = "*"
@@ -89,6 +87,13 @@ moto = "^5"
 Flask = "^3"
 Flask-Cors = "^4"
 matplotlib = "*"
+# Brought in by matplotlib. Without a split, one numpy must serve the whole
+# Python range, and numpy >= 2.3 needs Python >= 3.11, so every version got
+# 2.2.6, which has no wheels for 3.14. Delete when the floor reaches 3.11.
+numpy = [
+    { version = ">=2.3", python = ">=3.11" },
+    { version = "*", python = "<3.11" },
+]
 
 [tool.poetry.scripts]
 fair = 'fair.cli:cli'

fair_cli-0.9.8/CHANGELOG.md

@@ -1,33 +0,0 @@
-# Unreleased
-- Fix virtual environment bug in `fair` binaries where `venv` assumes the main executable is `python`.
-- Registration of `author` and `namespace` objects.
-- Allow specification of tag to install registry from during `fair registry install`.
-- Wildcard '*' parsing introduced for data products.
-- Ability to `push` to a registry added.
-- Added `--dirty` option to `fair run` to allow running with uncommitted changes.
-- Added `config.yaml` file validation.
-- Added initialisation from existing registry.
-- Switch to setting port not local URI during initialisation.
-- Added option to specify port on `fair registry start`.
-- Added `cli-config.yaml` file validation during initialisation.
-- Added `fair reset` to list of commands.
-- Add tab completion for staging data products.
-
-# 2021-11-17 [v0.2.3](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.3)
-- Move handling of the user `config.yaml` file to a separate class `JobConfiguration`.
-- Added various fixes to improve functionality within Windows.
-- Move registry installation from script execution to internal function which sets up virtual environment etc.
-- Added a test suite to the project.
-- Added additional recognised identifiers for author setup from an organisation GRID and ROR.
-
-# 2021-10-06 [v0.2.2](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.2)
-- Update to package metadata for PyPi
-
-# 2021-10-06 [v0.2.1](https://github.com/FAIRDataPipeline/FAIR-CLI/releases/tag/v0.2.1)
-- Automatic starter `config.yaml` generation.
-- Local and Global CLI configurations.
-- Start/stop local registry either explicitly or during synchronisations.
-- Run logs available via git-like interface.
-- Added ability to add/remove files.
-- Repository style handling, acts like another git-like tool per project.
-- Creation of an interface for `fair` using `click`.