h5rdmtoolbox 2.8.1__tar.gz → 2.8.2__tar.gz

{h5rdmtoolbox-2.8.1 → h5rdmtoolbox-2.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: h5rdmtoolbox
-Version: 2.8.1
+Version: 2.8.2
 Summary: Supporting a FAIR Research Data lifecycle using Python and HDF5.
 Author-email: Matthias Probst <matth.probst@gmail.com>
 License-Expression: MIT
@@ -41,7 +41,7 @@ Requires-Dist: typer>=0.12
 Requires-Dist: cftime>=1.6.4
 Requires-Dist: pyshacl<1.0.0,>=0.30.1
 Provides-Extra: database
-Requires-Dist: pymongo<=4.10.1,>=4.2.0; extra == "database"
+Requires-Dist: pymongo<=4.10.1,>=4.6.3; extra == "database"
 Provides-Extra: layout-validation
 Requires-Dist: tabulate<=0.9.0,>=0.8.10; extra == "layout-validation"
 Provides-Extra: csv
@@ -81,7 +81,7 @@ Requires-Dist: h5rdmtoolbox[complete,docs]; extra == "complete-with-docs"
 Provides-Extra: server
 Requires-Dist: fastapi>=0.95; extra == "server"
 Requires-Dist: uvicorn[standard]>=0.22; extra == "server"
-Requires-Dist: jinja2>=3.1; extra == "server"
+Requires-Dist: jinja2>=3.1.6; extra == "server"
 Dynamic: license-file
 
 # HDF5 Research Data Management Toolbox
@@ -91,6 +91,7 @@ Dynamic: license-file
 [![Documentation Status](https://readthedocs.org/projects/h5rdmtoolbox/badge/?version=latest)](https://h5rdmtoolbox.readthedocs.io/en/latest/?badge=latest)
 [![PyPI version](https://badge.fury.io/py/h5RDMtoolbox.svg)](https://badge.fury.io/py/h5RDMtoolbox)
 ![pyvers](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)
+[![Paper](https://img.shields.io/badge/Paper-View-green?style=flat&logo=adobeacrobatreader)](https://preprints.inggrid.org/repository/view/23/)
 
 A Python package that helps researchers achieve sustainable 
 [**FAIR (Findable, Accessible, Interoperable, Reusable)**](https://www.nature.com/articles/sdata201618) data management with HDF5 files. The toolbox provides a comprehensive
@@ -257,7 +258,7 @@ h5tbx serve data/ --h5ext=.h5 --h5ext=.hdf5
 ```
 This allows you to serve a single HDF5 file or a directory of HDF5 files, and view their metadata and structure in a web interface.
 
-You may before SPARQL queries or view the File in a graph as shown below:
+You may perform SPARQL queries or view the file as a graph, as shown below:
 
 <p align="center">
   <img src="docs/_static/screenshot-served-hdf5-as-graph.png" alt="Serve UI screenshot" width="640">
@@ -318,7 +319,7 @@ module implements the main interface between the user and the HDF5 file. It
 extends the features of the underlying `h5py` library. Some of the features are implemented in other modules, hence the
 wrapper module depends on the convention, database and linked data (ld) module.
 
-<a href="https://h5rdmtoolbox.readthedocs.io/en/latest/"><img src="docs/_static/h5tbx_modules.svg" alt="H5TBX modules" style="widht:600px;"></a>
+<a href="https://h5rdmtoolbox.readthedocs.io/en/latest/"><img src="docs/_static/h5tbx_modules.svg" alt="H5TBX modules" style="width:600px;"></a>
 
 Current implementation highlights in the modules:
 
@@ -332,7 +333,7 @@ Current implementation highlights in the modules:
 - For the **repository** module, a Zenodo interface is implemented. Zenodo is a repository, which allows to upload and
   download data with a persistent identifier.
 - For the **convention** module,
-  the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
+  the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html)
   are implemented.
 - The **layout** module allows to define expectations on the internal layout (object names, location, attributes,
   properties) of HDF5 files.
@@ -356,7 +357,7 @@ A paper is published in the journal [inggrid](https://preprints.inggrid.org/repo
 
 ## Installation
 
-Use python 3.9 or higher (automatic testing is performed until 3.13). If you are a regular user, you can install the
+Use Python 3.9 through 3.13. If you are a regular user, you can install the
 package via pip:
 
     pip install h5RDMtoolbox
@@ -365,7 +366,7 @@ package via pip:
 
 Developers may clone the repository and install the package from source. Clone the repository first:
 
-    git clone https://github.com/matthiasprobst/h5RDMtoolbox.git@main
+    git clone --branch main https://github.com/matthiasprobst/h5RDMtoolbox.git
 
 Then, run
 
@@ -387,6 +388,7 @@ specific to the features of the package:
 - `numpy`: Scientific computing, handling of arrays
 - `matplotlib`: Plotting
 - `appdirs`: Managing user and application directories
+- `importlib_resources`: Accessing package resources consistently
 - `packaging`: Version handling
 - `IPython`: Pretty display of data in notebooks
 - `regex`: Working with regular expressions
@@ -399,14 +401,17 @@ specific to the features of the package:
 - `pint`: Allows working with units
 - `pint_xarray`: Working with units for usage with xarray
 - `python-forge`: Used to update function signatures when using
-  the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
+  the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html)
 - `pydantic`: Used to
-  validate [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
+  validate [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html)
 - `pyyaml`: Reading and writing of yaml files, e.g. metadata definitions (conventions). Note, lower versions
   collide with python 3.11
 - `requests`: Used to download files from the internet or validate URLs, e.g. metadata definitions (conventions)
 - `rdflib`: Used to enable working with RDF
 - `ontolutils`: Required to work with RDF and derive semantic description of HDF5 file content
+- `cftime`: Handling non-standard calendars and time coordinates
+- `pyshacl`: Validating RDF graphs against SHACL shapes
+- `typer`: Providing the `h5tbx` command-line interface
 
 #### Optional dependencies
 
@@ -414,9 +419,9 @@ To run unit tests or to enable certain features, additional dependencies must be
 
 Install optional dependencies by specifying them in square brackets after the package name, e.g.:
 
-    pip install h5RDMtoolbox[mongodb]
+    pip install h5RDMtoolbox[database]
 
-[mongodb]
+[database]
 
 - `pymongo`: Database solution for HDF5 files
 
@@ -431,6 +436,10 @@ Install optional dependencies by specifying them in square brackets after the pa
 - `python-gitlab`: Access to gitlab repositories
 - `pypandoc`: Conversion of markdown files to html
 
+Other supported extras are `layout_validation`, `catalog`, `gui`, `server`, `test`, `docs`, `complete`, and
+`complete-with-docs`. See the [installation guide](https://h5rdmtoolbox.readthedocs.io/en/latest/gettingstarted/installation.html)
+for their purpose. The authoritative dependency lists and version constraints are in [`pyproject.toml`](./pyproject.toml).
+
 ## Citing the package
 
 If you intend to use the package in your work, you may cite the software itself as published on paper in the
@@ -463,10 +472,10 @@ Here is the BibTeX entry:
 Feel free to contribute. Make sure to write `docstrings` to your methods and classes and please write tests and use PEP
 8 (https://peps.python.org/pep-0008/)
 
-Please write tests for your code and put them into the `test/` folder. Visit the [README file](./tests/README.md) in the
-test-folder for more information.
+Please write tests for your code and put them into the `tests/` folder. Visit the [README file](./tests/README.md) in the
+tests folder for more information.
 
-Pleas also add a jupyter notebook in the `docs/` folder in order to document your code. Please visit
+Please also add a Jupyter notebook in the `docs/` folder in order to document your code. Please visit
 the [README file](./docs/README.md) in the docs-folder for more information on how to compile the documentation.
 
 Please use the **numpy style for the docstrings**:

{h5rdmtoolbox-2.8.1 → h5rdmtoolbox-2.8.2}/README.md

@@ -5,6 +5,7 @@
 [![Documentation Status](https://readthedocs.org/projects/h5rdmtoolbox/badge/?version=latest)](https://h5rdmtoolbox.readthedocs.io/en/latest/?badge=latest)
 [![PyPI version](https://badge.fury.io/py/h5RDMtoolbox.svg)](https://badge.fury.io/py/h5RDMtoolbox)
 ![pyvers](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)
+[![Paper](https://img.shields.io/badge/Paper-View-green?style=flat&logo=adobeacrobatreader)](https://preprints.inggrid.org/repository/view/23/)
 
 A Python package that helps researchers achieve sustainable 
 [**FAIR (Findable, Accessible, Interoperable, Reusable)**](https://www.nature.com/articles/sdata201618) data management with HDF5 files. The toolbox provides a comprehensive
@@ -171,7 +172,7 @@ h5tbx serve data/ --h5ext=.h5 --h5ext=.hdf5
 ```
 This allows you to serve a single HDF5 file or a directory of HDF5 files, and view their metadata and structure in a web interface.
 
-You may before SPARQL queries or view the File in a graph as shown below:
+You may perform SPARQL queries or view the file as a graph, as shown below:
 
 <p align="center">
   <img src="docs/_static/screenshot-served-hdf5-as-graph.png" alt="Serve UI screenshot" width="640">
@@ -232,7 +233,7 @@ module implements the main interface between the user and the HDF5 file. It
 extends the features of the underlying `h5py` library. Some of the features are implemented in other modules, hence the
 wrapper module depends on the convention, database and linked data (ld) module.
 
-<a href="https://h5rdmtoolbox.readthedocs.io/en/latest/"><img src="docs/_static/h5tbx_modules.svg" alt="H5TBX modules" style="widht:600px;"></a>
+<a href="https://h5rdmtoolbox.readthedocs.io/en/latest/"><img src="docs/_static/h5tbx_modules.svg" alt="H5TBX modules" style="width:600px;"></a>
 
 Current implementation highlights in the modules:
 
@@ -246,7 +247,7 @@ Current implementation highlights in the modules:
 - For the **repository** module, a Zenodo interface is implemented. Zenodo is a repository, which allows to upload and
   download data with a persistent identifier.
 - For the **convention** module,
-  the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
+  the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html)
   are implemented.
 - The **layout** module allows to define expectations on the internal layout (object names, location, attributes,
   properties) of HDF5 files.
@@ -270,7 +271,7 @@ A paper is published in the journal [inggrid](https://preprints.inggrid.org/repo
 
 ## Installation
 
-Use python 3.9 or higher (automatic testing is performed until 3.13). If you are a regular user, you can install the
+Use Python 3.9 through 3.13. If you are a regular user, you can install the
 package via pip:
 
     pip install h5RDMtoolbox
@@ -279,7 +280,7 @@ package via pip:
 
 Developers may clone the repository and install the package from source. Clone the repository first:
 
-    git clone https://github.com/matthiasprobst/h5RDMtoolbox.git@main
+    git clone --branch main https://github.com/matthiasprobst/h5RDMtoolbox.git
 
 Then, run
 
@@ -301,6 +302,7 @@ specific to the features of the package:
 - `numpy`: Scientific computing, handling of arrays
 - `matplotlib`: Plotting
 - `appdirs`: Managing user and application directories
+- `importlib_resources`: Accessing package resources consistently
 - `packaging`: Version handling
 - `IPython`: Pretty display of data in notebooks
 - `regex`: Working with regular expressions
@@ -313,14 +315,17 @@ specific to the features of the package:
 - `pint`: Allows working with units
 - `pint_xarray`: Working with units for usage with xarray
 - `python-forge`: Used to update function signatures when using
-  the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
+  the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html)
 - `pydantic`: Used to
-  validate [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
+  validate [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html)
 - `pyyaml`: Reading and writing of yaml files, e.g. metadata definitions (conventions). Note, lower versions
   collide with python 3.11
 - `requests`: Used to download files from the internet or validate URLs, e.g. metadata definitions (conventions)
 - `rdflib`: Used to enable working with RDF
 - `ontolutils`: Required to work with RDF and derive semantic description of HDF5 file content
+- `cftime`: Handling non-standard calendars and time coordinates
+- `pyshacl`: Validating RDF graphs against SHACL shapes
+- `typer`: Providing the `h5tbx` command-line interface
 
 #### Optional dependencies
 
@@ -328,9 +333,9 @@ To run unit tests or to enable certain features, additional dependencies must be
 
 Install optional dependencies by specifying them in square brackets after the package name, e.g.:
 
-    pip install h5RDMtoolbox[mongodb]
+    pip install h5RDMtoolbox[database]
 
-[mongodb]
+[database]
 
 - `pymongo`: Database solution for HDF5 files
 
@@ -345,6 +350,10 @@ Install optional dependencies by specifying them in square brackets after the pa
 - `python-gitlab`: Access to gitlab repositories
 - `pypandoc`: Conversion of markdown files to html
 
+Other supported extras are `layout_validation`, `catalog`, `gui`, `server`, `test`, `docs`, `complete`, and
+`complete-with-docs`. See the [installation guide](https://h5rdmtoolbox.readthedocs.io/en/latest/gettingstarted/installation.html)
+for their purpose. The authoritative dependency lists and version constraints are in [`pyproject.toml`](./pyproject.toml).
+
 ## Citing the package
 
 If you intend to use the package in your work, you may cite the software itself as published on paper in the
@@ -377,10 +386,10 @@ Here is the BibTeX entry:
 Feel free to contribute. Make sure to write `docstrings` to your methods and classes and please write tests and use PEP
 8 (https://peps.python.org/pep-0008/)
 
-Please write tests for your code and put them into the `test/` folder. Visit the [README file](./tests/README.md) in the
-test-folder for more information.
+Please write tests for your code and put them into the `tests/` folder. Visit the [README file](./tests/README.md) in the
+tests folder for more information.
 
-Pleas also add a jupyter notebook in the `docs/` folder in order to document your code. Please visit
+Please also add a Jupyter notebook in the `docs/` folder in order to document your code. Please visit
 the [README file](./docs/README.md) in the docs-folder for more information on how to compile the documentation.
 
 Please use the **numpy style for the docstrings**:

{h5rdmtoolbox-2.8.1 → h5rdmtoolbox-2.8.2}/h5rdmtoolbox/convention/core.py

@@ -306,7 +306,10 @@ class Convention(AbstractConvention):
 
     @classmethod
     def from_yaml(
-        cls, yaml_filename: Union[str, pathlib.Path], overwrite: bool = False
+        cls,
+        yaml_filename: Union[str, pathlib.Path],
+        overwrite: bool = False,
+        allow_python_validators: bool = True,
     ):
         """Create a convention from a yaml file.
         The YAML file must have the following structure:
@@ -333,6 +336,8 @@ class Convention(AbstractConvention):
             path to the yaml file
         overwrite: bool
             if True, overwrite an existing (registered) convention with the same name
+        allow_python_validators: bool
+            Whether to copy and execute an adjacent ``*_vfuncs.py`` module.
 
         Returns
         -------
@@ -364,8 +369,10 @@ class Convention(AbstractConvention):
                 f'YAML file {yaml_filename} does not contain "__contact__". Is the file a valid convention?'
             )
 
+        from . import generate
+
         # check if name already exists!
-        convention_name = attrs["__name__"].lower().replace("-", "_")
+        convention_name = generate.normalize_convention_name(attrs["__name__"])
         if convention_name in [d.name for d in CV_DIR.glob("*")]:
             if not overwrite:
                 return _get_convention_from_dir(attrs["__name__"])
@@ -375,10 +382,10 @@ class Convention(AbstractConvention):
                 f'Convention exists and overwrite is True: Deleting convention "{convention_name}"'
             )
 
-        from . import generate
-
         generate.write_convention_module_from_yaml(
-            yaml_filename, name=attrs["__name__"]
+            yaml_filename,
+            name=attrs["__name__"],
+            allow_python_validators=allow_python_validators,
         )
 
         # add_convention(yaml_filename, name=attrs['__name__'])
@@ -504,9 +511,6 @@ class Convention(AbstractConvention):
         """
         from ..wrapper.core import File
 
-        if not isinstance(file_or_filename, (str, pathlib.Path)):
-            with File(file_or_filename, "r") as f:
-                return self.check(f)
         failed = []
 
         convention = self
@@ -575,13 +579,19 @@ class Convention(AbstractConvention):
                                         )
                                     )
 
-        with File(file_or_filename, "r") as f:
+        def _validate_file(f):
             logger.debug(
                 f"Checking file {file_or_filename} for compliance with convention {self.name}"
             )
             _validate_convention("/", f)
             f.visititems(_validate_convention)
 
+        if isinstance(file_or_filename, (str, pathlib.Path)):
+            with File(file_or_filename, "r") as f:
+                _validate_file(f)
+        else:
+            _validate_file(file_or_filename)
+
         return failed
 
 
@@ -621,8 +631,9 @@ def _import_convention(convention_name) -> "module":
 
 
 def _get_convention_from_dir(convention_name: str) -> "Convention":
-    _convention_name = convention_name.lower().replace("-", "_")
-    assert "-" not in _convention_name
+    from .generate import normalize_convention_name
+
+    _convention_name = normalize_convention_name(convention_name)
     if _convention_name in get_registered_conventions():
         return get_registered_conventions()[convention_name]
     _convention_py_filename = CV_DIR / f"{_convention_name}" / f"{_convention_name}.py"
@@ -799,19 +810,22 @@ def delete(convention: Union[str, Convention]):
         convention_name = convention.name
     else:
         convention_name = convention
-    cv_dir = CV_DIR / convention_name
+    from .generate import normalize_convention_name
+
+    safe_convention_name = normalize_convention_name(convention_name)
+    cv_dir = CV_DIR / safe_convention_name
     if cv_dir.exists():
-        shutil.rmtree(CV_DIR / convention_name)
+        shutil.rmtree(cv_dir)
     cfg._registered_conventions.pop(convention_name, None)
     if convention_name in sys.modules:
         # if the convention (py script) already has been imported, remove it from the list of imported modules:
         del sys.modules[convention_name]
 
 
-def from_file(filename) -> Convention:
+def from_file(filename, allow_python_validators: bool = True) -> Convention:
     """Load a convention from a file. Currently yaml and json files are supported"""
     if filename.suffix == ".yaml":
-        return from_yaml(filename)
+        return from_yaml(filename, allow_python_validators=allow_python_validators)
     elif filename.suffix == ".json":
         return from_json(filename)
     else:
@@ -819,11 +833,17 @@ def from_file(filename) -> Convention:
 
 
 def from_yaml(
-    filename: Union[str, pathlib.Path], overwrite: bool = False
+    filename: Union[str, pathlib.Path],
+    overwrite: bool = False,
+    allow_python_validators: bool = True,
 ) -> Convention:
     """Load a convention from a YAML file. See Convention.from_yaml() for details"""
     logger.debug(f"Reading Convention from yaml file: {filename}")
-    return Convention.from_yaml(filename, overwrite=overwrite)
+    return Convention.from_yaml(
+        filename,
+        overwrite=overwrite,
+        allow_python_validators=allow_python_validators,
+    )
 
 
 def from_json(
@@ -833,7 +853,11 @@ def from_json(
     return Convention.from_json(filename, overwrite=overwrite)
 
 
-def from_repo(repo_interface: RepositoryInterface, name: str):
+def from_repo(
+        repo_interface: RepositoryInterface,
+        name: str,
+        allow_python_validators: bool = False,
+):
     """Download a YAML file from a repository
 
     Parameters
@@ -842,6 +866,9 @@ def from_repo(repo_interface: RepositoryInterface, name: str):
         The repository interface to use for downloading the file
     name: str
         Name of the file to download
+    allow_python_validators: bool
+        Whether to download and execute a repository-provided ``*_vfuncs.py``
+        file. Disabled by default because it is executable code.
     """
     logger.debug(f"Downloading file {name} from repository {repo_interface}")
 
@@ -849,19 +876,20 @@ def from_repo(repo_interface: RepositoryInterface, name: str):
     _suffix = pathlib.Path(name).suffix
     vfunc_filename = name.split(_suffix, 1)
     has_vfunc = False
-    if len(vfunc_filename) == 2:
+    downloaded_vfunc_filename = None
+    if allow_python_validators and len(vfunc_filename) == 2:
         vfunc_filename = f"{vfunc_filename[0]}_vfuncs.py"
         try:
             downloaded_vfunc_filename = repo_interface.download_file(vfunc_filename)
             has_vfunc = True
         except Exception as e:
             logger.debug(f"No vfuncs file found for {name}: {e}")
-    if has_vfunc:
+    if has_vfunc and downloaded_vfunc_filename is not None:
         shutil.copy(
             downloaded_vfunc_filename, filename.parent / downloaded_vfunc_filename.name
         )
     logger.debug(f"File downloaded to {filename}. Now loading convention from file.")
-    return from_file(filename)
+    return from_file(filename, allow_python_validators=allow_python_validators)
 
 
 def from_zenodo(
@@ -869,6 +897,7 @@ def from_zenodo(
     name: str = None,
     overwrite: bool = False,
     force_download: bool = False,
+    allow_python_validators: bool = False,
 ) -> Convention:
     """Download a YAML file from a zenodo repository
 
@@ -885,6 +914,9 @@ def from_zenodo(
         Whether to overwrite existing convention with the same name. Default is False
     force_download: bool
         Whether to force download the file even if it is already cached. Default is False
+    allow_python_validators: bool
+        Whether to download and execute repository-provided validator Python.
+        Disabled by default.
 
     Returns
     -------
@@ -910,10 +942,18 @@ def from_zenodo(
             yaml_matches = [
                 file for file in filenames if pathlib.Path(file).suffix == ".yaml"
             ]
-            vfuns_matches = [file for file in filenames if file.endswith("vfuncs.py")]
+            vfuns_matches = (
+                [file for file in filenames if file.endswith("vfuncs.py")]
+                if allow_python_validators
+                else []
+            )
         else:
             yaml_matches = [file for file in filenames if file == name]
-            vfuns_matches = [file for file in filenames if file == f"{name}_vfuncs.py"]
+            vfuns_matches = (
+                [file for file in filenames if file == f"{name}_vfuncs.py"]
+                if allow_python_validators
+                else []
+            )
             if len(yaml_matches) == 0:
                 raise ValueError(
                     f'No file with name "{name}" found in record {doi_or_recid}'
@@ -927,7 +967,11 @@ def from_zenodo(
             )
             shutil.move(_filename, match)
 
-    return from_yaml(yaml_matches[0], overwrite=overwrite)
+    return from_yaml(
+        yaml_matches[0],
+        overwrite=overwrite,
+        allow_python_validators=allow_python_validators,
+    )
 
 
 def yaml2jsonld(