h5rdmtoolbox 1.2.2__tar.gz → 1.3.0__tar.gz

{h5rdmtoolbox-1.2.2 → h5rdmtoolbox-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: h5rdmtoolbox
-Version: 1.2.2
+Version: 1.3.0
 Summary: Supporting a FAIR Research Data lifecycle using Python and HDF5.
 Home-page: https://h5rdmtoolbox.readthedocs.io/en/latest/
 Author: Matthias Probst
@@ -29,12 +29,15 @@ Requires-Dist: pint==0.21.1
 Requires-Dist: pint_xarray
 Requires-Dist: regex
 Requires-Dist: packaging
+Requires-Dist: ontolutils>=0.2.20
 Requires-Dist: python-forge
 Requires-Dist: requests
 Requires-Dist: pydantic>=2.3.0
 Requires-Dist: rdflib
 Provides-Extra: database
 Requires-Dist: pymongo; extra == "database"
+Provides-Extra: layout-validation
+Requires-Dist: tabulate; extra == "layout-validation"
 Provides-Extra: csv
 Requires-Dist: pandas; extra == "csv"
 Provides-Extra: snt
@@ -42,6 +45,8 @@ Requires-Dist: xmltodict; extra == "snt"
 Requires-Dist: tabulate; extra == "snt"
 Requires-Dist: python-gitlab; extra == "snt"
 Requires-Dist: pypandoc; extra == "snt"
+Provides-Extra: gui
+Requires-Dist: PyQt5; extra == "gui"
 Provides-Extra: test
 Requires-Dist: pytest>=7.1.2; extra == "test"
 Requires-Dist: pytest-cov; extra == "test"
@@ -82,7 +87,9 @@ Requires-Dist: sphinx-design; extra == "docs"
 Requires-Dist: simplejson; extra == "docs"
 Requires-Dist: myst-nb; extra == "docs"
 Requires-Dist: sphinxcontrib-bibtex; extra == "docs"
+Requires-Dist: kglab; extra == "docs"
 Provides-Extra: complete
+Requires-Dist: PyQt5; extra == "complete"
 Requires-Dist: pytest>=7.1.2; extra == "complete"
 Requires-Dist: pytest-cov; extra == "complete"
 Requires-Dist: pylint; extra == "complete"
@@ -121,6 +128,7 @@ Requires-Dist: sphinx-design; extra == "complete"
 Requires-Dist: simplejson; extra == "complete"
 Requires-Dist: myst-nb; extra == "complete"
 Requires-Dist: sphinxcontrib-bibtex; extra == "complete"
+Requires-Dist: kglab; extra == "complete"
 
 # HDF5 Research Data Management Toolbox
 
@@ -131,7 +139,7 @@ Requires-Dist: sphinxcontrib-bibtex; extra == "complete"
 
 *Note, that the project is still under development!*
 
-The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a python package supporting everybody who is working with
+The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a Python package supporting everybody who is working with
 HDF5 to achieve a sustainable data lifecycle which follows
 the [FAIR (Findable, Accessible, Interoperable, Reusable)](https://www.nature.com/articles/sdata201618)
 principles. It specifically supports the five main steps of *planning*, *collecting*, *analyzing*, *sharing* and
@@ -142,13 +150,14 @@ information of try the [quickstart using colab](#quickstart).
 
 - Combining HDF5 and [xarray](https://docs.xarray.dev/en/stable/) to allow easy access to metadata and data during
   analysis and processing (
-  see [here](https://h5rdmtoolbox.readthedocs.io/en/latest/gettingstarted/quickoverview.html#datasets-xarray-interface).
+  see [here](https://h5rdmtoolbox.readthedocs.io/en/latest/gettingstarted/quickoverview.html#datasets-xarray-interface)).
 - Assigning [metadata with "globally unique and persistent identifiers"]() as required
-  by [F1 of the FAIR principles](https://www.go-fair.org/fair-principles/f1-meta-data-assigned-globally-unique-persistent-identifiers/)
-  . This "remove[s] ambiguity in the meaning of your published data...".
+  by [F1 of the FAIR principles](https://www.go-fair.org/fair-principles/f1-meta-data-assigned-globally-unique-persistent-identifiers/).
+  This can be achieved by using [RDF triples](https://www.w3.org/RDF/), which removes "ambiguity in the meaning of your
+  published data".
 - Define standard attributes through
   [conventions](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html) and enforce users to use
-  them
+  certain attributes in their HDF5 files, such as units and a description, for example.
 - Upload HDF5 files directly
   to [repositories](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/repository/index.html)
   like [Zenodo](https://zenodo.org/)
@@ -159,20 +168,21 @@ information of try the [quickstart using colab](#quickstart).
 
 For everybody, who is...
 
-- ... looking for a management approach for his or her data
-- ... community has not yet established a stable convention
-- ... working with small and big data, that fits into HDF5 files
-- ... looking for an easy way to work with HDF5, especially through Jupyter Notebooks
-- ... looking to integrate HDF5 with repositories and databases
-- ... looking for a way to do all the above whiles not needing to learn a new syntax
-- ... new to HDF5 and wants to learn about it, especially with respect to the FAIR principles and data management
+- ... looking for a management approach for his or her data.
+- ... community has not yet established a stable convention.
+- ... working with small and big data, that fits into HDF5 files.
+- ... looking for an easy way to work with HDF5, especially through Jupyter Notebooks.
+- ... trying to integrate HDF5 with repositories and databases.
+- ... wishing to enrich data semantically with the RDF standard.
+- ... looking for a way to do all the above whiles not needing to learn a new syntax.
+- ... new to HDF5 and wants to learn about it, especially with respect to the FAIR principles and data management.
 
 ## Who is it not for?
 
 For everybody, who ...
 
 - ... is looking for a management approach which at the same time allows high-performance and/or parallel work with HDF5
-- ... has established conventions and managements approaches in his or her community
+- ... has already well-established conventions and managements approaches in his or her community
 
 ## Package Architecture/structure
 
@@ -185,8 +195,9 @@ of each other.
 Current implementation highlights in the modules:
 
 - The **wrapper** module adds functionality on top of the `h5py` package. It allows to include so-called standard names,
-  which are defined in conventions. And it implements an interface with the package `xarray`, which allows to carry
-  metadata from HDF5 to the user.
+  which are defined in conventions. And it implements interfaces, such as to the package `xarray`, which allows to carry
+  metadata from HDF5 to the user. Other high-level interfaces like `.rdf` allows assigning semantic information to the
+  HDF5 file.
 - For the **database** module, `hdfDB` and `mongoDB` are implemented. The `hdfDB` module allows to use HDF5 files as a
   database. The `mongoDB` module allows to use mongoDB as a database by mapping the metadata of HDF5 files to the
   database.
@@ -258,9 +269,13 @@ specific to the features of the package:
 - `pint_xarray>=0.2.1`: Working with units for usage with xarray
 - `python-forge==18.6.0`: Used to update function signatures when using
   the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
+- `pydantic`: Used to
+  validate [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
 - `pyyaml>6.0.0`: Reading and writing of yaml files, e.g. metadata definitions (conventions). Note, lower versions
   collide with python 3.11
 - `requests`: Used to download files from the internet or validate URLs, e.g. metadata definitions (conventions)
+- `rdflib`: Used to enable working with RDF
+- `ontolutils`: Required to work with RDF and derive semantic description of HDF5 file content
 
 #### Optional dependencies
 
@@ -285,7 +300,6 @@ Install optional dependencies by specifying them in square brackets after the pa
 - `python-gitlab`: Access to gitlab repositories
 - `pypandoc>=2.3`: Conversion of markdown files to html
 
-
 ## Citing the package
 
 If you intend to use the package in your work, you may cite the paper in the

{h5rdmtoolbox-1.2.2 → h5rdmtoolbox-1.3.0}/README.md

@@ -7,7 +7,7 @@
 
 *Note, that the project is still under development!*
 
-The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a python package supporting everybody who is working with
+The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a Python package supporting everybody who is working with
 HDF5 to achieve a sustainable data lifecycle which follows
 the [FAIR (Findable, Accessible, Interoperable, Reusable)](https://www.nature.com/articles/sdata201618)
 principles. It specifically supports the five main steps of *planning*, *collecting*, *analyzing*, *sharing* and
@@ -18,13 +18,14 @@ information of try the [quickstart using colab](#quickstart).
 
 - Combining HDF5 and [xarray](https://docs.xarray.dev/en/stable/) to allow easy access to metadata and data during
   analysis and processing (
-  see [here](https://h5rdmtoolbox.readthedocs.io/en/latest/gettingstarted/quickoverview.html#datasets-xarray-interface).
+  see [here](https://h5rdmtoolbox.readthedocs.io/en/latest/gettingstarted/quickoverview.html#datasets-xarray-interface)).
 - Assigning [metadata with "globally unique and persistent identifiers"]() as required
-  by [F1 of the FAIR principles](https://www.go-fair.org/fair-principles/f1-meta-data-assigned-globally-unique-persistent-identifiers/)
-  . This "remove[s] ambiguity in the meaning of your published data...".
+  by [F1 of the FAIR principles](https://www.go-fair.org/fair-principles/f1-meta-data-assigned-globally-unique-persistent-identifiers/).
+  This can be achieved by using [RDF triples](https://www.w3.org/RDF/), which removes "ambiguity in the meaning of your
+  published data".
 - Define standard attributes through
   [conventions](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html) and enforce users to use
-  them
+  certain attributes in their HDF5 files, such as units and a description, for example.
 - Upload HDF5 files directly
   to [repositories](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/repository/index.html)
   like [Zenodo](https://zenodo.org/)
@@ -35,20 +36,21 @@ information of try the [quickstart using colab](#quickstart).
 
 For everybody, who is...
 
-- ... looking for a management approach for his or her data
-- ... community has not yet established a stable convention
-- ... working with small and big data, that fits into HDF5 files
-- ... looking for an easy way to work with HDF5, especially through Jupyter Notebooks
-- ... looking to integrate HDF5 with repositories and databases
-- ... looking for a way to do all the above whiles not needing to learn a new syntax
-- ... new to HDF5 and wants to learn about it, especially with respect to the FAIR principles and data management
+- ... looking for a management approach for his or her data.
+- ... community has not yet established a stable convention.
+- ... working with small and big data, that fits into HDF5 files.
+- ... looking for an easy way to work with HDF5, especially through Jupyter Notebooks.
+- ... trying to integrate HDF5 with repositories and databases.
+- ... wishing to enrich data semantically with the RDF standard.
+- ... looking for a way to do all the above whiles not needing to learn a new syntax.
+- ... new to HDF5 and wants to learn about it, especially with respect to the FAIR principles and data management.
 
 ## Who is it not for?
 
 For everybody, who ...
 
 - ... is looking for a management approach which at the same time allows high-performance and/or parallel work with HDF5
-- ... has established conventions and managements approaches in his or her community
+- ... has already well-established conventions and managements approaches in his or her community
 
 ## Package Architecture/structure
 
@@ -61,8 +63,9 @@ of each other.
 Current implementation highlights in the modules:
 
 - The **wrapper** module adds functionality on top of the `h5py` package. It allows to include so-called standard names,
-  which are defined in conventions. And it implements an interface with the package `xarray`, which allows to carry
-  metadata from HDF5 to the user.
+  which are defined in conventions. And it implements interfaces, such as to the package `xarray`, which allows to carry
+  metadata from HDF5 to the user. Other high-level interfaces like `.rdf` allows assigning semantic information to the
+  HDF5 file.
 - For the **database** module, `hdfDB` and `mongoDB` are implemented. The `hdfDB` module allows to use HDF5 files as a
   database. The `mongoDB` module allows to use mongoDB as a database by mapping the metadata of HDF5 files to the
   database.
@@ -134,9 +137,13 @@ specific to the features of the package:
 - `pint_xarray>=0.2.1`: Working with units for usage with xarray
 - `python-forge==18.6.0`: Used to update function signatures when using
   the [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
+- `pydantic`: Used to
+  validate [standard attributes](https://h5rdmtoolbox.readthedocs.io/en/latest/conventions/standard_attributes_and_conventions.html)
 - `pyyaml>6.0.0`: Reading and writing of yaml files, e.g. metadata definitions (conventions). Note, lower versions
   collide with python 3.11
 - `requests`: Used to download files from the internet or validate URLs, e.g. metadata definitions (conventions)
+- `rdflib`: Used to enable working with RDF
+- `ontolutils`: Required to work with RDF and derive semantic description of HDF5 file content
 
 #### Optional dependencies
 
@@ -161,7 +168,6 @@ Install optional dependencies by specifying them in square brackets after the pa
 - `python-gitlab`: Access to gitlab repositories
 - `pypandoc>=2.3`: Conversion of markdown files to html
 
-
 ## Citing the package
 
 If you intend to use the package in your work, you may cite the paper in the

{h5rdmtoolbox-1.2.2 → h5rdmtoolbox-1.3.0}/h5rdmtoolbox/__init__.py

@@ -1,6 +1,31 @@
 """h5rdtoolbox repository"""
-import atexit
+
+import appdirs
+import logging
 import pathlib
+from logging.handlers import RotatingFileHandler
+
+_logdir = pathlib.Path(appdirs.user_log_dir('h5rdmtoolbox'))
+_logdir.mkdir(parents=True, exist_ok=True)
+
+DEFAULT_LOGGING_LEVEL = logging.WARNING
+_formatter = logging.Formatter(
+    '%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
+    datefmt='%Y-%m-%d_%H:%M:%S')
+
+_stream_handler = logging.StreamHandler()
+_stream_handler.setLevel(DEFAULT_LOGGING_LEVEL)
+_stream_handler.setFormatter(_formatter)
+
+_file_handler = RotatingFileHandler(_logdir / 'h5rdmtoolbox.log')
+_file_handler.setLevel(logging.DEBUG)  # log everything to file!
+_file_handler.setFormatter(_formatter)
+
+logger = logging.getLogger(__package__)
+logger.addHandler(_stream_handler)
+logger.addHandler(_file_handler)
+
+import atexit
 # noinspection PyUnresolvedReferences
 import pint_xarray
 import shutil
@@ -20,16 +45,24 @@ from . import utils
 from .wrapper.core import lower, Lower, File, Group, Dataset
 from . import warnings, errors
 from .wrapper import jsonld
-from .wrapper.accessory import register_special_dataset
+from .wrapper.lazy import lazy
+from .wrapper.h5attr import Attribute
+# from h5rdmtoolbox.wrapper.accessor import register_accessor
+import json
+from .wrapper.accessor import register_accessor
 
 name = 'h5rdmtoolbox'
 __this_dir__ = pathlib.Path(__file__).parent
 __author__ = 'Matthias Probst'
 __author_orcid__ = 'https://orcid.org/0000-0001-8729-0482'
 
-logger = utils.create_tbx_logger('h5rdmtoolbox')
 
-logger.setLevel(get_config()['init_logger_level'])
+def get_package_meta():
+    """Reads codemeta.json and returns it as dict"""
+    with open(__this_dir__ / '../codemeta.json', 'r') as f:
+        codemeta = json.loads(f.read())
+    return codemeta
+
 
 cv_h5py = convention.Convention(name='h5py',
                                 contact=__author_orcid__)
@@ -40,7 +73,7 @@ use(None)
 
 
 def dump(src: Union[str, File, pathlib.Path],
-         **kwargs) -> None:
+         *args, **kwargs) -> None:
     """Call h5.dump() on the provided HDF5 file
 
     Parameters
@@ -52,7 +85,7 @@ def dump(src: Union[str, File, pathlib.Path],
     """
     if isinstance(src, File):
         with File(src.hdf_filename) as h5:
-            return h5.dump(**kwargs)
+            return h5.dump(*args, **kwargs)
 
     if isinstance(src, (str, pathlib.Path)):
         pass
@@ -86,10 +119,58 @@ def dumps(src: Union[str, File, pathlib.Path]):
         return h5.dumps()
 
 
+def dump_jsonld(hdf_filename: Union[str, pathlib.Path],
+                skipND: int = 1,
+                structural: bool = True,
+                semantic: bool = True,
+                resolve_keys: bool = False,
+                **kwargs) -> str:
+    """Dump the JSON-LD representation of the file. With semantic=True and structural=False, the JSON-LD
+    represents the semantic content only. To get a pure structural representation, set semantic=False, which
+    will ignore any RDF content. If both are set to True, the JSON-LD will contain both structural and semantic.
+
+    Parameters
+    ----------
+    hdf_filename : str, pathlib.Path
+        the HDF5 file to dump.
+    skipND : int=1
+        Skip writing data of datasets with more then skipND dimensions. Only
+        considered if structural=True.
+    structural : bool=True
+        Include structural information in the JSON-LD output.
+    semantic : bool=True
+        Include semantic information in the JSON-LD output.
+    resolve_keys : bool=False
+        Resolve keys in the JSON-LD output. This is used when semantic=True.
+        If resolve_keys is False and an attribute name in the HDF5 file, which has
+        a predicate is different in its name from the predicate, the attribute name is used.
+        Example: an attribute "name" is associated with "foaf:lastName", then "name" is used
+        and "name": "https://xmlns.com/foaf/0.1/lastName" is added to the context.
+
+    """
+    from .wrapper import jsonld
+    if not structural and not semantic:
+        raise ValueError('At least one of structural or semantic must be True.')
+    if structural and not semantic:
+        return jsonld.dump_file(hdf_filename, skipND=skipND)
+    with File(hdf_filename) as h5:
+        return jsonld.dumps(h5, structural=structural, resolve_keys=resolve_keys, **kwargs)
+
+
+def get_filesize(hdf_filename: Union[str, pathlib.Path]) -> int:
+    """Get the size of the HDF5 file in bytes"""
+    return utils.get_filesize(hdf_filename)
+
+
+def get_checksum(hdf_filename: Union[str, pathlib.Path]) -> str:
+    """Get the checksum of the HDF5 file"""
+    return utils.get_checksum(hdf_filename)
+
+
 def register_dataset_decoder(decoder: Callable, decoder_name: str = None, overwrite: bool = False):
     """A decoder function takes a xarray.DataArray and a dataset as input and returns a xarray.DataArray
     It is called after the dataset is loaded into memory and before being returned to the user. Be careful:
-    Multiple decoders can be registered and they are called in the order of registration. Hence, your decoder
+    Multiple decoders can be registered, and they are called in the order of registration. Hence, your decoder
     may behave unexpectedly!
     """
     from .wrapper import ds_decoder
@@ -105,6 +186,15 @@ def register_dataset_decoder(decoder: Callable, decoder_name: str = None, overwr
 atexit_verbose = False
 
 
+def set_loglevel(level: Union[int, str]):
+    """Set the logging level of the h5rdmtoolbox logger"""
+    import logging
+    _logger = logging.getLogger('h5rdmtoolbox')
+    _logger.setLevel(level)
+    for h in _logger.handlers:
+        h.setLevel(level)
+
+
 @atexit.register
 def clean_temp_data(full: bool = False):
     """cleaning up the tmp directory"""
@@ -163,8 +253,7 @@ xr.set_options(display_expand_data=False)
 
 __all__ = ('__version__', '__author__', '__author_orcid__',
            'UserDir', 'use',
-           'File', 'Group', 'Dataset',
+           'File', 'Group', 'Dataset', 'Attribute',
            'dump', 'dumps', 'cv_h5py', 'lower', 'Lower',
            'set_config', 'get_config', 'get_ureg',
-           'Convention', 'namespace',
-           'jsonld')
+           'Convention', 'jsonld', 'lazy')

{h5rdmtoolbox-1.2.2 → h5rdmtoolbox-1.3.0}/h5rdmtoolbox/_cfg.py

@@ -17,26 +17,28 @@ def is_valid_logger_level(level: Union[str, int]):
 
 
 CONFIG = {
-          'return_xarray': True,
-          'advanced_shape_repr': True,
-          'natural_naming': True,
-          'hdf_compression': None,  # 'gzip',
-          'hdf_compression_opts': None,  # 5,
-          'adjusting_plotting_labels': True,
-          'xarray_unit_repr_in_plots': 'in',
-          'plotting_name_order': ('plot_name', 'long_name', 'standard_name'),
-          'require_unit': True,  # datasets require units
-          'ureg_format': 'C~',
-          'init_logger_level': 'ERROR',
-          'dtime_fmt': '%Y%m%d%H%M%S%f',
-          'expose_user_prop_to_attrs': True,
-          'add_provenance': False,
-          'ignore_set_std_attr_err': False,
-
-          # if a standard attribute is defined and cannot be retrieved because the value is invalid, ignore it:
-          'ignore_get_std_attr_err': False,
-          'allow_deleting_standard_attributes': False,
-          }
+    'return_xarray': True,
+    'advanced_shape_repr': True,
+    'natural_naming': True,
+    'hdf_compression': None,  # 'gzip',
+    'hdf_compression_opts': None,  # 5,
+    'adjusting_plotting_labels': True,
+    'xarray_unit_repr_in_plots': 'in',
+    'plotting_name_order': ('plot_name', 'long_name', 'standard_name'),
+    'require_unit': True,  # datasets require units
+    'ureg_format': 'C~',
+    'init_logger_level': 'ERROR',
+    'dtime_fmt': '%Y%m%d%H%M%S%f',
+    'expose_user_prop_to_attrs': True,
+    'add_provenance': False,
+    'ignore_set_std_attr_err': False,
+    'auto_create_h5tbx_version': False,  # automatically creates the group h5rdmtoolbox with the version attribute
+    'uuid_name': 'uuid',  # attribute name used for UUIDs
+    # if a standard attribute is defined and cannot be retrieved because the value is invalid, ignore it:
+    'ignore_get_std_attr_err': False,
+    'allow_deleting_standard_attributes': False,
+    'ignore_none': False
+}
 
 _VALIDATORS = {
     'return_xarray': lambda x: isinstance(x, bool),
@@ -54,8 +56,8 @@ _VALIDATORS = {
     'expose_user_prop_to_attrs': lambda x: isinstance(x, bool),
     'add_provenance': lambda x: isinstance(x, bool),
     'ignore_set_std_attr_err': lambda x: isinstance(x, bool),
-
     'ignore_get_std_attr_err': lambda x: isinstance(x, bool),
+    'ignore_none': lambda x: isinstance(x, bool)
 }