h5rdmtoolbox 2.6.0__tar.gz → 2.6.2__tar.gz

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2012-2023 Scott Chacon and others
+Copyright (c) 2022-2026 h5rdmtoolbox contributors
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: h5rdmtoolbox
-Version: 2.6.0
+Version: 2.6.2
 Summary: Supporting a FAIR Research Data lifecycle using Python and HDF5.
 Author-email: Matthias Probst <matth.probst@gmail.com>
 License-Expression: MIT
@@ -85,34 +85,109 @@ Dynamic: license-file
 ![Tests](https://github.com/matthiasprobst/h5RDMtoolbox/actions/workflows/tests.yml/badge.svg)
 [![codecov](https://codecov.io/gh/matthiasprobst/h5RDMtoolbox/graph/badge.svg?token=IVG4AQEW47)](https://codecov.io/gh/matthiasprobst/h5RDMtoolbox)
 [![Documentation Status](https://readthedocs.org/projects/h5rdmtoolbox/badge/?version=latest)](https://h5rdmtoolbox.readthedocs.io/en/latest/?badge=latest)
+[![PyPI version](https://badge.fury.io/py/h5RDMtoolbox.svg)](https://badge.fury.io/py/h5RDMtoolbox)
 ![pyvers](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)
 
-*Note, that the project is still under development!*
-
-The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a Python package supporting everybody who is working with
-HDF5 to achieve a sustainable data lifecycle which follows
-the [FAIR (Findable, Accessible, Interoperable, Reusable)](https://www.nature.com/articles/sdata201618)
-principles. It specifically supports the five main steps of *planning*, *collecting*, *analyzing*, *sharing* and
-*reusing* data. Please visit the [documentation](https://h5rdmtoolbox.readthedocs.io/en/latest/) for detailed
-information of try the [quickstart using colab](#quickstart).
-
-## Highlights
-
-- Combining HDF5 and [xarray](https://docs.xarray.dev/en/stable/) to allow easy access to metadata and data during
-  analysis and processing (
-  see [here](https://h5rdmtoolbox.readthedocs.io/en/latest/gettingstarted/quickoverview.html#datasets-xarray-interface)).
-- Assigning [metadata with "globally unique and persistent identifiers"]() as required
-  by [F1 of the FAIR principles](https://www.go-fair.org/fair-principles/f1-meta-data-assigned-globally-unique-persistent-identifiers/).
-  This can be achieved by using [RDF triples](https://www.w3.org/RDF/), which removes "ambiguity in the meaning of your
-  published data".
-- Define standard attributes through
-  [conventions](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html) and enforce users to use
-  certain attributes in their HDF5 files, such as units and a description, for example.
-- Upload HDF5 files directly
-  to [repositories](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/repository/index.html)
-  like [Zenodo](https://zenodo.org/)
-  or [use them with noSQL databases](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/database/index.html) like
-  [mongoDB](https://www.mongodb.com/).
+A Python package that helps researchers achieve sustainable 
+[**FAIR (Findable, Accessible, Interoperable, Reusable)**](https://www.nature.com/articles/sdata201618) data management with HDF5 files. The toolbox provides a comprehensive
+approach to the research data lifecycle through five main stages: **planning**, **collecting**, **analyzing**,
+**sharing**, and **reusing** data.
+
+> **Note**: This project is actively under development.
+
+
+
+
+## ✨ Quick Start
+
+
+One of the core features of the toolbox and success factors for achieving FAIR data, is using semantic metadata.
+
+Let's see how we can use RDF (semantic metadata) together with HDF5 files using the toolbox:
+
+```bash
+# Install the package
+pip install h5RDMtoolbox
+```
+
+```python
+import h5rdmtoolbox as h5tbx
+import rdflib
+import numpy as np
+
+# Create a new HDF5 file with semantic metadata
+M4I = rdflib.Namespace("http://w3id.org/nfdi4ing/metadata4ing#")
+
+# Create a new HDF5 file with FAIR metadata
+with h5tbx.File("example.h5", "w") as h5:
+    h5.create_dataset("temperature", data=np.array([20, 21, 19, 22]))
+    h5.attrs["units", M4I.hasUnit] = "degree_Celsius"
+    h5.rdf["units"].object = "http://qudt.org/vocab/unit/DEG_C"
+    h5.attrs["description", "https://schema.org/description"] = "Room temperature measurements"
+
+    ttl = h5.serialize("ttl")
+
+```
+The serialization in Turtle (ttl) is teh RDF serialization of the HDF5 data (without arrays):
+```ttl
+@prefix hdf: <http://purl.allotrope.org/ontologies/hdf5/1.8#> .
+@prefix m4i: <http://w3id.org/nfdi4ing/metadata4ing#> .
+@prefix schema: <https://schema.org/> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+
+hdf:H5T_INTEL_I64 a hdf:Datatype .
+
+[] a hdf:File ;
+    hdf:rootGroup [ a hdf:Group ;
+            hdf:attribute [ a hdf:StringAttribute ;
+                    hdf:data "Room temperature measurements" ;
+                    hdf:name "description" ],
+                [ a hdf:StringAttribute ;
+                    hdf:data "degree_Celsius" ;
+                    hdf:name "units" ] ;
+            hdf:member [ a hdf:Dataset ;
+                    hdf:dataspace [ a hdf:SimpleDataspace ;
+                            hdf:dimension [ a hdf:DataspaceDimension ;
+                                    hdf:dimensionIndex 0 ;
+                                    hdf:size 4 ] ] ;
+                    hdf:datatype hdf:H5T_INTEGER,
+                        hdf:H5T_INTEL_I64 ;
+                    hdf:layout hdf:H5D_CONTIGUOUS ;
+                    hdf:maximumSize 4 ;
+                    hdf:name "/temperature" ;
+                    hdf:rank 1 ;
+                    hdf:size 4 ] ;
+            hdf:name "/" ;
+            m4i:hasUnit <http://qudt.org/vocab/unit/DEG_C> ;
+            schema:description "Room temperature measurements" ] .
+```
+
+[//]: # (The "HDF5 Research Data Management Toolbox" &#40;h5RDMtoolbox&#41; is a Python package supporting everybody who is working with)
+
+[//]: # (HDF5 to achieve a sustainable data lifecycle which follows)
+
+[//]: # (the [FAIR &#40;Findable, Accessible, Interoperable, Reusable&#41;]&#40;https://www.nature.com/articles/sdata201618&#41;)
+
+[//]: # (principles. It specifically supports the five main steps of *planning*, *collecting*, *analyzing*, *sharing* and)
+
+[//]: # (*reusing* data. Please visit the [documentation]&#40;https://h5rdmtoolbox.readthedocs.io/en/latest/&#41; for detailed)
+
+[//]: # (information of try the [quickstart using colab]&#40;#quickstart&#41;.)
+
+
+## 🚀 Key Features
+
+- **🔗 HDF5 + [Xarray](https://docs.xarray.dev/en/stable/) Integration**: Seamless access to metadata during data analysis with native xarray support
+- **🏷️ Persistent Identifiers**: Assign globally unique identifiers using RDF triples for FAIR compliance  
+- **📋 Standardized Conventions**: Define and enforce community-specific metadata standards
+- **☁️ Repository Integration**: Direct upload to Zenodo and other research repositories
+- **🗄️ Database Support**: Use HDF5 files with MongoDB or native HDF5 databases
+- **🔒 Semantic Enrichment**: Add RDF-based semantic meaning to your data
+- **🌐 Catalog Integration**: Search and discover datasets through SPARQL-based catalogs
+
+
+Find an example code in  [![In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/matthiasprobst/h5RDMtoolbox/blob/main/docs/colab/quickstart.ipynb) or check out the 
+[📚 full Documentation](https://h5rdmtoolbox.readthedocs.io/en/latest/)
 
 ## Who is the package for?
 

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/README.md

@@ -3,34 +3,109 @@
 ![Tests](https://github.com/matthiasprobst/h5RDMtoolbox/actions/workflows/tests.yml/badge.svg)
 [![codecov](https://codecov.io/gh/matthiasprobst/h5RDMtoolbox/graph/badge.svg?token=IVG4AQEW47)](https://codecov.io/gh/matthiasprobst/h5RDMtoolbox)
 [![Documentation Status](https://readthedocs.org/projects/h5rdmtoolbox/badge/?version=latest)](https://h5rdmtoolbox.readthedocs.io/en/latest/?badge=latest)
+[![PyPI version](https://badge.fury.io/py/h5RDMtoolbox.svg)](https://badge.fury.io/py/h5RDMtoolbox)
 ![pyvers](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)
 
-*Note, that the project is still under development!*
-
-The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a Python package supporting everybody who is working with
-HDF5 to achieve a sustainable data lifecycle which follows
-the [FAIR (Findable, Accessible, Interoperable, Reusable)](https://www.nature.com/articles/sdata201618)
-principles. It specifically supports the five main steps of *planning*, *collecting*, *analyzing*, *sharing* and
-*reusing* data. Please visit the [documentation](https://h5rdmtoolbox.readthedocs.io/en/latest/) for detailed
-information of try the [quickstart using colab](#quickstart).
-
-## Highlights
-
-- Combining HDF5 and [xarray](https://docs.xarray.dev/en/stable/) to allow easy access to metadata and data during
-  analysis and processing (
-  see [here](https://h5rdmtoolbox.readthedocs.io/en/latest/gettingstarted/quickoverview.html#datasets-xarray-interface)).
-- Assigning [metadata with "globally unique and persistent identifiers"]() as required
-  by [F1 of the FAIR principles](https://www.go-fair.org/fair-principles/f1-meta-data-assigned-globally-unique-persistent-identifiers/).
-  This can be achieved by using [RDF triples](https://www.w3.org/RDF/), which removes "ambiguity in the meaning of your
-  published data".
-- Define standard attributes through
-  [conventions](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/convention/index.html) and enforce users to use
-  certain attributes in their HDF5 files, such as units and a description, for example.
-- Upload HDF5 files directly
-  to [repositories](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/repository/index.html)
-  like [Zenodo](https://zenodo.org/)
-  or [use them with noSQL databases](https://h5rdmtoolbox.readthedocs.io/en/latest/userguide/database/index.html) like
-  [mongoDB](https://www.mongodb.com/).
+A Python package that helps researchers achieve sustainable 
+[**FAIR (Findable, Accessible, Interoperable, Reusable)**](https://www.nature.com/articles/sdata201618) data management with HDF5 files. The toolbox provides a comprehensive
+approach to the research data lifecycle through five main stages: **planning**, **collecting**, **analyzing**,
+**sharing**, and **reusing** data.
+
+> **Note**: This project is actively under development.
+
+
+
+
+## ✨ Quick Start
+
+
+One of the core features of the toolbox and success factors for achieving FAIR data, is using semantic metadata.
+
+Let's see how we can use RDF (semantic metadata) together with HDF5 files using the toolbox:
+
+```bash
+# Install the package
+pip install h5RDMtoolbox
+```
+
+```python
+import h5rdmtoolbox as h5tbx
+import rdflib
+import numpy as np
+
+# Create a new HDF5 file with semantic metadata
+M4I = rdflib.Namespace("http://w3id.org/nfdi4ing/metadata4ing#")
+
+# Create a new HDF5 file with FAIR metadata
+with h5tbx.File("example.h5", "w") as h5:
+    h5.create_dataset("temperature", data=np.array([20, 21, 19, 22]))
+    h5.attrs["units", M4I.hasUnit] = "degree_Celsius"
+    h5.rdf["units"].object = "http://qudt.org/vocab/unit/DEG_C"
+    h5.attrs["description", "https://schema.org/description"] = "Room temperature measurements"
+
+    ttl = h5.serialize("ttl")
+
+```
+The serialization in Turtle (ttl) is teh RDF serialization of the HDF5 data (without arrays):
+```ttl
+@prefix hdf: <http://purl.allotrope.org/ontologies/hdf5/1.8#> .
+@prefix m4i: <http://w3id.org/nfdi4ing/metadata4ing#> .
+@prefix schema: <https://schema.org/> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+
+hdf:H5T_INTEL_I64 a hdf:Datatype .
+
+[] a hdf:File ;
+    hdf:rootGroup [ a hdf:Group ;
+            hdf:attribute [ a hdf:StringAttribute ;
+                    hdf:data "Room temperature measurements" ;
+                    hdf:name "description" ],
+                [ a hdf:StringAttribute ;
+                    hdf:data "degree_Celsius" ;
+                    hdf:name "units" ] ;
+            hdf:member [ a hdf:Dataset ;
+                    hdf:dataspace [ a hdf:SimpleDataspace ;
+                            hdf:dimension [ a hdf:DataspaceDimension ;
+                                    hdf:dimensionIndex 0 ;
+                                    hdf:size 4 ] ] ;
+                    hdf:datatype hdf:H5T_INTEGER,
+                        hdf:H5T_INTEL_I64 ;
+                    hdf:layout hdf:H5D_CONTIGUOUS ;
+                    hdf:maximumSize 4 ;
+                    hdf:name "/temperature" ;
+                    hdf:rank 1 ;
+                    hdf:size 4 ] ;
+            hdf:name "/" ;
+            m4i:hasUnit <http://qudt.org/vocab/unit/DEG_C> ;
+            schema:description "Room temperature measurements" ] .
+```
+
+[//]: # (The "HDF5 Research Data Management Toolbox" &#40;h5RDMtoolbox&#41; is a Python package supporting everybody who is working with)
+
+[//]: # (HDF5 to achieve a sustainable data lifecycle which follows)
+
+[//]: # (the [FAIR &#40;Findable, Accessible, Interoperable, Reusable&#41;]&#40;https://www.nature.com/articles/sdata201618&#41;)
+
+[//]: # (principles. It specifically supports the five main steps of *planning*, *collecting*, *analyzing*, *sharing* and)
+
+[//]: # (*reusing* data. Please visit the [documentation]&#40;https://h5rdmtoolbox.readthedocs.io/en/latest/&#41; for detailed)
+
+[//]: # (information of try the [quickstart using colab]&#40;#quickstart&#41;.)
+
+
+## 🚀 Key Features
+
+- **🔗 HDF5 + [Xarray](https://docs.xarray.dev/en/stable/) Integration**: Seamless access to metadata during data analysis with native xarray support
+- **🏷️ Persistent Identifiers**: Assign globally unique identifiers using RDF triples for FAIR compliance  
+- **📋 Standardized Conventions**: Define and enforce community-specific metadata standards
+- **☁️ Repository Integration**: Direct upload to Zenodo and other research repositories
+- **🗄️ Database Support**: Use HDF5 files with MongoDB or native HDF5 databases
+- **🔒 Semantic Enrichment**: Add RDF-based semantic meaning to your data
+- **🌐 Catalog Integration**: Search and discover datasets through SPARQL-based catalogs
+
+
+Find an example code in  [![In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/matthiasprobst/h5RDMtoolbox/blob/main/docs/colab/quickstart.ipynb) or check out the 
+[📚 full Documentation](https://h5rdmtoolbox.readthedocs.io/en/latest/)
 
 ## Who is the package for?
 

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/h5rdmtoolbox/catalog/__init__.py

@@ -100,7 +100,7 @@ def _download_catalog_datasets(
 
     results = g.query(GET_ALL_METADATA_CATALOG_DATASETS.query)
 
-    print("found datasets:", len(results))
+    logger.debug(f"found datasets: {len(results)}")
     list_of_ttl_web_resources = [
         WebResource(
             download_url=row.download,
@@ -113,11 +113,13 @@ def _download_catalog_datasets(
     ]
 
     n_ttl_resources = len(list_of_ttl_web_resources)
-    print(
+    logger.debug(
         f"Found {n_ttl_resources} TTL web resources to download. Downloading to {download_directory}..."
     )
     return download(
-        download_directory=download_directory, web_resources=list_of_ttl_web_resources
+        download_directory=download_directory,
+        web_resources=list_of_ttl_web_resources,
+        use_agent_header=True
     )
 
 
@@ -421,7 +423,6 @@ class CatalogManager:
             logger.debug("Adding Wikidata knowledge to the main RDF store...")
 
             if res.data.empty:
-                print("empty")
                 logger.info(
                     "No Wikidata entities found in the main RDF store. Skipping Wikidata knowledge addition."
                 )

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/h5rdmtoolbox/catalog/_initialization.py

@@ -84,5 +84,6 @@ def database_initialization(
         f"Found {len(list_of_ttl_web_resources)} TTL web resources to download. Downloading to {download_directory}...")
     return download(
         download_directory=download_directory,
-        web_resources=list_of_ttl_web_resources
+        web_resources=list_of_ttl_web_resources,
+        use_agent_header=True
     )

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/h5rdmtoolbox/catalog/core.py

@@ -383,6 +383,21 @@ class GraphDB(RemoteSparqlStore):
         )
         resp.raise_for_status()
 
+    def _post_update(self, update):
+        url = f"{self.endpoint}/repositories/{self.repository}/statements"
+        headers = {"Content-Type": "application/sparql-update"}
+        auth = (self.username, self.password) if self.username and self.password else None
+        r = requests.post(
+            url,
+            data=update.encode("utf-8"),
+            headers=headers,
+            auth=auth,
+        )
+        if not r.ok:
+            raise RuntimeError(
+                f"GraphDB SPARQL update failed "
+                f"(status {r.status_code}):\n{r.text}"
+            )
 
 class InMemoryRDFStore(RDFStore):
     """In-memory RDF database that can upload files and return a combined graph."""
@@ -565,6 +580,27 @@ class InMemoryRDFStore(RDFStore):
         self._combined_graph = rdflib.Graph()
         return self
 
+    def save(self, filename: Union[str, pathlib.Path]):
+        """Saves the combined graph to a file.
+
+        Parameters
+        ----------
+        filename : Union[str, pathlib.Path]
+            The path to the file where the graph will be saved.
+        """
+        filename = pathlib.Path(filename).resolve().absolute()
+        filename_suffix = filename.suffix.lower()
+        if filename_suffix == ".ttl":
+            _format = "turtle"
+        elif filename_suffix == ".rdf" or filename_suffix == ".xml":
+            _format = "xml"
+        elif filename_suffix == ".jsonld":
+            _format = "json-ld"
+        elif filename_suffix == ".nt":
+            _format = "nt"
+        else:
+            raise ValueError(f"File format '{filename_suffix}' not supported for saving.")
+        self.graph.serialize(destination=str(filename), format=_format)
 
 class AbstractQuery(ABC):
 

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/h5rdmtoolbox/catalog/utils.py

@@ -17,6 +17,7 @@ from pydantic.v1 import HttpUrl
 from rdflib import Graph
 
 from .. import logger
+from .._version import __version__
 
 # --- minimal caster for common XSD datatypes ---
 _XSD = "http://www.w3.org/2001/XMLSchema#"
@@ -28,6 +29,10 @@ _NUM_DT = {
 _FLOAT_DT = {_XSD + "float", _XSD + "double"}
 _DEC_DT = {_XSD + "decimal"}
 
+USER_AGENT_HEADER = {
+    "User-Agent": f"h5rdmtoolbox/{__version__} (https://github.com/matthiasprobst/h5rdmtoolbox)",
+}
+
 
 def file_uri_or_path_to_path(value) -> pathlib.Path:
     # Keep behavior close to your code
@@ -190,40 +195,8 @@ def extract_record_id(value: str) -> str:
     raise ValueError(f"Could not extract a numeric record ID from '{value}'")
 
 
-def get_file_metadata(record: str, sandbox: bool = False) -> Iterator[Tuple[str, Dict]]:
-    """
-    Fetch the Zenodo record JSON and yield (filename, file-metadata-dict) tuples.
-    """
-    record_id = extract_record_id(record)
-    base_url = "https://sandbox.zenodo.org/api/records" if sandbox else "https://zenodo.org/api/records"
-    url = f"{base_url}/{record_id}"
-
-    resp = requests.get(url)
-    if resp.status_code != 200:
-        raise RuntimeError(
-            f"Failed to fetch record {record_id}: HTTP {resp.status_code} - {resp.text}"
-        )
 
-    data = resp.json()
 
-    # Zenodo records normally have 'files' at the top level
-    files = data.get("files", [])
-
-    if not files:
-        # Some records may store file info differently; handle gracefully
-        raise RuntimeError(f"No files found in record {record_id} (or unexpected schema).")
-
-    for f in files:
-        # Common fields in Zenodo API
-        name = f.get("key") or f.get("filename") or "<unknown>"
-        yield name, f
-
-
-def _is_sandbox_doi(doi: str) -> bool:
-    """Return True if the provided doi string looks like a sandbox URL/DOI."""
-    if "10.5281/zenodo" in doi:
-        return True
-    return False
 
 
 def download_and_verify_file(file_meta: Dict, dest_dir: pathlib.Path) -> Tuple[pathlib.Path, bool, str, str]:
@@ -312,7 +285,8 @@ def parse_checksum(raw: str) -> Tuple[str, str]:
     return algo.lower(), val.lower()
 
 
-def download(download_directory: pathlib.Path, web_resources: List[WebResource]) -> List[DownloadStatus]:
+def download(download_directory: pathlib.Path, web_resources: List[WebResource], use_agent_header=True) -> List[
+    DownloadStatus]:
     """Download and verify a list of WebResource items.
 
     For each resource:
@@ -403,7 +377,11 @@ def download(download_directory: pathlib.Path, web_resources: List[WebResource])
         # stream download to tmp file while computing md5
         tmp_path = record_dir / (filename + ".tmp")
         try:
-            resp = requests.get(url, stream=True, timeout=60)
+            if use_agent_header:
+                kwargs_request = {"headers": USER_AGENT_HEADER}
+            else:
+                kwargs_request = {}
+            resp = requests.get(url, stream=True, timeout=60, **kwargs_request)
             resp.raise_for_status()
 
             md5 = hashlib.md5()
@@ -426,27 +404,27 @@ def download(download_directory: pathlib.Path, web_resources: List[WebResource])
                         # move tmp to final
                         tmp_path.replace(dest_path)
                         message = f"saved to {dest_path}"
-                        print(f" > OK: {filename} -> {dest_path} (md5 matches)")
+                        logger.debug(f" > OK: {filename} -> {dest_path} (md5 matches)")
                     else:
                         # conflict: keep both, rename new file with record id + short hash
                         new_name = f"{pathlib.Path(filename).stem}__{rec_id}__{computed[:8]}{pathlib.Path(filename).suffix}"
                         new_path = record_dir / new_name
                         tmp_path.replace(new_path)
                         message = f"checksum mismatch: expected={expected_val}, computed={computed}; saved as {new_path}"
-                        print(
+                        logger.error(
                             f" > MISMATCH: {filename} -> {new_path} (expected md5: {expected_val}, computed md5: {computed})")
                 else:
                     # no expected checksum provided — accept and move
                     tmp_path.replace(dest_path)
                     ok = True
                     message = f"saved to {dest_path} (no expected checksum)"
-                    print(f" > SAVED (no expected checksum): {filename} -> {dest_path}")
+                    logger.debug(f" > SAVED (no expected checksum): {filename} -> {dest_path}")
             else:
                 # unsupported algorithm: save file but mark as not-verified
                 tmp_path.replace(dest_path)
                 ok = False
                 message = f"saved to {dest_path} (unsupported checksum algorithm: {expected_algo})"
-                print(f" > SAVED (unsupported checksum alg '{expected_algo}'): {filename} -> {dest_path}")
+                logger.debug(f" > SAVED (unsupported checksum alg '{expected_algo}'): {filename} -> {dest_path}")
 
             overall.append(DownloadStatus(rec_id, dest_path, ok, message))
 
@@ -457,13 +435,13 @@ def download(download_directory: pathlib.Path, web_resources: List[WebResource])
                     tmp_path.unlink()
             except Exception:
                 pass
-            print(f" > ERROR downloading {filename}: {exc}")
+            logger.error(f" > ERROR downloading {filename}: {exc}")
             overall.append(DownloadStatus(rec_id, dest_path, False, str(exc)))
 
     # summary
     total = len(overall)
     ok_count = sum(1 for o in overall if o.ok)
-    print(f"Summary: {ok_count}/{total} files verified successfully")
+    logger.debug(f"Summary: {ok_count}/{total} files verified successfully")
 
     return overall
 
@@ -542,7 +520,7 @@ def database_initialization(
         mediaType="text/turtle"
     ) for row in results]
 
-    print(
+    logger.debug(
         f"Found {len(list_of_ttl_web_resources)} TTL web resources to download. Downloading to {download_directory}...")
     return download(
         download_directory=download_directory,

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/h5rdmtoolbox/convention/references.py

@@ -1,5 +1,11 @@
 import requests
 
+from h5rdmtoolbox import __version__
+
+USER_AGENT_HEADER = {
+    "User-Agent": f"h5rdmtoolbox/{__version__} (https://github.com/matthiasprobst/h5rdmtoolbox)",
+}
+
 
 def validate_url(url: str) -> bool:
     """Validate URL
@@ -15,7 +21,7 @@ def validate_url(url: str) -> bool:
         True if URL
     """
     try:
-        response = requests.get(url)
+        response = requests.get(url, headers=USER_AGENT_HEADER)
     except requests.exceptions.MissingSchema:
         return False
     if response.status_code == 200:

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/h5rdmtoolbox/protocols.py

@@ -149,6 +149,8 @@ class H5TbxAttributeManager(Protocol):
     def parent(self):
         ...
 
+    def items(self):
+        pass
 
 class H5TbxHLObject(Protocol):
     name: str

{h5rdmtoolbox-2.6.0 → h5rdmtoolbox-2.6.2}/h5rdmtoolbox/repository/utils.py

@@ -3,7 +3,7 @@ import logging
 import pathlib
 import uuid
 import warnings
-from typing import Optional, Union
+from typing import Optional, Union, Dict
 
 import requests
 
@@ -15,7 +15,8 @@ def download_file(file_url,
                   target_folder: Union[str, pathlib.Path] = None,
                   access_token: Optional[str] = None,
                   checksum: Optional[str] = None,
-                  checksum_algorithm: Optional[str] = None) -> pathlib.Path:
+                  checksum_algorithm: Optional[str] = None,
+                  headers: Optional[Dict]=None) -> pathlib.Path:
     logger.debug(f'Attempting to provide file from URL (download or return from cache): {file_url}')
     from ..utils import DownloadFileManager
     dfm = DownloadFileManager()
@@ -54,11 +55,11 @@ def download_file(file_url,
                 raise ValueError(f"Unsupported checksum algorithm: {checksum_algorithm}")
 
         try:
-            response = requests.get(file_url, stream=True)
+            response = requests.get(file_url, stream=True, headers=headers)
             if response.status_code == 403:
-                response = requests.get(file_url, stream=True, params={'access_token': access_token})
+                response = requests.get(file_url, stream=True, params={'access_token': access_token}, headers=headers)
         except requests.RequestException as e:
-            response = requests.get(file_url, stream=True, params={'access_token': access_token})
+            response = requests.get(file_url, stream=True, params={'access_token': access_token}, headers=headers)
         response.raise_for_status()
 
         chunk_size = 1024  # Define chunk size for download
@@ -107,7 +108,7 @@ def download_file(file_url,
         #                      f'but got: {file_checksum}')
     else:
         try:
-            r = requests.get(file_url)
+            r = requests.get(file_url, headers=headers)
         except requests.RequestException as e:
             r = requests.get(file_url, params={'access_token': access_token})
 
@@ -126,9 +127,9 @@ def download_file(file_url,
             file.write(r.content)
         if links_content:
             try:
-                _content_response = requests.get(links_content)
+                _content_response = requests.get(links_content, headers=headers)
             except requests.RequestException:
-                _content_response = requests.get(links_content, params={'access_token': access_token})
+                _content_response = requests.get(links_content, params={'access_token': access_token}, headers=headers)
             if _content_response.ok:
                 with open(target_filename, 'wb') as file:
                     file.write(_content_response.content)