h5rdmtoolbox 2.7.4__tar.gz → 2.8.0__tar.gz

{h5rdmtoolbox-2.7.4 → h5rdmtoolbox-2.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: h5rdmtoolbox
-Version: 2.7.4
+Version: 2.8.0
 Summary: Supporting a FAIR Research Data lifecycle using Python and HDF5.
 Author-email: Matthias Probst <matth.probst@gmail.com>
 License-Expression: MIT
@@ -78,6 +78,10 @@ Provides-Extra: complete
 Requires-Dist: h5rdmtoolbox[gui,test]; extra == "complete"
 Provides-Extra: complete-with-docs
 Requires-Dist: h5rdmtoolbox[complete,docs]; extra == "complete-with-docs"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.95; extra == "server"
+Requires-Dist: uvicorn[standard]>=0.22; extra == "server"
+Requires-Dist: jinja2>=3.1; extra == "server"
 Dynamic: license-file
 
 # HDF5 Research Data Management Toolbox
@@ -233,18 +237,37 @@ hdf:H5T_INTEL_I64 a hdf:Datatype .
     hdf:size 4 .
 ```
 
-[//]: # (The "HDF5 Research Data Management Toolbox" &#40;h5RDMtoolbox&#41; is a Python package supporting everybody who is working with)
+## Command Line Interface
 
-[//]: # (HDF5 to achieve a sustainable data lifecycle which follows)
+The `h5tbx` command provides quick access to linked-data serialization and local inspection from the shell.
 
-[//]: # (the [FAIR &#40;Findable, Accessible, Interoperable, Reusable&#41;]&#40;https://www.nature.com/articles/sdata201618&#41;)
+Use `h5tbx ld dump example.h5` to print Turtle RDF, or write a specific format with:
 
-[//]: # (principles. It specifically supports the five main steps of *planning*, *collecting*, *analyzing*, *sharing* and)
+```bash
+h5tbx ld dump example.h5 -o example.jsonld
+```
+
+The dump command accepts `--format`, `--file-uri`, `--prefix`, `--structural=false`, and `--contextual=false`, so you can choose RDF output, assign stable RDF subject URIs, bind a compact prefix, and include only structural or user-provided contextual metadata when needed.
+
+To inspect files in a browser, run:
+
+```bash
+h5tbx serve example.h5
+h5tbx serve data/ --h5ext=.h5 --h5ext=.hdf5
+```
+
+The local web app links to Turtle, JSON-LD, N-Triples, RDF/XML, an interactive graph view, SPARQL query editor, graph metrics, and SHACL validation. Per-file buttons are scoped to that file; the landing page also offers a separate combined graph section for all served files plus loaded enrichment. Combined graph visualization is reduced by default with node/edge limits and search controls, and combined metrics skip exact largest-distance computation on very large graphs to keep the local server responsive. HDF5 object URLs such as `/example.h5/group/dataset` dereference to a browser HTML page by default, while `Accept: text/turtle`, `Accept: application/ld+json`, or `?format=ttl|jsonld|nt|xml|html` return the requested representation. For published DOI-style file URIs, use `/resolve?iri=https://doi.org/...%23path` to resolve an exact external RDF subject locally. The resolver uses the first matching served HDF5 subject, caches all served files in one shared query graph, and enriches that graph with known ontology TTL sources, QUDT unit definitions, Zenodo RDF attachments, generic ontology documents, and Wikidata direct claims where applicable. URI values in resource pages link back into `/resolve?iri=...` so browsing stays inside the local app; if a browser request cannot be resolved locally, the app opens the original IRI in a new tab as a fallback. `h5tbx serve` accepts files and folders; folder discovery uses `.h5`, `.hdf`, and `.hdf5` by default, or repeated `--h5ext` values. `--local-iri-pattern` is optional and only limits which external IRIs get local resolver links in the graph view. If no filename is provided, `h5tbx serve` lists all matching HDF5 files in the current directory.
+
+The same graph metrics are available from Python:
 
-[//]: # (*reusing* data. Please visit the [documentation]&#40;https://h5rdmtoolbox.readthedocs.io/en/latest/&#41; for detailed)
+```python
+import h5rdmtoolbox as h5tbx
 
-[//]: # (information of try the [quickstart using colab]&#40;#quickstart&#41;.)
+metrics = h5tbx.compute_metrics("example.h5")
 
+with h5tbx.File("example.h5") as h5:
+    metrics = h5.metrics()
+```
 
 ## 🚀 Key Features
 

{h5rdmtoolbox-2.7.4 → h5rdmtoolbox-2.8.0}/README.md

@@ -151,18 +151,37 @@ hdf:H5T_INTEL_I64 a hdf:Datatype .
     hdf:size 4 .
 ```
 
-[//]: # (The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a Python package supporting everybody who is working with)
+## Command Line Interface
 
-[//]: # (HDF5 to achieve a sustainable data lifecycle which follows)
+The `h5tbx` command provides quick access to linked-data serialization and local inspection from the shell.
 
-[//]: # (the [FAIR (Findable, Accessible, Interoperable, Reusable)](https://www.nature.com/articles/sdata201618))
+Use `h5tbx ld dump example.h5` to print Turtle RDF, or write a specific format with:
 
-[//]: # (principles. It specifically supports the five main steps of *planning*, *collecting*, *analyzing*, *sharing* and)
+```bash
+h5tbx ld dump example.h5 -o example.jsonld
+```
+
+The dump command accepts `--format`, `--file-uri`, `--prefix`, `--structural=false`, and `--contextual=false`, so you can choose RDF output, assign stable RDF subject URIs, bind a compact prefix, and include only structural or user-provided contextual metadata when needed.
+
+To inspect files in a browser, run:
+
+```bash
+h5tbx serve example.h5
+h5tbx serve data/ --h5ext=.h5 --h5ext=.hdf5
+```
+
+The local web app links to Turtle, JSON-LD, N-Triples, RDF/XML, an interactive graph view, SPARQL query editor, graph metrics, and SHACL validation. Per-file buttons are scoped to that file; the landing page also offers a separate combined graph section for all served files plus loaded enrichment. Combined graph visualization is reduced by default with node/edge limits and search controls, and combined metrics skip exact largest-distance computation on very large graphs to keep the local server responsive. HDF5 object URLs such as `/example.h5/group/dataset` dereference to a browser HTML page by default, while `Accept: text/turtle`, `Accept: application/ld+json`, or `?format=ttl|jsonld|nt|xml|html` return the requested representation. For published DOI-style file URIs, use `/resolve?iri=https://doi.org/...%23path` to resolve an exact external RDF subject locally. The resolver uses the first matching served HDF5 subject, caches all served files in one shared query graph, and enriches that graph with known ontology TTL sources, QUDT unit definitions, Zenodo RDF attachments, generic ontology documents, and Wikidata direct claims where applicable. URI values in resource pages link back into `/resolve?iri=...` so browsing stays inside the local app; if a browser request cannot be resolved locally, the app opens the original IRI in a new tab as a fallback. `h5tbx serve` accepts files and folders; folder discovery uses `.h5`, `.hdf`, and `.hdf5` by default, or repeated `--h5ext` values. `--local-iri-pattern` is optional and only limits which external IRIs get local resolver links in the graph view. If no filename is provided, `h5tbx serve` lists all matching HDF5 files in the current directory.
+
+The same graph metrics are available from Python:
 
-[//]: # (*reusing* data. Please visit the [documentation](https://h5rdmtoolbox.readthedocs.io/en/latest/) for detailed)
+```python
+import h5rdmtoolbox as h5tbx
 
-[//]: # (information of try the [quickstart using colab](#quickstart).)
+metrics = h5tbx.compute_metrics("example.h5")
 
+with h5tbx.File("example.h5") as h5:
+    metrics = h5.metrics()
+```
 
 ## 🚀 Key Features
 

{h5rdmtoolbox-2.7.4 → h5rdmtoolbox-2.8.0}/h5rdmtoolbox/__init__.py

@@ -19,7 +19,7 @@ _stream_handler = logging.StreamHandler()
 _stream_handler.setLevel(DEFAULT_LOGGING_LEVEL)
 _stream_handler.setFormatter(_formatter)
 
-_file_handler = RotatingFileHandler(USER_LOG_DIR / 'h5rdmtoolbox.log')
+_file_handler = RotatingFileHandler(USER_LOG_DIR / 'h5rdmtoolbox.log', delay=True)
 _file_handler.setLevel(logging.DEBUG)  # log everything to file!
 _file_handler.setFormatter(_formatter)
 
@@ -146,6 +146,28 @@ def get_ld(
     )
 
 
+def compute_metrics(
+        hdf_filename: Union[str, pathlib.Path],
+        structural: bool = True,
+        contextual: bool = True,
+        skipND: int = 1,
+        file_uri: Optional[str] = None,
+        context: Optional[Dict] = None,
+        rdf_mappings: Dict[str, RDFMappingEntry] = None,
+) -> Dict[str, object]:
+    """Compute RDF knowledge graph metrics for an HDF5 file."""
+    from .ld.metrics import compute_metrics as _compute_metrics
+    return _compute_metrics(
+        hdf_filename,
+        structural=structural,
+        contextual=contextual,
+        skipND=skipND,
+        file_uri=file_uri,
+        context=context,
+        rdf_mappings=rdf_mappings,
+    )
+
+
 def _optimize_ld_context(graph: rdflib.Graph, context: Optional[Dict]) -> Dict:
     from .ld import optimize_context
     return optimize_context(graph, context or {})

h5rdmtoolbox-2.8.0/h5rdmtoolbox/cli.py

@@ -0,0 +1,4 @@
+from h5rdmtoolbox_cli import h5tbx
+
+if __name__ == '__main__':
+    h5tbx(obj={})

{h5rdmtoolbox-2.7.4 → h5rdmtoolbox-2.8.0}/h5rdmtoolbox/ld/__init__.py

@@ -12,7 +12,7 @@ from .utils import optimize_context
 
 BINARY_AS_STRING = True
 
-__all__ = ["get_ld", "hdf2jsonld", "hdf2ttl", "sparql"]
+__all__ = ["get_ld", "hdf2jsonld", "hdf2ttl", "sparql", "compute_metrics", "compute_graph_metrics"]
 
 
 def _validate_file_uri(file_uri: Optional[str]) -> None:
@@ -280,3 +280,6 @@ def _hdf2ld(
         f.write(_serialize_graph(graph, fmt=fmt, indent=indent, context=context))
 
     return metadata_filename
+
+
+from .metrics import compute_graph_metrics, compute_metrics  # noqa: E402

h5rdmtoolbox-2.8.0/h5rdmtoolbox/ld/metrics.py

@@ -0,0 +1,279 @@
+"""Knowledge graph metrics for RDF extracted from HDF5 files."""
+
+import pathlib
+import re
+from typing import Dict, Optional, Union
+
+import rdflib
+
+from . import get_ld
+
+STANDARD_PREFIXES = {
+    "dcat": "http://www.w3.org/ns/dcat#",
+    "dcterms": "http://purl.org/dc/terms/",
+    "doi": "https://doi.org/",
+    "foaf": "http://xmlns.com/foaf/0.1/",
+    "prof": "http://www.w3.org/ns/dx/prof/",
+    "prov": "http://www.w3.org/ns/prov#",
+    "qudt": "http://qudt.org/schema/qudt/",
+    "quantitykind": "http://qudt.org/vocab/quantitykind/",
+    "schema": "https://schema.org/",
+    "skos": "http://www.w3.org/2004/02/skos/core#",
+    "ssno": "https://matthiasprobst.github.io/ssno#",
+    "unit": "http://qudt.org/vocab/unit/",
+    "zenodo": "https://zenodo.org/records/",
+    "zenodo_record": "https://zenodo.org/record/",
+}
+
+
+def bind_standard_prefixes(graph: rdflib.Graph) -> None:
+    """Bind common namespaces without replacing prefixes already defined by the graph."""
+    for prefix, namespace in STANDARD_PREFIXES.items():
+        graph.bind(prefix, rdflib.Namespace(namespace), override=False, replace=False)
+
+
+def graph_label(value, rdf_graph: Optional[rdflib.Graph] = None) -> str:
+    """Return a compact display label for an RDF term."""
+    if isinstance(value, rdflib.URIRef):
+        namespace_manager = rdf_graph.namespace_manager if rdf_graph is not None else rdflib.Graph().namespace_manager
+        text = str(value)
+        try:
+            normalized = str(namespace_manager.normalizeUri(value))
+            if not normalized.startswith("<") and not re.match(r"^ns\d+:", normalized):
+                return normalized
+        except Exception:
+            pass
+        for prefix, namespace in STANDARD_PREFIXES.items():
+            if text.startswith(namespace) and text != namespace:
+                return f"{prefix}:{text[len(namespace):]}"
+        try:
+            return namespace_manager.qname(value)
+        except Exception:
+            pass
+        if rdf_graph is not None:
+            namespaces = sorted(
+                ((prefix, str(namespace)) for prefix, namespace in rdf_graph.namespaces() if prefix),
+                key=lambda item: len(item[1]),
+                reverse=True,
+            )
+            for prefix, namespace in namespaces:
+                if text.startswith(namespace) and text != namespace:
+                    return f"{prefix}:{text[len(namespace):]}"
+        return text.rsplit("#", 1)[-1].rsplit("/", 1)[-1] or text
+    text = str(value)
+    return text if len(text) <= 48 else f"{text[:45]}..."
+
+
+def _percent(numerator: int, denominator: int) -> float:
+    return (numerator / denominator * 100.0) if denominator else 0.0
+
+
+def _external_namespace(value) -> str:
+    text = str(value)
+    if "#" in text:
+        return text.rsplit("#", 1)[0] + "#"
+    if "/" in text:
+        return text.rsplit("/", 1)[0] + "/"
+    return text
+
+
+def _resource_terms(subject, predicate, obj) -> list:
+    return [
+        term
+        for term in (subject, predicate, obj)
+        if isinstance(term, (rdflib.URIRef, rdflib.BNode))
+    ]
+
+
+def compute_graph_metrics(
+        rdf_graph: rdflib.Graph,
+        base_namespace: Optional[str] = None,
+        compute_distances: bool = True,
+        distance_node_limit: int = 2000,
+) -> Dict[str, object]:
+    """Compute RDF knowledge graph metrics for an existing graph."""
+    bind_standard_prefixes(rdf_graph)
+    triples = list(rdf_graph)
+    subjects = set()
+    objects = set()
+    predicates = set()
+    iris = set()
+    blank_nodes = set()
+    resource_nodes = set()
+    literal_count = 0
+    untyped_literal_count = 0
+    empty_literal_count = 0
+    predicate_counts = {}
+    subject_triple_counts = {}
+    class_instances = {}
+    datatype_counts = {}
+    language_counts = {}
+    labels = set()
+    typed_subjects = set()
+    same_as_count = 0
+    external_iri_counts = {}
+    out_degree = {}
+    in_degree = {}
+    adjacency = {}
+
+    for subject, predicate, obj in triples:
+        subjects.add(subject)
+        predicates.add(predicate)
+        objects.add(obj)
+        subject_triple_counts[subject] = subject_triple_counts.get(subject, 0) + 1
+
+        for term in _resource_terms(subject, predicate, obj):
+            if isinstance(term, rdflib.URIRef):
+                iris.add(term)
+                namespace = _external_namespace(term)
+                if not base_namespace or not str(term).startswith(base_namespace):
+                    external_iri_counts[namespace] = external_iri_counts.get(namespace, 0) + 1
+            elif isinstance(term, rdflib.BNode):
+                blank_nodes.add(term)
+
+        predicate_label = graph_label(predicate, rdf_graph)
+        predicate_counts[predicate_label] = predicate_counts.get(predicate_label, 0) + 1
+
+        if predicate == rdflib.RDF.type:
+            typed_subjects.add(subject)
+            class_label = graph_label(obj, rdf_graph)
+            class_instances.setdefault(class_label, set()).add(subject)
+
+        if predicate == rdflib.RDFS.label:
+            labels.add(subject)
+
+        if predicate == rdflib.OWL.sameAs:
+            same_as_count += 1
+
+        resource_nodes.add(subject)
+
+        if isinstance(obj, rdflib.Literal):
+            literal_count += 1
+            datatype_label = graph_label(obj.datatype, rdf_graph) if obj.datatype else "None"
+            datatype_counts[datatype_label] = datatype_counts.get(datatype_label, 0) + 1
+            language_label = obj.language or "None"
+            language_counts[language_label] = language_counts.get(language_label, 0) + 1
+            if obj.datatype is None and obj.language is None:
+                untyped_literal_count += 1
+            if str(obj) == "":
+                empty_literal_count += 1
+            continue
+
+        resource_nodes.add(obj)
+        out_degree[subject] = out_degree.get(subject, 0) + 1
+        in_degree[obj] = in_degree.get(obj, 0) + 1
+        adjacency.setdefault(subject, set()).add(obj)
+        adjacency.setdefault(obj, set()).add(subject)
+
+    seen = set()
+    component_sizes = []
+    for node in resource_nodes:
+        if node in seen:
+            continue
+        stack = [node]
+        seen.add(node)
+        size = 0
+        while stack:
+            current = stack.pop()
+            size += 1
+            for neighbor in adjacency.get(current, set()):
+                if neighbor not in seen:
+                    seen.add(neighbor)
+                    stack.append(neighbor)
+        component_sizes.append(size)
+
+    resource_count = len(resource_nodes)
+    largest_distance = None
+    largest_distance_computed = compute_distances and resource_count <= distance_node_limit
+    if largest_distance_computed:
+        largest_distance = 0
+        for start in resource_nodes:
+            distances = {start: 0}
+            queue = [start]
+            for current in queue:
+                for neighbor in adjacency.get(current, set()):
+                    if neighbor not in distances:
+                        distances[neighbor] = distances[current] + 1
+                        queue.append(neighbor)
+            if distances:
+                largest_distance = max(largest_distance, max(distances.values()))
+
+    subjects_without_type = len(subjects - typed_subjects)
+    subjects_with_labels = len(labels)
+    subjects_missing_labels = len(subjects - labels)
+    duplicate_triples = len(triples) - len(set(triples))
+    subjects_with_one_triple = sum(1 for count in subject_triple_counts.values() if count == 1)
+    weakly_connected_nodes = sum(
+        1 for node in resource_nodes
+        if in_degree.get(node, 0) + out_degree.get(node, 0) <= 1
+    )
+    class_counts = {label: len(instances) for label, instances in class_instances.items()}
+
+    return {
+        "triples": len(triples),
+        "subjects": len(subjects),
+        "predicates": len(predicates),
+        "objects": len(objects),
+        "iri_count": len(iris),
+        "blank_nodes": len(blank_nodes),
+        "literals": literal_count,
+        "avg_triples_per_subject": len(triples) / len(subjects) if subjects else 0.0,
+        "classes": len(class_counts),
+        "subjects_without_type": subjects_without_type,
+        "resource_nodes": resource_count,
+        "avg_out_degree": sum(out_degree.values()) / resource_count if resource_count else 0.0,
+        "avg_in_degree": sum(in_degree.values()) / resource_count if resource_count else 0.0,
+        "weakly_connected_nodes": weakly_connected_nodes,
+        "components": len(component_sizes),
+        "largest_component": max(component_sizes, default=0),
+        "largest_distance": largest_distance,
+        "largest_distance_computed": largest_distance_computed,
+        "largest_distance_node_limit": distance_node_limit,
+        "datatype_distribution": sorted(datatype_counts.items(), key=lambda item: (-item[1], item[0])),
+        "language_distribution": sorted(language_counts.items(), key=lambda item: (-item[1], item[0])),
+        "untyped_literals": untyped_literal_count,
+        "empty_literals": empty_literal_count,
+        "subjects_with_labels": subjects_with_labels,
+        "subjects_missing_labels": subjects_missing_labels,
+        "label_coverage": _percent(subjects_with_labels, len(subjects)),
+        "duplicate_triples": duplicate_triples,
+        "subjects_with_one_triple": subjects_with_one_triple,
+        "blank_node_percentage": _percent(len(blank_nodes), resource_count),
+        "same_as_count": same_as_count,
+        "external_iri_count": sum(external_iri_counts.values()),
+        "top_external_namespaces": sorted(external_iri_counts.items(), key=lambda item: (-item[1], item[0]))[:10],
+        "top_predicates": sorted(predicate_counts.items(), key=lambda item: (-item[1], item[0]))[:10],
+        "rare_predicates": sorted(predicate_counts.items(), key=lambda item: (item[1], item[0]))[:10],
+        "predicate_distribution": sorted(predicate_counts.items(), key=lambda item: (-item[1], item[0])),
+        "top_classes": sorted(class_counts.items(), key=lambda item: (-item[1], item[0]))[:10],
+        "top_out_degree": [
+            (graph_label(node, rdf_graph), count)
+            for node, count in sorted(out_degree.items(), key=lambda item: (-item[1], graph_label(item[0], rdf_graph)))[:10]
+        ],
+        "top_in_degree": [
+            (graph_label(node, rdf_graph), count)
+            for node, count in sorted(in_degree.items(), key=lambda item: (-item[1], graph_label(item[0], rdf_graph)))[:10]
+        ],
+    }
+
+
+def compute_metrics(
+        hdf_filename: Union[str, pathlib.Path],
+        structural: bool = True,
+        contextual: bool = True,
+        file_uri: Optional[str] = None,
+        skipND: Optional[int] = 1,
+        context: Optional[Dict] = None,
+        rdf_mappings: Optional[Dict] = None,
+) -> Dict[str, object]:
+    """Compute RDF knowledge graph metrics for an HDF5 file."""
+    graph = get_ld(
+        hdf_filename,
+        structural=structural,
+        contextual=contextual,
+        file_uri=file_uri,
+        skipND=skipND,
+        context=context,
+        rdf_mappings=rdf_mappings,
+    )
+    return compute_graph_metrics(graph, base_namespace=file_uri)

h5rdmtoolbox-2.8.0/h5rdmtoolbox/llm/__init__.py

@@ -0,0 +1,15 @@
+"""LLM-based chat support for HDF5 metadata graphs."""
+
+from .chat import chat
+from .models import ChatResponse, ChatSession, ChatSessionState, QueryAttempt
+from .providers.openai_adapter import OpenAIChatClient
+
+__all__ = [
+    "chat",
+    "ChatResponse",
+    "ChatSession",
+    "ChatSessionState",
+    "QueryAttempt",
+    "OpenAIChatClient",
+]
+