ddigraph 0.4.0__py3-none-any.whl

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Philippe Bisson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

NOTICE

@@ -0,0 +1,2 @@
+ddigraph (formerly neo4ddi) is distributed under the MIT License.
+See the accompanying LICENSE file for the full license text.

README.md

@@ -0,0 +1,160 @@
+# ddigraph
+
+[![CI](https://img.shields.io/github/actions/workflow/status/pbisson44/ddigraph/ci.yml?label=CI&logo=github)](https://github.com/pbisson44/ddigraph/actions)
+[![codecov](https://codecov.io/gh/pbisson44/ddigraph/branch/main/graph/badge.svg)](https://codecov.io/gh/pbisson44/ddigraph)
+[![PyPI](https://img.shields.io/pypi/v/ddigraph?logo=pypi&logoColor=white)](https://pypi.org/project/ddigraph/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/Python-3.12%E2%80%933.14-blue?logo=python)](pyproject.toml)
+[![Neo4j](https://img.shields.io/badge/Neo4j-5.x-green?logo=neo4j)](https://neo4j.com/docs/)
+[![Code style](https://img.shields.io/badge/code%20style-ruff-000000?logo=ruff&logoColor=white)](https://docs.astral.sh/ruff/)
+[![Type checking](https://img.shields.io/badge/type%20checking-mypy-1678be?logo=mypy&logoColor=white)](https://mypy-lang.org/)
+
+A modern Python toolkit that transforms [DDI](https://ddialliance.org/) (Data Documentation
+Initiative) XML metadata into knowledge graphs. Supports **DDI Codebook** and **DDI-L
+FragmentInstance** formats with streaming parsing, batched writes, and full async I/O across
+multiple graph backends.
+
+[Documentation](https://pbisson44.github.io/ddigraph/) |
+[Getting Started](https://pbisson44.github.io/ddigraph/getting-started/installation/) |
+[PyPI](https://pypi.org/project/ddigraph/) |
+[Source Code](https://github.com/pbisson44/ddigraph)
+
+---
+
+## Features
+
+- **Multi-backend support** -- Neo4j, RDF/SPARQL, Gremlin, NetworkX, and pandas
+- **Streaming XML processing** -- Memory-bounded `iterparse` for files of any size
+- **Batched writes** -- UNWIND-based Cypher for 10-100x fewer database round trips
+- **Async I/O** -- Concurrent parsing and writing with back-pressure control
+- **Format auto-detection** -- Automatically identifies DDI Codebook vs Lifecycle format
+- **Unified schema** -- Single source of truth for all node and relationship definitions
+- **Adapter pattern** -- Plug in custom graph backends via `GraphWriteAdapter` protocol
+- **Production-ready** -- Retry logic, observability hooks, pydantic-based configuration
+
+## Quick Start
+
+### Install
+
+```bash
+pip install ddigraph
+```
+
+### Load DDI metadata (CLI)
+
+```bash
+# Set Neo4j connection
+export DDIGRAPH_NEO4J_URI=bolt://localhost:7687
+export DDIGRAPH_NEO4J_USER=neo4j
+export DDIGRAPH_NEO4J_PASSWORD=secret
+
+# Bootstrap schema and load data (format is auto-detected)
+ddigraph bootstrap
+ddigraph load survey.xml --dataset-id my-survey
+```
+
+### Load DDI metadata (Python)
+
+```python
+import asyncio
+from neo4j import AsyncGraphDatabase
+from ddigraph import DDILoader, DDIFragmentLoader, detect_ddi_format
+from ddigraph.config import Settings
+
+async def main():
+    settings = Settings()
+    driver = AsyncGraphDatabase.driver(
+        settings.neo4j_uri,
+        auth=(settings.neo4j_user, settings.neo4j_password.get_secret_value()),
+    )
+    path = "survey.xml"
+    if detect_ddi_format(path) == "lifecycle":
+        loader = DDIFragmentLoader(driver, settings=settings)
+        result = await loader.load(path)
+    else:
+        loader = DDILoader(driver, settings=settings)
+        result = await loader.load(path, dataset_id="my-survey")
+    print(result)  # {'Instrument': 1, 'Sequence': 388, 'QuestionItem': 373, ...}
+    await driver.close()
+
+asyncio.run(main())
+```
+
+## Supported Formats
+
+| Format | Description | Use Case |
+| ------ | ----------- | -------- |
+| **DDI Codebook** | Traditional flat format with central Dataset node | Survey archives, data catalogs |
+| **DDI-L FragmentInstance** | Lifecycle 3.x format with reusable fragments | Questionnaire design, CAPI/CAWI instruments |
+| **DDI-CDI 1.0** | Cross-Domain Integration metadata | Data integration, statistical production |
+
+### XSD Coverage
+
+`ddigraph` ships with 100 % coverage of every concrete identifiable element
+declared in the bundled XSD schemas (`schemas/`).  Coverage is enforced by the
+audit script and a pytest guardrail so new schema releases surface any gaps:
+
+| Flavor      | Scope                                                                 | Target | Covered |
+| ----------- | --------------------------------------------------------------------- | -----: | ------: |
+| DDI-L 3.x   | Concrete Maintainable + Versionable + Identifiable elements           |    189 |  100 %  |
+| DDI-C 2.x   | Codebook elements with the `GLOBALS` attribute group (no layout tags) |     73 |  100 %  |
+| DDI-CDI 1.0 | Concrete top-level entity elements (associations excluded)            |    210 |  100 %  |
+
+Run `python scripts/xsd_coverage.py` to regenerate the audit or
+`python scripts/xsd_coverage.py --json` for machine-readable output.
+
+## Supported Backends
+
+| Backend | Description | Use Case |
+| ------- | ----------- | -------- |
+| **Neo4j** | Native graph database (Bolt) | Production deployments, complex queries |
+| **RDF/SPARQL** | Semantic web triplestores | Linked data, ontology integration |
+| **Gremlin** | Graph traversal language | JanusGraph, Neptune, Cosmos DB |
+| **NetworkX** | Python graph library | Local analysis, prototyping |
+| **pandas** | DataFrame-based | Tabular analysis, Excel export |
+
+## Docker Quick Start
+
+```bash
+docker run --rm --name neo4j-demo \
+  -p 7474:7474 -p 7687:7687 \
+  -e NEO4J_AUTH=neo4j/password \
+  neo4j:5
+
+export DDIGRAPH_NEO4J_URI=bolt://localhost:7687
+export DDIGRAPH_NEO4J_USER=neo4j
+export DDIGRAPH_NEO4J_PASSWORD=password
+
+ddigraph bootstrap
+ddigraph load your-file.xml --dataset-id demo
+```
+
+## Documentation
+
+Full documentation is available at **[pbisson44.github.io/ddigraph](https://pbisson44.github.io/ddigraph/)** in English and French.
+
+- [Getting Started](https://pbisson44.github.io/ddigraph/getting-started/installation/) -- Installation, quick start, 10-minute tutorial
+- [User Guide](https://pbisson44.github.io/ddigraph/user-guide/architecture/) -- Architecture, DDI formats, relationships, adapters
+- [Graph Backends](https://pbisson44.github.io/ddigraph/backends/neo4j/) -- Neo4j, RDF/SPARQL, Gremlin, NetworkX
+- [Reference](https://pbisson44.github.io/ddigraph/reference/cli/) -- CLI commands, configuration
+- [Advanced](https://pbisson44.github.io/ddigraph/advanced/tuning/) -- Performance tuning, AI readiness, standards interoperability
+- [Contributing](https://pbisson44.github.io/ddigraph/project/contributing/) -- How to contribute
+
+## Development
+
+```bash
+git clone https://github.com/pbisson44/ddigraph.git
+cd ddigraph
+pip install -e ".[dev,docs]"
+
+ruff check . && ruff format .
+# Docstring linting is currently enforced for src/ddigraph only.
+pydocstyle src/ddigraph
+mypy .
+pytest
+mkdocs serve
+```
+
+## License
+
+MIT -- see [LICENSE](LICENSE) for details.

ddigraph/__init__.py

@@ -0,0 +1,108 @@
+"""ddigraph - DDI to Knowledge Graph transformation toolkit.
+
+This package transforms DDI (Data Documentation Initiative) XML
+metadata into a Neo4j knowledge graph. Streaming parsers also emit
+records that can drive other backends through the parser tier --
+see ``demo/load_rdf.py``, ``demo/load_gremlin.py``,
+``demo/load_networkx.py``, and ``demo/load_pandas.py`` for examples.
+The high-level entry points are:
+
+* :func:`ddigraph.load` -- sync load of a DDI file into a Neo4j target.
+* :func:`ddigraph.aload` -- async equivalent of ``load``.
+* :func:`ddigraph.detect` -- identify the DDI flavor (codebook,
+  lifecycle, cdi) of a file without loading.
+* :func:`ddigraph.bootstrap` -- create the indexes/constraints DDI
+  ingestion needs.
+
+Typical usage::
+
+    import ddigraph
+
+    ddigraph.bootstrap(target="bolt://localhost:7687")
+    result = ddigraph.load("survey.xml", target="bolt://localhost:7687")
+    print(result.nodes_written, "nodes,", result.relationships_written, "relationships")
+
+When ``target`` is omitted, connection details come from the env-driven
+:class:`~ddigraph.config.Settings` model (``DDIGRAPH_NEO4J_URI``,
+``DDIGRAPH_NEO4J_USER``, ``DDIGRAPH_NEO4J_PASSWORD``).
+
+The public API surface ships in two tiers:
+
+* **Supported** -- ``load``, ``aload``, ``detect``, ``bootstrap``,
+  ``abootstrap``, ``LoadResult``, ``Settings``, ``__version__``.
+  These names follow semantic versioning across minor releases.
+* **Power-user** -- ``DDILoader``, ``DDIFragmentLoader``,
+  ``DDIFragmentParser``, ``DDIBatch``, ``CDIBatch``,
+  ``CDIBatchStream``, ``DDISchema``, ``Fragment``,
+  ``FragmentReference``, ``FlavorName``, ``detect_ddi_format``,
+  ``is_cdi_format``, ``parse_ddi_batches``, ``parse_cdi_batches``.
+  Importable from ``ddigraph`` for fine-grained control, but they
+  carry no stability guarantee across minor releases. Pin a version
+  if you depend on them.
+"""
+
+from importlib.metadata import PackageNotFoundError, version
+
+from ddigraph.api import (
+    FlavorName,
+    LoadResult,
+    abootstrap,
+    aload,
+    bootstrap,
+    detect,
+    load,
+)
+from ddigraph.config import Settings
+from ddigraph.ingest.cdi_loader import (
+    CDIBatch,
+    CDIBatchStream,
+    is_cdi_format,
+    parse_cdi_batches,
+)
+from ddigraph.ingest.fragment_loader import (
+    DDIFragmentLoader,
+    DDIFragmentParser,
+    Fragment,
+    FragmentReference,
+    detect_ddi_format,
+)
+from ddigraph.ingest.loader import DDIBatch, DDILoader, parse_ddi_batches
+from ddigraph.schema.definitions import DDISchema
+
+try:
+    __version__ = version("ddigraph")
+except PackageNotFoundError:
+    # Package not installed (development mode)
+    __version__ = "0.4.0"
+
+# Intentionally split into two tiers (supported / power-user) with a
+# blank-line break instead of alphabetised; see the module docstring.
+__all__ = [  # noqa: RUF022 (tier ordering is intentional)
+    # Supported public API -- the 90 % case, semver-stable across
+    # minor releases. See the module docstring for details.
+    "LoadResult",
+    "Settings",
+    "__version__",
+    "abootstrap",
+    "aload",
+    "bootstrap",
+    "detect",
+    "load",
+    # Power-user surface -- the parser tier, batch types, and the
+    # shared schema container. Importable from ``ddigraph`` but carries
+    # no stability guarantee across minor releases.
+    "CDIBatch",
+    "CDIBatchStream",
+    "DDIBatch",
+    "DDIFragmentLoader",
+    "DDIFragmentParser",
+    "DDILoader",
+    "DDISchema",
+    "FlavorName",
+    "Fragment",
+    "FragmentReference",
+    "detect_ddi_format",
+    "is_cdi_format",
+    "parse_cdi_batches",
+    "parse_ddi_batches",
+]

ddigraph/api.py

@@ -0,0 +1,296 @@
+"""CRUD-simple public API for ``ddigraph``.
+
+The four functions below cover the 90 % case. Power users still have
+:class:`~ddigraph.ingest.loader.DDILoader`,
+:class:`~ddigraph.ingest.fragment_loader.DDIFragmentLoader`, and the
+CDI ``parse_cdi_batches`` family available for fine-grained control;
+this module just spares ordinary callers from building drivers, picking
+a flavor-specific loader, and chaining ``asyncio.run`` themselves.
+
+Typical usage::
+
+    import ddigraph
+
+    # Bootstrap the target's schema once.
+    ddigraph.bootstrap(target="bolt://localhost:7687")
+
+    # Stream a DDI file into the target. Format auto-detected.
+    result = ddigraph.load("survey.xml", target="bolt://localhost:7687")
+    print(result.nodes_written, "nodes,", result.relationships_written, "relationships")
+
+Connection credentials default to the env-driven
+:class:`~ddigraph.config.Settings` model when ``target`` is omitted, so
+the existing ``DDIGRAPH_NEO4J_*`` (or legacy ``NEO4J_*``) variables
+continue to work.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from dataclasses import dataclass
+from pathlib import Path
+from time import perf_counter
+from typing import Literal
+
+from neo4j import AsyncDriver, AsyncGraphDatabase
+
+from ddigraph.config import Settings
+from ddigraph.graph.bootstrap import ensure_schema as _ensure_schema
+from ddigraph.ingest.fragment_loader import (
+    DDIFragmentLoader,
+    detect_ddi_format,
+)
+from ddigraph.ingest.loader import DDILoader
+
+type FlavorName = Literal["codebook", "lifecycle", "cdi", "unknown"]
+
+
+@dataclass(slots=True)
+class LoadResult:
+    """Summary of a single ``ddigraph.load`` invocation.
+
+    Attributes:
+        flavor: One of ``"codebook"`` or ``"lifecycle"`` (CDI ingestion
+            is not yet covered by this entry point).
+        target: The connection URL the load wrote to.
+        dataset_id: Identifier assigned to the ingested dataset (the
+            codebook flavor always sets one; lifecycle leaves it
+            ``None``).
+        nodes_written: Number of graph nodes the loader recorded.
+        relationships_written: Number of relationships recorded.
+        duration_s: Wall-clock seconds the load took.
+        dry_run: True if the load ran in dry-run mode (no writes).
+        totals: The raw per-entity counts the underlying loader returned.
+    """
+
+    flavor: FlavorName
+    target: str
+    dataset_id: str | None
+    nodes_written: int
+    relationships_written: int
+    duration_s: float
+    dry_run: bool
+    totals: dict[str, int]
+
+
+def detect(path: str | Path) -> FlavorName:
+    """Return the DDI flavor of ``path``.
+
+    Thin typed wrapper over
+    :func:`ddigraph.ingest.fragment_loader.detect_ddi_format` so callers
+    get a real ``Literal`` instead of a free-form string.
+    """
+    raw = detect_ddi_format(path)
+    if raw in ("codebook", "lifecycle", "cdi"):
+        return raw  # type: ignore[return-value]
+    return "unknown"
+
+
+def _resolve_settings(target: str | None, settings: Settings | None) -> tuple[Settings, str]:
+    """Materialise a ``Settings`` instance and the URI it points at.
+
+    If ``settings`` is given, ``target`` overrides its ``neo4j_uri``;
+    otherwise a fresh ``Settings()`` is built from environment.
+    """
+    base = settings or Settings()
+    if target is not None:
+        # pydantic models are frozen by default; rebuild via model_copy.
+        base = base.model_copy(update={"neo4j_uri": target})
+    return base, base.neo4j_uri
+
+
+def _driver(settings: Settings) -> AsyncDriver:
+    """Build an ``AsyncDriver`` from a ``Settings`` instance."""
+    return AsyncGraphDatabase.driver(
+        settings.neo4j_uri,
+        auth=(settings.neo4j_user, settings.neo4j_password.get_secret_value()),
+    )
+
+
+def _default_dataset_id(path: str | Path) -> str:
+    """Derive a dataset identifier from a file name (codebook flavor only)."""
+    stem = Path(path).stem.replace(" ", "_")
+    # A path ending in ``.xml`` with no real stem (e.g. ``/path/.xml``)
+    # produces a leading-dot stem; treat that as a missing identifier.
+    if not stem or stem.startswith("."):
+        return "default"
+    return stem
+
+
+async def aload(
+    path: str | Path,
+    *,
+    target: str | None = None,
+    dataset_id: str | None = None,
+    dataset_name: str | None = None,
+    dry_run: bool = False,
+    replace: bool = False,
+    settings: Settings | None = None,
+) -> LoadResult:
+    """Async load of a DDI file into the configured Neo4j target.
+
+    Format auto-detection picks DDI-Codebook or DDI-L Lifecycle and
+    dispatches to the matching loader. DDI-CDI is parsed but not yet
+    persisted by this entry point (use ``ddigraph.parse_cdi_batches``).
+    Non-Neo4j backends (RDF, Gremlin, NetworkX, pandas) are not driven
+    through ``load``; use the parser tier and a backend-specific
+    adapter (see the ``demo/load_*.py`` examples).
+
+    Args:
+        path: Filesystem path to the DDI XML.
+        target: Neo4j URL (``bolt://...`` or ``neo4j://...``). When
+            omitted the env-driven ``DDIGRAPH_NEO4J_URI`` is used.
+        dataset_id: Dataset identifier (codebook flavor). Defaults to
+            the file stem when not supplied.
+        dataset_name: Human-readable dataset name (codebook flavor).
+        dry_run: When True, parse and validate without writing.
+        replace: When True, purge existing dataset content before
+            loading (codebook flavor only; lifecycle ``clear_first``).
+        settings: Optional pre-built ``Settings`` instance.
+
+    Returns:
+        A :class:`LoadResult` describing the load outcome.
+
+    Raises:
+        ValueError: If ``path`` does not point at a readable XML file.
+        NotImplementedError: If ``path`` is a CDI document (not yet
+            persisted by this entry point).
+    """
+    resolved_settings, target_uri = _resolve_settings(target, settings)
+    flavor = detect(path)
+    if flavor == "cdi":
+        raise NotImplementedError(
+            "ddigraph.load/aload does not yet persist DDI-CDI documents. "
+            "Use ddigraph.parse_cdi_batches and a custom adapter."
+        )
+
+    start = perf_counter()
+    driver = _driver(resolved_settings)
+    try:
+        totals: dict[str, int]
+        if flavor == "lifecycle":
+            loader = DDIFragmentLoader(driver, settings=resolved_settings)
+            totals = await loader.load(path=path, clear_first=replace)
+            resolved_dataset_id: str | None = None
+        else:
+            # Codebook flavor uses the sync DDILoader under the hood.
+            from neo4j import GraphDatabase
+
+            sync_driver = GraphDatabase.driver(
+                resolved_settings.neo4j_uri,
+                auth=(
+                    resolved_settings.neo4j_user,
+                    resolved_settings.neo4j_password.get_secret_value(),
+                ),
+            )
+            try:
+                codebook_loader = DDILoader(sync_driver, settings=resolved_settings)
+                resolved_dataset_id = dataset_id or _default_dataset_id(path)
+                totals = await codebook_loader.load(
+                    path=path,
+                    dataset_id=resolved_dataset_id,
+                    dataset_name=dataset_name,
+                    dry_run=dry_run,
+                    replace=replace,
+                )
+            finally:
+                sync_driver.close()
+    finally:
+        await driver.close()
+
+    nodes = sum(v for k, v in totals.items() if "relationship" not in k.lower())
+    rels = sum(v for k, v in totals.items() if "relationship" in k.lower())
+
+    return LoadResult(
+        flavor=flavor,
+        target=target_uri,
+        dataset_id=resolved_dataset_id,
+        nodes_written=nodes,
+        relationships_written=rels,
+        duration_s=perf_counter() - start,
+        dry_run=dry_run,
+        totals=totals,
+    )
+
+
+def load(
+    path: str | Path,
+    *,
+    target: str | None = None,
+    dataset_id: str | None = None,
+    dataset_name: str | None = None,
+    dry_run: bool = False,
+    replace: bool = False,
+    settings: Settings | None = None,
+) -> LoadResult:
+    """Synchronously load a DDI file into the configured Neo4j target.
+
+    Internally drives :func:`aload` via :func:`asyncio.run`. Use
+    :func:`aload` directly when calling from already-async code.
+
+    See :func:`aload` for argument details.
+    """
+    return asyncio.run(
+        aload(
+            path,
+            target=target,
+            dataset_id=dataset_id,
+            dataset_name=dataset_name,
+            dry_run=dry_run,
+            replace=replace,
+            settings=settings,
+        )
+    )
+
+
+async def abootstrap(
+    *,
+    target: str | None = None,
+    include_fragments: bool = True,
+    settings: Settings | None = None,
+) -> None:
+    """Async equivalent of :func:`bootstrap`."""
+    resolved_settings, _ = _resolve_settings(target, settings)
+    driver = _driver(resolved_settings)
+    try:
+        await _ensure_schema(
+            driver,
+            database=resolved_settings.neo4j_database,
+            include_fragments=include_fragments,
+        )
+    finally:
+        await driver.close()
+
+
+def bootstrap(
+    *,
+    target: str | None = None,
+    include_fragments: bool = True,
+    settings: Settings | None = None,
+) -> None:
+    """Create the indexes and constraints DDI ingestion needs.
+
+    Args:
+        target: Neo4j URL. Defaults to env-driven settings.
+        include_fragments: When True, also create DDI-L Lifecycle
+            constraints alongside the Codebook ones.
+        settings: Optional pre-built ``Settings`` instance.
+    """
+    asyncio.run(
+        abootstrap(
+            target=target,
+            include_fragments=include_fragments,
+            settings=settings,
+        )
+    )
+
+
+__all__ = [
+    "FlavorName",
+    "LoadResult",
+    "abootstrap",
+    "aload",
+    "bootstrap",
+    "detect",
+    "load",
+]