sdmxlib 0.8.0__tar.gz

sdmxlib-0.8.0/PKG-INFO

@@ -0,0 +1,102 @@
+Metadata-Version: 2.3
+Name: sdmxlib
+Version: 0.8.0
+Summary: SDMX structural metadata library for Python
+Keywords: sdmx,statistics,metadata,datastructure
+Author: gabrielgellner
+Author-email: gabrielgellner <gabrielgellner@gmail.com>
+License: Apache-2.0
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Dist: attrs>=26.1.0
+Requires-Dist: duckdb>=1.0.0
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: lxml>=6.0.2
+Requires-Dist: polars>=1.39.3
+Requires-Python: >=3.12
+Project-URL: Homepage, https://gitlab.com/gabrielgellner/sdmxlib
+Project-URL: Repository, https://gitlab.com/gabrielgellner/sdmxlib
+Project-URL: Documentation, https://gabrielgellner.gitlab.io/sdmxlib
+Project-URL: Changelog, https://gitlab.com/gabrielgellner/sdmxlib/-/blob/main/CHANGELOG.md
+Description-Content-Type: text/markdown
+
+# sdmxlib
+
+> **Note:** This library has been written extensively with AI assistance
+> (Claude Code). Users who are not comfortable with AI-generated code should
+> take that into account before adopting it.
+
+SDMX structural metadata library for Python. Fetch and navigate Data Structure
+Definitions, Codelists, Dataflows, and related artefacts from any SDMX REST
+endpoint — with plain attribute access, no XML or URN chasing required.
+
+## Quick start
+
+```python
+import sdmxlib as sl
+
+with sl.RestRegistry(sl.Provider.BIS) as reg:
+    dsd = reg.get(sl.DataStructure, agency="BIS", id="WS_CBPOL").resolve()
+
+    for dim in dsd.dimensions:
+        if dim.is_enumerated:
+            cl = dim.representation()
+            print(f"{dim.id}: {cl.id} ({len(cl.codes)} codes)")
+```
+
+```
+FREQ: CL_FREQ (7 codes)
+REF_AREA: CL_BIS_IF_REF_AREA (44 codes)
+INSTR_ASSET: CL_INSTR_ASSET (12 codes)
+BORROWER_CTY: CL_BIS_IF_REF_AREA (44 codes)
+```
+
+## Installation
+
+```bash
+uv add "sdmxlib @ git+https://gitlab.com/gabrielgellner/sdmxlib.git"
+```
+
+## What it does
+
+- Connects to any SDMX 2.1 or 3.0 REST endpoint; built-in shortcuts for ECB,
+  BIS, ESTAT, IMF, OECD, and the SDMX Global Registry via `sl.Provider`
+- Three resolution modes: `browse()` for catalogue scanning, `fetch()` for
+  unresolved artefacts, `resolve()` for the full reference graph (with optional
+  constraint-trimming via `partial=True/False`)
+- Returns fully resolved domain objects with plain attribute access — no XML,
+  no URN chasing
+- Filter collections with lambdas or the `item()` expression API
+- Build SDMX artefacts locally and serialise to SDMX-ML or SDMX-JSON
+- `LocalRegistry` -- a DuckDB-backed persistent store for offline access
+  with lazy codelist loading, automatic Ref resolution, and direct SQL
+  code maps; supports shared connections and configurable language columns
+- Integrates with [Polars](https://pola.rs) for schema generation and data
+  validation
+- Fully typed — IDE completion and static analysis work throughout
+
+## Documentation
+
+See the [docs](https://gabrielgellner.gitlab.io/sdmxlib/) for the full cookbook,
+including:
+
+- Connecting to registries and choosing a resolution mode
+- Persisting artefacts in a local DuckDB store with `LocalRegistry`
+- Fetching observation data with `sl.dim()` filters and pivoting to wide format
+- Inspecting DSDs and building code maps for ETL pipelines
+- Generating Polars schemas from SDMX Data Structure Definitions
+- Browsing data catalogs and filtering dataflows
+- Working with constraints, hierarchies, and serialisation
+
+## Development
+
+```bash
+just check           # lint + typecheck
+just test            # unit tests
+just test-integration  # live endpoint tests (requires network)
+just ci              # full pre-commit gate
+```

sdmxlib-0.8.0/README.md

@@ -0,0 +1,76 @@
+# sdmxlib
+
+> **Note:** This library has been written extensively with AI assistance
+> (Claude Code). Users who are not comfortable with AI-generated code should
+> take that into account before adopting it.
+
+SDMX structural metadata library for Python. Fetch and navigate Data Structure
+Definitions, Codelists, Dataflows, and related artefacts from any SDMX REST
+endpoint — with plain attribute access, no XML or URN chasing required.
+
+## Quick start
+
+```python
+import sdmxlib as sl
+
+with sl.RestRegistry(sl.Provider.BIS) as reg:
+    dsd = reg.get(sl.DataStructure, agency="BIS", id="WS_CBPOL").resolve()
+
+    for dim in dsd.dimensions:
+        if dim.is_enumerated:
+            cl = dim.representation()
+            print(f"{dim.id}: {cl.id} ({len(cl.codes)} codes)")
+```
+
+```
+FREQ: CL_FREQ (7 codes)
+REF_AREA: CL_BIS_IF_REF_AREA (44 codes)
+INSTR_ASSET: CL_INSTR_ASSET (12 codes)
+BORROWER_CTY: CL_BIS_IF_REF_AREA (44 codes)
+```
+
+## Installation
+
+```bash
+uv add "sdmxlib @ git+https://gitlab.com/gabrielgellner/sdmxlib.git"
+```
+
+## What it does
+
+- Connects to any SDMX 2.1 or 3.0 REST endpoint; built-in shortcuts for ECB,
+  BIS, ESTAT, IMF, OECD, and the SDMX Global Registry via `sl.Provider`
+- Three resolution modes: `browse()` for catalogue scanning, `fetch()` for
+  unresolved artefacts, `resolve()` for the full reference graph (with optional
+  constraint-trimming via `partial=True/False`)
+- Returns fully resolved domain objects with plain attribute access — no XML,
+  no URN chasing
+- Filter collections with lambdas or the `item()` expression API
+- Build SDMX artefacts locally and serialise to SDMX-ML or SDMX-JSON
+- `LocalRegistry` -- a DuckDB-backed persistent store for offline access
+  with lazy codelist loading, automatic Ref resolution, and direct SQL
+  code maps; supports shared connections and configurable language columns
+- Integrates with [Polars](https://pola.rs) for schema generation and data
+  validation
+- Fully typed — IDE completion and static analysis work throughout
+
+## Documentation
+
+See the [docs](https://gabrielgellner.gitlab.io/sdmxlib/) for the full cookbook,
+including:
+
+- Connecting to registries and choosing a resolution mode
+- Persisting artefacts in a local DuckDB store with `LocalRegistry`
+- Fetching observation data with `sl.dim()` filters and pivoting to wide format
+- Inspecting DSDs and building code maps for ETL pipelines
+- Generating Polars schemas from SDMX Data Structure Definitions
+- Browsing data catalogs and filtering dataflows
+- Working with constraints, hierarchies, and serialisation
+
+## Development
+
+```bash
+just check           # lint + typecheck
+just test            # unit tests
+just test-integration  # live endpoint tests (requires network)
+just ci              # full pre-commit gate
+```

sdmxlib-0.8.0/pyproject.toml

@@ -0,0 +1,79 @@
+[project]
+name = "sdmxlib"
+version = "0.8.0"
+description = "SDMX structural metadata library for Python"
+readme = "README.md"
+license = { text = "Apache-2.0" }
+authors = [{ name = "gabrielgellner", email = "gabrielgellner@gmail.com" }]
+requires-python = ">=3.12"
+keywords = ["sdmx", "statistics", "metadata", "datastructure"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+dependencies = [
+    "attrs>=26.1.0",
+    "duckdb>=1.0.0",
+    "httpx>=0.28.1",
+    "lxml>=6.0.2",
+    "polars>=1.39.3",
+]
+
+[project.urls]
+Homepage      = "https://gitlab.com/gabrielgellner/sdmxlib"
+Repository    = "https://gitlab.com/gabrielgellner/sdmxlib"
+Documentation = "https://gabrielgellner.gitlab.io/sdmxlib"
+Changelog     = "https://gitlab.com/gabrielgellner/sdmxlib/-/blob/main/CHANGELOG.md"
+
+[build-system]
+requires = ["uv_build>=0.11.2,<0.12.0"]
+build-backend = "uv_build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+  "unit: fast unit tests (no I/O)",
+  "docs: tests that verify documentation examples",
+  "integration: live endpoint tests (require network)",
+]
+
+[tool.coverage.run]
+source = ["src/sdmxlib"]
+omit = ["src/sdmxlib/contrib/*"]
+
+[tool.coverage.report]
+show_missing = true
+skip_covered = false
+fail_under = 80
+exclude_lines = [
+  "pragma: no cover",
+  "if TYPE_CHECKING:",
+  "raise NotImplementedError",
+  "@overload",
+]
+
+[tool.basedpyright]
+pythonVersion = "3.12"
+pythonPlatform = "Linux"
+venvPath = "."
+venv = ".venv"
+typeCheckingMode = "standard"
+reportUnusedCallResult = false
+
+[dependency-groups]
+dev = [
+  "basedpyright>=1.38.4",
+  "catppuccin[pygments]>=2.5.0",
+  "lxml-stubs>=0.5.1",
+  "mkdocs-material>=9.7.6",
+  "mkdocstrings[python]>=1.0.3",
+  "polars>=1.39.3",
+  "pytest>=9.0.2",
+  "pytest-cov>=7.1.0",
+  "pytest-recording>=0.13.4",
+  "ruff>=0.15.8",
+]

sdmxlib-0.8.0/src/sdmxlib/__init__.py

@@ -0,0 +1,155 @@
+"""sdmxlib — SDMX structural metadata library for Python."""
+
+from sdmxlib.api.filters import DimFilter, dim
+from sdmxlib.api.providers import Provider
+from sdmxlib.api.query import DataQuery
+from sdmxlib.api.registry import BearerToken, FmrRegistry, RestRegistry
+from sdmxlib.formats import parse, read, serialize, to_json, to_xml
+from sdmxlib.local import LazyCodelist, LocalRegistry
+from sdmxlib.model.convert import from_dict, to_dict
+from sdmxlib.model.validation import ReferenceIssue, ValidationIssue, validate, validate_references
+from sdmxlib.model import (
+    Agency,
+    AgencyScheme,
+    Annotation,
+    Annotations,
+    ArtefactList,
+    AttachmentLevel,
+    Attribute,
+    Categorisation,
+    Category,
+    CategoryScheme,
+    Code,
+    Codelist,
+    Concept,
+    ConceptScheme,
+    CategoryMap,
+    CategorySchemeMap,
+    CodeMap,
+    CodelistMap,
+    ComponentMap,
+    ConceptMap,
+    ConceptSchemeMap,
+    ConstraintAttachment,
+    Contact,
+    ContentConstraint,
+    CubeRegion,
+    ReferencePeriod,
+    ReleaseCalendar,
+    DataProvider,
+    DataProviderScheme,
+    DataStructure,
+    Dataset,
+    DataStructureMap,
+    Dataflow,
+    DataflowMap,
+    Dimension,
+    Hierarchy,
+    HierarchicalCode,
+    Group,
+    InternationalString,
+    ItemExpr,
+    ItemList,
+    ItemPredicate,
+    KeyValue,
+    item,
+    Maintainable,
+    MaintainableArtefact,
+    Measure,
+    MeasureDimension,
+    ProvisionAgreement,
+    DataType,
+    Facet,
+    Ref,
+    Registry,
+    Representation,
+    SdmxRef,
+    SdmxUrn,
+    StructureMessage,
+    StructureSet,
+    TimeDimension,
+    UnresolvedRef,
+)
+
+__all__ = [
+    "Agency",
+    "AgencyScheme",
+    "Annotation",
+    "Annotations",
+    "ArtefactList",
+    "AttachmentLevel",
+    "Attribute",
+    "BearerToken",
+    "Categorisation",
+    "Category",
+    "CategoryMap",
+    "CategoryScheme",
+    "CategorySchemeMap",
+    "Code",
+    "CodeMap",
+    "Codelist",
+    "CodelistMap",
+    "ComponentMap",
+    "Concept",
+    "ConceptMap",
+    "ConceptScheme",
+    "ConceptSchemeMap",
+    "ConstraintAttachment",
+    "Contact",
+    "ContentConstraint",
+    "CubeRegion",
+    "DataProvider",
+    "DataProviderScheme",
+    "DataQuery",
+    "DataStructure",
+    "DataStructureMap",
+    "DataType",
+    "Dataflow",
+    "DataflowMap",
+    "Dataset",
+    "DimFilter",
+    "Dimension",
+    "Facet",
+    "FmrRegistry",
+    "Group",
+    "HierarchicalCode",
+    "Hierarchy",
+    "InternationalString",
+    "ItemExpr",
+    "ItemList",
+    "ItemPredicate",
+    "KeyValue",
+    "LazyCodelist",
+    "LocalRegistry",
+    "Maintainable",
+    "MaintainableArtefact",
+    "Measure",
+    "MeasureDimension",
+    "Provider",
+    "ProvisionAgreement",
+    "Ref",
+    "ReferenceIssue",
+    "ReferencePeriod",
+    "Registry",
+    "ReleaseCalendar",
+    "Representation",
+    "RestRegistry",
+    "SdmxRef",
+    "SdmxUrn",
+    "StructureMessage",
+    "StructureSet",
+    "TimeDimension",
+    "UnresolvedRef",
+    "ValidationIssue",
+    "dim",
+    "from_dict",
+    "item",
+    "parse",
+    "read",
+    "serialize",
+    "to_dict",
+    "to_json",
+    "to_xml",
+    "validate",
+    "validate_references",
+]

sdmxlib-0.8.0/src/sdmxlib/api/__init__.py

@@ -0,0 +1,7 @@
+"""sdmxlib.api — HTTP client and registry."""
+
+from sdmxlib.api.providers import Provider
+from sdmxlib.api.query import GetQuery, Query
+from sdmxlib.api.registry import BearerToken, FmrRegistry, RestRegistry
+
+__all__ = ["BearerToken", "FmrRegistry", "GetQuery", "Provider", "Query", "RestRegistry"]

sdmxlib-0.8.0/src/sdmxlib/api/client.py

@@ -0,0 +1,246 @@
+"""SdmxClient — thin httpx wrapper around the SDMX REST API."""
+
+from pathlib import Path
+from typing import Self
+
+import httpx
+
+from sdmxlib.formats import parse
+from sdmxlib.model.message import StructureMessage
+
+# SDMX REST resource path segments (lowercase endpoint name)
+_RESOURCE_PATH: dict[str, str] = {
+    "DataStructure": "datastructure",
+    "Codelist": "codelist",
+    "Dataflow": "dataflow",
+    "ConceptScheme": "conceptscheme",
+    "ProvisionAgreement": "provisionagreement",
+    "CategoryScheme": "categoryscheme",
+    "Categorisation": "categorisation",
+    "ContentConstraint": "contentconstraint",
+    "AgencyScheme": "agencyscheme",
+    "DataProviderScheme": "dataproviderorganisationscheme",
+    "Hierarchy": "hierarchy",
+    "StructureSet": "structureset",
+}
+
+# Wire parameter presets — (detail, references)
+_BROWSE_PARAMS = {"detail": "allstubs", "references": "none"}
+_FETCH_PARAMS = {"detail": "full", "references": "none"}
+_RESOLVE_PARAMS = {"detail": "referencepartial", "references": "descendants"}
+_RESOLVE_FULL = {"detail": "full", "references": "descendants"}
+
+
+class SdmxClient:
+    """Thin httpx wrapper that fetches SDMX REST resources and parses them.
+
+    Handles URL construction, Accept headers, and response parsing.
+    Does not cache — caching is the Session's responsibility.
+
+    Args:
+        endpoint: Base URL of the SDMX REST endpoint.
+        api: SDMX REST API version — ``"2.1"`` (default) or ``"3.0"``.
+             Controls URL scheme and response format.
+        timeout: HTTP request timeout in seconds.
+        headers: Extra HTTP headers merged with defaults.
+    """
+
+    def __init__(
+        self,
+        endpoint: str,
+        *,
+        api: str = "2.1",
+        timeout: float = 30.0,
+        headers: dict[str, str] | None = None,
+        **httpx_kwargs: object,
+    ) -> None:
+        self._endpoint = endpoint.rstrip("/")
+        self._api = api
+        self._v3 = api.startswith(("3", "2.0"))
+        default_headers: dict[str, str]
+        if self._v3:
+            default_headers = {"Accept": "application/vnd.sdmx.structure+xml;version=3.0"}
+        else:
+            default_headers = {"Accept": "application/xml"}
+        if headers:
+            default_headers.update(headers)
+        self._client = httpx.Client(
+            headers=default_headers,
+            timeout=timeout,
+            **httpx_kwargs,  # type: ignore[arg-type]
+        )
+
+    def close(self) -> None:
+        """Close the underlying HTTP client."""
+        self._client.close()
+
+    def __enter__(self) -> Self:
+        return self
+
+    def __exit__(self, *args: object) -> None:
+        self.close()
+
+    # ── URL building ──────────────────────────────────────────────────────────
+
+    def _resource_url(
+        self,
+        resource: str,
+        agency: str,
+        id: str,  # noqa: A002
+        version: str,
+    ) -> str:
+        """Build the resource URL for the given API version."""
+        path = _RESOURCE_PATH.get(resource, resource.lower())
+        if self._v3:
+            # v2.x: /structure/{resource}/{agency}/{id}/{version}
+            # "latest" sentinel becomes "~" on the wire
+            ver = "~" if version == "latest" else version
+            return f"{self._endpoint}/structure/{path}/{agency}/{id}/{ver}"
+        # v1.5: /{resource}/{agency}/{id}/{version}
+        return f"{self._endpoint}/{path}/{agency}/{id}/{version}"
+
+    def _collection_url(self, resource: str, agency: str | None) -> str:
+        """Build the collection URL (all artefacts of a type)."""
+        path = _RESOURCE_PATH.get(resource, resource.lower())
+        agency_seg = agency or ("*" if self._v3 else "all")
+        if self._v3:
+            return f"{self._endpoint}/structure/{path}/{agency_seg}"
+        return f"{self._endpoint}/{path}/{agency_seg}"
+
+    # ── Parse version ─────────────────────────────────────────────────────────
+
+    @property
+    def _parse_version(self) -> str:
+        """Format version string passed to the parser."""
+        return "3.0" if self._v3 else "2.1"
+
+    # ── HTTP methods ──────────────────────────────────────────────────────────
+
+    def get_artefact(
+        self,
+        resource: str,
+        agency: str,
+        id: str,  # noqa: A002
+        version: str = "latest",
+        *,
+        detail: str = "full",
+        references: str = "none",
+    ) -> StructureMessage:
+        """Fetch a single maintainable artefact."""
+        url = self._resource_url(resource, agency, id, version)
+        params: dict[str, str] = {"detail": detail}
+        if references != "none":
+            params["references"] = references
+        response = self._client.get(url, params=params)
+        response.raise_for_status()
+        return parse(response.content, version=self._parse_version)
+
+    def get_all(
+        self,
+        resource: str,
+        agency: str | None = None,
+        *,
+        detail: str = "full",
+        references: str = "none",
+    ) -> StructureMessage:
+        """Fetch all artefacts of a type, optionally filtered by agency."""
+        url = self._collection_url(resource, agency)
+        params: dict[str, str] = {"detail": detail}
+        if references != "none":
+            params["references"] = references
+        response = self._client.get(url, params=params)
+        response.raise_for_status()
+        return parse(response.content, version=self._parse_version)
+
+    def _data_url(
+        self,
+        agency: str,
+        id: str,  # noqa: A002
+        version: str,
+        key: str,
+    ) -> str:
+        """Build the data URL for the given API version."""
+        if self._v3:
+            ver = "~" if version == "latest" else version
+            return f"{self._endpoint}/data/dataflow/{agency}/{id}/{ver}/{key}"
+        return f"{self._endpoint}/data/{agency},{id},{version}/{key}"
+
+    def get_data(
+        self,
+        agency: str,
+        id: str,  # noqa: A002
+        version: str = "latest",
+        *,
+        key: str = "all",
+        start_period: str | None = None,
+        end_period: str | None = None,
+    ) -> bytes:
+        """Fetch observation data as SDMX-CSV bytes.
+
+        Args:
+            agency: Dataflow maintainer agency id.
+            id: Dataflow id.
+            version: Dataflow version. Defaults to ``"latest"``.
+            key: SDMX key filter string, e.g. ``"M.AT+DE.INDPRO"``.
+                 Defaults to ``"all"`` (all observations).
+            start_period: ISO period start, e.g. ``"2020-01"``.
+            end_period: ISO period end.
+
+        Returns:
+            Raw SDMX-CSV bytes.
+        """
+        url = self._data_url(agency, id, version, key)
+        params: dict[str, str] = {}
+        if start_period is not None:
+            params["startPeriod"] = start_period
+        if end_period is not None:
+            params["endPeriod"] = end_period
+        response = self._client.get(
+            url,
+            params=params,
+            headers={"Accept": "text/csv"},
+        )
+        response.raise_for_status()
+        return response.content
+
+    def stream_data_to(
+        self,
+        agency: str,
+        id: str,  # noqa: A002
+        version: str = "latest",
+        *,
+        path: Path,
+        key: str = "all",
+        start_period: str | None = None,
+        end_period: str | None = None,
+    ) -> None:
+        """Stream observation data directly to a file.
+
+        Uses httpx chunked streaming — the response is never fully loaded into
+        memory. Suitable for multi-GB datasets.
+
+        Args:
+            agency: Dataflow maintainer agency id.
+            id: Dataflow id.
+            version: Dataflow version.
+            path: Destination file path.
+            key: SDMX key filter string.
+            start_period: ISO period start.
+            end_period: ISO period end.
+        """
+        url = self._data_url(agency, id, version, key)
+        params: dict[str, str] = {}
+        if start_period is not None:
+            params["startPeriod"] = start_period
+        if end_period is not None:
+            params["endPeriod"] = end_period
+        with self._client.stream(
+            "GET",
+            url,
+            params=params,
+            headers={"Accept": "text/csv"},
+        ) as response:
+            response.raise_for_status()
+            with path.open("wb") as f:
+                for chunk in response.iter_bytes():
+                    f.write(chunk)