statwrapper 0.1.0__tar.gz

statwrapper-0.1.0/PKG-INFO

@@ -0,0 +1,123 @@
+Metadata-Version: 2.4
+Name: statwrapper
+Version: 0.1.0
+Summary: A completely dependency-free unified wrapper around common statistical api types
+Author-email: Nordic Intel <info@nordicintel.com>
+License-Expression: Apache-2.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Requires-Dist: pytest>=8.3.5; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.26.0; extra == "dev"
+Requires-Dist: build>=1.2.2; extra == "dev"
+Requires-Dist: ruff>=0.11.0; extra == "dev"
+
+# statwrapper
+
+`statwrapper` is a dependency-free Python library that provides a unified interface for common statistical API families. It currently ships wrapper support for:
+
+- `pxweb`
+- `pxweb2`
+- `dst`
+- `eurostat`
+
+The package standardizes three operations across providers:
+
+- health checks
+- dataset discovery
+- dataset metadata resolution
+
+## Installation
+
+```bash
+pip install statwrapper
+```
+
+## Quick Start
+
+```python
+import asyncio
+
+from statwrapper import StatWrapper
+
+
+async def main() -> None:
+    wrapper = StatWrapper()
+
+    discovered = await wrapper.discover_datasets("scb", "en")
+    first = discovered[0]
+
+    metadata = await wrapper.resolve_dataset_metadata(first)
+    print(metadata.label)
+    print(metadata.dimension_ids)
+
+
+asyncio.run(main())
+```
+
+## Public API
+
+### `StatWrapper`
+
+```python
+from statwrapper import StatWrapper
+
+wrapper = StatWrapper()
+```
+
+Methods:
+
+- `await wrapper.discover_datasets(provider_code, language, task_id=None)`
+- `await wrapper.resolve_dataset_metadata(discovered, task_id=None)`
+- `wrapper.get_provider(provider_code)`
+- `wrapper.get_wrapper(provider_code, language)`
+
+### Convenience Functions
+
+```python
+from statwrapper import discover_provider_datasets, get_datasets
+```
+
+- `await discover_provider_datasets(provider_code, language)`
+- `get_datasets(provider_code, language)`
+
+## Request Layer
+
+The default request layer is a stdlib-backed async helper:
+
+```python
+from statwrapper import RateLimitedSession
+```
+
+`StatWrapper` uses `RateLimitedSession` automatically, but you can inject your own session object if it exposes:
+
+- `async def get_json(url, **kwargs)`
+- `async def get_text(url, **kwargs)`
+- `async def get_bytes(url, **kwargs)`
+
+## Provider Registry
+
+Provider metadata is loaded from `PROVIDERS.json`. During development, the package reads the repository file. In built distributions, the same data is bundled inside the package.
+
+## Development
+
+Run tests:
+
+```bash
+python -m pytest tests/unit tests/integration -q
+```
+
+Build locally:
+
+```bash
+python -m build
+```
+
+## Publishing
+
+PyPI publishing is handled by GitHub Actions through `.github/workflows/pypi_publish.yml` and the repository secret `PYPI_API_TOKEN`.

statwrapper-0.1.0/README.md

@@ -0,0 +1,104 @@
+# statwrapper
+
+`statwrapper` is a dependency-free Python library that provides a unified interface for common statistical API families. It currently ships wrapper support for:
+
+- `pxweb`
+- `pxweb2`
+- `dst`
+- `eurostat`
+
+The package standardizes three operations across providers:
+
+- health checks
+- dataset discovery
+- dataset metadata resolution
+
+## Installation
+
+```bash
+pip install statwrapper
+```
+
+## Quick Start
+
+```python
+import asyncio
+
+from statwrapper import StatWrapper
+
+
+async def main() -> None:
+    wrapper = StatWrapper()
+
+    discovered = await wrapper.discover_datasets("scb", "en")
+    first = discovered[0]
+
+    metadata = await wrapper.resolve_dataset_metadata(first)
+    print(metadata.label)
+    print(metadata.dimension_ids)
+
+
+asyncio.run(main())
+```
+
+## Public API
+
+### `StatWrapper`
+
+```python
+from statwrapper import StatWrapper
+
+wrapper = StatWrapper()
+```
+
+Methods:
+
+- `await wrapper.discover_datasets(provider_code, language, task_id=None)`
+- `await wrapper.resolve_dataset_metadata(discovered, task_id=None)`
+- `wrapper.get_provider(provider_code)`
+- `wrapper.get_wrapper(provider_code, language)`
+
+### Convenience Functions
+
+```python
+from statwrapper import discover_provider_datasets, get_datasets
+```
+
+- `await discover_provider_datasets(provider_code, language)`
+- `get_datasets(provider_code, language)`
+
+## Request Layer
+
+The default request layer is a stdlib-backed async helper:
+
+```python
+from statwrapper import RateLimitedSession
+```
+
+`StatWrapper` uses `RateLimitedSession` automatically, but you can inject your own session object if it exposes:
+
+- `async def get_json(url, **kwargs)`
+- `async def get_text(url, **kwargs)`
+- `async def get_bytes(url, **kwargs)`
+
+## Provider Registry
+
+Provider metadata is loaded from `PROVIDERS.json`. During development, the package reads the repository file. In built distributions, the same data is bundled inside the package.
+
+## Development
+
+Run tests:
+
+```bash
+python -m pytest tests/unit tests/integration -q
+```
+
+Build locally:
+
+```bash
+python -m build
+```
+
+## Publishing
+
+PyPI publishing is handled by GitHub Actions through `.github/workflows/pypi_publish.yml` and the repository secret `PYPI_API_TOKEN`.

statwrapper-0.1.0/pyproject.toml

@@ -0,0 +1,74 @@
+[project]
+name = "statwrapper"
+version = "0.1.0"
+description = "A completely dependency-free unified wrapper around common statistical api types"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "Apache-2.0"
+authors = [
+  { name = "Nordic Intel", email = "info@nordicintel.com" },
+]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+]
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.3.5",
+  "pytest-asyncio>=0.26.0",
+  "build>=1.2.2",
+  "ruff>=0.11.0",
+]
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.setuptools.packages.find]
+include = ["statwrapper*"]
+
+[tool.setuptools.package-data]
+statwrapper = ["providers.json"]
+
+[tool.ruff]
+target-version = "py310"
+
+[tool.ruff.lint]
+select = [
+    "E",  # pycodestyle errors
+    "W",  # pycodestyle warnings
+    "F",  # pyflakes
+    "I",  # isort
+    "B",  # flake8-bugbear
+    "C4",  # flake8-comprehensions
+    "UP",  # pyupgrade
+    "ARG001", # unused arguments in functions
+    "T201",   # print statements are not allowed
+]
+ignore = [
+    "E501",  # line too long, handled by black
+    "B008",  # do not perform function calls in argument defaults
+    "W191",  # indentation contains tabs
+    "B904",  # Allow raising exceptions without from e, for HTTPException
+]
+
+[tool.ruff.lint.pyupgrade]
+# Preserve types, even if a file imports `from __future__ import annotations`.
+keep-runtime-typing = true
+
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "session"
+asyncio_default_test_loop_scope = "session"
+
+# live wrapper tests are excluded from the default run; invoke them explicitly:
+#   pytest tests/live/ -v
+testpaths = ["tests/unit", "tests/integration"]

statwrapper-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

statwrapper-0.1.0/statwrapper/__init__.py

@@ -0,0 +1,31 @@
+from .base_api_client import APIWrapper
+from .http import RateLimitedSession
+from .models import DiscoveredDataset, Provider, ResolvedDatasetMetadata
+from .provider_registry import (
+    create_wrapper,
+    get_provider,
+    get_wrapper_class,
+    load_providers,
+)
+from .statwrapper import (
+    StatWrapper,
+    discover_provider_datasets,
+    get_datasets,
+    resolve_dataset_metadata,
+)
+
+__all__ = [
+    "APIWrapper",
+    "DiscoveredDataset",
+    "Provider",
+    "RateLimitedSession",
+    "ResolvedDatasetMetadata",
+    "StatWrapper",
+    "create_wrapper",
+    "discover_provider_datasets",
+    "get_datasets",
+    "get_provider",
+    "get_wrapper_class",
+    "load_providers",
+    "resolve_dataset_metadata",
+]

statwrapper-0.1.0/statwrapper/api_clients/__init__.py

@@ -0,0 +1,11 @@
+from .dst_client import DstClient
+from .eurostat_client import EurostatClient
+from .pxweb2_client import PxWeb2Client
+from .pxweb_client import PxWebClient
+
+__all__ = [
+    "DstClient",
+    "EurostatClient",
+    "PxWebClient",
+    "PxWeb2Client",
+]

statwrapper-0.1.0/statwrapper/api_clients/dst_client.py

@@ -0,0 +1,249 @@
+from __future__ import annotations
+
+import uuid
+from typing import Any
+
+from ..base_api_client import APIWrapper
+from ..models import DiscoveredDataset, Provider, ResolvedDatasetMetadata
+from ..utils import (
+    detect_role,
+    determine_time_unit,
+    normalize_contact,
+    normalize_note,
+    parse_dt,
+)
+
+
+def _collect_tables(subjects_tree: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
+    tables_by_id: dict[str, dict[str, Any]] = {}
+
+    def visit(node: dict[str, Any]) -> None:
+        for table in node.get("tables") or []:
+            if not isinstance(table, dict):
+                continue
+            table_id = str(table.get("id") or "").strip()
+            if not table_id:
+                continue
+            existing = tables_by_id.setdefault(table_id, {"id": table_id})
+            for key, value in table.items():
+                if existing.get(key) in (None, "", []) and value not in (None, "", []):
+                    existing[key] = value
+        for child in node.get("subjects") or []:
+            if isinstance(child, dict):
+                visit(child)
+
+    for root in subjects_tree:
+        if isinstance(root, dict):
+            visit(root)
+    return tables_by_id
+
+
+def _parse_paths(subjects_tree: list[dict[str, Any]]) -> dict[str, list[list[dict[str, str]]]]:
+    code_to_paths: dict[str, list[list[dict[str, str]]]] = {}
+
+    def visit(node: dict[str, Any], path: list[dict[str, str]]) -> None:
+        node_id = str(node.get("id") or "").strip()
+        node_label = str(node.get("description") or node.get("text") or node_id).strip()
+        current_path = path + ([{"id": node_id, "label": node_label}] if node_id else [])
+        for table in node.get("tables") or []:
+            if not isinstance(table, dict):
+                continue
+            table_id = str(table.get("id") or "").strip()
+            if not table_id:
+                continue
+            code_to_paths.setdefault(table_id, []).append(
+                current_path + [{"id": table_id, "label": str(table.get("text") or table_id).strip()}]
+            )
+        for child in node.get("subjects") or []:
+            if isinstance(child, dict):
+                visit(child, current_path)
+
+    for root in subjects_tree:
+        if isinstance(root, dict):
+            visit(root, [])
+    return code_to_paths
+
+
+class DstClient(APIWrapper):
+    def __init__(self, provider: Provider, **kwargs: Any) -> None:
+        super().__init__(
+            provider_code=provider.provider_code,
+            label=provider.label,
+            language=kwargs.pop("language"),
+            json_request_handler=kwargs.pop("json_request_handler"),
+            text_request_handler=kwargs.pop("text_request_handler", None),
+            bytes_request_handler=kwargs.pop("bytes_request_handler", None),
+            logger=kwargs.pop("logger", None),
+        )
+        self.provider = provider
+        self.api_type = provider.api_type
+        self.base_api_url = provider.base_api_url or ""
+        self.base_web_url = provider.base_web_url
+
+    def _subjects_url(self) -> str:
+        return f"{self.base_api_url.rstrip('/')}/subjects"
+
+    def _tableinfo_url(self, dataset_code: str) -> str:
+        return f"{self.base_api_url.rstrip('/')}/tableinfo/{dataset_code}"
+
+    def _data_url(self, dataset_code: str) -> str:
+        return f"{self.base_api_url.rstrip('/')}/data/{dataset_code}/JSONSTAT"
+
+    async def health_check(self, dataset_code: str | None = None) -> bool:
+        url = self._subjects_url() if dataset_code is None else self._tableinfo_url(dataset_code)
+        payload = await self._get_json(url, params={"lang": self.language})
+        return payload is not None
+
+    async def discover_datasets(
+        self,
+        task_id: uuid.UUID,
+        **_: Any,
+    ) -> list[DiscoveredDataset]:
+        payload = await self._get_json(
+            self._subjects_url(),
+            params={
+                "lang": self.language,
+                "recursive": "true",
+                "omitSubjectsWithoutTables": "true",
+                "includeTables": "true",
+                "format": "JSON",
+            },
+        )
+        if not isinstance(payload, list):
+            return []
+        tables_by_id = _collect_tables(payload)
+        paths_by_id = _parse_paths(payload)
+        output: list[DiscoveredDataset] = []
+        for dataset_code, table in sorted(tables_by_id.items()):
+            paths = paths_by_id.get(dataset_code, [])
+            primary_path = paths[0] if paths else []
+            subject = primary_path[0] if primary_path else None
+            first_period = str(table.get("firstPeriod") or "").strip() or None
+            last_period = str(table.get("latestPeriod") or table.get("lastPeriod") or "").strip() or None
+            updated = parse_dt(table.get("updated")) or parse_dt("1970-01-01T00:00:00+00:00")
+            if updated is None:
+                continue
+            output.append(
+                DiscoveredDataset(
+                    task_id=task_id,
+                    provider_code=self.provider_code,
+                    dataset_code=dataset_code,
+                    language=self.language,
+                    updated=updated,
+                    label=str(table.get("text") or dataset_code).strip(),
+                    description=str(table.get("description")).strip() if table.get("description") is not None else None,
+                    source=f"Statistics Denmark - statbank.dk/{dataset_code}",
+                    subject_code=subject["id"] if subject else None,
+                    subject_label=subject["label"] if subject else None,
+                    time_unit=determine_time_unit(first_period, last_period) if first_period and last_period else None,
+                    first_period=first_period,
+                    last_period=last_period,
+                    discontinued=(not bool(table.get("active")) if isinstance(table.get("active"), bool) else False),
+                    metadata_url=f"{self._tableinfo_url(dataset_code)}?lang={self.language}",
+                    data_url=f"{self._data_url(dataset_code)}?lang={self.language}",
+                    web_url=f"{self.base_web_url.rstrip('/')}/{dataset_code}" if self.base_web_url else None,
+                    paths=paths or None,
+                    extension={"unit": table.get("unit"), "variable_names": table.get("variables")},
+                )
+            )
+        return output
+
+    async def resolve_dataset_metadata(
+        self,
+        discovered: DiscoveredDataset,
+        task_id: uuid.UUID | None = None,
+        **_: Any,
+    ) -> ResolvedDatasetMetadata:
+        payload = await self._get_json(
+            self._tableinfo_url(discovered.dataset_code),
+            params={"lang": self.language},
+        )
+        dimensions: dict[str, dict[str, Any]] = {}
+        dimension_ids: list[str] = []
+        required_dimensions: dict[str, bool | None] = {}
+        role_map: dict[str, list[str]] = {}
+        extension = dict(discovered.extension)
+        if isinstance(payload, dict):
+            for position, variable in enumerate(payload.get("variables") or []):
+                if not isinstance(variable, dict):
+                    continue
+                dimension_id = str(variable.get("id") or "").strip()
+                if not dimension_id:
+                    continue
+                values = variable.get("values") if isinstance(variable.get("values"), list) else []
+                index: dict[str, int] = {}
+                labels: dict[str, str] = {}
+                for ordinal, item in enumerate(values):
+                    if not isinstance(item, dict):
+                        continue
+                    value_id = str(item.get("id") or "").strip()
+                    if not value_id:
+                        continue
+                    index[value_id] = ordinal
+                    labels[value_id] = str(item.get("text") or value_id).strip()
+                if not index:
+                    continue
+                label = str(variable.get("text") or dimension_id).strip()
+                dimensions[dimension_id] = {
+                    "label": label,
+                    "category": {"index": index, "label": labels},
+                }
+                dimension_ids.append(dimension_id)
+                elimination = variable.get("elimination")
+                required_dimensions[dimension_id] = None if elimination is None else not bool(elimination)
+                role = "geo" if variable.get("map") else detect_role(dimension_id, label, bool(variable.get("time")))
+                if role:
+                    role_map.setdefault(role, []).append(dimension_id)
+                extension.setdefault("dimension_extensions", {})[dimension_id] = {
+                    "extension": {"position": position}
+                }
+        label = (
+            str(payload.get("text")).strip()
+            if isinstance(payload, dict) and payload.get("text") is not None
+            else (discovered.label or discovered.dataset_code)
+        )
+        description = (
+            str(payload.get("description")).strip()
+            if isinstance(payload, dict) and payload.get("description") is not None
+            else discovered.description
+        )
+        updated = (
+            parse_dt(payload.get("updated"))
+            if isinstance(payload, dict)
+            else None
+        ) or discovered.updated
+        return ResolvedDatasetMetadata(
+            task_id=task_id or discovered.task_id,
+            provider_code=discovered.provider_code,
+            dataset_code=discovered.dataset_code,
+            language=discovered.language,
+            updated=updated,
+            label=label,
+            time_unit=discovered.time_unit or "Other",
+            first_period=discovered.first_period or "",
+            last_period=discovered.last_period or discovered.first_period or "",
+            paths=discovered.paths or [],
+            role=role_map,
+            metadata_url=discovered.metadata_url or f"{self._tableinfo_url(discovered.dataset_code)}?lang={self.language}",
+            data_url=discovered.data_url or f"{self._data_url(discovered.dataset_code)}?lang={self.language}",
+            dimension=dimensions,
+            required_dimensions=required_dimensions,
+            note=(normalize_note(payload.get("footnote")) if isinstance(payload, dict) else None) or discovered.note,
+            source=discovered.source,
+            description=description,
+            discontinued=not bool(payload.get("active")) if isinstance(payload, dict) and isinstance(payload.get("active"), bool) else discovered.discontinued,
+            subject_code=discovered.subject_code,
+            subject_label=discovered.subject_label,
+            web_url=discovered.web_url,
+            doc_url=(
+                str(payload.get("documentation", {}).get("url")).strip()
+                if isinstance(payload, dict)
+                and isinstance(payload.get("documentation"), dict)
+                and payload["documentation"].get("url") is not None
+                else discovered.doc_url
+            ),
+            official_statistics=discovered.official_statistics,
+            contact=(normalize_contact(payload.get("contacts")) if isinstance(payload, dict) else None) or discovered.contact,
+            dimension_ids=dimension_ids or None,
+            extension=extension,
+        )