statcatlib 0.8.0__tar.gz

statcatlib-0.8.0/LICENSE

@@ -0,0 +1,180 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship made available under
+      the License, as indicated by a copyright notice that is included in
+      or attached to the work (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean, as submitted to the Licensor for inclusion
+      in the Work by the copyright owner or by an individual or Legal Entity
+      authorized to submit on behalf of the copyright owner. For the purposes
+      of this definition, "submitted" means any form of electronic, verbal,
+      or written communication sent to the Licensor or its representatives,
+      including but not limited to communication on electronic mailing lists,
+      source code control systems, and issue tracking systems that are managed
+      by, or on behalf of, the Licensor for the purpose of discussing and
+      improving the Work, but excluding communication that is conspicuously
+      marked or designated in writing by the copyright owner as "Not a
+      Contribution."
+
+      "Contributor" shall mean Licensor and any Legal Entity on behalf of
+      whom a Contribution has been received by the Licensor and included
+      within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by the combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a cross-claim
+      or counterclaim in a lawsuit) alleging that the Work or any other
+      Contribution embodied in the Work constitutes direct or contributory
+      patent infringement, then any patent licenses granted to You under
+      this License for that Work shall terminate as of the date such
+      litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or Derivative
+          Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, You must include a readable copy of the
+          attribution notices contained within such NOTICE file, in
+          at least one of the following places: within a NOTICE text
+          file distributed as part of the Derivative Works; within
+          the Source form or documentation, if provided along with the
+          Derivative Works; or, within a display generated by the
+          Derivative Works, if and wherever such third-party notices
+          normally appear. The contents of the NOTICE file are for
+          informational purposes only and do not modify the License.
+          You may add Your own attribution notices within Derivative
+          Works that You distribute, alongside or in addition to the
+          NOTICE text from the Work, provided that such additional
+          attribution notices cannot be construed as modifying the
+          License.
+
+      You may add Your own license statement for Your modifications and
+      may provide additional grant of rights to use, copy, modify, merge,
+      publish, distribute, sublicense, and/or sell copies of the
+      Contribution, either on its own behalf or on behalf of the Licensor,
+      and only if You accept these terms.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any conditions of TITLE,
+      NONINFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR
+      PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or exemplary damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (even if such Contributor has been advised of the possibility
+      of such damages).
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may offer only
+      conditions consistent with this License, and charge a fee for,
+      such acceptance.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2026 gabrielgellner
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

statcatlib-0.8.0/PKG-INFO

@@ -0,0 +1,429 @@
+Metadata-Version: 2.4
+Name: statcatlib
+Version: 0.8.0
+Summary: Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX
+Keywords: sdmx,dcat,open-data,catalog,duckdb,statistical-data,ckan
+Author: gabrielgellner
+Author-email: gabrielgellner <gabrielgellner@gmail.com>
+License-Expression: Apache-2.0
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Dist: attrs>=26.1.0
+Requires-Dist: duckdb>=1.1,<2
+Requires-Dist: httpx>=0.28
+Requires-Dist: sdmxlib>=0.67
+Requires-Dist: lancedb>=0.39 ; extra == 'search'
+Requires-Dist: sentence-transformers>=3.0 ; extra == 'search'
+Requires-Dist: typer>=0.15 ; extra == 'search'
+Requires-Dist: rich>=15.0 ; extra == 'search'
+Requires-Python: >=3.13
+Project-URL: Homepage, https://github.com/StatCat-Suite/statcatlib
+Project-URL: Repository, https://github.com/StatCat-Suite/statcatlib
+Project-URL: Documentation, https://statcat-suite.github.io/statcatlib
+Project-URL: Issue Tracker, https://github.com/StatCat-Suite/statcatlib/issues
+Project-URL: Changelog, https://github.com/StatCat-Suite/statcatlib/blob/main/CHANGELOG.md
+Provides-Extra: search
+Description-Content-Type: text/markdown
+
+# statcatlib
+
+> **Note:** This library has been written extensively with AI assistance
+> (Claude Code). Users who are not comfortable with AI-generated code should
+> take that into account before adopting it.
+
+**statcatlib** is a Python library for building, managing, and querying
+statistical metadata catalogs. Formerly published as `statcatlib`.
+
+Inspired by DCAT and the broader landscape of metadata standards (SDMX, DDI),
+statcatlib provides a general-purpose engine for catalog storage, discovery, and
+retrieval — designed to be embedded in ETL pipelines, data platforms, and
+analytical tooling rather than used as a standalone application. It pairs
+naturally with domain-specific libraries like
+[sdmxlib](https://github.com/StatCat-Suite/sdmxlib) for standards-aware
+workflows.
+
+```python
+import statcatlib as sc
+```
+
+## What it does
+
+statcatlib materializes remote catalog and structural metadata into a local
+DuckDB database and exposes a fluent Python API for discovery queries —
+filtering by publisher, theme, dimension, code, free text, or provenance lineage.
+
+**The catalog is a materialized graph.** Rather than federated queries across
+separate REST endpoints, ingest pulls the full structural model into one database.
+SQL JOINs are the graph traversal. No network round-trips at query time.
+
+## Installation
+
+Requires Python 3.13+. Managed with [uv](https://docs.astral.sh/uv/).
+
+```bash
+uv add statcatlib
+# or: pip install statcatlib
+```
+
+Published on PyPI as `statcatlib` (formerly `pinaxlib`, imported as `pinax`).
+
+statcatlib is pre-1.0: until 1.0, a minor release may break the API.
+
+## Quick start
+
+```python
+import statcatlib as sc
+import statcatlib.query as q
+import sdmxlib as sl
+
+# Creates my_catalog/bundle.duckdb and my_catalog/parquet/
+with sc.Catalog("my_catalog").open() as store:
+    with sl.RestRegistry(sl.Provider.ESTAT) as reg:
+        reg.get(sl.Dataflow.urn_for(agency="ESTAT", id="NAMA_10_GDP"), resolve=True)
+        sc.ingest_sdmx(store, reg.registry)
+
+    # Catalog query — no network needed
+    results = (
+        store.query(sc.AggregateDataset)
+        .filter(q.has_code("geo", "DE"))
+        .search("GDP")
+        .with_facets("themes", "frequency")
+        .execute()
+    )
+    print(results.total, "datasets found")
+    print(results.facets["themes"])
+```
+
+## Five dataset kinds
+
+statcatlib uses a discriminated union of dataset types aligned with DCAT-AP and
+StatDCAT-AP:
+
+```python
+# Generic datasets (CKAN, open portals)
+sc.OpenDataset(identifier="co2-2024", title=i18n("CO2 Emissions 2024"), ...)
+
+# Statistical tables with SDMX structure (dimensions, codelists)
+sc.AggregateDataset(identifier="ESTAT:UNE_RT_M(1.0)", ..., sdmx_dataflow_urn="urn:...")
+
+# Survey microdata with variable-level metadata
+sc.MicrodataDataset(identifier="lfs-2023", ..., variables=[...])
+
+# Spatial datasets with bounding box and CRS
+sc.GeospatialDataset(identifier="boundaries-2024", ..., crs="EPSG:4326")
+
+# Articles, reports, and analytical publications
+sc.PublicationDataset(identifier="pub-71-607-x", ..., doi="10.25318/...", authors=[...])
+```
+
+All five share the same store and query API. Use `sc.BaseDataset` as the bound
+for generic code; `sc.Dataset` is a type alias for the full union.
+
+## Source connectors
+
+### SDMX (Eurostat, OECD, BIS, ABS, ...)
+
+```python
+import sdmxlib as sl
+import statcatlib as sc
+
+with sc.Catalog("my_catalog").open() as store:
+    with sl.RestRegistry(sl.Provider.ESTAT) as reg:
+        for df_id in ["NAMA_10_GDP", "UNE_RT_M", "PRIC_HPI_IDX"]:
+            reg.get(sl.Dataflow.urn_for(agency="ESTAT", id=df_id), resolve=True)
+        sc.ingest_sdmx(store, reg.registry)
+
+        # Stream observation data to Parquet, attach as Distribution
+        sc.ingest_data(store, reg, "ESTAT:NAMA_10_GDP(latest)", measure_dim="na_item")
+```
+
+### CKAN (open.canada.ca, data.gov, ...)
+
+```python
+from statcatlib.sources.ckan import CkanClient
+
+with sc.Catalog("open_canada").open() as store:
+    with CkanClient("https://open.canada.ca/data") as client:
+        sc.ingest_ckan(store, client, organization="statcan", rows=500)
+```
+
+## Scope-based graph traversal
+
+statcatlib exposes a lazy, scope-based API for navigating the catalog graph.
+Navigation builds scope objects without executing SQL; only terminal methods
+(`.collect()`, `.count()`) hit the database.
+
+```python
+import statcatlib as sc
+
+store = sc.Catalog("my_catalog").open()
+
+# Navigate themes — no SQL until .collect()
+concepts = store.themes["statcan"].collect()          # ItemList[Concept]
+concept = store.themes["statcan"]["13"].collect()     # Concept
+
+# Cross-entity navigation — .datasets returns a lazy QueryBuilder
+datasets = store.themes["statcan"]["13"].datasets.collect()
+
+# Enrich with sub-traversal expressions (like Polars' pl.col())
+store.themes["statcan"].enrich(
+    n=sc.each("datasets").count(),
+    has_data=sc.each("datasets").exists(),
+).collect()
+
+# Codelist navigation — sc.urn builds URN strings for you
+codes = store.codelist(sc.urn.codelist("SDMX", "CL_GEO")).collect()    # ItemList[Code]
+code = store.codelist(sc.urn.codelist("SDMX", "CL_GEO"))["ON"].collect()  # Code
+
+# CodelistsScope — parallel to ThemesScope, supports enrich
+# Enriched output includes labels resolved via sdmx.localized_text
+store.codelists.lang("en").enrich(n=sc.each("datasets").count()).collect()
+# → [{"urn": "...", "label": "Geography", "n": 5}, ...]
+
+# Filter codelists by label text (SQL-level, case-insensitive)
+store.codelists.filter(text_contains="geo").enrich(n=sc.each("datasets").count()).collect()
+
+store.codelist(urn).label("en")   # quick name lookup
+
+# Enriched per-code output includes code name labels
+store.codelist(urn).lang("en").enrich(n=sc.each("datasets").count()).collect()
+# → [{"code_id": "ON", "label": "Ontario", "n": 5}, ...]
+
+# Batch label resolution — single SQL query for many codes
+store.codelist(geo_urn).batch_labels(["CA", "US", "DE"])
+# → {"CA": "Canada", "US": "United States", "DE": "Germany"}
+
+# Across multiple codelists at once
+store.codelists.batch_labels([(geo_urn, "CA"), (freq_urn, "A")], lang="en")
+# → {(geo_urn, "CA"): "Canada", (freq_urn, "A"): "Annual"}
+
+# Dimension traversal
+dims = store.dimensions(ds).collect()              # ItemList[DimensionInfo]
+codelist = store.dimensions(ds)["GEO"].codelist    # CodelistScope (lazy)
+```
+
+Scope classes: `ConceptSchemesScope`, `ConceptSchemeScope`, `ThemesScope`,
+`SchemeScope`, `ConceptScope`, `CodelistsScope`, `CodelistScope`, `CodeScope`,
+`DimensionsScope`, `DimensionScope`.
+
+Expression types: `sc.each("edge")` creates a context-free sub-traversal
+expression. Reusable across `.enrich()`, `.filter()`, and `.sort_by()`.
+
+## Query API
+
+### Structured filters
+
+```python
+import statcatlib.query as q
+
+# Field filters (keyword arguments)
+store.query(sc.AggregateDataset).filter(publisher="ESTAT", status="current").all()
+
+# Composable filter objects
+store.query(sc.AggregateDataset).filter(
+    q.has_code("geo", "DE"),          # datasets with GEO=DE in their codelist
+    q.has_dimensions(["geo", "freq"]), # datasets with both GEO and FREQ dimensions
+).all()
+
+# Hierarchy descent — any geography *under* Canada, strict (CA itself excluded).
+# The applicable Hierarchy is resolved per dataflow; add include_self=True to
+# match the root as well.
+store.query(sc.AggregateDataset).filter(q.within("REF_AREA", "CA")).all()
+
+# Distribution and service filters
+store.query(sc.OpenDataset).filter(
+    q.distribution(format="CSV"),
+    q.has_service(endpoint_url="https://..."),
+).all()
+```
+
+### Cross-entity queries
+
+```python
+# Agents that publish datasets with code GEO=CA
+store.query(sc.Agent).filter(
+    q.publishes(q.has_code("geo", "CA"), kind="aggregate")
+).all()
+
+# Data services serving aggregate datasets
+store.query(sc.DataService).filter(
+    q.serves(kind="aggregate")
+).all()
+```
+
+### MAP column filters
+
+```python
+# Spatial coverage filter
+store.query(sc.BaseDataset).filter(q.has_spatial("Canada")).all()
+
+# Keyword filter
+store.query(sc.BaseDataset).filter(q.has_keyword("employment")).all()
+
+# Title/description search (case-insensitive ILIKE)
+store.query(sc.BaseDataset).filter(q.title_contains("GDP")).all()
+store.query(sc.BaseDataset).filter(q.description_contains("quarterly")).all()
+
+# Sort by multilingual title
+store.query(sc.BaseDataset).sort_by("title", lang="en").all()
+store.query(sc.BaseDataset).sort_by("title", lang="en", desc=True).all()
+```
+
+### Selective relationship loading
+
+By default, querying a list of datasets loads all relationships (~17 queries).
+Use `.include()` to declare exactly which relationships to batch-load — the rest
+are set to an `UNLOADED` sentinel:
+
+```python
+# Only load publisher and themes — 3 queries instead of ~17
+results = (
+    store.query(sc.BaseDataset)
+    .filter(status="published")
+    .include("publisher", "themes", "keywords")
+    .sort_by("issued", desc=True)
+    .limit(20)
+    .all()
+)
+
+# Explicit full hydration — useful to make the cost visible at the call site
+results = store.query(sc.BaseDataset).filter(...).full().all()
+
+# get() always loads everything — no include() needed
+ds = store.get(sc.AggregateDataset, "ESTAT:NAMA_10_GDP(1.0)")
+```
+
+Unloaded fields raise `sc.NotLoadedError` on access. Use `sc.is_unloaded(value)`
+to check before accessing:
+
+```python
+ds = store.query(sc.BaseDataset).include("publisher").first()
+ds.publisher.name    # OK
+ds.themes[0]         # raises NotLoadedError: 'themes' was not loaded
+
+if not sc.is_unloaded(ds.themes):
+    print(ds.themes)
+```
+
+Valid relationship names: `publisher`, `contact_point`, `frequency`, `licence`,
+`themes`, `subject`, `dataset_type`, `keywords`, `spatial_coverage`,
+`distributions`, `conforms_to`, `quality_annotations`, `provenance`,
+`dimension_names`, `variables`, `feature_types`, `authors`.
+
+### Lightweight projections
+
+When you need just a few columns (e.g. autocomplete), projection modifiers
+bypass full object reconstruction — a single SQL query:
+
+```python
+# Row projection — .select() + .rows() returns Row objects (dict subclass)
+rows = (
+    store.query(sc.BaseDataset)
+    .filter(q.title_contains("GDP"))
+    .sort_by("title", lang="en")
+    .limit(3)
+    .select("identifier", "title", lang="en")
+    .rows()
+)
+# → [Row({"identifier": "GDP", "title": "GDP Growth"}), ...]
+rows[0].identifier   # attribute-style access
+rows[0]["title"]     # dict-style access — both work
+
+# Flat value projection — .scalars() + .values() returns bare values
+ids = store.query(sc.BaseDataset).filter(status="current").scalars("identifier").values()
+# → ["EXR", "M1", "UNEMP", ...]
+
+# Existence check — no object reconstruction
+if store.query(sc.AggregateDataset).filter(q.has_code("geo", "DE")).exists():
+    ...
+```
+
+### Full-text search and facets
+
+```python
+# BM25 search across titles, descriptions, keywords, themes, and dimensions
+results = store.search("unemployment", limit=20)
+
+# Combined search and filter
+results = (
+    store.query(sc.AggregateDataset)
+    .filter(publisher="ESTAT")
+    .search("labour force")
+    .with_facets("themes", "frequency")
+    .execute()
+)
+print(results.facets["themes"])   # {"Labour": 18, "Economy": 6, ...}
+
+# Aggregation counts
+counts = store.facets("publisher", "themes", "frequency", "status")
+```
+
+### Lineage and provenance (PROV-O)
+
+```python
+# Record derivation relationships between datasets
+store.add_lineage(
+    "lfs-microdata",
+    "14100287",
+    "aggregated_from",
+    activity_type="aggregation",
+    activity_label="LFS monthly tabulation",
+    confidence="asserted",
+)
+
+# Transitive upstream/downstream traversal — returns QueryBuilder for chaining
+ancestors = store.dataset("14100287").upstream(depth=5).collect()
+dependents = store.dataset("CL_GEO").downstream(relationship="uses_classification").collect()
+
+# Chain additional filters after traversal
+current = store.dataset("14100287").upstream().filter(status="current").collect()
+
+# Inspect lineage records
+rows = store.dataset("14100287").lineage_records(role="target", relationship="aggregated_from")
+```
+
+## Architecture
+
+Three layers share one DuckDB database:
+
+```
+Discovery layer  — Catalog            (dataset, agent, concept_scheme, concept, distribution, lineage)
+Structural layer — sdmxlib tables     (dataflows, dsd_components, codes, codelists)
+Observation layer — Polars / Parquet  (actual time-series data)
+```
+
+statcatlib owns the discovery layer. sdmxlib owns the structural layer. Both write
+to the same DuckDB connection — queries JOIN freely across both. Parquet files live
+alongside the database and are referenced via DCAT `Distribution` records.
+
+## Development
+
+Run once per clone — installs a pre-push hook that runs the full gate, since
+hosted CI only builds the docs site:
+
+```bash
+just hooks-install
+```
+
+```bash
+just ci                # full gate: index guard + lint + format + docstrings + Any ratchet + both type checkers + unit tests + strict docs build
+just test              # unit tests
+just test-integration  # integration tests (no network)
+just test-live         # live tests against real SDMX endpoints
+just lint              # ruff check
+just typecheck         # basedpyright
+just check-indexes     # only public PyPI is named anywhere in the repo
+just docs              # local docs server
+```
+
+`uv.lock` is not committed: statcatlib is a library, so consumers resolve against
+`pyproject.toml`, and a lockfile would record whichever index resolved it.
+`just check-indexes` fails the build if a non-public package index is named in
+any tracked file.