dclimate-tabular-py 0.1.0__tar.gz

dclimate_tabular_py-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.coverage
+htmlcov/

dclimate_tabular_py-0.1.0/PKG-INFO

@@ -0,0 +1,126 @@
+Metadata-Version: 2.5
+Name: dclimate-tabular-py
+Version: 0.1.0
+Summary: Content-addressed tabular/telemetry manifest for climate data on IPFS (dclimate-tabular/0)
+Project-URL: Homepage, https://github.com/dClimate/tabular-py
+Project-URL: Repository, https://github.com/dClimate/tabular-py
+License: MIT
+Requires-Python: >=3.12
+Requires-Dist: blake3>=0.4.1
+Requires-Dist: dag-cbor>=0.3.3
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: multiformats[full]>=0.3.1.post4
+Requires-Dist: py-hamt>=3.6.0
+Requires-Dist: pyarrow>=18.0.0
+Description-Content-Type: text/markdown
+
+# tabular-py
+
+Python reader for `dclimate-tabular/0` — content-addressed station/telemetry data on IPFS.
+
+The Python counterpart to [`tabular-js`](https://github.com/dClimate/tabular-js), reading the
+same format from the same CIDs. Built on [`py-hamt`](https://github.com/dClimate/py-hamt),
+which supplies the HAMT and the content-addressed store the same way
+`@dclimate/ipld-index` does for JS.
+
+```
+tabular-js  ->  @dclimate/ipld-index      (HAMT, CAS, range reads)
+tabular-py  ->  py-hamt                   (same, already existed)
+```
+
+## Status
+
+**Reader only.** Publishing, compaction, and rollup stay in `tabular-js` — the ETL that
+writes these datasets is JS, and a second writer would be a second thing to keep
+byte-identical for no current gain. Everything needed to *read* a dataset published by
+`tabular-js` is here.
+
+## Install
+
+```bash
+uv pip install dclimate-tabular-py
+```
+
+## Usage
+
+```python
+import asyncio
+from tabular_py import GatewayRangeSource, StationDataset
+
+async def main():
+    source = GatewayRangeSource("https://ipfs-gateway.dclimate.net")
+    ds = await StationDataset.open(source, "bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a")
+
+    # nearest station to a point, then a year of readings
+    near = await ds.nearest(34.05, -118.24, max_km=100)
+    rows = await near.time_range("2024-01-01", "2024-12-31").elements("PRCP").rows()
+    for row in rows[:5]:
+        print(row.station_id, row.ts, row.values)
+
+asyncio.run(main())
+```
+
+The chainable selection API mirrors `tabular-js`:
+
+```python
+ds.select("USW00023174")           # explicit station ids
+ds.circle(34.05, -118.24, 50)      # within 50 km
+ds.rectangle(33.0, -119.0, 35.0, -117.0)
+ds.polygon([[(lon, lat), ...]])
+await ds.nearest(lat, lon)         # async: reads the geo index
+ds.time_range(start, end)
+ds.elements("PRCP", "TMAX")
+ds.where(gt("TMAX", 300))          # pushed down to fragment statistics
+```
+
+Selections are immutable — each call returns a new `StationDataset`, so a base dataset can
+be reused across queries.
+
+Terminal operations:
+
+| Call | Returns |
+|---|---|
+| `await ds.rows()` | `list[ResultRow]` |
+| `await ds.to_records()` | `list[dict]` — `station_id`, `time`, `values` |
+| `await ds.to_records("TMAX")` | `list[dict]` — `station_id`, `time`, `value` |
+| `await ds.to_arrow()` | `pyarrow.Table` |
+| `await ds.plan()` | `QueryPlan` — what *would* be fetched, without fetching |
+| `await ds.list_stations()` | `list[StationInfo]` |
+
+## How reads stay small
+
+A query never scans the dataset. Three things prune before any Parquet byte is fetched:
+
+1. **The station index** (a HAMT keyed by station id) resolves named stations directly.
+2. **The geo projection** answers region queries by reading one or two shard blocks
+   instead of walking all 132k stations.
+3. **Fragment statistics in the manifest** — per-column min/max and null counts — let a
+   predicate skip whole fragments unread.
+
+What survives is fetched with HTTP range requests against the exact column-chunk byte
+ranges the manifest records, so a query for one column of one year moves kilobytes.
+
+### One deviation from `tabular-js`, and why
+
+`tabular-js` synthesizes Parquet `FileMetaData` client-side from the manifest and reads a
+fragment with **zero** footer fetches. PyArrow exposes no public `FileMetaData`
+constructor, so that trick does not transfer.
+
+Instead this reader fetches the footer by its manifest-recorded
+`footer_offset`/`footer_length` in a single ranged GET, verifies it against the
+manifest's `footer_digest`, and hands the parsed metadata to PyArrow. Cost is one extra
+range request per fragment — and it buys a corruption check `tabular-js` does not
+perform. Footers are cached per fragment CID, so a repeated query pays it once.
+
+## Development
+
+```bash
+uv sync
+uv run pytest                     # unit tests, no network
+uv run pytest -m network          # conformance against the live gateway
+uv run ruff check . && uv run mypy tabular_py
+```
+
+`tests/test_conformance.py` reads a real published GHCNd dataset
+(`bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a`, 132,437 stations,
+1.15 B rows) and asserts this implementation agrees with `tabular-js` on decoded values.

dclimate_tabular_py-0.1.0/README.md

@@ -0,0 +1,110 @@
+# tabular-py
+
+Python reader for `dclimate-tabular/0` — content-addressed station/telemetry data on IPFS.
+
+The Python counterpart to [`tabular-js`](https://github.com/dClimate/tabular-js), reading the
+same format from the same CIDs. Built on [`py-hamt`](https://github.com/dClimate/py-hamt),
+which supplies the HAMT and the content-addressed store the same way
+`@dclimate/ipld-index` does for JS.
+
+```
+tabular-js  ->  @dclimate/ipld-index      (HAMT, CAS, range reads)
+tabular-py  ->  py-hamt                   (same, already existed)
+```
+
+## Status
+
+**Reader only.** Publishing, compaction, and rollup stay in `tabular-js` — the ETL that
+writes these datasets is JS, and a second writer would be a second thing to keep
+byte-identical for no current gain. Everything needed to *read* a dataset published by
+`tabular-js` is here.
+
+## Install
+
+```bash
+uv pip install dclimate-tabular-py
+```
+
+## Usage
+
+```python
+import asyncio
+from tabular_py import GatewayRangeSource, StationDataset
+
+async def main():
+    source = GatewayRangeSource("https://ipfs-gateway.dclimate.net")
+    ds = await StationDataset.open(source, "bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a")
+
+    # nearest station to a point, then a year of readings
+    near = await ds.nearest(34.05, -118.24, max_km=100)
+    rows = await near.time_range("2024-01-01", "2024-12-31").elements("PRCP").rows()
+    for row in rows[:5]:
+        print(row.station_id, row.ts, row.values)
+
+asyncio.run(main())
+```
+
+The chainable selection API mirrors `tabular-js`:
+
+```python
+ds.select("USW00023174")           # explicit station ids
+ds.circle(34.05, -118.24, 50)      # within 50 km
+ds.rectangle(33.0, -119.0, 35.0, -117.0)
+ds.polygon([[(lon, lat), ...]])
+await ds.nearest(lat, lon)         # async: reads the geo index
+ds.time_range(start, end)
+ds.elements("PRCP", "TMAX")
+ds.where(gt("TMAX", 300))          # pushed down to fragment statistics
+```
+
+Selections are immutable — each call returns a new `StationDataset`, so a base dataset can
+be reused across queries.
+
+Terminal operations:
+
+| Call | Returns |
+|---|---|
+| `await ds.rows()` | `list[ResultRow]` |
+| `await ds.to_records()` | `list[dict]` — `station_id`, `time`, `values` |
+| `await ds.to_records("TMAX")` | `list[dict]` — `station_id`, `time`, `value` |
+| `await ds.to_arrow()` | `pyarrow.Table` |
+| `await ds.plan()` | `QueryPlan` — what *would* be fetched, without fetching |
+| `await ds.list_stations()` | `list[StationInfo]` |
+
+## How reads stay small
+
+A query never scans the dataset. Three things prune before any Parquet byte is fetched:
+
+1. **The station index** (a HAMT keyed by station id) resolves named stations directly.
+2. **The geo projection** answers region queries by reading one or two shard blocks
+   instead of walking all 132k stations.
+3. **Fragment statistics in the manifest** — per-column min/max and null counts — let a
+   predicate skip whole fragments unread.
+
+What survives is fetched with HTTP range requests against the exact column-chunk byte
+ranges the manifest records, so a query for one column of one year moves kilobytes.
+
+### One deviation from `tabular-js`, and why
+
+`tabular-js` synthesizes Parquet `FileMetaData` client-side from the manifest and reads a
+fragment with **zero** footer fetches. PyArrow exposes no public `FileMetaData`
+constructor, so that trick does not transfer.
+
+Instead this reader fetches the footer by its manifest-recorded
+`footer_offset`/`footer_length` in a single ranged GET, verifies it against the
+manifest's `footer_digest`, and hands the parsed metadata to PyArrow. Cost is one extra
+range request per fragment — and it buys a corruption check `tabular-js` does not
+perform. Footers are cached per fragment CID, so a repeated query pays it once.
+
+## Development
+
+```bash
+uv sync
+uv run pytest                     # unit tests, no network
+uv run pytest -m network          # conformance against the live gateway
+uv run ruff check . && uv run mypy tabular_py
+```
+
+`tests/test_conformance.py` reads a real published GHCNd dataset
+(`bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a`, 132,437 stations,
+1.15 B rows) and asserts this implementation agrees with `tabular-js` on decoded values.

dclimate_tabular_py-0.1.0/pyproject.toml

@@ -0,0 +1,88 @@
+[project]
+# PyPI rejects "tabular-py": it normalises to the existing "tabularpy" project.
+# The distribution name is namespaced instead, matching @dclimate/tabular on npm.
+# The import name stays `tabular_py`, which PyPI has no opinion about.
+name = "dclimate-tabular-py"
+version = "0.1.0"
+description = "Content-addressed tabular/telemetry manifest for climate data on IPFS (dclimate-tabular/0)"
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.12"
+dependencies = [
+    "dag-cbor>=0.3.3",
+    "multiformats[full]>=0.3.1.post4",
+    # Declared explicitly rather than relied on through multiformats[full]:
+    # py-hamt imports blake3 the same way without declaring it, and a transitive
+    # dep that the format's only hash function depends on is not one to inherit.
+    "blake3>=0.4.1",
+    "py-hamt>=3.6.0",
+    "pyarrow>=18.0.0",
+    "httpx>=0.28.1",
+]
+
+[project.urls]
+Homepage = "https://github.com/dClimate/tabular-py"
+Repository = "https://github.com/dClimate/tabular-py"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["tabular_py"]
+
+[dependency-groups]
+dev = [
+    "pytest>=8.3.3",
+    "pytest-asyncio>=1.0.0",
+    "pytest-cov>=6.0.0",
+    "ruff>=0.7.1",
+    "mypy>=1.15.0",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+# py-hamt pins every async fixture/test to a function-scoped loop; the same rule
+# applies here because KuboCAS caches an httpx client per running loop.
+asyncio_default_fixture_loop_scope = "function"
+testpaths = ["tests"]
+markers = [
+    "network: hits a real IPFS gateway; deselect with -m 'not network'",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM"]
+# TC00x is deliberately off. `from __future__ import annotations` is on
+# everywhere, so annotations are already lazy, and the types these rules want
+# moved into TYPE_CHECKING blocks are needed at runtime anyway -- CID and the
+# model classes back isinstance checks, and the dataclass field types are
+# resolved when slots are built.
+
+[tool.ruff.lint.per-file-ignores]
+# Tests reach into private attributes to corrupt a block and assert the
+# integrity check fires; that is the point of the test.
+"tests/*" = ["SLF001"]
+
+[tool.mypy]
+python_version = "3.12"
+strict = true
+# Off because the exhaustiveness fallbacks at the end of the predicate
+# dispatchers are unreachable only as long as the Literal unions stay in sync
+# with the branches above them. Keeping the guard is worth more than the check.
+warn_unreachable = false
+
+[[tool.mypy.overrides]]
+# Neither ships a py.typed marker, so everything they export arrives as Any.
+module = ["py_hamt.*", "pyarrow.*"]
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+# The wire decoders funnel every value through helpers that narrow `Any` by
+# hand and raise otherwise. mypy cannot see through that, so it reports each
+# validated return as "returning Any"; the validation is the point.
+module = ["tabular_py.wire", "tabular_py.source", "tabular_py.reader"]
+warn_return_any = false

dclimate_tabular_py-0.1.0/scripts/capture_fixture.py

@@ -0,0 +1,106 @@
+"""Capture a real published dataset into an offline fixture.
+
+Records every block a small set of queries touches, so the test suite exercises
+genuine `tabular-js`-written bytes without needing a gateway. Re-run when the
+reference dataset changes:
+
+    uv run python scripts/capture_fixture.py
+
+Blocks are stored base64 in a single JSON file, keyed by CID string.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import json
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from multiformats import CID  # noqa: E402
+
+from tabular_py import (  # noqa: E402
+    Circle,
+    GatewayRangeSource,
+    StationDataset,
+    cid_str,
+)
+
+GATEWAY = "https://ipfs-gateway.dclimate.net"
+ROOT_CID = "bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a"
+OUT = Path(__file__).resolve().parent.parent / "tests" / "fixtures" / "ghcnd_sample.json"
+
+# Stations chosen for coverage rather than convenience: one with pre-epoch
+# timestamps, one modern, one in the LA cluster the geo tests query.
+STATIONS = ["US1GAFS0003", "USC00045111"]
+
+# Ids the dataset does not contain, so the not-found path is covered offline.
+MISSING_STATIONS = ["NO_SUCH_STATION", "ZZZZZZZZZZZ9"]
+
+
+class RecordingSource:
+    """Passes through to a real source and records every block it serves."""
+
+    def __init__(self, inner: GatewayRangeSource) -> None:
+        self._inner = inner
+        self.blocks: dict[str, bytes] = {}
+
+    async def get_block(self, cid: CID) -> bytes:
+        key = cid_str(cid)
+        if key not in self.blocks:
+            self.blocks[key] = await self._inner.get_block(cid)
+        return self.blocks[key]
+
+    async def get_range(self, cid: CID, offset: int, length: int) -> bytes:
+        # Record the whole block so the fixture can serve any range of it.
+        data = await self.get_block(cid)
+        return data[offset : offset + length]
+
+
+async def main() -> None:
+    async with GatewayRangeSource(GATEWAY) as gateway:
+        source = RecordingSource(gateway)
+        ds = await StationDataset.open(source, ROOT_CID)
+
+        for station_id in STATIONS:
+            await ds.info_for(station_id)
+            await ds.columns_for(station_id)
+            await ds.select(station_id).elements("PRCP").rows()
+
+        # A lookup that misses still descends the trie, and lands on different
+        # nodes than any successful lookup does. Capturing that path is what
+        # lets the offline suite test "station is not in this dataset".
+        from tabular_py import lookup_station
+
+        for absent in MISSING_STATIONS:
+            await lookup_station(ds.reader.root.stations, source, absent)
+
+        # Geo paths: the projection directory plus the shards around LA.
+        await ds.find_nearest_station(34.05, -118.24)
+        from tabular_py import lookup_geo_stations
+
+        projection = ds.reader.root.projections.get("geo")
+        if projection is not None:
+            await lookup_geo_stations(
+                projection, source, Circle(34_052_000, -118_243_000, 25_000)
+            )
+
+    OUT.parent.mkdir(parents=True, exist_ok=True)
+    payload = {
+        "root": ROOT_CID,
+        "gateway": GATEWAY,
+        "stations": STATIONS,
+        "blocks": {
+            key: base64.b64encode(data).decode("ascii")
+            for key, data in sorted(source.blocks.items())
+        },
+    }
+    OUT.write_text(json.dumps(payload, indent=1))
+    total = sum(len(v) for v in source.blocks.values())
+    print(f"wrote {OUT} -- {len(source.blocks)} blocks, {total} bytes")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

dclimate_tabular_py-0.1.0/tabular_py/__init__.py

@@ -0,0 +1,245 @@
+"""tabular-py -- Python reader for ``dclimate-tabular/0``.
+
+The Python counterpart to ``@dclimate/tabular``, reading the same content-addressed
+station datasets from the same CIDs, built on ``py-hamt``.
+
+    from tabular_py import GatewayRangeSource, StationDataset
+
+    source = GatewayRangeSource("https://ipfs-gateway.dclimate.net")
+    ds = await StationDataset.open(source, root_cid)
+    rows = await ds.select("USW00023174").time_range("2024-01-01", "2024-12-31").rows()
+"""
+
+from __future__ import annotations
+
+from .codec import (
+    BLOCK_HARD_LIMIT_BYTES,
+    BLOCK_TARGET_BYTES,
+    blake3_digest,
+    cid_for_bytes,
+    cid_str,
+    decode_block,
+    encode_block,
+    make_block,
+    verify_cid,
+)
+from .errors import (
+    BlockSizeError,
+    CidMismatchError,
+    CodecError,
+    DatasetIntegrityError,
+    DatasetReaderError,
+    DclimateTabularError,
+    GeoFilterError,
+    GeoIndexError,
+    PredicateError,
+    RangeSourceError,
+    StationIndexError,
+    StationSelectionError,
+    WireError,
+)
+from .geo import (
+    BBox,
+    Circle,
+    GeoFilter,
+    Polygon,
+    haversine_meters,
+    matches_geo_filter,
+    validate_geo_filter,
+)
+from .geo_index import (
+    bounding_boxes_for_filter,
+    first_accepted_in_order,
+    lookup_geo_stations,
+    nearest_station,
+    nearest_station_where,
+)
+from .model import (
+    SPEC_VERSION,
+    CellValue,
+    ColumnStat,
+    DatasetRoot,
+    FragmentEntry,
+    GeoProjection,
+    GeoShardRef,
+    HamtStationIndex,
+    InlineStationIndex,
+    StationEntry,
+    StationIndex,
+    StationSummary,
+    TableField,
+    TableSchema,
+    columns_from_fragments,
+    correction_schema,
+    default_column_key,
+)
+from .predicate import (
+    And,
+    Comparison,
+    Not,
+    NullCheck,
+    Or,
+    Predicate,
+    and_,
+    eq,
+    evaluate_row_predicate,
+    evaluate_stats_predicate,
+    ge,
+    gt,
+    is_null,
+    le,
+    lt,
+    ne,
+    not_,
+    not_null,
+    or_,
+    predicate_elements,
+)
+from .reader import (
+    DatasetReader,
+    PlanRange,
+    Query,
+    QueryPlan,
+    QueryPlanFragment,
+    ResultRow,
+)
+from .source import (
+    CasRangeSource,
+    CountingSource,
+    GatewayRangeSource,
+    MemoryRangeSource,
+    RangeSource,
+)
+from .station_dataset import NearestStation, StationDataset, StationInfo, WithinRange
+from .station_index import (
+    list_station_entries,
+    list_stations,
+    lookup_station,
+    station_count,
+)
+from .wire import (
+    fragment_from_wire,
+    fragment_to_wire,
+    geo_projection_from_wire,
+    geo_projection_to_wire,
+    geo_shard_from_wire,
+    geo_shard_to_wire,
+    root_from_wire,
+    root_to_wire,
+    station_entry_from_wire,
+    station_entry_to_wire,
+    station_summary_from_wire,
+    station_summary_to_wire,
+    table_schema_from_wire,
+    table_schema_to_wire,
+)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "BLOCK_HARD_LIMIT_BYTES",
+    "BLOCK_TARGET_BYTES",
+    "SPEC_VERSION",
+    "And",
+    "BBox",
+    "BlockSizeError",
+    "CasRangeSource",
+    "CellValue",
+    "CidMismatchError",
+    "Circle",
+    "CodecError",
+    "ColumnStat",
+    "Comparison",
+    "CountingSource",
+    "DatasetIntegrityError",
+    "DatasetReader",
+    "DatasetReaderError",
+    "DatasetRoot",
+    "DclimateTabularError",
+    "FragmentEntry",
+    "GatewayRangeSource",
+    "GeoFilter",
+    "GeoFilterError",
+    "GeoIndexError",
+    "GeoProjection",
+    "GeoShardRef",
+    "HamtStationIndex",
+    "InlineStationIndex",
+    "MemoryRangeSource",
+    "NearestStation",
+    "Not",
+    "NullCheck",
+    "Or",
+    "PlanRange",
+    "Polygon",
+    "Predicate",
+    "PredicateError",
+    "Query",
+    "QueryPlan",
+    "QueryPlanFragment",
+    "RangeSource",
+    "RangeSourceError",
+    "ResultRow",
+    "StationDataset",
+    "StationEntry",
+    "StationIndex",
+    "StationIndexError",
+    "StationInfo",
+    "StationSelectionError",
+    "StationSummary",
+    "TableField",
+    "TableSchema",
+    "WireError",
+    "WithinRange",
+    "__version__",
+    "and_",
+    "blake3_digest",
+    "bounding_boxes_for_filter",
+    "cid_for_bytes",
+    "cid_str",
+    "columns_from_fragments",
+    "correction_schema",
+    "decode_block",
+    "default_column_key",
+    "encode_block",
+    "eq",
+    "first_accepted_in_order",
+    "evaluate_row_predicate",
+    "evaluate_stats_predicate",
+    "fragment_from_wire",
+    "fragment_to_wire",
+    "ge",
+    "geo_projection_from_wire",
+    "geo_projection_to_wire",
+    "geo_shard_from_wire",
+    "geo_shard_to_wire",
+    "gt",
+    "haversine_meters",
+    "is_null",
+    "le",
+    "list_station_entries",
+    "list_stations",
+    "lookup_geo_stations",
+    "lookup_station",
+    "lt",
+    "make_block",
+    "matches_geo_filter",
+    "ne",
+    "nearest_station",
+    "nearest_station_where",
+    "not_",
+    "not_null",
+    "or_",
+    "predicate_elements",
+    "root_from_wire",
+    "root_to_wire",
+    "station_count",
+    "station_entry_from_wire",
+    "station_entry_to_wire",
+    "station_summary_from_wire",
+    "station_summary_to_wire",
+    "table_schema_from_wire",
+    "table_schema_to_wire",
+    "validate_geo_filter",
+    "verify_cid",
+]