dclimate-tabular-py 0.1.0__py3-none-any.whl

dclimate_tabular_py-0.1.0.dist-info/METADATA

@@ -0,0 +1,126 @@
+Metadata-Version: 2.5
+Name: dclimate-tabular-py
+Version: 0.1.0
+Summary: Content-addressed tabular/telemetry manifest for climate data on IPFS (dclimate-tabular/0)
+Project-URL: Homepage, https://github.com/dClimate/tabular-py
+Project-URL: Repository, https://github.com/dClimate/tabular-py
+License: MIT
+Requires-Python: >=3.12
+Requires-Dist: blake3>=0.4.1
+Requires-Dist: dag-cbor>=0.3.3
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: multiformats[full]>=0.3.1.post4
+Requires-Dist: py-hamt>=3.6.0
+Requires-Dist: pyarrow>=18.0.0
+Description-Content-Type: text/markdown
+
+# tabular-py
+
+Python reader for `dclimate-tabular/0` — content-addressed station/telemetry data on IPFS.
+
+The Python counterpart to [`tabular-js`](https://github.com/dClimate/tabular-js), reading the
+same format from the same CIDs. Built on [`py-hamt`](https://github.com/dClimate/py-hamt),
+which supplies the HAMT and the content-addressed store the same way
+`@dclimate/ipld-index` does for JS.
+
+```
+tabular-js  ->  @dclimate/ipld-index      (HAMT, CAS, range reads)
+tabular-py  ->  py-hamt                   (same, already existed)
+```
+
+## Status
+
+**Reader only.** Publishing, compaction, and rollup stay in `tabular-js` — the ETL that
+writes these datasets is JS, and a second writer would be a second thing to keep
+byte-identical for no current gain. Everything needed to *read* a dataset published by
+`tabular-js` is here.
+
+## Install
+
+```bash
+uv pip install dclimate-tabular-py
+```
+
+## Usage
+
+```python
+import asyncio
+from tabular_py import GatewayRangeSource, StationDataset
+
+async def main():
+    source = GatewayRangeSource("https://ipfs-gateway.dclimate.net")
+    ds = await StationDataset.open(source, "bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a")
+
+    # nearest station to a point, then a year of readings
+    near = await ds.nearest(34.05, -118.24, max_km=100)
+    rows = await near.time_range("2024-01-01", "2024-12-31").elements("PRCP").rows()
+    for row in rows[:5]:
+        print(row.station_id, row.ts, row.values)
+
+asyncio.run(main())
+```
+
+The chainable selection API mirrors `tabular-js`:
+
+```python
+ds.select("USW00023174")           # explicit station ids
+ds.circle(34.05, -118.24, 50)      # within 50 km
+ds.rectangle(33.0, -119.0, 35.0, -117.0)
+ds.polygon([[(lon, lat), ...]])
+await ds.nearest(lat, lon)         # async: reads the geo index
+ds.time_range(start, end)
+ds.elements("PRCP", "TMAX")
+ds.where(gt("TMAX", 300))          # pushed down to fragment statistics
+```
+
+Selections are immutable — each call returns a new `StationDataset`, so a base dataset can
+be reused across queries.
+
+Terminal operations:
+
+| Call | Returns |
+|---|---|
+| `await ds.rows()` | `list[ResultRow]` |
+| `await ds.to_records()` | `list[dict]` — `station_id`, `time`, `values` |
+| `await ds.to_records("TMAX")` | `list[dict]` — `station_id`, `time`, `value` |
+| `await ds.to_arrow()` | `pyarrow.Table` |
+| `await ds.plan()` | `QueryPlan` — what *would* be fetched, without fetching |
+| `await ds.list_stations()` | `list[StationInfo]` |
+
+## How reads stay small
+
+A query never scans the dataset. Three things prune before any Parquet byte is fetched:
+
+1. **The station index** (a HAMT keyed by station id) resolves named stations directly.
+2. **The geo projection** answers region queries by reading one or two shard blocks
+   instead of walking all 132k stations.
+3. **Fragment statistics in the manifest** — per-column min/max and null counts — let a
+   predicate skip whole fragments unread.
+
+What survives is fetched with HTTP range requests against the exact column-chunk byte
+ranges the manifest records, so a query for one column of one year moves kilobytes.
+
+### One deviation from `tabular-js`, and why
+
+`tabular-js` synthesizes Parquet `FileMetaData` client-side from the manifest and reads a
+fragment with **zero** footer fetches. PyArrow exposes no public `FileMetaData`
+constructor, so that trick does not transfer.
+
+Instead this reader fetches the footer by its manifest-recorded
+`footer_offset`/`footer_length` in a single ranged GET, verifies it against the
+manifest's `footer_digest`, and hands the parsed metadata to PyArrow. Cost is one extra
+range request per fragment — and it buys a corruption check `tabular-js` does not
+perform. Footers are cached per fragment CID, so a repeated query pays it once.
+
+## Development
+
+```bash
+uv sync
+uv run pytest                     # unit tests, no network
+uv run pytest -m network          # conformance against the live gateway
+uv run ruff check . && uv run mypy tabular_py
+```
+
+`tests/test_conformance.py` reads a real published GHCNd dataset
+(`bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a`, 132,437 stations,
+1.15 B rows) and asserts this implementation agrees with `tabular-js` on decoded values.

dclimate_tabular_py-0.1.0.dist-info/RECORD

@@ -0,0 +1,15 @@
+tabular_py/__init__.py,sha256=srZln4BumpfdFIy4m2k37VLy1aNiyOflQxURMMAVNPk,5071
+tabular_py/codec.py,sha256=R7pZCr_3X_8fIxDbRDcxcQGUifmLyerjUyNhh6AiYUQ,4284
+tabular_py/errors.py,sha256=ZEKqfXRCcoWPnX8QHQwGM9B34f7HNCMYAwVeQgbH_p4,2709
+tabular_py/geo.py,sha256=ehCu5uRRzhyzaT4DdwwaN6iCqjfoF7UD8Ey_NicdPdI,5771
+tabular_py/geo_index.py,sha256=wI2957JzSszjjmydmsV5-OrDtlH7dFpvREesE6FDZ7I,16747
+tabular_py/model.py,sha256=2qstncwWNCLRp_OPm2giCiaxG5DB3x9cq9ibPwKCXcU,13022
+tabular_py/predicate.py,sha256=OzBXRoWCAXqsiv-iLsrHBnbFrusdEbGHaQb-Ioq_CVQ,11105
+tabular_py/reader.py,sha256=rEDB62L_bDFIe5UZ-3-y534m66dFVmz6uWVWTWcCSM0,26466
+tabular_py/source.py,sha256=bRw2EwknrAx92I2Fix5KZQ4pJ2HE1Rm3CoAlHhadCpk,7965
+tabular_py/station_dataset.py,sha256=c-onhTzzcCdtv45CEVarZgAMpLJab6Sni3VoWnZ2SIM,23192
+tabular_py/station_index.py,sha256=0ckMC5cu6o8CCCxMFwOolMEPUPp6CFGpqw93_YJpH4g,5901
+tabular_py/wire.py,sha256=WKDmzVQJkDHvfucKVzaL2gt7FWDMRIC8tp3uYCXLqdY,19788
+dclimate_tabular_py-0.1.0.dist-info/METADATA,sha256=Zpu909niwS3h6iG5HsPx2zz_vkDkwLXvG6X4qUGiFWY,4780
+dclimate_tabular_py-0.1.0.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+dclimate_tabular_py-0.1.0.dist-info/RECORD,,

dclimate_tabular_py-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any

tabular_py/__init__.py

@@ -0,0 +1,245 @@
+"""tabular-py -- Python reader for ``dclimate-tabular/0``.
+
+The Python counterpart to ``@dclimate/tabular``, reading the same content-addressed
+station datasets from the same CIDs, built on ``py-hamt``.
+
+    from tabular_py import GatewayRangeSource, StationDataset
+
+    source = GatewayRangeSource("https://ipfs-gateway.dclimate.net")
+    ds = await StationDataset.open(source, root_cid)
+    rows = await ds.select("USW00023174").time_range("2024-01-01", "2024-12-31").rows()
+"""
+
+from __future__ import annotations
+
+from .codec import (
+    BLOCK_HARD_LIMIT_BYTES,
+    BLOCK_TARGET_BYTES,
+    blake3_digest,
+    cid_for_bytes,
+    cid_str,
+    decode_block,
+    encode_block,
+    make_block,
+    verify_cid,
+)
+from .errors import (
+    BlockSizeError,
+    CidMismatchError,
+    CodecError,
+    DatasetIntegrityError,
+    DatasetReaderError,
+    DclimateTabularError,
+    GeoFilterError,
+    GeoIndexError,
+    PredicateError,
+    RangeSourceError,
+    StationIndexError,
+    StationSelectionError,
+    WireError,
+)
+from .geo import (
+    BBox,
+    Circle,
+    GeoFilter,
+    Polygon,
+    haversine_meters,
+    matches_geo_filter,
+    validate_geo_filter,
+)
+from .geo_index import (
+    bounding_boxes_for_filter,
+    first_accepted_in_order,
+    lookup_geo_stations,
+    nearest_station,
+    nearest_station_where,
+)
+from .model import (
+    SPEC_VERSION,
+    CellValue,
+    ColumnStat,
+    DatasetRoot,
+    FragmentEntry,
+    GeoProjection,
+    GeoShardRef,
+    HamtStationIndex,
+    InlineStationIndex,
+    StationEntry,
+    StationIndex,
+    StationSummary,
+    TableField,
+    TableSchema,
+    columns_from_fragments,
+    correction_schema,
+    default_column_key,
+)
+from .predicate import (
+    And,
+    Comparison,
+    Not,
+    NullCheck,
+    Or,
+    Predicate,
+    and_,
+    eq,
+    evaluate_row_predicate,
+    evaluate_stats_predicate,
+    ge,
+    gt,
+    is_null,
+    le,
+    lt,
+    ne,
+    not_,
+    not_null,
+    or_,
+    predicate_elements,
+)
+from .reader import (
+    DatasetReader,
+    PlanRange,
+    Query,
+    QueryPlan,
+    QueryPlanFragment,
+    ResultRow,
+)
+from .source import (
+    CasRangeSource,
+    CountingSource,
+    GatewayRangeSource,
+    MemoryRangeSource,
+    RangeSource,
+)
+from .station_dataset import NearestStation, StationDataset, StationInfo, WithinRange
+from .station_index import (
+    list_station_entries,
+    list_stations,
+    lookup_station,
+    station_count,
+)
+from .wire import (
+    fragment_from_wire,
+    fragment_to_wire,
+    geo_projection_from_wire,
+    geo_projection_to_wire,
+    geo_shard_from_wire,
+    geo_shard_to_wire,
+    root_from_wire,
+    root_to_wire,
+    station_entry_from_wire,
+    station_entry_to_wire,
+    station_summary_from_wire,
+    station_summary_to_wire,
+    table_schema_from_wire,
+    table_schema_to_wire,
+)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "BLOCK_HARD_LIMIT_BYTES",
+    "BLOCK_TARGET_BYTES",
+    "SPEC_VERSION",
+    "And",
+    "BBox",
+    "BlockSizeError",
+    "CasRangeSource",
+    "CellValue",
+    "CidMismatchError",
+    "Circle",
+    "CodecError",
+    "ColumnStat",
+    "Comparison",
+    "CountingSource",
+    "DatasetIntegrityError",
+    "DatasetReader",
+    "DatasetReaderError",
+    "DatasetRoot",
+    "DclimateTabularError",
+    "FragmentEntry",
+    "GatewayRangeSource",
+    "GeoFilter",
+    "GeoFilterError",
+    "GeoIndexError",
+    "GeoProjection",
+    "GeoShardRef",
+    "HamtStationIndex",
+    "InlineStationIndex",
+    "MemoryRangeSource",
+    "NearestStation",
+    "Not",
+    "NullCheck",
+    "Or",
+    "PlanRange",
+    "Polygon",
+    "Predicate",
+    "PredicateError",
+    "Query",
+    "QueryPlan",
+    "QueryPlanFragment",
+    "RangeSource",
+    "RangeSourceError",
+    "ResultRow",
+    "StationDataset",
+    "StationEntry",
+    "StationIndex",
+    "StationIndexError",
+    "StationInfo",
+    "StationSelectionError",
+    "StationSummary",
+    "TableField",
+    "TableSchema",
+    "WireError",
+    "WithinRange",
+    "__version__",
+    "and_",
+    "blake3_digest",
+    "bounding_boxes_for_filter",
+    "cid_for_bytes",
+    "cid_str",
+    "columns_from_fragments",
+    "correction_schema",
+    "decode_block",
+    "default_column_key",
+    "encode_block",
+    "eq",
+    "first_accepted_in_order",
+    "evaluate_row_predicate",
+    "evaluate_stats_predicate",
+    "fragment_from_wire",
+    "fragment_to_wire",
+    "ge",
+    "geo_projection_from_wire",
+    "geo_projection_to_wire",
+    "geo_shard_from_wire",
+    "geo_shard_to_wire",
+    "gt",
+    "haversine_meters",
+    "is_null",
+    "le",
+    "list_station_entries",
+    "list_stations",
+    "lookup_geo_stations",
+    "lookup_station",
+    "lt",
+    "make_block",
+    "matches_geo_filter",
+    "ne",
+    "nearest_station",
+    "nearest_station_where",
+    "not_",
+    "not_null",
+    "or_",
+    "predicate_elements",
+    "root_from_wire",
+    "root_to_wire",
+    "station_count",
+    "station_entry_from_wire",
+    "station_entry_to_wire",
+    "station_summary_from_wire",
+    "station_summary_to_wire",
+    "table_schema_from_wire",
+    "table_schema_to_wire",
+    "validate_geo_filter",
+    "verify_cid",
+]

tabular_py/codec.py

@@ -0,0 +1,114 @@
+"""Canonical DAG-CBOR blocks and blake3 CIDs.
+
+Ported from ``tabular-js`` ``src/codec.ts``.
+
+Every block in the format is strict canonical DAG-CBOR addressed by a CIDv1 with
+a 32-byte blake3 multihash. Decoding re-encodes and compares, so a block that
+round-trips to different bytes is rejected rather than silently accepted -- two
+encodings of the same value would otherwise produce two CIDs for one logical
+block, and structural dedup across commits depends on that not happening.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Literal
+
+import dag_cbor
+from blake3 import blake3
+from multiformats import CID
+
+from .errors import BlockSizeError, CodecError
+
+# Blocks target 512 KiB and must never exceed 1 MiB. The target is what the
+# builders split on; the hard limit is what the format guarantees.
+BLOCK_TARGET_BYTES = 512 * 1024
+BLOCK_HARD_LIMIT_BYTES = 1024 * 1024
+
+BLAKE3_CODE = 0x1E
+DAG_CBOR_CODE = 0x71
+RAW_CODE = 0x55
+DAG_PB_CODE = 0x70
+
+Codec = Literal["dag-cbor", "raw"]
+
+_CODEC_NAMES: dict[str, int] = {"dag-cbor": DAG_CBOR_CODE, "raw": RAW_CODE}
+
+
+def blake3_digest(data: bytes) -> bytes:
+    """32-byte blake3, the only hash this format uses."""
+    return blake3(data).digest(length=32)
+
+
+def encode_block(value: Any) -> bytes:
+    """Encode a wire value to canonical DAG-CBOR."""
+    try:
+        return dag_cbor.encode(value)
+    except Exception as error:  # noqa: BLE001 - re-raised with context below
+        raise CodecError(f"DAG-CBOR encode failed: {error}") from error
+
+
+def decode_block(data: bytes) -> Any:
+    """Decode canonical DAG-CBOR, rejecting any non-canonical encoding.
+
+    The re-encode check is the reason this exists rather than a bare
+    ``dag_cbor.decode``. A block whose bytes are not the canonical encoding of
+    their own value cannot be re-derived from that value, so its CID could never
+    be reproduced -- accepting it would let a dataset carry blocks this
+    implementation can read but not rewrite.
+    """
+    try:
+        value = dag_cbor.decode(data)
+    except Exception as error:  # noqa: BLE001 - re-raised with context below
+        raise CodecError(f"DAG-CBOR decode failed: {error}") from error
+
+    try:
+        canonical = dag_cbor.encode(value)
+    except Exception as error:  # noqa: BLE001 - re-raised with context below
+        raise CodecError(f"DAG-CBOR decode failed: re-encode failed: {error}") from error
+
+    if canonical != data:
+        raise CodecError("DAG-CBOR decode failed: input is not canonical DAG-CBOR")
+    return value
+
+
+def cid_for_bytes(data: bytes, codec: Codec) -> CID:
+    """The CIDv1 that addresses ``data`` under ``codec``."""
+    code = _CODEC_NAMES.get(codec)
+    if code is None:
+        raise CodecError(f"Unsupported codec: {codec}")
+    digest = blake3_digest(data)
+    # multiformats wants the full multihash (code + length prefix + digest).
+    multihash = bytes([BLAKE3_CODE, len(digest)]) + digest
+    return CID("base32", 1, code, multihash)
+
+
+def make_block(value: Any) -> tuple[CID, bytes]:
+    """Encode a wire value and address it, enforcing the hard block limit."""
+    data = encode_block(value)
+    if len(data) > BLOCK_HARD_LIMIT_BYTES:
+        raise BlockSizeError(
+            f"Block is {len(data)} bytes, above the {BLOCK_HARD_LIMIT_BYTES}-byte hard limit"
+        )
+    return cid_for_bytes(data, "dag-cbor"), data
+
+
+def cid_str(cid: CID) -> str:
+    """Base32 string form, the spelling used for gateway paths and dict keys."""
+    return str(cid.set(base="base32"))
+
+
+def verify_cid(cid: CID, data: bytes) -> None:
+    """Raise unless ``data`` hashes to ``cid``.
+
+    Only meaningful for whole blocks. Ranged reads return a slice, which by
+    construction does not hash to the block's CID; those are covered by the
+    fragment footer digest instead.
+    """
+    if cid.codec.code == DAG_PB_CODE:
+        # UnixFS: the CID addresses a dag-pb node whose digest covers the DAG
+        # structure, not the concatenated file bytes, so this check does not
+        # apply. The caller gets the same guarantee from the footer digest.
+        return
+    # `.raw_digest` is the hash bytes; `.digest` carries the multihash prefix.
+    if blake3_digest(data) != bytes(cid.raw_digest):
+        raise CodecError(f"Block bytes do not match {cid_str(cid)}")

tabular_py/errors.py

@@ -0,0 +1,85 @@
+"""Error hierarchy, ported from ``tabular-js`` ``src/errors.ts``.
+
+The shape matters as much as the names: ``dclimate-client-py`` will translate
+these into its own error types the same way ``dclimate-client-js`` does, so a
+class moving between branches of this tree changes how a caller's ``except``
+behaves.
+"""
+
+from __future__ import annotations
+
+
+class DclimateTabularError(Exception):
+    """Base for everything this package raises."""
+
+
+class CodecError(DclimateTabularError):
+    """DAG-CBOR encode/decode failed, or the bytes were not canonical."""
+
+
+class BlockSizeError(DclimateTabularError):
+    """A block exceeded the hard limit."""
+
+
+class WireError(DclimateTabularError):
+    """A wire value did not match the schema for its position in the manifest."""
+
+
+class CidMismatchError(DclimateTabularError):
+    """Fetched bytes did not hash to the CID that addressed them."""
+
+
+class RangeSourceError(DclimateTabularError):
+    """A range request could not be satisfied."""
+
+
+class StationIndexError(DclimateTabularError):
+    """The station index could not be read or built."""
+
+
+class GeoIndexError(DclimateTabularError):
+    """The geo projection could not be read or built."""
+
+
+class GeoFilterError(DclimateTabularError):
+    """A geo filter was malformed."""
+
+
+class PredicateError(DclimateTabularError):
+    """A predicate was malformed."""
+
+
+class DatasetReaderError(DclimateTabularError):
+    """The caller asked for something this dataset cannot answer.
+
+    A bad request: an unknown column, an incoherent predicate, a selection that
+    names no station. Consumers translate this wholesale into their own
+    "invalid request" type.
+    """
+
+
+class DatasetIntegrityError(DclimateTabularError):
+    """The dataset contradicts itself.
+
+    A root naming a schema that is not there, a correction fragment whose commit
+    sequence contradicts the entry pointing at it, a fragment whose bytes do not
+    match the manifest that describes them.
+
+    Deliberately **not** a subclass of :class:`DatasetReaderError`: consumers
+    translate that type wholesale into their own "invalid request" error, so
+    inheriting would report a corrupt dataset as the caller's mistake and send
+    them looking for a bug they did not write.
+    """
+
+
+class StationSelectionError(DatasetReaderError):
+    """A station selection could not be resolved.
+
+    ``reason`` distinguishes "this station is not in this dataset" from "this
+    selection is malformed", because the client maps the first to a not-found
+    error and the second to an invalid-selection error.
+    """
+
+    def __init__(self, message: str, reason: str = "invalid") -> None:
+        super().__init__(message)
+        self.reason = reason