usdata 0.5.0__tar.gz → 0.6.0__tar.gz

{usdata-0.5.0 → usdata-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.5.0
+Version: 0.6.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -14,9 +14,11 @@ Requires-Dist: httpx>=0.28.1
 Requires-Dist: pydantic>=2.7
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: typer>=0.12
+Requires-Dist: pandas>=3.0 ; extra == 'pandas'
 Requires-Python: >=3.11
 Project-URL: Homepage, https://github.com/jakeryderv/usdata
 Project-URL: Repository, https://github.com/jakeryderv/usdata
+Provides-Extra: pandas
 Description-Content-Type: text/markdown
 
 # usdata
@@ -24,22 +26,22 @@ Description-Content-Type: text/markdown
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.5 supports GHCN-Daily, NEXRAD Level II, USGS daily
+> Status: pre-alpha. v0.6 supports GHCN-Daily, NEXRAD Level II, USGS daily
 > values, and CoastWatch SST subsets with provenance, plus Census state/county
-> lookup. Other datasets are planned.
+> lookup and optional pandas CSV readers. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
 
 <!-- registry:start -->
-| Provider | Available | Stub | Planned | Next up (0.6) | Datasets |
+| Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | `gfs`, `hrrr`, `oisst`, `etopo` | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | — | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
 | [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
-| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | `gpm-imerg` | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
 | [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
 
 Available datasets are in `code`, stubs in _italics_; planned ones are counted. Available means implemented in this source checkout; consult the [releases](https://github.com/jakeryderv/usdata/releases) for published support. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
@@ -130,6 +132,14 @@ sources:
     end: 2024-05-31
 ```
 
+## Opening CSV data
+
+`FetchedAsset.open()` is available since v0.6 with the optional pandas
+extra (`pip install "usdata[pandas]"`). It reads cached CSV into a DataFrame,
+preserves identifier strings, and keeps CoastWatch units as metadata.
+See the [reader reference](docs/reference/readers.md)
+and [fetch → open → analyze example](examples/sst-analysis/README.md).
+
 ## Development
 
 Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
@@ -138,16 +148,18 @@ Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
-just check     # format, lint, typecheck, offline tests, generated docs
+just check     # format, lint, typecheck, offline tests, generated docs, release notices
+just check-pandas  # install the CSV extra and run the same checks
 just build     # build wheel and sdist
-just smoke     # install and exercise the built wheel outside the checkout
+just smoke     # exercise core and pandas wheel installations outside the checkout
 just run search radar
 ```
 
 Unit tests mechanically block network connections. Integration tests that hit
 live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
-Linux and smoke-tests the installed wheel on Linux, macOS, and Windows. The
-full unit and live-service suites currently run on Linux.
+Linux, both with and without pandas, and smoke-tests both installed-wheel
+profiles on Linux, macOS, and Windows. The full unit and live-service suites currently run on Linux. `just setup` restores
+a core-only development environment; `just check-pandas` installs the extra.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.5.0 → usdata-0.6.0}/README.md

@@ -3,22 +3,22 @@
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.5 supports GHCN-Daily, NEXRAD Level II, USGS daily
+> Status: pre-alpha. v0.6 supports GHCN-Daily, NEXRAD Level II, USGS daily
 > values, and CoastWatch SST subsets with provenance, plus Census state/county
-> lookup. Other datasets are planned.
+> lookup and optional pandas CSV readers. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
 
 <!-- registry:start -->
-| Provider | Available | Stub | Planned | Next up (0.6) | Datasets |
+| Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | `gfs`, `hrrr`, `oisst`, `etopo` | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | — | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
 | [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
-| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | `gpm-imerg` | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
 | [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
 
 Available datasets are in `code`, stubs in _italics_; planned ones are counted. Available means implemented in this source checkout; consult the [releases](https://github.com/jakeryderv/usdata/releases) for published support. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
@@ -109,6 +109,14 @@ sources:
     end: 2024-05-31
 ```
 
+## Opening CSV data
+
+`FetchedAsset.open()` is available since v0.6 with the optional pandas
+extra (`pip install "usdata[pandas]"`). It reads cached CSV into a DataFrame,
+preserves identifier strings, and keeps CoastWatch units as metadata.
+See the [reader reference](docs/reference/readers.md)
+and [fetch → open → analyze example](examples/sst-analysis/README.md).
+
 ## Development
 
 Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
@@ -117,16 +125,18 @@ Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
-just check     # format, lint, typecheck, offline tests, generated docs
+just check     # format, lint, typecheck, offline tests, generated docs, release notices
+just check-pandas  # install the CSV extra and run the same checks
 just build     # build wheel and sdist
-just smoke     # install and exercise the built wheel outside the checkout
+just smoke     # exercise core and pandas wheel installations outside the checkout
 just run search radar
 ```
 
 Unit tests mechanically block network connections. Integration tests that hit
 live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
-Linux and smoke-tests the installed wheel on Linux, macOS, and Windows. The
-full unit and live-service suites currently run on Linux.
+Linux, both with and without pandas, and smoke-tests both installed-wheel
+profiles on Linux, macOS, and Windows. The full unit and live-service suites currently run on Linux. `just setup` restores
+a core-only development environment; `just check-pandas` installs the extra.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.5.0 → usdata-0.6.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.5.0"
+version = "0.6.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -30,6 +30,9 @@ dependencies = [
 name = "Jake Van Slyke"
 email = "jakervanslyke@gmail.com"
 
+[project.optional-dependencies]
+pandas = ["pandas>=3.0"]
+
 [project.urls]
 Homepage = "https://github.com/jakeryderv/usdata"
 Repository = "https://github.com/jakeryderv/usdata"

{usdata-0.5.0 → usdata-0.6.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.5.0"
+version = "0.6.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -22,6 +22,9 @@ dependencies = [
     "typer>=0.12",
 ]
 
+[project.optional-dependencies]
+pandas = ["pandas>=3.0"]
+
 [project.urls]
 Homepage = "https://github.com/jakeryderv/usdata"
 Repository = "https://github.com/jakeryderv/usdata"

{usdata-0.5.0 → usdata-0.6.0}/src/usdata/data/registry.yaml

@@ -293,7 +293,7 @@ datasets:
     provider: noaa
     status: planned
     domain: weather-models
-    target: "0.6"
+    target: later
     title: HRRR Forecast Model Output
     description: >-
       High-Resolution Rapid Refresh 3 km hourly forecasts in GRIB2 from the
@@ -310,7 +310,7 @@ datasets:
     provider: noaa
     status: planned
     domain: weather-models
-    target: "0.6"
+    target: later
     title: GFS Forecast Model Output
     description: >-
       Global Forecast System output in GRIB2 from the public noaa-gfs-bdp-pds
@@ -342,7 +342,7 @@ datasets:
     provider: noaa
     status: planned
     domain: ocean-physics
-    target: "0.6"
+    target: later
     title: OISST Daily Sea Surface Temperature
     description: >-
       Optimum Interpolation SST v2.1: daily global 0.25 degree analysis since
@@ -375,7 +375,7 @@ datasets:
     provider: noaa
     status: planned
     domain: bathymetry
-    target: "0.6"
+    target: later
     title: ETOPO 2022 Global Relief
     description: >-
       Global topography and bathymetry at 15, 30, and 60 arc-seconds as
@@ -591,7 +591,7 @@ datasets:
     provider: nasa
     status: planned
     domain: weather-satellites
-    target: "0.6"
+    target: later
     title: GPM IMERG Precipitation
     description: >-
       Global half-hourly and daily merged satellite precipitation estimates.

{usdata-0.5.0 → usdata-0.6.0}/src/usdata/fetch.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 from pathlib import Path
+from typing import Any
 
 from pydantic import BaseModel
 
@@ -25,6 +26,27 @@ class FetchedAsset(BaseModel):
     provenance: Provenance
     from_cache: bool
 
+    def open(
+        self,
+        *,
+        reader: str | None = None,
+        dtype: dict[str, str] | None = None,
+        parse_dates: list[str] | None = None,
+        usecols: list[str] | None = None,
+        nrows: int | None = None,
+    ) -> Any:
+        """Open this local CSV as a DataFrame; requires the ``pandas`` extra.
+
+        ERDDAP units are kept in ``frame.attrs["units"]`` and source provenance
+        in ``frame.attrs["usdata"]``. See ``usdata.readers.open_asset`` for options.
+        Cached files and provenance sidecars are never changed.
+        """
+        from usdata.readers import open_asset
+
+        return open_asset(
+            self, reader=reader, dtype=dtype, parse_dates=parse_dates, usecols=usecols, nrows=nrows
+        )
+
 
 def _fetch_asset(
     dataset: Dataset,

usdata-0.6.0/src/usdata/readers.py

@@ -0,0 +1,103 @@
+"""Optional readers for local fetched files; never fetch or modify cached bytes."""
+
+from __future__ import annotations
+
+import csv
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+from usdata.models import Protocol
+
+if TYPE_CHECKING:
+    from usdata.fetch import FetchedAsset
+
+CSV_MEDIA_TYPES = {"text/csv", "application/csv"}
+IDENTIFIER_COLUMNS = {
+    "station",
+    "station_id",
+    "site_no",
+    "monitoring_location_id",
+    "parameter_code",
+    "statistic_id",
+}
+
+
+class MissingReaderDependency(ImportError):
+    """The optional dependency required to open an asset is not installed."""
+
+
+class UnsupportedFormat(ValueError):
+    """No reader is implemented for this asset's format."""
+
+
+def open_asset(
+    fetched: FetchedAsset,
+    *,
+    reader: str | None = None,
+    dtype: dict[str, str] | None = None,
+    parse_dates: list[str] | None = None,
+    usecols: list[str] | None = None,
+    nrows: int | None = None,
+) -> Any:
+    """Read a local CSV into a pandas DataFrame, retaining units and provenance.
+
+    Infer ``csv`` or ``erddap-csv`` from media type and protocol, or use an
+    explicit reader for ambiguous metadata. Identifier columns default to pandas
+    strings; explicit dtype entries override those defaults. Dates remain strings
+    unless named in parse_dates. No checksum verification or downloading occurs.
+    """
+    if reader is None:
+        media_type = (fetched.asset.media_type or "").split(";", 1)[0].strip().lower()
+        if media_type not in CSV_MEDIA_TYPES:
+            raise UnsupportedFormat(
+                f"no reader for {fetched.asset.media_type!r}; supported formats are CSV and "
+                "ERDDAP CSV. For a known CSV with ambiguous metadata, pass reader='csv' "
+                "or reader='erddap-csv'; otherwise use fetched.path with a format-specific reader"
+            )
+        reader = "erddap-csv" if fetched.asset.protocol is Protocol.ERDDAP else "csv"
+    if reader not in {"csv", "erddap-csv"}:
+        raise UnsupportedFormat(f"unsupported reader {reader!r}; use 'csv' or 'erddap-csv'")
+    try:
+        pandas = import_module("pandas")
+    except ModuleNotFoundError as error:
+        if error.name != "pandas":
+            raise
+        raise MissingReaderDependency(
+            'CSV reading requires pandas; install it with: pip install "usdata[pandas]" '
+            '(or uv add "usdata[pandas]")'
+        ) from error
+
+    # Pass a file object to pandas: reading a fetched asset is strictly local.
+    with fetched.path.open(encoding="utf-8-sig", newline="") as stream:
+        records = csv.reader(stream)
+        columns = next(records, [])
+        if (
+            not columns
+            or any(not column for column in columns)
+            or len(set(columns)) != len(columns)
+        ):
+            raise ValueError("CSV must have a non-empty header with unique column names")
+        units = {}
+        if reader == "erddap-csv":
+            values = next(records, [])
+            if len(values) != len(columns):
+                raise ValueError("ERDDAP CSV must have a units row matching the header")
+            units = dict(zip(columns, values, strict=True))
+        types = {name: "string" for name in columns if name.casefold() in IDENTIFIER_COLUMNS}
+        types.update(dtype or {})
+        frame = pandas.read_csv(
+            stream,
+            header=None,
+            names=columns,
+            dtype=types,
+            parse_dates=parse_dates,
+            usecols=usecols,
+            nrows=nrows,
+        )
+    if units:
+        frame.attrs["units"] = {name: units[name] for name in frame.columns}
+    frame.attrs["usdata"] = {
+        "asset_id": fetched.asset.id,
+        "provenance": fetched.provenance.model_dump(mode="json"),
+    }
+    return frame