usdata 0.7.0__tar.gz → 0.8.0__tar.gz

{usdata-0.7.0 → usdata-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.7.0
+Version: 0.8.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -14,11 +14,16 @@ Requires-Dist: httpx>=0.28.1
 Requires-Dist: pydantic>=2.7
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: typer>=0.12
+Requires-Dist: xarray>=2025.1 ; extra == 'netcdf'
+Requires-Dist: h5netcdf[h5py]>=1.8.1 ; extra == 'netcdf'
 Requires-Dist: pandas>=3.0 ; extra == 'pandas'
+Requires-Dist: xradar>=0.12.0 ; extra == 'radar'
 Requires-Python: >=3.11
 Project-URL: Homepage, https://github.com/jakeryderv/usdata
 Project-URL: Repository, https://github.com/jakeryderv/usdata
+Provides-Extra: netcdf
 Provides-Extra: pandas
+Provides-Extra: radar
 Description-Content-Type: text/markdown
 
 # usdata
@@ -26,10 +31,11 @@ Description-Content-Type: text/markdown
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.7 supports GHCN-Daily, GSOM monthly summaries,
-> NEXRAD Level II, USGS daily values, and CoastWatch SST subsets with provenance,
-> plus Census state/county lookup, optional pandas CSV readers, and terminal
-> download progress. Other datasets are planned.
+> Status: pre-alpha. v0.8 supports GHCN-Daily, GSOM monthly summaries,
+> NEXRAD Level II, GOES ABI CONUS imagery, Storm Events annual archives,
+> USGS daily values, and CoastWatch SST subsets with provenance. It includes
+> optional CSV, radar, and NetCDF4 readers, six executed notebooks, Census
+> state/county lookup, and terminal download progress. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
@@ -37,7 +43,7 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 <!-- registry:start -->
 | Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 4 | 0 | 25 | — | `ghcn-daily`, `gsom`, `nexrad-level2`, `coastwatch-sst`, +25 planned |
+| [NOAA](docs/providers/noaa.md) | 6 | 0 | 23 | — | `ghcn-daily`, `gsom`, `storm-events`, `nexrad-level2`, `goes-abi`, `coastwatch-sst`, +23 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
@@ -94,6 +100,17 @@ usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
 usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
+Storm Events bulk access is available since v0.8. Dates select complete
+annual details archives; filter rows locally after opening the gzip CSV. For example:
+
+```sh
+usdata fetch noaa:storm-events --start 2024-05-01 --end 2024-05-31 --dry-run
+```
+
+This lists the entire 2024 archive. Location and variable filters are rejected;
+see the [executed Storm Events notebook](examples/storm-events/example.ipynb)
+for local filtering and reporting limitations.
+
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
@@ -143,6 +160,15 @@ size. Adapters that assemble files from metadata requests show asset-level progr
 Use `--no-progress` to disable it. Progress is automatically disabled when either
 stdout or stderr is redirected; existing output lines and exit codes are unchanged.
 
+For single-channel GOES CONUS imagery (available since v0.8):
+
+```sh
+usdata fetch noaa:goes-abi --start 2024-05-06T12:01:18.1Z --end 2024-05-06T12:01:18.1Z -p satellite=18 -p channel=6
+```
+
+The download is a whole NetCDF scene. See [GOES access notes](docs/providers/noaa.md#goes-abi-conus-imagery)
+for supported selectors and scan-start time semantics.
+
 ## Opening CSV data
 
 `FetchedAsset.open()` is available since v0.6 with the optional pandas
@@ -151,16 +177,32 @@ preserves identifier strings, and keeps CoastWatch units as metadata.
 See the [reader reference](docs/reference/readers.md)
 and [fetch → open → analyze example](examples/sst-analysis/README.md).
 
+The [examples directory](examples/README.md) contains executed Jupyter notebooks
+with saved data previews, small plots, and source provenance. Start with weather
+and streamflow for manifest workflows, SST for gridded CSV reading, or monthly
+climate for GSOM observations.
+
+NetCDF4 scene opening is available since v0.8 with `usdata[netcdf]`.
+See the executed [GOES infrared notebook](examples/goes-imagery/example.ipynb).
+
 ## Development
 
 Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 
+`just setup` uses the tested Python 3.14.7 pin in `.python-version`. Older Linux
+uv Python 3.14 builds can crash during NumPy array operations; see
+[the upstream fix](https://github.com/astral-sh/python-build-standalone/issues/991).
+
 ```sh
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
 just check     # format, lint, typecheck, offline tests, generated docs, release notices
 just check-pandas  # install the CSV extra and run the same checks
+just check-radar   # install the radar extra and run the same checks
+just check-netcdf  # install the NetCDF4 extra and run the same checks
+just notebooks    # launch the optional Jupyter examples environment
+just run-notebooks # execute notebooks live in fresh kernels and temporary caches
 just build     # build wheel and sdist
 just smoke     # exercise core and pandas wheel installations outside the checkout
 just run search radar
@@ -168,9 +210,11 @@ just run search radar
 
 Unit tests mechanically block network connections. Integration tests that hit
 live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
-Linux, both with and without pandas, and smoke-tests both installed-wheel
-profiles on Linux, macOS, and Windows. The full unit and live-service suites currently run on Linux. `just setup` restores
-a core-only development environment; `just check-pandas` installs the extra.
+Linux with core-only, pandas, radar, and NetCDF dependency profiles. Installed-wheel
+checks cover all four profiles on Linux, macOS, and Windows. The full unit and
+live-service suites run on Linux. `just setup` restores a core-only development
+environment; the `check-pandas`, `check-radar`, and `check-netcdf` commands install
+their respective extras.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.7.0 → usdata-0.8.0}/README.md

@@ -3,10 +3,11 @@
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.7 supports GHCN-Daily, GSOM monthly summaries,
-> NEXRAD Level II, USGS daily values, and CoastWatch SST subsets with provenance,
-> plus Census state/county lookup, optional pandas CSV readers, and terminal
-> download progress. Other datasets are planned.
+> Status: pre-alpha. v0.8 supports GHCN-Daily, GSOM monthly summaries,
+> NEXRAD Level II, GOES ABI CONUS imagery, Storm Events annual archives,
+> USGS daily values, and CoastWatch SST subsets with provenance. It includes
+> optional CSV, radar, and NetCDF4 readers, six executed notebooks, Census
+> state/county lookup, and terminal download progress. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
@@ -14,7 +15,7 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 <!-- registry:start -->
 | Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 4 | 0 | 25 | — | `ghcn-daily`, `gsom`, `nexrad-level2`, `coastwatch-sst`, +25 planned |
+| [NOAA](docs/providers/noaa.md) | 6 | 0 | 23 | — | `ghcn-daily`, `gsom`, `storm-events`, `nexrad-level2`, `goes-abi`, `coastwatch-sst`, +23 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
@@ -71,6 +72,17 @@ usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
 usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
+Storm Events bulk access is available since v0.8. Dates select complete
+annual details archives; filter rows locally after opening the gzip CSV. For example:
+
+```sh
+usdata fetch noaa:storm-events --start 2024-05-01 --end 2024-05-31 --dry-run
+```
+
+This lists the entire 2024 archive. Location and variable filters are rejected;
+see the [executed Storm Events notebook](examples/storm-events/example.ipynb)
+for local filtering and reporting limitations.
+
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
@@ -120,6 +132,15 @@ size. Adapters that assemble files from metadata requests show asset-level progr
 Use `--no-progress` to disable it. Progress is automatically disabled when either
 stdout or stderr is redirected; existing output lines and exit codes are unchanged.
 
+For single-channel GOES CONUS imagery (available since v0.8):
+
+```sh
+usdata fetch noaa:goes-abi --start 2024-05-06T12:01:18.1Z --end 2024-05-06T12:01:18.1Z -p satellite=18 -p channel=6
+```
+
+The download is a whole NetCDF scene. See [GOES access notes](docs/providers/noaa.md#goes-abi-conus-imagery)
+for supported selectors and scan-start time semantics.
+
 ## Opening CSV data
 
 `FetchedAsset.open()` is available since v0.6 with the optional pandas
@@ -128,16 +149,32 @@ preserves identifier strings, and keeps CoastWatch units as metadata.
 See the [reader reference](docs/reference/readers.md)
 and [fetch → open → analyze example](examples/sst-analysis/README.md).
 
+The [examples directory](examples/README.md) contains executed Jupyter notebooks
+with saved data previews, small plots, and source provenance. Start with weather
+and streamflow for manifest workflows, SST for gridded CSV reading, or monthly
+climate for GSOM observations.
+
+NetCDF4 scene opening is available since v0.8 with `usdata[netcdf]`.
+See the executed [GOES infrared notebook](examples/goes-imagery/example.ipynb).
+
 ## Development
 
 Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 
+`just setup` uses the tested Python 3.14.7 pin in `.python-version`. Older Linux
+uv Python 3.14 builds can crash during NumPy array operations; see
+[the upstream fix](https://github.com/astral-sh/python-build-standalone/issues/991).
+
 ```sh
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
 just check     # format, lint, typecheck, offline tests, generated docs, release notices
 just check-pandas  # install the CSV extra and run the same checks
+just check-radar   # install the radar extra and run the same checks
+just check-netcdf  # install the NetCDF4 extra and run the same checks
+just notebooks    # launch the optional Jupyter examples environment
+just run-notebooks # execute notebooks live in fresh kernels and temporary caches
 just build     # build wheel and sdist
 just smoke     # exercise core and pandas wheel installations outside the checkout
 just run search radar
@@ -145,9 +182,11 @@ just run search radar
 
 Unit tests mechanically block network connections. Integration tests that hit
 live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
-Linux, both with and without pandas, and smoke-tests both installed-wheel
-profiles on Linux, macOS, and Windows. The full unit and live-service suites currently run on Linux. `just setup` restores
-a core-only development environment; `just check-pandas` installs the extra.
+Linux with core-only, pandas, radar, and NetCDF dependency profiles. Installed-wheel
+checks cover all four profiles on Linux, macOS, and Windows. The full unit and
+live-service suites run on Linux. `just setup` restores a core-only development
+environment; the `check-pandas`, `check-radar`, and `check-netcdf` commands install
+their respective extras.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.7.0 → usdata-0.8.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.7.0"
+version = "0.8.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -32,6 +32,11 @@ email = "jakervanslyke@gmail.com"
 
 [project.optional-dependencies]
 pandas = ["pandas>=3.0"]
+radar = ["xradar>=0.12.0"]
+netcdf = [
+    "xarray>=2025.1",
+    "h5netcdf[h5py]>=1.8.1",
+]
 
 [project.urls]
 Homepage = "https://github.com/jakeryderv/usdata"
@@ -48,6 +53,13 @@ dev = [
     "respx>=0.23.1",
     "ruff>=0.6",
 ]
+examples = [
+    "ipykernel>=6.29",
+    "jupyterlab>=4.3",
+    "matplotlib>=3.9",
+    "nbclient>=0.10",
+    "pandas>=3.0",
+]
 
 [build-system]
 requires = ["uv_build>=0.12.5,<0.13.0"]

{usdata-0.7.0 → usdata-0.8.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.7.0"
+version = "0.8.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -24,6 +24,8 @@ dependencies = [
 
 [project.optional-dependencies]
 pandas = ["pandas>=3.0"]
+radar = ["xradar>=0.12.0"]
+netcdf = ["xarray>=2025.1", "h5netcdf[h5py]>=1.8.1"]
 
 [project.urls]
 Homepage = "https://github.com/jakeryderv/usdata"
@@ -40,6 +42,13 @@ dev = [
     "respx>=0.23.1",
     "ruff>=0.6",
 ]
+examples = [
+    "ipykernel>=6.29",
+    "jupyterlab>=4.3",
+    "matplotlib>=3.9",
+    "nbclient>=0.10",
+    "pandas>=3.0",
+]
 
 [build-system]
 requires = ["uv_build>=0.12.5,<0.13.0"]

usdata-0.8.0/src/usdata/_netcdf.py

@@ -0,0 +1,36 @@
+"""Local, eagerly loaded NetCDF4 reading behind the netcdf extra."""
+
+from __future__ import annotations
+
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+from usdata.readers import MissingReaderDependency
+
+if TYPE_CHECKING:
+    from usdata.fetch import FetchedAsset
+
+
+def open_netcdf(fetched: FetchedAsset) -> Any:
+    """Load a NetCDF4 root Dataset, then close every source file handle."""
+    try:
+        xarray = import_module("xarray")
+        import_module("h5netcdf")
+        import_module("h5py")
+    except ModuleNotFoundError as error:
+        if error.name not in {"xarray", "h5netcdf", "h5py"}:
+            raise
+        raise MissingReaderDependency(
+            'NetCDF4 reading requires xarray and h5netcdf; install: pip install "usdata[netcdf]"'
+        ) from error
+    # A local file object and fixed engine prevent interpretation as an OPeNDAP URL.
+    with (
+        fetched.path.open("rb") as stream,
+        xarray.open_dataset(stream, engine="h5netcdf", chunks=None) as dataset,
+    ):
+        dataset.load()
+    dataset.attrs["usdata"] = {
+        "asset_id": fetched.asset.id,
+        "provenance": fetched.provenance.model_dump(mode="json"),
+    }
+    return dataset

usdata-0.8.0/src/usdata/_radar.py

@@ -0,0 +1,73 @@
+"""Local NEXRAD Level II decoding behind the radar extra."""
+
+from __future__ import annotations
+
+import bz2
+import gzip
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+from usdata.readers import MissingReaderDependency
+
+# NOAA RDA/RPG ICD 2620002Y, Table XVII-I notes 21 and 30.
+MOMENT_FLAG_COUNTS = {
+    "DBZH": 2,
+    "VRADH": 2,
+    "WRADH": 2,
+    "ZDR": 2,
+    "PHIDP": 2,
+    "RHOHV": 2,
+    "CCORH": 8,
+}
+
+if TYPE_CHECKING:
+    from usdata.fetch import FetchedAsset
+
+
+def open_nexrad(fetched: FetchedAsset) -> Any:
+    """Decode a local volume into a fully loaded xarray DataTree."""
+    try:
+        xradar = import_module("xradar")
+    except ModuleNotFoundError as error:
+        if error.name != "xradar":
+            raise
+        raise MissingReaderDependency(
+            'NEXRAD reading requires xradar; install it with: pip install "usdata[radar]" '
+            '(or uv add "usdata[radar]")'
+        ) from error
+
+    # Bytes prevent remote URL interpretation and work across the backend's
+    # repeated sweep reads. Compressed source files stay unchanged in the cache.
+    content = fetched.path.read_bytes()
+    if content.startswith(b"\x1f\x8b"):
+        content = gzip.decompress(content)
+    elif content.startswith(b"BZh"):
+        content = bz2.decompress(content)
+    radar = xradar.io.open_nexradlevel2_datatree(content, incomplete_sweep="pad")
+    try:
+        radar.load()
+    finally:
+        radar.close()
+    for node in radar.subtree:
+        for name, variable in node.ds.variables.items():
+            # The backend records the entire input byte string as `source`.
+            # Provenance below is the durable reference, not that decoder buffer.
+            variable.encoding.pop("source", None)
+            if name in MOMENT_FLAG_COUNTS and "range" in variable.dims:
+                scale = variable.encoding.get("scale_factor")
+                offset = variable.encoding.get("add_offset")
+                if scale is not None and offset is not None:
+                    # xradar 0.12 does not supply _FillValue for NEXRAD flags.
+                    # Compare using each moment's native scale, not fixed units.
+                    data = node[name]
+                    valid = data.notnull()
+                    for code in range(MOMENT_FLAG_COUNTS[name]):
+                        valid = valid & (data != offset + code * scale)
+                    masked = data.where(valid)
+                    masked.encoding = data.encoding.copy()
+                    node[name] = masked
+    radar.attrs["usdata"] = {
+        "asset_id": fetched.asset.id,
+        "provenance": fetched.provenance.model_dump(mode="json"),
+    }
+    return radar

{usdata-0.7.0 → usdata-0.8.0}/src/usdata/data/registry.yaml

@@ -105,21 +105,22 @@ datasets:
 
   - id: noaa:goes-abi
     provider: noaa
-    status: planned
+    status: available
     domain: weather-satellites
-    target: later
-    title: GOES-R ABI Satellite Imagery
-    description: >-
-      Advanced Baseline Imager products from GOES-16, 18, and 19 in the public
-      noaa-goes16/18/19 S3 buckets, laid out as PRODUCT/YYYY/DDD/HH/ with one
-      NetCDF per scan (for example ABI-L2-CMIPC). Same anonymous S3 pattern as
-      NEXRAD, plus product, satellite, and channel selection.
-    keywords: [satellite, imagery, goes, abi, clouds, fire, radiance, netcdf]
+    since: "0.8"
+    title: GOES-R ABI CONUS Cloud and Moisture Imagery
+    description: >-
+      Single-channel CONUS Cloud and Moisture Imagery (ABI-L2-CMIPC) from
+      GOES-16, 17, 18, and 19 in anonymous NOAA S3 buckets. Select an explicit
+      satellite, channel, and scan-start interval; each asset is a complete
+      NetCDF scene with no geographic or variable subsetting.
+    keywords: [satellite, imagery, goes, abi, clouds, infrared, reflectance, netcdf, conus]
     protocol: s3
     homepage: https://registry.opendata.aws/noaa-goes/
     license: US Government Work (public domain)
-    temporal_extent: { start: "2017-01-01T00:00:00Z" }
-    capabilities: { spatial_subset: false, temporal_subset: true, variable_subset: true }
+    temporal_extent: { start: "2017-02-28T00:00:00Z" }
+    capabilities: { spatial_subset: false, temporal_subset: true, variable_subset: false }
+    adapter: usdata.providers.noaa.goes:GoesAbi
 
   - id: noaa:goes-glm
     provider: noaa
@@ -139,21 +140,24 @@ datasets:
 
   - id: noaa:storm-events
     provider: noaa
-    status: planned
+    status: available
     domain: severe-weather
-    target: later
+    since: "0.8"
     title: Storm Events Database
     description: >-
-      NCEI's record of significant weather events since 1950 (tornadoes, hail,
-      wind, floods, and more) with locations, damage, and narratives. Published
-      as per-year gzipped CSV files (details, fatalities, locations) in a plain
-      HTTPS directory; the first NCEI bulk-directory dataset.
+      NCEI's significant-weather event details since 1950, with locations,
+      impacts, and narratives. Anonymous whole-year gzipped CSV archives;
+      select the latest creation-date revision for each requested year.
+      No server-side row, location, or variable subsetting. Historical event
+      coverage and reporting practices vary; fatalities and locations tables
+      are separate products not included by this adapter.
     keywords: [storms, tornado, hail, wind, flood, damage, severe weather, events]
     protocol: http
-    homepage: https://www.ncdc.noaa.gov/stormevents/
+    homepage: https://www.ncei.noaa.gov/access/storm-events-database/
     license: US Government Work (public domain)
     temporal_extent: { start: "1950-01-01T00:00:00Z" }
-    capabilities: { spatial_subset: false, temporal_subset: true, variable_subset: false }
+    capabilities: { spatial_subset: false, temporal_subset: false, variable_subset: false }
+    adapter: usdata.providers.noaa.storm_events:StormEvents
 
   - id: noaa:hurdat2
     provider: noaa

{usdata-0.7.0 → usdata-0.8.0}/src/usdata/fetch.py

@@ -35,10 +35,12 @@ class FetchedAsset(BaseModel):
         usecols: list[str] | None = None,
         nrows: int | None = None,
     ) -> Any:
-        """Open this local CSV as a DataFrame; requires the ``pandas`` extra.
+        """Open local data with an optional ``pandas``, ``radar``, or ``netcdf`` reader.
 
         ERDDAP units are kept in ``frame.attrs["units"]`` and source provenance
-        in ``frame.attrs["usdata"]``. See ``usdata.readers.open_asset`` for options.
+        in ``frame.attrs["usdata"]``. NEXRAD returns a xarray DataTree with provenance
+        in ``radar.attrs["usdata"]``. NetCDF4 returns a loaded xarray Dataset with
+        matching provenance in its attributes. See ``usdata.readers.open_asset`` for options.
         Cached files and provenance sidecars are never changed.
         """
         from usdata.readers import open_asset

usdata-0.8.0/src/usdata/providers/noaa/goes.py

@@ -0,0 +1,126 @@
+"""GOES ABI CONUS Cloud and Moisture Imagery from anonymous NOAA S3 buckets.
+
+Require ``satellite`` (16, 17, 18, or 19), ``channel`` (1--16 or C01--C16), and
+both timestamps. The optional ``product`` must be ``ABI-L2-CMIPC``. Select whole
+single-channel NetCDF files by inclusive scan-start time, never by scan overlap.
+Geographic and variable subsetting are not available for these archived files.
+"""
+
+from __future__ import annotations
+
+import re
+from datetime import UTC, datetime, timedelta
+from pathlib import Path
+
+import httpx
+
+from usdata.models import Asset, Dataset, Protocol, Query, TimeRange
+from usdata.protocols import http, s3
+from usdata.providers.base import Provider, QueryError
+
+PRODUCT = "ABI-L2-CMIPC"
+PUBLIC_START = datetime(2017, 2, 28, tzinfo=UTC)
+KEY_RE = re.compile(
+    r"OR_ABI-L2-CMIPC-M[346]C(?P<channel>0[1-9]|1[0-6])_G(?P<satellite>1[6-9])"
+    r"_s(?P<start>\d{14})_e(?P<end>\d{14})_c\d{14}\.nc"
+)
+
+
+def _timestamp(raw: str) -> datetime:
+    stamp = datetime.strptime(raw, "%Y%j%H%M%S%f").replace(tzinfo=UTC)
+    # strptime accepts day 366 in a non-leap year by spilling into the next year.
+    if stamp.strftime("%Y%j%H%M%S") + str(stamp.microsecond // 100000) != raw:
+        raise ValueError("invalid ABI timestamp")
+    return stamp
+
+
+def _number(raw: object, label: str, low: int, high: int) -> int:
+    if isinstance(raw, bool) or not isinstance(raw, (str, int)):
+        raise QueryError(f"{label} must be an integer from {low} to {high}")
+    text = str(raw).strip()
+    if label == "channel" and text.startswith("C"):
+        text = text[1:]
+    if not text.isascii() or not text.isdigit() or not low <= int(text) <= high:
+        raise QueryError(f"{label} must be an integer from {low} to {high}")
+    return int(text)
+
+
+class GoesAbi(Provider):
+    """Single-channel CONUS ABI imagery; params: satellite, channel, product."""
+
+    def __init__(self, dataset: Dataset, client: httpx.Client | None = None) -> None:
+        super().__init__(dataset)
+        self._client = client
+        self._owns_client = client is None
+
+    def close(self) -> None:
+        """Close the owned HTTP client, leaving injected clients to their caller."""
+        if self._owns_client and self._client is not None:
+            self._client.close()
+            self._client = None
+
+    def _http(self) -> httpx.Client:
+        if self._client is None:
+            self._client = http.client()
+        return self._client
+
+    def list_assets(self, query: Query) -> list[Asset]:
+        """List complete scenes whose scan starts fall inside the inclusive UTC interval."""
+        if unknown := set(query.params) - {"satellite", "channel", "product"}:
+            raise QueryError(f"unsupported GOES params: {', '.join(sorted(unknown))}")
+        if query.params.get("product", PRODUCT) != PRODUCT:
+            raise QueryError(f"only product={PRODUCT} is supported")
+        if query.bbox is not None:
+            raise QueryError(
+                "GOES imagery has no geographic subsetting; omit location/bbox/lat/lon"
+            )
+        if query.text:
+            raise QueryError("GOES imagery has no text filtering; select satellite and channel")
+        if query.variables:
+            raise QueryError("GOES imagery has no variable subsetting; select a channel instead")
+        if query.time is None or query.time.start is None or query.time.end is None:
+            raise QueryError(f"{self.dataset.id} requires both start and end times")
+        satellite = _number(query.params.get("satellite"), "satellite", 16, 19)
+        channel = _number(query.params.get("channel"), "channel", 1, 16)
+        start = query.time.start.replace(tzinfo=query.time.start.tzinfo or UTC).astimezone(UTC)
+        end = query.time.end.replace(tzinfo=query.time.end.tzinfo or UTC).astimezone(UTC)
+        if end < PUBLIC_START:
+            raise QueryError("GOES CMIPC public observations begin on 2017-02-28")
+        start = max(start, PUBLIC_START)
+        hour = start.replace(minute=0, second=0, microsecond=0)
+        bucket = f"noaa-goes{satellite}"
+        assets: dict[str, Asset] = {}
+        while hour <= end:
+            prefix = f"{PRODUCT}/{hour:%Y/%j/%H}/"
+            for obj in s3.list_objects(bucket, prefix, self._http()):
+                if not obj.key.startswith(prefix):
+                    continue
+                name = obj.key.removeprefix(prefix)
+                match = KEY_RE.fullmatch(name)
+                if (
+                    match is None
+                    or int(match["satellite"]) != satellite
+                    or int(match["channel"]) != channel
+                ):
+                    continue
+                try:
+                    scan_start, scan_end = _timestamp(match["start"]), _timestamp(match["end"])
+                except ValueError:
+                    continue
+                if scan_end < scan_start or not start <= scan_start <= end:
+                    continue
+                assets[obj.key] = Asset(
+                    id=name,
+                    dataset_id=self.dataset.id,
+                    href=f"s3://{bucket}/{obj.key}",
+                    protocol=Protocol.S3,
+                    media_type="application/x-netcdf",
+                    size=obj.size,
+                    time=TimeRange(start=scan_start, end=scan_end),
+                )
+            hour += timedelta(hours=1)
+        return sorted(assets.values(), key=lambda asset: asset.id)
+
+    def fetch(self, asset: Asset, dest: Path) -> Path:
+        """Download the complete archived NetCDF object without modification."""
+        return s3.download(asset.href, dest, self._http())

usdata-0.8.0/src/usdata/providers/noaa/storm_events.py

@@ -0,0 +1,134 @@
+"""Annual Storm Events details archives from the NCEI bulk directory.
+
+Both dates are required. Every UTC calendar year touched by the interval is
+selected in full; geographic, variable, text, and provider-specific filters are
+unsupported. The most recently created supported details file wins per year.
+"""
+
+from __future__ import annotations
+
+import re
+from datetime import UTC, datetime
+from html.parser import HTMLParser
+from pathlib import Path
+
+import httpx
+
+from usdata.models import Asset, Dataset, Protocol, Query, TimeRange
+from usdata.protocols import http
+from usdata.providers.base import Provider, QueryError
+
+DIRECTORY_URL = "https://www.ncei.noaa.gov/pub/data/swdi/stormevents/csvfiles/"
+DETAILS_NAME = re.compile(r"StormEvents_details-ftp_v1\.0_d(\d{4})_c(\d{8})\.csv\.gz", re.ASCII)
+
+
+class _Directory(HTMLParser):
+    """Read filenames and exact byte sizes from NCEI's HTML directory table."""
+
+    def __init__(self) -> None:
+        super().__init__()
+        self.files: list[tuple[str, int | None]] = []
+        self._cells: list[str] = []
+        self._name: str | None = None
+        self._in_cell = False
+
+    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
+        if tag == "tr":
+            self._cells, self._name = [], None
+        elif tag == "td":
+            self._cells.append("")
+            self._in_cell = True
+        elif tag == "a":
+            href = dict(attrs).get("href", "") or ""
+            # Only literal local filenames; never follow arbitrary links from a listing.
+            if DETAILS_NAME.fullmatch(href):
+                self._name = href
+
+    def handle_data(self, data: str) -> None:
+        if self._in_cell:
+            self._cells[-1] += data
+
+    def handle_endtag(self, tag: str) -> None:
+        if tag == "td":
+            self._in_cell = False
+        elif tag == "tr" and self._name is not None:
+            size = self._cells[2].strip() if len(self._cells) > 2 else ""
+            self.files.append(
+                (self._name, int(size) if size.isascii() and size.isdigit() else None)
+            )
+
+
+class StormEvents(Provider):
+    """Resolve whole-year details archives; preserve the original gzip bytes."""
+
+    def __init__(self, dataset: Dataset, *, client: httpx.Client | None = None) -> None:
+        super().__init__(dataset)
+        self._client = client
+        self._owns_client = client is None
+
+    def _http(self) -> httpx.Client:
+        if self._client is None:
+            self._client = http.client()
+        return self._client
+
+    def close(self) -> None:
+        """Close an internally created HTTP client; injected clients stay open."""
+        if self._owns_client and self._client is not None:
+            self._client.close()
+            self._client = None
+
+    def list_assets(self, query: Query) -> list[Asset]:
+        """Select the latest supported details revision for every requested year."""
+        if query.params:
+            raise QueryError(f"unsupported Storm Events params: {', '.join(sorted(query.params))}")
+        if query.bbox is not None or query.variables or query.text:
+            raise QueryError(
+                "Storm Events downloads whole annual details files; location/bbox, variables, "
+                "and text filters are unsupported. Filter locally after opening the CSV."
+            )
+        if query.time is None or query.time.start is None or query.time.end is None:
+            raise QueryError(f"{self.dataset.id} requires both start and end dates")
+        start = query.time.start.replace(tzinfo=query.time.start.tzinfo or UTC).astimezone(UTC)
+        end = query.time.end.replace(tzinfo=query.time.end.tzinfo or UTC).astimezone(UTC)
+        if start.year < 1950:
+            raise QueryError("Storm Events annual details files start in 1950")
+        years = range(start.year, end.year + 1)
+        listing = _Directory()
+        listing.feed(http.get(DIRECTORY_URL, self._http()).text)
+        selected: dict[int, tuple[str, int | None]] = {}
+        for name, size in listing.files:
+            match = DETAILS_NAME.fullmatch(name)
+            assert match is not None
+            year = int(match[1])
+            if year not in years:
+                continue
+            try:
+                datetime.strptime(match[2], "%Y%m%d")
+            except ValueError:
+                continue
+            # Fixed-width YYYYMMDD names sort by creation date. Duplicate rows are harmless.
+            if year not in selected or name > selected[year][0]:
+                selected[year] = name, size
+        if missing := [str(year) for year in years if year not in selected]:
+            raise QueryError(
+                "no supported Storm Events details file for year(s): " + ", ".join(missing)
+            )
+        return [
+            Asset(
+                id=selected[year][0],
+                dataset_id=self.dataset.id,
+                href=DIRECTORY_URL + selected[year][0],
+                protocol=Protocol.HTTP,
+                media_type="application/gzip",
+                size=selected[year][1],
+                time=TimeRange(
+                    start=datetime(year, 1, 1, tzinfo=UTC),
+                    end=datetime(year, 12, 31, 23, 59, 59, 999999, tzinfo=UTC),
+                ),
+            )
+            for year in years
+        ]
+
+    def fetch(self, asset: Asset, dest: Path) -> Path:
+        """Download the pinned annual archive, without decompressing or subsetting."""
+        return http.download(asset.href, dest, self._http())

usdata-0.8.0/src/usdata/readers.py

@@ -0,0 +1,141 @@
+"""Optional readers for local fetched files; never fetch or modify cached bytes."""
+
+from __future__ import annotations
+
+import csv
+import gzip
+import io
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+from usdata.models import Protocol
+
+if TYPE_CHECKING:
+    from usdata.fetch import FetchedAsset
+
+NETCDF_MEDIA_TYPES = {"application/x-netcdf", "application/netcdf", "application/x-netcdf4"}
+CSV_MEDIA_TYPES = {"text/csv", "application/csv"}
+GZIP_MEDIA_TYPES = {"application/gzip", "application/x-gzip"}
+IDENTIFIER_COLUMNS = {
+    "station",
+    "station_id",
+    "site_no",
+    "monitoring_location_id",
+    "parameter_code",
+    "statistic_id",
+    "event_id",
+    "episode_id",
+    "state_fips",
+    "cz_fips",
+    "tor_other_cz_fips",
+}
+
+
+class MissingReaderDependency(ImportError):
+    """The optional dependency required to open an asset is not installed."""
+
+
+class UnsupportedFormat(ValueError):
+    """No reader is implemented for this asset's format."""
+
+
+def open_asset(
+    fetched: FetchedAsset,
+    *,
+    reader: str | None = None,
+    dtype: dict[str, str] | None = None,
+    parse_dates: list[str] | None = None,
+    usecols: list[str] | None = None,
+    nrows: int | None = None,
+) -> Any:
+    """Open local CSV, NetCDF4, or NEXRAD data, retaining units and provenance.
+
+    Gzip CSVs are decompressed locally without changing cached bytes.
+    Infer ``csv`` or ``erddap-csv`` from media type and protocol, or use an
+    explicit reader for ambiguous metadata. Identifier columns default to pandas
+    strings; explicit dtype entries override those defaults. Dates remain strings
+    unless named in parse_dates. No checksum verification or downloading occurs.
+    """
+    if reader is None:
+        media_type = (fetched.asset.media_type or "").split(";", 1)[0].strip().lower()
+        gzip_csv = media_type in GZIP_MEDIA_TYPES and fetched.asset.id.lower().endswith(".csv.gz")
+        if fetched.asset.dataset_id == "noaa:nexrad-level2":
+            reader = "nexrad-level2"
+        elif media_type in NETCDF_MEDIA_TYPES:
+            reader = "netcdf"
+        elif media_type in CSV_MEDIA_TYPES or gzip_csv:
+            reader = "erddap-csv" if fetched.asset.protocol is Protocol.ERDDAP else "csv"
+        else:
+            raise UnsupportedFormat(
+                f"no reader for {fetched.asset.media_type!r}; supported formats are CSV, "
+                "ERDDAP CSV, NetCDF4, and NEXRAD Level II. "
+                "For a known CSV with ambiguous metadata, "
+                "pass reader='csv' "
+                "or reader='erddap-csv'; otherwise use fetched.path with a format-specific reader"
+            )
+    if reader == "netcdf":
+        if any(value is not None for value in (dtype, parse_dates, usecols, nrows)):
+            raise ValueError(
+                "CSV options dtype, parse_dates, usecols and nrows do not apply to NetCDF"
+            )
+        from usdata._netcdf import open_netcdf
+
+        return open_netcdf(fetched)
+    if reader == "nexrad-level2":
+        if any(value is not None for value in (dtype, parse_dates, usecols, nrows)):
+            raise ValueError("dtype, parse_dates, usecols, and nrows apply only to CSV readers")
+        from usdata._radar import open_nexrad
+
+        return open_nexrad(fetched)
+    if reader not in {"csv", "erddap-csv"}:
+        raise UnsupportedFormat(
+            f"unsupported reader {reader!r}; use 'csv', 'erddap-csv', 'netcdf', or 'nexrad-level2'"
+        )
+    try:
+        pandas = import_module("pandas")
+    except ModuleNotFoundError as error:
+        if error.name != "pandas":
+            raise
+        raise MissingReaderDependency(
+            'CSV reading requires pandas; install it with: pip install "usdata[pandas]" '
+            '(or uv add "usdata[pandas]")'
+        ) from error
+
+    # Pass a file object to pandas: reading a fetched asset is strictly local.
+    with fetched.path.open("rb") as raw:
+        compressed = raw.read(2) == b"\x1f\x8b"
+        raw.seek(0)
+        binary = gzip.GzipFile(fileobj=raw) if compressed else raw
+        with io.TextIOWrapper(binary, encoding="utf-8-sig", newline="") as stream:
+            records = csv.reader(stream)
+            columns = next(records, [])
+            if (
+                not columns
+                or any(not column for column in columns)
+                or len(set(columns)) != len(columns)
+            ):
+                raise ValueError("CSV must have a non-empty header with unique column names")
+            units = {}
+            if reader == "erddap-csv":
+                values = next(records, [])
+                if len(values) != len(columns):
+                    raise ValueError("ERDDAP CSV must have a units row matching the header")
+                units = dict(zip(columns, values, strict=True))
+            types = {name: "string" for name in columns if name.casefold() in IDENTIFIER_COLUMNS}
+            types.update(dtype or {})
+            frame = pandas.read_csv(
+                stream,
+                header=None,
+                names=columns,
+                dtype=types,
+                parse_dates=parse_dates,
+                usecols=usecols,
+                nrows=nrows,
+            )
+    if units:
+        frame.attrs["units"] = {name: units[name] for name in frame.columns}
+    frame.attrs["usdata"] = {
+        "asset_id": fetched.asset.id,
+        "provenance": fetched.provenance.model_dump(mode="json"),
+    }
+    return frame

usdata-0.7.0/src/usdata/readers.py

@@ -1,103 +0,0 @@
-"""Optional readers for local fetched files; never fetch or modify cached bytes."""
-
-from __future__ import annotations
-
-import csv
-from importlib import import_module
-from typing import TYPE_CHECKING, Any
-
-from usdata.models import Protocol
-
-if TYPE_CHECKING:
-    from usdata.fetch import FetchedAsset
-
-CSV_MEDIA_TYPES = {"text/csv", "application/csv"}
-IDENTIFIER_COLUMNS = {
-    "station",
-    "station_id",
-    "site_no",
-    "monitoring_location_id",
-    "parameter_code",
-    "statistic_id",
-}
-
-
-class MissingReaderDependency(ImportError):
-    """The optional dependency required to open an asset is not installed."""
-
-
-class UnsupportedFormat(ValueError):
-    """No reader is implemented for this asset's format."""
-
-
-def open_asset(
-    fetched: FetchedAsset,
-    *,
-    reader: str | None = None,
-    dtype: dict[str, str] | None = None,
-    parse_dates: list[str] | None = None,
-    usecols: list[str] | None = None,
-    nrows: int | None = None,
-) -> Any:
-    """Read a local CSV into a pandas DataFrame, retaining units and provenance.
-
-    Infer ``csv`` or ``erddap-csv`` from media type and protocol, or use an
-    explicit reader for ambiguous metadata. Identifier columns default to pandas
-    strings; explicit dtype entries override those defaults. Dates remain strings
-    unless named in parse_dates. No checksum verification or downloading occurs.
-    """
-    if reader is None:
-        media_type = (fetched.asset.media_type or "").split(";", 1)[0].strip().lower()
-        if media_type not in CSV_MEDIA_TYPES:
-            raise UnsupportedFormat(
-                f"no reader for {fetched.asset.media_type!r}; supported formats are CSV and "
-                "ERDDAP CSV. For a known CSV with ambiguous metadata, pass reader='csv' "
-                "or reader='erddap-csv'; otherwise use fetched.path with a format-specific reader"
-            )
-        reader = "erddap-csv" if fetched.asset.protocol is Protocol.ERDDAP else "csv"
-    if reader not in {"csv", "erddap-csv"}:
-        raise UnsupportedFormat(f"unsupported reader {reader!r}; use 'csv' or 'erddap-csv'")
-    try:
-        pandas = import_module("pandas")
-    except ModuleNotFoundError as error:
-        if error.name != "pandas":
-            raise
-        raise MissingReaderDependency(
-            'CSV reading requires pandas; install it with: pip install "usdata[pandas]" '
-            '(or uv add "usdata[pandas]")'
-        ) from error
-
-    # Pass a file object to pandas: reading a fetched asset is strictly local.
-    with fetched.path.open(encoding="utf-8-sig", newline="") as stream:
-        records = csv.reader(stream)
-        columns = next(records, [])
-        if (
-            not columns
-            or any(not column for column in columns)
-            or len(set(columns)) != len(columns)
-        ):
-            raise ValueError("CSV must have a non-empty header with unique column names")
-        units = {}
-        if reader == "erddap-csv":
-            values = next(records, [])
-            if len(values) != len(columns):
-                raise ValueError("ERDDAP CSV must have a units row matching the header")
-            units = dict(zip(columns, values, strict=True))
-        types = {name: "string" for name in columns if name.casefold() in IDENTIFIER_COLUMNS}
-        types.update(dtype or {})
-        frame = pandas.read_csv(
-            stream,
-            header=None,
-            names=columns,
-            dtype=types,
-            parse_dates=parse_dates,
-            usecols=usecols,
-            nrows=nrows,
-        )
-    if units:
-        frame.attrs["units"] = {name: units[name] for name in frame.columns}
-    frame.attrs["usdata"] = {
-        "asset_id": fetched.asset.id,
-        "provenance": fetched.provenance.model_dump(mode="json"),
-    }
-    return frame