aihydro-lsh 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- aihydro_lsh-0.2.1/PKG-INFO +270 -0
- aihydro_lsh-0.2.1/README.md +228 -0
- aihydro_lsh-0.2.1/aihydro_lsh/__init__.py +192 -0
- aihydro_lsh-0.2.1/aihydro_lsh/_pipeline.py +372 -0
- aihydro_lsh-0.2.1/aihydro_lsh/batch.py +442 -0
- aihydro_lsh-0.2.1/aihydro_lsh/cli.py +356 -0
- aihydro_lsh-0.2.1/aihydro_lsh/contracts.py +429 -0
- aihydro_lsh-0.2.1/aihydro_lsh/mcp.py +882 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/__init__.py +57 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/_streamflow.py +166 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/climate.py +221 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/dynamic.py +378 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/events.py +379 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/forcing.py +145 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/geology.py +71 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/geomorphic.py +350 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/hydrology.py +360 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/soil.py +157 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/topography.py +90 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/trends.py +324 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/vegetation.py +194 -0
- aihydro_lsh-0.2.1/aihydro_lsh/recipes/watershed.py +89 -0
- aihydro_lsh-0.2.1/aihydro_lsh/reliability.py +83 -0
- aihydro_lsh-0.2.1/aihydro_lsh/visualization.py +640 -0
- aihydro_lsh-0.2.1/aihydro_lsh.egg-info/PKG-INFO +270 -0
- aihydro_lsh-0.2.1/aihydro_lsh.egg-info/SOURCES.txt +45 -0
- aihydro_lsh-0.2.1/aihydro_lsh.egg-info/dependency_links.txt +1 -0
- aihydro_lsh-0.2.1/aihydro_lsh.egg-info/entry_points.txt +5 -0
- aihydro_lsh-0.2.1/aihydro_lsh.egg-info/requires.txt +28 -0
- aihydro_lsh-0.2.1/aihydro_lsh.egg-info/top_level.txt +1 -0
- aihydro_lsh-0.2.1/pyproject.toml +129 -0
- aihydro_lsh-0.2.1/setup.cfg +4 -0
- aihydro_lsh-0.2.1/tests/test_attribute_metadata_coverage.py +118 -0
- aihydro_lsh-0.2.1/tests/test_basin_ref_passthrough.py +117 -0
- aihydro_lsh-0.2.1/tests/test_batch_and_mcp.py +260 -0
- aihydro_lsh-0.2.1/tests/test_conus_parity.py +307 -0
- aihydro_lsh-0.2.1/tests/test_dynamic_recipe.py +294 -0
- aihydro_lsh-0.2.1/tests/test_events_recipe.py +401 -0
- aihydro_lsh-0.2.1/tests/test_geology_tile_status.py +33 -0
- aihydro_lsh-0.2.1/tests/test_layering.py +100 -0
- aihydro_lsh-0.2.1/tests/test_modelled_streamflow_caveats.py +161 -0
- aihydro_lsh-0.2.1/tests/test_product_homogeneity.py +43 -0
- aihydro_lsh-0.2.1/tests/test_recipes_offline.py +493 -0
- aihydro_lsh-0.2.1/tests/test_reliability_matrix.py +47 -0
- aihydro_lsh-0.2.1/tests/test_served_provenance.py +145 -0
- aihydro_lsh-0.2.1/tests/test_streamflow_identity.py +500 -0
- aihydro_lsh-0.2.1/tests/test_trends_recipe.py +269 -0
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: aihydro-lsh
|
|
3
|
+
Version: 0.2.1
|
|
4
|
+
Summary: Global large-sample hydrology — CAMELS-like catchment attributes for any watershed (recipe layer over aihydro-data + aihydro-watershed)
|
|
5
|
+
Author-email: Mohammad Galib <mgalib@purdue.edu>
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/AI-Hydro/aihydro-lsh
|
|
8
|
+
Project-URL: Repository, https://github.com/AI-Hydro/aihydro-lsh
|
|
9
|
+
Keywords: hydrology,CAMELS,large-sample,catchment-attributes,global,watershed,geospatial,agentic
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Science/Research
|
|
12
|
+
Classifier: Topic :: Scientific/Engineering :: Hydrology
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Requires-Python: >=3.10
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
Requires-Dist: aihydro-core<0.3,>=0.2.3
|
|
20
|
+
Requires-Dist: aihydro-data<0.3,>=0.2.3
|
|
21
|
+
Requires-Dist: aihydro-watershed<0.2,>=0.1.3
|
|
22
|
+
Requires-Dist: numpy>=1.24.0
|
|
23
|
+
Requires-Dist: pandas>=2.0.0
|
|
24
|
+
Requires-Dist: geopandas>=0.13.0
|
|
25
|
+
Requires-Dist: shapely>=2.0.0
|
|
26
|
+
Requires-Dist: pydantic>=2.0.0
|
|
27
|
+
Provides-Extra: validate
|
|
28
|
+
Requires-Dist: camels-attrs>=1.0.4; extra == "validate"
|
|
29
|
+
Provides-Extra: mcp
|
|
30
|
+
Requires-Dist: fastmcp>=2.0.0; extra == "mcp"
|
|
31
|
+
Provides-Extra: trends
|
|
32
|
+
Requires-Dist: pymannkendall>=1.4.3; extra == "trends"
|
|
33
|
+
Provides-Extra: dev
|
|
34
|
+
Requires-Dist: pytest>=7.4.0; extra == "dev"
|
|
35
|
+
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
|
|
36
|
+
Requires-Dist: ruff>=0.1.0; extra == "dev"
|
|
37
|
+
Requires-Dist: mypy>=1.0.0; extra == "dev"
|
|
38
|
+
Requires-Dist: build>=1.0.0; extra == "dev"
|
|
39
|
+
Requires-Dist: import-linter>=2.0; extra == "dev"
|
|
40
|
+
Provides-Extra: all
|
|
41
|
+
Requires-Dist: aihydro-lsh[mcp,trends,validate]; extra == "all"
|
|
42
|
+
|
|
43
|
+
# aihydro-lsh
|
|
44
|
+
|
|
45
|
+
**Global large-sample hydrology — CAMELS-like catchment attributes for globally routed watersheds, with source-specific caveats.**
|
|
46
|
+
|
|
47
|
+
A recipe layer over [`aihydro-data`](../aihydro-data/) and [`aihydro-watershed`](../aihydro-watershed/) that computes **101 static scalar attributes** (8 families) plus **daily meteorological forcing timeseries** and **temporal-tier extensions** (trends, annual dynamics, drought/flood events) for basin geometries through region-aware product routing. The [`camels-attrs`](../camels-attrs/) CONUS reference implementation is used to validate recipes against USGS gauge data; use `aihydro_lsh.reliability.reliability_matrix()` for the family/region caveats that qualify global interpretation.
|
|
48
|
+
|
|
49
|
+
## Quick start
|
|
50
|
+
|
|
51
|
+
```python
|
|
52
|
+
import geopandas as gpd
|
|
53
|
+
from shapely.geometry import box
|
|
54
|
+
import aihydro_lsh
|
|
55
|
+
|
|
56
|
+
# Any watershed polygon (EPSG:4326)
|
|
57
|
+
geom = gpd.GeoDataFrame(geometry=[box(-79.5, 38.5, -77.5, 39.5)], crs="EPSG:4326")
|
|
58
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31")
|
|
59
|
+
|
|
60
|
+
print(result.attrs["p_mean"]) # mean daily precip (mm/day)
|
|
61
|
+
print(result.attrs["elev_mean"]) # mean elevation (m)
|
|
62
|
+
print(result.attrs["geol_porosity"]) # GLHYMPS porosity
|
|
63
|
+
df = result.to_dataframe() # single-row DataFrame, 101 columns
|
|
64
|
+
|
|
65
|
+
# Full provenance report (agent-relay verbatim — package is the authority)
|
|
66
|
+
print(result.to_report())
|
|
67
|
+
|
|
68
|
+
# Fully JSON-serializable dict (NaN → null)
|
|
69
|
+
d = result.to_json()
|
|
70
|
+
print(d["meta"]["content_hash"]) # SHA-256 fingerprint of attrs
|
|
71
|
+
|
|
72
|
+
# USGS gauge — auto-delineate then extract (outlet coords wired automatically)
|
|
73
|
+
from aihydro_watershed.delineation.router import delineate_from_point
|
|
74
|
+
ws = delineate_from_point(39.27, -77.54) # Potomac Point of Rocks
|
|
75
|
+
result = aihydro_lsh.attributes(ws, "1990-01-01", "2020-12-31",
|
|
76
|
+
gauge_id="01638500")
|
|
77
|
+
# ws is a HydroResult — outlet_lat/outlet_lon extracted automatically for geomorphic attrs
|
|
78
|
+
|
|
79
|
+
# Daily forcing timeseries (for ML workflows — LSTM, HBV training)
|
|
80
|
+
df = aihydro_lsh.forcing(geom, "1990-01-01", "2020-12-31")
|
|
81
|
+
print(df.columns.tolist())
|
|
82
|
+
# ['prcp_mm', 'tmin_C', 'tmax_C', 'tavg_C', 'pet_mm', 'srad_Wm2', 'wind_ms', 'sph_kgkg']
|
|
83
|
+
|
|
84
|
+
# Combined: scalar attrs + daily forcing in one call
|
|
85
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31", include_forcing=True)
|
|
86
|
+
print(result.forcing) # pd.DataFrame with DatetimeIndex
|
|
87
|
+
|
|
88
|
+
# Temporal extensions: Mann-Kendall trend scalars (T2.5)
|
|
89
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31", include_trends=True)
|
|
90
|
+
print(result.trends) # pd.DataFrame — 16 slope/p-value scalars
|
|
91
|
+
|
|
92
|
+
# Temporal extensions: annual dynamic panels (T3)
|
|
93
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31", include_dynamic=True)
|
|
94
|
+
print(result.dynamic) # pd.DataFrame (year × 22 variables)
|
|
95
|
+
|
|
96
|
+
# Temporal extensions: drought and flood event inventory (T3)
|
|
97
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31", include_events=True)
|
|
98
|
+
print(result.events["drought"]) # DataFrame of SPI-drought episodes
|
|
99
|
+
print(result.events["flood"]) # DataFrame of POT-flood exceedances
|
|
100
|
+
|
|
101
|
+
# Everything at once
|
|
102
|
+
result = aihydro_lsh.attributes(
|
|
103
|
+
geom, "1990-01-01", "2020-12-31",
|
|
104
|
+
include_forcing=True,
|
|
105
|
+
include_trends=True,
|
|
106
|
+
include_dynamic=True,
|
|
107
|
+
include_events=True,
|
|
108
|
+
)
|
|
109
|
+
|
|
110
|
+
# Batch extraction (list of pour-points → single DataFrame)
|
|
111
|
+
df, errors = aihydro_lsh.batch_attributes(
|
|
112
|
+
[{"id": "01638500", "lat": 39.27, "lon": -77.54},
|
|
113
|
+
{"id": "02108000", "lat": 35.53, "lon": -79.09}],
|
|
114
|
+
start="1990-01-01", end="2020-12-31",
|
|
115
|
+
)
|
|
116
|
+
df.to_csv("camels_global.csv", index=False)
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
## Installation
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
pip install -e . # editable (ecosystem tree)
|
|
123
|
+
pip install -e .[validate] # + camels-attrs CONUS reference
|
|
124
|
+
pip install -e .[mcp] # + MCP server tools
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
## Attribute families (101 static scalars, 8 families + 1 timeseries family)
|
|
128
|
+
|
|
129
|
+
| Family | Attrs | Data source | Notes |
|
|
130
|
+
|---|---|---|---|
|
|
131
|
+
| **watershed** | 5 | GDF geometry | area, perimeter, compactness, elongation, form_factor |
|
|
132
|
+
| **topography** | 7 | GLO30 / DEM3DEP / SRTM | elev stats, slope stats, area |
|
|
133
|
+
| **climate** | 14 | GRIDMET / ERA5-Land / CHIRPS | aridity, seasonality, snow frac, high/low precip stats |
|
|
134
|
+
| **soil** | 9 | POLARIS / SoilGrids | texture, Ksat; porosity/AWC/FC from Saxton-Rawls PTF |
|
|
135
|
+
| **geomorphic** | 28 | 3DEP/GLO30 + NLCD/ESA WorldCover | shape, relief, stream network, CN, sinks, land cover % |
|
|
136
|
+
| **vegetation** | 13 | MODIS LAI/NDVI + ESA WorldCover | LAI/GVF seasonal stats, LC fractions, root depth |
|
|
137
|
+
| **geology** | 9 | pygeoglim (GLiM + GLHYMPS) | lithology classes, carbonate frac, permeability, porosity; global path is permission/tile-gated |
|
|
138
|
+
| **hydrology** | 17 | NWIS (CONUS) / GEOGLOWS (global) | q_mean, BFI, FDC slope, half-flow date, … |
|
|
139
|
+
| **forcing** | timeseries | GRIDMET (CONUS) / ERA5-Land (global) | Daily: prcp_mm, tmin_C, tmax_C, tavg_C, pet_mm [+ srad, wind, sph] |
|
|
140
|
+
|
|
141
|
+
`geomorphic` runs after `soil` so `clay_frac` is available for NRCS Curve Number soil group estimation.
|
|
142
|
+
`forcing` is opt-in (not in the default `RECIPE_ORDER`) — request via `include_forcing=True` or `families=["forcing"]`.
|
|
143
|
+
|
|
144
|
+
## Temporal-tier extensions
|
|
145
|
+
|
|
146
|
+
The static families above are **T1** (permanent) and **T2** (window-mean over the caller's date range). Three optional tiers extend the attribute landscape:
|
|
147
|
+
|
|
148
|
+
| Tier | Flag | Output | Description |
|
|
149
|
+
|---|---|---|---|
|
|
150
|
+
| **T2.5 — Trends** | `include_trends=True` | `result.trends` — up to 16 slope/p-value scalars | Mann–Kendall p-value and Sen slope for eight derived series: precipitation mean, temperature mean, PET mean, snow fraction, LAI maximum, GVF mean, discharge mean and runoff ratio. At least ten valid annual values are required. P-values do not yet adjust for serial dependence or multiple comparisons. |
|
|
151
|
+
| **T3 — Dynamic** | `include_dynamic=True` | `result.dynamic` — year × 22-column schema | Annual climate, vegetation and streamflow summaries. Three annual land-cover columns are currently unavailable and filled with NaN. Other columns can also be missing by source/year; this is a schema, not 22 measured values in every year. Annual `q5` is high flow (5% exceedance) and `q95` is low flow. |
|
|
152
|
+
| **T3 — Events** | `include_events=True` | `result.events["drought"]` + `result.events["flood"]` | SPI-based drought episodes (McKee et al. 1993) and peaks-over-threshold flood exceedances (Coles 2001). Each row is a discrete event with start/end date, duration, severity/magnitude. |
|
|
153
|
+
|
|
154
|
+
The temporal extensions are **opt-in** so standard `attributes()` calls remain fast. All three can be combined with `include_forcing=True` in a single call.
|
|
155
|
+
|
|
156
|
+
### Geometry input forms
|
|
157
|
+
|
|
158
|
+
`attributes()` accepts any of:
|
|
159
|
+
- `HydroResult` from `delineate_from_point()` / `delineate_watershed()` — outlet coords auto-wired
|
|
160
|
+
- `GeoDataFrame` (EPSG:4326, single or multi-row)
|
|
161
|
+
- Shapely Polygon / MultiPolygon
|
|
162
|
+
- `(lat, lon)` tuple — wrapped as 0.001° buffer (use full delineation for accurate basin-mean attrs)
|
|
163
|
+
- `(minx, miny, maxx, maxy)` bbox tuple
|
|
164
|
+
|
|
165
|
+
### Provenance + reproducibility
|
|
166
|
+
|
|
167
|
+
Every `AttributeResult` carries:
|
|
168
|
+
- **`.provenance`** — per-family `AttrProvenance` (source, product, region, resolution_m, is_observed)
|
|
169
|
+
- **`.meta`** — `ResultMeta` with package versions, bbox, timing, and a SHA-256 content hash
|
|
170
|
+
- **`.to_report()`** — human-readable report; agents relay this verbatim (package is the authority)
|
|
171
|
+
- **`.to_json()`** — fully JSON-serializable dict (NaN → null)
|
|
172
|
+
|
|
173
|
+
The `is_observed` flag distinguishes gauge data (NWIS, `is_observed=True`) from modelled or
|
|
174
|
+
reanalysis products (GEOGLOWS, ERA5-Land, `is_observed=False`) — critical for uncertainty-aware
|
|
175
|
+
interpretation by agents. For hydrology specifically, gauge-less/global streamflow signatures
|
|
176
|
+
carry `quality_flags` including `modelled_streamflow`; `AttributeResult.to_report()` emits a
|
|
177
|
+
mandatory **MODELLED STREAMFLOW CAVEAT** so BFI/FDC/flood-frequency/timing metrics are not
|
|
178
|
+
mistaken for observed catchment properties.
|
|
179
|
+
|
|
180
|
+
## Layering
|
|
181
|
+
|
|
182
|
+
```
|
|
183
|
+
aihydro-core ← aihydro-data ← aihydro-lsh
|
|
184
|
+
aihydro-watershed ↗
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
`aihydro-lsh` may depend DOWN on core/data/watershed, never UP on aihydro-tools.
|
|
188
|
+
Enforced by the AST guard in `tests/test_layering.py` and import-linter.
|
|
189
|
+
|
|
190
|
+
## MCP tools (C5)
|
|
191
|
+
|
|
192
|
+
When `aihydro-lsh[mcp]` is installed alongside `aihydro-tools`, ten tools are
|
|
193
|
+
auto-registered on the shared server:
|
|
194
|
+
|
|
195
|
+
| Tool | Description |
|
|
196
|
+
|---|---|
|
|
197
|
+
| `lsh_attributes` | All 101 static scalar attrs for a polygon or pour-point |
|
|
198
|
+
| `lsh_forcing` | Daily basin-averaged meteorological timeseries (LSTM/HBV) |
|
|
199
|
+
| `lsh_dynamic_attributes` | Annual dynamic panels (T3) — year × 22 variables |
|
|
200
|
+
| `lsh_events` | Drought (SPI) and flood (POT) event inventories (T3) |
|
|
201
|
+
| `lsh_families` | List families + attribute keys |
|
|
202
|
+
| `lsh_attribute_info` | Unit, source, family, formula, CAMELS equivalent, and caveat for one attribute |
|
|
203
|
+
| `lsh_batch` | Multi-basin extraction from JSON array |
|
|
204
|
+
| `lsh_compare_to_ref` | lsh vs camels-attrs side-by-side (CONUS) |
|
|
205
|
+
| `lsh_recipe_status` | Check which recipes can run (auth/deps) |
|
|
206
|
+
| `lsh_reliability_matrix` | Family/region availability status and interpretation caveats |
|
|
207
|
+
|
|
208
|
+
For comparative studies, keep product chains explicit. The helper
|
|
209
|
+
`aihydro_lsh.batch.summarize_product_homogeneity(results)` reports whether each
|
|
210
|
+
attribute family used one product across a result set or mixed products such as
|
|
211
|
+
CHIRPS and ERA5-Land. Mixed products are valid but should be reported as a
|
|
212
|
+
source of uncertainty rather than interpreted as purely hydrologic contrast.
|
|
213
|
+
|
|
214
|
+
## Reliability matrix
|
|
215
|
+
|
|
216
|
+
`aihydro_lsh.reliability.reliability_matrix()` returns a code-readable status
|
|
217
|
+
for each attribute family by region (`reliable`, `available_with_caveats`, or
|
|
218
|
+
`not_available`). It is intentionally static and import-light: it does not prove
|
|
219
|
+
live network/data availability, but it prevents binary “global” claims from
|
|
220
|
+
dropping caveats such as modelled streamflow, product-switching uncertainty, and
|
|
221
|
+
permission/tile availability for geology. Geology provenance includes
|
|
222
|
+
`global_tiles_status=...` from `pygeoglim.global_tiles_status()` where available.
|
|
223
|
+
|
|
224
|
+
## CONUS parity validation (C3)
|
|
225
|
+
|
|
226
|
+
```bash
|
|
227
|
+
# CI gate — offline, reads the pre-generated fixture, no network required
|
|
228
|
+
pytest tests/test_conus_parity.py -k Offline --tb=short
|
|
229
|
+
|
|
230
|
+
# Regenerate the fixture (network + GEE + pygeoglim HF token required;
|
|
231
|
+
# several minutes per gauge)
|
|
232
|
+
python scripts/gen_conus_parity_fixture.py
|
|
233
|
+
|
|
234
|
+
# Live spot-check against a fresh live run instead of the cached fixture
|
|
235
|
+
pytest tests/test_conus_parity.py -m live --tb=short
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
The parity suite is parameterized by `tests/fixtures/conus_parity_panel.json`
|
|
239
|
+
— inspect that file for the current basin panel and how each gauge was
|
|
240
|
+
selected (stratified by aridity x snow fraction x relief using
|
|
241
|
+
`pygeohydro.get_camels()`'s real 671-station CAMELS metadata). It checks:
|
|
242
|
+
- Topography: ±10 % of camels-attrs reference
|
|
243
|
+
- Climate: ±10 %
|
|
244
|
+
- Geomorphic: shape ratios within geodesic tolerance
|
|
245
|
+
- Geology: ±5 % (same pygeoglim backend)
|
|
246
|
+
- NaN guard: ≤5 NaN numeric attrs for a healthy extraction (geomorphic sinks/CN may NaN on global path)
|
|
247
|
+
|
|
248
|
+
**What this does and does not establish:** this validates that aihydro-lsh
|
|
249
|
+
reproduces camels-attrs on CONUS — it does not validate the global attribute
|
|
250
|
+
claim (no non-CONUS ground truth exists in this chain, and gauge-less
|
|
251
|
+
hydrology signatures are computed from modelled streamflow with no observed
|
|
252
|
+
reference to check against anywhere). See `papers/DECISIONS.md` PD-4 for the
|
|
253
|
+
claim-scoping decision this follows, and
|
|
254
|
+
`scripts/gen_parity_tolerance_table.py` for the per-attribute divergence
|
|
255
|
+
table generated from the fixture.
|
|
256
|
+
|
|
257
|
+
## Citations
|
|
258
|
+
|
|
259
|
+
Attribute-level citations are returned in `AttributeResult.citations`.
|
|
260
|
+
Key references:
|
|
261
|
+
- Saxton & Rawls (2006) — soil PTF
|
|
262
|
+
- Hartmann & Moosdorf (2012) — GLiM lithology
|
|
263
|
+
- Gleeson et al. (2014) — GLHYMPS permeability
|
|
264
|
+
- Gutman & Ignatov (1998) — NDVI→GVF conversion
|
|
265
|
+
- Jackson et al. (1996) — rooting-depth profiles
|
|
266
|
+
- Addor et al. (2017) — CAMELS attribute definitions
|
|
267
|
+
- Mann (1945); Kendall (1975) — Mann-Kendall trend test
|
|
268
|
+
- Sen (1968) — Sen's slope estimator
|
|
269
|
+
- McKee, Doesken & Kleist (1993) — Standardised Precipitation Index (SPI)
|
|
270
|
+
- Coles (2001) — peaks-over-threshold extreme value analysis
|
|
@@ -0,0 +1,228 @@
|
|
|
1
|
+
# aihydro-lsh
|
|
2
|
+
|
|
3
|
+
**Global large-sample hydrology — CAMELS-like catchment attributes for globally routed watersheds, with source-specific caveats.**
|
|
4
|
+
|
|
5
|
+
A recipe layer over [`aihydro-data`](../aihydro-data/) and [`aihydro-watershed`](../aihydro-watershed/) that computes **101 static scalar attributes** (8 families) plus **daily meteorological forcing timeseries** and **temporal-tier extensions** (trends, annual dynamics, drought/flood events) for basin geometries through region-aware product routing. The [`camels-attrs`](../camels-attrs/) CONUS reference implementation is used to validate recipes against USGS gauge data; use `aihydro_lsh.reliability.reliability_matrix()` for the family/region caveats that qualify global interpretation.
|
|
6
|
+
|
|
7
|
+
## Quick start
|
|
8
|
+
|
|
9
|
+
```python
|
|
10
|
+
import geopandas as gpd
|
|
11
|
+
from shapely.geometry import box
|
|
12
|
+
import aihydro_lsh
|
|
13
|
+
|
|
14
|
+
# Any watershed polygon (EPSG:4326)
|
|
15
|
+
geom = gpd.GeoDataFrame(geometry=[box(-79.5, 38.5, -77.5, 39.5)], crs="EPSG:4326")
|
|
16
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31")
|
|
17
|
+
|
|
18
|
+
print(result.attrs["p_mean"]) # mean daily precip (mm/day)
|
|
19
|
+
print(result.attrs["elev_mean"]) # mean elevation (m)
|
|
20
|
+
print(result.attrs["geol_porosity"]) # GLHYMPS porosity
|
|
21
|
+
df = result.to_dataframe() # single-row DataFrame, 101 columns
|
|
22
|
+
|
|
23
|
+
# Full provenance report (agent-relay verbatim — package is the authority)
|
|
24
|
+
print(result.to_report())
|
|
25
|
+
|
|
26
|
+
# Fully JSON-serializable dict (NaN → null)
|
|
27
|
+
d = result.to_json()
|
|
28
|
+
print(d["meta"]["content_hash"]) # SHA-256 fingerprint of attrs
|
|
29
|
+
|
|
30
|
+
# USGS gauge — auto-delineate then extract (outlet coords wired automatically)
|
|
31
|
+
from aihydro_watershed.delineation.router import delineate_from_point
|
|
32
|
+
ws = delineate_from_point(39.27, -77.54) # Potomac Point of Rocks
|
|
33
|
+
result = aihydro_lsh.attributes(ws, "1990-01-01", "2020-12-31",
|
|
34
|
+
gauge_id="01638500")
|
|
35
|
+
# ws is a HydroResult — outlet_lat/outlet_lon extracted automatically for geomorphic attrs
|
|
36
|
+
|
|
37
|
+
# Daily forcing timeseries (for ML workflows — LSTM, HBV training)
|
|
38
|
+
df = aihydro_lsh.forcing(geom, "1990-01-01", "2020-12-31")
|
|
39
|
+
print(df.columns.tolist())
|
|
40
|
+
# ['prcp_mm', 'tmin_C', 'tmax_C', 'tavg_C', 'pet_mm', 'srad_Wm2', 'wind_ms', 'sph_kgkg']
|
|
41
|
+
|
|
42
|
+
# Combined: scalar attrs + daily forcing in one call
|
|
43
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31", include_forcing=True)
|
|
44
|
+
print(result.forcing) # pd.DataFrame with DatetimeIndex
|
|
45
|
+
|
|
46
|
+
# Temporal extensions: Mann-Kendall trend scalars (T2.5)
|
|
47
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31", include_trends=True)
|
|
48
|
+
print(result.trends) # pd.DataFrame — 16 slope/p-value scalars
|
|
49
|
+
|
|
50
|
+
# Temporal extensions: annual dynamic panels (T3)
|
|
51
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31", include_dynamic=True)
|
|
52
|
+
print(result.dynamic) # pd.DataFrame (year × 22 variables)
|
|
53
|
+
|
|
54
|
+
# Temporal extensions: drought and flood event inventory (T3)
|
|
55
|
+
result = aihydro_lsh.attributes(geom, "1990-01-01", "2020-12-31", include_events=True)
|
|
56
|
+
print(result.events["drought"]) # DataFrame of SPI-drought episodes
|
|
57
|
+
print(result.events["flood"]) # DataFrame of POT-flood exceedances
|
|
58
|
+
|
|
59
|
+
# Everything at once
|
|
60
|
+
result = aihydro_lsh.attributes(
|
|
61
|
+
geom, "1990-01-01", "2020-12-31",
|
|
62
|
+
include_forcing=True,
|
|
63
|
+
include_trends=True,
|
|
64
|
+
include_dynamic=True,
|
|
65
|
+
include_events=True,
|
|
66
|
+
)
|
|
67
|
+
|
|
68
|
+
# Batch extraction (list of pour-points → single DataFrame)
|
|
69
|
+
df, errors = aihydro_lsh.batch_attributes(
|
|
70
|
+
[{"id": "01638500", "lat": 39.27, "lon": -77.54},
|
|
71
|
+
{"id": "02108000", "lat": 35.53, "lon": -79.09}],
|
|
72
|
+
start="1990-01-01", end="2020-12-31",
|
|
73
|
+
)
|
|
74
|
+
df.to_csv("camels_global.csv", index=False)
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Installation
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
pip install -e . # editable (ecosystem tree)
|
|
81
|
+
pip install -e .[validate] # + camels-attrs CONUS reference
|
|
82
|
+
pip install -e .[mcp] # + MCP server tools
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## Attribute families (101 static scalars, 8 families + 1 timeseries family)
|
|
86
|
+
|
|
87
|
+
| Family | Attrs | Data source | Notes |
|
|
88
|
+
|---|---|---|---|
|
|
89
|
+
| **watershed** | 5 | GDF geometry | area, perimeter, compactness, elongation, form_factor |
|
|
90
|
+
| **topography** | 7 | GLO30 / DEM3DEP / SRTM | elev stats, slope stats, area |
|
|
91
|
+
| **climate** | 14 | GRIDMET / ERA5-Land / CHIRPS | aridity, seasonality, snow frac, high/low precip stats |
|
|
92
|
+
| **soil** | 9 | POLARIS / SoilGrids | texture, Ksat; porosity/AWC/FC from Saxton-Rawls PTF |
|
|
93
|
+
| **geomorphic** | 28 | 3DEP/GLO30 + NLCD/ESA WorldCover | shape, relief, stream network, CN, sinks, land cover % |
|
|
94
|
+
| **vegetation** | 13 | MODIS LAI/NDVI + ESA WorldCover | LAI/GVF seasonal stats, LC fractions, root depth |
|
|
95
|
+
| **geology** | 9 | pygeoglim (GLiM + GLHYMPS) | lithology classes, carbonate frac, permeability, porosity; global path is permission/tile-gated |
|
|
96
|
+
| **hydrology** | 17 | NWIS (CONUS) / GEOGLOWS (global) | q_mean, BFI, FDC slope, half-flow date, … |
|
|
97
|
+
| **forcing** | timeseries | GRIDMET (CONUS) / ERA5-Land (global) | Daily: prcp_mm, tmin_C, tmax_C, tavg_C, pet_mm [+ srad, wind, sph] |
|
|
98
|
+
|
|
99
|
+
`geomorphic` runs after `soil` so `clay_frac` is available for NRCS Curve Number soil group estimation.
|
|
100
|
+
`forcing` is opt-in (not in the default `RECIPE_ORDER`) — request via `include_forcing=True` or `families=["forcing"]`.
|
|
101
|
+
|
|
102
|
+
## Temporal-tier extensions
|
|
103
|
+
|
|
104
|
+
The static families above are **T1** (permanent) and **T2** (window-mean over the caller's date range). Three optional tiers extend the attribute landscape:
|
|
105
|
+
|
|
106
|
+
| Tier | Flag | Output | Description |
|
|
107
|
+
|---|---|---|---|
|
|
108
|
+
| **T2.5 — Trends** | `include_trends=True` | `result.trends` — up to 16 slope/p-value scalars | Mann–Kendall p-value and Sen slope for eight derived series: precipitation mean, temperature mean, PET mean, snow fraction, LAI maximum, GVF mean, discharge mean and runoff ratio. At least ten valid annual values are required. P-values do not yet adjust for serial dependence or multiple comparisons. |
|
|
109
|
+
| **T3 — Dynamic** | `include_dynamic=True` | `result.dynamic` — year × 22-column schema | Annual climate, vegetation and streamflow summaries. Three annual land-cover columns are currently unavailable and filled with NaN. Other columns can also be missing by source/year; this is a schema, not 22 measured values in every year. Annual `q5` is high flow (5% exceedance) and `q95` is low flow. |
|
|
110
|
+
| **T3 — Events** | `include_events=True` | `result.events["drought"]` + `result.events["flood"]` | SPI-based drought episodes (McKee et al. 1993) and peaks-over-threshold flood exceedances (Coles 2001). Each row is a discrete event with start/end date, duration, severity/magnitude. |
|
|
111
|
+
|
|
112
|
+
The temporal extensions are **opt-in** so standard `attributes()` calls remain fast. All three can be combined with `include_forcing=True` in a single call.
|
|
113
|
+
|
|
114
|
+
### Geometry input forms
|
|
115
|
+
|
|
116
|
+
`attributes()` accepts any of:
|
|
117
|
+
- `HydroResult` from `delineate_from_point()` / `delineate_watershed()` — outlet coords auto-wired
|
|
118
|
+
- `GeoDataFrame` (EPSG:4326, single or multi-row)
|
|
119
|
+
- Shapely Polygon / MultiPolygon
|
|
120
|
+
- `(lat, lon)` tuple — wrapped as 0.001° buffer (use full delineation for accurate basin-mean attrs)
|
|
121
|
+
- `(minx, miny, maxx, maxy)` bbox tuple
|
|
122
|
+
|
|
123
|
+
### Provenance + reproducibility
|
|
124
|
+
|
|
125
|
+
Every `AttributeResult` carries:
|
|
126
|
+
- **`.provenance`** — per-family `AttrProvenance` (source, product, region, resolution_m, is_observed)
|
|
127
|
+
- **`.meta`** — `ResultMeta` with package versions, bbox, timing, and a SHA-256 content hash
|
|
128
|
+
- **`.to_report()`** — human-readable report; agents relay this verbatim (package is the authority)
|
|
129
|
+
- **`.to_json()`** — fully JSON-serializable dict (NaN → null)
|
|
130
|
+
|
|
131
|
+
The `is_observed` flag distinguishes gauge data (NWIS, `is_observed=True`) from modelled or
|
|
132
|
+
reanalysis products (GEOGLOWS, ERA5-Land, `is_observed=False`) — critical for uncertainty-aware
|
|
133
|
+
interpretation by agents. For hydrology specifically, gauge-less/global streamflow signatures
|
|
134
|
+
carry `quality_flags` including `modelled_streamflow`; `AttributeResult.to_report()` emits a
|
|
135
|
+
mandatory **MODELLED STREAMFLOW CAVEAT** so BFI/FDC/flood-frequency/timing metrics are not
|
|
136
|
+
mistaken for observed catchment properties.
|
|
137
|
+
|
|
138
|
+
## Layering
|
|
139
|
+
|
|
140
|
+
```
|
|
141
|
+
aihydro-core ← aihydro-data ← aihydro-lsh
|
|
142
|
+
aihydro-watershed ↗
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
`aihydro-lsh` may depend DOWN on core/data/watershed, never UP on aihydro-tools.
|
|
146
|
+
Enforced by the AST guard in `tests/test_layering.py` and import-linter.
|
|
147
|
+
|
|
148
|
+
## MCP tools (C5)
|
|
149
|
+
|
|
150
|
+
When `aihydro-lsh[mcp]` is installed alongside `aihydro-tools`, ten tools are
|
|
151
|
+
auto-registered on the shared server:
|
|
152
|
+
|
|
153
|
+
| Tool | Description |
|
|
154
|
+
|---|---|
|
|
155
|
+
| `lsh_attributes` | All 101 static scalar attrs for a polygon or pour-point |
|
|
156
|
+
| `lsh_forcing` | Daily basin-averaged meteorological timeseries (LSTM/HBV) |
|
|
157
|
+
| `lsh_dynamic_attributes` | Annual dynamic panels (T3) — year × 22 variables |
|
|
158
|
+
| `lsh_events` | Drought (SPI) and flood (POT) event inventories (T3) |
|
|
159
|
+
| `lsh_families` | List families + attribute keys |
|
|
160
|
+
| `lsh_attribute_info` | Unit, source, family, formula, CAMELS equivalent, and caveat for one attribute |
|
|
161
|
+
| `lsh_batch` | Multi-basin extraction from JSON array |
|
|
162
|
+
| `lsh_compare_to_ref` | lsh vs camels-attrs side-by-side (CONUS) |
|
|
163
|
+
| `lsh_recipe_status` | Check which recipes can run (auth/deps) |
|
|
164
|
+
| `lsh_reliability_matrix` | Family/region availability status and interpretation caveats |
|
|
165
|
+
|
|
166
|
+
For comparative studies, keep product chains explicit. The helper
|
|
167
|
+
`aihydro_lsh.batch.summarize_product_homogeneity(results)` reports whether each
|
|
168
|
+
attribute family used one product across a result set or mixed products such as
|
|
169
|
+
CHIRPS and ERA5-Land. Mixed products are valid but should be reported as a
|
|
170
|
+
source of uncertainty rather than interpreted as purely hydrologic contrast.
|
|
171
|
+
|
|
172
|
+
## Reliability matrix
|
|
173
|
+
|
|
174
|
+
`aihydro_lsh.reliability.reliability_matrix()` returns a code-readable status
|
|
175
|
+
for each attribute family by region (`reliable`, `available_with_caveats`, or
|
|
176
|
+
`not_available`). It is intentionally static and import-light: it does not prove
|
|
177
|
+
live network/data availability, but it prevents binary “global” claims from
|
|
178
|
+
dropping caveats such as modelled streamflow, product-switching uncertainty, and
|
|
179
|
+
permission/tile availability for geology. Geology provenance includes
|
|
180
|
+
`global_tiles_status=...` from `pygeoglim.global_tiles_status()` where available.
|
|
181
|
+
|
|
182
|
+
## CONUS parity validation (C3)
|
|
183
|
+
|
|
184
|
+
```bash
|
|
185
|
+
# CI gate — offline, reads the pre-generated fixture, no network required
|
|
186
|
+
pytest tests/test_conus_parity.py -k Offline --tb=short
|
|
187
|
+
|
|
188
|
+
# Regenerate the fixture (network + GEE + pygeoglim HF token required;
|
|
189
|
+
# several minutes per gauge)
|
|
190
|
+
python scripts/gen_conus_parity_fixture.py
|
|
191
|
+
|
|
192
|
+
# Live spot-check against a fresh live run instead of the cached fixture
|
|
193
|
+
pytest tests/test_conus_parity.py -m live --tb=short
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The parity suite is parameterized by `tests/fixtures/conus_parity_panel.json`
|
|
197
|
+
— inspect that file for the current basin panel and how each gauge was
|
|
198
|
+
selected (stratified by aridity x snow fraction x relief using
|
|
199
|
+
`pygeohydro.get_camels()`'s real 671-station CAMELS metadata). It checks:
|
|
200
|
+
- Topography: ±10 % of camels-attrs reference
|
|
201
|
+
- Climate: ±10 %
|
|
202
|
+
- Geomorphic: shape ratios within geodesic tolerance
|
|
203
|
+
- Geology: ±5 % (same pygeoglim backend)
|
|
204
|
+
- NaN guard: ≤5 NaN numeric attrs for a healthy extraction (geomorphic sinks/CN may NaN on global path)
|
|
205
|
+
|
|
206
|
+
**What this does and does not establish:** this validates that aihydro-lsh
|
|
207
|
+
reproduces camels-attrs on CONUS — it does not validate the global attribute
|
|
208
|
+
claim (no non-CONUS ground truth exists in this chain, and gauge-less
|
|
209
|
+
hydrology signatures are computed from modelled streamflow with no observed
|
|
210
|
+
reference to check against anywhere). See `papers/DECISIONS.md` PD-4 for the
|
|
211
|
+
claim-scoping decision this follows, and
|
|
212
|
+
`scripts/gen_parity_tolerance_table.py` for the per-attribute divergence
|
|
213
|
+
table generated from the fixture.
|
|
214
|
+
|
|
215
|
+
## Citations
|
|
216
|
+
|
|
217
|
+
Attribute-level citations are returned in `AttributeResult.citations`.
|
|
218
|
+
Key references:
|
|
219
|
+
- Saxton & Rawls (2006) — soil PTF
|
|
220
|
+
- Hartmann & Moosdorf (2012) — GLiM lithology
|
|
221
|
+
- Gleeson et al. (2014) — GLHYMPS permeability
|
|
222
|
+
- Gutman & Ignatov (1998) — NDVI→GVF conversion
|
|
223
|
+
- Jackson et al. (1996) — rooting-depth profiles
|
|
224
|
+
- Addor et al. (2017) — CAMELS attribute definitions
|
|
225
|
+
- Mann (1945); Kendall (1975) — Mann-Kendall trend test
|
|
226
|
+
- Sen (1968) — Sen's slope estimator
|
|
227
|
+
- McKee, Doesken & Kleist (1993) — Standardised Precipitation Index (SPI)
|
|
228
|
+
- Coles (2001) — peaks-over-threshold extreme value analysis
|