meteoraster 2.5__tar.gz → 3.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- meteoraster-3.1/PKG-INFO +626 -0
- meteoraster-3.1/README.md +573 -0
- meteoraster-3.1/meteoraster/_lazy.py +73 -0
- meteoraster-3.1/meteoraster/io.py +493 -0
- meteoraster-3.1/meteoraster/meteoraster.py +1555 -0
- meteoraster-3.1/meteoraster/resample.py +528 -0
- meteoraster-3.1/meteoraster/utils.py +182 -0
- meteoraster-3.1/meteoraster/zonal.py +1105 -0
- meteoraster-3.1/meteoraster.egg-info/PKG-INFO +626 -0
- {meteoraster-2.5 → meteoraster-3.1}/meteoraster.egg-info/SOURCES.txt +12 -1
- {meteoraster-2.5 → meteoraster-3.1}/meteoraster.egg-info/requires.txt +2 -1
- {meteoraster-2.5 → meteoraster-3.1}/pyproject.toml +7 -3
- meteoraster-3.1/tests/conftest.py +315 -0
- {meteoraster-2.5 → meteoraster-3.1}/tests/test_deprecations.py +14 -0
- meteoraster-3.1/tests/test_io.py +353 -0
- meteoraster-3.1/tests/test_kml_catchments.py +375 -0
- {meteoraster-2.5 → meteoraster-3.1}/tests/test_meteo_raster_base.py +57 -0
- {meteoraster-2.5 → meteoraster-3.1}/tests/test_meteo_raster_plots.py +21 -28
- meteoraster-3.1/tests/test_quantiles.py +239 -0
- meteoraster-3.1/tests/test_real_rasters.py +292 -0
- meteoraster-3.1/tests/test_regressions.py +293 -0
- {meteoraster-2.5 → meteoraster-3.1}/tests/test_resample.py +4 -0
- meteoraster-3.1/tests/test_resample_semantics.py +441 -0
- meteoraster-3.1/tests/test_utils_readers.py +258 -0
- meteoraster-3.1/tests/test_zonal.py +688 -0
- meteoraster-2.5/PKG-INFO +0 -419
- meteoraster-2.5/README.md +0 -367
- meteoraster-2.5/meteoraster/meteoraster.py +0 -1267
- meteoraster-2.5/meteoraster/resample.py +0 -269
- meteoraster-2.5/meteoraster/utils.py +0 -359
- meteoraster-2.5/meteoraster.egg-info/PKG-INFO +0 -419
- meteoraster-2.5/tests/conftest.py +0 -149
- {meteoraster-2.5 → meteoraster-3.1}/LICENSE +0 -0
- {meteoraster-2.5 → meteoraster-3.1}/MANIFEST.in +0 -0
- {meteoraster-2.5 → meteoraster-3.1}/NOTICE +0 -0
- {meteoraster-2.5 → meteoraster-3.1}/meteoraster/__init__.py +0 -0
- {meteoraster-2.5 → meteoraster-3.1}/meteoraster.egg-info/dependency_links.txt +0 -0
- {meteoraster-2.5 → meteoraster-3.1}/meteoraster.egg-info/top_level.txt +0 -0
- {meteoraster-2.5 → meteoraster-3.1}/setup.cfg +0 -0
- {meteoraster-2.5 → meteoraster-3.1}/tests/test_meteo_raster_extra.py +0 -0
meteoraster-3.1/PKG-INFO
ADDED
|
@@ -0,0 +1,626 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: meteoraster
|
|
3
|
+
Version: 3.1
|
|
4
|
+
Summary: Module for handling distributed ensemble and probabilistic forecast data (5-D meteorological rasters)
|
|
5
|
+
Author-email: Jose Pedro Matos <jose.matos@tecnico.ulisboa.pt>
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/FORESIGHT-ULisboa/Tethys-meteoraster
|
|
8
|
+
Project-URL: Repository, https://github.com/FORESIGHT-ULisboa/Tethys-meteoraster
|
|
9
|
+
Project-URL: Issues, https://github.com/FORESIGHT-ULisboa/Tethys-meteoraster/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/FORESIGHT-ULisboa/Tethys-meteoraster/releases
|
|
11
|
+
Project-URL: FORESIGHT, https://foresight.tecnico.ulisboa.pt/
|
|
12
|
+
Keywords: meteorology,forecast,ensemble,raster,netcdf,grib,ERA5,ERA5-Land
|
|
13
|
+
Classifier: Development Status :: 4 - Beta
|
|
14
|
+
Classifier: Intended Audience :: Science/Research
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
|
|
20
|
+
Requires-Python: >=3.11
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
License-File: LICENSE
|
|
23
|
+
License-File: NOTICE
|
|
24
|
+
Requires-Dist: numpy>=1.26
|
|
25
|
+
Requires-Dist: scipy
|
|
26
|
+
Requires-Dist: pandas
|
|
27
|
+
Requires-Dist: xarray
|
|
28
|
+
Requires-Dist: netcdf4
|
|
29
|
+
Requires-Dist: h5netcdf
|
|
30
|
+
Requires-Dist: h5py
|
|
31
|
+
Requires-Dist: cfgrib
|
|
32
|
+
Requires-Dist: cdsapi
|
|
33
|
+
Requires-Dist: scikit-learn
|
|
34
|
+
Requires-Dist: matplotlib
|
|
35
|
+
Requires-Dist: seaborn
|
|
36
|
+
Requires-Dist: statsmodels
|
|
37
|
+
Requires-Dist: cartopy
|
|
38
|
+
Requires-Dist: beautifulsoup4
|
|
39
|
+
Requires-Dist: openpyxl
|
|
40
|
+
Provides-Extra: test
|
|
41
|
+
Requires-Dist: pytest; extra == "test"
|
|
42
|
+
Requires-Dist: pytest-cov; extra == "test"
|
|
43
|
+
Provides-Extra: build
|
|
44
|
+
Requires-Dist: build; extra == "build"
|
|
45
|
+
Requires-Dist: twine; extra == "build"
|
|
46
|
+
Provides-Extra: dev
|
|
47
|
+
Requires-Dist: pytest; extra == "dev"
|
|
48
|
+
Requires-Dist: pytest-cov; extra == "dev"
|
|
49
|
+
Requires-Dist: build; extra == "dev"
|
|
50
|
+
Requires-Dist: twine; extra == "dev"
|
|
51
|
+
Requires-Dist: jupyterlab; extra == "dev"
|
|
52
|
+
Dynamic: license-file
|
|
53
|
+
|
|
54
|
+
<p align="center">
|
|
55
|
+
<img src="images/foresight.png" alt="FORESIGHT" width="320">
|
|
56
|
+
</p>
|
|
57
|
+
|
|
58
|
+
# Tethys meteoraster
|
|
59
|
+
|
|
60
|
+
<p align="center">
|
|
61
|
+
<a href="https://pypi.org/project/meteoraster/"><img src="https://img.shields.io/pypi/v/meteoraster.svg" alt="PyPI version"></a>
|
|
62
|
+
<a href="https://pypi.org/project/meteoraster/"><img src="https://img.shields.io/pypi/pyversions/meteoraster.svg" alt="Supported Python versions"></a>
|
|
63
|
+
<a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-blue.svg" alt="License: Apache-2.0"></a>
|
|
64
|
+
</p>
|
|
65
|
+
|
|
66
|
+
**`meteoraster`** is a Python library, developed by [FORESIGHT — Forecasting and
|
|
67
|
+
Optimization for Resilient Environmental Systems through Investigation with
|
|
68
|
+
Groundbreaking Hydrological Tools](https://foresight.tecnico.ulisboa.pt/), for
|
|
69
|
+
**handling distributed ensemble and probabilistic meteorological forecasts**.
|
|
70
|
+
|
|
71
|
+
Its core data structure, `MeteoRaster`, holds a single **5-D** array indexed by
|
|
72
|
+
`[production_datetime, ensemble_member, leadtime, latitude, longitude]`, and
|
|
73
|
+
wraps it with everything you need to turn raw reanalysis/forecast files into
|
|
74
|
+
report-ready time series, maps and aggregated statistics:
|
|
75
|
+
|
|
76
|
+
- read ERA5-Land GRIB (monthly means and hourly, with de-accumulation) into a common model,
|
|
77
|
+
- crop, join, trim, resample and check the completeness of the data,
|
|
78
|
+
- extract point time series or **catchment/zone aggregates from KML polygons**
|
|
79
|
+
(area-weighted, missing-data aware), as means or as exact weighted quantiles,
|
|
80
|
+
- persist to compressed NetCDF (`h5netcdf`) and reload losslessly — with a read
|
|
81
|
+
path that survives an install on a different xarray,
|
|
82
|
+
- **stream a file too large to hold in memory**, one production at a time,
|
|
83
|
+
- and plot means, seasonal cycles, coordinates and availability with `cartopy`.
|
|
84
|
+
|
|
85
|
+
Only `numpy`, `scipy`, `pandas`, `xarray`, `h5netcdf` and `h5py` are needed to read,
|
|
86
|
+
write, subset and aggregate. GDAL, matplotlib, seaborn and cartopy are imported the
|
|
87
|
+
moment you use them, so `import meteoraster` works in a slim container that has
|
|
88
|
+
none of them.
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## Features
|
|
93
|
+
|
|
94
|
+
| Category | What you get |
|
|
95
|
+
|---|---|
|
|
96
|
+
| **Data model** | `MeteoRaster` — 5-D `[production_datetime, ensemble_member, leadtime, lat, lon]`; 2-D lat/lon grids; automatic Greenwich (0–360 → −180–180) fix and latitude flip |
|
|
97
|
+
| **Format readers** | `meteoraster.utils` — `readERA5Land_monthly`, `read_ERA5Land_hourly` (GRIB via `cfgrib`) |
|
|
98
|
+
| **Subsetting / merging** | `get_cropped`, `join`, `trim`, `adjust_leadtimes` |
|
|
99
|
+
| **Resampling** | `resample_timestep` (pandas rules) and the standalone `Resample` class |
|
|
100
|
+
| **Extraction** | `get_values_from_latlon`, `get_values_from_KML`, `get_quantiles_from_KML`, `get_values_from_latlon_by_event` |
|
|
101
|
+
| **Spatial aggregation** | `meteoraster.zonal` — sparse coverage by exact fractional overlap, area-weighted means and quantiles, missing-data aware per zone |
|
|
102
|
+
| **Quality / diagnostics** | `is_complete`, `get_complete_index`, `get_missing`, `get_completeness` |
|
|
103
|
+
| **I/O** | `save` / `load` (compressed NetCDF via `h5netcdf`), `to_xarray`, and `meteoraster.io` for header-only and streaming reads |
|
|
104
|
+
| **Visualisation** | `plot_mean`, `plot_mean_projected`, `plot_seasonal`, `plot_coordinates`, `plot_availability`, `add_shapefile`, `add_shape` |
|
|
105
|
+
|
|
106
|
+
---
|
|
107
|
+
|
|
108
|
+
## Installation
|
|
109
|
+
|
|
110
|
+
The **core** — reading, writing, subsetting, streaming and aggregating — needs only
|
|
111
|
+
`numpy`, `scipy`, `pandas`, `xarray`, `h5netcdf` and `h5py`, all pip-installable:
|
|
112
|
+
|
|
113
|
+
```bat
|
|
114
|
+
pip install meteoraster
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
The **geospatial and plotting extras** (**GDAL/`osgeo`**, `cartopy`, `cfgrib`) are
|
|
118
|
+
most reliably installed with **conda**, and are imported only when used. Zonal
|
|
119
|
+
aggregation from KML or PostGIS geometry needs GDAL; the map plots need cartopy.
|
|
120
|
+
`import meteoraster` works without either — a missing one raises at the call site
|
|
121
|
+
with the command to install it.
|
|
122
|
+
|
|
123
|
+
Two paths follow: install the **released package from PyPI** on top of a conda
|
|
124
|
+
environment (use the package), or install **from source** (develop / run the tests).
|
|
125
|
+
|
|
126
|
+
### A · Install from PyPI
|
|
127
|
+
|
|
128
|
+
Create a conda environment that provides the geospatial libraries, then install
|
|
129
|
+
[`meteoraster`](https://pypi.org/project/meteoraster/) into it:
|
|
130
|
+
|
|
131
|
+
```bat
|
|
132
|
+
conda env create -f environment.yml
|
|
133
|
+
conda activate tethys_rasters
|
|
134
|
+
pip install meteoraster
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Upgrading later needs no URL — `pip install --upgrade meteoraster` always picks up
|
|
138
|
+
the newest release.
|
|
139
|
+
|
|
140
|
+
> The package declares its pure-Python dependencies, but **GDAL is intentionally
|
|
141
|
+
> not** a pip dependency (it is a system/conda package). The conda environment
|
|
142
|
+
> above provides it. If you do not need zonal aggregation from vector geometry,
|
|
143
|
+
> `pip install meteoraster` into any environment is enough — including a slim
|
|
144
|
+
> container image, where `import meteoraster`, `save`, `load` and the streaming
|
|
145
|
+
> readers all work with no GDAL, cartopy, matplotlib or seaborn present.
|
|
146
|
+
|
|
147
|
+
Every release is also attached to the
|
|
148
|
+
[GitHub releases page](https://github.com/FORESIGHT-ULisboa/Tethys-meteoraster/releases),
|
|
149
|
+
which is useful to pin an exact build without going through PyPI:
|
|
150
|
+
|
|
151
|
+
```bat
|
|
152
|
+
pip install https://github.com/FORESIGHT-ULisboa/Tethys-meteoraster/releases/download/v3.0/meteoraster-3.0-py3-none-any.whl
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
### B · Install from source (development)
|
|
156
|
+
|
|
157
|
+
#### 1 · Create the conda environment
|
|
158
|
+
|
|
159
|
+
```bat
|
|
160
|
+
conda env create -f environment.yml
|
|
161
|
+
conda activate tethys_rasters
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
#### 2 · Install the package (editable) with the dev extras
|
|
165
|
+
|
|
166
|
+
From the repository root:
|
|
167
|
+
|
|
168
|
+
```bat
|
|
169
|
+
pip install -e ".[dev]"
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
That adds `pytest`, `pytest-cov`, `build`, `twine` and `jupyterlab` on top of the
|
|
173
|
+
runtime dependencies.
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## Quick start
|
|
178
|
+
|
|
179
|
+
> **Worked examples:** [`notebooks/`](notebooks/) holds seven short Jupyter notebooks —
|
|
180
|
+
> the 5-D data model, I/O and xarray, cropping, plotting, point time series, catchment
|
|
181
|
+
> aggregation from KML, and resampling. They run on the real files in
|
|
182
|
+
> [tests/resources](tests/resources) and are committed **with their outputs**, so every
|
|
183
|
+
> table and figure is readable on GitHub without running anything.
|
|
184
|
+
|
|
185
|
+
```python
|
|
186
|
+
import numpy as np
|
|
187
|
+
import pandas as pd
|
|
188
|
+
from meteoraster import MeteoRaster
|
|
189
|
+
|
|
190
|
+
# data axes: [production_datetime, ensemble_member, leadtime, lat, lon]
|
|
191
|
+
data = np.random.rand(2, 3, 2, 4, 5)
|
|
192
|
+
|
|
193
|
+
mr = MeteoRaster(
|
|
194
|
+
data=data,
|
|
195
|
+
latitudes=np.linspace(40, 30, 4), # 1-D is auto-meshed to 2-D
|
|
196
|
+
longitudes=np.linspace(-10, 10, 5),
|
|
197
|
+
production_datetime=pd.to_datetime(["2023-01-01", "2023-01-02"]).values,
|
|
198
|
+
leadtimes=np.array([pd.Timedelta(days=1), pd.Timedelta(days=2)]),
|
|
199
|
+
units="mm",
|
|
200
|
+
variable="precip",
|
|
201
|
+
)
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
### Read a real dataset
|
|
205
|
+
|
|
206
|
+
The ERA5-Land readers live in `meteoraster.utils` and return a ready-to-use
|
|
207
|
+
`MeteoRaster`:
|
|
208
|
+
|
|
209
|
+
```python
|
|
210
|
+
from meteoraster.utils import readERA5Land_monthly, read_ERA5Land_hourly
|
|
211
|
+
|
|
212
|
+
# monthly means: tp comes back as mm/month, t2m as degC
|
|
213
|
+
era5 = readERA5Land_monthly("era5land_monthly.grib", "tp")
|
|
214
|
+
|
|
215
|
+
# hourly: accumulations are de-accumulated to per-hour increments (mm/hr).
|
|
216
|
+
# A CDS request omits the last hour of its final day - it is delivered with the
|
|
217
|
+
# next period, so pass that file to fill it in.
|
|
218
|
+
hourly = read_ERA5Land_hourly("era5land_2025_11.grib", "tp",
|
|
219
|
+
file_next_year="era5land_2025_12.grib")
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
These need `cfgrib` and the ecCodes binary. On Windows ecCodes lives in
|
|
223
|
+
`<env>/Library/bin` and is only found through `PATH`, so **activate** the conda
|
|
224
|
+
environment rather than calling its `python.exe` directly — otherwise `import cfgrib`
|
|
225
|
+
raises `RuntimeError: Cannot find the ecCodes library`.
|
|
226
|
+
|
|
227
|
+
Anything already converted by Tethys needs no reader — `MeteoRaster.load` reads it
|
|
228
|
+
directly.
|
|
229
|
+
|
|
230
|
+
### Extract a point time series
|
|
231
|
+
|
|
232
|
+
```python
|
|
233
|
+
# DataFrame: index = production_datetime, columns = MultiIndex(leadtime, ensemble_member)
|
|
234
|
+
ts = mr.get_values_from_latlon(lat=35.0, lon=0.0)
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
### Aggregate over KML zones (e.g. catchments)
|
|
238
|
+
|
|
239
|
+
```python
|
|
240
|
+
# agg: DataFrame indexed by production date, columns MultiIndex(zone, leadtime, ensemble_member)
|
|
241
|
+
# centroids: DataFrame of zone centroids (x, y) and grid_fraction (share of the zone on the grid)
|
|
242
|
+
agg, centroids = mr.get_values_from_KML("zones.kml", nameField="zone_id")
|
|
243
|
+
|
|
244
|
+
# v3.0 numbers (square-degree weights) instead of true WGS84 areas
|
|
245
|
+
agg, centroids = mr.get_values_from_KML("zones.kml", nameField="zone_id",
|
|
246
|
+
area_weighting="planar")
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
Coordinates are WGS84 lon/lat of cell **centres** (curvilinear 2-D grids included);
|
|
250
|
+
KML edges are straight lines in lon/lat. Weights are true WGS84 areas by default.
|
|
251
|
+
A zone partly outside the grid is averaged over the part inside and warns
|
|
252
|
+
(`on_partial=`); a reused `coverage_info` must match the raster's grid.
|
|
253
|
+
|
|
254
|
+
Spatial **quantiles** over the same zones, area-weighted so a cell half inside a
|
|
255
|
+
zone carries half the probability mass:
|
|
256
|
+
|
|
257
|
+
```python
|
|
258
|
+
agg, centroids = mr.get_quantiles_from_KML("zones.kml", nameField="zone_id",
|
|
259
|
+
quantiles=[0.1, 0.5, 0.9])
|
|
260
|
+
|
|
261
|
+
# Pool an hourly forecast into one distribution per day. `pool` names the axis
|
|
262
|
+
# explicitly because "time" is the leadtime axis for a forecast and the production
|
|
263
|
+
# axis for an observation raster - the event time is production + leadtime.
|
|
264
|
+
agg, _ = mr.get_quantiles_from_KML("zones.kml", nameField="zone_id",
|
|
265
|
+
pool=("space", "leadtime"), resampling="1D")
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
The coverage is reusable, and the lower-level API needs no `MeteoRaster` at all —
|
|
269
|
+
useful for caching it, or for feeding geometry straight out of PostGIS:
|
|
270
|
+
|
|
271
|
+
```python
|
|
272
|
+
from meteoraster import zonal
|
|
273
|
+
|
|
274
|
+
coverage = zonal.coverage_from_kml("zones.kml", mr.latitudes, mr.longitudes,
|
|
275
|
+
name_field="zone_id")
|
|
276
|
+
means = zonal.apply_coverage(coverage, mr.data) # [..., y, x] -> [..., zone]
|
|
277
|
+
|
|
278
|
+
coverage.save("coverage.npz") # pickle-free
|
|
279
|
+
coverage = zonal.Coverage.load("coverage.npz")
|
|
280
|
+
assert coverage.matches(mr.latitudes, mr.longitudes) # fingerprints the coordinates
|
|
281
|
+
|
|
282
|
+
# WKB bytes or WKT work too, so PostGIS geometry needs no intermediate file
|
|
283
|
+
coverage = zonal.coverage_from_geometries(mr.latitudes, mr.longitudes,
|
|
284
|
+
[feature.geom.wkb], ["basin"])
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
### Crop, save and reload
|
|
288
|
+
|
|
289
|
+
```python
|
|
290
|
+
sub = mr.get_cropped(from_lat=32, to_lat=38, from_lon=-2, to_lon=7,
|
|
291
|
+
from_prod_date=pd.Timestamp("2023-01-02"))
|
|
292
|
+
|
|
293
|
+
sub.save("forecast.nc") # compressed NetCDF (h5netcdf)
|
|
294
|
+
reloaded = MeteoRaster.load("forecast.nc")
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
### Read a file that does not fit in memory
|
|
298
|
+
|
|
299
|
+
`load()` holds the whole cube. For a large archive, read the metadata first and
|
|
300
|
+
then stream one production at a time — peak memory is one block whatever the file
|
|
301
|
+
size (measured 126 MB against 467 MB for `load()` on a 352 MB cube):
|
|
302
|
+
|
|
303
|
+
```python
|
|
304
|
+
from meteoraster import io as mrio, zonal
|
|
305
|
+
|
|
306
|
+
header = mrio.read_header("archive.nct") # coordinates, shape, units; no values
|
|
307
|
+
print(header.shape, header.n_leadtimes, header.complete)
|
|
308
|
+
|
|
309
|
+
coverage = zonal.coverage_from_kml("zones.kml", header.latitudes, header.longitudes,
|
|
310
|
+
name_field="zone_id")
|
|
311
|
+
|
|
312
|
+
for production, block in mrio.iter_productions("archive.nct", reduce_members="mean"):
|
|
313
|
+
series = zonal.apply_coverage(coverage, block) # [leadtime, zone]
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
`load()` also takes a subset directly, and casting on the way in genuinely halves
|
|
317
|
+
peak memory for a float64 file rather than casting after it is already resident:
|
|
318
|
+
|
|
319
|
+
```python
|
|
320
|
+
mr = MeteoRaster.load("archive.nct", leadtimes=slice(0, 6), dtype="float32")
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
Reads try `h5netcdf`, then `netcdf4`, then `scipy`, and pin the decoding so a file
|
|
324
|
+
written by another install returns the same dtypes. If HDF5 locking fails on a
|
|
325
|
+
network share or a bind mount, set `HDF5_USE_FILE_LOCKING=FALSE` in the
|
|
326
|
+
environment; the library will not set it for you.
|
|
327
|
+
|
|
328
|
+
### Resample in time
|
|
329
|
+
|
|
330
|
+
```python
|
|
331
|
+
import numpy as np
|
|
332
|
+
|
|
333
|
+
mr.resample_timestep("MS", fun=np.nanmean) # to month start, ensemble-wise
|
|
334
|
+
|
|
335
|
+
# Or resample a production×leadtime DataFrame with the standalone helper:
|
|
336
|
+
from meteoraster import Resample
|
|
337
|
+
resampled = Resample.resample(df, timestep_frequency="1D", resamplingType="sum")
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
### Plot
|
|
341
|
+
|
|
342
|
+
```python
|
|
343
|
+
ax, cbar = mr.plot_mean(central_longitude=0, coastline=True, borders=True,
|
|
344
|
+
colorbar=True, cmap="viridis")
|
|
345
|
+
mr.add_shapefile(ax, "basin.shp")
|
|
346
|
+
mr.plot_seasonal(lat=35.0, lon=0.0)
|
|
347
|
+
mr.plot_availability()
|
|
348
|
+
```
|
|
349
|
+
|
|
350
|
+
---
|
|
351
|
+
|
|
352
|
+
## Project structure
|
|
353
|
+
|
|
354
|
+
```
|
|
355
|
+
Tethys-meteoraster/
|
|
356
|
+
├── meteoraster/
|
|
357
|
+
│ ├── __init__.py # exports MeteoRaster, Resample
|
|
358
|
+
│ ├── meteoraster.py # MeteoRaster core class
|
|
359
|
+
│ ├── io.py # portable NetCDF reads: header-only and streaming
|
|
360
|
+
│ ├── zonal.py # sparse coverage, area-weighted means and quantiles
|
|
361
|
+
│ ├── resample.py # Resample (time-step resampling of DataFrames)
|
|
362
|
+
│ ├── _lazy.py # on-demand GDAL / matplotlib / seaborn / cartopy
|
|
363
|
+
│ └── utils.py # ERA5-Land GRIB readers
|
|
364
|
+
├── notebooks/ # seven worked examples, executed (see notebooks/README.md)
|
|
365
|
+
├── tests/ # pytest suite + fixtures (conftest.py)
|
|
366
|
+
│ └── resources/ # real GRIB/NetCDF inputs (see PROVENANCE.md)
|
|
367
|
+
├── tools/make_fixtures.py # regenerates the subsetted test fixtures
|
|
368
|
+
├── images/foresight.png # README header
|
|
369
|
+
├── environment.yml # conda environment (tethys_rasters)
|
|
370
|
+
├── pyproject.toml # packaging + pytest configuration
|
|
371
|
+
├── AGENTS.md # canonical guidance for humans & AI agents
|
|
372
|
+
├── CLAUDE.md # → points to AGENTS.md
|
|
373
|
+
├── .github/copilot-instructions.md # → points to AGENTS.md
|
|
374
|
+
├── .github/workflows/release.yml # build + publish to PyPI on tag push
|
|
375
|
+
├── .github/scripts/check_version.py # release guard: version declared consistently
|
|
376
|
+
├── LICENSE # Apache 2.0
|
|
377
|
+
├── NOTICE # attribution required by Apache 2.0
|
|
378
|
+
└── README.md
|
|
379
|
+
```
|
|
380
|
+
|
|
381
|
+
---
|
|
382
|
+
|
|
383
|
+
## Running the tests
|
|
384
|
+
|
|
385
|
+
```bat
|
|
386
|
+
conda activate tethys_rasters
|
|
387
|
+
pip install -e ".[dev]"
|
|
388
|
+
pytest -ra
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
The suite covers construction, cropping, point/KML extraction, NetCDF round-trips,
|
|
392
|
+
completeness checks, resampling and the deprecated-alias warnings. Since v3.0 it also
|
|
393
|
+
runs against **real inputs** in `tests/resources` — ERA5-Land GRIB for the readers,
|
|
394
|
+
converted Tethys forecasts (ECMWF SEAS5, IPMA, NOAA GFS) for the monthly adaptation and
|
|
395
|
+
the resampling units contract, and a real 26-catchment KML layer for zonal aggregation.
|
|
396
|
+
Those files are subsets; see
|
|
397
|
+
[tests/resources/PROVENANCE.md](tests/resources/PROVENANCE.md) and
|
|
398
|
+
`tools/make_fixtures.py`.
|
|
399
|
+
|
|
400
|
+
Tests skip — by name, never error — when a resource is missing or when an optional
|
|
401
|
+
dependency is absent, so a partial clone and a slim install both stay green. **Activate**
|
|
402
|
+
the conda environment rather than calling its `python.exe` directly, or the GRIB tests
|
|
403
|
+
skip on a `PATH` that hides ecCodes.
|
|
404
|
+
|
|
405
|
+
---
|
|
406
|
+
|
|
407
|
+
## Building a distribution (wheel + sdist)
|
|
408
|
+
|
|
409
|
+
The package builds with the standard [PEP 517](https://peps.python.org/pep-0517/)
|
|
410
|
+
toolchain:
|
|
411
|
+
|
|
412
|
+
```bat
|
|
413
|
+
conda activate tethys_rasters
|
|
414
|
+
pip install -e ".[build]"
|
|
415
|
+
python -m build
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
This produces both artifacts in `dist/`:
|
|
419
|
+
|
|
420
|
+
```
|
|
421
|
+
dist/
|
|
422
|
+
├── meteoraster-3.0-py3-none-any.whl
|
|
423
|
+
└── meteoraster-3.0.tar.gz
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
Install the wheel anywhere (provided GDAL is available):
|
|
427
|
+
|
|
428
|
+
```bat
|
|
429
|
+
pip install dist/meteoraster-3.0-py3-none-any.whl
|
|
430
|
+
```
|
|
431
|
+
|
|
432
|
+
Notes:
|
|
433
|
+
- The version lives in [pyproject.toml](pyproject.toml) (`project.version`) and is
|
|
434
|
+
mirrored in `MeteoRaster.VERSION` — bump both together.
|
|
435
|
+
[.github/scripts/check_version.py](.github/scripts/check_version.py) enforces
|
|
436
|
+
that at release time.
|
|
437
|
+
- Building by hand is only needed to inspect an artifact locally. Actual releases
|
|
438
|
+
are produced by CI — see [Releasing](#releasing) below.
|
|
439
|
+
|
|
440
|
+
---
|
|
441
|
+
|
|
442
|
+
## Releasing
|
|
443
|
+
|
|
444
|
+
Releases are automated by
|
|
445
|
+
[.github/workflows/release.yml](.github/workflows/release.yml). To cut version
|
|
446
|
+
`X.Y`:
|
|
447
|
+
|
|
448
|
+
1. Bump the version in **both** [pyproject.toml](pyproject.toml) and
|
|
449
|
+
`MeteoRaster.VERSION`, then commit.
|
|
450
|
+
2. Tag and push:
|
|
451
|
+
|
|
452
|
+
```bat
|
|
453
|
+
git tag vX.Y
|
|
454
|
+
git push origin vX.Y
|
|
455
|
+
```
|
|
456
|
+
|
|
457
|
+
The workflow then verifies the two version declarations agree with the tag, builds
|
|
458
|
+
the wheel and sdist, runs `twine check --strict`, publishes to
|
|
459
|
+
[PyPI](https://pypi.org/project/meteoraster/), and attaches both artifacts to the
|
|
460
|
+
GitHub release.
|
|
461
|
+
|
|
462
|
+
To rehearse without touching PyPI, run the workflow manually from the **Actions**
|
|
463
|
+
tab and leave the target as `testpypi`; it builds and uploads to
|
|
464
|
+
[TestPyPI](https://test.pypi.org/project/meteoraster/) with no tag required.
|
|
465
|
+
|
|
466
|
+
> **PyPI uploads are immutable** — a version number can never be reused, even
|
|
467
|
+
> after deletion. Get the version right before pushing the tag.
|
|
468
|
+
|
|
469
|
+
### One-time setup (maintainers)
|
|
470
|
+
|
|
471
|
+
Publishing is authenticated with
|
|
472
|
+
[PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/) (OIDC):
|
|
473
|
+
GitHub mints a short-lived credential at publish time that PyPI accepts only for
|
|
474
|
+
this repository, this workflow file and this environment. **No API token is
|
|
475
|
+
created, stored or pasted anywhere.** Every value below is a public fact about
|
|
476
|
+
the repository — none of it is a secret, which is precisely the point.
|
|
477
|
+
|
|
478
|
+
**On PyPI** — once per index (repeat on [TestPyPI](https://test.pypi.org/) if you
|
|
479
|
+
want the rehearsal path):
|
|
480
|
+
|
|
481
|
+
1. *Your account → Publishing → Add a pending publisher* → GitHub, and fill in:
|
|
482
|
+
|
|
483
|
+
| Field | Value |
|
|
484
|
+
|---|---|
|
|
485
|
+
| PyPI project name | `meteoraster` |
|
|
486
|
+
| Owner | `FORESIGHT-ULisboa` |
|
|
487
|
+
| Repository | `Tethys-meteoraster` |
|
|
488
|
+
| Workflow name | `release.yml` |
|
|
489
|
+
| Environment | `pypi` (on TestPyPI: `testpypi`) |
|
|
490
|
+
|
|
491
|
+
The environment name must match the `environment:` key of the corresponding
|
|
492
|
+
job in [release.yml](.github/workflows/release.yml), or PyPI rejects the
|
|
493
|
+
exchange. A *pending* publisher is what you register before the project exists
|
|
494
|
+
on an index; it turns into an ordinary one on the first successful upload.
|
|
495
|
+
|
|
496
|
+
**On GitHub** — once:
|
|
497
|
+
|
|
498
|
+
2. *Settings → Environments* → create `pypi` (and `testpypi`).
|
|
499
|
+
3. Add a **required reviewer** to `pypi`. Every publish then pauses for an
|
|
500
|
+
explicit approval, so a tag push alone cannot ship a release.
|
|
501
|
+
|
|
502
|
+
Things worth keeping that way:
|
|
503
|
+
|
|
504
|
+
- **Do not add a PyPI API token to repository secrets.** Trusted Publishing
|
|
505
|
+
exists to remove that long-lived credential; a token in secrets is usable by
|
|
506
|
+
any workflow in the repository and stays valid until someone revokes it.
|
|
507
|
+
Nothing here needs one.
|
|
508
|
+
- The publisher trusts the workflow **by filename**, so anyone able to change
|
|
509
|
+
[release.yml](.github/workflows/release.yml) or push a `v*` tag can trigger a
|
|
510
|
+
publish. Protect `main` and restrict who may push tags; the environment
|
|
511
|
+
reviewer from step 3 is the backstop.
|
|
512
|
+
- Only the two publish jobs request `id-token: write`; the build job is
|
|
513
|
+
`contents: read` and the workflow default is `permissions: {}`. Keep that split
|
|
514
|
+
when editing it — the job that produces the artifacts never holds the identity
|
|
515
|
+
that can upload them.
|
|
516
|
+
|
|
517
|
+
---
|
|
518
|
+
|
|
519
|
+
## API reference
|
|
520
|
+
|
|
521
|
+
### `MeteoRaster`
|
|
522
|
+
|
|
523
|
+
**Construction & conversion**
|
|
524
|
+
|
|
525
|
+
| Member | Description |
|
|
526
|
+
|---|---|
|
|
527
|
+
| `MeteoRaster(data, latitudes, longitudes, production_datetime, leadtimes, units='unknown', variable='unknown', …)` | Build from a 5-D array (or a dict with the same keys). 1-D lat/lon are auto-meshed to 2-D; longitudes are wrapped to [−180, 180] and latitudes flipped if needed. |
|
|
528
|
+
| `copy()` | Deep copy. |
|
|
529
|
+
| `to_xarray()` | Convert to a labelled `xarray.DataArray`. |
|
|
530
|
+
|
|
531
|
+
**Subsetting, merging & resampling**
|
|
532
|
+
|
|
533
|
+
| Member | Description |
|
|
534
|
+
|---|---|
|
|
535
|
+
| `get_cropped(from_prod_date, to_prod_date, from_lat, to_lat, from_lon, to_lon, from_leadtime, to_leadtime)` | Crop in time and space; returns a new `MeteoRaster`. |
|
|
536
|
+
| `join(meteoRaster, strickt=False, trim=False)` | Concatenate another raster along production dates (aligns ensembles/leadtimes). |
|
|
537
|
+
| `trim()` | Drop leading/trailing all-NaN production dates. |
|
|
538
|
+
| `adjust_leadtimes(period='months')` | Align leadtimes to relative periods (experimental). |
|
|
539
|
+
| `resample_timestep(rule, fun=np.mean)` | Resample production dates with pandas rules (experimental). |
|
|
540
|
+
|
|
541
|
+
**Extraction & aggregation**
|
|
542
|
+
|
|
543
|
+
| Member | Description |
|
|
544
|
+
|---|---|
|
|
545
|
+
| `get_values_from_latlon(lat, lon)` | Time series from the nearest pixel as a `DataFrame`. |
|
|
546
|
+
| `get_values_from_KML(kml, nameField=None, …)` | Area-weighted aggregate over KML polygons → `(agg, centroids)`. |
|
|
547
|
+
| `get_quantiles_from_KML(kml, nameField=None, quantiles=…, pool=('space',), resampling=None, …)` | Area-weighted spatial quantiles over KML zones → `(agg, centroids)`. `pool` names the axes folded into the sample; `resampling` is the block size within a pooled time axis. |
|
|
548
|
+
| `get_values_from_latlon_by_event(production_date_dataframe)` | *(static)* Reindex a production-date frame by event date. |
|
|
549
|
+
|
|
550
|
+
Since **v3.0** `get_values_from_KML` runs on `meteoraster.zonal` (below). Its numbers
|
|
551
|
+
changed for any zone touching the edge of the grid: the legacy builder gave the
|
|
552
|
+
outermost row and column no weight and took its validity threshold from the first
|
|
553
|
+
zone. `ensemble_member` labels are now 0-based, matching the other extractors.
|
|
554
|
+
|
|
555
|
+
Since **v3.1** the default weights are true WGS84 areas (`area_weighting='ellipsoidal'`)
|
|
556
|
+
instead of square degrees. Square degrees over-weight the poleward side of a zone by
|
|
557
|
+
roughly tan(lat) x lat-span: measured 1 % less rain over a Zambezi-sized basin, 0.02 %
|
|
558
|
+
on the Vesdre catchments, 8 % on a single cell weight for a 5 deg zone at 60 N. Pass
|
|
559
|
+
`area_weighting='planar'` to reproduce v3.0 exactly.
|
|
560
|
+
|
|
561
|
+
**`meteoraster.zonal`** — needs no `MeteoRaster`; takes plain coordinate arrays.
|
|
562
|
+
|
|
563
|
+
| Member | Description |
|
|
564
|
+
|---|---|
|
|
565
|
+
| `coverage_from_kml(kml, latitudes, longitudes, name_field=None, …)` | Build a `Coverage` from KML Placemark polygons. Namespace-agnostic, and it does not touch the input file. |
|
|
566
|
+
| `coverage_from_geometries(latitudes, longitudes, geometries, names=None, …)` | Same from `ogr.Geometry`, WKB bytes or WKT — so PostGIS geometry feeds straight in. |
|
|
567
|
+
| `apply_coverage(coverage, values, min_coverage=0.8)` | Area-weighted mean per zone of any array whose last two axes are (y, x). Renormalises for missing data and applies the threshold per zone and per slice. |
|
|
568
|
+
| `quantiles_from_coverage(coverage, values, quantiles, pool_axes=(), …)` | Area-weighted spatial quantiles per zone. |
|
|
569
|
+
| `weighted_quantiles(values, weights, quantiles, method='hazen')` | The quantile primitive. Reduces exactly to `np.quantile` for equal weights. |
|
|
570
|
+
| `Coverage.save/load`, `Coverage.matches`, `grid_fingerprint` | Persist a coverage to a pickle-free `.npz` and check it against a grid; the fingerprint hashes the coordinates, so a shifted grid of the same shape is a miss. |
|
|
571
|
+
| `cell_corners`, `cell_areas`, `cell_bounds` | The grid geometry, extrapolated half a cell at the boundary so no cell is skipped. `cell_areas(..., 'ellipsoidal')` gives km². |
|
|
572
|
+
| `ring_area`, `geometry_area`, `authalic_latitude` | Exact WGS84 areas (km²) of lon/lat rings and ogr polygons, used by `area_weighting='ellipsoidal'`. |
|
|
573
|
+
|
|
574
|
+
**`meteoraster.io`** — the portable read path.
|
|
575
|
+
|
|
576
|
+
| Member | Description |
|
|
577
|
+
|---|---|
|
|
578
|
+
| `read_header(file)` | `Header` with coordinates, shape, dtype, units and the `complete` flag; reads no values. |
|
|
579
|
+
| `iter_productions(file, productions=None, leadtimes=None, reduce_members=None, dtype=None)` | Yield `(production_datetime, block)`; peak memory is one block. |
|
|
580
|
+
| `read_cube(file, …)` | Fill a whole or partial cube production by production. |
|
|
581
|
+
| `open_raster(file, engine=None, **kwargs)` | Context manager over `xr.open_dataset` with engine fallback and pinned decoding. |
|
|
582
|
+
| `read_completeness(file)` | The `complete` flag alone; `None` when the file has none. |
|
|
583
|
+
| `available_engines()` | Which of `h5netcdf`, `netcdf4`, `scipy` this install can use. |
|
|
584
|
+
|
|
585
|
+
**Quality & diagnostics**
|
|
586
|
+
|
|
587
|
+
| Member | Description |
|
|
588
|
+
|---|---|
|
|
589
|
+
| `is_complete(full_ensemble=True, space_completeness=False)` | Whether every production×leadtime slice has finite data. |
|
|
590
|
+
| `get_complete_index(full_ensemble=True, space_completeness=False)` | Boolean `DataFrame` (production × leadtime) of completeness. |
|
|
591
|
+
| `get_missing()` | Fraction of missing pixels per leadtime/ensemble member. |
|
|
592
|
+
| `get_completeness(file)` | *(classmethod)* Read the `complete` flag from a saved file. |
|
|
593
|
+
|
|
594
|
+
**I/O**
|
|
595
|
+
|
|
596
|
+
| Member | Description |
|
|
597
|
+
|---|---|
|
|
598
|
+
| `save(file, complevel=1, complete=None, tmp_dir=None, mode=0o644, engine=None)` | Write compressed NetCDF via `h5netcdf`. Staged in the destination directory and swapped in with `os.replace`, so nothing goes through the system temp dir and the swap is atomic. |
|
|
599
|
+
| `load(file, verbose=None, productions=None, leadtimes=None, dtype=None, engine=None)` | *(classmethod)* Reload a NetCDF written by `save`, optionally only part of it. |
|
|
600
|
+
| `read_header(file)` / `iter_productions(file, …)` | *(classmethods)* Metadata-only and streaming reads; see `meteoraster.io`. |
|
|
601
|
+
|
|
602
|
+
**Visualisation**
|
|
603
|
+
|
|
604
|
+
| Member | Description |
|
|
605
|
+
|---|---|
|
|
606
|
+
| `create_plot(central_longitude, …)` | Create a `cartopy` axes. |
|
|
607
|
+
| `plot_mean(ax=None, coastline=False, borders=False, colorbar=True, cmap='viridis', …)` | Map of the ensemble/temporal mean → `(ax, cbar)`. |
|
|
608
|
+
| `plot_mean_projected`, `plot_coordinates`, `plot_seasonal`, `plot_availability` | Projected mean, grid coordinates, seasonal cycle, data availability. |
|
|
609
|
+
| `add_shapefile(ax, shapefile_path, …)` / `add_shape(ax, path, …)` | Overlay shapefile geometries/boundaries. |
|
|
610
|
+
|
|
611
|
+
> Deprecated camelCase aliases (`getCropped`, `getValuesFromKML`,
|
|
612
|
+
> `getDataFromLatLon`, `getQuantilesFromKML`, `resampleTimeStep`) still work but
|
|
613
|
+
> emit a `DeprecationWarning`; use the snake_case names above.
|
|
614
|
+
|
|
615
|
+
### `Resample`
|
|
616
|
+
|
|
617
|
+
| Member | Description |
|
|
618
|
+
|---|---|
|
|
619
|
+
| `Resample.resample(data, timestep_frequency, resamplingType, date_from=None, date_to=None, print_func=None)` | *(classmethod)* Resample a production×leadtime `DataFrame`. `resamplingType` ∈ {`'sum'`, `'mean'`, `'linear'`, `'max'`}. |
|
|
620
|
+
|
|
621
|
+
---
|
|
622
|
+
|
|
623
|
+
## License
|
|
624
|
+
|
|
625
|
+
Licensed under the **Apache License, Version 2.0** — see [LICENSE](LICENSE) and
|
|
626
|
+
[NOTICE](NOTICE).
|