pyozwald 0.1.0__tar.gz

pyozwald-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Borevitz Lab, Australian National University, and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pyozwald-0.1.0/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.4
+Name: pyozwald
+Version: 0.1.0
+Summary: Cached OzWALD daily meteorology and 8-day biophysical series for Australia — fetch once per grid point, never twice
+Author: Borevitz Lab, Australian National University
+Author-email: Yasar Adeel Ansari <u6737670@anu.edu.au>
+License: MIT
+Project-URL: Homepage, https://github.com/thestochasticman/pyozwald
+Project-URL: Repository, https://github.com/thestochasticman/pyozwald
+Project-URL: Issues, https://github.com/thestochasticman/pyozwald/issues
+Keywords: ozwald,climate,australia,remote-sensing,agriculture,time-series
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: attrs
+Requires-Dist: typing_extensions
+Requires-Dist: pandas
+Requires-Dist: xarray
+Requires-Dist: netcdf4
+Dynamic: license-file
+
+# pyozwald
+
+**Cached [OzWALD](https://www.wenfo.org/ozwald/) time series for
+Australia — fetch once per grid point, never twice.** OzWALD is ANU's
+Water and Landscape Dynamics dataset: modelled daily meteorology
+(~5 km) and 8-day biophysical variables (~500 m, MODIS-derived) served
+as one OPeNDAP NetCDF per variable per year. Every observation this
+machine ever samples lands in one SQLite store, so repeat requests,
+nearby coordinates in the same cell, and extended date ranges all
+reuse the same rows. Part of the
+[Borevitz Lab](https://borevitzlab.anu.edu.au/) ecosystem.
+
+## How it works
+
+```
+{data_root}/ozwald_store/
+└── ozwald.db
+    ├── observations(point, cadence, variable, date, value)
+    └── coverage(point, cadence, variable, year, through)
+```
+
+- Coordinates snap to a dedup grid matching each product's native
+  resolution — 0.05° for daily meteorology, 0.005° for the 8-day
+  variables — so nearby requests share one stored series per cadence.
+- OzWALD's unit of delivery is one NetCDF per (variable, year), so
+  that's the unit of the coverage ledger. `Store.get_df(...)` diffs
+  the requested years × variables against it and samples **only the
+  missing cells**; a whole year is stored even when a sub-range was
+  requested, since the marginal cost is nil and it maximises reuse.
+- `through` records the last date a year's file actually contained —
+  an in-progress year keeps being re-fetched until complete, then
+  never again.
+- Writes are transactional (SQLite/WAL): a crash mid-fetch leaves the
+  cell unrecorded, and the next run re-fetches it.
+
+## Usage
+
+The core API is **troi-agnostic** — a coordinate, dates, and a cadence:
+
+```python
+from datetime import date
+from pyozwald.store import Store
+
+store = Store()
+
+met = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))
+#     daily meteorology: time, Pg, Tmax, Tmin, Uavg, Ueff, VPeff, ...
+
+veg = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31),
+                   cadence='8day', variables=['NDVI', 'LAI', 'GPP'])
+#     8-day biophysical series on the ~500 m grid
+
+store.fill(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))  # → 0: already local
+```
+
+Pipelines that speak the shared `troi.troi.Troi` use the
+adapters (evaluated at the bbox centre):
+
+```python
+df = store.get_df_troi(troi, cadence='daily')
+```
+
+`download_ozwald_daily(troi)` and `download_ozwald_8day(troi)`
+remain as thin wrappers.
+
+## Performance
+
+Live measurements against NCI THREDDS — one grid point:
+
+| Scenario | Fetched | Time |
+|---|---|---|
+| Cold fill — 2 daily variables × 1 year | 2 cells | 2.3 s |
+| Same request again | nothing | **0.0 s** |
+| Nearby coordinate, same ~5 km cell | nothing | **0.0 s** |
+| Date range extended −1 year | 2 cells — *the new year only* | 1.8 s |
+| 8-day NDVI, one year | 1 cell | 0.7 s |
+| Read cached year (365 × 2) | — | 0.01 s |
+
+(One *cell* = one variable × one year at one grid point.) Store
+footprint: ~100 KB for the five cells above. Absolute times vary with
+network and THREDDS load; the zeros are the point — they are ledger
+lookups, no network involved.
+
+## Install
+
+### pip
+
+```bash
+pip install git+https://github.com/thestochasticman/pyozwald.git
+```
+
+Dependencies (the `troi` core included, pulled from GitHub) are
+declared in `pyproject.toml` and installed automatically.
+
+### From source
+
+```bash
+git clone https://github.com/thestochasticman/pyozwald.git
+cd pyozwald
+pip install -e .
+```
+
+Package design (shared across the lab's packages — no inheritance,
+composition only):
+
+- **`Troi`** (from `troi`) — identity: what region, what dates.
+- **`OzWALD`** (`pyozwald.ozwald`) — config: endpoint, variable
+  catalogs per cadence, dedup grid steps.
+- **`Paths`** (`pyozwald.paths`) — derived location of the store for a
+  given `Config`.
+- **`grid`** — the dedup grids (pure, offline-testable math).
+- **`Store`** (`pyozwald.store`) — ties them together.
+
+## Test
+
+```bash
+# offline (pure math + synthetic store):
+python pyozwald/grid.py     # True
+python pyozwald/paths.py    # True
+python pyozwald/store.py    # True
+
+# live (small real samples from NCI THREDDS, incl. dedup assertions):
+python pyozwald/download_ozwald.py  # True
+```

pyozwald-0.1.0/README.md

@@ -0,0 +1,124 @@
+# pyozwald
+
+**Cached [OzWALD](https://www.wenfo.org/ozwald/) time series for
+Australia — fetch once per grid point, never twice.** OzWALD is ANU's
+Water and Landscape Dynamics dataset: modelled daily meteorology
+(~5 km) and 8-day biophysical variables (~500 m, MODIS-derived) served
+as one OPeNDAP NetCDF per variable per year. Every observation this
+machine ever samples lands in one SQLite store, so repeat requests,
+nearby coordinates in the same cell, and extended date ranges all
+reuse the same rows. Part of the
+[Borevitz Lab](https://borevitzlab.anu.edu.au/) ecosystem.
+
+## How it works
+
+```
+{data_root}/ozwald_store/
+└── ozwald.db
+    ├── observations(point, cadence, variable, date, value)
+    └── coverage(point, cadence, variable, year, through)
+```
+
+- Coordinates snap to a dedup grid matching each product's native
+  resolution — 0.05° for daily meteorology, 0.005° for the 8-day
+  variables — so nearby requests share one stored series per cadence.
+- OzWALD's unit of delivery is one NetCDF per (variable, year), so
+  that's the unit of the coverage ledger. `Store.get_df(...)` diffs
+  the requested years × variables against it and samples **only the
+  missing cells**; a whole year is stored even when a sub-range was
+  requested, since the marginal cost is nil and it maximises reuse.
+- `through` records the last date a year's file actually contained —
+  an in-progress year keeps being re-fetched until complete, then
+  never again.
+- Writes are transactional (SQLite/WAL): a crash mid-fetch leaves the
+  cell unrecorded, and the next run re-fetches it.
+
+## Usage
+
+The core API is **troi-agnostic** — a coordinate, dates, and a cadence:
+
+```python
+from datetime import date
+from pyozwald.store import Store
+
+store = Store()
+
+met = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))
+#     daily meteorology: time, Pg, Tmax, Tmin, Uavg, Ueff, VPeff, ...
+
+veg = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31),
+                   cadence='8day', variables=['NDVI', 'LAI', 'GPP'])
+#     8-day biophysical series on the ~500 m grid
+
+store.fill(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))  # → 0: already local
+```
+
+Pipelines that speak the shared `troi.troi.Troi` use the
+adapters (evaluated at the bbox centre):
+
+```python
+df = store.get_df_troi(troi, cadence='daily')
+```
+
+`download_ozwald_daily(troi)` and `download_ozwald_8day(troi)`
+remain as thin wrappers.
+
+## Performance
+
+Live measurements against NCI THREDDS — one grid point:
+
+| Scenario | Fetched | Time |
+|---|---|---|
+| Cold fill — 2 daily variables × 1 year | 2 cells | 2.3 s |
+| Same request again | nothing | **0.0 s** |
+| Nearby coordinate, same ~5 km cell | nothing | **0.0 s** |
+| Date range extended −1 year | 2 cells — *the new year only* | 1.8 s |
+| 8-day NDVI, one year | 1 cell | 0.7 s |
+| Read cached year (365 × 2) | — | 0.01 s |
+
+(One *cell* = one variable × one year at one grid point.) Store
+footprint: ~100 KB for the five cells above. Absolute times vary with
+network and THREDDS load; the zeros are the point — they are ledger
+lookups, no network involved.
+
+## Install
+
+### pip
+
+```bash
+pip install git+https://github.com/thestochasticman/pyozwald.git
+```
+
+Dependencies (the `troi` core included, pulled from GitHub) are
+declared in `pyproject.toml` and installed automatically.
+
+### From source
+
+```bash
+git clone https://github.com/thestochasticman/pyozwald.git
+cd pyozwald
+pip install -e .
+```
+
+Package design (shared across the lab's packages — no inheritance,
+composition only):
+
+- **`Troi`** (from `troi`) — identity: what region, what dates.
+- **`OzWALD`** (`pyozwald.ozwald`) — config: endpoint, variable
+  catalogs per cadence, dedup grid steps.
+- **`Paths`** (`pyozwald.paths`) — derived location of the store for a
+  given `Config`.
+- **`grid`** — the dedup grids (pure, offline-testable math).
+- **`Store`** (`pyozwald.store`) — ties them together.
+
+## Test
+
+```bash
+# offline (pure math + synthetic store):
+python pyozwald/grid.py     # True
+python pyozwald/paths.py    # True
+python pyozwald/store.py    # True
+
+# live (small real samples from NCI THREDDS, incl. dedup assertions):
+python pyozwald/download_ozwald.py  # True
+```

pyozwald-0.1.0/pyozwald/__init__.py

@@ -0,0 +1,8 @@
+from pyozwald.ozwald import OzWALD, defaultozwald
+from pyozwald.paths import Paths
+
+__all__ = [
+    'OzWALD',
+    'defaultozwald',
+    'Paths',
+]

pyozwald-0.1.0/pyozwald/download_ozwald.py

@@ -0,0 +1,82 @@
+"""Fetch OzWALD tables for a troi — via the machine-wide store.
+
+Thin compatibility wrappers: the heavy lifting (grid snapping,
+(variable x year) coverage ledger, OPeNDAP sampling) lives in
+:class:`pyozwald.store.Store`. Kept as a module so the familiar
+``download_ozwald_daily(troi)`` / ``download_ozwald_8day(troi)``
+entry points survive.
+"""
+import pandas as pd
+from troi import Troi
+from pyozwald.ozwald import OzWALD, defaultozwald
+
+
+def download_ozwald_daily(troi: Troi, variables: list[str] = None,
+                          ozwald: OzWALD = defaultozwald) -> pd.DataFrame:
+    """Daily OzWALD meteorology for the centre of ``troi.bbox``.
+
+    Fetches only the (variable, year) cells of the grid point that no
+    previous request has covered.
+
+    Returns:
+        pandas.DataFrame: One row per day with a ``time`` column and one
+        column per variable.
+    """
+    from pyozwald.store import Store
+    store = Store(config=troi.config, ozwald=ozwald)
+    return store.get_df_troi(troi, cadence='daily', variables=variables)
+
+
+def download_ozwald_8day(troi: Troi, variables: list[str] = None,
+                         ozwald: OzWALD = defaultozwald) -> pd.DataFrame:
+    """8-day OzWALD biophysical variables for the centre of ``troi.bbox``."""
+    from pyozwald.store import Store
+    store = Store(config=troi.config, ozwald=ozwald)
+    return store.get_df_troi(troi, cadence='8day', variables=variables)
+
+
+def test_live_fetch_and_dedup():
+    """Live: cold fetch of one daily variable-year; repeats and nearby
+    coordinates fetch nothing; extending a year fetches only the new year."""
+    import tempfile
+    from datetime import date
+    from troi import Config
+    from pyozwald.store import Store
+
+    tmpdir = tempfile.mkdtemp(prefix='pyozwald_live_test_')
+    cfg = Config(out_dir=tmpdir, tmp_dir=tmpdir)
+    store = Store(config=cfg)
+    lat, lon = -33.516, 148.373
+
+    fetched = store.fill(lat, lon, date(2023, 1, 1), date(2023, 12, 31),
+                         variables=['Tmax'])
+    if fetched != 1:  # one (variable, year) cell
+        return False
+    df = store.get_df(lat, lon, date(2023, 1, 1), date(2023, 12, 31),
+                      variables=['Tmax'])
+    if len(df) != 365 or 'Tmax' not in df.columns or df['Tmax'].isna().all():
+        return False
+    # identical repeat -> nothing
+    if store.fill(lat, lon, date(2023, 1, 1), date(2023, 12, 31),
+                  variables=['Tmax']) != 0:
+        return False
+    # ~300 m away, same ~5 km daily cell -> nothing
+    if store.fill(-33.514, 148.371, date(2023, 6, 1), date(2023, 6, 30),
+                  variables=['Tmax']) != 0:
+        return False
+    # extend one year back -> exactly one new cell
+    if store.fill(lat, lon, date(2022, 1, 1), date(2023, 12, 31),
+                  variables=['Tmax']) != 1:
+        return False
+    # an 8day variable is its own cell on its own grid
+    veg = store.get_df(lat, lon, date(2023, 1, 1), date(2023, 12, 31),
+                       cadence='8day', variables=['NDVI'])
+    return len(veg) > 30 and 'NDVI' in veg.columns
+
+
+def test():
+    return test_live_fetch_and_dedup()
+
+
+if __name__ == '__main__':
+    print(test())

pyozwald-0.1.0/pyozwald/grid.py

@@ -0,0 +1,62 @@
+"""Dedup grids every stored OzWALD series is keyed to.
+
+OzWALD serves two products at different native resolutions — daily
+meteorology on a ~5 km (0.05°) grid, 8-day biophysical variables on a
+~500 m (0.005°) grid. Requested coordinates snap to the matching grid
+spacing, so nearby coordinates inside one cell resolve to the same
+point and a point's series is only ever fetched once per
+(variable, year).
+
+All functions here are pure — no I/O, no store access.
+"""
+
+LON_MIN, LON_MAX = 110.0, 155.0
+LAT_MIN, LAT_MAX = -45.0, -9.0
+
+
+def snap(lat: float, lon: float, step: float) -> tuple[float, float]:
+    """Nearest grid point ``(lat, lon)`` at ``step`` degrees spacing."""
+    return (round(lat / step) * step, round(lon / step) * step)
+
+
+def point_id(lat: float, lon: float, step: float) -> str:
+    """Stable string key for the grid cell containing ``(lat, lon)``."""
+    slat, slon = snap(lat, lon, step)
+    return f'{slat:.3f},{slon:.3f}'
+
+
+def in_bounds(lat: float, lon: float) -> bool:
+    """True iff the coordinate lies inside OzWALD's Australian extent."""
+    return LAT_MIN <= lat <= LAT_MAX and LON_MIN <= lon <= LON_MAX
+
+
+def test_snap_is_idempotent():
+    slat, slon = snap(-33.51606, 148.37265, 0.05)
+    return snap(slat, slon, 0.05) == (slat, slon)
+
+
+def test_nearby_points_share_a_cell():
+    # ~1 km apart inside one ~5 km daily cell -> same point
+    return point_id(-33.514, 148.371, 0.05) == point_id(-33.516, 148.373, 0.05)
+
+
+def test_fine_grid_separates_them():
+    # ...but on the ~500 m 8-day grid they are different cells
+    return point_id(-33.514, 148.371, 0.005) != point_id(-33.516, 148.373, 0.005)
+
+
+def test_bounds():
+    return in_bounds(-33.5, 148.4) and not in_bounds(-33.5, 100.0)
+
+
+def test():
+    return all([
+        test_snap_is_idempotent(),
+        test_nearby_points_share_a_cell(),
+        test_fine_grid_separates_them(),
+        test_bounds(),
+    ])
+
+
+if __name__ == '__main__':
+    print(test())

pyozwald-0.1.0/pyozwald/ozwald.py

@@ -0,0 +1,59 @@
+from attrs import frozen
+
+@frozen
+class OzWALD:
+    """Endpoint and variable configuration for OzWALD OPeNDAP requests.
+
+    Two products, two cadences: ``daily`` meteorology (~5 km grid) and
+    ``8day`` biophysical variables (~500 m grid). ``step`` gives the
+    dedup grid spacing per cadence — requests are snapped to it so
+    nearby coordinates share one stored series (matching the native
+    resolution of each product).
+    """
+
+    base_url: str = 'https://thredds.nci.org.au/thredds/dodsC/ub8/au/OzWALD'
+
+    daily = {
+        'Pg':      {'path': 'daily/meteo/Pg',      'file': 'OzWALD.daily.Pg.{year}.nc'},
+        'Tmax':    {'path': 'daily/meteo/Tmax',    'file': 'OzWALD.Tmax.{year}.nc'},
+        'Tmin':    {'path': 'daily/meteo/Tmin',    'file': 'OzWALD.Tmin.{year}.nc'},
+        'Uavg':    {'path': 'daily/meteo/Uavg',    'file': 'OzWALD.Uavg.{year}.nc'},
+        'Ueff':    {'path': 'daily/meteo/Ueff',    'file': 'OzWALD.Ueff.{year}.nc'},
+        'VPeff':   {'path': 'daily/meteo/VPeff',   'file': 'OzWALD.VPeff.{year}.nc'},
+        'kTavg':   {'path': 'daily/meteo/kTavg',   'file': 'OzWALD.kTavg.{year}.nc'},
+        'kTeff':   {'path': 'daily/meteo/kTeff',   'file': 'OzWALD.kTeff.{year}.nc'},
+        'DWLReff': {'path': 'daily/meteo/DWLReff', 'file': 'OzWALD.DWLReff.{year}.nc'},
+    }
+
+    eight_day = {
+        'LAI':   {'path': '8day/LAI',   'file': 'OzWALD.LAI.{year}.nc'},
+        'GPP':   {'path': '8day/GPP',   'file': 'OzWALD.GPP.{year}.nc'},
+        'NDVI':  {'path': '8day/NDVI',  'file': 'OzWALD.NDVI.{year}.nc'},
+        'EVI':   {'path': '8day/EVI',   'file': 'OzWALD.EVI.{year}.nc'},
+        'PV':    {'path': '8day/PV',    'file': 'OzWALD.PV.{year}.nc'},
+        'NPV':   {'path': '8day/NPV',   'file': 'OzWALD.NPV.{year}.nc'},
+        'BS':    {'path': '8day/BS',    'file': 'OzWALD.BS.{year}.nc'},
+        'FMC':   {'path': '8day/FMC',   'file': 'OzWALD.FMC.{year}.nc'},
+        'Qtot':  {'path': '8day/Qtot',  'file': 'OzWALD.Qtot.{year}.nc'},
+        'Ssoil': {'path': '8day/Ssoil', 'file': 'OzWALD.Ssoil.{year}.nc'},
+        'OW':    {'path': '8day/OW',    'file': 'OzWALD.OW.{year}.nc'},
+        'SN':    {'path': '8day/SN',    'file': 'OzWALD.SN.{year}.nc'},
+        'Alb':   {'path': '8day/Alb',   'file': 'OzWALD.Alb.{year}.nc'},
+    }
+
+    # Dedup grid spacing (degrees) per cadence, matching each product's
+    # native resolution.
+    steps = {'daily': 0.05, '8day': 0.005}
+
+    def catalog(self, cadence: str) -> dict:
+        if cadence == 'daily':
+            return self.daily
+        if cadence == '8day':
+            return self.eight_day
+        raise ValueError(f"cadence must be 'daily' or '8day', got {cadence!r}")
+
+    def get_url(self, cadence: str, variable: str, year: int) -> str:
+        info = self.catalog(cadence)[variable]
+        return f'{self.base_url}/{info["path"]}/{info["file"].format(year=year)}'
+
+defaultozwald = OzWALD()

pyozwald-0.1.0/pyozwald/paths.py

@@ -0,0 +1,55 @@
+"""Derived on-disk location of the machine-wide OzWALD store.
+
+The store is keyed by :class:`troi.Config` (one store per
+data root, shared by every request on this machine). Rule of thumb
+across the lab's packages: user-settable inputs → Config, derived
+locations → Paths. No inheritance — composition only.
+"""
+from attrs import frozen, field
+from troi import Config, config as default_config
+
+
+@frozen
+class Paths:
+    """Where the pyozwald store lives for a given Config.
+
+    Attributes:
+        config: The :class:`troi.Config` supplying the data root.
+        root: Store directory (``{config.tmp_dir}/ozwald_store``).
+        db: The SQLite database holding every observation and the
+            coverage ledger.
+
+    Example:
+        ```python
+        from pyozwald.paths import Paths
+
+        Paths().db  # '~/Downloads/Troi-Tmp/ozwald_store/ozwald.db'
+        ```
+    """
+
+    config: Config = default_config
+
+    root: str = field(init=False)
+    db: str = field(init=False)
+
+    root.default(lambda s: f'{s.config.tmp_dir}/ozwald_store')
+    db.default(lambda s: f'{s.root}/ozwald.db')
+
+
+def test_paths_derive_from_config():
+    import tempfile
+    tmpdir = tempfile.mkdtemp(prefix='pyozwald_paths_test_')
+    cfg = Config(out_dir=tmpdir, tmp_dir=tmpdir)
+    paths = Paths(cfg)
+    return (
+        paths.root == f'{tmpdir}/ozwald_store'
+        and paths.db == f'{tmpdir}/ozwald_store/ozwald.db'
+    )
+
+
+def test():
+    return test_paths_derive_from_config()
+
+
+if __name__ == '__main__':
+    print(test())

pyozwald-0.1.0/pyozwald/store.py

@@ -0,0 +1,328 @@
+"""One machine-wide OzWALD store that fills itself on demand.
+
+Every observation this machine ever fetches lands in a single SQLite
+database, keyed by dedup grid point (:mod:`pyozwald.grid`), cadence,
+variable and date:
+
+    {config.tmp_dir}/ozwald_store/
+    └── ozwald.db
+        ├── observations(point, cadence, variable, date, value)
+        └── coverage(point, cadence, variable, year, through)
+
+OzWALD is served as one OPeNDAP NetCDF per (variable, year), so the
+unit of fetching — and of the coverage ledger — is the
+``(point, variable, year)`` cell. ``Store.get_df(lat, lon, start, end)``
+diffs the requested years x variables against the ledger and fetches
+only the missing cells; a whole year is stored even when a sub-range
+was requested, since the marginal cost is nil and it maximises reuse.
+``through`` records the last date a year's file actually contained —
+an in-progress year keeps being re-fetched until it is complete, then
+never again.
+"""
+import sqlite3
+from attrs import frozen, field
+from datetime import date
+from os import makedirs
+
+import pandas as pd
+
+from troi import Config, config as default_config
+from pyozwald import grid
+from pyozwald.ozwald import OzWALD, defaultozwald
+from pyozwald.paths import Paths
+
+_SCHEMA = """
+CREATE TABLE IF NOT EXISTS observations (
+    point    TEXT NOT NULL,
+    cadence  TEXT NOT NULL,
+    variable TEXT NOT NULL,
+    date     TEXT NOT NULL,
+    value    REAL,
+    PRIMARY KEY (point, cadence, variable, date)
+) WITHOUT ROWID;
+
+CREATE TABLE IF NOT EXISTS coverage (
+    point    TEXT NOT NULL,
+    cadence  TEXT NOT NULL,
+    variable TEXT NOT NULL,
+    year     INTEGER NOT NULL,
+    through  TEXT NOT NULL,
+    PRIMARY KEY (point, cadence, variable, year)
+) WITHOUT ROWID;
+"""
+
+
+def year_covered(through: str | None, year: int, end: date) -> bool:
+    """Is a year's cell already sufficient for a request ending at ``end``?
+
+    ``through`` is the last date the year's file contained when last
+    fetched (None = never fetched). The cell satisfies the request iff
+    it reaches the earlier of the requested end and the year's own end —
+    so completed years are fetched exactly once, and an in-progress year
+    is re-fetched only while the request actually needs newer days.
+    """
+    if through is None:
+        return False
+    return date.fromisoformat(through) >= min(end, date(year, 12, 31))
+
+
+@frozen
+class Store:
+    """The machine-wide OzWALD store: one ledger, zero re-fetches.
+
+    Composed from :class:`troi.Config` (where the store
+    lives) and :class:`pyozwald.ozwald.OzWALD` (endpoint + catalogs +
+    dedup grid steps). No inheritance.
+
+    Example:
+        ```python
+        from datetime import date
+        from pyozwald.store import Store
+
+        store = Store()
+        met = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))
+        veg = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31),
+                           cadence='8day', variables=['NDVI', 'LAI'])
+        ```
+    """
+
+    config: Config = default_config
+    ozwald: OzWALD = defaultozwald
+    paths: Paths = field(init=False)
+
+    paths.default(lambda s: Paths(s.config))
+
+    def __attrs_post_init__(s):
+        makedirs(s.paths.root, exist_ok=True)
+
+    def _db(s) -> sqlite3.Connection:
+        db = sqlite3.connect(s.paths.db)
+        db.execute('PRAGMA journal_mode=WAL')
+        db.executescript(_SCHEMA)
+        return db
+
+    def _resolve(s, lat, lon, cadence, variables):
+        if not grid.in_bounds(lat, lon):
+            raise ValueError(f'({lat}, {lon}) is outside the OzWALD extent')
+        catalog = s.ozwald.catalog(cadence)
+        variables = list(variables) if variables else list(catalog)
+        unknown = set(variables) - set(catalog)
+        if unknown:
+            raise ValueError(f'Unknown {cadence} variable(s): {sorted(unknown)}')
+        return grid.point_id(lat, lon, s.ozwald.steps[cadence]), variables
+
+    # -- fill -------------------------------------------------------------
+
+    def fill(s, lat: float, lon: float, start: date, end: date,
+             cadence: str = 'daily', variables: list[str] = None) -> int:
+        """Ensure every (variable x year) cell covering the request is
+        populated for the grid point containing ``(lat, lon)``.
+
+        Returns the number of (variable, year) cells actually fetched —
+        0 means full coverage already existed and no network was touched.
+        """
+        pid, variables = s._resolve(lat, lon, cadence, variables)
+        db = s._db()
+        try:
+            fetched = 0
+            for variable in variables:
+                for year in range(start.year, end.year + 1):
+                    row = db.execute(
+                        'SELECT through FROM coverage WHERE point=? AND cadence=? '
+                        'AND variable=? AND year=?', (pid, cadence, variable, year),
+                    ).fetchone()
+                    if year_covered(row[0] if row else None, year, end):
+                        continue
+                    s._fetch_year(db, pid, cadence, variable, year)
+                    fetched += 1
+            return fetched
+        finally:
+            db.close()
+
+    def _fetch_year(s, db, pid: str, cadence: str, variable: str, year: int) -> None:
+        """Sample one (variable, year) NetCDF at the grid point and store
+        the whole year's series."""
+        import xarray as xr
+        slat, slon = (float(v) for v in pid.split(','))
+        url = s.ozwald.get_url(cadence, variable, year)
+        ds = xr.open_dataset(url)
+        try:
+            data = ds.sel(latitude=slat, longitude=slon, method='nearest')[variable].load()
+        finally:
+            ds.close()
+        # Some yearly files carry int32 time — normalise to datetime64.
+        times = pd.to_datetime(data.time.values)
+        values = data.values
+        with db:
+            db.executemany(
+                'INSERT OR REPLACE INTO observations (point, cadence, variable, date, value) '
+                'VALUES (?, ?, ?, ?, ?)',
+                [(pid, cadence, variable, str(t.date()),
+                  None if pd.isna(v) else float(v))
+                 for t, v in zip(times, values)],
+            )
+            db.execute(
+                'INSERT OR REPLACE INTO coverage (point, cadence, variable, year, through) '
+                'VALUES (?, ?, ?, ?, ?)',
+                (pid, cadence, variable, year, str(times.max().date())),
+            )
+
+    # -- read -------------------------------------------------------------
+
+    def get_df(s, lat: float, lon: float, start: date, end: date,
+               cadence: str = 'daily', variables: list[str] = None) -> pd.DataFrame:
+        """Return the OzWALD table for ``(lat, lon)`` x ``[start, end]``,
+        fetching only what's missing first.
+
+        Troi-agnostic — the data layer of the package. Pipelines that
+        speak :class:`troi.Troi` use :meth:`get_df_troi`.
+
+        Args:
+            lat: Latitude in decimal degrees (EPSG:4326).
+            lon: Longitude in decimal degrees.
+            start: Inclusive start date.
+            end: Inclusive end date.
+            cadence: ``'daily'`` (meteorology, ~5 km) or ``'8day'``
+                (biophysical, ~500 m).
+            variables: Subset of the cadence's catalog; all of it if None.
+
+        Returns:
+            pandas.DataFrame: One row per timestep, a ``time`` column
+            (datetime64) plus one column per variable.
+        """
+        s.fill(lat, lon, start, end, cadence=cadence, variables=variables)
+        pid, variables = s._resolve(lat, lon, cadence, variables)
+        db = s._db()
+        try:
+            long = pd.read_sql_query(
+                'SELECT date, variable, value FROM observations '
+                'WHERE point=? AND cadence=? AND variable IN (%s) '
+                'AND date >= ? AND date <= ? ORDER BY date'
+                % ','.join('?' * len(variables)),
+                db, params=(pid, cadence, *variables, str(start), str(end)),
+            )
+        finally:
+            db.close()
+        df = long.pivot(index='date', columns='variable', values='value').reset_index()
+        df.columns.name = None
+        df = df.rename(columns={'date': 'time'})
+        df['time'] = pd.to_datetime(df['time'])
+        return df
+
+    # -- Troi adapters (the reproducibility layer speaks Troi) ----------
+
+    def fill_troi(s, troi, cadence: str = 'daily', variables: list[str] = None) -> int:
+        """:meth:`fill` at the centre of a :class:`troi.Troi`."""
+        return s.fill(troi.centre_lat, troi.centre_lon, troi.start, troi.end,
+                      cadence=cadence, variables=variables)
+
+    def get_df_troi(s, troi, cadence: str = 'daily', variables: list[str] = None) -> pd.DataFrame:
+        """:meth:`get_df` at the centre of a :class:`troi.Troi`."""
+        return s.get_df(troi.centre_lat, troi.centre_lon, troi.start, troi.end,
+                        cadence=cadence, variables=variables)
+
+
+# -- offline tests (synthetic rows, no network) -----------------------------
+
+def _tmp_store() -> Store:
+    import tempfile
+    tmpdir = tempfile.mkdtemp(prefix='pyozwald_store_test_')
+    return Store(config=Config(out_dir=tmpdir, tmp_dir=tmpdir))
+
+
+def _prime(store: Store, pid: str, cadence: str, variable: str, year: int,
+           through: date, value: float = 1.0):
+    """Insert a synthetic (variable, year) cell directly, bypassing the network."""
+    db = store._db()
+    days = pd.date_range(date(year, 1, 1), through, freq='D' if cadence == 'daily' else '8D')
+    with db:
+        db.executemany(
+            'INSERT OR REPLACE INTO observations (point, cadence, variable, date, value) '
+            'VALUES (?, ?, ?, ?, ?)',
+            [(pid, cadence, variable, str(d.date()), value) for d in days],
+        )
+        db.execute(
+            'INSERT OR REPLACE INTO coverage (point, cadence, variable, year, through) '
+            'VALUES (?, ?, ?, ?, ?)',
+            (pid, cadence, variable, year, str(through)),
+        )
+    db.close()
+
+
+def test_year_covered_semantics():
+    complete = year_covered('2023-12-31', 2023, date(2024, 6, 30))
+    lagging = year_covered('2024-05-01', 2024, date(2024, 6, 30))
+    enough = year_covered('2024-05-01', 2024, date(2024, 4, 1))
+    never = year_covered(None, 2023, date(2023, 6, 30))
+    return complete and not lagging and enough and not never
+
+
+def test_fill_skips_covered_cells():
+    store = _tmp_store()
+    lat, lon = -33.516, 148.373
+    pid = grid.point_id(lat, lon, 0.05)
+    _prime(store, pid, 'daily', 'Tmax', 2023, date(2023, 12, 31))
+    return store.fill(lat, lon, date(2023, 3, 1), date(2023, 9, 30),
+                      variables=['Tmax']) == 0
+
+
+def test_read_pivots_wide():
+    store = _tmp_store()
+    lat, lon = -33.516, 148.373
+    pid = grid.point_id(lat, lon, 0.05)
+    _prime(store, pid, 'daily', 'Tmax', 2023, date(2023, 12, 31), value=31.5)
+    _prime(store, pid, 'daily', 'Tmin', 2023, date(2023, 12, 31), value=12.5)
+    df = store.get_df(lat, lon, date(2023, 1, 1), date(2023, 12, 31),
+                      variables=['Tmax', 'Tmin'])
+    return (
+        len(df) == 365
+        and float(df['Tmax'].iloc[0]) == 31.5
+        and float(df['Tmin'].iloc[0]) == 12.5
+        and str(df['time'].dtype).startswith('datetime64')
+    )
+
+
+def test_nearby_coordinate_shares_daily_cell():
+    store = _tmp_store()
+    pid = grid.point_id(-33.514, 148.371, 0.05)
+    _prime(store, pid, 'daily', 'Pg', 2023, date(2023, 12, 31))
+    return store.fill(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31),
+                      variables=['Pg']) == 0
+
+
+def test_cadences_are_independent():
+    """A covered daily cell must not satisfy an 8day request."""
+    store = _tmp_store()
+    lat, lon = -33.516, 148.373
+    _prime(store, grid.point_id(lat, lon, 0.05), 'daily', 'Tmax', 2023, date(2023, 12, 31))
+    db = store._db()
+    row = db.execute(
+        'SELECT through FROM coverage WHERE point=? AND cadence=? AND variable=? AND year=?',
+        (grid.point_id(lat, lon, 0.005), '8day', 'NDVI', 2023),
+    ).fetchone()
+    db.close()
+    return row is None
+
+
+def test_unknown_variable_raises():
+    store = _tmp_store()
+    try:
+        store.fill(-33.5, 148.4, date(2023, 1, 1), date(2023, 2, 1), variables=['NDVI'])
+    except ValueError:
+        return True  # NDVI is 8day, not daily
+    return False
+
+
+def test():
+    return all([
+        test_year_covered_semantics(),
+        test_fill_skips_covered_cells(),
+        test_read_pivots_wide(),
+        test_nearby_coordinate_shares_daily_cell(),
+        test_cadences_are_independent(),
+        test_unknown_variable_raises(),
+    ])
+
+
+if __name__ == '__main__':
+    print(test())

pyozwald-0.1.0/pyozwald.egg-info/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.4
+Name: pyozwald
+Version: 0.1.0
+Summary: Cached OzWALD daily meteorology and 8-day biophysical series for Australia — fetch once per grid point, never twice
+Author: Borevitz Lab, Australian National University
+Author-email: Yasar Adeel Ansari <u6737670@anu.edu.au>
+License: MIT
+Project-URL: Homepage, https://github.com/thestochasticman/pyozwald
+Project-URL: Repository, https://github.com/thestochasticman/pyozwald
+Project-URL: Issues, https://github.com/thestochasticman/pyozwald/issues
+Keywords: ozwald,climate,australia,remote-sensing,agriculture,time-series
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: attrs
+Requires-Dist: typing_extensions
+Requires-Dist: pandas
+Requires-Dist: xarray
+Requires-Dist: netcdf4
+Dynamic: license-file
+
+# pyozwald
+
+**Cached [OzWALD](https://www.wenfo.org/ozwald/) time series for
+Australia — fetch once per grid point, never twice.** OzWALD is ANU's
+Water and Landscape Dynamics dataset: modelled daily meteorology
+(~5 km) and 8-day biophysical variables (~500 m, MODIS-derived) served
+as one OPeNDAP NetCDF per variable per year. Every observation this
+machine ever samples lands in one SQLite store, so repeat requests,
+nearby coordinates in the same cell, and extended date ranges all
+reuse the same rows. Part of the
+[Borevitz Lab](https://borevitzlab.anu.edu.au/) ecosystem.
+
+## How it works
+
+```
+{data_root}/ozwald_store/
+└── ozwald.db
+    ├── observations(point, cadence, variable, date, value)
+    └── coverage(point, cadence, variable, year, through)
+```
+
+- Coordinates snap to a dedup grid matching each product's native
+  resolution — 0.05° for daily meteorology, 0.005° for the 8-day
+  variables — so nearby requests share one stored series per cadence.
+- OzWALD's unit of delivery is one NetCDF per (variable, year), so
+  that's the unit of the coverage ledger. `Store.get_df(...)` diffs
+  the requested years × variables against it and samples **only the
+  missing cells**; a whole year is stored even when a sub-range was
+  requested, since the marginal cost is nil and it maximises reuse.
+- `through` records the last date a year's file actually contained —
+  an in-progress year keeps being re-fetched until complete, then
+  never again.
+- Writes are transactional (SQLite/WAL): a crash mid-fetch leaves the
+  cell unrecorded, and the next run re-fetches it.
+
+## Usage
+
+The core API is **troi-agnostic** — a coordinate, dates, and a cadence:
+
+```python
+from datetime import date
+from pyozwald.store import Store
+
+store = Store()
+
+met = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))
+#     daily meteorology: time, Pg, Tmax, Tmin, Uavg, Ueff, VPeff, ...
+
+veg = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31),
+                   cadence='8day', variables=['NDVI', 'LAI', 'GPP'])
+#     8-day biophysical series on the ~500 m grid
+
+store.fill(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))  # → 0: already local
+```
+
+Pipelines that speak the shared `troi.troi.Troi` use the
+adapters (evaluated at the bbox centre):
+
+```python
+df = store.get_df_troi(troi, cadence='daily')
+```
+
+`download_ozwald_daily(troi)` and `download_ozwald_8day(troi)`
+remain as thin wrappers.
+
+## Performance
+
+Live measurements against NCI THREDDS — one grid point:
+
+| Scenario | Fetched | Time |
+|---|---|---|
+| Cold fill — 2 daily variables × 1 year | 2 cells | 2.3 s |
+| Same request again | nothing | **0.0 s** |
+| Nearby coordinate, same ~5 km cell | nothing | **0.0 s** |
+| Date range extended −1 year | 2 cells — *the new year only* | 1.8 s |
+| 8-day NDVI, one year | 1 cell | 0.7 s |
+| Read cached year (365 × 2) | — | 0.01 s |
+
+(One *cell* = one variable × one year at one grid point.) Store
+footprint: ~100 KB for the five cells above. Absolute times vary with
+network and THREDDS load; the zeros are the point — they are ledger
+lookups, no network involved.
+
+## Install
+
+### pip
+
+```bash
+pip install git+https://github.com/thestochasticman/pyozwald.git
+```
+
+Dependencies (the `troi` core included, pulled from GitHub) are
+declared in `pyproject.toml` and installed automatically.
+
+### From source
+
+```bash
+git clone https://github.com/thestochasticman/pyozwald.git
+cd pyozwald
+pip install -e .
+```
+
+Package design (shared across the lab's packages — no inheritance,
+composition only):
+
+- **`Troi`** (from `troi`) — identity: what region, what dates.
+- **`OzWALD`** (`pyozwald.ozwald`) — config: endpoint, variable
+  catalogs per cadence, dedup grid steps.
+- **`Paths`** (`pyozwald.paths`) — derived location of the store for a
+  given `Config`.
+- **`grid`** — the dedup grids (pure, offline-testable math).
+- **`Store`** (`pyozwald.store`) — ties them together.
+
+## Test
+
+```bash
+# offline (pure math + synthetic store):
+python pyozwald/grid.py     # True
+python pyozwald/paths.py    # True
+python pyozwald/store.py    # True
+
+# live (small real samples from NCI THREDDS, incl. dedup assertions):
+python pyozwald/download_ozwald.py  # True
+```

pyozwald-0.1.0/pyozwald.egg-info/SOURCES.txt

@@ -0,0 +1,14 @@
+LICENSE
+README.md
+pyproject.toml
+pyozwald/__init__.py
+pyozwald/download_ozwald.py
+pyozwald/grid.py
+pyozwald/ozwald.py
+pyozwald/paths.py
+pyozwald/store.py
+pyozwald.egg-info/PKG-INFO
+pyozwald.egg-info/SOURCES.txt
+pyozwald.egg-info/dependency_links.txt
+pyozwald.egg-info/requires.txt
+pyozwald.egg-info/top_level.txt

pyozwald-0.1.0/pyozwald.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

pyozwald-0.1.0/pyozwald.egg-info/requires.txt

@@ -0,0 +1,5 @@
+attrs
+typing_extensions
+pandas
+xarray
+netcdf4

pyozwald-0.1.0/pyozwald.egg-info/top_level.txt

@@ -0,0 +1 @@
+pyozwald

pyozwald-0.1.0/pyproject.toml

@@ -0,0 +1,51 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "pyozwald"
+version = "0.1.0"
+description = "Cached OzWALD daily meteorology and 8-day biophysical series for Australia — fetch once per grid point, never twice"
+readme = "README.md"
+authors = [
+  { name = "Borevitz Lab, Australian National University" },
+  { name = "Yasar Adeel Ansari", email = "u6737670@anu.edu.au" },
+]
+license = { text = "MIT" }
+requires-python = ">=3.11"
+keywords = [
+  "ozwald",
+  "climate",
+  "australia",
+  "remote-sensing",
+  "agriculture",
+  "time-series",
+]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Scientific/Engineering :: Atmospheric Science",
+]
+# The scientific stack (pandas, xarray, netcdf4) is provided by conda via
+# environment.yml; only the lab core is declared here so `pip install -e .`
+# is fast.
+dependencies = [
+  "attrs",
+  "typing_extensions",
+  "pandas",
+  "xarray",
+  "netcdf4",
+]
+
+[project.urls]
+Homepage = "https://github.com/thestochasticman/pyozwald"
+Repository = "https://github.com/thestochasticman/pyozwald"
+Issues = "https://github.com/thestochasticman/pyozwald/issues"
+
+[tool.setuptools.packages.find]
+include = ["pyozwald", "pyozwald.*"]

pyozwald-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+