forcingkit 0.9.0__tar.gz → 0.11.0__tar.gz

{forcingkit-0.9.0 → forcingkit-0.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: forcingkit
-Version: 0.9.0
+Version: 0.11.0
 Summary: Spatiotemporal forcing for computational Earth-system models: selects, regrids and serves model-ready time series with provenance.
 Project-URL: Documentation, https://forcingkit.docs.lhzn.io
 Project-URL: Issues, https://github.com/lhzn-io/forcingkit/issues

{forcingkit-0.9.0 → forcingkit-0.11.0}/docs/source/conf.py

@@ -9,7 +9,7 @@ sys.path.insert(0, os.path.abspath("../../src"))
 project = "forcingkit"
 copyright = "2026, Long Horizon Observatory"
 author = "Daniel Fry"
-release = "0.9.0"
+release = "0.11.0"
 
 extensions = [
     "sphinx.ext.autodoc",

{forcingkit-0.9.0 → forcingkit-0.11.0}/docs/source/fetchers.rst

@@ -25,6 +25,15 @@ Atmospheric forcing (``/api/v1/atmosphere``)
        <https://cds.climate.copernicus.eu/>`__ through `NumericalEarth
        <https://github.com/NumericalEarth/NumericalEarth.jl>`__ in the model, outside this
        service. See :doc:`atmospheric_forcing`.
+   * - **HRRR levels**
+     - ``/api/v1/atmosphere/levels`` reads the lowest hybrid levels of the same cycles' native-level
+       file (``wrfnatf01``) by byte range: U and V (rotated to east and north), vertical velocity
+       (converted from omega with the hydrostatic relation) and temperature every hour, and the
+       levels' heights and pressures once, from the first hour, since they move only a few metres
+       with the surface pressure. Over open water level 1 is about 10 m above the surface (its
+       wind agrees with the surface delivery's 10 m wind), level 6 about 430 m and level 14 about
+       2.8 km. Each CONUS message is fetched whole (1 to 2.4 MB), so 14 levels read about 63 MB an
+       hour: a 48 h window took 3 minutes and made a 7 MB store (schema ``hrrr-levels-v1``).
 
 Parent ocean (``/api/v1/obc``)
 ------------------------------

{forcingkit-0.9.0 → forcingkit-0.11.0}/docs/source/index.rst

@@ -63,6 +63,11 @@ What it serves
      - HRRR surface fields (10 m wind, 2 m temperature and humidity, pressure, radiation,
        precipitation) on a regular 0.03 degree grid, hourly, as schema ``hrrr-atm-v1``. See
        :doc:`atmospheric_forcing`.
+   * - ``POST /api/v1/atmosphere/levels``
+     - The air above: HRRR wind, vertical velocity and temperature on its lowest native levels
+       (14 by default, about 10 m to 2.8 km above open water), over the same window and grid as
+       ``/api/v1/atmosphere``, as schema ``hrrr-levels-v1``. For vertical sections and air-parcel
+       tracks in a viewer; not a forcing. See :doc:`fetchers`.
    * - ``POST /api/v1/rivers``
      - Freshwater discharge at each listed river mouth in the box, hourly, from the tide-free
        `USGS <https://waterdata.usgs.gov/>`__ stream gauges upstream (chosen per request) scaled
@@ -167,5 +172,6 @@ PyPI as ``forcingkit``.
    :caption: Reference
 
    mesh_frames
+   layered_mesh
    sources
    roadmap

forcingkit-0.11.0/docs/source/layered_mesh.rst

@@ -0,0 +1,110 @@
+Layered mesh: 3D model output on its own mesh
+==============================================
+
+``forcingkit.layered_mesh`` holds every layer of a 3D model on the model's own horizontal mesh,
+with no regridding: the triangles, a vertical coordinate, and any number of named fields. It is
+the 3D companion of :doc:`mesh_frames` (``MeshFrames`` stays the single-layer surface-current
+contract; ``LayeredMesh.surface()`` returns one). A viewer can draw a model's own cells from it,
+cut sections through the layers, and colour several models on one shared scale.
+
+Data contract
+-------------
+
+``LayeredMesh`` holds:
+
+- ``times`` (UTC; naive input is taken as UTC);
+- ``node_lon``, ``node_lat`` (degrees) and ``tri`` (``n_elem`` x 3, zero-based node indices);
+- one vertical coordinate:
+
+  - sigma (terrain-following): ``h`` (``n_node``, depth below the datum, positive down),
+    ``siglay`` (``[n_layer, n_node]``, sigma at layer centres, surface first: 0 at the surface,
+    -1 at the bed), optional ``siglev`` (``[n_layer + 1, n_node]``, at the interfaces) and
+    ``zeta`` (``[t, n_node]``, surface elevation). A layer centre is at height
+    ``siglay * (h + zeta) + zeta``;
+  - fixed heights: ``layer_z`` (``[n_layer, n_elem]``, height of each layer centre in metres,
+    positive up) and an optional ``bed_z`` (``n_elem``);
+  - neither, for a single layer at the surface;
+
+- ``fields``: ``name -> [t, n_layer, n_elem]`` or ``[t, n_layer, n_node]``, NaN where a cell has
+  no value (dry, below the bed, masked);
+- ``meta``: a ``FieldMeta`` per field, whose ``kind`` (``"element"`` or ``"node"``) says which;
+- ``source_index`` (optional, ``n_elem``): each element's index in the source's own numbering
+  (an FVCOM element id, or a flat grid-cell index).
+
+``element_values(name)`` gives any field per element: as held for an element field, and as the
+mean of the element's three nodes for a node field, which is the flat value a viewer draws for
+that cell. ``speed`` is derived from ``u`` and ``v`` when the mesh holds them.
+
+Field metadata
+--------------
+
+``FieldMeta`` carries ``name, label, units, lo, hi, log, ticks``, the entries a viewer manifest
+lists for each field (``manifest_entry()`` returns exactly those), plus ``kind`` and
+``symmetric`` (a signed field whose scale is centred on zero). ``encode(values)`` maps values to
+one byte on the scale (1 at or below ``lo``, 255 at or above ``hi``, linear or logarithmic, 0 for
+NaN) and ``decode(codes)`` maps them back, so per-field u8 codes and their legend come from one
+place. Without explicit ``ticks``, legends use round values (1, 2 or 5 times a power of ten).
+
+Sections
+--------
+
+``section(points, n=200, fields=None, node_values="element")`` samples the mesh at ``n`` points
+spaced evenly by distance along a polyline ``[(lon, lat), ...]`` and returns a ``Section`` with
+``distance`` (m), ``lon``, ``lat``, the host ``element`` (-1 outside the mesh), ``z`` (layer
+centres, ``[t, n_layer, n]``), ``z_interfaces`` (sigma meshes with ``siglev``), ``bed_z`` and
+``values[name]`` (``[t, n_layer, n]``). Element fields take the host element's value; node fields
+take the element mean by default or, with ``node_values="linear"``, are interpolated within the
+element. Heights on a sigma mesh are interpolated within the element and include ``zeta``. Points
+outside the mesh are NaN. Point location uses matplotlib's trapezoid map, and falls back to a
+search over every element for a triangulation it cannot index.
+
+Shared colour scales
+--------------------
+
+``shared_scales(meshes, names=None, window=None)`` sets ``lo`` and ``hi`` for each field from
+every element value of every mesh that holds it, at the times inside ``window``, weighted by
+element area: the 1st to 99th percentiles, or for a symmetric field ``-hi`` to ``hi`` with ``hi``
+the 99.5th percentile of the magnitude. Weighting by area matters on unstructured meshes, whose
+rivers and harbours hold many small elements; counted per element, those elements set the scale
+to their own extremes (on NECOFS around Long Island Sound on 2 April 2026, temperature reached
+19 degC and salinity 5 PSU that way, against 1.7 to 8.0 degC and 28.5 to 34.2 PSU by area).
+``shared_scale`` does one field and accepts other percentiles.
+
+Particle tracks
+---------------
+
+``Tracks`` holds particle positions ``xyz`` (``[t, n_particle, 3]``: longitude, latitude, height
+in metres) and an ``alive`` mask (``[t, n_particle]``; a position that is not finite is never
+alive). ``local_xyz(lon0, lat0)`` gives metres about an origin, and ``quantize(lon0, lat0)``
+stores them as int16 per axis with an offset and scale (error at most half a step, about 0.3 m
+across 20 km; samples that are not alive get code 0). Tracking itself is the caller's: any
+tracker that reports positions fits.
+
+Adapters
+--------
+
+``necofs(bbox, start, end)`` reads NECOFS FVCOM GOM7 through
+``forcingkit.fetchers.necofs.volume`` (see :doc:`necofs`) for the elements whose centres lie in
+the box, hourly: 45 sigma layers with ``zeta``, element ``u``, ``v`` and ``w`` (FVCOM ``ww``,
+m/s, up positive) and node ``T`` (degC) and ``S`` (PSU). ``from_necofs_volume(ds)`` converts a
+dataset already read.
+
+``structured(lon, lat, mask, z, times=..., fields=...)`` turns a regular grid (1-D axes) or a
+curvilinear one (2-D centres) into two triangles per cell, both carrying the cell's own value, so
+cells are drawn flat, as the model holds them. Cell corners lie midway between centres (for a
+curvilinear grid, the mean of the four centres around each corner), and the outer corners half a
+cell beyond the outermost centres. ``mask`` keeps water cells; ``z`` is None for one surface
+layer, fixed heights ``[n_layer]`` (z-level models), or ``[n_layer, ny, nx]``; ``fields`` are
+``[t, n_layer, ny, nx]`` or ``[t, ny, nx]``. This is the basis for the NYOFS and DBOFS
+(curvilinear), HYCOM (regular, z levels) and HRRR (regular, surface only for now) adapters.
+
+Agreement with an independent pipeline
+--------------------------------------
+
+Built from the same NECOFS volumes (2 April 2026, 02:00 to 21:00 UTC, four boxes around Long
+Island Sound), sections along four lines reproduce byte for byte the per-field codes of a
+separate pipeline written for the same data (speed, temperature, salinity and vertical velocity,
+45 layers, every hourly frame: about 11.4 million codes), and its layer interfaces and bed to
+within 0.1 mm. Its area-weighted shared scales agree to 0.02 percent (temperature 1.7408 to
+7.9941 degC against 1.7406 to 7.9932): that pipeline measured element areas in the model's
+Cartesian projection, and this module about each element's own latitude.

{forcingkit-0.9.0 → forcingkit-0.11.0}/docs/source/mesh_frames.rst

@@ -45,12 +45,18 @@ Colour
   last row take the last colour; NaN is transparent. ``load_lut_csv(path)`` reads such a table
   from a ``value,r,g,b[,a]`` CSV;
 - ``packaged_color("currents_lis")``: the surface-current table shipped with the package
-  (``forcingkit/ramps/currents_lis.csv``), 0.02 to 3.1 m/s on a log axis, pale blue through
-  yellow to dark green with CIELAB lightness falling monotonically. Its CSV header records the
+  (``forcingkit/ramps/currents_lis.csv``), 0.02 to 3.1 m/s on a log axis, pale aqua through
+  teal to deep blue-green with CIELAB lightness falling monotonically. Its CSV header records the
   knots and their source;
 - ``packaged_color("currents_lis_dark")``: its twin for a dark basemap, on the same speeds, from
-  midnight blue through steel blue to pale seafoam with lightness rising monotonically, so the
-  two swap by theme alone.
+  deep blue-teal through teal to pale seafoam with lightness rising monotonically, so the two
+  swap by theme alone;
+- ``packaged_color("currents_lis_deep")`` and ``packaged_color("currents_lis_haline")``: multi-hue
+  alternatives on the same speeds, after cmocean "deep" (light backgrounds: pale green through
+  teal and navy to deep purple) and "haline" (dark backgrounds: indigo through blue and green to
+  pale yellow). They separate speeds more strongly than the single-hue pair, but they are not
+  mirror images: purple is fast in one and slow in the other. Use them for a field shown alone
+  with its legend, not beside other colour-mapped layers.
 
 Python
 ------
@@ -83,7 +89,7 @@ Command line
      --size 1200x800 --ramp viridis --vmax 1.5 \
      --edges 0,0,0,40 --background 242,243,240 --out frames/
 
-``--ramp`` takes a built-in name, a packaged table (``currents_lis``, ``currents_lis_dark``) or a path to a
+``--ramp`` takes a built-in name, a packaged table (``currents_lis``, ``currents_lis_dark``, ``currents_lis_deep``, ``currents_lis_haline``) or a path to a
 ``value,r,g,b[,a]`` CSV (with ``--log`` and ``--floor``). The output directory receives
 ``frame_0000.png``, ``frame_0001.png``, ... and ``frames.json``, which lists the frame times, the box
 and the size.

{forcingkit-0.9.0 → forcingkit-0.11.0}/docs/source/necofs.rst

@@ -247,6 +247,84 @@ From Python, either stream records yourself or assemble them in memory:
 
    ds = fetch_necofs_boundary_conditions("2026-09-26T00:00:00Z", 24, bbox)
 
+Full-depth subset on the native mesh
+------------------------------------
+
+For work that needs FVCOM's own geometry rather than a regular grid (3D rendering of the sigma
+prisms, particle tracking with FVCOM's own interpolation, comparisons element by element),
+``volume(bbox, start, end)`` returns every sigma layer for the elements whose centroids lie in
+the box, with no regridding and no vertical conversion. ``read_volume(bbox, start_time, hours,
+dap_url=None)`` does the same for one file.
+
+.. code-block:: python
+
+   from forcingkit.fetchers.necofs import volume
+
+   bbox = [-72.22, 41.10, -71.75, 41.395]  # Fishers Island and The Race, with a margin
+   vol = volume(bbox, "2026-04-02T00Z", "2026-04-04T00Z")  # 49 hourly records
+
+.. list-table::
+   :header-rows: 1
+   :widths: 22 26 52
+
+   * - Name
+     - Dimensions
+     - Meaning
+   * - ``u``, ``v``, ``ww``
+     - (time, siglay, element)
+     - Velocities at element layer centres, m s-1. ``ww`` is the vertical velocity in z, which
+       FVCOM derives from the sigma-coordinate velocity omega; omega itself is not archived.
+   * - ``temp``, ``salinity``
+     - (time, siglay, node)
+     - At node layer centres.
+   * - ``zeta``; ``h``
+     - (time, node); (node)
+     - Surface elevation; depth below the geoid, positive down.
+   * - ``siglay``, ``siglev``
+     - (siglay, node), (siglev, node)
+     - Sigma at layer centres and interfaces, surface first. The depth of a layer centre is
+       :math:`\sigma (h + \zeta) + \zeta`.
+   * - ``x``, ``y``; ``xc``, ``yc``
+     - (node); (element)
+     - The model's own Cartesian coordinates in metres, NAD83 / Maine West (EPSG:26984).
+   * - ``node_lon``, ``node_lat``; ``lonc``, ``latc``
+     - (node); (element)
+     - Degrees.
+   * - ``triangles``
+     - (element, vertex)
+     - Zero-based indices into the returned nodes.
+   * - ``neighbours``
+     - (element, vertex)
+     - The element across the edge opposite each vertex (FVCOM's ``NBE`` order), as an index into
+       the returned elements; ``NO_NEIGHBOUR`` (-1) on the mesh boundary, ``OUTSIDE_SUBSET`` (-2)
+       where the neighbour exists but was not selected.
+   * - ``on_boundary``
+     - (node)
+     - The node lies on the mesh boundary (coast or open boundary).
+   * - ``element_id``, ``node_id``
+     - (element); (node)
+     - Indices in the full mesh.
+
+The archive does not carry FVCOM's edge-neighbour table, so ``neighbours`` and ``on_boundary`` are
+rebuilt from ``nv`` over the **whole** mesh (``forcingkit.mesh_frames.element_neighbours``): on
+the subset alone, every element along the subset's edge would look like a coastal element.
+
+Every requested hour must be found, and every archive file must select the same elements;
+otherwise ``volume`` raises. Results are cached as Zarr under
+``$FORCINGKIT_CACHE_DIR/necofs/volume/``, keyed by the schema (``volume-v1``), the box and the
+window. Hyperslabs are fetched with ``FORCINGKIT_MAX_WORKERS`` threads. Even so, the 48 hours
+above (2,831 elements, 1,829 nodes, 45 layers, about 110 MB) took about five minutes from SMAST
+on 2026-10-22, almost all of it server round trips, and 0.1 s from the cache.
+
+Wetting and drying
+~~~~~~~~~~~~~~~~~~
+
+GOM7 runs with FVCOM's wet/dry treatment, although no file attribute says so and the wet/dry
+masks are not archived. The data show it: the total depth :math:`h + \zeta` never falls below
+0.200 m, thousands of nodes sit exactly at that floor at low water (5,641 in the record of
+2026-04-02 12:00 UTC), and 99 % of the elements whose three nodes are at the floor carry exactly
+zero ``u``, ``v`` and ``ww``. Treat an element as dry when all three of its nodes are at the floor.
+
 Notes and limits
 ----------------
 
@@ -276,6 +354,13 @@ tag, and the passing of donor options through the dispatcher.
 
    uv run pytest tests/unit/test_necofs_parent.py -v
 
+``tests/unit/test_necofs_volume.py`` covers the full-depth subset on a synthetic FVCOM file:
+every value lands on its own element, node, layer and hour; neighbours are mutual, and the
+subset's edge is never mistaken for the coast; boundary nodes; windows assembled across
+archive files (the midnight record in its own day's file); a missing hour raising; and the
+cache. ``tests/integration/test_necofs_volume_live.py`` reads three hours across a file
+boundary from SMAST.
+
 References
 ----------
 

{forcingkit-0.9.0 → forcingkit-0.11.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "forcingkit"
-version = "0.9.0"
+version = "0.11.0"
 description = "Spatiotemporal forcing for computational Earth-system models: selects, regrids and serves model-ready time series with provenance."
 readme = "README.md"
 requires-python = ">=3.11"

{forcingkit-0.9.0 → forcingkit-0.11.0}/service/forcingkit_serve/main.py

@@ -595,6 +595,63 @@ def generate_atmosphere(request: AtmosphereRequest) -> Dict[str, Any]:
     }
 
 
+class AtmosphereLevelsRequest(BaseModel):
+    bbox: BoundingBox
+    start_time: str  # ISO 8601, UTC
+    hours: int
+    n_levels: int = 14
+    resolution_deg: float = 0.03
+    margin_deg: float = 0.25
+    cache_bust: bool = False
+
+
+@app.post("/api/v1/atmosphere/levels")
+def generate_atmosphere_levels(request: AtmosphereLevelsRequest) -> Dict[str, Any]:
+    """The lower atmosphere above the surface delivery: HRRR wind, vertical velocity and
+    temperature on its lowest `n_levels` native levels (14 reach about 2.9 km over water), hourly
+    over the same window and grid as /api/v1/atmosphere, streamed to a Zarr store. About 63 MB of
+    HRRR is read per hour at 14 levels."""
+    from forcingkit.dispatcher import dispatch_levels_request, levels_key
+    from forcingkit.fetchers.hrrr_levels import MAX_LEVELS
+
+    if request.hours < 1:
+        raise HTTPException(status_code=400, detail="hours must be at least 1")
+    if not 1 <= request.n_levels <= MAX_LEVELS:
+        raise HTTPException(
+            status_code=400, detail=f"n_levels must be 1 to {MAX_LEVELS}"
+        )
+    bbox_list = [
+        request.bbox.min_lon,
+        request.bbox.min_lat,
+        request.bbox.max_lon,
+        request.bbox.max_lat,
+    ]
+    zarr_id = levels_key(
+        bbox_list,
+        request.start_time,
+        request.hours,
+        request.n_levels,
+        request.resolution_deg,
+        request.margin_deg,
+    )
+    zarr_path = os.path.join(_atmosphere_cache_dir(), f"{zarr_id}.zarr")
+    try:
+        final_path = dispatch_levels_request(
+            bbox_list,
+            request.start_time,
+            request.hours,
+            zarr_path,
+            n_levels=request.n_levels,
+            resolution_deg=request.resolution_deg,
+            margin_deg=request.margin_deg,
+            cache_bust=request.cache_bust,
+        )
+    except Exception as e:
+        logger.error(f"Atmosphere levels generation failed: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    return {"status": "success", "zarr_id": zarr_id, "zarr_path": final_path}
+
+
 @app.get("/api/v1/atmosphere/download/{zarr_id}")
 def download_atmosphere(zarr_id: str):
     cache_dir = _atmosphere_cache_dir()

{forcingkit-0.9.0 → forcingkit-0.11.0}/src/forcingkit/dispatcher.py

@@ -424,6 +424,82 @@ def dispatch_atmosphere_request(
         raise
 
 
+def levels_key(
+    bbox: list[float],
+    start_time: str,
+    hours: int,
+    n_levels: int,
+    resolution_deg: float,
+    margin_deg: float,
+) -> str:
+    """Cache id of a lower-atmosphere delivery on native levels; the schema is part of it."""
+    import hashlib
+
+    from forcingkit.fetchers.hrrr_levels import HRRR_LEVELS_SCHEMA
+
+    key = (
+        f"lev_hrrr_{HRRR_LEVELS_SCHEMA}_{bbox[0]}_{bbox[1]}_{bbox[2]}_{bbox[3]}_"
+        f"{start_time}_{hours}_{n_levels}_{resolution_deg}_{margin_deg}"
+    )
+    return "lev_" + hashlib.sha256(key.encode()).hexdigest()[:12]
+
+
+def dispatch_levels_request(
+    bbox: list[float],
+    start_time: str,
+    hours: int,
+    zarr_path: str,
+    n_levels: int = 14,
+    resolution_deg: float = 0.03,
+    margin_deg: float = 0.25,
+    cache_bust: bool = False,
+) -> str:
+    """HRRR's lowest `n_levels` native levels for `hours` from `start_time`, streamed to
+    `zarr_path` (schema hrrr-levels-v1)."""
+    from forcingkit.fetchers import hrrr_levels
+    from forcingkit.zarr_stream import StreamingZarrWriter, store_is_complete
+
+    if not cache_bust and store_is_complete(
+        zarr_path, (hrrr_levels.HRRR_LEVELS_SCHEMA,)
+    ):
+        logger.info(f"Cache hit for atmosphere levels: {zarr_path}")
+        return zarr_path
+
+    writer = None
+    try:
+        for item in hrrr_levels.iter_levels(
+            start_time,
+            hours,
+            bbox,
+            n_levels=n_levels,
+            resolution_deg=resolution_deg,
+            margin_deg=margin_deg,
+        ):
+            if item[0] == "static":
+                writer = StreamingZarrWriter(
+                    zarr_path,
+                    item[1],
+                    hrrr_levels.RECORD_DIMS,
+                    expected_records=hours + 3,
+                    attrs=dict(item[1].attrs),
+                    record_attrs={
+                        k: {"units": u, "standard_name": sn}
+                        for k, (u, sn) in hrrr_levels.UNITS.items()
+                    },
+                )
+                continue
+            if writer is None:
+                raise RuntimeError("level records arrived before the static fields")
+            writer.append(item[1], item[2])
+        if writer is None:
+            raise RuntimeError("HRRR yielded no data")
+        return writer.close()
+    except Exception:
+        if writer is not None:
+            writer.abort()
+        raise
+
+
 PROVISIONAL_MAX_AGE_HOURS = 24