forcingkit 0.1.0__tar.gz

forcingkit-0.1.0/.agents/forcingkit.md

@@ -0,0 +1,98 @@
+# forcingkit Reference
+
+## Mission
+
+Python microservice providing real-time and historical forcing (the parent ocean and
+the HRRR atmosphere) and validation observations to the `coastal-sim` Julia physics
+engine.
+
+## Environment
+
+- **Python 3.11+** via `uv`. Always use `uv run` — never `pip install`.
+- **FastAPI** microservice in `service/forcingkit_serve/`.
+- Tests run with `uv run pytest`.
+
+## Architecture
+
+### Core Pipeline
+
+```text
+HTTP Request → service/forcingkit_serve/main.py
+             → src/forcingkit/dispatcher.py
+             → fetchers/*.py
+             → Zarr (~/.cache/forcingkit/)
+```
+
+- **Entry point**: `service/forcingkit_serve/main.py` — FastAPI application.
+- **Dispatcher**: `src/forcingkit/dispatcher.py` — interprets
+  requests and routes to the appropriate fetcher via tiered fallback.
+  Uses `_rank_obc_candidates(bbox)` to select the best parent by
+  resolution and domain overlap.
+- **Fetchers**: `src/forcingkit/fetchers/` — isolated modules per data source.
+
+## Fetcher Tier (Parent Ocean Priority Order)
+
+| Fetcher | Resolution | Domain | Notes |
+|---|---|---|---|
+| `nyofs.py` | ~100 m | NY/NJ Harbor | POM curvilinear grid, C-grid stagger |
+| `necofs.py` | ~200 m | New England | FVCOM unstructured |
+| `hycom.py` | ~9 km | Global | Final fallback (Operational + Historical) |
+
+*Note: Because the architecture is modular, a future roadmap item includes the integration of TPXO (via `tpxo.py`) given usage rights.*
+
+Atmospheric forcing: HRRR from 2014-07-30 (`hrrr_atmosphere.py`, `/api/v1/atmosphere`).
+Earlier runs use ERA5 through NumericalEarth in coastal-sim, not this service.
+
+## Testing
+
+```bash
+# Unit tests (no network I/O, fast)
+uv run pytest tests/unit/ -v
+
+# Single test file
+uv run pytest tests/unit/test_nyofs.py -v
+
+# Integration tests (live APIs, slower)
+uv run pytest tests/integration/ -v
+
+# Pre-commit checks (ruff + mypy) on modified files
+uv run pre-commit run --files <file1> <file2> ...
+```
+
+## Commands
+
+```bash
+# Sync dependencies
+uv sync
+
+# Start data service
+uv run python service/run_server.py
+
+# Run one-off fetch
+uv run python -c "from forcingkit.fetchers import nyofs; print(nyofs.get_metadata())"
+```
+
+## Gotchas
+
+- **Telemetry performance**: Use `.csvp` endpoints for ERDDAP
+  tabular time-series to avoid heavy NetCDF binary parsing overhead.
+- **OPeNDAP engine**: Always pass `engine="pydap"` to `xr.open_dataset` for THREDDS
+  endpoints. The default netcdf4 engine fails on remote DAP URLs.
+- **NYOFS FMRC vs NCEI tiering**: FMRC aggregation covers rolling 7-day window
+  (< 31 days). Older data requires per-hour NCEI file enumeration with naming
+  conventions that changed on 2024-09-09.
+- **C-grid interpolation**: NYOFS u/v are on Arakawa C-grid face
+  points. After `.where(mask, drop=True)`, the subset is already
+  zero-indexed — use the `_c_grid_to_rho()` helper that averages
+  adjacent cells on the subset array, not the old absolute-index
+  approach.
+- **Mocking strategy**: Dispatcher unit tests check tuple/dict return
+  boundaries carefully. When modifying mocked fetchers, track
+  keyword-argument vs positional argument boundaries.
+- **Grid normalization**: `coastal-sim` expects elevations positive
+  up (LMSL/NAVD88). Normalize any inverted datasets in the fetcher
+  tier before the dispatcher sees them.
+- **Cache keys**: Cache Zarr keys are deterministic hashes of bbox
+  - time window. Changing fetcher output schema (variable names, dims)
+  will miss existing cache entries — purge
+  `~/.cache/forcingkit/` when making breaking schema changes.

forcingkit-0.1.0/.claude/CLAUDE.md

@@ -0,0 +1,10 @@
+# Agent Instructions
+
+> **Redirect**: Please refer to [AGENTS.md](../AGENTS.md) for the
+> canonical source of truth.
+
+## Expected Follow-on Context
+
+- `/.agents/forcingkit.md`
+
+Use this file as a compatibility entry point only.

forcingkit-0.1.0/.env.template

@@ -0,0 +1,17 @@
+# forcingkit environment variables
+# Copy this file to .env and fill in your credentials.
+# This file is loaded by coastal-sim/docker-compose.yml via env_file.
+
+# No credentials are required at present: HRRR, NOAA OFS, NECOFS, HYCOM, NDBC and
+# CO-OPS are all served without authentication.
+
+# Cache root for every store (default ~/.cache/forcingkit). The old names
+# ECODATA_CACHE_CACHE_DIR and COASTAL_SIM_DATA_CACHE_DIR are still read, with a
+# warning, until the next release.
+# FORCINGKIT_CACHE_DIR=~/.cache/forcingkit
+
+# Threads for parallel OPeNDAP reads (default 4; was ECODATA_CACHE_MAX_WORKERS).
+# FORCINGKIT_MAX_WORKERS=4
+
+# The topobathykit elevation service (default http://localhost:9595; was TOPOBATHYSIM_URL).
+# TOPOBATHYKIT_URL=http://localhost:9595

forcingkit-0.1.0/.github/copilot-instructions.md

@@ -0,0 +1,16 @@
+# forcingkit — Copilot Compatibility
+
+This file exists for tools that specifically load `.github/copilot-instructions.md`.
+
+## Canonical Guidance
+
+- Treat `/AGENTS.md` as the primary instruction entry point.
+- Treat `/.agents/` as the vendor-neutral location for deeper project guidance.
+
+## Copilot-Specific Compatibility Notes
+
+- Always use `uv run` for Python commands and `uv run pytest` for tests.
+- Run `uv run pre-commit run --files <modified-files>` before staging any changes.
+- Never commit without explicit user approval: present the change set and commit
+  message first, then wait for positive confirmation.
+- Stage files explicitly (`git add <file>`). Never `git add .`.

forcingkit-0.1.0/.github/workflows/docs.yml

@@ -0,0 +1,49 @@
+name: Deploy Docs
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "docs/**"
+      - "src/forcingkit/**"
+      - ".github/workflows/docs.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install Sphinx dependencies
+        run: pip install -r docs/requirements-docs.txt
+
+      - name: Build HTML docs
+        run: sphinx-build -W --keep-going -b html docs/source docs/build/html
+
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: docs/build/html
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@v4

forcingkit-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,51 @@
+name: Publish to PyPI
+
+# Pushing a tag vX.Y.Z builds the sdist and wheel and publishes them to PyPI through trusted
+# publishing (OIDC): no API token is stored anywhere. PyPI must list this repository, this
+# workflow file and the `pypi` environment as a trusted publisher for forcingkit.
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v6
+
+      - name: Check the tag matches the package version
+        run: |
+          version=$(python3 -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')
+          if [ "v$version" != "$GITHUB_REF_NAME" ]; then
+            echo "Tag $GITHUB_REF_NAME does not match pyproject.toml version $version" >&2
+            exit 1
+          fi
+
+      - name: Build
+        run: uv build
+
+      - name: Check the distributions
+        run: uvx twine check --strict dist/*
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/forcingkit
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+      - uses: pypa/gh-action-pypi-publish@release/v1

forcingkit-0.1.0/.github/workflows/tests.yml

@@ -0,0 +1,35 @@
+name: Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  unit:
+    name: Unit tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Sync dependencies
+        run: uv sync --dev
+
+      # Gating. Runs the pinned hooks from .pre-commit-config.yaml rather than
+      # whatever ruff the venv resolves, so CI and a local commit apply the same
+      # version and the same rule selection. Grading against ambient-latest is
+      # what made the first run of this workflow fail on untouched code.
+      - name: Lint
+        run: uv run pre-commit run --all-files --show-diff-on-failure
+
+      # Integration tests are excluded: they call live NOAA, NECOFS and HYCOM
+      # endpoints and take several minutes.
+      # Run them locally or via workflow_dispatch.
+      - name: Unit tests
+        run: uv run python -m pytest tests/unit -m "not integration" --durations=10

forcingkit-0.1.0/.gitignore

@@ -0,0 +1,19 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+.env
+
+# Testing and Logs
+*.log
+.pytest_cache/
+.coverage
+htmlcov/
+.vscode/
+.mypy_cache/

forcingkit-0.1.0/.markdownlint.yaml

@@ -0,0 +1,5 @@
+default: true
+MD013: false  # line-length: Disable (too strict for documentation/URLs)
+MD033: false  # no-inline-html: Disable (needed for centering/resizing images)
+MD003:        # heading-style
+  style: atx  # Enforce hash-style headers

forcingkit-0.1.0/.pre-commit-config.yaml

@@ -0,0 +1,48 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+        args: ["--maxkb=500000"]
+      - id: check-toml
+      - id: detect-private-key
+
+  # Pinned deliberately. This rev is the authoritative linter version for the
+  # repo; CI runs these same hooks so there is one standard everywhere. Bump it
+  # as its own reviewed change, never implicitly. The rule selection is pinned
+  # alongside it in pyproject.toml [tool.ruff.lint], because ruff's built-in
+  # defaults are not stable across versions.
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.16.9
+    hooks:
+      - id: ruff-check
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: local
+    hooks:
+      - id: mypy
+        name: mypy
+        entry: mypy
+        language: system
+        types: [python]
+        require_serial: true
+        args: ["--ignore-missing-imports", "--explicit-package-bases"]
+
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.39.0
+    hooks:
+      - id: markdownlint
+        language_version: 20.10.0
+        args: ["--fix"]
+
+  - repo: local
+    hooks:
+      - id: block-cds-secrets
+        name: Block Copernicus Secrets
+        entry: 'url: https://cds.climate.copernicus.eu'
+        language: pygrep
+        files: \.jl$|\.py$|\.env$|\.cdsapirc$

forcingkit-0.1.0/.python-version

@@ -0,0 +1 @@
+3.13

forcingkit-0.1.0/AGENTS.md

@@ -0,0 +1,71 @@
+# forcingkit Agent Guide
+
+This file is the canonical, vendor-neutral entry point for agent
+guidance in this repository.
+
+## Scope
+
+- Prefer standards and shared conventions over vendor-specific agent layouts.
+- Treat `AGENTS.md` and `.agents/` as the primary instruction surface.
+- Keep agent-specific files as thin compatibility shims when needed by a tool.
+
+## Tool Discovery Map
+
+- Canonical source of truth: `AGENTS.md` and `/.agents/`.
+- Claude Code:
+  - Compatibility shim: `/.claude/CLAUDE.md`.
+- VS Code GitHub Copilot Agent:
+  - Compatibility shim: `/.github/copilot-instructions.md`.
+- Rule for all tool-specific files: redirect to this file and
+  `/.agents/`, and only add minimal tool-local behavior.
+
+## Repository Focus
+
+forcingkit is a Python microservice that fetches, harmonizes, regrids, and serves
+real-time and historical forcing to the `coastal-sim` Julia physics engine: the parent
+ocean (`/api/v1/obc`, schema z-v2), the HRRR atmosphere (`/api/v1/atmosphere`), tides,
+and station telemetry and NDBC observations for validation.
+
+## Core Rules
+
+- **Dependency management**: Always use `uv run` - never `pip install` directly.
+- **Linting/formatting**: Run `uv run pre-commit run --files <files>` (ruff + mypy)
+  before staging any changes.
+- **Testing**: Run tests with `uv run python -m pytest tests/unit/` (unit) or
+  `uv run python -m pytest tests/integration/` (live API, roughly 10 minutes).
+  Use `python -m pytest` rather than the bare `pytest` entry point: the latter can
+  resolve to a system interpreter outside the venv. `tests/conftest.py` puts
+  `src/` and `service/` on `sys.path`, so no `PYTHONPATH` export is needed.
+- **Commits**: Conventional commits (`feat:`, `fix:`, `docs:`, `chore:`).
+  Author and committer are always the human contributor (`dfry-lhzn <dfry@lhzn.io>`,
+  from `git config`), so the log shows who was behind each change. No `Co-Authored-By:`
+  trailers or non-human identities: GitHub parses those into the Contributors list,
+  which is reserved for people. Noting the agent harness or model that collaborated
+  is welcome as plain text in the commit body. Verify with `git var GIT_AUTHOR_IDENT`
+  before the first commit in a shell.
+- **Git staging**: Stage files explicitly (`git add <file>`). Never `git add .`.
+- **Never commit without approval**: Present the proposed change set and commit message
+  to the user and wait for explicit confirmation before running any `git commit`.
+
+## Domain Constraints
+
+- **Grid normalization**: Elevations and water levels are positive up. Normalize
+  external dataset quirks at the fetcher tier - never let them propagate into the
+  dispatcher.
+- **OPeNDAP access**: Use the `pydap` engine (`engine="pydap"`) for all THREDDS/OPeNDAP
+  endpoints. Prefer `.csvp` endpoints for tabular time-series to avoid NetCDF overhead.
+- **C-grid stagger**: NYOFS (POM) uses an Arakawa C-grid. Interpolation to rho-points
+  must be done in the fetcher before returning data to the dispatcher.
+- **Precision**: Output arrays should default to `float32` (`<f4`) and use Little-Endian
+  endianness for compatibility with `Zarr.jl` and `Oceananigans.jl`.
+
+## Where To Read Next
+
+- `.agents/forcingkit.md`: architecture pipeline, fetcher tier, testing strategy
+  and gotchas
+
+## Compatibility
+
+If a tool only reads `.github/copilot-instructions.md`, `CLAUDE.md`, or another
+vendor-specific file, that file redirects here and adds only the minimum compatibility
+details required by that tool.

forcingkit-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,30 @@
+# Contributing to forcingkit
+
+Thank you for considering contributing to `forcingkit`. This repository is a community resource for providing clean, standardized ocean and atmospheric forcing boundaries for high-fidelity coastal hydrodynamic models. Whether you're fixing a bug, improving documentation, or adding a new data node, contributions are welcome.
+
+## Development Workflow
+
+### 1. Environment Setup
+
+We use `uv` for Python package management.
+
+1. Install `uv`.
+2. Sync the environment: `uv sync`
+3. We use `pre-commit` to ensure code format standardization. Run `pre-commit install` to set up your git hooks.
+
+### 2. Making Changes
+
+- All code must pass `ruff` (for formatting and linting) and `mypy` (for typing).
+- Ensure your changes are covered by tests where applicable (we use `pytest`). Tests are located in the `tests/` directory.
+- `forcingkit.fetchers` logic handles API integrations. If adding a new telemetry source (e.g., a new regional IOOS node), follow the patterns established in `ndbc.py` (observations) or `dbofs.py` (a parent ocean).
+- Keep the Sphinx documentation up to date.
+
+### 3. Pull Requests
+
+1. Create a descriptive branch name: `git checkout -b feature/ioos-integration`
+2. Make your commits clear and logical.
+3. Open a Pull Request.
+
+### 4. Code of Conduct
+
+We encourage an inclusive, patient, and highly respectful environment for all collaborators.

forcingkit-0.1.0/Dockerfile

@@ -0,0 +1,48 @@
+# Tested on: NVIDIA Jetson AGX Orin (JetPack 6.x / l4t r36) and x86_64 (Ubuntu 24.04)
+# Override BASE_IMAGE at build time for your target platform:
+#   Jetson Orin:  --build-arg BASE_IMAGE=nvcr.io/nvidia/l4t-base:r36.2.0
+#   x86_64/WSL2:  --build-arg BASE_IMAGE=ubuntu:24.04  (default)
+ARG BASE_IMAGE=ubuntu:24.04
+FROM ${BASE_IMAGE}
+
+WORKDIR /app
+
+# Install python and dev dependencies
+RUN apt-get update && apt-get install -y \
+    python3 \
+    python3-pip \
+    python3-venv \
+    gcc \
+    curl \
+    libeccodes-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Alias python to python3
+RUN ln -s /usr/bin/python3 /usr/bin/python
+
+# Install UV
+RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_UNMANAGED_INSTALL="/usr/local/bin" sh
+
+# Configure environment
+ENV UV_PROJECT_ENVIRONMENT="/app/.venv" \
+    PYTHONPATH="/app/src:/app/service"
+
+COPY pyproject.toml uv.lock ./
+COPY README.md ./
+COPY src/ /app/src/
+COPY service/ /app/service/
+COPY static/ /app/static/
+
+# Downgrade python was done in pyproject.toml.
+# We use --frozen to respect uv.lock
+RUN uv sync --frozen --no-dev --compile-bytecode
+
+# Fix ownership so non-root users can write to .venv
+ARG UID=1000
+ARG GID=1000
+RUN chown -R ${UID}:${GID} /app
+
+# Default port 9598
+EXPOSE 9598
+
+CMD ["uv", "run", "python", "-m", "forcingkit_serve.main", "--host", "0.0.0.0", "--port", "9598"]