h2mare 0.1.0__tar.gz

h2mare-0.1.0/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 h2ugoparra
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

h2mare-0.1.0/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.4
+Name: h2mare
+Version: 0.1.0
+Summary: Downloader and data management tools for climate and ocean datasets.
+Author-email: Hugo Parra <h2ugo.parra@pm.me>
+License-Expression: MIT
+Project-URL: Repository, https://github.com/h2ugoparra/h2mare
+Keywords: ocean,climate,geospatial,zarr,netcdf,cmems,copernicus,era5
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
+Classifier: Topic :: Scientific/Engineering :: GIS
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: cartopy>=0.23.0
+Requires-Dist: cdsapi>=0.7.0
+Requires-Dist: cfgrib>=0.9.10
+Requires-Dist: copernicusmarine>=2.0.0
+Requires-Dist: dask>=2024.1.0
+Requires-Dist: duckdb>=1.0.0
+Requires-Dist: eccodes>=1.7.0
+Requires-Dist: ephem>=4.1.0
+Requires-Dist: geopandas>=1.0.0
+Requires-Dist: global_land_mask>=1.0.0
+Requires-Dist: ipython>=8.0.0
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: matplotlib>=3.8.0
+Requires-Dist: msgspec>=0.18.0
+Requires-Dist: netCDF4>=1.6.0
+Requires-Dist: plotly>=5.18.0
+Requires-Dist: polars>=1.0.0
+Requires-Dist: PyYAML>=6.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: rioxarray>=0.17.0
+Requires-Dist: scipy>=1.13.0
+Requires-Dist: tqdm>=4.66.0
+Requires-Dist: typer>=0.12.0
+Requires-Dist: xarray>=2024.1.0
+Dynamic: license-file
+
+# H2MARE - Geospatial Processing for Climate and Ocean Data
+
+![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)
+
+A Python pipeline for downloading and preprocessing multi-source oceanographic and atmospheric data into analysis-ready formats. H2MARE streamlines the acquisition and harmonization of data from major climate and ocean observation services, optimized for large-scale spatiotemporal analysis.
+
+## Features
+
+- **Multi-source data integration**: Download and process data from CMEMS, AVISO, and ERA5.
+- **Variable grouping**: Organize related variables using configurable keys.
+- **Format conversion**: Automated conversion from NetCDF/GRIB to optimized Zarr and Parquet format
+- **Data compilation**: Regrid and interpolate multi-resolution datasets to a common grid
+- **Point and geometry extraction**: Extract time series for specific locations or spatial features
+
+## Data Sources
+
+H2MARE supports the following data providers API keys and authentication are required for each:
+
+- **[CMEMS](https://marine.copernicus.eu/)** - Copernicus Marine Service: Satellite and in-situ ocean observations
+- **[AVISO](https://www.aviso.altimetry.fr/en/home.html)** - Archiving, Validation and Interpretation of Satellite Oceanographic data
+- **[CDS-ERA5](https://cds.climate.copernicus.eu/)** - ERA5 hourly atmospheric reanalysis (1940-present)  
+  *Hersbach, H., et al. (2023). DOI: 10.24381/cds.adbb2d47*
+
+**Note**: Refer to each provider's documentation for authentication setup before use.
+
+## Installation
+
+### Prerequisites
+
+- Python >= 3.9
+- [uv](https://docs.astral.sh/uv/) — fast Python package and project manager
+- Sufficient disk space for downloaded datasets (varies by region and time range)
+
+### Install from source
+
+```bash
+git clone https://github.com/h2ugoparra/h2mare.git
+cd h2mare
+uv sync          # installs all dependencies into .venv
+```
+
+For development (includes pytest, black, isort):
+
+```bash
+uv sync --extra dev
+```
+
+## Configuration
+
+Create .env file with external storage path:
+
+```env
+STORE_DIR=/path/to/your/storage
+```
+
+### Key variables groups
+
+Edit `config.yaml` to define variable groups and processing parameters.
+
+### Data Flow
+
+- **Dowload** -  Raw NetCDF/GRIB files are fetched from configurated sources and saved at specified time resolution (monthly or yearly) as native-resolution Zarr files.
+- **Compilation** (`h2mare/processing/compiler.py`) - Preprocessed data is regridded to a defined spatial/temporal resolution and geographic extent (configured via 'h2ds' key in `config.yaml`)
+- **Extraction** (`h2mare/processing/extractor.py`) - Point (CSV files) or geometry (SHP files) data extraction from xarray datasets.
+
+## Quick Start
+
+```bash
+# Download and process a single variable for a specific date range
+uv run h2mare run sst --start-date 2021-01-01 --end-date 2021-12-31
+
+# Multiple variables at once (space-separated)
+uv run h2mare run seapodym mld o2 chl
+
+# Infer missing dates from the existing store and download what's new
+uv run h2mare run sst
+
+# Download only (skip Zarr conversion)
+uv run h2mare run sst --no-process
+
+# Validate configuration without downloading
+uv run h2mare run sst --dry-run
+
+# Process all configured variables
+uv run h2mare run
+```
+
+## Development
+
+```bash
+# Run the full test suite
+uv run pytest tests/
+
+# Run a single test file
+uv run pytest tests/test_zarr_catalog.py -v
+
+# Format code
+uv run black h2mare/
+uv run isort h2mare/
+```
+
+## Built with
+
+| Library | Role |
+|---------|------|
+| [xarray](https://xarray.dev/) | N-dimensional labelled arrays and NetCDF/Zarr I/O |
+| [zarr](https://zarr.dev/) | Chunked, compressed array storage |
+| [dask](https://www.dask.org/) | Parallel and out-of-core computation |
+| [polars](https://pola.rs/) | Fast DataFrame engine for extracted time series |
+| [geopandas](https://geopandas.org/) | Geometry-based spatial extraction |
+| [copernicusmarine](https://pypi.org/project/copernicusmarine/) | CMEMS dataset access |
+| [cdsapi](https://pypi.org/project/cdsapi/) | ERA5 / CDS dataset access |
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit issues or pull requests on [GitHub](https://github.com/h2ugoparra/h2mare.git).
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## AI Assistance
+
+Parts of this codebase were developed with the help of [Claude](https://claude.ai) (Anthropic).
+
+## Acknowledgments
+
+This project was developed under the framework of [COSTA project](https://costaproject.org/en/). This project relies on data from Copernicus Marine Service, AVISO, Copernicus Climate Data Store, and NOAA NCEI. We gratefully acknowledge these organizations for providing open access to their datasets.

h2mare-0.1.0/README.md

@@ -0,0 +1,128 @@
+# H2MARE - Geospatial Processing for Climate and Ocean Data
+
+![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)
+
+A Python pipeline for downloading and preprocessing multi-source oceanographic and atmospheric data into analysis-ready formats. H2MARE streamlines the acquisition and harmonization of data from major climate and ocean observation services, optimized for large-scale spatiotemporal analysis.
+
+## Features
+
+- **Multi-source data integration**: Download and process data from CMEMS, AVISO, and ERA5.
+- **Variable grouping**: Organize related variables using configurable keys.
+- **Format conversion**: Automated conversion from NetCDF/GRIB to optimized Zarr and Parquet format
+- **Data compilation**: Regrid and interpolate multi-resolution datasets to a common grid
+- **Point and geometry extraction**: Extract time series for specific locations or spatial features
+
+## Data Sources
+
+H2MARE supports the following data providers API keys and authentication are required for each:
+
+- **[CMEMS](https://marine.copernicus.eu/)** - Copernicus Marine Service: Satellite and in-situ ocean observations
+- **[AVISO](https://www.aviso.altimetry.fr/en/home.html)** - Archiving, Validation and Interpretation of Satellite Oceanographic data
+- **[CDS-ERA5](https://cds.climate.copernicus.eu/)** - ERA5 hourly atmospheric reanalysis (1940-present)  
+  *Hersbach, H., et al. (2023). DOI: 10.24381/cds.adbb2d47*
+
+**Note**: Refer to each provider's documentation for authentication setup before use.
+
+## Installation
+
+### Prerequisites
+
+- Python >= 3.9
+- [uv](https://docs.astral.sh/uv/) — fast Python package and project manager
+- Sufficient disk space for downloaded datasets (varies by region and time range)
+
+### Install from source
+
+```bash
+git clone https://github.com/h2ugoparra/h2mare.git
+cd h2mare
+uv sync          # installs all dependencies into .venv
+```
+
+For development (includes pytest, black, isort):
+
+```bash
+uv sync --extra dev
+```
+
+## Configuration
+
+Create .env file with external storage path:
+
+```env
+STORE_DIR=/path/to/your/storage
+```
+
+### Key variables groups
+
+Edit `config.yaml` to define variable groups and processing parameters.
+
+### Data Flow
+
+- **Dowload** -  Raw NetCDF/GRIB files are fetched from configurated sources and saved at specified time resolution (monthly or yearly) as native-resolution Zarr files.
+- **Compilation** (`h2mare/processing/compiler.py`) - Preprocessed data is regridded to a defined spatial/temporal resolution and geographic extent (configured via 'h2ds' key in `config.yaml`)
+- **Extraction** (`h2mare/processing/extractor.py`) - Point (CSV files) or geometry (SHP files) data extraction from xarray datasets.
+
+## Quick Start
+
+```bash
+# Download and process a single variable for a specific date range
+uv run h2mare run sst --start-date 2021-01-01 --end-date 2021-12-31
+
+# Multiple variables at once (space-separated)
+uv run h2mare run seapodym mld o2 chl
+
+# Infer missing dates from the existing store and download what's new
+uv run h2mare run sst
+
+# Download only (skip Zarr conversion)
+uv run h2mare run sst --no-process
+
+# Validate configuration without downloading
+uv run h2mare run sst --dry-run
+
+# Process all configured variables
+uv run h2mare run
+```
+
+## Development
+
+```bash
+# Run the full test suite
+uv run pytest tests/
+
+# Run a single test file
+uv run pytest tests/test_zarr_catalog.py -v
+
+# Format code
+uv run black h2mare/
+uv run isort h2mare/
+```
+
+## Built with
+
+| Library | Role |
+|---------|------|
+| [xarray](https://xarray.dev/) | N-dimensional labelled arrays and NetCDF/Zarr I/O |
+| [zarr](https://zarr.dev/) | Chunked, compressed array storage |
+| [dask](https://www.dask.org/) | Parallel and out-of-core computation |
+| [polars](https://pola.rs/) | Fast DataFrame engine for extracted time series |
+| [geopandas](https://geopandas.org/) | Geometry-based spatial extraction |
+| [copernicusmarine](https://pypi.org/project/copernicusmarine/) | CMEMS dataset access |
+| [cdsapi](https://pypi.org/project/cdsapi/) | ERA5 / CDS dataset access |
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit issues or pull requests on [GitHub](https://github.com/h2ugoparra/h2mare.git).
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## AI Assistance
+
+Parts of this codebase were developed with the help of [Claude](https://claude.ai) (Anthropic).
+
+## Acknowledgments
+
+This project was developed under the framework of [COSTA project](https://costaproject.org/en/). This project relies on data from Copernicus Marine Service, AVISO, Copernicus Climate Data Store, and NOAA NCEI. We gratefully acknowledge these organizations for providing open access to their datasets.

h2mare-0.1.0/h2mare/__init__.py

@@ -0,0 +1,29 @@
+"""
+h2mare - Geospatial Processing for Climate and Ocean Data
+
+Main components:
+- config: Project paths and settings
+- models: Data models for configuration
+"""
+
+__version__ = "0.1.0"
+
+from .config import settings
+from .models import AppConfig, KeyVarConfigEntry, VariablesConfig
+from .types import BBox, DateLike, DateRange, DownloadTask, TimeResolution
+from .validators import validate_time_resolution, validate_var_key, validate_var_keys
+
+__all__ = [
+    "settings",
+    "AppConfig",
+    "VariablesConfig",
+    "KeyVarConfigEntry",
+    "DateLike",
+    "DateRange",
+    "BBox",
+    "TimeResolution",
+    "DownloadTask",
+    "validate_var_key",
+    "validate_var_keys",
+    "validate_time_resolution",
+]

h2mare-0.1.0/h2mare/cli/__init__.py

@@ -0,0 +1,24 @@
+"""H2GIS command-line interface."""
+
+import typer
+
+from h2mare.cli.catalog import catalog
+from h2mare.cli.compile import compile
+from h2mare.cli.main import run
+from h2mare.cli.nc2zarr import convert
+
+app = typer.Typer(
+    name="h2mare",
+    help="Climate and ocean data pipeline — download, convert, and inspect.",
+    no_args_is_help=True,
+)
+
+app.command("run", help="Download and convert data for one or more variable keys.")(run)
+app.command(
+    "convert", help="Convert downloaded NetCDF/GRIB files to Zarr (no download)."
+)(convert)
+app.command("catalog", help="Inspect ZarrCatalog metadata for a variable.")(catalog)
+app.command(
+    "compile",
+    help="Merge per-variable Zarr stores into the unified h2ds compiled dataset.",
+)(compile)

h2mare-0.1.0/h2mare/cli/catalog.py

@@ -0,0 +1,121 @@
+"""
+h2mare catalog — inspect ZarrCatalog metadata for a variable.
+
+Shows coverage, file count, variables, and per-dataset breakdown from the
+local Parquet index without opening any Zarr files.
+
+Examples
+--------
+    # Summary for SST
+    uv run h2mare catalog sst
+
+    # Summary for all configured variables
+    uv run h2mare catalog --all
+
+    # Show individual catalog rows
+    uv run h2mare catalog sst --rows
+"""
+
+from typing import Optional
+
+import pandas as pd
+import typer
+from loguru import logger
+
+from h2mare.config import settings
+
+app = typer.Typer()
+
+
+def _print_catalog(var_key: str, show_rows: bool) -> None:
+    from h2mare.storage.zarr_catalog import ZarrCatalog
+
+    try:
+        cat = ZarrCatalog(var_key)
+    except Exception as e:
+        typer.echo(f"  [{var_key}] Could not load catalog: {e}", err=True)
+        return
+
+    df = cat.df
+    summary = cat.summary()
+    cov = summary.get("time_coverage")
+
+    typer.echo(f"\nZarrCatalog — {var_key.upper()}")
+    typer.echo(f"  Files      : {summary['num_files']}")
+
+    if cov and cov != "No data":
+        typer.echo(f"  Coverage   : {cov.start.date()} → {cov.end.date()}")
+    else:
+        typer.echo(f"  Coverage   : No data")
+
+    variables = summary.get("variables") or set()
+    typer.echo(f"  Variables  : {', '.join(sorted(variables)) if variables else '—'}")
+    typer.echo(f"  Timesteps  : {summary.get('total_timesteps', '—')}")
+    typer.echo(f"  Store      : {summary['store_root']}")
+    typer.echo(f"  Catalog    : {summary['catalog_path']}")
+    last = summary.get("last_scanned")
+    typer.echo(
+        f"  Scanned    : {last.strftime('%Y-%m-%d %H:%M:%S') if pd.notna(last) else '—'}"
+    )
+
+    if not df.empty and "dataset" in df.columns:
+        typer.echo("\n  Dataset breakdown:")
+        for dataset, group in df.groupby("dataset", sort=True):
+            start = group["start_date"].min()
+            end = group["end_date"].max()
+            n_ts = (
+                group["num_timesteps"].sum()
+                if "num_timesteps" in group.columns
+                else "—"
+            )
+            typer.echo(f"    {dataset}")
+            typer.echo(f"      {start.date()} → {end.date()}  ({n_ts} timesteps)")
+
+    if show_rows and not df.empty:
+        cols = [
+            c
+            for c in ["filename", "dataset", "start_date", "end_date", "num_timesteps"]
+            if c in df.columns
+        ]
+        typer.echo(f"\n  Rows:\n{df[cols].to_string(index=False)}")
+
+
+def catalog(
+    var_key: Optional[str] = typer.Argument(
+        None,
+        help="Variable key to inspect (e.g. sst, ssh). Omit with --all to show every variable.",
+    ),
+    all_vars: bool = typer.Option(
+        False,
+        "--all",
+        "-a",
+        is_flag=True,
+        help="Show catalog summary for all variables configured in config.yaml.",
+    ),
+    show_rows: bool = typer.Option(
+        False,
+        "--rows",
+        "-r",
+        is_flag=True,
+        help="Print individual catalog rows (filename, dataset, dates, timesteps).",
+    ),
+) -> None:
+    """Inspect ZarrCatalog metadata: coverage, file count, and per-dataset breakdown."""
+
+    if not var_key and not all_vars:
+        typer.echo("Provide a variable key or use --all.", err=True)
+        raise typer.Exit(code=1)
+
+    keys = list(settings.app_config.variables.keys()) if all_vars else [var_key]
+
+    for key in keys:
+        if key not in settings.app_config.variables:
+            typer.echo(
+                f"Unknown variable key '{key}'. Available: {', '.join(settings.app_config.variables)}.",
+                err=True,
+            )
+            continue
+        _print_catalog(key, show_rows)
+
+
+app.command()(catalog)

h2mare-0.1.0/h2mare/cli/compile.py

@@ -0,0 +1,99 @@
+"""
+h2mare compile — merge per-variable Zarr stores into a unified h2ds dataset.
+
+Reads the individual per-variable Zarr stores and interpolates them to a
+common 0.25° daily grid, writing the result as the h2ds compiled dataset.
+When no dates are given the step infers what is missing from the local store.
+
+Examples
+--------
+    # Compile all available variables (dates inferred from store)
+    uv run h2mare compile
+
+    # Compile specific variables over a date range
+    uv run h2mare compile -v sst -v ssh -v mld --start-date 2024-01-01 --end-date 2024-12-31
+
+    # Compile with a custom store path
+    uv run h2mare compile --store-path D:/GlobalData
+"""
+
+from pathlib import Path
+from typing import List, Optional
+
+import pandas as pd
+import typer
+from loguru import logger
+
+from h2mare.config import settings
+
+app = typer.Typer()
+
+
+def compile(
+    vars: Optional[List[str]] = typer.Option(
+        None,
+        "--vars",
+        "-v",
+        help=(
+            "Variable key(s) to compile (repeat for multiple: -v sst -v ssh). "
+            "Defaults to all available keys."
+        ),
+    ),
+    start_date: Optional[str] = typer.Option(
+        None,
+        "--start-date",
+        help="Start date (YYYY-MM-DD). Must be paired with --end-date.",
+    ),
+    end_date: Optional[str] = typer.Option(
+        None,
+        "--end-date",
+        help="End date (YYYY-MM-DD). Must be paired with --start-date.",
+    ),
+    store_path: Optional[Path] = typer.Option(
+        None,
+        "--store-path",
+        help="Override the Zarr store root (defaults to STORE_DIR from .env).",
+    ),
+) -> None:
+    """Merge per-variable Zarr stores into the unified h2ds compiled dataset."""
+
+    log_path = settings.LOGS_DIR / "h2mare.log"
+    logger.add(log_path, level="INFO")
+
+    if bool(start_date) ^ bool(end_date):
+        typer.echo(
+            "Error: --start-date and --end-date must be provided together.", err=True
+        )
+        raise typer.Exit(code=1)
+
+    if start_date and end_date:
+        start_ts = pd.Timestamp(start_date)
+        end_ts = pd.Timestamp(end_date)
+        if start_ts >= end_ts:
+            typer.echo(
+                f"Error: --start-date ({start_date}) must be before --end-date ({end_date}).",
+                err=True,
+            )
+            raise typer.Exit(code=1)
+
+    if vars:
+        available = set(settings.app_config.variables.keys())
+        unknown = set(vars) - available
+        if unknown:
+            typer.echo(
+                f"Error: unknown variable key(s): {', '.join(sorted(unknown))}. "
+                f"Available: {', '.join(sorted(available))}.",
+                err=True,
+            )
+            raise typer.Exit(code=1)
+
+    from h2mare.processing.compiler import Compiler
+
+    Compiler(remote_store_root=store_path or settings.STORE_DIR).run(
+        start_date=start_date,
+        end_date=end_date,
+        var_keys=list(vars) if vars else None,
+    )
+
+
+app.command()(compile)