geoextract-gee 0.2.0__tar.gz

geoextract_gee-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bruno Cocitto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

geoextract_gee-0.2.0/PKG-INFO

@@ -0,0 +1,123 @@
+Metadata-Version: 2.4
+Name: geoextract-gee
+Version: 0.2.0
+Summary: Google Earth Engine backend for geoprepare (geoextract)
+Author: Bruno Cocitto
+License-Expression: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: GIS
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: geoprepare>=0.6.331
+Requires-Dist: earthengine-api
+Requires-Dist: google-auth
+Requires-Dist: google-cloud-storage
+Requires-Dist: geopandas
+Requires-Dist: pandas
+Requires-Dist: shapely
+Requires-Dist: arrow
+Requires-Dist: tenacity
+Requires-Dist: tqdm
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# geoextract_gee
+
+Google Earth Engine backend extension for `geoprepare` (geoextract). 
+
+This package provides an optional Google Earth Engine (GEE) execution path for Earth Observation (EO) data extraction. It orchestrates task combinations, processes them using GEE, polls the tasks, and downloads the resulting CSVs from Google Cloud Storage (GCS).
+
+## Prerequisites
+
+- **Google Earth Engine Account**: Ensure you have access and a project set up.
+- **Google Cloud Storage**: A GCS bucket is required to export the results before downloading.
+- **Environment**: It is recommended to refer to [installer](https://github.com/ritviksahajpal/installer) for setting up the environment required for `geocif` and `geoprepare`.
+
+## Installation
+
+```bash
+pip install geoextract-gee
+```
+
+This also installs `geoprepare`, which it extends. geoprepare switches to this backend when `use_gee = True` (see Configuration).
+
+## Authentication
+
+Set `key` in `[GEE]` to a service account JSON key, as a path relative to geoprepare's metadata folder. Without it (or with `key = none`), the standard Earth Engine sign-in (`ee.Authenticate()`) runs the first time; follow its instructions.
+
+## Configuration
+
+Turn the backend on in `[DEFAULT]` of your geoprepare config, and put its settings in a `[GEE]` section:
+
+```ini
+[DEFAULT]
+use_gee = True
+redo = False
+
+[GEE]
+project = your-gcp-project-name
+bucket = your-gcs-bucket-name
+
+# Optional
+# Service account key, relative to the metadata folder. Default: interactive sign-in.
+key = gee_key.json
+# Compute extra counts and sums per region, for checking results. Costs more EECU. Default False.
+audit_stats = False
+# One GEE task per country instead of one per region (see Architecture). Default False.
+parallel_regions = True
+# Price of one EECU-hour, in dollars, for the cost shown while polling. Default 0.4.
+eecu_h_pricing = 0.4
+# Extract variables that have no GEE handler with the local tiles backend
+# (see "Variables not in Earth Engine" below). Default False.
+fallback_local = False
+# Write GEE outputs to dir_output/gee/ instead of dir_output/, for comparing the
+# two backends. A debugging aid; geomerge reads dir_output. Default False.
+use_subdir = False
+```
+
+Configs from before version 0.2.0 have these settings in `[DEFAULT]` with a `gee_` prefix (`gee_project`, `gee_bucket`, `gee_key`, `gee_audit_stats`, `gee_parallel_regions`, `gee_eecu_h_pricing`, `gee_fallback_local`) and `use_gee_subdir`. They still work, with a warning naming the new setting.
+
+## Supported Datasets
+
+The GEE backend currently supports extraction for the following `geoprepare` datasets:
+
+| Dataset | Description | Notes |
+|---------|-------------|-------|
+| **AEF** (`aef`) | AlphaEarth Foundations satellite embeddings | Extracted as a multi-year static average. The GEE dataset has different scaling and quantization compared to the raw files from Source Cooperative. |
+| **AgERA5** (`agera5_*`) | Agrometeorological indicators | Extracted from the Climate Engine Community Catalog. Supports `agera5_snow_thickness_lwe` and `agera5_dewpoint` (converted from Kelvin to °C, as in the tiles backend). Only variables the tiles backend also extracts are added, so both backends give the same columns. |
+| **CHIRPS** (`chirps`) | Precipitation data | CHIRPS v3, IMERG-based daily (`UCSB-CHC/CHIRPS/V3/DAILY_SAT`). Arrives in Earth Engine weeks after the CHC server: on 2026-10-11 the last image was 2026-08-31. |
+| **CPC** (`cpc_*`) | Temperature and precipitation | Supports `cpc_tmax`, `cpc_tmin` and `cpc_precip`. Earth Engine stores precipitation in 0.1 mm; it is divided by 10 to give mm. Its current version starts in 2006, so earlier years are empty. |
+| **ESI** (`esi`) | Evaporative Stress Index | ESI data in GEE often has a processing lag. The pipeline extracts the last available data for the given period. Supports `esi_4wk` and `esi_12wk`. |
+| **NDVI** (`ndvi`) | Vegetation index (MODIS) | Because the GEE pipeline pulls daily observations rather than the 8-day composite blocks used by the legacy `tiles` backend, the timing and number of missing observations at the start or end of a time-series will differ. `geomerge` naturally handles linear interpolation of these gaps across year boundaries. |
+| **NSIDC** (`nsidc`) | SMAP L4 soil moisture (`NASA/SMAP/SPL4SMGP/008`) | Supports `nsidc_surface` and `nsidc_rootzone`. |
+| **SoilGrids** (`soilgrids`) | Soil properties | Extracted as a multi-year static dataset. |
+| **VIIRS** (`viirs`) | High-res VIIRS NDVI data | |
+
+## Variables not in Earth Engine
+
+Some geoprepare variables have no copy in Earth Engine or its community catalog, so this backend cannot extract them:
+
+| Variable | Why |
+|----------|-----|
+| `chirts_era5_tmax`, `chirts_era5_tmin` | Earth Engine has only CHIRTS-daily v1, which ends in 2016. |
+| `chirps_gefs` | CHIRPS-GEFS is not in Earth Engine. |
+| `etref` | NCEP RefET is not in Earth Engine. AgERA5's FAO-56 reference ET is, but it is a different product and geoprepare has no variable for it, so it is not offered as a replacement. |
+
+Other variables without a handler (for example `lst`, `fpar`, `ndvi_cdr`) are treated the same way.
+
+By default (`fallback_local = False`) the run stops with an error listing these variables, before any GEE task is submitted. With `fallback_local = True`, each one is logged with a warning and extracted by geoprepare's local tiles backend while the GEE tasks run. The local backend reads files that geodownload has already written to `dir_intermed`; geoextract does not download them. If they are missing, the run stops before submitting any task and names the geodownload dataset to run (for example `CHIRTS-ERA5`). Fallback variables are written to `dir_output`, not to `dir_output/gee/`, even with `use_subdir = True`. They are extracted one combination at a time.
+
+## Architecture
+
+1.  **Task Submission**: The pipeline extracts data using one of two modes, controlled by `parallel_regions` in `[GEE]`:
+    - **Sequential Mode** (`parallel_regions = False`): For each region and variable combination, a separate GEE task is created. 
+    - **Parallel Batch Mode** (`parallel_regions = True`): All regions within a country are bundled into a single `ee.FeatureCollection` and processed simultaneously by GEE via `reduceRegion` mapping. This submits a single "bulk" GEE task per year/variable, reducing the task queue size and may also avoid quota limits.
+      1. **Note:** For variables requiring heavy reductions (e.g., `nsidc` converting 3-hourly images to daily means), Parallel Batch Mode may cause GEE tasks to take more time to run, time out or run out of memory. In case of error, using Sequential Mode (`parallel_regions = False`) is recommended for these variables.
+      2. **Note:** In Parallel Batch Mode, incremental updates for the current year do not generate partial files like Sequential Mode. Instead, the pipeline automatically re-extracts the entire year and overwrites the local CSVs to ensure seamless data coverage without complex local file merging and different time windows required by each variable.
+2.  **Polling**: The process tracks the submitted tasks and polls Earth Engine until all tasks are marked as `COMPLETED` or `FAILED`. To prevent unexpected billing, polling monitors EECU usage and cancels runaway tasks exceeding defined limits.
+3.  **Download and Format**: Once tasks are complete, the resulting CSVs are downloaded concurrently from Google Cloud Storage using `transfer_manager`. If Parallel Batch Mode was used, the bulk CSV is automatically parsed and split locally into the individual region CSVs expected by `geomerge`.

geoextract_gee-0.2.0/README.md

@@ -0,0 +1,95 @@
+# geoextract_gee
+
+Google Earth Engine backend extension for `geoprepare` (geoextract). 
+
+This package provides an optional Google Earth Engine (GEE) execution path for Earth Observation (EO) data extraction. It orchestrates task combinations, processes them using GEE, polls the tasks, and downloads the resulting CSVs from Google Cloud Storage (GCS).
+
+## Prerequisites
+
+- **Google Earth Engine Account**: Ensure you have access and a project set up.
+- **Google Cloud Storage**: A GCS bucket is required to export the results before downloading.
+- **Environment**: It is recommended to refer to [installer](https://github.com/ritviksahajpal/installer) for setting up the environment required for `geocif` and `geoprepare`.
+
+## Installation
+
+```bash
+pip install geoextract-gee
+```
+
+This also installs `geoprepare`, which it extends. geoprepare switches to this backend when `use_gee = True` (see Configuration).
+
+## Authentication
+
+Set `key` in `[GEE]` to a service account JSON key, as a path relative to geoprepare's metadata folder. Without it (or with `key = none`), the standard Earth Engine sign-in (`ee.Authenticate()`) runs the first time; follow its instructions.
+
+## Configuration
+
+Turn the backend on in `[DEFAULT]` of your geoprepare config, and put its settings in a `[GEE]` section:
+
+```ini
+[DEFAULT]
+use_gee = True
+redo = False
+
+[GEE]
+project = your-gcp-project-name
+bucket = your-gcs-bucket-name
+
+# Optional
+# Service account key, relative to the metadata folder. Default: interactive sign-in.
+key = gee_key.json
+# Compute extra counts and sums per region, for checking results. Costs more EECU. Default False.
+audit_stats = False
+# One GEE task per country instead of one per region (see Architecture). Default False.
+parallel_regions = True
+# Price of one EECU-hour, in dollars, for the cost shown while polling. Default 0.4.
+eecu_h_pricing = 0.4
+# Extract variables that have no GEE handler with the local tiles backend
+# (see "Variables not in Earth Engine" below). Default False.
+fallback_local = False
+# Write GEE outputs to dir_output/gee/ instead of dir_output/, for comparing the
+# two backends. A debugging aid; geomerge reads dir_output. Default False.
+use_subdir = False
+```
+
+Configs from before version 0.2.0 have these settings in `[DEFAULT]` with a `gee_` prefix (`gee_project`, `gee_bucket`, `gee_key`, `gee_audit_stats`, `gee_parallel_regions`, `gee_eecu_h_pricing`, `gee_fallback_local`) and `use_gee_subdir`. They still work, with a warning naming the new setting.
+
+## Supported Datasets
+
+The GEE backend currently supports extraction for the following `geoprepare` datasets:
+
+| Dataset | Description | Notes |
+|---------|-------------|-------|
+| **AEF** (`aef`) | AlphaEarth Foundations satellite embeddings | Extracted as a multi-year static average. The GEE dataset has different scaling and quantization compared to the raw files from Source Cooperative. |
+| **AgERA5** (`agera5_*`) | Agrometeorological indicators | Extracted from the Climate Engine Community Catalog. Supports `agera5_snow_thickness_lwe` and `agera5_dewpoint` (converted from Kelvin to °C, as in the tiles backend). Only variables the tiles backend also extracts are added, so both backends give the same columns. |
+| **CHIRPS** (`chirps`) | Precipitation data | CHIRPS v3, IMERG-based daily (`UCSB-CHC/CHIRPS/V3/DAILY_SAT`). Arrives in Earth Engine weeks after the CHC server: on 2026-10-11 the last image was 2026-08-31. |
+| **CPC** (`cpc_*`) | Temperature and precipitation | Supports `cpc_tmax`, `cpc_tmin` and `cpc_precip`. Earth Engine stores precipitation in 0.1 mm; it is divided by 10 to give mm. Its current version starts in 2006, so earlier years are empty. |
+| **ESI** (`esi`) | Evaporative Stress Index | ESI data in GEE often has a processing lag. The pipeline extracts the last available data for the given period. Supports `esi_4wk` and `esi_12wk`. |
+| **NDVI** (`ndvi`) | Vegetation index (MODIS) | Because the GEE pipeline pulls daily observations rather than the 8-day composite blocks used by the legacy `tiles` backend, the timing and number of missing observations at the start or end of a time-series will differ. `geomerge` naturally handles linear interpolation of these gaps across year boundaries. |
+| **NSIDC** (`nsidc`) | SMAP L4 soil moisture (`NASA/SMAP/SPL4SMGP/008`) | Supports `nsidc_surface` and `nsidc_rootzone`. |
+| **SoilGrids** (`soilgrids`) | Soil properties | Extracted as a multi-year static dataset. |
+| **VIIRS** (`viirs`) | High-res VIIRS NDVI data | |
+
+## Variables not in Earth Engine
+
+Some geoprepare variables have no copy in Earth Engine or its community catalog, so this backend cannot extract them:
+
+| Variable | Why |
+|----------|-----|
+| `chirts_era5_tmax`, `chirts_era5_tmin` | Earth Engine has only CHIRTS-daily v1, which ends in 2016. |
+| `chirps_gefs` | CHIRPS-GEFS is not in Earth Engine. |
+| `etref` | NCEP RefET is not in Earth Engine. AgERA5's FAO-56 reference ET is, but it is a different product and geoprepare has no variable for it, so it is not offered as a replacement. |
+
+Other variables without a handler (for example `lst`, `fpar`, `ndvi_cdr`) are treated the same way.
+
+By default (`fallback_local = False`) the run stops with an error listing these variables, before any GEE task is submitted. With `fallback_local = True`, each one is logged with a warning and extracted by geoprepare's local tiles backend while the GEE tasks run. The local backend reads files that geodownload has already written to `dir_intermed`; geoextract does not download them. If they are missing, the run stops before submitting any task and names the geodownload dataset to run (for example `CHIRTS-ERA5`). Fallback variables are written to `dir_output`, not to `dir_output/gee/`, even with `use_subdir = True`. They are extracted one combination at a time.
+
+## Architecture
+
+1.  **Task Submission**: The pipeline extracts data using one of two modes, controlled by `parallel_regions` in `[GEE]`:
+    - **Sequential Mode** (`parallel_regions = False`): For each region and variable combination, a separate GEE task is created. 
+    - **Parallel Batch Mode** (`parallel_regions = True`): All regions within a country are bundled into a single `ee.FeatureCollection` and processed simultaneously by GEE via `reduceRegion` mapping. This submits a single "bulk" GEE task per year/variable, reducing the task queue size and may also avoid quota limits.
+      1. **Note:** For variables requiring heavy reductions (e.g., `nsidc` converting 3-hourly images to daily means), Parallel Batch Mode may cause GEE tasks to take more time to run, time out or run out of memory. In case of error, using Sequential Mode (`parallel_regions = False`) is recommended for these variables.
+      2. **Note:** In Parallel Batch Mode, incremental updates for the current year do not generate partial files like Sequential Mode. Instead, the pipeline automatically re-extracts the entire year and overwrites the local CSVs to ensure seamless data coverage without complex local file merging and different time windows required by each variable.
+2.  **Polling**: The process tracks the submitted tasks and polls Earth Engine until all tasks are marked as `COMPLETED` or `FAILED`. To prevent unexpected billing, polling monitors EECU usage and cancels runaway tasks exceeding defined limits.
+3.  **Download and Format**: Once tasks are complete, the resulting CSVs are downloaded concurrently from Google Cloud Storage using `transfer_manager`. If Parallel Batch Mode was used, the bulk CSV is automatically parsed and split locally into the individual region CSVs expected by `geomerge`.

geoextract_gee-0.2.0/geoextract_gee/asset_manager.py

@@ -0,0 +1,128 @@
+import hashlib
+import time
+import uuid
+from pathlib import Path
+
+import ee
+from google.cloud import storage
+
+from .datasets.common import gee_setting
+
+def get_file_md5(file_path: Path) -> str:
+    """Calculates the MD5 hash of a local file."""
+    hash_md5 = hashlib.md5()
+    with open(file_path, "rb") as f:
+        for chunk in iter(lambda: f.read(4096), b""):
+            hash_md5.update(chunk)
+    return hash_md5.hexdigest()
+
+def check_asset_exists_and_matches(asset_id: str, local_md5: str) -> bool:
+    """Checks if an EE asset exists and has a matching local_md5 property."""
+    try:
+        asset = ee.data.getAsset(asset_id)
+        asset_md5 = asset.get('properties', {}).get('local_md5')
+        return asset_md5 == local_md5
+    except ee.ee_exception.EEException:
+        # Asset doesn't exist or we don't have access
+        return False
+
+def upload_to_gcs(local_path: Path, bucket_name: str, logger) -> str:
+    """Uploads a local file to GCS and returns the gs:// URI."""
+    client = storage.Client()
+    bucket = client.bucket(bucket_name)
+    # Put them in a staging folder so they don't clutter the export directory
+    dest_blob_name = f"gee_staging/{local_path.name}"
+    blob = bucket.blob(dest_blob_name)
+    
+    logger.info(f"Uploading {local_path.name} to gs://{bucket_name}/{dest_blob_name}...")
+    blob.upload_from_filename(str(local_path))
+    return f"gs://{bucket_name}/{dest_blob_name}"
+
+def start_ingestion(gcs_uri: str, asset_id: str, local_md5: str):
+    """Triggers an Earth Engine ingestion task from a GCS URI."""
+    manifest = {
+        "name": asset_id,
+        "tilesets": [
+            {"id": "tileset_1", "sources": [{"uris": [gcs_uri]}]}
+        ],
+        "properties": {
+            "local_md5": local_md5
+        }
+    }
+    request_id = str(uuid.uuid4())
+    ee.data.startIngestion(request_id, manifest)
+
+def sync_cropmasks(params, unique_paths: set) -> dict:
+    """
+    Synchronizes local TIFF cropmasks with Earth Engine.
+    Returns a dict mapping the local Path string to the final GEE Asset ID.
+    """
+    params.logger.info(f"Synchronizing {len(unique_paths)} unique cropmasks with Earth Engine...")
+    
+    project_name = gee_setting(params, "project")
+    bucket_name = gee_setting(params, "bucket")
+    
+    # 1. Ensure the parent folder exists in GEE
+    folder_id = f"projects/{project_name}/assets/geoextract"
+    try:
+        ee.data.createAsset({'type': 'FOLDER'}, folder_id)
+        params.logger.info(f"Created new GEE folder: {folder_id}")
+    except ee.ee_exception.EEException:
+        pass # Already exists
+        
+    cropmask_map = {}
+    pending_ingestions = []
+    
+    # 2. Check and stage files
+    for path in unique_paths:
+        path_obj = Path(path)
+        asset_name = path_obj.stem # Removes the path and .tif extension
+        asset_id = f"{folder_id}/{asset_name}"
+        cropmask_map[str(path)] = asset_id
+        
+        if not path_obj.exists():
+            params.logger.warning(f"Cropmask file not found locally: {path_obj}")
+            continue
+            
+        md5 = get_file_md5(path_obj)
+        
+        # Check if it already exists in GEE and matches our hash
+        if check_asset_exists_and_matches(asset_id, md5):
+            params.logger.debug(f"Asset {asset_name} already up to date in GEE.")
+            continue
+            
+        # It needs to be uploaded
+        gcs_uri = upload_to_gcs(path_obj, bucket_name, params.logger)
+        
+        params.logger.info(f"Triggering ingestion for {asset_name}...")
+        start_ingestion(gcs_uri, asset_id, md5)
+        pending_ingestions.append(asset_id)
+        
+    # 3. Block and wait for ingestions to finish
+    if pending_ingestions:
+        params.logger.info(f"Waiting for {len(pending_ingestions)} cropmasks to finish ingesting...")
+        
+        while True:
+            ops = ee.data.listOperations()
+            active_ingestions = 0
+            
+            for op in ops:
+                metadata = op.get('metadata', {})
+                state = metadata.get('state')
+                
+                # Check if it's an image ingestion task
+                if metadata.get('type') == 'INGEST_IMAGE':
+                    if state in ['PENDING', 'RUNNING']:
+                        active_ingestions += 1
+                    elif state == 'FAILED':
+                        # Optionally check if it's one of ours, but usually safe to log
+                        params.logger.error(f"An ingestion failed: {op.get('name')}")
+                        
+            if active_ingestions == 0:
+                params.logger.info("All cropmask ingestions have completed.")
+                break
+                
+            params.logger.info(f"{active_ingestions} assets still ingesting. Waiting 30s...")
+            time.sleep(30)
+            
+    return cropmask_map

geoextract_gee-0.2.0/geoextract_gee/datasets/aef.py

@@ -0,0 +1,113 @@
+from __future__ import annotations
+
+import ee
+from .common import TaskConfig, transform_geometry
+
+# --- Dataset Constants ---
+AEF_COLLECTION_ID = "GOOGLE/SATELLITE_EMBEDDING/V1/ANNUAL"
+
+def _aef_stats_feature_batch(
+    avg_img: ee.Image,
+    config: TaskConfig,
+    mask_threshold_percent: float,
+) -> ee.FeatureCollection:
+    """
+    One row per region representing the multi-year average of 
+    the 64 AEF embedding bands over the crop-masked area.
+    """
+    avg_img = ee.Image(avg_img)
+
+    afi_thresh = ee.Number(float(mask_threshold_percent * 100))
+    fc_bounds = config.geometry_r.geometry().bounds(1)
+    w_raw = ee.Image(config.cropmask_asset).float().clip(fc_bounds)
+    w = w_raw.updateMask(w_raw.gt(afi_thresh))
+
+    # Mask the AEF image to the crop area
+    new_names = [f"aef_{i+1}" for i in range(64)]
+    
+    p = avg_img.rename(new_names).float().clip(fc_bounds)
+    pf = p.updateMask(w.mask())
+
+    reducer_common_parms = {
+        "crs": config.reduce_crs,
+        "scale": config.reduce_scale,
+        "maxPixels": 1e9,
+        "tileScale": 4,
+        "bestEffort": True,
+    }
+
+    def _process_region(feat):
+        geom = feat.geometry()
+
+        # Returns a dict like {"A00": 0.123, "A01": 0.456, ...}
+        d_mean = pf.reduceRegion(
+            reducer=ee.Reducer.mean().unweighted(),
+            geometry=geom,
+            **reducer_common_parms
+        )
+        
+        props = ee.Dictionary({
+            "region_label": feat.get("region_label"),
+            "region_id": feat.get("region_id")
+        }).combine(d_mean)
+
+        return ee.Feature(None, props)
+
+    return ee.FeatureCollection(config.geometry_r).map(_process_region)
+
+
+def create_task(config: TaskConfig, mask_threshold_percent: float) -> ee.batch.Task:
+    """Creates a GEE Export task for AEF (AlphaEarth Foundations) 64-band data."""
+    
+    # AEF has data from 2018 onwards
+    start_year = int(config.date_from.split('-')[0])
+    end_year = int(config.date_to.split('-')[0])
+    
+    start_year = max(2018, start_year)
+    end_year = min(2025, end_year - 1)
+    
+    # If the user requested a year completely outside AEF bounds (e.g., 2026-2027),
+    # we fall back to the global 2018-2025 average so geomerge still gets the columns.
+    # This mimics extract_EO using the cached aef_avg.tif for future years
+    if start_year > end_year:
+        start_year = 2018
+        end_year = 2025
+        
+    # Annual images are at 01-01T00:00:00Z, 
+    ic = ee.ImageCollection(AEF_COLLECTION_ID).filterDate(f"{start_year}-01-01", f"{end_year}-01-02")
+    
+    avg_image = ic.mean()
+
+    # ensure projection equals dataset projection to decrease reducers cost
+    aef_proj = ee.Projection(
+        ee.Algorithms.If(
+            ic.size().gt(0), 
+            ic.first().select("A00").projection(), 
+            ee.Projection('EPSG:4326')
+        )
+    )
+    
+    config.geometry_r = transform_geometry(config.geometry_ee, aef_proj)
+    config.reduce_crs = aef_proj.crs()
+    
+    # Force scale to 5600m (0.05 degrees) to match local equivalents
+    # and leverage GEE Image Pyramids for the 64-band reduction
+    config.reduce_scale = 5600
+
+    # We only want one row per region
+    # So wrap the single average image in a single-feature FeatureCollection.
+    col = _aef_stats_feature_batch(avg_image, config, mask_threshold_percent)
+
+    # just region_label and 64 AEF bands
+    selectors = ["region_label"] + [f"aef_{i+1}" for i in range(64)]
+
+    task_desc = config.task_desc
+    
+    return ee.batch.Export.table.toCloudStorage(
+        collection=col,
+        description=task_desc[:100],
+        bucket=config.export_bucket,
+        fileNamePrefix=config.export_prefix,
+        fileFormat="CSV",
+        selectors=selectors,
+    )

geoextract_gee-0.2.0/geoextract_gee/datasets/agera5.py

@@ -0,0 +1,182 @@
+from __future__ import annotations
+
+import ee
+from .common import TaskConfig, transform_geometry
+
+# --- Dataset Constants ---
+AGERA5_COLLECTION_ID = "projects/climate-engine-pro/assets/ce-ag-era5-v2/daily"
+
+# var -> band. Only variables the tiles backend also extracts, so both backends
+# give the same columns (docs/decisions/001).
+AGERA5_BANDS = {
+    "agera5_snow_thickness_lwe": "Snow_Thickness_LWE_Mean_24h",
+    "agera5_dewpoint": "Dew_Point_Temperature_2m_Mean_24h",
+}
+
+def _agera5_daily_stats_feature_batch(
+    img: ee.Image,
+    config: TaskConfig,
+    band_name: str,
+    mask_threshold_percent: float,
+) -> ee.FeatureCollection:
+    """
+    One row per day per region, aligned with geoextract geom_extract + arr_stats.
+    """
+    img = ee.Image(img)
+
+    afi_thresh = ee.Number(float(mask_threshold_percent * 100))
+    fc_bounds = config.geometry_r.geometry().bounds(1)
+    w_raw = ee.Image(config.cropmask_asset).float().clip(fc_bounds)
+    w = w_raw.updateMask(w_raw.gt(afi_thresh))
+
+
+    p = (
+        img.select(band_name)
+        .float()
+        .clip(fc_bounds)
+    )
+    
+    # Same masks and units as the tiles backend (geoprepare extract/stats.py)
+    if config.var == 'agera5_snow_thickness_lwe':
+        p = p.updateMask(p.gte(0))
+    elif config.var == 'agera5_dewpoint':
+        # Kelvin -> deg C; below 150 K is fill, not weather
+        p = p.updateMask(p.gte(150)).subtract(273.15)
+        
+    pf = p.updateMask(w.mask())
+    pf_and_w = pf.addBands(w.rename("weight"))
+
+    if config.include_audit:
+        stack_audit = ee.Image.cat([
+            ee.Image.constant(1).clip(fc_bounds).rename("total_w"),
+            ee.Image.constant(1).updateMask(p.mask()).rename("valid_p_w"),
+            w.rename("w_all"),
+            w.updateMask(pf.mask()).rename("w_used"),
+            pf.pow(2).multiply(w).rename("p2w"),
+        ]).float()
+    else:
+        stack_audit = None
+
+    reducer_common_parms = {
+        "crs": config.reduce_crs,
+        "scale": config.reduce_scale,
+        "maxPixels": 1e9,
+        "tileScale": 4,
+        "bestEffort": True,
+    }
+
+    def _process_region(feat):
+        geom = feat.geometry()
+
+        # Common stats: mean, min, max, median, count (valid after mask)
+        d_mean = pf_and_w.reduceRegion(
+            reducer=ee.Reducer.mean().splitWeights(),
+            geometry=geom,
+            **reducer_common_parms
+        )
+        
+        # Combined min, max, median, and unweighted count
+        d_stats = pf.reduceRegion(
+            reducer=ee.Reducer.minMax()
+                .combine(ee.Reducer.median().unweighted(), sharedInputs=True)
+                .combine(ee.Reducer.count().unweighted(), sharedInputs=True),
+            geometry=geom,
+            **reducer_common_parms
+        )
+
+        valid_after = d_stats.getNumber(f"{band_name}_count")
+        mean_raw = d_mean.getNumber("mean")
+        
+        props = ee.Dictionary({
+            "date": img.date().format("YYYY-MM-dd"),
+            "region_label": feat.get("region_label"),
+            "region_id": feat.get("region_id"),
+            "stats_mean": mean_raw,
+            "stats_min": d_stats.get(f"{band_name}_min"),
+            "stats_max": d_stats.get(f"{band_name}_max"),
+            "stats_median": d_stats.get(f"{band_name}_median"),
+            "stats_count": valid_after,
+        })
+
+        if config.include_audit:
+            d_audit = stack_audit.reduceRegion(
+                reducer=ee.Reducer.sum().unweighted(),
+                geometry=geom,
+                **reducer_common_parms
+            )
+
+            w_used_sum = d_audit.getNumber("w_used")
+            sum_p2w = d_audit.getNumber("p2w")
+            
+            # Weighted Std Dev logic
+            var_raw = ee.Number(
+                ee.Algorithms.If(
+                    w_used_sum.gt(0),
+                    sum_p2w.divide(w_used_sum).subtract(mean_raw.pow(2)),
+                    0,
+                )
+            )
+            std_mm = ee.Number(
+                ee.Algorithms.If(
+                    w_used_sum.lte(0),
+                    None,
+                    ee.Algorithms.If(valid_after.gt(1), var_raw.max(0).sqrt(), 0),
+                )
+            )
+
+            props = props.combine(ee.Dictionary({
+                "counts_total": d_audit.getNumber("total_w"),
+                "counts_valid_data": d_audit.getNumber("valid_p_w"),
+                "counts_valid_data_after_masking": valid_after,
+                "counts_weight_sum": d_audit.getNumber("w_all"),
+                "counts_weight_sum_used": w_used_sum,
+                "stats_std": std_mm,
+            }))
+
+        return ee.Feature(None, props)
+
+    return ee.FeatureCollection(config.geometry_r).map(_process_region)
+
+
+def create_task(config: TaskConfig, mask_threshold_percent: float) -> ee.batch.Task:
+    """Creates a GEE Export task for AgERA5 data."""
+    ic = (
+        ee.ImageCollection(AGERA5_COLLECTION_ID)
+        .filterDate(config.date_from, config.date_to)
+        .filterBounds(config.geometry_ee)
+    )
+
+    band_name = AGERA5_BANDS[config.var]
+
+    # ensure projection equals dataset projection to decrease reducers cost
+    agera5_proj = ee.Projection(
+        ee.Algorithms.If(
+            ic.size().gt(0), 
+            ic.first().select(band_name).projection(), 
+            ee.Projection('EPSG:4326')
+        )
+    )
+    
+    config.geometry_r = transform_geometry(config.geometry_ee, agera5_proj)
+    config.reduce_crs = agera5_proj.crs()
+    config.reduce_scale = agera5_proj.nominalScale()
+
+    col = ic.map(lambda im: _agera5_daily_stats_feature_batch(im, config, band_name, mask_threshold_percent)).flatten()
+
+    selectors = ["date", "region_label", "stats_mean", "stats_min", "stats_max", "stats_median", "stats_count"]
+    if config.include_audit:
+        selectors += [
+            "counts_total", "counts_valid_data", "counts_valid_data_after_masking",
+            "counts_weight_sum", "counts_weight_sum_used", "stats_std"
+        ]
+
+    task_desc = config.task_desc
+    
+    return ee.batch.Export.table.toCloudStorage(
+        collection=col,
+        description=task_desc[:100],
+        bucket=config.export_bucket,
+        fileNamePrefix=config.export_prefix,
+        fileFormat="CSV",
+        selectors=selectors,
+    )