cavapy 0.1.0__tar.gz

cavapy-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c)  Food and Agriculture Organization of the United Nations. All rights reserved.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cavapy-0.1.0/PKG-INFO

@@ -0,0 +1,77 @@
+Metadata-Version: 2.1
+Name: cavapy
+Version: 0.1.0
+Summary: CAVA Python package. Retrive and analyze climate data.
+Home-page: https://github.com/Risk-Team/cavapy
+License: MIT
+Author: Riccardo Soldan
+Author-email: riccardosoldan@hotmail.it
+Requires-Python: >=3.11,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: bottleneck (>=1.4.2,<2.0.0)
+Requires-Dist: dask (>=2024.11.2,<2025.0.0)
+Requires-Dist: geopandas (>=0.14.4,<0.15.0)
+Requires-Dist: llvmlite (==0.43.0)
+Requires-Dist: matplotlib (>=3.9.2,<4.0.0)
+Requires-Dist: netcdf4 (>=1.7.2,<2.0.0)
+Requires-Dist: xclim (>=0.53.2,<0.54.0)
+Project-URL: Repository, https://github.com/Risk-Team/cavapy
+Description-Content-Type: text/markdown
+
+#### **Status
+
+**Beta testing**
+
+# Introduction
+This project is about creating a Python function to automatically get the required climate data needed to run pyAEZ. This can be an optional feature of the pyAEZ climate module. 
+
+## Data source
+The climate data is available at the THREDDS data server of the University of Cantabria as part of the CAVA (Climate and Agriculture Risk Visualization and Assessment) product developed by FAO, the University of Cantabria, the University of Cape Town and Predictia. 
+CAVA has available CORDEX-CORE climate models, the high resolution (25 Km) dynamically-downscaled climate models used in the IPCC report AR5. Additionally, CAVA  offers access to state-of-the-art reanalyses datasets, such as W5E5 and ERA5.
+
+The currently available data is:
+
+- CORDEX-CORE simulations (3 GCMs donwscaled with 2 RCMs for two RCPs)
+- W5E5 and ERA5 reanalyses datasets
+  
+Available variables are:
+
+- Daily maximum temperature (tasmax)
+- Daily minimum temperature (tasmin)
+- Daily precipitation (pr)
+- Daily relative humidity (hurs)
+- Daily wind speed (sfcWind)
+- Daily solar radiation (rsds)
+
+
+
+## Usage
+The function can be downloaded from the script folder and imported, for example, as follow:
+
+```
+import os
+os.chdir('/path/to/function')
+import cavapy
+# check documentation
+help(cavapy.get_climate_data)
+
+```
+Depending on the interest, downloading climate data can be done in a few different ways. Note that GCM stands for General Circulation Model while RCP stands for Regional Climate Model. As the climate data comes from the CORDEX-CORE initiative, users can choose between 3 different GCMs downscaled with two RCMs. In total, there are six simulations for any given domain (except for CAS-22 where only three are available).
+
+Since bias-correction requires both the historical run of the CORDEX model and the observational dataset (in this case ERA5), even when the historical argument is set to False, the historical run will be used for learning the bias-correction factor.
+
+```
+### Bias-corrected climate projections
+Zambia_climate_data = cavapy.get_climate_data(country="Zambia", cordex_domain="AFR-22", rcp="rcp26", gcm="MPI", rcm="REMO", years_up_to=2030, obs=False, bias_correction=True, historical=False)
+### Non bias-corrected climate projections
+Zambia_climate_data = cavapy.get_climate_data(country="Zambia", cordex_domain="AFR-22", rcp="rcp26", gcm="MPI", rcm="REMO", years_up_to=2030, obs=False, bias_correction=False, historical=False)
+### Bias-corrected climate projections plus the historical run. This is useful when assessing changes in crop yield from the historical period. In this case, we provide the bias-corrected # historical run of the climate models plus the projections. 
+Zambia_climate_data = cavapy.get_climate_data(country="Zambia", cordex_domain="AFR-22", rcp="rcp26", gcm="MPI", rcm="REMO", years_up_to=2030, obs=False, bias_correction=True, historical=True)
+### Observations only (ERA5)
+Zambia_climate_data = cavapy.get_climate_data(country="Zambia", cordex_domain="AFR-22", rcp="rcp26", gcm="MPI", rcm="REMO", years_up_to=2030, obs=True, bias_correction=True, historical=True, years_obs=range(1980,2019))
+```

cavapy-0.1.0/README.md

@@ -0,0 +1,52 @@
+#### **Status
+
+**Beta testing**
+
+# Introduction
+This project is about creating a Python function to automatically get the required climate data needed to run pyAEZ. This can be an optional feature of the pyAEZ climate module. 
+
+## Data source
+The climate data is available at the THREDDS data server of the University of Cantabria as part of the CAVA (Climate and Agriculture Risk Visualization and Assessment) product developed by FAO, the University of Cantabria, the University of Cape Town and Predictia. 
+CAVA has available CORDEX-CORE climate models, the high resolution (25 Km) dynamically-downscaled climate models used in the IPCC report AR5. Additionally, CAVA  offers access to state-of-the-art reanalyses datasets, such as W5E5 and ERA5.
+
+The currently available data is:
+
+- CORDEX-CORE simulations (3 GCMs donwscaled with 2 RCMs for two RCPs)
+- W5E5 and ERA5 reanalyses datasets
+  
+Available variables are:
+
+- Daily maximum temperature (tasmax)
+- Daily minimum temperature (tasmin)
+- Daily precipitation (pr)
+- Daily relative humidity (hurs)
+- Daily wind speed (sfcWind)
+- Daily solar radiation (rsds)
+
+
+
+## Usage
+The function can be downloaded from the script folder and imported, for example, as follow:
+
+```
+import os
+os.chdir('/path/to/function')
+import cavapy
+# check documentation
+help(cavapy.get_climate_data)
+
+```
+Depending on the interest, downloading climate data can be done in a few different ways. Note that GCM stands for General Circulation Model while RCP stands for Regional Climate Model. As the climate data comes from the CORDEX-CORE initiative, users can choose between 3 different GCMs downscaled with two RCMs. In total, there are six simulations for any given domain (except for CAS-22 where only three are available).
+
+Since bias-correction requires both the historical run of the CORDEX model and the observational dataset (in this case ERA5), even when the historical argument is set to False, the historical run will be used for learning the bias-correction factor.
+
+```
+### Bias-corrected climate projections
+Zambia_climate_data = cavapy.get_climate_data(country="Zambia", cordex_domain="AFR-22", rcp="rcp26", gcm="MPI", rcm="REMO", years_up_to=2030, obs=False, bias_correction=True, historical=False)
+### Non bias-corrected climate projections
+Zambia_climate_data = cavapy.get_climate_data(country="Zambia", cordex_domain="AFR-22", rcp="rcp26", gcm="MPI", rcm="REMO", years_up_to=2030, obs=False, bias_correction=False, historical=False)
+### Bias-corrected climate projections plus the historical run. This is useful when assessing changes in crop yield from the historical period. In this case, we provide the bias-corrected # historical run of the climate models plus the projections. 
+Zambia_climate_data = cavapy.get_climate_data(country="Zambia", cordex_domain="AFR-22", rcp="rcp26", gcm="MPI", rcm="REMO", years_up_to=2030, obs=False, bias_correction=True, historical=True)
+### Observations only (ERA5)
+Zambia_climate_data = cavapy.get_climate_data(country="Zambia", cordex_domain="AFR-22", rcp="rcp26", gcm="MPI", rcm="REMO", years_up_to=2030, obs=True, bias_correction=True, historical=True, years_obs=range(1980,2019))
+```

cavapy-0.1.0/cavapy.py

@@ -0,0 +1,631 @@
+import os
+import multiprocessing as mp
+from concurrent.futures import ThreadPoolExecutor
+from functools import partial
+import logging
+import warnings
+
+warnings.filterwarnings(
+    "ignore",
+    category=FutureWarning,
+    message=".*geopandas.dataset module is deprecated.*",
+)
+import geopandas as gpd  # noqa: E402
+import pandas as pd  # noqa: E402
+import xarray as xr  # noqa: E402
+import numpy as np  # noqa: E402
+from xclim import sdba  # noqa: E402
+
+
+logger = logging.getLogger("climate")
+logger.handlers = []  # Remove any existing handlers
+handler = logging.StreamHandler()
+formatter = logging.Formatter(
+    "%(asctime)s | %(name)s | %(process)d:%(thread)d [%(levelname)s]: %(message)s"
+)
+handler.setFormatter(formatter)
+for hdlr in logger.handlers[:]:  # remove all old handlers
+    logger.removeHandler(hdlr)
+logger.addHandler(handler)
+logger.setLevel(logging.DEBUG)
+
+VARIABLES_MAP = {
+    "pr": "tp",
+    "tasmax": "t2mx",
+    "tasmin": "t2mn",
+    "hurs": "hurs",
+    "sfcWind": "sfcwind",
+    "rsds": "ssrd",
+}
+VALID_VARIABLES = list(VARIABLES_MAP)
+# TODO: Throw an error if the selected country is not in the selected domain
+VALID_DOMAINS = [
+    "NAM-22",
+    "EUR-22",
+    "AFR-22",
+    "EAS-22",
+    "SEA-22",
+    "WAS-22",
+    "AUS-22",
+    "SAM-22",
+    "CAM-22",
+]
+VALID_RCPS = ["rcp26", "rcp85"]
+VALID_GCM = ["MOHC", "MPI", "NCC"]
+VALID_RCM = ["REMO", "Reg"]
+
+INVENTORY_DATA_REMOTE_URL = (
+    "https://hub.ipcc.ifca.es/thredds/fileServer/inventories/cava.csv"
+)
+INVENTORY_DATA_LOCAL_PATH = os.path.join(
+    os.path.expanduser("~"), "shared/inventories/cava/inventory.csv"
+)
+ERA5_DATA_REMOTE_URL = (
+    "https://hub.ipcc.ifca.es/thredds/dodsC/fao/observations/ERA5/0.25/ERA5_025.ncml"
+)
+ERA5_DATA_LOCAL_PATH = os.path.join(
+    os.path.expanduser("~"), "shared/data/observations/ERA5/0.25/ERA5_025.ncml"
+)
+DEFAULT_YEARS_OBS = range(1980, 2006)
+
+
+def get_climate_data(
+    *,
+    country: str | None,
+    cordex_domain: str,
+    rcp: str,
+    gcm: str,
+    rcm: str,
+    years_up_to: int,
+    years_obs: range | None = None,
+    bias_correction: bool = False,
+    historical: bool = False,
+    obs: bool = False,
+    buffer: int = 0,
+    xlim: tuple[float, float] | None = None,
+    ylim: tuple[float, float] | None = None,
+    remote: bool = True,
+    num_processes: int = len(VALID_VARIABLES),
+    max_threads_per_process: int = 8,
+) -> dict[str, xr.DataArray]:
+    f"""
+    Process climate data required by pyAEZ climate module.
+    The function automatically access CORDEX-CORE models at 0.25° and the ERA5 datasets.
+
+    Args:
+    country (str): Name of the country for which data is to be processed.
+        Use None if specifying a region using xlim and ylim.
+    cordex_domain (str): CORDEX domain of the climate data. One of {VALID_DOMAINS}.
+    rcp (str): Representative Concentration Pathway. One of {VALID_RCPS}.
+    gcm (str): GCM name. One of {VALID_GCM}.
+    rcm (str): RCM name. One of {VALID_RCM}.
+    years_up_to (int): The ending year for the projected data. Projections start in 2006 and ends in 2100.
+        Hence, if years_up_to is set to 2030, data will be downloaded for the 2006-2030 period.
+    years_obs (range): Range of years for observational data (ERA5 only). Only used when obs is True. (default: None).
+    bias_correction (bool): Whether to apply bias correction (default: False).
+    historical (bool): Flag to indicate if processing historical data (default: False).
+        If True, historical data is provided together with projections.
+        Historical simulation runs for CORDEX-CORE initiative are provided for the 1980-2005 time period.
+    obs (bool): Flag to indicate if processing observational data (default: False).
+    buffer (int): Buffer distance to expand the region of interest (default: 0).
+    xlim (tuple or None): Longitudinal bounds of the region of interest. Use only when country is None (default: None).
+    ylim (tuple or None): Latitudinal bounds of the region of interest. Use only when country is None (default: None).
+    remote (bool): Flag to work with remote data or not (default: True).
+    num_processes (int): Number of processes to use, one per variable.
+        By default equals to the number of all possible variables. (default: {len(VALID_VARIABLES)}).
+    max_threads_per_process (int): Max number of threads within each process. (default: 8).
+
+    Returns:
+    dict: A dictionary containing processed climate data for each variable as an xarray object.
+    """
+
+    if xlim is None and ylim is not None or xlim is not None and ylim is None:
+        raise ValueError(
+            "xlim and ylim mismatch: they must be both specified or both unspecified"
+        )
+    if country is None and xlim is None:
+        raise ValueError("You must specify a country or (xlim, ylim)")
+    if country is not None and xlim is not None:
+        raise ValueError("You must specify either country or (xlim, ylim), not both")
+    verify_variables = {
+        "cordex_domain": VALID_DOMAINS,
+        "rcp": VALID_RCPS,
+        "gcm": VALID_GCM,
+        "rcm": VALID_RCM,
+    }
+    for var_name, valid_values in verify_variables.items():
+        var_value = locals()[var_name]
+        if var_value not in valid_values:
+            raise ValueError(
+                f"Invalid {var_name}={var_value}. Must be one of {valid_values}"
+            )
+    if years_up_to <= 2006:
+        raise ValueError("years_up_to must be greater than 2006")
+    if years_obs is not None and not (1980 <= min(years_obs) <= max(years_obs) <= 2020):
+        raise ValueError("Years in years_obs must be within the range 1980 to 2020")
+    if obs and years_obs is None:
+        raise ValueError("years_obs must be provided when obs is True")
+    if not obs or years_obs is None:
+        # Make sure years_obs is set to default when obs=False
+        years_obs = DEFAULT_YEARS_OBS
+
+    _validate_urls(gcm, rcm, rcp, remote, cordex_domain, obs)
+
+    bbox = _geo_localize(country, xlim, ylim, buffer, cordex_domain)
+
+    with mp.Pool(processes=num_processes) as pool:
+        futures = []
+        for variable in VALID_VARIABLES:
+            futures.append(
+                pool.apply_async(
+                    process_worker,
+                    args=(max_threads_per_process,),
+                    kwds={
+                        "variable": variable,
+                        "bbox": bbox,
+                        "cordex_domain": cordex_domain,
+                        "rcp": rcp,
+                        "gcm": gcm,
+                        "rcm": rcm,
+                        "years_up_to": years_up_to,
+                        "years_obs": years_obs,
+                        "obs": obs,
+                        "bias_correction": bias_correction,
+                        "historical": historical,
+                        "remote": remote,
+                    },
+                )
+            )
+
+        results = {
+            variable: futures[i].get() for i, variable in enumerate(VALID_VARIABLES)
+        }
+
+        pool.close()  # Prevent any more tasks from being submitted to the pool
+        pool.join()  # Wait for all worker processes to finish
+
+    return results
+
+
+def _validate_urls(
+    gcm: str = None,
+    rcm: str = None,
+    rcp: str = None,
+    remote: bool = True,
+    cordex_domain: str = None,
+    obs: bool = False,
+):
+    # Load the data
+    log = logger.getChild("URLs validation")
+
+    if obs is False:
+        inventory_csv_url = (
+            INVENTORY_DATA_REMOTE_URL if remote else INVENTORY_DATA_LOCAL_PATH
+        )
+        data = pd.read_csv(inventory_csv_url)
+
+        # Set the column to use based on whether the data is remote or local
+        column_to_use = "location" if remote else "hub"
+
+        # Filter the data based on the conditions
+        filtered_data = data[
+            lambda x: (
+                x["activity"].str.contains("FAO", na=False)
+                & (x["domain"] == cordex_domain)
+                & (x["model"].str.contains(gcm, na=False))
+                & (x["rcm"].str.contains(rcm, na=False))
+                & (x["experiment"].isin([rcp, "historical"]))
+            )
+        ][["experiment", column_to_use]]
+
+        # Extract the column values as a list
+        num_rows = filtered_data.shape[0]
+        column_values = filtered_data[column_to_use]
+
+        if num_rows == 1:
+            # Log the output for one row
+            row1 = column_values.iloc[0]
+            log.info(f"Projections: {row1}")
+        else:
+            # Log the output for two rows
+            row1 = column_values.iloc[0]
+            row2 = column_values.iloc[1]
+            log.info(f"Historical simulation: {row1}")
+            log.info(f"Projections: {row2}")
+    else:  # when obs is True
+        log.info(f"Observations: {ERA5_DATA_REMOTE_URL}")
+
+
+def _geo_localize(
+    country: str = None,
+    xlim: tuple[float, float] = None,
+    ylim: tuple[float, float] = None,
+    buffer: int = 0,
+    cordex_domain: str = None,
+) -> dict[str, tuple[float, float]]:
+    if country:
+        if xlim or ylim:
+            raise ValueError(
+                "Specify either a country or bounding box limits (xlim, ylim), but not both."
+            )
+        # Load country shapefile and extract bounds
+        world = gpd.read_file(gpd.datasets.get_path("naturalearth_lowres"))
+        country_shp = world[world.name == country]
+        if country_shp.empty:
+            raise ValueError(f"Country '{country}' is unknown.")
+        bounds = country_shp.total_bounds  # [minx, miny, maxx, maxy]
+        xlim, ylim = (bounds[0], bounds[2]), (bounds[1], bounds[3])
+    elif not (xlim and ylim):
+        raise ValueError(
+            "Either a country or bounding box limits (xlim, ylim) must be specified."
+        )
+
+    # Apply buffer
+    xlim = (xlim[0] - buffer, xlim[1] + buffer)
+    ylim = (ylim[0] - buffer, ylim[1] + buffer)
+
+    # Always validate CORDEX domain
+    if cordex_domain:
+        _validate_cordex_domain(xlim, ylim, cordex_domain)
+
+    return {"xlim": xlim, "ylim": ylim}
+
+
+def _validate_cordex_domain(xlim, ylim, cordex_domain):
+
+    # CORDEX domains data
+    cordex_domains_df = pd.DataFrame(
+        {
+            "min_lon": [
+                -33,
+                -28.3,
+                89.25,
+                86.75,
+                19.25,
+                44.0,
+                -106.25,
+                -115.0,
+                -24.25,
+                10.75,
+            ],
+            "min_lat": [
+                -28,
+                -23,
+                -15.25,
+                -54.25,
+                -15.75,
+                -4.0,
+                -58.25,
+                -14.5,
+                -46.25,
+                17.75,
+            ],
+            "max_lon": [
+                20,
+                18,
+                147.0,
+                -152.75,
+                116.25,
+                -172.0,
+                -16.25,
+                -30.5,
+                59.75,
+                140.25,
+            ],
+            "max_lat": [28, 21.7, 26.5, 13.75, 45.75, 65.0, 18.75, 28.5, 42.75, 69.75],
+            "cordex_domain": [
+                "NAM-22",
+                "EUR-22",
+                "SEA-22",
+                "AUS-22",
+                "WAS-22",
+                "EAS-22",
+                "SAM-22",
+                "CAM-22",
+                "AFR-22",
+                "CAS-22",
+            ],
+        }
+    )
+
+    def is_bbox_contained(bbox, domain):
+        """Check if bbox is contained within the domain bounding box."""
+        return (
+            bbox[0] >= domain["min_lon"]
+            and bbox[1] >= domain["min_lat"]
+            and bbox[2] <= domain["max_lon"]
+            and bbox[3] <= domain["max_lat"]
+        )
+
+    user_bbox = [xlim[0], ylim[0], xlim[1], ylim[1]]
+    domain_row = cordex_domains_df[cordex_domains_df["cordex_domain"] == cordex_domain]
+
+    if domain_row.empty:
+        raise ValueError(f"CORDEX domain '{cordex_domain}' is not recognized.")
+
+    domain_bbox = domain_row.iloc[0]
+
+    if not is_bbox_contained(user_bbox, domain_bbox):
+        suggested_domains = cordex_domains_df[
+            cordex_domains_df.apply(
+                lambda row: is_bbox_contained(user_bbox, row), axis=1
+            )
+        ]
+
+        if suggested_domains.empty:
+            raise ValueError(
+                f"The bounding box {user_bbox} is outside of all available CORDEX domains."
+            )
+
+        suggested_domain = suggested_domains.iloc[0]["cordex_domain"]
+
+        raise ValueError(
+            f"Bounding box {user_bbox} is not within '{cordex_domain}'. Suggested domain: '{suggested_domain}'."
+        )
+
+
+def process_worker(num_threads, **kwargs) -> xr.DataArray:
+    variable = kwargs["variable"]
+    log = logger.getChild(variable)
+    try:
+        with ThreadPoolExecutor(
+            max_workers=num_threads, thread_name_prefix="climate"
+        ) as executor:
+            return _climate_data_for_variable(executor, **kwargs)
+    except Exception as e:
+        log.exception(f"Process worker failed: {e}")
+        raise
+
+
+def _climate_data_for_variable(
+    executor: ThreadPoolExecutor,
+    *,
+    variable: str,
+    bbox: dict[str, tuple[float, float]],
+    cordex_domain: str,
+    rcp: str,
+    gcm: str,
+    rcm: str,
+    years_up_to: int,
+    years_obs: range,
+    obs: bool,
+    bias_correction: bool,
+    historical: bool,
+    remote: bool,
+) -> xr.DataArray:
+    log = logger.getChild(variable)
+
+    pd.options.mode.chained_assignment = None
+    inventory_csv_url = (
+        INVENTORY_DATA_REMOTE_URL if remote else INVENTORY_DATA_LOCAL_PATH
+    )
+    data = pd.read_csv(inventory_csv_url)
+    column_to_use = "location" if remote else "hub"
+    filtered_data = data[
+        lambda x: (x["activity"].str.contains("FAO", na=False))
+        & (x["domain"] == cordex_domain)
+        & (x["model"].str.contains(gcm, na=False))
+        & (x["rcm"].str.contains(rcm, na=False))
+        & (x["experiment"].isin([rcp, "historical"]))
+    ][["experiment", column_to_use]]
+
+    future_obs = None
+    if obs or bias_correction:
+        future_obs = executor.submit(
+            _thread_download_data,
+            url=None,
+            bbox=bbox,
+            variable=variable,
+            obs=True,
+            years_up_to=years_up_to,
+            years_obs=years_obs,
+            remote=remote,
+        )
+
+    if not obs:
+        download_fn = partial(
+            _thread_download_data,
+            bbox=bbox,
+            variable=variable,
+            obs=False,
+            years_obs=years_obs,
+            years_up_to=years_up_to,
+            remote=remote,
+        )
+        downloaded_models = list(
+            executor.map(download_fn, filtered_data[column_to_use])
+        )
+
+        # Add the downloaded models to the DataFrame
+        filtered_data["models"] = downloaded_models
+        log.info("Interpolating missing values")
+        hist = (
+            filtered_data["models"].iloc[0].interpolate_na(dim="time", method="linear")
+        )
+        proj = (
+            filtered_data["models"].iloc[1].interpolate_na(dim="time", method="linear")
+        )
+        log.info("Missing values interpolated")
+
+        if bias_correction and historical:
+            # Load observations for bias correction
+            ref = future_obs.result()
+            log.info("Training eqm with historical data")
+            QM_mo = sdba.EmpiricalQuantileMapping.train(
+                ref,
+                hist,
+                group="time.month",
+                kind="*" if variable in ["pr", "rsds", "sfcWind"] else "+",
+            )
+            log.info("Performing bias correction with eqm")
+            hist_bs = QM_mo.adjust(hist, extrapolation="constant", interp="linear")
+            proj_bs = QM_mo.adjust(proj, extrapolation="constant", interp="linear")
+            log.info("Done!")
+            if variable == "hurs":
+                hist_bs = hist_bs.where(hist_bs <= 100, 100)
+                hist_bs = hist_bs.where(hist_bs >= 0, 0)
+            combined = xr.concat([hist_bs, proj_bs], dim="time")
+            return combined
+
+        elif not bias_correction and historical:
+            combined = xr.concat([hist, proj], dim="time")
+            return combined
+
+        elif bias_correction and not historical:
+            ref = future_obs.result()
+            log.info("Training eqm with historical data")
+            QM_mo = sdba.EmpiricalQuantileMapping.train(
+                ref,
+                hist,
+                group="time.month",
+                kind="*" if variable in ["pr", "rsds", "sfcWind"] else "+",
+            )  # multiplicative approach for pr, rsds and wind speed
+            log.info("Performing bias correction with eqm")
+            proj_bs = QM_mo.adjust(proj, extrapolation="constant", interp="linear")
+            log.info("Done!")
+            if variable == "hurs":
+                proj_bs = proj_bs.where(proj_bs <= 100, 100)
+                proj_bs = proj_bs.where(proj_bs >= 0, 0)
+            return proj_bs
+
+        return proj
+
+    else:  # when observations are True
+        downloaded_obs = future_obs.result()
+        log.info("Done!")
+        return downloaded_obs
+
+
+def _thread_download_data(url: str | None, **kwargs):
+    variable = kwargs["variable"]
+    log = logger.getChild(variable)
+    try:
+        return _download_data(url=url, **kwargs)
+    except Exception as e:
+        log.exception(f"Failed to download data from {url}: {e}")
+        raise
+
+
+def _download_data(
+    url: str | None,
+    bbox: dict[str, tuple[float, float]],
+    variable: str,
+    obs: bool,
+    years_obs: range,
+    years_up_to: int,
+    remote: bool,
+) -> xr.DataArray:
+    log = logger.getChild(variable)
+    if obs:
+        var = VARIABLES_MAP[variable]
+        log.info(f"Downloading observational data for {variable}({var})")
+        if remote:
+            ds_var = xr.open_dataset(ERA5_DATA_REMOTE_URL)[var]
+        else:
+            ds_var = xr.open_dataset(ERA5_DATA_LOCAL_PATH)[var]
+        log.info(f"Observational data for {variable}({var}) has been downloaded")
+
+        # Coordinate normalization and renaming for 'hurs'
+        if var == "hurs":
+            ds_var = ds_var.rename({"lat": "latitude", "lon": "longitude"})
+            ds_cropped = ds_var.sel(
+                longitude=slice(bbox["xlim"][0], bbox["xlim"][1]),
+                latitude=slice(bbox["ylim"][0], bbox["ylim"][1]),
+            )
+        else:
+            ds_var.coords["longitude"] = (ds_var.coords["longitude"] + 180) % 360 - 180
+            ds_var = ds_var.sortby(ds_var.longitude)
+            ds_cropped = ds_var.sel(
+                longitude=slice(bbox["xlim"][0], bbox["xlim"][1]),
+                latitude=slice(bbox["ylim"][1], bbox["ylim"][0]),
+            )
+
+        # Unit conversion
+        if var in ["t2mx", "t2mn", "t2m"]:
+            ds_cropped -= 273.15  # Convert from Kelvin to Celsius
+            ds_cropped.attrs["units"] = "°C"
+        elif var == "tp":
+            ds_cropped *= 1000  # Convert precipitation
+            ds_cropped.attrs["units"] = "mm"
+        elif var == "ssrd":
+            ds_cropped /= 86400  # Convert from J/m^2 to W/m^2
+            ds_cropped.attrs["units"] = "W m-2"
+        elif var == "sfcwind":
+            ds_cropped = ds_cropped * (
+                4.87 / np.log((67.8 * 10) - 5.42)
+            )  # Convert wind speed from 10 m to 2 m
+            ds_cropped.attrs["units"] = "m s-1"
+
+        # Select years
+        years = [x for x in years_obs]
+        time_mask = (ds_cropped["time"].dt.year >= years[0]) & (
+            ds_cropped["time"].dt.year <= years[-1]
+        )
+
+    else:
+        log.info(f"Downloading CORDEX data for {variable}")
+        ds_var = xr.open_dataset(url)[variable]
+        log.info(f"CORDEX data for {variable} has been downloaded")
+        ds_cropped = ds_var.sel(
+            longitude=slice(bbox["xlim"][0], bbox["xlim"][1]),
+            latitude=slice(bbox["ylim"][1], bbox["ylim"][0]),
+        )
+
+        # Unit conversion
+        if variable in ["tas", "tasmax", "tasmin"]:
+            ds_cropped -= 273.15  # Convert from Kelvin to Celsius
+            ds_cropped.attrs["units"] = "°C"
+        elif variable == "pr":
+            ds_cropped *= 86400  # Convert from kg m^-2 s^-1 to mm/day
+            ds_cropped.attrs["units"] = "mm"
+        elif variable == "rsds":
+            ds_cropped.attrs["units"] = "W m-2"
+        elif variable == "sfcWind":
+            ds_cropped = ds_cropped * (
+                4.87 / np.log((67.8 * 10) - 5.42)
+            )  # Convert wind speed from 10 m to 2 m
+            ds_cropped.attrs["units"] = "m s-1"
+
+        # Select years based on rcp
+        if "rcp" in url:
+            years = [x for x in range(2006, years_up_to + 1)]
+        else:
+            years = [x for x in DEFAULT_YEARS_OBS]
+
+        # Add missing dates
+        ds_cropped = ds_cropped.convert_calendar(
+            calendar="gregorian", missing=np.nan, align_on="date"
+        )
+        log.debug(
+            "360-calendar converted into Gregorian calendar and missing values linearly interpolated"
+        )
+
+        time_mask = (ds_cropped["time"].dt.year >= years[0]) & (
+            ds_cropped["time"].dt.year <= years[-1]
+        )
+
+    # subset years
+    ds_cropped = ds_cropped.sel(time=time_mask)
+
+    assert isinstance(ds_cropped, xr.DataArray)
+
+    log.info(
+        f"{'Observational' if obs else 'CORDEX'} data for {variable} has been processed"
+    )
+
+    return ds_cropped
+
+
+if __name__ == "__main__":
+    data = get_climate_data(
+        country="Zambia",
+        cordex_domain="AFR-22",
+        rcp="rcp26",
+        gcm="MPI",
+        rcm="REMO",
+        years_up_to=2030,
+        obs=False,
+        bias_correction=True,
+        historical=False,
+    )
+    print(data)

cavapy-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[tool.poetry]
+name = "cavapy"
+version = "0.1.0"
+description = "CAVA Python package. Retrive and analyze climate data."
+authors = ["Riccardo Soldan <riccardosoldan@hotmail.it>", "Oleh Lokshyn <olokshyn@gmail.com>"]
+readme = "README.md"
+license = "MIT"
+repository = "https://github.com/Risk-Team/cavapy"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+
+[tool.poetry.dependencies]
+python = "^3.11"
+xclim = "^0.53.2"
+geopandas = "^0.14.4"
+matplotlib = "^3.9.2"
+netcdf4 = "^1.7.2"
+dask = "^2024.11.2"
+bottleneck = "^1.4.2"
+llvmlite = "0.43.0"
+
+
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"