chap-core 0.0.8__tar.gz

chap_core-0.0.8/HISTORY.rst

@@ -0,0 +1,8 @@
+=======
+History
+=======
+
+0.0.1 (2024-01-23)
+------------------
+
+* First release on PyPI.

chap_core-0.0.8/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2024, Sandvelab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

chap_core-0.0.8/MANIFEST.in

@@ -0,0 +1,10 @@
+include CONTRIBUTING.rst
+include HISTORY.rst
+include LICENSE
+include README.rst
+
+recursive-include tests *
+recursive-exclude * __pycache__
+recursive-exclude * *.py[co]
+
+recursive-include docs *.rst conf.py Makefile make.bat *.jpg *.png *.gif

chap_core-0.0.8/PKG-INFO

@@ -0,0 +1,55 @@
+Metadata-Version: 2.1
+Name: chap_core
+Version: 0.0.8
+Summary: Main repository for joint programming project on climate health
+Home-page: https://github.com/dhis2/chap-core
+Author: Sandvelab
+Author-email: knutdrand@gmail.com
+License: MIT license
+Keywords: chap_core
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Requires-Python: >=3.9
+License-File: LICENSE
+Requires-Dist: virtualenv
+Requires-Dist: numpy<2.0
+Requires-Dist: bionumpy
+Requires-Dist: pandas
+Requires-Dist: plotly
+Requires-Dist: scikit-learn
+Requires-Dist: matplotlib
+Requires-Dist: diskcache
+Requires-Dist: geopy
+Requires-Dist: pooch
+Requires-Dist: python-dateutil
+Requires-Dist: meteostat
+Requires-Dist: cyclopts
+Requires-Dist: requests
+Requires-Dist: fastapi
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml
+Requires-Dist: geopandas
+Requires-Dist: libpysal
+Requires-Dist: docker
+Requires-Dist: scipy
+Requires-Dist: gitpython
+Requires-Dist: earthengine-api
+Requires-Dist: python-dotenv
+Requires-Dist: rq
+Requires-Dist: python-multipart
+Requires-Dist: uvicorn
+Requires-Dist: pydantic-geojson
+Requires-Dist: annotated_types
+Requires-Dist: pycountry
+Requires-Dist: unidecode
+Requires-Dist: httpx
+Requires-Dist: earthengine-api
+Requires-Dist: mlflow
+Requires-Dist: gluonts
+Requires-Dist: xarray
+
+Chap Core

chap_core-0.0.8/README.md

@@ -0,0 +1,67 @@
+# Climate Health Analysis Platform (CHAP)
+CHAP offers a platform for analysing the relationship between climate and health. The platform is designed to be modular and flexible, allowing for easy integration of new models and data sources. The platform is designed to be used by researchers and public health professionals to forecast and assess the impact of climate on health outcomes.
+
+# Installation
+
+Basic installation can be done using pip:
+
+    $ pip install git+https://github.com/dhis2/chap-core.git
+
+If running models, you may also need to install:
+
+- Docker, if running a model that runs through docker
+- [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation), if running a model that uses Python virtual environments
+
+# Documentation
+
+The main documentation is located at [https://dhis2.github.io/chap-core/](https://dhis2.github.io/chap-core/).
+
+# Usage
+
+The following shows basic usage of the platform. Follow the link to the documentation above for more details.
+
+## Evaluate a public model on public data
+CHAP supports evaluating models that are defined using the MLflow specification for machine learning models (link coming). Such models can e.g. exist in Github repositories. CHAP also has some built-in example data that can be used to evaluate models. The following example shows how to evaluate an Ewars model located on Github ([https://github.com/sandvelab/chap_auto_ewars](https://github.com/sandvelab/chap_auto_ewars))  using the ISMIP dataset:
+
+```bash
+chap evaluate --model-name https://github.com/sandvelab/chap_auto_ewars --dataset-name ISIMIP_dengue_harmonized --dataset-country brazil
+```
+
+The above example requires that you have installed chap with pip and also that you have Docker available.
+
+
+## Fetching Polygon Data
+Fetch polygons for regions of interest in a country (on admin1 level). The following example fetches polygons for two regions in Norway
+
+```python
+    import chap_core.fetch
+
+polygons = chap_core.fetch.get_area_polygons('Norway', ['Oslo', 'Akershus'])
+assert [feature.id for feature in polygons.features] == ['Oslo', 'Akershus']
+```
+Region names that are not recognized are skipped:
+
+```python
+    polygons = climate_health.fetch.get_area_polygons('Norway', ['Oslo', 'Akershus', 'Unknown'])
+    assert [feature.id for feature in polygons.features] == ['Oslo', 'Akershus']
+```
+
+## Fetching climate data
+Fetching climate data through Google Earth Engine. The following example fetches temperature data from the ERA5 dataset for the regions of interest. You need to have an an account at gee, and pass the credentials to the method to make this work: https://developers.google.com/earth-engine/guides/auth
+
+```python
+
+import chap_core.fetch
+
+credentials = dict(account='demoaccount@demo.gserviceaccount.com', private_key='private_key')
+polygons = open("polygon_file.geojson").read()
+start_period = '202001'  # January 2020
+end_period = '202011'  # December 2020
+band_names = ['temperature_2m', 'total_precipitation_sum']
+data = chap_core.fetch.gee_era5(credentials, polygons, start_period, end_period, band_names)
+print(data)
+start_week = '2020W03'  # Week 3 of 2020
+end_week = '2020W05'  # Week 5 of 2020
+data = chap_core.fetch.gee_era5(credentials, polygons, start_week, end_week, band_names)
+print(data)
+```

chap_core-0.0.8/chap_core/__init__.py

@@ -0,0 +1,8 @@
+"""Top-level package for chap-core."""
+
+__author__ = """Sandvelab"""
+__email__ = "knutdrand@gmail.com"
+__version__ = "0.0.8"
+
+from . import fetch
+from . import data

chap_core-0.0.8/chap_core/_legacy/file_io.py

@@ -0,0 +1,50 @@
+from typing import List
+
+import chap_core.time_period.dataclasses as dc
+from chap_core.time_period import TimePeriod, Month, Day, Year
+
+
+def parse_period_string(time_string: str) -> TimePeriod:
+    period = TimePeriod.parse(time_string)
+    return period
+
+
+def write_time_series_data(data):
+    def topandas(self):
+        data = pd.DataFrame(
+            {
+                "time_period": self.time_period.topandas(),
+                "rainfall": self.rainfall,
+                "mean_temperature": self.mean_temperature,
+                "disease_cases": self.disease_cases,
+            }
+        )
+        return data
+
+    to_pandas = topandas
+
+    def to_csv(self, csv_file: str, **kwargs):
+        """Write data to a csv file."""
+        data = self.to_pandas()
+        data.to_csv(csv_file, index=False, **kwargs)
+
+
+def parse_periods_strings(time_strings: List[str]) -> dc.Period:
+    periods = [parse_period_string(time_string) for time_string in time_strings]
+    if not periods:
+        return dc.Period.empty()
+    t = type(periods[0])
+    assert all(type(period) == t for period in periods), periods
+
+    if t == Year:
+        return dc.Year([period.year for period in periods])
+    if t == Month:
+        return dc.Month(
+            [period.year for period in periods], [period.month for period in periods]
+        )
+    elif t == Day:
+        return dc.Day(
+            [period.year for period in periods],
+            [period.month for period in periods],
+            [period.day for period in periods],
+        )

chap_core-0.0.8/chap_core/_legacy_dataset.py

@@ -0,0 +1,101 @@
+from typing import Protocol, TypeAlias, Union, Iterable, TypeVar, Tuple
+
+import pandas as pd
+from pydantic import BaseModel
+
+from chap_core.datatypes import Location
+from chap_core.time_period.dataclasses import Period
+
+SpatialIndexType: TypeAlias = Union[str, Location]
+TemporalIndexType: TypeAlias = Union[Period, Iterable[Period], slice]
+
+
+FeaturesT = TypeVar("FeaturesT")
+
+
+class ClimateData(BaseModel):
+    temperature: float
+    rainfall: float
+    humidity: float
+
+
+class DataType(BaseModel):
+    disease_cases: int
+    climate_data: ClimateData
+
+
+class IsTemporalDataSet(Protocol[FeaturesT]):
+    def restrict_time_period(
+        self, start_period: Period = None, end_period: Period = None
+    ) -> "IsTemporalDataSet[FeaturesT]": ...
+
+    def to_tidy_dataframe(self) -> pd.DataFrame: ...
+
+    @classmethod
+    def from_tidy_dataframe(cls, df: pd.DataFrame) -> "IsSpatialDataSet[FeaturesT]": ...
+
+
+class TemporalArray:
+    def __init__(self, time_index, data):
+        self._time_index = time_index
+        self._data = data
+
+    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
+        return ufunc
+
+
+class IsSpatialDataSet(Protocol[FeaturesT]):
+    def get_locations(
+        self, location: Iterable[SpatialIndexType]
+    ) -> "IsSpatialDataSet[FeaturesT]": ...
+
+    def get_location(self, location: SpatialIndexType) -> FeaturesT: ...
+
+    def locations(self) -> Iterable[SpatialIndexType]: ...
+
+    def data(self) -> Iterable[FeaturesT]: ...
+
+    def location_items(self) -> Iterable[Tuple[SpatialIndexType, FeaturesT]]: ...
+
+    def to_tidy_dataframe(self) -> pd.DataFrame: ...
+
+    @classmethod
+    def from_tidy_dataframe(cls, df: pd.DataFrame) -> "IsSpatialDataSet[FeaturesT]": ...
+
+
+class IsSpatioTemporalDataSet(Protocol[FeaturesT]):
+    dataclass = ...
+
+    def get_data_for_locations(
+        self, location: Iterable[SpatialIndexType]
+    ) -> "IsSpatioTemporalDataSet[FeaturesT]": ...
+
+    def get_data_for_location(self, location: SpatialIndexType) -> FeaturesT: ...
+
+    def restrict_time_period(
+        self, start_period: Period = None, end_period: Period = None
+    ) -> "IsSpatioTemporalDataSet[FeaturesT]": ...
+
+    def start_time(self) -> Period: ...
+
+    def end_time(self) -> Period: ...
+
+    def locations(self) -> Iterable[SpatialIndexType]: ...
+
+    def data(self) -> Iterable[FeaturesT]: ...
+
+    def location_items(
+        self,
+    ) -> Iterable[Tuple[SpatialIndexType, IsTemporalDataSet[FeaturesT]]]: ...
+
+    def to_tidy_dataframe(self) -> pd.DataFrame: ...
+
+    @classmethod
+    def from_tidy_dataframe(
+        cls, df: pd.DataFrame
+    ) -> "IsSpatioTemporalDataSet[FeaturesT]": ...
+
+    def to_csv(self, file_name: str): ...
+
+    @classmethod
+    def from_csv(self, file_name: str) -> "IsSpatioTemporalDataSet[FeaturesT]": ...

chap_core-0.0.8/chap_core/adaptors/gluonts.py

@@ -0,0 +1,50 @@
+from gluonts.model.estimator import Estimator
+from gluonts.dataset.common import ListDataset
+from gluonts.model.predictor import Predictor
+from ..data import DataSet
+from ..data.gluonts_adaptor.dataset import DataSetAdaptor
+from ..datatypes import Samples
+from ..time_period import PeriodRange
+from pathlib import Path
+from dataclasses import dataclass
+
+
+@dataclass
+class GluonTSPredictor:
+    gluonts_predictor: Predictor
+
+    def predict(
+        self, history: DataSet, future_data: DataSet, num_samples=100
+    ) -> DataSet:
+        gluonts_dataset = DataSetAdaptor.to_gluonts_testinstances(
+            history, future_data, self.gluonts_predictor.prediction_length
+        )
+        forecasts = self.gluonts_predictor.predict(
+            gluonts_dataset, num_samples=num_samples
+        )
+        data = {
+            location: Samples(
+                PeriodRange.from_pandas(forecast.index), forecast.samples.T
+            )
+            for location, forecast in zip(history.keys(), forecasts)
+        }
+        return DataSet(data)
+
+    def save(self, filename: str):
+        filepath = Path(filename)
+        filepath.mkdir(exist_ok=True, parents=True)
+        self.gluonts_predictor.serialize(filepath)
+
+    @classmethod
+    def load(cls, filename: str):
+        return GluonTSPredictor(Predictor.deserialize(Path(filename)))
+
+
+@dataclass
+class GluonTSEstimator:
+    gluont_ts_estimator: Estimator
+
+    def train(self, dataset: DataSet) -> GluonTSPredictor:
+        gluonts_dataset = DataSetAdaptor.to_gluonts(dataset)
+        ds = ListDataset(gluonts_dataset, freq="m")
+        return GluonTSPredictor(self.gluont_ts_estimator.train(ds))

chap_core-0.0.8/chap_core/alarms.py

@@ -0,0 +1,18 @@
+from typing import Iterable
+
+import numpy as np
+from pydantic import BaseModel
+
+# Epidemioglical week: different from calendar week
+# https://www.cdc.gov/flu/weekly/overview.htm
+
+
+class OutbreakParameters(BaseModel):
+    endemic_factor: float
+    probability_threshold: float
+
+
+def outbreak_prediction(
+    parameters: OutbreakParameters, case_samples: Iterable[float]
+) -> bool:
+    return np.mean()

chap_core-0.0.8/chap_core/api.py

@@ -0,0 +1,264 @@
+import logging
+import json
+
+from .assessment.forecast import forecast as do_forecast
+import zipfile
+from pathlib import Path
+from typing import Optional, List
+
+import numpy as np
+
+from .assessment.dataset_splitting import train_test_split_with_weather
+from .datatypes import (
+    HealthData,
+    ClimateData,
+    HealthPopulationData,
+    SimpleClimateData,
+    FullData,
+)
+from .dhis2_interface.json_parsing import (
+    predictions_to_datavalue,
+    parse_disease_data,
+    json_to_pandas,
+    parse_population_data,
+)
+from .external.external_model import get_model_from_directory_or_github_url
+from .file_io.example_data_set import DataSetType, datasets
+from .geojson import geojson_to_graph, NeighbourGraph
+from .plotting.prediction_plot import plot_forecast_from_summaries
+from .predictor import get_model
+from .spatio_temporal_data.temporal_dataclass import DataSet
+import dataclasses
+
+from .time_period.date_util_wrapper import delta_month, Month
+
+from .transformations.covid_mask import mask_covid_data
+
+logger = logging.getLogger(__name__)
+
+
+class DummyControl:
+    def set_status(self, status):
+        pass
+
+    @property
+    def current_control(self):
+        return None
+
+
+@dataclasses.dataclass
+class AreaPolygons:
+    shape_file: str
+
+
+@dataclasses.dataclass
+class PredictionData:
+    area_polygons: AreaPolygons = None
+    health_data: DataSet[HealthData] = None
+    climate_data: DataSet[ClimateData] = None
+    population_data: DataSet[HealthPopulationData] = None
+    disease_id: Optional[str] = None
+    features: List[object] = None
+
+
+def extract_disease_name(health_data: dict) -> str:
+    return health_data["rows"][0][0]
+
+
+def read_zip_folder(zip_file_path: str) -> PredictionData:
+    # read zipfile, create PredictionData
+    print(zip_file_path)
+    ziparchive = zipfile.ZipFile(zip_file_path)
+    expected_files = {
+        "area_polygons": "orgUnits.geojson",
+        "disease": "disease.json",
+        "population": "population.json",
+        "temperature": "temperature.json",
+        "precipitation": "precipitation.json",
+    }
+    json_data = json.load(ziparchive.open(expected_files["disease"]))
+    name_mapping = {"time_period": 2, "disease_cases": 3, "location": 1}
+    disease = parse_disease_data(json_data, name_mapping=name_mapping)
+    disease_id = extract_disease_name(json_data)
+    temperature_json = json.load(ziparchive.open(expected_files["temperature"]))
+    name_mapping = {"time_period": 2, "mean_temperature": 3, "location": 1}
+    temperature = json_to_pandas(temperature_json, name_mapping)
+
+    precipitation_json = json.load(ziparchive.open(expected_files["temperature"]))
+    name_mapping = {"time_period": 2, "precipitation": 3, "location": 1}
+    precipitation = json_to_pandas(precipitation_json, name_mapping)
+
+    assert np.all(precipitation.time_period == temperature.time_period)
+    assert np.all(precipitation.location == temperature.location)
+
+    temperature["rainfall"] = precipitation["precipitation"]
+    temperature["rainfall"] = temperature["rainfall"].astype(float)
+    temperature["mean_temperature"] = temperature["mean_temperature"].astype(float)
+
+    features = json.load(ziparchive.open(expected_files["area_polygons"]))["features"]
+    climate = DataSet.from_pandas(temperature, dataclass=SimpleClimateData)
+
+    population_json = json.load(ziparchive.open(expected_files["population"]))
+    population = parse_population_data(population_json)
+    graph_file_name = ""
+    graph = NeighbourGraph.from_geojson_file(
+        ziparchive.open(expected_files["area_polygons"])
+    )
+    print(graph)
+    if False:
+        graph_file_name = Path(zip_file_path).with_suffix(".graph")
+        area_polygons_file = ziparchive.open(expected_files["area_polygons"])
+        geojson_to_graph(area_polygons_file, graph_file_name)
+    # geojson_to_shape(area_polygons_file, shape_file_name)
+
+    # geojson_to_shape(str(zip_file_path) + "!area_polygons", shape_file_name)
+
+    return PredictionData(
+        health_data=disease,
+        climate_data=climate,
+        population_data=population,
+        area_polygons=graph,
+        disease_id=disease_id,
+        features=features,
+    )
+
+    out_data = {}
+
+
+#    ...
+
+
+def dhis_zip_flow(
+    zip_file_path: str,
+    out_json: Optional[str] = None,
+    model_name=None,
+    n_months=4,
+    docker_filename: Optional[str] = None,
+) -> List[dict] | None:
+    data: PredictionData = read_zip_folder(zip_file_path)
+    json_body = train_on_prediction_data(data, model_name, n_months, docker_filename)
+    if out_json is not None:
+        with open(out_json, "w") as f:
+            json.dump(json_body, f)
+        return None
+    else:
+        return json_body
+
+
+def train_on_prediction_data(
+    data,
+    model_name=None,
+    n_months=4,
+    docker_filename=None,
+    model_path=None,
+    control=None,
+):
+    if control is None:
+        control = DummyControl()
+    control.set_status("Preprocessing")
+    if model_name == "external":
+        model = get_model_from_directory_or_github_url(model_path)
+    else:
+        model = get_model(model_name)()
+    start_timestamp = min(
+        data.health_data.start_timestamp, data.climate_data.start_timestamp
+    )
+    end_timestamp = max(data.health_data.end_timestamp, data.climate_data.end_timestamp)
+    new_dict = {}
+    for location in data.health_data.locations():
+        health = data.health_data.get_location(location).fill_to_range(
+            start_timestamp, end_timestamp
+        )
+        climate = data.climate_data.get_location(location).fill_to_range(
+            start_timestamp, end_timestamp
+        )
+        assert (
+            location in data.population_data
+        ), f"Location {location} not in population data: {data.population_data.keys()}"
+        population = data.population_data[location]
+        new_dict[location] = FullData.combine(health.data(), climate.data(), population)
+
+    climate_health_data = DataSet(new_dict)
+    prediction_start = Month(climate_health_data.end_timestamp) - n_months * delta_month
+    train_data, _, future_weather = train_test_split_with_weather(
+        climate_health_data, prediction_start
+    )
+    logger.info(f"Training model {model_name} on {len(train_data.items())} locations")
+    control.set_status("Training")
+    if hasattr(model, "set_training_control"):
+        model.set_training_control(control.current_control)
+    if hasattr(model, "set_graph"):
+        model.set_graph(data.area_polygons)
+
+    model.train(train_data)  # , extra_args=data.area_polygons)
+    logger.info(
+        f"Forecasting using {model_name} on {len(train_data.items())} locations"
+    )
+    control.set_status("Forecasting")
+    predictions = model.forecast(future_weather, forecast_delta=n_months * delta_month)
+    attrs = ["median", "quantile_high", "quantile_low"]
+    logger.info("Converting predictions to json")
+    control.set_status("Postprocessing")
+    data_values = predictions_to_datavalue(
+        predictions, attribute_mapping=dict(zip(attrs, attrs))
+    )
+    json_body = [dataclasses.asdict(element) for element in data_values]
+    diseaseId = data.disease_id
+    return {"diseaseId": diseaseId, "dataValues": json_body}
+    # return json_body
+
+
+def train_with_validation(model_name, dataset_name, n_months=12):
+    dataset = datasets[dataset_name].load()
+    # assert not np.any(np.any(np.isnan(data.to_array()[:, 1:])) for data in dataset.values()), "Dataset contains NaN values"
+    # assert not any(np.any(np.isnan(data.mean_temperature) | np.isnan(data.rainfall)) for data in dataset.values()), "Dataset contains NaN values"
+    dataset = mask_covid_data(dataset)
+    model = get_model(model_name)(n_iter=32000)
+    # split_point = dataset.end_timestamp - n_months * delta_month
+    # train_data, test_data, future_weather = train_test_split_with_weather(dataset, split_point)
+    prediction_length = n_months * delta_month
+    split_point = dataset.end_timestamp - prediction_length
+    split_period = Month(split_point.year, split_point.month)
+    train_data, test_set, future_weather = train_test_split_with_weather(
+        dataset, split_period
+    )
+    model.set_validation_data(test_set)
+    model.train(train_data)
+    predictions = model.forecast(
+        future_weather, forecast_delta=n_months * delta_month, n_samples=100
+    )
+    # plot predictions
+    figs = []
+    for location, prediction in predictions.items():
+        fig = plot_forecast_from_summaries(
+            prediction.data(), dataset.get_location(location).data()
+        )  # , lambda x: np.log(x+1))
+        figs.append(fig)
+    return figs
+
+
+def forecast(
+    model_name: str,
+    dataset_name: DataSetType,
+    n_months: int,
+    model_path: Optional[str] = None,
+):
+    logging.basicConfig(level=logging.INFO)
+    dataset = datasets[dataset_name].load()
+
+    if model_name == "external":
+        model = get_model_from_directory_or_github_url(model_path)
+    else:
+        model = get_model(model_name)
+        model = model()
+
+    # model = get_model(model_name)()
+    predictions = do_forecast(model, dataset, n_months * delta_month)
+
+    figs = []
+    for location, prediction in predictions.items():
+        fig = plot_forecast_from_summaries(
+            prediction.data(), dataset.get_location(location).data()
+        )  # , lambda x: np.log(x+1))
+        figs.append(fig)
+    return figs