copro 0.1.2__tar.gz → 2.0.2__tar.gz

{copro-0.1.2 → copro-2.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: copro
-Version: 0.1.2
+Version: 2.0.2
 Summary: Python-model build on scikit-learn functions, designed to facilitate the set-up, execution, and evaluation of machine-learning models for the study of the climate-conflict nexus.
 Home-page: https://copro.readthedocs.io/
 Author: Jannis M. Hoch
@@ -16,6 +16,27 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Requires-Python: >=3.9
 License-File: LICENSE
+Requires-Dist: click==8.1.7
+Requires-Dist: xarray==2023.12.0
+Requires-Dist: pandas==2.1.4
+Requires-Dist: rasterio==1.2.10
+Requires-Dist: rioxarray==0.15.0
+Requires-Dist: rasterstats==0.19.0
+Requires-Dist: geopandas==1.1.2
+Requires-Dist: numpy==1.26.3
+Requires-Dist: scikit-learn==1.5.0
+Requires-Dist: netcdf4==1.6.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 ===============
 CoPro
@@ -23,9 +44,6 @@ CoPro
 
 Welcome to CoPro, a machine-learning tool for conflict risk projections based on climate, environmental, and societal drivers.
 
-.. image:: https://travis-ci.com/JannisHoch/copro.svg?branch=dev
-    :target: https://travis-ci.com/JannisHoch/copro
-
 .. image:: https://img.shields.io/badge/License-MIT-blue.svg
     :target: https://github.com/JannisHoch/copro/blob/dev/LICENSE
 
@@ -33,10 +51,10 @@ Welcome to CoPro, a machine-learning tool for conflict risk projections based on
     :target: https://copro.readthedocs.io/en/latest/?badge=latest
 
 .. image:: https://img.shields.io/github/v/release/JannisHoch/copro
-    :target: https://github.com/JannisHoch/copro/releases/tag/v0.0.8
+    :target: https://github.com/JannisHoch/copro/releases/tag/v2.0.1
 
 .. image:: https://zenodo.org/badge/254407279.svg
-    :target: https://zenodo.org/badge/latestdoi/254407279
+    :target: https://doi.org/10.5281/zenodo.4061705
 
 .. image:: https://badges.frapsoft.com/os/v2/open-source.svg?v=103
     :target: https://github.com/ellerbrock/open-source-badges/
@@ -44,8 +62,8 @@ Welcome to CoPro, a machine-learning tool for conflict risk projections based on
 .. image:: https://joss.theoj.org/papers/1f03334e56413ff71f65092ecc609aa4/status.svg
     :target: https://joss.theoj.org/papers/1f03334e56413ff71f65092ecc609aa4
 
-.. image:: https://mybinder.org/badge_logo.svg
-    :target: https://mybinder.org/v2/gh/JannisHoch/copro/dev?filepath=%2Fexample%2Fnb_binder.ipynb
+.. .. image:: https://mybinder.org/badge_logo.svg
+..     :target: https://mybinder.org/v2/gh/JannisHoch/copro/dev?filepath=%2Fexample%2Fnb_binder.ipynb
 
 Model purpose
 --------------
@@ -143,6 +161,11 @@ With this (or other) data, the provided configuration-files (cfg-files) can be u
 All output is stored in the output directory specified in the cfg-files. 
 In the output directory, two folders are created: one name `_REF` for output from the reference run, and `_PROJ` for output for projections.
 
+.. important::
+
+    The example data works with copro version < 2.0.0. 
+    For the latest version, the data needs to be realigned with the new config-file requirements.
+
 Jupyter notebooks
 ^^^^^^^^^^^^^^^^^^
 
@@ -209,8 +232,6 @@ CoPro is released under the MIT license.
 Authors
 ----------------
 
-* Jannis M. Hoch (Utrecht University)
-* Sophie de Bruin (Utrecht University, PBL)
+* Jannis M. Hoch (Utrecht University, now Fathom)
+* Sophie de Bruin (Utrecht University & PBL)
 * Niko Wanders (Utrecht University)
-
-Corresponding author: Jannis M. Hoch (j.m.hoch@uu.nl)

{copro-0.1.2 → copro-2.0.2}/README.rst

@@ -4,9 +4,6 @@ CoPro
 
 Welcome to CoPro, a machine-learning tool for conflict risk projections based on climate, environmental, and societal drivers.
 
-.. image:: https://travis-ci.com/JannisHoch/copro.svg?branch=dev
-    :target: https://travis-ci.com/JannisHoch/copro
-
 .. image:: https://img.shields.io/badge/License-MIT-blue.svg
     :target: https://github.com/JannisHoch/copro/blob/dev/LICENSE
 
@@ -14,10 +11,10 @@ Welcome to CoPro, a machine-learning tool for conflict risk projections based on
     :target: https://copro.readthedocs.io/en/latest/?badge=latest
 
 .. image:: https://img.shields.io/github/v/release/JannisHoch/copro
-    :target: https://github.com/JannisHoch/copro/releases/tag/v0.0.8
+    :target: https://github.com/JannisHoch/copro/releases/tag/v2.0.1
 
 .. image:: https://zenodo.org/badge/254407279.svg
-    :target: https://zenodo.org/badge/latestdoi/254407279
+    :target: https://doi.org/10.5281/zenodo.4061705
 
 .. image:: https://badges.frapsoft.com/os/v2/open-source.svg?v=103
     :target: https://github.com/ellerbrock/open-source-badges/
@@ -25,8 +22,8 @@ Welcome to CoPro, a machine-learning tool for conflict risk projections based on
 .. image:: https://joss.theoj.org/papers/1f03334e56413ff71f65092ecc609aa4/status.svg
     :target: https://joss.theoj.org/papers/1f03334e56413ff71f65092ecc609aa4
 
-.. image:: https://mybinder.org/badge_logo.svg
-    :target: https://mybinder.org/v2/gh/JannisHoch/copro/dev?filepath=%2Fexample%2Fnb_binder.ipynb
+.. .. image:: https://mybinder.org/badge_logo.svg
+..     :target: https://mybinder.org/v2/gh/JannisHoch/copro/dev?filepath=%2Fexample%2Fnb_binder.ipynb
 
 Model purpose
 --------------
@@ -124,6 +121,11 @@ With this (or other) data, the provided configuration-files (cfg-files) can be u
 All output is stored in the output directory specified in the cfg-files. 
 In the output directory, two folders are created: one name `_REF` for output from the reference run, and `_PROJ` for output for projections.
 
+.. important::
+
+    The example data works with copro version < 2.0.0. 
+    For the latest version, the data needs to be realigned with the new config-file requirements.
+
 Jupyter notebooks
 ^^^^^^^^^^^^^^^^^^
 
@@ -190,8 +192,6 @@ CoPro is released under the MIT license.
 Authors
 ----------------
 
-* Jannis M. Hoch (Utrecht University)
-* Sophie de Bruin (Utrecht University, PBL)
+* Jannis M. Hoch (Utrecht University, now Fathom)
+* Sophie de Bruin (Utrecht University & PBL)
 * Niko Wanders (Utrecht University)
-
-Corresponding author: Jannis M. Hoch (j.m.hoch@uu.nl)

copro-2.0.2/copro/__init__.py

@@ -0,0 +1,4 @@
+"""Top-level package."""
+
+__author__ = """Jannis M. Hoch, Sophie de Bruin, Niko Wanders"""
+__version__ = "2.0.2"

copro-2.0.2/copro/conflict.py

@@ -0,0 +1,308 @@
+import os
+import warnings
+from configparser import RawConfigParser
+from pathlib import Path
+from typing import Literal, Union
+
+import click
+import geopandas as gpd
+import numpy as np
+import pandas as pd
+
+from copro import nb, utils
+
+
+def conflict_in_year_bool(
+    config: RawConfigParser,
+    conflict_gdf: gpd.GeoDataFrame,
+    extent_gdf: gpd.GeoDataFrame,
+    sim_year: int,
+    out_dir: click.Path,
+    poly_identifier="watprovID",
+    conflict_identifier="event_id_cnty",
+) -> list:
+    """Creates a list for each timestep with boolean information whether a conflict took place in a polygon or not.
+
+    Args:
+        config (ConfigParser-object): object containing the parsed configuration-settings of the model.
+        conflict_gdf (geodataframe): geo-dataframe containing georeferenced information of conflict.
+        extent_gdf (geodataframe): geo-dataframe containing one or more polygons \
+            with geometry information for which values are extracted.
+        sim_year (int): year for which data is extracted.
+        out_dir (str): path to output folder. If 'None', no output is stored.
+        identifier (str, optional): unique polygon identifier column name. Defaults to 'watprovID'.
+
+    Returns:
+        list: list containing 0/1 per polygon depending on conflict occurence.
+    """
+
+    click.echo(f"Checking for conflict events which occured in {sim_year}.")
+    # select the entries which occured in this year
+    temp_sel_year = conflict_gdf.loc[conflict_gdf.year == sim_year]
+    if temp_sel_year.empty:
+        warnings.warn(
+            f"No conflicts were found in sampled conflict data set for year {sim_year}."
+        )
+
+    # merge the dataframes with polygons and conflict information, creating a sub-set of polygons/regions
+    data_merged = gpd.sjoin(temp_sel_year, extent_gdf)
+
+    # determine the aggregated amount of fatalities in one region (e.g. water province)
+    fatalities_per_poly = (
+        data_merged[conflict_identifier]
+        .groupby(data_merged[poly_identifier])
+        .count()
+        .to_frame()
+        .rename(columns={conflict_identifier: "total_fatalities"})
+    )
+
+    out_dir = os.path.join(out_dir, "files")
+    Path.mkdir(Path(out_dir), exist_ok=True)
+
+    if sim_year == config["general"]["y_end"]:
+        _store_boolean_conflict_data_to_csv(
+            fatalities_per_poly, extent_gdf, sim_year, out_dir
+        )
+
+    # loop through all regions and check if exists in sub-set
+    # if so, this means that there was conflict and thus assign value 1
+    list_out = []
+    for i, _ in extent_gdf.iterrows():
+        i_poly = extent_gdf.iloc[i][poly_identifier]
+        if i_poly in fatalities_per_poly.index.values:
+            list_out.append(1)
+        else:
+            list_out.append(0)
+
+    return list_out
+
+
+def conflict_in_previous_year_bool(
+    conflict_gdf: gpd.GeoDataFrame,
+    extent_gdf: gpd.GeoDataFrame,
+    sim_year: int,
+    check_neighbors: bool = False,
+    neighboring_matrix: Union[None, pd.DataFrame] = None,
+    poly_identifier="watprovID",  # TODO: no kwarg, should come from config
+    conflict_identifier="event_id_cnty",  # TODO: no kwarg, should come from config
+) -> list:
+    """Creates a list for each timestep with boolean information whether 
+    a conflict took place in the previous year in a polygon or not.
+
+    Args:
+        conflict_gdf (gpd.GeoDataFrame): georeferenced information of conflict.
+        extent_gdf (gpd.GeoDataFrame):  geo-dataframe containing one or more polygons \
+            with geometry information for which values are extracted.
+        sim_year (int): year for which data is extracted.
+        check_neighbors (bool, optional): whether to check for conflict in neighbouring polygons. Defaults to False.
+        neighboring_matrix (Union[None, pd.DataFrame], optional): dataframe with neighbouring polygons per polygon. \
+            Defaults to None.
+        identifier (str, optional): unique polygon identifier column name. Defaults to 'watprovID'.
+
+    Returns:
+        list: list containing 0/1 per polygon depending on conflict occurence.
+    """
+
+    if check_neighbors:
+        click.echo("Checking for conflicts in neighboring polygons at t-1")
+    else:
+        click.echo("Checking for conflict event in polygon at t-1")
+
+    # TODO: screening whether there is any conflict data in sim_year should be done earlier
+    # get conflicts at t-1
+    temp_sel_year = conflict_gdf[conflict_gdf.year == sim_year - 1]
+    if temp_sel_year.empty:
+        warnings.warn(
+            f"No conflicts were found in sampled conflict data set for year {sim_year - 1}."
+        )
+    # merge the dataframes with polygons and conflict information, creating a sub-set of polygons/regions
+    data_merged = gpd.sjoin(temp_sel_year, extent_gdf)
+    conflicts_per_poly = (
+        data_merged[conflict_identifier]
+        .groupby(data_merged[poly_identifier])
+        .count()
+        .to_frame()
+        .rename(columns={conflict_identifier: "conflict_count"})
+    )
+    # NOTE: WORKS UNTIL HERE
+
+    # loop through all polygons
+    list_out = []
+    for i in range(len(extent_gdf)):
+        i_poly = extent_gdf[poly_identifier].iloc[i]
+        # check if polygon is in list with conflict polygons
+        if i_poly in conflicts_per_poly.index.values:
+            # if so, check if neighboring polygons contain conflict and assign boolean value
+            if check_neighbors:
+                val = calc_conflicts_nb(i_poly, neighboring_matrix, conflicts_per_poly)
+                # append resulting value
+                list_out.append(val)
+            # if not, assign 1 directly
+            else:
+                list_out.append(1)
+        else:
+            # if polygon not in list with conflict polygons, assign 0
+            list_out.append(0)
+
+    return list_out
+
+
+def read_projected_conflict(
+    extent_gdf: gpd.GeoDataFrame,
+    bool_conflict: pd.DataFrame,
+    check_neighbors=False,
+    neighboring_matrix=None,
+    identifier="watprovID",
+) -> list:
+    """Creates a list for each timestep with boolean information 
+    whether a conflict took place in a polygon or not.
+    Input conflict data (`bool_conflict`) must contain an index with IDs 
+    corresponding with the `watprovID` values of extent_gdf.
+    Optionally, the algorithm can be extended to the neighboring polygons.
+
+    Args:
+        extent_gdf (gpd.GeoDataFrame): geo-dataframe containing one or more polygons \
+            with geometry information for which values are extracted.
+        bool_conflict (pd.DataFrame): dataframe with boolean values (1) for each polygon with conflict.
+        check_neighbors (bool, optional): whether or not to check for conflict in neighboring polygons. \
+            Defaults to `False`.
+        neighboring_matrix (pd.DataFrame, optional): look-up dataframe listing all neighboring polygons. \
+            Defaults to `None`.
+        identifier (str, optional): unique polygon identifier column name. Defaults to 'watprovID'.
+
+    Returns:
+        list: 1 and 0 values for each polygon with conflict respectively without conflict. \
+            If `check_neighbors=True`, then 1 if neighboring polygon contains conflict and 0 is not.
+    """
+
+    # loop through all polygons and check if exists in sub-set
+    list_out = []
+    for i in range(len(extent_gdf)):
+        i_poly = extent_gdf[identifier].iloc[i]
+        if i_poly in bool_conflict.index.values:
+            if check_neighbors:
+                # determine log-scaled number of conflict events in neighboring polygons
+                val = calc_conflicts_nb(i_poly, neighboring_matrix, bool_conflict)
+                # append resulting value
+                list_out.append(val)
+            else:
+                list_out.append(1)
+        else:
+            # if polygon not in list with conflict polygons, assign 0
+            list_out.append(0)
+
+    return list_out
+
+
+def calc_conflicts_nb(
+    i_poly: int, neighboring_matrix: pd.DataFrame, conflicts_per_poly: pd.DataFrame
+) -> Literal[0, 1]:
+    """Determines whether in the neighbouring polygons of a polygon i_poly conflict took place.
+    If so, a value 1 is returned, otherwise 0.
+
+    Args:
+        i_poly (int): ID number of polygon under consideration.
+        neighboring_matrix (pd.DataFrame): look-up dataframe listing all neighboring polygons.
+        conflicts_per_poly (pd.DataFrame): dataframe with conflict data per polygon.
+
+    Returns:
+        Literal: 1 if conflict took place in neighboring polygon, 0 if not.
+    """
+
+    # find neighbors of this polygon
+    nbs = nb.find_neighbors(i_poly, neighboring_matrix)
+
+    # initiate list
+    nb_count = []
+    # loop through neighbors
+    for k in nbs:
+        # check if there was conflict at t-1
+        if k in conflicts_per_poly.index.values:
+            nb_count.append(1)
+    # if more than one neighboring polygon has conflict, return 0
+    if np.sum(nb_count) > 0:
+        val = 1
+    # otherwise, return 0
+    else:
+        val = 0
+
+    return val
+
+
+def check_for_correct_prediction(
+    X_test_ID: np.ndarray,
+    X_test_geom: np.ndarray,
+    y_test: np.ndarray,
+    y_pred: np.ndarray,
+    y_prob_0: np.ndarray,
+    y_prob_1: np.ndarray,
+) -> pd.DataFrame:
+    """Stacks together the arrays with unique identifier, geometry, test data, and predicted data into a dataframe.
+    Contains therefore only the data points used in the test-sample, not in the training-sample.
+    Additionally computes whether a correct prediction was made.
+
+    Args:
+        X_test_ID (np.ndarray): _description_
+        X_test_geom (np.ndarray): _description_
+        y_test (np.ndarray): _description_
+        y_pred (np.ndarray): _description_
+        y_prob_0 (np.ndarray): _description_
+        y_prob_1 (np.ndarray): _description_
+
+    Returns:
+        pd.DataFrame: _description_
+    """
+
+    # stack separate columns horizontally
+    arr = np.column_stack((X_test_ID, X_test_geom, y_test, y_pred, y_prob_0, y_prob_1))
+    # convert array to dataframe
+    df = pd.DataFrame(
+        arr, columns=["ID", "geometry", "y_test", "y_pred", "y_prob_0", "y_prob_1"]
+    )
+    # compute whether a prediction is correct
+    # if so, assign 1; otherwise, assign 0
+    df["correct_pred"] = np.where(df["y_test"] == df["y_pred"], 1, 0)
+
+    return df
+
+
+def _store_boolean_conflict_data_to_csv(
+    fatalities_per_poly: pd.DataFrame,
+    extent_gdf: gpd.GeoDataFrame,
+    sim_year: int,
+    out_dir: click.Path,
+    identifier="watprovID",
+):
+    """Stores boolean conflict data to csv-file at the end of reference period.
+    Used as initial conditions for projections from there.
+
+    Args:
+        fatalities_per_poly (pd.DataFrame): Fatalities per polygon in `sim_year`.
+        extent_gdf (gpd.GeoDataFrame): All polygons considered in analysis, also those w/o conflict.
+        sim_year (int): Simulation year for which data is stored.
+        out_dir (click.Path): Path to output folder.
+        identifier (str, optional): Unique polygon identifier column name. Defaults to 'watprovID'.
+    """
+
+    # get a 1 for each polygon where there was conflict
+    bool_per_poly = fatalities_per_poly / fatalities_per_poly
+    # change column name and dtype
+    bool_per_poly = bool_per_poly.rename(
+        columns={"total_fatalities": "bool_conflict"}
+    ).astype(int)
+    # change index name to fit global_df
+    bool_per_poly.index = bool_per_poly.index.rename("ID")
+    # get list of all polygon IDs with their geometry information
+    global_df = utils.get_ID_geometry_lookup(extent_gdf)
+    # merge the boolean info with geometry
+    # for all polygons without conflict, set a 0
+    click.echo(
+        f"Storing boolean conflict map of year {sim_year} \
+            to file {os.path.join(out_dir, f'conflicts_in_{sim_year}.csv')}"
+    )
+
+    data_stored = pd.merge(bool_per_poly, global_df, on="ID", how="right").dropna()
+    data_stored.index = data_stored.index.rename(identifier)
+    data_stored = data_stored.drop("geometry", axis=1)
+    data_stored = data_stored.astype(int)
+    data_stored.to_csv(os.path.join(out_dir, f"conflicts_in_{sim_year}.csv"))

copro-2.0.2/copro/evaluation.py

@@ -0,0 +1,151 @@
+from typing import Union
+
+import geopandas as gpd
+import pandas as pd
+from sklearn import metrics
+
+
+def init_out_dict(scores: Union[list[str], None] = None) -> dict:
+    """Initiates the main model evaluatoin dictionary for a range of model metric scores.
+    The scores should match the scores used in the dictioary created in 'evaluation.evaluate_prediction()'.
+
+    Args:
+        scores (list, None): list containing metric scores. If 'None', a default list is used. Defaults to 'None'.
+
+    Returns:
+        dict: empty dictionary with metrics as keys.
+    """
+
+    if scores is None:
+        scores = [
+            "Accuracy",
+            "Precision",
+            "Recall",
+            "F1 score",
+            "Cohen-Kappa score",
+            "Brier loss score",
+            "ROC AUC score",
+            "AP score",
+        ]
+
+    # initialize empty dictionary with one emtpy list per score
+    out_dict = {}
+    for score in scores:
+        out_dict[score] = []
+
+    return out_dict
+
+
+def fill_out_dict(
+    out_dict: dict,
+    eval_dict: dict,
+):
+    """Appends the computed metric score per run to the main output dictionary.
+    All metrics are initialized in init_out_dict().
+
+    Args:
+        out_dict (dict): main output dictionary.
+        eval_dict (dict): dictionary containing scores per simulation.
+
+    Returns:
+        dict: dictionary with collected scores for each simulation
+    """
+
+    for key in out_dict:
+        out_dict[key].append(eval_dict[key])
+
+    return out_dict
+
+
+def polygon_model_accuracy(
+    df: pd.DataFrame, global_df: pd.DataFrame, make_proj=False
+) -> gpd.GeoDataFrame:
+    """Determines a range of model accuracy values for each polygon.
+    Reduces dataframe with results from each simulation to values per unique polygon identifier.
+    Determines the total number of predictions made per polygon 
+    as well as fraction of correct predictions made for overall and conflict-only data.
+
+    Args:
+        df (pd.DataFrame): output dataframe containing results of all simulations.
+        global_df (pd.DataFrame): global look-up dataframe to associate unique identifier with geometry.
+        make_proj (bool, optional): whether or not this function is used to make a projection. \
+            If True, a couple of calculations are skipped as no observed data is available for projections. \
+                Defaults to 'False'.
+
+    Returns:
+        gpd.GeoDataFrame: model accuracy data per polygon.
+    """
+
+    # - create a dataframe containing the number of occurence per ID
+    ID_count = (
+        df.ID.value_counts().to_frame().rename(columns={"count": "nr_predictions"})
+    )
+
+    df_count = pd.DataFrame()
+
+    # - per polygon ID, compute sum of overall correct predictions and rename column name
+    if not make_proj:
+        df_count["nr_correct_predictions"] = df.correct_pred.groupby(df.ID).sum()
+
+    # - per polygon ID, compute sum of all conflict data points and add to dataframe
+    if not make_proj:
+        df_count["nr_observed_conflicts"] = df.y_test.groupby(df.ID).sum().astype(float)
+
+    # - per polygon ID, compute sum of all conflict data points and add to dataframe
+    df_count["nr_predicted_conflicts"] = df.y_pred.groupby(df.ID).sum().astype(float)
+    # - per polygon ID, compute average probability that conflict occurs
+    df_count["min_prob_1"] = pd.to_numeric(df.y_prob_1).groupby(df.ID).min()
+    df_count["probability_of_conflict"] = (
+        pd.to_numeric(df.y_prob_1).groupby(df.ID).mean()
+    )
+    df_count["max_prob_1"] = pd.to_numeric(df.y_prob_1).groupby(df.ID).max()
+
+    # - merge the two dataframes with ID as key
+    df_temp = pd.merge(ID_count, df_count, on="ID")
+
+    # - compute average correct prediction rate by dividing sum of correct predictions with number of all predicionts
+    if not make_proj:
+        df_temp["fraction_correct_predictions"] = (
+            df_temp.nr_correct_predictions / df_temp.nr_predictions
+        )
+
+    # - compute average correct prediction rate by dividing sum of correct predictions with number of all predicionts
+    df_temp["chance_of_conflict"] = (
+        df_temp.nr_predicted_conflicts / df_temp.nr_predictions
+    )
+
+    # - merge with global dataframe containing IDs and geometry, and keep only those polygons occuring in test sample
+    df_hit = pd.merge(df_temp, global_df, on="ID", how="left")
+    # #- convert to geodataframe
+    gdf_hit = gpd.GeoDataFrame(df_hit, geometry=df_hit.geometry)
+
+    return gdf_hit
+
+
+def evaluate_prediction(y_test: list, y_pred: list, y_prob: list) -> dict:
+    """Computes a range of model evaluation metrics and appends the resulting scores to a dictionary.
+    This is done for each model execution separately.
+    Output will be stored to stderr if possible.
+
+    Args:
+        y_test (list): list containing test-sample conflict data.
+        y_pred (list): list containing predictions.
+        y_prob (array): array resulting probabilties of predictions.
+
+    Returns:
+        dict: dictionary with scores for each simulation
+    """
+
+    # compute value per evaluation metric and assign to list
+    eval_dict = {
+        "Accuracy": metrics.accuracy_score(y_test, y_pred),
+        "Precision": metrics.precision_score(y_test, y_pred),
+        "Recall": metrics.recall_score(y_test, y_pred),
+        "F1 score": metrics.f1_score(y_test, y_pred),
+        "Cohen-Kappa score": metrics.cohen_kappa_score(y_test, y_pred),
+        "Brier loss score": metrics.brier_score_loss(y_test, y_prob[:, 1]),
+        "ROC AUC score": metrics.roc_auc_score(y_test, y_prob[:, 1]),
+        "AP score": metrics.average_precision_score(y_test, y_prob[:, 1]),
+    }
+
+    return eval_dict