openstef 3.4.16__tar.gz → 3.4.30__tar.gz

{openstef-3.4.16 → openstef-3.4.30}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: openstef
-Version: 3.4.16
+Version: 3.4.30
 Summary: Open short term energy forecaster
 Home-page: https://github.com/OpenSTEF/openstef
 Author: Alliander N.V
@@ -27,6 +27,7 @@ Requires-Dist: pandas~=2.2.0
 Requires-Dist: plotly~=5.18
 Requires-Dist: pvlib==0.9.4
 Requires-Dist: pydantic~=2.4
+Requires-Dist: pydantic-settings~=2.3
 Requires-Dist: pymsteams~=0.2.2
 Requires-Dist: scikit-learn~=1.3
 Requires-Dist: scipy~=1.10
@@ -40,9 +41,11 @@ SPDX-FileCopyrightText: 2017-2023 Contributors to the OpenSTEF project <korte.te
 SPDX-License-Identifier: MPL-2.0
 -->
 
-<!-- Github Actions badges -->
-[![Python Build](https://github.com/openstef/openstef/actions/workflows/python-build.yaml/badge.svg)](https://github.com/openstef/openstef/actions/workflows/python-build.yaml)
-[![REUSE Compliance Check](https://github.com/openstef/openstef/actions/workflows/reuse-compliance.yaml/badge.svg)](https://github.com/openstef/openstef/actions/workflows/reuse-compliance.yaml)
+# OpenSTEF
+<!-- Badges -->
+[![Downloads](https://static.pepy.tech/badge/openstef)](https://pepy.tech/project/openstef)
+[![Downloads](https://static.pepy.tech/badge/openstef/month)](https://pepy.tech/project/openstef)
+[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/5585/badge)](https://bestpractices.coreinfrastructure.org/projects/5585)
 <!-- SonarCloud badges -->
 [![Bugs](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=bugs)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
 [![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=code_smells)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
@@ -53,11 +56,6 @@ SPDX-License-Identifier: MPL-2.0
 [![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=security_rating)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
 [![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=sqale_index)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
 [![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=vulnerabilities)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
-[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/5585/badge)](https://bestpractices.coreinfrastructure.org/projects/5585)
-[![Downloads](https://static.pepy.tech/badge/openstef)](https://pepy.tech/project/openstef)
-[![Downloads](https://static.pepy.tech/badge/openstef/month)](https://pepy.tech/project/openstef)
-
-# OpenSTEF
 
 OpenSTEF is a Python package designed for generating short-term forecasts in the energy sector. The repository includes all the essential components required for machine learning pipelines that facilitate the forecasting process. To utilize the package, users are required to furnish their own data storage and retrieval interface.
 
@@ -65,26 +63,22 @@ OpenSTEF is a Python package designed for generating short-term forecasts in the
 - [OpenSTEF](#openstef)
 - [Table of contents](#table-of-contents)
 - [External information sources](#external-information-sources)
-- [Installation](#installation)
-  - [Install the openstef package](#install-the-openstef-package)
-    - [Remark regarding installation within a **conda environment on Windows**:](#remark-regarding-installation-within-a-conda-environment-on-windows)
+- [Installation](#install)
 - [Usage](#usage)
-  - [Reference Implementation](#reference-implementation)
-  - [Openstef-dbc - Database connector for openstef](#openstef-dbc---database-connector-for-openstef)
-  - [Example notebooks](#example-notebooks)
-  - [License](#license)
-  - [Licenses third-party libraries](#licenses-third-party-libraries)
-  - [Contributing](#contributing)
-  - [Contact](#contact)
+    - [Example notebooks](#example-notebooks)
+    - [Reference Implementation](#reference-implementation)
+    - [Database connector for OpenSTEF](#database-connector-for-openstef)
+- [License](license)
+- [Contributing](#contributing)
+- [Contact](#contact)
 
 # External information sources
 - [Documentation website](https://openstef.github.io/openstef/index.html);
 - [Python package](https://pypi.org/project/openstef/);
-- [Project website](https://www.lfenergy.org/projects/openstef/);
+- [Linux Foundation project page](https://www.lfenergy.org/projects/openstef/);
 - [Documentation on dashboard](https://raw.githack.com/OpenSTEF/.github/main/profile/html/openstef_dashboard_doc.html);
-- [Linux Foundation project page](https://openstef.github.io/openstef/index.html)
 - [Video about OpenSTEF](https://www.lfenergy.org/forecasting-to-create-a-more-resilient-optimized-grid/);
-- [Teams channel](https://teams.microsoft.com/l/team/19%3ac08a513650524fc988afb296cd0358cc%40thread.tacv2/conversations?groupId=bfcb763a-3a97-4938-81d7-b14512aa537d&tenantId=697f104b-d7cb-48c8-ac9f-bd87105bafdc) 
+- [Teams channel](https://teams.microsoft.com/l/team/19%3ac08a513650524fc988afb296cd0358cc%40thread.tacv2/conversations?groupId=bfcb763a-3a97-4938-81d7-b14512aa537d&tenantId=697f104b-d7cb-48c8-ac9f-bd87105bafdc)
 
 # Installation
 
@@ -104,33 +98,34 @@ For more information on this issue see the [readme of pywin32](https://github.co
 
 # Usage
 
-To run a task use:
-
-```shell
-python -m openstef task <task_name>
-```
+## Example notebooks
+To help you get started, a set of fundamental example notebooks has been created. You can access these offline examples [here](https://github.com/OpenSTEF/openstef-offline-example).
 
 ## Reference Implementation
 A complete implementation including databases, user interface, example data, etc. is available at: https://github.com/OpenSTEF/openstef-reference
 
 ![screenshot](https://user-images.githubusercontent.com/60883372/146760483-29af3ac7-62af-4f13-98c7-982a79c517d1.jpg)
 Screenshot of the operational dashboard showing the key functionality of OpenSTEF.
-Dashboard documentation can be found [here](https://github.com/OpenSTEF/.github/blob/main/profile/README.md).
+Dashboard documentation can be found [here](https://raw.githack.com/OpenSTEF/.github/main/profile/html/openstef_dashboard_doc.html).
 
-## Openstef-dbc - Database connector for openstef
-This repository provides an interface to OpenSTEF (reference) databases. The repository can be found [here](https://github.com/OpenSTEF/openstef-dbc).
+To run a task use:
 
-## Example notebooks 
-To help you get started, a set of fundamental example notebooks has been created. You can access these offline examples [here](https://github.com/OpenSTEF/openstef-offline-example).
+```shell
+python -m openstef task <task_name>
+```
+
+## Database connector for openstef
+This repository provides an interface to OpenSTEF (reference) databases. The repository can be found [here](https://github.com/OpenSTEF/openstef-dbc).
 
-## License
+# License
 This project is licensed under the Mozilla Public License, version 2.0 - see LICENSE for details.
 
 ## Licenses third-party libraries
 This project includes third-party libraries, which are licensed under their own respective Open-Source licenses. SPDX-License-Identifier headers are used to show which license is applicable. The concerning license files can be found in the LICENSES directory.
 
-## Contributing
+# Contributing
 Please read [CODE_OF_CONDUCT.md](https://github.com/OpenSTEF/.github/blob/main/CODE_OF_CONDUCT.md), [CONTRIBUTING.md](https://github.com/OpenSTEF/.github/blob/main/CONTRIBUTING.md) and [PROJECT_GOVERNANCE.md](https://github.com/OpenSTEF/.github/blob/main/PROJECT_GOVERNANCE.md) for details on the process for submitting pull requests to us.
 
-## Contact
+# Contact
 Please read [SUPPORT.md](https://github.com/OpenSTEF/.github/blob/main/SUPPORT.md) for how to connect and get into contact with the OpenSTEF project
+ 

{openstef-3.4.16 → openstef-3.4.30}/README.md

@@ -4,9 +4,11 @@ SPDX-FileCopyrightText: 2017-2023 Contributors to the OpenSTEF project <korte.te
 SPDX-License-Identifier: MPL-2.0
 -->
 
-<!-- Github Actions badges -->
-[![Python Build](https://github.com/openstef/openstef/actions/workflows/python-build.yaml/badge.svg)](https://github.com/openstef/openstef/actions/workflows/python-build.yaml)
-[![REUSE Compliance Check](https://github.com/openstef/openstef/actions/workflows/reuse-compliance.yaml/badge.svg)](https://github.com/openstef/openstef/actions/workflows/reuse-compliance.yaml)
+# OpenSTEF
+<!-- Badges -->
+[![Downloads](https://static.pepy.tech/badge/openstef)](https://pepy.tech/project/openstef)
+[![Downloads](https://static.pepy.tech/badge/openstef/month)](https://pepy.tech/project/openstef)
+[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/5585/badge)](https://bestpractices.coreinfrastructure.org/projects/5585)
 <!-- SonarCloud badges -->
 [![Bugs](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=bugs)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
 [![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=code_smells)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
@@ -17,11 +19,6 @@ SPDX-License-Identifier: MPL-2.0
 [![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=security_rating)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
 [![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=sqale_index)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
 [![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=OpenSTEF_openstef&metric=vulnerabilities)](https://sonarcloud.io/dashboard?id=OpenSTEF_openstef)
-[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/5585/badge)](https://bestpractices.coreinfrastructure.org/projects/5585)
-[![Downloads](https://static.pepy.tech/badge/openstef)](https://pepy.tech/project/openstef)
-[![Downloads](https://static.pepy.tech/badge/openstef/month)](https://pepy.tech/project/openstef)
-
-# OpenSTEF
 
 OpenSTEF is a Python package designed for generating short-term forecasts in the energy sector. The repository includes all the essential components required for machine learning pipelines that facilitate the forecasting process. To utilize the package, users are required to furnish their own data storage and retrieval interface.
 
@@ -29,26 +26,22 @@ OpenSTEF is a Python package designed for generating short-term forecasts in the
 - [OpenSTEF](#openstef)
 - [Table of contents](#table-of-contents)
 - [External information sources](#external-information-sources)
-- [Installation](#installation)
-  - [Install the openstef package](#install-the-openstef-package)
-    - [Remark regarding installation within a **conda environment on Windows**:](#remark-regarding-installation-within-a-conda-environment-on-windows)
+- [Installation](#install)
 - [Usage](#usage)
-  - [Reference Implementation](#reference-implementation)
-  - [Openstef-dbc - Database connector for openstef](#openstef-dbc---database-connector-for-openstef)
-  - [Example notebooks](#example-notebooks)
-  - [License](#license)
-  - [Licenses third-party libraries](#licenses-third-party-libraries)
-  - [Contributing](#contributing)
-  - [Contact](#contact)
+    - [Example notebooks](#example-notebooks)
+    - [Reference Implementation](#reference-implementation)
+    - [Database connector for OpenSTEF](#database-connector-for-openstef)
+- [License](license)
+- [Contributing](#contributing)
+- [Contact](#contact)
 
 # External information sources
 - [Documentation website](https://openstef.github.io/openstef/index.html);
 - [Python package](https://pypi.org/project/openstef/);
-- [Project website](https://www.lfenergy.org/projects/openstef/);
+- [Linux Foundation project page](https://www.lfenergy.org/projects/openstef/);
 - [Documentation on dashboard](https://raw.githack.com/OpenSTEF/.github/main/profile/html/openstef_dashboard_doc.html);
-- [Linux Foundation project page](https://openstef.github.io/openstef/index.html)
 - [Video about OpenSTEF](https://www.lfenergy.org/forecasting-to-create-a-more-resilient-optimized-grid/);
-- [Teams channel](https://teams.microsoft.com/l/team/19%3ac08a513650524fc988afb296cd0358cc%40thread.tacv2/conversations?groupId=bfcb763a-3a97-4938-81d7-b14512aa537d&tenantId=697f104b-d7cb-48c8-ac9f-bd87105bafdc) 
+- [Teams channel](https://teams.microsoft.com/l/team/19%3ac08a513650524fc988afb296cd0358cc%40thread.tacv2/conversations?groupId=bfcb763a-3a97-4938-81d7-b14512aa537d&tenantId=697f104b-d7cb-48c8-ac9f-bd87105bafdc)
 
 # Installation
 
@@ -68,33 +61,34 @@ For more information on this issue see the [readme of pywin32](https://github.co
 
 # Usage
 
-To run a task use:
-
-```shell
-python -m openstef task <task_name>
-```
+## Example notebooks
+To help you get started, a set of fundamental example notebooks has been created. You can access these offline examples [here](https://github.com/OpenSTEF/openstef-offline-example).
 
 ## Reference Implementation
 A complete implementation including databases, user interface, example data, etc. is available at: https://github.com/OpenSTEF/openstef-reference
 
 ![screenshot](https://user-images.githubusercontent.com/60883372/146760483-29af3ac7-62af-4f13-98c7-982a79c517d1.jpg)
 Screenshot of the operational dashboard showing the key functionality of OpenSTEF.
-Dashboard documentation can be found [here](https://github.com/OpenSTEF/.github/blob/main/profile/README.md).
+Dashboard documentation can be found [here](https://raw.githack.com/OpenSTEF/.github/main/profile/html/openstef_dashboard_doc.html).
 
-## Openstef-dbc - Database connector for openstef
-This repository provides an interface to OpenSTEF (reference) databases. The repository can be found [here](https://github.com/OpenSTEF/openstef-dbc).
+To run a task use:
 
-## Example notebooks 
-To help you get started, a set of fundamental example notebooks has been created. You can access these offline examples [here](https://github.com/OpenSTEF/openstef-offline-example).
+```shell
+python -m openstef task <task_name>
+```
+
+## Database connector for openstef
+This repository provides an interface to OpenSTEF (reference) databases. The repository can be found [here](https://github.com/OpenSTEF/openstef-dbc).
 
-## License
+# License
 This project is licensed under the Mozilla Public License, version 2.0 - see LICENSE for details.
 
 ## Licenses third-party libraries
 This project includes third-party libraries, which are licensed under their own respective Open-Source licenses. SPDX-License-Identifier headers are used to show which license is applicable. The concerning license files can be found in the LICENSES directory.
 
-## Contributing
+# Contributing
 Please read [CODE_OF_CONDUCT.md](https://github.com/OpenSTEF/.github/blob/main/CODE_OF_CONDUCT.md), [CONTRIBUTING.md](https://github.com/OpenSTEF/.github/blob/main/CONTRIBUTING.md) and [PROJECT_GOVERNANCE.md](https://github.com/OpenSTEF/.github/blob/main/PROJECT_GOVERNANCE.md) for details on the process for submitting pull requests to us.
 
-## Contact
+# Contact
 Please read [SUPPORT.md](https://github.com/OpenSTEF/.github/blob/main/SUPPORT.md) for how to connect and get into contact with the OpenSTEF project
+ 

openstef-3.4.30/openstef/app_settings.py

@@ -0,0 +1,19 @@
+# SPDX-FileCopyrightText: 2017-2023 Contributors to the OpenSTEF project <korte.termijn.prognoses@alliander.com> # noqa E501>
+#
+# SPDX-License-Identifier: MPL-2.0
+
+from pydantic import Field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class AppSettings(BaseSettings):
+    """Global app settings."""
+
+    model_config = SettingsConfigDict(
+        env_prefix="openstef_", env_file=".env", extra="ignore"
+    )
+
+    post_teams_messages: bool = True
+
+    # Logging settings.
+    log_level: str = Field("INFO", description="Log level used for logging statements.")

{openstef-3.4.16 → openstef-3.4.30}/openstef/data_classes/prediction_job.py

@@ -25,10 +25,15 @@ class PredictionJobDataClass(BaseModel):
         - ``"xgb_quantile"``
         - ``"lgb"``
         - ``"linear"``
+        - ``"linear_quantile"``
+        - ``"xgb_multioutput_quantile"``
+        - ``"flatliner"``
 
     If unsure what to pick, choose ``"xgb"``.
 
     """
+    model_kwargs: Optional[dict]
+    """The model parameters that should be used."""
     forecast_type: str
     """The type of forecasts that should be made.
 

{openstef-3.4.16 → openstef-3.4.30}/openstef/enums.py

@@ -8,9 +8,12 @@ from enum import Enum
 class MLModelType(Enum):
     XGB = "xgb"
     XGB_QUANTILE = "xgb_quantile"
+    XGB_MULTIOUTPUT_QUANTILE = "xgb_multioutput_quantile"
     LGB = "lgb"
     LINEAR = "linear"
+    LINEAR_QUANTILE = "linear_quantile"
     ARIMA = "arima"
+    FLATLINER = "flatliner"
 
 
 class ForecastType(Enum):

{openstef-3.4.16 → openstef-3.4.30}/openstef/exceptions.py

@@ -5,7 +5,7 @@
 """Openstef custom exceptions."""
 
 
-# Define custom exception
+# Define custom exceptions
 class NoPredictedLoadError(Exception):
     """No predicted load for given datatime range."""
 

{openstef-3.4.16 → openstef-3.4.30}/openstef/feature_engineering/data_preparation.py

@@ -1,6 +1,7 @@
 # SPDX-FileCopyrightText: 2017-2023 Alliander N.V. <korte.termijn.prognoses@alliander.com> # noqa E501>
 #
 # SPDX-License-Identifier: MPL-2.0
+import logging
 from abc import ABC, abstractmethod
 from datetime import timedelta
 from typing import Optional
@@ -20,6 +21,7 @@ from openstef.feature_engineering.general import (
 )
 from openstef.model.regressors.regressor import OpenstfRegressor
 from openstef.pipeline.utils import generate_forecast_datetime_range
+from openstef.settings import Settings
 
 
 class AbstractDataPreparation(ABC):
@@ -120,6 +122,11 @@ class ARDataPreparation(AbstractDataPreparation):
     def prepare_forecast_data(
         self, data: pd.DataFrame
     ) -> tuple[pd.DataFrame, pd.DataFrame]:
+        structlog.configure(
+            wrapper_class=structlog.make_filtering_bound_logger(
+                logging.getLevelName(Settings.log_level)
+            )
+        )
         logger = structlog.get_logger(__name__)
         self.check_model()
         # Prep forecast input by selecting only the forecast datetime interval (this is much smaller than the input range)

{openstef-3.4.16 → openstef-3.4.30}/openstef/feature_engineering/general.py

@@ -3,10 +3,14 @@
 # SPDX-License-Identifier: MPL-2.0
 """This modelu contains various helper functions."""
 
+import logging
+
 import numpy as np
 import pandas as pd
 import structlog
 
+from openstef.settings import Settings
+
 
 def add_missing_feature_columns(
     input_data: pd.DataFrame, features: list[str]
@@ -30,6 +34,11 @@ def add_missing_feature_columns(
         Input dataframe with missing columns filled with ``np.N=nan``.
 
     """
+    structlog.configure(
+        wrapper_class=structlog.make_filtering_bound_logger(
+            logging.getLevelName(Settings.log_level)
+        )
+    )
     logger = structlog.get_logger(__name__)
 
     if features is None:
@@ -61,6 +70,11 @@ def remove_non_requested_feature_columns(
         Model input data with features.
 
     """
+    structlog.configure(
+        wrapper_class=structlog.make_filtering_bound_logger(
+            logging.getLevelName(Settings.log_level)
+        )
+    )
     logger = structlog.get_logger(__name__)
 
     if requested_features is None:

openstef-3.4.30/openstef/feature_engineering/missing_values_transformer.py

@@ -0,0 +1,99 @@
+# SPDX-FileCopyrightText: 2017-2023 Contributors to the OpenSTEF project <korte.termijn.prognoses@alliander.com> # noqa E501>
+#
+# SPDX-License-Identifier: MPL-2.0
+from typing import Union, List, Optional
+
+import numpy as np
+import pandas as pd
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import FunctionTransformer
+from sklearn.utils.validation import check_array
+
+
+class MissingValuesTransformer:
+    """MissingValuesTransformer handles missing values in data by imputing them with a given strategy.
+
+    It also removes columns that are always null from the data.
+
+    """
+
+    in_feature_names: Optional[List[str]] = None
+    _n_in_features: Optional[int] = None
+
+    non_null_feature_names: List[str] = None
+
+    def __init__(
+        self,
+        missing_values: Union[int, float, str, None] = np.nan,
+        imputation_strategy: str = None,
+        fill_value: Union[str, int, float] = None,
+    ):
+        """Initialize missing values handler.
+
+        Args:
+            missing_values: The placeholder for the missing values. All occurrences of
+                `missing_values` will be imputed.
+            imputation_strategy: The imputation strategy to use
+                Can be one of "mean", "median", "most_frequent", "constant" or None.
+            fill_value: When strategy == "constant", fill_value is used to replace all
+                occurrences of missing_values.
+
+        """
+        self.missing_values = missing_values
+        self.imputation_strategy = imputation_strategy
+        self.fill_value = fill_value
+
+    def fit(self, x, y=None):
+        """Fit the imputer on the input data."""
+        _ = check_array(x, force_all_finite="allow-nan")
+        if not isinstance(x, pd.DataFrame):
+            x = pd.DataFrame(np.asarray(x))
+
+        self.in_feature_names = list(x.columns)
+        self._n_in_features = x.shape[1]
+
+        # Remove always null columns
+        is_column_null = x.isnull().all(axis="index")
+        self.non_null_feature_names = list(x.columns[~is_column_null])
+
+        # Build the proper imputation transformer
+        # - Identity function if strategy is None
+        # - SimpleImputer with the dedicated strategy
+        if self.imputation_strategy is None:
+            self.imputer_ = FunctionTransformer(func=self._identity)
+        else:
+            self.imputer_ = SimpleImputer(
+                missing_values=self.missing_values,
+                strategy=self.imputation_strategy,
+                fill_value=self.fill_value,
+            ).set_output(transform="pandas")
+
+        # Imputers do not support labels
+        self.imputer_.fit(X=x, y=None)
+
+    def transform(self, x) -> pd.DataFrame:
+        """Transform the input data by imputing missing values."""
+        _ = check_array(x, force_all_finite="allow-nan")
+        if not isinstance(x, pd.DataFrame):
+            x = pd.DataFrame(np.asarray(x))
+
+        x = x[self.non_null_feature_names]
+
+        return self.imputer_.transform(x)
+
+    def fit_transform(self, x, y=None):
+        """Fit the imputer on the input data and transform it.
+
+        Returns:
+            The data with missing values imputed.
+
+        """
+        self.fit(x, y)
+        return self.transform(x)
+
+    @classmethod
+    def _identity(cls, x):
+        return x
+
+    def __sklearn_is_fitted__(self) -> bool:
+        return self.in_feature_names is not None

{openstef-3.4.16 → openstef-3.4.30}/openstef/feature_engineering/weather_features.py

@@ -3,6 +3,7 @@
 # SPDX-License-Identifier: MPL-2.0
 
 """This module contains all wheather related functions used for feature engineering."""
+import logging
 from typing import Union
 
 import numpy as np
@@ -12,7 +13,13 @@ import structlog
 from pvlib.location import Location
 
 from openstef.data_classes.prediction_job import PredictionJobDataClass
+from openstef.settings import Settings
 
+structlog.configure(
+    wrapper_class=structlog.make_filtering_bound_logger(
+        logging.getLevelName(Settings.log_level)
+    )
+)
 logger = structlog.get_logger(__name__)
 
 

{openstef-3.4.16 → openstef-3.4.30}/openstef/metrics/metrics.py

@@ -431,3 +431,54 @@ def xgb_quantile_obj(
     hess = np.ones_like(preds)
 
     return grad, hess
+
+
+def arctan_loss(y_true, y_pred, taus, s=0.1):
+    """Compute the arctan pinball loss.
+
+    Note that XGBoost outputs the predictions in a slightly peculiar manner.
+    Suppose we have 100 data points and we predict 10 quantiles. The predictions
+    will be an array of size (1000 x 1). We first resize this to a (100x10) array
+    where each row corresponds to the 10 predicted quantile for a single data
+    point. We then use a for-loop (over the 10 columns) to calculate the gradients
+    and second derivatives. Legibility was chosen over efficiency. This part
+    can be made more efficient.
+
+    Args:
+        y_true: An array containing the true observations.
+        y_pred: An array containing the predicted quantiles.
+        taus: A list containing the true desired coverage of the quantiles.
+        s: A smoothing parameter.
+
+    Returns:
+        grad: An array containing the (negative) gradients with respect to y_pred.
+        hess: An array containing the second derivative with respect to y_pred.
+
+    """
+    size = len(y_true)
+    n_dim = len(taus)  # The number of columns
+    n_rows = size // n_dim
+
+    # Resize the predictions and targets.
+    # Each column corresponds to a quantile, each row to a data point.
+    y_pred = np.reshape(y_pred, (n_rows, n_dim))
+    y_true = np.reshape(y_true, (n_rows, n_dim))
+
+    # Calculate the differences
+    u = y_true - y_pred
+
+    # Calculate the gradient and second derivatives
+    grad = np.zeros_like(y_pred)
+    hess = np.zeros_like(y_pred)
+    z = u / s
+    for i, tau in enumerate(taus):
+        x = 1 + z[:, i] ** 2
+        grad[:, i] = (
+            tau - 0.5 + 1 / np.pi * np.arctan(z[:, i]) + z[:, i] / (np.pi) * x**-1
+        )
+        hess[:, i] = 2 / (np.pi * s) * x ** (-2)
+
+    # Reshape back to the original shape.
+    grad = grad.reshape(size)
+    hess = hess.reshape(size)
+    return -grad / n_dim, hess / n_dim

{openstef-3.4.16 → openstef-3.4.30}/openstef/metrics/reporter.py

@@ -2,6 +2,7 @@
 #
 # SPDX-License-Identifier: MPL-2.0
 """Defines reporter class."""
+import logging
 import os
 import warnings
 from dataclasses import dataclass
@@ -16,6 +17,7 @@ from plotly.graph_objects import Figure
 from openstef.metrics import figure
 from openstef.metrics.metrics import bias, mae, nsme, r_mae, rmse
 from openstef.model.regressors.regressor import OpenstfRegressor
+from openstef.settings import Settings
 
 
 @dataclass
@@ -167,6 +169,11 @@ class Reporter:
     def write_report_to_disk(report: Report, report_folder: str):
         """Write report to disk; e.g. for viewing report of latest models using grafana."""
         # Initialize logger and serializer
+        structlog.configure(
+            wrapper_class=structlog.make_filtering_bound_logger(
+                logging.getLevelName(Settings.log_level)
+            )
+        )
         logger = structlog.get_logger(__name__)
         if report_folder:
             # create path if does not exist

{openstef-3.4.16 → openstef-3.4.30}/openstef/model/confidence_interval_applicator.py

@@ -1,6 +1,7 @@
 # SPDX-FileCopyrightText: 2017-2023 Contributors to the OpenSTEF project <korte.termijn.prognoses@alliander.com> # noqa E501>
 #
 # SPDX-License-Identifier: MPL-2.0
+import logging
 from datetime import datetime
 
 import numpy as np
@@ -11,12 +12,18 @@ from sklearn.base import RegressorMixin
 
 from openstef.data_classes.prediction_job import PredictionJobDataClass
 from openstef.exceptions import ModelWithoutStDev
+from openstef.settings import Settings
 
 
 class ConfidenceIntervalApplicator:
     def __init__(self, model: RegressorMixin, forecast_input_data: pd.DataFrame):
         self.model = model
         self.forecast_input_data = forecast_input_data
+        structlog.configure(
+            wrapper_class=structlog.make_filtering_bound_logger(
+                logging.getLevelName(Settings.log_level)
+            )
+        )
         self.logger = structlog.get_logger(self.__class__.__name__)
 
     def add_confidence_interval(
@@ -54,9 +61,24 @@ class ConfidenceIntervalApplicator:
         temp_forecast = self._add_standard_deviation_to_forecast(forecast)
 
         if self.model.can_predict_quantiles:
-            return self._add_quantiles_to_forecast_quantile_regression(
-                temp_forecast, pj["quantiles"]
-            )
+            # Try to generate the quantiles that were requested
+            try:
+                result = self._add_quantiles_to_forecast_quantile_regression(
+                    temp_forecast, pj["quantiles"]
+                )
+                return result
+            except Exception:
+                # Fallback on quantiles of the model if the requested quantiles cant be generated by the model.
+                # Can happen when the model was trained on different quantiles than are requested
+                result = self._add_quantiles_to_forecast_quantile_regression(
+                    temp_forecast, self.model.quantiles
+                )
+                self.logger.warning(
+                    "Quantiles are requested the model was not trained on. Using the quantiles the model was trained on",
+                    requested_quantiles=pj["quantiles"],
+                    trained_quantiles=self.model.quantiles,
+                )
+                return result
 
         return self._add_quantiles_to_forecast_default(temp_forecast, pj["quantiles"])