rtc-tools-interface 0.8.0b4__tar.gz → 0.9.0a1__tar.gz

{rtc-tools-interface-0.8.0b4 → rtc-tools-interface-0.9.0a1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: rtc-tools-interface
-Version: 0.8.0b4
+Version: 0.9.0a1
 Summary: Toolbox for user interfaces for RTC-Tools
 Home-page: UNKNOWN
 Author: Deltares

{rtc-tools-interface-0.8.0b4 → rtc-tools-interface-0.9.0a1}/README.md

@@ -172,30 +172,72 @@ The table could thus look like:
 After running the model, in your output folder the folder `figures` containing the figures is created.
 
 ## Closed loop runner
-To run a closed loop experiment one can use the `run_optimization_problem_closed_loop` function from `run_closed_loop`.
-This function is a drop-in replacement for the `run_optimization_problem` of rtc-tools. The user only needs to specify the `closed_loop_dates.csv` in the input folder of the optimization problem.
+
+To run a closed loop experiment
+one can use the `run_optimization_problem_closed_loop` function
+from `rtctools_interface.closed_loop.runner`.
+This function is a drop-in replacement for the `run_optimization_problem` of rtc-tools.
+The user needs to specify a `ClosedLoopConfig` configuration
+from `rtctools_interface.closed_loop.config`
+to specify the time ranges for which to subsequentially solve the optimization problem.
 
 ### Setup
-Import `run_optimization_problem_closed_loop` with:
+
+Import `ClosedLoopConfig` and `run_optimization_problem_closed_loop` with:
+
+```python
+from rtctools_interface.closed_loop.config import ClosedLoopConfig
+from rtctools_interface.closed_loop.runner import run_optimization_problem_closed_loop
+```
+
+A `ClosedLoopConfig` configuration can be created from a csv file or
+from a given forecast timestep (time between each time range)
+and optimization period (duration of each time range).
+An option `round_to_dates` can be used to round the start and end time of each range to a date,
+i.e. the start time is rounded to the start of the day
+and the end time is rounded to the end of a day.
+Examples of creating a configuration are given below.
+
 ```python
-from rtctools_interface import run_optimization_problem_closed_loop
+from datetime import timedelta
+
+from rtctools_interface.closed_loop.config import ClosedLoopConfig
+
+config_from_file = ClosedLoopConfig(
+    file="path/to/closed_loop_dates.csv",
+    round_to_dates=True
+)
+
+config_from_fixed_periods = ClosedLoopConfig.from_fixed_periods(
+    optimization_period=timedelta(days=3),
+    forecast_timestep=timedelta(days=2)
+)
 ```
-Add a table named `closed_loop_dates.csv` to the input folder of your optimization problem. The table should contain two columns: `start_date` and `end_date`.
-Each row of the table corresponds to one modelling period. 
 
-Example table `closed_loop_dates.csv`:
+The CSV file `closed_loop_dates.csv` has two columns `start_date` and `end_date`
+and looks as follows.
+
 ```
 start_date, end_date
 2024-05-19, 2024-05-23
 2024-05-23, 2024-05-25
 ```
-With this table rtc-tools will run two optimization problems (modelling periods): one with the data from 2024-05-19 upto and including 2024-05-23 and one from 2024-05-23 upto and including 2024-05-25. 
-The `run_optimization_problem_closed_loop` will automatically set the final results from the previous as initial conditions of the next run. Note that this happens for:
-- All variables available at the first time step in original timeseries_import, but not available at any timestep in the modelling period.
+
+The `run_optimization_problem_closed_loop` will solve the optimization problem
+for each time range subseqentually.
+It will use the final results from the previous run to set the initial values of the next run.
+Note that this happens for:
+
+- All variables available at the first time step in original timeseries_import,
+    but not available at any timestep in the modelling period.
 - All variables in the `initial_state.csv` (if the csv_mixin is used).
 
 ### Notes
-- The first start_date in your `closed_loop_dates.csv` should be equal to the start_date of your timeseries_import.
-- The different horizons should overlap with at least one day (to allow retrieving and setting initial values). An overlap of more days is allowed.
-- Currently only a single timestep is copied as an initial value.
-- The closed_loop runner only works in combination with the CSVMixin or the PIMixin. The CDFMixin is not supported.
+
+- The start time of the first time range should coincide
+    with the start time of the input timeseries.
+- The start time of the next time range should be less or equal
+    to the end time of the current time range.
+- Currently, only the initial values of the first time step in a given time range are set.
+- The closed_loop runner only works in combination with the CSVMixin or the PIMixin.
+    The CDFMixin is not supported.

{rtc-tools-interface-0.8.0b4 → rtc-tools-interface-0.9.0a1}/rtc_tools_interface.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: rtc-tools-interface
-Version: 0.8.0b4
+Version: 0.9.0a1
 Summary: Toolbox for user interfaces for RTC-Tools
 Home-page: UNKNOWN
 Author: Deltares

{rtc-tools-interface-0.8.0b4 → rtc-tools-interface-0.9.0a1}/rtc_tools_interface.egg-info/SOURCES.txt

@@ -11,7 +11,8 @@ rtc_tools_interface.egg-info/top_level.txt
 rtctools_interface/__init__.py
 rtctools_interface/_version.py
 rtctools_interface/closed_loop/__init__.py
-rtctools_interface/closed_loop/closed_loop_dates.py
+rtctools_interface/closed_loop/config.py
+rtctools_interface/closed_loop/optimization_ranges.py
 rtctools_interface/closed_loop/results_construction.py
 rtctools_interface/closed_loop/runner.py
 rtctools_interface/closed_loop/time_series_handler.py
@@ -41,6 +42,7 @@ rtctools_interface/utils/serialization.py
 rtctools_interface/utils/type_definitions.py
 tests/__init__.py
 tests/closed_loop/__init__.py
+tests/closed_loop/test_optimization_ranges.py
 tests/closed_loop/test_read_xml.py
 tests/closed_loop/test_run_optization_problem_closed_loop.py
 tests/optimization/__init__.py

{rtc-tools-interface-0.8.0b4 → rtc-tools-interface-0.9.0a1}/rtctools_interface/_version.py

@@ -8,11 +8,11 @@ import json
 
 version_json = '''
 {
- "date": "2024-06-11T11:21:50+0200",
+ "date": "2024-06-18T15:06:29+0200",
  "dirty": false,
  "error": null,
- "full-revisionid": "6bc6e859ff0f928c2f4c1127a2aca08ce9720051",
- "version": "0.8.0b4"
+ "full-revisionid": "478b74b0fc61a477f80531ea4e3f54f1054bc603",
+ "version": "0.9.0a1"
 }
 '''  # END VERSION_JSON
 

rtc-tools-interface-0.9.0a1/rtctools_interface/closed_loop/config.py

@@ -0,0 +1,73 @@
+"""Module for configuring a closed-loop optimization problem."""
+from pathlib import Path
+from datetime import timedelta
+
+
+class ClosedLoopConfig():
+    """Configuration of a closed-loop optimization problem."""
+
+    def __init__(
+        self,
+        file: Path = None,
+        round_to_dates: bool = False,
+    ):
+        """
+        Create a configuration for closed-loop optimization.
+
+        :param file: CSV file with two columns 'start_date' and 'end_date'.
+            Each row indicates a time range for which to optimize a given problem.
+            Note:
+            * The start time of the first time range should coincide with the
+              start time of the input timeseries.
+            * The start time of the next time range should be less or equal
+              to the end time of the current time range.
+        :param round_to_dates: If true, the start time and end time of each time range
+            is rounded to just the date.
+            In particular, the start time is rounded to the start of the day
+            and the end time is rounded to the end of the day.
+        """
+        if file is not None:
+            file = Path(file).resolve()
+        self._file = file
+        self._forecast_timestep = None
+        self._optimization_period = None
+        self.round_to_dates = round_to_dates
+
+    @classmethod
+    def from_fixed_periods(
+        cls,
+        forecast_timestep: timedelta,
+        optimization_period: timedelta,
+        round_to_dates: bool = False,
+    ):
+        """
+        Create a closed loop configuration based on fixed periods.
+
+        :param forecast_timestep: The time between the start of each optimization time range.
+        :param optimization_period: the duration of each optimization time range
+        """
+        if forecast_timestep > optimization_period:
+            raise ValueError(
+                f"The forecast timestep ({forecast_timestep}) should be less than or equal to"
+                f" the optimization period ({optimization_period})."
+            )
+        config = cls()
+        config._forecast_timestep = forecast_timestep
+        config._optimization_period = optimization_period
+        config.round_to_dates = round_to_dates
+        return config
+
+    @property
+    def file(self):
+        """Get the file that defines the closed-loop periods."""
+        return self._file
+
+    @property
+    def forecast_timestep(self):
+        """Get the forecast timestep of the closed-loop periods."""
+        return self._forecast_timestep
+
+    @property
+    def optimization_period(self):
+        """Get the optimization period of the closed-loop periods."""
+        return self._optimization_period

rtc-tools-interface-0.9.0a1/rtctools_interface/closed_loop/optimization_ranges.py

@@ -0,0 +1,131 @@
+"""Module for calculating optimization periods."""
+import bisect
+import datetime
+from pathlib import Path
+
+import pandas as pd
+
+
+def get_optimization_ranges_from_file(
+    file_path: Path, model_time_range: tuple[datetime.datetime, datetime.datetime]
+):
+    """Read horizon config from a csv file"""
+    if not file_path.exists():
+        raise FileNotFoundError(
+            f"The closed_loop_dates csv does not exist. Please create a horizon config file in {file_path}."
+        )
+    try:
+        closed_loop_dates = pd.read_csv(file_path)
+    except pd.errors.EmptyDataError:
+        raise ValueError(
+            "The closed_loop_dates csv is empty. Please provide a valid file with start_date and end_date column."
+        )
+    closed_loop_dates.columns = closed_loop_dates.columns.str.replace(" ", "")
+    if not all([col in closed_loop_dates.columns for col in ["start_date", "end_date"]]):
+        raise ValueError("The closed_loop_dates csv should have both 'start_date' and 'end_date' columns.")
+    closed_loop_dates["start_date"] = pd.to_datetime(closed_loop_dates["start_date"])
+    closed_loop_dates["end_date"] = pd.to_datetime(closed_loop_dates["end_date"])
+    for i in range(1, len(closed_loop_dates)):
+        if closed_loop_dates["start_date"].iloc[i] > closed_loop_dates["end_date"].iloc[i - 1]:
+            raise ValueError(f"Closed loop date table: Start date at row {i} is later than the previous end date. ")
+    if any(closed_loop_dates["start_date"] < closed_loop_dates["start_date"].shift(1)):
+        raise ValueError("Closed loop date table: The start dates are not in ascending order.")
+    if any(closed_loop_dates["end_date"] < closed_loop_dates["end_date"].shift(1)):
+        raise ValueError("Closed loop date table: The end dates are not in ascending order.")
+    if any(closed_loop_dates["end_date"] < closed_loop_dates["start_date"]):
+        raise ValueError("Closed loop date table: For one or more rows the end date is before the start date.")
+    if any(closed_loop_dates["start_date"] > closed_loop_dates["end_date"]):
+        raise ValueError("Closed loop date table: For one or more rows the start date is after the end date.")
+    if (
+        any(closed_loop_dates["start_date"].dt.hour != 0)
+        or any(closed_loop_dates["start_date"].dt.minute != 0)
+        or any(closed_loop_dates["end_date"].dt.hour != 0)
+        or any(closed_loop_dates["end_date"].dt.minute != 0)
+    ):
+        raise ValueError(
+            "Closed loop date table: Currently, the date ranges can only be specific up to the level of days."
+        )
+    assert (
+        min(closed_loop_dates["start_date"]).date() == model_time_range[0].date()
+    ), (
+        "The start day of the first optimization run is not equal"
+        " to the start day of the timeseries import."
+    )
+    assert (
+        max(closed_loop_dates["end_date"]).date() <= model_time_range[1].date()
+    ), (
+        "The end date of one or more optimization runs is later"
+        " than the end date of the timeseries import."
+    )
+    closed_loop_dates = [
+        (optimization_range["start_date"], optimization_range["end_date"])
+        for _, optimization_range in closed_loop_dates.iterrows()
+    ]
+    return closed_loop_dates
+
+
+def _get_next_time_index(
+    times: list[datetime.date],
+    i_current: int,
+    timestep_size: datetime.timedelta
+) -> int:
+    """
+    Get the next timestep index.
+
+    The next timestep index i_next is the highest index such that:
+    * times[i_next] > times[i_current]
+    * times[i_next] <= times[i_current] + timestep_size
+    In case i_current >= i_max, i_next will be set to i_max, where i_max = len(times) - 1.
+    """
+    i_max = len(times) - 1
+    if i_current >= i_max:
+        return i_max
+    current_time = times[i_current]
+    next_time = current_time + timestep_size
+    i_next = bisect.bisect_right(times, next_time) - 1
+    i_next = max(i_current + 1, i_next)
+    return i_next
+
+
+def get_optimization_ranges(
+    model_times: list[datetime.date],
+    start_time: datetime.datetime,
+    forecast_timestep: datetime.timedelta,
+    optimization_period: datetime.timedelta
+) -> list[tuple[datetime.datetime, datetime.datetime]]:
+    """Calculate a list of optimization periods."""
+    if forecast_timestep > optimization_period:
+        raise ValueError(
+            f"Forecast timestep {forecast_timestep} cannot be larger than"
+            f" the optimization period {optimization_period}."
+        )
+    if start_time not in model_times:
+        raise ValueError(f"Start time {start_time} is not in the given model times.")
+    i_max = len(model_times) - 1
+    i_start = model_times.index(start_time)
+    i_end = _get_next_time_index(model_times, i_start, optimization_period)
+    optimization_periods = []
+    optimization_periods.append((model_times[i_start], model_times[i_end]))
+    while i_end < i_max:
+        i_start = _get_next_time_index(model_times, i_start, forecast_timestep)
+        i_end = _get_next_time_index(model_times, i_start, optimization_period)
+        optimization_periods.append((model_times[i_start], model_times[i_end]))
+    return optimization_periods
+
+
+def round_datetime_ranges_to_days(
+    datetime_ranges: list[tuple[datetime.datetime, datetime.datetime]]
+) -> list[tuple[datetime.datetime, datetime.datetime]]:
+    """Round datetimes to dats in datetime ranges.
+
+    The start of the range is rounded to the start of the day
+    and the end of the range is rounded to the end of the day.
+    """
+    rounded_ranges = []
+    for start, end in datetime_ranges:
+        if start.date() == end.date():
+            continue
+        start = datetime.datetime.combine(start.date(), datetime.time.min)
+        end = datetime.datetime.combine(end.date(), datetime.time.max)
+        rounded_ranges.append((start, end))
+    return rounded_ranges

{rtc-tools-interface-0.8.0b4 → rtc-tools-interface-0.9.0a1}/rtctools_interface/closed_loop/runner.py

@@ -9,7 +9,8 @@ from rtctools.data.pi import DiagHandler
 from rtctools.optimization.pi_mixin import PIMixin
 from rtctools.optimization.csv_mixin import CSVMixin
 from rtctools.util import run_optimization_problem, _resolve_folder
-from rtctools_interface.closed_loop.closed_loop_dates import read_closed_loop_dates
+from rtctools_interface.closed_loop.config import ClosedLoopConfig
+import rtctools_interface.closed_loop.optimization_ranges as opt_ranges
 from rtctools_interface.closed_loop.results_construction import combine_csv_exports, combine_xml_exports
 from rtctools_interface.closed_loop.time_series_handler import XMLTimeSeriesFile, CSVTimeSeriesFile, TimeSeriesHandler
 import logging
@@ -53,26 +54,49 @@ def write_input_folder(
     timeseries_import.write(modelling_period_input_folder_i)
 
 
+def _get_optimization_ranges(
+    config: ClosedLoopConfig,
+    input_timeseries: TimeSeriesHandler,
+) -> list[tuple[datetime.datetime, datetime.datetime]]:
+    """Return a list of optimization periods."""
+    if config.file is not None:
+        datetime_range = input_timeseries.get_datetime_range()
+        optimization_ranges = opt_ranges.get_optimization_ranges_from_file(
+            config.file, datetime_range
+        )
+    elif config.optimization_period is not None:
+        datetimes = input_timeseries.get_datetimes()
+        optimization_ranges = opt_ranges.get_optimization_ranges(
+            model_times=datetimes,
+            start_time=datetimes[0],
+            forecast_timestep=config.forecast_timestep,
+            optimization_period=config.optimization_period,
+        )
+    else:
+        raise ValueError(
+            "The closed-loop configuration should have either a file or optimization_period set."
+        )
+    if config.round_to_dates:
+        optimization_ranges = opt_ranges.round_datetime_ranges_to_days(optimization_ranges)
+    return optimization_ranges
+
+
 def run_optimization_problem_closed_loop(
     optimization_problem_class,
     base_folder="..",
     log_level=logging.INFO,
     profile=False,
+    config: ClosedLoopConfig = None,
     **kwargs,
 ):
-    """Runs an optimization problem in closed loop mode.
-    This function is a drop-in replacement for the run_optimization_problem of rtc-tools. The user
-    needs to specify a closed_loop_dates.csv in the input folder of the optimization problem. This
-    CSV file should contain two columns: start_date and end_date, each row corresponding to one
-    modelling period. The function will run optimization problems for each modelling period specified
-    in the CSV file, setting the final results from the previous run as initial conditions for the next.
-    See the readme.md for more details.
-
-    Notes:
-        - The first start_date in your closed_loop_dates.csv should be equal to the start_date of
-          your timeseries_import.
-        - The different horizons should overlap with at least one day to allow retrieving and setting
-          initial values. More days of overlap are allowed.
+    """
+    Runs an optimization problem in closed loop mode.
+
+    This function is a drop-in replacement for the run_optimization_problem of rtc-tools.
+    The user needs to specify a closed-loop configuration that describes the time ranges
+    for which to subsequentially solve the optimization problem.
+    The results from the previous run will be used to set the initial values of the next run.
+    See README.md for more details.
     """
     base_folder = Path(base_folder)
     if not os.path.isabs(base_folder):
@@ -112,24 +136,15 @@ def run_optimization_problem_closed_loop(
         shutil.rmtree(modelling_periods_output_folder)
     modelling_periods_output_folder.mkdir(exist_ok=True)
 
-    original_date_range = original_import.get_datetime_range()
-    closed_loop_dates = read_closed_loop_dates(original_input_folder / "closed_loop_dates.csv")
-    assert (
-        min(closed_loop_dates["start_date"]).date() == original_date_range[0].date()
-    ), "The start day of the first optimization run is not equal to the start day of the timeseries import."
-    assert (
-        max(closed_loop_dates["end_date"]).date() <= original_date_range[1].date()
-    ), "The end date of one or more optimization runs is later than the end date of the timeseries import."
-
+    if config is None:
+        config = ClosedLoopConfig(original_input_folder / "closed_loop_dates.csv")
+    optimization_ranges = _get_optimization_ranges(config, original_import)
     results_previous_run = None
     previous_run_datetimes = None
-    for i, date_range in closed_loop_dates.iterrows():
+    for i, (start_time, end_time) in enumerate(optimization_ranges):
         timeseries_import = copy.deepcopy(original_import)
 
-        timeseries_import.select_time_range(
-            start_date=date_range["start_date"],
-            end_date=date_range["end_date"] + datetime.timedelta(days=1) - datetime.timedelta(seconds=1),
-        )
+        timeseries_import.select_time_range(start_date=start_time, end_date=end_time)
 
         set_initial_values_from_previous_run(results_previous_run, timeseries_import, previous_run_datetimes)
 

{rtc-tools-interface-0.8.0b4 → rtc-tools-interface-0.9.0a1}/rtctools_interface/closed_loop/time_series_handler.py

@@ -42,6 +42,10 @@ class TimeSeriesHandler(ABC):
     def get_timestep(self) -> datetime.timedelta:
         """Get the timestep of the timeseries data."""
 
+    @abstractmethod
+    def get_datetimes(self) -> List[datetime.datetime]:
+        """Get the dates of the timeseries."""
+
     @abstractmethod
     def get_datetime_range(self) -> Tuple[datetime.datetime, datetime.datetime]:
         """Get the date range of the timeseries data (min, max)."""
@@ -124,6 +128,9 @@ class CSVTimeSeriesFile(TimeSeriesHandler):
     def get_timestep(self):
         return self.data[self.date_col].diff().min()
 
+    def get_datetimes(self):
+        return self.data[self.date_col].to_list()
+
     def get_datetime_range(self):
         return self.data[self.date_col].min(), self.data[self.date_col].max()
 
@@ -323,6 +330,18 @@ class XMLTimeSeriesFile(TimeSeriesHandler):
         ) = self.get_external_id_from_series(series)
         return self.get_single_date_range(location_id, parameter_id, qualifier_ids)
 
+    def get_datetimes(self):
+        """Get the dates of all timeseries data."""
+        datetimes = set()
+        for series in self.root.findall("pi:series", ns):
+            events = series.findall("pi:event", ns)
+            for event in events:
+                event_datetime = self.__parse_date_time(event)
+                datetimes.add(event_datetime)
+        datetimes = list(datetimes)
+        datetimes.sort()
+        return datetimes
+
     def get_datetime_range(self):
         """Get the date range of the timeseries data, minimum and maximum over all series"""
         min_date = datetime.datetime.max

rtc-tools-interface-0.9.0a1/tests/closed_loop/test_optimization_ranges.py

@@ -0,0 +1,64 @@
+"""Tests for getting optimization periods."""
+import unittest
+from datetime import datetime, timedelta
+
+import rtctools_interface.closed_loop.optimization_ranges as opt_ranges
+
+
+class TestGetOptimizationPeriods(unittest.TestCase):
+    """Test calculating optimization periods."""
+    def check_ranges(
+        self,
+        ranges: list[tuple[datetime, datetime]],
+        expected_ranges: list[tuple[datetime, datetime]]
+    ):
+        """Check if two lists of ranges are the same."""
+        n_exp_ranges = len(expected_ranges)
+        n_ranges = len(ranges)
+        self.assertEqual(n_ranges, n_exp_ranges)
+        for i_period in range(n_ranges):
+            exp_start, exp_end = expected_ranges[i_period]
+            start, end = expected_ranges[i_period]
+            self.assertEqual(start, exp_start)
+            self.assertEqual(end, exp_end)
+
+    def test_get_optimization_ranges(self):
+        """Test get_optimization_ranges."""
+        model_times = [
+            datetime(2024,1,1),
+            datetime(2024,1,2),
+            datetime(2024,1,4),
+            datetime(2024,1,5),
+            datetime(2024,1,7),
+            datetime(2024,1,8),
+            datetime(2024,1,10),
+        ]
+        start_time = datetime(2024,1,2)
+        forecast_timestep = timedelta(days=2)
+        optimization_period = timedelta(days=5)
+        ranges = opt_ranges.get_optimization_ranges(
+            model_times=model_times,
+            start_time=start_time,
+            forecast_timestep=forecast_timestep,
+            optimization_period=optimization_period
+        )
+        expected_ranges = [
+            (datetime(2024,1,2), datetime(2024,1,7)),
+            (datetime(2024,1,4), datetime(2024,1,8)),
+            (datetime(2024,1,5), datetime(2024,1,10)),
+        ]
+        self.check_ranges(ranges, expected_ranges)
+
+    def test_round_datetime_ranges_to_days(self):
+        """Test round_datetime_ranges_to_days."""
+        ranges = [
+            (datetime(2020,1,1), datetime(2020,1,2)),
+            (datetime(2020,1,2), datetime(2020,1,2)),
+            (datetime(2020,1,2,12), datetime(2020,1,5,23)),
+        ]
+        rounded_ranges = opt_ranges.round_datetime_ranges_to_days(ranges)
+        expected_ranges = [
+            (datetime(2020,1,1), datetime(2020,1,2,23,59,59,999999)),
+            (datetime(2020,1,2,12), datetime(2020,1,5,23,59,59,999999)),
+        ]
+        self.check_ranges(rounded_ranges, expected_ranges)

rtc-tools-interface-0.9.0a1/tests/closed_loop/test_run_optization_problem_closed_loop.py

@@ -0,0 +1,180 @@
+"""Test the closed loop runner"""
+import math
+import xml.etree.ElementTree as ET
+from datetime import timedelta
+from pathlib import Path
+from unittest import TestCase
+
+import pandas as pd
+
+from rtctools_interface.closed_loop.config import ClosedLoopConfig
+from rtctools_interface.closed_loop.runner import run_optimization_problem_closed_loop
+from .test_models.goal_programming_xml.src.example import Example as ExampleXml
+from .test_models.goal_programming_csv.src.example import Example as ExampleCsv
+
+ns = {"fews": "http://www.wldelft.nl/fews", "pi": "http://www.wldelft.nl/fews/PI"}
+
+# Elementwise comparisons are practially disabled.
+A_TOL = 0.1
+R_TOL = 0.1
+
+
+def compare_xml_file(file_result: Path, file_ref: Path):
+    """Compare two timeseries_export files elementwise."""	
+    tree_result = ET.parse(file_result)
+    tree_ref = ET.parse(file_ref)
+    series_result = tree_result.findall("pi:series", ns)
+    series_ref = tree_ref.findall("pi:series", ns)
+    assert len(series_result) == len(series_ref), "Different number of series found in exports."
+    for serie_result, serie_ref in zip(series_result, series_ref):
+        for event_result, event_ref in zip(serie_result.findall("pi:event", ns), serie_ref.findall("pi:event", ns)):
+            value_result = float(event_result.attrib["value"])
+            value_ref = float(event_ref.attrib["value"])
+            assert math.isclose(
+                value_result, value_ref, rel_tol=R_TOL, abs_tol=A_TOL
+            ), f"Difference found in event: {value_result} != {value_ref}"
+
+
+def compare_xml_files(output_modelling_period_folder: Path, reference_folder: Path):
+    """Compare the timeseries_export.xml files in the output and reference folders."""
+    for folder in output_modelling_period_folder.iterdir():
+        if not folder.is_dir():
+            continue
+        file_name = "timeseries_export.xml"
+        file_result = folder / file_name
+        file_ref = reference_folder / folder.name / file_name
+        compare_xml_file(file_result, file_ref)
+
+
+class TestClosedLoop(TestCase):
+    """
+    Class for testing closed loop runner.
+    """
+
+    def compare_csv_files(self, output_folder: Path, reference_folder: Path, n_periods: int):
+        """Compare the csv files in output and reference subfolders."""
+        self.assertTrue(
+            output_folder.exists(),
+            "Output modelling period folder should be created."
+        )
+        self.assertEqual(
+            len(list(output_folder.iterdir())),
+            n_periods,
+            f"Error: {n_periods} modelling periods should be created."
+        )
+        for folder in output_folder.iterdir():
+            self.assertTrue((folder / "timeseries_export.csv").exists())
+        for folder in output_folder.iterdir():
+            reference_folder_i = reference_folder / folder.name
+            for file in folder.iterdir():
+                df_result = pd.read_csv(file)
+                df_ref = pd.read_csv(reference_folder_i / file.name)
+                pd.testing.assert_frame_equal(df_result, df_ref, atol=A_TOL, rtol=R_TOL)
+
+    def compare_csv_file(self, output_file: Path, reference_file: Path):
+        """Compare the main csv output file in output and reference folder."""
+        df_result = pd.read_csv(output_file)
+        df_ref = pd.read_csv(reference_file)
+        pd.testing.assert_frame_equal(df_result, df_ref, atol=A_TOL, rtol=R_TOL)
+
+    def compare_xml_files(self, output_folder: Path, reference_folder: Path, n_periods: int):
+        """Compare the xml files in output and reference subfolders."""
+        self.assertTrue(
+            output_folder.exists(),
+            "Output modelling period folder should be created."
+        )
+        self.assertEqual(
+            len([f for f in output_folder.iterdir() if f.is_dir()]),
+            n_periods,
+            f"Error: {n_periods} modelling periods should be created.",
+        )
+        for folder in output_folder.iterdir():
+            if folder.is_dir():
+                self.assertTrue((folder / "timeseries_export.xml").exists())
+        compare_xml_files(output_folder, reference_folder)
+
+    def compare_xml_file(self, output_file: Path, reference_file: Path):
+        """Compare the main xml output file in output and reference folder."""
+        compare_xml_file(output_file, reference_file)
+
+    def test_running_closed_loop_csv(self):
+        """
+        Check if test model runs without problems and generates same results.
+        """
+        base_folder = Path(__file__).parent / "test_models" / "goal_programming_csv"
+        config = ClosedLoopConfig(
+            file=base_folder / "input" / "closed_loop_dates.csv",
+            round_to_dates=True
+        )
+        run_optimization_problem_closed_loop(ExampleCsv, base_folder=base_folder, config=config)
+        self.compare_csv_files(
+            output_folder=base_folder / "output" / "output_modelling_periods",
+            reference_folder=base_folder / "output" / "output_modelling_periods_reference",
+            n_periods=3
+        )
+        self.compare_csv_file(
+            output_file=base_folder / "output" / "timeseries_export.csv",
+            reference_file=base_folder / "output" / "timeseries_export_reference.csv"
+        )
+
+    def test_running_closed_loop_csv_fixed_periods(self):
+        """
+        Check if test model runs for fixed optimization periods.
+        """
+        base_folder = Path(__file__).parent / "test_models" / "goal_programming_csv"
+        output_folder = "output_fixed_periods"
+        config = ClosedLoopConfig.from_fixed_periods(
+            optimization_period=timedelta(days=3),
+            forecast_timestep=timedelta(days=2)
+        )
+        run_optimization_problem_closed_loop(
+            ExampleCsv,
+            base_folder=base_folder,
+            config=config,
+            output_folder=output_folder
+        )
+        self.compare_csv_file(
+            output_file=base_folder / output_folder / "timeseries_export.csv",
+            reference_file=base_folder / "output" / "timeseries_export_reference.csv"
+        )
+
+    def test_running_closed_loop_xml(self):
+        """
+        Check if test model runs without problems and generates same results.
+        """
+        base_folder = Path(__file__).parent / "test_models" / "goal_programming_xml"
+        config = ClosedLoopConfig(
+            file=base_folder / "input" / "closed_loop_dates.csv",
+            round_to_dates=True
+        )
+        run_optimization_problem_closed_loop(ExampleXml, base_folder=base_folder, config=config)
+        self.compare_xml_files(
+            output_folder=base_folder / "output" / "output_modelling_periods",
+            reference_folder=base_folder / "output" / "output_modelling_periods_reference",
+            n_periods=3
+        )
+        self.compare_xml_file(
+            output_file=base_folder / "output" / "timeseries_export.xml",
+            reference_file=base_folder / "output" / "timeseries_export_reference.xml"
+        )
+
+    def test_running_closed_loop_xml_fixed_periods(self):
+        """
+        Check if test model runs for fixed optimization periods.
+        """
+        base_folder = Path(__file__).parent / "test_models" / "goal_programming_xml"
+        output_folder = "output_fixed_periods"
+        config = ClosedLoopConfig.from_fixed_periods(
+            optimization_period=timedelta(days=3),
+            forecast_timestep=timedelta(days=2)
+        )
+        run_optimization_problem_closed_loop(
+            ExampleXml,
+            base_folder=base_folder,
+            config=config,
+            output_folder=output_folder
+        )
+        self.compare_xml_file(
+            output_file=base_folder / output_folder / "timeseries_export.xml",
+            reference_file=base_folder / "output" / "timeseries_export_reference.xml"
+        )

rtc-tools-interface-0.8.0b4/rtctools_interface/closed_loop/closed_loop_dates.py

@@ -1,43 +0,0 @@
-from pathlib import Path
-
-import pandas as pd
-
-
-def read_closed_loop_dates(file_path: Path):
-    """Read horizon config from a csv file"""
-    if not file_path.exists():
-        raise FileNotFoundError(
-            f"The closed_loop_dates csv does not exist. Please create a horizon config file in {file_path}."
-        )
-    try:
-        closed_loop_dates = pd.read_csv(file_path)
-    except pd.errors.EmptyDataError:
-        raise ValueError(
-            "The closed_loop_dates csv is empty. Please provide a valid file with start_date and end_date column."
-        )
-    closed_loop_dates.columns = closed_loop_dates.columns.str.replace(" ", "")
-    if not all([col in closed_loop_dates.columns for col in ["start_date", "end_date"]]):
-        raise ValueError("The closed_loop_dates csv should have both 'start_date' and 'end_date' columns.")
-    closed_loop_dates["start_date"] = pd.to_datetime(closed_loop_dates["start_date"])
-    closed_loop_dates["end_date"] = pd.to_datetime(closed_loop_dates["end_date"])
-    for i in range(1, len(closed_loop_dates)):
-        if closed_loop_dates["start_date"].iloc[i] > closed_loop_dates["end_date"].iloc[i - 1]:
-            raise ValueError(f"Closed loop date table: Start date at row {i} is later than the previous end date. ")
-    if any(closed_loop_dates["start_date"] < closed_loop_dates["start_date"].shift(1)):
-        raise ValueError("Closed loop date table: The start dates are not in ascending order.")
-    if any(closed_loop_dates["end_date"] < closed_loop_dates["end_date"].shift(1)):
-        raise ValueError("Closed loop date table: The end dates are not in ascending order.")
-    if any(closed_loop_dates["end_date"] < closed_loop_dates["start_date"]):
-        raise ValueError("Closed loop date table: For one or more rows the end date is before the start date.")
-    if any(closed_loop_dates["start_date"] > closed_loop_dates["end_date"]):
-        raise ValueError("Closed loop date table: For one or more rows the start date is after the end date.")
-    if (
-        any(closed_loop_dates["start_date"].dt.hour != 0)
-        or any(closed_loop_dates["start_date"].dt.minute != 0)
-        or any(closed_loop_dates["end_date"].dt.hour != 0)
-        or any(closed_loop_dates["end_date"].dt.minute != 0)
-    ):
-        raise ValueError(
-            "Closed loop date table: Currently, the date ranges can only be specific up to the level of days."
-        )
-    return closed_loop_dates

rtc-tools-interface-0.8.0b4/tests/closed_loop/test_run_optization_problem_closed_loop.py

@@ -1,98 +0,0 @@
-"""Test the closed loop runner"""
-import math
-import xml.etree.ElementTree as ET
-from unittest import TestCase
-from pathlib import Path
-import pandas as pd
-from rtctools_interface.closed_loop.runner import run_optimization_problem_closed_loop
-from .test_models.goal_programming_xml.src.example import Example as ExampleXml
-from .test_models.goal_programming_csv.src.example import Example as ExampleCsv
-
-ns = {"fews": "http://www.wldelft.nl/fews", "pi": "http://www.wldelft.nl/fews/PI"}
-
-# Elementwise comparisons are practially disabled.
-A_TOL = 0.1
-R_TOL = 0.1
-
-
-def compare_xml_file(file_result: Path, file_ref: Path):
-    """Compare two timeseries_export files elementwise."""	
-    tree_result = ET.parse(file_result)
-    tree_ref = ET.parse(file_ref)
-    series_result = tree_result.findall("pi:series", ns)
-    series_ref = tree_ref.findall("pi:series", ns)
-    assert len(series_result) == len(series_ref), "Different number of series found in exports."
-    for serie_result, serie_ref in zip(series_result, series_ref):
-        for event_result, event_ref in zip(serie_result.findall("pi:event", ns), serie_ref.findall("pi:event", ns)):
-            value_result = float(event_result.attrib["value"])
-            value_ref = float(event_ref.attrib["value"])
-            assert math.isclose(
-                value_result, value_ref, rel_tol=R_TOL, abs_tol=A_TOL
-            ), f"Difference found in event: {value_result} != {value_ref}"
-
-
-def compare_xml_files(output_modelling_period_folder: Path, reference_folder: Path):
-    """Compare the timeseries_export.xml files in the output and reference folders."""
-    for folder in output_modelling_period_folder.iterdir():
-        if not folder.is_dir():
-            continue
-        file_name = "timeseries_export.xml"
-        file_result = folder / file_name
-        file_ref = reference_folder / folder.name / file_name
-        compare_xml_file(file_result, file_ref)
-
-
-class TestClosedLoop(TestCase):
-    """
-    Class for testing closed loop runner.
-    """
-
-    def test_running_closed_loop_csv(self):
-        """
-        Check if test model runs without problems and generates same results.
-        """
-        base_folder = Path(__file__).parent / "test_models" / "goal_programming_csv"
-        run_optimization_problem_closed_loop(ExampleCsv, base_folder=base_folder)
-
-        output_modelling_period_folder = base_folder / "output" / "output_modelling_periods"
-        self.assertTrue(output_modelling_period_folder.exists(), "Output modelling period folder should be created.")
-        self.assertEqual(
-            len(list(output_modelling_period_folder.iterdir())), 3, "Three modelling periods should be created."
-        )
-        for folder in output_modelling_period_folder.iterdir():
-            self.assertTrue((folder / "timeseries_export.csv").exists())
-        reference_folder = base_folder / "output" / "output_modelling_periods_reference"
-        for folder in output_modelling_period_folder.iterdir():
-            reference_folder_i = reference_folder / folder.name
-            for file in folder.iterdir():
-                df_result = pd.read_csv(file)
-                df_ref = pd.read_csv(reference_folder_i / file.name)
-                pd.testing.assert_frame_equal(df_result, df_ref, atol=A_TOL, rtol=R_TOL)
-        # Also compare the combined timeseries_export.csv
-        df_result = pd.read_csv(base_folder / "output" / "timeseries_export.csv")
-        df_ref = pd.read_csv(base_folder / "output" / "timeseries_export_reference.csv")
-        pd.testing.assert_frame_equal(df_result, df_ref, atol=A_TOL, rtol=R_TOL)
-
-    def test_running_closed_loop_xml(self):
-        """
-        Check if test model runs without problems and generates same results.
-        """
-        base_folder = Path(__file__).parent / "test_models" / "goal_programming_xml"
-        run_optimization_problem_closed_loop(ExampleXml, base_folder=base_folder)
-
-        output_modelling_period_folder = base_folder / "output" / "output_modelling_periods"
-        self.assertTrue(output_modelling_period_folder.exists(), "Output modelling period folder should be created.")
-        self.assertEqual(
-            len([f for f in output_modelling_period_folder.iterdir() if f.is_dir()]),
-            3,
-            "Three modelling periods should be created.",
-        )
-        for folder in output_modelling_period_folder.iterdir():
-            if folder.is_dir():
-                self.assertTrue((folder / "timeseries_export.xml").exists())
-        reference_folder = base_folder / "output" / "output_modelling_periods_reference"
-        compare_xml_files(output_modelling_period_folder, reference_folder)
-        # Also compare the combined timeseries_export
-        compare_xml_file(
-            base_folder / "output" / "timeseries_export.xml", base_folder / "output" / "timeseries_export_reference.xml"
-        )