rtc-tools-interface 0.10.2a3__tar.gz → 0.10.2a4__tar.gz

{rtc_tools_interface-0.10.2a3 → rtc_tools_interface-0.10.2a4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rtc_tools_interface
-Version: 0.10.2a3
+Version: 0.10.2a4
 Summary: Toolbox for user interfaces for RTC-Tools
 Author: Deltares
 Maintainer: Deltares

{rtc_tools_interface-0.10.2a3 → rtc_tools_interface-0.10.2a4}/README.md

@@ -11,8 +11,9 @@ pip install rtc-tools-interface
 ## Table of Contents
 1. [Goal generator](#goal-generator)
 2. [Goal performance metrics](#goal-performance-metrics)
-3. [Automatic plotting of results](#automatic-plotting-of-results)
-4. [Closed loop runner](#closed-loop-runner)
+3. [Active constraint diagnostics](#active-constraint-diagnostics)
+4. [Automatic plotting of results](#automatic-plotting-of-results)
+5. [Closed loop runner](#closed-loop-runner)
 
 ## Goal generator
 The `goal generator` can be used to automatically add goals based on a csv file. Currently, the following goal types are supported:
@@ -91,18 +92,19 @@ by minimizing, if any, the sum of exceedances for the timesteps. For more detail
 
 
 #### range_rate_of_change
-The range_rate_of_change goal can be used to set a target range on ramp rate. Like the range goal, one needs to set the `target_min` and `target_max` for that. **Importantly** for the range_rate_of_change goal, the supplied values are relative to the nominal of the function. So supplying a `target_max` of `10` corresponds to the aim of having a maximum increase per timestep of 10% * `nominal`, where the nominal automatically set to `maximum rate of change`/2 or specified manually. To formulate the target of having a maximum increase and decrease by of 10% per timestep, one would set the `target_min` to `-10` and the `target_max` to `10`.
+The range_rate_of_change goal can be used to set a target range on ramp rate for a model state. Like the range goal, one needs to set the `target_min` and `target_max` for that. **Importantly** for the range_rate_of_change goal, the supplied values are relative to the nominal of the function. So supplying a `target_max` of `10` corresponds to the aim of having a maximum increase per timestep of 10% * `nominal`, where the nominal automatically set to `maximum rate of change`/2 or specified manually. To formulate the target of having a maximum increase and decrease by of 10% per timestep, one would set the `target_min` to `-10` and the `target_max` to `10`.
 
-The equations for the range_rate_of_change goal are almost the same as for the range goal, which can be found above. The only difference is that $x_t$ is replaced by the derivative of $x$ with respect to $t$, calculated as $\frac{x_t - x_{t-1}}{\Delta t}$.  
+The equations for the range_rate_of_change goal are almost the same as for the range goal, which can be found above. The only difference is that $x_t$ is replaced by the derivative of $x$ with respect to $t$, calculated as $\frac{x_t - x_{t-1}}{\Delta t}$. RTC-Tools computes the derivative (the ramp rate) internally for the state variable specified in the goal table. 
 
 ### Example goal table
 See the table below for an example content of the `goal_table.csv`.
 
 | id     | state | active | goal_type    | function_min | function_max | function_nominal | target_data_type | target_min | target_max | priority | weight | order |
 |--------|-------|--------|--------------|--------------|--------------|------------------|------------------|------------|------------|----------|--------|-------|
-| goal_1 | reservoir_1_waterlevel     | 1      | range        | 0            | 15           | 10               | value            | 5.0        | 10.0       | 5       |        |       |
-| goal_2 | reservoir_2_waterlevel     | 1      | range        | 0            | 15           | 10               | timeseries            | "target_series"        | "target_series"       | 10       |        |       |
-| goal_3 | electricity_cost     | 1      | minimization_path |              |              |                  |                  |            |            | 20       |        |       |
+| goal_1 | reservoir_1_waterlevel     | 1      | range        | 0            | 15           | 10               | value            | 5.0        | 10.0       | 5       |    1    |    1   |
+| goal_2 | reservoir_2_waterlevel     | 1      | range        | 0            | 15           | 10               | timeseries            | "target_series"        | "target_series"       | 10       |    1    |   1    |
+| goal_3 | electricity_cost     | 1      | minimization_path |              |              |                  |                  |            |            | 20       |   1     |   1    |
+| goal_4 | reservoir_2_volume | 1 | range_rate_of_change | -100 | 100 | 10 | value | -10 | 10 | 30 | 1 | 1 |  
 
 ## Goal performance metrics
 For all goals defined with the goal generator this rtc-tools-interface module will also calculate performance metrics. By default, these performance metrics are saved to a .csv in the folder `output/perfomance_metrics`, with one csv file per goal. With the class variable `calculate_performance_metrics` this functionality can be disabled (by default it is enabled).
@@ -116,6 +118,23 @@ The calculated metrics are:
 - `mean_absolute_percentual_difference`: The mean of the absolute percentual difference per timestep over all timesteps (only for range goals).
 - `mean_absolute_difference`: The mean absolute difference per timestep of the state variable over all timesteps (only for range goals).
 
+## Active constraint diagnostics
+The `ActiveConstraintMixin` writes diagnostics about active constraints after each RTC-Tools goal-programming priority. A constraint is considered active when its evaluated value hits either its lower or upper bound within `active_constraint_tolerance` (default `1e-6`).
+
+To enable the diagnostics, import the mixin and add it before `BaseOptimizationProblem` or before RTC-Tools' `GoalProgrammingMixin` in the class inheritance order:
+
+```python
+from rtctools_interface.optimization.active_constraint_mixin import ActiveConstraintMixin
+from rtctools_interface.optimization.base_optimization_problem import BaseOptimizationProblem
+
+
+class MyOptimizationProblem(ActiveConstraintMixin, BaseOptimizationProblem):
+    pass
+```
+
+After optimization, the mixin creates `output/active_constraints/active_constraints_of_previous_goals.csv`.
+This file contains detailed rows for active constraints that RTC-Tools created from goals optimized in previous priorities. For each priority with active previous-goal constraints, it reports the total number of previous-goal constraints, how many are active, which bound (`lower`, `upper`, or `both`) was hit, and the corresponding bound value. Path-goal constraints are reported in one row per active path goal; the `active_times` column lists the timesteps at which that path-goal constraint is active.
+
 
 ## Automatic plotting of results
 With the `PlotMixin` one can easily make plots of the results of rtc-tools. This functionality can be used both for optimization and simulation problems. For optimization problems, use:

{rtc_tools_interface-0.10.2a3 → rtc_tools_interface-0.10.2a4}/rtc_tools_interface.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rtc_tools_interface
-Version: 0.10.2a3
+Version: 0.10.2a4
 Summary: Toolbox for user interfaces for RTC-Tools
 Author: Deltares
 Maintainer: Deltares

{rtc_tools_interface-0.10.2a3 → rtc_tools_interface-0.10.2a4}/rtc_tools_interface.egg-info/SOURCES.txt

@@ -17,6 +17,8 @@ rtctools_interface/closed_loop/results_construction.py
 rtctools_interface/closed_loop/runner.py
 rtctools_interface/closed_loop/time_series_handler.py
 rtctools_interface/optimization/__init__.py
+rtctools_interface/optimization/active_constraint_helpers.py
+rtctools_interface/optimization/active_constraint_mixin.py
 rtctools_interface/optimization/base_goal.py
 rtctools_interface/optimization/base_optimization_problem.py
 rtctools_interface/optimization/goal_generator_mixin.py
@@ -42,10 +44,14 @@ rtctools_interface/utils/serialization.py
 rtctools_interface/utils/type_definitions.py
 tests/__init__.py
 tests/closed_loop/__init__.py
+tests/closed_loop/test_combine_xml_exports.py
 tests/closed_loop/test_optimization_ranges.py
 tests/closed_loop/test_read_xml.py
 tests/closed_loop/test_run_optization_problem_closed_loop.py
+tests/closed_loop/test_set_initial_values_from_previous_run.py
 tests/optimization/__init__.py
+tests/optimization/test_active_constraint_helpers.py
+tests/optimization/test_active_constraint_mixin.py
 tests/optimization/test_base_optimization_problem.py
 tests/optimization/test_passing_goals_directly.py
 tests/optimization/test_plot_goals_mixin.py

{rtc_tools_interface-0.10.2a3 → rtc_tools_interface-0.10.2a4}/rtctools_interface/_version.py

@@ -8,11 +8,11 @@ import json
 
 version_json = '''
 {
- "date": "2026-01-19T15:25:58+0100",
+ "date": "2026-09-17T07:52:27+0200",
  "dirty": false,
  "error": null,
- "full-revisionid": "b1cd2bed75dbdf57577a1474771953e8983e405d",
- "version": "0.10.2a3"
+ "full-revisionid": "8d59f3b94a76dba915ac4488e060b927d4820acc",
+ "version": "0.10.2a4"
 }
 '''  # END VERSION_JSON
 

{rtc_tools_interface-0.10.2a3 → rtc_tools_interface-0.10.2a4}/rtctools_interface/closed_loop/results_construction.py

@@ -36,8 +36,8 @@ def combine_xml_exports(
     )
     if ts_import_orig.forecast_datetime > ts_import_orig.start_datetime:
         logger.info(
-            "Timeseries export will start at original forecast date, "
-            "disregarding data before forecast date."
+            "Timeseries export will start at original forecast date, disregarding data before"
+            " forecast date."
         )
         ts_import_orig.resize(ts_import_orig.forecast_datetime, ts_import_orig.end_datetime)
         ts_import_orig.times = ts_import_orig.times[
@@ -67,6 +67,21 @@ def combine_xml_exports(
         all_times = ts_import_orig.times  # Workaround to map indices to times, as ts_export does
         # not contain all times. TODO Check whether the assumption that these times map to
         # the correct indices for ts_export always holds.
+        new_times = ts_export_step.times
+        if len(new_times) <= 1:
+            logger.debug(f"Skipping model horizon {i}: only a single timestep was exported.")
+            i += 1
+            continue
+        try:
+            start_new_data_index = all_times.index(new_times[1])
+        except ValueError:
+            if all_times[-1] + ts_export.dt == new_times[0]:
+                start_new_data_index = len(all_times)
+            else:
+                raise ValueError(
+                    "Could not match the start data of the timeseries export file "
+                    + "with the end of the previous."
+                )
         for loc_par in variables:
             try:
                 current_values = ts_export.get(loc_par)
@@ -74,21 +89,10 @@ def combine_xml_exports(
             except KeyError:
                 logger.debug(f"Variable {loc_par} not found in output of model horizon: {i}")
                 continue
-            new_times = ts_export_step.times
-            try:
-                start_new_data_index = all_times.index(new_times[0])
-            except ValueError:
-                if all_times[-1] + ts_export.dt == new_times[0]:
-                    start_new_data_index = len(all_times)
-                else:
-                    raise ValueError(
-                        "Could not match the start data of the timeseries export file "
-                        + "with the end of the previous."
-                    )
             combined_values = copy.deepcopy(current_values)
-            combined_values[start_new_data_index : start_new_data_index + len(new_values)] = (
-                new_values  # noqa
-            )
+            combined_values[start_new_data_index : start_new_data_index + len(new_values[1:])] = (
+                new_values[1:]
+            )  # noqa
             ts_export.set(loc_par, combined_values)
         i += 1
     ts_export.write(output_folder=output_base_path.parent, output_filename="timeseries_export")
@@ -109,8 +113,8 @@ def combine_xml_exports(
 
 def combine_dataframes(dfs: list[pd.DataFrame], index_col: str = "time"):
     """Combine multiple dataframes with the same index column.
-    The dataframes are combined in the order they are passed,
-    with the last dataframe taking precedence in case of overlapping indices."""
+    The dataframes are combined in the order they are passed, with the last dataframe taking
+    precedence in case of overlapping indices."""
     combined_df = pd.DataFrame()
     for df in dfs:
         df.set_index(index_col, inplace=True)

{rtc_tools_interface-0.10.2a3 → rtc_tools_interface-0.10.2a4}/rtctools_interface/closed_loop/runner.py

@@ -187,15 +187,27 @@ def run_optimization_problem_closed_loop(
         )
 
         logger.info(f"Running optimization for period {i}: {(str(start_time), str(end_time))}.")
-        result = run_optimization_problem(
-            optimization_problem_class,
-            base_folder,
-            log_level,
-            profile,
-            input_folder=modelling_period_input_folder_i,
-            output_folder=modelling_period_output_folder_i,
-            **kwargs,
-        )
+        run_number_in_fallback_list = 1
+        # Fallback mechanism: allows optimization class to request a retry
+        # run_number_in_fallback_list: 1=first attempt, 2=retry with fallback, 3=done
+        for _k in range(2):
+            if run_number_in_fallback_list < 3:
+                result = run_optimization_problem(
+                    optimization_problem_class,
+                    base_folder,
+                    log_level,
+                    profile,
+                    input_folder=modelling_period_input_folder_i,
+                    output_folder=modelling_period_output_folder_i,
+                    run_number_in_fallback_list=run_number_in_fallback_list,
+                    **kwargs,
+                )
+                if hasattr(result, "run_number_in_fallback_list"):
+                    run_number_in_fallback_list = result.run_number_in_fallback_list
+                else:
+                    run_number_in_fallback_list = 3
+                if run_number_in_fallback_list == 2:
+                    logger.info("The model failed, it will run it again with a fallback option")
         period = f"period {i} {(str(start_time), str(end_time))}"
         if result.solver_stats["success"]:
             logger.info(f"Successful optimization for {period}.")
@@ -207,11 +219,21 @@ def run_optimization_problem_closed_loop(
             logger.error(message)
             raise Exception(message)
 
-        results_previous_run = {
-            key: result.extract_results().get(key)
-            for key in variables_in_import
-            if key not in fixed_input_series
-        }
+        extracted_results = result.extract_results() or {}
+        results_previous_run = {}
+        for key in variables_in_import:
+            if key not in fixed_input_series:
+                extracted_value = extracted_results.get(key)
+                if extracted_value is not None:
+                    results_previous_run[key] = extracted_value
+                else:
+                    try:
+                        fallback_value = result.get_timeseries(key)
+                    except KeyError:
+                        fallback_value = None
+                    results_previous_run[key] = (
+                        fallback_value.values if fallback_value is not None else None
+                    )
         previous_run_datetimes = result.io.datetimes
 
     logger.info("Finished all optimization runs.")

rtc_tools_interface-0.10.2a4/rtctools_interface/optimization/active_constraint_helpers.py

@@ -0,0 +1,102 @@
+"""Helper functions for active-constraint diagnostics."""
+
+import csv
+from collections.abc import Iterable
+from pathlib import Path
+
+import casadi as ca
+import numpy as np
+from rtctools.optimization.timeseries import Timeseries
+
+from rtctools_interface.utils.type_definitions import PreviousGoalConstraintRow
+
+
+def as_flat_float_array(value: object) -> np.ndarray:
+    """Convert CasADi/numeric values to a one-dimensional float array."""
+    if isinstance(value, Timeseries):
+        value = value.values
+    if isinstance(value, (list, tuple)):
+        if not value:
+            return np.array([], dtype=float)
+        value = ca.veccat(*value)
+    array = np.array(value, dtype=float)
+    return array.reshape(-1)
+
+
+def bound_to_array(bound: object, size: int) -> np.ndarray:
+    """Return a flat bound array matching an evaluated constraint size."""
+    if isinstance(bound, Timeseries):
+        bound = bound.values
+    array = np.array(bound, dtype=float)
+    if array.size == 1 and size != 1:
+        return np.full(size, float(array.reshape(-1)[0]))
+    return array.reshape(-1)
+
+
+def component_time(
+    times: np.ndarray | None, component_index: int, n_components_per_time: int
+) -> float | str:
+    """Return the time associated with a flattened path-constraint component."""
+    if times is None or len(times) == 0:
+        return ""
+    return times[component_index // n_components_per_time]
+
+
+def format_active_times(
+    times: np.ndarray | None, active_indices: np.ndarray, values_size: int
+) -> str:
+    """Format unique active timesteps for a flattened path-constraint vector."""
+    if times is None or len(times) == 0:
+        return ""
+    if values_size % len(times) != 0:
+        raise ValueError(
+            "Cannot map path-goal constraint components to times: "
+            f"values size {values_size} is not divisible by {len(times)} timesteps."
+        )
+    n_components_per_time = values_size // len(times)
+    active_times = [
+        component_time(times, int(index), n_components_per_time) for index in active_indices
+    ]
+    return format_values(active_times)
+
+
+def format_indexed_values(values: np.ndarray, indices: np.ndarray) -> str:
+    """Format values at selected indices for aggregated CSV cells."""
+    return format_values([values[index] for index in indices])
+
+
+def format_values(values: Iterable[object]) -> str:
+    """Format unique, non-empty values as a semicolon-separated string."""
+    formatted_values = []
+    for value in values:
+        if value == "" or (isinstance(value, float) and np.isnan(value)):
+            continue
+        formatted_value = str(value)
+        if formatted_value not in formatted_values:
+            formatted_values.append(formatted_value)
+    return ";".join(formatted_values)
+
+
+def active_bound_description(
+    lower_active: bool, upper_active: bool, lower_bound: float, upper_bound: float
+) -> tuple[str, str | float]:
+    """Return a text label and value for the bound hit by a constraint."""
+    if lower_active and upper_active:
+        if np.isclose(lower_bound, upper_bound, rtol=0.0, atol=0.0):
+            return "both", lower_bound
+        return "both", f"{lower_bound};{upper_bound}"
+    if lower_active:
+        return "lower", lower_bound
+    if upper_active:
+        return "upper", upper_bound
+    return "", ""
+
+
+def write_csv(
+    file_path: Path, fieldnames: list[str], rows: list[PreviousGoalConstraintRow]
+) -> None:
+    """Write rows to a CSV file with a stable header."""
+    with file_path.open("w", newline="") as csv_file:
+        writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
+        writer.writeheader()
+        writer.writerows(rows)

rtc_tools_interface-0.10.2a4/rtctools_interface/optimization/active_constraint_mixin.py

@@ -0,0 +1,302 @@
+"""Mixin for reporting active constraints after goal-programming priorities."""
+
+from pathlib import Path
+from typing import Any, cast
+
+import casadi as ca
+import numpy as np
+
+from rtctools_interface.optimization.active_constraint_helpers import (
+    active_bound_description,
+    as_flat_float_array,
+    bound_to_array,
+    format_active_times,
+    format_indexed_values,
+    format_values,
+    write_csv,
+)
+from rtctools_interface.utils.type_definitions import PreviousGoalConstraintRow
+
+
+class _PreviousGoalConstraintInternalRow(PreviousGoalConstraintRow):
+    """Previous-goal constraint row while active-state filtering is still needed."""
+
+    is_active: bool
+
+
+class ActiveConstraintMixin:
+    """Write active-constraint diagnostics for RTC-Tools optimizations.
+
+    Add this mixin before ``GoalProgrammingMixin``/``BaseOptimizationProblem`` in
+    the inheritance list. After optimization,
+    ``active_constraints_of_previous_goals.csv`` is written to
+    ``<output_folder>/active_constraints`` with details for active constraints that
+    RTC-Tools created from goals optimized in previous priorities.
+    """
+
+    active_constraint_output_folder = "active_constraints"
+    active_constraint_tolerance = 1e-6
+
+    def __init__(self, **kwargs):
+        self._previous_goal_constraint_rows: list[PreviousGoalConstraintRow] = []
+        super().__init__(**kwargs)
+
+    def priority_completed(self, priority: int) -> None:
+        """Collect active-constraint diagnostics after a priority is solved."""
+        self._collect_previous_goal_constraint_details(priority)
+        super().priority_completed(priority)
+
+    def post(self) -> None:
+        """Write active-constraint diagnostics after optimization."""
+        super().post()
+        if self.solver_stats["success"]:
+            self._write_active_constraint_csv_files()
+
+    def _active_bound_masks(
+        self, values: np.ndarray, lower_bounds: np.ndarray, upper_bounds: np.ndarray
+    ) -> tuple[np.ndarray, np.ndarray]:
+        """Return masks for lower-bound and upper-bound active components."""
+        tolerance = self.active_constraint_tolerance
+        lower_active = np.isfinite(lower_bounds) & np.isclose(
+            values, lower_bounds, rtol=0, atol=tolerance
+        )
+        upper_active = np.isfinite(upper_bounds) & np.isclose(
+            values, upper_bounds, rtol=0, atol=tolerance
+        )
+        return lower_active, upper_active
+
+    def _collect_previous_goal_constraint_details(self, priority: int) -> None:
+        """Collect rows for constraints created from goals of previous priorities."""
+        rows = []
+        rows.extend(self._goal_constraint_rows(priority, "goal", is_path_goal=False))
+        rows.extend(self._goal_constraint_rows(priority, "path_goal", is_path_goal=True))
+        rows.extend(
+            self._goal_constraint_rows(
+                priority, "kept_soft_goal", is_path_goal=False, include_problem_constraints=True
+            )
+        )
+        rows.extend(
+            self._goal_constraint_rows(
+                priority,
+                "kept_soft_path_goal",
+                is_path_goal=True,
+                include_problem_constraints=True,
+            )
+        )
+
+        total_constraints = len(rows)
+        active_rows = [row for row in rows if row.pop("is_active")]
+        active_constraints = len(active_rows)
+        if active_constraints == 0:
+            self._previous_goal_constraint_rows.append(
+                self._inactive_priority_row(priority, total_constraints)
+            )
+            return
+
+        for row in active_rows:
+            row["total_previous_goal_constraints"] = total_constraints
+            row["active_previous_goals_constraints"] = active_constraints
+        self._previous_goal_constraint_rows.extend(active_rows)
+
+    @staticmethod
+    def _inactive_priority_row(priority: int, total_constraints: int) -> PreviousGoalConstraintRow:
+        """Build a summary row for a priority without active previous-goal constraints."""
+        row = cast(
+            PreviousGoalConstraintRow,
+            dict.fromkeys(PreviousGoalConstraintRow.__annotations__, ""),
+        )
+        row.update(
+            {
+                "priority": priority,
+                "total_previous_goal_constraints": total_constraints,
+                "active_previous_goals_constraints": 0,
+            }
+        )
+        return row
+
+    def _goal_constraint_rows(
+        self,
+        priority: int,
+        constraint_source: str,
+        *,
+        is_path_goal: bool,
+        include_problem_constraints: bool = False,
+    ) -> list[_PreviousGoalConstraintInternalRow]:
+        """Evaluate private RTC-Tools goal constraints and build detail rows."""
+        rows: list[_PreviousGoalConstraintInternalRow] = []
+        constraint_collections = self._get_goal_constraint_collections(
+            is_path_goal=is_path_goal, include_problem_constraints=include_problem_constraints
+        )
+        times = self.times() if is_path_goal else None
+
+        for ensemble_member, constraints in enumerate(constraint_collections):
+            for function_key, constraint in constraints:
+                goal = getattr(constraint, "goal", None)
+                if goal is not None and int(goal.priority) >= int(priority):
+                    continue
+
+                values = self._evaluate_goal_constraint(
+                    constraint, ensemble_member, is_path_goal=is_path_goal
+                )
+                lower_bounds = bound_to_array(constraint.min, values.size)
+                upper_bounds = bound_to_array(constraint.max, values.size)
+                lower_active, upper_active = self._active_bound_masks(
+                    values, lower_bounds, upper_bounds
+                )
+
+                if is_path_goal:
+                    rows.append(
+                        self._path_goal_constraint_row(
+                            priority,
+                            constraint_source,
+                            function_key,
+                            ensemble_member,
+                            goal,
+                            times,
+                            values,
+                            lower_bounds,
+                            upper_bounds,
+                            lower_active,
+                            upper_active,
+                        )
+                    )
+                    continue
+
+                for component_index, value in enumerate(values):
+                    active_bound, active_bound_value = active_bound_description(
+                        lower_active[component_index],
+                        upper_active[component_index],
+                        lower_bounds[component_index],
+                        upper_bounds[component_index],
+                    )
+                    rows.append(
+                        {
+                            "priority": priority,
+                            "total_previous_goal_constraints": "",
+                            "active_previous_goals_constraints": "",
+                            "ensemble_member": ensemble_member,
+                            "constraint_source": constraint_source,
+                            "function_key": function_key,
+                            "goal_priority": (int(goal.priority) if goal is not None else ""),
+                            "goal_class": goal.__class__.__name__ if goal is not None else "",
+                            "active_times": "",
+                            "value": float(value),
+                            "lower_bound": float(lower_bounds[component_index]),
+                            "upper_bound": float(upper_bounds[component_index]),
+                            "is_active": bool(
+                                lower_active[component_index] or upper_active[component_index]
+                            ),
+                            "active_bound": active_bound,
+                            "active_bound_value": active_bound_value,
+                        }
+                    )
+        return rows
+
+    def _path_goal_constraint_row(
+        self,
+        priority: int,
+        constraint_source: str,
+        function_key: str,
+        ensemble_member: int,
+        goal: Any,
+        times: np.ndarray | None,
+        values: np.ndarray,
+        lower_bounds: np.ndarray,
+        upper_bounds: np.ndarray,
+        lower_active: np.ndarray,
+        upper_active: np.ndarray,
+    ) -> _PreviousGoalConstraintInternalRow:
+        """Build a single summary row for a path-goal constraint."""
+        active = lower_active | upper_active
+        active_indices = np.flatnonzero(active)
+        active_bound_descriptions = [
+            active_bound_description(
+                lower_active[index],
+                upper_active[index],
+                lower_bounds[index],
+                upper_bounds[index],
+            )
+            for index in active_indices
+        ]
+
+        return {
+            "priority": priority,
+            "total_previous_goal_constraints": "",
+            "active_previous_goals_constraints": "",
+            "ensemble_member": ensemble_member,
+            "constraint_source": constraint_source,
+            "function_key": function_key,
+            "goal_priority": int(goal.priority) if goal is not None else "",
+            "goal_class": goal.__class__.__name__ if goal is not None else "",
+            "active_times": format_active_times(times, active_indices, values.size),
+            "value": format_indexed_values(values, active_indices),
+            "lower_bound": format_indexed_values(lower_bounds, active_indices),
+            "upper_bound": format_indexed_values(upper_bounds, active_indices),
+            "is_active": bool(np.any(active)),
+            "active_bound": format_values(
+                [description[0] for description in active_bound_descriptions]
+            ),
+            "active_bound_value": format_values(
+                [description[1] for description in active_bound_descriptions]
+            ),
+        }
+
+    def _get_goal_constraint_collections(
+        self, *, is_path_goal: bool, include_problem_constraints: bool
+    ) -> list[list[tuple[str, Any]]]:
+        """Read goal-constraint stores from RTC-Tools goal programming internals."""
+        if include_problem_constraints:
+            attribute = (
+                "_GoalProgrammingMixin__problem_path_constraints"
+                if is_path_goal
+                else "_GoalProgrammingMixin__problem_constraints"
+            )
+        else:
+            attribute = (
+                "_GoalProgrammingMixin__path_constraint_store"
+                if is_path_goal
+                else "_GoalProgrammingMixin__constraint_store"
+            )
+
+        constraint_store = getattr(self, attribute, [])
+        collections = []
+        for ensemble_member in range(self.ensemble_size):
+            try:
+                constraints = constraint_store[ensemble_member]
+            except (IndexError, KeyError, TypeError):
+                constraints = []
+            if hasattr(constraints, "items"):
+                collections.append(list(constraints.items()))
+            else:
+                collections.append(
+                    [
+                        (f"{attribute}_{ensemble_member}_{i}", constraint)
+                        for i, constraint in enumerate(constraints)
+                    ]
+                )
+        return collections
+
+    def _evaluate_goal_constraint(
+        self, constraint: Any, ensemble_member: int, *, is_path_goal: bool
+    ) -> np.ndarray:
+        """Evaluate a goal constraint at the current solver solution."""
+        expression = constraint.function(self)
+        if is_path_goal:
+            expression = self.map_path_expression(expression, ensemble_member)
+        function = ca.Function(
+            f"active_goal_constraint_{ensemble_member}_{id(constraint)}",
+            [self.solver_input],
+            [expression],
+        )
+        return as_flat_float_array(function(self.solver_output))
+
+    def _write_active_constraint_csv_files(self) -> None:
+        """Write collected active-constraint diagnostics to CSV files."""
+        output_folder = Path(self._output_folder) / self.active_constraint_output_folder
+        output_folder.mkdir(parents=True, exist_ok=True)
+
+        write_csv(
+            output_folder / "active_constraints_of_previous_goals.csv",
+            list(PreviousGoalConstraintRow.__annotations__),
+            self._previous_goal_constraint_rows,
+        )

{rtc_tools_interface-0.10.2a3 → rtc_tools_interface-0.10.2a4}/rtctools_interface/utils/type_definitions.py

@@ -58,6 +58,25 @@ class IntermediateResult(TypedDict):
     timeseries_data: dict[str, np.ndarray]
 
 
+class PreviousGoalConstraintRow(TypedDict):
+    """CSV row for active constraints created from previous goals."""
+
+    priority: int
+    total_previous_goal_constraints: int | str
+    active_previous_goals_constraints: int | str
+    ensemble_member: int | str
+    constraint_source: str
+    function_key: str
+    goal_priority: int | str
+    goal_class: str
+    active_times: str
+    value: float | str
+    lower_bound: float | str
+    upper_bound: float | str
+    active_bound: str
+    active_bound_value: float | str
+
+
 class PlotDataAndConfig(TypedDict):
     """All data and options required to create all plots for one optimization run."""
 

rtc_tools_interface-0.10.2a4/tests/closed_loop/test_combine_xml_exports.py

@@ -0,0 +1,106 @@
+"""Tests for the base optimization problem class."""
+
+import os
+import unittest
+import xml.etree.ElementTree as ET
+from datetime import timedelta
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+from rtctools_interface.closed_loop.results_construction import combine_xml_exports
+
+
+def read_timeseries_data_from_xml(path, location_id_to_extract):
+    tree = ET.parse(path)
+    root = tree.getroot()
+
+    ns = {"pi": "http://www.wldelft.nl/fews/PI"}
+
+    for series in root.findall("pi:series", ns):
+        header = series.find("pi:header", ns)
+        location_id = header.find("pi:locationId", ns)
+        if location_id_to_extract in location_id.text:
+            values = []
+            for event in series.findall("pi:event", ns):
+                values.append(float(event.get("value")))
+            break
+    return values
+
+
+class TestCombineXmlExports(unittest.TestCase):
+    """Combining XML output files."""
+
+    def test_combine_xml_exports(self):
+        TEST_DIR = Path(__file__).parent
+
+        original_input_timeseries_path = TEST_DIR / "test_models" / "goal_programming_xml" / "input"
+
+        output_base_path = (
+            TEST_DIR
+            / "test_models"
+            / "goal_programming_xml"
+            / "output"
+            / "output_modelling_periods_reference"
+        )
+
+        combine_xml_exports(output_base_path, original_input_timeseries_path)
+        forecast_timestep = timedelta(days=2)
+        timestep = timedelta(hours=8)
+
+        values_per_period = {}
+
+        for period_number in range(3):
+            xml_path = os.path.join(
+                output_base_path, Path("period_" + str(period_number) + "/timeseries_export.xml")
+            )
+            values = read_timeseries_data_from_xml(xml_path, "Q_orifice")
+            values_per_period[period_number] = values
+
+        # Logic for glueing the series together without taking the first timestep
+        aggregates_values = values_per_period[0][0 : int(forecast_timestep / timestep) + 1]
+        aggregates_values = (
+            aggregates_values + values_per_period[1][1 : int(forecast_timestep / timestep) + 1]
+        )
+        aggregates_values = aggregates_values + values_per_period[2][1:]
+
+        reference_values = aggregates_values
+
+        xml_path = os.path.join(output_base_path, Path("../timeseries_export.xml"))
+        values_to_be_compared = read_timeseries_data_from_xml(xml_path, "Q_orifice")
+
+        # Ensure we checked multiple periods
+        assert reference_values == values_to_be_compared
+
+    @patch("rtctools_interface.closed_loop.results_construction.pi.Timeseries")
+    @patch("rtctools_interface.closed_loop.results_construction.rtc.DataConfig")
+    @patch("rtctools_interface.closed_loop.results_construction.os.path.isfile")
+    def test_combine_xml_exports_skips_single_timestep_period(
+        self, isfile_mock, dataconfig_mock, timeseries_mock
+    ):
+        dataconfig = MagicMock()
+        dataconfig.pi_variable_ids.return_value = ("location", "parameter")
+        dataconfig_mock.return_value = dataconfig
+
+        ts_import_orig = MagicMock()
+        ts_import_orig.forecast_datetime = 0
+        ts_import_orig.start_datetime = 0
+        ts_import_orig.end_datetime = 2
+        ts_import_orig.times = [0, 1, 2]
+
+        ts_export = MagicMock()
+        ts_export.items.return_value = [("Q_orifice", None)]
+        ts_export.get.return_value = [1.0, 2.0, 3.0]
+
+        ts_export_step = MagicMock()
+        ts_export_step.times = [1]
+        ts_export_step.get.return_value = [9.0]
+
+        timeseries_mock.side_effect = [ts_import_orig, ts_export, ts_export_step]
+        isfile_mock.side_effect = [True, False]
+
+        combine_xml_exports(Path("/unused/output"), Path("/unused/input"))
+
+        ts_export.set.assert_not_called()
+        ts_export.write.assert_called_once_with(
+            output_folder=Path("/unused/output").parent, output_filename="timeseries_export"
+        )

rtc_tools_interface-0.10.2a4/tests/closed_loop/test_set_initial_values_from_previous_run.py

@@ -0,0 +1,61 @@
+"""Tests for set_initial_values_from_previous_run."""
+
+import datetime
+import unittest
+from unittest.mock import MagicMock
+
+import pytest
+
+from rtctools_interface.closed_loop.runner import set_initial_values_from_previous_run
+
+
+class TestSetInitialValuesFromPreviousRun(unittest.TestCase):
+    """set_initial_values_from_previous_run should fail loudly when no initial value is
+    available for a variable, rather than silently skipping it."""
+
+    def _make_timeseries(self, forecast_date):
+        timeseries = MagicMock()
+        timeseries.forecast_date = forecast_date
+        timeseries.is_set.return_value = False
+        return timeseries
+
+    def test_sets_initial_value_when_present(self):
+        forecast_date = datetime.datetime(2020, 1, 1)
+        timeseries = self._make_timeseries(forecast_date)
+
+        set_initial_values_from_previous_run(
+            results_previous_run={"var_a": [1.0, 2.0, 3.0]},
+            timeseries=timeseries,
+            previous_run_datetimes=[forecast_date],
+        )
+
+        timeseries.set_initial_value.assert_called_once_with("var_a", 1.0)
+
+    def test_raises_when_value_missing_for_variable(self):
+        """A variable with no value from either extract_results() or get_timeseries() must
+        be represented as None, not omitted, so the missing-value check still fires."""
+        forecast_date = datetime.datetime(2020, 1, 1)
+        timeseries = self._make_timeseries(forecast_date)
+
+        with pytest.raises(ValueError, match="Could not find initial value for var_a"):
+            set_initial_values_from_previous_run(
+                results_previous_run={"var_a": None},
+                timeseries=timeseries,
+                previous_run_datetimes=[forecast_date],
+            )
+
+    def test_absent_key_is_invisible_to_consumer(self):
+        """A key left out of results_previous_run entirely (instead of being set to None)
+        cannot be detected here: no error is raised and no initial value is set for it.
+        Producers must therefore always set missing values to None rather than omitting the
+        key, so this consumer's missing-value check can still fire."""
+        forecast_date = datetime.datetime(2020, 1, 1)
+        timeseries = self._make_timeseries(forecast_date)
+
+        set_initial_values_from_previous_run(
+            results_previous_run={},
+            timeseries=timeseries,
+            previous_run_datetimes=[forecast_date],
+        )
+
+        timeseries.set_initial_value.assert_not_called()

rtc_tools_interface-0.10.2a4/tests/optimization/test_active_constraint_helpers.py

@@ -0,0 +1,87 @@
+"""Tests for active constraint helper functions."""
+
+import importlib.util
+import sys
+import types
+import unittest
+from pathlib import Path
+
+import numpy as np
+
+
+def _load_active_constraint_helpers():
+    """Load active constraint helpers with runtime dependencies stubbed."""
+    module_names = [
+        "casadi",
+        "rtctools",
+        "rtctools.optimization",
+        "rtctools.optimization.timeseries",
+        "rtctools_interface",
+        "rtctools_interface.utils",
+        "rtctools_interface.utils.type_definitions",
+    ]
+    previous_modules = {name: sys.modules.get(name) for name in module_names}
+
+    casadi = types.ModuleType("casadi")
+    casadi.veccat = lambda *values: values
+    casadi.DM = types.SimpleNamespace(zeros=lambda size: np.zeros(size))
+
+    timeseries = types.ModuleType("rtctools.optimization.timeseries")
+    timeseries.Timeseries = type("Timeseries", (), {})
+
+    type_definitions = types.ModuleType("rtctools_interface.utils.type_definitions")
+    type_definitions.PreviousGoalConstraintRow = dict
+
+    sys.modules["casadi"] = casadi
+    sys.modules["rtctools"] = types.ModuleType("rtctools")
+    sys.modules["rtctools.optimization"] = types.ModuleType("rtctools.optimization")
+    sys.modules["rtctools.optimization.timeseries"] = timeseries
+    sys.modules["rtctools_interface"] = types.ModuleType("rtctools_interface")
+    sys.modules["rtctools_interface.utils"] = types.ModuleType("rtctools_interface.utils")
+    sys.modules["rtctools_interface.utils.type_definitions"] = type_definitions
+
+    try:
+        helper_path = (
+            Path(__file__).parents[2]
+            / "rtctools_interface"
+            / "optimization"
+            / "active_constraint_helpers.py"
+        )
+        spec = importlib.util.spec_from_file_location(
+            "active_constraint_helpers_under_test", helper_path
+        )
+        module = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(module)
+    finally:
+        for name, previous_module in previous_modules.items():
+            if previous_module is None:
+                sys.modules.pop(name, None)
+            else:
+                sys.modules[name] = previous_module
+
+    return module
+
+
+active_constraint_helpers = _load_active_constraint_helpers()
+
+
+class TestActiveConstraintHelpers(unittest.TestCase):
+    """Test active constraint helper functions."""
+
+    def test_format_active_times_maps_vector_components_to_timesteps(self):
+        """Map flattened vector-valued path-goal components to their timestep."""
+        times = np.array([10.0, 20.0, 30.0])
+        active_indices = np.array([1, 2, 4])
+
+        self.assertEqual(
+            active_constraint_helpers.format_active_times(times, active_indices, values_size=6),
+            "10.0;20.0;30.0",
+        )
+
+    def test_format_active_times_rejects_uneven_component_count(self):
+        """Reject path-goal values that cannot be evenly mapped to timesteps."""
+        times = np.array([10.0, 20.0, 30.0])
+        active_indices = np.array([0])
+
+        with self.assertRaisesRegex(ValueError, "not divisible"):
+            active_constraint_helpers.format_active_times(times, active_indices, values_size=5)

rtc_tools_interface-0.10.2a4/tests/optimization/test_active_constraint_mixin.py

@@ -0,0 +1,89 @@
+"""Tests for the active constraint mixin."""
+
+import shutil
+import unittest
+from pathlib import Path
+
+import pandas as pd
+
+from rtctools_interface.optimization.active_constraint_mixin import ActiveConstraintMixin
+from rtctools_interface.optimization.base_optimization_problem import BaseOptimizationProblem
+from tests.utils.get_test import get_test_data
+
+
+class ActiveConstraintProblem(ActiveConstraintMixin, BaseOptimizationProblem):
+    """Optimization problem with active constraint diagnostics enabled."""
+
+
+class TestActiveConstraintMixin(unittest.TestCase):
+    """Test active constraint CSV generation."""
+
+    def test_active_constraint_csv_files_are_written(self):
+        """Solve a goal-programming problem and inspect generated CSV files."""
+        test_data = get_test_data("basic", optimization=True)
+        output_folder = Path(test_data["output_folder"])
+        active_constraint_folder = output_folder / "active_constraints"
+        shutil.rmtree(active_constraint_folder, ignore_errors=True)
+
+        problem = ActiveConstraintProblem(
+            goal_table_file=test_data["goals_file"],
+            model_folder=test_data["model_folder"],
+            model_name=test_data["model_name"],
+            input_folder=test_data["model_input_folder"],
+            output_folder=output_folder,
+        )
+
+        self.assertTrue(problem.optimize())
+
+        previous_goal_file = active_constraint_folder / "active_constraints_of_previous_goals.csv"
+        self.assertFalse((active_constraint_folder / "active_constraints_by_priority.csv").exists())
+        self.assertFalse((active_constraint_folder / "previous_goal_constraints.csv").exists())
+        self.assertTrue(previous_goal_file.exists())
+
+        previous_goal_constraints = pd.read_csv(previous_goal_file)
+        self.assertNotIn("goal_id", previous_goal_constraints.columns)
+        self.assertNotIn("component_index", previous_goal_constraints.columns)
+        self.assertNotIn("time", previous_goal_constraints.columns)
+        self.assertNotIn("is_active", previous_goal_constraints.columns)
+        self.assertIn("active_times", previous_goal_constraints.columns)
+        self.assertIn("active_bound_value", previous_goal_constraints.columns)
+        self.assertNotIn("active_previous_goal_constraints", previous_goal_constraints.columns)
+        self.assertIn("active_previous_goals_constraints", previous_goal_constraints.columns)
+        self.assertEqual(set(previous_goal_constraints["priority"].to_list()), {10, 15, 20})
+        self.assertTrue((previous_goal_constraints["total_previous_goal_constraints"] >= 0).all())
+        self.assertTrue(
+            (
+                previous_goal_constraints["total_previous_goal_constraints"]
+                >= previous_goal_constraints["active_previous_goals_constraints"]
+            ).all()
+        )
+        self.assertEqual(
+            previous_goal_constraints[previous_goal_constraints["priority"] == 10][
+                "active_previous_goals_constraints"
+            ].iloc[0],
+            0,
+        )
+        self.assertEqual(
+            previous_goal_constraints[previous_goal_constraints["priority"] == 15][
+                "active_previous_goals_constraints"
+            ].iloc[0],
+            0,
+        )
+        self.assertTrue(
+            (
+                previous_goal_constraints[previous_goal_constraints["priority"] == 20][
+                    "active_previous_goals_constraints"
+                ]
+                > 0
+            ).all()
+        )
+        path_goal_constraints = previous_goal_constraints[
+            previous_goal_constraints["constraint_source"] == "path_goal"
+        ]
+        self.assertEqual(len(path_goal_constraints[path_goal_constraints["priority"] == 15]), 0)
+        self.assertEqual(len(path_goal_constraints[path_goal_constraints["priority"] == 20]), 2)
+        self.assertTrue(
+            path_goal_constraints[path_goal_constraints["priority"] == 20]["active_times"]
+            .notna()
+            .all()
+        )