emhass 0.10.0__tar.gz → 0.10.2__tar.gz

{emhass-0.10.0 → emhass-0.10.2}/CHANGELOG.md

@@ -1,8 +1,25 @@
 # Changelog
 
+## 0.10.2 - 2024-07-06
+### Improvement
+- Weather forecast caching and Solcast method fix by @GeoDerp
+- Added a new configuration parameter to control wether we compute PV curtailment or not
+- Added hybrid inverter to data publish
+- It is now possible to pass these battery parameters at runtime: `SOCmin`, `SOCmax`, `Pd_max` and `Pc_max`
+### Fix
+- Fixed problem with negative PV forecast values in optimization.py, by @GeoDerp
+
+## 0.10.1 - 2024-06-03
+### Fix
+- Fixed PV curtailment maximum possible value constraint
+- Added PV curtailement to variable to publish to HA
+
 ## 0.10.0 - 2024-06-02
+### BREAKING CHANGE
+- In this new version we have added support for PV curtailment computation. While doing this the nominal PV peak power is needed. The easiest way find this information is by directly using the `inverter_model` defined in the configuration. As this is needed in the optimization to correctly compute PV curtailment, this parameter need to be properly defined for your installation. Before this chage this parameter was only needed if using the PV forecast method `scrapper`, but now it is not optional as it is directly used in the optimization. 
+Use the dedicated webapp to find the correct model for your inverter, if you cannot find your exact brand/model then just pick an inverter with the same nominal power as yours: [https://emhass-pvlib-database.streamlit.app/](https://emhass-pvlib-database.streamlit.app/)
 ### Improvement
-- Added support for hybrid inverters
+- Added support for hybrid inverters and PV curtailment computation
 - Implemented a new `continual_publish` service that avoid the need of setting a special automation for data publish. Thanks to @GeoDerp
 - Implement a deferrable load start penalty functionality. Thanks to @werdnum
   - This feature also implement a `def_current_state` that can be passed at runtime to let the optimization consider that a deferrable load is currently scheduled or under operation when launching the optimization task

{emhass-0.10.0 → emhass-0.10.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: emhass
-Version: 0.10.0
+Version: 0.10.2
 Summary: An Energy Management System for Home Assistant
 Home-page: https://github.com/davidusb-geek/emhass
 Author: David HERNANDEZ
@@ -28,7 +28,7 @@ Requires-Dist: h5py==3.11.0
 Requires-Dist: pulp>=2.4
 Requires-Dist: pyyaml>=5.4.1
 Requires-Dist: tables<=3.9.1
-Requires-Dist: skforecast==0.12.0
+Requires-Dist: skforecast==0.12.1
 Requires-Dist: flask>=2.0.3
 Requires-Dist: waitress>=2.1.1
 Requires-Dist: plotly>=5.6.0
@@ -179,13 +179,9 @@ docker run -it --restart always -p 5000:5000  -e TZ="Europe/Paris"  -e LOCAL_COS
 ### Method 3) Legacy method using a Python virtual environment
 
 With this method it is recommended to install on a virtual environment.
-For this you will need `virtualenv`, install it using:
+Create and activate a virtual environment:
 ```bash
-sudo apt install python3-virtualenv
-```
-Then create and activate the virtual environment:
-```bash
-virtualenv -p /usr/bin/python3 emhassenv
+python3 -m venv emhassenv
 cd emhassenv
 source bin/activate
 ```
@@ -496,7 +492,7 @@ curl -i -H 'Content-Type:application/json' -X POST -d '{"publish_prefix":"all"}'
 ```
 This action will publish the dayahead (_dh) and MPC (_mpc) optimization results from the optimizations above.
 
-### Forecast data
+### Forecast data at runtime
 
 It is possible to provide EMHASS with your own forecast data. For this just add the data as list of values to a data dictionary during the call to `emhass` using the `runtimeparams` option. 
 
@@ -519,7 +515,7 @@ The possible dictionary keys to pass data are:
 
 - `prod_price_forecast` for the PV production selling price forecast.
 
-### Passing other data
+### Passing other data at runtime
 
 It is possible to also pass other data during runtime in order to automate the energy management. For example, it could be useful to dynamically update the total number of hours for each deferrable load (`def_total_hours`) using for instance a correlation with the outdoor temperature (useful for water heater for example). 
 
@@ -535,6 +531,8 @@ Here is the list of the other additional dictionary keys that can be passed at r
 
 - `def_end_timestep` for the timestep before which each deferrable load should operate (if you don't want the deferrable load to use the whole optimization timewindow).
 
+- `def_current_state` Pass this as a list of booleans (True/False) to indicate the current deferrable load state. This is used internally to avoid incorrectly penalizing a deferrable load start if a forecast is run when that load is already running.
+
 - `treat_def_as_semi_cont` to define if we should treat each deferrable load as a semi-continuous variable.
 
 - `set_def_constant` to define if we should set each deferrable load as a constant fixed value variable with just one startup for each optimization task.
@@ -545,8 +543,16 @@ Here is the list of the other additional dictionary keys that can be passed at r
 
 - `solar_forecast_kwp` for the PV peak installed power in kW used for the solar.forecast API call. 
 
+- `SOCmin` the minimum possible SOC.
+
+- `SOCmax` the maximum possible SOC.
+
 - `SOCtarget` for the desired target value of initial and final SOC.
 
+- `Pd_max` for the maximum battery discharge power.
+
+- `Pc_max` for the maximum battery charge power.
+
 - `publish_prefix` use this key to pass a common prefix to all published data. This will add a prefix to the sensor name but also to the forecasts attributes keys within the sensor.
 
 ## A naive Model Predictive Controller

{emhass-0.10.0 → emhass-0.10.2}/README.md

@@ -144,13 +144,9 @@ docker run -it --restart always -p 5000:5000  -e TZ="Europe/Paris"  -e LOCAL_COS
 ### Method 3) Legacy method using a Python virtual environment
 
 With this method it is recommended to install on a virtual environment.
-For this you will need `virtualenv`, install it using:
+Create and activate a virtual environment:
 ```bash
-sudo apt install python3-virtualenv
-```
-Then create and activate the virtual environment:
-```bash
-virtualenv -p /usr/bin/python3 emhassenv
+python3 -m venv emhassenv
 cd emhassenv
 source bin/activate
 ```
@@ -461,7 +457,7 @@ curl -i -H 'Content-Type:application/json' -X POST -d '{"publish_prefix":"all"}'
 ```
 This action will publish the dayahead (_dh) and MPC (_mpc) optimization results from the optimizations above.
 
-### Forecast data
+### Forecast data at runtime
 
 It is possible to provide EMHASS with your own forecast data. For this just add the data as list of values to a data dictionary during the call to `emhass` using the `runtimeparams` option. 
 
@@ -484,7 +480,7 @@ The possible dictionary keys to pass data are:
 
 - `prod_price_forecast` for the PV production selling price forecast.
 
-### Passing other data
+### Passing other data at runtime
 
 It is possible to also pass other data during runtime in order to automate the energy management. For example, it could be useful to dynamically update the total number of hours for each deferrable load (`def_total_hours`) using for instance a correlation with the outdoor temperature (useful for water heater for example). 
 
@@ -500,6 +496,8 @@ Here is the list of the other additional dictionary keys that can be passed at r
 
 - `def_end_timestep` for the timestep before which each deferrable load should operate (if you don't want the deferrable load to use the whole optimization timewindow).
 
+- `def_current_state` Pass this as a list of booleans (True/False) to indicate the current deferrable load state. This is used internally to avoid incorrectly penalizing a deferrable load start if a forecast is run when that load is already running.
+
 - `treat_def_as_semi_cont` to define if we should treat each deferrable load as a semi-continuous variable.
 
 - `set_def_constant` to define if we should set each deferrable load as a constant fixed value variable with just one startup for each optimization task.
@@ -510,8 +508,16 @@ Here is the list of the other additional dictionary keys that can be passed at r
 
 - `solar_forecast_kwp` for the PV peak installed power in kW used for the solar.forecast API call. 
 
+- `SOCmin` the minimum possible SOC.
+
+- `SOCmax` the maximum possible SOC.
+
 - `SOCtarget` for the desired target value of initial and final SOC.
 
+- `Pd_max` for the maximum battery discharge power.
+
+- `Pc_max` for the maximum battery charge power.
+
 - `publish_prefix` use this key to pass a common prefix to all published data. This will add a prefix to the sensor name but also to the forecasts attributes keys within the sensor.
 
 ## A naive Model Predictive Controller

emhass-0.10.2/data/opt_res_latest.csv

@@ -0,0 +1,49 @@
+timestamp,P_PV,P_Load,P_deferrable0,P_deferrable1,P_grid_pos,P_grid_neg,P_grid,P_PV_curtailment,unit_load_cost,unit_prod_price,cost_profit,cost_fun_profit,optim_status
+2024-06-03 23:00:00+02:00,0.0,2124.8911797752808,0.0,750.0,2874.8912,0.0,2874.8912,0.0,0.1419,0.065,-0.20397353064,-0.20397353064,Optimal
+2024-06-03 23:30:00+02:00,0.0,393.7693220338983,3000.0,750.0,4143.7693,0.0,4143.7693,0.0,0.1419,0.065,-0.29400043183499996,-0.29400043183499996,Optimal
+2024-06-04 00:00:00+02:00,0.0,329.5656571428571,0.0,750.0,1079.5657,0.0,1079.5657,0.0,0.1419,0.065,-0.076595186415,-0.076595186415,Optimal
+2024-06-04 00:30:00+02:00,0.0,214.95473988439306,0.0,750.0,964.95474,0.0,964.95474,0.0,0.1419,0.065,-0.068463538803,-0.068463538803,Optimal
+2024-06-04 01:00:00+02:00,0.0,254.92180790960455,3000.0,0.0,3254.9218,0.0,3254.9218,0.0,0.1419,0.065,-0.23093670171,-0.23093670171,Optimal
+2024-06-04 01:30:00+02:00,0.0,653.0385393258427,0.0,0.0,653.03854,0.0,653.03854,0.0,0.1419,0.065,-0.046333084413000006,-0.046333084413000006,Optimal
+2024-06-04 02:00:00+02:00,0.0,694.8668181818182,0.0,0.0,694.86682,0.0,694.86682,0.0,0.1419,0.065,-0.049300800879,-0.049300800879,Optimal
+2024-06-04 02:30:00+02:00,0.0,856.8446739130435,0.0,0.0,856.84467,0.0,856.84467,0.0,0.1419,0.065,-0.060793129336499996,-0.060793129336499996,Optimal
+2024-06-04 03:00:00+02:00,0.0,914.380597826087,0.0,0.0,914.3806,0.0,914.3806,0.0,0.1907,0.065,-0.08718619021000001,-0.08718619021000001,Optimal
+2024-06-04 03:30:00+02:00,0.0,599.8399421965318,0.0,0.0,599.83994,0.0,599.83994,0.0,0.1907,0.065,-0.057194738279,-0.057194738279,Optimal
+2024-06-04 04:00:00+02:00,0.0,703.5027607361963,0.0,0.0,703.50276,0.0,703.50276,0.0,0.1907,0.065,-0.067078988166,-0.067078988166,Optimal
+2024-06-04 04:30:00+02:00,0.0,646.7419879518072,0.0,0.0,646.74199,0.0,646.74199,0.0,0.1907,0.065,-0.061666848746500004,-0.061666848746500004,Optimal
+2024-06-04 05:00:00+02:00,0.0,1009.152816091954,0.0,0.0,1009.1528,0.0,1009.1528,0.0,0.1907,0.065,-0.09622271948,-0.09622271948,Optimal
+2024-06-04 05:30:00+02:00,0.0,967.1363841807911,0.0,0.0,967.13638,0.0,967.13638,0.0,0.1907,0.065,-0.092216453833,-0.092216453833,Optimal
+2024-06-04 06:00:00+02:00,0.0,935.1571508379889,0.0,0.0,935.15715,0.0,935.15715,0.0,0.1907,0.065,-0.0891672342525,-0.0891672342525,Optimal
+2024-06-04 06:30:00+02:00,60.0,3267.5106703910615,0.0,0.0,3207.5107,0.0,3207.5107,0.0,0.1907,0.065,-0.305836145245,-0.305836145245,Optimal
+2024-06-04 07:00:00+02:00,840.0,3286.2027777777776,0.0,0.0,2446.2028,0.0,2446.2028,0.0,0.1907,0.065,-0.23324543698000003,-0.23324543698000003,Optimal
+2024-06-04 07:30:00+02:00,660.0,1496.1914772727273,0.0,0.0,836.19148,0.0,836.19148,0.0,0.1907,0.065,-0.079730857618,-0.079730857618,Optimal
+2024-06-04 08:00:00+02:00,620.0,794.2991620111732,0.0,0.0,174.29916,0.0,174.29916,0.0,0.1907,0.065,-0.016619424906,-0.016619424906,Optimal
+2024-06-04 08:30:00+02:00,620.0,832.2424719101124,0.0,0.0,212.24247,0.0,212.24247,0.0,0.1907,0.065,-0.020237319514500002,-0.020237319514500002,Optimal
+2024-06-04 09:00:00+02:00,6380.0,788.9761235955057,3000.0,750.0,0.0,-1841.0239,-1841.0239,0.0,0.1907,0.065,0.05983327675,0.05983327675,Optimal
+2024-06-04 09:30:00+02:00,1095.5620000000001,781.2152298850575,0.0,750.0,435.65323,0.0,435.65323,0.0,0.1907,0.065,-0.0415395354805,-0.0415395354805,Optimal
+2024-06-04 10:00:00+02:00,811.4380000000002,664.0545197740113,0.0,0.0,0.0,-147.38348,-147.38348,0.0,0.1907,0.065,0.0047899631,0.0047899631,Optimal
+2024-06-04 10:30:00+02:00,681.0759999999999,666.1989265536723,0.0,0.0,0.0,-14.877073,-14.877073,0.0,0.1907,0.065,0.00048350487250000003,0.00048350487250000003,Optimal
+2024-06-04 11:00:00+02:00,671.9846666666667,669.4183146067417,0.0,0.0,0.0,-2.5663521,-2.5663521,0.0,0.1907,0.065,8.340644325000001e-05,8.340644325000001e-05,Optimal
+2024-06-04 11:30:00+02:00,6469.634666666666,579.2235294117647,3000.0,750.0,0.0,0.0,0.0,2140.4111,0.1907,-0.07,-0.0,-0.0,Optimal
+2024-06-04 12:00:00+02:00,2992.012,642.7344318181817,3000.0,0.0,650.72243,0.0,650.72243,0.0,0.1907,-0.07,-0.06204638370050001,-0.06204638370050001,Optimal
+2024-06-04 12:30:00+02:00,1867.9053333333331,637.1688636363637,0.0,750.0,0.0,0.0,0.0,480.73647,0.1907,-0.07,-0.0,-0.0,Optimal
+2024-06-04 13:00:00+02:00,2067.554666666667,649.3890173410405,3000.0,0.0,1581.8344,0.0,1581.8344,0.0,0.1907,-0.07,-0.15082791004,-0.15082791004,Optimal
+2024-06-04 13:30:00+02:00,622.756,509.79664739884396,0.0,0.0,0.0,0.0,0.0,112.95935,0.1907,-0.07,-0.0,-0.0,Optimal
+2024-06-04 14:00:00+02:00,1518.7553333333335,500.53686046511626,0.0,750.0,0.0,-268.21847,-268.21847,0.0,0.1907,0.065,0.008717100275000002,0.008717100275000002,Optimal
+2024-06-04 14:30:00+02:00,2551.502,520.944,3000.0,0.0,969.442,0.0,969.442,0.0,0.1907,0.065,-0.0924362947,-0.0924362947,Optimal
+2024-06-04 15:00:00+02:00,4160.0,415.32341040462427,3000.0,750.0,5.3234104,0.0,5.3234104,0.0,0.1907,0.065,-0.00050758718164,-0.00050758718164,Optimal
+2024-06-04 15:30:00+02:00,4240.0,321.6410285714286,3000.0,750.0,0.0,-168.35897,-168.35897,0.0,0.1419,0.065,0.005471666525,0.005471666525,Optimal
+2024-06-04 16:00:00+02:00,560.0,3778.0933888888894,3000.0,750.0,6968.0934,0.0,6968.0934,0.0,0.1419,0.065,-0.49438622673,-0.49438622673,Optimal
+2024-06-04 16:30:00+02:00,380.0,3990.816179775281,0.0,750.0,4360.8162,0.0,4360.8162,0.0,0.1419,0.065,-0.30939990939,-0.30939990939,Optimal
+2024-06-04 17:00:00+02:00,0.0,3909.8039890710384,0.0,750.0,4659.804,0.0,4659.804,0.0,0.1419,0.065,-0.33061309380000004,-0.33061309380000004,Optimal
+2024-06-04 17:30:00+02:00,0.0,4206.869447513812,0.0,0.0,4206.8694,0.0,4206.8694,0.0,0.1907,0.065,-0.40112499728999995,-0.40112499728999995,Optimal
+2024-06-04 18:00:00+02:00,0.0,486.26584269662925,0.0,0.0,486.26584,0.0,486.26584,0.0,0.1907,0.065,-0.046365447844000006,-0.046365447844000006,Optimal
+2024-06-04 18:30:00+02:00,0.0,402.43446927374305,0.0,0.0,402.43447,0.0,402.43447,0.0,0.1907,0.065,-0.0383721267145,-0.0383721267145,Optimal
+2024-06-04 19:00:00+02:00,0.0,316.17875,0.0,0.0,316.17875,0.0,316.17875,0.0,0.1907,0.065,-0.0301476438125,-0.0301476438125,Optimal
+2024-06-04 19:30:00+02:00,0.0,867.4,0.0,0.0,867.4,0.0,867.4,0.0,0.1907,0.065,-0.08270659000000001,-0.08270659000000001,Optimal
+2024-06-04 20:00:00+02:00,0.0,340.8070760233918,0.0,0.0,340.80708,0.0,340.80708,0.0,0.1907,0.065,-0.032495955078,-0.032495955078,Optimal
+2024-06-04 20:30:00+02:00,0.0,349.07406779661017,0.0,750.0,1099.0741,0.0,1099.0741,0.0,0.1419,0.065,-0.077979307395,-0.077979307395,Optimal
+2024-06-04 21:00:00+02:00,0.0,1790.5224581005587,0.0,750.0,2540.5225,0.0,2540.5225,0.0,0.1419,0.065,-0.18025007137500001,-0.18025007137500001,Optimal
+2024-06-04 21:30:00+02:00,0.0,2612.0882857142856,0.0,0.0,2612.0883,0.0,2612.0883,0.0,0.1419,0.065,-0.185327664885,-0.185327664885,Optimal
+2024-06-04 22:00:00+02:00,0.0,2617.098882681564,0.0,0.0,2617.0989,0.0,2617.0989,0.0,0.1419,0.065,-0.185683166955,-0.185683166955,Optimal
+2024-06-04 22:30:00+02:00,0.0,2254.2344375000002,0.0,0.0,2254.2344,0.0,2254.2344,0.0,0.1419,0.065,-0.15993793068,-0.15993793068,Optimal

{emhass-0.10.0 → emhass-0.10.2}/setup.py

@@ -19,7 +19,7 @@ long_description = (here / 'README.md').read_text(encoding='utf-8')
 
 setup(
     name='emhass',  # Required
-    version='0.10.0',  # Required
+    version='0.10.2',  # Required
     description='An Energy Management System for Home Assistant',  # Optional
     long_description=long_description,  # Optional
     long_description_content_type='text/markdown',  # Optional (see note above)
@@ -52,7 +52,7 @@ setup(
         'pulp>=2.4',
         'pyyaml>=5.4.1',
         'tables<=3.9.1',
-        'skforecast==0.12.0',
+        'skforecast==0.12.1',
         'flask>=2.0.3',
         'waitress>=2.1.1',
         'plotly>=5.6.0'

{emhass-0.10.0 → emhass-0.10.2}/src/emhass/command_line.py

@@ -52,7 +52,7 @@ def set_input_data_dict(emhass_conf: dict, costfun: str,
     logger.info("Setting up needed data")
     # Parsing yaml
     retrieve_hass_conf, optim_conf, plant_conf = utils.get_yaml_parse(
-        emhass_conf, use_secrets=not (get_data_from_file), params=params)
+        emhass_conf, use_secrets=not(get_data_from_file), params=params)
     # Treat runtimeparams
     params, retrieve_hass_conf, optim_conf, plant_conf = utils.treat_runtimeparams(
         runtimeparams, params, retrieve_hass_conf, optim_conf, plant_conf, set_type, logger)
@@ -97,6 +97,8 @@ def set_input_data_dict(emhass_conf: dict, costfun: str,
         # Get PV and load forecasts
         df_weather = fcst.get_weather_forecast(
             method=optim_conf["weather_forecast_method"])
+        if isinstance(df_weather, bool) and not df_weather:
+            return False
         P_PV_forecast = fcst.get_power_from_weather(df_weather)
         P_load_forecast = fcst.get_load_forecast(
             method=optim_conf['load_forecast_method'])
@@ -142,6 +144,8 @@ def set_input_data_dict(emhass_conf: dict, costfun: str,
         # Get PV and load forecasts
         df_weather = fcst.get_weather_forecast(
             method=optim_conf['weather_forecast_method'])
+        if isinstance(df_weather, bool) and not df_weather:
+            return False
         P_PV_forecast = fcst.get_power_from_weather(
             df_weather, set_mix_forecast=True, df_now=df_input_data)
         P_load_forecast = fcst.get_load_forecast(
@@ -243,6 +247,50 @@ def set_input_data_dict(emhass_conf: dict, costfun: str,
     }
     return input_data_dict
 
+def weather_forecast_cache(emhass_conf: dict, params: str, 
+                   runtimeparams: str, logger: logging.Logger) -> bool:
+    """
+    Perform a call to get forecast function, intend to save results to cache.
+
+    :param emhass_conf: Dictionary containing the needed emhass paths
+    :type emhass_conf: dict
+    :param params: Configuration parameters passed from data/options.json
+    :type params: str
+    :param runtimeparams: Runtime optimization parameters passed as a dictionary
+    :type runtimeparams: str
+    :param logger: The passed logger object
+    :type logger: logging object
+    :return: A bool for function completion
+    :rtype: bool
+
+    """
+    
+    # Parsing yaml
+    retrieve_hass_conf, optim_conf, plant_conf = utils.get_yaml_parse(
+        emhass_conf, use_secrets=True, params=params)
+    
+    # Treat runtimeparams
+    params, retrieve_hass_conf, optim_conf, plant_conf = utils.treat_runtimeparams(
+        runtimeparams, params, retrieve_hass_conf, optim_conf, plant_conf, "forecast", logger)
+    
+    # Make sure weather_forecast_cache is true
+    if (params != None) and (params != "null"):
+        params = json.loads(params)
+    else:
+        params = {}
+    params["passed_data"]["weather_forecast_cache"] = True
+    params = json.dumps(params)
+
+    # Create Forecast object
+    fcst = Forecast(retrieve_hass_conf, optim_conf, plant_conf,
+                params, emhass_conf, logger)
+
+    result = fcst.get_weather_forecast(optim_conf["weather_forecast_method"])
+    if isinstance(result, bool) and not result:
+        return False
+
+    return True
+
 
 def perfect_forecast_optim(input_data_dict: dict, logger: logging.Logger,
                            save_data_to_file: Optional[bool] = True, 
@@ -683,8 +731,6 @@ def regressor_model_predict(input_data_dict: dict, logger: logging.Logger,
             type_var="mlregressor")
     return prediction
 
-
-
 def publish_data(input_data_dict: dict, logger: logging.Logger, 
                  save_data_to_file: Optional[bool] = False, 
                  opt_res_latest: Optional[pd.DataFrame] = None, 
@@ -708,12 +754,10 @@ def publish_data(input_data_dict: dict, logger: logging.Logger,
 
     """
     logger.info("Publishing data to HASS instance")
-
     if not isinstance(input_data_dict["params"],dict):
         params = json.loads(input_data_dict["params"])
     else:
         params = input_data_dict["params"]
-    
     # Check if a day ahead optimization has been performed (read CSV file)
     if save_data_to_file:
         today = datetime.now(timezone.utc).replace(
@@ -726,7 +770,6 @@ def publish_data(input_data_dict: dict, logger: logging.Logger,
         opt_res_list_names = []
         publish_prefix = params["passed_data"]["publish_prefix"]
         entity_path = input_data_dict['emhass_conf']['data_path'] / "entities"
-        
         # Check if items in entity_path
         if os.path.exists(entity_path) and len(os.listdir(entity_path)) > 0:
             # Obtain all files in entity_path
@@ -805,6 +848,36 @@ def publish_data(input_data_dict: dict, logger: logging.Logger,
         dont_post=dont_post
     )
     cols_published = ["P_PV", "P_Load"]
+    # Publish PV curtailment
+    if input_data_dict["fcst"].plant_conf['compute_curtailment']:
+        custom_pv_curtailment_id = params["passed_data"]["custom_pv_curtailment_id"]
+        input_data_dict["rh"].post_data(
+            opt_res_latest["P_PV_curtailment"],
+            idx_closest,
+            custom_pv_curtailment_id["entity_id"],
+            custom_pv_curtailment_id["unit_of_measurement"],
+            custom_pv_curtailment_id["friendly_name"],
+            type_var="power",
+            publish_prefix=publish_prefix,
+            save_entities=entity_save,
+            dont_post=dont_post
+        )
+        cols_published = cols_published + ["P_PV_curtailment"]
+    # Publish P_hybrid_inverter
+    if input_data_dict["fcst"].plant_conf['inverter_is_hybrid']:
+        custom_hybrid_inverter_id = params["passed_data"]["custom_hybrid_inverter_id"]
+        input_data_dict["rh"].post_data(
+            opt_res_latest["P_hybrid_inverter"],
+            idx_closest,
+            custom_hybrid_inverter_id["entity_id"],
+            custom_hybrid_inverter_id["unit_of_measurement"],
+            custom_hybrid_inverter_id["friendly_name"],
+            type_var="power",
+            publish_prefix=publish_prefix,
+            save_entities=entity_save,
+            dont_post=dont_post
+        )
+        cols_published = cols_published + ["P_hybrid_inverter"]
     # Publish deferrable loads
     custom_deferrable_forecast_id = params["passed_data"][
         "custom_deferrable_forecast_id"
@@ -944,7 +1017,7 @@ def publish_data(input_data_dict: dict, logger: logging.Logger,
         opt_res_latest.index[idx_closest]]]
     return opt_res
 
-def continual_publish(input_data_dict,entity_path,logger):
+def continual_publish(input_data_dict: dict, entity_path: pathlib.Path, logger: logging.Logger):
     """
     If continual_publish is true and a entity file saved in /data_path/entities, continually publish sensor on freq rate, updating entity current state value based on timestamp
 
@@ -959,23 +1032,22 @@ def continual_publish(input_data_dict,entity_path,logger):
     logger.info("Continual publish thread service started")
     freq = input_data_dict['retrieve_hass_conf'].get("freq", pd.to_timedelta(1, "minutes"))
     entity_path_contents = []
-
     while True:
         # Sleep for x seconds (using current time as a reference for time left)
         time.sleep(max(0,freq.total_seconds() - (datetime.now(input_data_dict["retrieve_hass_conf"]["time_zone"]).timestamp() % 60)))
-
         # Loop through all saved entity files
         if os.path.exists(entity_path) and len(os.listdir(entity_path)) > 0:
             entity_path_contents =  os.listdir(entity_path)    
             for entity in entity_path_contents:
                 if entity != "metadata.json":
                     # Call publish_json with entity file, build entity, and publish                     
-                    publish_json(entity,input_data_dict,entity_path,logger,"continual_publish")
+                    publish_json(entity, input_data_dict, entity_path, logger, "continual_publish")
         pass 
     # This function should never return           
     return False 
 
-def publish_json(entity,input_data_dict,entity_path,logger,reference: Optional[str] = ""):
+def publish_json(entity: dict, input_data_dict: dict, entity_path: pathlib.Path, 
+                 logger: logging.Logger, reference: Optional[str] = ""):
     """
     Extract saved entity data from .json (in data_path/entities), build entity, post results to post_data
 
@@ -989,9 +1061,8 @@ def publish_json(entity,input_data_dict,entity_path,logger,reference: Optional[s
     :type logger: logging.Logger
     :param reference: String for identifying who ran the function  
     :type reference: str, optional
-
+    
     """
-
     # Retrieve entity metadata from file
     if os.path.isfile(entity_path / "metadata.json"):
         with open(entity_path / "metadata.json", "r") as file:
@@ -1001,16 +1072,12 @@ def publish_json(entity,input_data_dict,entity_path,logger,reference: Optional[s
     else:
         logger.error("unable to located metadata.json in:" + entity_path)
         return False            
-
     # Round current timecode (now)
     now_precise = datetime.now(input_data_dict["retrieve_hass_conf"]["time_zone"]).replace(second=0, microsecond=0)
-
     # Retrieve entity data from file
     entity_data = pd.read_json(entity_path / entity , orient='index')
-    
     # Remove ".json" from string for entity_id
     entity_id = entity.replace(".json", "")
-    
     # Adjust Dataframe from received entity json file
     entity_data.columns = [metadata[entity_id]["name"]]
     entity_data.index.name = "timestamp"
@@ -1025,15 +1092,13 @@ def publish_json(entity,input_data_dict,entity_path,logger,reference: Optional[s
         idx_closest = entity_data.index.get_indexer([now_precise], method="bfill")[0]
     if idx_closest == -1:
         idx_closest = entity_data.index.get_indexer([now_precise], method="nearest")[0]
-    
     # Call post data 
     if reference == "continual_publish":
         logger.debug("Auto Published sensor:")
         logger_levels = "DEBUG"
     else: 
         logger_levels = "INFO"
-    
-    #post/save entity
+    # post/save entity
     input_data_dict["rh"].post_data(
         data_df=entity_data[metadata[entity_id]["name"]],
         idx=idx_closest,
@@ -1125,7 +1190,7 @@ def main():
         logger.error("Could not find emhass/src foulder in: " + str(root_path))
         logger.error("Try setting emhass root path with --root")
         return False
-    # Additionnal argument
+    # Additional argument
     try:
         parser.add_argument(
             "--version",

{emhass-0.10.0 → emhass-0.10.2}/src/emhass/forecast.py

@@ -182,6 +182,7 @@ class Forecast(object):
         
         """
         csv_path  = self.emhass_conf['data_path'] / csv_path
+        w_forecast_cache_path = os.path.abspath(self.emhass_conf['data_path'] / "weather_forecast_data.pkl")
 
         self.logger.info("Retrieving weather forecast data using method = "+method)
         self.weather_forecast_method = method # Saving this attribute for later use to identify csv method usage
@@ -223,40 +224,97 @@ class Forecast(object):
             data['relative_humidity'] = raw_data['Relative Humidity (%)']
             data['precipitable_water'] = pvlib.atmosphere.gueymard94_pw(
                 data['temp_air'], data['relative_humidity'])
-        elif method == 'solcast': # using solcast API
-            # Retrieve data from the solcast API
-            if 'solcast_api_key' not in self.retrieve_hass_conf:
-                self.logger.warning("The solcast_api_key parameter was not defined, using dummy values for testing")
-                self.retrieve_hass_conf['solcast_api_key'] = "123456"
-            if 'solcast_rooftop_id' not in self.retrieve_hass_conf:
-                self.logger.warning("The solcast_rooftop_id parameter was not defined, using dummy values for testing")
-                self.retrieve_hass_conf['solcast_rooftop_id'] = "123456"
-            headers = {
-                "Authorization": "Bearer " + self.retrieve_hass_conf['solcast_api_key'],
-                "content-type": "application/json",
-                }
-            days_solcast = int(len(self.forecast_dates)*self.freq.seconds/3600)
-            url = "https://api.solcast.com.au/rooftop_sites/"+self.retrieve_hass_conf['solcast_rooftop_id']+"/forecasts?hours="+str(days_solcast)
-            response = get(url, headers=headers)
-            '''import bz2 # Uncomment to save a serialized data for tests
-            import _pickle as cPickle
-            with bz2.BZ2File("data/test_response_solcast_get_method.pbz2", "w") as f: 
-                cPickle.dump(response, f)'''
-            data = response.json()
-            data_list = []
-            for elm in data['forecasts']:
-                data_list.append(elm['pv_estimate']*1000) # Converting kW to W
-            # Check if the retrieved data has the correct length
-            if len(data_list) < len(self.forecast_dates):
-                self.logger.error("Not enough data retrived from SolCast service, try increasing the time step or use MPC")
+        elif method == 'solcast': # using Solcast API
+            # Check if weather_forecast_cache is true or if forecast_data file does not exist
+            if self.params["passed_data"]["weather_forecast_cache"] or not os.path.isfile(w_forecast_cache_path):
+                # Check if weather_forecast_cache_only is true, if so produce error for not finding cache file
+                if not self.params["passed_data"]["weather_forecast_cache_only"]:
+                    # Retrieve data from the Solcast API
+                    if 'solcast_api_key' not in self.retrieve_hass_conf:
+                        self.logger.error("The solcast_api_key parameter was not defined")
+                        return False
+                    if 'solcast_rooftop_id' not in self.retrieve_hass_conf:
+                        self.logger.error("The solcast_rooftop_id parameter was not defined")
+                        return False
+                    headers = {
+                        'User-Agent': 'EMHASS',
+                        "Authorization": "Bearer " + self.retrieve_hass_conf['solcast_api_key'],
+                        "content-type": "application/json",
+                        }
+                    days_solcast = int(len(self.forecast_dates)*self.freq.seconds/3600)
+                    # If weather_forecast_cache, set request days as twice as long to avoid length issues (add a buffer) 
+                    if self.params["passed_data"]["weather_forecast_cache"]:
+                        days_solcast = min((days_solcast * 2), 336)
+                    url = "https://api.solcast.com.au/rooftop_sites/"+self.retrieve_hass_conf['solcast_rooftop_id']+"/forecasts?hours="+str(days_solcast)
+                    response = get(url, headers=headers)
+                    '''import bz2 # Uncomment to save a serialized data for tests
+                    import _pickle as cPickle
+                    with bz2.BZ2File("data/test_response_solcast_get_method.pbz2", "w") as f: 
+                        cPickle.dump(response, f)'''
+                    # Verify the request passed
+                    if int(response.status_code) == 200:
+                        data = response.json()
+                    elif int(response.status_code) == 402 or int(response.status_code) == 429:
+                        self.logger.error("Solcast error: May have exceeded your subscription limit.")
+                        return False  
+                    elif int(response.status_code) >= 400 or int(response.status_code) >= 202:
+                        self.logger.error("Solcast error: There was a issue with the solcast request, check solcast API key and rooftop ID.")
+                        self.logger.error("Solcast error: Check that your subscription is valid and your network can connect to Solcast.")
+                        return False  
+                    data_list = []
+                    for elm in data['forecasts']:
+                        data_list.append(elm['pv_estimate']*1000) # Converting kW to W
+                    # Check if the retrieved data has the correct length
+                    if len(data_list) < len(self.forecast_dates):
+                        self.logger.error("Not enough data retried from Solcast service, try increasing the time step or use MPC.")
+                    else:
+                        # If runtime weather_forecast_cache is true save forecast result to file as cache
+                        if self.params["passed_data"]["weather_forecast_cache"]:
+                            # Add x2 forecast periods for cached results. This adds a extra delta_forecast amount of days for a buffer
+                            cached_forecast_dates =  self.forecast_dates.union(pd.date_range(self.forecast_dates[-1], periods=(len(self.forecast_dates) +1), freq=self.freq)[1:])
+                            cache_data_list = data_list[0:len(cached_forecast_dates)]
+                            cache_data_dict = {'ts':cached_forecast_dates, 'yhat':cache_data_list}
+                            data_cache = pd.DataFrame.from_dict(cache_data_dict)
+                            data_cache.set_index('ts', inplace=True)
+                            with open(w_forecast_cache_path, "wb") as file:    
+                                cPickle.dump(data_cache, file)
+                            if not os.path.isfile(w_forecast_cache_path):
+                                self.logger.warning("Solcast forecast data could not be saved to file.")
+                            else:
+                                self.logger.info("Saved the Solcast results to cache, for later reference.")    
+                        # Trim request results to forecast_dates        
+                        data_list = data_list[0:len(self.forecast_dates)]
+                        data_dict = {'ts':self.forecast_dates, 'yhat':data_list}
+                        # Define DataFrame
+                        data = pd.DataFrame.from_dict(data_dict)
+                        # Define index
+                        data.set_index('ts', inplace=True)
+            # Else, notify user to update cache
+                else:
+                    self.logger.error("Unable to obtain Solcast cache file.")
+                    self.logger.error("Try running optimization again with 'weather_forecast_cache_only': false")
+                    self.logger.error("Optionally, obtain new Solcast cache with runtime parameter 'weather_forecast_cache': true in an optimization, or run the `forecast-cache` action, to pull new data from Solcast and cache.")
+                    return False
+            # Else, open stored weather_forecast_data.pkl file for previous forecast data (cached data)
             else:
-                # Ensure correct length
-                data_list = data_list[0:len(self.forecast_dates)]
-                # Define DataFrame
-                data_dict = {'ts':self.forecast_dates, 'yhat':data_list}
-                data = pd.DataFrame.from_dict(data_dict)
-                # Define index
-                data.set_index('ts', inplace=True)
+                with open(w_forecast_cache_path, "rb") as file:
+                    data = cPickle.load(file)
+                    if not isinstance(data, pd.DataFrame) or len(data) < len(self.forecast_dates):
+                        self.logger.error("There has been a error obtaining cached Solcast forecast data.")
+                        self.logger.error("Try running optimization again with 'weather_forecast_cache': true, or run action `forecast-cache`, to pull new data from Solcast and cache.")
+                        self.logger.warning("Removing old Solcast cache file. Next optimization will pull data from Solcast, unless 'weather_forecast_cache_only': true")
+                        os.remove(w_forecast_cache_path)
+                        return False
+                    # Filter cached forecast data to match current forecast_dates start-end range (reduce forecast Dataframe size to appropriate length)
+                    if self.forecast_dates[0] in data.index and self.forecast_dates[-1] in data.index:
+                        data = data.loc[self.forecast_dates[0]:self.forecast_dates[-1]]
+                        self.logger.info("Retrieved Solcast data from the previously saved cache.")
+                    else:
+                        self.logger.error("Unable to obtain cached Solcast forecast data within the requested timeframe range.")
+                        self.logger.error("Try running optimization again (not using cache). Optionally, add runtime parameter 'weather_forecast_cache': true to pull new data from Solcast and cache.")
+                        self.logger.warning("Removing old Solcast cache file. Next optimization will pull data from Solcast, unless 'weather_forecast_cache_only': true")
+                        os.remove(w_forecast_cache_path)
+                        return False    
         elif method == 'solar.forecast': # using the solar.forecast API
             # Retrieve data from the solar.forecast API
             if 'solar_forecast_kwp' not in self.retrieve_hass_conf:
@@ -423,9 +481,9 @@ class Forecast(object):
                 # Setting the main parameters of the PV plant
                 location = Location(latitude=self.lat, longitude=self.lon)
                 temp_params = TEMPERATURE_MODEL_PARAMETERS['sapm']['close_mount_glass_glass']
-                cec_modules = bz2.BZ2File(self.emhass_conf['root_path'] / 'data/cec_modules.pbz2', "rb")
+                cec_modules = bz2.BZ2File(self.emhass_conf['root_path'] / 'data' / 'cec_modules.pbz2', "rb")
                 cec_modules = cPickle.load(cec_modules)
-                cec_inverters = bz2.BZ2File(self.emhass_conf['root_path'] / 'data/cec_inverters.pbz2', "rb")
+                cec_inverters = bz2.BZ2File(self.emhass_conf['root_path'] / 'data' /  'cec_inverters.pbz2', "rb")
                 cec_inverters = cPickle.load(cec_inverters)
                 if type(self.plant_conf['module_model']) == list:
                     P_PV_forecast = pd.Series(0, index=df_weather.index)
@@ -838,5 +896,4 @@ class Forecast(object):
         else:
             self.logger.error("Passed method is not valid")
             return False
-        return df_final
-    
+        return df_final