py-ewr 2.2.5__tar.gz → 2.2.6__tar.gz

{py_ewr-2.2.5 → py_ewr-2.2.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: py_ewr
-Version: 2.2.5
+Version: 2.2.6
 Summary: Environmental Water Requirement calculator
 Home-page: https://github.com/MDBAuth/EWR_tool
 Author: Martin Job
@@ -12,34 +12,38 @@ Classifier: Operating System :: OS Independent
 Classifier: Development Status :: 4 - Beta
 Classifier: Programming Language :: Python
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Framework :: Pytest
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ipython==8.8.0
-Requires-Dist: ipywidgets==7.7.0
-Requires-Dist: pandas==2.0.3
-Requires-Dist: requests==2.25.1
+Requires-Dist: pandas>2
+Requires-Dist: requests>2
 Requires-Dist: mdba-gauge-getter==0.5.1
-Requires-Dist: cachetools==5.2.0
-Requires-Dist: xarray==2023.01.0
-Requires-Dist: netCDF4==1.6.4
-Requires-Dist: numpy<2
+Requires-Dist: cachetools>5
+Requires-Dist: xarray
+Requires-Dist: h5py
+Requires-Dist: netCDF4
+Requires-Dist: numpy
 
 [![CI](https://github.com/MDBAuth/EWR_tool/actions/workflows/test-release.yml/badge.svg)]()
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/py-ewr)](https://pypi.org/project/py-ewr/)
 [![PyPI](https://img.shields.io/pypi/v/py-ewr)](https://pypi.org/project/py-ewr/)
 [![DOI](https://zenodo.org/badge/342122359.svg)](https://zenodo.org/badge/latestdoi/342122359)
 
-### **EWR tool version 2.2.5 README**
+### **EWR tool version 2.2.6 README**
 
 ### **Notes on recent version updates**
+- Including draft objective mapping files in the package (see below sub heading **Objective mapping** for more information). Objective mapping has been therefore pulled out of the parameter sheet
+- Including an example parallel processing script for running the EWR tool
+- Adding handling for cases where there are single MDBA bigmod site IDs mapping to multiple different gauges
+- Fix SDL resource unit mapping in the parameter sheet
+- Adding lat and lon to the parameter sheet
 - ten thousand year handling - this has been brought back online.
 - Remove TQDM loading bars
-- Handle duplicate sites in MDBA siteID file - where a duplicate exists, the first match is used and the rest are skipped over
 - Adding new model format handling - 'IQQM - netcdf'
 - Standard time-series handling added - each column needs a gauge, followed by and underscore, followed by either flow or level (e.g. 409025_flow). This handling also has missing date filling - so any missing dates will be filled with NaN values in all columns.
 - bug fixes: spells of length equal to the minimum required spell length were getting filtered out of the successful events table and successful interevents table, fixed misclassification of some gauges to flow, level, and lake level categories
@@ -47,7 +51,7 @@ Requires-Dist: numpy<2
 
 ### **Installation**
 
-Note - requires Python 3.8 or newer
+Note - requires Python 3.9 to 3.13 (inclusive)
 
 Step 1. 
 Upgrade pip
@@ -109,7 +113,30 @@ all_successful_interEvents = ewr_oh.get_all_successful_interEvents()
 ### Option 2: Running model scenarios through the EWR tool
 
 1. Tell the tool where the model files are (can either be local or in a remote location)
-2. Tell the tool what format the model files are in (Current model format options: 'Bigmod - MDBA', 'Source - NSW (res.csv)', 'Standard time-series' - see manual for formatting requirements)
+2. Tell the tool what format the model files are in. The current model format options are: 
+    - 'Bigmod - MDBA'
+        Bigmod formatted outputs
+    - 'Source - NSW (res.csv)'
+        Source res.csv formatted outputs
+    - 'Standard time-series'
+        The first column header should be *Date* with the date values in the YYYY-MM-DD format.
+        The next columns should have the *gauge* followed by *_* followed by either *flow* or *level*
+        E.g.
+        | Date | 409025_flow | 409025_level | 414203_flow |
+        | --- | --- | --- | --- |
+        | 1895-07-01 | 8505 | 5.25 | 8500 |
+        | 1895-07-02 | 8510 | 5.26 | 8505 |
+
+    - 'ten thousand year'
+        This has the same formatting requirements as the 'Standard time-series'. This can handle ten thousand years worth of hydrology data.
+        The first column header should be *Date* with the date values in the YYYY-MM-DD format.
+        The next columns should have the *gauge* followed by *_* followed by either *flow* or *level*
+        E.g.
+        | Date | 409025_flow | 409025_level | 414203_flow |
+        | --- | --- | --- | --- |
+        | 105-07-01 | 8505 | 5.25 | 8500 |
+        | 105-07-02 | 8510 | 5.26 | 8505 |
+
 
 ```python
 #USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
@@ -199,11 +226,11 @@ For issues relating to the script, a tutorial, or feedback please contact Lara P
 
 
 **Disclaimer**
-Every effort has been taken to ensure the EWR database represents the original EWRs from state long term water plans as best as possible, and that the code within this tool has been developed to interpret and analyse these EWRs in an accurate way. However, there may still be unresolved bugs in the EWR parameter sheet and/or EWR tool. Please report any bugs to the issues tab under the GitHub project so we can investigate further. 
+Every effort has been taken to ensure the EWR database represents the original EWRs from state Long Term Water Plans (LTWPs) and Environmental Water Management Plans (EWMPs) as best as possible, and that the code within this tool has been developed to interpret and analyse these EWRs in an accurate way. However, there may still be unresolved bugs in the EWR parameter sheet and/or EWR tool. Please report any bugs to the issues tab under the GitHub project so we can investigate further. 
 
 
 **Notes on development of the dataset of EWRs**
-The MDBA has worked with Basin state representatives to ensure scientific integrity of EWRs has been maintained when translating from raw EWRs in the Basin state Long Term Water Plans (LTWPs) to the machine readable format found in the parameter sheet within this tool. 
+The MDBA has worked with Basin state representatives to ensure scientific integrity of EWRs has been maintained when translating from raw EWRs in the Basin state LTWPs and EWMPs to the machine readable format found in the parameter sheet within this tool. 
 
 **Compatibility**
 
@@ -226,11 +253,8 @@ NSW:
 
 Consult the user manual for instructions on how to run the tool. Please email the above email addresses for a copy of the user manual.
 
-To disable progress bars, as for example when running remote scripted runs, use 
-
-``` python
-import os
-os.environ["TQDM_DISABLE"] = "1"
-```
-*before* importing py-ewr in your script.
-
+**Objective mapping**
+Objective mapping csv files are now included in the EWR tool package. Currently this objective mapping is in an early draft format. The objective mapping will be finalised after consultation with relevant state representatives. The files are intended to be used together to link EWRs to the detailed objectives, theme level targets and specific goals. The three sheets are located in the py_ewr/parameter_metadata folder:
+- ewr2obj.csv: For each planning unit, gauge, ewr combination there are either one or many env_obj codes. These env_obj codes come under one of five different theme level targets (Native Fish, Native vegetation, Waterbirds, Other species or Ecosystem functions)
+- obj2target.csv: env_obj's are unique to their planning unit in the LTWP (noting there are often a lot of similarities between env_obj's in the same states). The plain english wording of the env objectives is also contained in this csv. The LTWP, planning unit and env_obj rows are repeated for each specific goal related to that LTWP, planning unit and env_obj. 
+- obj2yrtarget.csv: The environmental objectives are related to 5, 10 and 20 year targets

{py_ewr-2.2.5 → py_ewr-2.2.6}/README.md

@@ -3,12 +3,16 @@
 [![PyPI](https://img.shields.io/pypi/v/py-ewr)](https://pypi.org/project/py-ewr/)
 [![DOI](https://zenodo.org/badge/342122359.svg)](https://zenodo.org/badge/latestdoi/342122359)
 
-### **EWR tool version 2.2.5 README**
+### **EWR tool version 2.2.6 README**
 
 ### **Notes on recent version updates**
+- Including draft objective mapping files in the package (see below sub heading **Objective mapping** for more information). Objective mapping has been therefore pulled out of the parameter sheet
+- Including an example parallel processing script for running the EWR tool
+- Adding handling for cases where there are single MDBA bigmod site IDs mapping to multiple different gauges
+- Fix SDL resource unit mapping in the parameter sheet
+- Adding lat and lon to the parameter sheet
 - ten thousand year handling - this has been brought back online.
 - Remove TQDM loading bars
-- Handle duplicate sites in MDBA siteID file - where a duplicate exists, the first match is used and the rest are skipped over
 - Adding new model format handling - 'IQQM - netcdf'
 - Standard time-series handling added - each column needs a gauge, followed by and underscore, followed by either flow or level (e.g. 409025_flow). This handling also has missing date filling - so any missing dates will be filled with NaN values in all columns.
 - bug fixes: spells of length equal to the minimum required spell length were getting filtered out of the successful events table and successful interevents table, fixed misclassification of some gauges to flow, level, and lake level categories
@@ -16,7 +20,7 @@
 
 ### **Installation**
 
-Note - requires Python 3.8 or newer
+Note - requires Python 3.9 to 3.13 (inclusive)
 
 Step 1. 
 Upgrade pip
@@ -78,7 +82,30 @@ all_successful_interEvents = ewr_oh.get_all_successful_interEvents()
 ### Option 2: Running model scenarios through the EWR tool
 
 1. Tell the tool where the model files are (can either be local or in a remote location)
-2. Tell the tool what format the model files are in (Current model format options: 'Bigmod - MDBA', 'Source - NSW (res.csv)', 'Standard time-series' - see manual for formatting requirements)
+2. Tell the tool what format the model files are in. The current model format options are: 
+    - 'Bigmod - MDBA'
+        Bigmod formatted outputs
+    - 'Source - NSW (res.csv)'
+        Source res.csv formatted outputs
+    - 'Standard time-series'
+        The first column header should be *Date* with the date values in the YYYY-MM-DD format.
+        The next columns should have the *gauge* followed by *_* followed by either *flow* or *level*
+        E.g.
+        | Date | 409025_flow | 409025_level | 414203_flow |
+        | --- | --- | --- | --- |
+        | 1895-07-01 | 8505 | 5.25 | 8500 |
+        | 1895-07-02 | 8510 | 5.26 | 8505 |
+
+    - 'ten thousand year'
+        This has the same formatting requirements as the 'Standard time-series'. This can handle ten thousand years worth of hydrology data.
+        The first column header should be *Date* with the date values in the YYYY-MM-DD format.
+        The next columns should have the *gauge* followed by *_* followed by either *flow* or *level*
+        E.g.
+        | Date | 409025_flow | 409025_level | 414203_flow |
+        | --- | --- | --- | --- |
+        | 105-07-01 | 8505 | 5.25 | 8500 |
+        | 105-07-02 | 8510 | 5.26 | 8505 |
+
 
 ```python
 #USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
@@ -168,11 +195,11 @@ For issues relating to the script, a tutorial, or feedback please contact Lara P
 
 
 **Disclaimer**
-Every effort has been taken to ensure the EWR database represents the original EWRs from state long term water plans as best as possible, and that the code within this tool has been developed to interpret and analyse these EWRs in an accurate way. However, there may still be unresolved bugs in the EWR parameter sheet and/or EWR tool. Please report any bugs to the issues tab under the GitHub project so we can investigate further. 
+Every effort has been taken to ensure the EWR database represents the original EWRs from state Long Term Water Plans (LTWPs) and Environmental Water Management Plans (EWMPs) as best as possible, and that the code within this tool has been developed to interpret and analyse these EWRs in an accurate way. However, there may still be unresolved bugs in the EWR parameter sheet and/or EWR tool. Please report any bugs to the issues tab under the GitHub project so we can investigate further. 
 
 
 **Notes on development of the dataset of EWRs**
-The MDBA has worked with Basin state representatives to ensure scientific integrity of EWRs has been maintained when translating from raw EWRs in the Basin state Long Term Water Plans (LTWPs) to the machine readable format found in the parameter sheet within this tool. 
+The MDBA has worked with Basin state representatives to ensure scientific integrity of EWRs has been maintained when translating from raw EWRs in the Basin state LTWPs and EWMPs to the machine readable format found in the parameter sheet within this tool. 
 
 **Compatibility**
 
@@ -195,11 +222,8 @@ NSW:
 
 Consult the user manual for instructions on how to run the tool. Please email the above email addresses for a copy of the user manual.
 
-To disable progress bars, as for example when running remote scripted runs, use 
-
-``` python
-import os
-os.environ["TQDM_DISABLE"] = "1"
-```
-*before* importing py-ewr in your script.
-
+**Objective mapping**
+Objective mapping csv files are now included in the EWR tool package. Currently this objective mapping is in an early draft format. The objective mapping will be finalised after consultation with relevant state representatives. The files are intended to be used together to link EWRs to the detailed objectives, theme level targets and specific goals. The three sheets are located in the py_ewr/parameter_metadata folder:
+- ewr2obj.csv: For each planning unit, gauge, ewr combination there are either one or many env_obj codes. These env_obj codes come under one of five different theme level targets (Native Fish, Native vegetation, Waterbirds, Other species or Ecosystem functions)
+- obj2target.csv: env_obj's are unique to their planning unit in the LTWP (noting there are often a lot of similarities between env_obj's in the same states). The plain english wording of the env objectives is also contained in this csv. The LTWP, planning unit and env_obj rows are repeated for each specific goal related to that LTWP, planning unit and env_obj. 
+- obj2yrtarget.csv: The environmental objectives are related to 5, 10 and 20 year targets

{py_ewr-2.2.5 → py_ewr-2.2.6}/py_ewr/data_inputs.py

@@ -121,7 +121,7 @@ def get_MDBA_codes() -> pd.DataFrame:
         pd.DataFrame: dataframe for linking MDBA model nodes to gauges
 
     '''
-    metadata = pd.read_csv( BASE_PATH / 'model_metadata/SiteID_MDBA.csv', engine = 'python', dtype=str, encoding='windows-1252')
+    metadata = pd.read_csv( BASE_PATH / 'model_metadata/SiteID_MDBA.csv', engine = 'python', dtype=str)#, encoding='windows-1252')
 
     return metadata
   
@@ -392,12 +392,12 @@ def get_gauges(category: str, ewr_table_path: str = None) -> set:
     multi_gauges = get_multi_gauges('gauges')
     multi_gauges = list(multi_gauges.values())
     if category == 'all gauges':
-        return set(EWR_table['Gauge'].to_list() + menindee_gauges + wp_gauges + multi_gauges)
+        return set(EWR_table['Gauge'].to_list()+menindee_gauges+wp_gauges+multi_gauges+flow_barrage_gauges+level_barrage_gauges+qld_flow_gauges+qld_level_gauges+vic_level_gauges)
     elif category == 'flow gauges':
         return set(EWR_table['Gauge'].to_list() + multi_gauges + flow_barrage_gauges + qld_flow_gauges) 
     elif category == 'level gauges':
         level_gauges = EWR_table[EWR_table['FlowLevelVolume']=='L']['Gauge'].to_list()
-        return set(menindee_gauges + wp_gauges + level_barrage_gauges + qld_level_gauges + level_gauges)
+        return set(menindee_gauges + wp_gauges + level_barrage_gauges + qld_level_gauges + level_gauges + vic_level_gauges)
     else:
         raise ValueError('''No gauge category sent to the "get_gauges" function''')
     
@@ -439,4 +439,18 @@ def gauge_groups(parameter_sheet: pd.DataFrame) -> dict:
 
     return flow_gauges, level_gauges, lake_level_gauges
 
-# def gauges_to_measurand()
+# def gauges_to_measurand()
+
+def get_causal_ewr() -> dict:
+
+    ewr2obj_path = os.path.join(BASE_PATH, "parameter_metadata/ewr2obj.csv")
+    obj2target_path = os.path.join(BASE_PATH, "parameter_metadata/obj2target.csv")
+    obj2yrtarget_path = os.path.join(BASE_PATH, "parameter_metadata/obj2yrtarget.csv")
+
+    causal_ewr = {
+        "ewr2obj": pd.read_csv(ewr2obj_path),
+        "obj2target": pd.read_csv(obj2target_path),
+        "obj2yrtarget":pd.read_csv(obj2yrtarget_path)
+    }
+
+    return causal_ewr

{py_ewr-2.2.5 → py_ewr-2.2.6}/py_ewr/evaluate_EWRs.py

@@ -4859,7 +4859,7 @@ def event_stats(df:pd.DataFrame, PU_df:pd.DataFrame, gauge:str, EWR:str, EWR_inf
         years_with_events = get_event_years_volume_achieved(events, unique_water_years)
 
     YWE = pd.Series(name = str(EWR + '_eventYears'), data = years_with_events, index = unique_water_years)
-    PU_df = pd.concat([PU_df, YWE], axis = 1)
+    # PU_df = pd.concat([PU_df, YWE], axis = 1)
     # Number of event achievements:
     num_event_achievements = get_achievements(EWR_info, events, unique_water_years, durations)
 
@@ -4867,64 +4867,84 @@ def event_stats(df:pd.DataFrame, PU_df:pd.DataFrame, gauge:str, EWR:str, EWR_inf
         num_event_achievements = get_achievements_connecting_events(events, unique_water_years)
 
     NEA = pd.Series(name = str(EWR + '_numAchieved'), data= num_event_achievements, index = unique_water_years)
-    PU_df = pd.concat([PU_df, NEA], axis = 1)
+    # PU_df = pd.concat([PU_df, NEA], axis = 1)
     # Total number of events THIS ONE IS ONLY ACHIEVED due to Filter Applied
     num_events = get_number_events(EWR_info, events, unique_water_years, durations)
     NE = pd.Series(name = str(EWR + '_numEvents'), data= num_events, index = unique_water_years)
-    PU_df = pd.concat([PU_df, NE], axis = 1)
+    # PU_df = pd.concat([PU_df, NE], axis = 1)
     # Total number of events THIS ONE IS ALL EVENTS
     num_events_all = get_all_events(events)
     NEALL = pd.Series(name = str(EWR + '_numEventsAll'), data= num_events_all, index = unique_water_years)
-    PU_df = pd.concat([PU_df, NEALL], axis = 1)
+    # PU_df = pd.concat([PU_df, NEALL], axis = 1)
     # Max inter event period
     max_inter_period = get_max_inter_event_days(no_events, unique_water_years)
     MIP = pd.Series(name = str(EWR + '_maxInterEventDays'), data= max_inter_period, index = unique_water_years)
-    PU_df = pd.concat([PU_df, MIP], axis = 1)
+    # PU_df = pd.concat([PU_df, MIP], axis = 1)
     # Max inter event period achieved
     max_inter_period_achieved = get_event_max_inter_event_achieved(EWR_info, no_events, unique_water_years)
     MIPA = pd.Series(name = str(EWR + '_maxInterEventDaysAchieved'), data= max_inter_period_achieved, index = unique_water_years)
-    PU_df = pd.concat([PU_df, MIPA], axis = 1)
+    # PU_df = pd.concat([PU_df, MIPA], axis = 1)
     # Average length of events
     av_length = get_average_event_length(events, unique_water_years)
     AL = pd.Series(name = str(EWR + '_eventLength'), data = av_length, index = unique_water_years)
-    PU_df = pd.concat([PU_df, AL], axis = 1)
+    # PU_df = pd.concat([PU_df, AL], axis = 1)
     # Average length of events ONLY the ACHIEVED
     av_length_achieved = get_average_event_length_achieved(EWR_info, events)
     ALA = pd.Series(name = str(EWR + '_eventLengthAchieved' ), data = av_length_achieved, index = unique_water_years)
-    PU_df = pd.concat([PU_df, ALA], axis = 1)
+    # PU_df = pd.concat([PU_df, ALA], axis = 1)
     # Total event days
     total_days = get_total_days(events, unique_water_years)
-    TD = pd.Series(name = str(EWR + '_totalEventDays'), data = total_days, index = unique_water_years)
-    PU_df = pd.concat([PU_df, TD], axis = 1)
+    TD_A = pd.Series(name = str(EWR + '_totalEventDays'), data = total_days, index = unique_water_years)
+    # PU_df = pd.concat([PU_df, TD], axis = 1)
     # Total event days ACHIEVED
     total_days_achieved = get_achieved_event_days(EWR_info, events)
     TDA = pd.Series(name = str(EWR + '_totalEventDaysAchieved'), data = total_days_achieved, index = unique_water_years)
-    PU_df = pd.concat([PU_df, TDA], axis = 1)
+    # PU_df = pd.concat([PU_df, TDA], axis = 1)
     # Max event days
     max_days = get_max_event_days(events, unique_water_years)
     MD = pd.Series(name = str(EWR + '_maxEventDays'), data = max_days, index = unique_water_years)
-    PU_df = pd.concat([PU_df, MD], axis = 1)
+    # PU_df = pd.concat([PU_df, MD], axis = 1)
     # Max rolling consecutive event days
     try:
         max_consecutive_days = get_max_consecutive_event_days(events, unique_water_years)
         MR = pd.Series(name = str(EWR + '_maxRollingEvents'), data = max_consecutive_days, index = unique_water_years)
-        PU_df = pd.concat([PU_df, MR], axis = 1)
+        # PU_df = pd.concat([PU_df, MR], axis = 1)
     except Exception as e:
         max_consecutive_days = [0]*len(unique_water_years)
         MR = pd.Series(name = str(EWR + '_maxRollingEvents'), data = max_consecutive_days, index = unique_water_years)
-        PU_df = pd.concat([PU_df, MR], axis = 1)
+        # PU_df = pd.concat([PU_df, MR], axis = 1)
         log.error(e)
     # Max rolling duration achieved
     achieved_max_rolling_duration = get_max_rolling_duration_achievement(durations, max_consecutive_days)
     MRA = pd.Series(name = str(EWR + '_maxRollingAchievement'), data = achieved_max_rolling_duration, index = unique_water_years)
-    PU_df = pd.concat([PU_df, MRA], axis = 1)
+    # PU_df = pd.concat([PU_df, MRA], axis = 1)
     # Append information around available and missing data:
     yearly_gap = get_data_gap(df, water_years, gauge)
     total_days = get_total_series_days(water_years)
     YG = pd.Series(name = str(EWR + '_missingDays'), data = yearly_gap, index = unique_water_years)
-    TD = pd.Series(name = str(EWR + '_totalPossibleDays'), data = total_days, index = unique_water_years)
-    PU_df = pd.concat([PU_df, YG], axis = 1)
-    PU_df = pd.concat([PU_df, TD], axis = 1)
+    TD_B = pd.Series(name = str(EWR + '_totalPossibleDays'), data = total_days, index = unique_water_years)
+    # PU_df = pd.concat([PU_df, YG], axis = 1)
+    # PU_df = pd.concat([PU_df, TD], axis = 1)
+    PU_df = pd.concat(
+        [PU_df,
+         YWE,
+         NEA,
+         NE,
+         NEALL,
+         MIP,
+         MIPA,
+         AL,
+         ALA,
+         TD_A,
+         TDA,
+         MD,
+         MR,
+         MRA,
+         YG,
+         TD_B
+         ],
+        axis=1
+    )
     
     return PU_df