py-ewr 2.3.7__tar.gz → 2.3.9__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. {py_ewr-2.3.7 → py_ewr-2.3.9}/PKG-INFO +90 -55
  2. {py_ewr-2.3.7 → py_ewr-2.3.9}/README.md +87 -53
  3. py_ewr-2.3.9/py_ewr/data_inputs.py +584 -0
  4. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/evaluate_EWRs.py +529 -1149
  5. py_ewr-2.3.9/py_ewr/model_metadata/EWR_Sitelist_FIRM_20250718.csv +255 -0
  6. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/observed_handling.py +13 -13
  7. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/parameter_metadata/ewr_calc_config.json +66 -5
  8. py_ewr-2.3.9/py_ewr/parameter_metadata/objective_reference.csv +776 -0
  9. py_ewr-2.3.9/py_ewr/parameter_metadata/parameter_sheet.csv +3325 -0
  10. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/scenario_handling.py +166 -43
  11. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/summarise_results.py +76 -88
  12. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr.egg-info/PKG-INFO +90 -55
  13. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr.egg-info/SOURCES.txt +2 -3
  14. {py_ewr-2.3.7 → py_ewr-2.3.9}/setup.py +1 -1
  15. {py_ewr-2.3.7 → py_ewr-2.3.9}/tests/test_data_inputs.py +3 -16
  16. {py_ewr-2.3.7 → py_ewr-2.3.9}/tests/test_evaluate_ewr_rest.py +177 -648
  17. {py_ewr-2.3.7 → py_ewr-2.3.9}/tests/test_evaluate_ewrs.py +86 -290
  18. {py_ewr-2.3.7 → py_ewr-2.3.9}/tests/test_observed_handling.py +3 -3
  19. {py_ewr-2.3.7 → py_ewr-2.3.9}/tests/test_scenario_handling.py +21 -16
  20. {py_ewr-2.3.7 → py_ewr-2.3.9}/tests/test_summarise_results.py +30 -31
  21. py_ewr-2.3.7/py_ewr/data_inputs.py +0 -456
  22. py_ewr-2.3.7/py_ewr/parameter_metadata/ewr2obj.csv +0 -43331
  23. py_ewr-2.3.7/py_ewr/parameter_metadata/obj2target.csv +0 -7941
  24. py_ewr-2.3.7/py_ewr/parameter_metadata/obj2yrtarget.csv +0 -106
  25. py_ewr-2.3.7/py_ewr/parameter_metadata/parameter_sheet.csv +0 -3445
  26. {py_ewr-2.3.7 → py_ewr-2.3.9}/LICENSE +0 -0
  27. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/__init__.py +0 -0
  28. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/io.py +0 -0
  29. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/model_metadata/SiteID_MDBA.csv +0 -0
  30. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/model_metadata/SiteID_NSW.csv +0 -0
  31. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr/model_metadata/iqqm_stations.csv +0 -0
  32. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr.egg-info/dependency_links.txt +0 -0
  33. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr.egg-info/requires.txt +0 -0
  34. {py_ewr-2.3.7 → py_ewr-2.3.9}/py_ewr.egg-info/top_level.txt +0 -0
  35. {py_ewr-2.3.7 → py_ewr-2.3.9}/pyproject.toml +0 -0
  36. {py_ewr-2.3.7 → py_ewr-2.3.9}/setup.cfg +0 -0
@@ -1,6 +1,6 @@
1
- Metadata-Version: 2.2
1
+ Metadata-Version: 2.4
2
2
  Name: py_ewr
3
- Version: 2.3.7
3
+ Version: 2.3.9
4
4
  Summary: Environmental Water Requirement calculator
5
5
  Home-page: https://github.com/MDBAuth/EWR_tool
6
6
  Author: Martin Job
@@ -34,6 +34,7 @@ Dynamic: classifier
34
34
  Dynamic: description
35
35
  Dynamic: description-content-type
36
36
  Dynamic: home-page
37
+ Dynamic: license-file
37
38
  Dynamic: project-url
38
39
  Dynamic: requires-dist
39
40
  Dynamic: summary
@@ -43,23 +44,29 @@ Dynamic: summary
43
44
  [![PyPI](https://img.shields.io/pypi/v/py-ewr)](https://pypi.org/project/py-ewr/)
44
45
  [![DOI](https://zenodo.org/badge/342122359.svg)](https://zenodo.org/badge/latestdoi/342122359)
45
46
 
46
- ### **EWR tool version 2.3.7 README**
47
+ ### **ewr tool version 2.3.9 README**
47
48
 
48
49
  ### **Notes on recent version updates**
49
- - Including metadata report (this is still being ironed out and tested)
50
- - CLLMM_c and CLLMM_d ewrs are now able to be calculated without all barrage level gauges being present in the model file.
51
- - Including draft objective mapping files in the package (see below sub heading **Objective mapping** for more information). Objective mapping has been therefore pulled out of the parameter sheet
52
- - Including an example parallel processing script for running the EWR tool
50
+
51
+ #### EWR handling and outputs
53
52
  - Adding handling for cases where there are single MDBA bigmod site IDs mapping to multiple different gauges
54
- - Fix SDL resource unit mapping in the parameter sheet
55
- - Adding lat and lon to the parameter sheet
56
- - ten thousand year handling - this has been brought back online.
57
- - Remove TQDM loading bars
58
- - Adding new model format handling - 'IQQM - netcdf'
59
- - Standard time-series handling added - each column needs a gauge, followed by and underscore, followed by either flow or level (e.g. 409025_flow). This handling also has missing date filling - so any missing dates will be filled with NaN values in all columns.
60
- - bug fixes: spells of length equal to the minimum required spell length were getting filtered out of the successful events table and successful interevents table, fixed misclassification of some gauges to flow, level, and lake level categories
61
53
  - New EWRs: New Qld EWRs - SF_FD and BF_FD used to look into the FD EWRs in closer detail.
62
54
  - Adding state and Surface Water SDL (SWSDL) to py-ewr output tables
55
+ - Including metadata report (this is still being ironed out and tested)
56
+ - New handling capabilities for FIRM model formated files
57
+
58
+ #### Model metadata
59
+ - Added new FIRM ID file mapping FIRM ID to gauge number.
60
+
61
+ #### parameter metadata
62
+ - Updated parameter sheet and objective mapping correcting some mismatched links between environmental objectives and EWRs
63
+ - Fix SDL resource unit mapping in the parameter sheet
64
+ - Adding lat and lon to the parameter sheet
65
+ - Added in model handling for FIRM model outputs
66
+ - Various minor variable renamings for consistency
67
+ - Renamed MaxLevelRise to MaxLevelChange
68
+ - Removed AnnualFlowSum and MinLevelRise column from parameter sheet
69
+ - New format of objective mapping includes the adding of objective mapping back into the parameter sheet and one secondary dataframe (objective_reference.csv) in parameter metadata.
63
70
 
64
71
  ### **Installation**
65
72
 
@@ -77,13 +84,14 @@ pip install py-ewr
77
84
  ```
78
85
 
79
86
  ### Option 1: Running the observed mode of the tool
80
- The EWR tool will use a second program called gauge getter to first download the river data at the locations and dates selected and then run this through the EWR tool
87
+ The ewr tool will use a second program called gauge getter to first download the river data at the locations and dates selected and then run this through the ewr tool.
88
+ For more information please visit the [MDBA Gauge Getter](https://github.com/MDBAuth/MDBA_Gauge_Getter) github page.
81
89
 
82
90
  ```python
83
91
 
84
92
  from datetime import datetime
85
93
 
86
- #USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
94
+ # USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
87
95
 
88
96
  dates = {'start_date': datetime(YYYY, 7, 1),
89
97
  'end_date': datetime(YYYY, 6, 30)}
@@ -98,15 +106,15 @@ gauges = ['Gauge1', 'Gauge2']
98
106
 
99
107
  from py_ewr.observed_handling import ObservedHandler
100
108
 
101
- # Running the EWR tool:
109
+ # Running the ewr tool:
102
110
  ewr_oh = ObservedHandler(gauges=gauges, dates=dates)
103
111
 
104
112
  # Generating tables:
105
- # Table 1: Summarised EWR results for the entire timeseries
113
+ # Table 1: Summarised ewr results for the entire timeseries
106
114
  ewr_results = ewr_oh.get_ewr_results()
107
115
 
108
- # Table 2: Summarised EWR results, aggregated to water years:
109
- yearly_ewr_results = ewr_oh.get_yearly_ewr_results()
116
+ # Table 2: Summarised ewr results, aggregated to water years:
117
+ yearly_results = ewr_oh.get_yearly_results()
110
118
 
111
119
  # Table 3: All events details regardless of duration
112
120
  all_events = ewr_oh.get_all_events()
@@ -122,14 +130,23 @@ all_successful_interEvents = ewr_oh.get_all_successful_interEvents()
122
130
 
123
131
  ```
124
132
 
125
- ### Option 2: Running model scenarios through the EWR tool
133
+ ### Option 2: Running model scenarios through the ewr tool
126
134
 
127
- 1. Tell the tool where the model files are (can either be local or in a remote location)
135
+ 1. Tell the tool where the model files are (can either be local or in a remote location). For each scenario list the file path of each file. Python will default to the path relative to the directory it is being run from, but if you put a full file path you can choose any file.
136
+ ```python
137
+ scenarios = {'Scenario1': ['path/to/file1', 'path/to/file2', 'path/to/file3'],
138
+ 'Scenario2': ['path/to/file1', 'path/to/file2', 'path/to/file3']}
139
+ ```
128
140
  2. Tell the tool what format the model files are in. The current model format options are:
129
141
  - 'Bigmod - MDBA'
130
142
  Bigmod formatted outputs
143
+
131
144
  - 'Source - NSW (res.csv)'
132
145
  Source res.csv formatted outputs
146
+
147
+ - 'FIRM - MDBA'
148
+ FIRM ID formatted outputs
149
+
133
150
  - 'Standard time-series'
134
151
  The first column header should be *Date* with the date values in the YYYY-MM-DD format.
135
152
  The next columns should have the *gauge* followed by *_* followed by either *flow* or *level*
@@ -151,23 +168,31 @@ all_successful_interEvents = ewr_oh.get_all_successful_interEvents()
151
168
 
152
169
 
153
170
  ```python
154
- #USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
171
+ # USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
155
172
 
156
173
  # Minimum 1 scenario and 1 related file required
157
- scenarios = {'Scenario1': ['file/location/1', 'file/location/2', 'file/location/3'],
158
- 'Scenario2': ['file/location/1', 'file/location/2', 'file/location/3']}
174
+ scenarios = {'Scenario1': ['path/to/file1', 'path/to/file2', 'path/to/file3'],
175
+ 'Scenario2': ['path/to/file1', 'path/to/file2', 'path/to/file3']}
159
176
 
160
177
  model_format = 'Bigmod - MDBA'
161
178
 
179
+ #----- other model formats include -----#
180
+ # 'Bigmod - MDBA'
181
+ # 'Source - NSW (res.csv)'
182
+ # 'FIRM - MDBA'
183
+ # 'Standard time-series'
184
+ # 'ten thousand year'
185
+
162
186
  # END USER INPUT<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
163
187
 
164
188
  ```
189
+ File names will be incorporated into the Scenario column of the ewr tool output tables for tracebility of ewr tool output and corresponding model file. In the code written below, scenario name will be appendeded to the file name. eg. Scenario_name1_all_results.csv, therefore it is suggested to have informative scenario names that can traceback to your model files as well.
165
190
 
166
191
  ``` python
167
192
  from py_ewr.scenario_handling import ScenarioHandler
168
193
  import pandas as pd
169
194
 
170
- ewr_results_dict = {}
195
+ summary_results_dict = {}
171
196
  yearly_results_dict = {}
172
197
  all_events_dict = {}
173
198
  all_interEvents_dict = {}
@@ -175,25 +200,25 @@ all_successful_Events_dict = {}
175
200
  all_successful_interEvents_dict = {}
176
201
 
177
202
  for scenario_name, scenario_list in scenarios.items():
178
- ewr_results = pd.DataFrame()
179
- yearly_ewr_results = pd.DataFrame()
203
+ summary_results = pd.DataFrame()
204
+ yearly_results = pd.DataFrame()
180
205
  all_events = pd.DataFrame()
181
206
  all_interEvents = pd.DataFrame()
182
207
  all_successful_Events = pd.DataFrame()
183
208
  all_successful_interEvents = pd.DataFrame()
184
209
  for file in scenarios[scenario_name]:
185
210
 
186
- # Running the EWR tool:
211
+ # Running the ewr tool:
187
212
  ewr_sh = ScenarioHandler(scenario_file = file,
188
213
  model_format = model_format)
189
214
 
190
215
  # Return each table and stitch the different files of the same scenario together:
191
- # Table 1: Summarised EWR results for the entire timeseries
192
- temp_ewr_results = ewr_sh.get_ewr_results()
193
- ewr_results = pd.concat([ewr_results, temp_ewr_results], axis = 0)
194
- # Table 2: Summarised EWR results, aggregated to water years:
195
- temp_yearly_ewr_results = ewr_sh.get_yearly_ewr_results()
196
- yearly_ewr_results = pd.concat([yearly_ewr_results, temp_yearly_ewr_results], axis = 0)
216
+ # Table 1: Summarised ewr results for the entire timeseries
217
+ temp_summary_results = ewr_sh.get_ewr_results()
218
+ summary_results = pd.concat([summary_results, temp_summary_results], axis = 0)
219
+ # Table 2: Summarised ewr results, aggregated to water years:
220
+ temp_yearly_results = ewr_sh.get_yearly_ewr_results()
221
+ yearly_results = pd.concat([yearly_results, temp_yearly_results], axis = 0)
197
222
  # Table 3: All events details regardless of duration
198
223
  temp_all_events = ewr_sh.get_all_events()
199
224
  all_events = pd.concat([all_events, temp_all_events], axis = 0)
@@ -209,64 +234,74 @@ for scenario_name, scenario_list in scenarios.items():
209
234
 
210
235
 
211
236
  # Optional code to output results to csv files:
212
- ewr_results.to_csv(scenario_name + 'all_results.csv')
213
- yearly_ewr_results.to_csv(scenario_name + 'yearly_ewr_results.csv')
237
+ summary_results.to_csv(scenario_name + 'summary_results.csv')
238
+ yearly_results.to_csv(scenario_name + 'yearly_results.csv')
214
239
  all_events.to_csv(scenario_name + 'all_events.csv')
215
240
  all_interEvents.to_csv(scenario_name + 'all_interevents.csv')
216
241
  all_successful_Events.to_csv(scenario_name + 'all_successful_Events.csv')
217
242
  all_successful_interEvents.to_csv(scenario_name + 'all_successful_interEvents.csv')
218
243
 
219
244
  # Save the final tables to the dictionaries:
220
- ewr_results_dict[scenario_name] = ewr_results
221
- yearly_results_dict[scenario_name] = yearly_ewr_results
222
- all_events_dict[scenario_name] = all_events_dict
245
+ summary_results_dict[scenario_name] = summary_results
246
+ yearly_results_dict[scenario_name] = yearly_results
247
+ all_events_dict[scenario_name] = all_events
223
248
  all_interEvents_dict[scenario_name] = all_interEvents
224
249
  all_successful_Events_dict[scenario_name] = all_successful_Events
225
250
  all_successful_interEvents_dict[scenario_name] = all_successful_interEvents
226
251
 
227
252
 
228
253
  ```
229
-
254
+ #### Optional arguments for ScenarioHandler
255
+ ```python
256
+ ewr_sh = ScenarioHandler(scenario_file = file,
257
+ model_format = model_format,
258
+ parameter_sheet = parameter_sheet,
259
+ calc_config_path = calc_config_path)
260
+ ```
261
+ You may add a custom parameter sheet and or calc_config_file to your EWR tool run using the ```parameter_sheet``` and ```calc_config_path``` arguments. These arguments take a string file path pointing to files. Please check this ewr_calc_config.json file found in parameter metadata to see if any EWRs in your custom parameter sheet are not represented in the calc_config_file. For an EWR to be calculated, it must be found in both calc_config.json and the parameter sheet.
230
262
 
231
263
  ### **Purpose**
232
264
  This tool has two purposes:
233
- 1. Operational: Tracking EWR success at gauges of interest in real time - option 1 above.
234
- 2. Planning: Comparing EWR success between scenarios (i.e. model runs) - option 2 above.
265
+ 1. Operational: Tracking ewr success at gauges of interest in real time - option 1 above.
266
+ 2. Planning: Comparing ewr success between scenarios (i.e. model runs) - option 2 above.
235
267
 
236
268
  **Support**
237
- For issues relating to the script, a tutorial, or feedback please contact Lara Palmer at lara.palmer@mdba.gov.au, Martin Job at martin.job@mdba.gov.au, or Joel Bailey at joel.bailey@mdba.gov.au
269
+ For issues relating to the script, a tutorial, or feedback please contact Martin Job at martin.job@mdba.gov.au, Sirous Safari Pour at sirous.safaripour@mdba.gov.au, Elisha Freedman at elisha.freedman@mdba.gov.au, Joel Bailey at joel.bailey@mdba.gov.au, or Lara Palmer at lara.palmer@mdba.gov.au.
238
270
 
239
271
 
240
272
  **Disclaimer**
241
- Every effort has been taken to ensure the EWR database represents the original EWRs from state Long Term Water Plans (LTWPs) and Environmental Water Management Plans (EWMPs) as best as possible, and that the code within this tool has been developed to interpret and analyse these EWRs in an accurate way. However, there may still be unresolved bugs in the EWR parameter sheet and/or EWR tool. Please report any bugs to the issues tab under the GitHub project so we can investigate further.
273
+ Every effort has been taken to ensure the ewr database represents the original EWRs from state Long Term Water Plans (LTWPs) and Environmental Water Management Plans (EWMPs) as best as possible, and that the code within this tool has been developed to interpret and analyse these EWRs in an accurate way. However, there may still be unresolved bugs in the ewr parameter sheet and/or ewr tool. Please report any bugs to the issues tab under the GitHub project so we can investigate further.
242
274
 
243
275
 
244
276
  **Notes on development of the dataset of EWRs**
245
277
  The MDBA has worked with Basin state representatives to ensure scientific integrity of EWRs has been maintained when translating from raw EWRs in the Basin state LTWPs and EWMPs to the machine readable format found in the parameter sheet within this tool.
246
278
 
279
+ Environmental Water Requirements (EWRs) in the tool are subject to change when the relevant documents including Long Term Water Plans (LTWPs) and Environmental Water Management Plans (EWMPs) are updated or move from draft to final versions. LTWPs that are currently in draft form include the ACT and the upper Murrumbidgee section of the NSW Murrumbidgee LTWP.
280
+
247
281
  **Compatibility**
248
282
 
249
- NSW:
250
283
  - All Queensland catchments
251
284
  - All New South Wales catchments
252
285
  - All South Australian catchments
253
286
  - All EWRs from river based Environmental Water Management Plans (EWMPs) in Victoria*
287
+ - All EWRs for ACT (draft version)
254
288
 
255
- *Currently the wetland EWMPS and mixed wetland-river EWMPs in Victoria contain EWRs that cannot be evaluated by an automated EWR tool so the EWRs from these plans have been left out for now. The MDBA will work with our Victorian colleagues to ensure any updated EWRs in these plans are integrated into the tool where possible.
289
+ *The wetland EWMPS and mixed wetland-river EWMPs in Victoria are in progress. The MDBA will work with our Victorian colleagues to ensure any updated EWRs in these plans are integrated into the tool where possible.
256
290
 
257
291
  **Input data**
258
292
 
259
293
  - Gauge data from the relevant Basin state websites and the Bureau of Meteorology website
260
294
  - Scenario data input by the user
261
295
  - Model metadata for location association between gauge ID's and model nodes
262
- - EWR parameter sheet
296
+ #### optional
297
+ - ewr parameter sheet
298
+ - calc_config.json
263
299
 
264
- **Running the tool**
300
+ **Objective mapping**
301
+ The objective mapping is located in the EWR tool package. This is intended to be used to link EWRs to the detailed objectives, theme level targets and specific goals. The objective reference file is located in the py_ewr/parameter_metadata folder:
265
302
 
266
- Consult the user manual for instructions on how to run the tool. Please email the above email addresses for a copy of the user manual.
303
+ parameter_sheet.csv (EnvObj column)
304
+ obj_reference.csv
267
305
 
268
- **Objective mapping**
269
- Objective mapping csv files are now included in the EWR tool package. Currently this objective mapping is in an early draft format. The objective mapping will be finalised after consultation with relevant state representatives. The files are intended to be used together to link EWRs to the detailed objectives, theme level targets and specific goals. The three sheets are located in the py_ewr/parameter_metadata folder:
270
- - ewr2obj.csv: For each planning unit, gauge, ewr combination there are either one or many env_obj codes. These env_obj codes come under one of five different theme level targets (Native Fish, Native vegetation, Waterbirds, Other species or Ecosystem functions)
271
- - obj2target.csv: env_obj's are unique to their planning unit in the LTWP (noting there are often a lot of similarities between env_obj's in the same states). The plain english wording of the env objectives is also contained in this csv. The LTWP, planning unit and env_obj rows are repeated for each specific goal related to that LTWP, planning unit and env_obj.
272
- - obj2yrtarget.csv: The environmental objectives are related to 5, 10 and 20 year targets
306
+ Contains the individual environmnetal objectives listed in the 'EnvObj' column of the parameter sheet and their ecological targets (Target) and plain english description of objectives (Objectives) for each planning unit, long term water plan (LTWPShortName), and surface water sustainable diversion limit (SWSDLName).
307
+ the function ```get_obj_mapping()``` is available to automatically merge the information from obj_reference.csv with the parameter sheet to link these objectives with their specific ewr_codes.
@@ -3,23 +3,29 @@
3
3
  [![PyPI](https://img.shields.io/pypi/v/py-ewr)](https://pypi.org/project/py-ewr/)
4
4
  [![DOI](https://zenodo.org/badge/342122359.svg)](https://zenodo.org/badge/latestdoi/342122359)
5
5
 
6
- ### **EWR tool version 2.3.7 README**
6
+ ### **ewr tool version 2.3.9 README**
7
7
 
8
8
  ### **Notes on recent version updates**
9
- - Including metadata report (this is still being ironed out and tested)
10
- - CLLMM_c and CLLMM_d ewrs are now able to be calculated without all barrage level gauges being present in the model file.
11
- - Including draft objective mapping files in the package (see below sub heading **Objective mapping** for more information). Objective mapping has been therefore pulled out of the parameter sheet
12
- - Including an example parallel processing script for running the EWR tool
9
+
10
+ #### EWR handling and outputs
13
11
  - Adding handling for cases where there are single MDBA bigmod site IDs mapping to multiple different gauges
14
- - Fix SDL resource unit mapping in the parameter sheet
15
- - Adding lat and lon to the parameter sheet
16
- - ten thousand year handling - this has been brought back online.
17
- - Remove TQDM loading bars
18
- - Adding new model format handling - 'IQQM - netcdf'
19
- - Standard time-series handling added - each column needs a gauge, followed by and underscore, followed by either flow or level (e.g. 409025_flow). This handling also has missing date filling - so any missing dates will be filled with NaN values in all columns.
20
- - bug fixes: spells of length equal to the minimum required spell length were getting filtered out of the successful events table and successful interevents table, fixed misclassification of some gauges to flow, level, and lake level categories
21
12
  - New EWRs: New Qld EWRs - SF_FD and BF_FD used to look into the FD EWRs in closer detail.
22
13
  - Adding state and Surface Water SDL (SWSDL) to py-ewr output tables
14
+ - Including metadata report (this is still being ironed out and tested)
15
+ - New handling capabilities for FIRM model formated files
16
+
17
+ #### Model metadata
18
+ - Added new FIRM ID file mapping FIRM ID to gauge number.
19
+
20
+ #### parameter metadata
21
+ - Updated parameter sheet and objective mapping correcting some mismatched links between environmental objectives and EWRs
22
+ - Fix SDL resource unit mapping in the parameter sheet
23
+ - Adding lat and lon to the parameter sheet
24
+ - Added in model handling for FIRM model outputs
25
+ - Various minor variable renamings for consistency
26
+ - Renamed MaxLevelRise to MaxLevelChange
27
+ - Removed AnnualFlowSum and MinLevelRise column from parameter sheet
28
+ - New format of objective mapping includes the adding of objective mapping back into the parameter sheet and one secondary dataframe (objective_reference.csv) in parameter metadata.
23
29
 
24
30
  ### **Installation**
25
31
 
@@ -37,13 +43,14 @@ pip install py-ewr
37
43
  ```
38
44
 
39
45
  ### Option 1: Running the observed mode of the tool
40
- The EWR tool will use a second program called gauge getter to first download the river data at the locations and dates selected and then run this through the EWR tool
46
+ The ewr tool will use a second program called gauge getter to first download the river data at the locations and dates selected and then run this through the ewr tool.
47
+ For more information please visit the [MDBA Gauge Getter](https://github.com/MDBAuth/MDBA_Gauge_Getter) github page.
41
48
 
42
49
  ```python
43
50
 
44
51
  from datetime import datetime
45
52
 
46
- #USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
53
+ # USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
47
54
 
48
55
  dates = {'start_date': datetime(YYYY, 7, 1),
49
56
  'end_date': datetime(YYYY, 6, 30)}
@@ -58,15 +65,15 @@ gauges = ['Gauge1', 'Gauge2']
58
65
 
59
66
  from py_ewr.observed_handling import ObservedHandler
60
67
 
61
- # Running the EWR tool:
68
+ # Running the ewr tool:
62
69
  ewr_oh = ObservedHandler(gauges=gauges, dates=dates)
63
70
 
64
71
  # Generating tables:
65
- # Table 1: Summarised EWR results for the entire timeseries
72
+ # Table 1: Summarised ewr results for the entire timeseries
66
73
  ewr_results = ewr_oh.get_ewr_results()
67
74
 
68
- # Table 2: Summarised EWR results, aggregated to water years:
69
- yearly_ewr_results = ewr_oh.get_yearly_ewr_results()
75
+ # Table 2: Summarised ewr results, aggregated to water years:
76
+ yearly_results = ewr_oh.get_yearly_results()
70
77
 
71
78
  # Table 3: All events details regardless of duration
72
79
  all_events = ewr_oh.get_all_events()
@@ -82,14 +89,23 @@ all_successful_interEvents = ewr_oh.get_all_successful_interEvents()
82
89
 
83
90
  ```
84
91
 
85
- ### Option 2: Running model scenarios through the EWR tool
92
+ ### Option 2: Running model scenarios through the ewr tool
86
93
 
87
- 1. Tell the tool where the model files are (can either be local or in a remote location)
94
+ 1. Tell the tool where the model files are (can either be local or in a remote location). For each scenario list the file path of each file. Python will default to the path relative to the directory it is being run from, but if you put a full file path you can choose any file.
95
+ ```python
96
+ scenarios = {'Scenario1': ['path/to/file1', 'path/to/file2', 'path/to/file3'],
97
+ 'Scenario2': ['path/to/file1', 'path/to/file2', 'path/to/file3']}
98
+ ```
88
99
  2. Tell the tool what format the model files are in. The current model format options are:
89
100
  - 'Bigmod - MDBA'
90
101
  Bigmod formatted outputs
102
+
91
103
  - 'Source - NSW (res.csv)'
92
104
  Source res.csv formatted outputs
105
+
106
+ - 'FIRM - MDBA'
107
+ FIRM ID formatted outputs
108
+
93
109
  - 'Standard time-series'
94
110
  The first column header should be *Date* with the date values in the YYYY-MM-DD format.
95
111
  The next columns should have the *gauge* followed by *_* followed by either *flow* or *level*
@@ -111,23 +127,31 @@ all_successful_interEvents = ewr_oh.get_all_successful_interEvents()
111
127
 
112
128
 
113
129
  ```python
114
- #USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
130
+ # USER INPUT REQUIRED>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
115
131
 
116
132
  # Minimum 1 scenario and 1 related file required
117
- scenarios = {'Scenario1': ['file/location/1', 'file/location/2', 'file/location/3'],
118
- 'Scenario2': ['file/location/1', 'file/location/2', 'file/location/3']}
133
+ scenarios = {'Scenario1': ['path/to/file1', 'path/to/file2', 'path/to/file3'],
134
+ 'Scenario2': ['path/to/file1', 'path/to/file2', 'path/to/file3']}
119
135
 
120
136
  model_format = 'Bigmod - MDBA'
121
137
 
138
+ #----- other model formats include -----#
139
+ # 'Bigmod - MDBA'
140
+ # 'Source - NSW (res.csv)'
141
+ # 'FIRM - MDBA'
142
+ # 'Standard time-series'
143
+ # 'ten thousand year'
144
+
122
145
  # END USER INPUT<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
123
146
 
124
147
  ```
148
+ File names will be incorporated into the Scenario column of the ewr tool output tables for tracebility of ewr tool output and corresponding model file. In the code written below, scenario name will be appendeded to the file name. eg. Scenario_name1_all_results.csv, therefore it is suggested to have informative scenario names that can traceback to your model files as well.
125
149
 
126
150
  ``` python
127
151
  from py_ewr.scenario_handling import ScenarioHandler
128
152
  import pandas as pd
129
153
 
130
- ewr_results_dict = {}
154
+ summary_results_dict = {}
131
155
  yearly_results_dict = {}
132
156
  all_events_dict = {}
133
157
  all_interEvents_dict = {}
@@ -135,25 +159,25 @@ all_successful_Events_dict = {}
135
159
  all_successful_interEvents_dict = {}
136
160
 
137
161
  for scenario_name, scenario_list in scenarios.items():
138
- ewr_results = pd.DataFrame()
139
- yearly_ewr_results = pd.DataFrame()
162
+ summary_results = pd.DataFrame()
163
+ yearly_results = pd.DataFrame()
140
164
  all_events = pd.DataFrame()
141
165
  all_interEvents = pd.DataFrame()
142
166
  all_successful_Events = pd.DataFrame()
143
167
  all_successful_interEvents = pd.DataFrame()
144
168
  for file in scenarios[scenario_name]:
145
169
 
146
- # Running the EWR tool:
170
+ # Running the ewr tool:
147
171
  ewr_sh = ScenarioHandler(scenario_file = file,
148
172
  model_format = model_format)
149
173
 
150
174
  # Return each table and stitch the different files of the same scenario together:
151
- # Table 1: Summarised EWR results for the entire timeseries
152
- temp_ewr_results = ewr_sh.get_ewr_results()
153
- ewr_results = pd.concat([ewr_results, temp_ewr_results], axis = 0)
154
- # Table 2: Summarised EWR results, aggregated to water years:
155
- temp_yearly_ewr_results = ewr_sh.get_yearly_ewr_results()
156
- yearly_ewr_results = pd.concat([yearly_ewr_results, temp_yearly_ewr_results], axis = 0)
175
+ # Table 1: Summarised ewr results for the entire timeseries
176
+ temp_summary_results = ewr_sh.get_ewr_results()
177
+ summary_results = pd.concat([summary_results, temp_summary_results], axis = 0)
178
+ # Table 2: Summarised ewr results, aggregated to water years:
179
+ temp_yearly_results = ewr_sh.get_yearly_ewr_results()
180
+ yearly_results = pd.concat([yearly_results, temp_yearly_results], axis = 0)
157
181
  # Table 3: All events details regardless of duration
158
182
  temp_all_events = ewr_sh.get_all_events()
159
183
  all_events = pd.concat([all_events, temp_all_events], axis = 0)
@@ -169,64 +193,74 @@ for scenario_name, scenario_list in scenarios.items():
169
193
 
170
194
 
171
195
  # Optional code to output results to csv files:
172
- ewr_results.to_csv(scenario_name + 'all_results.csv')
173
- yearly_ewr_results.to_csv(scenario_name + 'yearly_ewr_results.csv')
196
+ summary_results.to_csv(scenario_name + 'summary_results.csv')
197
+ yearly_results.to_csv(scenario_name + 'yearly_results.csv')
174
198
  all_events.to_csv(scenario_name + 'all_events.csv')
175
199
  all_interEvents.to_csv(scenario_name + 'all_interevents.csv')
176
200
  all_successful_Events.to_csv(scenario_name + 'all_successful_Events.csv')
177
201
  all_successful_interEvents.to_csv(scenario_name + 'all_successful_interEvents.csv')
178
202
 
179
203
  # Save the final tables to the dictionaries:
180
- ewr_results_dict[scenario_name] = ewr_results
181
- yearly_results_dict[scenario_name] = yearly_ewr_results
182
- all_events_dict[scenario_name] = all_events_dict
204
+ summary_results_dict[scenario_name] = summary_results
205
+ yearly_results_dict[scenario_name] = yearly_results
206
+ all_events_dict[scenario_name] = all_events
183
207
  all_interEvents_dict[scenario_name] = all_interEvents
184
208
  all_successful_Events_dict[scenario_name] = all_successful_Events
185
209
  all_successful_interEvents_dict[scenario_name] = all_successful_interEvents
186
210
 
187
211
 
188
212
  ```
189
-
213
+ #### Optional arguments for ScenarioHandler
214
+ ```python
215
+ ewr_sh = ScenarioHandler(scenario_file = file,
216
+ model_format = model_format,
217
+ parameter_sheet = parameter_sheet,
218
+ calc_config_path = calc_config_path)
219
+ ```
220
+ You may add a custom parameter sheet and or calc_config_file to your EWR tool run using the ```parameter_sheet``` and ```calc_config_path``` arguments. These arguments take a string file path pointing to files. Please check this ewr_calc_config.json file found in parameter metadata to see if any EWRs in your custom parameter sheet are not represented in the calc_config_file. For an EWR to be calculated, it must be found in both calc_config.json and the parameter sheet.
190
221
 
191
222
  ### **Purpose**
192
223
  This tool has two purposes:
193
- 1. Operational: Tracking EWR success at gauges of interest in real time - option 1 above.
194
- 2. Planning: Comparing EWR success between scenarios (i.e. model runs) - option 2 above.
224
+ 1. Operational: Tracking ewr success at gauges of interest in real time - option 1 above.
225
+ 2. Planning: Comparing ewr success between scenarios (i.e. model runs) - option 2 above.
195
226
 
196
227
  **Support**
197
- For issues relating to the script, a tutorial, or feedback please contact Lara Palmer at lara.palmer@mdba.gov.au, Martin Job at martin.job@mdba.gov.au, or Joel Bailey at joel.bailey@mdba.gov.au
228
+ For issues relating to the script, a tutorial, or feedback please contact Martin Job at martin.job@mdba.gov.au, Sirous Safari Pour at sirous.safaripour@mdba.gov.au, Elisha Freedman at elisha.freedman@mdba.gov.au, Joel Bailey at joel.bailey@mdba.gov.au, or Lara Palmer at lara.palmer@mdba.gov.au.
198
229
 
199
230
 
200
231
  **Disclaimer**
201
- Every effort has been taken to ensure the EWR database represents the original EWRs from state Long Term Water Plans (LTWPs) and Environmental Water Management Plans (EWMPs) as best as possible, and that the code within this tool has been developed to interpret and analyse these EWRs in an accurate way. However, there may still be unresolved bugs in the EWR parameter sheet and/or EWR tool. Please report any bugs to the issues tab under the GitHub project so we can investigate further.
232
+ Every effort has been taken to ensure the ewr database represents the original EWRs from state Long Term Water Plans (LTWPs) and Environmental Water Management Plans (EWMPs) as best as possible, and that the code within this tool has been developed to interpret and analyse these EWRs in an accurate way. However, there may still be unresolved bugs in the ewr parameter sheet and/or ewr tool. Please report any bugs to the issues tab under the GitHub project so we can investigate further.
202
233
 
203
234
 
204
235
  **Notes on development of the dataset of EWRs**
205
236
  The MDBA has worked with Basin state representatives to ensure scientific integrity of EWRs has been maintained when translating from raw EWRs in the Basin state LTWPs and EWMPs to the machine readable format found in the parameter sheet within this tool.
206
237
 
238
+ Environmental Water Requirements (EWRs) in the tool are subject to change when the relevant documents including Long Term Water Plans (LTWPs) and Environmental Water Management Plans (EWMPs) are updated or move from draft to final versions. LTWPs that are currently in draft form include the ACT and the upper Murrumbidgee section of the NSW Murrumbidgee LTWP.
239
+
207
240
  **Compatibility**
208
241
 
209
- NSW:
210
242
  - All Queensland catchments
211
243
  - All New South Wales catchments
212
244
  - All South Australian catchments
213
245
  - All EWRs from river based Environmental Water Management Plans (EWMPs) in Victoria*
246
+ - All EWRs for ACT (draft version)
214
247
 
215
- *Currently the wetland EWMPS and mixed wetland-river EWMPs in Victoria contain EWRs that cannot be evaluated by an automated EWR tool so the EWRs from these plans have been left out for now. The MDBA will work with our Victorian colleagues to ensure any updated EWRs in these plans are integrated into the tool where possible.
248
+ *The wetland EWMPS and mixed wetland-river EWMPs in Victoria are in progress. The MDBA will work with our Victorian colleagues to ensure any updated EWRs in these plans are integrated into the tool where possible.
216
249
 
217
250
  **Input data**
218
251
 
219
252
  - Gauge data from the relevant Basin state websites and the Bureau of Meteorology website
220
253
  - Scenario data input by the user
221
254
  - Model metadata for location association between gauge ID's and model nodes
222
- - EWR parameter sheet
255
+ #### optional
256
+ - ewr parameter sheet
257
+ - calc_config.json
223
258
 
224
- **Running the tool**
259
+ **Objective mapping**
260
+ The objective mapping is located in the EWR tool package. This is intended to be used to link EWRs to the detailed objectives, theme level targets and specific goals. The objective reference file is located in the py_ewr/parameter_metadata folder:
225
261
 
226
- Consult the user manual for instructions on how to run the tool. Please email the above email addresses for a copy of the user manual.
262
+ parameter_sheet.csv (EnvObj column)
263
+ obj_reference.csv
227
264
 
228
- **Objective mapping**
229
- Objective mapping csv files are now included in the EWR tool package. Currently this objective mapping is in an early draft format. The objective mapping will be finalised after consultation with relevant state representatives. The files are intended to be used together to link EWRs to the detailed objectives, theme level targets and specific goals. The three sheets are located in the py_ewr/parameter_metadata folder:
230
- - ewr2obj.csv: For each planning unit, gauge, ewr combination there are either one or many env_obj codes. These env_obj codes come under one of five different theme level targets (Native Fish, Native vegetation, Waterbirds, Other species or Ecosystem functions)
231
- - obj2target.csv: env_obj's are unique to their planning unit in the LTWP (noting there are often a lot of similarities between env_obj's in the same states). The plain english wording of the env objectives is also contained in this csv. The LTWP, planning unit and env_obj rows are repeated for each specific goal related to that LTWP, planning unit and env_obj.
232
- - obj2yrtarget.csv: The environmental objectives are related to 5, 10 and 20 year targets
265
+ Contains the individual environmnetal objectives listed in the 'EnvObj' column of the parameter sheet and their ecological targets (Target) and plain english description of objectives (Objectives) for each planning unit, long term water plan (LTWPShortName), and surface water sustainable diversion limit (SWSDLName).
266
+ the function ```get_obj_mapping()``` is available to automatically merge the information from obj_reference.csv with the parameter sheet to link these objectives with their specific ewr_codes.