deepdiscover-hydro 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- deepdiscover_hydro-1.0.0/CHANGELOG.md +13 -0
- deepdiscover_hydro-1.0.0/CITATION.cff +29 -0
- deepdiscover_hydro-1.0.0/LICENSE +29 -0
- deepdiscover_hydro-1.0.0/MANIFEST.in +15 -0
- deepdiscover_hydro-1.0.0/PKG-INFO +197 -0
- deepdiscover_hydro-1.0.0/README.md +164 -0
- deepdiscover_hydro-1.0.0/check.py +51 -0
- deepdiscover_hydro-1.0.0/docs/boundaries.md +48 -0
- deepdiscover_hydro-1.0.0/docs/camels.md +68 -0
- deepdiscover_hydro-1.0.0/docs/customization.md +54 -0
- deepdiscover_hydro-1.0.0/docs/data.md +26 -0
- deepdiscover_hydro-1.0.0/docs/getting_started.md +26 -0
- deepdiscover_hydro-1.0.0/docs/index.md +7 -0
- deepdiscover_hydro-1.0.0/docs/installation.md +34 -0
- deepdiscover_hydro-1.0.0/docs/limitations.md +17 -0
- deepdiscover_hydro-1.0.0/docs/model_configuration.md +49 -0
- deepdiscover_hydro-1.0.0/docs/observations.md +35 -0
- deepdiscover_hydro-1.0.0/docs/routing.md +27 -0
- deepdiscover_hydro-1.0.0/docs/scientific_formulation.md +77 -0
- deepdiscover_hydro-1.0.0/docs/structural_comparison.md +22 -0
- deepdiscover_hydro-1.0.0/docs/topologies.md +57 -0
- deepdiscover_hydro-1.0.0/docs/training_and_evaluation.md +62 -0
- deepdiscover_hydro-1.0.0/examples/01_streamflow.ipynb +220 -0
- deepdiscover_hydro-1.0.0/examples/02_groundwater_level.ipynb +178 -0
- deepdiscover_hydro-1.0.0/examples/03_sources_losses_routing.ipynb +164 -0
- deepdiscover_hydro-1.0.0/examples/04_camels_us.ipynb +240 -0
- deepdiscover_hydro-1.0.0/mkdocs.yml +18 -0
- deepdiscover_hydro-1.0.0/pyproject.toml +35 -0
- deepdiscover_hydro-1.0.0/run_examples.py +156 -0
- deepdiscover_hydro-1.0.0/setup.cfg +4 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/__init__.py +33 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/api.py +419 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/boundaries.py +86 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/data.py +182 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/dataset.py +165 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/metrics.py +105 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/model.py +1416 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/observations.py +62 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/routing.py +234 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/snow.py +310 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover/structural.py +104 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover_hydro.egg-info/PKG-INFO +197 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover_hydro.egg-info/SOURCES.txt +54 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover_hydro.egg-info/dependency_links.txt +1 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover_hydro.egg-info/requires.txt +19 -0
- deepdiscover_hydro-1.0.0/src/deepdiscover_hydro.egg-info/top_level.txt +1 -0
- deepdiscover_hydro-1.0.0/tests/conftest.py +16 -0
- deepdiscover_hydro-1.0.0/tests/test_camels_data.py +62 -0
- deepdiscover_hydro-1.0.0/tests/test_camels_windows.py +40 -0
- deepdiscover_hydro-1.0.0/tests/test_dataset.py +46 -0
- deepdiscover_hydro-1.0.0/tests/test_driver_channels.py +123 -0
- deepdiscover_hydro-1.0.0/tests/test_equations_numpy.py +41 -0
- deepdiscover_hydro-1.0.0/tests/test_physics.py +207 -0
- deepdiscover_hydro-1.0.0/tests/test_topologies.py +48 -0
- deepdiscover_hydro-1.0.0/tests/test_training.py +47 -0
- deepdiscover_hydro-1.0.0/verify_camels.py +102 -0
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 1.0.0
|
|
4
|
+
|
|
5
|
+
First public release of DeepDiscover for water-quantity modeling.
|
|
6
|
+
|
|
7
|
+
- Conservative DiscoverUnits with Causal Relationship Constraints.
|
|
8
|
+
- Multiple hydrological sources, losses, finite storage capacities and overflow.
|
|
9
|
+
- Instantaneous, fixed, learned and state-dependent temporal routing.
|
|
10
|
+
- Optional informational memory, snow accumulation and melt.
|
|
11
|
+
- Streamflow, groundwater level and other observation operators.
|
|
12
|
+
- Multivariable calibration, custom loss functions and numerical diagnostics.
|
|
13
|
+
- CAMELS-US and synthetic examples, documentation and tests.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
cff-version: 1.2.0
|
|
2
|
+
message: "Please cite the Multiple-DU methodological manuscript when using DeepDiscover."
|
|
3
|
+
title: "DeepDiscover"
|
|
4
|
+
version: 1.0.0
|
|
5
|
+
type: software
|
|
6
|
+
license: BSD-3-Clause
|
|
7
|
+
repository-code: "https://github.com/avadombi/DeepDiscover"
|
|
8
|
+
authors:
|
|
9
|
+
- family-names: Adombi
|
|
10
|
+
given-names: Adoubi Vincent De Paul
|
|
11
|
+
preferred-citation:
|
|
12
|
+
type: unpublished
|
|
13
|
+
title: "Can discharge identify internal hydrological processes? Neural equifinality in the Multiple-DU extension of DeepDiscover"
|
|
14
|
+
authors:
|
|
15
|
+
- family-names: Adombi
|
|
16
|
+
given-names: Adoubi Vincent De Paul
|
|
17
|
+
year: 2026
|
|
18
|
+
status: submitted
|
|
19
|
+
notes: "Manuscript submitted."
|
|
20
|
+
references:
|
|
21
|
+
- type: article
|
|
22
|
+
title: "DeepDiscover: towards autonomous discovery of bucket-type conceptual models – a proof of concept applied to hydrology"
|
|
23
|
+
authors:
|
|
24
|
+
- family-names: Adombi
|
|
25
|
+
given-names: Adoubi Vincent De Paul
|
|
26
|
+
journal: Journal of Hydrology
|
|
27
|
+
year: 2026
|
|
28
|
+
volume: "671"
|
|
29
|
+
doi: 10.1016/j.jhydrol.2026.135249
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026, Vincent Adombi
|
|
4
|
+
All rights reserved.
|
|
5
|
+
|
|
6
|
+
Redistribution and use in source and binary forms, with or without
|
|
7
|
+
modification, are permitted provided that the following conditions are met:
|
|
8
|
+
|
|
9
|
+
1. Redistributions of source code must retain the above copyright notice,
|
|
10
|
+
this list of conditions and the following disclaimer.
|
|
11
|
+
|
|
12
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
13
|
+
this list of conditions and the following disclaimer in the documentation
|
|
14
|
+
and/or other materials provided with the distribution.
|
|
15
|
+
|
|
16
|
+
3. Neither the name of the copyright holder nor the names of its contributors
|
|
17
|
+
may be used to endorse or promote products derived from this software
|
|
18
|
+
without specific prior written permission.
|
|
19
|
+
|
|
20
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
21
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
22
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
23
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
24
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
25
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
26
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
27
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
28
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
29
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
include LICENSE
|
|
2
|
+
include CHANGELOG.md
|
|
3
|
+
include CITATION.cff
|
|
4
|
+
include mkdocs.yml
|
|
5
|
+
include check.py
|
|
6
|
+
include run_examples.py
|
|
7
|
+
include verify_camels.py
|
|
8
|
+
recursive-include docs *.md
|
|
9
|
+
recursive-include examples *.ipynb
|
|
10
|
+
recursive-include tests *.py
|
|
11
|
+
prune data
|
|
12
|
+
prune results
|
|
13
|
+
prune .venv
|
|
14
|
+
prune .pytest_cache
|
|
15
|
+
global-exclude __pycache__ *.py[cod] *.log
|
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: deepdiscover-hydro
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Conservative differentiable hydrological modeling with DiscoverUnits
|
|
5
|
+
Author: Vincent Adombi
|
|
6
|
+
License-Expression: BSD-3-Clause
|
|
7
|
+
Project-URL: Repository, https://github.com/avadombi/DeepDiscover
|
|
8
|
+
Project-URL: Issues, https://github.com/avadombi/DeepDiscover/issues
|
|
9
|
+
Keywords: hydrology,groundwater,scientific machine learning,conservation
|
|
10
|
+
Classifier: Intended Audience :: Science/Research
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Topic :: Scientific/Engineering
|
|
13
|
+
Requires-Python: <3.13,>=3.11
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
License-File: LICENSE
|
|
16
|
+
Requires-Dist: numpy>=1.26
|
|
17
|
+
Requires-Dist: tensorflow<2.22,>=2.16
|
|
18
|
+
Requires-Dist: keras>=3.0
|
|
19
|
+
Provides-Extra: examples
|
|
20
|
+
Requires-Dist: pandas>=2.0; extra == "examples"
|
|
21
|
+
Requires-Dist: matplotlib>=3.7; extra == "examples"
|
|
22
|
+
Requires-Dist: nbformat>=5.9; extra == "examples"
|
|
23
|
+
Requires-Dist: nbclient>=0.9; extra == "examples"
|
|
24
|
+
Requires-Dist: ipykernel>=6.0; extra == "examples"
|
|
25
|
+
Requires-Dist: jupyterlab>=4.0; extra == "examples"
|
|
26
|
+
Provides-Extra: docs
|
|
27
|
+
Requires-Dist: mkdocs>=1.6; extra == "docs"
|
|
28
|
+
Provides-Extra: test
|
|
29
|
+
Requires-Dist: pytest>=8.0; extra == "test"
|
|
30
|
+
Requires-Dist: build>=1.0; extra == "test"
|
|
31
|
+
Requires-Dist: pandas>=2.0; extra == "test"
|
|
32
|
+
Dynamic: license-file
|
|
33
|
+
|
|
34
|
+
# DeepDiscover
|
|
35
|
+
|
|
36
|
+
DeepDiscover is a Python library for differentiable hydrological modeling with explicit water conservation. A model contains one or more DiscoverUnits (DUs), each with a water store and constitutive relationships that determine water releases. Each release process has one declared destination. Temporal routing is specified separately.
|
|
37
|
+
|
|
38
|
+
The same hydrological system can be calibrated against streamflow, groundwater levels, soil moisture and other measured variables using observation operators. Multiple observation types can be used together.
|
|
39
|
+
|
|
40
|
+
## Installation
|
|
41
|
+
|
|
42
|
+
Python 3.11 or 3.12 is recommended. DeepDiscover uses TensorFlow and Keras.
|
|
43
|
+
|
|
44
|
+
Install from this repository:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
python -m pip install .
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Install the optional dependencies for the notebooks and documentation:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
python -m pip install ".[examples,docs]"
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Once published on [PyPI](https://pypi.org/), install the distribution with `python -m pip install deepdiscover-hydro`.
|
|
57
|
+
|
|
58
|
+
## Example
|
|
59
|
+
|
|
60
|
+
```python
|
|
61
|
+
import numpy as np
|
|
62
|
+
from deepdiscover import CRCNet, DeepDiscover, DiscoverUnit, MultiDUModel
|
|
63
|
+
|
|
64
|
+
n_days = 365
|
|
65
|
+
precipitation = np.maximum(2 + np.sin(np.arange(n_days) / 9), 0).astype("float32")
|
|
66
|
+
pet = np.ones(n_days, dtype="float32")
|
|
67
|
+
|
|
68
|
+
water = precipitation[:, None]
|
|
69
|
+
drivers = np.column_stack([precipitation, pet])
|
|
70
|
+
|
|
71
|
+
release = CRCNet(gamma=[1, -1, 1], x_ref=[5, 3, 100])
|
|
72
|
+
unit = DiscoverUnit(process_nets=[release], context_indices=[(0, 1)])
|
|
73
|
+
physical_model = MultiDUModel([unit], destinations=[["outlet"]])
|
|
74
|
+
model = DeepDiscover(physical_model, observations={"streamflow": "outlet"})
|
|
75
|
+
|
|
76
|
+
prediction = model.predict(water[None], drivers[None])["streamflow"]
|
|
77
|
+
print(prediction.shape) # (1, 365, 1)
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
To calibrate the model, provide observed discharge as an array and create training windows:
|
|
81
|
+
|
|
82
|
+
```python
|
|
83
|
+
from deepdiscover import HydroDataset
|
|
84
|
+
|
|
85
|
+
data = HydroDataset(water, drivers, {"streamflow": observed_discharge})
|
|
86
|
+
train = data.make_windows(warmup=30, score_length=60, stride=30)
|
|
87
|
+
model.fit(train, epochs=30, batch_size=4, loss="mse")
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Here `observed_discharge` is a measured series containing one value per day.
|
|
91
|
+
|
|
92
|
+
## Main features
|
|
93
|
+
|
|
94
|
+
| Component | Options |
|
|
95
|
+
| --- | --- |
|
|
96
|
+
| DiscoverUnits | Multiple storages, processes and fixed or learned storage capacities |
|
|
97
|
+
| Constitutive relationships | Causal Relationship Constraints (CRC), custom Keras layers |
|
|
98
|
+
| Sources | Prescribed water inputs, lateral inflow, managed recharge, custom source layers |
|
|
99
|
+
| Losses | Evapotranspiration, pumping, prescribed extraction, custom sink layers |
|
|
100
|
+
| Routes | Instantaneous, prescribed, learned and state-dependent temporal routing |
|
|
101
|
+
| Additional states | Optional information memory and storage deficit |
|
|
102
|
+
| Snow | Differentiable degree-day snow model |
|
|
103
|
+
| Observations | Direct outputs, groundwater-head operators, soil-moisture operators, custom layers |
|
|
104
|
+
| Training | Time windows, warm-up, observation masks, multivariable and custom losses |
|
|
105
|
+
| Diagnostics | States, fluxes, routing storage, numerical water balance and model scores |
|
|
106
|
+
| Model comparison | Multi-seed fitting and validation-based comparison of declared structures |
|
|
107
|
+
|
|
108
|
+
Water losses belong to the physical balance. Training losses measure the agreement between simulations and observations. Both can be configured independently.
|
|
109
|
+
|
|
110
|
+
## Model topologies
|
|
111
|
+
|
|
112
|
+
Each release process has one destination, and several processes may lead to the same destination. The library supports both a direct outlet pathway plus an internal DU transfer, and a cascade with two downstream outlet processes. See [model topologies](docs/topologies.md) for explicit configurations and [structural comparison](docs/structural_comparison.md) for validation-based comparisons.
|
|
113
|
+
|
|
114
|
+
## Examples
|
|
115
|
+
|
|
116
|
+
The `examples/` directory contains four notebooks:
|
|
117
|
+
|
|
118
|
+
- `01_streamflow.ipynb`: simulation and calibration of streamflow.
|
|
119
|
+
- `02_groundwater_level.ipynb`: joint calibration using streamflow and groundwater levels.
|
|
120
|
+
- `03_sources_losses_routing.ipynb`: custom physical boundaries and delayed routing.
|
|
121
|
+
- `04_camels_us.ipynb`: a CAMELS-US case study with snow and chronological data partitions.
|
|
122
|
+
|
|
123
|
+
To run the three self-contained notebooks:
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
python run_examples.py
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
To run the CAMELS-US example, obtain the original forcing and streamflow files and follow [the data instructions](docs/camels.md). Three execution modes are provided:
|
|
130
|
+
|
|
131
|
+
```bash
|
|
132
|
+
# Short check of loading, simulation, calibration and evaluation
|
|
133
|
+
python run_examples.py --only 04_camels_us
|
|
134
|
+
|
|
135
|
+
# Measure the cost of full-length training windows (three epochs)
|
|
136
|
+
python run_examples.py --only 04_camels_us --camels-mode benchmark
|
|
137
|
+
|
|
138
|
+
# Full chronological calibration with the original training-window settings
|
|
139
|
+
python run_examples.py --only 04_camels_us --camels-mode full
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
The short check uses a limited number of training windows and two epochs. Its scores should not be used as scientific results. The benchmark and full modes use 19 parallel windows with 365 warm-up and 365 scored days. Full mode uses up to 400 epochs and validates every five epochs. It saves the validation-selected model and its training history under `results/camels/01013500/two_du_serial_seed42/`. During notebook execution, progress is printed for each code cell and each optimization batch.
|
|
143
|
+
|
|
144
|
+
Output notebooks and execution reports are written to `results/notebooks/`, which is excluded from version control.
|
|
145
|
+
|
|
146
|
+
A saved CAMELS-US checkpoint can be verified independently, without repeating model fitting or consulting the test period:
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
python verify_camels.py
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
This command requires the CAMELS-US files and the `checkpoint/` and `history.json` created by the full example.
|
|
153
|
+
|
|
154
|
+
## Documentation
|
|
155
|
+
|
|
156
|
+
The `docs/` directory covers the [scientific equations](docs/scientific_formulation.md), [model configuration](docs/model_configuration.md), [topologies](docs/topologies.md), [data](docs/data.md), [routing](docs/routing.md), [observation operators](docs/observations.md), [custom components](docs/customization.md), [calibration](docs/training_and_evaluation.md) and [model comparison](docs/structural_comparison.md).
|
|
157
|
+
|
|
158
|
+
To build the documentation locally:
|
|
159
|
+
|
|
160
|
+
```bash
|
|
161
|
+
python -m pip install ".[docs]"
|
|
162
|
+
mkdocs build
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
## Tests
|
|
166
|
+
|
|
167
|
+
```bash
|
|
168
|
+
python -m pip install ".[test]"
|
|
169
|
+
python check.py
|
|
170
|
+
python -m pytest -q
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
The tests cover conservation, routing, gradients, observation operators, training, model saving, temporal windows and CAMELS data preparation.
|
|
174
|
+
|
|
175
|
+
## Reporting bugs
|
|
176
|
+
|
|
177
|
+
Report bugs and request features through [GitHub Issues](https://github.com/avadombi/DeepDiscover/issues). Include the Python, TensorFlow and Keras versions, the model configuration, a minimal example and the complete error message. Avoid sharing private datasets or credentials.
|
|
178
|
+
|
|
179
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.
|
|
180
|
+
|
|
181
|
+
## Scientific references and license
|
|
182
|
+
|
|
183
|
+
DeepDiscover is intended for water-quantity modeling. Its latent states and processes require scientific interpretation in the context of the available observations. See [limitations](docs/limitations.md).
|
|
184
|
+
|
|
185
|
+
The methodological article is identified in [CITATION.cff](CITATION.cff). Source code is distributed under the [BSD 3-Clause License](LICENSE). CAMELS-US data must be obtained separately from their original provider.
|
|
186
|
+
|
|
187
|
+
## Citation
|
|
188
|
+
|
|
189
|
+
The current Multiple-DU methodology is described in:
|
|
190
|
+
|
|
191
|
+
Adombi, A. V. D. P. (2026). *Can discharge identify internal hydrological processes? Neural equifinality in the Multiple-DU extension of DeepDiscover*. Manuscript submitted to *Journal of Hydrology*.
|
|
192
|
+
|
|
193
|
+
The original DeepDiscover formulation is published in:
|
|
194
|
+
|
|
195
|
+
Adombi, A. V. D. P. (2026). *DeepDiscover: towards autonomous discovery of bucket-type conceptual models – a proof of concept applied to hydrology*. *Journal of Hydrology*, 671, 135249. https://doi.org/10.1016/j.jhydrol.2026.135249
|
|
196
|
+
|
|
197
|
+
The submitted manuscript currently has no publication DOI. Citation information will be updated when it is published.
|
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# DeepDiscover
|
|
2
|
+
|
|
3
|
+
DeepDiscover is a Python library for differentiable hydrological modeling with explicit water conservation. A model contains one or more DiscoverUnits (DUs), each with a water store and constitutive relationships that determine water releases. Each release process has one declared destination. Temporal routing is specified separately.
|
|
4
|
+
|
|
5
|
+
The same hydrological system can be calibrated against streamflow, groundwater levels, soil moisture and other measured variables using observation operators. Multiple observation types can be used together.
|
|
6
|
+
|
|
7
|
+
## Installation
|
|
8
|
+
|
|
9
|
+
Python 3.11 or 3.12 is recommended. DeepDiscover uses TensorFlow and Keras.
|
|
10
|
+
|
|
11
|
+
Install from this repository:
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
python -m pip install .
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Install the optional dependencies for the notebooks and documentation:
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
python -m pip install ".[examples,docs]"
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
Once published on [PyPI](https://pypi.org/), install the distribution with `python -m pip install deepdiscover-hydro`.
|
|
24
|
+
|
|
25
|
+
## Example
|
|
26
|
+
|
|
27
|
+
```python
|
|
28
|
+
import numpy as np
|
|
29
|
+
from deepdiscover import CRCNet, DeepDiscover, DiscoverUnit, MultiDUModel
|
|
30
|
+
|
|
31
|
+
n_days = 365
|
|
32
|
+
precipitation = np.maximum(2 + np.sin(np.arange(n_days) / 9), 0).astype("float32")
|
|
33
|
+
pet = np.ones(n_days, dtype="float32")
|
|
34
|
+
|
|
35
|
+
water = precipitation[:, None]
|
|
36
|
+
drivers = np.column_stack([precipitation, pet])
|
|
37
|
+
|
|
38
|
+
release = CRCNet(gamma=[1, -1, 1], x_ref=[5, 3, 100])
|
|
39
|
+
unit = DiscoverUnit(process_nets=[release], context_indices=[(0, 1)])
|
|
40
|
+
physical_model = MultiDUModel([unit], destinations=[["outlet"]])
|
|
41
|
+
model = DeepDiscover(physical_model, observations={"streamflow": "outlet"})
|
|
42
|
+
|
|
43
|
+
prediction = model.predict(water[None], drivers[None])["streamflow"]
|
|
44
|
+
print(prediction.shape) # (1, 365, 1)
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
To calibrate the model, provide observed discharge as an array and create training windows:
|
|
48
|
+
|
|
49
|
+
```python
|
|
50
|
+
from deepdiscover import HydroDataset
|
|
51
|
+
|
|
52
|
+
data = HydroDataset(water, drivers, {"streamflow": observed_discharge})
|
|
53
|
+
train = data.make_windows(warmup=30, score_length=60, stride=30)
|
|
54
|
+
model.fit(train, epochs=30, batch_size=4, loss="mse")
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Here `observed_discharge` is a measured series containing one value per day.
|
|
58
|
+
|
|
59
|
+
## Main features
|
|
60
|
+
|
|
61
|
+
| Component | Options |
|
|
62
|
+
| --- | --- |
|
|
63
|
+
| DiscoverUnits | Multiple storages, processes and fixed or learned storage capacities |
|
|
64
|
+
| Constitutive relationships | Causal Relationship Constraints (CRC), custom Keras layers |
|
|
65
|
+
| Sources | Prescribed water inputs, lateral inflow, managed recharge, custom source layers |
|
|
66
|
+
| Losses | Evapotranspiration, pumping, prescribed extraction, custom sink layers |
|
|
67
|
+
| Routes | Instantaneous, prescribed, learned and state-dependent temporal routing |
|
|
68
|
+
| Additional states | Optional information memory and storage deficit |
|
|
69
|
+
| Snow | Differentiable degree-day snow model |
|
|
70
|
+
| Observations | Direct outputs, groundwater-head operators, soil-moisture operators, custom layers |
|
|
71
|
+
| Training | Time windows, warm-up, observation masks, multivariable and custom losses |
|
|
72
|
+
| Diagnostics | States, fluxes, routing storage, numerical water balance and model scores |
|
|
73
|
+
| Model comparison | Multi-seed fitting and validation-based comparison of declared structures |
|
|
74
|
+
|
|
75
|
+
Water losses belong to the physical balance. Training losses measure the agreement between simulations and observations. Both can be configured independently.
|
|
76
|
+
|
|
77
|
+
## Model topologies
|
|
78
|
+
|
|
79
|
+
Each release process has one destination, and several processes may lead to the same destination. The library supports both a direct outlet pathway plus an internal DU transfer, and a cascade with two downstream outlet processes. See [model topologies](docs/topologies.md) for explicit configurations and [structural comparison](docs/structural_comparison.md) for validation-based comparisons.
|
|
80
|
+
|
|
81
|
+
## Examples
|
|
82
|
+
|
|
83
|
+
The `examples/` directory contains four notebooks:
|
|
84
|
+
|
|
85
|
+
- `01_streamflow.ipynb`: simulation and calibration of streamflow.
|
|
86
|
+
- `02_groundwater_level.ipynb`: joint calibration using streamflow and groundwater levels.
|
|
87
|
+
- `03_sources_losses_routing.ipynb`: custom physical boundaries and delayed routing.
|
|
88
|
+
- `04_camels_us.ipynb`: a CAMELS-US case study with snow and chronological data partitions.
|
|
89
|
+
|
|
90
|
+
To run the three self-contained notebooks:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
python run_examples.py
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
To run the CAMELS-US example, obtain the original forcing and streamflow files and follow [the data instructions](docs/camels.md). Three execution modes are provided:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
# Short check of loading, simulation, calibration and evaluation
|
|
100
|
+
python run_examples.py --only 04_camels_us
|
|
101
|
+
|
|
102
|
+
# Measure the cost of full-length training windows (three epochs)
|
|
103
|
+
python run_examples.py --only 04_camels_us --camels-mode benchmark
|
|
104
|
+
|
|
105
|
+
# Full chronological calibration with the original training-window settings
|
|
106
|
+
python run_examples.py --only 04_camels_us --camels-mode full
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
The short check uses a limited number of training windows and two epochs. Its scores should not be used as scientific results. The benchmark and full modes use 19 parallel windows with 365 warm-up and 365 scored days. Full mode uses up to 400 epochs and validates every five epochs. It saves the validation-selected model and its training history under `results/camels/01013500/two_du_serial_seed42/`. During notebook execution, progress is printed for each code cell and each optimization batch.
|
|
110
|
+
|
|
111
|
+
Output notebooks and execution reports are written to `results/notebooks/`, which is excluded from version control.
|
|
112
|
+
|
|
113
|
+
A saved CAMELS-US checkpoint can be verified independently, without repeating model fitting or consulting the test period:
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
python verify_camels.py
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
This command requires the CAMELS-US files and the `checkpoint/` and `history.json` created by the full example.
|
|
120
|
+
|
|
121
|
+
## Documentation
|
|
122
|
+
|
|
123
|
+
The `docs/` directory covers the [scientific equations](docs/scientific_formulation.md), [model configuration](docs/model_configuration.md), [topologies](docs/topologies.md), [data](docs/data.md), [routing](docs/routing.md), [observation operators](docs/observations.md), [custom components](docs/customization.md), [calibration](docs/training_and_evaluation.md) and [model comparison](docs/structural_comparison.md).
|
|
124
|
+
|
|
125
|
+
To build the documentation locally:
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
python -m pip install ".[docs]"
|
|
129
|
+
mkdocs build
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
## Tests
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
python -m pip install ".[test]"
|
|
136
|
+
python check.py
|
|
137
|
+
python -m pytest -q
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
The tests cover conservation, routing, gradients, observation operators, training, model saving, temporal windows and CAMELS data preparation.
|
|
141
|
+
|
|
142
|
+
## Reporting bugs
|
|
143
|
+
|
|
144
|
+
Report bugs and request features through [GitHub Issues](https://github.com/avadombi/DeepDiscover/issues). Include the Python, TensorFlow and Keras versions, the model configuration, a minimal example and the complete error message. Avoid sharing private datasets or credentials.
|
|
145
|
+
|
|
146
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.
|
|
147
|
+
|
|
148
|
+
## Scientific references and license
|
|
149
|
+
|
|
150
|
+
DeepDiscover is intended for water-quantity modeling. Its latent states and processes require scientific interpretation in the context of the available observations. See [limitations](docs/limitations.md).
|
|
151
|
+
|
|
152
|
+
The methodological article is identified in [CITATION.cff](CITATION.cff). Source code is distributed under the [BSD 3-Clause License](LICENSE). CAMELS-US data must be obtained separately from their original provider.
|
|
153
|
+
|
|
154
|
+
## Citation
|
|
155
|
+
|
|
156
|
+
The current Multiple-DU methodology is described in:
|
|
157
|
+
|
|
158
|
+
Adombi, A. V. D. P. (2026). *Can discharge identify internal hydrological processes? Neural equifinality in the Multiple-DU extension of DeepDiscover*. Manuscript submitted to *Journal of Hydrology*.
|
|
159
|
+
|
|
160
|
+
The original DeepDiscover formulation is published in:
|
|
161
|
+
|
|
162
|
+
Adombi, A. V. D. P. (2026). *DeepDiscover: towards autonomous discovery of bucket-type conceptual models – a proof of concept applied to hydrology*. *Journal of Hydrology*, 671, 135249. https://doi.org/10.1016/j.jhydrol.2026.135249
|
|
163
|
+
|
|
164
|
+
The submitted manuscript currently has no publication DOI. Citation information will be updated when it is published.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
"""Short installation check for a new DeepDiscover environment.
|
|
2
|
+
|
|
3
|
+
Run from the repository root:
|
|
4
|
+
python check.py
|
|
5
|
+
python -m pytest -q
|
|
6
|
+
"""
|
|
7
|
+
from pathlib import Path
|
|
8
|
+
import numpy as np
|
|
9
|
+
import tensorflow as tf
|
|
10
|
+
import keras
|
|
11
|
+
from deepdiscover import (
|
|
12
|
+
CRCNet, DiscoverUnit, MultiDUModel, DeepDiscover,
|
|
13
|
+
PrescribedSink, HydroDataset,
|
|
14
|
+
)
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
def main():
|
|
18
|
+
print('TensorFlow', tf.__version__, 'Keras', keras.__version__)
|
|
19
|
+
t = np.arange(45, dtype=np.float32)
|
|
20
|
+
rain = 2.0 + np.sin(t / 5)
|
|
21
|
+
pet = np.ones_like(rain)
|
|
22
|
+
drivers = np.stack([rain, pet], axis=-1)[None]
|
|
23
|
+
water = rain[None, :, None]
|
|
24
|
+
du = DiscoverUnit(
|
|
25
|
+
process_nets=[CRCNet([1, -1, 1], [5, 5, 100])],
|
|
26
|
+
context_indices=[(0, 1)],
|
|
27
|
+
exchanges=[PrescribedSink(driver_index=1, scale=0.1)],
|
|
28
|
+
)
|
|
29
|
+
model = DeepDiscover(MultiDUModel([du], [['outlet']]))
|
|
30
|
+
out = model.simulate(water, drivers)
|
|
31
|
+
error = float(tf.reduce_max(tf.abs(out['mass_residual'])).numpy())
|
|
32
|
+
print('Shapes:', {k: tuple(out[k].shape) for k in
|
|
33
|
+
('outlet', 'storage', 'process_fluxes', 'sink_total', 'mass_residual')})
|
|
34
|
+
print('Maximum mass residual:', error)
|
|
35
|
+
assert error < 1e-4
|
|
36
|
+
with tf.GradientTape() as tape:
|
|
37
|
+
y = model.simulate(water, drivers)['outlet']
|
|
38
|
+
objective = tf.reduce_mean(tf.square(y))
|
|
39
|
+
grad = tape.gradient(objective, model._variables())
|
|
40
|
+
assert any(g is not None for g in grad)
|
|
41
|
+
assert all(np.isfinite(g.numpy()).all() for g in grad if g is not None)
|
|
42
|
+
dataset = HydroDataset(rain, np.stack([rain, pet], axis=-1), {'streamflow': rain})
|
|
43
|
+
train = dataset.make_windows(warmup=5, score_length=10, stride=5)
|
|
44
|
+
model.fit(train, epochs=1, batch_size=2, verbose=0)
|
|
45
|
+
score = model.evaluate(train)
|
|
46
|
+
print('Training score:', score['loss'])
|
|
47
|
+
print('Installation check passed')
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
if __name__ == '__main__':
|
|
51
|
+
main()
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Physical water sources and losses
|
|
2
|
+
|
|
3
|
+
DeepDiscover distinguishes **boundary exchanges** from **training loss functions**. A boundary exchange changes the physical water balance and must be accounted for in the simulated system.
|
|
4
|
+
|
|
5
|
+
## Sources
|
|
6
|
+
|
|
7
|
+
`water` supplies the external conserved inputs directly to DUs. Additional sources can be specified in a DU's `exchanges` list:
|
|
8
|
+
|
|
9
|
+
- `PrescribedSource(driver_index, scale=1)` adds a non-negative amount from a driver column.
|
|
10
|
+
- `FunctionalSource(my_layer)` adds a non-negative amount computed by a differentiable Keras layer.
|
|
11
|
+
- A custom `keras.layers.Layer` declaring `kind = "source"` can implement the same `call(drivers)` contract.
|
|
12
|
+
|
|
13
|
+
Applications include precipitation, prescribed recharge, irrigation, lateral inflow and water transferred into the modeled domain. A physical source cannot be silently included in both `water` and `exchanges`, as that would count the same water twice.
|
|
14
|
+
|
|
15
|
+
## Losses
|
|
16
|
+
|
|
17
|
+
- `PrescribedSink(driver_index, scale=1)` requests an externally specified amount, such as pumping or managed extraction.
|
|
18
|
+
- `DemandLimitedSink(efficiency_net, demand_index, context_indices)` requests a loss bounded by demand and an efficiency between zero and one.
|
|
19
|
+
- `FunctionalSink(my_layer)` uses a custom layer returning a non-negative requested amount.
|
|
20
|
+
- A custom `keras.layers.Layer` with `kind = "sink"` implements `call(drivers, storage, available)`.
|
|
21
|
+
|
|
22
|
+
Loss requests from the same DU are combined. If demand exceeds water availability, each request is reduced by the same fraction. Simultaneous evaporation and groundwater pumping are therefore admissible and conserve water.
|
|
23
|
+
|
|
24
|
+
## Example
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
from deepdiscover import (
|
|
28
|
+
CRCNet, DiscoverUnit, PrescribedSource, PrescribedSink,
|
|
29
|
+
DemandLimitedSink,
|
|
30
|
+
)
|
|
31
|
+
|
|
32
|
+
et_efficiency = CRCNet(gamma=[1, 1], x_ref=[10, 100], output="sigmoid")
|
|
33
|
+
du = DiscoverUnit(
|
|
34
|
+
process_nets=[CRCNet([1, 1], [10, 100])],
|
|
35
|
+
context_indices=[(0,)],
|
|
36
|
+
exchanges=[
|
|
37
|
+
PrescribedSource(driver_index=2), # lateral inflow
|
|
38
|
+
DemandLimitedSink(et_efficiency, demand_index=1, context_indices=[0]),
|
|
39
|
+
PrescribedSink(driver_index=3), # pumping
|
|
40
|
+
],
|
|
41
|
+
)
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Here the driver columns represent precipitation information, PET, lateral inflow and pumping request. The user must decide which entries are conserved in `water`, and which are introduced as boundary sources. This choice determines the physical system boundary.
|
|
45
|
+
|
|
46
|
+
## Requirements for custom exchanges
|
|
47
|
+
|
|
48
|
+
Sources and sinks return one non-negative `(B,1)` tensor in quantity-per-step units. Non-finite or negative values raise errors. Custom sinks return **requested losses**; the DU applies the available-water constraint. These exchanges may carry trainable Keras parameters. If they introduce water from a separate physical reservoir outside the graph, that reservoir is not included in the system storage balance.
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# CAMELS-US data
|
|
2
|
+
|
|
3
|
+
The CAMELS-US example uses the forcing and observed streamflow files for basin `01013500`. Download the original data from the CAMELS-US provider. The files are not part of this repository.
|
|
4
|
+
|
|
5
|
+
Arrange the files as follows:
|
|
6
|
+
|
|
7
|
+
```text
|
|
8
|
+
data/
|
|
9
|
+
camels/
|
|
10
|
+
01013500/
|
|
11
|
+
01013500_lump_cida_forcing_leap.txt
|
|
12
|
+
01013500_streamflow_qc.txt
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
```python
|
|
16
|
+
from deepdiscover import load_camels_dataset
|
|
17
|
+
|
|
18
|
+
data = load_camels_dataset(
|
|
19
|
+
".", "01013500", n_du=2,
|
|
20
|
+
start="1980-10-01", end="2010-09-30",
|
|
21
|
+
)
|
|
22
|
+
print(data.water.shape, data.drivers.shape)
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The data reader computes mean temperature, Hamon potential evapotranspiration and catchment-equivalent streamflow depth. Predictor order is precipitation, potential evapotranspiration and temperature. External precipitation enters DU 0 by default.
|
|
26
|
+
|
|
27
|
+
A value of `-999` or quality flag `M` in the CAMELS streamflow file indicates a missing observation. These values are excluded from calibration through the observation mask. Other negative discharge values cause an error. Missing precipitation or meteorological drivers must be handled explicitly before simulation.
|
|
28
|
+
|
|
29
|
+
The published case-study chronology uses 365 warm-up days from 1980-10-01, training from 1981-10-01 to 2000-09-30, validation from 2000-10-01 to 2005-09-30 and testing from 2005-10-01 to 2010-09-30. The notebook selects these dates explicitly.
|
|
30
|
+
|
|
31
|
+
Review the original CAMELS-US data documentation and quality flags before using alternative basins or periods. Cite the original data sources in scientific publications.
|
|
32
|
+
|
|
33
|
+
## Running the notebook
|
|
34
|
+
|
|
35
|
+
For a short execution check:
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
python run_examples.py --only 04_camels_us
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
To measure training speed on the complete set of 19 windows:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
python run_examples.py --only 04_camels_us --camels-mode benchmark
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
For full calibration:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
python run_examples.py --only 04_camels_us --camels-mode full
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
The physical model in this example has two DUs and three release processes: one direct outlet pathway, one transfer to the second DU, and one downstream outlet pathway. The snow module separates precipitation into rain and snowmelt. These quantities and PET form the process drivers, in that order. The temperature series is used by the snow module.
|
|
54
|
+
|
|
55
|
+
Training data have shapes `water=(19,730,2)`, `drivers=(19,730,3)` and `streamflow=(19,730,1)`. Each window contains 365 unscored warm-up days and 365 scored days. The 19 complete scored blocks cover 6,935 of the 6,940 available training days; the final five days are not included in these nonoverlapping blocks. Full calibration fits all 19 windows together in one gradient update per epoch, validates every five epochs, and uses an upper limit of 400 epochs with early stopping. The notebook uses a learning rate of 0.01. The benchmark mode uses the same windows but only three epochs to report execution times.
|
|
56
|
+
|
|
57
|
+
The check and benchmark modes verify execution and measure timing. Their calibration scores should not be used in scientific analysis. The independent test period is evaluated only in full mode, after checkpoint selection on validation data. Do not use test results for model selection. For comparisons between models, report the random seeds, model parameters, calibration periods and evaluation conventions.
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
## Verifying a saved checkpoint
|
|
61
|
+
|
|
62
|
+
After a full notebook run, the selected weights and validation history are saved. To check their consistency without fitting again, run:
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
python verify_camels.py
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
This procedure reconstructs the notebook architecture, loads the saved parameters, evaluates the validation period, and checks the water balance. The recorded validation objective and selected epoch must agree with the saved model. It does not evaluate the independent test period. The checkpoint format requires the same model configuration and a compatible DeepDiscover version.
|