dapper 1.5.1__tar.gz → 1.7.0__tar.gz

{dapper-1.5.1 → dapper-1.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dapper
-Version: 1.5.1
+Version: 1.7.0
 Summary: DAPPER benchmarks the performance of data assimilation (DA) methods.
 Author: Patrick N. Raanes
 Author-email: patrick.n.raanes@gmail.com

{dapper-1.5.1 → dapper-1.7.0}/README.md

@@ -10,10 +10,8 @@
 
 <img src="/docs/imgs/logo_wtxt.png" align="left" width="250"/>
 
-DAPPER is a set of templates for **benchmarking** the performance of
-**data assimilation** (DA) methods.
-The tests provide experimental support and guidance for
-new developments in DA.
+DAPPER is a set of templates for **benchmarking** the performance of **data assimilation** (DA) methods.
+The numerical experiments provide support and guidance for new developments in DA.
 The typical set-up is a **synthetic (twin) experiment**, where you
 specify a dynamic model and an observational model,
 and use these to generate a synthetic truth (multivariate time series),
@@ -30,19 +28,18 @@ and then estimate that truth given the models and noisy observations.
 
 [Install](#installation), then
 read, run and try to understand `examples/basic_{1,2,3}.py`.
-Some of the examples can also be opened in Jupyter, and thereby run in the cloud
-(i.e. *without installation*, but requiring Google login): [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](http://colab.research.google.com/github/nansencenter/DAPPER).
+Some of the examples also exist as Jupyter notebooks, and can be run in the cloud
+*without installation* (but requiring Google login): [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](http://colab.research.google.com/github/nansencenter/DAPPER).
 This [screencast](https://www.youtube.com/watch?v=YtalK0Zkzvg&t=6475s)
-provides an introduction.
+provides an overview to DAPPER.
 The [documentation](https://nansencenter.github.io/DAPPER)
-includes general guidelines and the API,
-but for any serious use you will want to read and adapt the code yourself.
-If you use it in a publication, please cite, e.g.,
-*The experiments used (inspiration from) DAPPER [ref], version 1.2.1*,
-where [ref] points to [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.2029296.svg)](https://doi.org/10.5281/zenodo.2029296).
-Lastly, for an introduction to DA theory also using Python,
-see these [tutorials](https://github.com/nansencenter/DA-tutorials).
-
+includes general guidelines and the API reference,
+but most users must expect to read the code as well.
+If used towards a publication, please cite as
+*The experiments used (inspiration from) DAPPER [ref], version 1.6.0*,
+or similar, where [ref] points to [![DOI](https://joss.theoj.org/papers/10.21105/joss.05150/status.svg)](https://doi.org/10.21105/joss.05150).
+Also see the interactive [tutorials on DA theory](https://github.com/nansencenter/DA-tutorials)
+with Python.
 
 ## Highlights
 
@@ -52,26 +49,30 @@ through a variety of typical [test cases](#test-cases-models) and statistics. It
 (b) facilitates comparative studies, thus promoting the
 (a) reliability and
 (b) relevance of the results.
-For example, this figure is generated by `examples/basic_3.py`,
-making use of built-in tools for experiment and result management,
+For example, the figure below is generated by `examples/basic_3.py`,
 reproduces figure 5.7 of [these lecture notes](http://cerea.enpc.fr/HomePages/bocquet/teaching/assim-mb-en.pdf).
-
-![Comparative benchmarks with Lorenz'96 plotted as a function of the ensemble size (N)](./docs/imgs/ex3.svg)
-
-DAPPER is (c) open source, written in Python, and (d) focuses on readability;
-this promotes the (c) reproduction and (d) dissemination of the underlying science,
+DAPPER is
+(c) open source, written in Python, and
+(d) focuses on readability;
+this promotes the
+(c) reproduction and
+(d) dissemination of the underlying science,
 and makes it easy to adapt and extend.
 
-It also illustrates how to parallelise ensemble forecasts (e.g. the QG model),
-local analyses (e.g. the LETKF), and independent experiments (e.g. `examples/basic_3.py`).
-It comes with a battery of diagnostics and statistics.
-These all get averaged over subdomains (e..g "ocean" and "land") and then in time.
+![Comparative benchmarks with Lorenz-96 plotted as a function of the ensemble size (N)](./docs/imgs/ex3.svg)
+
+DAPPER demonstrates how to parallelise ensemble forecasts (e.g., the QG model),
+local analyses (e.g., the LETKF), and independent experiments (e.g., `examples/basic_3.py`).
+It includes a battery of diagnostics and statistics,
+which all get averaged over subdomains (e.g., "ocean" and "land") and then in time.
 Confidence intervals are computed, including correction for auto-correlations,
 and used for uncertainty quantification, and significant digits printing.
 Several diagnostics are included in the on-line "liveplotting" illustrated below,
 which may be paused for further interactive inspection.
+In summary, DAPPER is well suited for teaching and fundamental DA research.
+Also see its [drawbacks](#similar-projects).
 
-![EnKF - Lorenz'63](./docs/imgs/ex1.jpg)
+![EnKF - Lorenz-96](./docs/imgs/ex1.jpg)
 
 <!-- Non-highlighted features:
 - Time sequences use via `tools.chronos.Chronology` and `tools.chronos.Ticker`.
@@ -81,12 +82,9 @@ which may be paused for further interactive inspection.
   provides input flexibility/overloading,
   lazy eval that facilitates the use of non-diagonal
   covariance matrices (whether sparse or full).
+- built-in tools for experiment and result management,
 -->
 
-In summary, DAPPER is well suited for teaching and fundamental DA research.
-Also see its [drawbacks](#similar-projects).
-
-
 ## Installation
 
 Successfully tested on Linux/Mac/Windows.
@@ -100,12 +98,12 @@ open the [Anaconda terminal](https://docs.conda.io/projects/conda/en/latest/user
 and run the following commands:
 
 ```sh
-conda create --yes --name dapper-env python=3.9
+conda create --yes --name dapper-env python=3.12
 conda activate dapper-env
 python --version
 ```
 
-Ensure the printed version is 3.9 or more.  
+Ensure the printed version is as desired.
 *Keep using the same terminal for the commands below.*
 
 ### Install
@@ -116,19 +114,17 @@ Ensure the printed version is 3.9 or more.
 
 - Download and unzip (or `git clone`) DAPPER.
 - Move the resulting folder wherever you like,  
-  and `cd` into it
-  *(ensure you're in the folder with a `setup.py` file)*.
-- `pip install -e '.[dev]'`  
-  You can omit `[dev]` if you don't need to do serious development.
+  and `cd` into it *(ensure you're in the folder with a `setup.py` file)*.
+- `pip install -e '.'`  
 
 #### *Or*: Install as library
 
 *Do you just want to run a script that requires DAPPER?* Then
 
-- If the script comes with a `requirements.txt` file, then do  
+- If the script comes with a `requirements.txt` file that lists DAPPER, then do  
   `pip install -r path/to/requirements.txt`.
 - If not, hopefully you know the version of DAPPER needed. Run  
-  `pip install dapper==1.0.0` to get version `1.2.3` (as an example).
+  `pip install dapper==1.6.0` to get version `1.6.0` (as an example).
 
 #### *Finally*: Test the installation
 
@@ -176,7 +172,7 @@ The particle filter is tuned with "effective-N monitoring",
 "regularization/jittering" strength, and more.
 
 For a list of ready-made experiments with suitable,
-tuned settings for a given method (e.g. the `iEnKS`), use:
+tuned settings for a given method (e.g., the `iEnKS`), use:
 
 ```sh
 grep -r "xp.*iEnKS" dapper/mods
@@ -185,6 +181,9 @@ grep -r "xp.*iEnKS" dapper/mods
 
 ## Test cases (models)
 
+Simple models facilitate the reliability, reproducibility,
+and interpretability of experiment results.
+
 Model                | Lin | TLM** | PDE?  | Phys.dim. | State len | Lyap≥0 | Implementer
 -----------          | --- | ----- | ----  | --------- | --------- | ------ | ----------
 Id                   | Yes | Yes   | No    | N/A       | *         | 0      | Raanes
@@ -233,9 +232,7 @@ DAPPER is aimed at research and teaching (see discussion up top).
 Example of limitations:
 
 - It is not suited for very big models (>60k unknowns).
-- Time-dependent error covariances and changes in lengths of state/obs
-  (although the `Dyn` and `Obs` models may otherwise be time-dependent).
-- Non-uniform time sequences not fully supported.
+- Non-uniform time sequences.
 
 The scope of DAPPER is restricted because
 
@@ -321,7 +318,24 @@ in the development of DAPPER.
 [26]: https://github.com/CliMA/EnsembleKalmanProcesses.jl
 
 
-## Contributors
+## Contributing
+
+### Issues and Pull requests
+
+Do not hesitate to open an issue, whether to report a problem or ask a question.
+It may take some time for us to get back to you,
+since DAPPER is primarily a volunteer effort.
+Please start by perusing the [documentation](https://nansencenter.github.io/DAPPER/dapper.html)
+and searching the issue tracker for similar items.
+
+Pull requests are very welcome.
+Examples: adding a new DA method, dynamical models,
+experimental configuration reproducing literature results,
+or improving the features and capabilities of DAPPER.
+Please keep in mind the intentional [limitations](https://github.com/nansencenter/DAPPER#similar-projects)
+and read the [developers guidelines](https://nansencenter.github.io/DAPPER/dev_guide).
+
+### Contributors
 
 Patrick N. Raanes,
 Yumeng Chen,
@@ -345,7 +359,7 @@ and the Center for Western Weather and Water Extremes (CW3E).
 <img src="./docs/imgs/CW3E-Logo-Horizontal-FullColor.png?raw=true" width="400">
 <!-- markdownlint-restore -->
 
-## Publication list
+## Publications
 
 - [Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: A case study with the Lorenz 96 model](https://doi.org/10.1016/j.jocs.2020.101171)
 - [Adaptive covariance inflation in the ensemble Kalman filter by Gaussian scale mixtures](https://doi.org/10.1002/qj.3386)

{dapper-1.5.1 → dapper-1.7.0}/dapper/README.md

@@ -5,11 +5,11 @@ the docs for the various modules.
 
 ## Installation
 
-See [README/Installation](https://github.com/nansencenter/DAPPER#Installation)
+See [README/Installation](https://github.com/nansencenter/DAPPER#installation)
 
 ## Usage
 
-See [README/Getting-started](https://github.com/nansencenter/DAPPER#Getting-started)
+See [README/Getting-started](https://github.com/nansencenter/DAPPER#getting-started)
 
 ##### Examples
 
@@ -26,8 +26,8 @@ your own **model** or **method**, then
   rather than trying to squeeze everything into its templates.
 - If it is relatively simple, however, you may well want to use DAPPER.
   In that case, read this:
-    - `mods`
-    - `da_methods`
+    - `dapper.mods`
+    - `dapper.da_methods`
 
 Since the generality of DAPPER is
 [limited](https://github.com/nansencenter/DAPPER#similar-projects)

{dapper-1.5.1 → dapper-1.7.0}/dapper/__init__.py

@@ -4,12 +4,12 @@
 .. include:: ./README.md
 """
 
-__version__ = "1.5.1"
+__version__ = "1.7.0"
 
 # A parsimonious list of imports used in the examples
 from .dpr_config import rc
 from .tools.datafiles import find_latest_run, load_xps
 from .tools.rounding import round2sigfig
-from .tools.seeding import set_seed
+from .tools.seeding import rng, set_seed
 from .xp_launch import combinator, seed_and_simulate, xpList
 from .xp_process import xpSpace

{dapper-1.5.1 → dapper-1.7.0}/dapper/da_methods/README.md

@@ -1,5 +1,6 @@
-See the README section on
+Also see the section on
 [DA Methods](https://github.com/nansencenter/DAPPER#DA-Methods)
+in the main README
 for an overview of the methods included with DAPPER.
 
 ## Defining your own method

{dapper-1.5.1 → dapper-1.7.0}/dapper/da_methods/__init__.py

@@ -2,6 +2,7 @@
 
 .. include:: ./README.md
 """
+
 from pathlib import Path
 
 
@@ -82,7 +83,7 @@ def da_method(*default_dataclasses):
         def set_field(name, type_, val):
             """Set the inherited (i.e. default, i.e. has value) field."""
             # Ensure annotations
-            cls.__annotations__ = getattr(cls, '__annotations__', {})
+            cls.__annotations__ = getattr(cls, "__annotations__", {})
             # Set annotation
             cls.__annotations__[name] = type_
             # Set value
@@ -102,7 +103,7 @@ def da_method(*default_dataclasses):
         # Define the new assimilate method (has bells and whistles)
         def assimilate(self, HMM, xx, yy, desc=None, fail_gently=False, **stat_kwargs):
             # Progressbar name
-            pb_name_hook = self.da_method if desc is None else desc # noqa
+            pb_name_hook = self.da_method if desc is None else desc  # noqa
 
             # Init stats
             self.stats = dapper.stats.Stats(self, HMM, xx, yy, **stat_kwargs)
@@ -120,7 +121,7 @@ def da_method(*default_dataclasses):
                     # Don't use _print_cropped_traceback here -- It would
                     # crop out errors in the DAPPER infrastructure itself.
                     raise
-            self.stat("duration", time.time()-time0)
+            self.stat("duration", time.time() - time0)
 
         # Overwrite the assimilate method with the new one
         try:
@@ -128,12 +129,14 @@ def da_method(*default_dataclasses):
         except AttributeError as error:
             raise AttributeError(
                 "Classes decorated by da_method()"
-                " must define a method called 'assimilate'.") from error
+                " must define a method called 'assimilate'."
+            ) from error
         cls.assimilate = functools.wraps(_assimilate)(assimilate)
 
         # Shortcut for register_stat
         def stat(self, name, value):
             dapper.stats.register_stat(self.stats, name, value)
+
         cls.stat = stat
 
         # Make self.__class__.__name__ an attrib.
@@ -141,6 +144,7 @@ def da_method(*default_dataclasses):
         cls.da_method = cls.__name__
 
         return cls
+
     return dataclass_with_defaults
 
 
@@ -155,15 +159,17 @@ def _print_cropped_traceback(ERR):
         msg = "Traceback (most recent call last):\n"
         try:
             # If in IPython, use its coloring functionality
-            __IPYTHON__  # type: ignore
+            __IPYTHON__  # type: ignore # noqa: B018
         except (NameError, ImportError):
             msg += "".join(traceback.format_tb(ERR.__traceback__))
         else:
             from IPython.core.debugger import Pdb
+
             pdb_instance = Pdb()
             pdb_instance.curframe = inspect.currentframe()
 
             import dapper.da_methods
+
             keep = False
             for frame in traceback.walk_tb(ERR.__traceback__):
                 if keep:
@@ -175,10 +181,12 @@ def _print_cropped_traceback(ERR):
         return msg
 
     msg = crop_traceback(ERR) + "\nError message: " + str(ERR)
-    msg += ("\n\nResuming execution."
-            "\nIf instead you wish to raise the exceptions as usual,"
-            "\nwhich will halt the execution (and enable post-mortem debug),"
-            "\nthen use `fail_gently=False`")
+    msg += (
+        "\n\nResuming execution."
+        "\nIf instead you wish to raise the exceptions as usual,"
+        "\nwhich will halt the execution (and enable post-mortem debug),"
+        "\nthen use `fail_gently=False`"
+    )
     print(msg, file=sys.stderr)
 
 

{dapper-1.5.1 → dapper-1.7.0}/dapper/da_methods/baseline.py

@@ -2,6 +2,7 @@
 
 Many are based on `bib.raanes2016thesis`.
 """
+
 from typing import Callable, Optional
 
 import numpy as np
@@ -25,14 +26,14 @@ class Climatology:
 
     def assimilate(self, HMM, xx, yy):
         muC = np.mean(xx, 0)
-        AC  = xx - muC
-        PC  = CovMat(AC, 'A')
+        AC = xx - muC
+        PC = CovMat(AC, "A")
 
         self.stats.assess(0, mu=muC, Cov=PC)
         self.stats.trHK[:] = 0
 
         for k, ko, _, _ in progbar(HMM.tseq.ticker):
-            fau = 'u' if ko is None else 'fau'
+            fau = "u" if ko is None else "fau"
             self.stats.assess(k, ko, fau, mu=muC, Cov=PC)
 
 
@@ -49,13 +50,13 @@ class OptInterp:
 
         # Compute "climatological" Kalman gain
         muC = np.mean(xx, 0)
-        AC  = xx - muC
-        PC  = (AC.T @ AC) / (xx.shape[0] - 1)
+        AC = xx - muC
+        PC = (AC.T @ AC) / (xx.shape[0] - 1)
 
         # Setup scalar "time-series" covariance dynamics.
         # ONLY USED FOR DIAGNOSTICS, not to affect the Kalman gain.
-        L  = series.estimate_corr_length(AC.ravel(order='F'))
-        SM = fit_sigmoid(1/2, L, 0)
+        L = series.estimate_corr_length(AC.ravel(order="F"))
+        SM = fit_sigmoid(1 / 2, L, 0)
 
         # Init
         mu = muC
@@ -63,19 +64,19 @@ class OptInterp:
 
         for k, ko, t, dt in progbar(HMM.tseq.ticker):
             # Forecast
-            mu = HMM.Dyn(mu, t-dt, dt)
+            mu = HMM.Dyn(mu, t - dt, dt)
             if ko is not None:
-                self.stats.assess(k, ko, 'f', mu=muC, Cov=PC)
+                self.stats.assess(k, ko, "f", mu=muC, Cov=PC)
 
                 # Analysis
-                H  = HMM.Obs.linear(muC, t)
-                KG  = mrdiv(PC@H.T, H@PC@H.T + HMM.Obs.noise.C.full)
-                mu = muC + KG@(yy[ko] - HMM.Obs(muC, t))
+                H = HMM.Obs(ko).linear(muC)
+                KG = mrdiv(PC @ H.T, H @ PC @ H.T + HMM.Obs(ko).noise.C.full)
+                mu = muC + KG @ (yy[ko] - HMM.Obs(ko)(muC))
 
-                P  = (Id - KG@H) @ PC
-                SM = fit_sigmoid(P.trace()/PC.trace(), L, k)
+                P = (Id - KG @ H) @ PC
+                SM = fit_sigmoid(P.trace() / PC.trace(), L, k)
 
-            self.stats.assess(k, ko, mu=mu, Cov=2*PC*SM(k))
+            self.stats.assess(k, ko, mu=mu, Cov=2 * PC * SM(k))
 
 
 @da_method()
@@ -88,27 +89,27 @@ class Var3D:
     """
 
     B: Optional[np.ndarray] = None
-    xB: float               = 1.0
+    xB: float = 1.0
 
     def assimilate(self, HMM, xx, yy):
         Id = np.eye(HMM.Nx)
         if isinstance(self.B, np.ndarray):
             # compare ndarray 1st to avoid == error for ndarray
             B = self.B.astype(float)
-        elif self.B in (None, 'clim'):
+        elif self.B in (None, "clim"):
             # Use climatological cov, estimated from truth
             B = np.cov(xx.T)
-        elif self.B == 'eye':
+        elif self.B == "eye":
             B = Id
         else:
             raise ValueError("Bad input B.")
         B *= self.xB
 
         # ONLY USED FOR DIAGNOSTICS, not to change the Kalman gain.
-        CC = 2*np.cov(xx.T)
-        L  = series.estimate_corr_length(center(xx)[0].ravel(order='F'))
-        P  = HMM.X0.C.full
-        SM = fit_sigmoid(P.trace()/CC.trace(), L, 0)
+        CC = 2 * np.cov(xx.T)
+        L = series.estimate_corr_length(center(xx)[0].ravel(order="F"))
+        P = HMM.X0.C.full
+        SM = fit_sigmoid(P.trace() / CC.trace(), L, 0)
 
         # Init
         mu = HMM.X0.mu
@@ -116,20 +117,20 @@ class Var3D:
 
         for k, ko, t, dt in progbar(HMM.tseq.ticker):
             # Forecast
-            mu = HMM.Dyn(mu, t-dt, dt)
-            P  = CC*SM(k)
+            mu = HMM.Dyn(mu, t - dt, dt)
+            P = CC * SM(k)
 
             if ko is not None:
-                self.stats.assess(k, ko, 'f', mu=mu, Cov=P)
+                self.stats.assess(k, ko, "f", mu=mu, Cov=P)
 
                 # Analysis
-                H  = HMM.Obs.linear(mu, t)
-                KG = mrdiv(B@H.T, H@B@H.T + HMM.Obs.noise.C.full)
-                mu = mu + KG@(yy[ko] - HMM.Obs(mu, t))
+                H = HMM.Obs(ko).linear(mu)
+                KG = mrdiv(B @ H.T, H @ B @ H.T + HMM.Obs(ko).noise.C.full)
+                mu = mu + KG @ (yy[ko] - HMM.Obs(ko)(mu))
 
                 # Re-calibrate fit_sigmoid with new W0 = Pa/B
-                P = (Id - KG@H) @ B
-                SM = fit_sigmoid(P.trace()/CC.trace(), L, k)
+                P = (Id - KG @ H) @ B
+                SM = fit_sigmoid(P.trace() / CC.trace(), L, k)
 
             self.stats.assess(k, ko, mu=mu, Cov=P)
 
@@ -151,14 +152,17 @@ def fit_sigmoid(Sb, L, kb):
     - `b` to match values of `S(kb)` and `Sb`
     """
 
-    def sigmoid(k): return 1/(1+np.exp(-k))  # normalized sigmoid
-    def inv_sig(s): return np.log(s/(1-s))  # its inverse
+    def sigmoid(k):
+        return 1 / (1 + np.exp(-k))  # normalized sigmoid
+
+    def inv_sig(s):
+        return np.log(s / (1 - s))  # its inverse
 
-    a = 1/L
+    a = 1 / L
     b = inv_sig(Sb)
 
     def S(k):
-        return sigmoid(b + a*(k-kb))
+        return sigmoid(b + a * (k - kb))
 
     return S
 
@@ -176,9 +180,9 @@ class Persistence:
         prev = xx[0]
         self.stats.assess(0, mu=prev)
         for k, ko, _t, _dt in progbar(HMM.tseq.ticker):
-            self.stats.assess(k, ko, 'fu', mu=xx[k-1])
+            self.stats.assess(k, ko, "fu", mu=xx[k - 1])
             if ko is not None:
-                self.stats.assess(k, ko, 'a', mu=prev)
+                self.stats.assess(k, ko, "a", mu=prev)
                 prev = xx[k]
 
 
@@ -196,6 +200,6 @@ class PreProg:
     def assimilate(self, HMM, xx, yy):
         self.stats.assess(0, mu=self.schedule(0, xx, yy))
         for k, ko, _t, _dt in progbar(HMM.tseq.ticker):
-            self.stats.assess(k, ko, 'fu', mu=self.schedule(k, xx, yy))
+            self.stats.assess(k, ko, "fu", mu=self.schedule(k, xx, yy))
             if ko is not None:
-                self.stats.assess(k, ko, 'a', mu=self.schedule(k, xx, yy))
+                self.stats.assess(k, ko, "a", mu=self.schedule(k, xx, yy))