dapper 1.6.0__tar.gz → 1.7.0__tar.gz

dapper-1.7.0/PKG-INFO

@@ -0,0 +1,29 @@
+Metadata-Version: 2.1
+Name: dapper
+Version: 1.7.0
+Summary: DAPPER benchmarks the performance of data assimilation (DA) methods.
+Author: Patrick N. Raanes
+Author-email: patrick.n.raanes@gmail.com
+Project-URL: Documentation, https://nansencenter.github.io/DAPPER/
+Project-URL: Source, https://github.com/nansencenter/DAPPER
+Project-URL: Tracker, https://github.com/nansencenter/DAPPER/issues
+Keywords: data-assimilation enkf kalman-filtering state-estimation particle-filter kalman bayesian-methods bayesian-filter chaos
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Mathematics
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: Qt
+Provides-Extra: debug
+Provides-Extra: test
+Provides-Extra: lint
+Provides-Extra: build
+Provides-Extra: dev
+License-File: LICENCE.txt
+
+It is usually best to install from source (github),
+so that you the code is readily available to play with.
+See full README on [github](https://github.com/nansencenter/DAPPER).

{dapper-1.6.0 → dapper-1.7.0}/README.md

@@ -28,19 +28,18 @@ and then estimate that truth given the models and noisy observations.
 
 [Install](#installation), then
 read, run and try to understand `examples/basic_{1,2,3}.py`.
-Some of the examples can also be opened in Jupyter, and thereby run in the cloud
-(i.e. *without installation*, but requiring Google login): [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](http://colab.research.google.com/github/nansencenter/DAPPER).
+Some of the examples also exist as Jupyter notebooks, and can be run in the cloud
+*without installation* (but requiring Google login): [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](http://colab.research.google.com/github/nansencenter/DAPPER).
 This [screencast](https://www.youtube.com/watch?v=YtalK0Zkzvg&t=6475s)
-provides an introduction.
+provides an overview to DAPPER.
 The [documentation](https://nansencenter.github.io/DAPPER)
-includes general guidelines and the API,
-but for any serious use you will want to read and adapt the code yourself.
-If you use it in a publication, please cite, e.g.,
+includes general guidelines and the API reference,
+but most users must expect to read the code as well.
+If used towards a publication, please cite as
 *The experiments used (inspiration from) DAPPER [ref], version 1.6.0*,
-where [ref] points to [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10710355.svg)](https://doi.org/10.5281/zenodo.10710355).
-Lastly, for an introduction to DA theory also using Python,
-see these [tutorials](https://github.com/nansencenter/DA-tutorials).
-
+or similar, where [ref] points to [![DOI](https://joss.theoj.org/papers/10.21105/joss.05150/status.svg)](https://doi.org/10.21105/joss.05150).
+Also see the interactive [tutorials on DA theory](https://github.com/nansencenter/DA-tutorials)
+with Python.
 
 ## Highlights
 
@@ -50,26 +49,30 @@ through a variety of typical [test cases](#test-cases-models) and statistics. It
 (b) facilitates comparative studies, thus promoting the
 (a) reliability and
 (b) relevance of the results.
-For example, this figure is generated by `examples/basic_3.py`,
-making use of built-in tools for experiment and result management,
+For example, the figure below is generated by `examples/basic_3.py`,
 reproduces figure 5.7 of [these lecture notes](http://cerea.enpc.fr/HomePages/bocquet/teaching/assim-mb-en.pdf).
-
-![Comparative benchmarks with Lorenz'96 plotted as a function of the ensemble size (N)](./docs/imgs/ex3.svg)
-
-DAPPER is (c) open source, written in Python, and (d) focuses on readability;
-this promotes the (c) reproduction and (d) dissemination of the underlying science,
+DAPPER is
+(c) open source, written in Python, and
+(d) focuses on readability;
+this promotes the
+(c) reproduction and
+(d) dissemination of the underlying science,
 and makes it easy to adapt and extend.
 
-It also illustrates how to parallelise ensemble forecasts (e.g. the QG model),
-local analyses (e.g. the LETKF), and independent experiments (e.g. `examples/basic_3.py`).
-It comes with a battery of diagnostics and statistics.
-These all get averaged over subdomains (e..g "ocean" and "land") and then in time.
+![Comparative benchmarks with Lorenz-96 plotted as a function of the ensemble size (N)](./docs/imgs/ex3.svg)
+
+DAPPER demonstrates how to parallelise ensemble forecasts (e.g., the QG model),
+local analyses (e.g., the LETKF), and independent experiments (e.g., `examples/basic_3.py`).
+It includes a battery of diagnostics and statistics,
+which all get averaged over subdomains (e.g., "ocean" and "land") and then in time.
 Confidence intervals are computed, including correction for auto-correlations,
 and used for uncertainty quantification, and significant digits printing.
 Several diagnostics are included in the on-line "liveplotting" illustrated below,
 which may be paused for further interactive inspection.
+In summary, DAPPER is well suited for teaching and fundamental DA research.
+Also see its [drawbacks](#similar-projects).
 
-![EnKF - Lorenz'63](./docs/imgs/ex1.jpg)
+![EnKF - Lorenz-96](./docs/imgs/ex1.jpg)
 
 <!-- Non-highlighted features:
 - Time sequences use via `tools.chronos.Chronology` and `tools.chronos.Ticker`.
@@ -79,12 +82,9 @@ which may be paused for further interactive inspection.
   provides input flexibility/overloading,
   lazy eval that facilitates the use of non-diagonal
   covariance matrices (whether sparse or full).
+- built-in tools for experiment and result management,
 -->
 
-In summary, DAPPER is well suited for teaching and fundamental DA research.
-Also see its [drawbacks](#similar-projects).
-
-
 ## Installation
 
 Successfully tested on Linux/Mac/Windows.
@@ -98,12 +98,12 @@ open the [Anaconda terminal](https://docs.conda.io/projects/conda/en/latest/user
 and run the following commands:
 
 ```sh
-conda create --yes --name dapper-env python=3.9
+conda create --yes --name dapper-env python=3.12
 conda activate dapper-env
 python --version
 ```
 
-Ensure the printed version is 3.9 or more.  
+Ensure the printed version is as desired.
 *Keep using the same terminal for the commands below.*
 
 ### Install
@@ -114,19 +114,17 @@ Ensure the printed version is 3.9 or more.
 
 - Download and unzip (or `git clone`) DAPPER.
 - Move the resulting folder wherever you like,  
-  and `cd` into it
-  *(ensure you're in the folder with a `setup.py` file)*.
-- `pip install -e '.[dev]'`  
-  You can omit `[dev]` if you don't need to do serious development.
+  and `cd` into it *(ensure you're in the folder with a `setup.py` file)*.
+- `pip install -e '.'`  
 
 #### *Or*: Install as library
 
 *Do you just want to run a script that requires DAPPER?* Then
 
-- If the script comes with a `requirements.txt` file, then do  
+- If the script comes with a `requirements.txt` file that lists DAPPER, then do  
   `pip install -r path/to/requirements.txt`.
 - If not, hopefully you know the version of DAPPER needed. Run  
-  `pip install dapper==1.5.1` to get version `1.5.1` (as an example).
+  `pip install dapper==1.6.0` to get version `1.6.0` (as an example).
 
 #### *Finally*: Test the installation
 
@@ -174,7 +172,7 @@ The particle filter is tuned with "effective-N monitoring",
 "regularization/jittering" strength, and more.
 
 For a list of ready-made experiments with suitable,
-tuned settings for a given method (e.g. the `iEnKS`), use:
+tuned settings for a given method (e.g., the `iEnKS`), use:
 
 ```sh
 grep -r "xp.*iEnKS" dapper/mods
@@ -183,6 +181,9 @@ grep -r "xp.*iEnKS" dapper/mods
 
 ## Test cases (models)
 
+Simple models facilitate the reliability, reproducibility,
+and interpretability of experiment results.
+
 Model                | Lin | TLM** | PDE?  | Phys.dim. | State len | Lyap≥0 | Implementer
 -----------          | --- | ----- | ----  | --------- | --------- | ------ | ----------
 Id                   | Yes | Yes   | No    | N/A       | *         | 0      | Raanes

{dapper-1.6.0 → dapper-1.7.0}/dapper/__init__.py

@@ -4,7 +4,7 @@
 .. include:: ./README.md
 """
 
-__version__ = "1.6.0"
+__version__ = "1.7.0"
 
 # A parsimonious list of imports used in the examples
 from .dpr_config import rc

{dapper-1.6.0 → dapper-1.7.0}/dapper/da_methods/__init__.py

@@ -2,6 +2,7 @@
 
 .. include:: ./README.md
 """
+
 from pathlib import Path
 
 
@@ -82,7 +83,7 @@ def da_method(*default_dataclasses):
         def set_field(name, type_, val):
             """Set the inherited (i.e. default, i.e. has value) field."""
             # Ensure annotations
-            cls.__annotations__ = getattr(cls, '__annotations__', {})
+            cls.__annotations__ = getattr(cls, "__annotations__", {})
             # Set annotation
             cls.__annotations__[name] = type_
             # Set value
@@ -102,7 +103,7 @@ def da_method(*default_dataclasses):
         # Define the new assimilate method (has bells and whistles)
         def assimilate(self, HMM, xx, yy, desc=None, fail_gently=False, **stat_kwargs):
             # Progressbar name
-            pb_name_hook = self.da_method if desc is None else desc # noqa
+            pb_name_hook = self.da_method if desc is None else desc  # noqa
 
             # Init stats
             self.stats = dapper.stats.Stats(self, HMM, xx, yy, **stat_kwargs)
@@ -120,7 +121,7 @@ def da_method(*default_dataclasses):
                     # Don't use _print_cropped_traceback here -- It would
                     # crop out errors in the DAPPER infrastructure itself.
                     raise
-            self.stat("duration", time.time()-time0)
+            self.stat("duration", time.time() - time0)
 
         # Overwrite the assimilate method with the new one
         try:
@@ -128,12 +129,14 @@ def da_method(*default_dataclasses):
         except AttributeError as error:
             raise AttributeError(
                 "Classes decorated by da_method()"
-                " must define a method called 'assimilate'.") from error
+                " must define a method called 'assimilate'."
+            ) from error
         cls.assimilate = functools.wraps(_assimilate)(assimilate)
 
         # Shortcut for register_stat
         def stat(self, name, value):
             dapper.stats.register_stat(self.stats, name, value)
+
         cls.stat = stat
 
         # Make self.__class__.__name__ an attrib.
@@ -141,6 +144,7 @@ def da_method(*default_dataclasses):
         cls.da_method = cls.__name__
 
         return cls
+
     return dataclass_with_defaults
 
 
@@ -155,15 +159,17 @@ def _print_cropped_traceback(ERR):
         msg = "Traceback (most recent call last):\n"
         try:
             # If in IPython, use its coloring functionality
-            __IPYTHON__  # type: ignore
+            __IPYTHON__  # type: ignore # noqa: B018
         except (NameError, ImportError):
             msg += "".join(traceback.format_tb(ERR.__traceback__))
         else:
             from IPython.core.debugger import Pdb
+
             pdb_instance = Pdb()
             pdb_instance.curframe = inspect.currentframe()
 
             import dapper.da_methods
+
             keep = False
             for frame in traceback.walk_tb(ERR.__traceback__):
                 if keep:
@@ -175,10 +181,12 @@ def _print_cropped_traceback(ERR):
         return msg
 
     msg = crop_traceback(ERR) + "\nError message: " + str(ERR)
-    msg += ("\n\nResuming execution."
-            "\nIf instead you wish to raise the exceptions as usual,"
-            "\nwhich will halt the execution (and enable post-mortem debug),"
-            "\nthen use `fail_gently=False`")
+    msg += (
+        "\n\nResuming execution."
+        "\nIf instead you wish to raise the exceptions as usual,"
+        "\nwhich will halt the execution (and enable post-mortem debug),"
+        "\nthen use `fail_gently=False`"
+    )
     print(msg, file=sys.stderr)
 
 

{dapper-1.6.0 → dapper-1.7.0}/dapper/da_methods/baseline.py

@@ -2,6 +2,7 @@
 
 Many are based on `bib.raanes2016thesis`.
 """
+
 from typing import Callable, Optional
 
 import numpy as np
@@ -25,14 +26,14 @@ class Climatology:
 
     def assimilate(self, HMM, xx, yy):
         muC = np.mean(xx, 0)
-        AC  = xx - muC
-        PC  = CovMat(AC, 'A')
+        AC = xx - muC
+        PC = CovMat(AC, "A")
 
         self.stats.assess(0, mu=muC, Cov=PC)
         self.stats.trHK[:] = 0
 
         for k, ko, _, _ in progbar(HMM.tseq.ticker):
-            fau = 'u' if ko is None else 'fau'
+            fau = "u" if ko is None else "fau"
             self.stats.assess(k, ko, fau, mu=muC, Cov=PC)
 
 
@@ -49,13 +50,13 @@ class OptInterp:
 
         # Compute "climatological" Kalman gain
         muC = np.mean(xx, 0)
-        AC  = xx - muC
-        PC  = (AC.T @ AC) / (xx.shape[0] - 1)
+        AC = xx - muC
+        PC = (AC.T @ AC) / (xx.shape[0] - 1)
 
         # Setup scalar "time-series" covariance dynamics.
         # ONLY USED FOR DIAGNOSTICS, not to affect the Kalman gain.
-        L  = series.estimate_corr_length(AC.ravel(order='F'))
-        SM = fit_sigmoid(1/2, L, 0)
+        L = series.estimate_corr_length(AC.ravel(order="F"))
+        SM = fit_sigmoid(1 / 2, L, 0)
 
         # Init
         mu = muC
@@ -63,19 +64,19 @@ class OptInterp:
 
         for k, ko, t, dt in progbar(HMM.tseq.ticker):
             # Forecast
-            mu = HMM.Dyn(mu, t-dt, dt)
+            mu = HMM.Dyn(mu, t - dt, dt)
             if ko is not None:
-                self.stats.assess(k, ko, 'f', mu=muC, Cov=PC)
+                self.stats.assess(k, ko, "f", mu=muC, Cov=PC)
 
                 # Analysis
-                H  = HMM.Obs(ko).linear(muC)
-                KG  = mrdiv(PC@H.T, H@PC@H.T + HMM.Obs(ko).noise.C.full)
-                mu = muC + KG@(yy[ko] - HMM.Obs(ko)(muC))
+                H = HMM.Obs(ko).linear(muC)
+                KG = mrdiv(PC @ H.T, H @ PC @ H.T + HMM.Obs(ko).noise.C.full)
+                mu = muC + KG @ (yy[ko] - HMM.Obs(ko)(muC))
 
-                P  = (Id - KG@H) @ PC
-                SM = fit_sigmoid(P.trace()/PC.trace(), L, k)
+                P = (Id - KG @ H) @ PC
+                SM = fit_sigmoid(P.trace() / PC.trace(), L, k)
 
-            self.stats.assess(k, ko, mu=mu, Cov=2*PC*SM(k))
+            self.stats.assess(k, ko, mu=mu, Cov=2 * PC * SM(k))
 
 
 @da_method()
@@ -88,27 +89,27 @@ class Var3D:
     """
 
     B: Optional[np.ndarray] = None
-    xB: float               = 1.0
+    xB: float = 1.0
 
     def assimilate(self, HMM, xx, yy):
         Id = np.eye(HMM.Nx)
         if isinstance(self.B, np.ndarray):
             # compare ndarray 1st to avoid == error for ndarray
             B = self.B.astype(float)
-        elif self.B in (None, 'clim'):
+        elif self.B in (None, "clim"):
             # Use climatological cov, estimated from truth
             B = np.cov(xx.T)
-        elif self.B == 'eye':
+        elif self.B == "eye":
             B = Id
         else:
             raise ValueError("Bad input B.")
         B *= self.xB
 
         # ONLY USED FOR DIAGNOSTICS, not to change the Kalman gain.
-        CC = 2*np.cov(xx.T)
-        L  = series.estimate_corr_length(center(xx)[0].ravel(order='F'))
-        P  = HMM.X0.C.full
-        SM = fit_sigmoid(P.trace()/CC.trace(), L, 0)
+        CC = 2 * np.cov(xx.T)
+        L = series.estimate_corr_length(center(xx)[0].ravel(order="F"))
+        P = HMM.X0.C.full
+        SM = fit_sigmoid(P.trace() / CC.trace(), L, 0)
 
         # Init
         mu = HMM.X0.mu
@@ -116,20 +117,20 @@ class Var3D:
 
         for k, ko, t, dt in progbar(HMM.tseq.ticker):
             # Forecast
-            mu = HMM.Dyn(mu, t-dt, dt)
-            P  = CC*SM(k)
+            mu = HMM.Dyn(mu, t - dt, dt)
+            P = CC * SM(k)
 
             if ko is not None:
-                self.stats.assess(k, ko, 'f', mu=mu, Cov=P)
+                self.stats.assess(k, ko, "f", mu=mu, Cov=P)
 
                 # Analysis
-                H  = HMM.Obs(ko).linear(mu)
-                KG = mrdiv(B@H.T, H@B@H.T + HMM.Obs(ko).noise.C.full)
-                mu = mu + KG@(yy[ko] - HMM.Obs(ko)(mu))
+                H = HMM.Obs(ko).linear(mu)
+                KG = mrdiv(B @ H.T, H @ B @ H.T + HMM.Obs(ko).noise.C.full)
+                mu = mu + KG @ (yy[ko] - HMM.Obs(ko)(mu))
 
                 # Re-calibrate fit_sigmoid with new W0 = Pa/B
-                P = (Id - KG@H) @ B
-                SM = fit_sigmoid(P.trace()/CC.trace(), L, k)
+                P = (Id - KG @ H) @ B
+                SM = fit_sigmoid(P.trace() / CC.trace(), L, k)
 
             self.stats.assess(k, ko, mu=mu, Cov=P)
 
@@ -151,14 +152,17 @@ def fit_sigmoid(Sb, L, kb):
     - `b` to match values of `S(kb)` and `Sb`
     """
 
-    def sigmoid(k): return 1/(1+np.exp(-k))  # normalized sigmoid
-    def inv_sig(s): return np.log(s/(1-s))  # its inverse
+    def sigmoid(k):
+        return 1 / (1 + np.exp(-k))  # normalized sigmoid
+
+    def inv_sig(s):
+        return np.log(s / (1 - s))  # its inverse
 
-    a = 1/L
+    a = 1 / L
     b = inv_sig(Sb)
 
     def S(k):
-        return sigmoid(b + a*(k-kb))
+        return sigmoid(b + a * (k - kb))
 
     return S
 
@@ -176,9 +180,9 @@ class Persistence:
         prev = xx[0]
         self.stats.assess(0, mu=prev)
         for k, ko, _t, _dt in progbar(HMM.tseq.ticker):
-            self.stats.assess(k, ko, 'fu', mu=xx[k-1])
+            self.stats.assess(k, ko, "fu", mu=xx[k - 1])
             if ko is not None:
-                self.stats.assess(k, ko, 'a', mu=prev)
+                self.stats.assess(k, ko, "a", mu=prev)
                 prev = xx[k]
 
 
@@ -196,6 +200,6 @@ class PreProg:
     def assimilate(self, HMM, xx, yy):
         self.stats.assess(0, mu=self.schedule(0, xx, yy))
         for k, ko, _t, _dt in progbar(HMM.tseq.ticker):
-            self.stats.assess(k, ko, 'fu', mu=self.schedule(k, xx, yy))
+            self.stats.assess(k, ko, "fu", mu=self.schedule(k, xx, yy))
             if ko is not None:
-                self.stats.assess(k, ko, 'a', mu=self.schedule(k, xx, yy))
+                self.stats.assess(k, ko, "a", mu=self.schedule(k, xx, yy))