simdec 1.1.0__tar.gz → 1.2.0__tar.gz

{simdec-1.1.0 → simdec-1.2.0}/.gitignore

@@ -7,6 +7,9 @@ __pycache__
 .cache/*
 **/.ipynb_checkpoints
 
+# Env variables with direnv
+.envrc
+
 # OS generated files
 .DS_Store*
 
@@ -29,4 +32,6 @@ htmlcov/*
 build/*
 dist/*
 sdist/*
-docs/_build
+docs/html
+docs/jupyter_execute
+app.html

{simdec-1.1.0 → simdec-1.2.0}/CONTRIBUTING.md

@@ -161,3 +161,39 @@ a release:
 5. Check that release has been deployed to PyPI
 6. Check documentation is built and deployed to readthedocs
 7. Check that auto-generated PR is auto-merged on the conda-forge feedstock repo
+
+## Dashboard
+A live dashboard is available at:
+
+https://simdec.io
+
+The DNS records are available on CPanel and the rest is hosted on Google Cloud
+Platform.
+
+Developing locally requires the installation of GCP CLI and Docker engine.
+
+A few helper commands are provided in the Makefile.
+
+### Local use
+
+The dashboard can be run locally using:
+
+    make serve-dev
+
+If you want to test OAuth, you need to export the following env variables:
+
+    export PANEL_OAUTH_REDIRECT_URI=http://localhost:5006/app
+    export PANEL_OAUTH_KEY=[VALUE IN GCP Secret Manager]
+    export PANEL_OAUTH_SECRET=[VALUE IN GCP Secret Manager]
+    export PANEL_OAUTH_ENCRYPTION=[VALUE IN GCP Secret Manager]
+
+Use the CLI tool `direnv` for convenience. Then you can serve with
+OAuth support:
+
+    make serve-oauth
+
+### Deployment
+
+New version can either be deployed from the CI with `cloudbuild` or locally:
+
+    make production

{simdec-1.1.0 → simdec-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.3
 Name: simdec
-Version: 1.1.0
+Version: 1.2.0
 Summary: Sensitivity analysis using simulation decomposition
 Project-URL: homepage, https://www.simdec.fi/
 Project-URL: documentation, https://simdec.readthedocs.io
@@ -16,22 +16,32 @@ Classifier: License :: OSI Approved :: BSD License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
-Requires-Python: >=3.9
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
 Requires-Dist: numpy
 Requires-Dist: pandas
-Requires-Dist: panel
 Requires-Dist: salib
 Requires-Dist: seaborn
-Requires-Dist: typer[all]
+Provides-Extra: dashboard
+Requires-Dist: cryptography; extra == 'dashboard'
+Requires-Dist: panel>=1.3.8; extra == 'dashboard'
 Provides-Extra: dev
+Requires-Dist: accessible-pygments; extra == 'dev'
+Requires-Dist: cryptography; extra == 'dev'
 Requires-Dist: hatch; extra == 'dev'
+Requires-Dist: myst-nb; extra == 'dev'
+Requires-Dist: numpydoc; extra == 'dev'
+Requires-Dist: panel>=1.3.8; extra == 'dev'
 Requires-Dist: pre-commit; extra == 'dev'
-Requires-Dist: simdec[doc,test]; extra == 'dev'
+Requires-Dist: pydata-sphinx-theme; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: sphinx; extra == 'dev'
 Provides-Extra: doc
 Requires-Dist: accessible-pygments; extra == 'doc'
+Requires-Dist: myst-nb; extra == 'doc'
 Requires-Dist: numpydoc; extra == 'doc'
 Requires-Dist: pydata-sphinx-theme; extra == 'doc'
 Requires-Dist: sphinx; extra == 'doc'
@@ -69,30 +79,11 @@ The library is distributed on PyPi and can be installed with:
 
     pip install simdec
 
-To install the development version, download the source and from the root of
-the repository:
-
-    pip install -e .[dev]
-
 ## Dashboard
 A live dashboard is available at:
 
 https://simdec.io
 
-### Local use
-
-The dashboard can be run locally using:
-
-    make serve
-
-### Deployment
-
-The live version of the dashboard is hosted on GCP. To deploy a new version:
-
-    PANEL_TOKEN=... make production
-
-###
-
 ## Citations
 
 The algorithms and visualizations used in this package came primarily out of
@@ -103,13 +94,6 @@ Foundation, and Finnish Foundation for Economic Education.
 If you use SimDec in your research we would appreciate a citation to the
 following publications:
 
-- Kozlova, M., & Yeomans, J. S. (2022). Monte Carlo Enhancement via Simulation
-  Decomposition: A “Must-Have” Inclusion for Many Disciplines. _INFORMS
-  Transactions on Education, 22_(3), 147-159. DOI:10.1287/ited.2019.0240.
-- Kozlova, M., Moss, R. J., Yeomans, J. S., & Caers, J. (forthcoming).
-  Uncovering Heterogeneous Effects in Computational Models for Sustainable
-  Decision-making. _Environmental Modelling & Software_.
-- Kozlova, M., Moss, R. J., Roy, P., Alam, A., & Yeomans, J. S. (forthcoming).
-  SimDec algorithm. In M. Kozlova & J. S. Yeomans (Eds.), _Sensitivity Analysis
-  for Business, Technology, and Policymaking Made Easy with Simulation
-  Decomposition_. Routledge.
+- Kozlova, M., & Yeomans, J. S. (2022). Monte Carlo Enhancement via Simulation Decomposition: A “Must-Have” Inclusion for Many Disciplines. _INFORMS Transactions on Education, 22_(3), 147-159. [Available here](https://pubsonline.informs.org/doi/10.1287/ited.2019.0240).
+- Kozlova, M., Moss, R. J., Yeomans, J. S., & Caers, J. (2024). Uncovering Heterogeneous Effects in Computational Models for Sustainable Decision-making. _Environmental Modelling & Software_, 171, 105898. [https://doi.org/10.1016/j.envsoft.2023.105898](https://doi.org/10.1016/j.envsoft.2023.105898)
+- Kozlova, M., Moss, R. J., Roy, P., Alam, A., & Yeomans, J. S. (forthcoming). SimDec algorithm and guidelines for its usage and interpretation. In M. Kozlova & J. S. Yeomans (Eds.), _Sensitivity Analysis for Business, Technology, and Policymaking. Made Easy with Simulation Decomposition_. Routledge.

{simdec-1.1.0 → simdec-1.2.0}/README.md

@@ -27,30 +27,11 @@ The library is distributed on PyPi and can be installed with:
 
     pip install simdec
 
-To install the development version, download the source and from the root of
-the repository:
-
-    pip install -e .[dev]
-
 ## Dashboard
 A live dashboard is available at:
 
 https://simdec.io
 
-### Local use
-
-The dashboard can be run locally using:
-
-    make serve
-
-### Deployment
-
-The live version of the dashboard is hosted on GCP. To deploy a new version:
-
-    PANEL_TOKEN=... make production
-
-###
-
 ## Citations
 
 The algorithms and visualizations used in this package came primarily out of
@@ -61,13 +42,6 @@ Foundation, and Finnish Foundation for Economic Education.
 If you use SimDec in your research we would appreciate a citation to the
 following publications:
 
-- Kozlova, M., & Yeomans, J. S. (2022). Monte Carlo Enhancement via Simulation
-  Decomposition: A “Must-Have” Inclusion for Many Disciplines. _INFORMS
-  Transactions on Education, 22_(3), 147-159. DOI:10.1287/ited.2019.0240.
-- Kozlova, M., Moss, R. J., Yeomans, J. S., & Caers, J. (forthcoming).
-  Uncovering Heterogeneous Effects in Computational Models for Sustainable
-  Decision-making. _Environmental Modelling & Software_.
-- Kozlova, M., Moss, R. J., Roy, P., Alam, A., & Yeomans, J. S. (forthcoming).
-  SimDec algorithm. In M. Kozlova & J. S. Yeomans (Eds.), _Sensitivity Analysis
-  for Business, Technology, and Policymaking Made Easy with Simulation
-  Decomposition_. Routledge.
+- Kozlova, M., & Yeomans, J. S. (2022). Monte Carlo Enhancement via Simulation Decomposition: A “Must-Have” Inclusion for Many Disciplines. _INFORMS Transactions on Education, 22_(3), 147-159. [Available here](https://pubsonline.informs.org/doi/10.1287/ited.2019.0240).
+- Kozlova, M., Moss, R. J., Yeomans, J. S., & Caers, J. (2024). Uncovering Heterogeneous Effects in Computational Models for Sustainable Decision-making. _Environmental Modelling & Software_, 171, 105898. [https://doi.org/10.1016/j.envsoft.2023.105898](https://doi.org/10.1016/j.envsoft.2023.105898)
+- Kozlova, M., Moss, R. J., Roy, P., Alam, A., & Yeomans, J. S. (forthcoming). SimDec algorithm and guidelines for its usage and interpretation. In M. Kozlova & J. S. Yeomans (Eds.), _Sensitivity Analysis for Business, Technology, and Policymaking. Made Easy with Simulation Decomposition_. Routledge.

simdec-1.2.0/joss/paper.bib

@@ -0,0 +1,115 @@
+@article{Herman2017,
+  doi = {10.21105/joss.00097},
+  year  = {2017},
+  month = {jan},
+  publisher = {The Open Journal},
+  volume = {2},
+  number = {9},
+  author = {Jon Herman and Will Usher},
+  title = {{SALib}: An open-source {Python} library for Sensitivity Analysis},
+  journal = {The Journal of Open Source Software}
+}
+
+@article{Kozlova2024,
+  title = {Uncovering heterogeneous effects in computational models for sustainable decision-making},
+  journal = {Environmental Modelling & Software},
+  volume = {171},
+  pages = {105898},
+  year = {2024},
+  issn = {1364-8152},
+  doi = {10.1016/j.envsoft.2023.105898},
+  author = {Mariia Kozlova and Robert J. Moss and Julian Scott Yeomans and Jef Caers},
+  keywords = {Global sensitivity analysis, Simulation decomposition, Monte Carlo simulation, Decision-making under uncertainty}
+}
+
+@article{Roy2023,
+  doi = {10.21105/joss.05309},
+  year = {2023},
+  publisher = {The Open Journal},
+  volume = {8},
+  number = {84},
+  pages = {5309},
+  author = {Pamphile T. Roy and Art B. Owen and Maximilian Balandat and Matt Haberland},
+  title = {Quasi-Monte Carlo Methods in {Python}},
+  journal = {Journal of Open Source Software}
+}
+
+@article{Virtanen2020,
+  title={SciPy 1.0: fundamental algorithms for scientific computing in {P}ython},
+  author={Virtanen, Pauli and Gommers, Ralf and Oliphant, Travis E and Haberland, Matt and Reddy, Tyler and Cournapeau, David and Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and Bright, Jonathan et al.},
+  journal={Nature methods},
+  volume={17},
+  number={3},
+  pages={261--272},
+  year={2020},
+  publisher={Nature Publishing Group},
+  doi={10.1038/s41592-019-0686-2}
+}
+
+@book{Saltelli2007,
+author = {Saltelli, Andrea and Ratto, Marco and Andres, Terry and Campolongo, Francesca and Cariboni, Jessica and Gatelli, Debora and Saisana, Michaela and Tarantola, Stefano},
+booktitle = {Global Sensitivity Analysis. The Primer},
+doi = {10.1002/9780470725184},
+isbn = {9780470725184},
+month = {dec},
+pages = {237--275},
+publisher = {John Wiley {\&} Sons, Ltd},
+title = {{Global Sensitivity Analysis. The Primer}},
+year = {2007}
+}
+
+@article{sobol1993,
+  title={Sensitivity analysis for non-linear mathematical models, originally “Sensitivity estimates for non-linear mathematical models”},
+  author={Sobol, Ilya M},
+  journal={Math Model Comput Exp},
+  volume={1},
+  pages={407--414},
+  year={1993}
+}
+
+@report{europeancommission2021,
+  title = {Better {{Regulation Toolbox}}},
+  author = {{European Commission}},
+  date = {2021-11},
+  url = {https://ec.europa.eu/info/law/law-making-process/planning-and-proposing-law/better-regulation-why-and-how/better-regulation-guidelines-and-toolbox_en},
+  city = {Brussels},
+  keywords = {\#nosource}
+}
+
+@software{panel,
+  author       = {Philipp Rudiger and
+                  Marc Skov Madsen and
+                  Simon Høxbro Hansen and
+                  Maxime Liquet and
+                  Andrew and
+                  Xavier Artusi and
+                  James A. Bednar and
+                  Chris B and
+                  Jean-Luc Stevens and
+                  Christoph Deil and
+                  Demetris Roumis and
+                  Julia Signell and
+                  Mateusz Paprocki and
+                  Jerry Wu and
+                  Jon Mease and
+                  Arne and
+                  Coderambling and
+                  Hugues-Yanis Amanieu and
+                  thuydotm and
+                  Simon and
+                  sdc50 and
+                  Luca Fabbri and
+                  kbowen and
+                  Theom and
+                  Joel Ostblom and
+                  Govinda Totla and
+                  Niko Föhr and
+                  TBym},
+  title        = {holoviz/panel: Version 1.4.3},
+  month        = may,
+  year         = 2024,
+  publisher    = {Zenodo},
+  version      = {v1.4.3},
+  doi          = {10.5281/zenodo.11261266},
+  url          = {https://doi.org/10.5281/zenodo.11261266}
+}

simdec-1.2.0/joss/paper.md

@@ -0,0 +1,76 @@
+---
+title: 'Simulation Decomposition in Python'
+tags:
+  - Python
+  - SimDec
+  - statistics
+  - Sensitivity Analysis
+  - Visualization
+authors:
+  - name: Pamphile T. Roy
+    affiliation: 1
+    corresponding: true
+    orcid: 0000-0001-9816-1416
+  - name: Mariia Kozlova
+    affiliation: 2
+    orcid: 0000-0002-6952-7682
+affiliations:
+    - name: Consulting Manao GmbH, Vienna, Austria
+      index: 1
+    - name: LUT Business School, LUT University, Lappeenranta, Finland
+      index: 2
+date: 1 April 2024
+bibliography: paper.bib
+
+---
+
+# Summary
+
+Uncertainties are everywhere. Whether you are developing a new Artificial Intelligence (AI) system,
+running complex simulations or making an experiment in a lab, uncertainties
+influence the system. Therefore, an approach is needed to understand how these uncertainties impact the system's performance.
+
+SimDec offers a novel visual way to understand the intricate role that
+uncertainties play. A clear Python Application Programming Interface (API) and a no-code interactive web
+dashboard make uncertainty analysis with SimDec accessible to everyone.
+
+# Statement of need
+
+From real life experiments to numerical simulations, uncertainties play a
+crucial role in the system under study. With the advent of AI
+and new regulations such as the [AI Act](https://artificialintelligenceact.eu) or the
+*Better Regulation Guideline* [@europeancommission2021], there is a growing need for explainability and
+impact assessments of systems under uncertainties.
+
+Traditional methods to analyse the uncertainties focus on quantitative methods
+to compare the importance of factors, there is a large body of literature and
+the field is known as: Sensitivity Analysis (SA) [@Saltelli2007]. The indices of Sobol' are a
+prominent example of such methods [@sobol1993].
+
+Simulation Decomposition or SimDec moves the field of SA forward by supplementing the computation of sensitivity indices with the visualization of the type of interactions involved, which proves critical for understanding the system's behavior and decision-making [@Kozlova2024].
+In short, SimDec is a hybrid uncertainty-sensitivity analysis approach
+that reveals the critical behavior of a computational model or an empirical
+dataset. It decomposes the distribution of the output
+(target variable) by automatically forming scenarios that reveal the most critical behavior of the system. The scenarios are formed out of the most influential input variables (defined with variance-based sensitivity indices) by breaking down their numeric ranges into states (e.g. _low_ and _high_) and creating an exhaustive list of their combinations (e.g. (i) _low_ _**A**_ & _low_ _**B**_, (ii) _low_ _**A**_ & _high_ _**B**_, (iii) _high_ **_A_** & _low_ **_B_**, and (iv) _high_ **_A_** and _high_ **_B_**). The resulting visualization shows how different
+output ranges can be achieved and what kind of critical interactions affect
+the output—as seen in \autoref{fig:simdec}. The method has shown value for
+various computational models from different fields, including business,
+environment, and engineering, as well as an emerging evidence of use for
+empirical data and AI.
+
+![SimDec: explanation of output by most important inputs. A simulation dataset of a structural reliability model with one key output variable and four input variables is used for this case. Inputs 3 and 1 have the highest sensitivity indices and thus are automatically chosen for decomposition. The most influential input 3 divides the distribution of the output into three main states with distinct colors. Input 1 further subdivides them into shades. From the graph, it becomes obvious that input 1 influences the output when input 3 is low, but has a negligible effect when input 3 is medium or high.\label{fig:simdec}](simdec_presentation.png)
+
+Besides proposing a comprehensive yet simple API through a Python package
+available on PyPi, SimDec is also made available
+to practitioners through an online dashboard at [https://simdec.io](https://simdec.io). The project
+relies on powerful variance-based sensitivity analysis methods from SALib [@Herman2017] and
+SciPy [@Virtanen2020; @Roy2023]—notably the Quasi-Monte Carlo capabilities with
+`sp.stats.qmc` and in the future sensitivity indices with `sp.stats.sensitivity_indices`.
+The dashboard is made possible thanks to Panel [@panel].
+
+# Acknowledgements
+
+The work on this open-source software was supported by grant #220177 from
+Finnish Foundation for Economic Foundation.
+
+# References

{simdec-1.1.0 → simdec-1.2.0}/pyproject.toml

@@ -4,10 +4,10 @@ build-backend = "hatchling.build"
 
 [project]
 name = "simdec"
-version = "1.1.0"
+version = "1.2.0"
 description = "Sensitivity analysis using simulation decomposition"
 readme = "README.md"
-requires-python = ">=3.9"
+requires-python = ">=3.10"
 license = "BSD-3-Clause"
 authors = [
     { name = "Pamphile Roy" },
@@ -23,9 +23,9 @@ classifiers = [
     "Operating System :: OS Independent",
     "Programming Language :: Python",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
 ]
 
 dependencies = [
@@ -33,11 +33,14 @@ dependencies = [
     "pandas",
     "SALib",
     "seaborn",
-    "typer[all]",
-    "panel",
 ]
 
 [project.optional-dependencies]
+dashboard = [
+    "panel>=1.3.8",
+    "cryptography",
+]
+
 test = [
     "pytest",
     "pytest-cov",
@@ -48,32 +51,37 @@ doc = [
     "pydata-sphinx-theme",
     "accessible-pygments",
     "numpydoc",
+    "myst-nb",
 ]
 
 dev = [
-    "simdec[doc,test]",
+    "simdec[doc,test,dashboard]",
     "pre-commit",
     "hatch",
 ]
 
-[project.scripts]
-simdec = "simdec.workflow:app"
-
 [project.urls]
 homepage = "https://www.simdec.fi/"
 documentation = "https://simdec.readthedocs.io"
 source = "https://github.com/Simulation-Decomposition/simdec-python"
 
 [tool.hatch]
+build.targets.wheel.packages = ["src/simdec"]
 build.targets.sdist.exclude = [
   ".github",
   "docs",
+  "panel",
   "tests",
   "*.rst",
   "*.yml",
   ".*",
+  "Makefile",
+  "Dockerfile",
 ]
 
+[project.entry-points."panel.auth"]
+custom_google = "simdec.auth:CustomGoogleLoginHandler"
+
 [tool.pytest.ini_options]
 addopts = "--durations 10"
 testpaths = [

simdec-1.2.0/src/simdec/auth.py

@@ -0,0 +1,17 @@
+from panel.auth import GoogleLoginHandler
+from panel.io.resources import CDN_DIST
+
+
+class CustomGoogleLoginHandler(GoogleLoginHandler):
+    def _simple_get(self):
+        html = self._login_template.render(errormessage="", PANEL_CDN=CDN_DIST)
+        self.write(html)
+
+    async def get(self):
+        if "login" in self.request.uri and "state" not in self.request.uri:
+            self._simple_get()
+        else:
+            await super().get()
+
+    async def post(self):
+        await super().get()

{simdec-1.1.0 → simdec-1.2.0}/src/simdec/decomposition.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
+from hashlib import blake2b
 from typing import Literal
 
 import numpy as np
@@ -14,13 +15,15 @@ __all__ = ["decomposition", "states_expansion"]
 def states_expansion(states: list[int], inputs: pd.DataFrame) -> list[list[str]]:
     """Expand states list to fully represent all scenarios."""
     inputs = pd.DataFrame(inputs)
-    for i, state in enumerate(states):
+    expanded_states = []
+    for state in states:
         if isinstance(state, int):
-            states: list
             if state == 2:
-                states[i] = ["low", "high"]
+                expanded_states.append(["low", "high"])
             elif state == 3:
-                states[i] = ["low", "medium", "high"]
+                expanded_states.append(["low", "medium", "high"])
+        else:
+            expanded_states.append(state)
 
     # categorical for a given variable
     cat_cols = inputs.select_dtypes(exclude=["number"])
@@ -33,9 +36,9 @@ def states_expansion(states: list[int], inputs: pd.DataFrame) -> list[list[str]]
 
     for i, states_cat_ in zip(cat_cols_idx, states_cats_):
         n_unique = np.unique(inputs.iloc[:, i]).size
-        states[i] = list(states_cat_) if n_unique < 5 else states[i]
+        expanded_states[i] = list(states_cat_) if n_unique < 5 else expanded_states[i]
 
-    return states
+    return expanded_states
 
 
 @dataclass
@@ -46,6 +49,17 @@ class DecompositionResult:
     states: list[int]
     bin_edges: np.ndarray
 
+    def __reduce__(self):
+        h = blake2b(key=b"result hashing", digest_size=20)
+
+        h.update(str(self.var_names).encode())
+        h.update(str(self.statistic).encode())
+        h.update(str(self.bins).encode())
+        h.update(str(self.states).encode())
+        h.update(str(self.bin_edges).encode())
+
+        return [h.hexdigest()]
+
 
 def decomposition(
     inputs: pd.DataFrame,
@@ -95,11 +109,9 @@ def decomposition(
     var_names = inputs.columns
 
     cat_cols = inputs.select_dtypes(exclude=["number"])
-    cat_cols_idx = []
     for cat_col in cat_cols:
         codes, cat_states_ = pd.factorize(inputs[cat_col])
         inputs[cat_col] = codes
-        cat_cols_idx.append(inputs.columns.get_loc(cat_col))
 
     inputs = inputs.to_numpy()
     output = output.to_numpy()
@@ -109,19 +121,22 @@ def decomposition(
 
     # only keep the explained variance corresponding to `dec_limit`
     sensitivity_indices = sensitivity_indices[var_order]
-    n_var_dec = np.where(np.cumsum(sensitivity_indices) < dec_limit)[0].size
-    n_var_dec = max(1, n_var_dec)  # keep at least one variable
-    n_var_dec = min(5, n_var_dec)  # use at most 5 variables
+
+    if auto_ordering:
+        n_var_dec = np.where(np.cumsum(sensitivity_indices) < dec_limit)[0].size
+        n_var_dec = max(1, n_var_dec)  # keep at least one variable
+        n_var_dec = min(5, n_var_dec)  # use at most 5 variables
+    else:
+        n_var_dec = inputs.shape[1]
 
     # 2. states formation
     if states is None:
         states = 3 if n_var_dec < 3 else 2
         states = [states] * n_var_dec
 
-        # categorical for a given variable
-        for i in cat_cols_idx:
+        for i in range(n_var_dec):
             n_unique = np.unique(inputs[:, i]).size
-            states[i] = n_unique if n_unique < 5 else states[i]
+            states[i] = n_unique if n_unique <= 5 else states[i]
 
     if auto_ordering:
         var_names = var_names[var_order[:n_var_dec]].tolist()
@@ -145,8 +160,20 @@ def decomposition(
         bins.append(inputs)
         return statistic_method(inputs)
 
+    # make bins with equal number of samples for a given dimension
+    # sort and then split in n-state
+    sorted_inputs = np.sort(inputs, axis=0)
+    bin_edges = []
+    for i, states_ in enumerate(states):
+        splits = np.array_split(sorted_inputs[:, i], states_)
+        bin_edges_ = [splits_[0] for splits_ in splits]
+        bin_edges_.append(splits[-1][-1])  # last point to close the edges
+        # bin_edges_ = np.unique(bin_edges_)  # remove duplicate points, sorted
+        bin_edges_ += 1e-10 * np.linspace(0, 1, len(bin_edges_))
+        bin_edges.append(bin_edges_)
+
     res = stats.binned_statistic_dd(
-        inputs, values=output, statistic=statistic_, bins=states
+        inputs, values=output, statistic=statistic_, bins=bin_edges
     )
 
     bins = pd.DataFrame(bins[1:]).T

{simdec-1.1.0 → simdec-1.2.0}/src/simdec/visualization.py

@@ -1,6 +1,9 @@
+import copy
+import functools
 import itertools
 from typing import Literal
 
+import colorsys
 import matplotlib as mpl
 import matplotlib.pyplot as plt
 import numpy as np
@@ -11,7 +14,12 @@ from pandas.io.formats.style import Styler
 __all__ = ["visualization", "tableau", "palette"]
 
 
-sequential_palettes = [
+SEQUENTIAL_PALETTES = [
+    "#DC267F",
+    "#E8EA2F",
+    "#26DCD1",
+    "#C552E4",
+    "#3F45D0",
     "Oranges",
     "Purples",
     "Reds",
@@ -33,7 +41,58 @@ sequential_palettes = [
 ]
 
 
-def palette(states: list[int]) -> list[list[float]]:
+@functools.cache
+def sequential_cmaps():
+    cmaps = []
+    for cmap in SEQUENTIAL_PALETTES:
+        try:
+            cmap_ = mpl.colormaps[cmap]
+        except KeyError:
+            color = mpl.colors.hex2color(cmap)
+            cmap_ = single_color_to_colormap(color)
+        cmaps.append(cmap_)
+    return cmaps
+
+
+def single_color_to_colormap(
+    rgba_color: list[float] | str, *, factor: float = 0.5
+) -> mpl.colors.LinearSegmentedColormap:
+    """Create a linear colormap using a single color."""
+    if isinstance(rgba_color, str):
+        rgba_color = mpl.colors.hex2color(rgba_color)
+    # discard alpha channel
+    if len(rgba_color) == 4:
+        *rgb_color, alpha = rgba_color
+    else:
+        alpha = 1.0
+        rgb_color = rgba_color
+        rgba_color = list(rgba_color) + [1]
+
+    # lighten and darken from factor around single color
+    hls_color = colorsys.rgb_to_hls(*rgb_color)
+
+    lightness = hls_color[1]
+    lightened_hls_color = (hls_color[0], lightness * (1 + factor), hls_color[2])
+    lightened_rgb_color = list(colorsys.hls_to_rgb(*lightened_hls_color))
+
+    darkened_hls_color = (hls_color[0], lightness * (1 - factor), hls_color[2])
+    darkened_rgb_color = list(colorsys.hls_to_rgb(*darkened_hls_color))
+
+    lightened_rgba_color = lightened_rgb_color + [alpha]
+    darkened_rgba_color = darkened_rgb_color + [alpha]
+
+    # convert to CMAP
+    cmap = mpl.colors.LinearSegmentedColormap.from_list(
+        "CustomSingleColor",
+        [lightened_rgba_color, rgba_color, darkened_rgba_color],
+        N=3,
+    )
+    return cmap
+
+
+def palette(
+    states: list[int], cmaps: list[mpl.colors.LinearSegmentedColormap] = None
+) -> list[list[float]]:
     """Colour palette.
 
     The product of the states gives the number of scenarios. For each
@@ -43,20 +102,32 @@ def palette(states: list[int]) -> list[list[float]]:
     ----------
     states : list of int
         List of possible states for the considered parameter.
-
+    cmaps : list of LinearSegmentedColormap
+        List of colormaps. Must have the same number of colormaps as the number
+        of first level of states.
     Returns
     -------
-    palette : list of int of size (n, 4)
-        List of colours corresponding to scenarios.
+    palette : list of float of size (n, 4)
+        List of colors corresponding to scenarios. RGBA formatted.
     """
+    n_cmaps = states[0]
+    if cmaps is None:
+        cmaps = sequential_cmaps()[:n_cmaps]
+    else:
+        cmaps = cmaps[:n_cmaps]
+        if len(cmaps) != n_cmaps:
+            raise ValueError(
+                f"Must have the same number of cmaps ({len(cmaps)}) as the "
+                f"number of first states ({n_cmaps})"
+            )
+
     colors = []
     # one palette per first level state, could use more palette when there are
     # many levels
     n_shades = int(np.prod(states[1:]))
-    for i in range(states[0]):
-        palette_ = sequential_palettes[i]
-        cmap = mpl.colormaps[palette_].resampled(n_shades + 1)
-        colors.append(cmap(range(1, n_shades + 1)))
+    for i in range(n_cmaps):
+        cmap = cmaps[i].resampled(n_shades)
+        colors.append(cmap(np.linspace(0, 1, n_shades)))
 
     return np.concatenate(colors).tolist()
 
@@ -108,7 +179,7 @@ def visualization(
     elif kind == "boxplot":
         ax = sns.boxplot(
             bins,
-            palette=palette[::-1],
+            palette=palette,
             orient="h",
             order=list(bins.columns)[::-1],
             ax=ax,
@@ -132,6 +203,8 @@ def tableau(
     ----------
     var_names : list of str
         Variables name.
+    statistic : ndarray of shape (n_factors, 1)
+        Statistic in each bin.
     states : list of int or list of str
         For each variable, number of states. Can either be a scalar or a list.
 
@@ -155,16 +228,17 @@ def tableau(
     table.rename(columns={"index": "colour"}, inplace=True)
 
     # Default states for 2 or 3
-    for i, state in enumerate(states):
+    states_ = copy.deepcopy(states)
+    for i, state in enumerate(states_):
         if isinstance(state, int):
-            states: list
+            states_: list
             if state == 2:
-                states[i] = ["low", "high"]
+                states_[i] = ["low", "high"]
             elif state == 3:
-                states[i] = ["low", "medium", "high"]
+                states_[i] = ["low", "medium", "high"]
 
     # get the list of states
-    gen_states = [range(x) if isinstance(x, int) else x for x in states]
+    gen_states = [range(x) if isinstance(x, int) else x for x in states_]
     states_ = np.asarray(list(itertools.product(*gen_states)))
     for i, var_name in enumerate(var_names):
         table.insert(loc=i + 1, column=var_name, value=states_[:, i])
@@ -180,10 +254,10 @@ def tableau(
     # only select/ordering interesting columns
     table = table[["colour", "std", "min", "mean", "max", "probability"]]
 
-    table.insert(loc=0, column="N°", value=np.arange(1, stop=len(table) + 1))
+    table.insert(loc=0, column="N°", value=np.arange(1, stop=len(table) + 1)[::-1])
 
     # style the colour background with palette
-    cmap = mpl.colors.ListedColormap(palette[::-1])
+    cmap = mpl.colors.ListedColormap(palette)
     styler = table.style
     styler.format(precision=2)
     styler.background_gradient(subset=["colour"], cmap=cmap)

simdec-1.1.0/Dockerfile

@@ -1,47 +0,0 @@
-## Step 1/3: Base image
-FROM python:3.11-slim-bookworm as base
-RUN \
-    --mount=type=cache,target=/var/cache/apt \
-    apt-get update && apt-get upgrade -y && apt install curl -y
-
-## Step 2/3: Build/Install app dependencies
-FROM base as builder
-# Allow statements and log messages to immediately appear in the Knative logs
-ENV PYTHONUNBUFFERED True
-
-COPY pyproject.toml README.md /
-
-RUN --mount=type=cache,target=/var/cache/apt apt install -y gcc git
-RUN --mount=type=cache,mode=0777,target=/root/.cache/pip \
-    pip install --upgrade pip setuptools wheel && \
-    pip install -e .  && \
-    mkdir -p /app/src
-
-COPY tests/data /app/tests/data
-COPY src/simdec /app/src/simdec
-COPY app.py /app/app.py
-# matplotlib cache
-RUN mkdir -p /app/.config/matplotlib
-
-# Step 3/3: Image
-FROM  base as panel
-# Allow statements and log messages to immediately appear in the Knative logs
-ENV PYTHONUNBUFFERED True
-ENV PYTHONPATH=/app/src
-ENV PYTHONIOENCODING=utf-8
-ENV ENV=production
-ENV PORT=8080
-
-ARG PANEL_TOKEN
-
-COPY --from=builder /usr/local/lib/python3.11/site-packages/ /usr/local/lib/python3.11/site-packages/
-COPY --from=builder /usr/local/bin/panel /usr/local/bin/panel
-COPY --from=builder /app/src /app/src
-COPY --from=builder /app/tests/data /app/tests/data
-COPY --from=builder /app/app.py /app/app.py
-
-WORKDIR /app
-RUN useradd app && usermod -a -G app app
-USER app
-# Run the web service on container startup.
-CMD panel serve app.py --address 0.0.0.0 --port $PORT --allow-websocket-origin="*" --basic-auth=$PANEL_TOKEN --num-procs 2 --cookie-secret my_super_safe_cookie_secret

simdec-1.1.0/Makefile

@@ -1,95 +0,0 @@
-.PHONY: help prepare doc test serve build publish-production
-.DEFAULT_GOAL := help
-SHELL:=/bin/bash
-
-ifndef version
-   override version = development
-endif
-
-ifndef region
-   override region = europe-north1
-endif
-
-ifndef project
-   override project = delta-entity-401706
-endif
-
-ifndef PANEL_TOKEN
-   # only for local dev, another token is used for deployment
-   override PANEL_TOKEN = e41ea4c145bf13a60c8779c24356
-endif
-
-
-# Add help text after each target name starting with '\#\#'
-help:   ## show this help
-	@echo -e "Help for this makefile\n"
-	@echo "Possible commands are:"
-	@grep -h "##" $(MAKEFILE_LIST) | grep -v grep | sed -e 's/\(.*\):.*##\(.*\)/    \1: \2/'
-
-prepare:  ## Install dependencies and pre-commit hook
-	pip install -e ".[dev]"
-	pre-commit install
-	gcloud init
-	gcloud auth configure-docker
-
-doc:  ## Build Sphinx documentation
-	sphinx-build -b html docs docs/html
-
-test:  ## Run tests with coverage
-	pytest --cov simdec --cov-report term-missing
-
-serve-dev:  ## Serve Panel dashboard - Dev mode
-	panel serve app.py --show --autoreload
-
-serve:  ## Serve Panel dashboard - Prod mode
-	panel serve app.py
-
-build-local:
-	docker build -f ./Dockerfile \
-		--build-arg PANEL_TOKEN=$(PANEL_TOKEN) \
-		--tag simdec-panel:$(version) \
-	    --pull \
-	    ./.
-
-run-local: build-local
-	docker run --rm -it \
-    --name=simdec-panel \
-	--memory=1g \
-	--cpuset-cpus=0 \
-	-e ENV=development \
-	-p "8080:8080" \
-	simdec-panel:$(version)
-
-build:
-	docker build -f ./Dockerfile \
-		--platform linux/amd64 \
-		--build-arg PANEL_TOKEN=$(PANEL_TOKEN) \
-		--tag simdec-panel:$(version) \
-	    --pull \
-	    ./.
-
-run: build
-	docker run --rm -it \
-    --name=simdec-panel \
-	--memory=1g \
-	--cpuset-cpus=0 \
-	-e ENV=development \
-	-p "8080:8080" \
-	simdec-panel:$(version)
-
-publish-production: build
-	docker tag simdec-panel:$(version) $(region)-docker.pkg.dev/$(project)/simdec-panel/simdec-panel:$(version)
-	docker push $(region)-docker.pkg.dev/$(project)/simdec-panel/simdec-panel:$(version)
-
-production: publish-production
-	@echo "Deploying '$(version)'."
-	gcloud run deploy simdec-panel \
-	                  --concurrency=10 \
-	                  --max-instances=1 \
-	                  --region=$(region) \
-	                  --port=8080 \
-	                  --set-env-vars ENV=production \
-	                  --allow-unauthenticated \
-	                  --timeout=600 \
-	                  --image=$(region)-docker.pkg.dev/$(project)/simdec-panel/simdec-panel:$(version) \
-	                  --memory 1Gi

simdec-1.1.0/app.py

@@ -1,358 +0,0 @@
-import bisect
-import io
-
-from bokeh.models import PrintfTickFormatter
-from bokeh.models.widgets.tables import NumberFormatter
-import matplotlib.pyplot as plt
-import numpy as np
-import pandas as pd
-import panel as pn
-
-import simdec as sd
-
-
-# panel app
-pn.extension(template="material")
-pn.extension("tabulator")
-
-pn.config.sizing_mode = "stretch_width"
-pn.config.throttled = True
-font_size = "12pt"
-
-
-@pn.cache
-def load_data(text_fname):
-    if text_fname is None:
-        text_fname = "tests/data/stress.csv"
-    else:
-        text_fname = io.BytesIO(text_fname)
-
-    data = pd.read_csv(text_fname)
-    pn.state.clear_caches()
-    return data
-
-
-@pn.cache
-def column_inputs(data, output):
-    inputs = list(data.columns)
-    inputs.remove(output)
-    return inputs
-
-
-@pn.cache
-def filtered_inputs(data, input_names):
-    return data[input_names]
-
-
-@pn.cache
-def column_output(data):
-    return list(data.columns)
-
-
-@pn.cache
-def filtered_output(data, output_name):
-    return data[output_name]
-
-
-@pn.cache
-def sensitivity_indices(inputs, output):
-    sensitivity_indices = sd.sensitivity_indices(inputs=inputs, output=output)
-    if 0.01 < sum(sensitivity_indices.si) < 2.0:
-        indices = sensitivity_indices.si
-    else:
-        indices = sensitivity_indices.first_order
-    return indices
-
-
-def sensitivity_indices_table(si, inputs):
-    var_names = inputs.columns
-    var_order = np.argsort(si)[::-1]
-    var_names = var_names[var_order].tolist()
-
-    si = list(si[var_order])
-    sum_si = sum(si)
-
-    var_names.append("Sum of Indices")
-    si_numerics = si.copy()
-    si.append(0)
-    si_numerics.append(sum_si)
-
-    d = {"Inputs": var_names, "Indices": si, "": si_numerics}
-    df = pd.DataFrame(data=d)
-    formatters = {
-        "Indices": {"type": "progress", "max": sum_si, "color": "#007eff"},
-        "": NumberFormatter(format="0.00"),
-    }
-    widget = pn.widgets.Tabulator(
-        df,
-        show_index=False,
-        formatters=formatters,
-        theme="bulma",
-    )
-    widget.style.apply(
-        lambda x: ["font-style: italic"] * 3, axis=1, subset=df.index[-1]
-    )
-    widget.style.apply(lambda x: ["font-size: 11pt"] * len(si))
-    return widget
-
-
-@pn.cache
-def explained_variance(si):
-    return sum(si) + np.finfo(np.float64).eps
-
-
-def filtered_si(sensitivity_indices_table, input_names):
-    df = sensitivity_indices_table.value
-    si = []
-    for input_name in input_names:
-        si.append(df.loc[df["Inputs"] == input_name, "Indices"])
-    return np.asarray(si).flatten()
-
-
-def explained_variance_80(sensitivity_indices_table):
-    si = sensitivity_indices_table.value["Indices"]
-    pos_80 = bisect.bisect_right(np.cumsum(si), 0.8)
-
-    # pos_80 = max(2, pos_80)
-    # pos_80 = min(len(si), pos_80)
-
-    input_names = sensitivity_indices_table.value["Inputs"]
-    return input_names.to_list()[: pos_80 + 1]
-
-
-def decomposition(dec_limit, si, inputs, output):
-    return sd.decomposition(
-        inputs=inputs,
-        output=output,
-        sensitivity_indices=si,
-        dec_limit=dec_limit,
-        auto_ordering=False,
-    )
-
-
-def palette(res):
-    return sd.palette(res.states)
-
-
-def n_bins_auto(res):
-    min_ = np.nanmin(res.bins)
-    max_ = np.nanmax(res.bins)
-    return len(np.histogram_bin_edges(res.bins, bins="auto", range=(min_, max_))) - 1
-
-
-def display_n_bins(kind):
-    if kind != "Stacked histogram":
-        return
-
-    return selector_n_bins
-
-
-def figure(res, palette, n_bins, kind, output_name):
-    kind = "histogram" if kind == "Stacked histogram" else "boxplot"
-    plt.close("all")
-    fig, ax = plt.subplots()
-    _ = sd.visualization(
-        bins=res.bins, palette=palette, n_bins=n_bins, kind=kind, ax=ax
-    )
-    ax.set(xlabel=output_name)
-    return fig
-
-
-def states_from_data(res, inputs):
-    return sd.states_expansion(states=res.states, inputs=inputs)
-
-
-def tableau(res, states, palette):
-    # use a notebook to see the styling
-    _, styler = sd.tableau(
-        statistic=res.statistic,
-        var_names=res.var_names,
-        states=states,
-        bins=res.bins,
-        palette=palette,
-    )
-    return styler
-
-
-def tableau_states(res, states):
-    data = []
-    for var_name, states_, bin_edges in zip(res.var_names, states, res.bin_edges):
-        for i, state in enumerate(states_):
-            data.append([var_name, state, bin_edges[i], bin_edges[i + 1]])
-
-    table = pd.DataFrame(data, columns=["variable", "state", "min", "max"])
-    table.set_index(["variable", "state"], inplace=True)
-
-    styler = table.style
-    styler.format(precision=2)
-    styler.set_table_styles([{"selector": "th", "props": [("text-align", "center")]}])
-    return styler
-
-
-# Bindings
-text_fname = pn.widgets.FileInput(sizing_mode="stretch_width", accept=".csv")
-
-interactive_file = pn.bind(load_data, text_fname)
-
-interactive_column_output = pn.bind(column_output, interactive_file)
-selector_output = pn.widgets.Select(name="Output", options=interactive_column_output)
-interactive_output = pn.bind(filtered_output, interactive_file, selector_output)
-
-interactive_column_input = pn.bind(column_inputs, interactive_file, selector_output)
-selector_inputs_sensitivity = pn.widgets.MultiSelect(
-    name="Inputs", value=interactive_column_input, options=interactive_column_input
-)
-interactive_inputs = pn.bind(
-    filtered_output, interactive_file, selector_inputs_sensitivity
-)
-
-interactive_sensitivity_indices = pn.bind(
-    sensitivity_indices, interactive_inputs, interactive_output
-)
-interactive_explained_variance = pn.bind(
-    explained_variance, interactive_sensitivity_indices
-)
-
-interactive_sensitivity_indices_table = pn.bind(
-    sensitivity_indices_table, interactive_sensitivity_indices, interactive_inputs
-)
-
-interactive_explained_variance_80 = pn.bind(
-    explained_variance_80, interactive_sensitivity_indices_table
-)
-selector_inputs_decomposition = pn.widgets.MultiChoice(
-    name="Select inputs for decomposition",
-    value=interactive_explained_variance_80,
-    options=selector_inputs_sensitivity,
-    solid=False,
-)
-interactive_inputs_decomposition = pn.bind(
-    filtered_output, interactive_file, selector_inputs_decomposition
-)
-
-interactive_filtered_si = pn.bind(
-    filtered_si, interactive_sensitivity_indices_table, selector_inputs_decomposition
-)
-interactive_filtered_explained_variance = pn.bind(
-    explained_variance, interactive_filtered_si
-)
-indicator_explained_variance = pn.indicators.Number(
-    name="Explained variance ratio from selected inputs:",
-    value=interactive_filtered_explained_variance,
-    title_size=font_size,
-    font_size=font_size,
-    format="{value:.2f}",
-)
-
-
-interactive_decomposition = pn.bind(
-    decomposition,
-    interactive_explained_variance,
-    interactive_filtered_si,
-    interactive_inputs_decomposition,
-    interactive_output,
-)
-interactive_palette = pn.bind(palette, interactive_decomposition)
-
-switch_histogram_boxplot = pn.widgets.RadioButtonGroup(
-    name="Switch histogram - boxplot",
-    options=["Stacked histogram", "Boxplot"],
-)
-
-interactive_n_bins_auto = pn.bind(n_bins_auto, interactive_decomposition)
-selector_n_bins = pn.widgets.EditableIntSlider(
-    name="Number of bins",
-    start=0,
-    end=100,
-    value=interactive_n_bins_auto,
-    step=10,
-    # bar_color="#FFFFFF",  # does not work
-    format=PrintfTickFormatter(format="%d bins"),
-)
-conditional_selector_n_bins = pn.bind(display_n_bins, switch_histogram_boxplot)
-
-
-interactive_figure = pn.bind(
-    figure,
-    interactive_decomposition,
-    interactive_palette,
-    selector_n_bins,
-    switch_histogram_boxplot,
-    selector_output,
-)
-
-
-interactive_states = pn.bind(
-    states_from_data, interactive_decomposition, interactive_inputs_decomposition
-)
-interactive_tableau = pn.bind(
-    tableau, interactive_decomposition, interactive_states, interactive_palette
-)
-interactive_tableau_states = pn.bind(
-    tableau_states, interactive_decomposition, interactive_states
-)
-
-# App layout
-
-top_description = """
-## Data
-"""
-
-si_description = """
-## Sensitivity Indices
-"""
-
-decomposition_description = """
-## Decomposition
-"""
-
-table_description = """
-## Scenarios
-"""
-
-states_description = """
-## Details on inputs' states
-"""
-
-
-pn_params = pn.layout.WidgetBox(
-    pn.pane.Markdown(top_description, styles={"color": "#0072b5"}),
-    text_fname,
-    selector_output,
-    selector_inputs_sensitivity,
-    pn.pane.Markdown(decomposition_description, styles={"color": "#0072b5"}),
-    selector_inputs_decomposition,
-    indicator_explained_variance,
-    pn.pane.Markdown("## Visualization", styles={"color": "#0072b5"}),
-    switch_histogram_boxplot,
-    conditional_selector_n_bins,
-    max_width=350,
-    sizing_mode="stretch_width",
-).servable(area="sidebar")
-
-pn_app = pn.Column(
-    pn.Row(
-        pn.Column(
-            pn.panel(
-                pn.pane.Matplotlib(
-                    interactive_figure,
-                    tight=True,
-                    format="svg",
-                    sizing_mode="stretch_both",
-                    max_height=500,
-                    height_policy="min",
-                )
-            ),
-            pn.Spacer(height=50),
-            pn.pane.Markdown(si_description, styles={"color": "#0072b5"}),
-            pn.Column(interactive_sensitivity_indices_table, width=400),
-        ),
-        pn.Column(
-            pn.pane.Markdown(table_description, styles={"color": "#0072b5"}),
-            pn.panel(interactive_tableau),
-            pn.Spacer(height=125),
-            pn.pane.Markdown(states_description, styles={"color": "#0072b5"}),
-            pn.panel(interactive_tableau_states),
-        ),
-    )
-).servable(title="Simulation Decomposition Dashboard")