ml-peg 0.2.1__tar.gz

ml_peg-0.2.1/PKG-INFO

@@ -0,0 +1,198 @@
+Metadata-Version: 2.3
+Name: ml-peg
+Version: 0.2.1
+Summary: ML potential usability and performance guide
+Author: Elliott Kasoar, Joseph Hart, Ilyes Batatia, Alin M. Elena, Gábor Csányi
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
+Classifier: Natural Language :: English
+Classifier: Development Status :: 3 - Alpha
+Requires-Dist: boto3>=1.40.49,<2
+Requires-Dist: dash>=3.1.1
+Requires-Dist: janus-core>=0.8.2,<1.0.0
+Requires-Dist: kaleido>=1.0.0
+Requires-Dist: mlipx>=0.1.5,<0.2
+Requires-Dist: scikit-learn>=1.7.1
+Requires-Dist: typer>=0.19.1,<1.0.0
+Requires-Dist: matcalc
+Requires-Dist: matminer
+Requires-Dist: mdanalysis
+Requires-Dist: chgnet==0.4.0 ; extra == 'chgnet'
+Requires-Dist: torch-dftd==0.5.1 ; extra == 'd3'
+Requires-Dist: deepmd-kit==3.1.0 ; extra == 'dpa3'
+Requires-Dist: tensorpotential==0.5.1 ; python_full_version < '3.13' and extra == 'grace'
+Requires-Dist: mace-torch==0.3.14 ; extra == 'mace'
+Requires-Dist: mattersim==1.2.0 ; extra == 'mattersim'
+Requires-Dist: orb-models==0.5.5 ; python_full_version < '3.13' and sys_platform != 'win32' and extra == 'orb'
+Requires-Dist: pet-mad==1.4.4 ; sys_platform != 'win32' and extra == 'pet-mad'
+Requires-Dist: fairchem-core==2.10.0 ; extra == 'uma'
+Requires-Python: >=3.10
+Project-URL: Repository, https://github.com/ddmms/ml-peg/
+Project-URL: Documentation, https://ddmms.github.io/ml-peg/
+Provides-Extra: chgnet
+Provides-Extra: d3
+Provides-Extra: dpa3
+Provides-Extra: grace
+Provides-Extra: mace
+Provides-Extra: mattersim
+Provides-Extra: orb
+Provides-Extra: pet-mad
+Provides-Extra: uma
+Description-Content-Type: text/markdown
+
+# ML-PEG: ML potential usability and performance guide
+
+[![PyPI version][pypi-badge]][pypi-link]
+[![Python versions][python-badge]][python-link]
+[![Build Status][ci-badge]][ci-link]
+[![Docs status][docs-badge]][docs-link]
+[![License][license-badge]][license-link]
+[![DOI][doi-badge]][doi-link]
+
+🔗 See our live guide: https://ml-peg.stfc.ac.uk
+
+> [!NOTE]
+> Migration in progress! The live benchmarks are currently run and analysed using
+> [mlipx](https://github.com/basf/mlipx) nodes defined in this repository:
+> https://github.com/joehart2001/mlipx.
+>
+> New benchmarks are expected to be added following the format defined in this
+> repository, and work is ongoing to migrate all existing benchmarks to this format.
+> Our original interactive analysis suite is currently hosted at: http://mlip-testing.stfc.ac.uk:8050
+
+## Contents
+- [Getting started](#getting-started)
+- [Features](#features)
+- [Docker/Podman images](#dockerpodman-images)
+- [Development](#development)
+- [License](#license)
+
+## Getting started
+
+### Dependencies
+
+All required and optional dependencies can be found in [pyproject.toml](pyproject.toml).
+
+
+### Installation
+
+The latest stable release of ML-PEG, including its dependencies, will be installable from PyPI by running:
+
+```
+python3 -m pip install ml-peg
+```
+
+To get all the latest changes, ML-PEG can be installed from GitHub:
+
+```
+python3 -m pip install git+https://github.com/ddmms/ml-peg.git
+```
+
+
+## Features
+
+Coming soon!
+
+
+## Docker/Podman images
+
+You can use [Docker](https://www.docker.com) or [Podman](https://podman.io/) to build
+and/or run the ML-PEG app yourself.
+
+> [!TIP]
+> The commands below will assume you are using Docker. To use Podman, replace `docker`
+> with `podman`, e.g. `podman pull`, `podman build`, and `podman run`.
+
+A Docker image with the latest changes can be pulled from the
+GitHub container registry, following the command that can be found under this
+repository's [packages](https://github.com/ddmms/ml-peg/pkgs/container/ml-peg-app).
+
+
+> [!NOTE]
+> Currently, this repository only contains images for the linux/amd64 platform.
+> On MacOS with ARM silicon, this can often still be run by setting
+> `--platform linux/amd64` when using `docker run`.
+
+
+Alternatively, to build the container yourself, you can use the
+[Dockerfile](containers/Dockerfile) provided. From the `ml-peg` directory, run:
+
+```
+docker build -t ml-peg-app -f containers/Dockerfile .
+```
+
+Once built, you can mount your current application data and start the app by running:
+
+```
+docker run --volume ./ml_peg/app/data:/app/ml_peg/app/data  --publish 8050:8050 ml-peg-app
+```
+
+> [!TIP]
+> Ensure `ml_peg/app/data` is populated with results before running the container.
+>
+> A compressed zip file containing the current live data can be found at
+> http://s3.echo.stfc.ac.uk/ml-peg-data/app/data/data.tar.gz.
+>
+> This may also be downloaded through the command line using
+> ```
+> ml_peg download --key app/data/data.tar.gz  --filename data.tar.gz
+> ```
+
+
+Alternatively, you can use the [compose.yml](containers/compose.yml) file provided, via
+Docker Compose:
+
+```
+docker compose -f containers/compose.yml up -d
+```
+
+The app should now be accessible at http://localhost:8050.
+
+## Development
+
+Please ensure you have consulted our
+[contribution guidelines](contributing.md)
+and
+[coding style](coding_style.md)
+before proceeding.
+
+We recommend installing `uv` for dependency management when developing for ML-PEG:
+
+1. Install [uv](https://docs.astral.sh/uv/getting-started/installation)
+2. Install ML-PEG with dependencies in a virtual environment:
+
+```shell
+git clone https://github.com/ddmms/ml-peg
+cd ml-peg
+uv sync # Create a virtual environment and install dependencies
+source .venv/bin/activate
+pre-commit install  # Install pre-commit hooks
+pytest -v  # Discover and run all tests
+```
+
+Please refer to the [online documentation](https://ddmms.github.io/ml-peg/developer_guide/index.html)
+for information about contributing new benchmarks and models.
+
+## License
+
+[GNU General Public License version 3](LICENSE)
+
+[pypi-badge]: https://badge.fury.io/py/ml-peg.svg
+[pypi-link]: https://pypi.org/project/ml-peg/
+[python-badge]: https://img.shields.io/pypi/pyversions/ml-peg.svg
+[python-link]: https://pypi.org/project/ml-peg/
+[ci-badge]: https://github.com/ddmms/ml-peg/actions/workflows/ci.yml/badge.svg?branch=main
+[ci-link]: https://github.com/ddmms/ml-peg/actions
+[cov-badge]: https://coveralls.io/repos/github/ddmms/ml-peg/badge.svg?branch=main
+[cov-link]: https://coveralls.io/github/ddmms/ml-peg?branch=main
+[docs-badge]: https://github.com/ddmms/ml-peg/actions/workflows/docs.yml/badge.svg
+[docs-link]: https://ddmms.github.io/ml-peg/
+[license-badge]: https://img.shields.io/badge/License-GPLv3-blue.svg
+[license-link]: https://opensource.org/license/gpl-3-0
+[doi-link]: https://doi.org/10.5281/zenodo.16904444
+[doi-badge]: https://zenodo.org/badge/DOI/10.5281/zenodo.16904444.svg

ml_peg-0.2.1/README.md

@@ -0,0 +1,151 @@
+# ML-PEG: ML potential usability and performance guide
+
+[![PyPI version][pypi-badge]][pypi-link]
+[![Python versions][python-badge]][python-link]
+[![Build Status][ci-badge]][ci-link]
+[![Docs status][docs-badge]][docs-link]
+[![License][license-badge]][license-link]
+[![DOI][doi-badge]][doi-link]
+
+🔗 See our live guide: https://ml-peg.stfc.ac.uk
+
+> [!NOTE]
+> Migration in progress! The live benchmarks are currently run and analysed using
+> [mlipx](https://github.com/basf/mlipx) nodes defined in this repository:
+> https://github.com/joehart2001/mlipx.
+>
+> New benchmarks are expected to be added following the format defined in this
+> repository, and work is ongoing to migrate all existing benchmarks to this format.
+> Our original interactive analysis suite is currently hosted at: http://mlip-testing.stfc.ac.uk:8050
+
+## Contents
+- [Getting started](#getting-started)
+- [Features](#features)
+- [Docker/Podman images](#dockerpodman-images)
+- [Development](#development)
+- [License](#license)
+
+## Getting started
+
+### Dependencies
+
+All required and optional dependencies can be found in [pyproject.toml](pyproject.toml).
+
+
+### Installation
+
+The latest stable release of ML-PEG, including its dependencies, will be installable from PyPI by running:
+
+```
+python3 -m pip install ml-peg
+```
+
+To get all the latest changes, ML-PEG can be installed from GitHub:
+
+```
+python3 -m pip install git+https://github.com/ddmms/ml-peg.git
+```
+
+
+## Features
+
+Coming soon!
+
+
+## Docker/Podman images
+
+You can use [Docker](https://www.docker.com) or [Podman](https://podman.io/) to build
+and/or run the ML-PEG app yourself.
+
+> [!TIP]
+> The commands below will assume you are using Docker. To use Podman, replace `docker`
+> with `podman`, e.g. `podman pull`, `podman build`, and `podman run`.
+
+A Docker image with the latest changes can be pulled from the
+GitHub container registry, following the command that can be found under this
+repository's [packages](https://github.com/ddmms/ml-peg/pkgs/container/ml-peg-app).
+
+
+> [!NOTE]
+> Currently, this repository only contains images for the linux/amd64 platform.
+> On MacOS with ARM silicon, this can often still be run by setting
+> `--platform linux/amd64` when using `docker run`.
+
+
+Alternatively, to build the container yourself, you can use the
+[Dockerfile](containers/Dockerfile) provided. From the `ml-peg` directory, run:
+
+```
+docker build -t ml-peg-app -f containers/Dockerfile .
+```
+
+Once built, you can mount your current application data and start the app by running:
+
+```
+docker run --volume ./ml_peg/app/data:/app/ml_peg/app/data  --publish 8050:8050 ml-peg-app
+```
+
+> [!TIP]
+> Ensure `ml_peg/app/data` is populated with results before running the container.
+>
+> A compressed zip file containing the current live data can be found at
+> http://s3.echo.stfc.ac.uk/ml-peg-data/app/data/data.tar.gz.
+>
+> This may also be downloaded through the command line using
+> ```
+> ml_peg download --key app/data/data.tar.gz  --filename data.tar.gz
+> ```
+
+
+Alternatively, you can use the [compose.yml](containers/compose.yml) file provided, via
+Docker Compose:
+
+```
+docker compose -f containers/compose.yml up -d
+```
+
+The app should now be accessible at http://localhost:8050.
+
+## Development
+
+Please ensure you have consulted our
+[contribution guidelines](contributing.md)
+and
+[coding style](coding_style.md)
+before proceeding.
+
+We recommend installing `uv` for dependency management when developing for ML-PEG:
+
+1. Install [uv](https://docs.astral.sh/uv/getting-started/installation)
+2. Install ML-PEG with dependencies in a virtual environment:
+
+```shell
+git clone https://github.com/ddmms/ml-peg
+cd ml-peg
+uv sync # Create a virtual environment and install dependencies
+source .venv/bin/activate
+pre-commit install  # Install pre-commit hooks
+pytest -v  # Discover and run all tests
+```
+
+Please refer to the [online documentation](https://ddmms.github.io/ml-peg/developer_guide/index.html)
+for information about contributing new benchmarks and models.
+
+## License
+
+[GNU General Public License version 3](LICENSE)
+
+[pypi-badge]: https://badge.fury.io/py/ml-peg.svg
+[pypi-link]: https://pypi.org/project/ml-peg/
+[python-badge]: https://img.shields.io/pypi/pyversions/ml-peg.svg
+[python-link]: https://pypi.org/project/ml-peg/
+[ci-badge]: https://github.com/ddmms/ml-peg/actions/workflows/ci.yml/badge.svg?branch=main
+[ci-link]: https://github.com/ddmms/ml-peg/actions
+[cov-badge]: https://coveralls.io/repos/github/ddmms/ml-peg/badge.svg?branch=main
+[cov-link]: https://coveralls.io/github/ddmms/ml-peg?branch=main
+[docs-badge]: https://github.com/ddmms/ml-peg/actions/workflows/docs.yml/badge.svg
+[docs-link]: https://ddmms.github.io/ml-peg/
+[license-badge]: https://img.shields.io/badge/License-GPLv3-blue.svg
+[license-link]: https://opensource.org/license/gpl-3-0
+[doi-link]: https://doi.org/10.5281/zenodo.16904444
+[doi-badge]: https://zenodo.org/badge/DOI/10.5281/zenodo.16904444.svg

ml_peg-0.2.1/ml_peg/__init__.py

@@ -0,0 +1,7 @@
+"""Testing framework for machine learnt interatomic potentials."""
+
+from __future__ import annotations
+
+from importlib.metadata import version
+
+__version__ = version("ml-peg")

ml_peg-0.2.1/ml_peg/analysis/__init__.py

@@ -0,0 +1,7 @@
+"""Run analysis for testing framework."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+ANALYSIS_ROOT = Path(__file__).parent

ml_peg-0.2.1/ml_peg/analysis/bulk_crystal/elasticity/analyse_elasticity.py

@@ -0,0 +1,235 @@
+"""Analyse elasticity benchmark."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import pandas as pd
+import pytest
+
+from ml_peg.analysis.utils.decorators import (
+    build_table,
+    plot_density_scatter,
+)
+from ml_peg.analysis.utils.utils import (
+    build_density_inputs,
+    load_metrics_config,
+    mae,
+)
+from ml_peg.app import APP_ROOT
+from ml_peg.calcs import CALCS_ROOT
+from ml_peg.models.get_models import get_model_names
+from ml_peg.models.models import current_models
+
+MODELS = get_model_names(current_models)
+CALC_PATH = CALCS_ROOT / "bulk_crystal" / "elasticity" / "outputs"
+OUT_PATH = APP_ROOT / "data" / "bulk_crystal" / "elasticity"
+
+METRICS_CONFIG_PATH = Path(__file__).with_name("metrics.yml")
+DEFAULT_THRESHOLDS, DEFAULT_TOOLTIPS, DEFAULT_WEIGHTS = load_metrics_config(
+    METRICS_CONFIG_PATH
+)
+
+K_COLUMN = "K_vrh"
+G_COLUMN = "G_vrh"
+
+
+def _filter_results(df: pd.DataFrame, model_name: str) -> tuple[pd.DataFrame, int]:
+    """
+    Filter outlier predictions and return remaining data with exclusion count.
+
+    Parameters
+    ----------
+    df
+        Dataframe containing raw benchmark results.
+    model_name
+        Model whose columns should be filtered.
+
+    Returns
+    -------
+    tuple[pd.DataFrame, int]
+        Filtered dataframe and number of excluded systems.
+    """
+    mask_bulk = df[f"{K_COLUMN}_{model_name}"].between(-50, 600)
+    mask_shear = df[f"{G_COLUMN}_{model_name}"].between(-50, 600)
+    valid = df[mask_bulk & mask_shear].copy()
+    excluded = len(df) - len(valid)
+    return valid, excluded
+
+
+@pytest.fixture
+def elasticity_stats() -> dict[str, dict[str, Any]]:
+    """
+    Load and cache processed benchmark statistics per model.
+
+    Returns
+    -------
+    dict[str, dict[str, Any]]
+        Processed information per model (bulk, shear, exclusion counts).
+    """
+    OUT_PATH.mkdir(parents=True, exist_ok=True)
+    stats: dict[str, dict[str, Any]] = {}
+    for model_name in MODELS:
+        results_path = CALC_PATH / model_name / "moduli_results.csv"
+        df = pd.read_csv(results_path)
+
+        filtered, excluded = _filter_results(df, model_name)
+
+        stats[model_name] = {
+            "bulk": {
+                "ref": filtered[f"{K_COLUMN}_DFT"].tolist(),
+                "pred": filtered[f"{K_COLUMN}_{model_name}"].tolist(),
+            },
+            "shear": {
+                "ref": filtered[f"{G_COLUMN}_DFT"].tolist(),
+                "pred": filtered[f"{G_COLUMN}_{model_name}"].tolist(),
+            },
+            "excluded": excluded,
+        }
+
+    return stats
+
+
+@pytest.fixture
+def bulk_mae(elasticity_stats: dict[str, dict[str, Any]]) -> dict[str, float | None]:
+    """
+    Mean absolute error for bulk modulus predictions.
+
+    Parameters
+    ----------
+    elasticity_stats
+        Aggregated bulk/shear data per model.
+
+    Returns
+    -------
+    dict[str, float | None]
+        MAE values for each model (``None`` if no data).
+    """
+    results: dict[str, float | None] = {}
+    for model_name in MODELS:
+        prop = elasticity_stats.get(model_name, {}).get("bulk")
+        results[model_name] = mae(prop["ref"], prop["pred"])
+    return results
+
+
+@pytest.fixture
+def shear_mae(elasticity_stats: dict[str, dict[str, Any]]) -> dict[str, float | None]:
+    """
+    Mean absolute error for shear modulus predictions.
+
+    Parameters
+    ----------
+    elasticity_stats
+        Aggregated bulk/shear data per model.
+
+    Returns
+    -------
+    dict[str, float | None]
+        MAE values for each model (``None`` if no data).
+    """
+    results: dict[str, float | None] = {}
+    for model_name in MODELS:
+        prop = elasticity_stats.get(model_name, {}).get("shear")
+        results[model_name] = mae(prop["ref"], prop["pred"])
+    return results
+
+
+@pytest.fixture
+@plot_density_scatter(
+    filename=OUT_PATH / "figure_bulk_density.json",
+    title="Bulk modulus density plot",
+    x_label="Reference bulk modulus / GPa",
+    y_label="Predicted bulk modulus / GPa",
+)
+def bulk_density(elasticity_stats: dict[str, dict[str, Any]]) -> dict[str, dict]:
+    """
+    Density scatter inputs for bulk modulus.
+
+    Parameters
+    ----------
+    elasticity_stats
+        Aggregated bulk/shear data per model.
+
+    Returns
+    -------
+    dict[str, dict]
+        Mapping of model name to density-scatter data.
+    """
+    return build_density_inputs(MODELS, elasticity_stats, "bulk", metric_fn=mae)
+
+
+@pytest.fixture
+@plot_density_scatter(
+    filename=OUT_PATH / "figure_shear_density.json",
+    title="Shear modulus density plot",
+    x_label="Reference shear modulus / GPa",
+    y_label="Predicted shear modulus / GPa",
+)
+def shear_density(elasticity_stats: dict[str, dict[str, Any]]) -> dict[str, dict]:
+    """
+    Density scatter inputs for shear modulus.
+
+    Parameters
+    ----------
+    elasticity_stats
+        Aggregated bulk/shear data per model.
+
+    Returns
+    -------
+    dict[str, dict]
+        Mapping of model name to density-scatter data.
+    """
+    return build_density_inputs(MODELS, elasticity_stats, "shear", metric_fn=mae)
+
+
+@pytest.fixture
+@build_table(
+    filename=OUT_PATH / "elasticity_metrics_table.json",
+    metric_tooltips=DEFAULT_TOOLTIPS,
+    thresholds=DEFAULT_THRESHOLDS,
+    weights=DEFAULT_WEIGHTS,
+)
+def metrics(
+    bulk_mae: dict[str, float | None],
+    shear_mae: dict[str, float | None],
+) -> dict[str, dict]:
+    """
+    All elasticity metrics.
+
+    Parameters
+    ----------
+    bulk_mae
+        Bulk modulus MAE per model.
+    shear_mae
+        Shear modulus MAE per model.
+
+    Returns
+    -------
+    dict[str, dict]
+        Mapping of metric name to model-value dictionaries.
+    """
+    return {
+        "Bulk modulus MAE": bulk_mae,
+        "Shear modulus MAE": shear_mae,
+    }
+
+
+def test_elasticity(
+    metrics: dict[str, dict],
+    bulk_density: dict[str, dict],
+    shear_density: dict[str, dict],
+) -> None:
+    """
+    Run elasticity analysis.
+
+    Parameters
+    ----------
+    metrics
+        Benchmark metric values.
+    bulk_density
+        Density scatter inputs for bulk modulus.
+    shear_density
+        Density scatter inputs for shear modulus.
+    """
+    return

ml_peg-0.2.1/ml_peg/analysis/bulk_crystal/elasticity/metrics.yml

@@ -0,0 +1,13 @@
+metrics:
+  Bulk modulus MAE:
+    good: 2.0
+    bad: 30.0
+    unit: GPa
+    tooltip: Mean absolute error of VRH bulk modulus (lower is better). Excludes systems with bulk moduli < -50 GPa and > 500 GPa.
+    level_of_theory: PBE
+  Shear modulus MAE:
+    good: 2.0
+    bad: 30.0
+    unit: GPa
+    tooltip: Mean absolute error of VRH shear modulus (lower is better). Excludes systems with shear moduli < -50 GPa and > 500 GPa.
+    level_of_theory: PBE