@mat3ra/esse 2023.12.25-2

package/.babelrc

@@ -0,0 +1,10 @@
+{
+    "presets": [
+        "@babel/preset-env"
+    ],
+    "plugins": [
+        [
+            "@babel/plugin-proposal-class-properties"
+        ]
+    ]
+}

package/.eslintrc.json

@@ -0,0 +1,3 @@
+{
+    "extends": ["@exabyte-io/eslint-config"]
+}

package/.pre-commit-config.yaml

@@ -0,0 +1,20 @@
+repos:
+  - repo: https://github.com/Exabyte-io/pre-commit-hooks
+    rev: 2023.6.13
+    hooks:
+      - id: ruff
+        exclude: ^src/py/mat3ra/esse/data
+      - id: black
+        exclude: ^src/py/mat3ra/esse/data
+  - repo: local
+    hooks:
+      - id: generate-python-modules
+        name: Regenerate data modules classes from static assets
+        # yamllint disable rule:line-length
+        entry: |
+          bash -c 'if ! [ "$(git diff --cached --name-only | grep -e "example|schema")" ]; then echo "Skipping model generation because static files were not changed."; exit 0; fi'
+          bash -c 'python build_schemas.py'
+        # yamllint enable rule:line-length
+        language: system
+        pass_filenames: false
+        verbose: true

package/.prettierrc

@@ -0,0 +1,6 @@
+{
+    "singleQuote": false,
+    "printWidth": 100,
+    "trailingComma": "all",
+    "tabWidth": 4
+}

package/LICENSE.md

@@ -0,0 +1,15 @@
+# LICENSE
+
+Copyright 2019 Exabyte Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

package/README.md

@@ -0,0 +1,157 @@
+[![PyPI version](https://badge.fury.io/py/esse.svg)](https://badge.fury.io/py/esse)
+[![npm version](https://badge.fury.io/js/%40exabyte-io%2Fesse.js.svg)](https://badge.fury.io/js/%40exabyte-io%2Fesse.js)
+[![License: Apache](https://img.shields.io/badge/License-Apache-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
+
+# ESSE
+
+Essential Source of Schemas and Examples (ESSE) contains data formats and examples for common entities used in digital materials science (see refs. [1, 2](#links) below).
+
+Although, the schemas are used to facilitate the operations of [mat3ra.com](https://mat3ra.com), they are designed to be generic and can be used in other applications. The open-source packages developed by Mat3ra.com use the schemas available in this repository.
+
+The latest variants of schemas and examples are available at [schemas.mat3ra.com](https://schemas.mat3ra.com/).
+
+ESSE has a dual-nature as both a Python and a Node.js package.
+
+
+## 1. Installation
+
+### 1.1. Python
+
+ESSE is compatible with Python 3.8+.
+
+#### 1.1.1. PyPI
+
+```bash
+pip install mat3ra-esse
+```
+
+#### 1.1.2. Repository
+
+```bash
+virtualenv .venv
+source .venv/bin/activate
+pip install -e PATH_TO_ESSE_REPOSITORY
+```
+
+### 1.2. Node
+
+#### 1.2.1. NPM
+
+```bash
+npm install @exabyte-io/esse.js
+```
+
+
+## 2. Usage
+
+ESSE contains separate but equivalent interfaces for Python and Javascript.
+The package provides `ESSE` class that can be initialized and used as below.
+
+### 2.1. Usage in Python
+
+```python
+from mat3ra.esse import ESSE
+
+helper = ESSE()
+schema = helper.get_schema_by_id("material")
+```
+
+### 2.2. Usage in Node/JS/TS
+
+```javascript
+import {ESSE} from "esse-js";
+
+const helper = new ESSE();
+const schema = helper.getSchemaById("material");
+```
+
+
+## 3. Directory Structure
+
+ESSE contains 3 main directories, [schema](schema), [example](example) and [src](src) outlined below.
+
+### 3.1. Schema
+
+The schema directory contains the schemas specifying the rules to structure data. A set of core schemas, outlined below, are defined to facilitate the schema modularity.
+
+- [Primitive](schema/core/primitive) directory contains a set of custom primitives that extends default standard primitive types allowed by schema, such as String and Number.
+Primitives are solely defined by the default primitives and can not be re-constructed from each other.
+- [Abstract](schema/core/abstract) directory contains unit-less schemas that are constructed from default and custom primitives.
+- [Reusable](schema/core/reusable) directory contains the schemas that are widely used in other schemas to avoid duplication, constructed from the abstract and primitive schemas.
+- [Reference](schema/core/reference) directory contains the schemas defining the rules to structure the references to data sources.
+
+### 3.2. Example
+
+This directory contains the examples formed according to the schemas and implements the same directory structure as the schema directory.
+
+### 3.3. src
+
+This directory contains Python and Javascript interfaces implementing the functionality to access and validate schemas and examples.
+
+
+## 4. Conventions
+
+### 4.1. Generative vs Non-generative keys
+Generative keys are the fields which allow for user input prior to calculation of the final property values. A flag is included in the schema comments on the fields in [property schemas](schema/properties_directory): `isGenerative:true` marks which fields to use as subschemas in the generation of a user input schema. On properties allowing user inputs, additional fields may be tagged, as in [the `file_content` property](schema/properties_directory/non-scalar/file_content.json)
+
+
+## 5. Development
+
+The schemas and examples are stored as JSON assets. The JSON assets are used to generate JS/TS and PY modules that can be used to access the schemas and examples in the corresponding runtimes. The modules are generated using the [build_schemas.py](./build_schemas.py) and [build_schema.js](./build_schema.js) scripts. The JS modules are generated during the transpilation step of the npm. The PY modules are generated during the development and distributed within the pip package.
+
+The following outlines the development process workflow:
+
+1. Setup: clone the repository and install the dependencies for both JS and PY (as explained below).
+2. Edit code and commit changes.
+3. Pre commit is used to regenerate the modules.
+4. Push the changes to GitHub.
+5. GH workflow is used to generate the fully resolved file (without "$ref"s and "$allOf" etc.) and examples and publish them to [schemas.mat3ra.com](http://schemas.mat3ra.com/).
+6. Publish the new version of the package to PyPI and npm.
+
+The [pre-commit](.husky/pre-commit) is using both JS and PY runtime(s) to regenerate the schemas and examples. 
+
+[//]: # (TODO: consider reusing JS runtime and schemas build script for PY modules for consistency)
+NOTE: The PY and JS modules are built from the same JSON sources, but using different runtimes (scripts) and thus may still be different. Only for JS the fully resolved schemas (with merged "$allOf") are created. They are used for the docs website. 
+
+### 5.1. Development in Python
+
+When developing in python the following should be taken into account:
+
+1. The modules containing the schemas and examples are generated using the [build-schemas.py](./build_schemas.py) script. There is a setup for it to be run automatically on every commit, but it is recommended to run it manually before committing to make sure that the changes are reflected in the modules. This can be done with `pre-commit run --all-files`. The pre-commit package can be installed with `pip install pre-commit`. To rebuild schemas manually, run (note `-e` in install):
+    ```bash
+    virtualenv .venv
+    source .venv/bin/activate
+    pip install -e ."[tests]"
+    python build_schemas.py
+    ```
+2. Tests can be run using the following commands:
+    ```bash
+    virtualenv .venv
+    source .venv/bin/activate
+    pip install ."[tests]"
+    python -m unittest discover --verbose --catch --start-directory tests/py/esse/   
+    ```
+
+### 5.2. Development in Javascript/Typescript
+
+See [package.json](package.json) for the list of available npm commands. The JS modules are generated using the [build_schema.js](./build_schema.js) script. There is a setup for it to be run automatically when the package is installed (see "transpile" directive). To rebuild schemas manually, run:
+```bash
+npm install
+npm run transpile
+```
+
+### 5.3. General Dev Suggestions
+
+This repository is an [open-source](LICENSE.md) work-in-progress and we welcome contributions. We suggest forking this repository and introducing the adjustments there, the changes in the fork can further be considered for merging into this repository as it is commonly done on GitHub (see [3](#links) below).
+
+Other suggestions:
+
+- Use unique IDs for schemas
+- Do not use circular references in the schemas, instead leave the type as object and add explanation to description.
+
+
+## 6. Links
+
+1: [Data-centric online ecosystem for digital materials science](https://arxiv.org/pdf/1902.10838.pdf)
+2: [CateCom: A Practical Data-Centric Approach to Categorization of Computational Models](https://pubs.acs.org/doi/abs/10.1021/acs.jcim.2c00112)
+3: [GitHub Standard Fork & Pull Request Workflow](https://gist.github.com/Chaser324/ce0505fbed06b947d962)

package/build_schemas.js

@@ -0,0 +1,42 @@
+/**
+ * build_schemas uses node API to read all schemas from the FS
+ * at build time and writes them out to a single schemas.js file for
+ * downstream consumption to avoid FS calls in the browser.
+ */
+const fs = require("fs");
+const path = require("path");
+const mergeAllOf = require("json-schema-merge-allof");
+const { ESSE } = require("./lib/js/esse");
+
+const esse = new ESSE();
+const { schemas, wrappedExamples } = esse;
+const schema = esse.buildGlobalSchema();
+
+fs.writeFileSync(
+    "./schemas.js",
+    "module.exports = {schemas: " + JSON.stringify(schemas) + "}",
+    "utf8",
+);
+
+fs.writeFileSync("./schema.js", "module.exports = " + JSON.stringify(schema), "utf8");
+
+if (process.env.BUILD_ASSETS !== "true") {
+    process.exit(0);
+}
+
+const subfolder = process.env.BUILD_PATH || "./docs/js/";
+schemas.forEach((s) => {
+    if (process.env.SKIP_MERGE_ALLOF !== "true") {
+        s = mergeAllOf(s, { resolvers: { defaultResolver: mergeAllOf.options.resolvers.title } });
+    }
+    const id_as_path = s.$id.replace("-", "_");
+    const full_path = `${subfolder}/${id_as_path}.json`;
+    fs.mkdirSync(path.dirname(full_path), { recursive: true });
+    fs.writeFileSync(full_path, JSON.stringify(s, null, 4), "utf8");
+});
+wrappedExamples.forEach((e) => {
+    const id_as_path = e.path.replace("-", "_");
+    const full_path = `${subfolder}/${id_as_path}.json`;
+    fs.mkdirSync(path.dirname(full_path), { recursive: true });
+    fs.writeFileSync(full_path, JSON.stringify(e.data, null, 4), "utf8");
+});

package/build_schemas.py

@@ -0,0 +1,61 @@
+"""
+This script builds the schemas and examples modules.
+The schemas module contains all the schemas in the SCHEMAS_DIR.
+The examples module contains all the examples in the EXAMPLES_DIR.
+The script is aware of the include and reference statements and is using the json_include library to resolve them.
+The script creates the python modules containing the schemas and examples and avoids
+having the dependency on the filesystem during the calls to the ESSE class.
+Usage (NOTE: "-e" in pip install is required for the file references to be resolved):
+    virtualenv .venv
+    source .venv/bin/activate
+    pip install -e ."[tests]"
+    python build_schemas.py
+After that, check for the files `src/py/mat3ra/esse/data/schemas.py` and `src/py/mat3ra/esse/data/examples.py`.
+"""
+import os
+import json
+import yaml
+from pathlib import Path
+
+from mat3ra.esse.utils import parse_include_reference_statements_by_dir
+
+TOP_DIR = os.path.dirname(__file__)
+SCHEMAS = parse_include_reference_statements_by_dir(TOP_DIR)
+# Wrap examples into {"data": example, "path": schema_id} format to relate them to schemas.
+EXAMPLES = parse_include_reference_statements_by_dir(TOP_DIR, True)
+
+with open(os.path.join(TOP_DIR, "manifest/properties.yaml")) as f:
+    PROPERTIES_MANIFEST = yaml.load(f.read(), Loader=yaml.FullLoader)
+    RESULTS = [k for k, v in PROPERTIES_MANIFEST.items() if v.get("isResult")]
+
+with open("src/py/mat3ra/esse/data/schemas.py", "w") as f:
+    f.write("import json\n" + f"SCHEMAS = json.loads(json.dumps({SCHEMAS}))")
+
+with open("src/py/mat3ra/esse/data/examples.py", "w") as f:
+    f.write("import json\n" + f"EXAMPLES = json.loads(json.dumps({EXAMPLES}))")
+
+with open("src/py/mat3ra/esse/data/properties.py", "w") as f:
+    content = (
+        "import json\n"
+        + f"PROPERTIES_MANIFEST = json.loads(json.dumps({PROPERTIES_MANIFEST}))\n"
+        + f"RESULTS = json.loads(json.dumps({RESULTS}))\n"
+    )
+    f.write(content)
+
+if os.environ.get("BUILD_DOCS") != "true":
+    exit(0)
+
+for schema in SCHEMAS:
+    id_as_path = schema["$id"].replace("-", "_")
+    full_path = os.path.join(TOP_DIR, "docs/py/schema", id_as_path + ".json")
+    Path(full_path).parent.mkdir(parents=True, exist_ok=True)
+    with open(full_path, "w") as f:
+        f.write(json.dumps(schema, sort_keys=True, indent=4, separators=(",", ": ")))
+
+for example in EXAMPLES:
+    id_as_path = example["path"].replace("-", "_")
+    full_path = os.path.join(TOP_DIR, "docs/py/example", id_as_path + ".json")
+    Path(full_path).parent.mkdir(parents=True, exist_ok=True)
+    with open(full_path, "w") as f:
+        f.write(json.dumps(example, sort_keys=True, indent=4, separators=(",", ": ")))
+

package/docs/README.md

@@ -0,0 +1,3 @@
+# Overview
+
+This folder is used for the fully build (without "$ref", "$allOf", "include()" statements) JSON assets and is populated by the [build_schemas.js](../build_schemas.js) (with and appropriate flag).

package/example/3pse/file/applications/espresso/7.2/pw.x.json

@@ -0,0 +1,94 @@
+{
+    "&CONTROL": {
+        "calculation": "scf",
+        "title": "",
+        "verbosity": "low",
+        "restart_mode": "from_scratch",
+        "wf_collect": true,
+        "tstress": true,
+        "tprnfor": true,
+        "outdir": "{{ JOB_WORK_DIR }}/outdir",
+        "wfcdir": "{{ JOB_WORK_DIR }}/outdir",
+        "prefix": "__prefix__",
+        "pseudo_dir": "{{ JOB_WORK_DIR }}/pseudo"
+    },
+    "&SYSTEM": {
+        "ibrav": 0,
+        "nat": 2,
+        "ntyp": 2,
+        "ecutwfc": 40,
+        "ecutrho": 200,
+        "occupations": "smearing",
+        "degauss": 0.005,
+        "starting_magnetization": [
+            0.01,
+            0.02
+        ],
+        "Hubbard_occ": [
+            [
+                1,
+                1,
+                0.00
+            ],
+            [
+                1,
+                2,
+                0.02
+            ]
+        ]
+    },
+    "&ELECTRONS": {
+        "diagonalization": "david",
+        "diago_david_ndim": 4,
+        "diago_full_acc": true,
+        "mixing_beta": 0.3,
+        "startingwfc": "atomic+random"
+    },
+    "&IONS": null,
+    "&CELL": null,
+    "ATOMIC_SPECIES": {
+        "values": [
+            {
+                "X": "Cs",
+                "Mass_X": 132.90543,
+                "PseudoPot_X": "Cs.upf"
+            },
+            {
+                "X": "Cl",
+                "Mass_X": 35.4527,
+                "PseudoPot_X": "Cl.upf"
+            }
+        ]
+    },
+    "CELL_PARAMETERS": {
+        "card_option": "angstrom",
+        "values": {
+            "v1": [
+                4.324582724,
+                0.000000000,
+                2.496799000
+            ],
+            "v2": [
+                1.441527575,
+                4.077255694,
+                2.496799000
+            ],
+            "v3": [
+                0.000000000,
+                0.000000000,
+                4.993598000
+            ]
+        }
+    },
+    "K_POINTS": {
+        "card_option": "automatic",
+        "values": {
+            "nk1": 2,
+            "nk2": 2,
+            "nk3": 2,
+            "sk1": 0,
+            "sk2": 0,
+            "sk3": 0
+        }
+    }
+}

package/example/core/abstract/2d_data.json

@@ -0,0 +1,19 @@
+{
+    "xDataArray": [
+        0,
+        1,
+        2
+    ],
+    "yDataSeries": [
+        [
+            0,
+            0.5,
+            1
+        ],
+        [
+            0,
+            2.5,
+            5
+        ]
+    ]
+}

package/example/core/abstract/2d_plot.json

@@ -0,0 +1,26 @@
+{
+    "xAxis": {
+        "label": "kpoint index"
+    },
+    "xDataArray": [
+        0,
+        1,
+        2
+    ],
+    "yAxis": {
+        "label": "eigenvalues",
+        "units": "eV"
+    },
+    "yDataSeries": [
+        [
+            0,
+            0.5,
+            1
+        ],
+        [
+            0,
+            2.5,
+            5
+        ]
+    ]
+}

package/example/core/abstract/3d_grid.json

@@ -0,0 +1,12 @@
+{
+    "dimensions": [
+        2,
+        2,
+        2
+    ],
+    "shifts": [
+        0,
+        0,
+        0
+    ]
+}

package/example/core/abstract/3d_tensor.json

@@ -0,0 +1,17 @@
+[
+    [
+        1,
+        0,
+        0
+    ],
+    [
+        0,
+        1,
+        0
+    ],
+    [
+        0,
+        0,
+        1
+    ]
+]

package/example/core/abstract/3d_vector_basis.json

@@ -0,0 +1,17 @@
+{
+    "a": [
+        5.0,
+        0.0,
+        0.0
+    ],
+    "b": [
+        0.0,
+        5.0,
+        0.0
+    ],
+    "c": [
+        0.0,
+        0.0,
+        5.0
+    ]
+}

package/example/core/abstract/point.json

@@ -0,0 +1,5 @@
+[
+    0.0,
+    5.5,
+    0.0
+]

package/example/core/abstract/vector.json

@@ -0,0 +1,5 @@
+[
+    1.0,
+    0.0,
+    0.0
+]

package/example/core/primitive/1d_data_series.json

@@ -0,0 +1,12 @@
+[
+    [
+        0,
+        0.5,
+        1
+    ],
+    [
+        0,
+        2.5,
+        5
+    ]
+]

package/example/core/primitive/3d_lattice.json

@@ -0,0 +1,8 @@
+{
+    "a": 5.14,
+    "alpha": 90.0,
+    "b": 5.14,
+    "beta": 90.0,
+    "c": 5.14,
+    "gamma": 90.0
+}

package/example/core/primitive/array_of_3_booleans.json

@@ -0,0 +1,5 @@
+[
+    true,
+    false,
+    false
+]

package/example/core/primitive/array_of_3_numbers.json

@@ -0,0 +1,5 @@
+[
+    1.0,
+    0.0,
+    0.0
+]

package/example/core/primitive/array_of_ids.json

@@ -0,0 +1,8 @@
+[
+    {
+        "id": 1
+    },
+    {
+        "id": 2
+    }
+]

package/example/core/primitive/axis.json

@@ -0,0 +1,4 @@
+{
+    "label": "energy",
+    "units": "eV"
+}

package/example/core/primitive/scalar.json

@@ -0,0 +1,3 @@
+{
+    "value": 0.1
+}

package/example/core/primitive/string.json

@@ -0,0 +1,3 @@
+{
+    "value": "example string"
+}

package/example/core/reference/experiment/condition.json

@@ -0,0 +1,12 @@
+{
+    "name": "pressure",
+    "scalar": [
+        {
+            "value": "1.2"
+        },
+        {
+            "value": "3.3"
+        }
+    ],
+    "units": "kbar"
+}

package/example/core/reference/experiment/location.json

@@ -0,0 +1,4 @@
+{
+    "latitude": 24.3423424,
+    "longitude": 43.234232
+}

package/example/core/reference/experiment.json

@@ -0,0 +1,22 @@
+{
+    "authors": [
+        {
+            "affiliation": "Exabyte Inc.",
+            "first": "John",
+            "last": "Doe"
+        }
+    ],
+    "conditions": [
+        {
+            "...": "include(experiment/condition.json)"
+        }
+    ],
+    "method": "DFT",
+    "references": [
+        {
+            "...": "include(literature.json)"
+        }
+    ],
+    "timestamp": 141182979832,
+    "title": "Experiment 1"
+}

package/example/core/reference/literature/name.json

@@ -0,0 +1,5 @@
+{
+    "affiliation": "Exabyte Inc.",
+    "first": "John",
+    "last": "Doe"
+}

package/example/core/reference/literature/pages.json

@@ -0,0 +1,4 @@
+{
+    "end": "2",
+    "start": "1"
+}

package/example/core/reference/literature.json

@@ -0,0 +1,8 @@
+{
+    "authors": [
+        {
+            "...": "include(literature/name.json)"
+        }
+    ],
+    "doi": "10.1000/xyz123"
+}

package/example/core/reference/modeling/exabyte.json

@@ -0,0 +1,7 @@
+{
+    "_id": "nDAavgjrT5ezwFgod",
+    "owner": {
+        "...": "include(../../../system/owner.json)"
+    },
+    "title": "My Calculation"
+}