@hestia-earth/engine-models 0.81.1 → 0.81.4

package/README.md

@@ -0,0 +1,144 @@
+# HESTIA Engine Models
+
+[![Pipeline Status](https://gitlab.com/hestia-earth/hestia-engine-models/badges/master/pipeline.svg)](https://gitlab.com/hestia-earth/hestia-engine-models/commits/master)
+[![Coverage Report](https://gitlab.com/hestia-earth/hestia-engine-models/badges/master/coverage.svg)](https://gitlab.com/hestia-earth/hestia-engine-models/commits/master)
+
+HESTIA's set of models for running calculations or retrieving data using external datasets and internal lookups.
+
+## Documentation
+
+Documentation for every model can be found in the [HESTIA Guide](https://hestia.earth/guide/models).
+
+## Install
+
+1. Install python `3.12` minimum
+2. Install the module:
+```bash
+pip install hestia_earth.models
+```
+
+### Usage
+
+```python
+from hestia_earth.models.pooreNemecek2018 import run
+
+cycle_data = {"@type": "Cycle", ...}
+# cycle is a JSONLD node Cycle
+result = run('no3ToGroundwaterSoilFlux', cycle_data)
+print(result)
+```
+
+This will display only the result of the `no3ToGroundwaterSoilFlux` model (Emission).
+
+Additionally, to reduce the number of queries to the HESTIA API and run the models faster, prefetching can be enabled:
+```python
+from hestia_earth.models.preload_requests import enable_preload
+
+enable_preload()
+```
+
+#### Using the orchestrator
+
+The models come with an "orchestrator", which allows you to run a pre-configured set of models instead of a single one.
+
+The configuration for each Node (Cycle, Site or ImpactAssessment) can be found in the [config](./config) folder.
+
+Usage:
+```python
+from hestia_earth.orchestrator import run
+from hestia_earth.models.config import load_config
+
+cycle = {"@type": "Cycle", ...}
+result = run(cycle, load_config(cycle))
+print(result)
+```
+
+This will display the Cycle recalculated with all HESTIA default models running.
+
+#### Using Spatial Models
+
+We have models that can gap-fill geographical information on a `Site`. If you want to use these models:
+1. Install the library: `pip install hestia-earth-earth-engine`
+2. Follow the [Getting Started instructions](https://gitlab.com/hestia-earth/hestia-earth-engine#getting-started).
+
+### Collecting logs in JSON format
+
+:info: This is an experimental feature and is not yet integrated in all the HESTIA tools.
+
+To collect logs in JSON format:
+1. Set the env variable `LOG_JSON_ENABLED` to `true`
+1. Use the following code to collect contribution logs:
+```python
+from hestia_earth.orchestrator import run
+from hestia_earth.models.config import load_config
+from hestia_earth.models.jlog import reset_collected_logs, get_collected_logs
+
+# make sure the logs are reset before each run
+reset_collected_logs()
+
+# run the models as usual
+impact = {"@type": "ImpactAssessment", ...}
+result = run(impact, load_config(impact))
+print(result)
+
+# write the logs to a JSON file
+logs = get_collected_logs()
+with open("logs.json", "w") as f:
+    f.write(to_string(logs, indent=2))
+```
+
+When a node is recalculated in separate passes (e.g. stage 1 then stage 2 in different
+processes), seed the later pass with the earlier one's logs so stage-1-only placements
+are not lost:
+```python
+from hestia_earth.models.jlog import load_collected_logs, get_collected_logs
+
+# stage 2: load the logs collected during stage 1, then run stage 2 as usual
+load_collected_logs(stage1_logs)
+result = run(impact, load_config(impact), stage=2)
+logs = get_collected_logs()  # contains both stage 1 and stage 2
+```
+
+### Collecting contribution data
+
+:info: This is an experimental feature and is not yet integrated in all the HESTIA tools.
+
+To collect contribution data of impacts/endpoints on ImpactAssessment:
+1. Set the env variable `LOG_CONTRIBUTION_ENABLED` to `true`
+1. Use the following code to collect contribution logs:
+```python
+from hestia_earth.orchestrator import run
+from hestia_earth.models.config import load_config
+from hestia_earth.models.log_contribution import reset_contributions, get_contributions
+
+# make sure the contributions are reset before each run
+reset_contributions()
+
+# run the models as usual
+impact = {"@type": "ImpactAssessment", ...}
+result = run(impact, load_config(impact))
+print(result)
+
+# write the contributions to a JSON file
+contributions = get_contributions()
+with open("contributions.json", "w") as f:
+    f.write(to_string(contributions, indent=2))
+```
+
+The contribution data will store the factors used for each `emissionsResourceUse` that contributes to specific `impacts`, and likewise `impacts` that contribute to `endpoints`. The index in the data represents the index of the blank node in the list of contributing blank nodes.
+
+#### Using the ecoinventV3 model
+
+:warning: This model has been deprecated and no further support will be provided.
+
+ecoinvent is a consistent, transparent, and well validated life cycle inventory database.
+We use ecoinvent data to ascertain the environmental impacts of activities that occur outside of our system boundary, for example data on the environmental impacts of extracting oil and producing diesel, or the impacts of manufacturing plastics.
+
+The `ecoinventV3` model requires a valid [license](https://ecoinvent.org/offerings/licences/) to run. We are currently working on a way to enable users of this code with a valid ecoinvent licence to run these models themselves, but for now, these models are only available on the public platform.
+
+### Amortising Land Use Change Emissions Using Linear Discounting
+
+:info: This is an experimental feature and is not yet integrated in all the HESTIA tools. All carbon stock change models (`ipcc2019.aboveGroundBiomass`, _etc._) and their associated emission models (`ipcc2019.co2ToAirAboveGroundBiomassStockChange`, _etc._) have been updated; however, changes to `hestia.landCover` are required to make linear discounting fully functional (see work item [#1623](https://gitlab.com/hestia-earth/hestia-engine-models/-/work_items/1623) for further information).
+
+To amortise the impacts of land use change using linear discounting across IPCC (2019) models:
+1. Set the env variable `USE_LINEAR_DISCOUNTING` to `true`

package/cjs/version.d.ts

@@ -0,0 +1 @@
+export declare const ENGINE_VERSION = "0.81.4";

package/{dist → cjs}/version.js

@@ -1,4 +1,4 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.ENGINE_VERSION = void 0;
-exports.ENGINE_VERSION = '0.81.1';
+exports.ENGINE_VERSION = '0.81.4';

package/config/Cycle.json

@@ -2607,6 +2607,14 @@
       },
       "stage": 2
     },
+    {
+      "key": "completeness",
+      "model": "cycle",
+      "value": "completeness",
+      "runStrategy": "always",
+      "mergeStrategy": "node",
+      "stage": 2
+    },
     {
       "key": "emissions",
       "model": "cycle",

package/esm/config.d.ts

@@ -0,0 +1,31 @@
+import { NodeType } from '@hestia-earth/schema';
+export interface IOrchestratorModelConfig {
+    key: string;
+    model: string;
+    value: string;
+    runStrategy: 'always' | 'add_key_if_missing' | 'add_blank_node_if_missing';
+    runArgs?: Record<string, any>;
+    mergeStrategy: 'default' | 'append' | 'list' | 'node';
+    mergeArgs?: Record<string, any>;
+}
+export interface IOrchestratorConfig {
+    models: (IOrchestratorModelConfig | IOrchestratorModelConfig[])[];
+}
+export type allowedType = NodeType.Cycle | NodeType.Site | NodeType.ImpactAssessment;
+export declare const allowedTypes: allowedType[];
+/**
+ * Load orchestrator configuration.
+ *
+ * @param nodeType NodeType associated with configuration.
+ * @returns
+ */
+export declare const loadConfig: (nodeType: allowedType) => IOrchestratorConfig;
+interface ICalculationConfigData {
+    key: 'related' | 'nested';
+    type: allowedType;
+    stage: number;
+}
+export declare const loadRunConfig: (nodeType: allowedType, stage: number) => ICalculationConfigData[];
+export declare const loadTriggerConfig: (nodeType: allowedType, stage: number) => ICalculationConfigData[];
+export declare const getMaxStage: (nodeType: allowedType) => any;
+export {};

package/esm/config.js

@@ -0,0 +1,39 @@
+import { NodeType } from '@hestia-earth/schema';
+import * as runConfig from '../config/run-calculations.json';
+import * as triggerConfig from '../config/trigger-calculations.json';
+import * as Cycle from '../config/Cycle.json';
+import * as ImpactAssessment from '../config/ImpactAssessment.json';
+import * as Site from '../config/Site.json';
+export const allowedTypes = [
+    NodeType.Cycle,
+    NodeType.Site,
+    NodeType.ImpactAssessment
+];
+const validateType = (nodeType) => allowedTypes.includes(nodeType) ||
+    (() => {
+        throw new Error(`Invalid type ${nodeType}. Allowed types: ${allowedTypes.join(', ')}`);
+    })();
+const typeToConfig = {
+    [NodeType.Cycle]: Cycle,
+    [NodeType.ImpactAssessment]: ImpactAssessment,
+    [NodeType.Site]: Site
+};
+/**
+ * Load orchestrator configuration.
+ *
+ * @param nodeType NodeType associated with configuration.
+ * @returns
+ */
+export const loadConfig = (nodeType) => validateType(nodeType) && typeToConfig[nodeType];
+const validateStage = (config, nodeType, stage) => `stage-${stage}` in config[nodeType]
+    ? config[nodeType][`stage-${stage}`]
+    : (() => {
+        throw new Error(`Invalid stage configuration for ${nodeType}: ${stage}`);
+    })();
+export const loadRunConfig = (nodeType, stage) => validateType(nodeType) &&
+    validateStage(runConfig, nodeType, stage);
+export const loadTriggerConfig = (nodeType, stage) => validateType(nodeType) &&
+    validateStage(triggerConfig, nodeType, stage);
+export const getMaxStage = (nodeType) => validateType(nodeType)
+    ? Math.max.apply(Math.max, Object.keys(runConfig[nodeType]).map((key) => Number(key.replace('stage-', ''))))
+    : null;

package/esm/index.js

@@ -0,0 +1,5 @@
+export * from './config';
+export * from './models';
+export * from './utils';
+export * from './validate-config';
+export * from './version';

package/esm/models.d.ts

@@ -0,0 +1,41 @@
+import { EmissionMethodTier } from '@hestia-earth/schema';
+export interface IModel {
+    /**
+     * Path to the implementation (code) of the model.
+     */
+    path: string;
+    /**
+     * Path to the documentation of the model.
+     */
+    docPath: string;
+    /**
+     * Path to the documentation of the model on the guide.
+     */
+    guidePath: string;
+    /**
+     * The name of the model used as `methodModel`.
+     */
+    model: string;
+    /**
+     * The term the model is associated with.
+     */
+    term?: string;
+    /**
+     * The methodTier of the Emission (if applicable).
+     */
+    methodTier?: EmissionMethodTier;
+    /**
+     * A key in the model if term does not exist.
+     */
+    modelKey?: string;
+    /**
+     * List of other models this model needs to run.
+     */
+    dependencies?: string[];
+}
+export type modelLookup = Record<string, Pick<IModel, 'model' | 'term' | 'modelKey'>[]>;
+export interface IModelLinks {
+    links: IModel[];
+    lookups: Record<string, modelLookup>;
+}
+export declare const models: IModelLinks;

package/esm/models.js

@@ -0,0 +1,2 @@
+import * as data from '../model-links.json';
+export const models = data;

package/esm/utils.d.ts

@@ -0,0 +1,6 @@
+import { IOrchestratorModelConfig } from './config';
+import { IModel } from './models';
+export declare const findMatchingModel: (model: Partial<IModel>, config?: {
+    useDotValues?: boolean;
+}) => IModel;
+export declare const findMatchingModelFromConfig: (model: Pick<IOrchestratorModelConfig, "model" | "value">) => IModel;

package/esm/utils.js

@@ -0,0 +1,33 @@
+import { models } from './models';
+const matchWithDot = (model, key, value) => typeof value === 'string' && value.split('.').pop() === model[key];
+const matchPath = (model, key, value) => key === 'path' && typeof value === 'string' && model[key].startsWith(value);
+export const findMatchingModel = (model, config = {
+    useDotValues: false
+}) => Object.keys(model).length > 0
+    ? models.links.find((m) => Object.entries(model).every(([key, value]) => value === m[key] ||
+        (config.useDotValues && matchWithDot(m, key, value)) ||
+        matchPath(m, key, value)))
+    : null;
+const modelMatchOrder = ({ model, value }) => [
+    { model, term: value },
+    { model, modelKey: value },
+    {
+        model,
+        modelKey: value.split('.').pop()
+    },
+    {
+        model,
+        term: value.split('.').pop()
+    },
+    {
+        model,
+        path: ['hestia_earth', 'models', model, value].join('/')
+    },
+    !value || value === 'all' ? { model } : null
+].filter(Boolean);
+export const findMatchingModelFromConfig = (model) => {
+    var _a, _b;
+    return (_b = (_a = modelMatchOrder(model)
+        .map((params) => findMatchingModel(params) ||
+        findMatchingModel(params, { useDotValues: true }))) === null || _a === void 0 ? void 0 : _a.filter(Boolean)) === null || _b === void 0 ? void 0 : _b.shift();
+};

package/esm/validate-config.d.ts

@@ -0,0 +1,19 @@
+import { allowedType, IOrchestratorModelConfig } from './config';
+/**
+ * Use inter-dependencies to detect models that are in the wrong order.
+ *
+ * @param nodeType Cycle, Site, or ImpactAssessment
+ * @returns List of models that have dependencies that could not be found prior to running.
+ */
+export declare const validateConfigOrder: (nodeType: allowedType) => {
+    model: IOrchestratorModelConfig;
+    existing: string[];
+    missing: string[];
+}[];
+/**
+ * Detect multiple models running in parallel when they should not.
+ * Note: we use the `mergeArgs` to handle special cases when multiple identical models CAN run in parallel
+ *
+ * @param nodeType Cycle, Site, or ImpactAssessment
+ */
+export declare const validateConfigDuplicates: (nodeType: allowedType) => IOrchestratorModelConfig[];

package/esm/validate-config.js

@@ -0,0 +1,65 @@
+import { loadConfig } from './config';
+import { findMatchingModelFromConfig } from './utils';
+// models not included in the links
+const skipModels = ['emissions.deleted', 'transformations'];
+const isHestiaResidueCyclicModel = ({ model, value }) => model === 'hestia' && (value === null || value === void 0 ? void 0 : value.startsWith('residue'));
+const isOrganicCarbonCyclicModel = ({ model, value }) => model === 'hestia' && (value === null || value === void 0 ? void 0 : value.startsWith('organicCarbonPer'));
+const includeModel = (model) => [
+    !skipModels.includes(model.model),
+    // cyclic dependencies
+    !isHestiaResidueCyclicModel(model),
+    !isOrganicCarbonCyclicModel(model),
+    !model.key.startsWith('cache')
+].every(Boolean);
+// dependencies that cannot be validated
+const skipDependencies = [
+    // completeness gap-fills after other models run
+    'completeness'
+];
+const matchDependencies = (models) => (model) => {
+    var _a, _b, _c;
+    const { dependencies } = findMatchingModelFromConfig(model);
+    const previousModels = models.slice(0, models.indexOf(model));
+    const existing = (_a = dependencies === null || dependencies === void 0 ? void 0 : dependencies.filter((value) => previousModels.some((config) => { var _a; return ((_a = config.value) === null || _a === void 0 ? void 0 : _a.split('.').pop()) === value; }))) !== null && _a !== void 0 ? _a : [];
+    return {
+        model,
+        existing,
+        missing: (_c = (_b = dependencies === null || dependencies === void 0 ? void 0 : dependencies
+        // skip dependencies that dont exist at all
+        .filter((value) => models.some((config) => config.value === value))) === null || _b === void 0 ? void 0 : _b.filter((v) => !existing.includes(v) &&
+            !skipDependencies.includes(v) &&
+            v !== model.value)) !== null && _c !== void 0 ? _c : []
+    };
+};
+/**
+ * Use inter-dependencies to detect models that are in the wrong order.
+ *
+ * @param nodeType Cycle, Site, or ImpactAssessment
+ * @returns List of models that have dependencies that could not be found prior to running.
+ */
+export const validateConfigOrder = (nodeType) => {
+    const config = loadConfig(nodeType);
+    // flatten all models to process one by one, starting with the last one
+    const models = config.models.flat();
+    const results = models.filter(includeModel).map(matchDependencies(models));
+    return results.filter((r) => r.missing.length > 0);
+};
+const findDuplicatedConfigs = (models) => {
+    const modelsByValue = models.reduce((prev, curr) => (Object.assign(Object.assign({}, prev), { [curr.value]: [...(prev[curr.value] || []), curr] })), {});
+    return Object.values(modelsByValue)
+        .map((values) => values.filter((v) => { var _a; return !((_a = v.mergeArgs) === null || _a === void 0 ? void 0 : _a.sameMethodModel); }))
+        .filter((values) => values.length > 1);
+};
+/**
+ * Detect multiple models running in parallel when they should not.
+ * Note: we use the `mergeArgs` to handle special cases when multiple identical models CAN run in parallel
+ *
+ * @param nodeType Cycle, Site, or ImpactAssessment
+ */
+export const validateConfigDuplicates = (nodeType) => {
+    const config = loadConfig(nodeType);
+    const duplicates = config.models
+        .map((value) => (Array.isArray(value) ? findDuplicatedConfigs(value) : []))
+        .flat(2);
+    return duplicates;
+};

package/esm/version.d.ts

@@ -0,0 +1 @@
+export declare const ENGINE_VERSION = "0.81.4";

package/esm/version.js

@@ -0,0 +1 @@
+export const ENGINE_VERSION = '0.81.4';