@grnsft/if 0.6.0 → 0.7.0-beta.0

package/Refactor-migration-guide.md

@@ -105,10 +105,10 @@ There have also been some changes to the structure of manifest files. Some of th
     method: SciEmbodied
   ```
 
-- **Global config**
-  We have introduced the concept of global config to the plugins. This is truly global configuration data that should be kept constant regardless of where the plugin is invoked across the manifest file.
+- **Config**
+  We have introduced the concept of config to the plugins. This is truly configuration data that should be kept constant regardless of where the plugin is invoked across the manifest file.
 
-  A good example is the `interpolation` method to use in the Teads curve plugin - this is not expected to vary from component to component and can therefore be defined in global config. The plugin code itself must expect the global config. Then, the config can be passed in the `Initialize` block, for example:
+  A good example is the `interpolation` method to use in the Teads curve plugin - this is not expected to vary from component to component and can therefore be defined in config. The plugin code itself must expect the config. Then, the config can be passed in the `Initialize` block, for example:
 
   ```yaml
   initialize:
@@ -116,34 +116,13 @@ There have also been some changes to the structure of manifest files. Some of th
       'time-sync':
         method: TimeSync
         path: 'builtin'
-        global-config:
+        config:
           start-time: '2023-12-12T00:00:00.000Z'
           end-time: '2023-12-12T00:01:00.000Z'
           interval: 5
           allow-padding: true
   ```
 
-- **Node level config**
-
-  We have also introduced the concept of node-level config. This is designed for pluin configuration that might vary between components in the tree. For example, for each child in the tree you might wish to use the `regroup` feature to group the outputs according to a different set of keys.
-
-  ```yaml
-  tree:
-    children:
-      child-1:
-        pipeline:
-          compute:
-            - teads-curve
-            - sci-e
-            - sci-embodied
-            - sci-o
-            - time-sync
-            - sci
-          regroup:
-            - region
-            - cloud/instance-type
-  ```
-
 - **Defaults**
 
   We have also introduced the concept of `defaults`. This is a section in each component's definition that can be used to provide fallbacks for missing input data. For example, perhaps you have a value arriving from an external API that should be present in every observation in your inputs array, but for soem reason the API fails to deliver a value for some timestamps. In this case, IF would fallback to the value provided for that metric in the `defaults` section of the manifest for that component.
@@ -178,7 +157,7 @@ There have also been some changes to the structure of manifest files. Some of th
 
 Technically time-sync is not a new feature as it was present in IF before the refactor, but there are some tweaks to how the plugin is configured that are worth explaining here. Time sync snaps all input arrays across an entire graph to a common time grid.
 
-This means you have to define a global start time, end time and interval to use everywhere. There is also a boolean to toggle whether you should allow the time sync model to pad the start and end of your time series with zeroes. You should default to `true` unless you have a specific reason not to. In the refactored IF we expect this information to be provided in global config, as follows:
+This means you have to define a start time, end time and interval to use everywhere. There is also a boolean to toggle whether you should allow the time sync model to pad the start and end of your time series with zeroes. You should default to `true` unless you have a specific reason not to. In the refactored IF we expect this information to be provided in config, as follows:
 
 ```yaml
 initialize:
@@ -186,7 +165,7 @@ initialize:
     'time-sync':
       method: TimeSync
       path: 'builtin'
-      global-config:
+      config:
         start-time: '2023-12-12T00:00:00.000Z'
         end-time: '2023-12-12T00:01:00.000Z'
         interval: 5
@@ -220,68 +199,75 @@ Details tbc...
 
 ## Plugins
 
-The plugins themselves require some changes to keep them compatible with the refactored IF.
+Plugins require some modifications to remain compatible with the refactored IF interface.
 
-Instead of the old class-based model, plugins are now functions. They conform to the following interface:
+Each plugin follows the `PluginFactory` interface, which is a higher-order function that accepts a `params` object of type `PluginFactoryParams`. This function returns another function (the inner function), which handles the plugin’s `config`, `parametersMetadata`, and `mapping`.
 
 ```ts
-export type PluginInterface = {
-  execute: (
-    inputs: PluginParams[],
-    config?: Record<string, any>
-  ) => PluginParams[];
-  metadata: {
-    kind: string;
-  };
-  [key: string]: any;
-};
+export const PluginFactory =
+  (params: PluginFactoryParams) =>
+  (
+    config: ConfigParams = {},
+    parametersMetadata: PluginParametersMetadata,
+    mapping: MappingParams
+  ) => ({
+    metadata: {
+      inputs: {...params.metadata.inputs, ...parametersMetadata?.inputs},
+      outputs: parametersMetadata?.outputs || params.metadata.outputs,
+    },
+    execute: async (inputs: PluginParams[]) => {
+      // Generic plugin functionality goes here
+      // E.g., mapping, arithmetic operations, validation
+      // Process inputs and mapping logic
+    });
+  })
 ```
 
-The plugin still requires an execute function. This is where you implement the plugin logic.
+Inner Function Parameters:
+
+- `config`: This is of type `ConfigParams` and has a default value of an empty object ({}). This might hold configuration settings for the plugin.
+- `parametersMetadata`: A `PluginParametersMetadata` object that describes the metadata for the plugin’s parameters.
+- `mapping`: A `MappingParams` object, describing parameters are mapped.
+
+Implementation Function:
 
-Here's a minimal example for a plugin that sums some inputs defined in global config - see inline comments for some important notes:
+The plugin requires an `implementation` function, where the actual plugin logic is defined.
+Here’s a minimal example of a plugin that sums inputs as defined in the config. See the inline comments for further clarification.
 
 ```ts
-// Here's the function definition - notice that global config is passed in here!
-export const Sum = (globalConfig: SumConfig): PluginInterface => {
-  const inputParameters = globalConfig['input-parameters'] || [];
-  const outputParameter = globalConfig['output-parameter'];
-
-  // we also return metadata now too - you can add more or just use this default
-  const metadata = {
-    kind: 'execute',
-  };
-
-  /**
-   * Calculate the sum of the input metrics for each timestamp.
-   */
-  const execute = async (inputs: PluginParams[]): Promise<PluginParams[]> => {
-    inputs.map(input => {
-      return calculateSum(input, inputParameters, outputParameter);
+// Here's the function definition!
+export const Sum = PluginFactory({
+  configValidation: z.object({
+    'input-parameters': z.array(z.string()),
+    'output-parameter': z.string().min(1),
+  }),
+  inputValidation: (input: PluginParams, config: ConfigParams) => {
+    return validate(validationSchema, inputData);
+  },
+  implementation: async (inputs: PluginParams[], config: ConfigParams) => {
+    const {
+      'input-parameters': inputParameters,
+      'output-parameter': outputParameter,
+    } = config;
+
+    return inputs.map(input => {
+      const calculatedResult = calculateSum(input, inputParameters);
+
+      return {
+        ...input,
+        [outputParameter]: calculatedResult,
+      };
     });
-    return inputs;
-  };
-
-  /**
-   * Calculates the sum of the energy components.
-   */
-  const calculateSum = (
-    input: PluginParams,
-    inputParameters: string[],
-    outputParameter: string
-  ) => {
-    input[outputParameter] = inputParameters.reduce(
-      (accumulator, metricToSum) => {
-        return accumulator + input[metricToSum];
-      },
-      0
-    );
-  };
-
-  // return the metadata and the execute function
-  return {
-    metadata,
-    execute,
-  };
-};
+  },
+  allowArithmeticExpressions: [],
+});
+
+/**
+ * Calculates the sum of the energy components.
+ */
+const calculateSum = (input: PluginParams, inputParameters: string[]) =>
+  inputParameters.reduce(
+    (accumulator, metricToSum) => accumulator + input[metricToSum],
+    0
+  );
 ```

package/github-processes.md

@@ -3,24 +3,18 @@
 
 - [`if`](https://github.com/Green-Software-Foundation/if)
     - source code for the IF
+- [`if-core`](https://github.com/Green-Software-Foundation/if-core)
+    - helper types, interfaces and utilities for IF and plugin development
 - [`if-plugins`](https://github.com/Green-Software-Foundation/if-plugins) **DEPRECATED**
     - source code for standard library of plugins
     - IF core team commit to maintaining these plugins 
-- [`if-unofficial-plugins`](https://github.com/Green-Software-Foundation/if-unofficial-plugins)
+- [`if-unofficial-plugins`](https://github.com/Green-Software-Foundation/if-unofficial-plugins) **DEPRECATED**
     - source code for plugins relying on third-party data/APIs
     - intended to be deprecated and removed in mid-term future
     - plugins in this repo should be handed over to relevant organizations to maintain
 - [`if-plugin-template`](https://github.com/Green-Software-Foundation/if-plugin-template)
     - template for new plugins
     - intended for builders to bootstrap IF-compatible plugin development
-- [`if-standards`](https://github.com/Green-Software-Foundation/if-standards)
-    - not currently used, but intended to be the home of params.ts
-    - will have a dedicated discussion board and governance to set IF standards
-- [`if-exhaust plugins`](https://github.com/Green-Software-Foundation/if-exhaust-plugins)
-    - not currently used
-    - intended to become a separate repo just for exhaust plugins
-    - requires strict rules from if
-
 
 ## Branch names and purposes
 

package/manifests/examples/builtins/coefficient/failure-invalid-config-input-param.yml

@@ -1,12 +1,12 @@
 name: coefficient-demo
-description: failure with ivalid `global-config.input-parameter`
+description: failure with ivalid `config.input-parameter`
 tags:
 initialize:
   plugins:
     coefficient:
       method: Coefficient
       path: "builtin"
-      global-config:
+      config:
         input-parameter: 4
         coefficient: 3
         output-parameter: "carbon-product"
@@ -16,8 +16,6 @@ tree:
       pipeline:
         compute:
           - coefficient
-      config:
-        sum:
       inputs:
         - timestamp: 2023-08-06T00:00
           duration: 3600

package/manifests/examples/builtins/coefficient/failure-output-param-is-null.yaml

@@ -6,7 +6,7 @@ initialize:
     coefficient:
       method: Coefficient
       path: "builtin"
-      global-config:
+      config:
         input-parameter: "carbon"
         coefficient: 3
         output-parameter:
@@ -16,8 +16,6 @@ tree:
       pipeline:
         compute:
           - coefficient
-      config:
-        sum:
       inputs:
         - timestamp: 2023-08-06T00:00
           duration: 3600

package/manifests/examples/builtins/coefficient/success.yml

@@ -6,7 +6,7 @@ initialize:
     coefficient:
       method: Coefficient
       path: "builtin"
-      global-config:
+      config:
         input-parameter: "carbon"
         coefficient: 3
         output-parameter: "carbon-product"
@@ -16,8 +16,6 @@ tree:
       pipeline:
         compute:
           - coefficient
-      config:
-        sum:
       inputs:
         - timestamp: 2023-08-06T00:00
           duration: 3600

package/manifests/examples/builtins/csv-lookup/cloud-metadata/failure-invalid-instance-type.yaml

@@ -6,7 +6,7 @@ initialize:
     cloud-metadata:
       path: builtin
       method: CSVLookup
-      global-config:
+      config:
         filepath: >-
           https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
         query:
@@ -18,7 +18,6 @@ tree:
       pipeline:
         compute:
           - cloud-metadata
-      config:
       inputs:
         - timestamp: 2023-07-06T00:00 # [KEYWORD] [NO-SUBFIELDS] time when measurement occurred
           cloud/vendor: aws

package/manifests/examples/builtins/csv-lookup/cloud-metadata/failure-invalid-vendor.yaml

@@ -6,19 +6,18 @@ initialize:
     cloud-metadata:
       path: builtin
       method: CSVLookup
-      global-config:
+      config:
         filepath: >-
           https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
         query:
-          instance-class: cloud/instance-type
-        output: ["cpu-cores-utilized", "vcpus-allocated"]
+          instance-class: cloud/vendor
+        output: ["cpu-cores-utilized", "cloud/instance-type"]
 tree:
   children:
     child:
       pipeline:
         compute:
           - cloud-metadata
-      config:
       inputs:
         - timestamp: 2023-07-06T00:00 # [KEYWORD] [NO-SUBFIELDS] time when measurement occurred
           cloud/vendor: gcp

package/manifests/examples/builtins/csv-lookup/cloud-metadata/failure-missing-cloud-vendor.yml

@@ -6,11 +6,11 @@ initialize:
     cloud-metadata:
       path: builtin
       method: CSVLookup
-      global-config:
+      config:
         filepath: >-
           https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
         query:
-          instance-class: cloud/instance-type
+          instance-class: cloud/vendor
         output: ["cpu-cores-utilized", "vcpus-allocated"]
 tree:
   children:
@@ -18,7 +18,6 @@ tree:
       pipeline:
         compute:
           - cloud-metadata
-      config:
       inputs:
         - timestamp: 2023-07-06T00:00 # [KEYWORD] [NO-SUBFIELDS] time when measurement occurred
           #cloud/vendor: aws

package/manifests/examples/builtins/csv-lookup/cloud-metadata/success.yml

@@ -6,7 +6,7 @@ initialize:
     cloud-metadata:
       path: builtin
       method: CSVLookup
-      global-config:
+      config:
         filepath: >-
           https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
         query:
@@ -18,7 +18,6 @@ tree:
       pipeline:
         compute:
           - cloud-metadata
-      config:
       inputs:
         - timestamp: 2023-07-06T00:00 # [KEYWORD] [NO-SUBFIELDS] time when measurement occurred
           cloud/vendor: aws

package/manifests/examples/builtins/csv-lookup/region-metadata/failure-missing-column.yml

@@ -6,7 +6,7 @@ initialize:
     cloud-metadata:
       method: CSVLookup
       path: "builtin"
-      global-config:
+      config:
         filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
         query:
           cloud-provider: "nonexistant"

package/manifests/examples/builtins/csv-lookup/region-metadata/failure-missing-output.yml

@@ -6,12 +6,12 @@ initialize:
     cloud-metadata:
       method: CSVLookup
       path: "builtin"
-      global-config:
+      config:
         filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
         query:
           cloud-provider: "cloud/provider"
           cloud-region: "cloud/region"
-        output: "*"
+        output:
 tree:
   children:
     child:

package/manifests/examples/builtins/csv-lookup/region-metadata/success-renaming.yml

@@ -6,7 +6,7 @@ initialize:
     cloud-metadata:
       method: CSVLookup
       path: "builtin"
-      global-config:
+      config:
         filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
         query:
           cloud-provider: "cloud/provider"

package/manifests/examples/builtins/csv-lookup/region-metadata/success.yml

@@ -6,7 +6,7 @@ initialize:
     cloud-metadata:
       method: CSVLookup
       path: "builtin"
-      global-config:
+      config:
         filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
         query:
           cloud-provider: "cloud/provider"

package/manifests/examples/builtins/csv-lookup/tdp-finder/failure-missing-input-param.yml

@@ -6,7 +6,7 @@ initialize:
     tdp-finder:
       method: CSVLookup
       path: "builtin"
-      global-config:
+      config:
         filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/tdp-data-1.csv
         query:
           name: physical-processor
@@ -17,7 +17,6 @@ tree:
       pipeline:
         compute:
           - tdp-finder
-      config:
       inputs:
         - timestamp: 2023-07-06T00:00
           duration: 300

package/manifests/examples/builtins/csv-lookup/tdp-finder/failure-unsupported-physical-processor.yml

@@ -6,7 +6,7 @@ initialize:
     tdp-finder:
       method: CSVLookup
       path: "builtin"
-      global-config:
+      config:
         filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/tdp-data-1.csv
         query:
           name: physical-processor
@@ -17,7 +17,6 @@ tree:
       pipeline:
         compute:
           - tdp-finder
-      config:
       inputs:
         - timestamp: 2023-07-06T00:00
           duration: 300

package/manifests/examples/builtins/csv-lookup/tdp-finder/success.yml

@@ -6,7 +6,7 @@ initialize:
     tdp-finder:
       method: CSVLookup
       path: "builtin"
-      global-config:
+      config:
         filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/tdp-data-1.csv
         query:
           name: physical-processor
@@ -17,7 +17,6 @@ tree:
       pipeline:
         compute:
           - tdp-finder
-      config:
       inputs:
         - timestamp: 2023-07-06T00:00
           duration: 300

package/manifests/examples/builtins/divide/failure-invalid-config-denominator.yml

@@ -1,12 +1,12 @@
 name: divide
-description: failure when `global-config.denominator` is string
+description: failure when `config.denominator` is string
 tags:
 initialize:
   plugins:
     cloud-metadata:
       path: builtin
       method: CSVLookup
-      global-config:
+      config:
         filepath: >-
           https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
         query:
@@ -15,7 +15,7 @@ initialize:
     divide:
       method: Divide
       path: "builtin"
-      global-config:
+      config:
         numerator: vcpus-allocated
         denominator: "vcpus"
         output: cpu/number-cores
@@ -26,8 +26,6 @@ tree:
         compute:
           - cloud-metadata
           - divide
-      config:
-        divide:
       defaults:
         cloud/vendor: aws
         cloud/instance-type: m5n.large

package/manifests/examples/builtins/divide/failure-missing-numerator.yml

@@ -6,7 +6,7 @@ initialize:
     cloud-metadata:
       path: builtin
       method: CSVLookup
-      global-config:
+      config:
         filepath: >-
           https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
         query:
@@ -15,7 +15,7 @@ initialize:
     divide:
       method: Divide
       path: "builtin"
-      global-config:
+      config:
         #numerator: vcpus-allocated
         denominator: 2
         output: cpu/number-cores
@@ -26,8 +26,6 @@ tree:
         compute:
           - cloud-metadata
           - divide
-      config:
-        divide:
       defaults:
         cloud/vendor: aws
         cloud/instance-type: m5n.large

package/manifests/examples/builtins/divide/success.yml

@@ -6,7 +6,7 @@ initialize:
     cloud-metadata:
       path: builtin
       method: CSVLookup
-      global-config:
+      config:
         filepath: >-
           https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
         query:
@@ -15,7 +15,7 @@ initialize:
     divide:
       method: Divide
       path: "builtin"
-      global-config:
+      config:
         numerator: vcpus-allocated
         denominator: 2
         output: cpu/number-cores

package/manifests/examples/builtins/exponent/success.yml

@@ -6,7 +6,7 @@ initialize:
     exponent:
       method: Exponent
       path: "builtin"
-      global-config:
+      config:
         input-parameter: "cpu/energy"
         exponent: 2
         output-parameter: "energy"

package/manifests/examples/builtins/interpolation/success.yml

@@ -6,7 +6,7 @@ initialize:
     interpolation:
       method: Interpolation
       path: "builtin"
-      global-config:
+      config:
         method: linear
         x: [0, 10, 50, 100]
         y: [0.12, 0.32, 0.75, 1.02]

package/manifests/examples/builtins/mock-observations/failure-invalid-config-cpu-range.yml

@@ -1,5 +1,5 @@
 name: mock-observation-demo
-description: failure with `global-config->generators->randint->cpu/utilization->min` is greater than `max`
+description: failure with `config->generators->randint->cpu/utilization->min` is greater than `max`
 tags:
 initialize:
   plugins:
@@ -7,7 +7,7 @@ initialize:
       kind: plugin
       method: MockObservations
       path: "builtin"
-      global-config:
+      config:
         timestamp-from: 2023-07-06T00:00
         timestamp-to: 2023-07-06T00:10
         duration: 60

package/manifests/examples/builtins/mock-observations/failure-invalid-memory-utilization-range.yml

@@ -7,7 +7,7 @@ initialize:
       kind: plugin
       method: MockObservations
       path: "builtin"
-      global-config:
+      config:
         timestamp-from: 2023-07-06T00:00
         timestamp-to: 2023-07-06T00:10
         duration: 60

package/manifests/examples/builtins/mock-observations/failure-missing-timestamp-from-param.yml

@@ -7,7 +7,7 @@ initialize:
       kind: plugin
       method: MockObservations
       path: "builtin"
-      global-config:
+      config:
         #timestamp-from: 2023-07-06T00:00
         timestamp-to: 2023-07-06T00:10
         duration: 60

package/manifests/examples/builtins/mock-observations/success.yml

@@ -7,7 +7,7 @@ initialize:
       kind: plugin
       method: MockObservations
       path: "builtin"
-      global-config:
+      config:
         timestamp-from: 2023-07-06T00:00
         timestamp-to: 2023-07-06T00:10
         duration: 60

package/manifests/examples/builtins/multiply/failure-input-parameter-is-missing.yml

@@ -6,7 +6,7 @@ initialize:
     multiply:
       method: Multiply
       path: "builtin"
-      global-config:
+      config:
         input-parameters: ["cpu/energy", "network/energy"]
         output-parameter: "energy-product"
 tree:

package/manifests/examples/builtins/multiply/success-with-multiple-inputs.yml

@@ -6,7 +6,7 @@ initialize:
     multiply:
       method: Multiply
       path: "builtin"
-      global-config:
+      config:
         input-parameters: ["cpu/energy", "network/energy"]
         output-parameter: "energy-product"
 tree:

package/manifests/examples/builtins/multiply/success.yml

@@ -7,7 +7,7 @@ initialize:
     multiply:
       method: Multiply
       path: builtin
-      global-config:
+      config:
         input-parameters: ["cpu/energy", "network/energy"]
         output-parameter: "energy-product"
 tree:

package/manifests/examples/builtins/regex/failure-missing-input-param.yml

@@ -6,7 +6,7 @@ initialize:
     regex:
       method: Regex
       path: "builtin"
-      global-config:
+      config:
         parameter: physical-processor
         match: ^(.*),
         output: cpu/name