@grnsft/if 0.3.4 → 0.4.0-beta.1

package/src/builtins/csv-lookup/README.md

@@ -0,0 +1,142 @@
+# CSV Lookup Plugin
+
+`csv-lookup` is a generic plugin that enables you to select arbitrary data from a given csv file and add it to your manifest file's `input` data.
+
+You provide path to the target csv file pus some query parameters. The filepath can point to a location on the local filesystem or it can be a URL for an online resource. The query parameters include the column names for the target data you want to return (can be one column name, multiple column names or all column names, indicated using `"*"`), plus the column names and values you want to use as selectors.
+
+For example, for the following CSV:
+
+|      |                |                 |            |            |              |           |                  |            |            |                        |            |                            |                                   |                                |                                  |                       |
+| ---- | -------------- | --------------- | ---------- | ---------- | ------------ | --------- | ---------------- | ---------- | ---------- | ---------------------- | ---------- | -------------------------- | --------------------------------- | ------------------------------ | -------------------------------- | --------------------- |
+| year | cloud-provider | cloud-region    | cfe-region | em-zone-id | wt-region-id | location  | geolocation      | cfe-hourly | cfe-annual | power-usage-efficiency | net-carbon | grid-carbon-intensity-24x7 | grid-carbon-intensity-consumption | grid-carbon-intensity-marginal | grid-carbon-intensity-production | grid-carbon-intensity |
+| 2022 | Google Cloud   | asia-east1      | Taiwan     | TW         | TW           | Taiwan    | 25.0375,121.5625 | 0.18       |            |                        | 0          | 453                        |                                   |                                |                                  | 453                   |
+| 2022 | Google Cloud   | asia-east2      | Hong Kong  | HK         | HK           | Hong Kong | 22.3,114.2       | 0.28       |            |                        | 0          | 453                        |                                   |                                |                                  | 360                   |
+| 2022 | Google Cloud   | asia-northeast1 | Tokyo      | JP-TK      | JP-TK        | Tokyo     | 35.6897,139.692  | 0.28       |            |                        | 0          | 463                        |                                   |                                |                                  | 463                   |
+
+
+You could select all the data for the cloud provider `Google Cloud` in the region `asia-east2` using the following configuration:
+
+```yaml
+filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
+query:
+  cloud-provider: "cloud/provider"
+  cloud-region: "cloud/region"
+output: "*"
+```
+
+Notice that the query parameters are key/value pairs where the key is the column name in the target CSV and the value is a **reference to a value** in your `input` data (*not* an actual value - a reference). This is to enable you to chain CSV lookups together based on information from other plugins in your pipeline.
+
+
+## Parameters
+
+### Plugin config
+
+- `filepath` - path to a csv file, either on the local filesystem or on the internet
+- `query` - an array of key/value pairs where the key is a column name in the target csv and the value is a parameter from inputs
+- `output` - the columns to grab data from and add to output data - should support wildcard or multiple values. 
+
+The plugin also supports data renaming. This means you can grab data from a named column but push it into your manifest file data under another name, for example, maybe we want to grab data from the `processor-name` column int he target csv and add it to the manifest file data as `processor-id` because this is the name expected by some other plugin in your piepline. You can do this by passing comma separated values in arrays. 
+
+```yaml
+output:
+  ["processor-name": "processor-id"]
+```
+
+You can nest arrays to do this renaming for multiple columns.
+
+```yaml
+output:
+  [["processor-name", "processor-model-id"],["tdp","thermal-design-power"]]
+```
+
+All the following values are valid for the `output` field:
+- `"*"`
+- `"tdp"`
+- `["processor-name", "processor-model-id"]`
+- `[["processor-name", "processor-model-id"],["tdp","thermal-design-power"]]`
+
+### Inputs
+
+There are no strict requirements on input for this plugin because they depend upon the contents of the target CSV and your input data at the time the CSV lookup is invoked. Please make sure you are requesting data from columns that exist in the target csv file and that your query values are available in your `input` data.
+
+## Returns
+
+The input data with the requested csv content appended to it.
+
+## Plugin logic
+
+1. Validates global config which contains `filepath`, `query` and `output`.
+2. Tries to retrieve given file (with url or local path).
+3. Parses given CSV.
+4. Filters requested information from CSV.
+5. Adds new key-value pairs to the input data
+6. Returns enriched input data
+
+## Implementation
+
+To run the plugin, you must first create an instance of `CSVLookup`. Then, you can call `execute()`.
+
+```typescript
+const globalConfig = {
+  filepath: 'https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv',
+  query: {
+    'cloud-provider': 'cloud/provider'
+	  region: 'cloud/region'
+	  'instance-type': 'cloud/instance-type'
+  },
+  output: ['cpu-tdp', 'tdp'],
+};
+const divide = CSVLookup(globalConfig);
+
+const input = [
+  {
+    timestamp: '2023-08-06T00:00'
+    duration: 3600
+    'cpu/energy': 0.001
+    'cloud/provider': gcp
+    'cloud/region': asia-east
+  },
+];
+```
+
+## Example manifest
+
+IF users will typically call the plugin as part of a pipeline defined in a manifest file. In this case, instantiating the plugin is handled by `if` and does not have to be done explicitly by the user. The following is an example manifest that calls `csv-lookup`:
+
+```yaml
+name: csv-lookup-demo
+description:
+tags:
+initialize:
+  outputs:
+    - yaml
+  plugins:
+    cloud-metadata:
+      method: CSVLookup
+      path: 'builtin'
+      global-config:
+        filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
+        query:
+          cloud-provider: "cloud/provider"
+          cloud-region: "cloud/region"
+        output: "*"
+tree:
+  children:
+    child:
+      pipeline:
+        - cloud-metadata
+      inputs:
+        - timestamp: 2023-08-06T00:00
+          duration: 3600
+          cloud/provider: Google Cloud
+          cloud/region: europe-north1
+```
+
+You can run this example by saving it as `./examples/manifests/csv-lookup.yml` and executing the following command from the project root:
+
+```sh
+npm i -g @grnsft/if
+ie --manifest manifests/plugins/csv-lookup.yml --output manifests/outputs/csv-lookup
+```
+
+The results will be saved to a new `yaml` file in `manifests/outputs`.

package/src/builtins/divide/README.md

@@ -0,0 +1,95 @@
+# Divide
+
+`divide` is a generic plugin for doing arithmetic division of two values in an `input` array.
+
+You provide the names of the values you want to divide, and a name to use to add the divide to the output array.
+
+For example, `boavizta-cpu` need `cpu/number-cores` to work, however `cloud-metadata` returns `vcpus-allocated`, to get number of cores you divide `vcpus-allocated` by 2.
+
+## Parameters
+
+### Plugin config
+
+- `numerator` - a parameter by a specific configured number
+- `denominator` - a parameter by a specific configured number or the number by which `numerator` is divided
+- `output` - the number to a configured output parameter
+
+### Inputs
+
+- `numerator` - as input parameter, must be available in the input array
+- `denominator` - must be available in the input array if is an input parameter
+- `output` - as input parameter, must be available in the input array
+
+## Returns
+
+- `output`: the division of `numerator` with the parameter name into `denominator` with the parameter name defined by `output` in global config.
+
+The plugin throws an exception if the division result is not a number.
+
+## Calculation
+
+```pseudocode
+output = input0 / input1
+```
+
+## Implementation
+
+To run the plugin, you must first create an instance of `Divide`. Then, you can call `execute()`.
+
+```typescript
+const globalConfig = {
+  numerator: 'vcpus-allocated',
+  denominator: 2,
+  output: 'cpu/number-cores',
+};
+const divide = Divide(globalConfig);
+
+const input = [
+  {
+    timestamp: '2021-01-01T00:00:00Z',
+    duration: 3600,
+    'vcpus-allocated': 24,
+  },
+];
+```
+
+## Example manifest
+
+IF users will typically call the plugin as part of a pipeline defined in a manifest file. In this case, instantiating the plugin is handled by `if` and does not have to be done explicitly by the user. The following is an example manifest that calls `divide`:
+
+```yaml
+name: divide-demo
+description:
+tags:
+initialize:
+  outputs:
+    - yaml
+  plugins:
+    divide:
+      method: Divide
+      path: 'builtin'
+      global-config:
+        numerator: vcpus-allocated
+        denominator: 2
+        output: cpu/number-cores
+tree:
+  children:
+    child:
+      pipeline:
+        - divide
+      config:
+        divide:
+      inputs:
+        - timestamp: 2023-08-06T00:00
+          duration: 3600
+          vcpus-allocated: 24
+```
+
+You can run this example by saving it as `./examples/manifests/divide.yml` and executing the following command from the project root:
+
+```sh
+npm i -g @grnsft/if
+ie --manifest ./examples/manifests/divide.yml --output ./examples/outputs/divide.yml
+```
+
+The results will be saved to a new `yaml` file in `./examples/outputs`.

package/src/builtins/exponent/README.md

@@ -0,0 +1,97 @@
+# Exponent
+
+`exponent` is a generic plugin for calculating exponent of an input param (as base) and another (as the exponent) in an `input` array.
+
+You provide the names of the values you want to use for the exponent calculation, and a name to use to add the exponent result to the output array.
+
+For example, you use `cpu/energy` as base and `network/energy` as and name the result `energy`. `energy` would then be added to every observation in your input array as `cpu/energy` raised by the exponent `network/energy`.
+
+## Parameters
+
+### Plugin config
+
+Three parameters are required in global config: `input-parameter`, `exponent` and `output-parameter`.
+
+`input-parameter`: a string defining the base. Must match an existing key in the `inputs` array
+`exponent`: a number defining the exponent.
+`output-parameter`: a string defining the name to use to add the result of the exponent to the output array.
+
+### Inputs
+
+`input-parameter` and `exponent` must be available in the input array.
+
+## Returns
+
+- `output-parameter`: `input-parameter` raised by `exponent` with the parameter name defined by `output-parameter` in global config.
+
+## Calculation
+
+```pseudocode
+output = input ^ exponent
+```
+
+## Implementation
+
+To run the plugin, you must first create an instance of `Exponent`. Then, you can call `execute()`.
+
+```typescript
+import {Exponent} from 'builtins';
+
+const config = {
+  inputParameter: ['cpu/energy'],
+  exponent: 2
+  outputParameter: 'energy',
+};
+
+const exponent = Exponent(config);
+const result = await exponent.execute([
+  {
+    duration: 3600,
+    timestamp: '2021-01-01T00:00:00Z',
+    'cpu/energy': 0.1,
+    'energy': 0.01,
+  },
+]);
+```
+
+## Example manifest
+
+IF users will typically call the plugin as part of a pipeline defined in a manifest file. In this case, instantiating the plugin is handled by and does not have to be done explicitly by the user. The following is an example manifest that calls `exponent`:
+
+```yaml
+name: exponent demo
+description:
+tags:
+initialize:
+  outputs:
+    - yaml
+  plugins:
+    exponent:
+      method: Exponent
+      path: 'builtin'
+      global-config:
+        input-parameter: 'cpu/energy'
+        exponent: 2
+        output-parameter: 'energy'
+tree:
+  children:
+    child:
+      pipeline:
+        - exponent
+      config:
+        exponent:
+      inputs:
+        - timestamp: 2023-08-06T00:00
+          duration: 3600
+          cpu/energy: 0.001
+          network/energy: 0.001
+```
+
+You can run this example by saving it as `manifests/examples/test/exponent.yml` and executing the following command from the project root:
+
+```sh
+npm i -g @grnsft/if
+ie --manifest manifests/examples/test/exponent.yml --output manifests/outputs/exponent.yml
+```
+
+The results will be saved to a new `yaml` file in `manifests/outputs`.

package/src/builtins/interpolation/README.md

@@ -0,0 +1,168 @@
+# Interpolation Plugin
+
+## Overview
+
+Interpolation is a way to infer new data points from a previously known set of points.
+This plugin provides the `y` value at a given `x` by interpolating between known points. You provide a set of known `x`s and `y`s and a target `x`, the plugin returns the `y` for your `x`.
+
+## Usage
+
+To employ the `Interpolation` plugin, adhere to these steps:
+
+1. **Initialize Plugin**: Import the `Interpolation` function and initialize it with global configuration parameters `method`, `x`, `y`, `input-parameter` and `output-parameter`.
+
+2. **Execute Plugin**: Invoke the `execute` method of the initialized plugin instance with an array of input parameters. Each input parameter should include a `timestamp`, `duration` and `[input-parameter]` information.
+
+3. **Result**: The plugin will return an array of plugin parameters enriched with the calculated average carbon intensity for each input.
+
+## Global Config
+
+- `method`: specifies the interpolation method for the data. Acceptable values are 'linear', 'spline', or 'polynomial'. The default method is linear. (optional)
+- `x`: array of x points. Numbers should be in ascending order (required).
+- `y`: array of y points. Numbers should be in ascending order (required).
+- `input-parameter`: a string defining the name to use its value to calculate the interpolation. It should match an existing key in the inputs array (required).
+- `output-parameter`: a string defining the name to use to add the result of interpolation with additional calculation (required).
+
+`x` and `y` arrays must be equal lengths.
+
+## Input Parameters
+
+The plugin expects the following input parameters:
+
+- `timestamp`: a timestamp for the input (required)
+- `duration`: the amount of time, in seconds, that the input covers. (required)
+- `[input-parameter]` - a field whose name matches the string provided to input-parameter in global config (i.e. if the input-parameter in global config is cpu/utilisation then cpu-utilisation must exist in the input data)
+
+## Output
+
+The plugin outputs a value `z` that is the result of looking up the `y` value at `x = 'input-parameter'` using the user-defined interpolation method in `kWh` units. `z` is assigned to a field with a name defined by `output-parameter` in the output data.
+
+## Error Handling
+
+The plugin conducts input validation using the `zod` library and may throw errors if the provided parameters are invalid.
+
+## Plugin Algorithm
+
+1. **Execution**:
+
+   - Validate Global config
+
+     - `method` - validates if the method is one of these methods: `linear`, `spline`, or `polynomial`. If the method isn’t provided, it sets to `linear`.
+     - `x` and `y` should be arrays of numbers, the length should be equal, and elements should be ordered in the ascendant order.
+     - `input-parameter` - validates if the parameter is string.
+     - `output-parameter` - validates if the parameter is string.
+
+   - Iterate through each input, and do corresponding validation and calculation.
+
+   - Validate input parameters
+
+     - `duration` - validate if the duration is a number
+     - `timestamp` - should be in string or date format
+     - `[input-parameter]` - validates whether the parameter name is included in the input, and if its value should be a number within the range of x points.
+
+   - Calculation
+
+     - If the `method` is provided, choose the right way to calculate. For the `linear` and `polynomial` methods, calculate according to their formulas. For spline interpolation, use the npm module `typescript-cubic-spline`.
+
+2. **Output**: Output the provided input along with the calculated `cpu/energy`
+
+### TypeScript Usage
+
+```ts
+const globalConfig = {
+  method: 'linear',
+  x: [0, 10, 50, 100],
+  y: [0.12, 0.32, 0.75, 1.02],
+  'input-parameter': 'cpu/utilization'
+  'output-parameter': 'cpu/energy'
+
+};
+
+const interpolationPlugin = Interpolation(globalConfig);
+
+const inputs = [
+  {
+    timestamp: '2024-04-16T12:00:00Z',
+    duration: 3600,
+    'cpu/utilization': 45
+  },
+];
+
+const results = interpolationPlugin.execute(inputs);
+
+console.log(results);
+```
+
+### Manifest Usage
+
+#### Input
+
+```yaml
+name: interpolation-demo
+description: simple demo of interpolation plugin
+tags:
+initialize:
+  outputs:
+    - yaml
+  plugins:
+    interpolation:
+      method: Interpolation
+      path: 'builtin'
+      global-config:
+        method: linear
+        x: [0, 10, 50, 100]
+        y: [0.12, 0.32, 0.75, 1.02]
+        input-parameter: 'cpu/utilization'
+        output-parameter: 'cpu/energy'
+tree:
+  children:
+    child:
+      pipeline:
+        - interpolation
+      inputs:
+        - timestamp: 2023-07-06T00:00
+          duration: 3600
+          cpu/utilization: 45
+```
+
+#### Output
+
+```yaml
+name: interpolation-demo
+description: simple demo of interpolation plugin
+tags:
+initialize:
+  outputs:
+    - yaml
+  plugins:
+    interpolation:
+      method: Interpolation
+      path: 'builtin'
+      global-config:
+        method: linear
+        x: [0, 10, 50, 100]
+        y: [0.12, 0.32, 0.75, 1.02]
+        input-parameter: 'cpu/utilization'
+        output-parameter: 'cpu/energy'
+tree:
+  children:
+    child:
+      pipeline:
+        - interpolation
+      inputs:
+        - timestamp: 2023-07-06T00:00
+          duration: 3600
+          cpu/utilization: 45
+      outputs:
+        - timestamp: 2023-07-06T00:00
+          duration: 3600
+          cpu/utilization: 45
+          cpu/energy: 0.00069625
+```
+
+You can execute this by passing it to `ie`. Run the impact using the following command from the project root:
+
+```sh
+npm i -g @grnsft/if
+ie --manifest ./manifests/examples/interpolation.yml --output ./manifests/outputs/interpolation.yml
+```

package/src/builtins/mock-observations/README.md

@@ -0,0 +1,97 @@
+# Mock Observations Plugin
+
+## Introduction
+
+A plugin for mocking observations (inputs) for testing and demo purposes
+
+## Scope
+
+The mode currently mocks 2 types of observation data:
+
+- Common key-value pairs, that are generated statically and are the same for each generated observation/input (see 'helpers/CommonGenerator.ts')
+- Randomly generated integer values for predefined keys (see 'helpers/RandIntGenerator.ts')
+
+### Plugin global config
+
+- `timestamp-from`, `timestamp-to` and `duration` define time buckets for which to generate observations.
+- `generators` define which fields to generate for each observation
+- `components` define the components for which to generate observations. The observations generated according to `timestamp-from`, `timestamp-to`, `duration` and `generators` will be duplicated for each component.
+
+### Authentication
+
+N/A
+
+### Inputs
+
+The plugin's `global-config` section in the manifest file determines its behaviour.
+'inputs' section is ignored.
+
+### Typescript Usage
+
+```typescript
+const mockObservations = MockObservations({
+  'timestamp-from': '2023-07-06T00:00',
+  'timestamp-to': '2023-07-06T00:10',
+  duration: 60,
+  components: {
+    'instance-type': 'A1',
+  },
+  generators: {
+    common: {
+      region: 'uk-west',
+    },
+  },
+});
+const result = await mockObservations.execute([]);
+```
+
+### manifest Example
+
+IF users will typically call the plugin as part of a pipeline defined in a `manifest` file. In this case, instantiating the plugin is handled by `ie` and does not have to be done explicitly by the user. The following is an example `manifest` that calls `mock-observation`:
+
+```yaml
+name: mock-observation-demo
+description: example invoking mock-observation plugin
+tags:
+initialize:
+  outputs:
+    - yaml
+  plugins:
+    mock-observations:
+      kind: plugin
+      method: MockObservations
+      path: 'builtin'
+      global-config:
+        timestamp-from: 2023-07-06T00:00
+        timestamp-to: 2023-07-06T00:10
+        duration: 60
+        components:
+          - instance-type: A1
+          - instance-type: B1
+        generators:
+          common:
+            region: uk-west
+            common-key: common-val
+          randint:
+            cpu/utilization:
+              min: 1
+              max: 99
+            memory/utilization:
+              min: 1
+              max: 99
+tree:
+  children:
+    child:
+      pipeline:
+        - mock-observations
+      inputs:
+```
+
+You can run this example `manifest` by saving it as `manifests/plugins/mock-observation.yml` and executing the following command from the project root:
+
+```sh
+npm i -g @grnsft/if
+ie --manifest ./examples/manifests/test/mock-observation.yml --output ./examples/outputs/mock-observation
+```
+
+The results will be saved to a new `yaml` file in `./examples/outputs`.

package/src/builtins/multiply/README.md

@@ -0,0 +1,94 @@
+# Multiply
+
+`multiply` is a generic plugin for multiplying two or more values in an `input` array.
+
+You provide the names of the values you want to multiply, and a name to use to append the product to the output array.
+
+For example, you could multiply `cpu/energy` and `network/energy` and name the result `energy-product`. `energy-product` would then be added to every observation in your input array as the product of `cpu/energy` and `network/energy`.
+
+## Parameters
+
+### Plugin config
+
+Two parameters are required in global config: `input-parameters` and `output-parameter`.
+
+`input-parameters`: an array of strings. Each string should match an existing key in the `inputs` array
+`output-parameter`: a string defining the name to use to add the product of the input parameters to the output array.
+
+### Inputs
+
+All of `input-parameters` must be available in the input array.
+
+## Returns
+
+- `output-parameter`: the product of all `input-parameters` with the parameter name defined by `output-parameter` in global config.
+
+## Calculation
+
+```pseudocode
+output = input0 * input1 * input2 ... inputN
+```
+
+## Implementation
+
+To run the plugin, you must first create an instance of `Multiply`. Then, you can call `execute()`.
+
+```typescript
+import {Multiply} from 'builtins';
+
+const config = {
+  inputParameters: ['cpu/energy', 'network/energy'],
+  outputParameter: 'energy-product',
+};
+
+const mult = Multiply(config);
+const result = await mult.execute([
+  {
+    duration: 3600,
+    timestamp: '2021-01-01T00:00:00Z',
+    'cpu/energy': 0.001,
+    'memory/energy': 0.0005,
+  },
+]);
+```
+
+## Example manifest
+
+IF users will typically call the plugin as part of a pipeline defined in a manifest file. In this case, instantiating the plugin is handled by `ie` and does not have to be done explicitly by the user. The following is an example manifest that calls `multiply`:
+
+```yaml
+name: multiply-demo
+description:
+tags:
+initialize:
+  outputs:
+    - yaml
+  plugins:
+    multiply:
+      method: Multiply
+      path: 'builtin'
+      global-config:
+        input-parameters: ['cpu/energy', 'network/energy']
+        output-parameter: 'energy-product'
+tree:
+  children:
+    child:
+      pipeline:
+        - multiply
+      config:
+        multiply:
+      inputs:
+        - timestamp: 2023-08-06T00:00
+          duration: 3600
+          cpu/energy: 0.001
+          network/energy: 0.001
+```
+
+You can run this example by saving it as `./examples/manifests/test/multiply.yml` and executing the following command from the project root:
+
+```sh
+npm i -g @grnsft/if
+ie --manifest ./examples/manifests/test/multiply.yml --output ./examples/outputs/multiply.yml
+```
+
+The results will be saved to a new `yaml` file in `./examples/outputs`