@constructor-io/constructorio-connect-cli 1.0.0

package/boilerplate-repo/README.md

@@ -0,0 +1,650 @@
+# Constructor.io Connect CLI Repository
+
+This repository was created using the [constructorio-connect-cli CLI](https://www.npmjs.com/package/@constructor-io/constructorio-connect-cli). It is intended to be used as a workspace for developing custom templates for the Constructor Connect platform.
+
+## Table of Contents
+
+- [Constructor.io Connect CLI Repository](#constructorio-connect-cli-repository)
+  - [Table of Contents](#table-of-contents)
+- [Getting Started](#getting-started)
+  - [Configuring Authentication with the Connect API](#configuring-authentication-with-the-connect-api)
+  - [Associating Templates with Connections](#associating-templates-with-connections)
+- [Executing Templates](#executing-templates)
+  - [`npm run execute [FLAGS]`](#npm-run-execute-flags)
+- [Testing templates](#testing-templates)
+- [Deploying Templates](#deploying-templates)
+  - [`npm run deploy ENV`](#npm-run-deploy-env)
+- [Developing Templates](#developing-templates)
+  - [➡️ Expected Transformer Implementation](#️-expected-transformer-implementation)
+  - [🗑️ Removing Entities](#️-removing-entities)
+  - [🍙 Grouping logic](#-grouping-logic)
+  - [🗺️ Mapping logic](#️-mapping-logic)
+    - [Mapping items](#mapping-items)
+    - [Mapping variations](#mapping-variations)
+    - [Mapping item groups](#mapping-item-groups)
+  - [💡 Valid transformer examples](#-valid-transformer-examples)
+    - [Item transformer](#item-transformer)
+    - [Variation transformer](#variation-transformer)
+    - [Item Group transformer](#item-group-transformer)
+    - [Grouping transformer](#grouping-transformer)
+  - [⚙️ Template Language](#️-template-language)
+    - [1️⃣ Adding a single facet](#1️⃣-adding-a-single-facet)
+    - [2️⃣ Using built in helper functions](#2️⃣-using-built-in-helper-functions)
+    - [3️⃣ Using helper functions provided by connectors](#3️⃣-using-helper-functions-provided-by-connectors)
+      - [Helper Function](#helper-function)
+    - [4️⃣ Defining new functions and/or variables inside templates](#4️⃣-defining-new-functions-andor-variables-inside-templates)
+      - [Connector data](#connector-data)
+      - [Template](#template)
+      - [Result](#result)
+    - [5️⃣ Sharing code across templates](#5️⃣-sharing-code-across-templates)
+
+# Getting Started
+
+## Configuring Authentication with the Connect API
+
+Before you can deploy and test templates on the Constructor Connect platform, you'll need to setup your local environment. Primarily, you will need
+to create a .env file with the contents of .env.example and fill in the `CONNECT_AUTH_TOKEN` variable with your Constructor Connect Auth token. This will be necessary to test templates against the Connect API and deploy them to the platform.
+
+## Associating Templates with Connections
+
+With that configured, visit the `connectrc.js` file. This is where you can describe the relationships between your templates and the connections you want to use them with. Any arbitrary list of connections (identified by their ID) can be connected with any of your template files. You can create as many of these relationships as you need, with the following restrictions:
+- A connection can only be associated with one set of templates.
+- The environment declared for each relationship must match the environment for all connections that belong to it (e.g. only development connections may be listed in a relationship configured with `environment: "development"`)
+
+An example with a single placeholder connection ID is provided in the `connectrc.js` by default. Replacing the example with your connection ID(s) and adjusting the example template source code according to your needs is the quickest way to get a set of working templates.
+
+# Executing Templates
+
+Once you have written some templates and you would like to quickly test their output, you can use the `execute` command. This command will allow you to see the result of executing your template against a specific fixture of JSON catalog data.
+
+Based on the template you provide, the connections existing on your account, and the configuration in your `connectrc.js` file, you will be prompted with choices of fixture files and connections to execute against. Alternatively, you can provide all of these upfront as flags to the command (see below). This is especially useful if you want to repeat execution on the same set of templates and fixtures as you iterate without having to step through the prompts each time.
+
+**💡 Tip:** this repo also provides a built in VSCode integration. If you have any template file open, just hit `F5` or launch it from the debugger menu to execute the template and see the results immediately.
+
+## `npm run execute [FLAGS]`
+
+```
+USAGE
+  $ npm run execute -- [--template-path <value>] [--fixture-path <value>] [--external-data-path <value>] [--connection-id <value>]
+
+FLAGS
+  --template-path=<value>      The path to the template to execute. Must be in the 'src/templates' directory.
+  --fixture-path=<value>       The path to the fixture to execute the template against. Must be in the 'src/fixtures' directory.
+  --external-data-path=<value>  The path to the external data to execute the template against. Must be in the 'src/fixtures' directory.
+  --connection-id=<value>      The ID of the connection to execute the template against.
+
+DESCRIPTION
+  Execute a template against a connection and fixture to see the resulting transformed data. Each value not provided as a flag will be prompted for when the command is executed.
+
+EXAMPLES
+  $ npm run execute
+
+  $ npm run execute -- --template-path=variation/test_variation.jsonata
+
+  $ npm run execute -- --template-path=item_group/item_group.jsonata --fixture-path=item_group.json
+
+  $ npm run execute -- --template-path=grouping/grouping.jsonata --connection-id=example-connection-id
+```
+
+# Testing templates
+
+_This section is under active development and will be updated soon. Keep an eye out for new versions!_
+
+# Deploying Templates
+
+Once you have developed and tested your templates, you can deploy them to the Constructor Connect platform. To do this, you can use the `deploy` command.
+
+## `npm run deploy ENV`
+
+```
+USAGE
+  $ npm run deploy ENV
+
+ARGUMENTS
+  ENV  (development|qa|production) The target Constructor environment to deploy to.
+
+DESCRIPTION
+
+  Deploys all templates defined on the `connectrc.js` file to the specified environment.
+
+  The script will use the `CONNECT_AUTH_TOKEN` environment variable to authenticate with Constructor.
+
+EXAMPLES
+  $ npm run deploy development
+
+  $ npm run deploy qa
+
+  $ npm run deploy production
+```
+
+# Developing Templates
+
+## ➡️ Expected Transformer Implementation
+
+To implement your very own transformers, you just need to edit the `.jsonata` files inside the `templates` folder.
+
+Keep in mind that the template will have access to `externalData` and `targetData`. You can use the `targetData` to access the data that is being ingested, and `externalData` to access the data available in the connector.
+
+Note that all template files must return the expected data types. To know which properties you can override inside `item`, `variation` or `item_group`, take a look into the type definitions:
+
+- [Item](https://docs-beta.constructor.io/docs/integrating-with-constructor-product-catalog-csv-csv-feed-format-items-products)
+- [Variation](https://docs-beta.constructor.io/docs/integrating-with-constructor-product-catalog-csv-csv-feed-format-variations)
+- [Item Group](https://docs-beta.constructor.io/docs/integrating-with-constructor-product-catalog-csv-csv-feed-format-item-groups-categories)
+
+You can also refer to the [Constructor API docs](https://docs-beta.constructor.io/reference/v1-catalog-create-or-replace-catalog) for more details.
+
+## 🗑️ Removing Entities
+
+You can also remove any item, variation, or item group through our templates. Suppose each product on your catalog has a `visible` boolean field. It could make sense to not even ingest them into [Constructor.io](https://constructor.io/).
+
+We can make this happen returning `{ "remove": true }` whenever we find an item we want to drop from the ingestion:
+
+```js
+(
+  /* item.jsonata */
+
+  targetData.visible
+    ? { "active": true } /* Visible = true  => we simply ingest it as an active item */
+    : { "remove": true } /* Visible = false => we don't even ingest it */
+)
+```
+
+## 🍙 Grouping logic
+
+You can use the `grouping.jsonata` template to group variations into the item level. This can be helpful, for example, if you want to ingest data while grouping your items with a specific criteria (e.g. color).
+
+To do this, you need to define a grouping template with the `grouping` key.
+
+Refer to the examples below for more information.
+
+## 🗺️ Mapping logic
+
+You can use the `mapping.jsonata` template to map your data before transforming it. This has a few use cases:
+
+- **Ingesting data of any shape**: Say you're simply ingesting data from a JSON file. Since you don't have the base transformation layer that a partner connector would usually provide, you need to map your data first.
+- **Customizing the connector mappings**: Say you're using a partner connector and you want to customize how the data is mapped. For example, you want to ingest content, or you want to grab the categories that would normally be ingested as `item_groups` and ingest those into a custom index section.
+- **Filtering out data**: While you can filter out data during the transformations, it might be computationally expensive. You can simply drop any data during the mapping phase.
+- **Creating data**: Say you want to have a default variation in every item, or a new item group as parent of all others. You can create data out of thin air during the mapping phase.
+- **Grouping data**: When dealing with variation ingestions we may want to ingest partial data, maybe some pricing or inventory delta updates. Sometimes this data is not in the variation level yet, we can solve this by using the `__group_by` custom fields to group all records into a specific variation identifier. That way when transforming the data you'll have access to an array of all data points that should be ingested as variations.
+- ⚠️ This is currently only supported when performing variation-only ingestions.
+
+Essentially, mapping templates empower you to fully customize how the connector works. It's up to you to map and transform the data.
+
+The mapping template can return three arrays:
+
+- `items`: Items to be ingested.
+- `variations`: Variations to be ingested.
+- `item_groups`: Item groups to be ingested.
+
+All objects inside these arrays can have **any data structure**. In the transformation templates, you'll receive the same structure you defined here.
+
+It's important to note that there are a few requirements and optional fields for each of these, as described below.
+
+### Mapping items
+
+When you map items, you can append a field named `__variations` in case you can easily map the variations from the input data. This will automatically group the variations into the item level, and will heavily speed up the ingestion performance.
+
+### Mapping variations
+
+When you map variations, you can optionally provide a `__parent_id` field to each variation. If you do that and the item is also present in the ingestion, you'll be able to access the variations under the `__variations` field during the item template transformations.
+
+When no item is present you may also add a `__group_by` field to group all related data into a single variation. This is useful when you're ingesting partial data, like pricing or inventory updates.
+
+### Mapping item groups
+
+Mapping item groups is simpler. Since they can be ingested separately, you can define any shape you want and transform them later.
+
+## 💡 Valid transformer examples
+
+### Item transformer
+
+In the `templates/item.jsonata` file, you can customize how items are transformed:
+
+```js
+(
+  {
+    "item_name": "This overrides the item name",
+
+    "facets": [
+      {
+        "key": "color",
+        "value": "red"
+      }
+    ],
+
+    "metadata": [
+      {
+        "key": 'id',
+        "value": targetData.id
+      }
+    ]
+  }
+)
+```
+
+### Variation transformer
+
+In the `templates/variation.jsonata` file, you can customize how variations are transformed:
+
+```js
+{
+  "facets": [
+    {
+      "key": "color",
+      "value": "orange"
+    }
+  ],
+
+  "metadata": [
+    {
+      "key": "sku",
+      "value": $getAttributeValueFromNames(["sku", "fallback-sku"])
+    }
+  ]
+}
+```
+
+### Item Group transformer
+
+In the `templates/item_group.jsonata` file, you can customize how item groups are transformed:
+
+```js
+{
+  "name": "foobar"
+}
+```
+
+### Grouping transformer
+
+The `grouping` template is expected to override the same keys as the item transformer, and ideally it should override the `id` property.
+
+In this example, we're assuming that the connector data has properties such as `color`, `id`, and `parent` (in case of variations). With the code below, if a product has the ID `foo` with the color `blue`, the new ID would become `foo-blue`.
+
+Also note that in this scenario, you'll also need to update the `item_id` property in the variant level to make sure it'll match the overridden item ids.
+
+In the `templates/grouping.jsonata` file, you can customize how item groups are transformed:
+
+```js
+/* grouping.jsonata */
+(
+  $color := targetData.color;
+  $id := targetData.id;
+
+  {
+    "id": $join([$id, '-', $color]);
+  }
+)
+```
+
+We also need to update variations to point to the correct item:
+
+```js
+/* variation.jsonata */
+{
+  $color := targetData.color;
+  $parentId := targetData.parent.id;
+
+  {
+    "item_id": $join([$parentId, '-', $color]);
+  }
+}
+```
+
+Finally, we likely want to remove excess items so that only new, grouped items are kept:
+
+```js
+/* item.jsonata */
+{
+  "remove": true
+}
+```
+
+## ⚙️ Template Language
+
+To power the catalog customizations, we use a template language that allows for mapping data.
+
+To do this, we use [JSONata](https://jsonata.org/) templates to support customizing your catalog data. JSONata provides a powerful template language with many built-in functions, but we also provide additional helper functions to deal with the external data for each connector.
+
+> JSONata is a lightweight query and transformation language for JSON data. Inspired by the 'location path' semantics of XPath 3.1, it allows sophisticated queries to be expressed in a compact and intuitive notation. A rich complement of built in operators and functions is provided for manipulating and combining extracted data, and the results of queries can be formatted into any JSON output structure using familiar JSON object and array syntax. Coupled with the facility to create user defined functions, advanced expressions can be built to tackle any JSON query and transformation task.
+
+For more details, check out the [documentation 🔍](https://docs.jsonata.org/overview.html)
+
+Here are some good examples of basic to complex scenarios:
+
+### 1️⃣ Adding a single facet
+
+We want to map the variant sku to a specific metadata
+
+[👉 Try it for yourself!](https://try.jsonata.org/ufT7N8znc)
+
+<table>
+<tr>
+  <td>Connector data</td>
+  <td>Template</td>
+  <td>Result</td>
+</tr>
+<tr>
+<td>
+
+```ts
+{
+  variant: {
+    sku: "CW21001"
+  }
+}
+```
+
+</td>
+
+<td>
+
+```json
+{
+  "item": {
+    "facets": [
+      {
+        "key": "sku",
+        "value": variant.sku
+      }
+    ]
+  }
+}
+```
+
+</td>
+
+<td>
+
+```ts
+{
+  item: {
+    // ... all other transformed properties should be present
+    facets: [
+      // ... all other facets should be present too
+      {
+        key: "sku",
+        value: "CW21001"
+      }
+    ]
+  }
+}
+```
+
+</td>
+</tr>
+</table>
+
+### 2️⃣ Using built in helper functions
+
+JSONata already provides a powerful set of helper functions that can be used in the templates.
+
+To check all available functions, please take a look at [the documentation](https://docs.jsonata.org/overview.html).
+
+For example, let's say that we want to add `min-price` and `max-price` metadata
+using just one field called `prices`.
+
+[👉 Try it for yourself!](https://try.jsonata.org/b-ddoNC3p)
+
+<table>
+<tr>
+  <td>Connector data</td>
+  <td>Template</td>
+  <td>Result</td>
+</tr>
+<tr>
+<td>
+
+```ts
+{
+  "product": {
+    "prices": [
+      {
+        "value": 19.99,
+        "currency": "USD"
+      },
+      {
+        "value": 25.99,
+        "currency": "USD"
+      },
+      {
+        "value": 59.99,
+        "currency": "USD"
+      }
+    ]
+  }
+}
+```
+
+</td>
+
+<td>
+
+```json
+{
+  "item": {
+    "metadata": [
+      {
+        "key": "min-price",
+        "value": $min(product.prices.value)
+      },
+      {
+        "key": "max-price",
+        "value": $max(product.prices.value)
+      }
+    ]
+  }
+}
+```
+</td>
+
+<td>
+
+```ts
+{
+  item: {
+    // ... all other transformed properties should be present
+    metadata: [
+      // ... all other metadata should be present too
+      {
+        key: "min-price",
+        value: 19.99
+      },
+      {
+        key: "max-price",
+        value: 59.99
+      }
+    ]
+  }
+}
+```
+</td>
+</tr>
+</table>
+
+### 3️⃣ Using helper functions provided by connectors
+
+To make things even easier, our connectors provide helper functions that allow you to easily access
+and parse the external data that comes from the partner.
+
+To get more details into which helper functions are available, please check out the documentation
+for each connector.
+
+For example, let's say we want to define a `description` facet with the right
+translation, with a fallback to English if not found.
+
+[👉 Try it for yourself!](https://try.jsonata.org/yfsTJnbkF)
+
+#### Helper Function
+
+As described before, each connector provides helper functions to make it easier to map your data.
+
+Suppose that our connector exposes the following function below:
+
+```ts
+function getProductDescriptionWithFallback(locale) {
+  const description = product.description.find((desc) => desc.locale === locale);
+
+  if (description) return description;
+
+  // Return fallback if not found
+  return product.description.find((desc) => desc.locale === "en_US");
+}
+```
+
+<table>
+<tr>
+  <td>Connector data</td>
+  <td>Template</td>
+  <td>Result</td>
+</tr>
+<tr>
+<td>
+
+```ts
+{
+  "product": {
+    "description": [
+      {
+        "text": "This is a lipstick",
+        "locale": "en_US"
+      },
+      {
+        "text": "Isso é um batom",
+        "locale": "pt_BR"
+      },
+      {
+        "text": "Ceci est un rouge à lèvres",
+        "locale": "fr_FR"
+      },
+    ]
+  }
+}
+```
+
+</td>
+
+<td>
+
+```js
+{
+  "item": {
+    "facets": [
+      {
+        "key": "description_pt_BR",
+        "value": $getProductDescriptionWithFallback("pt_BR")
+      },
+      {
+        "key": "description_foo",
+        "value": $getProductDescriptionWithFallback("foo")
+      }
+    ]
+  }
+}
+```
+
+</td>
+
+<td>
+
+```ts
+{
+  item: {
+    // ... all other transformed properties should be present
+    facets: [
+      // ... all other facets should be present too
+      {
+        key: "description_pt_BR",
+        value: "Isso é um batom"
+      },
+      {
+        key: "description_foo",
+        value: "This is a lipstick"
+      }
+    ]
+  }
+}
+```
+
+</td>
+</tr>
+</table>
+
+### 4️⃣ Defining new functions and/or variables inside templates
+
+Sometimes you want to do something really complex and the connector might not have a helper
+function ready for this.
+
+In this case, JSONata provides a powerful expression set so that you can create new variables,
+functions and more inside your template to easily handle complex cases.
+
+To get more info on this, please refer to [the documentation](https://docs.jsonata.org/programming).
+
+For example, let's say that for some reason you want to add `bar` to a specific string.
+
+Notice how in the example below we're defining a variable named `bar` and a function named `addBar`,
+that will then be called to override the product id (resulting in `foobar`).
+
+[👉 Try it for yourself!](https://try.jsonata.org/fYov5XxnY)
+
+#### Connector data
+
+```ts
+{
+  product: {
+    id: "foo"
+  }
+}
+```
+
+#### Template
+
+```js
+(
+  $bar := "bar";
+
+  $addBar := function($value) {
+    $join([$value, $bar])
+  };
+
+  {
+    "item": {
+      "id": $addBar(product.id)
+    }
+  }
+)
+```
+
+#### Result
+
+```ts
+{
+  item: {
+    id: "foobar"
+  }
+}
+```
+
+### 5️⃣ Sharing code across templates
+
+Sometimes is handy to move some duplicated code into a separate file and import it whenever needed. You can do this through our `helpers.jsonata` file. And then, reuse it anywhere by simply importing it using the `{{helpers}}` syntax.
+
+Note: This feature is implemented under our custom build process, so the JSONata exerciser will not parse this syntax correctly.
+
+```js
+/* helpers.jsonata */
+
+$getProductUrl := function () {
+  /* fancy code */
+};
+```
+
+```js
+/* any other .jsonata file */
+
+/* imports the helpers */
+{{helpers}}
+
+{
+  "url": $getProductUrl()
+}
+```

package/boilerplate-repo/connectrc.js

@@ -0,0 +1,7 @@
+// @ts-check
+
+const connections = "{{REPLACE_CONNECTIONS}}";
+
+const config = "{{REPLACE_CONFIG}}";
+
+module.exports = config;

package/boilerplate-repo/package.json

@@ -0,0 +1,21 @@
+{
+  "name": "{{REPLACE_REPO_NAME}}",
+  "version": "1.0.0",
+  "description": "🔧 Repository to interface with the Constructor Connect ecosystem.",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1",
+    "lint": "eslint . --ext .js",
+    "lint:fix": "eslint . --ext .js --fix",
+    "execute": "constructorio-connect-cli execute",
+    "deploy": "constructorio-connect-cli deploy"
+  },
+  "devDependencies": {
+    "eslint": "^8.47.0",
+    "eslint-plugin-prettier": "^5.1.2",
+    "eslint-config-prettier": "^9.1.0",
+    "prettier": "^3.1.1"
+  },
+  "keywords": [],
+  "author": "",
+  "license": "ISC"
+}