@jupiterone/integration-sdk-cli 12.7.1 → 12.8.3

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jupiterone/integration-sdk-cli",
-  "version": "12.7.1",
+  "version": "12.8.3",
   "description": "The SDK for developing JupiterOne integrations",
   "main": "dist/src/index.js",
   "types": "dist/src/index.d.ts",
@@ -17,18 +17,20 @@
     "access": "public"
   },
   "scripts": {
+    "test": "jest",
     "prebuild:dist": "rm -rf dist && mkdir dist",
-    "build:dist": "tsc -p tsconfig.dist.json --declaration && yarn copy-files",
-    "prepack": "yarn build:dist",
-    "copy-files": "cp -r src/generator/template dist/src/generator/template && cp -r src/generator/stepTemplate dist/src/generator/stepTemplate",
+    "build:dist": "tsc -p tsconfig.dist.json --declaration && npm run copy-files",
+    "prepack": "npm run build:dist",
+    "copy-files": "cp -r src/generator/template dist/src/generator/template && cp -r src/generator/stepTemplate dist/src/generator/stepTemplate && cp -r src/bocchi/templates dist/src/bocchi/templates",
     "plop": "plop --plopfile dist/src/generator/newIntegration.js"
   },
   "dependencies": {
-    "@jupiterone/data-model": "^0.55.0",
-    "@jupiterone/integration-sdk-core": "^12.7.1",
-    "@jupiterone/integration-sdk-runtime": "^12.7.1",
+    "@jupiterone/data-model": "^0.61.3",
+    "@jupiterone/integration-sdk-core": "^12.8.3",
+    "@jupiterone/integration-sdk-runtime": "^12.8.3",
     "chalk": "^4",
     "commander": "^9.4.0",
+    "ejs": "^3.1.9",
     "fs-extra": "^10.1.0",
     "globby": "^11.0.0",
     "inquirer-checkbox-plus-prompt": "^1.4.2",
@@ -43,7 +45,7 @@
     "url-exists": "^1.0.3"
   },
   "devDependencies": {
-    "@jupiterone/integration-sdk-private-test-utils": "^12.7.1",
+    "@jupiterone/integration-sdk-private-test-utils": "^12.8.3",
     "@pollyjs/adapter-node-http": "^6.0.5",
     "@pollyjs/core": "^6.0.5",
     "@pollyjs/persister-fs": "^6.0.5",
@@ -56,5 +58,5 @@
     "neo-forgery": "^2.0.0",
     "vis": "^4.21.0-EOL"
   },
-  "gitHead": "e4d3dbe3b95d47e043d10f508355195a99ea4da3"
+  "gitHead": "aa9dfabc973f1b046424b742b34cafc91261c3f5"
 }

package/src/bocchi/README.md

@@ -0,0 +1,95 @@
+# Bocchi - a JupiterOne graph project generator
+
+Developing a graph project from scratch for a new JupiterOne integration can be
+time consuming, and typically involves writing a lot of boilerplate code that is
+similar across all integrations. This is where Bocchi comes in. It's purpose is
+to eliminate the need to write boilerplate code, as well as standardize much of
+the logic and practices our graph projects use. Moreover, Bocchi allows us to
+more quickly expand JupiterOne’s integration roster without spending a ton of
+resources and development time.
+
+Bocchi is a one-time use tool to be used when creating a new graph project. To
+use Bocchi, a developer provides a JSON template that contains key components of
+the new integration, such as instance configuration fields, the authentication
+method, and the steps the new integration should run. Bocchi will interpret this
+JSON template and generate a fully-functioning graph project - things like an
+http-service client, step handlers, authentication functions, etc. will all be
+written for the developer. After using Bocchi to generate a new graph project,
+the only code the developer (may) need to write is the more involved,
+vendor-specific logic.
+
+To try Bocchi out, you can use the example template
+[here](./docs/template/examples/signalSciences.json).
+
+## The Template
+
+To create a new graph project using Bocchi, a developer needs to provide a JSON
+template that Bocchi will interpret to generate graph code. The JSON template
+defines key components of the integration, such as instance configuration
+fields, the authentication method, as well as what steps the integration should
+have. Moreoever, the steps outlined in the template contain information on what
+API endpoints should be used for the step, as well as what entities and
+relationships should be ingested.
+
+<b>Currently, Bocchi only supports creating graph projects that ingest data from
+REST APIs!</b>
+
+Documentation for the JSON template can be found
+[here](./docs/template/README.md).
+
+Additionally, example template files can be found
+[here](./docs/template/examples/).
+
+## Usage
+
+```
+yarn j1-integration bocchi
+```
+
+## Additional Notes
+
+### Authentication
+
+Bocchi currently supports two different methods of authentication for
+integrations: Endpoint authentication and Config Value authentication.
+
+Endpoint authentication is used when the product we're integrating with requires
+an API request to a special endpoint to receive authentication headers that will
+be used in all future requests. However, if the product does not require a
+special authentication endpoint to be hit and instead something like a pre-made
+API token can be sent on all API requests, then Config Value authentication can
+be used.
+
+Note that if Config Value authentication is used for the integration - that is,
+authentication information is provided in the instance config - then the graph
+project Bocchi generates will have an incomplete `verifyAuthentication()`
+function in the client. This is because the graph-project generator does not
+know what endpoint to use for verifying authentication.
+
+<b>The developer will need to manually update the generated
+verifyAuthentication() function and choose what endpoint should be used for
+verifying authentication.</b>
+
+### Types
+
+After running Bocchi, the generated graph project will have a `types.ts` file
+that contains TS types for the entities the integration will produce. The types
+however will be `any`. This is because Bocchi cannot guess what properties will
+be on the different entities the integration will produce. So for prosperity and
+to follow best coding practices, <b>The types should be updated manually by the
+developer.</b>
+
+### Mapped Relationships
+
+If a developer defines a mapped relationship(s) in the template, then they may
+notice the following warning when running the integration locally:
+
+```
+The following types were encountered but are not declared in the step's "types" field:
+```
+
+If a step produces a mapped relationship(s), the developer will need to manually
+add the `mappedRelationships` field in the generated stepMap for that step. The
+`mappedRelationships` field will need to contain a valid
+`StepMappedRelationshipMetadata` object (from
+`@jupiterone/integration-sdk-core`).

package/src/bocchi/actions/steps.ts

@@ -0,0 +1,17 @@
+import { Step, StepType } from '../utils/types';
+
+export function stepTemplateHelper(step: Step) {
+  return {
+    stepTemplateFile: `${determineTypeOfStep(step)}.ts.hbs`,
+  };
+}
+
+function determineTypeOfStep(step: Step): string {
+  return step.response.responseType === 'SINGLETON'
+    ? step.parentAssociation
+      ? StepType.CHILD_SINGLETON
+      : StepType.SINGLETON
+    : step.parentAssociation
+      ? StepType.FETCH_CHILD_ENTITIES
+      : StepType.FETCH_ENTITIES;
+}

package/src/bocchi/bocchi.ts

@@ -0,0 +1,311 @@
+import { NodePlopAPI } from 'plop';
+import checkboxPlus from 'inquirer-checkbox-plus-prompt';
+import path from 'path';
+import { kebabCase } from 'lodash';
+import {
+  yarnFormat,
+  yarnInstall,
+  yarnLint,
+  j1Document,
+} from '../generator/actions';
+import { Template } from './utils/types';
+import * as fs from 'fs';
+import { stepTemplateHelper } from './actions/steps';
+import {
+  IntegrationInstanceConfigFieldMap,
+  RelationshipDirection,
+  generateRelationshipType,
+} from '@jupiterone/integration-sdk-core';
+
+function bocchi(plop: NodePlopAPI) {
+  plop.setActionType('yarnFormat', yarnFormat);
+  plop.setActionType('yarnInstall', yarnInstall);
+  plop.setActionType('yarnLint', yarnLint);
+  plop.setActionType('j1Document', j1Document);
+  plop.setPrompt('checkbox-plus', checkboxPlus);
+  plop.setHelper('getDirectRelationships', (step, options) => {
+    return step.directRelationships.map((relationship) => {
+      const step2 = options.data.root.template.steps.find(
+        (s) => s.entity._type === relationship.targetType,
+      );
+      return {
+        step,
+        sourceStep:
+          relationship.direction === RelationshipDirection.FORWARD
+            ? step
+            : step2,
+        targetStep:
+          relationship.direction === RelationshipDirection.FORWARD
+            ? step2
+            : step,
+        relationshipClass: relationship._class,
+        targetKey: relationship.targetKey,
+        forward: relationship.direction === RelationshipDirection.FORWARD,
+      };
+    });
+  });
+  plop.setHelper('getStepByType', (type, options) => {
+    return options.data.root.template.steps.find(
+      (s) => s.entity._type === type,
+    );
+  });
+  /**
+   * (_class, fromType, toType) => string
+   */
+  plop.setHelper('getRelationshipType', generateRelationshipType);
+  plop.setHelper('sanitizeUrlPath', (urlTemplate: string): string => {
+    const regex = /%parent\.(.+?)%/g;
+    for (const match of urlTemplate.matchAll(regex)) {
+      urlTemplate = urlTemplate.replace(
+        match[0],
+        '${parentEntity.' + match[1] + '}',
+      );
+    }
+    // looks at the urlTemplate provided and replaces "%nextToken%" with "${nextToken}",
+    // which is the string the client file will use to reference the nextToken
+    return urlTemplate.replace('%nextToken%', '${nextToken}');
+  });
+  plop.setHelper('escape', (data) => {
+    if (typeof data === 'string') {
+      return "'data'";
+    } else if (Array.isArray(data)) {
+      return '[' + data.toString() + ']';
+    } else {
+      return data;
+    }
+  });
+  plop.setHelper('sanitizeHttpMethod', (method: string = 'GET'): string => {
+    // if the user puts nothing in request.method, we need to default to GET
+    return method.toUpperCase();
+  });
+  plop.setHelper(
+    'sanitizeHttpBody',
+    (body: Record<string, any>): Record<string, any> => {
+      // Also, we need to check if the body contains "%nextToken%" and replace it with
+      // "nextToken" since that is the var in the client file that references the nextToken.
+      for (const key in body) {
+        if (typeof body[key] === 'string') {
+          body[key] = body[key].replace('%nextToken%', '${nextToken}');
+        }
+      }
+      return body;
+    },
+  );
+  plop.setHelper('isSingletonRequest', (responseType: string): boolean => {
+    return responseType === 'SINGLETON';
+  });
+  plop.setHelper('configTypeToType', (type: string) => {
+    if (type.toLowerCase() === 'json') {
+      return 'Record<string, any>';
+    }
+    return type;
+  });
+  plop.setHelper(
+    'requiredConfig',
+    (configFields: IntegrationInstanceConfigFieldMap) => {
+      return Object.entries(configFields)
+        .filter(([_, value]) => !value.optional)
+        .map(([key, _]) => key);
+    },
+  );
+  plop.setHelper('isNotEndpointAuth', (authStrategy: string): boolean => {
+    return authStrategy !== 'endpoint';
+  });
+  plop.setHelper(
+    'sanitizeAuthObject',
+    (authObj: Record<string, any>): Record<string, any> => {
+      const regex = /%(response|config)\.(.+?)%/g;
+      for (const key in authObj) {
+        if (typeof authObj[key] === 'string') {
+          for (const match of authObj[key].matchAll(regex)) {
+            authObj[key] = authObj[key].replace(
+              match[0],
+              '${' +
+                (match[1] === 'config' ? 'this.' : '') +
+                match[1] +
+                '.' +
+                match[2] +
+                '}',
+            );
+          }
+        }
+      }
+      return authObj;
+    },
+  );
+
+  for (const partial of fs.readdirSync(
+    path.join(__dirname, 'templates/partials'),
+  )) {
+    plop.setPartial(
+      partial.replace('.hbs', ''),
+      fs.readFileSync(
+        path.join(__dirname, 'templates/partials', partial),
+        'utf8',
+      ),
+    );
+  }
+
+  plop.setGenerator('bocchi', {
+    description: 'Create a new integration graph project',
+    prompts: async function (inquirer) {
+      const { vendorName } = await inquirer.prompt({
+        type: 'input',
+        name: 'vendorName',
+        message:
+          'The integration vendor name (ex. AWS, Google Workspace, CrowdStrike)',
+        validate(input) {
+          if (!input) {
+            return 'Cannot be an empty string';
+          }
+          return true;
+        },
+      });
+
+      const { packageName } = await inquirer.prompt({
+        type: 'input',
+        name: 'packageName',
+        default: `@jupiterone/graph-${kebabCase(vendorName)}`,
+        message: 'The npm package name',
+        validate(input) {
+          if (!input) {
+            return 'Cannot be an empty string';
+          }
+          return true;
+        },
+      });
+
+      const { packageDescription } = await inquirer.prompt({
+        type: 'input',
+        name: 'packageDescription',
+        message: 'A description for package',
+        default: `An integration and graph conversion project for ${vendorName}`,
+        validate(input) {
+          if (!input) {
+            return 'Cannot be an empty string';
+          }
+          return true;
+        },
+      });
+
+      const { templateFile } = await inquirer.prompt({
+        type: 'input',
+        name: 'templateFile',
+        message: 'Template file for the integration',
+        validate(input) {
+          if (!input) {
+            return 'Cannot be an empty string';
+          }
+          return true;
+        },
+      });
+
+      return {
+        vendorName,
+        packageName,
+        packageDescription,
+        templateFile,
+      };
+    },
+    actions: function (data) {
+      if (!data) {
+        return [];
+      }
+
+      const { templateFile } = data;
+
+      // @jupiterone/graph-foo -> graph-foo
+      // graph-foo -> graph-foo
+      const directoryName = path.join(
+        process.cwd(),
+        path.basename(data.packageName),
+      );
+
+      const template: Template = JSON.parse(
+        fs.readFileSync(templateFile, 'utf8'),
+      );
+
+      const actions: any[] = [];
+
+      actions.push({
+        type: 'addMany',
+        destination: directoryName,
+        base: path.join(__dirname, '/templates/top-level'),
+        templateFiles: path.join(__dirname + '/templates/top-level/**'),
+        globOptions: { dot: true },
+        force: true,
+        data: { ...data, template },
+      });
+
+      for (const step of template.steps) {
+        const { stepTemplateFile } = stepTemplateHelper(step);
+        actions.push({
+          type: 'add',
+          path: path.join(
+            directoryName,
+            path.normalize(`src/steps/${kebabCase(step.id)}/index.ts`),
+          ),
+          templateFile: path.join(
+            __dirname,
+            `templates/steps/${stepTemplateFile}`,
+          ),
+          data: { template, step },
+          force: true,
+        });
+
+        actions.push({
+          type: 'add',
+          path: path.join(
+            directoryName,
+            path.normalize(`src/steps/${kebabCase(step.id)}/index.test.ts`),
+          ),
+          templateFile: path.join(
+            __dirname,
+            `templates/steps/index.test.ts.hbs`,
+          ),
+          data: { template, step },
+          force: true,
+        });
+
+        actions.push({
+          type: 'add',
+          path: path.join(
+            directoryName,
+            path.normalize(`docs/spec/${kebabCase(step.id)}/index.ts`),
+          ),
+          templateFile: path.join(__dirname, `templates/steps/spec.ts.hbs`),
+          data: { template, step },
+          force: true,
+        });
+      }
+
+      actions.push({
+        type: 'yarnInstall',
+        path: directoryName,
+        verbose: true,
+      });
+
+      actions.push({
+        type: 'j1Document',
+        path: directoryName,
+        verbose: true,
+      });
+
+      actions.push({
+        type: 'yarnFormat',
+        path: directoryName,
+        verbose: true,
+      });
+
+      actions.push({
+        type: 'yarnLint',
+        path: directoryName,
+        verbose: true,
+      });
+
+      return actions;
+    },
+  });
+}
+
+module.exports = bocchi;

package/src/bocchi/docs/template/README.md

@@ -0,0 +1,140 @@
+# The Template File
+
+In order to generate the graph project code, Bocchi interprets a
+developer-provided JSON template file. This JSON template defines key components
+of the integration, such as instance configuration fields, the authentication
+method, as well as what steps the integration should have.
+
+Contents:
+
+- [Format](#format)
+- [Fields](#fields)
+  - [Instance Config Fields](#instance-config-fields)
+  - [Base URL](#base-url)
+  - [Token Bucket](#token-bucket)
+  - [Authentication](#authentication)
+  - [Steps](#steps)
+
+## Format
+
+```ts
+{
+    instanceConfigFields: Object,
+    baseUrl: string,
+    tokenBucket: Object
+    authentication: Object,
+    steps: Array<Object>
+}
+```
+
+## Fields
+
+### Instance Config Fields
+
+Lists out the config fields used by this integration.
+
+Schema:
+
+|          |                        |
+| -------- | ---------------------- |
+| Property | `instanceConfigFields` |
+| Type     | `Object`               |
+| Required | `true`                 |
+
+```ts
+{
+    [configFieldName]: {
+        type?: 'string' | 'json' | 'boolean',
+        mask?: Boolean,
+        optional?: Boolean
+    }
+}
+```
+
+Example:
+
+```json
+{
+  "accessKey": {
+    "type": "string"
+  },
+  "accessKeyId": {
+    "type": "string",
+    "optional": false
+  },
+  "example": {
+    "type": "boolean",
+    "mask": true,
+    "optional": true
+  }
+}
+```
+
+### Base URL
+
+Template for the base URL of the API.
+
+Schema:
+
+|                     |           |
+| ------------------- | --------- |
+| Property            | `baseUrl` |
+| Type                | `string`  |
+| Required            | `true`    |
+| Available templates | `config`  |
+
+Examples:
+
+`https://api.github.com/octocat`
+
+`https://%config.tenant%.example.com`
+
+### Token Bucket
+
+Allows you to define a rate limit across all steps. Useful for an API where the
+rate limit is per API key rather than per endpoint.
+
+Can be used in combination with step-level token buckets.
+
+Schema:
+
+|          |               |
+| -------- | ------------- |
+| Property | `tokenBucket` |
+| Type     | `Object`      |
+| Required | `false`       |
+
+```ts
+{
+    maximumCapacity: number,
+    refillRate: number
+}
+```
+
+Examples:
+
+10 API calls / second
+
+```json
+{
+  "maximumCapacity": 10,
+  "refillRate": 10
+}
+```
+
+10 API calls / second, 50 Burst
+
+```json
+{
+  "maximumCapacity": 50,
+  "refillRate": 10
+}
+```
+
+### Authentication
+
+[Authentication](./authentication.md)
+
+### Steps
+
+[Steps](./steps.md)