@devrev/ts-adaas 1.2.6 → 1.4.0

package/README.md

@@ -23,275 +23,4 @@ npm install @devrev/ts-adaas
 
 ## Reference
 
-### `spawn` function
-
-This function initializes a new worker thread and oversees its lifecycle. 
-It should be invoked when the snap-in receives a message from the Airdrop platform.
-The worker script provided then handles the event accordingly.
-
-#### Usage
-
-```typescript
-spawn({ event, initialState, workerPath, options })
-```
-
-#### Parameters
-
-* _event_
-  
-  Required. An object of type __AirdropEvent__ that is received from the Airdrop platform.
-
-* _initialState_
-
-  Required. Object of __any__ type that represents the initial state of the snap-in.
-
-* _workerPath_
-
-  Required. A __string__ that represents the path to the worker file.
-
-* _options_
-
-  Optional. An object of type **WorkerAdapterOptions**, which will be passed to the newly created worker. This worker will then initialize a `WorkerAdapter` by invoking the `processTask` function. The options include:
-  
-  * `isLocalDevelopment`
-  
-    A __boolean__ flag. If set to `true`, intermediary files containing extracted data will be stored on the local machine, which is useful during development. The default value is `false`.
-
-  * `timeout`
-  
-    A __number__ that specifies the timeout duration for the lambda function, in milliseconds. The default is 10 minutes (10 * 60 * 1000 milliseconds), with a maximum allowable duration of 13 minutes (13 * 60 * 1000 milliseconds).
-  
-  * `batchSize`
-
-    A __number__ that determines the maximum number of items to be processed and saved to an intermediary file before being sent to the Airdrop platform. The default batch size is 2,000.
-
-#### Return value
-
-A __promise__ that resolves once the worker has completed processing.
-
-#### Example
-
-```typescript
-const run = async (events: AirdropEvent[]) => {
-  for (const event of events) {
-    const file = getWorkerPerExtractionPhase(event);
-    await spawn<ExtractorState>({
-      event,
-      initialState,
-      workerPath: file,
-    });
-  }
-};
-```
-
-### `processTask` function
-
-The `processTask` function retrieves the current state from the Airdrop platform and initializes a new `WorkerAdapter`.
-It executes the code specified in the `task` parameter, which contains the worker's functionality.
-If a timeout occurs, the function handles it by executing the `onTimeout` callback, ensuring the worker exits gracefully.
-Both functions receive an `adapter` parameter, representing the initialized `WorkerAdapter` object.
-
-
-#### Usage
-```typescript
-processTask({ task, onTimeout })
-```
-
-#### Parameters
-
-* _task_
-  
-  Required. A __function__ that defines the logic associated with the given event type.
-
-* _onTimeout_
-  
-  Required. A __function__ managing the timeout of the lambda invocation, including saving any necessary progress at the time of timeout.
-
-#### Example
-
-````typescript
-// External sync units extraction
-processTask({
-  task: async ({ adapter }) => {
-    const httpClient = new HttpClient(adapter.event);
-
-    const todoLists = await httpClient.getTodoLists();
-
-    const externalSyncUnits: ExternalSyncUnit[] = todoLists.map((todoList) => normalizeTodoList(todoList));
-
-    await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsDone, {
-      external_sync_units: externalSyncUnits,
-    });
-  },
-  onTimeout: async ({ adapter }) => {
-    await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsError, {
-      error: {
-        message: 'Failed to extract external sync units. Lambda timeout.',
-      },
-    });
-  },
-});
-````
-
-### `WorkerAdapter` class
-
-Used to interact with Airdrop platform. 
-Provides utilities to emit events to the Airdrop platform, update the state of the snap-in and upload artifacts (files with data) to the platform.
-
-### Usage
-
-```typescript
-new WorkerAdapter({
-  event,
-  adapterState,
-  options,
-});
-```
-
-#### Parameters
-
-* _event_
-  
-  Required. An object of type __AirdropEvent__ that is received from the Airdrop platform.
-
-* _adapterState_
-  
-  Required. An object of type __State__, which represents the initial state of the adapter.
-
-* _options_
-  
-  Optional. An object of type __WorkerAdapterOptions__ that specifies additional configuration options for the `WorkerAdapter`. This object is passed via the `spawn` function.
-
-#### Example
-
-```typescript
-const adapter = new WorkerAdapter<ConnectorState>({
-  event,
-  adapterState,
-  options,
-});
-```
-
-### `WorkerAdapter.state` property
-
-Getter and setter methods for working with the adapter state.
-
-### Usage
-
-```typescript
-// get state
-const adapterState = adapter.state;
-
-// set state
-adapter.state = newAdapterState;
-```
-
-#### Example
-
-```typescript
-export const initialState: ExtractorState = {
-  users: { completed: false },
-  tasks: { completed: false },
-  attachments: { completed: false },
-};
-
-adapter.state = initialState;
-```
-
-### `WorkerAdapter.initializeRepos` method
-
-Initializes a `Repo` object for each item provided.
-
-### Usage
-
-```typescript
-adapter.initializeRepos(repos);
-```
-
-#### Parameters
-
-* _repos_
-  
-  Required. An array of objects of type `RepoInterface`. 
-
-#### Example
-
-This should typically be called within the function passed as a parameter to the `processTask` function in the data extraction phase.
-
-```typescript
-const repos = [
-  {
-    itemType: 'tasks',
-    normalize: normalizeTask,
-  }
-];
-
-adapter.initializeRepos(repos);
-```
-
-### `WorkerAdapter.getRepo` method
-
-Finds a Repo from the initialized repos.
-
-### Usage
-
-```typescript
-adapter.getRepo(itemType);
-```
-
-#### Parameters
-
-* _itemType_
-  
-  Required. A __string__ that represents the itemType property for the searched repo.
-
-#### Return value
-
-An object of type __Repo__ if the repo is found, otherwise __undefined__.
-
-#### Example
-
-This should typically be called within the function passed as a parameter to the `processTask` function.
-
-```typescript
-// Push users to the repository designated for 'users' data.
-await adapter.getRepo('users')?.push(users);
-```
-
-### `WorkerAdapter.emit` method
-
-Emits an event to the Airdrop platform.
-
-### Usage
-
-```typescript
-adapter.emit( newEventType, data ):
-```
-
-#### Parameters
-
-* _newEventType_
-  
-  Required. The event type to be emitted, of type __ExtractorEventType__ or __LoaderEventType__.
-
-* _data_
-  
-  Optional. An object of type __EventData__ which represents the data to be sent with the event.
-
-#### Return value
-
-A __promise__, which resolves to undefined after the emit function completes its execution or rejects with an error.
-
-#### Example
-
-This should typically be called within the function passed as a parameter to the `processTask` function.
-
-```typescript
-// Emitting successfully finished data extraction.
-await adapter.emit(ExtractorEventType.ExtractionDataDone);
-
-// Emitting a delay in attachments extraction phase.
-await adapter.emit(ExtractorEventType.ExtractionAttachmentsDelay, {
-  delay: 10,
-});
-```
+Please refer to the [REFERENCE.md](./REFERENCE.md) file for more information on the types, interfaces and functions used in the library.

package/dist/common/install-initial-domain-mapping.d.ts

@@ -1,3 +1,3 @@
-import { FunctionInput } from '@devrev/typescript-sdk/dist/snap-ins';
+import { AirdropEvent } from '../types/extraction';
 import { InitialDomainMapping } from '../types/common';
-export declare function installInitialDomainMapping(event: FunctionInput, initialDomainMappingJson: InitialDomainMapping): Promise<void>;
+export declare function installInitialDomainMapping(event: AirdropEvent, initialDomainMappingJson: InitialDomainMapping): Promise<void>;

package/dist/http/axios-client.d.ts

@@ -1,3 +1,26 @@
+/**
+ * Axios client setup with retry capabilities using axios-retry.
+ *
+ * This module exports an Axios client instance (`axiosClient`) that is configured to automatically retry
+ * failed requests under certain conditions.
+ *
+ * Retry Conditions:
+ * 1. Network errors (where no response is received).
+ * 2. Idempotent requests (defaults include GET, HEAD, OPTIONS, PUT).
+ * 3. All 5xx server errors.
+ *
+ * Retry Strategy:
+ * - A maximum of 5 retries are attempted.
+ * - Exponential backoff delay is applied between retries, increasing with each retry attempt.
+ *
+ * Additional Features:
+ * - When the maximum number of retry attempts is reached, sensitive headers (like authorization)
+ *   are removed from error logs for security reasons.
+ *
+ * Exported:
+ * - `axios`: Original axios instance for additional customizations or direct use.
+ * - `axiosClient`: Configured axios instance with retry logic.
+ */
 import axios from 'axios';
 declare const axiosClient: import("axios").AxiosInstance;
 export { axios, axiosClient };

package/dist/http/axios-client.js

@@ -1,4 +1,27 @@
 "use strict";
+/**
+ * Axios client setup with retry capabilities using axios-retry.
+ *
+ * This module exports an Axios client instance (`axiosClient`) that is configured to automatically retry
+ * failed requests under certain conditions.
+ *
+ * Retry Conditions:
+ * 1. Network errors (where no response is received).
+ * 2. Idempotent requests (defaults include GET, HEAD, OPTIONS, PUT).
+ * 3. All 5xx server errors.
+ *
+ * Retry Strategy:
+ * - A maximum of 5 retries are attempted.
+ * - Exponential backoff delay is applied between retries, increasing with each retry attempt.
+ *
+ * Additional Features:
+ * - When the maximum number of retry attempts is reached, sensitive headers (like authorization)
+ *   are removed from error logs for security reasons.
+ *
+ * Exported:
+ * - `axios`: Original axios instance for additional customizations or direct use.
+ * - `axiosClient`: Configured axios instance with retry logic.
+ */
 var __importDefault = (this && this.__importDefault) || function (mod) {
     return (mod && mod.__esModule) ? mod : { "default": mod };
 };
@@ -19,9 +42,10 @@ exports.axiosClient = axiosClient;
         return delay;
     },
     retryCondition: (error) => {
-        var _a;
-        return (axios_retry_1.default.isNetworkOrIdempotentRequestError(error) &&
-            ((_a = error.response) === null || _a === void 0 ? void 0 : _a.status) !== 429);
+        var _a, _b, _c;
+        return ((axios_retry_1.default.isNetworkOrIdempotentRequestError(error) &&
+            ((_a = error.response) === null || _a === void 0 ? void 0 : _a.status) !== 429) ||
+            ((_c = (_b = error.response) === null || _b === void 0 ? void 0 : _b.status) !== null && _c !== void 0 ? _c : 0) >= 500);
     },
     onMaxRetryTimesExceeded(error) {
         var _a, _b, _c, _d;

package/dist/repo/repo.d.ts

@@ -11,5 +11,5 @@ export declare class Repo {
     constructor({ event, itemType, normalize, onUpload, options, }: RepoFactoryInterface);
     getItems(): (NormalizedItem | NormalizedAttachment | Item)[];
     upload(batch?: (NormalizedItem | NormalizedAttachment | Item)[]): Promise<void | ErrorRecord>;
-    push(items: Item[]): Promise<boolean | ErrorRecord>;
+    push(items: Item[]): Promise<boolean>;
 }

package/dist/state/state.d.ts

@@ -1,6 +1,6 @@
 import { ErrorRecord } from '../types/common';
 import { AdapterState, StateInterface } from './state.interfaces';
-export declare function createAdapterState<ConnectorState>({ event, initialState, options, }: StateInterface<ConnectorState>): Promise<State<ConnectorState>>;
+export declare function createAdapterState<ConnectorState>({ event, initialState, initialDomainMapping, options, }: StateInterface<ConnectorState>): Promise<State<ConnectorState>>;
 export declare class State<ConnectorState> {
     private _state;
     private initialSdkState;

package/dist/state/state.interfaces.d.ts

@@ -1,20 +1,24 @@
 import { AirdropEvent } from '../types/extraction';
 import { FileToLoad } from '../types/loading';
 import { WorkerAdapterOptions } from '../types/workers';
+import { InitialDomainMapping } from '../types/common';
 export interface SdkState {
     lastSyncStarted?: string;
     lastSuccessfulSyncStarted?: string;
     toDevRev?: ToDevRev;
     fromDevRev?: FromDevRev;
+    snapInVersionId?: string;
 }
 /**
- * AdapterState is an interface that defines the structure of the adapter state that is used by the external extractor. It extends the connector state with additional fields: lastSyncStarted, lastSuccessfulSyncStarted, and attachmentsMetadata.
+ * AdapterState is an interface that defines the structure of the adapter state that is used by the external extractor.
+ * It extends the connector state with additional fields: lastSyncStarted, lastSuccessfulSyncStarted, snapInVersionId and attachmentsMetadata.
  */
 export type AdapterState<ConnectorState> = ConnectorState & SdkState;
 export interface ToDevRev {
     attachmentsMetadata: {
         artifactIds: string[];
         lastProcessed: number;
+        lastProcessedAttachmentsIdsList?: string[];
     };
 }
 export interface FromDevRev {
@@ -23,5 +27,6 @@ export interface FromDevRev {
 export interface StateInterface<ConnectorState> {
     event: AirdropEvent;
     initialState: ConnectorState;
+    initialDomainMapping?: InitialDomainMapping;
     options?: WorkerAdapterOptions;
 }

package/dist/state/state.js

@@ -6,16 +6,36 @@ const axios_client_1 = require("../http/axios-client");
 const extraction_1 = require("../types/extraction");
 const constants_1 = require("../common/constants");
 const logger_1 = require("../logger/logger");
+const install_initial_domain_mapping_1 = require("../common/install-initial-domain-mapping");
 const helpers_1 = require("../common/helpers");
-async function createAdapterState({ event, initialState, options, }) {
+async function createAdapterState({ event, initialState, initialDomainMapping, options, }) {
+    var _a;
     const newInitialState = structuredClone(initialState);
     const as = new State({
         event,
         initialState: newInitialState,
+        initialDomainMapping,
         options,
     });
     if (!constants_1.STATELESS_EVENT_TYPES.includes(event.payload.event_type)) {
         await as.fetchState(newInitialState);
+        const snapInVersionId = event.context.snap_in_version_id;
+        const hasSnapInVersionInState = 'snapInVersionId' in as.state;
+        const shouldUpdateIDM = !hasSnapInVersionInState || as.state.snapInVersionId !== snapInVersionId;
+        if (shouldUpdateIDM) {
+            console.log(`Snap-in version in state (${(_a = as.state) === null || _a === void 0 ? void 0 : _a.snapInVersionId}) differs from the version in event context (${snapInVersionId}) - initial domain mapping needs to be updated.`);
+            if (initialDomainMapping) {
+                await (0, install_initial_domain_mapping_1.installInitialDomainMapping)(event, initialDomainMapping);
+                as.state.snapInVersionId = snapInVersionId;
+                console.log('Successfully installed new initial domain mapping.');
+            }
+            else {
+                console.warn('No initial domain mapping was passed to spawn function. Skipping initial domain mapping installation.');
+            }
+        }
+        else {
+            console.log(`Snap-in version in state matches the version in event context (${snapInVersionId}). Skipping initial domain mapping installation.`);
+        }
         if (event.payload.event_type === extraction_1.EventType.ExtractionDataStart &&
             !as.state.lastSyncStarted) {
             as.state.lastSyncStarted = new Date().toISOString();
@@ -29,6 +49,7 @@ class State {
         this.initialSdkState =
             (0, helpers_1.getSyncDirection)({ event }) === extraction_1.SyncMode.LOADING
                 ? {
+                    snapInVersionId: '',
                     fromDevRev: {
                         filesToLoad: [],
                     },
@@ -36,10 +57,12 @@ class State {
                 : {
                     lastSyncStarted: '',
                     lastSuccessfulSyncStarted: '',
+                    snapInVersionId: '',
                     toDevRev: {
                         attachmentsMetadata: {
                             artifactIds: [],
                             lastProcessed: 0,
+                            lastProcessedAttachmentsIdsList: [],
                         },
                     },
                 };
@@ -96,7 +119,7 @@ class State {
             this.event.payload.event_context.sync_unit_id +
             '.');
         try {
-            const response = await axios_client_1.axiosClient.post(this.workerUrl + '.get', {}, {
+            const response = await axios_client_1.axiosClient.get(this.workerUrl + '.get', {
                 headers: {
                     Authorization: this.devrevToken,
                 },

package/dist/tests/test-helpers.js

@@ -10,7 +10,7 @@ function createEvent({ eventType, externalSyncUnits = [], progress, error, delay
     return {
         context: Object.assign({ secrets: {
                 service_account_token: 'test_token',
-            }, snap_in_version_id: 'test_snap_in_version_id' }, contextOverrides),
+            }, snap_in_version_id: 'test_snap_in_version_id', snap_in_id: 'test_snap_in_id' }, contextOverrides),
         payload: Object.assign({ connection_data: {
                 org_id: 'test_org_id',
                 org_name: 'test_org_name',

package/dist/types/extraction.d.ts

@@ -197,6 +197,7 @@ export interface AirdropEvent {
             service_account_token: string;
         };
         snap_in_version_id: string;
+        snap_in_id: string;
     };
     payload: AirdropMessage;
     execution_metadata: {
@@ -257,9 +258,10 @@ export type StreamAttachmentsReturnType = {
     delay?: number;
     error?: ErrorRecord;
 } | undefined;
-export type ExternalSystemAttachmentReducerFunction<Batch, NewBatch, ConnectorState> = ({ attachments, adapter, }: {
+export type ExternalSystemAttachmentReducerFunction<Batch, NewBatch, ConnectorState> = ({ attachments, adapter, batchSize, }: {
     attachments: Batch;
     adapter: WorkerAdapter<ConnectorState>;
+    batchSize?: number;
 }) => NewBatch;
 export type ExternalProcessAttachmentFunction = ({ attachment, stream, }: {
     attachment: NormalizedAttachment;

package/dist/types/workers.d.ts

@@ -3,6 +3,7 @@ import { State } from '../state/state';
 import { WorkerAdapter } from '../workers/worker-adapter';
 import { ExtractorEventType, AirdropEvent } from './extraction';
 import { LoaderEventType } from './loading';
+import { InitialDomainMapping } from './common';
 /**
  * WorkerAdapterInterface is an interface for WorkerAdapter class.
  * @interface WorkerAdapterInterface
@@ -52,6 +53,7 @@ export interface SpawnInterface {
  * @param {AirdropEvent} event - The event object received from the platform
  * @param {object=} initialState - The initial state of the adapter
  * @param {string} workerPath - The path to the worker file
+ * @param {string} initialDomainMapping - The initial domain mapping
  * @param {WorkerAdapterOptions} options - The options to create a new instance of Spawn class
  */
 export interface SpawnFactoryInterface<ConnectorState> {
@@ -59,6 +61,7 @@ export interface SpawnFactoryInterface<ConnectorState> {
     initialState: ConnectorState;
     workerPath?: string;
     options?: WorkerAdapterOptions;
+    initialDomainMapping?: InitialDomainMapping;
 }
 /**
  * TaskAdapterInterface is an interface for TaskAdapter class.
@@ -130,6 +133,7 @@ export interface WorkerData<ConnectorState> {
     event: AirdropEvent;
     initialState: ConnectorState;
     workerPath: string;
+    initialDomainMapping?: InitialDomainMapping;
     options?: WorkerAdapterOptions;
 }
 /**

package/dist/workers/process-task.js

@@ -11,11 +11,13 @@ function processTask({ task, onTimeout, }) {
         void (async () => {
             const event = node_worker_threads_1.workerData.event;
             const initialState = node_worker_threads_1.workerData.initialState;
+            const initialDomainMapping = node_worker_threads_1.workerData.initialDomainMapping;
             const options = node_worker_threads_1.workerData.options;
             console = new logger_1.Logger({ event, options });
             const adapterState = await (0, state_1.createAdapterState)({
                 event,
                 initialState,
+                initialDomainMapping,
                 options,
             });
             if (node_worker_threads_1.parentPort && node_worker_threads_1.workerData.event) {

package/dist/workers/spawn.d.ts

@@ -10,7 +10,7 @@ import { SpawnFactoryInterface, SpawnInterface } from '../types/workers';
  * @param {string} workerPath - The path to the worker file
  * @returns {Promise<Spawn>} - A new instance of Spawn class
  */
-export declare function spawn<ConnectorState>({ event, initialState, workerPath, options, }: SpawnFactoryInterface<ConnectorState>): Promise<void>;
+export declare function spawn<ConnectorState>({ event, initialState, workerPath, initialDomainMapping, options, }: SpawnFactoryInterface<ConnectorState>): Promise<void>;
 export declare class Spawn {
     private event;
     private alreadyEmitted;

package/dist/workers/spawn.js

@@ -71,7 +71,7 @@ function getWorkerPath({ event, connectorWorkerPath, }) {
  * @param {string} workerPath - The path to the worker file
  * @returns {Promise<Spawn>} - A new instance of Spawn class
  */
-async function spawn({ event, initialState, workerPath, options, }) {
+async function spawn({ event, initialState, workerPath, initialDomainMapping, options, }) {
     const logger = new logger_1.Logger({ event, options });
     const script = getWorkerPath({
         event,
@@ -91,6 +91,7 @@ async function spawn({ event, initialState, workerPath, options, }) {
                 event,
                 initialState,
                 workerPath: script,
+                initialDomainMapping,
                 options,
             });
             return new Promise((resolve) => {

package/dist/workers/worker-adapter.d.ts

@@ -68,15 +68,37 @@ export declare class WorkerAdapter<ConnectorState> {
         item: ExternalSystemAttachment;
         create: ExternalSystemLoadingFunction<ExternalSystemAttachment>;
     }): Promise<LoadItemResponse>;
+    /**
+     * Transforms an array of attachments into array of batches of the specified size.
+     *
+     * @param {Object} parameters - The parameters object
+     * @param {NormalizedAttachment[]} parameters.attachments - Array of attachments to be processed
+     * @param {number} [parameters.batchSize=1] - The size of each batch (defaults to 1)
+     * @param {ConnectorState} parameters.adapter - The adapter instance
+     * @returns {NormalizedAttachment[][]} An array of attachment batches
+     */
+    private defaultAttachmentsReducer;
+    /**
+     * This iterator function processes attachments batch by batch, saves progress to state, and handles rate limiting.
+     *
+     * @param {Object} parameters - The parameters object
+     * @param {NormalizedAttachment[][]} parameters.reducedAttachments - Array of attachment batches to process
+     * @param {Object} parameters.adapter - The connector adapter that contains state and processing methods
+     * @param {Object} parameters.stream - Stream object for logging or progress reporting
+     * @returns {Promise<{delay?: number} | void>} Returns an object with delay information if rate-limited, otherwise void
+     * @throws Will not throw exceptions but will log warnings for processing failures
+     */
+    private defaultAttachmentsIterator;
     /**
      * Streams the attachments to the DevRev platform.
      * The attachments are streamed to the platform and the artifact information is returned.
      * @param {{ stream, processors }: { stream: ExternalSystemAttachmentStreamingFunction, processors?: ExternalSystemAttachmentProcessors  }} Params - The parameters to stream the attachments
-     * @returns {Promise<StreamAttachmentsReturnType>} - The response object containing the ssoAttachment artifact information
+     * @returns {Promise<StreamAttachmentsReturnType>} - The response object containing the ssorAttachment artifact information
      * or error information if there was an error
      */
-    streamAttachments<NewBatch>({ stream, processors, }: {
+    streamAttachments<NewBatch>({ stream, processors, batchSize, }: {
         stream: ExternalSystemAttachmentStreamingFunction;
         processors?: ExternalSystemAttachmentProcessors<ConnectorState, NormalizedAttachment[], NewBatch>;
+        batchSize?: number;
     }): Promise<StreamAttachmentsReturnType>;
 }