@devrev/ts-adaas 1.2.4 → 1.2.6

package/README.md

@@ -1,235 +1,124 @@
-# ADaaS Library
-
-## Release Notes
-
-### v1.2.4
-
-- Do not fail the extraction of attachments if streaming of single attachment fails.
-
-### v1.2.3
-
-- Add `local` flag to use for local development of the ADaaS snap-ins.
-- Send library version, snap-in version and snap-in slug in headers while emitting.
-- Make `actor_id` field optional for `SsorAttachment` interface.
-- Fix bugs related to event handling, error logging.
-
-### v1.2.2
-
-- Add library version as a part of control protocol.
-- Improve axios client and adapter logging.
-- Fix bugs related to state handling.
-
-### v1.2.1
-
-- Reduced the `delayFactor` to minimize unnecessary delays.
-- Correct the setting of the `lastSyncStarted` timestamp.
-- Improve logging for attachment extraction and loading.
-- Fix several bugs related to the control protocol.
-
-### v1.2.0
-
-- Add support for loading attachments from DevRev to external system.
-
-### v1.1.6
-
-- Add exponential retry and handle rate-limiting towards DevRev.
-- Gracefully handle failure to upload extracted attachments.
-
-### v1.1.5
-
-- Increase `delayFactor` and number of retries for the exponential backoff retry mechanism for HTTP requests.
-- Provide an inject function for streaming attachments.
-- Fix the attachments streaming bug.
-
-### v1.1.4
-
-- Provide log lines and stack traces for runtime worker errors.
-
-### v1.1.3
-
-- Export `axios` and `axiosClient` with the exponential backoff retry mechanism for HTTP requests and omit Authorization headers from Axios errors.
-- Resolve circular structure logging issues.
-- Fix the attachments metadata normalization bug.
-- Improve repository logging.
-
-### v1.1.2
-
-- Unify incoming and outgoing event context.
-- Add `dev_oid` to logger tags.
-
-### v1.1.1
-
-- Add default workers for loading deletion events.
-
-### v1.1.0
-
-- Support sync from DevRev to the external system. (Known limitations: no support for loading attachments.)
-
-### v1.0.4
-
-- Fix logging from worker threads.
-
-### v1.0.3
-
-- Add release notes.
-
-### v1.0.2
-
-- Fix bugs and improve local development.
-- Expose `formatAxiosError` function for error handling.
-
-### v1.0.1
-
-- Fix bugs and improve logging.
-
-### v1.0.0
-
-- Enable extractions to use the full lambda runtime and gracefully handle execution context timeout.
-- Simplify metadata and data normalization and uploading with the repo implementation.
-- Provide default handling of the attachment extraction phase in the ADaaS SDK library.
-- Reduce file size and streamline processes with gzip compression.
-- Fix bugs and improve error handling.
-
-### v0.0.3
-
-- Support new recipe management.
-
-### v0.0.2
-
-- Support the State API.
-- Provide an HTTP client for API requests.
-- Create local artifact files in the local development environment.
-- Improve logging.
-
-### v0.0.1
-
-- Implement a demo of the ADaaS snap-in.
-- Add an adapter for the ADaaS control protocol with helper functions.
-- Provide an uploader for uploading artifacts.
-
-# Overview
+# Airdrop SDK
 
 [![Coverage Status](https://coveralls.io/repos/github/devrev/adaas-sdk/badge.svg?branch=main&t=s4Otlm)](https://coveralls.io/github/devrev/adaas-sdk?branch=main)
 
-The ADaaS (Airdrop-as-a-Service) Library for TypeScript helps developers build Snap-ins that integrate with DevRev’s ADaaS platform. This library simplifies the workflow for handling data extraction and loading, event-driven actions, state management, and artifact handling.
+## Overview
+
+The Airdrop SDK for TypeScript helps developers build snap-ins that integrate with DevRev’s Airdrop platform. 
+This SDK simplifies the workflow for handling data extraction and loading, event-driven actions, state management, and artifact handling.
 
 It provides features such as:
 
-- Type Definitions: Structured types for ADaaS control protocol
+- Type Definitions: Structured types for Airdrop control protocol
 - Event Management: Easily emit events for different extraction or loading phases
 - State Handling: Update and access state in real-time within tasks
 - Artifact Management: Supports batched storage of artifacts
 - Error & Timeout Support: Error handling and timeout management for long-running tasks
 
-# Installation
+## Installation
 
 ```bash
 npm install @devrev/ts-adaas
 ```
 
-# Usage
+## Reference
 
-ADaaS Snap-ins can import data in both directions: from external sources to DevRev and from DevRev to external sources. Both directions are composed of several phases.
+### `spawn` function
 
-From external source to DevRev:
+This function initializes a new worker thread and oversees its lifecycle. 
+It should be invoked when the snap-in receives a message from the Airdrop platform.
+The worker script provided then handles the event accordingly.
 
-- External Sync Units Extraction
-- Metadata Extraction
-- Data Extraction
-- Attachments Extraction
+#### Usage
 
-From DevRev to external source:
+```typescript
+spawn({ event, initialState, workerPath, options })
+```
 
-- Data Loading
+#### Parameters
 
-Each phase comes with unique requirements for processing task, and both timeout and error handling.
+* _event_
+  
+  Required. An object of type __AirdropEvent__ that is received from the Airdrop platform.
 
-The ADaaS library exports processTask to structure the work within each phase, and onTimeout function to handle timeouts.
+* _initialState_
 
-### ADaaS Snap-in Invocation
+  Required. Object of __any__ type that represents the initial state of the snap-in.
 
-Each ADaaS snap-in must handle all the phases of ADaaS extraction. In a Snap-in, you typically define a `run` function that iterates over events and invokes workers per extraction phase.
+* _workerPath_
 
-```typescript
-import { AirdropEvent, EventType, spawn } from '@devrev/ts-adaas';
+  Required. A __string__ that represents the path to the worker file.
 
-interface DummyExtractorState {
-  issues: { completed: boolean };
-  users: { completed: boolean };
-  attachments: { completed: boolean };
-}
+* _options_
 
-const initialState: DummyExtractorState = {
-  issues: { completed: false },
-  users: { completed: false },
-  attachments: { completed: false },
-};
+  Optional. An object of type **WorkerAdapterOptions**, which will be passed to the newly created worker. This worker will then initialize a `WorkerAdapter` by invoking the `processTask` function. The options include:
+  
+  * `isLocalDevelopment`
+  
+    A __boolean__ flag. If set to `true`, intermediary files containing extracted data will be stored on the local machine, which is useful during development. The default value is `false`.
 
-function getWorkerPerExtractionPhase(event: AirdropEvent) {
-  let path;
-  switch (event.payload.event_type) {
-    case EventType.ExtractionExternalSyncUnitsStart:
-      path = __dirname + '/workers/external-sync-units-extraction';
-      break;
-    case EventType.ExtractionMetadataStart:
-      path = __dirname + '/workers/metadata-extraction';
-      break;
-    case EventType.ExtractionDataStart:
-    case EventType.ExtractionDataContinue:
-      path = __dirname + '/workers/data-extraction';
-      break;
-  }
-  return path;
-}
+  * `timeout`
+  
+    A __number__ that specifies the timeout duration for the lambda function, in milliseconds. The default is 10 minutes (10 * 60 * 1000 milliseconds), with a maximum allowable duration of 13 minutes (13 * 60 * 1000 milliseconds).
+  
+  * `batchSize`
+
+    A __number__ that determines the maximum number of items to be processed and saved to an intermediary file before being sent to the Airdrop platform. The default batch size is 2,000.
 
+#### Return value
+
+A __promise__ that resolves once the worker has completed processing.
+
+#### Example
+
+```typescript
 const run = async (events: AirdropEvent[]) => {
   for (const event of events) {
     const file = getWorkerPerExtractionPhase(event);
-    await spawn<DummyExtractorState>({
+    await spawn<ExtractorState>({
       event,
       initialState,
       workerPath: file,
-      options: {
-        isLocalDevelopment: true,
-      },
     });
   }
 };
-
-export default run;
 ```
 
-## Extraction
+### `processTask` function
 
-The ADaaS snap-in extraction lifecycle consists of three main phases: External Sync Units Extraction, Metadata Extraction, and Data Extraction. Each phase is defined in a separate file and is responsible for fetching the respective data.
+The `processTask` function retrieves the current state from the Airdrop platform and initializes a new `WorkerAdapter`.
+It executes the code specified in the `task` parameter, which contains the worker's functionality.
+If a timeout occurs, the function handles it by executing the `onTimeout` callback, ensuring the worker exits gracefully.
+Both functions receive an `adapter` parameter, representing the initialized `WorkerAdapter` object.
 
-The ADaaS library provides a repository management system to handle artifacts in batches. The `initializeRepos` function initializes the repositories, and the `push` function uploads the artifacts to the repositories. The `postState` function is used to post the state of the extraction task.
 
-State management is crucial for ADaaS Snap-ins to maintain the state of the extraction task. The `postState` function is used to post the state of the extraction task. The state is stored in the adapter and can be retrieved using the `adapter.state` property.
+#### Usage
+```typescript
+processTask({ task, onTimeout })
+```
 
-### 1. External Sync Units Extraction
+#### Parameters
 
-This phase is defined in `external-sync-units-extraction.ts` and is responsible for fetching the external sync units.
+* _task_
+  
+  Required. A __function__ that defines the logic associated with the given event type.
 
-```typescript
-import {
-  ExternalSyncUnit,
-  ExtractorEventType,
-  processTask,
-} from '@devrev/ts-adaas';
+* _onTimeout_
+  
+  Required. A __function__ managing the timeout of the lambda invocation, including saving any necessary progress at the time of timeout.
 
-const externalSyncUnits: ExternalSyncUnit[] = [
-  {
-    id: 'devrev',
-    name: 'devrev',
-    description: 'Demo external sync unit',
-    item_count: 2,
-    item_type: 'issues',
-  },
-];
+#### Example
 
+````typescript
+// External sync units extraction
 processTask({
   task: async ({ adapter }) => {
+    const httpClient = new HttpClient(adapter.event);
+
+    const todoLists = await httpClient.getTodoLists();
+
+    const externalSyncUnits: ExternalSyncUnit[] = todoLists.map((todoList) => normalizeTodoList(todoList));
+
     await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsDone, {
       external_sync_units: externalSyncUnits,
     });
@@ -242,170 +131,167 @@ processTask({
     });
   },
 });
-```
+````
+
+### `WorkerAdapter` class
 
-### 2. Metadata Extraction
+Used to interact with Airdrop platform. 
+Provides utilities to emit events to the Airdrop platform, update the state of the snap-in and upload artifacts (files with data) to the platform.
 
-This phase is defined in `metadata-extraction.ts` and is responsible for fetching the metadata.
+### Usage
 
 ```typescript
-import { ExtractorEventType, processTask } from '@devrev/ts-adaas';
-import externalDomainMetadata from '../dummy-extractor/external_domain_metadata.json';
+new WorkerAdapter({
+  event,
+  adapterState,
+  options,
+});
+```
 
-const repos = [{ itemType: 'external_domain_metadata' }];
+#### Parameters
 
-processTask({
-  task: async ({ adapter }) => {
-    adapter.initializeRepos(repos);
-    await adapter
-      .getRepo('external_domain_metadata')
-      ?.push([externalDomainMetadata]);
-    await adapter.emit(ExtractorEventType.ExtractionMetadataDone);
-  },
-  onTimeout: async ({ adapter }) => {
-    await adapter.emit(ExtractorEventType.ExtractionMetadataError, {
-      error: { message: 'Failed to extract metadata. Lambda timeout.' },
-    });
-  },
+* _event_
+  
+  Required. An object of type __AirdropEvent__ that is received from the Airdrop platform.
+
+* _adapterState_
+  
+  Required. An object of type __State__, which represents the initial state of the adapter.
+
+* _options_
+  
+  Optional. An object of type __WorkerAdapterOptions__ that specifies additional configuration options for the `WorkerAdapter`. This object is passed via the `spawn` function.
+
+#### Example
+
+```typescript
+const adapter = new WorkerAdapter<ConnectorState>({
+  event,
+  adapterState,
+  options,
 });
 ```
 
-### 3. Data Extraction
+### `WorkerAdapter.state` property
 
-This phase is defined in `data-extraction.ts` and is responsible for fetching the data. In this phase also attachments metadata is extracted.
+Getter and setter methods for working with the adapter state.
+
+### Usage
 
 ```typescript
-import { EventType, ExtractorEventType, processTask } from '@devrev/ts-adaas';
-import { normalizeAttachment, normalizeIssue, normalizeUser } from '../dummy-extractor/data-normalization';
+// get state
+const adapterState = adapter.state;
 
-const issues = [
-  { id: 'issue-1', created_date: '1999-12-25T01:00:03+01:00', ... },
-  { id: 'issue-2', created_date: '1999-12-27T15:31:34+01:00', ... },
-];
+// set state
+adapter.state = newAdapterState;
+```
 
-const users = [
-  { id: 'user-1', created_date: '1999-12-25T01:00:03+01:00', ... },
-  { id: 'user-2', created_date: '1999-12-27T15:31:34+01:00', ... },
-];
+#### Example
 
-const attachments = [
-  { url: 'https://app.dev.devrev-eng.ai/favicon.ico', id: 'attachment-1', ... },
-  { url: 'https://app.dev.devrev-eng.ai/favicon.ico', id: 'attachment-2', ... },
-];
+```typescript
+export const initialState: ExtractorState = {
+  users: { completed: false },
+  tasks: { completed: false },
+  attachments: { completed: false },
+};
+
+adapter.state = initialState;
+```
+
+### `WorkerAdapter.initializeRepos` method
+
+Initializes a `Repo` object for each item provided.
+
+### Usage
+
+```typescript
+adapter.initializeRepos(repos);
+```
+
+#### Parameters
+
+* _repos_
+  
+  Required. An array of objects of type `RepoInterface`. 
+
+#### Example
+
+This should typically be called within the function passed as a parameter to the `processTask` function in the data extraction phase.
 
+```typescript
 const repos = [
-  { itemType: 'issues', normalize: normalizeIssue },
-  { itemType: 'users', normalize: normalizeUser },
-  { itemType: 'attachments', normalize: normalizeAttachment },
+  {
+    itemType: 'tasks',
+    normalize: normalizeTask,
+  }
 ];
 
-processTask({
-  task: async ({ adapter }) => {
-    adapter.initializeRepos(repos);
-
-    if (adapter.event.payload.event_type === EventType.ExtractionDataStart) {
-      await adapter.getRepo('issues')?.push(issues);
-      await adapter.emit(ExtractorEventType.ExtractionDataProgress, { progress: 50 });
-    } else {
-      await adapter.getRepo('users')?.push(users);
-      await adapter.getRepo('attachments')?.push(attachments);
-      await adapter.emit(ExtractorEventType.ExtractionDataDone, { progress: 100 });
-    }
-  },
-  onTimeout: async ({ adapter }) => {
-    await adapter.postState();
-    await adapter.emit(ExtractorEventType.ExtractionDataProgress, { progress: 50 });
-  },
-});
+adapter.initializeRepos(repos);
 ```
 
-### 4. Attachments Streaming
+### `WorkerAdapter.getRepo` method
 
-The ADaaS library handles attachments streaming to improve efficiency and reduce complexity for developers. During the extraction phase, developers need only to provide metadata in a specific format for each attachment, and the library manages the streaming process.
+Finds a Repo from the initialized repos.
 
-The Snap-in should provide attachment metadata following the `NormalizedAttachment` interface:
+### Usage
 
 ```typescript
-export interface NormalizedAttachment {
-  url: string;
-  id: string;
-  file_name: string;
-  author_id: string;
-  parent_id: string;
-}
+adapter.getRepo(itemType);
 ```
 
-## Loading phases
+#### Parameters
 
-### 1. Loading Data
+* _itemType_
+  
+  Required. A __string__ that represents the itemType property for the searched repo.
 
-This phase is defined in `load-data.ts` and is responsible for loading the data to the external system.
+#### Return value
 
-Loading is done by providing an ordered list of itemTypes to load and their respective create and update functions.
+An object of type __Repo__ if the repo is found, otherwise __undefined__.
+
+#### Example
+
+This should typically be called within the function passed as a parameter to the `processTask` function.
 
 ```typescript
-  processTask({
-    task: async ({ adapter }) => {
-      const { reports, processed_files } = await adapter.loadItemTypes({
-        itemTypesToLoad: [
-          {
-            itemType: 'tickets',
-            create: createTicket,
-            update: updateTicket,
-          },
-          {
-            itemType: 'conversations',
-            create: createConversation,
-            update: updateConversation,
-          },
-        ],
-      });
-
-    await adapter.emit(LoaderEventType.DataLoadingDone, {
-      reports,
-      processed_files,
-    });
-  },
-  onTimeout: async ({ adapter }) => {
-    await adapter.emit(LoaderEventType.DataLoadingProgress, {
-      reports: adapter.reports,
-      processed_files: adapter.processedFiles,
-    });
-});
+// Push users to the repository designated for 'users' data.
+await adapter.getRepo('users')?.push(users);
+```
+
+### `WorkerAdapter.emit` method
+
+Emits an event to the Airdrop platform.
+
+### Usage
+
+```typescript
+adapter.emit( newEventType, data ):
 ```
 
-The loading functions `create` and `update` provide loading to the external system. They provide denormalization of the records to the schema of the external system and provide HTTP calls to the external system. Both loading functions must handle rate limiting for the external system and handle errors.
+#### Parameters
 
-Functions return an ID and modified date of the record in the external system, or specify rate-liming offset or errors, if the record could not be created or updated.
+* _newEventType_
+  
+  Required. The event type to be emitted, of type __ExtractorEventType__ or __LoaderEventType__.
 
-### 2. Loading Attachments
+* _data_
+  
+  Optional. An object of type __EventData__ which represents the data to be sent with the event.
 
-This phase is defined in `load-attachments.ts` and is responsible for loading the attachments to the external system.
+#### Return value
 
-Loading is done by providing the create function to create attachments in the external system.
+A __promise__, which resolves to undefined after the emit function completes its execution or rejects with an error.
+
+#### Example
+
+This should typically be called within the function passed as a parameter to the `processTask` function.
 
 ```typescript
-processTask({
-  task: async ({ adapter }) => {
-    const { reports, processed_files } = await adapter.loadAttachments({
-      create,
-    });
+// Emitting successfully finished data extraction.
+await adapter.emit(ExtractorEventType.ExtractionDataDone);
 
-    await adapter.emit(LoaderEventType.AttachmentLoadingDone, {
-      reports,
-      processed_files,
-    });
-  },
-  onTimeout: async ({ adapter }) => {
-    await adapter.postState();
-    await adapter.emit(LoaderEventType.AttachmentLoadingProgress, {
-      reports: adapter.reports,
-      processed_files: adapter.processedFiles,
-    });
-  },
+// Emitting a delay in attachments extraction phase.
+await adapter.emit(ExtractorEventType.ExtractionAttachmentsDelay, {
+  delay: 10,
 });
 ```
-
-The loading function `create` provides loading to the external system, to make API calls to the external system to create the attachments and handle errors and external system's rate limiting.
-
-Functions return an ID and modified date of the record in the external system, specify rate-liming back-off, or log errors, if the attachment could not be created.

package/dist/deprecated/uploader/index.js

@@ -34,7 +34,7 @@ var __importStar = (this && this.__importStar) || (function () {
 })();
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.Uploader = void 0;
-const axios_devrev_client_1 = require("../../http/axios-devrev-client");
+const axios_client_1 = require("../../http/axios-client");
 const typescript_sdk_1 = require("@devrev/typescript-sdk");
 const fs_1 = __importStar(require("fs"));
 const helpers_1 = require("../common/helpers");
@@ -118,9 +118,9 @@ class Uploader {
     ) {
         const formData = (0, helpers_1.createFormData)(preparedArtifact, fetchedObjects);
         try {
-            const response = await axios_devrev_client_1.axiosDevRevClient.post(preparedArtifact.url, formData, {
+            const response = await axios_client_1.axiosClient.post(preparedArtifact.url, formData, {
                 headers: {
-                    'Content-Type': 'multipart/form',
+                    'Content-Type': 'multipart/form-data',
                 },
             });
             return response;

package/dist/repo/repo.d.ts

@@ -7,6 +7,7 @@ export declare class Repo {
     private normalize?;
     private uploader;
     private onUpload;
+    private options?;
     constructor({ event, itemType, normalize, onUpload, options, }: RepoFactoryInterface);
     getItems(): (NormalizedItem | NormalizedAttachment | Item)[];
     upload(batch?: (NormalizedItem | NormalizedAttachment | Item)[]): Promise<void | ErrorRecord>;

package/dist/repo/repo.js

@@ -10,6 +10,7 @@ class Repo {
         this.normalize = normalize;
         this.onUpload = onUpload;
         this.uploader = new uploader_1.Uploader({ event, options });
+        this.options = options;
     }
     getItems() {
         return this.items;
@@ -35,6 +36,7 @@ class Repo {
         }
     }
     async push(items) {
+        var _a;
         let recordsToPush;
         if (!items || items.length === 0) {
             console.log(`No items to push for type ${this.itemType}. Skipping push.`);
@@ -52,9 +54,10 @@ class Repo {
         // Add the new records to the items array
         this.items.push(...recordsToPush);
         // Upload in batches while the number of items exceeds the batch size
-        while (this.items.length >= constants_1.ARTIFACT_BATCH_SIZE) {
-            // Slice out a batch of ARTIFACT_BATCH_SIZE items to upload
-            const batch = this.items.splice(0, constants_1.ARTIFACT_BATCH_SIZE);
+        const batchSize = ((_a = this.options) === null || _a === void 0 ? void 0 : _a.batchSize) || constants_1.ARTIFACT_BATCH_SIZE;
+        while (this.items.length >= batchSize) {
+            // Slice out a batch of batchSize items to upload
+            const batch = this.items.splice(0, batchSize);
             try {
                 // Upload the batch
                 await this.upload(batch);

package/dist/repo/repo.test.js

@@ -15,7 +15,6 @@ describe('Repo class push method', () => {
             itemType: 'test_item_type',
             normalize,
             onUpload: jest.fn(),
-            options: {},
         });
     });
     afterEach(() => {
@@ -83,4 +82,36 @@ describe('Repo class push method', () => {
         expect(uploadSpy).toHaveBeenCalledTimes(2); // Check that upload was called twice
         uploadSpy.mockRestore();
     });
+    describe('should take batch size into account', () => {
+        beforeEach(() => {
+            repo = new repo_1.Repo({
+                event: (0, test_helpers_1.createEvent)({ eventType: types_1.EventType.ExtractionDataStart }),
+                itemType: 'test_item_type',
+                normalize,
+                onUpload: jest.fn(),
+                options: {
+                    batchSize: 50,
+                },
+            });
+        });
+        it('should empty the items array after pushing 50 items with batch size of 50', async () => {
+            const items = (0, test_helpers_1.createItems)(50);
+            await repo.push(items);
+            expect(repo.getItems()).toEqual([]);
+        });
+        it('should leave 5 items in the items array after pushing 205 items with batch size of 50', async () => {
+            const items = (0, test_helpers_1.createItems)(205);
+            await repo.push(items);
+            expect(repo.getItems().length).toBe(5);
+        });
+        it('should upload 4 batches of 50 and leave 5 items in the items array after pushing 205 items with batch size of 50', async () => {
+            const uploadSpy = jest.spyOn(repo, 'upload');
+            const items = (0, test_helpers_1.createItems)(205);
+            await repo.push(items);
+            expect(normalize).toHaveBeenCalledTimes(205);
+            expect(repo.getItems().length).toBe(5);
+            expect(uploadSpy).toHaveBeenCalledTimes(4);
+            uploadSpy.mockRestore();
+        });
+    });
 });

package/dist/types/workers.d.ts

@@ -22,10 +22,12 @@ export interface WorkerAdapterInterface<ConnectorState> {
  * @constructor
  * @param {boolean=} isLocalDevelopment - A flag to indicate if the adapter is being used in local development
  * @param {number=} timeout - The timeout for the worker thread
+ * @param {number=} batchSize - Maximum number of extracted items in a batch
  */
 export interface WorkerAdapterOptions {
     isLocalDevelopment?: boolean;
     timeout?: number;
+    batchSize?: number;
 }
 /**
  * SpawnInterface is an interface for Spawn class.

package/dist/uploader/uploader.d.ts

@@ -1,11 +1,11 @@
-import { betaSDK } from '@devrev/typescript-sdk';
 import { NormalizedAttachment } from '../repo/repo.interfaces';
-import { UploadResponse, UploaderFactoryInterface } from './uploader.interfaces';
+import { ArtifactsPrepareResponse, UploadResponse, UploaderFactoryInterface } from './uploader.interfaces';
 import { AxiosResponse } from 'axios';
 export declare class Uploader {
     private event;
-    private betaDevrevSdk;
     private isLocalDevelopment?;
+    private devrevApiEndpoint;
+    private devrevApiToken;
     constructor({ event, options }: UploaderFactoryInterface);
     /**
      * Uploads the fetched objects to the DevRev platform.
@@ -17,9 +17,9 @@ export declare class Uploader {
      * or error information if there was an error
      */
     upload(itemType: string, fetchedObjects: object[] | object): Promise<UploadResponse>;
-    prepareArtifact(filename: string, fileType: string): Promise<betaSDK.ArtifactsPrepareResponse | void>;
+    prepareArtifact(filename: string, fileType: string): Promise<ArtifactsPrepareResponse | void>;
     private uploadToArtifact;
-    streamToArtifact(preparedArtifact: betaSDK.ArtifactsPrepareResponse, fileStreamResponse: any): Promise<AxiosResponse | void>;
+    streamToArtifact(preparedArtifact: ArtifactsPrepareResponse, fileStreamResponse: any): Promise<AxiosResponse | void>;
     getAttachmentsFromArtifactId({ artifact, }: {
         artifact: string;
     }): Promise<{

package/dist/uploader/uploader.interfaces.d.ts

@@ -16,7 +16,6 @@ export interface Artifact {
 }
 /**
  * ArtifactsPrepareResponse is an interface that defines the structure of the response from the prepare artifacts endpoint.
- * @deprecated
  */
 export interface ArtifactsPrepareResponse {
     url: string;

package/dist/uploader/uploader.js

@@ -42,16 +42,13 @@ const axios_client_1 = require("../http/axios-client");
 const zlib_1 = __importDefault(require("zlib"));
 const js_jsonl_1 = require("js-jsonl");
 const form_data_1 = __importDefault(require("form-data"));
-const typescript_sdk_1 = require("@devrev/typescript-sdk");
 const constants_1 = require("../common/constants");
 const logger_1 = require("../logger/logger");
 class Uploader {
     constructor({ event, options }) {
         this.event = event;
-        this.betaDevrevSdk = typescript_sdk_1.client.setupBeta({
-            endpoint: event.execution_metadata.devrev_endpoint,
-            token: event.context.secrets.service_account_token,
-        });
+        this.devrevApiEndpoint = event.execution_metadata.devrev_endpoint;
+        this.devrevApiToken = event.context.secrets.service_account_token;
         this.isLocalDevelopment = options === null || options === void 0 ? void 0 : options.isLocalDevelopment;
     }
     /**
@@ -101,9 +98,13 @@ class Uploader {
     }
     async prepareArtifact(filename, fileType) {
         try {
-            const response = await this.betaDevrevSdk.artifactsPrepare({
+            const response = await axios_client_1.axiosClient.post(`${this.devrevApiEndpoint}/artifacts.prepare`, {
                 file_name: filename,
                 file_type: fileType,
+            }, {
+                headers: {
+                    Authorization: `Bearer ${this.devrevApiToken}`,
+                },
             });
             return response.data;
         }
@@ -116,9 +117,7 @@ class Uploader {
             }
         }
     }
-    async uploadToArtifact(preparedArtifact, file
-    // eslint-disable-next-line @typescript-eslint/no-explicit-any
-    ) {
+    async uploadToArtifact(preparedArtifact, file) {
         const formData = new form_data_1.default();
         for (const field of preparedArtifact.form_data) {
             formData.append(field.key, field.value);
@@ -139,22 +138,23 @@ class Uploader {
             }
         }
     }
-    async streamToArtifact(preparedArtifact, 
-    // eslint-disable-next-line @typescript-eslint/no-explicit-any
-    fileStreamResponse) {
+    async streamToArtifact(preparedArtifact, fileStreamResponse) {
         const formData = new form_data_1.default();
         for (const field of preparedArtifact.form_data) {
             formData.append(field.key, field.value);
         }
         formData.append('file', fileStreamResponse.data);
         if (fileStreamResponse.headers['content-length'] > constants_1.MAX_DEVREV_ARTIFACT_SIZE) {
+            console.warn(`File size exceeds the maximum limit of ${constants_1.MAX_DEVREV_ARTIFACT_SIZE} bytes.`);
             return;
         }
         try {
             const response = await axios_client_1.axiosClient.post(preparedArtifact.url, formData, {
-                headers: Object.assign(Object.assign({}, formData.getHeaders()), (!fileStreamResponse.headers['content-length'] && {
-                    'Content-Length': constants_1.MAX_DEVREV_ARTIFACT_SIZE,
-                })),
+                headers: Object.assign(Object.assign({}, formData.getHeaders()), (!fileStreamResponse.headers['content-length']
+                    ? {
+                        'Content-Length': constants_1.MAX_DEVREV_ARTIFACT_SIZE,
+                    }
+                    : {})),
             });
             return response;
         }
@@ -169,28 +169,28 @@ class Uploader {
         }
     }
     async getAttachmentsFromArtifactId({ artifact, }) {
-        // 1. Get the URL of the attachments metadata artifact
+        // Get the URL of the attachments metadata artifact
         const artifactUrl = await this.getArtifactDownloadUrl(artifact);
         if (!artifactUrl) {
             return {
                 error: { message: 'Error while getting artifact download URL.' },
             };
         }
-        // 2. Download artifact from the URL
+        // Download artifact from the URL
         const gzippedJsonlObject = await this.downloadArtifact(artifactUrl);
         if (!gzippedJsonlObject) {
             return {
                 error: { message: 'Error while downloading gzipped jsonl object.' },
             };
         }
-        // 3. Decompress the gzipped jsonl object
+        // Decompress the gzipped jsonl object
         const jsonlObject = this.decompressGzip(gzippedJsonlObject);
         if (!jsonlObject) {
             return {
                 error: { message: 'Error while decompressing gzipped jsonl object.' },
             };
         }
-        // 4. Parse the jsonl object to get the attachment metadata
+        // Parse the jsonl object to get the attachment metadata
         const jsonObject = this.parseJsonl(jsonlObject);
         if (!jsonObject) {
             return {
@@ -201,8 +201,12 @@ class Uploader {
     }
     async getArtifactDownloadUrl(artifactId) {
         try {
-            const response = await this.betaDevrevSdk.artifactsLocate({
+            const response = await axios_client_1.axiosClient.post(`${this.devrevApiEndpoint}/artifacts.locate`, {
                 id: artifactId,
+            }, {
+                headers: {
+                    Authorization: `Bearer ${this.devrevApiToken}`,
+                },
             });
             return response.data.url;
         }

package/dist/workers/default-workers/attachments-extraction.js

@@ -7,6 +7,9 @@ const getAttachmentStream = async ({ item, }) => {
     try {
         const fileStreamResponse = await axios_client_1.axiosClient.get(url, {
             responseType: 'stream',
+            headers: {
+                'Accept-Encoding': 'identity',
+            },
         });
         return { httpStream: fileStreamResponse };
     }
@@ -51,7 +54,6 @@ const getAttachmentStream = async ({ item, }) => {
         }
     },
     onTimeout: async ({ adapter }) => {
-        await adapter.postState();
         await adapter.emit(index_1.ExtractorEventType.ExtractionAttachmentsProgress, {
             progress: 50,
         });

package/dist/workers/default-workers/data-extraction.js

@@ -93,7 +93,6 @@ const repos = [
         }
     },
     onTimeout: async ({ adapter }) => {
-        await adapter.postState();
         await adapter.emit(index_1.ExtractorEventType.ExtractionDataProgress, {
             progress: 50,
         });

package/dist/workers/default-workers/load-attachments.js

@@ -11,7 +11,6 @@ const types_1 = require("../../types");
         });
     },
     onTimeout: async ({ adapter }) => {
-        await adapter.postState();
         await adapter.emit(types_1.LoaderEventType.AttachmentLoadingError, {
             reports: adapter.reports,
             processed_files: adapter.processedFiles,

package/dist/workers/default-workers/load-data.js

@@ -10,7 +10,6 @@ const loading_1 = require("../../types/loading");
         });
     },
     onTimeout: async ({ adapter }) => {
-        await adapter.postState();
         await adapter.emit(loading_1.LoaderEventType.DataLoadingError, {
             reports: adapter.reports,
             processed_files: adapter.processedFiles,

package/dist/workers/worker-adapter.js

@@ -2,7 +2,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.WorkerAdapter = void 0;
 exports.createWorkerAdapter = createWorkerAdapter;
-const axios_devrev_client_1 = require("../http/axios-devrev-client");
+const axios_client_1 = require("../http/axios-client");
 const extraction_1 = require("../types/extraction");
 const loading_1 = require("../types/loading");
 const constants_1 = require("../common/constants");
@@ -153,7 +153,7 @@ class WorkerAdapter {
             node_worker_threads_1.parentPort === null || node_worker_threads_1.parentPort === void 0 ? void 0 : node_worker_threads_1.parentPort.postMessage(message);
         }
         catch (error) {
-            if (axios_devrev_client_1.axios.isAxiosError(error)) {
+            if (axios_client_1.axios.isAxiosError(error)) {
                 console.error(`Error while emitting event with event type: ${newEventType}`, (0, logger_1.serializeAxiosError)(error));
             }
             else {
@@ -374,7 +374,7 @@ class WorkerAdapter {
                         });
                     }
                     catch (error) {
-                        if (axios_devrev_client_1.axios.isAxiosError(error)) {
+                        if (axios_client_1.axios.isAxiosError(error)) {
                             console.warn('Failed to update sync mapper record', (0, logger_1.serializeAxiosError)(error));
                             return {
                                 error: {
@@ -419,7 +419,7 @@ class WorkerAdapter {
             // TODO: Update mapper (optional)
         }
         catch (error) {
-            if (axios_devrev_client_1.axios.isAxiosError(error)) {
+            if (axios_client_1.axios.isAxiosError(error)) {
                 if (((_a = error.response) === null || _a === void 0 ? void 0 : _a.status) === 404) {
                     // Create item in external system if mapper record not found
                     const { id, delay, error } = await itemTypeToLoad.create({
@@ -444,7 +444,7 @@ class WorkerAdapter {
                             };
                         }
                         catch (error) {
-                            if (axios_devrev_client_1.axios.isAxiosError(error)) {
+                            if (axios_client_1.axios.isAxiosError(error)) {
                                 console.warn('Failed to create sync mapper record', (0, logger_1.serializeAxiosError)(error));
                                 return {
                                     error: {
@@ -511,14 +511,12 @@ class WorkerAdapter {
             const fileType = ((_a = httpStream.headers) === null || _a === void 0 ? void 0 : _a['content-type']) || 'application/octet-stream';
             const preparedArtifact = await this.uploader.prepareArtifact(attachment.file_name, fileType);
             if (!preparedArtifact) {
-                console.warn('Error while preparing artifact for attachment ID ' +
-                    attachment.id +
-                    '. Skipping attachment');
+                console.warn(`Error while preparing artifact for attachment ID ${attachment.id}. Skipping attachment.`);
                 return;
             }
             const uploadedArtifact = await this.uploader.streamToArtifact(preparedArtifact, httpStream);
             if (!uploadedArtifact) {
-                console.warn('Error while preparing artifact for attachment ID ' + attachment.id);
+                console.warn(`Error while streaming to artifact for attachment ID ${attachment.id}. Skipping attachment.`);
                 return;
             }
             const ssorAttachment = {
@@ -579,17 +577,24 @@ class WorkerAdapter {
      * or error information if there was an error
      */
     async streamAttachments({ stream, processors, }) {
-        var _a, _b, _c;
+        var _a, _b, _c, _d;
         const repos = [
             {
                 itemType: 'ssor_attachment',
             },
         ];
         this.initializeRepos(repos);
-        const attachmentsState = (((_a = this.state.toDevRev) === null || _a === void 0 ? void 0 : _a.attachmentsMetadata.artifactIds) || []).slice();
-        console.log('Attachments metadata artifact IDs', attachmentsState);
-        for (const attachmentsMetadataArtifactId of attachmentsState) {
-            console.log(`Started processing attachments for artifact ID: ${attachmentsMetadataArtifactId}.`);
+        const attachmentsMetadataArtifactIds = (_b = (_a = this.state.toDevRev) === null || _a === void 0 ? void 0 : _a.attachmentsMetadata) === null || _b === void 0 ? void 0 : _b.artifactIds;
+        if (!attachmentsMetadataArtifactIds ||
+            attachmentsMetadataArtifactIds.length === 0) {
+            console.log(`No attachments metadata artifact IDs found in state.`);
+            return;
+        }
+        else {
+            console.log(`Found ${attachmentsMetadataArtifactIds.length} attachments metadata artifact IDs in state.`);
+        }
+        for (const attachmentsMetadataArtifactId of attachmentsMetadataArtifactIds) {
+            console.log(`Started processing attachments for attachments metadata artifact ID: ${attachmentsMetadataArtifactId}.`);
             const { attachments, error } = await this.uploader.getAttachmentsFromArtifactId({
                 artifact: attachmentsMetadataArtifactId,
             });
@@ -601,6 +606,7 @@ class WorkerAdapter {
                 console.warn(`No attachments found for artifact ID: ${attachmentsMetadataArtifactId}.`);
                 continue;
             }
+            console.log(`Found ${attachments.length} attachments for artifact ID: ${attachmentsMetadataArtifactId}.`);
             if (processors) {
                 console.log(`Using custom processors for attachments.`);
                 const { reducer, iterator } = processors;
@@ -616,7 +622,7 @@ class WorkerAdapter {
             }
             else {
                 console.log(`Using default processors for attachments.`);
-                const attachmentsToProcess = attachments.slice((_c = (_b = this.state.toDevRev) === null || _b === void 0 ? void 0 : _b.attachmentsMetadata) === null || _c === void 0 ? void 0 : _c.lastProcessed, attachments.length);
+                const attachmentsToProcess = attachments.slice((_d = (_c = this.state.toDevRev) === null || _c === void 0 ? void 0 : _c.attachmentsMetadata) === null || _d === void 0 ? void 0 : _d.lastProcessed, attachments.length);
                 for (const attachment of attachmentsToProcess) {
                     const response = await this.processAttachment(attachment, stream);
                     if (response === null || response === void 0 ? void 0 : response.delay) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@devrev/ts-adaas",
-  "version": "1.2.4",
+  "version": "1.2.6",
   "description": "DevRev ADaaS (AirDrop-as-a-Service) Typescript SDK.",
   "type": "commonjs",
   "main": "./dist/index.js",

package/dist/http/axios-devrev-client.d.ts

@@ -1,3 +0,0 @@
-import axios from 'axios';
-declare const axiosDevRevClient: import("axios").AxiosInstance;
-export { axios, axiosDevRevClient };

package/dist/http/axios-devrev-client.js

@@ -1,37 +0,0 @@
-"use strict";
-var __importDefault = (this && this.__importDefault) || function (mod) {
-    return (mod && mod.__esModule) ? mod : { "default": mod };
-};
-Object.defineProperty(exports, "__esModule", { value: true });
-exports.axiosDevRevClient = exports.axios = void 0;
-const axios_1 = __importDefault(require("axios"));
-exports.axios = axios_1.default;
-const axios_retry_1 = __importDefault(require("axios-retry"));
-const axiosDevRevClient = axios_1.default.create();
-exports.axiosDevRevClient = axiosDevRevClient;
-(0, axios_retry_1.default)(axiosDevRevClient, {
-    retries: 5,
-    retryDelay: (retryCount, error) => {
-        var _a, _b;
-        console.warn('Retry attempt: ' + retryCount + ' to url: ' + ((_a = error.config) === null || _a === void 0 ? void 0 : _a.url) + '.');
-        if (error.response) {
-            const retry_after = (_b = error.response) === null || _b === void 0 ? void 0 : _b.headers['retry-after'];
-            if (retry_after) {
-                return retry_after;
-            }
-        }
-        // Exponential backoff algorithm: 1 * 2 ^ retryCount * 1000ms
-        return axios_retry_1.default.exponentialDelay(retryCount, error, 1000);
-    },
-    retryCondition: (error) => {
-        var _a;
-        return (axios_retry_1.default.isNetworkOrIdempotentRequestError(error) ||
-            ((_a = error.response) === null || _a === void 0 ? void 0 : _a.status) === 429);
-    },
-    onMaxRetryTimesExceeded(error, retryCount) {
-        var _a;
-        console.log(`Max retries attempted: ${retryCount}`);
-        (_a = error.config) === null || _a === void 0 ? true : delete _a.headers.Authorization;
-        delete error.request._header;
-    },
-});