node-es-transformer 1.0.0-alpha6 → 1.0.0-alpha8

package/README.md

@@ -12,7 +12,6 @@ If you're looking for a nodejs based tool which allows you to ingest large CSV/J
 
 While I'd generally recommend using [Logstash](https://www.elastic.co/products/logstash), [filebeat](https://www.elastic.co/products/beats/filebeat) or [Ingest Nodes](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html) for established use cases, this tool may be of help especially if you feel more at home in the JavaScript/nodejs universe and have use cases with customized ingestion and data transformation needs.
 
-
 **This is experimental code, use at your own risk. Nonetheless, I encourage you to give it a try so I can gather some feedback.**
 
 ### So why is this still _alpha_?
@@ -21,13 +20,13 @@ While I'd generally recommend using [Logstash](https://www.elastic.co/products/l
 - The code needs some more safety measures to avoid some possible accidental data loss scenarios.
 - No test coverage yet.
 
-----
+---
 
 Now that we've talked about the caveats, let's have a look what you actually get with this tool:
 
 ## Features
 
-- Buffering/Streaming for both reading and indexing. Files are read using streaming and Elasticsearch ingestion is done using buffered bulk indexing. This is tailored towards ingestion of large files. Successfully tested so far with JSON and CSV files in the range of 20-30 GBytes. On a single machine running both `node-es-transformer` and Elasticsearch ingestion rates up to 20k documents/second were achieved (2,9 GHz Intel Core i7, 16GByte RAM, SSD). 
+- Buffering/Streaming for both reading and indexing. Files are read using streaming and Elasticsearch ingestion is done using buffered bulk indexing. This is tailored towards ingestion of large files. Successfully tested so far with JSON and CSV files in the range of 20-30 GBytes. On a single machine running both `node-es-transformer` and Elasticsearch ingestion rates up to 20k documents/second were achieved (2,9 GHz Intel Core i7, 16GByte RAM, SSD).
 - Supports wildcards to ingest/transform a range of files in one go.
 - Supports fetching documents from existing indices using search/scroll. This allows you to reindex with custom data transformations just using JavaScript in the `transform` callback.
 - The `transform` callback gives you each source document, but you can split it up in multiple ones and return an array of documents. An example use case for this: Each source document is a Tweet and you want to transform that into an entity centric index based on Hashtags.
@@ -46,9 +45,8 @@ const transformer = require('node-es-transformer');
 transformer({
   fileName: 'filename.json',
   targetIndexName: 'my-index',
-  typeName: 'doc',
   mappings: {
-    doc: {
+    _doc: {
       properties: {
         '@timestamp': {
           type: 'date'
@@ -82,9 +80,9 @@ const transformer = require('node-es-transformer');
 transformer({
   sourceIndexName: 'my-source-index',
   targetIndexName: 'my-target-index',
-  typeName: 'doc',
+  // optional, if you skip mappings, they will be fetched from the source index.
   mappings: {
-    doc: {
+    _doc: {
       properties: {
         '@timestamp': {
           type: 'date'
@@ -112,15 +110,18 @@ transformer({
 
 ### Options
 
-- `deleteIndex`:  Setting to automatically delete an existing index, default is `false`.
-- `host`: Elasticsearch host, defaults to `localhost`.
-- `port`: Elasticsearch port, defaults to `9200`.
+- `deleteIndex`: Setting to automatically delete an existing index, default is `false`.
+- `protocol`/`targetProtocol`: Elasticsearch protocol, defaults to `http`.
+- `host`/`targetHost`: Elasticsearch host, defaults to `localhost`.
+- `port`/`targetPort`: Elasticsearch port, defaults to `9200`.
+- `auth`/`targetAuth`: Optional Elasticsearch authorization object, for example `{ username: 'elastic', password: 'changeme'}`.
+- `rejectUnauthorized`: Elasticsearch TLS option, defaults to `true`.
+- `ca`: Optional path to certificate used for TLS configuraiton.
 - `bufferSize`: The amount of documents inserted with each Elasticsearch bulk insert request, default is `1000`.
 - `fileName`: Source filename to ingest, supports wildcards. If this is set, `sourceIndexName` is not allowed.
 - `splitRegex`: Custom line split regex, defaults to `/\n/`.
 - `sourceIndexName`: The source Elasticsearch to reindex from. If this is set, `fileName` is not allowed.
 - `targetIndexName`: The target Elasticsearch index where documents will be indexed.
-- `typeName`: Elasticsearch document type name.
 - `mappings`: Elasticsearch document mapping.
 - `skipHeader`: If true, skips the first line of the source file. Defaults to `false`.
 - `transform(line)`: A callback function which allows the transformation of a source line into one or several documents.
@@ -138,10 +139,10 @@ yarn
 
 `yarn build` builds the library to `dist`, generating two files:
 
-* `dist/node-es-transformer.cjs.js`
-    A CommonJS bundle, suitable for use in Node.js, that `require`s the external dependency. This corresponds to the `"main"` field in package.json
-* `dist/node-es-transformer.esm.js`
-    an ES module bundle, suitable for use in other people's libraries and applications, that `import`s the external dependency. This corresponds to the `"module"` field in package.json
+- `dist/node-es-transformer.cjs.js`
+  A CommonJS bundle, suitable for use in Node.js, that `require`s the external dependency. This corresponds to the `"main"` field in package.json
+- `dist/node-es-transformer.esm.js`
+  an ES module bundle, suitable for use in other people's libraries and applications, that `import`s the external dependency. This corresponds to the `"module"` field in package.json
 
 `yarn dev` builds the library, then keeps rebuilding it whenever the source files change using [rollup-watch](https://github.com/rollup/rollup-watch).
 

package/dist/node-es-transformer.cjs.js

@@ -5,48 +5,55 @@ function _interopDefault (ex) { return (ex && (typeof ex === 'object') && 'defau
 var fs = _interopDefault(require('fs'));
 var es = _interopDefault(require('event-stream'));
 var glob = _interopDefault(require('glob'));
-var elasticsearch = _interopDefault(require('elasticsearch'));
+var cliProgress = _interopDefault(require('cli-progress'));
+var elasticsearch = _interopDefault(require('@elastic/elasticsearch'));
 
 function createMappingFactory(ref) {
-  var client = ref.client;
+  var sourceClient = ref.sourceClient;
+  var sourceIndexName = ref.sourceIndexName;
+  var targetClient = ref.targetClient;
   var targetIndexName = ref.targetIndexName;
   var mappings = ref.mappings;
   var verbose = ref.verbose;
 
-  return function () { return (new Promise(function (resolve, reject) {
-    console.log('targetIndexName', targetIndexName);
-    if (
-      typeof mappings === 'object'
-      && mappings !== null
-    ) {
-      client.indices.create({
-        index: targetIndexName,
-        body: { mappings: mappings },
-      }, function (err, resp) {
-        if (err) {
-          console.log('Error creating mapping', err);
-          reject();
-          return;
-        }
-        if (verbose) { console.log('Created mapping', resp); }
-        resolve();
-      });
-    } else {
-      resolve();
+  return async function () {
+    var targetMappings = mappings;
+
+    if (sourceClient && sourceIndexName && typeof targetMappings === 'undefined') {
+      try {
+        var mapping = await sourceClient.indices.getMapping({ index: sourceIndexName });
+        targetMappings = mapping[sourceIndexName].mappings;
+      } catch (err) {
+        console.log('Error reading source mapping', err);
+        return;
+      }
     }
-  })); };
+
+    if (typeof targetMappings === 'object' && targetMappings !== null) {
+      try {
+        var resp = await targetClient.indices.create(
+          {
+            index: targetIndexName,
+            body: { mappings: targetMappings },
+          }
+        );
+        if (verbose) { console.log('Created target mapping', resp); }
+      } catch (err) {
+        console.log('Error creating target mapping', err);
+      }
+    }
+  };
 }
 
 function fileReaderFactory(indexer, fileName, transform, splitRegex, verbose) {
   function startIndex(files) {
     var file = files.shift();
     var s = fs.createReadStream(file)
-      .pipe(es.split(/\n/))
+      .pipe(es.split(splitRegex))
       .pipe(es.mapSync(function (line) {
         s.pause();
         try {
           var doc = (typeof transform === 'function') ? transform(line) : line;
-          console.log('doc', doc);
           // if doc is undefined we'll skip indexing it
           if (typeof doc === 'undefined') {
             s.resume();
@@ -94,9 +101,8 @@ var queueEmitter = new EventEmitter();
 
 // a simple helper queue to bulk index documents
 function indexQueueFactory(ref) {
-  var client = ref.client;
+  var client = ref.targetClient;
   var targetIndexName = ref.targetIndexName;
-  var typeName = ref.typeName; if ( typeName === void 0 ) typeName = 'doc';
   var bufferSize = ref.bufferSize; if ( bufferSize === void 0 ) bufferSize = 1000;
   var skipHeader = ref.skipHeader; if ( skipHeader === void 0 ) skipHeader = false;
   var verbose = ref.verbose; if ( verbose === void 0 ) verbose = true;
@@ -105,36 +111,40 @@ function indexQueueFactory(ref) {
   var queue = [];
   var ingesting = false;
 
-  var ingest = function (b) {
+  var ingest = async function (b) {
     if (typeof b !== 'undefined') {
       queue.push(b);
+      queueEmitter.emit('queue-size', queue.length);
     }
 
     if (ingesting === false) {
       var docs = queue.shift();
+      queueEmitter.emit('queue-size', queue.length);
       ingesting = true;
       if (verbose) { console.log(("bulk ingest docs: " + (docs.length / 2) + ", queue length: " + (queue.length))); }
 
-      client.bulk({ body: docs }, function () {
+      try {
+        await client.bulk({ body: docs });
         ingesting = false;
         if (queue.length > 0) {
           ingest();
         }
-      });
+      } catch (err) {
+        console.log('bulk index error', err);
+      }
     }
 
     // console.log(`ingest: queue.length ${queue.length}`);
     if (queue.length === 0) {
+      queueEmitter.emit('queue-size', 0);
       queueEmitter.emit('resume');
     }
-
-    return [];
   };
 
   return {
     add: function (doc) {
       if (!skipHeader) {
-        var header = { index: { _index: targetIndexName, _type: typeName } };
+        var header = { index: { _index: targetIndexName } };
         buffer.push(header);
       }
       buffer.push(doc);
@@ -145,16 +155,24 @@ function indexQueueFactory(ref) {
       }
 
       if (buffer.length >= (bufferSize * 2)) {
-        buffer = ingest(buffer);
+        ingest(buffer);
+        buffer = [];
       }
     },
-    finish: function () {
-      buffer = ingest(buffer);
+    finish: async function () {
+      await ingest(buffer);
+      buffer = [];
+      queueEmitter.emit('finish');
     },
     queueEmitter: queueEmitter,
   };
 }
 
+var MAX_QUEUE_SIZE = 5;
+
+// create a new progress bar instance and use shades_classic theme
+var progressBar = new cliProgress.SingleBar({}, cliProgress.Presets.shades_classic);
+
 function indexReaderFactory(indexer, sourceIndexName, transform, client) {
   return async function indexReader() {
     var responseQueue = [];
@@ -164,12 +182,13 @@ function indexReaderFactory(indexer, sourceIndexName, transform, client) {
       return client.search({
         index: sourceIndexName,
         scroll: '30s',
+        size: 10000,
       });
     }
 
     function scroll(id) {
       return client.scroll({
-        scrollId: id,
+        scroll_id: id,
         scroll: '30s',
       });
     }
@@ -178,11 +197,12 @@ function indexReaderFactory(indexer, sourceIndexName, transform, client) {
     // our first response into the queue to be processed
     var se = await search();
     responseQueue.push(se);
+    progressBar.start(se.hits.total.value, 0);
 
     function processHit(hit) {
       docsNum += 1;
       try {
-        var doc = (typeof transform === 'function') ? transform(hit._source) : hit._source; // eslint-disable-line no-underscore-dangle,max-len
+        var doc = (typeof transform === 'function') ? transform(hit._source) : hit._source; // eslint-disable-line no-underscore-dangle
         // if doc is undefined we'll skip indexing it
         if (typeof doc === 'undefined') {
           return;
@@ -201,36 +221,86 @@ function indexReaderFactory(indexer, sourceIndexName, transform, client) {
       }
     }
 
-    while (responseQueue.length) {
-      var response = responseQueue.shift();
+    var ingestQueueSize = 0;
+    var scrollId = se._scroll_id; // eslint-disable-line no-underscore-dangle
+    var readActive = false;
+
+    async function processResponseQueue() {
+      while (responseQueue.length) {
+        readActive = true;
+        var response = responseQueue.shift();
+
+        // collect the docs from this response
+        response.hits.hits.forEach(processHit);
+
+        progressBar.update(docsNum);
+
+        // check to see if we have collected all of the docs
+        // console.log('check count', response.hits.total.value, docsNum);
+        if (response.hits.total.value === docsNum) {
+          indexer.finish();
+          progressBar.stop();
+          break;
+        }
+
+        if (ingestQueueSize < MAX_QUEUE_SIZE) {
+        // get the next response if there are more docs to fetch
+          var sc = await scroll(response._scroll_id); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+          scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+          responseQueue.push(sc);
+        } else {
+          readActive = false;
+        }
+      }
+    }
+
+    indexer.queueEmitter.on('queue-size', async function (size) {
+      ingestQueueSize = size;
+
+      if (!readActive && ingestQueueSize < MAX_QUEUE_SIZE) {
+      // get the next response if there are more docs to fetch
+        var sc = await scroll(scrollId); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+        scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+        responseQueue.push(sc);
+        processResponseQueue();
+      }
+    });
 
-      // collect the docs from this response
-      response.hits.hits.forEach(processHit);
+    indexer.queueEmitter.on('resume', async function () {
+      ingestQueueSize = 0;
 
-      // check to see if we have collected all of the docs
-      if (response.hits.total === docsNum) {
-        console.log('finished scrolling.');
-        indexer.finish();
-        break;
+      if (readActive) {
+        return;
       }
 
       // get the next response if there are more docs to fetch
-      var sc = await scroll(response._scroll_id); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+      var sc = await scroll(scrollId); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+      scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
       responseQueue.push(sc);
-    }
+      processResponseQueue();
+    });
+
+    processResponseQueue();
   };
 }
 
-function transformer(ref) {
+async function transformer(ref) {
   var deleteIndex = ref.deleteIndex; if ( deleteIndex === void 0 ) deleteIndex = false;
+  var protocol = ref.protocol; if ( protocol === void 0 ) protocol = 'http';
   var host = ref.host; if ( host === void 0 ) host = 'localhost';
   var port = ref.port; if ( port === void 0 ) port = '9200';
+  var auth = ref.auth;
+  var rejectUnauthorized = ref.rejectUnauthorized; if ( rejectUnauthorized === void 0 ) rejectUnauthorized = true;
+  var ca = ref.ca;
+  var targetProtocol = ref.targetProtocol;
+  var targetHost = ref.targetHost;
+  var targetPort = ref.targetPort;
+  var targetAuth = ref.targetAuth;
   var bufferSize = ref.bufferSize; if ( bufferSize === void 0 ) bufferSize = 1000;
   var fileName = ref.fileName;
   var splitRegex = ref.splitRegex; if ( splitRegex === void 0 ) splitRegex = /\n/;
   var sourceIndexName = ref.sourceIndexName;
   var targetIndexName = ref.targetIndexName;
-  var typeName = ref.typeName;
   var mappings = ref.mappings;
   var skipHeader = ref.skipHeader; if ( skipHeader === void 0 ) skipHeader = false;
   var transform = ref.transform;
@@ -240,30 +310,70 @@ function transformer(ref) {
     throw Error('targetIndexName must be specified.');
   }
 
-  var client = new elasticsearch.Client({ host: (host + ":" + port) });
+  var sourceNode = protocol + "://" + host + ":" + port;
+  var sourceClient = new elasticsearch.Client({
+    node: sourceNode,
+    auth: auth,
+    tls: { ca: ca, rejectUnauthorized: rejectUnauthorized },
+  });
+
+  var targetNode = (typeof targetProtocol === 'string' ? targetProtocol : protocol) + "://" + (typeof targetHost === 'string' ? targetHost : host) + ":" + (typeof targetPort === 'string' ? targetPort : port);
+  var targetClient = new elasticsearch.Client({
+    node: targetNode,
+    auth: targetAuth !== undefined ? targetAuth : auth,
+    tls: { ca: ca, rejectUnauthorized: rejectUnauthorized },
+  });
 
   var createMapping = createMappingFactory({
-    client: client, targetIndexName: targetIndexName, mappings: mappings, verbose: verbose,
+    sourceClient: sourceClient,
+    sourceIndexName: sourceIndexName,
+    targetClient: targetClient,
+    targetIndexName: targetIndexName,
+    mappings: mappings,
+    verbose: verbose,
   });
   var indexer = indexQueueFactory({
-    client: client, targetIndexName: targetIndexName, typeName: typeName, bufferSize: bufferSize, skipHeader: skipHeader, verbose: verbose,
+    targetClient: targetClient,
+    targetIndexName: targetIndexName,
+    bufferSize: bufferSize,
+    skipHeader: skipHeader,
+    verbose: verbose,
   });
 
   function getReader() {
-    if (typeof fileName !== 'undefined' && typeof sourceIndexName !== 'undefined') {
-      throw Error('Only either one of fileName or sourceIndexName can be specified.');
+    if (
+      typeof fileName !== 'undefined'
+      && typeof sourceIndexName !== 'undefined'
+    ) {
+      throw Error(
+        'Only either one of fileName or sourceIndexName can be specified.'
+      );
     }
 
-    if (typeof fileName === 'undefined' && typeof sourceIndexName === 'undefined') {
+    if (
+      typeof fileName === 'undefined'
+      && typeof sourceIndexName === 'undefined'
+    ) {
       throw Error('Either fileName or sourceIndexName must be specified.');
     }
 
     if (typeof fileName !== 'undefined') {
-      return fileReaderFactory(indexer, fileName, transform, splitRegex, verbose);
+      return fileReaderFactory(
+        indexer,
+        fileName,
+        transform,
+        splitRegex,
+        verbose
+      );
     }
 
     if (typeof sourceIndexName !== 'undefined') {
-      return indexReaderFactory(indexer, sourceIndexName, transform, client);
+      return indexReaderFactory(
+        indexer,
+        sourceIndexName,
+        transform,
+        sourceClient
+      );
     }
 
     return null;
@@ -271,17 +381,26 @@ function transformer(ref) {
 
   var reader = getReader();
 
-  client.indices.exists({ index: targetIndexName }, function (err, resp) {
-    if (resp === false) {
-      createMapping().then(reader);
+  try {
+    var indexExists = await targetClient.indices.exists({ index: targetIndexName });
+
+    if (indexExists === false) {
+      await createMapping();
+      reader();
     } else if (deleteIndex === true) {
-      client.indices.delete({ index: targetIndexName }, function () {
-        createMapping().then(reader);
-      });
+      await targetClient.indices.delete({ index: targetIndexName });
+      await createMapping();
+      reader();
     } else {
       reader();
     }
-  });
+  } catch (error) {
+    console.error('Error checking index existence:', error);
+  } finally {
+    // targetClient.close();
+  }
+
+  return { events: indexer.queueEmitter };
 }
 
 module.exports = transformer;

package/dist/node-es-transformer.esm.js

@@ -1,48 +1,55 @@
 import fs from 'fs';
 import es from 'event-stream';
 import glob from 'glob';
-import elasticsearch from 'elasticsearch';
+import cliProgress from 'cli-progress';
+import elasticsearch from '@elastic/elasticsearch';
 
 function createMappingFactory(ref) {
-  var client = ref.client;
+  var sourceClient = ref.sourceClient;
+  var sourceIndexName = ref.sourceIndexName;
+  var targetClient = ref.targetClient;
   var targetIndexName = ref.targetIndexName;
   var mappings = ref.mappings;
   var verbose = ref.verbose;
 
-  return function () { return (new Promise(function (resolve, reject) {
-    console.log('targetIndexName', targetIndexName);
-    if (
-      typeof mappings === 'object'
-      && mappings !== null
-    ) {
-      client.indices.create({
-        index: targetIndexName,
-        body: { mappings: mappings },
-      }, function (err, resp) {
-        if (err) {
-          console.log('Error creating mapping', err);
-          reject();
-          return;
-        }
-        if (verbose) { console.log('Created mapping', resp); }
-        resolve();
-      });
-    } else {
-      resolve();
+  return async function () {
+    var targetMappings = mappings;
+
+    if (sourceClient && sourceIndexName && typeof targetMappings === 'undefined') {
+      try {
+        var mapping = await sourceClient.indices.getMapping({ index: sourceIndexName });
+        targetMappings = mapping[sourceIndexName].mappings;
+      } catch (err) {
+        console.log('Error reading source mapping', err);
+        return;
+      }
     }
-  })); };
+
+    if (typeof targetMappings === 'object' && targetMappings !== null) {
+      try {
+        var resp = await targetClient.indices.create(
+          {
+            index: targetIndexName,
+            body: { mappings: targetMappings },
+          }
+        );
+        if (verbose) { console.log('Created target mapping', resp); }
+      } catch (err) {
+        console.log('Error creating target mapping', err);
+      }
+    }
+  };
 }
 
 function fileReaderFactory(indexer, fileName, transform, splitRegex, verbose) {
   function startIndex(files) {
     var file = files.shift();
     var s = fs.createReadStream(file)
-      .pipe(es.split(/\n/))
+      .pipe(es.split(splitRegex))
       .pipe(es.mapSync(function (line) {
         s.pause();
         try {
           var doc = (typeof transform === 'function') ? transform(line) : line;
-          console.log('doc', doc);
           // if doc is undefined we'll skip indexing it
           if (typeof doc === 'undefined') {
             s.resume();
@@ -90,9 +97,8 @@ var queueEmitter = new EventEmitter();
 
 // a simple helper queue to bulk index documents
 function indexQueueFactory(ref) {
-  var client = ref.client;
+  var client = ref.targetClient;
   var targetIndexName = ref.targetIndexName;
-  var typeName = ref.typeName; if ( typeName === void 0 ) typeName = 'doc';
   var bufferSize = ref.bufferSize; if ( bufferSize === void 0 ) bufferSize = 1000;
   var skipHeader = ref.skipHeader; if ( skipHeader === void 0 ) skipHeader = false;
   var verbose = ref.verbose; if ( verbose === void 0 ) verbose = true;
@@ -101,36 +107,40 @@ function indexQueueFactory(ref) {
   var queue = [];
   var ingesting = false;
 
-  var ingest = function (b) {
+  var ingest = async function (b) {
     if (typeof b !== 'undefined') {
       queue.push(b);
+      queueEmitter.emit('queue-size', queue.length);
     }
 
     if (ingesting === false) {
       var docs = queue.shift();
+      queueEmitter.emit('queue-size', queue.length);
       ingesting = true;
       if (verbose) { console.log(("bulk ingest docs: " + (docs.length / 2) + ", queue length: " + (queue.length))); }
 
-      client.bulk({ body: docs }, function () {
+      try {
+        await client.bulk({ body: docs });
         ingesting = false;
         if (queue.length > 0) {
           ingest();
         }
-      });
+      } catch (err) {
+        console.log('bulk index error', err);
+      }
     }
 
     // console.log(`ingest: queue.length ${queue.length}`);
     if (queue.length === 0) {
+      queueEmitter.emit('queue-size', 0);
       queueEmitter.emit('resume');
     }
-
-    return [];
   };
 
   return {
     add: function (doc) {
       if (!skipHeader) {
-        var header = { index: { _index: targetIndexName, _type: typeName } };
+        var header = { index: { _index: targetIndexName } };
         buffer.push(header);
       }
       buffer.push(doc);
@@ -141,16 +151,24 @@ function indexQueueFactory(ref) {
       }
 
       if (buffer.length >= (bufferSize * 2)) {
-        buffer = ingest(buffer);
+        ingest(buffer);
+        buffer = [];
       }
     },
-    finish: function () {
-      buffer = ingest(buffer);
+    finish: async function () {
+      await ingest(buffer);
+      buffer = [];
+      queueEmitter.emit('finish');
     },
     queueEmitter: queueEmitter,
   };
 }
 
+var MAX_QUEUE_SIZE = 5;
+
+// create a new progress bar instance and use shades_classic theme
+var progressBar = new cliProgress.SingleBar({}, cliProgress.Presets.shades_classic);
+
 function indexReaderFactory(indexer, sourceIndexName, transform, client) {
   return async function indexReader() {
     var responseQueue = [];
@@ -160,12 +178,13 @@ function indexReaderFactory(indexer, sourceIndexName, transform, client) {
       return client.search({
         index: sourceIndexName,
         scroll: '30s',
+        size: 10000,
       });
     }
 
     function scroll(id) {
       return client.scroll({
-        scrollId: id,
+        scroll_id: id,
         scroll: '30s',
       });
     }
@@ -174,11 +193,12 @@ function indexReaderFactory(indexer, sourceIndexName, transform, client) {
     // our first response into the queue to be processed
     var se = await search();
     responseQueue.push(se);
+    progressBar.start(se.hits.total.value, 0);
 
     function processHit(hit) {
       docsNum += 1;
       try {
-        var doc = (typeof transform === 'function') ? transform(hit._source) : hit._source; // eslint-disable-line no-underscore-dangle,max-len
+        var doc = (typeof transform === 'function') ? transform(hit._source) : hit._source; // eslint-disable-line no-underscore-dangle
         // if doc is undefined we'll skip indexing it
         if (typeof doc === 'undefined') {
           return;
@@ -197,36 +217,86 @@ function indexReaderFactory(indexer, sourceIndexName, transform, client) {
       }
     }
 
-    while (responseQueue.length) {
-      var response = responseQueue.shift();
+    var ingestQueueSize = 0;
+    var scrollId = se._scroll_id; // eslint-disable-line no-underscore-dangle
+    var readActive = false;
+
+    async function processResponseQueue() {
+      while (responseQueue.length) {
+        readActive = true;
+        var response = responseQueue.shift();
+
+        // collect the docs from this response
+        response.hits.hits.forEach(processHit);
+
+        progressBar.update(docsNum);
+
+        // check to see if we have collected all of the docs
+        // console.log('check count', response.hits.total.value, docsNum);
+        if (response.hits.total.value === docsNum) {
+          indexer.finish();
+          progressBar.stop();
+          break;
+        }
+
+        if (ingestQueueSize < MAX_QUEUE_SIZE) {
+        // get the next response if there are more docs to fetch
+          var sc = await scroll(response._scroll_id); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+          scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+          responseQueue.push(sc);
+        } else {
+          readActive = false;
+        }
+      }
+    }
+
+    indexer.queueEmitter.on('queue-size', async function (size) {
+      ingestQueueSize = size;
+
+      if (!readActive && ingestQueueSize < MAX_QUEUE_SIZE) {
+      // get the next response if there are more docs to fetch
+        var sc = await scroll(scrollId); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+        scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+        responseQueue.push(sc);
+        processResponseQueue();
+      }
+    });
 
-      // collect the docs from this response
-      response.hits.hits.forEach(processHit);
+    indexer.queueEmitter.on('resume', async function () {
+      ingestQueueSize = 0;
 
-      // check to see if we have collected all of the docs
-      if (response.hits.total === docsNum) {
-        console.log('finished scrolling.');
-        indexer.finish();
-        break;
+      if (readActive) {
+        return;
       }
 
       // get the next response if there are more docs to fetch
-      var sc = await scroll(response._scroll_id); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+      var sc = await scroll(scrollId); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+      scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
       responseQueue.push(sc);
-    }
+      processResponseQueue();
+    });
+
+    processResponseQueue();
   };
 }
 
-function transformer(ref) {
+async function transformer(ref) {
   var deleteIndex = ref.deleteIndex; if ( deleteIndex === void 0 ) deleteIndex = false;
+  var protocol = ref.protocol; if ( protocol === void 0 ) protocol = 'http';
   var host = ref.host; if ( host === void 0 ) host = 'localhost';
   var port = ref.port; if ( port === void 0 ) port = '9200';
+  var auth = ref.auth;
+  var rejectUnauthorized = ref.rejectUnauthorized; if ( rejectUnauthorized === void 0 ) rejectUnauthorized = true;
+  var ca = ref.ca;
+  var targetProtocol = ref.targetProtocol;
+  var targetHost = ref.targetHost;
+  var targetPort = ref.targetPort;
+  var targetAuth = ref.targetAuth;
   var bufferSize = ref.bufferSize; if ( bufferSize === void 0 ) bufferSize = 1000;
   var fileName = ref.fileName;
   var splitRegex = ref.splitRegex; if ( splitRegex === void 0 ) splitRegex = /\n/;
   var sourceIndexName = ref.sourceIndexName;
   var targetIndexName = ref.targetIndexName;
-  var typeName = ref.typeName;
   var mappings = ref.mappings;
   var skipHeader = ref.skipHeader; if ( skipHeader === void 0 ) skipHeader = false;
   var transform = ref.transform;
@@ -236,30 +306,70 @@ function transformer(ref) {
     throw Error('targetIndexName must be specified.');
   }
 
-  var client = new elasticsearch.Client({ host: (host + ":" + port) });
+  var sourceNode = protocol + "://" + host + ":" + port;
+  var sourceClient = new elasticsearch.Client({
+    node: sourceNode,
+    auth: auth,
+    tls: { ca: ca, rejectUnauthorized: rejectUnauthorized },
+  });
+
+  var targetNode = (typeof targetProtocol === 'string' ? targetProtocol : protocol) + "://" + (typeof targetHost === 'string' ? targetHost : host) + ":" + (typeof targetPort === 'string' ? targetPort : port);
+  var targetClient = new elasticsearch.Client({
+    node: targetNode,
+    auth: targetAuth !== undefined ? targetAuth : auth,
+    tls: { ca: ca, rejectUnauthorized: rejectUnauthorized },
+  });
 
   var createMapping = createMappingFactory({
-    client: client, targetIndexName: targetIndexName, mappings: mappings, verbose: verbose,
+    sourceClient: sourceClient,
+    sourceIndexName: sourceIndexName,
+    targetClient: targetClient,
+    targetIndexName: targetIndexName,
+    mappings: mappings,
+    verbose: verbose,
   });
   var indexer = indexQueueFactory({
-    client: client, targetIndexName: targetIndexName, typeName: typeName, bufferSize: bufferSize, skipHeader: skipHeader, verbose: verbose,
+    targetClient: targetClient,
+    targetIndexName: targetIndexName,
+    bufferSize: bufferSize,
+    skipHeader: skipHeader,
+    verbose: verbose,
   });
 
   function getReader() {
-    if (typeof fileName !== 'undefined' && typeof sourceIndexName !== 'undefined') {
-      throw Error('Only either one of fileName or sourceIndexName can be specified.');
+    if (
+      typeof fileName !== 'undefined'
+      && typeof sourceIndexName !== 'undefined'
+    ) {
+      throw Error(
+        'Only either one of fileName or sourceIndexName can be specified.'
+      );
     }
 
-    if (typeof fileName === 'undefined' && typeof sourceIndexName === 'undefined') {
+    if (
+      typeof fileName === 'undefined'
+      && typeof sourceIndexName === 'undefined'
+    ) {
       throw Error('Either fileName or sourceIndexName must be specified.');
     }
 
     if (typeof fileName !== 'undefined') {
-      return fileReaderFactory(indexer, fileName, transform, splitRegex, verbose);
+      return fileReaderFactory(
+        indexer,
+        fileName,
+        transform,
+        splitRegex,
+        verbose
+      );
     }
 
     if (typeof sourceIndexName !== 'undefined') {
-      return indexReaderFactory(indexer, sourceIndexName, transform, client);
+      return indexReaderFactory(
+        indexer,
+        sourceIndexName,
+        transform,
+        sourceClient
+      );
     }
 
     return null;
@@ -267,17 +377,26 @@ function transformer(ref) {
 
   var reader = getReader();
 
-  client.indices.exists({ index: targetIndexName }, function (err, resp) {
-    if (resp === false) {
-      createMapping().then(reader);
+  try {
+    var indexExists = await targetClient.indices.exists({ index: targetIndexName });
+
+    if (indexExists === false) {
+      await createMapping();
+      reader();
     } else if (deleteIndex === true) {
-      client.indices.delete({ index: targetIndexName }, function () {
-        createMapping().then(reader);
-      });
+      await targetClient.indices.delete({ index: targetIndexName });
+      await createMapping();
+      reader();
     } else {
       reader();
     }
-  });
+  } catch (error) {
+    console.error('Error checking index existence:', error);
+  } finally {
+    // targetClient.close();
+  }
+
+  return { events: indexer.queueEmitter };
 }
 
 export default transformer;

package/package.json

@@ -14,25 +14,26 @@
   "license": "Apache-2.0",
   "author": "Walter Rafelsberger <walter@rafelsberger.at>",
   "contributors": [],
-  "version": "1.0.0-alpha6",
+  "version": "1.0.0-alpha8",
   "main": "dist/node-es-transformer.cjs.js",
   "module": "dist/node-es-transformer.esm.js",
   "dependencies": {
-    "elasticsearch": "^15.0.0",
-    "event-stream": "^3.3.4",
-    "glob": "^7.1.2"
+    "@elastic/elasticsearch": "^8.8.1",
+    "cli-progress": "^3.12.0",
+    "event-stream": "3.3.4",
+    "glob": "7.1.2"
   },
   "devDependencies": {
-    "acorn": "^6.0.0",
-    "eslint": "^4.19.1",
-    "eslint-config-airbnb": "^17.1.0",
-    "eslint-plugin-jsx-a11y": "^6.1.1",
-    "eslint-plugin-react": "^7.11.0",
-    "eslint-plugin-import": "^2.12.0",
-    "rollup": "^0.66.6",
-    "rollup-plugin-buble": "^0.19.4",
-    "rollup-plugin-commonjs": "^8.0.2",
-    "rollup-plugin-node-resolve": "^3.0.0"
+    "acorn": "^6.4.2",
+    "eslint": "8.2.0",
+    "eslint-config-airbnb": "19.0.4",
+    "eslint-plugin-import": "2.27.5",
+    "eslint-plugin-jsx-a11y": "6.7.1",
+    "eslint-plugin-react": "7.32.2",
+    "rollup": "0.66.6",
+    "rollup-plugin-buble": "0.19.6",
+    "rollup-plugin-commonjs": "8.0.2",
+    "rollup-plugin-node-resolve": "3.0.0"
   },
   "scripts": {
     "build": "rollup -c",