node-es-transformer 1.0.0-alpha1 → 1.0.0-alpha11

package/CHANGELOG.md

@@ -0,0 +1,32 @@
+# Changelog
+
+All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.
+
+## [1.0.0-alpha11](https://github.com/walterra/node-es-transformer/compare/v1.0.0-alpha10...v1.0.0-alpha11) (2023-10-12)
+
+### Features
+
+- new option 'indexMappingTotalFieldsLimit' ([92edad1](https://github.com/walterra/node-es-transformer/commit/92edad18da7186d3881fc181e6e88b7929bed2d4))
+
+### Bug Fixes
+
+- fixes bufferSize to be applied to index reader too ([ffc3749](https://github.com/walterra/node-es-transformer/commit/ffc3749e296cd39f39924571c197986addc756ff))
+
+## [`v1.0.0-alpha10`](https://github.com/walterra/node-es-transformer/releases/tag/v1.0.0-alpha10)
+
+- New option `mappingsOverride` (0b951e1).
+- New option `query` (45f91db).
+
+## [`v1.0.0-alpha9`](https://github.com/walterra/node-es-transformer/releases/tag/v1.0.0-alpha9)
+
+- Source and target configs are now expected to be passed in as complete client configs instead of individual parameters (5e6d0c7).
+
+## [`v1.0.0-alpha8`](https://github.com/walterra/node-es-transformer/releases/tag/v1.0.0-alpha8)
+
+- Exposes events and introduces `finish` event (a3e5810).
+- Drop support for `_type` from `6.x` indices (3a26a84).
+
+## [`v1.0.0-alpha7`](https://github.com/walterra/node-es-transformer/releases/tag/v1.0.0-alpha7)
+
+- This version locks down `event-stream` to version `3.3.4` because of the security issue described here: https://github.com/dominictarr/event-stream/issues/116
+- Last version to support `_type` from `6.x` indices.

package/README.md

@@ -1,12 +1,36 @@
 [![npm](https://img.shields.io/npm/v/node-es-transformer.svg?maxAge=2592000)](https://www.npmjs.com/package/node-es-transformer)
 [![npm](https://img.shields.io/npm/l/node-es-transformer.svg?maxAge=2592000)](https://www.npmjs.com/package/node-es-transformer)
 [![npm](https://img.shields.io/npm/dt/node-es-transformer.svg?maxAge=2592000)](https://www.npmjs.com/package/node-es-transformer)
+[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)
 
 # node-es-transformer
 
 A nodejs based library to (re)index and transform data from/to Elasticsearch.
 
-**This is experimental code, use at your own risk.**
+### Why another reindex/ingestion tool?
+
+If you're looking for a nodejs based tool which allows you to ingest large CSV/JSON files in the GigaBytes you've come to the right place. Everything else I've tried with larger files runs out of JS heap, hammers ES with too many single requests, times out or tries to do everything with a single bulk request.
+
+While I'd generally recommend using [Logstash](https://www.elastic.co/products/logstash), [filebeat](https://www.elastic.co/products/beats/filebeat), [Ingest Nodes](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html), [Elastic Agent](https://www.elastic.co/guide/en/fleet/current/fleet-overview.html) or [Elasticsearch Transforms](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) for established use cases, this tool may be of help especially if you feel more at home in the JavaScript/nodejs universe and have use cases with customized ingestion and data transformation needs.
+
+**This is experimental code, use at your own risk. Nonetheless, I encourage you to give it a try so I can gather some feedback.**
+
+### So why is this still _alpha_?
+
+- The API is not quite final and might change from release to release.
+- The code needs some more safety measures to avoid some possible accidental data loss scenarios.
+- No test coverage yet.
+
+---
+
+Now that we've talked about the caveats, let's have a look what you actually get with this tool:
+
+## Features
+
+- Buffering/Streaming for both reading and indexing. Files are read using streaming and Elasticsearch ingestion is done using buffered bulk indexing. This is tailored towards ingestion of large files. Successfully tested so far with JSON and CSV files in the range of 20-30 GBytes. On a single machine running both `node-es-transformer` and Elasticsearch ingestion rates up to 20k documents/second were achieved (2,9 GHz Intel Core i7, 16GByte RAM, SSD), depending on document size.
+- Supports wildcards to ingest/transform a range of files in one go.
+- Supports fetching documents from existing indices using search/scroll. This allows you to reindex with custom data transformations just using JavaScript in the `transform` callback.
+- The `transform` callback gives you each source document, but you can split it up in multiple ones and return an array of documents. An example use case for this: Each source document is a Tweet and you want to transform that into an entity centric index based on Hashtags.
 
 ## Getting started
 
@@ -14,31 +38,90 @@ In your node-js project, add `node-es-transformer` as a dependency (`yarn add no
 
 Use the library in your code like:
 
+### Read from a file
+
 ```javascript
-const { transformer } = require('node-es-transformer');
+const transformer = require('node-es-transformer');
 
 transformer({
   fileName: 'filename.json',
-  indexName: 'my-index',
-  typeName: 'doc',
+  targetIndexName: 'my-index',
+  mappings: {
+    properties: {
+      '@timestamp': {
+        type: 'date'
+      },
+      'first_name': {
+        type: 'keyword'
+      },
+      'last_name': {
+        type: 'keyword'
+      }
+      'full_name': {
+        type: 'keyword'
+      }
+    }
+  },
+  transform(line) {
+    return {
+      ...line,
+      full_name: `${line.first_name} ${line.last_name}`
+    }
+  }
+});
+```
+
+### Read from another index
+
+```javascript
+const transformer = require('node-es-transformer');
+
+transformer({
+  sourceIndexName: 'my-source-index',
+  targetIndexName: 'my-target-index',
+  // optional, if you skip mappings, they will be fetched from the source index.
   mappings: {
-    doc: {
-      properties: {
-        '@timestamp': {
-          type: 'date'
-        },
-        'field1': {
-          type: 'keyword'
-        },
-        'field2': {
-          type: 'keyword'
-        }
+    properties: {
+      '@timestamp': {
+        type: 'date'
+      },
+      'first_name': {
+        type: 'keyword'
+      },
+      'last_name': {
+        type: 'keyword'
       }
+      'full_name': {
+        type: 'keyword'
+      }
+    }
+  },
+  transform(doc) {
+    return {
+      ...doc,
+      full_name: `${line.first_name} ${line.last_name}`
     }
   }
 });
 ```
 
+### Options
+
+- `deleteIndex`: Setting to automatically delete an existing index, default is `false`.
+- `sourceClientConfig`/`targetClientConfig`: Optional Elasticsearch client options, defaults to `{ node: 'http://localhost:9200' }`.
+- `bufferSize`: The amount of documents inserted with each Elasticsearch bulk insert request, default is `1000`.
+- `fileName`: Source filename to ingest, supports wildcards. If this is set, `sourceIndexName` is not allowed.
+- `splitRegex`: Custom line split regex, defaults to `/\n/`.
+- `sourceIndexName`: The source Elasticsearch index to reindex from. If this is set, `fileName` is not allowed.
+- `targetIndexName`: The target Elasticsearch index where documents will be indexed.
+- `mappings`: Optional Elasticsearch document mappings. If not set and you're reindexing from another index, the mappings from the existing index will be used.
+- `mappingsOverride`: If you're reindexing and this is set to `true`, `mappings` will be applied on top of the source index's mappings. Defaults to `false`.
+- `indexMappingTotalFieldsLimit`: Optional field limit for the target index to be created that will be passed on as the `index.mapping.total_fields.limit` setting.
+- `query`: Optional Elasticsearch [DSL query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html) to filter documents from the source index.
+- `skipHeader`: If true, skips the first line of the source file. Defaults to `false`.
+- `transform(line)`: A callback function which allows the transformation of a source line into one or several documents.
+- `verbose`: Logging verbosity, defaults to `true`
+
 ## Development
 
 Clone this repository and install its dependencies:
@@ -49,12 +132,12 @@ cd node-es-transformer
 yarn
 ```
 
-`yarn build` builds the library to `dist`, generating three files:
+`yarn build` builds the library to `dist`, generating two files:
 
-* `dist/node-es-transformer.cjs.js`
-    A CommonJS bundle, suitable for use in Node.js, that `require`s the external dependency. This corresponds to the `"main"` field in package.json
-* `dist/node-es-transformer.esm.js`
-    an ES module bundle, suitable for use in other people's libraries and applications, that `import`s the external dependency. This corresponds to the `"module"` field in package.json
+- `dist/node-es-transformer.cjs.js`
+  A CommonJS bundle, suitable for use in Node.js, that `require`s the external dependency. This corresponds to the `"main"` field in package.json
+- `dist/node-es-transformer.esm.js`
+  an ES module bundle, suitable for use in other people's libraries and applications, that `import`s the external dependency. This corresponds to the `"module"` field in package.json
 
 `yarn dev` builds the library, then keeps rebuilding it whenever the source files change using [rollup-watch](https://github.com/rollup/rollup-watch).
 

package/dist/node-es-transformer.cjs.js

@@ -1,104 +1,418 @@
 'use strict';
 
-Object.defineProperty(exports, '__esModule', { value: true });
-
 function _interopDefault (ex) { return (ex && (typeof ex === 'object') && 'default' in ex) ? ex['default'] : ex; }
 
 var fs = _interopDefault(require('fs'));
 var es = _interopDefault(require('event-stream'));
-var elasticsearch = _interopDefault(require('elasticsearch'));
-
-function transformer(ref) {
-	var deleteIndex = ref.deleteIndex; if ( deleteIndex === void 0 ) deleteIndex = false;
-	var host = ref.host; if ( host === void 0 ) host = 'localhost';
-	var port = ref.port; if ( port === void 0 ) port = '9200';
-	var fileName = ref.fileName;
-	var indexName = ref.indexName;
-	var typeName = ref.typeName;
-	var mappings = ref.mappings;
-	var transform = ref.transform;
-	var verbose = ref.verbose; if ( verbose === void 0 ) verbose = true;
-
-
-	var client = new elasticsearch.Client({
-		host: (host + ":" + port)
-	});
-
-	client.indices.exists({
-		index: indexName
-	}, function (err, resp) {
-		if (resp === false) {
-			createMapping();
-		} else {
-			if (deleteIndex === true) {
-				client.indices.delete({
-					index: indexName
-				}, function (err, resp) {
-					createMapping();
-				});
-			} else {
-				indexFile();
-			}
-		}
-	});
-
-	function createMapping() {
-		if (
-			typeof mappings === 'object' &&
-			mappings !== null
-		) {
-			client.indices.create({
-				index: indexName,
-				body: {
-					mappings: mappings
-				}
-			}, function (err, resp) {
-				console.log('create mapping', err, resp);
-				indexFile();
-			});
-		} else {
-			indexFile();
-		}
-	}
-
-	function indexFile() {
-		var docs = [];
-		var s = fs.createReadStream(fileName)
-			.pipe(es.split())
-			.pipe(es.mapSync(function (line) {
-				s.pause();
-
-				if (line) {
-					try {
-						var header = { index: { _index: indexName, _type: typeName } };
-
-						var doc = (typeof transform === 'function') ? transform(line) : line;
-
-						docs.push(header);
-						docs.push(doc);
-					} catch (e) {
-						console.log('error', e);
-					}
-				}
-
-				// resume the readstream, possibly from a callback
-				s.resume();
-			})
-				.on('error', function (err) {
-					console.log('Error while reading file.', err);
-				})
-				.on('end', function () {
-					verbose && console.log('Read entire file.');
-					client.bulk({
-						body: docs
-					}, function (err, resp) {
-						if (err) {
-							console.log('Ingest Error:', err);
-						}
-					});
-				})
-			);
-	}
+var glob = _interopDefault(require('glob'));
+var cliProgress = _interopDefault(require('cli-progress'));
+var elasticsearch = _interopDefault(require('@elastic/elasticsearch'));
+
+var DEFAULT_BUFFER_SIZE = 1000;
+
+function createMappingFactory(ref) {
+  var sourceClient = ref.sourceClient;
+  var sourceIndexName = ref.sourceIndexName;
+  var targetClient = ref.targetClient;
+  var targetIndexName = ref.targetIndexName;
+  var mappings = ref.mappings;
+  var mappingsOverride = ref.mappingsOverride;
+  var indexMappingTotalFieldsLimit = ref.indexMappingTotalFieldsLimit;
+  var verbose = ref.verbose;
+
+  return async function () {
+    var targetMappings = mappingsOverride ? undefined : mappings;
+
+    if (sourceClient && sourceIndexName && typeof targetMappings === 'undefined') {
+      try {
+        var mapping = await sourceClient.indices.getMapping({
+          index: sourceIndexName,
+        });
+        targetMappings = mapping[sourceIndexName].mappings;
+      } catch (err) {
+        console.log('Error reading source mapping', err);
+        return;
+      }
+    }
+
+    if (typeof targetMappings === 'object' && targetMappings !== null) {
+      if (mappingsOverride) {
+        targetMappings = Object.assign({}, targetMappings,
+          {properties: Object.assign({}, targetMappings.properties,
+            mappings)});
+      }
+
+      try {
+        var resp = await targetClient.indices.create({
+          index: targetIndexName,
+          body: Object.assign({}, {mappings: targetMappings},
+            (indexMappingTotalFieldsLimit !== undefined
+              ? {
+                  settings: {
+                    'index.mapping.total_fields.limit': indexMappingTotalFieldsLimit,
+                  },
+                }
+              : {})),
+        });
+        if (verbose) { console.log('Created target mapping', resp); }
+      } catch (err) {
+        console.log('Error creating target mapping', err);
+      }
+    }
+  };
+}
+
+function fileReaderFactory(indexer, fileName, transform, splitRegex, verbose) {
+  function startIndex(files) {
+    var file = files.shift();
+    var s = fs
+      .createReadStream(file)
+      .pipe(es.split(splitRegex))
+      .pipe(
+        es
+          .mapSync(function (line) {
+            s.pause();
+            try {
+              var doc = typeof transform === 'function' ? transform(line) : line;
+              // if doc is undefined we'll skip indexing it
+              if (typeof doc === 'undefined') {
+                s.resume();
+                return;
+              }
+
+              // the transform callback may return an array of docs so we can emit
+              // multiple docs from a single line
+              if (Array.isArray(doc)) {
+                doc.forEach(function (d) { return indexer.add(d); });
+                return;
+              }
+
+              indexer.add(doc);
+            } catch (e) {
+              console.log('error', e);
+            }
+          })
+          .on('error', function (err) {
+            console.log('Error while reading file.', err);
+          })
+          .on('end', function () {
+            if (verbose) { console.log('Read entire file: ', file); }
+            indexer.finish();
+            if (files.length > 0) {
+              startIndex(files);
+            }
+          })
+      );
+
+    indexer.queueEmitter.on('resume', function () {
+      s.resume();
+    });
+  }
+
+  return function () {
+    glob(fileName, function (er, files) {
+      startIndex(files);
+    });
+  };
+}
+
+var EventEmitter = require('events');
+
+var queueEmitter = new EventEmitter();
+
+// a simple helper queue to bulk index documents
+function indexQueueFactory(ref) {
+  var client = ref.targetClient;
+  var targetIndexName = ref.targetIndexName;
+  var bufferSize = ref.bufferSize; if ( bufferSize === void 0 ) bufferSize = DEFAULT_BUFFER_SIZE;
+  var skipHeader = ref.skipHeader; if ( skipHeader === void 0 ) skipHeader = false;
+  var verbose = ref.verbose; if ( verbose === void 0 ) verbose = true;
+
+  var buffer = [];
+  var queue = [];
+  var ingesting = false;
+
+  var ingest = async function (b) {
+    if (typeof b !== 'undefined') {
+      queue.push(b);
+      queueEmitter.emit('queue-size', queue.length);
+    }
+
+    if (ingesting === false) {
+      var docs = queue.shift();
+      queueEmitter.emit('queue-size', queue.length);
+      ingesting = true;
+      if (verbose)
+        { console.log(("bulk ingest docs: " + (docs.length / 2) + ", queue length: " + (queue.length))); }
+
+      try {
+        await client.bulk({ body: docs });
+        ingesting = false;
+        if (queue.length > 0) {
+          ingest();
+        }
+      } catch (err) {
+        console.log('bulk index error', err);
+      }
+    }
+
+    // console.log(`ingest: queue.length ${queue.length}`);
+    if (queue.length === 0) {
+      queueEmitter.emit('queue-size', 0);
+      queueEmitter.emit('resume');
+    }
+  };
+
+  return {
+    add: function (doc) {
+      if (!skipHeader) {
+        var header = { index: { _index: targetIndexName } };
+        buffer.push(header);
+      }
+      buffer.push(doc);
+
+      // console.log(`add: queue.length ${queue.length}`);
+      if (queue.length === 0) {
+        queueEmitter.emit('resume');
+      }
+
+      if (buffer.length >= bufferSize * 2) {
+        ingest(buffer);
+        buffer = [];
+      }
+    },
+    finish: async function () {
+      await ingest(buffer);
+      buffer = [];
+      queueEmitter.emit('finish');
+    },
+    queueEmitter: queueEmitter,
+  };
+}
+
+var MAX_QUEUE_SIZE = 5;
+
+// create a new progress bar instance and use shades_classic theme
+var progressBar = new cliProgress.SingleBar({}, cliProgress.Presets.shades_classic);
+
+function indexReaderFactory(
+  indexer,
+  sourceIndexName,
+  transform,
+  client,
+  query,
+  bufferSize
+) {
+  if ( bufferSize === void 0 ) bufferSize = DEFAULT_BUFFER_SIZE;
+
+  return async function indexReader() {
+    var responseQueue = [];
+    var docsNum = 0;
+
+    function search() {
+      return client.search({
+        index: sourceIndexName,
+        scroll: '30s',
+        size: bufferSize,
+        query: query,
+      });
+    }
+
+    function scroll(id) {
+      return client.scroll({
+        scroll_id: id,
+        scroll: '30s',
+      });
+    }
+
+    // start things off by searching, setting a scroll timeout, and pushing
+    // our first response into the queue to be processed
+    var se = await search();
+    responseQueue.push(se);
+    progressBar.start(se.hits.total.value, 0);
+
+    function processHit(hit) {
+      docsNum += 1;
+      try {
+        var doc = typeof transform === 'function' ? transform(hit._source) : hit._source; // eslint-disable-line no-underscore-dangle
+        // if doc is undefined we'll skip indexing it
+        if (typeof doc === 'undefined') {
+          return;
+        }
+
+        // the transform callback may return an array of docs so we can emit
+        // multiple docs from a single line
+        if (Array.isArray(doc)) {
+          doc.forEach(function (d) { return indexer.add(d); });
+          return;
+        }
+
+        indexer.add(doc);
+      } catch (e) {
+        console.log('error', e);
+      }
+    }
+
+    var ingestQueueSize = 0;
+    var scrollId = se._scroll_id; // eslint-disable-line no-underscore-dangle
+    var readActive = false;
+
+    async function processResponseQueue() {
+      while (responseQueue.length) {
+        readActive = true;
+        var response = responseQueue.shift();
+
+        // collect the docs from this response
+        response.hits.hits.forEach(processHit);
+
+        progressBar.update(docsNum);
+
+        // check to see if we have collected all of the docs
+        // console.log('check count', response.hits.total.value, docsNum);
+        if (response.hits.total.value === docsNum) {
+          indexer.finish();
+          progressBar.stop();
+          break;
+        }
+
+        if (ingestQueueSize < MAX_QUEUE_SIZE) {
+          // get the next response if there are more docs to fetch
+          var sc = await scroll(response._scroll_id); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+          scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+          responseQueue.push(sc);
+        } else {
+          readActive = false;
+        }
+      }
+    }
+
+    indexer.queueEmitter.on('queue-size', async function (size) {
+      ingestQueueSize = size;
+
+      if (!readActive && ingestQueueSize < MAX_QUEUE_SIZE) {
+        // get the next response if there are more docs to fetch
+        var sc = await scroll(scrollId); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+        scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+        responseQueue.push(sc);
+        processResponseQueue();
+      }
+    });
+
+    indexer.queueEmitter.on('resume', async function () {
+      ingestQueueSize = 0;
+
+      if (readActive) {
+        return;
+      }
+
+      // get the next response if there are more docs to fetch
+      var sc = await scroll(scrollId); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+      scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+      responseQueue.push(sc);
+      processResponseQueue();
+    });
+
+    processResponseQueue();
+  };
+}
+
+async function transformer(ref) {
+  var deleteIndex = ref.deleteIndex; if ( deleteIndex === void 0 ) deleteIndex = false;
+  var sourceClientConfig = ref.sourceClientConfig;
+  var targetClientConfig = ref.targetClientConfig;
+  var bufferSize = ref.bufferSize; if ( bufferSize === void 0 ) bufferSize = DEFAULT_BUFFER_SIZE;
+  var fileName = ref.fileName;
+  var splitRegex = ref.splitRegex; if ( splitRegex === void 0 ) splitRegex = /\n/;
+  var sourceIndexName = ref.sourceIndexName;
+  var targetIndexName = ref.targetIndexName;
+  var mappings = ref.mappings;
+  var mappingsOverride = ref.mappingsOverride; if ( mappingsOverride === void 0 ) mappingsOverride = false;
+  var indexMappingTotalFieldsLimit = ref.indexMappingTotalFieldsLimit;
+  var query = ref.query;
+  var skipHeader = ref.skipHeader; if ( skipHeader === void 0 ) skipHeader = false;
+  var transform = ref.transform;
+  var verbose = ref.verbose; if ( verbose === void 0 ) verbose = true;
+
+  if (typeof targetIndexName === 'undefined') {
+    throw Error('targetIndexName must be specified.');
+  }
+
+  var defaultClientConfig = {
+    node: 'http://localhost:9200',
+  };
+
+  var sourceClient = new elasticsearch.Client(sourceClientConfig || defaultClientConfig);
+  var targetClient = new elasticsearch.Client(
+    targetClientConfig || sourceClientConfig || defaultClientConfig
+  );
+
+  var createMapping = createMappingFactory({
+    sourceClient: sourceClient,
+    sourceIndexName: sourceIndexName,
+    targetClient: targetClient,
+    targetIndexName: targetIndexName,
+    mappings: mappings,
+    mappingsOverride: mappingsOverride,
+    indexMappingTotalFieldsLimit: indexMappingTotalFieldsLimit,
+    verbose: verbose,
+  });
+  var indexer = indexQueueFactory({
+    targetClient: targetClient,
+    targetIndexName: targetIndexName,
+    bufferSize: bufferSize,
+    skipHeader: skipHeader,
+    verbose: verbose,
+  });
+
+  function getReader() {
+    if (typeof fileName !== 'undefined' && typeof sourceIndexName !== 'undefined') {
+      throw Error('Only either one of fileName or sourceIndexName can be specified.');
+    }
+
+    if (typeof fileName === 'undefined' && typeof sourceIndexName === 'undefined') {
+      throw Error('Either fileName or sourceIndexName must be specified.');
+    }
+
+    if (typeof fileName !== 'undefined') {
+      return fileReaderFactory(indexer, fileName, transform, splitRegex, verbose);
+    }
+
+    if (typeof sourceIndexName !== 'undefined') {
+      return indexReaderFactory(
+        indexer,
+        sourceIndexName,
+        transform,
+        sourceClient,
+        query,
+        bufferSize
+      );
+    }
+
+    return null;
+  }
+
+  var reader = getReader();
+
+  try {
+    var indexExists = await targetClient.indices.exists({ index: targetIndexName });
+
+    if (indexExists === false) {
+      await createMapping();
+      reader();
+    } else if (deleteIndex === true) {
+      await targetClient.indices.delete({ index: targetIndexName });
+      await createMapping();
+      reader();
+    } else {
+      reader();
+    }
+  } catch (error) {
+    console.error('Error checking index existence:', error);
+  } finally {
+    // targetClient.close();
+  }
+
+  return { events: indexer.queueEmitter };
 }
 
-exports.transformer = transformer;
+module.exports = transformer;

package/dist/node-es-transformer.esm.js

@@ -1,98 +1,414 @@
 import fs from 'fs';
 import es from 'event-stream';
-import elasticsearch from 'elasticsearch';
-
-function transformer(ref) {
-	var deleteIndex = ref.deleteIndex; if ( deleteIndex === void 0 ) deleteIndex = false;
-	var host = ref.host; if ( host === void 0 ) host = 'localhost';
-	var port = ref.port; if ( port === void 0 ) port = '9200';
-	var fileName = ref.fileName;
-	var indexName = ref.indexName;
-	var typeName = ref.typeName;
-	var mappings = ref.mappings;
-	var transform = ref.transform;
-	var verbose = ref.verbose; if ( verbose === void 0 ) verbose = true;
-
-
-	var client = new elasticsearch.Client({
-		host: (host + ":" + port)
-	});
-
-	client.indices.exists({
-		index: indexName
-	}, function (err, resp) {
-		if (resp === false) {
-			createMapping();
-		} else {
-			if (deleteIndex === true) {
-				client.indices.delete({
-					index: indexName
-				}, function (err, resp) {
-					createMapping();
-				});
-			} else {
-				indexFile();
-			}
-		}
-	});
-
-	function createMapping() {
-		if (
-			typeof mappings === 'object' &&
-			mappings !== null
-		) {
-			client.indices.create({
-				index: indexName,
-				body: {
-					mappings: mappings
-				}
-			}, function (err, resp) {
-				console.log('create mapping', err, resp);
-				indexFile();
-			});
-		} else {
-			indexFile();
-		}
-	}
-
-	function indexFile() {
-		var docs = [];
-		var s = fs.createReadStream(fileName)
-			.pipe(es.split())
-			.pipe(es.mapSync(function (line) {
-				s.pause();
-
-				if (line) {
-					try {
-						var header = { index: { _index: indexName, _type: typeName } };
-
-						var doc = (typeof transform === 'function') ? transform(line) : line;
-
-						docs.push(header);
-						docs.push(doc);
-					} catch (e) {
-						console.log('error', e);
-					}
-				}
-
-				// resume the readstream, possibly from a callback
-				s.resume();
-			})
-				.on('error', function (err) {
-					console.log('Error while reading file.', err);
-				})
-				.on('end', function () {
-					verbose && console.log('Read entire file.');
-					client.bulk({
-						body: docs
-					}, function (err, resp) {
-						if (err) {
-							console.log('Ingest Error:', err);
-						}
-					});
-				})
-			);
-	}
+import glob from 'glob';
+import cliProgress from 'cli-progress';
+import elasticsearch from '@elastic/elasticsearch';
+
+var DEFAULT_BUFFER_SIZE = 1000;
+
+function createMappingFactory(ref) {
+  var sourceClient = ref.sourceClient;
+  var sourceIndexName = ref.sourceIndexName;
+  var targetClient = ref.targetClient;
+  var targetIndexName = ref.targetIndexName;
+  var mappings = ref.mappings;
+  var mappingsOverride = ref.mappingsOverride;
+  var indexMappingTotalFieldsLimit = ref.indexMappingTotalFieldsLimit;
+  var verbose = ref.verbose;
+
+  return async function () {
+    var targetMappings = mappingsOverride ? undefined : mappings;
+
+    if (sourceClient && sourceIndexName && typeof targetMappings === 'undefined') {
+      try {
+        var mapping = await sourceClient.indices.getMapping({
+          index: sourceIndexName,
+        });
+        targetMappings = mapping[sourceIndexName].mappings;
+      } catch (err) {
+        console.log('Error reading source mapping', err);
+        return;
+      }
+    }
+
+    if (typeof targetMappings === 'object' && targetMappings !== null) {
+      if (mappingsOverride) {
+        targetMappings = Object.assign({}, targetMappings,
+          {properties: Object.assign({}, targetMappings.properties,
+            mappings)});
+      }
+
+      try {
+        var resp = await targetClient.indices.create({
+          index: targetIndexName,
+          body: Object.assign({}, {mappings: targetMappings},
+            (indexMappingTotalFieldsLimit !== undefined
+              ? {
+                  settings: {
+                    'index.mapping.total_fields.limit': indexMappingTotalFieldsLimit,
+                  },
+                }
+              : {})),
+        });
+        if (verbose) { console.log('Created target mapping', resp); }
+      } catch (err) {
+        console.log('Error creating target mapping', err);
+      }
+    }
+  };
+}
+
+function fileReaderFactory(indexer, fileName, transform, splitRegex, verbose) {
+  function startIndex(files) {
+    var file = files.shift();
+    var s = fs
+      .createReadStream(file)
+      .pipe(es.split(splitRegex))
+      .pipe(
+        es
+          .mapSync(function (line) {
+            s.pause();
+            try {
+              var doc = typeof transform === 'function' ? transform(line) : line;
+              // if doc is undefined we'll skip indexing it
+              if (typeof doc === 'undefined') {
+                s.resume();
+                return;
+              }
+
+              // the transform callback may return an array of docs so we can emit
+              // multiple docs from a single line
+              if (Array.isArray(doc)) {
+                doc.forEach(function (d) { return indexer.add(d); });
+                return;
+              }
+
+              indexer.add(doc);
+            } catch (e) {
+              console.log('error', e);
+            }
+          })
+          .on('error', function (err) {
+            console.log('Error while reading file.', err);
+          })
+          .on('end', function () {
+            if (verbose) { console.log('Read entire file: ', file); }
+            indexer.finish();
+            if (files.length > 0) {
+              startIndex(files);
+            }
+          })
+      );
+
+    indexer.queueEmitter.on('resume', function () {
+      s.resume();
+    });
+  }
+
+  return function () {
+    glob(fileName, function (er, files) {
+      startIndex(files);
+    });
+  };
+}
+
+var EventEmitter = require('events');
+
+var queueEmitter = new EventEmitter();
+
+// a simple helper queue to bulk index documents
+function indexQueueFactory(ref) {
+  var client = ref.targetClient;
+  var targetIndexName = ref.targetIndexName;
+  var bufferSize = ref.bufferSize; if ( bufferSize === void 0 ) bufferSize = DEFAULT_BUFFER_SIZE;
+  var skipHeader = ref.skipHeader; if ( skipHeader === void 0 ) skipHeader = false;
+  var verbose = ref.verbose; if ( verbose === void 0 ) verbose = true;
+
+  var buffer = [];
+  var queue = [];
+  var ingesting = false;
+
+  var ingest = async function (b) {
+    if (typeof b !== 'undefined') {
+      queue.push(b);
+      queueEmitter.emit('queue-size', queue.length);
+    }
+
+    if (ingesting === false) {
+      var docs = queue.shift();
+      queueEmitter.emit('queue-size', queue.length);
+      ingesting = true;
+      if (verbose)
+        { console.log(("bulk ingest docs: " + (docs.length / 2) + ", queue length: " + (queue.length))); }
+
+      try {
+        await client.bulk({ body: docs });
+        ingesting = false;
+        if (queue.length > 0) {
+          ingest();
+        }
+      } catch (err) {
+        console.log('bulk index error', err);
+      }
+    }
+
+    // console.log(`ingest: queue.length ${queue.length}`);
+    if (queue.length === 0) {
+      queueEmitter.emit('queue-size', 0);
+      queueEmitter.emit('resume');
+    }
+  };
+
+  return {
+    add: function (doc) {
+      if (!skipHeader) {
+        var header = { index: { _index: targetIndexName } };
+        buffer.push(header);
+      }
+      buffer.push(doc);
+
+      // console.log(`add: queue.length ${queue.length}`);
+      if (queue.length === 0) {
+        queueEmitter.emit('resume');
+      }
+
+      if (buffer.length >= bufferSize * 2) {
+        ingest(buffer);
+        buffer = [];
+      }
+    },
+    finish: async function () {
+      await ingest(buffer);
+      buffer = [];
+      queueEmitter.emit('finish');
+    },
+    queueEmitter: queueEmitter,
+  };
+}
+
+var MAX_QUEUE_SIZE = 5;
+
+// create a new progress bar instance and use shades_classic theme
+var progressBar = new cliProgress.SingleBar({}, cliProgress.Presets.shades_classic);
+
+function indexReaderFactory(
+  indexer,
+  sourceIndexName,
+  transform,
+  client,
+  query,
+  bufferSize
+) {
+  if ( bufferSize === void 0 ) bufferSize = DEFAULT_BUFFER_SIZE;
+
+  return async function indexReader() {
+    var responseQueue = [];
+    var docsNum = 0;
+
+    function search() {
+      return client.search({
+        index: sourceIndexName,
+        scroll: '30s',
+        size: bufferSize,
+        query: query,
+      });
+    }
+
+    function scroll(id) {
+      return client.scroll({
+        scroll_id: id,
+        scroll: '30s',
+      });
+    }
+
+    // start things off by searching, setting a scroll timeout, and pushing
+    // our first response into the queue to be processed
+    var se = await search();
+    responseQueue.push(se);
+    progressBar.start(se.hits.total.value, 0);
+
+    function processHit(hit) {
+      docsNum += 1;
+      try {
+        var doc = typeof transform === 'function' ? transform(hit._source) : hit._source; // eslint-disable-line no-underscore-dangle
+        // if doc is undefined we'll skip indexing it
+        if (typeof doc === 'undefined') {
+          return;
+        }
+
+        // the transform callback may return an array of docs so we can emit
+        // multiple docs from a single line
+        if (Array.isArray(doc)) {
+          doc.forEach(function (d) { return indexer.add(d); });
+          return;
+        }
+
+        indexer.add(doc);
+      } catch (e) {
+        console.log('error', e);
+      }
+    }
+
+    var ingestQueueSize = 0;
+    var scrollId = se._scroll_id; // eslint-disable-line no-underscore-dangle
+    var readActive = false;
+
+    async function processResponseQueue() {
+      while (responseQueue.length) {
+        readActive = true;
+        var response = responseQueue.shift();
+
+        // collect the docs from this response
+        response.hits.hits.forEach(processHit);
+
+        progressBar.update(docsNum);
+
+        // check to see if we have collected all of the docs
+        // console.log('check count', response.hits.total.value, docsNum);
+        if (response.hits.total.value === docsNum) {
+          indexer.finish();
+          progressBar.stop();
+          break;
+        }
+
+        if (ingestQueueSize < MAX_QUEUE_SIZE) {
+          // get the next response if there are more docs to fetch
+          var sc = await scroll(response._scroll_id); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+          scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+          responseQueue.push(sc);
+        } else {
+          readActive = false;
+        }
+      }
+    }
+
+    indexer.queueEmitter.on('queue-size', async function (size) {
+      ingestQueueSize = size;
+
+      if (!readActive && ingestQueueSize < MAX_QUEUE_SIZE) {
+        // get the next response if there are more docs to fetch
+        var sc = await scroll(scrollId); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+        scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+        responseQueue.push(sc);
+        processResponseQueue();
+      }
+    });
+
+    indexer.queueEmitter.on('resume', async function () {
+      ingestQueueSize = 0;
+
+      if (readActive) {
+        return;
+      }
+
+      // get the next response if there are more docs to fetch
+      var sc = await scroll(scrollId); // eslint-disable-line no-await-in-loop,no-underscore-dangle,max-len
+      scrollId = sc._scroll_id; // eslint-disable-line no-underscore-dangle
+      responseQueue.push(sc);
+      processResponseQueue();
+    });
+
+    processResponseQueue();
+  };
+}
+
+async function transformer(ref) {
+  var deleteIndex = ref.deleteIndex; if ( deleteIndex === void 0 ) deleteIndex = false;
+  var sourceClientConfig = ref.sourceClientConfig;
+  var targetClientConfig = ref.targetClientConfig;
+  var bufferSize = ref.bufferSize; if ( bufferSize === void 0 ) bufferSize = DEFAULT_BUFFER_SIZE;
+  var fileName = ref.fileName;
+  var splitRegex = ref.splitRegex; if ( splitRegex === void 0 ) splitRegex = /\n/;
+  var sourceIndexName = ref.sourceIndexName;
+  var targetIndexName = ref.targetIndexName;
+  var mappings = ref.mappings;
+  var mappingsOverride = ref.mappingsOverride; if ( mappingsOverride === void 0 ) mappingsOverride = false;
+  var indexMappingTotalFieldsLimit = ref.indexMappingTotalFieldsLimit;
+  var query = ref.query;
+  var skipHeader = ref.skipHeader; if ( skipHeader === void 0 ) skipHeader = false;
+  var transform = ref.transform;
+  var verbose = ref.verbose; if ( verbose === void 0 ) verbose = true;
+
+  if (typeof targetIndexName === 'undefined') {
+    throw Error('targetIndexName must be specified.');
+  }
+
+  var defaultClientConfig = {
+    node: 'http://localhost:9200',
+  };
+
+  var sourceClient = new elasticsearch.Client(sourceClientConfig || defaultClientConfig);
+  var targetClient = new elasticsearch.Client(
+    targetClientConfig || sourceClientConfig || defaultClientConfig
+  );
+
+  var createMapping = createMappingFactory({
+    sourceClient: sourceClient,
+    sourceIndexName: sourceIndexName,
+    targetClient: targetClient,
+    targetIndexName: targetIndexName,
+    mappings: mappings,
+    mappingsOverride: mappingsOverride,
+    indexMappingTotalFieldsLimit: indexMappingTotalFieldsLimit,
+    verbose: verbose,
+  });
+  var indexer = indexQueueFactory({
+    targetClient: targetClient,
+    targetIndexName: targetIndexName,
+    bufferSize: bufferSize,
+    skipHeader: skipHeader,
+    verbose: verbose,
+  });
+
+  function getReader() {
+    if (typeof fileName !== 'undefined' && typeof sourceIndexName !== 'undefined') {
+      throw Error('Only either one of fileName or sourceIndexName can be specified.');
+    }
+
+    if (typeof fileName === 'undefined' && typeof sourceIndexName === 'undefined') {
+      throw Error('Either fileName or sourceIndexName must be specified.');
+    }
+
+    if (typeof fileName !== 'undefined') {
+      return fileReaderFactory(indexer, fileName, transform, splitRegex, verbose);
+    }
+
+    if (typeof sourceIndexName !== 'undefined') {
+      return indexReaderFactory(
+        indexer,
+        sourceIndexName,
+        transform,
+        sourceClient,
+        query,
+        bufferSize
+      );
+    }
+
+    return null;
+  }
+
+  var reader = getReader();
+
+  try {
+    var indexExists = await targetClient.indices.exists({ index: targetIndexName });
+
+    if (indexExists === false) {
+      await createMapping();
+      reader();
+    } else if (deleteIndex === true) {
+      await targetClient.indices.delete({ index: targetIndexName });
+      await createMapping();
+      reader();
+    } else {
+      reader();
+    }
+  } catch (error) {
+    console.error('Error checking index existence:', error);
+  } finally {
+    // targetClient.close();
+  }
+
+  return { events: indexer.queueEmitter };
 }
 
-export { transformer };
+export default transformer;

package/package.json

@@ -1,28 +1,58 @@
 {
   "name": "node-es-transformer",
-  "version": "1.0.0-alpha1",
+  "description": "A nodejs based library to (re)index and transform data from/to Elasticsearch.",
+  "keywords": [
+    "elasticsearch",
+    "data-transformation"
+  ],
+  "private": false,
+  "homepage": "https://github.com/walterra/node-es-transformer",
+  "repository": "https://github.com/walterra/node-es-transformer",
+  "bugs": {
+    "url": "https://github.com/walterra/node-es-transformer/issues"
+  },
+  "license": "Apache-2.0",
+  "author": "Walter Rafelsberger <walter@rafelsberger.at>",
+  "contributors": [],
+  "version": "1.0.0-alpha11",
   "main": "dist/node-es-transformer.cjs.js",
   "module": "dist/node-es-transformer.esm.js",
   "dependencies": {
-    "elasticsearch": "^15.0.0",
-    "event-stream": "^3.3.4"
+    "@elastic/elasticsearch": "^8.8.1",
+    "cli-progress": "^3.12.0",
+    "event-stream": "3.3.4",
+    "glob": "7.1.2"
   },
   "devDependencies": {
-    "eslint": "^4.19.1",
-    "eslint-config-airbnb": "^16.1.0",
-    "eslint-plugin-import": "^2.12.0",
-    "rollup": "^0.46.0",
-    "rollup-plugin-buble": "^0.15.0",
-    "rollup-plugin-commonjs": "^8.0.2",
-    "rollup-plugin-node-resolve": "^3.0.0"
+    "acorn": "^6.4.2",
+    "commit-and-tag-version": "^11.3.0",
+    "cz-conventional-changelog": "^3.3.0",
+    "eslint": "8.2.0",
+    "eslint-config-airbnb": "19.0.4",
+    "eslint-config-prettier": "^9.0.0",
+    "eslint-plugin-import": "2.27.5",
+    "eslint-plugin-jsx-a11y": "6.7.1",
+    "eslint-plugin-prettier": "^3.3.1",
+    "eslint-plugin-react": "7.32.2",
+    "prettier": "^2.2.1",
+    "rollup": "0.66.6",
+    "rollup-plugin-buble": "0.19.6",
+    "rollup-plugin-commonjs": "8.0.2",
+    "rollup-plugin-node-resolve": "3.0.0"
   },
   "scripts": {
     "build": "rollup -c",
     "dev": "rollup -c -w",
     "test": "node test/test.js",
-    "pretest": "npm run build"
+    "pretest": "npm run build",
+    "release": "commit-and-tag-version"
   },
   "files": [
     "dist"
-  ]
+  ],
+  "config": {
+    "commitizen": {
+      "path": "./node_modules/cz-conventional-changelog"
+    }
+  }
 }