mqtt2elasticsearch 1.0.0 → 2.0.0

package/README.md

@@ -2,44 +2,97 @@
 
 [![mqtt-smarthome](https://img.shields.io/badge/mqtt-smarthome-blue.svg)](https://github.com/mqtt-smarthome/mqtt-smarthome)
 [![NPM version](https://badge.fury.io/js/mqtt2elasticsearch.svg)](http://badge.fury.io/js/mqtt2elasticsearch)
-[![Dependency Status](https://img.shields.io/gemnasium/hobbyquaker/mqtt2elasticsearch.svg?maxAge=2592000)](https://gemnasium.com/github.com/hobbyquaker/mqtt2elasticsearch)
-[![Build Status](https://travis-ci.org/hobbyquaker/mqtt2elasticsearch.svg?branch=master)](https://travis-ci.org/hobbyquaker/mqtt2elasticsearch)
-[![XO code style](https://img.shields.io/badge/code_style-XO-5ed9c7.svg)](https://github.com/sindresorhus/xo)
+[![CI](https://github.com/hobbyquaker/mqtt2elasticsearch/actions/workflows/ci.yml/badge.svg)](https://github.com/hobbyquaker/mqtt2elasticsearch/actions/workflows/ci.yml)
 [![License][mit-badge]][mit-url]
 
-> Send MQTT messages to Elasticsearch
+Index MQTT messages in Elasticsearch — searchable history and Kibana dashboards for everything on
+your broker.
 
+Subscribes to the topics you name and writes one document per message, in bulk, into a daily index.
+Unlike [influx4mqtt](https://github.com/hobbyquaker/influx4mqtt), which keeps numbers only, this
+keeps **every** message: text payloads, JSON, the lot.
 
-### Install
+Built on [mqtt-interfaces-core](https://github.com/hobbyquaker/mqtt-interfaces-core), so it has the
+same options, `--install`, `<name>/info` and maintenance topics as the rest of the `xyz2mqtt` fleet.
 
-`$ sudo npm install -g mqtt2elasticsearch`
+Requires **Elasticsearch 7 or newer** (or OpenSearch). Version 1.x of this package wrote ES 6
+mapping types and does not work with anything released since 2019.
 
+## Install
 
-### Usage 
+```
+npm install -g mqtt2elasticsearch
+mqtt2elasticsearch -u mqtt://broker -e http://elastic:9200
+sudo mqtt2elasticsearch --install -n elastic -u mqtt://broker   # systemd service
+```
 
+### Docker
+
+```
+docker run -d --name mqtt2elasticsearch --restart unless-stopped \
+  -e MQTT2ELASTICSEARCH_MQTT_URL=mqtt://broker \
+  -e MQTT2ELASTICSEARCH_ES_URL=http://elastic:9200 \
+  -e MQTT2ELASTICSEARCH_SUBSCRIBE='+/status/#' \
+  ghcr.io/hobbyquaker/mqtt2elasticsearch
 ```
-Usage: mqtt2elasticsearch [options]
-
-Options:
-  -e, --elastic-url     elasticsearch url     [default: "http://localhost:9200"]
-  -v, --verbosity       possible values: "error", "warn", "info", "debug"
-                                                               [default: "info"]
-  -n, --name            instance name. used as connected topic and client id
-                        prefix                        [default: "elasticsearch"]
-  -u, --url             mqtt broker url. May contain user/password
-                                                   [default: "mqtt://localhost"]
-  -m, --mqtt-smarthome  parse mqtt-smarthome payloads                  [boolean]
-  -s, --subscribe       mqtt topic to subscribe. may be repeated      [required]
-  -h, --help            Show help                                      [boolean]
-  -k, --insecure        allow tls connections with invalid certificates[boolean]
-  --version             Show version number                            [boolean]
-  -i, --index                                                  [default: "mqtt"]
-  -t, --type                                                   [default: "mqtt"]
-
-```  
-
-Example: `$ mqtt2elasticsearch -s '#'`
 
+## What gets indexed
+
+One document per message, in `<index>-YYYY.MM.DD` (UTC), created from an index template the adapter
+writes on start:
+
+```json
+{
+  "@timestamp": 1699999999000,
+  "topic": "hm/status/Wohnzimmer/TEMPERATURE",
+  "payload": "{\"val\":21.5,\"ts\":1699999999000}",
+  "val_number": 21.5,
+  "ts": 1699999999000
+}
+```
+
+- **`{val, ts, lc}` payloads are unpacked** (`--mqtt-smarthome`, on by default). The value goes in a
+  **type-suffixed field** — `val_number`, `val_boolean`, `val_string` — because Elasticsearch fixes
+  a field's type on first use, and one shared `val` would take the type of whatever arrived first
+  and then reject everything else.
+- **The device's `ts` becomes `@timestamp`** when there is one, so a chart shows when a value was
+  measured rather than when it was indexed.
+- **Retained messages are skipped.** The broker replays them on every reconnect, which would file
+  bursts of duplicates under the reconnect time. `--retained-messages` keeps them.
+- **Daily indices** make retention a matter of deleting old indices, which is the only way
+  Elasticsearch deletes cheaply. Use ILM or a cron job.
+
+## Options
+
+| Option                 | Default                 | Meaning                                              |
+| ---------------------- | ----------------------- | ---------------------------------------------------- |
+| `-s, --subscribe`      | `#`                     | topic to index, `+`/`#` wildcards; repeat for more   |
+| `-u, --mqtt-url`       | `mqtt://localhost`      | broker url                                           |
+| `-n, --name`           | `elasticsearch`         | instance name = topic prefix of its own topics       |
+| `-e, --es-url`         | `http://localhost:9200` | Elasticsearch base url                               |
+| `-i, --index`          | `mqtt`                  | index name prefix; one index per day                 |
+| `--es-username`        |                         | http basic auth                                      |
+| `--es-password`        |                         | http basic auth                                      |
+| `--es-api-key`         |                         | api key, used instead of username/password           |
+| `-m, --mqtt-smarthome` | `true`                  | unpack `{val, ts, lc}` into `val_<type>`, `ts`, `lc` |
+| `--retained-messages`  | `false`                 | also index retained messages                         |
+| `--buf-length`         | `500`                   | write once this many documents are buffered          |
+| `--buf-interval`       | `10`                    | seconds between writes                               |
+| `-v, --verbosity`      | `info`                  | `error`, `warn`, `info`, `debug`                     |
+
+Every option is also an environment variable (`MQTT2ELASTICSEARCH_SUBSCRIBE`, …); several
+subscriptions are comma separated there. Put credentials in the instance's env file rather than on
+the command line, where a process list would show them — `--install` does that for you.
+
+## Topics of its own
+
+| Topic                             | Meaning                                                               |
+| --------------------------------- | --------------------------------------------------------------------- |
+| `<name>/connected`                | `2` Elasticsearch is accepting writes · `1` broker only · `0` stopped |
+| `<name>/info`                     | endpoint, index, subscriptions, documents indexed                     |
+| `<name>/maintenance/set/loglevel` | `error` \| `warn` \| `info` \| `debug` at runtime                     |
+| `<name>/maintenance/set/restart`  | graceful restart                                                      |
+| `<name>/maintenance/stats`        | memory, cpu, event loop lag, uptime                                   |
 
 ## License
 

package/config.js

@@ -1,37 +1,100 @@
-module.exports = require('yargs')
-    .usage('Usage: $0 [options]')
-    .describe('e', 'elasticsearch url')
-    .describe('v', 'possible values: "error", "warn", "info", "debug"')
-    .describe('n', 'instance name. used as connected topic and client id prefix')
-    .describe('u', 'mqtt broker url. See https://github.com/mqttjs/MQTT.js#connect-using-a-url')
-    .describe('m', 'parse mqtt-smarthome payloads')
-    .describe('s', 'mqtt topic to subscribe. may be repeated')
-    .describe('h', 'show help')
-    .describe('k', 'allow tls connections with invalid certificates')
-    .alias({
-        e: 'elastic-url',
-        i: 'index',
-        t: 'type',
-        h: 'help',
-        k: 'insecure',
-        n: 'name',
-        u: 'url',
-        v: 'verbosity',
-        c: 'cleanup',
-        m: 'mqtt-smarthome',
-        s: 'subscribe'
-    })
-    .demand('subscribe')
-    .boolean('mqtt-smarthome')
-    .boolean('insecure')
-    .default({
-        e: 'http://localhost:9200',
-        u: 'mqtt://localhost',
-        n: 'elasticsearch',
-        v: 'info',
-        i: 'mqtt',
-        t: 'mqtt'
-    })
-    .version()
-    .help('help')
-    .argv;
+/**
+ * Adapter options on top of the core's parseConfig(): the shared MQTT / name / maintenance
+ * options, MQTT2ELASTICSEARCH_* environment variables and --config-schema come from
+ * mqtt-interfaces-core; only what is specific to Elasticsearch is defined here.
+ *
+ * Targets Elasticsearch 7+ and OpenSearch. `--type` is gone: mapping types were removed in ES 7.
+ */
+
+import {parseConfig} from 'mqtt-interfaces-core';
+import pkg from './package.json' with {type: 'json'};
+
+export const OPTIONS = {
+    subscribe: {
+        alias: 's',
+        type: 'array',
+        describe: 'mqtt topic to index, with + and # wildcards (repeatable)',
+        default: ['#'],
+    },
+    'es-url': {
+        alias: 'e',
+        type: 'string',
+        describe: 'elasticsearch base url',
+        default: 'http://localhost:9200',
+    },
+    index: {
+        alias: 'i',
+        type: 'string',
+        describe: 'index name prefix; one index per day (<prefix>-YYYY.MM.DD)',
+        default: 'mqtt',
+    },
+    'es-username': {
+        type: 'string',
+        describe: 'elasticsearch username (http basic auth)',
+    },
+    'es-password': {
+        type: 'string',
+        describe: 'elasticsearch password',
+        secret: true,
+    },
+    'es-api-key': {
+        type: 'string',
+        describe: 'elasticsearch api key, used instead of username/password',
+        secret: true,
+    },
+    'mqtt-smarthome': {
+        alias: 'm',
+        type: 'boolean',
+        describe: 'unpack {val, ts, lc} payloads into val_<type>, ts and lc fields',
+        default: true,
+    },
+    'retained-messages': {
+        type: 'boolean',
+        describe: 'also index retained messages, which the broker replays on every reconnect',
+        default: false,
+    },
+    'buf-length': {
+        type: 'number',
+        describe: 'write to elasticsearch once this many documents are buffered',
+        default: 500,
+    },
+    'buf-interval': {
+        type: 'number',
+        describe: 'seconds between writes, however few documents are buffered',
+        default: 10,
+    },
+};
+
+/** yargs .check(): value ranges the option types cannot express. */
+export function check(argv) {
+    if (!(argv.bufLength >= 1)) {
+        throw new Error('--buf-length must be >= 1');
+    }
+    if (!(argv.bufInterval >= 1)) {
+        throw new Error('--buf-interval must be >= 1 second');
+    }
+    if (!/^https?:\/\//.test(String(argv.esUrl))) {
+        throw new Error('--es-url must start with http:// or https://');
+    }
+    if (!/^[a-z0-9][a-z0-9._-]*$/.test(String(argv.index))) {
+        throw new Error('--index must be lower case and start with a letter or digit');
+    }
+    for (const topic of argv.subscribe || []) {
+        if (!topic || /#.+/.test(topic)) {
+            throw new Error(`--subscribe ${topic}: # is only allowed as the last level`);
+        }
+    }
+    return true;
+}
+
+export default parseConfig({
+    pkg,
+    options: OPTIONS,
+    defaults: {name: 'elasticsearch'},
+    check,
+    examples: [
+        ['$0 -u mqtt://broker -e http://elastic:9200', 'index every topic'],
+        ['$0 -u mqtt://broker -s "+/status/#" -s "$SYS/#"', 'index two topic patterns'],
+        ['sudo $0 --install -n elastic -u mqtt://broker', 'install as mqtt2elasticsearch@elastic'],
+    ],
+});

package/index.js

@@ -1,133 +1,66 @@
 #!/usr/bin/env node
 
-const Mqtt = require('mqtt');
-const request =	require('request');
-const log = require('yalm');
-const pkg = require('./package.json');
-const config = require('./config.js');
-
-process.title = pkg.name;
-log.setLevel(config.verbosity);
-log.info(pkg.name + ' ' + pkg.version + ' starting');
-
-let mqttConnected;
-let url;
-let currentDate;
-
-createMapping();
-
-log.debug('mqtt trying to connect', config.url);
-
-const mqtt = Mqtt.connect(config.url, {
-    will: {topic: config.name + '/connected', payload: '0', retain: true},
-    clientId: config.name + '_' + Math.random().toString(16).substr(2, 8),
-    rejectUnauthorized: !config.insecure
-});
-
-mqtt.on('connect', () => {
-    mqttConnected = true;
-
-    log.info('mqtt connected', config.url);
-    mqtt.publish(config.name + '/connected', '2', {retain: true});
-
-    if (typeof config.subscribe === 'string') {
-        config.subscribe = [config.subscribe];
-    }
-    config.subscribe.forEach(topic => {
-        log.info('mqtt subscribe', topic);
-        mqtt.subscribe(topic);
-    });
-});
-
-mqtt.on('close', () => {
-    if (mqttConnected) {
-        mqttConnected = false;
-        log.info('mqtt closed ' + config.url);
-    }
-});
-
-mqtt.on('error', err => {
-    log.error('mqtt', err.message);
-});
-
-mqtt.on('message', (topic, payload, msg) => {
-    if (msg.retain) {
+/**
+ * mqtt2elasticsearch — index MQTT messages in Elasticsearch, on mqtt-interfaces-core.
+ *
+ * A *sink*: it has no device of its own, its subject is what every other adapter publishes. The
+ * core's `listen` gives it the subscriptions (absolute topics, not under `<name>/`);
+ * lib/document.js decides what a message becomes and lib/elastic.js writes documents in bulk.
+ *
+ * `<name>/connected` reports 2 while Elasticsearch is accepting writes and 1 while it is not.
+ */
+
+import {createAdapter} from 'mqtt-interfaces-core';
+import config from './config.js';
+import pkg from './package.json' with {type: 'json'};
+import {handle as handleInstall} from './lib/install.js';
+import {document} from './lib/document.js';
+import {createWriter, putTemplate} from './lib/elastic.js';
+
+handleInstall(config); // --install / --uninstall never reach the rest
+
+let writer;
+let indexed = 0;
+
+/**
+ * One message -> one document.
+ *
+ * Retained messages are skipped by default: the broker replays the retained value of every
+ * matching topic on each reconnect, and indexing those would file a burst of duplicates under the
+ * reconnect time. `--retained-messages` keeps them for anyone who wants the snapshot.
+ */
+function index(topic, value, raw, packet) {
+    if (packet.retain && !config.retainedMessages) {
         return;
     }
+    indexed++;
+    writer.add(document(topic, raw, {mqttSmarthome: config.mqttSmarthome}));
+}
 
-    payload = payload.toString();
-    log.debug('mqtt <', topic, payload);
-
-    const data = {
-        '@timestamp': (new Date()).getTime(),
-        topic,
-        payload
-    };
-
-    if (config.mqttSmarthome) {
-        try {
-            const json = JSON.parse(payload);
-            if (json && typeof json.val !== 'undefined') {
-                data['val_' + (typeof json.val).toLowerCase()] = json.val;
-                data.ts = json.ts;
-                data.lc = json.lc;
-            }
-        } catch (err) {}
-    }
-    const body = JSON.stringify(data);
-
-    const d = new Date();
-    const date = d.getDate();
-    if (date !== currentDate) {
-        const index = config.index + '-' + d.getFullYear() + '.' + ('0' + (d.getMonth() + 1)).substr(-2) + '.' + ('0' + date).substr(-2);
-        url = config.elasticUrl + '/' + index + '/' + config.type + '/';
-        currentDate = date;
-    }
-
-    log.debug('post', url, body);
-    request.post({
-        url,
-        headers: {
-            'Content-Type': 'application/json'
-        },
-        body,
-        strictSSL: !config.insecure
-    }, (err, res) => {
-        if (err) {
-            log.error(err);
-        } else if (res.statusCode !== 200 && res.statusCode !== 201) {
-            log.error(res.statusCode, res.body);
-        }
-    });
+const adapter = createAdapter({
+    pkg,
+    config,
+    deviceLabel: 'elasticsearch',
+    info: () => ({
+        elasticsearch: config.esUrl,
+        index: config.index,
+        subscribe: config.subscribe,
+        mqttSmarthome: config.mqttSmarthome,
+        buffered: writer ? writer.size : 0,
+        indexed,
+    }),
+    // a sink: the topics are the user's, anywhere on the broker, not under <name>/
+    listen: Object.fromEntries(config.subscribe.map((topic) => [topic, index])),
+    onShutdown: () => writer.stop(),
 });
 
-function createMapping() {
-    const data = {
-        index_patterns: [config.index + '-*'], // eslint-disable-line camelcase
-        mappings: {}
-    };
-    data.mappings[config.type] = {
-        properties: {
-            '@timestamp': {type: 'date'}
-        }
-    };
-    if (config.mqttSmarthome) {
-        data.mappings[config.type].properties.ts = {type: 'date'};
-        data.mappings[config.type].properties.lc = {type: 'date'};
-        data.mappings[config.type].properties.val_number = {type: 'float'}; // eslint-disable-line camelcase
-    }
-    log.debug('create mappings', JSON.stringify(data));
-    request.put({
-        url: config.elasticUrl + '/_template/' + config.index,
-        headers: {
-            'Content-Type': 'application/json'
-        },
-        body: JSON.stringify(data),
-        strictSSL: !config.insecure
-    }, err => {
-        if (err) {
-            log.error(err.message);
-        }
-    });
-}
+writer = createWriter(config, {
+    log: adapter.log,
+    onState: (up) => adapter.setDeviceConnected(up),
+});
 
+adapter.log.info(`elasticsearch at ${config.esUrl}, index ${config.index}-YYYY.MM.DD`);
+// the template decides the field types of every index created from now on; a failure is logged
+await putTemplate(config, {log: adapter.log});
+writer.start();
+adapter.start();

package/lib/document.js

@@ -0,0 +1,89 @@
+/**
+ * MQTT message -> the Elasticsearch document, and the index it goes in. Pure, so the mapping
+ * rules are testable without a cluster.
+ */
+
+/** Two digits, for the date in an index name. */
+const pad = (n) => String(n).padStart(2, '0');
+
+/**
+ * The index a message belongs in: one per day, so retention is a matter of deleting old indices
+ * rather than deleting documents — which is the only way Elasticsearch deletes cheaply.
+ *
+ * The date is **UTC**, as Logstash and Filebeat write theirs. 1.x used the host's local time,
+ * which puts an index boundary at 22:00 or 23:00 UTC depending on the season and makes the
+ * indices of a DST changeover overlap or gap. `@timestamp` is what Kibana charts against; the
+ * index name only has to be stable and sortable.
+ *
+ * @param {string} prefix `--index`
+ * @param {Date} date
+ */
+export function indexName(prefix, date) {
+    return `${prefix}-${date.getUTCFullYear()}.${pad(date.getUTCMonth() + 1)}.${pad(date.getUTCDate())}`;
+}
+
+/**
+ * The document.
+ *
+ * `topic` and `payload` are always there. With `--mqtt-smarthome` a `{val, ts, lc}` payload is
+ * unpacked as well, and the value is indexed under a **type-suffixed** field — `val_number`,
+ * `val_boolean`, `val_string`. Elasticsearch fixes a field's type on first use, so one shared
+ * `val` field would take the type of whatever arrived first and then reject everything else;
+ * splitting by type is what keeps a number searchable as a number.
+ *
+ * @param {string} topic
+ * @param {string} raw the payload as it arrived
+ * @param {{mqttSmarthome?: boolean, now?: number}} [options]
+ */
+export function document(topic, raw, {mqttSmarthome = false, now = Date.now()} = {}) {
+    const doc = {'@timestamp': now, topic, payload: raw};
+    if (!mqttSmarthome) {
+        return doc;
+    }
+    let json;
+    try {
+        json = JSON.parse(raw);
+    } catch {
+        return doc; // a plain payload; `payload` already holds it
+    }
+    if (!json || typeof json !== 'object' || json.val === undefined) {
+        return doc;
+    }
+    const type = json.val === null ? 'null' : typeof json.val;
+    doc[`val_${type}`] = json.val;
+    if (Number(json.ts) > 0) {
+        doc.ts = json.ts;
+        doc['@timestamp'] = json.ts; // the device's own time is the better one to chart against
+    }
+    if (Number(json.lc) > 0) {
+        doc.lc = json.lc;
+    }
+    return doc;
+}
+
+/**
+ * The index template every daily index is created from.
+ *
+ * Elasticsearch 7 removed mapping types and 8 removed the last of the legacy `_template` shape, so
+ * this is the modern one: `PUT _index_template/<name>` with the mapping directly under
+ * `template.mappings`, no type level anywhere. 1.x wrote the ES 6 shape and fails on anything
+ * current.
+ */
+export function indexTemplate(prefix, {mqttSmarthome = false} = {}) {
+    const properties = {
+        '@timestamp': {type: 'date'},
+        topic: {type: 'keyword'},
+        payload: {type: 'text'},
+    };
+    if (mqttSmarthome) {
+        properties.ts = {type: 'date'};
+        properties.lc = {type: 'date'};
+        properties.val_number = {type: 'double'};
+        properties.val_boolean = {type: 'boolean'};
+        properties.val_string = {type: 'text'};
+    }
+    return {
+        index_patterns: [`${prefix}-*`],
+        template: {mappings: {properties}},
+    };
+}

package/lib/elastic.js

@@ -0,0 +1,159 @@
+/**
+ * The Elasticsearch client: the index template, and a bulk writer.
+ *
+ * 1.x posted one HTTP request per MQTT message, which is a request per state change of every
+ * device in the house. This buffers and uses the `_bulk` API instead — the same trade as
+ * influx4mqtt, and the reason the buffer has to be flushed on shutdown.
+ *
+ * Targets Elasticsearch 7+ (and OpenSearch): typeless documents, `_index_template`. ES 6 and its
+ * mapping types are not supported.
+ */
+
+import {indexName, indexTemplate} from './document.js';
+
+/** Auth header from an api key or basic credentials, if any were given. */
+export function authHeader({esApiKey, esUsername, esPassword}) {
+    if (esApiKey) {
+        return {authorization: `ApiKey ${esApiKey}`};
+    }
+    if (esUsername) {
+        return {authorization: 'Basic ' + Buffer.from(`${esUsername}:${esPassword || ''}`).toString('base64')};
+    }
+    return {};
+}
+
+/**
+ * `PUT _index_template/<prefix>` — so every daily index gets the right field types.
+ *
+ * A failure is logged, not fatal: an existing template, or a cluster that only allows writes, is
+ * no reason to refuse to record anything.
+ */
+export async function putTemplate(config, {log = console, fetchImpl = globalThis.fetch} = {}) {
+    const base = String(config.esUrl).replace(/\/+$/, '');
+    const url = `${base}/_index_template/${encodeURIComponent(config.index)}`;
+    const body = JSON.stringify(indexTemplate(config.index, {mqttSmarthome: config.mqttSmarthome}));
+    try {
+        const response = await fetchImpl(url, {
+            method: 'PUT',
+            headers: {'content-type': 'application/json', ...authHeader(config)},
+            body,
+        });
+        if (response.status >= 200 && response.status < 300) {
+            log.debug('elasticsearch index template', config.index, 'is up to date');
+            return true;
+        }
+        log.warn('elasticsearch index template rejected', response.status, (await response.text()).slice(0, 200));
+    } catch (error) {
+        log.warn('elasticsearch index template failed —', error.message);
+    }
+    return false;
+}
+
+/**
+ * A buffer of documents, written with the `_bulk` API.
+ *
+ * @param {object} config parsed config
+ * @param {{log?: object, fetchImpl?: Function, onState?: (up: boolean) => void, now?: () => Date}} [deps]
+ */
+export function createWriter(
+    config,
+    {log = console, fetchImpl = globalThis.fetch, onState, now = () => new Date()} = {},
+) {
+    const base = String(config.esUrl).replace(/\/+$/, '');
+    const url = `${base}/_bulk`;
+    const headers = {'content-type': 'application/x-ndjson', ...authHeader(config)};
+    const max = Number(config.bufLength) > 0 ? Number(config.bufLength) : 500;
+    let buffer = [];
+    let timer = null;
+    let up = null;
+    let inFlight = null;
+
+    function state(next) {
+        if (next !== up) {
+            up = next;
+            if (onState) {
+                onState(next);
+            }
+        }
+    }
+
+    async function send(docs) {
+        // ndjson: an action line and a source line per document, and a trailing newline
+        const body =
+            docs
+                .map(({index, doc}) => JSON.stringify({index: {_index: index}}) + '\n' + JSON.stringify(doc))
+                .join('\n') + '\n';
+        let response;
+        try {
+            response = await fetchImpl(url, {method: 'POST', headers, body});
+        } catch (error) {
+            log.warn('elasticsearch unreachable —', error.message, `(${docs.length} documents dropped)`);
+            state(false);
+            return false;
+        }
+        if (response.status < 200 || response.status >= 300) {
+            log.error('elasticsearch bulk failed', response.status, (await response.text()).slice(0, 300));
+            state(false);
+            return false;
+        }
+        // a 200 can still contain per-document failures — a mapping conflict looks like this
+        try {
+            const result = await response.json();
+            if (result && result.errors) {
+                const first = (result.items || []).find((i) => i.index && i.index.error);
+                log.warn(
+                    'elasticsearch rejected some documents:',
+                    first ? JSON.stringify(first.index.error) : 'unknown',
+                );
+            }
+        } catch {
+            // a body we cannot parse does not change that the request succeeded
+        }
+        log.debug('elasticsearch wrote', docs.length, 'documents');
+        state(true);
+        return true;
+    }
+
+    function flush() {
+        if (buffer.length === 0) {
+            return inFlight || Promise.resolve(true);
+        }
+        const docs = buffer;
+        buffer = [];
+        inFlight = Promise.resolve(inFlight)
+            .catch(() => {})
+            .then(() => send(docs));
+        return inFlight;
+    }
+
+    return {
+        /** Buffer one document, in the index for today. */
+        add(doc) {
+            buffer.push({index: indexName(config.index, now()), doc});
+            if (buffer.length >= max) {
+                flush();
+            }
+        },
+        flush,
+        get size() {
+            return buffer.length;
+        },
+        get url() {
+            return url;
+        },
+        start() {
+            const seconds = Number(config.bufInterval) > 0 ? Number(config.bufInterval) : 10;
+            timer = setInterval(flush, seconds * 1000);
+            if (timer.unref) {
+                timer.unref();
+            }
+        },
+        async stop() {
+            if (timer) {
+                clearInterval(timer);
+                timer = null;
+            }
+            await flush();
+        },
+    };
+}

package/lib/install.js

@@ -0,0 +1,18 @@
+/**
+ * --install / --uninstall: systemd template service mqtt2elasticsearch@<name>
+ * (mqtt-interfaces-core installer). Nothing privileged and no local state.
+ */
+
+import {createInstaller} from 'mqtt-interfaces-core';
+
+export const SERVICE = 'mqtt2elasticsearch';
+export const ENV_PREFIX = 'MQTT2ELASTICSEARCH';
+
+const installer = createInstaller({
+    service: SERVICE,
+    envPrefix: ENV_PREFIX,
+    description: `${SERVICE} %i - MQTT to Elasticsearch`,
+    documentation: 'https://github.com/hobbyquaker/mqtt2elasticsearch',
+});
+
+export const {unitFile, envFile, installService, uninstallService, handle} = installer;

package/package.json

@@ -1,42 +1,63 @@
 {
   "name": "mqtt2elasticsearch",
-  "version": "1.0.0",
-  "description": "Send MQTT messages to Elasticsearch",
+  "version": "2.0.0",
+  "description": "Send MQTT messages to Elasticsearch. Follows the mqtt-smarthome architecture.",
+  "type": "module",
   "main": "index.js",
   "bin": {
     "mqtt2elasticsearch": "index.js"
   },
   "preferGlobal": true,
+  "files": [
+    "index.js",
+    "config.js",
+    "lib/"
+  ],
+  "engines": {
+    "node": "^20.19 || ^22.12 || >=24"
+  },
   "scripts": {
-    "test": "camo-purge ; xo"
+    "start": "node index.js",
+    "lint": "eslint . && prettier --check .",
+    "format": "prettier --write . && eslint --fix .",
+    "test": "node --test",
+    "deploy": "bash deploy.sh"
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/hobbyquaker/mqtt2elasticsearch"
+    "url": "git+https://github.com/hobbyquaker/mqtt2elasticsearch.git"
   },
+  "homepage": "https://github.com/hobbyquaker/mqtt2elasticsearch",
+  "bugs": {
+    "url": "https://github.com/hobbyquaker/mqtt2elasticsearch/issues"
+  },
+  "author": "Sebastian Raff <hobbyquaker@gmail.com> (https://github.com/hobbyquaker)",
+  "license": "MIT",
   "keywords": [
     "mqtt",
-    "smarthome",
+    "mqtt-smarthome",
+    "home-automation",
     "elasticsearch",
+    "opensearch",
+    "kibana",
     "log"
   ],
-  "author": "Sebastian Raff <hobbyquaker@gmail.com> (https://github.com/hobbyquaker)",
-  "license": "MIT",
-  "bugs": {
-    "url": "https://github.com/hobbyquaker/mqtt2elasticsearch/issues"
+  "mqttInterfaces": {
+    "spec": "2.0",
+    "envPrefix": "MQTT2ELASTICSEARCH",
+    "needs": [
+      "network"
+    ],
+    "serviceExtra": []
   },
-  "homepage": "https://github.com/hobbyquaker/mqtt2elasticsearch",
   "dependencies": {
-    "mqtt": "^2.16.0",
-    "request": "^2.85.0",
-    "yalm": "^4.1.0",
-    "yargs": "^11.0.0"
+    "mqtt-interfaces-core": "^0.13.0"
   },
   "devDependencies": {
-    "camo-purge": "^1.0.2",
-    "xo": "^0.20.3"
-  },
-  "xo": {
-    "space": 4
+    "@eslint/js": "^9",
+    "eslint": "^9",
+    "eslint-config-prettier": "^10",
+    "globals": "^16",
+    "prettier": "^3"
   }
 }

package/.npmignore

@@ -1,3 +0,0 @@
-node_modules
-.DS_Store
-.idea

package/.travis.yml

@@ -1,4 +0,0 @@
-anguage: node_js
-node_js:
-  - '6'
-