ecoflow2mqtt 0.0.1 → 0.2.0

package/README.md

@@ -1,14 +1,146 @@
 # ecoflow2mqtt
 
-Interface between EcoFlow PowerStream / STREAM micro-inverters and MQTT, following the
-[mqtt-smarthome](https://github.com/mqtt-smarthome/mqtt-smarthome) convention, with Home Assistant
-discovery. Built on [mqtt-interfaces-core](https://github.com/hobbyquaker/mqtt-interfaces-core).
+Interface between EcoFlow **STREAM Microinverter** / **PowerStream** micro-inverters and MQTT,
+following the [mqtt-smarthome](https://github.com/mqtt-smarthome/mqtt-smarthome) convention, with
+Home Assistant discovery. Built on
+[mqtt-interfaces-core](https://github.com/hobbyquaker/mqtt-interfaces-core).
 
-**Status: placeholder release to reserve the package name. Work in progress — nothing usable yet.**
+**0.2.0 publishes everything the inverter reports: PV power per input and in total, what goes into
+the grid, the voltages, currents, frequency, feed-in limit and Wi-Fi signal.** It is still read
+only — writing the feed-in limit is the next milestone ([ROADMAP.md](ROADMAP.md)); the protocol
+research is in [RESEARCH.md](RESEARCH.md). Values arrive every few seconds, the way the EcoFlow app
+shows them, without the app having to be open.
 
-Planned: continuous PV / inverter / battery readings via the official EcoFlow Developer API and the
-EcoFlow app MQTT protocol (protobuf), with keep-alive polling so values update every few seconds
-like in the app. See [RESEARCH.md](RESEARCH.md) in the repository for the protocol research.
+## Topics
+
+All under `<name>/status/`, retained, payload `{"val": …, "ts": …, "lc": …}`:
+
+| Item                                 | Example             | Meaning                                        |
+| ------------------------------------ | ------------------- | ---------------------------------------------- |
+| `pv1_watts`                          | `114.7`             | PV input 1, W                                  |
+| `pv2_watts`                          | `112`               | PV input 2, W                                  |
+| `pv_watts`                           | `226.7`             | both inputs together, W                        |
+| `grid_watts`                         | `226`               | what actually goes into the grid, W            |
+| `grid_status`                        | `feed_grid`         | `feed_grid`, `grid_in`, `offline` or `invalid` |
+| `pv1_volts`, `pv1_amps`              | `33.71`, `3.64`     | PV input 1, V and A                            |
+| `pv2_volts`, `pv2_amps`              | `33.38`, `3.58`     | PV input 2, V and A                            |
+| `grid_volts`, `grid_amps`, `grid_hz` | `238`, `1.01`, `50` | the mains side                                 |
+| `feed_limit_watts`                   | `600`               | feed-in limit the inverter runs with, W        |
+| `feed_limit_max_watts`               | `600`               | highest limit it accepts, W                    |
+| `wifi_rssi`                          | `-59`               | Wi-Fi signal of the inverter, dBm              |
+
+Plus, next to `status/`:
+
+| Topic                                          | Payload           | Meaning                        |
+| ---------------------------------------------- | ----------------- | ------------------------------ |
+| `<name>/connected`                             | `0` \| `1` \| `2` | 2 = frames are arriving        |
+| `<name>/info`                                  | JSON              | version, region, masked serial |
+| `<name>/maintenance/set/loglevel`, `…/restart` | `debug` …         | provided by the core           |
+
+`<name>` is the instance name, `ecoflow` by default (`--name`). Payloads are `{val, ts, lc}` JSON
+unless the instance runs with `--no-json-payloads`. There are no `set/` topics yet — 0.2.0 is read
+only.
+
+## Install
+
+Needs Node.js >= 20.19.
+
+```bash
+npm install -g ecoflow2mqtt
+ecoflow2mqtt --email you@example.com --password 'your-app-password' --sn BK01Z... -u mqtt://broker
+```
+
+As a systemd service (one instance per inverter, credentials in `/etc/ecoflow2mqtt/<name>.env`
+with mode 600, state in `/var/lib/ecoflow2mqtt/<name>/`):
+
+```bash
+sudo ecoflow2mqtt --install --name balcony \
+    --email you@example.com --password 'your-app-password' --sn BK01Z... -u mqtt://broker
+sudo systemctl status ecoflow2mqtt@balcony
+journalctl -fu ecoflow2mqtt@balcony
+```
+
+Every option can also be set as an environment variable (`ECOFLOW2MQTT_SN`, `ECOFLOW2MQTT_POLL`,
+…); `MQTT_URL`, `MQTT_USERNAME` and `MQTT_PASSWORD` are used as a fallback, so several adapters can
+share `/etc/mqtt-interfaces/broker.env`. Prefer the env file over the command line — a process list
+is world readable.
+
+### Docker
+
+Multi-arch image (amd64, arm64, armv7):
+
+```
+docker run -d --name ecoflow2mqtt --restart unless-stopped -v ecoflow2mqtt:/data \
+  -e ECOFLOW2MQTT_EMAIL=you@example.com -e ECOFLOW2MQTT_PASSWORD='your-app-password' \
+  -e ECOFLOW2MQTT_SN=BK01Z... -e ECOFLOW2MQTT_MQTT_URL=mqtt://broker \
+  ghcr.io/hobbyquaker/ecoflow2mqtt
+```
+
+The `/data` volume keeps the mqtt client id stable across restarts — without it EcoFlow's broker
+sees a new client on every start.
+
+## What you need
+
+- The **EcoFlow app account** the inverter is bound to (a shared device does not work — bind it to
+  this account in the app).
+- The **serial number**, as shown in the app or on the device.
+- The **region** the account was created in (`--region`, default `eu`). Accounts are region bound;
+  a wrong region answers "account doesn't exist or incorrect password", which is also what an
+  EcoFlow outage answers, so the adapter keeps retrying instead of giving up.
+
+No developer account, no API keys: the adapter logs in the way the app does.
+
+## Options
+
+| Option              | Default            | Meaning                                                         |
+| ------------------- | ------------------ | --------------------------------------------------------------- |
+| `--email`           | required           | EcoFlow app account                                             |
+| `--password`        | required           | its password                                                    |
+| `--sn`              | required           | serial number of the inverter                                   |
+| `--region`          | `eu`               | `eu`, `us`, `global`, `americas`, `cn`                          |
+| `--api-host`        | from region        | override the API host                                           |
+| `--mqtt-host`       | from login         | override the EcoFlow broker (it occasionally names a wrong one) |
+| `--poll`            | `60`               | seconds between full-frame refreshes; `0` = passive             |
+| `--stream-interval` | `0`                | seconds between "activate stream" frames; try `20` if it stalls |
+| `--timeout`         | `300`              | seconds without a frame before `connected` drops to `1`         |
+| `--capture`         | off                | directory for raw frames (serial and account id removed)        |
+| `--state-dir`       | `$STATE_DIRECTORY` | where the MQTT client id is kept                                |
+
+Plus the shared options of every adapter (`--name`, `-u/--mqtt-url`, `--json-payloads`,
+`--ha-discovery`, `--verbosity`, `--install`, `--config-schema`, …): `ecoflow2mqtt --help`.
+
+## Home Assistant
+
+Discovery is on by default: the inverter shows up as one device with fifteen sensors, available
+while `<name>/connected` is `2`. The four power values and the grid status are the primary
+entities; voltages, currents, frequency, the limits and the Wi-Fi signal are filed under
+diagnostics. Readings carry `state_class: measurement`, so HA keeps long-term statistics for them.
+Turn discovery off with `--no-ha-discovery`.
+
+For the energy dashboard you need kWh, and this firmware sends no energy counters — add a
+[Riemann sum helper](https://www.home-assistant.io/integrations/integration/) on `pv_watts` (or on
+`grid_watts` for what you actually feed in).
+
+## How it works, and what that means for you
+
+The adapter logs into EcoFlow's cloud with your app account, asks for the MQTT credentials the app
+uses, and subscribes to the inverter's telemetry topic. Frames are protobuf, obfuscated with a
+one-byte XOR; the decoder and the field numbers are in `lib/proto/`.
+
+- **This is the unofficial API the app uses**, not EcoFlow's developer platform. It has been stable
+  since 2023 and is what most EcoFlow integrations use, but EcoFlow can change it at any time.
+  ecoflow2mqtt keeps the protocol in data files so a change is a small fix, not a rewrite.
+- **Values update every 2–6 seconds** as long as the adapter is subscribed — measured over 20
+  minutes with the app closed. Reports of the stream throttling when nobody is watching did not
+  reproduce on this path; if yours does stall, set `--stream-interval 20` and please open an issue.
+- **Your credentials stay on your machine**; they go to EcoFlow's login endpoint only.
+- **Nothing is written to the inverter.** 0.2.0 only reads.
+
+## Contributing a capture
+
+If your firmware or your model sends different fields, run with `--capture <dir>` for a minute and
+attach the file to an issue. The capture module removes the serial number and the account id
+before writing, so it does not identify your device — the test suite checks that.
 
 ## License
 

package/config.js

@@ -0,0 +1,120 @@
+/**
+ * Adapter options on top of the core's parseConfig(): the shared MQTT / name / discovery /
+ * maintenance options, ECOFLOW2MQTT_* environment variables and --config-schema come from
+ * mqtt-interfaces-core; only the EcoFlow specific options are defined here.
+ *
+ * Credentials and the serial number are marked `secret: true`: they identify the account and the
+ * device, and management UIs mask them (ROADMAP.md E-2). Put them in the instance's env file
+ * rather than on the command line, where every process list would show them.
+ */
+
+import {parseConfig} from 'mqtt-interfaces-core';
+import pkg from './package.json' with {type: 'json'};
+import {REGIONS} from './lib/app/login.js';
+import {defaultStateDir} from './lib/clientid.js';
+
+export const OPTIONS = {
+    email: {
+        type: 'string',
+        describe: 'ecoflow app account the inverter is bound to',
+        demandOption: true,
+        secret: true,
+    },
+    password: {
+        type: 'string',
+        describe: 'password of the ecoflow app account',
+        demandOption: true,
+        secret: true,
+    },
+    sn: {
+        type: 'string',
+        describe: 'serial number of the inverter (as shown in the app)',
+        demandOption: true,
+        secret: true,
+    },
+    region: {
+        type: 'string',
+        describe: 'region the account was created in (accounts are region bound)',
+        choices: REGIONS,
+        default: 'eu',
+    },
+    'api-host': {
+        type: 'string',
+        describe: 'override the api host derived from --region',
+    },
+    'mqtt-host': {
+        type: 'string',
+        describe: 'override the ecoflow broker host returned at login',
+    },
+    poll: {
+        type: 'number',
+        describe: 'seconds between full-frame refreshes (0 = passive, only what the device pushes)',
+        default: 60,
+    },
+    'stream-interval': {
+        type: 'number',
+        describe: 'seconds between energy stream activate frames (0 = off; try 20 if the stream stalls)',
+        default: 0,
+    },
+    timeout: {
+        type: 'number',
+        describe: 'seconds without a frame before the device counts as disconnected',
+        default: 300,
+    },
+    capture: {
+        type: 'string',
+        describe: 'directory to append raw frames to, serial number removed (development)',
+    },
+    'state-dir': {
+        type: 'string',
+        describe: 'directory for the persisted mqtt client id (default: $STATE_DIRECTORY)',
+        default: process.env.STATE_DIRECTORY,
+    },
+};
+
+/** yargs .check(): value ranges the option types cannot express. */
+export function check(argv) {
+    if (!(argv.poll >= 0)) {
+        throw new Error('--poll must be >= 0 seconds');
+    }
+    if (argv.poll > 0 && argv.poll < 5) {
+        throw new Error('--poll must be 0 or >= 5 seconds');
+    }
+    if (!(argv.streamInterval >= 0)) {
+        throw new Error('--stream-interval must be >= 0 seconds');
+    }
+    if (!(argv.timeout >= 30)) {
+        throw new Error('--timeout must be >= 30 seconds');
+    }
+    return true;
+}
+
+/** The state directory actually used (option, $STATE_DIRECTORY, or ~/.ecoflow2mqtt). */
+export function stateDirOf(config) {
+    return config.stateDir || defaultStateDir();
+}
+
+/** Parse the command line + environment; `overrides` (argv, env, exit, print) are for tests. */
+export function parse(overrides = {}) {
+    return parseConfig({
+        pkg,
+        defaults: {name: 'ecoflow'},
+        options: OPTIONS,
+        check,
+        examples: [
+            ['$0 --email me@example.com --password secret --sn BK01Z... -u mqtt://broker', 'run in the foreground'],
+            ['sudo $0 --install -n balcony', 'install as service ecoflow2mqtt@balcony'],
+        ],
+        // the core's own epilog (environment variables) plus what is specific to this adapter
+        epilog:
+            'Every option can also be set via environment variable, e.g. ECOFLOW2MQTT_MQTT_URL, ' +
+            'ECOFLOW2MQTT_SN. The unprefixed MQTT_URL, MQTT_USERNAME, MQTT_PASSWORD are used as fallback.\n' +
+            'Credentials belong in /etc/ecoflow2mqtt/<name>.env (see --install), not on the command line —\n' +
+            'a process list is world readable.\n' +
+            'ecoflow2mqtt uses the same unofficial cloud api as the ecoflow app; ecoflow may change it.\n' +
+            pkg.homepage,
+        ...overrides,
+    });
+}
+
+export default parse();

package/index.js

@@ -1,4 +1,140 @@
 #!/usr/bin/env node
-console.error('ecoflow2mqtt 0.0.1 is a placeholder release; the adapter is work in progress.');
-console.error('See https://github.com/hobbyquaker/ecoflow2mqtt');
-process.exit(1);
+
+/**
+ * ecoflow2mqtt — EcoFlow micro-inverter to MQTT.
+ *
+ * Wiring only: the cloud client (lib/app/) emits decoded frames, the item table (lib/items.js)
+ * turns them into values, the core publishes them. 0.1.0 publishes PV power and nothing else
+ * (ROADMAP.md §1); there are no settable items yet.
+ */
+
+import {createAdapter} from 'mqtt-interfaces-core';
+import config, {stateDirOf} from './config.js';
+import pkg from './package.json' with {type: 'json'};
+import {handle as handleInstall} from './lib/install.js';
+import {EcoflowClient} from './lib/app/mqtt.js';
+import {loadClientUuid} from './lib/clientid.js';
+import {createItems} from './lib/items.js';
+import {createCapture} from './lib/capture.js';
+import {discoveryModel} from './lib/hadiscovery.js';
+import {maskSn, modelOf} from './lib/mask.js';
+import {apiHostOf} from './lib/app/login.js';
+
+handleInstall(config); // --install / --uninstall never reach the rest
+
+const items = createItems();
+/** (cmd_func, cmd_id) pairs already logged, so unknown frames are reported once (E-7) */
+const seenFrames = new Set();
+let lastFrame = 0;
+let lastProductId = null;
+let staleTimer = null;
+let capture = null;
+
+const adapter = createAdapter({
+    pkg,
+    config,
+    deviceLabel: 'inverter',
+    info: () => ({
+        sn: maskSn(config.sn),
+        model: modelOf(config.sn),
+        region: config.region,
+        api: apiHostOf(config),
+        broker: client?.broker?.host,
+        product_id: lastProductId,
+        poll: config.poll,
+    }),
+    discovery: () => discoveryModel({name: config.name, sn: config.sn, jsonPayloads: config.jsonPayloads}),
+    onSet: (parts) => {
+        throw new Error(`no settable items in ${pkg.version} (read only): set/${parts.join('/')}`);
+    },
+    onShutdown: async () => {
+        clearInterval(staleTimer);
+        await capture?.close();
+        await client.stop();
+    },
+});
+
+const {log, pubStatus, setDeviceConnected, publishInfo} = adapter;
+
+const client = new EcoflowClient({
+    config,
+    log,
+    uuid: loadClientUuid({stateDir: stateDirOf(config), name: config.name, log}),
+});
+
+if (config.capture) {
+    capture = createCapture({dir: config.capture, sn: config.sn, userId: () => client.userId, log});
+    client.on('raw', (topic, payload) => capture.write(topic, payload));
+}
+
+client.on('frames', (frames) => {
+    for (const frame of frames) {
+        handleFrame(frame);
+    }
+});
+
+client.on('close', () => setDeviceConnected(false));
+
+function handleFrame(frame) {
+    if (frame.error) {
+        log.debug(`inverter < ${frame.name}: undecodable (${frame.error})`);
+        return;
+    }
+    if (!frame.data) {
+        // not mapped in this version: log the pair once with its payload so it can be added later
+        if (!seenFrames.has(frame.key)) {
+            seenFrames.add(frame.key);
+            log.debug(
+                `inverter < ${frame.name} (${frame.key}), ${frame.pdata.length} bytes: ` +
+                    `${frame.pdata.toString('hex')}`,
+            );
+        }
+        return;
+    }
+    if (!frame.items) {
+        // decoded, but nothing to publish (RuntimePropertyUpload: the device's upload periods)
+        if (!seenFrames.has(frame.key)) {
+            seenFrames.add(frame.key);
+            log.debug(`inverter < ${frame.name}: ${JSON.stringify(frame.data)}`);
+        }
+        return;
+    }
+
+    lastFrame = Date.now();
+    setDeviceConnected(true);
+    if (frame.productId && frame.productId !== lastProductId) {
+        // <name>/info is published before the first frame arrives; refresh it once the device is known
+        lastProductId = frame.productId;
+        publishInfo();
+    }
+
+    const updates = items.update(frame.data);
+    if (updates.length === 0) {
+        return;
+    }
+    log.debug(`inverter < ${frame.name}: ${updates.map(({item, value}) => `${item}=${value}`).join(' ')}`);
+    for (const {item, value} of updates) {
+        pubStatus(item, value);
+    }
+}
+
+/** The device counts as connected while frames keep arriving (E-5) — the cloud's own flag lags. */
+function watchStaleness() {
+    clearInterval(staleTimer);
+    staleTimer = setInterval(() => {
+        if (lastFrame === 0 || !client.connected) {
+            return;
+        }
+        const age = (Date.now() - lastFrame) / 1000;
+        if (age > config.timeout) {
+            log.warn(`no frame from the inverter for ${Math.round(age)} s`);
+            setDeviceConnected(false);
+        }
+    }, 30_000);
+    staleTimer.unref?.();
+}
+
+log.info(`${pkg.name} ${pkg.version} starting, device ${maskSn(config.sn)} (${modelOf(config.sn) ?? 'unknown model'})`);
+adapter.start();
+watchStaleness();
+client.start();

package/lib/app/login.js

@@ -0,0 +1,161 @@
+/**
+ * Step 1-2 of the app path (ROADMAP.md §2): log in with the EcoFlow app account and ask for the
+ * MQTT credentials the app itself uses. No signature, no OAuth — the password goes base64 encoded
+ * (not encrypted) over TLS, exactly like the Android app (R §4.1).
+ *
+ * Accounts are region bound: an EU account must talk to `api-e.ecoflow.com`, and a wrong region
+ * answers "Account doesn't exist or incorrect password" — the same message a real outage returns
+ * (tolwi #902). Nothing here ever discards credentials because of an error; the caller retries.
+ */
+
+/** Region -> API host. Only `eu` is verified on hardware; `--api-host` overrides any of them. */
+export const API_HOSTS = {
+    eu: 'api-e.ecoflow.com',
+    us: 'api.ecoflow.com',
+    global: 'api.ecoflow.com',
+    americas: 'api-a.ecoflow.com',
+    cn: 'api-cn.ecoflow.com',
+};
+
+export const REGIONS = Object.keys(API_HOSTS);
+
+/** Headers the app sends; some EcoFlow endpoints answer differently without them. */
+const APP_HEADERS = {
+    'content-type': 'application/json',
+    lang: 'en_US',
+    platform: 'android',
+    version: '4.1.2.02',
+    'user-agent': 'okhttp/3.14.9',
+};
+
+export class EcoflowApiError extends Error {
+    constructor(message, {code, status, endpoint} = {}) {
+        super(message);
+        this.name = 'EcoflowApiError';
+        this.code = code;
+        this.status = status;
+        this.endpoint = endpoint;
+    }
+}
+
+/** The API host for a config: explicit `--api-host` wins, otherwise the region table. */
+export function apiHostOf({apiHost, region} = {}) {
+    if (apiHost) {
+        return apiHost;
+    }
+    const host = API_HOSTS[String(region ?? '').toLowerCase()];
+    if (!host) {
+        throw new EcoflowApiError(`unknown region '${region}' — use --api-host`, {code: 'EREGION'});
+    }
+    return host;
+}
+
+async function postJson({url, body, headers, fetchImpl}) {
+    let response;
+    try {
+        response = await fetchImpl(url, {method: 'POST', headers: {...APP_HEADERS, ...headers}, body});
+    } catch (error) {
+        throw new EcoflowApiError(`request failed: ${error.message}`, {code: 'ENETWORK', endpoint: url});
+    }
+    return finish(response, url);
+}
+
+async function getJson({url, headers, fetchImpl}) {
+    let response;
+    try {
+        response = await fetchImpl(url, {headers: {...APP_HEADERS, ...headers}});
+    } catch (error) {
+        throw new EcoflowApiError(`request failed: ${error.message}`, {code: 'ENETWORK', endpoint: url});
+    }
+    return finish(response, url);
+}
+
+async function finish(response, url) {
+    let json;
+    try {
+        json = await response.json();
+    } catch {
+        throw new EcoflowApiError(`http ${response.status}, no json body`, {
+            code: 'EFORMAT',
+            status: response.status,
+            endpoint: url,
+        });
+    }
+    if (String(json.code) !== '0') {
+        throw new EcoflowApiError(json.message || `api error ${json.code}`, {
+            code: `E${json.code}`,
+            status: response.status,
+            endpoint: url,
+        });
+    }
+    return json.data ?? {};
+}
+
+/**
+ * POST /auth/login — the app account.
+ *
+ * @returns {Promise<{token: string, userId: string}>}
+ */
+export async function login({host, email, password, fetchImpl = globalThis.fetch}) {
+    const data = await postJson({
+        url: `https://${host}/auth/login`,
+        fetchImpl,
+        body: JSON.stringify({
+            email,
+            password: Buffer.from(String(password)).toString('base64'),
+            scene: 'IOT_APP',
+            userType: 'ECOFLOW',
+            os: 'android',
+            osVersion: '30',
+            appVersion: '4.1.2.02',
+        }),
+    });
+    if (!data.token || !data.user?.userId) {
+        throw new EcoflowApiError('login succeeded but returned no token', {code: 'EFORMAT'});
+    }
+    return {token: data.token, userId: String(data.user.userId)};
+}
+
+/**
+ * GET /iot-auth/app/certification — the broker and the credentials for it.
+ *
+ * @returns {Promise<{host: string, port: number, protocol: string, username: string, password: string}>}
+ */
+export async function certification({host, token, userId, fetchImpl = globalThis.fetch}) {
+    const data = await getJson({
+        url: `https://${host}/iot-auth/app/certification?userId=${encodeURIComponent(userId)}`,
+        headers: {authorization: `Bearer ${token}`},
+        fetchImpl,
+    });
+    if (!data.url || !data.certificateAccount || !data.certificatePassword) {
+        throw new EcoflowApiError('certification returned no broker credentials', {code: 'EFORMAT'});
+    }
+    return {
+        host: data.url,
+        port: Number(data.port) || 8883,
+        protocol: data.protocol || 'mqtts',
+        username: data.certificateAccount,
+        password: data.certificatePassword,
+    };
+}
+
+/**
+ * Login + certification in one step.
+ *
+ * @param {{email: string, password: string, region?: string, apiHost?: string, mqttHost?: string}} config
+ * @returns {Promise<{userId: string, apiHost: string, broker: object}>}
+ */
+export async function authenticate(config, {fetchImpl = globalThis.fetch} = {}) {
+    const host = apiHostOf(config);
+    const {token, userId} = await login({
+        host,
+        email: config.email,
+        password: config.password,
+        fetchImpl,
+    });
+    const broker = await certification({host, token, userId, fetchImpl});
+    if (config.mqttHost) {
+        broker.host = config.mqttHost; // certification occasionally names the wrong region (R §4.1)
+    }
+    return {userId, apiHost: host, broker};
+}