frugal-iot-server 2.0.1 → 2.0.3

package/INSTALLATION.md

@@ -25,6 +25,12 @@ and go to [Upgrading](#upgrading).
 `npx --no frugal-iot-diagnostic` from your install directory — it inspects the whole installation and
 reports what looks wrong. See [When something does not work](#when-something-does-not-work).
 
+Adding `quiet` makes it say nothing at all unless something is wrong, and exit non-zero when it
+does, so it can be run periodically rather than only when you already have a problem:
+`npx --no frugal-iot-diagnostic quiet`. The word has no dash on purpose — npm parses the command
+line itself and never passes a flag through to the script, and `-q` is worse still, because it is
+npm's own abbreviation for `--quiet`.
+
 ---
 
 ## Which Raspberry Pi
@@ -189,17 +195,32 @@ bash install-pi.sh --org myfarm --name "My Farm" --email you@example.com --phone
 ```
 
 It asks for anything it needs that you did not pass, says what it is doing as it goes, and stops at
-the first thing that fails - leaving a log, and telling you to run `frugal-iot-diagnostic`. Running
-it again after a failure carries on rather than starting over, which matters on a Pi Zero W where one
-step takes about 40 minutes. `--dry-run` checks the arguments and shows the plan without touching the
-machine.
-
-**Passwords** are asked for, not invented, unless you say otherwise: it prompts for each (twice, not
-echoed) and generates one if you just press Enter. Pass them as `--superuser-password`,
-`--login-password` and `--broker-password` to skip the prompts, `--random-passwords` to have all
-three generated, or `--yes` to ask nothing at all — which is what to use over ssh with no terminal.
-A password you chose is not written to the log; a generated one has to be, since otherwise you would
-have no way of knowing it.
+the first thing that fails - leaving a log, and telling you to run `frugal-iot-diagnostic`.
+`--dry-run` checks the arguments and shows the plan without touching the machine.
+
+**It is safe to run again, as many times as you like.** Every step that changes something checks
+first and says `already done:` instead, so a run that stopped half way carries on from where it got
+to rather than starting over. That matters most on a Pi Zero W, where compiling `sqlite3` takes
+about 40 minutes and is not repeated once it has succeeded. It is also how to pick up a fix: re-run
+it after upgrading the package and it will put right anything the newer version installs differently.
+
+Two things it will tell you it cannot mend by itself, rather than looping:
+
+* a `sqlite3` that still will not load after being rebuilt;
+* a `/var/lib/mosquitto/dynamic-security.json` that exists while `config.d/secrets.yaml` has no
+  `dynsec_admin_password` - a previous run stopped between creating the one and recording the other,
+  and the password it chose was never written down. It tells you to delete that file and run again.
+
+**Passwords.** You are asked for the two you will actually type - the `superuser` login and the
+organization's web login - twice each, not echoed, and pressing Enter has one generated instead.
+`--superuser-password` and `--login-password` skip the prompts, `--random-passwords` generates both,
+and `--yes` asks nothing at all, which is what to use over ssh with no terminal. A password you
+chose is not written to the log; a generated one has to be, or you would have no way of knowing it.
+
+The organization's *broker* credential is no longer among them: it is generated and never shown,
+because nothing on the Pi needs you to know it. Nodes are issued their own credential when they
+enrol, and browsers derive their own when they log in. `--broker-password` still sets it, for an
+organization whose nodes were flashed before enrolment existed.
 
 **It tells you whether a reboot is needed**, rather than leaving you to wonder — it notices a kernel
 or boot-firmware package in the upgrade, the images in `/boot` changing underneath it, and
@@ -209,8 +230,10 @@ Fetch it with `curl -O` and then run it, rather than piping curl into bash - pip
 terminal to ask questions at.
 
 It finishes with a live server and a tested broker. It cannot do step 9, pointing your nodes at it,
-because a node learns its broker by being flashed with it - so it prints the settings to give them.
-Nor step 10, HTTPS.
+because a node learns its broker by being flashed with it - so it prints what to build into them:
+the broker host, the organization, and the organization's **enrolment secret**. A node presents that
+secret once, is issued a broker credential of its own, and stores it; the secret grants nothing else,
+no read and no write. Nor can it do step 10, HTTPS.
 
 The steps below are what it does, in the same order, if you would rather do it by hand or need to
 understand what went wrong, otherwise skip to Step 9.
@@ -242,23 +265,38 @@ which you may not have the password for.
 address does not change. Sensor nodes and phones then have something stable to talk to even where
 `.local` names do not work.
 
-Now the packages the server needs.
+Now the packages the server needs. Which set depends on whether this is a **32-bit or a 64-bit
+installation** — and that is not the same question as which board it is, so check rather than assume:
 
-**On a Pi 4** (or any 64-bit board):
+```
+dpkg --print-architecture
+```
+
+`arm64` is a 64-bit installation; `armhf` is a 32-bit one. **Do not use `uname -m` for this.** It
+reports the *kernel*, and Raspberry Pi OS 32-bit boots the 64-bit kernel by default on any board
+that can run it — so a Pi 4 running the 32-bit image answers `armv8l` there while every binary on
+it is 32-bit.
+
+**If it said `arm64`:**
 
 ```
 sudo apt install -y nodejs npm sqlite3 zsh
 node -v
 ```
 
-**On a Pi Zero W**, three more packages, because a 32-bit machine has to compile part of the server
-in step 4 and a Lite image has no compiler:
+**If it said `armhf`** (any 32-bit installation, on a Pi Zero W or a Pi 4 alike) — three more
+packages, because no ready-made binary is published for 32-bit ARM, so part of the server is
+compiled in step 4 and a Lite image has no compiler:
 
 ```
 sudo apt install -y nodejs npm sqlite3 zsh build-essential python3-dev python3-setuptools
 node -v
 ```
 
+> Skipping those three on a 32-bit installation does not fail until step 4, and then it fails deep
+> inside pages of `gyp` output, ending in `ModuleNotFoundError: No module named 'distutils'`. If you
+> are reading this after seeing that, install them and run step 4 again — nothing is lost.
+
 What each is for:
 
 * `nodejs` — the server needs **Node 18 or later**. Current Raspberry Pi OS (Debian 13, trixie)
@@ -321,8 +359,8 @@ The server is an npm package. Make a directory for this server to live in and in
 that directory will hold your configuration, your data, and your database, while npm looks after
 the software itself underneath it in `node_modules`.
 
-**On a Pi 4** (or any 64-bit board) — a few minutes, everything arrives as ready-made binaries and
-nothing is compiled:
+**On a 64-bit installation** (`dpkg --print-architecture` said `arm64`) — a few minutes, everything
+arrives as ready-made binaries and nothing is compiled:
 
 ```
 mkdir ~/frugal-iot
@@ -330,8 +368,10 @@ cd ~/frugal-iot
 npm install frugal-iot-server
 ```
 
-**On a Pi Zero W** — the extra flags make npm do one thing at a time, which lowers the peak memory
-as well as being kinder to the SD card. This is the slow part; leave it running:
+**On a 32-bit installation** (`armhf`) `sqlite3` is compiled here — a few minutes on a Pi 4, about
+40 on a Pi Zero W. On a board with under about a gigabyte of memory add `--maxsockets 1 --no-audit
+--no-fund`, which makes npm do one thing at a time: that lowers its peak memory as well as being
+kinder to the SD card. This is the slow part; leave it running:
 
 ```
 mkdir ~/frugal-iot
@@ -663,30 +703,50 @@ still answers. Your server is now installed.
 
 ### 9. Point your sensor nodes at the Pi
 
-Your ESP8266/ESP32 nodes are told which broker to use in their sketch — `main.cpp`, or the `.ino`
-file if you build in the Arduino IDE. Look for a line like:
+A node is flashed with an **enrolment secret**,
+presents that once, and is issued a broker credential of its own which it stores. The secret grants only one thing — "create a node in this organization" — so it is not the thing to guard, and a node that is stolen cannot be used to read anyone else's data.
 
-```cpp
-frugal_iot.configure_mqtt("frugaliot.naturalinnovation.org", "dev", "public");
+**First, get the enrolment secret.** On the dashboard on the Pi: **gear icon → Nodes → Enrolment secret**. It prints the line ready to paste. 
+(It is also in `config.d/secrets.yaml` as `enrolment_myfarm`.)
+
+**Second, check the project exists.** A node states which project it is in, and enrolling into one the server does not know is refused. The organization you made in step 6 starts with none, so create one on the dashboard before flashing.
+
+Then, on your workstation, in the PlatformIO project's `platformio-secrets-local.ini` for example ...:
+
+```ini
+[common]
+
+build_flags_local =
+    '-D SYSTEM_FRUGAL_ORG="myfarm"'
+    '-D SYSTEM_FRUGAL_PROJECT="lotus"'
+    '-D SYSTEM_MQTT_HOST="frugaliot.local"'
+    '-D SYSTEM_MQTT_ENROL_URL="http://frugaliot.local:8080/enrol"'
+    '-D SYSTEM_MQTT_ENROL_SECRET="<the secret from the dashboard>"'
 ```
 
-and point it at your Pi instead:
+Note `http`, not `https`, in the enrol URL: a Pi on your own network has no certificate, and step 10 is where that may change in the future. The defaults in `platformio.ini` point at the production server, so all five
+lines are needed to aim a build at your Pi.
+
+The sketch itself calls:
 
 ```cpp
-frugal_iot.configure_mqtt("frugaliot.local", "myfarm", "<broker-password>");
+frugal_iot.configure_mqtt_enrolled(SYSTEM_MQTT_HOST, SYSTEM_MQTT_ENROL_SECRET);
 ```
 
-The three arguments are the broker's host, the organization, and that organization's broker
-password. The organization must be the one you created in step 6, because it is the first part of
-every topic the node publishes to, and the password is the *broker* password from that step — not
-the login password. Then rebuild and flash the node as usual.
+which is already what the examples do. Then rebuild and flash as usual. The node enrols on first boot and appears on the dashboard by itself.
+
+> If the node does not connect, try the Pi's IP address in place of `frugaliot.local`, in both the
+> host and the enrol URL. Resolving `.local` names needs mDNS support in the firmware, which is not
+> something this guide has confirmed; an IP address avoids the question entirely, which is why step
+> 3 suggests reserving one for the Pi in your router.
+
+> A node whose enrolment is refused — wrong secret, unknown project, or awaiting approval — retries
+> every half hour rather than giving up, because nobody can reach a node in a field to reboot it.
 
-> If the node does not connect, try the Pi's IP address in place of `frugaliot.local`. Resolving
-> `.local` names needs mDNS support in the firmware, which is not something this guide has
-> confirmed; an IP address avoids the question entirely, which is why step 3 suggests reserving one
-> for the Pi in your router.
+**To rotate the secret**, add a new line above the old one in `config.d/secrets.yaml` and leave the
+old one there until every node flashed with it has enrolled. Deleting it first strands them.
 
-You can confirm nodes are reporting without involving the UI, using the subscriber from step 6:
+You can watch nodes arriving without involving the UI, using the subscriber from step 6:
 
 ```
 mosquitto_sub -h localhost -u myfarm -P '<broker-password>' -t '#' -v
@@ -698,8 +758,10 @@ port 1883 is reachable from off the Pi, which is what the nodes need.
 ### 9a. Optional: outgoing mail, so people can reset a forgotten password
 
 Without this, a forgotten password can only be fixed by you, on the Pi, with
-`npx --no frugal-iot-setpassword <username> <new-password>`. The login page's "Forgot password?"
-link says "Password reset is not available on this server" rather than pretending to send anything.
+`npx --no frugal-iot-setpassword <username> <new-password>`. 
+
+Until you set this up, the login page's "Forgot password?"
+link says "Password reset is not available on this server".
 
 An offline Pi cannot send mail at all, so skip this unless it has internet access.
 
@@ -819,10 +881,9 @@ surrounding work.
 It comes in three parts, and only the first is a one-off:
 
 * **11a — preparing a production server to accept bridges.** A TLS listener, a certificate, and an
-  access-control file. Done once for a given production server, and not again until there is
-  another one.
-* **11b — authorizing this Pi on that server.** One command, run there, once per Pi.
-* **11c — pointing this Pi at it.** One command, run here, once per Pi.
+  access-control file. Done once for a given production server.
+* **11b — authorizing this Pi on the production server.** One command, run on that server, once per Pi.
+* **11c — pointing this Pi at it.** One command, run on the Pi, once per Pi.
 
 **What you get, and what you do not.** While the link is up, readings appear on production within
 a second or so. While it is down, the Pi records everything as usual and production simply has a
@@ -839,7 +900,7 @@ asks for it; with it, this Pi pulls that organization's logins and permissions f
 fifteen minutes, so the same people can use this Pi's dashboard — including while production is
 unreachable, which is the point of the Pi. It is optional: leave the token blank and the bridge
 relays readings as before. Nothing derived from a password travels either way, and each server
-issues its own broker credentials, so the same person has different ones here and there. Revoking a
+issues its own broker credentials, so the same person has different ones on the Pi and on production. Revoking a
 permission on production removes it here on the next pull. See SECURITY.md.
 
 #### 11a. Preparing a production server to accept bridges
@@ -917,118 +978,68 @@ installed by hand, the written procedure for doing that.
 > Note the asymmetry with the Pi end below: a *certificate* change needs only `reload`, but adding
 > or changing a *bridge* needs a full `restart`, because Mosquitto does not reload bridges.
 
-**Turn on access control** — if this broker predates the release that ships it. Since
-`extras/aclfile` became part of the base install, a Pi set up with `install-pi.sh` already has this
-and there is nothing to do here; the steps below are for a broker built before that, or one whose
-configuration was written by hand.
-
-Without it, any account that can log in to the broker can publish and subscribe anywhere on it, so
-a per-Pi account would be no more confined than the organization's own. This is the part to plan
-carefully, because Mosquitto's ACL file is **deny-by-default**: the moment the broker names an
-`acl_file`, every existing account with no entry in it stops working. So the file that restricts new
-bridge accounts has to grant the existing organizations what they already have, in the same change.
+#### 11b. Authorizing this Pi (on the production server)
 
-```
-sudo tee /etc/mosquitto/aclfile >/dev/null <<'EOF'
-# Deny-by-default: an account with no rule here can connect but reach no topic at all.
+Once per Pi, and there are two ways. **Use the dashboard** unless that server's broker has no
+dynamic security plugin.
 
-# Each organization reaches only its own topic tree. "%u" is the connecting account name, and the
-# first element of every Frugal IoT topic is the organization id - so this is what the nodes,
-# dashboards and the server's own logger already do. It just stops being optional.
-pattern readwrite %u/#
+**From the dashboard** — no shell account on the production server needed, which is the point:
 
-# Whether a bridge is up, for frugal-iot-diagnostic. "pattern readwrite %u/#" does not match $SYS,
-# so without this the diagnostic reports every bridge as never having connected. Mosquitto logs a
-# warning that this pattern contains no %u - harmless, and a "topic" line would apply to anonymous
-# clients only, which is not what is wanted.
-pattern read $SYS/broker/connection/+/state
-EOF
-sudo chown mosquitto:mosquitto /etc/mosquitto/aclfile
-sudo chmod 600 /etc/mosquitto/aclfile
-```
+1. Log in to production as someone with ADMIN on the organization.
+2. Open the organization, gear icon → **Bridges**.
+3. Under "Add a bridge", give the Pi a short name — `northfield`, `shed`, `village2`. It only
+   distinguishes one Pi from another within an organization, and becomes part of the broker account
+   name, so it cannot be changed afterwards without reconfiguring that Pi.
+4. Copy the **broker password** and the **replica token** it shows. Both are shown once.
 
-The ownership matters: like the password file and the TLS key, the broker reads this *after*
-dropping to the `mosquitto` user, so a root-only file stops it starting. Mode 600 owned by
-`mosquitto` rather than 644 owned by root — 2.0.21 accepts the latter but warns on every start that
-a file it does not own, or one that is world readable, "will be refused by a future version". Then name it in the configuration
-and restart:
-
-```
-echo 'acl_file /etc/mosquitto/aclfile' | sudo tee /etc/mosquitto/conf.d/zy-frugal-iot-acl.conf
-sudo systemctl restart mosquitto
-```
-
-Check every existing organization can still log in **before** connecting any Pi — a wrong ACL file
-locks out every node and dashboard at once, and the reason appears only in Mosquitto's own log. On a
-Frugal IoT server the quickest check is its journal, which logs one line per organization:
-
-```
-sudo journalctl -u frugaliot -n 40 --no-pager | grep -iE "mqtt |not authoris"
-```
-
-Every organization should show `connect` and none should show `not authorized`. To back the change
-out, delete `/etc/mosquitto/conf.d/zy-frugal-iot-acl.conf` and restart.
-
-Nothing needs adding here per Pi: `frugal-iot-addbridge-prod` in 11b appends each bridge's own rule.
-
-> `pattern readwrite %u/#` on its own denies `$SYS`, which would stop `frugal-iot-diagnostic`
-> reporting whether a bridge is connected — it reads `$SYS/broker/connection/+/state` as the
-> organization. The second rule above restores exactly that one topic and nothing else. Reading all
-> of `$SYS` would also work and would hand every account the broker's client counts, subscription
-> counts and traffic totals, which on a multi-organization broker tells each organization about the
-> others.
-
-#### 11b. Authorizing this Pi (run on the production server)
-
-Once per Pi. On the production server, from its own directory:
+**From a shell on the production server**, which is what to use if its broker predates the plugin:
 
 ```
 npx --no frugal-iot-addbridge-prod <org-id> <site-name>
 ```
 
-The site name only distinguishes one Pi from another within an organization — `northfield`,
-`shed`, `village2` — and becomes part of the account name. The script creates the broker account,
-adds its access-control rule, reloads the broker, and prints both the password and the exact
-command to run in 11c.
+Either way you end up with a broker account `bridge-<site-name>`, its access-control rules, a
+replica token, and the command to run in 11c.
 
-Three things it checks, because each of them is a way this quietly fails:
+Three things worth knowing, because each is a way this quietly fails:
 
 * **The organization must already exist on the production server**, or the readings will arrive at
   its broker and nothing will record them — that server's logger subscribes per organization,
   driven by the files in its `config.d/organizations/`. If it is missing, add it there first with
   `frugal-iot-addorganization`, using the *same* organization id as this Pi.
-* **Each Pi gets its own account**, rather than sharing the organization's. The organization's
-  broker password is handed to every browser that logs in, so it is not a secret; a per-Pi account
-  can be confined to one site and revoked on its own.
-* **An existing account is left alone.** Re-running it for a site that already has one is refused
-  rather than quietly issuing a new password, which would stop that Pi relaying without anyone
-  touching it.
+* **Each Pi gets its own account**, rather than sharing the organization's, so it can be confined to
+  one site and revoked on its own.
+* **Doing it again for a site that already exists re-issues both credentials**, and the ones that Pi
+  is using stop working — from the dashboard it asks first. That is how to recover from having lost
+  them, and how to revoke a Pi you no longer trust; it is not a way to look them up again.
 
-The password is shown once and is not stored anywhere you can read back — copy it before you lose
-the output.
+Neither credential is stored anywhere you can read back, so copy both before you lose the output.
 
 #### 11c. Bridging this Pi (run on the Pi)
 
-Once per Pi. Run the command 11b printed, from this installation's directory:
+Once per Pi. Have both credentials from 11b to hand first — it asks for each in turn:
+
+* the **broker password** for `bridge-<site-name>`, which the MQTT bridge authenticates with;
+* the **replica token**, which lets this Pi pull production's logins so the same people can log in
+  here. Press Enter to skip it: a bridge relays readings perfectly well without one, and you can
+  re-run this later to add it.
+
+They are two different credentials for two different connections — MQTT on port 8883 for the first,
+HTTPS for the second — so neither substitutes for the other.
 
 ```
 cd ~/frugal-iot
 npx --no frugal-iot-addbridge-pi <org-id> <prod-host> bridge-<site-name>
 ```
 
-It asks for the password rather than taking it on the command line, so it stays out of your shell
-history. Then it checks the far end is reachable and that its certificate is valid for that name,
-writes `/etc/mosquitto/conf.d/frugal-iot-bridge.conf`, restarts the broker, and reports whether the
-bridge actually connected.
-
-The restart is unavoidable: Mosquitto does not pick up bridges on a reload signal, so `reload`
-appears to succeed and changes nothing. It briefly disconnects every node, which they recover from
-on their own.
+Having asked for those, it checks the far end is reachable and that its certificate is valid for
+that name, writes `/etc/mosquitto/conf.d/frugal-iot-bridge.conf` and — if you gave a token —
+`config.d/replica.yaml` with the token in `config.d/secrets.yaml`, restarts the broker, and reports
+whether the bridge actually connected. It briefly disconnects every node, which they recover from on
+their own.
 
 > The configuration goes into `/etc/mosquitto/conf.d/` and not into this installation's
-> `config.d/`, because it holds a password for the production server. Everything under `config.d/`
-> is served to any logged-in browser by `/config.json`, so a credential put there would not stay
-> private. The file is written mode 600 for the same reason.
+> `config.d/`, to keep the password out of `/config.json`. The file is written mode 600 for the same reason.
 
 To undo it: `sudo rm /etc/mosquitto/conf.d/frugal-iot-bridge.conf && sudo systemctl restart
 mosquitto`. The Pi goes back to being self-contained and keeps everything it has recorded.
@@ -1222,24 +1233,51 @@ journalctl -u frugaliot -n 30
 npm ls frugal-iot-server
 ```
 
-`frugal-iot-init` leaves your files alone, so watch its output for a section headed **"These files
-were left as you have them, but this release ships a different version"**. It gives you the `diff`
-command for each. That matters most for the files you copy somewhere else — a changed
-`extras/mosquitto.conf` or `extras/frugaliot.service` does nothing until you install it again:
+`frugal-iot-init` leaves your own files alone. Two sections of its output are worth reading.
+
+**"These files were left as you have them, but this release ships a different version"** covers the
+sensor schema under `config.d/schema/`, and gives you a `diff` command for each. If you added a
+sensor type of your own, that is exactly what it should say; if you did not, this release changed
+the schema and the difference is worth reading.
+
+**"Installed elsewhere from extras/, and older than this release"** covers the files that live
+somewhere else — the broker configuration, its ACL, the systemd unit. It compares what is actually
+in `/etc`, not the copy in `extras/`, because that copy is never overwritten and so differs for ever
+once a release changes it. To bring anything it names up to date, re-run the installer:
 
 ```
-sudo cp node_modules/frugal-iot-server/extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf
-sudo cp node_modules/frugal-iot-server/extras/aclfile /etc/mosquitto/aclfile
-sudo chown mosquitto:mosquitto /etc/mosquitto/aclfile && sudo chmod 600 /etc/mosquitto/aclfile
-sudo systemctl restart mosquitto
-sudo cp node_modules/frugal-iot-server/extras/frugaliot.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl restart frugaliot
+bash node_modules/frugal-iot-server/scripts/install-pi.sh
 ```
 
-Note those read from `node_modules/frugal-iot-server/extras/`, **not** from `extras/` in this
-directory. `frugal-iot-init` keeps your local copy and only tells you it differs, so the local one is
-the *old* version — copying it would reinstall what you already have. Re-running the install script
-(`bash node_modules/frugal-iot-server/scripts/install-pi.sh`) does the same job and reads from the
-same place.
+It is safe to re-run: it changes only what is out of date, and it is the *only* correct way to
+install these three, because none of them can simply be copied into place:
+
+| file | why a plain `cp` is wrong |
+|------|---------------------------|
+| `extras/mosquitto.conf` | carries `plugin PLUGIN_PATH_SET_BY_INSTALLER`. The path differs by distribution, so the installer substitutes it. Copied as-is, **the broker will not start.** |
+| `extras/aclfile` | `frugal-iot-addbridge-prod` appends a rule per Pi. Copying over it deletes them, and a mosquitto `acl_file` is deny-by-default — so every bridge goes on logging in and reaches nothing at all. |
+| `extras/frugaliot.service` | written for user `pi` in `/home/pi/frugal-iot`; the installer rewrites `User`, `WorkingDirectory` and `ExecStart` for wherever this actually went. |
+
+The installer will not overwrite an ACL file that has per-bridge rules in it. If this release adds a
+rule to a broker that has them, it says which lines are missing and leaves the file alone for you to
+add them by hand, then `sudo systemctl reload mosquitto`.
+
+> **If your broker's configuration was assembled by hand** — several small files in
+> `/etc/mosquitto/conf.d/` rather than the single `frugal-iot.conf` the installer writes — then
+> `frugal-iot-init` reports those files under "Nothing at the usual place, so these were not
+> compared", and re-running the installer is *not* what you want: it would add `frugal-iot.conf`
+> alongside what you already have, declaring the same listeners and the same plugin twice. Compare
+> by hand instead, and add only what is missing:
+>
+> ```
+> sudo grep -hvE '^\s*(#|$)' /etc/mosquitto/conf.d/*.conf | sort -u > /tmp/running.txt
+> grep -hvE '^\s*(#|$)' node_modules/frugal-iot-server/extras/mosquitto.conf | sort -u > /tmp/shipped.txt
+> diff /tmp/running.txt /tmp/shipped.txt
+> ```
+>
+> Lines only in the running config are usually this machine's own — TLS `certfile`/`keyfile`, a
+> bridge `listener`, a `password_file` in a different place. Lines only in the shipped one are what
+> this release added.
 
 > It does not compare `config.yaml`, `config.d/mqtt.yaml`, `config.d/logger.yaml` or
 > `config.d/server.yaml`, because those hold your own settings and would differ every time. If a

package/README.md

@@ -100,6 +100,9 @@ broker logins, what has been logged - and changes nothing:
 ```
 npx --no frugal-iot-diagnostic
 ```
+Add `quiet` and it says nothing unless something is wrong, exiting non-zero when it does, which is
+what makes it usable as a periodic check: `npx --no frugal-iot-diagnostic quiet`. No dash, because
+npm consumes flags before they reach the script - and `-q` is npm's own `--quiet`.
 
 To upgrade later: `npm update frugal-iot-server` then `npx --no frugal-iot-init`.
 

package/config.d/schema/modules.yaml

@@ -249,6 +249,21 @@ gps:
     - leaf:     hdop
     - leaf:     position
     - leaf:     utc_time
+# HC-SR04 ultrasonic module, timed trigger/echo - Sensor_HCSR04. The distance is the same shape
+# as the Modbus `ultrasonic` module's, but the module is only rated to 4m. The temperature is a
+# compensation input for the speed of sound, not a reading - so controlfloat, as for ens160.
+hcsr04:
+  name: "HC-SR04 Distance"
+  topics:
+    - leaf: hcsr04
+      leaf_from: ultrasonic
+      max:  4000
+    - leaf: temperature
+      leaf_from: controlfloat
+      name: Air Temperature
+      min:  -20
+      max:  50
+      color: "#ff0000"
 health:
   name: System
   insidefrugaliot: true

package/frugal-iot-createdb.sql

@@ -96,7 +96,8 @@ CREATE TABLE IF NOT EXISTS `bridges` (
 -- Keyed by (org, nodeid), NOT by project: a node states its own project, so keying on it would let
 -- a denied node return by claiming a different one.
 --
--- Persisted, unlike the record of failed attempts, because a restart must not un-deny a node.
+-- Persisted, because a restart must not un-deny a node. So is node_attempts below, for a
+-- different reason - see the note there.
 CREATE TABLE IF NOT EXISTS `node_grants` (
   `org` TEXT NOT NULL,
   `project` TEXT NOT NULL DEFAULT '',
@@ -193,3 +194,35 @@ CREATE TABLE IF NOT EXISTS notification_queue (
 -- silently becoming the superuser.
 INSERT OR IGNORE INTO `users` (`id`, `username`, `name`, `organization`) VALUES (0, 'everyone', 'Everyone', '');
 INSERT OR IGNORE INTO `users` (`id`, `username`, `name`, `organization`) VALUES (1, 'superuser', 'Superuser', '');
+
+-- Every node that asked to enrol and was refused, for the Nodes card on the dashboard.
+--
+-- This is the ONLY way an administrator learns that a node exists: a node id is not discoverable
+-- from the server, so a node nobody can reach physically can only announce itself by asking and
+-- being refused. The rows are therefore UNTRUSTED - anyone can make an id appear by attempting
+-- enrolment - so each keeps when it asked, from where, and why it was refused, for an administrator
+-- to recognise rather than to trust.
+--
+-- Persisted, although it was in memory first and the reasoning for that was not wrong: the nodes
+-- table has no last_seen column because that would be a database write per reading. But an
+-- enrolment attempt is not a reading. It is capped by the rate limiter at 20 per organization per
+-- hour, repeats from one node are throttled to one write a minute, and a rate-limited request is
+-- not written at all - so the write rate here is a few rows an hour, not one per reading.
+--
+-- What losing them on a restart cost was the thing this table exists to give: a node retries only
+-- every half hour, so a server restart made a node that was asking invisible for up to thirty
+-- minutes, which is indistinguishable from a node that is not asking at all. That is precisely the
+-- case an administrator is trying to diagnose.
+--
+-- Keyed by (org, nodeid) like node_grants, and for the same reason: a node states its own project,
+-- so keying on it would let one node occupy several rows by varying what it claims.
+CREATE TABLE IF NOT EXISTS `node_attempts` (
+  `org` TEXT NOT NULL,
+  `project` TEXT NOT NULL DEFAULT '',
+  `nodeid` TEXT NOT NULL,
+  `reason` TEXT NOT NULL DEFAULT '',
+  `from_addr` TEXT NOT NULL DEFAULT '',
+  `at` INTEGER NOT NULL,
+  `count` INTEGER NOT NULL DEFAULT 1,
+  UNIQUE(`org`, `nodeid`)
+);