frugal-iot-server 0.3.4 → 0.3.5

package/INSTALLATION.md

@@ -13,9 +13,13 @@ Two hardware paths are covered:
 If you already have a working Linux server (not a Pi), you do not need this document —
 see [README.md](https://github.com/mitra42/frugal-iot-server/blob/main/README.md) instead.
 
-> **Status:** first draft, written 2026-07. The Pi 4 path has been written against the current
-> code but not yet walked through end to end on hardware. Items still to confirm are collected
-> under [Open questions](#open-questions) — please add your findings there as you go.
+> **Status:** the Raspberry Pi 4 path has been followed end to end on hardware (see
+> [Tested on](#tested-on)), and corrected from what that run found. Part B, and HTTPS/OTA in step
+> A10, are not written yet. Remaining uncertainties are listed under
+> [Open questions](#open-questions) — please add your findings there as you go.
+
+**Already have a Frugal IoT server running and just want a newer version?** Skip everything below
+and go to [Upgrading](#upgrading).
 
 ---
 
@@ -25,8 +29,11 @@ see [README.md](https://github.com/mitra42/frugal-iot-server/blob/main/README.md
 
 **Hardware**
 
-* Raspberry Pi 4 Model B. 2 GB RAM or more is comfortable; 1 GB should work.
-* A microSD card, 16 GB or larger, Class 10 / A1 or better. (8 GB works but leaves little room for data.)
+* Raspberry Pi 4 Model B. Any RAM size — a running server uses around 250 MB, so even the 1 GB
+  model has plenty of room. (Measured on a 4 GB Pi 4: 231 MB in use with the server running.)
+* A microSD card. 8 GB is enough for the system and software — a working install occupies about
+  5 GB — but sensor data accumulates on this card for as long as the server runs, so 16 GB or larger
+  is the safer choice. Class 10 / A1 or better.
 * The official Raspberry Pi USB-C power supply (5 V / 3 A). Phone chargers frequently cause
   random reboots and corrupted SD cards — this is the single most common cause of "it doesn't work".
 * A way to write the SD card from your laptop: a built-in SD slot or a USB card reader.
@@ -48,10 +55,10 @@ see [README.md](https://github.com/mitra42/frugal-iot-server/blob/main/README.md
 
 **Have these to hand in case the headless setup does not come up**
 
-* A micro-HDMI to HDMI cable plus a monitor, and a USB keyboard — lets you log in directly, see boot
-  errors, and finish Wi-Fi setup by hand. Wi-Fi configured through Imager does not always connect on
-  the first boot (see step A2), so do not count on not needing these.
-* An Ethernet cable from the Pi to your router — bypasses all Wi-Fi problems.
+* An Ethernet cable from the Pi to your router — the simplest way in if the Pi does not appear on
+  the Wi-Fi, and you can then sort the Wi-Fi out over SSH (step A2).
+* A micro-HDMI to HDMI cable plus a monitor, and a USB keyboard — for when the Pi is nowhere near the
+  router, or you want to see boot messages.
 
 ### A1. Write the operating system to the SD card
 
@@ -72,9 +79,11 @@ a desktop would only consume memory and SD card space.
    * Type the SSID exactly as the network broadcasts it, including capitals. Imager does not store
      your Wi-Fi password; it converts it into a 64-character key using the SSID, so a mistyped SSID
      produces a key that fails even though the password was right.
-   * Do not be surprised if the Pi still does not join the network - this has been seen on a plain
-     WPA2 network with correct details, and is quick to fix at the console with `nmtui` (step A2).
-     A WPA3 network cannot use Imager's derived key at all, so there expect to use `nmtui`.
+   * If the Pi does not join the network, a mistyped SSID or password is much the likeliest cause -
+     neither is echoed back to you here, and Imager combines the two into a key, so a slip in either
+     looks the same later. Step A2 fixes it in a couple of minutes with `nmtui`.
+   * A WPA3 network cannot use the key Imager derives at all, so on one of those expect to finish the
+     Wi-Fi setup with `nmtui` regardless.
 10. Enable SSH, either choose *Use password authentication* or paste your public key if you already use SSH keys.
 11. Leave Raspberry Pi Connect off for now - feel free to experiment with this, as we haven't yet. 
 12. Confirm that you want to save settings and write to the card, and click through operating system prompts wanting to stop you ! 
@@ -170,12 +179,12 @@ address does not change. Sensor nodes and phones then have something stable to t
 ```
 sudo apt install -y nodejs npm sqlite3 zsh
 node -v
-npm -v
 ```
 
-* `nodejs` — the server needs **Node 18 or later**. Raspberry Pi OS currently provides 20.19.2,
-  which is fine. If `node -v` ever reports something older on your image, install a current
-  version from [NodeSource](https://github.com/nodesource/distributions) instead.
+* `nodejs` — the server needs **Node 18 or later**. Current Raspberry Pi OS (Debian 13, trixie)
+  provides 20.19.2, which is fine. Older images shipped Node 18, also fine. If `node -v` reports
+  anything below 18, install a current version from
+  [NodeSource](https://github.com/nodesource/distributions) instead.
 * `npm` — installs the server; it is a separate package from `nodejs` on Debian.
 * `sqlite3` — the database the server keeps its accounts in.
 * `zsh` — the setup commands in step A6 are zsh scripts.
@@ -257,15 +266,22 @@ sudo cp extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf
 `/etc/mosquitto/mosquitto.conf`, which keeps its own settings for logging and persistence.)
 
 That configuration names a password file, and Mosquitto will not start if the file is missing, so
-create an empty one. The accounts inside it get created for you in the next step, by
-`addorganization.zsh` — which runs as you rather than as root, hence the ownership:
+create an empty one. The accounts inside it get created for you in the next step:
 
 ```
-sudo touch /etc/mosquitto/mosquitto_passwords
-sudo chown ${USER}:mosquitto /etc/mosquitto/mosquitto_passwords
-sudo chmod 640 /etc/mosquitto/mosquitto_passwords
+sudo install -o mosquitto -g mosquitto -m 600 /dev/null /var/lib/mosquitto/passwords
 ```
 
+That makes an empty file belonging to the `mosquitto` user, readable by nobody else — one command
+instead of a `touch`, a `chown` and a `chmod`.
+
+> The ownership matters, and is easy to get wrong. Mosquitto warns unless the password file belongs
+> to whoever opened it, and the broker runs as the `mosquitto` user — so the file belongs to
+> `mosquitto`, and it lives under `/var/lib/mosquitto` (which that user owns) rather than
+> `/etc/mosquitto` (which root owns). `mosquitto_passwd` also writes a temporary backup file
+> alongside it, so it needs to write to that directory too, not just to the file. Step A6 runs it as
+> the right user for you.
+
 Start the broker and have it start at every boot:
 
 ```
@@ -525,6 +541,16 @@ network and everything is correct — which is what you will see during this ins
 cannot: Raspberry Pi OS saves the time periodically and restores that value at boot, so after a
 power cut the Pi comes up believing it is whenever it last saved, and the gap never gets made up.
 
+To see whether the Pi's clock is actually right at any moment:
+
+```
+timedatectl
+```
+
+`System clock synchronized: yes` means it has reached a time server and the clock is trustworthy.
+`no`, with `NTP service: active`, means it is trying but has not succeeded — normal on a Pi with no
+route to the internet, and the point at which the timestamps below become a concern.
+
 What that affects:
 
 * **Logged data is stamped with the wrong time**, so graphs and history drift after each power cut.
@@ -539,6 +565,54 @@ but close enough for sensor data.
 
 ---
 
+## Upgrading
+
+To move an existing server to a newer release, from the directory you installed into:
+
+```
+cd ~/frugal-iot
+npm update frugal-iot-server
+npx frugal-iot-init
+sudo systemctl restart frugaliot
+```
+
+Taking those in turn:
+
+* `npm update` fetches the new version of the server, and of the client and logger it depends on.
+* `frugal-iot-init` adds any configuration file or directory the new version expects and you do not
+  have yet. It never changes files you already have, so your settings, database and data are safe.
+* `systemctl restart frugaliot` is what actually puts the new version into service — without it the
+  old one keeps running from memory, and nothing appears to have changed. (Skip this if you are
+  running the server by hand rather than as a service; stop it with `Ctrl-C` and start it again.)
+
+Check it came back up, and is running the version you expect:
+
+```
+systemctl status frugaliot
+journalctl -u frugaliot -n 30
+npm ls frugal-iot-server
+```
+
+`frugal-iot-init` leaves your files alone, so watch its output for a section headed **"These files
+were left as you have them, but this release ships a different version"**. It gives you the `diff`
+command for each. That matters most for the files you copy somewhere else — a changed
+`extras/mosquitto.conf` or `extras/frugaliot.service` does nothing until you install it again:
+
+```
+sudo cp extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf && sudo systemctl restart mosquitto
+sudo cp extras/frugaliot.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl restart frugaliot
+```
+
+> It does not compare `config.yaml`, `config.d/mqtt.yaml` or `config.d/server.yaml`, because those
+> hold your own settings and would differ every time. If a release note mentions a new setting in
+> one of them, compare it yourself:
+> `diff config.d/server.yaml node_modules/frugal-iot-server/config.d/server.yaml`
+
+Your organizations, accounts, database and logged data are untouched by an upgrade — they live in
+this directory, not in `node_modules`.
+
+---
+
 ## Part B — Raspberry Pi Zero W
 
 **Not yet tested — do not follow this section expecting it to work.** It records what is known
@@ -547,8 +621,9 @@ and what has to be checked.
 First, work out which board you have, because they are very different:
 
 * **Raspberry Pi Zero 2 W** — 64-bit ARM (Cortex-A53), 512 MB RAM. Should follow **Part A**
-  almost unchanged: choose *Raspberry Pi Zero 2 W* in Imager, and everything else applies.
-  The concern is memory, not architecture.
+  almost unchanged: choose *Raspberry Pi Zero 2 W* in Imager, and everything else applies. Running
+  the server needs about 250 MB, so 512 MB should be enough; the pinch point is more likely to be
+  `npm install`, which is far hungrier than the running server.
 * **Raspberry Pi Zero W** (the original) — 32-bit ARMv6 (BCM2835), 512 MB RAM. This is the
   awkward one: it needs the 32-bit Raspberry Pi OS, and **the official Node.js builds no longer
   support ARMv6**. Unofficial ARMv6 builds exist at
@@ -565,8 +640,9 @@ Differences to expect on either Zero:
 * Everything will be slow. `npm install` may take a long time, especially if `sqlite3` has to be
   compiled from source — which is likely on ARMv6, since no ready-made binaries are published.
 
-**Before this section can be written, I need to know:** which Zero you intend to support (original
-W, 2 W, or both), and the results of trying a Node install on it.
+**Both boards are to be tested**, one of each having been ordered for the purpose. This section gets
+written once we know how they behave — in particular whether a usable Node exists for the original
+Zero W's ARMv6.
 
 ---
 
@@ -584,20 +660,18 @@ they get settled.
 
 **Needs information I do not have**
 
-2. **Which Pi Zero** (Part B) — original Zero W, Zero 2 W, or both.
+2. **Pi Zero W and Zero 2 W** (Part B) — both boards are on order; Part B stays unwritten until they
+    have been tried. The open technical question is Node on the original Zero W's ARMv6.
 3. **Bridging to the shared server** — the local broker could optionally bridge to
     naturalinnovation.org so data also reaches the shared server. Not covered here; a later task.
 4. **Organization naming** — this guide sets up exactly one organization named `dev`, because that
     is the node firmware's default. Is that the right default for a farm installation, or should the
     guide encourage a meaningful organization id (which then has to be set in the node firmware too)?
-5. **Upgrading an existing installation** — `npm update frugal-iot-server` followed by
-    `npx frugal-iot-init` should be all it takes, since init adds missing configuration without
-    touching what is there. Not yet tried on a server that has been running for a while.
 
-**Tested on**
+## Tested on
 
-Fill this in as you go, so we know what the guide has actually been proven against:
+What the guide has actually been proven against — add a row for each run:
 
-| Date | Board | OS image | Node version | Result |
-| --- | --- | --- | --- | --- |
-|  | Pi 4 Model B |  |  |  |
+| Date | Board | OS image | Node | Server | Result |
+| --- | --- | --- | --- | --- | --- |
+| 2026-07-31 | Pi 4 Model B, 4 GB | Raspberry Pi OS Lite 64-bit, Debian 13.6 (trixie) | 20.19.2 and npm 9.2.0, both from `apt` | frugal-iot-server 0.3.3 from npm | Steps A1–A8 completed, plus an upgrade over the top of it. Wi-Fi from Imager did not connect and was fixed with `nmtui` (A2). `sqlite3` installed as a prebuilt binary, nothing compiled. Mosquitto started with an empty password file, and `frugal-iot-addorganization` wrote the broker account without sudo. Dashboard reached over `frugaliot.local` from a Mac (Chrome, after allowing local network access) and from an iPhone, MQTT status *connected*. A9 (sensor nodes) and A10 (HTTPS/OTA) not exercised. |

package/config.d/server.yaml

@@ -1,10 +1,10 @@
 port: 8080
 #Production mode
-#nodemodulesdir: ./node_modules
-#htmldir: ./node_modules/frugal-iot-client
+nodemodulesdir: ./node_modules
+htmldir: ./node_modules/frugal-iot-client
 # Development mode (uncomment these two, and comment out the two above, when working on the client)
-htmldir: ../frugal-iot-client
-nodemodulesdir: ../frugal-iot-client/node_modules
+#htmldir: ../frugal-iot-client
+#nodemodulesdir: ../frugal-iot-client/node_modules
 # The path to the private directory is relative to the server's working directory.
 privatedir: ./private
 publicdir: ./public

package/extras/mosquitto.conf

@@ -3,4 +3,8 @@ protocol mqtt
 listener 9012
 protocol websockets
 #http_dir /home/mitra/github_mitra42/frugal-iot/html
-password_file /etc/mosquitto/mosquitto_passwords
+# Kept under /var/lib/mosquitto, which belongs to the mosquitto user, rather than /etc/mosquitto,
+# which belongs to root. Both the broker and mosquitto_passwd warn unless the file belongs to
+# whoever opened it, and mosquitto_passwd also writes a temporary backup beside it, so the
+# directory has to be writable by that same user. See INSTALLATION.md step A5.
+password_file /var/lib/mosquitto/passwords

package/frugal-iot-server.js

@@ -30,6 +30,7 @@
  *Δ /admin (get) dashboard for administrators - includes OTA and will include permission management
  P /ota_update (post) protected place to upload new binaries (OTAUPDATE)
  XP /ota_list/:org list all ota files for an organization
+ XP /ota_get/:org/*remainingpath download a binary so the client can flash it over USB
  O  /private Serve up private files under authentication - currently unused
  * /register (post) register a new user
  */
@@ -920,6 +921,24 @@ mqttLogger.readYamlConfig('.', (err, configobj) => {
             });
           }
         );
+        // Download an OTA binary so the client can flash it over USB (see FLASH_PLAN.md in
+        // frugal-iot-client). Deliberately not /ota_update, which is unauthenticated for devices
+        // and answers 304 rather than sending bytes.
+        app.get('/ota_get/:org/*remainingpath',
+          loggedInOrFail,
+          can_OTAUPDATE,
+          (req,res) => {
+            let remainingpath = req.params.remainingpath.join('/');
+            let filepath = `${config.server.otadir}/${req.params.org}/${sanitize(remainingpath)}/firmware.bin`;
+            console.log("Sending OTA file", filepath);
+            res.sendFile(filepath, {}, (err) => {
+              if (err) {
+                console.error("Error sending ota file:", filepath, err);
+                if (!res.headersSent) res.status(404).send(err.message);
+              }
+            });
+          }
+        );
         app.get('/people_list/:org',
           loggedInOrFail,
           can_ADMIN, // Gets org from URL

package/package.json

@@ -12,7 +12,7 @@
     "express": "^5.2.1",
     "express-session": "^1.19.0",
     "frugal-iot-client": "^1.3.11",
-    "frugal-iot-logger": "^1.1.17",
+    "frugal-iot-logger": "^1.1.18",
     "hash-wasm": "^4.12.0",
     "morgan": "^1.11.0",
     "multer": "^2.1.1",
@@ -24,7 +24,8 @@
     "frugal-iot-server": "frugal-iot-server.js",
     "frugal-iot-init": "scripts/init.zsh",
     "frugal-iot-addorganization": "scripts/addorganization.zsh",
-    "frugal-iot-setpassword": "scripts/setpassword.zsh"
+    "frugal-iot-setpassword": "scripts/setpassword.zsh",
+    "frugal-iot-diagnostic": "scripts/diagnostic.zsh"
   },
   "description": "Frugal IoT client and server",
   "//files": "An allowlist, not an ignore list - anything not named here is never published, so a stray file (a database backup, a credential) cannot leak by default. Check with: npm pack --dry-run",
@@ -66,5 +67,5 @@
     "test:coverage": "vitest run --coverage"
   },
   "type": "module",
-  "version": "0.3.4"
+  "version": "0.3.5"
 }

package/scripts/addorganization.zsh

@@ -75,13 +75,50 @@ if ! command -v mosquitto_passwd >/dev/null; then
   MOSQUITTO_PASSWD_MISSING=1
 fi
 
-# Prefer the real system password file; if it's not present (e.g. mosquitto isn't installed on this
-# machine), fall back to editing the local copy under extras/ so the script still does something useful.
-MOSQUITTO_PASSWD_FILE="/etc/mosquitto/mosquitto_passwords"
+# Ask the broker's own configuration where its password file is, rather than guessing - that way
+# this works whatever layout a machine uses, including installations older than this script.
+# Mosquitto reads /etc/mosquitto/mosquitto.conf and then the .conf files in conf.d, last one wins.
+MOSQUITTO_PASSWD_FILE=$(grep -hE '^[[:space:]]*password_file[[:space:]]' \
+  /etc/mosquitto/mosquitto.conf /etc/mosquitto/conf.d/*.conf(N) 2>/dev/null | tail -1 | awk '{print $2}')
 USED_LOCAL_MOSQUITTO_FILE=0
-if [[ ! -e "$MOSQUITTO_PASSWD_FILE" && -e "extras/mosquitto_passwords" ]]; then
-  MOSQUITTO_PASSWD_FILE="extras/mosquitto_passwords"
-  USED_LOCAL_MOSQUITTO_FILE=1
+if [[ -n "$MOSQUITTO_PASSWD_FILE" ]]; then
+  MOSQUITTO_PASSWD_FROM_CONFIG=1
+else
+  MOSQUITTO_PASSWD_FROM_CONFIG=0
+  # No mosquitto configuration here - fall back to the usual locations, and then to the copy under
+  # extras/ so that a machine without mosquitto installed still gets something useful.
+  for f in /var/lib/mosquitto/passwords /etc/mosquitto/mosquitto_passwords; do
+    if [[ -e "$f" ]]; then
+      MOSQUITTO_PASSWD_FILE="$f"
+      break
+    fi
+  done
+  if [[ -z "$MOSQUITTO_PASSWD_FILE" ]]; then
+    if [[ -e "extras/mosquitto_passwords" ]]; then
+      MOSQUITTO_PASSWD_FILE="extras/mosquitto_passwords"
+      USED_LOCAL_MOSQUITTO_FILE=1
+    else
+      MOSQUITTO_PASSWD_FILE="/var/lib/mosquitto/passwords"
+    fi
+  fi
+fi
+
+# Run mosquitto_passwd as whoever owns that file. Mosquitto warns unless the file belongs to the user
+# opening it, and mosquitto_passwd writes a temporary backup beside it, so it needs the directory too
+# - neither of which we get by running as ourselves against a file owned by mosquitto or by root.
+# On Raspberry Pi OS the first user has passwordless sudo, so this is invisible.
+MOSQUITTO_PASSWD_CMD=(mosquitto_passwd)
+if [[ ! -w "${MOSQUITTO_PASSWD_FILE:h}" || ( -e "$MOSQUITTO_PASSWD_FILE" && ! -w "$MOSQUITTO_PASSWD_FILE" ) ]]; then
+  if command -v sudo >/dev/null; then
+    # stat's spelling differs between Linux (-c) and BSD/macOS (-f)
+    MOSQUITTO_PASSWD_OWNER=$(stat -c '%U' "$MOSQUITTO_PASSWD_FILE" 2>/dev/null \
+      || stat -f '%Su' "$MOSQUITTO_PASSWD_FILE" 2>/dev/null || true)
+    if [[ -n "$MOSQUITTO_PASSWD_OWNER" && "$MOSQUITTO_PASSWD_OWNER" != "$(id -un)" && "$MOSQUITTO_PASSWD_OWNER" != "root" ]]; then
+      MOSQUITTO_PASSWD_CMD=(sudo -u "$MOSQUITTO_PASSWD_OWNER" mosquitto_passwd)
+    else
+      MOSQUITTO_PASSWD_CMD=(sudo mosquitto_passwd)
+    fi
+  fi
 fi
 
 # ---- Escaping helpers ----
@@ -168,17 +205,24 @@ MOSQUITTO_PASSWD_FAILED=0
 MOSQUITTO_PASSWD_ERROR=""
 if [[ "$MOSQUITTO_PASSWD_MISSING" -eq 1 ]]; then
   echo "Warning: mosquitto_passwd command not found - skipped setting MQTT broker password" >&2
+elif [[ "$MOSQUITTO_PASSWD_FROM_CONFIG" -eq 1 && ! -e "$MOSQUITTO_PASSWD_FILE" ]]; then
+  # Nothing to add an account to - and mosquitto will not be running either, for the same reason.
+  echo "Warning: mosquitto is configured to use ${MOSQUITTO_PASSWD_FILE}, but that file does not exist," >&2
+  echo "so the MQTT broker password was NOT set. Create it, then add this organization to it:" >&2
+  echo "  sudo install -o mosquitto -g mosquitto -m 600 /dev/null \"${MOSQUITTO_PASSWD_FILE}\"" >&2
+  echo "  sudo -u mosquitto mosquitto_passwd -b \"${MOSQUITTO_PASSWD_FILE}\" \"${ORG_ID}\" \"${PASSWORD}\"" >&2
+  MOSQUITTO_PASSWD_FAILED=1
 else
   # Capture stderr without letting `set -e` abort the whole script on failure - the org's config file
   # and DB rows are already written by this point, so a failure here should be reported, not fatal.
-  MOSQUITTO_PASSWD_ERROR=$(mosquitto_passwd -b "$MOSQUITTO_PASSWD_FILE" "${ORG_ID}" "${PASSWORD}" 2>&1 >/dev/null) || MOSQUITTO_PASSWD_FAILED=1
+  MOSQUITTO_PASSWD_ERROR=$("${MOSQUITTO_PASSWD_CMD[@]}" -b "$MOSQUITTO_PASSWD_FILE" "${ORG_ID}" "${PASSWORD}" 2>&1 >/dev/null) || MOSQUITTO_PASSWD_FAILED=1
   if [[ "$MOSQUITTO_PASSWD_FAILED" -eq 1 ]]; then
     echo "Warning: mosquitto_passwd failed to set the MQTT broker password:" >&2
     echo "  ${MOSQUITTO_PASSWD_ERROR}" >&2
-    if [[ "$MOSQUITTO_PASSWD_ERROR" == *"Permission denied"* || ! -w "$MOSQUITTO_PASSWD_FILE" ]]; then
-      echo "This looks like a permissions problem - re-run that command with sudo:" >&2
-      echo "  sudo mosquitto_passwd -b \"${MOSQUITTO_PASSWD_FILE}\" \"${ORG_ID}\" \"${PASSWORD}\"" >&2
-    fi
+    echo "Everything else was done, so set just the broker password by hand:" >&2
+    echo "  sudo mosquitto_passwd -b \"${MOSQUITTO_PASSWD_FILE}\" \"${ORG_ID}\" \"${PASSWORD}\"" >&2
+    echo "If it complains about creating a backup file, the directory holding that file is not" >&2
+    echo "writable by you - which is why the command above uses sudo." >&2
   else
     echo "Set mosquitto password for ${ORG_ID} in ${MOSQUITTO_PASSWD_FILE}"
   fi

package/scripts/diagnostic.zsh

@@ -0,0 +1,289 @@
+#!/usr/bin/env zsh
+#
+# Report the state of a Frugal IoT installation, for working out why something in INSTALLATION.md
+# did not do what it said it would. It only looks - it changes nothing.
+#
+# Run it from the directory the server is installed in (the one holding frugal-iot.db):
+#   zsh diagnostic.zsh
+# or, once installed from npm:
+#   npx frugal-iot-diagnostic
+#
+# Paste the whole output into a bug report. Passwords are deliberately not printed: the password
+# file is reported by account name only, and organization configs by name only.
+
+# No "set -e": a failing check should report and carry on, not stop the script.
+setopt no_unset 2>/dev/null
+
+section() { print -r -- ""; print -r -- "===== $* ====="; }
+item()    { print -r -- "  $*"; }
+have()    { command -v "$1" >/dev/null 2>&1; }
+# Ownership and mode of a file, spelled for either Linux or BSD/macOS stat
+fileinfo() {
+  local f=$1
+  if [[ ! -e "$f" ]]; then print -r -- "missing"; return; fi
+  stat -c '%U:%G %a %s bytes' "$f" 2>/dev/null || stat -f '%Su:%Sg %Lp %z bytes' "$f" 2>/dev/null || print -r -- "(cannot stat)"
+}
+
+PROBLEMS=()
+problem() { PROBLEMS+=("$1"); }
+
+section "When and where"
+item "date:        $(date 2>/dev/null)"
+item "host:        $(hostname 2>/dev/null)"
+item "user:        $(id -un 2>/dev/null) (groups: $(id -Gn 2>/dev/null))"
+item "directory:   $PWD"
+
+section "The machine"
+have uname && item "kernel:      $(uname -srm)"
+[[ -r /etc/os-release ]] && item "os:          $(grep PRETTY_NAME /etc/os-release | cut -d'"' -f2)"
+[[ -r /proc/device-tree/model ]] && item "board:       $(tr -d '\0' < /proc/device-tree/model)"
+have free && item "memory:      $(free -h | awk '/^Mem:/ {print $3 " used of " $2 ", " $7 " available"}')"
+have df && item "disk (/):    $(df -h / | awk 'NR==2 {print $3 " used of " $2 ", " $4 " free"}')"
+if have timedatectl; then
+  item "clock:       $(timedatectl show -p NTPSynchronized --value 2>/dev/null | sed 's/^yes$/synchronized with a time server/; s/^no$/NOT synchronized - timestamps may be wrong/')"
+fi
+
+section "Software versions"
+for c in node npm sqlite3 zsh mosquitto mosquitto_passwd mosquitto_sub; do
+  if have $c; then
+    case $c in
+      node|npm)            item "$c: $($c -v 2>&1 | head -1)" ;;
+      sqlite3)             item "$c: $(sqlite3 --version 2>&1 | awk '{print $1}')" ;;
+      zsh)                 item "$c: $ZSH_VERSION" ;;
+      mosquitto)           item "$c: $(mosquitto -h 2>&1 | head -1)" ;;
+      *)                   item "$c: present" ;;
+    esac
+  else
+    item "$c: NOT INSTALLED"
+    [[ $c == (node|npm|sqlite3) ]] && problem "$c is not installed - see INSTALLATION.md step A3"
+  fi
+done
+
+section "This installation"
+for f in config.yaml config.d frugal-iot.db data ota node_modules; do
+  if [[ -e $f ]]; then item "$f: present"; else item "$f: MISSING"; fi
+done
+[[ ! -e frugal-iot.db ]] && problem "No frugal-iot.db here - either the wrong directory, or 'npx frugal-iot-init' has not been run"
+if [[ -d node_modules/frugal-iot-server ]]; then
+  item "frugal-iot-server: $(node -e 'console.log(require("./node_modules/frugal-iot-server/package.json").version)' 2>/dev/null)"
+  item "frugal-iot-client: $(node -e 'console.log(require("./node_modules/frugal-iot-client/package.json").version)' 2>/dev/null)"
+  item "frugal-iot-logger: $(node -e 'console.log(require("./node_modules/frugal-iot-logger/package.json").version)' 2>/dev/null)"
+  item "commands:    $(ls node_modules/.bin 2>/dev/null | grep frugal | tr '\n' ' ')"
+else
+  item "frugal-iot-server is not installed in this directory"
+fi
+if [[ -f config.d/mqtt.yaml ]]; then item "broker configured as: $(grep -h '^broker:' config.d/mqtt.yaml 2>/dev/null)"; fi
+if [[ -d config.d/organizations ]]; then
+  # Names only - these files contain the broker password.
+  # (N) makes a non-matching glob expand to nothing rather than erroring - and the list has to be
+  # tested before use, because "ls" with no arguments would list the current directory instead.
+  ORGS=(config.d/organizations/*.yaml(N:t:r))
+  if (( ${#ORGS} )); then
+    item "organizations: ${ORGS}"
+  else
+    item "organizations: none defined yet"
+  fi
+fi
+if [[ -f frugal-iot.db ]] && have sqlite3; then
+  item "accounts:    $(sqlite3 frugal-iot.db 'SELECT group_concat(username, ", ") FROM users;' 2>/dev/null)"
+fi
+
+section "Mosquitto configuration"
+if [[ -f /etc/mosquitto/mosquitto.conf ]]; then
+  item "/etc/mosquitto/mosquitto.conf: $(fileinfo /etc/mosquitto/mosquitto.conf)"
+else
+  item "/etc/mosquitto/mosquitto.conf: missing - is mosquitto installed?"
+fi
+item "files in /etc/mosquitto/conf.d:"
+for f in /etc/mosquitto/conf.d/*(N); do
+  item "  ${f}: $(fileinfo $f)"
+  grep -nE '^[[:space:]]*(listener|protocol|password_file|allow_anonymous)' $f 2>/dev/null | sed 's/^/      /'
+done
+[[ -z "$(print -r -- /etc/mosquitto/conf.d/*(N))" ]] && problem "Nothing in /etc/mosquitto/conf.d - the Frugal IoT config was never copied there (step A5)"
+
+# Which password file does the running configuration actually name?
+# mosquitto's include_dir only loads files ending in .conf, so look at exactly those, last one wins
+CONFIGURED_PWFILE=$(grep -hE '^[[:space:]]*password_file[[:space:]]' /etc/mosquitto/mosquitto.conf /etc/mosquitto/conf.d/*.conf(N) 2>/dev/null | tail -1 | awk '{print $2}')
+section "Mosquitto password file"
+if [[ -n "$CONFIGURED_PWFILE" ]]; then
+  item "configuration names: $CONFIGURED_PWFILE"
+  item "  that file:         $(fileinfo $CONFIGURED_PWFILE)"
+  if [[ ! -e "$CONFIGURED_PWFILE" ]]; then
+    problem "Mosquitto is configured to use $CONFIGURED_PWFILE but that file does not exist - it will refuse to start"
+  else
+    # Account names only, never the hashes
+    item "  accounts in it:    $(cut -d: -f1 "$CONFIGURED_PWFILE" 2>/dev/null | tr '\n' ' ')"
+    [[ ! -s "$CONFIGURED_PWFILE" ]] && item "  (the file is empty - no accounts yet)"
+  fi
+else
+  item "no password_file line found in any mosquitto configuration"
+  problem "No password_file configured - the broker would allow anonymous access or reject everything"
+fi
+# The copy in this installation is what step A5 tells you to put into /etc - if the two disagree,
+# either the copy never happened or the installed package is older than the instructions.
+if [[ -f extras/mosquitto.conf ]]; then
+  item "this installation's extras/mosquitto.conf names: $(grep -hE '^[[:space:]]*password_file' extras/mosquitto.conf 2>/dev/null | awk '{print $2}')"
+  if [[ -n "$CONFIGURED_PWFILE" ]] && ! diff -q extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf >/dev/null 2>&1; then
+    problem "extras/mosquitto.conf here differs from /etc/mosquitto/conf.d/frugal-iot.conf - if you followed step A5 with an older installed package, copy it again after upgrading"
+  fi
+fi
+item "candidate locations, whether or not configured:"
+for f in /var/lib/mosquitto/passwords /etc/mosquitto/mosquitto_passwords; do
+  item "  ${f}: $(fileinfo $f)"
+done
+item "directories they live in:"
+for d in /var/lib/mosquitto /etc/mosquitto; do
+  item "  ${d}: $(fileinfo $d)"
+done
+
+section "Mosquitto service"
+if have systemctl; then
+  item "enabled:     $(systemctl is-enabled mosquitto 2>&1)"
+  item "active:      $(systemctl is-active mosquitto 2>&1)"
+  [[ "$(systemctl is-active mosquitto 2>/dev/null)" != "active" ]] && problem "Mosquitto is not running - see its own log, quoted above under 'Mosquitto service'"
+  print -r -- "  --- last 12 journal lines (systemd's view: usually only an exit code) ---"
+  journalctl -u mosquitto -n 12 --no-pager 2>&1 | sed 's/^/      /'
+else
+  item "systemctl not available on this machine"
+fi
+# Debian's mosquitto logs to a file rather than the journal, so this is where the real reason is.
+MOSQUITTO_LOG=$(grep -hE '^[[:space:]]*log_dest[[:space:]]+file' /etc/mosquitto/mosquitto.conf /etc/mosquitto/conf.d/*(N) 2>/dev/null | tail -1 | awk '{print $3}')
+[[ -z "$MOSQUITTO_LOG" ]] && MOSQUITTO_LOG=/var/log/mosquitto/mosquitto.log
+item "its own log file: ${MOSQUITTO_LOG} ($(fileinfo $MOSQUITTO_LOG))"
+if [[ -r "$MOSQUITTO_LOG" ]]; then
+  print -r -- "  --- last 12 lines of that log (this is where startup errors appear) ---"
+  tail -12 "$MOSQUITTO_LOG" 2>&1 | sed 's/^/      /'
+elif [[ -e "$MOSQUITTO_LOG" ]]; then
+  # The log belongs to the mosquitto user, so reading it needs privilege. Try without prompting -
+  # on Raspberry Pi OS the first user has passwordless sudo, so this usually just works.
+  if have sudo && MOSQUITTO_LOG_TAIL=$(sudo -n tail -12 "$MOSQUITTO_LOG" 2>/dev/null); then
+    print -r -- "  --- last 12 lines of that log, read with sudo (startup errors appear here) ---"
+    print -r -- "$MOSQUITTO_LOG_TAIL" | sed 's/^/      /'
+  else
+    item "  it belongs to $(fileinfo $MOSQUITTO_LOG | awk '{print $1}') so it cannot be read as $(id -un)."
+    item "  Re-run this whole script with sudo to include it:  sudo zsh $0"
+    problem "Could not read ${MOSQUITTO_LOG}, which is where Mosquitto explains itself - re-run with sudo"
+  fi
+fi
+
+section "Listening ports"
+PORTS=""
+if have ss; then
+  PORTS=$(ss -tln 2>/dev/null | grep -E ':(1883|9012|8080)\b')
+elif have netstat; then
+  PORTS=$(netstat -an 2>/dev/null | grep -E '[.:](1883|9012|8080) ')
+else
+  item "neither ss nor netstat available"
+fi
+if [[ -n "$PORTS" ]]; then
+  print -r -- "$PORTS" | sed 's/^/  /'
+else
+  item "none of 1883 (mqtt), 9012 (websockets), 8080 (web) are listening"
+fi
+
+section "Frugal IoT server service"
+if have systemctl; then
+  if systemctl list-unit-files 2>/dev/null | grep -q '^frugaliot.service'; then
+    item "enabled:     $(systemctl is-enabled frugaliot 2>&1)"
+    item "active:      $(systemctl is-active frugaliot 2>&1)"
+    print -r -- "  --- last 20 journal lines ---"
+    journalctl -u frugaliot -n 20 --no-pager 2>&1 | sed 's/^/      /'
+  else
+    item "no frugaliot service installed yet (step A8 not reached)"
+  fi
+else
+  item "systemctl not available on this machine"
+fi
+
+section "Name resolution (step A2, A4)"
+# The broker URL in config.d/mqtt.yaml has to resolve from every machine that uses it - this one,
+# and whatever browser or node talks to it.
+BROKER_URL=$(grep -h '^broker:' config.d/mqtt.yaml 2>/dev/null | awk '{print $2}')
+BROKER_HOST=${${BROKER_URL#*://}%%:*}
+if [[ -n "$BROKER_HOST" ]]; then
+  item "broker host: $BROKER_HOST"
+  if have getent && BROKER_IP=$(getent hosts "$BROKER_HOST" 2>/dev/null | head -1 | awk '{print $1}') && [[ -n "$BROKER_IP" ]]; then
+    item "  resolves to $BROKER_IP from this machine"
+  elif have ping && ping -c1 -W2 "$BROKER_HOST" >/dev/null 2>&1; then
+    item "  answers ping from this machine"
+  else
+    item "  DOES NOT RESOLVE from this machine"
+    problem "The broker host '$BROKER_HOST' does not resolve here - the server's own logger cannot connect either"
+  fi
+  [[ "$BROKER_HOST" == *.local ]] && item "  note: .local names do not resolve on most Android phones - use an IP address there"
+fi
+MYADDRS=$( (have ip && ip -4 -o addr show scope global | awk '{print $2"="$4}') 2>/dev/null | tr '\n' ' ')
+item "this machine's addresses: ${MYADDRS:-(could not determine)}"
+
+section "Broker authentication (steps A5, A6)"
+if have mosquitto_sub; then
+  # A wrong password must be refused - that is the check in step A5
+  WRONGOUT=$(mosquitto_sub -h localhost -u nobody -P wrong -t '#' -W 2 2>&1)
+  if [[ "$WRONGOUT" == *"not authorised"* ]]; then
+    item "wrong password: correctly refused"
+  elif [[ "$WRONGOUT" == *"Connection refused"* || "$WRONGOUT" == *"Error"* ]]; then
+    item "wrong password: broker did not answer - $WRONGOUT"
+    problem "The broker is not answering on localhost:1883"
+  else
+    item "wrong password: ACCEPTED - the broker is not requiring credentials"
+    problem "A wrong password was not refused - check password_file is set in the mosquitto config"
+  fi
+  # Then each configured organization should be able to connect with its own credentials.
+  # The password is read from the config and used, never printed.
+  for f in config.d/organizations/*.yaml(N); do
+    ORG=${f:t:r}
+    ORG_PW=$(sed -n 's/^mqtt_password:[[:space:]]*//p' "$f" 2>/dev/null | head -1 | tr -d '"'"'"'')
+    if [[ -z "$ORG_PW" ]]; then
+      item "organization $ORG: no mqtt_password in $f"
+      continue
+    fi
+    ORGOUT=$(mosquitto_sub -h localhost -u "$ORG" -P "$ORG_PW" -t '#' -W 2 2>&1)
+    if [[ "$ORGOUT" == *"not authorised"* ]]; then
+      item "organization $ORG: REFUSED by the broker"
+      problem "Organization '$ORG' cannot log in to the broker - its password in $f does not match the broker's password file (step A6)"
+    elif [[ "$ORGOUT" == *"Connection refused"* ]]; then
+      item "organization $ORG: broker not answering"
+    else
+      item "organization $ORG: authenticates, and saw $(print -r -- "$ORGOUT" | grep -c . ) message(s) in a 2 second sample"
+    fi
+  done
+else
+  item "mosquitto_sub not installed - cannot test broker logins (sudo apt install mosquitto-clients)"
+fi
+
+section "Web server (step A7)"
+WEBPORT=$(grep -h '^port:' config.d/server.yaml 2>/dev/null | awk '{print $2}')
+[[ -z "$WEBPORT" ]] && WEBPORT=8080
+if have curl; then
+  HOMECODE=$(curl -s -o /dev/null -w '%{http_code}' -m 5 http://localhost:${WEBPORT}/ 2>/dev/null)
+  if [[ "$HOMECODE" == "000" ]]; then
+    item "nothing is answering on port ${WEBPORT} - the server is not running"
+    item "  (expected if you have not reached step A7 yet; otherwise start it, or see the frugaliot service below)"
+  else
+    item "GET /            -> ${HOMECODE} (expect 200)"
+    item "GET /config.json -> $(curl -s -o /dev/null -w '%{http_code}' -m 5 http://localhost:${WEBPORT}/config.json 2>/dev/null) (expect 401 when not logged in)"
+    [[ "$HOMECODE" != "200" ]] && problem "The web server answered ${HOMECODE} rather than 200 on port ${WEBPORT}"
+  fi
+else
+  item "curl not installed - cannot test the web server"
+fi
+
+section "Logged data (step A9)"
+if [[ -d data ]]; then
+  DATADIRS=(data/*(N/))
+  if (( ${#DATADIRS} )); then
+    for d in $DATADIRS; do
+      item "${d}: $(find $d -type f 2>/dev/null | wc -l | tr -d ' ') files, newest $(ls -t $d/**/*(N.om[1]) 2>/dev/null | head -1)"
+    done
+  else
+    item "data/ is empty - no readings logged yet, which is expected until a node reports (step A9)"
+  fi
+fi
+
+section "Summary"
+if (( ${#PROBLEMS} == 0 )); then
+  item "No problems detected by these checks."
+else
+  for p in $PROBLEMS; do item "PROBLEM: $p"; done
+fi
+print -r -- ""

package/scripts/init.zsh

@@ -28,11 +28,24 @@ if [[ ! -f "${PKG}/frugal-iot-createdb.sql" ]]; then
 fi
 
 # ---- 1. Configuration files, copied only if not already here ----
-COPIED=()   # which files this run actually created, so existing ones are never rewritten below
+COPIED=()      # which files this run actually created, so existing ones are never rewritten below
+DIFFER_TO=()   # files kept that no longer match what this release ships, and where to compare them
+DIFFER_FROM=()
+# copy_if_missing <packaged file> <local file> [compare]
+# "compare" asks for a warning when the file exists but differs from the one this release ships.
+# Only pass it for files nobody is expected to edit - config.yaml, mqtt.yaml and server.yaml are
+# this installation's own settings, so of course they differ, and warning about them every time
+# would train you to ignore the warning that matters.
 copy_if_missing() {
-  local from=$1 to=$2
+  local from=$1 to=$2 compare=${3:-}
   if [[ -e "$to" ]]; then
-    echo "  kept    ${to} (already present)"
+    if [[ "$compare" == compare && -e "$from" ]] && ! cmp -s "$from" "$to"; then
+      DIFFER_TO+=("$to")
+      DIFFER_FROM+=("$from")
+      echo "  kept    ${to} (DIFFERS from this release - see below)"
+    else
+      echo "  kept    ${to} (already present)"
+    fi
   elif [[ -e "$from" ]]; then
     mkdir -p "${to:h}"
     cp "$from" "$to"
@@ -46,12 +59,14 @@ copy_if_missing "${PKG}/config.yaml" "./config.yaml"
 for f in logger.yaml mqtt.yaml server.yaml; do
   copy_if_missing "${PKG}/config.d/${f}" "./config.d/${f}"
 done
+# The schema describes the sensor types the software understands, so a release changing it matters
 for f in "${PKG}"/config.d/schema/*.yaml(N); do
-  copy_if_missing "$f" "./config.d/schema/${f:t}"
+  copy_if_missing "$f" "./config.d/schema/${f:t}" compare
 done
-# Copied so that mosquitto.conf and frugaliot.service can be edited and installed from here
+# Copied so that mosquitto.conf and frugaliot.service can be edited and installed from here. Worth
+# comparing: these get copied on somewhere else (/etc/...), where an old version lingers unnoticed.
 for f in "${PKG}"/extras/*(N); do
-  copy_if_missing "$f" "./extras/${f:t}"
+  copy_if_missing "$f" "./extras/${f:t}" compare
 done
 
 # ---- 1a. Point server.yaml at the web client this instance actually has ----
@@ -104,6 +119,18 @@ else
   echo "  created ${DB}"
 fi
 
+if (( ${#DIFFER_TO} )); then
+  echo
+  echo "These files were left as you have them, but this release ships a different version:"
+  for i in {1..${#DIFFER_TO}}; do
+    echo "  ${DIFFER_TO[$i]}"
+    echo "    compare with:  diff ${DIFFER_TO[$i]} ${DIFFER_FROM[$i]}"
+  done
+  echo "Usually that just means you edited it, and there is nothing to do. But a release can also"
+  echo "change one of these files - and anything installed elsewhere from it, such as"
+  echo "/etc/mosquitto/conf.d/frugal-iot.conf, keeps the old content until you copy it again."
+fi
+
 echo
 echo "Ready. Next:"
 echo "  npx frugal-iot-setpassword superuser <password>            # so you can log in as the administrator"