frugal-iot-server 0.3.3 → 0.3.5

package/INSTALLATION.md

@@ -13,9 +13,13 @@ Two hardware paths are covered:
 If you already have a working Linux server (not a Pi), you do not need this document —
 see [README.md](https://github.com/mitra42/frugal-iot-server/blob/main/README.md) instead.
 
-> **Status:** first draft, written 2026-07. The Pi 4 path has been written against the current
-> code but not yet walked through end to end on hardware. Items still to confirm are collected
-> under [Open questions](#open-questions) — please add your findings there as you go.
+> **Status:** the Raspberry Pi 4 path has been followed end to end on hardware (see
+> [Tested on](#tested-on)), and corrected from what that run found. Part B, and HTTPS/OTA in step
+> A10, are not written yet. Remaining uncertainties are listed under
+> [Open questions](#open-questions) — please add your findings there as you go.
+
+**Already have a Frugal IoT server running and just want a newer version?** Skip everything below
+and go to [Upgrading](#upgrading).
 
 ---
 
@@ -25,8 +29,11 @@ see [README.md](https://github.com/mitra42/frugal-iot-server/blob/main/README.md
 
 **Hardware**
 
-* Raspberry Pi 4 Model B. 2 GB RAM or more is comfortable; 1 GB should work.
-* A microSD card, 16 GB or larger, Class 10 / A1 or better. (8 GB works but leaves little room for data.)
+* Raspberry Pi 4 Model B. Any RAM size — a running server uses around 250 MB, so even the 1 GB
+  model has plenty of room. (Measured on a 4 GB Pi 4: 231 MB in use with the server running.)
+* A microSD card. 8 GB is enough for the system and software — a working install occupies about
+  5 GB — but sensor data accumulates on this card for as long as the server runs, so 16 GB or larger
+  is the safer choice. Class 10 / A1 or better.
 * The official Raspberry Pi USB-C power supply (5 V / 3 A). Phone chargers frequently cause
   random reboots and corrupted SD cards — this is the single most common cause of "it doesn't work".
 * A way to write the SD card from your laptop: a built-in SD slot or a USB card reader.
@@ -48,10 +55,10 @@ see [README.md](https://github.com/mitra42/frugal-iot-server/blob/main/README.md
 
 **Have these to hand in case the headless setup does not come up**
 
-* A micro-HDMI to HDMI cable plus a monitor, and a USB keyboard — lets you log in directly, see boot
-  errors, and finish Wi-Fi setup by hand. Wi-Fi configured through Imager does not always connect on
-  the first boot (see step A2), so do not count on not needing these.
-* An Ethernet cable from the Pi to your router — bypasses all Wi-Fi problems.
+* An Ethernet cable from the Pi to your router — the simplest way in if the Pi does not appear on
+  the Wi-Fi, and you can then sort the Wi-Fi out over SSH (step A2).
+* A micro-HDMI to HDMI cable plus a monitor, and a USB keyboard — for when the Pi is nowhere near the
+  router, or you want to see boot messages.
 
 ### A1. Write the operating system to the SD card
 
@@ -72,9 +79,11 @@ a desktop would only consume memory and SD card space.
    * Type the SSID exactly as the network broadcasts it, including capitals. Imager does not store
      your Wi-Fi password; it converts it into a 64-character key using the SSID, so a mistyped SSID
      produces a key that fails even though the password was right.
-   * Do not be surprised if the Pi still does not join the network - this has been seen on a plain
-     WPA2 network with correct details, and is quick to fix at the console with `nmtui` (step A2).
-     A WPA3 network cannot use Imager's derived key at all, so there expect to use `nmtui`.
+   * If the Pi does not join the network, a mistyped SSID or password is much the likeliest cause -
+     neither is echoed back to you here, and Imager combines the two into a key, so a slip in either
+     looks the same later. Step A2 fixes it in a couple of minutes with `nmtui`.
+   * A WPA3 network cannot use the key Imager derives at all, so on one of those expect to finish the
+     Wi-Fi setup with `nmtui` regardless.
 10. Enable SSH, either choose *Use password authentication* or paste your public key if you already use SSH keys.
 11. Leave Raspberry Pi Connect off for now - feel free to experiment with this, as we haven't yet. 
 12. Confirm that you want to save settings and write to the card, and click through operating system prompts wanting to stop you ! 
@@ -101,8 +110,14 @@ Say `yes` to the fingerprint question, then give the password you set in Imager.
 * Wait another minute and try again — the Pi may still be on its first boot.
 * Log in to your Wi-Fi router's admin page and look for a device called `frugaliot` in its
   list of connected clients; note its IP address and use that instead: `ssh pi@192.168.1.42`.
+* **Plug an Ethernet cable from the Pi into your router.** Nothing needs configuring — the Pi picks
+  up an address, and `ssh pi@frugaliot.local` then works over the cable. This is the least effort
+  way in if the Pi is within reach of the router, and once you are logged in you can sort the Wi-Fi
+  out over SSH with `nmtui` as below, no monitor or keyboard needed. Unplug the cable afterwards and
+  check that Wi-Fi alone still gets you in.
 * Plug in the HDMI and keyboard, log in at the console, and run `ip addr` to read the IP address,
-  and `sudo journalctl -b | grep -i wpa` to see why Wi-Fi failed.
+  and `sudo journalctl -b | grep -i wpa` to see why Wi-Fi failed. Use this when the Pi is nowhere
+  near the router, or when you want to see boot messages.
 
 **If Wi-Fi did not connect at all** — `ip addr` shows no address on `wlan0`, and the journal has
 `WPA: 4-Way Handshake failed - pre-shared key may be incorrect` — then the Pi found your network
@@ -145,20 +160,31 @@ sudo reboot
 
 Wait a minute, then `ssh pi@frugaliot.local` again.
 
-**Recommended:** in your router, give the Pi a fixed (reserved) IP address. Sensor nodes and
-phones then have a stable address to talk to even where `.local` names do not work.
+**Find the Pi's IP address**, because later steps can need it — for the broker URL if you will view
+the dashboard on a phone, and for your sensor nodes. From your laptop:
+
+```
+ping -c1 frugaliot.local
+```
+
+The address it prints is the Pi's. That is easier than hunting through your router's admin pages,
+which you may not have the password for.
+
+**Recommended:** if you can get into your router, give the Pi a fixed (reserved) IP address, so that
+address does not change. Sensor nodes and phones then have something stable to talk to even where
+`.local` names do not work.
 
 ### A3. Install the prerequisites
 
 ```
 sudo apt install -y nodejs npm sqlite3 zsh
 node -v
-npm -v
 ```
 
-* `nodejs` — the server needs **Node 18 or later**. Raspberry Pi OS currently provides 20.19.2,
-  which is fine. If `node -v` ever reports something older on your image, install a current
-  version from [NodeSource](https://github.com/nodesource/distributions) instead.
+* `nodejs` — the server needs **Node 18 or later**. Current Raspberry Pi OS (Debian 13, trixie)
+  provides 20.19.2, which is fine. Older images shipped Node 18, also fine. If `node -v` reports
+  anything below 18, install a current version from
+  [NodeSource](https://github.com/nodesource/distributions) instead.
 * `npm` — installs the server; it is a separate package from `nodejs` on Debian.
 * `sqlite3` — the database the server keeps its accounts in.
 * `zsh` — the setup commands in step A6 are zsh scripts.
@@ -186,9 +212,10 @@ This copies in the configuration files, creates the `data`, `ota` and `config.d/
 directories, and creates the database. It never overwrites anything already there, so it is also
 what you run after an upgrade to pick up newly added configuration.
 
-> If `npm install` fails while building `sqlite3`, it could not find a ready-made binary for this
-> platform and needs to compile one. Install the compiler toolchain and try again:
-> `sudo apt install -y build-essential python3` then `npm install frugal-iot-server`.
+> On a 64-bit Raspberry Pi OS this installs ready-made binaries and compiles nothing. If it ever
+> does stop while building `sqlite3`, no binary was available for your platform, so install the
+> compiler toolchain and try again: `sudo apt install -y build-essential python3` then
+> `npm install frugal-iot-server`.
 
 Everything from here on is run from `~/frugal-iot`, and `npx` is how you run the server's commands
 without having to know where npm put them.
@@ -210,9 +237,11 @@ This one URL is used both by the server's own logger and by the browser UI, so i
 WebSocket (`ws://`) address that **your phone or laptop browser** can reach, not just one that
 works on the Pi.
 
-> Many Android phones cannot resolve `.local` names. If you will view the dashboard from a phone,
-> use the Pi's IP address instead — for example `broker: ws://192.168.1.42:9012` — and give the Pi
-> a reserved address in your router so it does not change.
+> `frugaliot.local` is known to work from a laptop and from an iPhone. Android phones generally
+> cannot resolve `.local` names, so if you will view the dashboard on Android, put the Pi's IP
+> address here instead — `broker: ws://192.168.1.42:9012`, using the address `ping -c1
+> frugaliot.local` reported in step A2 — and reserve that address in your router if you can, so it
+> does not change under you.
 
 ### A5. Install and configure the MQTT broker (Mosquitto)
 
@@ -237,15 +266,22 @@ sudo cp extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf
 `/etc/mosquitto/mosquitto.conf`, which keeps its own settings for logging and persistence.)
 
 That configuration names a password file, and Mosquitto will not start if the file is missing, so
-create an empty one. The accounts inside it get created for you in the next step, by
-`addorganization.zsh` — which runs as you rather than as root, hence the ownership:
+create an empty one. The accounts inside it get created for you in the next step:
 
 ```
-sudo touch /etc/mosquitto/mosquitto_passwords
-sudo chown ${USER}:mosquitto /etc/mosquitto/mosquitto_passwords
-sudo chmod 640 /etc/mosquitto/mosquitto_passwords
+sudo install -o mosquitto -g mosquitto -m 600 /dev/null /var/lib/mosquitto/passwords
 ```
 
+That makes an empty file belonging to the `mosquitto` user, readable by nobody else — one command
+instead of a `touch`, a `chown` and a `chmod`.
+
+> The ownership matters, and is easy to get wrong. Mosquitto warns unless the password file belongs
+> to whoever opened it, and the broker runs as the `mosquitto` user — so the file belongs to
+> `mosquitto`, and it lives under `/var/lib/mosquitto` (which that user owns) rather than
+> `/etc/mosquitto` (which root owns). `mosquitto_passwd` also writes a temporary backup file
+> alongside it, so it needs to write to that directory too, not just to the file. Step A6 runs it as
+> the right user for you.
+
 Start the broker and have it start at every boot:
 
 ```
@@ -254,7 +290,8 @@ sudo systemctl restart mosquitto
 systemctl status mosquitto
 ```
 
-`systemctl status` should say `active (running)`. Press `q` to exit. If it is not running,
+`systemctl status` should say `active (running)`, and returns you to the prompt. (If the output is
+long enough that it opens a pager instead, `q` gets you out.) If it is not running,
 `sudo journalctl -u mosquitto -n 50` will show what it objected to.
 
 **Check both listeners are open:**
@@ -349,45 +386,71 @@ If you get `Connection Refused: not authorised`, the password does not match the
 
 ### A7. Start the server by hand and check it
 
+From your install directory — `npx` looks for the server in the current directory's `node_modules`,
+so this only works there:
+
 ```
+cd ~/frugal-iot
 npx frugal-iot-server
 ```
 
-You should see the configuration echoed back, then something like:
+> If npx answers with `Need to install the following packages: frugal-iot-server` and asks to
+> continue, you are in the wrong directory. Say no, `cd ~/frugal-iot`, and try again — otherwise
+> npx fetches a throwaway copy that has none of your configuration.
+
+It lists each configuration file as it reads it, then:
 
 ```
+readYamlConfigFile ./config.yaml
+readYamlConfigDir ./config.d
+    ... one line per configuration file ...
+Broker ws://frugaliot.local:9012 - organizations: dev
 Doing OTA updates at /ota_update from /home/pi/frugal-iot/ota
 Serving /node_modules from ./node_modules
 User Database exists
 Opened user database
+Exec-ed starting SQL
+Created logger client for API integration
+Created push manager for Farm-Platform data push
+Mounted API routes at /api
+Added API error handler
 Serving /data from ./data
 Server starting on port 8080
-Serving from ./node_modules/frugal-iot-client
 mqtt dev connecting
 mqtt dev connect
+Subscribing topic dev/# 0
 ```
 
-The two lines that matter most are `Server starting on port 8080` and `mqtt dev connect`.
-`mqtt dev connect` means the server reached your broker and authenticated. If instead you see
-repeated `mqtt dev close` or `offline`, the broker URL or the password is wrong — recheck
+Check the `Broker` line names your own broker and your organization. The lines that matter most are
+the last three: `mqtt dev connect` means the server reached the broker and authenticated, and
+`Subscribing topic dev/#` means it is listening for your nodes. If instead you see repeated
+`mqtt dev close`, `offline`, or `Not authorized`, the broker URL or the password is wrong — recheck
 `config.d/mqtt.yaml` and that the password in `config.d/organizations/dev.yaml` matches the `dev`
 broker account.
 
+Once nodes are reporting, each reading is logged as it arrives, so this output keeps scrolling.
+
 Now open a browser on your laptop or phone at:
 
 ```
 http://frugaliot.local:8080
 ```
 
-(or `http://<the Pi's IP>:8080`). You should get the Frugal IoT UI, and be able to log in as
-username `dev` with the login password you set with `frugal-iot-setpassword` — not the broker
-password. (`superuser` and its password work too.)
+(or `http://<the Pi's IP>:8080`).
 
-Once logged in and with your organization selected, the UI's MQTT status should show *connected*:
-that is the browser using the organization's broker credentials over the WebSocket listener on
-port 9012, which is the last untested piece of the chain. If it does not connect, check that the
-`broker:` URL in `config.d/mqtt.yaml` is one this browser can actually resolve — a phone that
-cannot look up `.local` names needs the Pi's IP address there instead.
+> On a Mac, the browser will ask something like *"Allow Google Chrome Helper to find devices on
+> local networks"* the first time. **Allow it** — without that permission the browser cannot look up
+> `frugaliot.local`, nor reach the broker at that name, so the page and the live data both fail.
+
+You land on the Frugal IoT home page. Click **Dashboard**, and log in as username `dev` with the
+login password you set with `frugal-iot-setpassword` — not the broker password. (`superuser` and its
+password work too.)
+
+Once you are through to the dashboard and have selected your organization, its MQTT status should
+show **connected**. That is the browser authenticating to the broker as your organization, over the
+WebSocket listener on port 9012, from a different machine — the last untested piece of the chain.
+If it does not connect, check that the `broker:` URL in `config.d/mqtt.yaml` is one this browser can
+actually resolve: a phone that cannot look up `.local` names needs the Pi's IP address there instead.
 
 Until a sensor node reports in there will be no data to look at, but the dashboard should load.
 
@@ -395,32 +458,26 @@ Stop the server with `Ctrl-C` before continuing.
 
 ### A8. Run the server as a service
 
-So that it starts automatically at boot and restarts if it crashes:
+So that it starts automatically at boot and restarts if it crashes. The file that
+`npx frugal-iot-init` put in `extras/` already describes this installation — user `pi`, installed
+into `/home/pi/frugal-iot` — so it needs no editing:
 
 ```
 sudo cp extras/frugaliot.service /etc/systemd/system/frugaliot.service
-sudo nano /etc/systemd/system/frugaliot.service
-```
-
-Three lines need to match your Pi:
-
-```
-User=pi
-WorkingDirectory=/home/pi/frugal-iot
-ExecStart=/home/pi/frugal-iot/node_modules/.bin/frugal-iot-server
-```
-
-`WorkingDirectory` must be the directory you installed into, because that is where the server
-finds its configuration and database. `ExecStart` is the command npm created for you there — check
-it exists with `ls node_modules/.bin/frugal-iot-server`. Then:
-
-```
 sudo systemctl daemon-reload
-sudo systemctl enable frugaliot
-sudo systemctl start frugaliot
+sudo systemctl enable --now frugaliot
 systemctl status frugaliot
 ```
 
+`enable --now` both starts it and sets it to start at boot. `status` should report
+`active (running)`.
+
+> Only if you departed from this guide — a different username, or a directory other than
+> `~/frugal-iot` — edit `User`, `WorkingDirectory` and `ExecStart` in
+> `/etc/systemd/system/frugaliot.service` to match, then `sudo systemctl daemon-reload` and
+> `sudo systemctl restart frugaliot`. `WorkingDirectory` is the important one: it is where the
+> server looks for its configuration and database.
+
 To watch its log output, which is where the `mqtt dev connect` and incoming-reading messages now go:
 
 ```
@@ -432,21 +489,37 @@ still answers. Your server is now installed.
 
 ### A9. Point your sensor nodes at the Pi
 
-Your ESP8266/ESP32 nodes need to be told to use the Pi's broker — hostname or IP of the Pi,
-port `1883`, username and password of the organization (`dev` / the broker password from step A6),
-and the organization and project names to publish under.
+Your ESP8266/ESP32 nodes are told which broker to use in their sketch — `main.cpp`, or the `.ino`
+file if you build in the Arduino IDE. Look for a line like:
 
-> **To be written.** The node firmware lives in a separate repo
-> ([mitra42/frugal-iot](https://github.com/mitra42/frugal-iot)) and its exact configuration
-> settings are not documented here yet. See [Open questions](#open-questions).
+```cpp
+frugal_iot.configure_mqtt("frugaliot.naturalinnovation.org", "dev", "public");
+```
 
-You can confirm nodes are reporting without the UI at all, using the subscriber from step A6:
+and point it at your Pi instead:
+
+```cpp
+frugal_iot.configure_mqtt("frugaliot.local", "dev", "<broker-password>");
+```
+
+The three arguments are the broker's host, the organization, and that organization's broker
+password. The organization must be the one you created in step A6, because it is the first part of
+every topic the node publishes to, and the password is the *broker* password from that step — not
+the login password. Then rebuild and flash the node as usual.
+
+> If the node does not connect, try the Pi's IP address in place of `frugaliot.local`. Resolving
+> `.local` names needs mDNS support in the firmware, which is not something this guide has
+> confirmed; an IP address avoids the question entirely, which is why step A2 suggests reserving one
+> for the Pi in your router.
+
+You can confirm nodes are reporting without involving the UI, using the subscriber from step A6:
 
 ```
 mosquitto_sub -h localhost -u dev -P '<broker-password>' -t '#' -v
 ```
 
-Every reading from every node should scroll past.
+Every reading from every node should scroll past. Seeing anything here also proves the broker's
+port 1883 is reachable from off the Pi, which is what the nodes need.
 
 ### A10. HTTPS and over-the-air firmware updates
 
@@ -461,6 +534,83 @@ That section will cover: a DNS name for the Pi, port forwarding on the router, n
 proxy in front of port 8080, and certificates from Let's Encrypt via certbot. Both require the Pi
 to have internet access, which is the opposite of the offline case this guide is aimed at.
 
+### Known limitation: the clock on an offline Pi
+
+A Raspberry Pi has no battery-backed clock. While it has internet access it sets its time from the
+network and everything is correct — which is what you will see during this install. Fully offline it
+cannot: Raspberry Pi OS saves the time periodically and restores that value at boot, so after a
+power cut the Pi comes up believing it is whenever it last saved, and the gap never gets made up.
+
+To see whether the Pi's clock is actually right at any moment:
+
+```
+timedatectl
+```
+
+`System clock synchronized: yes` means it has reached a time server and the clock is trustworthy.
+`no`, with `NTP service: active`, means it is trying but has not succeeded — normal on a Pi with no
+route to the internet, and the point at which the timestamps below become a concern.
+
+What that affects:
+
+* **Logged data is stamped with the wrong time**, so graphs and history drift after each power cut.
+* **Relaying MQTT and watching devices live are unaffected** — the dashboard shows current values
+  whatever the Pi believes the date to be.
+
+If timestamps matter to you on an installation with no internet, the fix today is a hardware RTC
+module on the Pi's GPIO header. A possible future fix within Frugal IoT itself: the dashboard knows
+the time of the phone or laptop viewing it, so it could hand that to the server, which could adopt
+it whenever its own clock looks implausible (a date far in the past). Not accurate to the second,
+but close enough for sensor data.
+
+---
+
+## Upgrading
+
+To move an existing server to a newer release, from the directory you installed into:
+
+```
+cd ~/frugal-iot
+npm update frugal-iot-server
+npx frugal-iot-init
+sudo systemctl restart frugaliot
+```
+
+Taking those in turn:
+
+* `npm update` fetches the new version of the server, and of the client and logger it depends on.
+* `frugal-iot-init` adds any configuration file or directory the new version expects and you do not
+  have yet. It never changes files you already have, so your settings, database and data are safe.
+* `systemctl restart frugaliot` is what actually puts the new version into service — without it the
+  old one keeps running from memory, and nothing appears to have changed. (Skip this if you are
+  running the server by hand rather than as a service; stop it with `Ctrl-C` and start it again.)
+
+Check it came back up, and is running the version you expect:
+
+```
+systemctl status frugaliot
+journalctl -u frugaliot -n 30
+npm ls frugal-iot-server
+```
+
+`frugal-iot-init` leaves your files alone, so watch its output for a section headed **"These files
+were left as you have them, but this release ships a different version"**. It gives you the `diff`
+command for each. That matters most for the files you copy somewhere else — a changed
+`extras/mosquitto.conf` or `extras/frugaliot.service` does nothing until you install it again:
+
+```
+sudo cp extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf && sudo systemctl restart mosquitto
+sudo cp extras/frugaliot.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl restart frugaliot
+```
+
+> It does not compare `config.yaml`, `config.d/mqtt.yaml` or `config.d/server.yaml`, because those
+> hold your own settings and would differ every time. If a release note mentions a new setting in
+> one of them, compare it yourself:
+> `diff config.d/server.yaml node_modules/frugal-iot-server/config.d/server.yaml`
+
+Your organizations, accounts, database and logged data are untouched by an upgrade — they live in
+this directory, not in `node_modules`.
+
 ---
 
 ## Part B — Raspberry Pi Zero W
@@ -471,8 +621,9 @@ and what has to be checked.
 First, work out which board you have, because they are very different:
 
 * **Raspberry Pi Zero 2 W** — 64-bit ARM (Cortex-A53), 512 MB RAM. Should follow **Part A**
-  almost unchanged: choose *Raspberry Pi Zero 2 W* in Imager, and everything else applies.
-  The concern is memory, not architecture.
+  almost unchanged: choose *Raspberry Pi Zero 2 W* in Imager, and everything else applies. Running
+  the server needs about 250 MB, so 512 MB should be enough; the pinch point is more likely to be
+  `npm install`, which is far hungrier than the running server.
 * **Raspberry Pi Zero W** (the original) — 32-bit ARMv6 (BCM2835), 512 MB RAM. This is the
   awkward one: it needs the 32-bit Raspberry Pi OS, and **the official Node.js builds no longer
   support ARMv6**. Unofficial ARMv6 builds exist at
@@ -489,8 +640,9 @@ Differences to expect on either Zero:
 * Everything will be slow. `npm install` may take a long time, especially if `sqlite3` has to be
   compiled from source — which is likely on ARMv6, since no ready-made binaries are published.
 
-**Before this section can be written, I need to know:** which Zero you intend to support (original
-W, 2 W, or both), and the results of trying a Node install on it.
+**Both boards are to be tested**, one of each having been ordered for the purpose. This section gets
+written once we know how they behave — in particular whether a usable Node exists for the original
+Zero W's ARMv6.
 
 ---
 
@@ -502,47 +654,24 @@ they get settled.
 
 **Part A, to check while installing**
 
-1. **`sqlite3` native build** (step A4) — does `npm install frugal-iot-server` find a prebuilt
-   ARM64 binary, or does it need `build-essential`? If it always needs compiling, that should move
-   up into step A3 as a normal prerequisite rather than a troubleshooting note.
-2. **Mosquitto listeners** (step A5) — confirm that copying `extras/mosquitto.conf` into `conf.d`
-   does not collide with the packaged default configuration, and that both 1883 and 9012 are
-   reachable from another machine (`mosquitto_sub -h frugaliot.local ...` from your laptop).
-3. **Empty password file** (step A5) - Mosquitto is started with a `password_file` that exists but
-   is empty, because the first account is not created until step A6. Confirm
-   it starts happily like that. If it refuses, the fix is to create a throwaway account with
-   `sudo mosquitto_passwd -c -b /etc/mosquitto/mosquitto_passwords unused unused` before starting it.
-4. **`frugal-iot-addorganization` writing the broker password** (step A6) — step A5 chowns
-   `/etc/mosquitto/mosquitto_passwords` to your user so the script can add the account without
-   sudo. Confirm it does, and that it reports "Set mosquitto password" rather than a warning.
-5. **`.local` name resolution** (step A2) — does `frugaliot.local` work from your laptop, and from
-   an Android phone? If Android fails as expected, the guide should recommend IP addresses more strongly.
-6. **First-run output** (step A7) — the expected startup output above is adapted from README.md and
-   a run on a development machine; replace it with the actual output from the Pi.
-7. **Time and dates while offline** — an offline Pi has no internet clock to sync with, and no
-   battery-backed clock, so after a power cut it starts with a wrong date until something corrects
-   it. For a data logger writing timestamped files, that matters. Does the logger cope? Should this
-   guide recommend a hardware RTC module, or a way for a phone or laptop on the LAN to set the time?
+1. **`.local` from an Android phone** (step A2, A4) - confirmed working from a laptop (with the
+   local-network permission granted on a Mac) and from an iPhone. Android is expected to fail, which
+   is what the note in A4 assumes; worth confirming on a real Android phone.
 
 **Needs information I do not have**
 
-8. **Sensor node configuration** (step A9) — what exactly does one set in the node firmware to
-   point it at a local broker (broker host/port, credentials, organization, project)? Once you tell
-   me, or point me at the right file in the `frugal-iot` repo, I can write that section properly.
-9. **Which Pi Zero** (Part B) — original Zero W, Zero 2 W, or both.
-10. **Bridging to the shared server** — the local broker could optionally bridge to
+2. **Pi Zero W and Zero 2 W** (Part B) — both boards are on order; Part B stays unwritten until they
+    have been tried. The open technical question is Node on the original Zero W's ARMv6.
+3. **Bridging to the shared server** — the local broker could optionally bridge to
     naturalinnovation.org so data also reaches the shared server. Not covered here; a later task.
-11. **Organization naming** — this guide sets up exactly one organization named `dev`, because that
+4. **Organization naming** — this guide sets up exactly one organization named `dev`, because that
     is the node firmware's default. Is that the right default for a farm installation, or should the
     guide encourage a meaningful organization id (which then has to be set in the node firmware too)?
-12. **Upgrading an existing installation** — `npm update frugal-iot-server` followed by
-    `npx frugal-iot-init` should be all it takes, since init adds missing configuration without
-    touching what is there. Not yet tried on a server that has been running for a while.
 
-**Tested on**
+## Tested on
 
-Fill this in as you go, so we know what the guide has actually been proven against:
+What the guide has actually been proven against — add a row for each run:
 
-| Date | Board | OS image | Node version | Result |
-| --- | --- | --- | --- | --- |
-|  | Pi 4 Model B |  |  |  |
+| Date | Board | OS image | Node | Server | Result |
+| --- | --- | --- | --- | --- | --- |
+| 2026-07-31 | Pi 4 Model B, 4 GB | Raspberry Pi OS Lite 64-bit, Debian 13.6 (trixie) | 20.19.2 and npm 9.2.0, both from `apt` | frugal-iot-server 0.3.3 from npm | Steps A1–A8 completed, plus an upgrade over the top of it. Wi-Fi from Imager did not connect and was fixed with `nmtui` (A2). `sqlite3` installed as a prebuilt binary, nothing compiled. Mosquitto started with an empty password file, and `frugal-iot-addorganization` wrote the broker account without sudo. Dashboard reached over `frugaliot.local` from a Mac (Chrome, after allowing local network access) and from an iPhone, MQTT status *connected*. A9 (sensor nodes) and A10 (HTTPS/OTA) not exercised. |

package/README.md

@@ -51,25 +51,20 @@ If its working correctly you should see something like
 readYamlConfigFile ./config.yaml
 ... and then reading each of the other files
 
-Config= {
-  server: { port: 8080 },
-  mqtt: { broker: 'wss://frugaliot.naturalinnovation.org/wss' },
-  organizations: { 
-    dev: { mqtt_password: 'public', projects: [Array] } 
-  }
-}
+Broker wss://frugaliot.naturalinnovation.org/wss - organizations: dev
 Doing OTA updates at /ota_update from ...some path.../ota
 Serving /node_modules from ./node_modules
 User Database exists
 Opened user database
+Exec-ed starting SQL
 Serving /data from ./data
 Server starting on port 8080
-Serving from ./node_modules/frugal-iot-client
 mqtt dev connecting
 mqtt dev connect
+Subscribing topic dev/# 0
 Received dev/lotus/esp8266-85ea2b/humidity   71.8
 ```
-Where the config is reported back, 
+Where the broker and organizations are reported back, 
 then it successfully connects to the mqtt server
 and receives data from nodes attached to it. 
 

package/config.d/server.yaml

@@ -1,10 +1,10 @@
 port: 8080
 #Production mode
-#nodemodulesdir: ./node_modules
-#htmldir: ./node_modules/frugal-iot-client
+nodemodulesdir: ./node_modules
+htmldir: ./node_modules/frugal-iot-client
 # Development mode (uncomment these two, and comment out the two above, when working on the client)
-htmldir: ../frugal-iot-client
-nodemodulesdir: ../frugal-iot-client/node_modules
+#htmldir: ../frugal-iot-client
+#nodemodulesdir: ../frugal-iot-client/node_modules
 # The path to the private directory is relative to the server's working directory.
 privatedir: ./private
 publicdir: ./public

package/extras/frugaliot.service

@@ -1,22 +1,33 @@
-# Copy to /etc/systemd/system/frugaliot.service (or /usr/lib/systemd/system on some distributions)
-# and edit User, WorkingDirectory and ExecStart for this machine.
+# Systemd unit for the Frugal IoT server.
+#
+# As shipped this matches the Raspberry Pi installation in INSTALLATION.md: user "pi", installed
+# with npm into /home/pi/frugal-iot. If that is what you have, install it unedited:
+#   sudo cp extras/frugaliot.service /etc/systemd/system/frugaliot.service
+#   sudo systemctl daemon-reload
+#   sudo systemctl enable --now frugaliot
+#
+# On any other machine, change User and WorkingDirectory to the account and the directory you
+# installed into, and ExecStart to match (see the git clone alternative below).
 [Unit]
 Description=frugal-iot server
-After=network-online.target
+# mosquitto is listed for the common case of the broker running on this same machine; harmless if
+# it is not installed, because systemd ignores ordering against units that do not exist.
+After=network-online.target mosquitto.service
+Wants=network-online.target
 
 [Service]
 Type=simple
-User=mitra
+User=pi
 # The directory holding this server's config.d, database and data - where npm install was run
-WorkingDirectory=/var/www/frugal-iot
+WorkingDirectory=/home/pi/frugal-iot
 # Installed with npm, this is the command npm put in that directory:
-ExecStart=/var/www/frugal-iot/node_modules/.bin/frugal-iot-server
+ExecStart=/home/pi/frugal-iot/node_modules/.bin/frugal-iot-server
 # From a git clone instead, run the file directly (check the node path with "which node"):
 #ExecStart=/usr/bin/node ./frugal-iot-server.js
 Restart=always
 RestartSec=10
-StandardOutput=syslog
-StandardError=syslog
+StandardOutput=journal
+StandardError=journal
 SyslogIdentifier=frugaliot
 
 [Install]

package/extras/mosquitto.conf

@@ -3,4 +3,8 @@ protocol mqtt
 listener 9012
 protocol websockets
 #http_dir /home/mitra/github_mitra42/frugal-iot/html
-password_file /etc/mosquitto/mosquitto_passwords
+# Kept under /var/lib/mosquitto, which belongs to the mosquitto user, rather than /etc/mosquitto,
+# which belongs to root. Both the broker and mosquitto_passwd warn unless the file belongs to
+# whoever opened it, and mosquitto_passwd also writes a temporary backup beside it, so the
+# directory has to be writable by that same user. See INSTALLATION.md step A5.
+password_file /var/lib/mosquitto/passwords

package/frugal-iot-server.js

@@ -30,6 +30,7 @@
  *Δ /admin (get) dashboard for administrators - includes OTA and will include permission management
  P /ota_update (post) protected place to upload new binaries (OTAUPDATE)
  XP /ota_list/:org list all ota files for an organization
+ XP /ota_get/:org/*remainingpath download a binary so the client can flash it over USB
  O  /private Serve up private files under authentication - currently unused
  * /register (post) register a new user
  */
@@ -313,7 +314,6 @@ function addProjectToConfig(org, id, name) {
 }
 // Read the projects table for every configured organization and add them to config.organizations
 function loadProjectsIntoConfig(cb) {
-  console.log("XXX loadProjectsIntoConfig");
   each(Object.keys(config.organizations), (org, cb) => {
     get_projects_list(org, (err, projects) => {
       if (err) {
@@ -659,13 +659,25 @@ function debugRoutes(req, res, next) {
   next();
 }
 // Main for server
+// Everything is read relative to the working directory, so being in the wrong one is the most
+// common way to fail to start - say so plainly rather than reporting ENOENT on ./config.yaml.
+access('./config.yaml', constants.R_OK, (err) => {
+  if (err) {
+    console.error(`No config.yaml in ${process.cwd()}`);
+    console.error("Run this from the directory this server was installed into - the one holding");
+    console.error("config.d and frugal-iot.db. If it is a new directory, set it up with: npx frugal-iot-init");
+    process.exit(1);
+  }
+});
 mqttLogger.readYamlConfig('.', (err, configobj) => {
   // Note side effect leaves copy of config in the mqttLogger
   if (err) {
     console.error(err);
   } else {
     /* global */ config = configobj;
-    console.log("Config=", config);
+    // Summarize rather than dumping the whole config: it is mostly the sensor schema, and it holds
+    // each organization's mqtt_password, which should not be going to the console and the journal.
+    console.log("Broker", config.mqtt.broker, "- organizations:", Object.keys(config.organizations).join(", ") || "(none)");
     // Could genericize config defaults
     if (!config.morgan) {
       config.morgan = ':method :url :req[range] :status :res[content-length] :response-time ms :req[referer]'
@@ -909,6 +921,24 @@ mqttLogger.readYamlConfig('.', (err, configobj) => {
             });
           }
         );
+        // Download an OTA binary so the client can flash it over USB (see FLASH_PLAN.md in
+        // frugal-iot-client). Deliberately not /ota_update, which is unauthenticated for devices
+        // and answers 304 rather than sending bytes.
+        app.get('/ota_get/:org/*remainingpath',
+          loggedInOrFail,
+          can_OTAUPDATE,
+          (req,res) => {
+            let remainingpath = req.params.remainingpath.join('/');
+            let filepath = `${config.server.otadir}/${req.params.org}/${sanitize(remainingpath)}/firmware.bin`;
+            console.log("Sending OTA file", filepath);
+            res.sendFile(filepath, {}, (err) => {
+              if (err) {
+                console.error("Error sending ota file:", filepath, err);
+                if (!res.headersSent) res.status(404).send(err.message);
+              }
+            });
+          }
+        );
         app.get('/people_list/:org',
           loggedInOrFail,
           can_ADMIN, // Gets org from URL

package/package.json

@@ -12,7 +12,7 @@
     "express": "^5.2.1",
     "express-session": "^1.19.0",
     "frugal-iot-client": "^1.3.11",
-    "frugal-iot-logger": "^1.1.17",
+    "frugal-iot-logger": "^1.1.18",
     "hash-wasm": "^4.12.0",
     "morgan": "^1.11.0",
     "multer": "^2.1.1",
@@ -24,7 +24,8 @@
     "frugal-iot-server": "frugal-iot-server.js",
     "frugal-iot-init": "scripts/init.zsh",
     "frugal-iot-addorganization": "scripts/addorganization.zsh",
-    "frugal-iot-setpassword": "scripts/setpassword.zsh"
+    "frugal-iot-setpassword": "scripts/setpassword.zsh",
+    "frugal-iot-diagnostic": "scripts/diagnostic.zsh"
   },
   "description": "Frugal IoT client and server",
   "//files": "An allowlist, not an ignore list - anything not named here is never published, so a stray file (a database backup, a credential) cannot leak by default. Check with: npm pack --dry-run",
@@ -66,5 +67,5 @@
     "test:coverage": "vitest run --coverage"
   },
   "type": "module",
-  "version": "0.3.3"
+  "version": "0.3.5"
 }

package/scripts/addorganization.zsh

@@ -75,13 +75,50 @@ if ! command -v mosquitto_passwd >/dev/null; then
   MOSQUITTO_PASSWD_MISSING=1
 fi
 
-# Prefer the real system password file; if it's not present (e.g. mosquitto isn't installed on this
-# machine), fall back to editing the local copy under extras/ so the script still does something useful.
-MOSQUITTO_PASSWD_FILE="/etc/mosquitto/mosquitto_passwords"
+# Ask the broker's own configuration where its password file is, rather than guessing - that way
+# this works whatever layout a machine uses, including installations older than this script.
+# Mosquitto reads /etc/mosquitto/mosquitto.conf and then the .conf files in conf.d, last one wins.
+MOSQUITTO_PASSWD_FILE=$(grep -hE '^[[:space:]]*password_file[[:space:]]' \
+  /etc/mosquitto/mosquitto.conf /etc/mosquitto/conf.d/*.conf(N) 2>/dev/null | tail -1 | awk '{print $2}')
 USED_LOCAL_MOSQUITTO_FILE=0
-if [[ ! -e "$MOSQUITTO_PASSWD_FILE" && -e "extras/mosquitto_passwords" ]]; then
-  MOSQUITTO_PASSWD_FILE="extras/mosquitto_passwords"
-  USED_LOCAL_MOSQUITTO_FILE=1
+if [[ -n "$MOSQUITTO_PASSWD_FILE" ]]; then
+  MOSQUITTO_PASSWD_FROM_CONFIG=1
+else
+  MOSQUITTO_PASSWD_FROM_CONFIG=0
+  # No mosquitto configuration here - fall back to the usual locations, and then to the copy under
+  # extras/ so that a machine without mosquitto installed still gets something useful.
+  for f in /var/lib/mosquitto/passwords /etc/mosquitto/mosquitto_passwords; do
+    if [[ -e "$f" ]]; then
+      MOSQUITTO_PASSWD_FILE="$f"
+      break
+    fi
+  done
+  if [[ -z "$MOSQUITTO_PASSWD_FILE" ]]; then
+    if [[ -e "extras/mosquitto_passwords" ]]; then
+      MOSQUITTO_PASSWD_FILE="extras/mosquitto_passwords"
+      USED_LOCAL_MOSQUITTO_FILE=1
+    else
+      MOSQUITTO_PASSWD_FILE="/var/lib/mosquitto/passwords"
+    fi
+  fi
+fi
+
+# Run mosquitto_passwd as whoever owns that file. Mosquitto warns unless the file belongs to the user
+# opening it, and mosquitto_passwd writes a temporary backup beside it, so it needs the directory too
+# - neither of which we get by running as ourselves against a file owned by mosquitto or by root.
+# On Raspberry Pi OS the first user has passwordless sudo, so this is invisible.
+MOSQUITTO_PASSWD_CMD=(mosquitto_passwd)
+if [[ ! -w "${MOSQUITTO_PASSWD_FILE:h}" || ( -e "$MOSQUITTO_PASSWD_FILE" && ! -w "$MOSQUITTO_PASSWD_FILE" ) ]]; then
+  if command -v sudo >/dev/null; then
+    # stat's spelling differs between Linux (-c) and BSD/macOS (-f)
+    MOSQUITTO_PASSWD_OWNER=$(stat -c '%U' "$MOSQUITTO_PASSWD_FILE" 2>/dev/null \
+      || stat -f '%Su' "$MOSQUITTO_PASSWD_FILE" 2>/dev/null || true)
+    if [[ -n "$MOSQUITTO_PASSWD_OWNER" && "$MOSQUITTO_PASSWD_OWNER" != "$(id -un)" && "$MOSQUITTO_PASSWD_OWNER" != "root" ]]; then
+      MOSQUITTO_PASSWD_CMD=(sudo -u "$MOSQUITTO_PASSWD_OWNER" mosquitto_passwd)
+    else
+      MOSQUITTO_PASSWD_CMD=(sudo mosquitto_passwd)
+    fi
+  fi
 fi
 
 # ---- Escaping helpers ----
@@ -168,17 +205,24 @@ MOSQUITTO_PASSWD_FAILED=0
 MOSQUITTO_PASSWD_ERROR=""
 if [[ "$MOSQUITTO_PASSWD_MISSING" -eq 1 ]]; then
   echo "Warning: mosquitto_passwd command not found - skipped setting MQTT broker password" >&2
+elif [[ "$MOSQUITTO_PASSWD_FROM_CONFIG" -eq 1 && ! -e "$MOSQUITTO_PASSWD_FILE" ]]; then
+  # Nothing to add an account to - and mosquitto will not be running either, for the same reason.
+  echo "Warning: mosquitto is configured to use ${MOSQUITTO_PASSWD_FILE}, but that file does not exist," >&2
+  echo "so the MQTT broker password was NOT set. Create it, then add this organization to it:" >&2
+  echo "  sudo install -o mosquitto -g mosquitto -m 600 /dev/null \"${MOSQUITTO_PASSWD_FILE}\"" >&2
+  echo "  sudo -u mosquitto mosquitto_passwd -b \"${MOSQUITTO_PASSWD_FILE}\" \"${ORG_ID}\" \"${PASSWORD}\"" >&2
+  MOSQUITTO_PASSWD_FAILED=1
 else
   # Capture stderr without letting `set -e` abort the whole script on failure - the org's config file
   # and DB rows are already written by this point, so a failure here should be reported, not fatal.
-  MOSQUITTO_PASSWD_ERROR=$(mosquitto_passwd -b "$MOSQUITTO_PASSWD_FILE" "${ORG_ID}" "${PASSWORD}" 2>&1 >/dev/null) || MOSQUITTO_PASSWD_FAILED=1
+  MOSQUITTO_PASSWD_ERROR=$("${MOSQUITTO_PASSWD_CMD[@]}" -b "$MOSQUITTO_PASSWD_FILE" "${ORG_ID}" "${PASSWORD}" 2>&1 >/dev/null) || MOSQUITTO_PASSWD_FAILED=1
   if [[ "$MOSQUITTO_PASSWD_FAILED" -eq 1 ]]; then
     echo "Warning: mosquitto_passwd failed to set the MQTT broker password:" >&2
     echo "  ${MOSQUITTO_PASSWD_ERROR}" >&2
-    if [[ "$MOSQUITTO_PASSWD_ERROR" == *"Permission denied"* || ! -w "$MOSQUITTO_PASSWD_FILE" ]]; then
-      echo "This looks like a permissions problem - re-run that command with sudo:" >&2
-      echo "  sudo mosquitto_passwd -b \"${MOSQUITTO_PASSWD_FILE}\" \"${ORG_ID}\" \"${PASSWORD}\"" >&2
-    fi
+    echo "Everything else was done, so set just the broker password by hand:" >&2
+    echo "  sudo mosquitto_passwd -b \"${MOSQUITTO_PASSWD_FILE}\" \"${ORG_ID}\" \"${PASSWORD}\"" >&2
+    echo "If it complains about creating a backup file, the directory holding that file is not" >&2
+    echo "writable by you - which is why the command above uses sudo." >&2
   else
     echo "Set mosquitto password for ${ORG_ID} in ${MOSQUITTO_PASSWD_FILE}"
   fi

package/scripts/diagnostic.zsh

@@ -0,0 +1,289 @@
+#!/usr/bin/env zsh
+#
+# Report the state of a Frugal IoT installation, for working out why something in INSTALLATION.md
+# did not do what it said it would. It only looks - it changes nothing.
+#
+# Run it from the directory the server is installed in (the one holding frugal-iot.db):
+#   zsh diagnostic.zsh
+# or, once installed from npm:
+#   npx frugal-iot-diagnostic
+#
+# Paste the whole output into a bug report. Passwords are deliberately not printed: the password
+# file is reported by account name only, and organization configs by name only.
+
+# No "set -e": a failing check should report and carry on, not stop the script.
+setopt no_unset 2>/dev/null
+
+section() { print -r -- ""; print -r -- "===== $* ====="; }
+item()    { print -r -- "  $*"; }
+have()    { command -v "$1" >/dev/null 2>&1; }
+# Ownership and mode of a file, spelled for either Linux or BSD/macOS stat
+fileinfo() {
+  local f=$1
+  if [[ ! -e "$f" ]]; then print -r -- "missing"; return; fi
+  stat -c '%U:%G %a %s bytes' "$f" 2>/dev/null || stat -f '%Su:%Sg %Lp %z bytes' "$f" 2>/dev/null || print -r -- "(cannot stat)"
+}
+
+PROBLEMS=()
+problem() { PROBLEMS+=("$1"); }
+
+section "When and where"
+item "date:        $(date 2>/dev/null)"
+item "host:        $(hostname 2>/dev/null)"
+item "user:        $(id -un 2>/dev/null) (groups: $(id -Gn 2>/dev/null))"
+item "directory:   $PWD"
+
+section "The machine"
+have uname && item "kernel:      $(uname -srm)"
+[[ -r /etc/os-release ]] && item "os:          $(grep PRETTY_NAME /etc/os-release | cut -d'"' -f2)"
+[[ -r /proc/device-tree/model ]] && item "board:       $(tr -d '\0' < /proc/device-tree/model)"
+have free && item "memory:      $(free -h | awk '/^Mem:/ {print $3 " used of " $2 ", " $7 " available"}')"
+have df && item "disk (/):    $(df -h / | awk 'NR==2 {print $3 " used of " $2 ", " $4 " free"}')"
+if have timedatectl; then
+  item "clock:       $(timedatectl show -p NTPSynchronized --value 2>/dev/null | sed 's/^yes$/synchronized with a time server/; s/^no$/NOT synchronized - timestamps may be wrong/')"
+fi
+
+section "Software versions"
+for c in node npm sqlite3 zsh mosquitto mosquitto_passwd mosquitto_sub; do
+  if have $c; then
+    case $c in
+      node|npm)            item "$c: $($c -v 2>&1 | head -1)" ;;
+      sqlite3)             item "$c: $(sqlite3 --version 2>&1 | awk '{print $1}')" ;;
+      zsh)                 item "$c: $ZSH_VERSION" ;;
+      mosquitto)           item "$c: $(mosquitto -h 2>&1 | head -1)" ;;
+      *)                   item "$c: present" ;;
+    esac
+  else
+    item "$c: NOT INSTALLED"
+    [[ $c == (node|npm|sqlite3) ]] && problem "$c is not installed - see INSTALLATION.md step A3"
+  fi
+done
+
+section "This installation"
+for f in config.yaml config.d frugal-iot.db data ota node_modules; do
+  if [[ -e $f ]]; then item "$f: present"; else item "$f: MISSING"; fi
+done
+[[ ! -e frugal-iot.db ]] && problem "No frugal-iot.db here - either the wrong directory, or 'npx frugal-iot-init' has not been run"
+if [[ -d node_modules/frugal-iot-server ]]; then
+  item "frugal-iot-server: $(node -e 'console.log(require("./node_modules/frugal-iot-server/package.json").version)' 2>/dev/null)"
+  item "frugal-iot-client: $(node -e 'console.log(require("./node_modules/frugal-iot-client/package.json").version)' 2>/dev/null)"
+  item "frugal-iot-logger: $(node -e 'console.log(require("./node_modules/frugal-iot-logger/package.json").version)' 2>/dev/null)"
+  item "commands:    $(ls node_modules/.bin 2>/dev/null | grep frugal | tr '\n' ' ')"
+else
+  item "frugal-iot-server is not installed in this directory"
+fi
+if [[ -f config.d/mqtt.yaml ]]; then item "broker configured as: $(grep -h '^broker:' config.d/mqtt.yaml 2>/dev/null)"; fi
+if [[ -d config.d/organizations ]]; then
+  # Names only - these files contain the broker password.
+  # (N) makes a non-matching glob expand to nothing rather than erroring - and the list has to be
+  # tested before use, because "ls" with no arguments would list the current directory instead.
+  ORGS=(config.d/organizations/*.yaml(N:t:r))
+  if (( ${#ORGS} )); then
+    item "organizations: ${ORGS}"
+  else
+    item "organizations: none defined yet"
+  fi
+fi
+if [[ -f frugal-iot.db ]] && have sqlite3; then
+  item "accounts:    $(sqlite3 frugal-iot.db 'SELECT group_concat(username, ", ") FROM users;' 2>/dev/null)"
+fi
+
+section "Mosquitto configuration"
+if [[ -f /etc/mosquitto/mosquitto.conf ]]; then
+  item "/etc/mosquitto/mosquitto.conf: $(fileinfo /etc/mosquitto/mosquitto.conf)"
+else
+  item "/etc/mosquitto/mosquitto.conf: missing - is mosquitto installed?"
+fi
+item "files in /etc/mosquitto/conf.d:"
+for f in /etc/mosquitto/conf.d/*(N); do
+  item "  ${f}: $(fileinfo $f)"
+  grep -nE '^[[:space:]]*(listener|protocol|password_file|allow_anonymous)' $f 2>/dev/null | sed 's/^/      /'
+done
+[[ -z "$(print -r -- /etc/mosquitto/conf.d/*(N))" ]] && problem "Nothing in /etc/mosquitto/conf.d - the Frugal IoT config was never copied there (step A5)"
+
+# Which password file does the running configuration actually name?
+# mosquitto's include_dir only loads files ending in .conf, so look at exactly those, last one wins
+CONFIGURED_PWFILE=$(grep -hE '^[[:space:]]*password_file[[:space:]]' /etc/mosquitto/mosquitto.conf /etc/mosquitto/conf.d/*.conf(N) 2>/dev/null | tail -1 | awk '{print $2}')
+section "Mosquitto password file"
+if [[ -n "$CONFIGURED_PWFILE" ]]; then
+  item "configuration names: $CONFIGURED_PWFILE"
+  item "  that file:         $(fileinfo $CONFIGURED_PWFILE)"
+  if [[ ! -e "$CONFIGURED_PWFILE" ]]; then
+    problem "Mosquitto is configured to use $CONFIGURED_PWFILE but that file does not exist - it will refuse to start"
+  else
+    # Account names only, never the hashes
+    item "  accounts in it:    $(cut -d: -f1 "$CONFIGURED_PWFILE" 2>/dev/null | tr '\n' ' ')"
+    [[ ! -s "$CONFIGURED_PWFILE" ]] && item "  (the file is empty - no accounts yet)"
+  fi
+else
+  item "no password_file line found in any mosquitto configuration"
+  problem "No password_file configured - the broker would allow anonymous access or reject everything"
+fi
+# The copy in this installation is what step A5 tells you to put into /etc - if the two disagree,
+# either the copy never happened or the installed package is older than the instructions.
+if [[ -f extras/mosquitto.conf ]]; then
+  item "this installation's extras/mosquitto.conf names: $(grep -hE '^[[:space:]]*password_file' extras/mosquitto.conf 2>/dev/null | awk '{print $2}')"
+  if [[ -n "$CONFIGURED_PWFILE" ]] && ! diff -q extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf >/dev/null 2>&1; then
+    problem "extras/mosquitto.conf here differs from /etc/mosquitto/conf.d/frugal-iot.conf - if you followed step A5 with an older installed package, copy it again after upgrading"
+  fi
+fi
+item "candidate locations, whether or not configured:"
+for f in /var/lib/mosquitto/passwords /etc/mosquitto/mosquitto_passwords; do
+  item "  ${f}: $(fileinfo $f)"
+done
+item "directories they live in:"
+for d in /var/lib/mosquitto /etc/mosquitto; do
+  item "  ${d}: $(fileinfo $d)"
+done
+
+section "Mosquitto service"
+if have systemctl; then
+  item "enabled:     $(systemctl is-enabled mosquitto 2>&1)"
+  item "active:      $(systemctl is-active mosquitto 2>&1)"
+  [[ "$(systemctl is-active mosquitto 2>/dev/null)" != "active" ]] && problem "Mosquitto is not running - see its own log, quoted above under 'Mosquitto service'"
+  print -r -- "  --- last 12 journal lines (systemd's view: usually only an exit code) ---"
+  journalctl -u mosquitto -n 12 --no-pager 2>&1 | sed 's/^/      /'
+else
+  item "systemctl not available on this machine"
+fi
+# Debian's mosquitto logs to a file rather than the journal, so this is where the real reason is.
+MOSQUITTO_LOG=$(grep -hE '^[[:space:]]*log_dest[[:space:]]+file' /etc/mosquitto/mosquitto.conf /etc/mosquitto/conf.d/*(N) 2>/dev/null | tail -1 | awk '{print $3}')
+[[ -z "$MOSQUITTO_LOG" ]] && MOSQUITTO_LOG=/var/log/mosquitto/mosquitto.log
+item "its own log file: ${MOSQUITTO_LOG} ($(fileinfo $MOSQUITTO_LOG))"
+if [[ -r "$MOSQUITTO_LOG" ]]; then
+  print -r -- "  --- last 12 lines of that log (this is where startup errors appear) ---"
+  tail -12 "$MOSQUITTO_LOG" 2>&1 | sed 's/^/      /'
+elif [[ -e "$MOSQUITTO_LOG" ]]; then
+  # The log belongs to the mosquitto user, so reading it needs privilege. Try without prompting -
+  # on Raspberry Pi OS the first user has passwordless sudo, so this usually just works.
+  if have sudo && MOSQUITTO_LOG_TAIL=$(sudo -n tail -12 "$MOSQUITTO_LOG" 2>/dev/null); then
+    print -r -- "  --- last 12 lines of that log, read with sudo (startup errors appear here) ---"
+    print -r -- "$MOSQUITTO_LOG_TAIL" | sed 's/^/      /'
+  else
+    item "  it belongs to $(fileinfo $MOSQUITTO_LOG | awk '{print $1}') so it cannot be read as $(id -un)."
+    item "  Re-run this whole script with sudo to include it:  sudo zsh $0"
+    problem "Could not read ${MOSQUITTO_LOG}, which is where Mosquitto explains itself - re-run with sudo"
+  fi
+fi
+
+section "Listening ports"
+PORTS=""
+if have ss; then
+  PORTS=$(ss -tln 2>/dev/null | grep -E ':(1883|9012|8080)\b')
+elif have netstat; then
+  PORTS=$(netstat -an 2>/dev/null | grep -E '[.:](1883|9012|8080) ')
+else
+  item "neither ss nor netstat available"
+fi
+if [[ -n "$PORTS" ]]; then
+  print -r -- "$PORTS" | sed 's/^/  /'
+else
+  item "none of 1883 (mqtt), 9012 (websockets), 8080 (web) are listening"
+fi
+
+section "Frugal IoT server service"
+if have systemctl; then
+  if systemctl list-unit-files 2>/dev/null | grep -q '^frugaliot.service'; then
+    item "enabled:     $(systemctl is-enabled frugaliot 2>&1)"
+    item "active:      $(systemctl is-active frugaliot 2>&1)"
+    print -r -- "  --- last 20 journal lines ---"
+    journalctl -u frugaliot -n 20 --no-pager 2>&1 | sed 's/^/      /'
+  else
+    item "no frugaliot service installed yet (step A8 not reached)"
+  fi
+else
+  item "systemctl not available on this machine"
+fi
+
+section "Name resolution (step A2, A4)"
+# The broker URL in config.d/mqtt.yaml has to resolve from every machine that uses it - this one,
+# and whatever browser or node talks to it.
+BROKER_URL=$(grep -h '^broker:' config.d/mqtt.yaml 2>/dev/null | awk '{print $2}')
+BROKER_HOST=${${BROKER_URL#*://}%%:*}
+if [[ -n "$BROKER_HOST" ]]; then
+  item "broker host: $BROKER_HOST"
+  if have getent && BROKER_IP=$(getent hosts "$BROKER_HOST" 2>/dev/null | head -1 | awk '{print $1}') && [[ -n "$BROKER_IP" ]]; then
+    item "  resolves to $BROKER_IP from this machine"
+  elif have ping && ping -c1 -W2 "$BROKER_HOST" >/dev/null 2>&1; then
+    item "  answers ping from this machine"
+  else
+    item "  DOES NOT RESOLVE from this machine"
+    problem "The broker host '$BROKER_HOST' does not resolve here - the server's own logger cannot connect either"
+  fi
+  [[ "$BROKER_HOST" == *.local ]] && item "  note: .local names do not resolve on most Android phones - use an IP address there"
+fi
+MYADDRS=$( (have ip && ip -4 -o addr show scope global | awk '{print $2"="$4}') 2>/dev/null | tr '\n' ' ')
+item "this machine's addresses: ${MYADDRS:-(could not determine)}"
+
+section "Broker authentication (steps A5, A6)"
+if have mosquitto_sub; then
+  # A wrong password must be refused - that is the check in step A5
+  WRONGOUT=$(mosquitto_sub -h localhost -u nobody -P wrong -t '#' -W 2 2>&1)
+  if [[ "$WRONGOUT" == *"not authorised"* ]]; then
+    item "wrong password: correctly refused"
+  elif [[ "$WRONGOUT" == *"Connection refused"* || "$WRONGOUT" == *"Error"* ]]; then
+    item "wrong password: broker did not answer - $WRONGOUT"
+    problem "The broker is not answering on localhost:1883"
+  else
+    item "wrong password: ACCEPTED - the broker is not requiring credentials"
+    problem "A wrong password was not refused - check password_file is set in the mosquitto config"
+  fi
+  # Then each configured organization should be able to connect with its own credentials.
+  # The password is read from the config and used, never printed.
+  for f in config.d/organizations/*.yaml(N); do
+    ORG=${f:t:r}
+    ORG_PW=$(sed -n 's/^mqtt_password:[[:space:]]*//p' "$f" 2>/dev/null | head -1 | tr -d '"'"'"'')
+    if [[ -z "$ORG_PW" ]]; then
+      item "organization $ORG: no mqtt_password in $f"
+      continue
+    fi
+    ORGOUT=$(mosquitto_sub -h localhost -u "$ORG" -P "$ORG_PW" -t '#' -W 2 2>&1)
+    if [[ "$ORGOUT" == *"not authorised"* ]]; then
+      item "organization $ORG: REFUSED by the broker"
+      problem "Organization '$ORG' cannot log in to the broker - its password in $f does not match the broker's password file (step A6)"
+    elif [[ "$ORGOUT" == *"Connection refused"* ]]; then
+      item "organization $ORG: broker not answering"
+    else
+      item "organization $ORG: authenticates, and saw $(print -r -- "$ORGOUT" | grep -c . ) message(s) in a 2 second sample"
+    fi
+  done
+else
+  item "mosquitto_sub not installed - cannot test broker logins (sudo apt install mosquitto-clients)"
+fi
+
+section "Web server (step A7)"
+WEBPORT=$(grep -h '^port:' config.d/server.yaml 2>/dev/null | awk '{print $2}')
+[[ -z "$WEBPORT" ]] && WEBPORT=8080
+if have curl; then
+  HOMECODE=$(curl -s -o /dev/null -w '%{http_code}' -m 5 http://localhost:${WEBPORT}/ 2>/dev/null)
+  if [[ "$HOMECODE" == "000" ]]; then
+    item "nothing is answering on port ${WEBPORT} - the server is not running"
+    item "  (expected if you have not reached step A7 yet; otherwise start it, or see the frugaliot service below)"
+  else
+    item "GET /            -> ${HOMECODE} (expect 200)"
+    item "GET /config.json -> $(curl -s -o /dev/null -w '%{http_code}' -m 5 http://localhost:${WEBPORT}/config.json 2>/dev/null) (expect 401 when not logged in)"
+    [[ "$HOMECODE" != "200" ]] && problem "The web server answered ${HOMECODE} rather than 200 on port ${WEBPORT}"
+  fi
+else
+  item "curl not installed - cannot test the web server"
+fi
+
+section "Logged data (step A9)"
+if [[ -d data ]]; then
+  DATADIRS=(data/*(N/))
+  if (( ${#DATADIRS} )); then
+    for d in $DATADIRS; do
+      item "${d}: $(find $d -type f 2>/dev/null | wc -l | tr -d ' ') files, newest $(ls -t $d/**/*(N.om[1]) 2>/dev/null | head -1)"
+    done
+  else
+    item "data/ is empty - no readings logged yet, which is expected until a node reports (step A9)"
+  fi
+fi
+
+section "Summary"
+if (( ${#PROBLEMS} == 0 )); then
+  item "No problems detected by these checks."
+else
+  for p in $PROBLEMS; do item "PROBLEM: $p"; done
+fi
+print -r -- ""

package/scripts/init.zsh

@@ -28,11 +28,24 @@ if [[ ! -f "${PKG}/frugal-iot-createdb.sql" ]]; then
 fi
 
 # ---- 1. Configuration files, copied only if not already here ----
-COPIED=()   # which files this run actually created, so existing ones are never rewritten below
+COPIED=()      # which files this run actually created, so existing ones are never rewritten below
+DIFFER_TO=()   # files kept that no longer match what this release ships, and where to compare them
+DIFFER_FROM=()
+# copy_if_missing <packaged file> <local file> [compare]
+# "compare" asks for a warning when the file exists but differs from the one this release ships.
+# Only pass it for files nobody is expected to edit - config.yaml, mqtt.yaml and server.yaml are
+# this installation's own settings, so of course they differ, and warning about them every time
+# would train you to ignore the warning that matters.
 copy_if_missing() {
-  local from=$1 to=$2
+  local from=$1 to=$2 compare=${3:-}
   if [[ -e "$to" ]]; then
-    echo "  kept    ${to} (already present)"
+    if [[ "$compare" == compare && -e "$from" ]] && ! cmp -s "$from" "$to"; then
+      DIFFER_TO+=("$to")
+      DIFFER_FROM+=("$from")
+      echo "  kept    ${to} (DIFFERS from this release - see below)"
+    else
+      echo "  kept    ${to} (already present)"
+    fi
   elif [[ -e "$from" ]]; then
     mkdir -p "${to:h}"
     cp "$from" "$to"
@@ -46,12 +59,14 @@ copy_if_missing "${PKG}/config.yaml" "./config.yaml"
 for f in logger.yaml mqtt.yaml server.yaml; do
   copy_if_missing "${PKG}/config.d/${f}" "./config.d/${f}"
 done
+# The schema describes the sensor types the software understands, so a release changing it matters
 for f in "${PKG}"/config.d/schema/*.yaml(N); do
-  copy_if_missing "$f" "./config.d/schema/${f:t}"
+  copy_if_missing "$f" "./config.d/schema/${f:t}" compare
 done
-# Copied so that mosquitto.conf and frugaliot.service can be edited and installed from here
+# Copied so that mosquitto.conf and frugaliot.service can be edited and installed from here. Worth
+# comparing: these get copied on somewhere else (/etc/...), where an old version lingers unnoticed.
 for f in "${PKG}"/extras/*(N); do
-  copy_if_missing "$f" "./extras/${f:t}"
+  copy_if_missing "$f" "./extras/${f:t}" compare
 done
 
 # ---- 1a. Point server.yaml at the web client this instance actually has ----
@@ -104,6 +119,18 @@ else
   echo "  created ${DB}"
 fi
 
+if (( ${#DIFFER_TO} )); then
+  echo
+  echo "These files were left as you have them, but this release ships a different version:"
+  for i in {1..${#DIFFER_TO}}; do
+    echo "  ${DIFFER_TO[$i]}"
+    echo "    compare with:  diff ${DIFFER_TO[$i]} ${DIFFER_FROM[$i]}"
+  done
+  echo "Usually that just means you edited it, and there is nothing to do. But a release can also"
+  echo "change one of these files - and anything installed elsewhere from it, such as"
+  echo "/etc/mosquitto/conf.d/frugal-iot.conf, keeps the old content until you copy it again."
+fi
+
 echo
 echo "Ready. Next:"
 echo "  npx frugal-iot-setpassword superuser <password>            # so you can log in as the administrator"