frugal-iot-server 0.3.7 → 0.3.9

package/INSTALLATION.md

@@ -40,6 +40,8 @@ Any of these work. What differs is only how long the install takes, and how much
 A Zero W runs the server perfectly well once installed. It is *installing* that is slow, because a
 single 1 GHz core has to unpack several hundred packages and compile one of them.
 
+
+---
 ### 0. What you need before you start
 
 **Hardware**
@@ -125,6 +127,7 @@ From your laptop's terminal:
 ```
 ssh pi@frugaliot.local
 ```
+> **Pi Zero W:** This can take a long time to connect, try `ping frugaliot.local` first to check it is alive
 
 Say `yes` to the fingerprint question, then give the password you set in Imager.
 
@@ -175,6 +178,44 @@ the password — either way `nmtui` is the fix, and it is not worth more time th
 (`sudo nmcli device wifi list` shows each nearby network's SSID and whether it is WPA2 or WPA3;
 WPA3 cannot use a derived key at all.)
 
+## 3-8. The short way: one script
+
+Steps 3 to 8 below can be done for you. Once you can ssh into the Pi (steps 1 and 2, which need a
+person with an SD card):
+
+```
+curl -fSLO https://raw.githubusercontent.com/mitra42/frugal-iot-server/main/scripts/install-pi.sh
+bash install-pi.sh --org myfarm --name "My Farm" --email you@example.com --phone +61123456789
+```
+
+It asks for anything it needs that you did not pass, says what it is doing as it goes, and stops at
+the first thing that fails - leaving a log, and telling you to run `frugal-iot-diagnostic`. Running
+it again after a failure carries on rather than starting over, which matters on a Pi Zero W where one
+step takes about 40 minutes. `--dry-run` checks the arguments and shows the plan without touching the
+machine.
+
+**Passwords** are asked for, not invented, unless you say otherwise: it prompts for each (twice, not
+echoed) and generates one if you just press Enter. Pass them as `--superuser-password`,
+`--login-password` and `--broker-password` to skip the prompts, `--random-passwords` to have all
+three generated, or `--yes` to ask nothing at all — which is what to use over ssh with no terminal.
+A password you chose is not written to the log; a generated one has to be, since otherwise you would
+have no way of knowing it.
+
+**It tells you whether a reboot is needed**, rather than leaving you to wonder — it notices a kernel
+or boot-firmware package in the upgrade, the images in `/boot` changing underneath it, and
+`/var/run/reboot-required`. If none of those happened it says so, and there is nothing more to do.
+
+Fetch it with `curl -O` and then run it, rather than piping curl into bash - piped, it has no
+terminal to ask questions at.
+
+It finishes with a live server and a tested broker. It cannot do step 9, pointing your nodes at it,
+because a node learns its broker by being flashed with it - so it prints the settings to give them.
+Nor step 10, HTTPS.
+
+The steps below are what it does, in the same order, if you would rather do it by hand or need to
+understand what went wrong, otherwise skip to Step 9.
+
+
 ### 3. Update the operating system and install the prerequisites
 
 Logged in, bring the system up to date and reboot:
@@ -654,6 +695,101 @@ mosquitto_sub -h localhost -u myfarm -P '<broker-password>' -t '#' -v
 Every reading from every node should scroll past. Seeing anything here also proves the broker's
 port 1883 is reachable from off the Pi, which is what the nodes need.
 
+### 9a. Optional: outgoing mail, so people can reset a forgotten password
+
+Without this, a forgotten password can only be fixed by you, on the Pi, with
+`npx --no frugal-iot-setpassword <username> <new-password>`. The login page's "Forgot password?"
+link says "Password reset is not available on this server" rather than pretending to send anything.
+
+An offline Pi cannot send mail at all, so skip this unless it has internet access.
+
+Mail sent straight from a home broadband connection is almost always treated as spam, so this
+relays through somebody else's SMTP server rather than sending directly. Gmail is the usual choice
+and is written out in full below; any other provider works the same way.
+
+#### With Gmail
+
+Use a Google account you are willing to have the Pi send as. A separate account for the purpose is
+better than your own mailbox — the password ends up in a file on an SD card in a shed.
+
+**1. Turn on 2-Step Verification** on that account, at
+<https://myaccount.google.com/signinoptions/twosv>.
+
+This is not optional and it is where most people get stuck: **app passwords do not exist until
+2-Step Verification is on**. Until then the page in step 2 simply says the setting is not available,
+without explaining why.
+
+**2. Create an app password** at <https://myaccount.google.com/apppasswords>.
+
+Type a name for it — anything you will recognise later, such as `Frugal IoT Pi` — and create it.
+Google shows you **16 lowercase letters, in four groups of four**, once. Copy them now; you cannot
+come back and read it again, only delete it and make another.
+
+The groups are only there to make it readable. **Type it into the config with the spaces removed**,
+as one 16-character word.
+
+**3. Fill in `config.d/email.yaml`:**
+
+```yaml
+host: smtp.gmail.com
+port: 587
+user: yourname@gmail.com
+pass: "abcdefghijklmnop"          # the 16 characters from step 2, spaces removed
+from: Frugal IoT <yourname@gmail.com>
+```
+
+Things that catch people out with Gmail specifically:
+
+* **`from:` must be the same address as `user:`.** Gmail rewrites the sender to the account you
+  authenticated as, so a different address here does not fail — it quietly arrives as something
+  else, which is harder to diagnose than an error. (An address you have set up under Gmail's
+  "Send mail as" is the one exception.)
+* **`user:` is the full address**, including `@gmail.com`, not just the part before it.
+* **`Username and Password not accepted`** in the server's log means the login password was used
+  instead of an app password. Gmail has not accepted account passwords over SMTP for years.
+* **Port 465 also works** if 587 is blocked where the Pi is; the software works out the encryption
+  from the port number, so change nothing else.
+* **A free Gmail account can send around 500 messages a day.** Password resets will not come close.
+* **Google Workspace administrators can switch app passwords off** for a whole domain. If step 2
+  offers you nothing on a work account, that is why, and you will need a real SMTP relay instead.
+
+#### With any other provider
+
+The same four settings, with that provider's SMTP host:
+
+```yaml
+host: smtp.example.org
+port: 587           # 465 is implicit TLS, 587 and 25 are STARTTLS
+user: frugaliot@example.org
+pass: an-app-password-not-your-login-password
+from: Frugal IoT <frugaliot@example.org>
+```
+
+Fastmail, Zoho, Proton (via its bridge) and the rest all issue app passwords in much the same way,
+and for the same reason: one leaking off a Pi then costs you one mailbox rather than the account.
+
+#### Checking it, either way
+
+Restart the server and look at its output:
+
+* `Sending mail via smtp.gmail.com:587 as ...` — configured.
+* `Not sending mail (no host/from in config.d/email.yaml) ...` — it did not find a `host` and a
+  `from`, so the reset link will say it is unavailable.
+
+Then use "Forgot password?" on the login page with your own account. If the mail never arrives, the
+reason is in the server's log — the page deliberately gives the same answer whether or not the
+address is known to it, so it cannot be used to find out who has an account here.
+
+Two more things worth knowing:
+
+* `pass` sits in a file on the SD card in plain text. Give that mailbox nothing else to lose.
+* If the Pi is behind a proxy that does not set `X-Forwarded-Host`, the link in the mail will point
+  at the wrong address. Set `baseurl: https://your.address` in the same file.
+
+The reset code itself is stored nowhere — it is a hash of the account, its current password and the
+time, valid between five and ten minutes and dead the moment it is used. There is no table to
+maintain and nothing to clean up. Restarting the server invalidates any code already sent.
+
 ### 10. HTTPS and over-the-air firmware updates
 
 **To be written.** Everything above gives you a plain HTTP server on your local network, which is
@@ -667,6 +803,238 @@ That section will cover: a DNS name for the Pi, port forwarding on the router, n
 proxy in front of port 8080, and certificates from Let's Encrypt via certbot. Both require the Pi
 to have internet access, which is the opposite of the offline case this guide is aimed at.
 
+### 11. Optional: bridge this Pi to a production server
+
+Everything above gives a self-contained Pi. This step also relays its readings to a production
+Frugal IoT server, so the same nodes appear on a dashboard elsewhere and can be controlled from
+it, while the Pi carries on working on its own whenever the link is down. Skip it if you do not
+want that — nothing else depends on it.
+
+The relaying is done by the broker, not by the server: Mosquitto has a "bridge" feature for
+connecting to another broker, and the server, the logger and the nodes need no configuration
+change at all. This installation ships an example bridge configuration in
+`extras/mosquitto-bridge.conf.example`, which explains each setting in place; this section is the
+surrounding work.
+
+It comes in three parts, and only the first is a one-off:
+
+* **11a — preparing a production server to accept bridges.** A TLS listener, a certificate, and an
+  access-control file. Done once for a given production server, and not again until there is
+  another one.
+* **11b — authorizing this Pi on that server.** One command, run there, once per Pi.
+* **11c — pointing this Pi at it.** One command, run here, once per Pi.
+
+**What you get, and what you do not.** While the link is up, readings appear on production within
+a second or so. While it is down, the Pi records everything as usual and production simply has a
+gap — the readings taken during an outage never reach it. That is a deliberate choice: the
+alternative is for the broker to hold them and deliver the backlog on reconnect, but production
+timestamps each reading as it arrives, so a day's backlog would arrive claiming to have happened
+in the instant the link returned. A gap is honest; that would not be. Controls are treated the
+other way round, and *are* queued, so turning something on from production while the Pi is offline
+takes effect when it reconnects.
+
+#### 11a. Preparing a production server to accept bridges
+
+Once per production server, not per Pi — and if bridging to `frugaliot.naturalinnovation.org`
+this has already been done. If so, skip to 11b.
+
+**Add a listener for bridges.** A Mosquitto bridge speaks MQTT or MQTT-over-TLS and cannot use
+WebSockets, so the existing `wss://` path that browsers and servers use cannot carry this. It needs
+a TLS listener of its own on 8883, and the sensible certificate to give it is the one the web
+server already uses for the same hostname — one certificate and one renewal, rather than two.
+
+Find where that certificate actually is, rather than assuming: certbot's own layout under
+`/etc/letsencrypt/live/` is only one of several, and a wildcard certificate covering more than one
+domain is often installed by hand somewhere else entirely.
+
+```
+sudo grep -rn 'SSLCertificateFile\|SSLCertificateKeyFile' /etc/apache2/   # Apache
+sudo nginx -T 2>/dev/null | grep ssl_certificate                          # nginx
+```
+
+Take the paths for the vhost serving this server's hostname — `sudo apache2ctl -S` says which
+vhost that is — and add to production's Mosquitto configuration:
+
+```
+listener 8883
+protocol mqtt
+certfile <the SSLCertificateFile path>
+keyfile  <the SSLCertificateKeyFile path>
+```
+
+then open 8883 on the firewall.
+
+> **`certfile` must contain the intermediate certificate as well as the server's own**, or bridges
+> will refuse to connect even though browsers are happy — a browser can often fill in a missing
+> intermediate from cache and a bridge never can. `sudo grep -c 'BEGIN CERTIFICATE' <certfile>`
+> should report 2 or more. If it reports 1, concatenate the issuer's chain onto a copy and point
+> `certfile` at that.
+
+> **The key's permissions are what usually goes wrong.** Unlike a web server, which reads its key
+> as root at startup, Mosquitto drops to the `mosquitto` user *first* and reads the key afterwards,
+> so a key left `root:root 0600` stops the broker starting:
+>
+> ```
+> Error: Unable to load server key file "...". Check keyfile.
+> OpenSSL Error[0]: error:8000000D:system library::Permission denied
+> ```
+>
+> On Debian the tidy fix is the `ssl-cert` group, which exists for exactly this and which
+> `/etc/ssl/private` is already set up to admit:
+>
+> ```
+> sudo adduser mosquitto ssl-cert
+> sudo chgrp ssl-cert <keyfile>
+> sudo chmod 640 <keyfile>
+> ```
+>
+> The web server is unaffected — root still reads the key as before. If you would rather not widen
+> access to a directory holding keys for other domains, copy the certificate and key into
+> `/etc/mosquitto/certs` owned `mosquitto:mosquitto` instead, and refresh that copy on renewal.
+
+**Add one line to however this certificate gets renewed:**
+
+```
+systemctl reload mosquitto
+```
+
+Mosquitto keeps the certificate it loaded at startup, so without this the broker goes on serving
+the old one until it happens to be restarted, and bridges start failing once it expires. A reload
+is enough — it picks up a replaced certificate without dropping a single connection. Where that
+line goes depends on how the certificate is renewed: a certbot deploy hook in
+`/etc/letsencrypt/renewal-hooks/deploy/`, `acme.sh`'s `--reloadcmd`, or, if the certificate is
+installed by hand, the written procedure for doing that.
+
+> Note the asymmetry with the Pi end below: a *certificate* change needs only `reload`, but adding
+> or changing a *bridge* needs a full `restart`, because Mosquitto does not reload bridges.
+
+**Turn on access control.** Without it, any account that can log in to the broker can publish and
+subscribe anywhere on it, so a per-Pi account would be no more confined than the organization's
+own. This is the part to plan carefully, because Mosquitto's ACL file is **deny-by-default**: the
+moment the broker names an `acl_file`, every existing account with no entry in it stops working.
+So the file that restricts new bridge accounts has to grant the existing organizations what they
+already have, in the same change.
+
+```
+sudo tee /etc/mosquitto/aclfile >/dev/null <<'EOF'
+# Deny-by-default: an account with no rule here can connect but reach no topic at all.
+
+# Each organization reaches only its own topic tree. "%u" is the connecting account name, and the
+# first element of every Frugal IoT topic is the organization id - so this is what the nodes,
+# dashboards and the server's own logger already do. It just stops being optional.
+pattern readwrite %u/#
+EOF
+sudo chmod 644 /etc/mosquitto/aclfile
+```
+
+`chmod 644` matters: like the password file and the TLS key, the broker reads this *after* dropping
+to the `mosquitto` user, so a root-only file stops it starting. Then name it in the configuration
+and restart:
+
+```
+echo 'acl_file /etc/mosquitto/aclfile' | sudo tee /etc/mosquitto/conf.d/zy-frugal-iot-acl.conf
+sudo systemctl restart mosquitto
+```
+
+Check every existing organization can still log in **before** connecting any Pi — a wrong ACL file
+locks out every node and dashboard at once, and the reason appears only in Mosquitto's own log. On a
+Frugal IoT server the quickest check is its journal, which logs one line per organization:
+
+```
+sudo journalctl -u frugaliot -n 40 --no-pager | grep -iE "mqtt |not authoris"
+```
+
+Every organization should show `connect` and none should show `not authorized`. To back the change
+out, delete `/etc/mosquitto/conf.d/zy-frugal-iot-acl.conf` and restart.
+
+Nothing needs adding here per Pi: `frugal-iot-addbridge-prod` in 11b appends each bridge's own rule.
+
+> One thing this costs: `pattern readwrite %u/#` also denies `$SYS`, so the production broker can no
+> longer tell anyone which bridges are currently connected. The Pi still reports its own bridge
+> state to `frugal-iot-diagnostic`, which is where you would look anyway, so this is a fair trade —
+> but it is why production's dashboard cannot show a site as up or down.
+
+#### 11b. Authorizing this Pi (run on the production server)
+
+Once per Pi. On the production server, from its own directory:
+
+```
+npx --no frugal-iot-addbridge-prod <org-id> <site-name>
+```
+
+The site name only distinguishes one Pi from another within an organization — `northfield`,
+`shed`, `village2` — and becomes part of the account name. The script creates the broker account,
+adds its access-control rule, reloads the broker, and prints both the password and the exact
+command to run in 11c.
+
+Three things it checks, because each of them is a way this quietly fails:
+
+* **The organization must already exist on the production server**, or the readings will arrive at
+  its broker and nothing will record them — that server's logger subscribes per organization,
+  driven by the files in its `config.d/organizations/`. If it is missing, add it there first with
+  `frugal-iot-addorganization`, using the *same* organization id as this Pi.
+* **Each Pi gets its own account**, rather than sharing the organization's. The organization's
+  broker password is handed to every browser that logs in, so it is not a secret; a per-Pi account
+  can be confined to one site and revoked on its own.
+* **An existing account is left alone.** Re-running it for a site that already has one is refused
+  rather than quietly issuing a new password, which would stop that Pi relaying without anyone
+  touching it.
+
+The password is shown once and is not stored anywhere you can read back — copy it before you lose
+the output.
+
+#### 11c. Bridging this Pi (run on the Pi)
+
+Once per Pi. Run the command 11b printed, from this installation's directory:
+
+```
+cd ~/frugal-iot
+npx --no frugal-iot-addbridge-pi <org-id> <prod-host> bridge-<site-name>
+```
+
+It asks for the password rather than taking it on the command line, so it stays out of your shell
+history. Then it checks the far end is reachable and that its certificate is valid for that name,
+writes `/etc/mosquitto/conf.d/frugal-iot-bridge.conf`, restarts the broker, and reports whether the
+bridge actually connected.
+
+The restart is unavoidable: Mosquitto does not pick up bridges on a reload signal, so `reload`
+appears to succeed and changes nothing. It briefly disconnects every node, which they recover from
+on their own.
+
+> The configuration goes into `/etc/mosquitto/conf.d/` and not into this installation's
+> `config.d/`, because it holds a password for the production server. Everything under `config.d/`
+> is served to any logged-in browser by `/config.json`, so a credential put there would not stay
+> private. The file is written mode 600 for the same reason.
+
+To undo it: `sudo rm /etc/mosquitto/conf.d/frugal-iot-bridge.conf && sudo systemctl restart
+mosquitto`. The Pi goes back to being self-contained and keeps everything it has recorded.
+
+#### Check it
+
+```
+npx --no frugal-iot-diagnostic
+```
+
+The **Bridge to a production server** section names the bridge, lists what it relays, and reports
+whether it is connected. `CONNECTED` means the link is up. `DOWN`, or a report that it has never
+connected, means the address, the credentials or the certificate is wrong — and Mosquitto's own log,
+quoted earlier in the same output, says which.
+
+Then confirm at the far end: log in to production's dashboard and look for this Pi's nodes. They
+should appear within a couple of minutes, which is however long it takes each node to next report.
+
+**One oddity worth expecting.** Each time the bridge reconnects, the most recent value of every
+reading is re-sent to production and recorded there with the reconnect time rather than the time
+it was measured. That is one row per topic per outage, and it happens because nodes publish
+readings retained. The `duplicates:` rules in `config.d/schema/topics.yaml` absorb it when the
+value has not moved and the outage was short; after a long outage the row is written.
+
+**The clock matters more once a Pi is bridged**, because its readings now sit alongside
+production's. A bridged Pi has internet access whenever the link is up, so it sets its time from
+the network and the problem below mostly goes away — but it still comes up after a power cut
+believing whatever it last saved, and anything recorded before it reaches a time server carries
+that wrong time.
+
 ### Known limitation: the clock on an offline Pi
 
 A Raspberry Pi has no battery-backed clock. While it has internet access it sets its time from the
@@ -755,6 +1123,8 @@ broken. It covers most of the checks scattered through this guide, in one pass:
 * whether Mosquitto and the `frugaliot` service are running, with the last lines of the journal
   **and of Mosquitto's own log**, which is where its startup errors actually appear
 * whether the broker refuses a wrong password, and accepts each organization's real one
+* whether a bridge to a production server is configured, what it relays, and whether it is
+  connected right now (step 11) — the bridge password is not printed
 * whether ports 1883, 9012 and 8080 are listening, and whether the web server answers
 * whether the broker host in `config.d/mqtt.yaml` resolves from this machine
 * how much data the logger has written
@@ -770,6 +1140,34 @@ If you have not reached step 4 yet, the command does not exist. Copy
 [scripts/diagnostic.zsh](https://github.com/mitra42/frugal-iot-server/blob/main/scripts/diagnostic.zsh)
 to the Pi and run `zsh diagnostic.zsh` instead.
 
+### A topic that will not go away
+
+Nodes publish almost everything **retained**, meaning the broker keeps the last value and gives it
+to every new subscriber — which is how a dashboard shows a reading immediately instead of waiting
+for the next one. The catch is that a topic published by mistake outlives the mistake. Misspell a
+field, rename a module, flash a node with the wrong id, and that topic sits on the broker for ever,
+appearing on every dashboard. Fixing the node does not remove it, and neither does restarting the
+broker: the only way is to publish an empty message to that exact topic.
+
+```
+cd ~/frugal-iot
+npx --no frugal-iot-clearretained 'myfarm/lotus/+/sht/temperture/#'
+```
+
+Quote the pattern or the shell will expand it. That lists what is retained and changes nothing;
+add the word `delete` at the end of the same command to remove it.
+
+> `delete` is a bare word rather than a `--delete` flag on purpose. npm parses the command line of
+> an `npx` invocation itself and swallows any `--flag` it does not recognise, so `--delete` would
+> never reach the script and it would quietly list instead of deleting.
+
+> Look before deleting. A node's `min`, `max`, `color` and `wired` settings are retained messages
+> too, and they are how the dashboard knows how to draw it — delete those and the node has to be
+> restarted before it looks right again. A pattern of just `#` is refused outright.
+
+On a Pi that is bridged to a production server (step 11), deleting here deletes there too: the
+bridge forwards the empty message like any other.
+
 ---
 
 ## Upgrading

package/README.md

@@ -3,7 +3,8 @@
 ## Installation
 
 For a self-contained server on a Raspberry Pi - including the MQTT broker, and starting from a
-blank SD card - see [INSTALLATION.md](INSTALLATION.md).
+blank SD card - see [INSTALLATION.md](INSTALLATION.md), which also has a script that does the whole
+thing in one go once you can ssh in.
 
 On a unix box that is already running ...
 
@@ -20,15 +21,20 @@ npm install frugal-iot-server
 npx --no frugal-iot-init
 ```
 `frugal-iot-init` copies in the configuration files, creates the `data`, `ota` and
-`config.d/organizations` directories, and creates the database. It never overwrites anything
-already there, so run it again after an upgrade to pick up newly added configuration.
+`config.d/organizations` directories, points `config.d/server.yaml` at the web client npm just
+installed, and creates the database. It never overwrites anything already there, so run it again
+after an upgrade to pick up newly added configuration.
 
 Edit `config.yaml` and `config.d/mqtt.yaml` (which broker to talk to) if the defaults do not suit.
 
-The database is created with a `superuser` account that has no password, so give it one:
+A newly created database has two accounts in it, `everyone` (which holds the permissions every
+logged-in user gets) and `superuser`, which has no password - so give it one:
 ```
 npx --no frugal-iot-setpassword superuser "<a-good-password>"
 ```
+On a database that existed before this account was introduced, that command reports
+`no user 'superuser'`, because id 1 was already taken by whoever registered first - and that
+account already has the superuser's privileges. Log in as that one instead.
 
 Then add an organization - this writes its yaml file into `config.d/organizations`, creates a
 login account of the same name with its permissions, and sets its password on the MQTT broker:
@@ -52,24 +58,49 @@ readYamlConfigFile ./config.yaml
 ... and then reading each of the other files
 
 Broker wss://frugaliot.naturalinnovation.org/wss - organizations: dev
+Not logging HTTP requests (morgan: false in config.d/server.yaml)
 Doing OTA updates at /ota_update from ...some path.../ota
-Serving /node_modules from ./node_modules
+Serving /node_modules from node_modules/frugal-iot-client/node_modules then ./node_modules
 User Database exists
 Opened user database
 Exec-ed starting SQL
+Created logger client for API integration
+Mounted API routes at /api
 Serving /data from ./data
 Server starting on port 8080
+Logger not reporting individual messages (verbose: false in config.d/logger.yaml)
+Collecting readings in memory, writing them out every 300 seconds
 mqtt dev connecting
 mqtt dev connect
 Subscribing topic dev/# 0
-Received dev/lotus/esp8266-85ea2b/humidity   71.8
 ```
-Where the broker and organizations are reported back, 
-then it successfully connects to the mqtt server
-and receives data from nodes attached to it. 
+Where the broker and organizations are reported back,
+then it successfully connects to the mqtt server.
+
+**It then goes quiet, and that is what should happen.** A new installation is set up to write as
+little as it can, because the usual home for this is a Raspberry Pi running from an SD card, and
+cards wear out from being written to. So it does not log a line per message received, nor a line
+per web request, and readings are collected in memory and written out every five minutes rather
+than one at a time. Seeing nothing after the lines above does not mean nothing is arriving.
+
+To watch the messages while setting up, or to work out why a node's readings are not appearing,
+turn it back on in `config.d/logger.yaml` and restart:
+```
+verbose: true
+```
+and each message reappears as `Received dev/lotus/esp8266-85ea2b/humidity   71.8`. `morgan: true`
+in `config.d/server.yaml` does the same for web requests. Turn both off again afterwards.
+There is more on what gets written, and how to see how much, under
+[Wear and tear on the SD card](INSTALLATION.md#wear-and-tear-on-the-sd-card).
 
 Open a browser pointing at for example `localhost:8080` and you should see the UI.
 
+If something is not right, this reports on the whole installation - versions, configuration,
+broker logins, what has been logged - and changes nothing:
+```
+npx --no frugal-iot-diagnostic
+```
+
 To upgrade later: `npm update frugal-iot-server` then `npx --no frugal-iot-init`.
 
 #### Developing the server, client or logger
@@ -85,22 +116,66 @@ The commands above have in-repo equivalents - `scripts/addorganization.zsh` and
 `scripts/setpassword.zsh` - and the server is `node frugal-iot-server.js`. All of them work on
 the directory you run them in, which for a clone is the top of the repo.
 
-To work on the client or logger from sibling checkouts, switch two places to their commented-out
-development lines:
-- `config.d/server.yaml` - `htmldir` and `nodemodulesdir` point at `../frugal-iot-client`
-- `frugal-iot-server.js` - the `MqttLogger` import points at `../frugal-iot-logger/index.js`
+To work on the client or the logger from sibling checkouts as well, link them in. This needs no
+edit to any file, so there is nothing to remember to change back:
+```
+cd ../frugal-iot-client && npm link
+cd ../frugal-iot-logger && npm link
+cd ../frugal-iot-server && npm link frugal-iot-client frugal-iot-logger
+```
+The server then loads both from your checkouts. `ls -l node_modules/frugal-iot-*` shows whether the
+links are in place - a later `npm install` or `npm update` can replace them with the published
+packages, which looks like your changes having no effect.
+
+There are commented-out alternatives in `config.d/server.yaml` (for `htmldir`) and in
+`frugal-iot-server.js` (for the `MqttLogger` import) that do the same job by editing instead. They
+work, but they are easy to publish by accident, so prefer the links.
 
-Take care not to commit those local switches, and note that `npm publish` packages your working
-tree - so check with `npm pack --dry-run` before publishing.
+Before publishing, run:
+```
+npm run prerelease
+```
+It checks the sensor schema and copies it into the examples that ship with `frugal-iot-logger`, sets
+the service worker's cache version from the client version this release installs, and refuses if:
+- the package is wired to a local checkout - a `file:` dependency, an `npm link`, a switched import
+  or a switched `htmldir`;
+- it requires a version of the client or logger that has not been published (a warning if published
+  but not the newest, since a fresh install resolves to the newest anyway while an existing one may
+  not move);
+- anything git ignores would be published. `files` in `package.json` is an allow-list, which fails
+  closed, but a file sitting in one of the listed directories goes out whether or not you meant it
+  to - which is how a working copy of the user database was published in 0.3.2.
+
+It also lists anything in the repository that would *not* be published and is not already accounted
+for in `scripts/not-published.txt`, so a newly added file that installers need does not go missing
+unnoticed. `npm run check-files` runs just that part.
+
+`npm run check-schema` on its own reports sensor topics that do not say whether they are logged, or
+that are logged with no rule about how often - both of which are easy to add by accident and
+expensive on an SD card.
 
 #### Running a production server
 To set it up as a service that runs at startup (and instructions vary between flavors of Linux)
 
-copy and edit `extras/frugaliot.service` to `/etc/systemd/system/frugaliot.service`
-you'll need to change the user, the directory you installed into, and `ExecStart`
+copy and edit `extras/frugaliot.service` to `/etc/systemd/system/frugaliot.service`. As shipped it
+matches the Raspberry Pi install in [INSTALLATION.md](INSTALLATION.md) - user `pi`, installed in
+`/home/pi/frugal-iot` - so on any other machine change `User`, `WorkingDirectory` and `ExecStart`
+to the account and directory you installed into.
+```
+sudo cp extras/frugaliot.service /etc/systemd/system/frugaliot.service
+sudo systemctl daemon-reload
+sudo systemctl enable --now frugaliot
+```
+`daemon-reload` is the step that is easy to miss - without it systemd keeps using the version it
+read before, and your edits appear to do nothing. After that `service frugaliot start|stop|restart`
+works as usual, and `systemctl status frugaliot` reports whether it is running.
+
+If you already have the service installed and only want to change one setting, `sudo systemctl edit
+frugaliot` writes an override rather than touching the file, which is easier to undo
+(`sudo systemctl revert frugaliot`).
 
-You can run`service frugaliot start` to start it
-and `systemctl enable frugaliot` to make sure it starts at boot. 
+Stop it with `service frugaliot stop` rather than killing it, so that readings still held in memory
+are written out first.
 
 Note that this will give you a HTTP server, but OTA on ESP32 requires HTTPS.
 

package/config.d/email.yaml

@@ -0,0 +1,23 @@
+# Outgoing mail. Only used by the password reset, which is unavailable until this is filled in -
+# the login page says so rather than pretending to have sent something.
+#
+# Leave it commented out on a server that should not send mail.
+#
+# For Gmail - the common case, written out step by step in INSTALLATION.md section 9a:
+#   1. Turn on 2-Step Verification: https://myaccount.google.com/signinoptions/twosv
+#      App passwords do not exist until you do, which is where most people get stuck.
+#   2. Create an app password:      https://myaccount.google.com/apppasswords
+#      You get 16 lowercase letters in four groups of four, shown once. Remove the spaces.
+#   3. "from" must be the same address as "user" - Gmail silently rewrites anything else.
+#
+# host: smtp.gmail.com
+# port: 587           # 465 is implicit TLS, 587 and 25 are STARTTLS; secure: is derived from this
+# user: yourname@gmail.com
+# pass: "abcdefghijklmnop"        # the app password, NOT your Google account password
+# from: Frugal IoT <yourname@gmail.com>
+#
+# Any other provider is the same four settings with its own SMTP host.
+#
+# The reset email contains a link back to this server. Set baseurl if the server cannot work out
+# its own public address from the request - behind a proxy that does not set X-Forwarded-Host, say.
+# baseurl: https://frugaliot.example.org