frugal-iot-server 0.3.8 → 2.0.0

package/INSTALLATION.md

@@ -40,42 +40,6 @@ Any of these work. What differs is only how long the install takes, and how much
 A Zero W runs the server perfectly well once installed. It is *installing* that is slow, because a
 single 1 GHz core has to unpack several hundred packages and compile one of them.
 
-## The short way: one script
-
-Steps 3 to 8 below can be done for you. Once you can ssh into the Pi (steps 1 and 2, which need a
-person with an SD card):
-
-```
-curl -fsSLO https://raw.githubusercontent.com/mitra42/frugal-iot-server/main/scripts/install-pi.sh
-bash install-pi.sh --org myfarm --name "My Farm" --email you@example.com --phone +61123456789
-```
-
-It asks for anything it needs that you did not pass, says what it is doing as it goes, and stops at
-the first thing that fails - leaving a log, and telling you to run `frugal-iot-diagnostic`. Running
-it again after a failure carries on rather than starting over, which matters on a Pi Zero W where one
-step takes about 40 minutes. `--dry-run` checks the arguments and shows the plan without touching the
-machine.
-
-**Passwords** are asked for, not invented, unless you say otherwise: it prompts for each (twice, not
-echoed) and generates one if you just press Enter. Pass them as `--superuser-password`,
-`--login-password` and `--broker-password` to skip the prompts, `--random-passwords` to have all
-three generated, or `--yes` to ask nothing at all — which is what to use over ssh with no terminal.
-A password you chose is not written to the log; a generated one has to be, since otherwise you would
-have no way of knowing it.
-
-**It tells you whether a reboot is needed**, rather than leaving you to wonder — it notices a kernel
-or boot-firmware package in the upgrade, the images in `/boot` changing underneath it, and
-`/var/run/reboot-required`. If none of those happened it says so, and there is nothing more to do.
-
-Fetch it with `curl -O` and then run it, rather than piping curl into bash - piped, it has no
-terminal to ask questions at.
-
-It finishes with a live server and a tested broker. It cannot do step 9, pointing your nodes at it,
-because a node learns its broker by being flashed with it - so it prints the settings to give them.
-Nor step 10, HTTPS.
-
-The steps below are what it does, in the same order, if you would rather do it by hand or need to
-understand what went wrong.
 
 ---
 ### 0. What you need before you start
@@ -163,6 +127,7 @@ From your laptop's terminal:
 ```
 ssh pi@frugaliot.local
 ```
+> **Pi Zero W:** This can take a long time to connect, try `ping frugaliot.local` first to check it is alive
 
 Say `yes` to the fingerprint question, then give the password you set in Imager.
 
@@ -213,6 +178,44 @@ the password — either way `nmtui` is the fix, and it is not worth more time th
 (`sudo nmcli device wifi list` shows each nearby network's SSID and whether it is WPA2 or WPA3;
 WPA3 cannot use a derived key at all.)
 
+## 3-8. The short way: one script
+
+Steps 3 to 8 below can be done for you. Once you can ssh into the Pi (steps 1 and 2, which need a
+person with an SD card):
+
+```
+curl -fSLO https://raw.githubusercontent.com/mitra42/frugal-iot-server/main/scripts/install-pi.sh
+bash install-pi.sh --org myfarm --name "My Farm" --email you@example.com --phone +61123456789
+```
+
+It asks for anything it needs that you did not pass, says what it is doing as it goes, and stops at
+the first thing that fails - leaving a log, and telling you to run `frugal-iot-diagnostic`. Running
+it again after a failure carries on rather than starting over, which matters on a Pi Zero W where one
+step takes about 40 minutes. `--dry-run` checks the arguments and shows the plan without touching the
+machine.
+
+**Passwords** are asked for, not invented, unless you say otherwise: it prompts for each (twice, not
+echoed) and generates one if you just press Enter. Pass them as `--superuser-password`,
+`--login-password` and `--broker-password` to skip the prompts, `--random-passwords` to have all
+three generated, or `--yes` to ask nothing at all — which is what to use over ssh with no terminal.
+A password you chose is not written to the log; a generated one has to be, since otherwise you would
+have no way of knowing it.
+
+**It tells you whether a reboot is needed**, rather than leaving you to wonder — it notices a kernel
+or boot-firmware package in the upgrade, the images in `/boot` changing underneath it, and
+`/var/run/reboot-required`. If none of those happened it says so, and there is nothing more to do.
+
+Fetch it with `curl -O` and then run it, rather than piping curl into bash - piped, it has no
+terminal to ask questions at.
+
+It finishes with a live server and a tested broker. It cannot do step 9, pointing your nodes at it,
+because a node learns its broker by being flashed with it - so it prints the settings to give them.
+Nor step 10, HTTPS.
+
+The steps below are what it does, in the same order, if you would rather do it by hand or need to
+understand what went wrong, otherwise skip to Step 9.
+
+
 ### 3. Update the operating system and install the prerequisites
 
 Logged in, bring the system up to date and reboot:
@@ -500,9 +503,9 @@ npx --no frugal-iot-addorganization myfarm "My Farm" you@example.com +6112345678
 
 The arguments are: organization id, display name, your email, your phone (`+` and digits only),
 and a password. That one command writes `config.d/organizations/myfarm.yaml`, creates a login account
-named after the organization (`myfarm`), grants it admin rights, creates its OTA directory, **and adds
-the organization's account to the broker's password file** — which is why the broker had to be
-installed first.
+named after the organization (`myfarm`), grants it admin, read and write rights, creates its OTA
+directory, **and adds the organization's account to the broker's password file** — which is why the
+broker had to be installed first.
 
 * **Organization id `myfarm`** — must be 1–10 lower-case letters or digits. It becomes the first part
   of every MQTT topic, so it must match what your sensor nodes are configured to publish to.
@@ -692,6 +695,101 @@ mosquitto_sub -h localhost -u myfarm -P '<broker-password>' -t '#' -v
 Every reading from every node should scroll past. Seeing anything here also proves the broker's
 port 1883 is reachable from off the Pi, which is what the nodes need.
 
+### 9a. Optional: outgoing mail, so people can reset a forgotten password
+
+Without this, a forgotten password can only be fixed by you, on the Pi, with
+`npx --no frugal-iot-setpassword <username> <new-password>`. The login page's "Forgot password?"
+link says "Password reset is not available on this server" rather than pretending to send anything.
+
+An offline Pi cannot send mail at all, so skip this unless it has internet access.
+
+Mail sent straight from a home broadband connection is almost always treated as spam, so this
+relays through somebody else's SMTP server rather than sending directly. Gmail is the usual choice
+and is written out in full below; any other provider works the same way.
+
+#### With Gmail
+
+Use a Google account you are willing to have the Pi send as. A separate account for the purpose is
+better than your own mailbox — the password ends up in a file on an SD card in a shed.
+
+**1. Turn on 2-Step Verification** on that account, at
+<https://myaccount.google.com/signinoptions/twosv>.
+
+This is not optional and it is where most people get stuck: **app passwords do not exist until
+2-Step Verification is on**. Until then the page in step 2 simply says the setting is not available,
+without explaining why.
+
+**2. Create an app password** at <https://myaccount.google.com/apppasswords>.
+
+Type a name for it — anything you will recognise later, such as `Frugal IoT Pi` — and create it.
+Google shows you **16 lowercase letters, in four groups of four**, once. Copy them now; you cannot
+come back and read it again, only delete it and make another.
+
+The groups are only there to make it readable. **Type it into the config with the spaces removed**,
+as one 16-character word.
+
+**3. Fill in `config.d/email.yaml`:**
+
+```yaml
+host: smtp.gmail.com
+port: 587
+user: yourname@gmail.com
+pass: "abcdefghijklmnop"          # the 16 characters from step 2, spaces removed
+from: Frugal IoT <yourname@gmail.com>
+```
+
+Things that catch people out with Gmail specifically:
+
+* **`from:` must be the same address as `user:`.** Gmail rewrites the sender to the account you
+  authenticated as, so a different address here does not fail — it quietly arrives as something
+  else, which is harder to diagnose than an error. (An address you have set up under Gmail's
+  "Send mail as" is the one exception.)
+* **`user:` is the full address**, including `@gmail.com`, not just the part before it.
+* **`Username and Password not accepted`** in the server's log means the login password was used
+  instead of an app password. Gmail has not accepted account passwords over SMTP for years.
+* **Port 465 also works** if 587 is blocked where the Pi is; the software works out the encryption
+  from the port number, so change nothing else.
+* **A free Gmail account can send around 500 messages a day.** Password resets will not come close.
+* **Google Workspace administrators can switch app passwords off** for a whole domain. If step 2
+  offers you nothing on a work account, that is why, and you will need a real SMTP relay instead.
+
+#### With any other provider
+
+The same four settings, with that provider's SMTP host:
+
+```yaml
+host: smtp.example.org
+port: 587           # 465 is implicit TLS, 587 and 25 are STARTTLS
+user: frugaliot@example.org
+pass: an-app-password-not-your-login-password
+from: Frugal IoT <frugaliot@example.org>
+```
+
+Fastmail, Zoho, Proton (via its bridge) and the rest all issue app passwords in much the same way,
+and for the same reason: one leaking off a Pi then costs you one mailbox rather than the account.
+
+#### Checking it, either way
+
+Restart the server and look at its output:
+
+* `Sending mail via smtp.gmail.com:587 as ...` — configured.
+* `Not sending mail (no host/from in config.d/email.yaml) ...` — it did not find a `host` and a
+  `from`, so the reset link will say it is unavailable.
+
+Then use "Forgot password?" on the login page with your own account. If the mail never arrives, the
+reason is in the server's log — the page deliberately gives the same answer whether or not the
+address is known to it, so it cannot be used to find out who has an account here.
+
+Two more things worth knowing:
+
+* `pass` sits in a file on the SD card in plain text. Give that mailbox nothing else to lose.
+* If the Pi is behind a proxy that does not set `X-Forwarded-Host`, the link in the mail will point
+  at the wrong address. Set `baseurl: https://your.address` in the same file.
+
+The reset code itself is stored nowhere — it is a hash of the account, its current password and the
+time, valid between five and ten minutes and dead the moment it is used. There is no table to
+maintain and nothing to clean up. Restarting the server invalidates any code already sent.
+
 ### 10. HTTPS and over-the-air firmware updates
 
 **To be written.** Everything above gives you a plain HTTP server on your local network, which is
@@ -726,9 +824,6 @@ It comes in three parts, and only the first is a one-off:
 * **11b — authorizing this Pi on that server.** One command, run there, once per Pi.
 * **11c — pointing this Pi at it.** One command, run here, once per Pi.
 
-If someone has already done 11a for the server you are bridging to — the usual case once the first
-Pi is working — **start at 11b**.
-
 **What you get, and what you do not.** While the link is up, readings appear on production within
 a second or so. While it is down, the Pi records everything as usual and production simply has a
 gap — the readings taken during an outage never reach it. That is a deliberate choice: the
@@ -738,10 +833,19 @@ in the instant the link returned. A gap is honest; that would not be. Controls a
 other way round, and *are* queued, so turning something on from production while the Pi is offline
 takes effect when it reconnects.
 
+**People, as well as readings.** A bridge relays topics, not accounts, so on its own it does not let
+someone registered on production log in here. Step 11b also prints a *replica token*, and step 11c
+asks for it; with it, this Pi pulls that organization's logins and permissions from production every
+fifteen minutes, so the same people can use this Pi's dashboard — including while production is
+unreachable, which is the point of the Pi. It is optional: leave the token blank and the bridge
+relays readings as before. Nothing derived from a password travels either way, and each server
+issues its own broker credentials, so the same person has different ones here and there. Revoking a
+permission on production removes it here on the next pull. See SECURITY.md.
+
 #### 11a. Preparing a production server to accept bridges
 
-Once per production server, not per Pi — and someone has probably already done it for
-`frugaliot.naturalinnovation.org`. If so, skip to 11b.
+Once per production server, not per Pi — and if bridging to `frugaliot.naturalinnovation.org`
+this has already been done. If so, skip to 11b.
 
 **Add a listener for bridges.** A Mosquitto bridge speaks MQTT or MQTT-over-TLS and cannot use
 WebSockets, so the existing `wss://` path that browsers and servers use cannot carry this. It needs
@@ -813,12 +917,16 @@ installed by hand, the written procedure for doing that.
 > Note the asymmetry with the Pi end below: a *certificate* change needs only `reload`, but adding
 > or changing a *bridge* needs a full `restart`, because Mosquitto does not reload bridges.
 
-**Turn on access control.** Without it, any account that can log in to the broker can publish and
-subscribe anywhere on it, so a per-Pi account would be no more confined than the organization's
-own. This is the part to plan carefully, because Mosquitto's ACL file is **deny-by-default**: the
-moment the broker names an `acl_file`, every existing account with no entry in it stops working.
-So the file that restricts new bridge accounts has to grant the existing organizations what they
-already have, in the same change.
+**Turn on access control** — if this broker predates the release that ships it. Since
+`extras/aclfile` became part of the base install, a Pi set up with `install-pi.sh` already has this
+and there is nothing to do here; the steps below are for a broker built before that, or one whose
+configuration was written by hand.
+
+Without it, any account that can log in to the broker can publish and subscribe anywhere on it, so
+a per-Pi account would be no more confined than the organization's own. This is the part to plan
+carefully, because Mosquitto's ACL file is **deny-by-default**: the moment the broker names an
+`acl_file`, every existing account with no entry in it stops working. So the file that restricts new
+bridge accounts has to grant the existing organizations what they already have, in the same change.
 
 ```
 sudo tee /etc/mosquitto/aclfile >/dev/null <<'EOF'
@@ -828,12 +936,21 @@ sudo tee /etc/mosquitto/aclfile >/dev/null <<'EOF'
 # first element of every Frugal IoT topic is the organization id - so this is what the nodes,
 # dashboards and the server's own logger already do. It just stops being optional.
 pattern readwrite %u/#
+
+# Whether a bridge is up, for frugal-iot-diagnostic. "pattern readwrite %u/#" does not match $SYS,
+# so without this the diagnostic reports every bridge as never having connected. Mosquitto logs a
+# warning that this pattern contains no %u - harmless, and a "topic" line would apply to anonymous
+# clients only, which is not what is wanted.
+pattern read $SYS/broker/connection/+/state
 EOF
-sudo chmod 644 /etc/mosquitto/aclfile
+sudo chown mosquitto:mosquitto /etc/mosquitto/aclfile
+sudo chmod 600 /etc/mosquitto/aclfile
 ```
 
-`chmod 644` matters: like the password file and the TLS key, the broker reads this *after* dropping
-to the `mosquitto` user, so a root-only file stops it starting. Then name it in the configuration
+The ownership matters: like the password file and the TLS key, the broker reads this *after*
+dropping to the `mosquitto` user, so a root-only file stops it starting. Mode 600 owned by
+`mosquitto` rather than 644 owned by root — 2.0.21 accepts the latter but warns on every start that
+a file it does not own, or one that is world readable, "will be refused by a future version". Then name it in the configuration
 and restart:
 
 ```
@@ -854,10 +971,12 @@ out, delete `/etc/mosquitto/conf.d/zy-frugal-iot-acl.conf` and restart.
 
 Nothing needs adding here per Pi: `frugal-iot-addbridge-prod` in 11b appends each bridge's own rule.
 
-> One thing this costs: `pattern readwrite %u/#` also denies `$SYS`, so the production broker can no
-> longer tell anyone which bridges are currently connected. The Pi still reports its own bridge
-> state to `frugal-iot-diagnostic`, which is where you would look anyway, so this is a fair trade —
-> but it is why production's dashboard cannot show a site as up or down.
+> `pattern readwrite %u/#` on its own denies `$SYS`, which would stop `frugal-iot-diagnostic`
+> reporting whether a bridge is connected — it reads `$SYS/broker/connection/+/state` as the
+> organization. The second rule above restores exactly that one topic and nothing else. Reading all
+> of `$SYS` would also work and would hand every account the broker's client counts, subscription
+> counts and traffic totals, which on a multi-organization broker tells each organization about the
+> others.
 
 #### 11b. Authorizing this Pi (run on the production server)
 
@@ -894,7 +1013,7 @@ Once per Pi. Run the command 11b printed, from this installation's directory:
 
 ```
 cd ~/frugal-iot
-npx --no frugal-iot-addbridge-pi --org <org-id> --host <prod-host> --account bridge-<site-name>
+npx --no frugal-iot-addbridge-pi <org-id> <prod-host> bridge-<site-name>
 ```
 
 It asks for the password rather than taking it on the command line, so it stays out of your shell
@@ -1060,7 +1179,11 @@ npx --no frugal-iot-clearretained 'myfarm/lotus/+/sht/temperture/#'
 ```
 
 Quote the pattern or the shell will expand it. That lists what is retained and changes nothing;
-add `--delete` to the same command to remove it.
+add the word `delete` at the end of the same command to remove it.
+
+> `delete` is a bare word rather than a `--delete` flag on purpose. npm parses the command line of
+> an `npx` invocation itself and swallows any `--flag` it does not recognise, so `--delete` would
+> never reach the script and it would quietly list instead of deleting.
 
 > Look before deleting. A node's `min`, `max`, `color` and `wired` settings are retained messages
 > too, and they are how the dashboard knows how to draw it — delete those and the node has to be
@@ -1105,10 +1228,19 @@ command for each. That matters most for the files you copy somewhere else — a
 `extras/mosquitto.conf` or `extras/frugaliot.service` does nothing until you install it again:
 
 ```
-sudo cp extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf && sudo systemctl restart mosquitto
-sudo cp extras/frugaliot.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl restart frugaliot
+sudo cp node_modules/frugal-iot-server/extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf
+sudo cp node_modules/frugal-iot-server/extras/aclfile /etc/mosquitto/aclfile
+sudo chown mosquitto:mosquitto /etc/mosquitto/aclfile && sudo chmod 600 /etc/mosquitto/aclfile
+sudo systemctl restart mosquitto
+sudo cp node_modules/frugal-iot-server/extras/frugaliot.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl restart frugaliot
 ```
 
+Note those read from `node_modules/frugal-iot-server/extras/`, **not** from `extras/` in this
+directory. `frugal-iot-init` keeps your local copy and only tells you it differs, so the local one is
+the *old* version — copying it would reinstall what you already have. Re-running the install script
+(`bash node_modules/frugal-iot-server/scripts/install-pi.sh`) does the same job and reads from the
+same place.
+
 > It does not compare `config.yaml`, `config.d/mqtt.yaml`, `config.d/logger.yaml` or
 > `config.d/server.yaml`, because those hold your own settings and would differ every time. If a
 > release note mentions a new setting in one of them, compare it yourself:
@@ -1124,6 +1256,32 @@ do. On a Pi, compare the two files above and copy across the settings you want.
 Your organizations, accounts, database and logged data are untouched by an upgrade — they live in
 this directory, not in `node_modules`.
 
+### One thing to do by hand after upgrading: grant WRITE
+
+Commanding a device through the API (`/devices/action`, `/devices/property`) now requires a **WRITE**
+permission, which is separate from READ and is *not* implied by ADMIN. New organizations get it
+automatically, but an upgrade does not touch your database, so on an existing server nobody has it
+yet and switching a relay on answers 401.
+
+Grant it from the dashboard — **Admin → People**, pick the person, choose `WRITE` — or in one
+statement, to everyone who already administers an organization:
+
+```
+sqlite3 frugal-iot.db "INSERT OR IGNORE INTO permissions (id, capability, org)
+  SELECT id, 'WRITE', org FROM permissions WHERE capability = 'ADMIN';"
+sudo systemctl restart frugaliot
+```
+
+Check who has it:
+
+```
+sqlite3 frugal-iot.db "SELECT id, org FROM permissions WHERE capability = 'WRITE';"
+```
+
+Do **not** grant WRITE to id `0`. That row means "every logged-in account", including anyone who has
+just registered themselves, and handing them write access is exactly what the WRITE check exists to
+prevent. (Read is a different matter — see the `(0, 'READ', ...)` rows, which are there on purpose.)
+
 ---
 
 ## Open questions

package/config.d/email.yaml

@@ -0,0 +1,23 @@
+# Outgoing mail. Only used by the password reset, which is unavailable until this is filled in -
+# the login page says so rather than pretending to have sent something.
+#
+# Leave it commented out on a server that should not send mail.
+#
+# For Gmail - the common case, written out step by step in INSTALLATION.md section 9a:
+#   1. Turn on 2-Step Verification: https://myaccount.google.com/signinoptions/twosv
+#      App passwords do not exist until you do, which is where most people get stuck.
+#   2. Create an app password:      https://myaccount.google.com/apppasswords
+#      You get 16 lowercase letters in four groups of four, shown once. Remove the spaces.
+#   3. "from" must be the same address as "user" - Gmail silently rewrites anything else.
+#
+# host: smtp.gmail.com
+# port: 587           # 465 is implicit TLS, 587 and 25 are STARTTLS; secure: is derived from this
+# user: yourname@gmail.com
+# pass: "abcdefghijklmnop"        # the app password, NOT your Google account password
+# from: Frugal IoT <yourname@gmail.com>
+#
+# Any other provider is the same four settings with its own SMTP host.
+#
+# The reset email contains a link back to this server. Set baseurl if the server cannot work out
+# its own public address from the request - behind a proxy that does not set X-Forwarded-Host, say.
+# baseurl: https://frugaliot.example.org

package/config.d/schema/devices.yaml

@@ -59,5 +59,3 @@ soil:
 sonoff:
   # The relay is what someone wants to see; the control that drives it comes second.
   front:    [relay/on, controlhysteresis]
-temp:
-  front:    [aht20/temperature, bmp280/pressure, controlhysteresis]